Pliant stream names
This article is explaining how to name a Pliant stream to access various available filesystems.
In all this article, we are assuming that your module and current function already contains:
Real operating system file
Accessing operating system '/tmp/test.bin' file is:
s open "file:/tmp/test.bin" out+safe
Please notice that under Windows, the accessed file will be operating system file '\tmp\test.bin'. In other words, if you are running Windows, you should use slash in your Pliant code, and Pliant will transparently translate it to Windows anti-slash notation.
When using 'file:', any name containing '/../' will be rejected. The reason is that this feature is not security wise. If you really want to allow '/../' in file names, use 'os_file:' instead of 'file:'
s open "os_file:/tmp/../pliant_security/this_computer.pdb" append
Under Unix operating system, accessing a device such as /dev/urandom can be:
s open "device:/urandom" in+safe
A name starting with a slash will be assumed to be the name of file in the Pliant source tree, so assuming that you unpacked Pliant tree in operating system root directory, then the two following are equivalent:
s open "/my_org/my_file.pli" in+safe
s open "file:/pliant/my_org/my_file.pli" in+safe
Then, a name starting with 'security:/' will be assumed to be the name of a file in the Pliant configuration directory, so the two following are equivalent:
s open "security:/my_org/my_file.pdb" in+safe
s open "file:/pliant_security/my_org/my_file.pdb" in+safe
Finally, a name starting with 'data:/' will be assumed to be the name of a file in the Pliant data directory, so the two following are equivalent:
s open "data:/my_org/my_file.bin" out+safe
s open "file:/pliant_data/my_org/my_file.bin" out+safe
TCP network connections
In order to open TCP port 8080, and wait for a client, just use:
s open "tcp:/server/8080" in+out+safe
In order to open to connect to TCP port 8080 on computer 'foo.fullpliant.org', just use:
s open "tcp://foo.fullpliant.org/client/8080" in+out+safe
When opening a Pliant connection, it is possible to specify a maximum connection time in seconds (the TCP default timeout is quite long):
s open "tcp://foo.fullpliant.org/client/8080" "timeout 15" in+out+safe
The most frequently used query is discover the IP address of the remote side of the connection:
var Str ip := s query "remote_ip_address"
It is possible to adjust the timeout (in seconds) at any time using:
s configure "timeout 120"
There are a lot of other query and configuration options that are less frequently used. Please read the 'query' and 'configure' methods in /pliant/language/stream/tcp.pli since it's fairly simple.
A server side TCP connection uses two operating system connections
In the article explaining how to use Pliant stream, I stated that Pliant streams are not thread safe, but that it is possible to have one thread reading and one thread writing with no special protection, provided the stream has been open with 'noautopost' flag set.
s configure "shutdown"
and the result will be force read and write operations attempted by the other thread to fail. The application might have been writen so that when the thread notices that the read or write operations fail, it exists the handling loop, release some semaphore to specify that it's not dealing anymore with the stream, so that the other thread can safely close it.
In Pliant, UDP interface is very similar to TCP one.
s open "udp://foo.fullpliant.org/client/53" in+out+safe
If you expect an answer from the remote, you should define a timeout:
s configure "timeout 15"
See 'query_provider' in /pliant/protocol/dns/common.pli for a full sample.
A loopback connection is a bit like a TCP connection, but within a single Pliant process. The advantage is it cannot be accessed remotely or by another process on the same computer, so is secured.
s open "loopback:/client/abc" in+out+safe
On a loopback connection, if you do:
var Str ip := s query "remote_ip_address"
the 'ip' variable will receive 'loopback' value instead of a dotted IP address.
s configure "shutdown"
is also available on loopack connections in order to enable dual threading.
Setting a read timeout is also supported:
s configure "timeout 15"
also it is currently ignored when writing.
Pliant secured channels provide secured connections that are expected to protect from man in middle listening or content modification.
Wait for a secured connection on TCP port 36:
'zpml:' means Zlib compression + RC4 cyphering whereas 'pml:' would mean cyphering only.
Securely connect to remote site:
s open "zpml:www.fullpliant.org" "service [dq]ui[dq] from_class [dq]user[dq] from_id [dq]me[dq] password [dq]my_key_password[dq] to_class [dq]site[dq]" in+out+safe
Supported classes are 'host' 'site' and 'user'. They are used to decide from which table to pick the keys corresponding to the provided ID. The 'to_id' parameter is provided just after 'pml:' or 'zpml:' instead of being provided in the options.
'service' option (if present) will tel the server what service to connect.
When the connection is established, any side can query the (local and) remote class and ID to check remote site identity:
var Str v
Once again, in case of troubles, use Pliant tracing mechanism to see what's going wrong.
Unix pipes and direct using operating system file handles
A Unix stream pipe is created through 'stream_pipe' function that returns the a name for each side. Each of them can used as the name of a stream, or passed to 'execute' to redirect the standard input or output.
Please study /pliant/appli/execute.ui applet for a usage example,
If you read the module code implementing Unix pipes, you will notice that the returned pipes names are 'handle:xxx'. Using an 'handle:xxx' name enables to use any operating system file handle for the stream input or output. As an example, the following code:
is equivalent to:
console "hello" eol
because operating system file handle 1 is always the standard output.
Accessing a remote FTP file (could be either 'in' or 'out'):
FTP client is implementing some extra filesystem functions, so you can also do:
'file_tree_create', 'file_tree_delete' and 'file_move' are also expected to work.
Also the problem is that FTP files directory listing report is not standardized in the RFC, so you are not granted that Pliant will guess correctly the listing format on the remote server.
Please also keep in mind that the right way to debug an FTP client connection is to use Pliant trace mechanism ('Configure' 'Trace' and 'FTP client').
Accessing a remote HTTP file (could be either 'in' or 'out'):
TCP port to use can be specified for both HTTP and FTP:
s open "http://foo.fullpliant.org:8080/my_directory/my_subdirectory/my_file.bin" in+safe
As opposed to FTP, with HTTP, only 'file_delete' is supported.
At any time, you can get the MIME type associated with the file through:
var Str mime := s query "mime"
Some features such as logging need to be added.
SMB is the network protocol used to share files under Windows.
Please notice that with SMB protocol, a server exports not a single true, but a set of subtrees, each associated with a name, and this name ('export_name' in the previous sample) has to be provided between the server name and the file path in the exported subtree.
Just like with FTP, all basic file operations are expected to work.
A second wrapper is provided in module '/pliant/protocol/smb/client2.pli' that use a different set of functions in 'libsmbclient'. Did not work any better for me. Make your own tests.
SMTP is the protocol for sending mails, described in RFC 821.
If 'server' option is not provided, it will be determined automatically using a DNS query on the mail target domain. There are two possible strategies for sending mails: either you provide no 'server' option so that Pliant SMTP client will try to post the mail directly in the recipient mailbox, or you specify a local or your Internet service provider SMTP server so that it will store then forward the mail. With the second solution, the SMTP server will do retry if the target mailbox SMTP server cannot be reached, but with the first solution, 'open' or 'close' method will return an error status and the application is responsible for resending at a later time. The spams invasion on the Internet led many mail servers to use gray lists that force to resend, so that the second strategy is now close to mandatory unless you are prepared to implement resend at application level.
When the mail has several recipents, you shoud use the options paramater:
s open "smtp:" "from [dq]my_maibox@my_domain.org[dq] to [dq]firstname.lastname@example.org[dq] cc [dq]email@example.com[dq] cc [dq]firstname.lastname@example.org[dq] subject [dq]Some test mail for you[dq]" out+safe
Known bug: I have to do encoding on the mail content in order to allow lines with only a dot sign (a line with a single dot sign is the standard way to end a mail in SMTP protocol).
POP3 is a protocol for fetching mails, described in RFC 1939 :
Alternative syntax is:
s open "pop3://foo.fullpliant.org" "user [dq]my_account[dq] password[dq]my_password[dq]" in+safe
If you want to read the mail content, but not remove it from the server, use 'keep' command:
s configure "keep"
LPR is a protocol for print job spooling, described in RFC 1179.
The file name is the name of the LPR queue on the server. On most servers, 'lp' is the name of default quere.
Several options can be used to provide a more complete job ticket set of informations:
s open "lpr://foo.fullpliant.org/lp/report.pdf" "title [dq]My report[dq] user [dq]me[dq] timeout 120" out+safe
where 'report.pdf' will be the file name of the file in the job ticket queue.
'lpng:' can be used instead of 'lpr:' with servers supporting zero file size as a convension for send files with unkown size. If not used, the file will be transparently stored in a temporary file before being sent.
the number provided in the name is the serial port number.
Timeout is supported on serial ports:
s configure "timeout 15"
A Zlib compressed stream can have three different effective encoding.
The compressed content, with a header, and tail CRC and size fields, as described in RFC 1952:
s open "gzip:file:/tmp/test.gz" "level 9 comment [dq]My comment[dq]" out+safe
The compressed content, with a header and a 32 bits CRC at end, as described in RFC 1950:
s open "zlib:file/tmp/test.zlib" out+safe
The compressed content, with nothing else, as described in RFC 1951:
s open "deflate:file:/tmp/test.deflate" out+safe
Of course, the compression 'level' option can be used with any of the three. It's value ranges from 1 (fastest) to 9 (best compression ration).
It is sometime difficult to know which one is really used because many documentation just call them all 'zlib'. The general rule is:
Please notice that, as explained in the 'Advanced opening' section of the article explaining Pliant streams usage, you can also open a Zlib compression stream through passing the support stream as a parameter:
Pliant enables to easily scan a ZIP tarball:
(var Stream raw) open "file:/tmp/test.zip" in+safe
In other words, a ZIP tarball is a set of compressed files one after the other. Each time you open with 'zip:', the next to come file header is parsed, so that you can get the name, date and size using 'query' method, then read the file content using standard reading commands. When closing, the file tail is consumed so that the file cursor is just at the beginning of the next file header.
There is no Pliant function to create a ZIP tarball.
Other transparent decoding
These decoding filters are documented here in order to avoid a restart from scratch if somebody need one of them, but they are not used much, so code quality has to be verified.
ASCII85 is one of the many encodings used in PDF files (a bit like base64).
CCITT is mainly used to compress 1 bit per pixel black and white images. It's an Huffman encoding with fixed dictionary. Was very popular because fast, probably easy to implement in very limited hardware, and used by facsimile machines and TIFF file format.
When decoding TIFF images, Pliant uses a libtiff wrapper, so effective decoding is handled by libtiff in facts.
Postscript Type1 fonts encoding
PFB is an encoding introduced by Adobe at a time it wanted Postscript type1 fonts to be secret:
Chunking has been introduced in HTTP protocol to enable keep alive connections and proper error report on dynamically built answers (the size of the answer is not known at the beginning of the answer). Pliant supports both encoding and decoding:
If 'pml' option is used, the chunking will use Pliant PML encapsulation instead of HTTP one.
Adding seek capability through a temporary file
When reading a file through a network connection, or using a compression filter, seeking might not be available. Using a temporary stream filter will enable seeking through copying the content to a temporary file before reading, or storing the content in a temporary file when writing.
Counting, limiting size and speed
Counting bytes transmited on a stream:
Using 'size' option forces 'atend' method to return true after specified number of bytes have been red:
s open "count:file:/tmp/test.bin" "size 1000" in
Limiting transmission speed (speed is specified in characters per second):
s open "count:tcp://foo.fullpliant.org/client/8080" "read_cps 500000 write_cps 80000" in+out+safe
Speed limit is not applied accurately at all: basically, the calling program will be suspend enough time after each buffer read or write in order to grant the specified speed limit not to be exceeded on average. If the program takes time between reading and writing instructions to perform other things, this time will just be ignored so that the effective transmission speed will be lower than the specified one in the end.
A filter is provided that enable to read a file, and instead of stopping as soon as the file end is reached, rather wait for the file content to extend. It is used by Pliant print spooler to enable sending the job content to the printer while receiving if over the network.
It is possible to write the stream content to several places at once:
The multiplexing code is not used, so poorly tested at the moment.
A null stream just drops the content writen to, and returns zeros bytes when red from.
This one does basic PML encapsulation. This is used to provide multiplexing (not that important) and tagging (user and timestamp, very important) in a Pliant storage file.