Pliant stream names

This article is explaining how to name a Pliant stream to access various available filesystems.
The API to use Pliant streams is documented in another article.

In all this article, we are assuming that your module and current function already contains:

module "/pliant/language/stream.pli"
var Stream s

Real operating system file

Accessing operating system '/tmp/test.bin' file is:

s open "file:/tmp/test.bin" out+safe

Please notice that under Windows, the accessed file will be operating system file '\tmp\test.bin'. In other words, if you are running Windows, you should use slash in your Pliant code, and Pliant will transparently translate it to Windows anti-slash notation.

When using 'file:', any name containing '/../' will be rejected. The reason is that this feature is not security wise. If you really want to allow '/../' in file names, use 'os_file:' instead of 'file:'

s open "os_file:/tmp/../pliant_security/this_computer.pdb" append

Under Unix operating system, accessing a device such as /dev/urandom can be:

s open "device:/urandom" in+safe

A name starting with a slash will be assumed to be the name of file in the Pliant source tree, so assuming that you unpacked Pliant tree in operating system root directory, then the two following are equivalent:

s open "/my_org/my_file.pli" in+safe

and

s open "file:/pliant/my_org/my_file.pli" in+safe

Then, a name starting with 'security:/' will be assumed to be the name of a file in the Pliant configuration directory, so the two following are equivalent:

s open "security:/my_org/my_file.pdb" in+safe

and

s open "file:/pliant_security/my_org/my_file.pdb" in+safe

Finally, a name starting with 'data:/' will be assumed to be the name of a file in the Pliant data directory, so the two following are equivalent:

s open "data:/my_org/my_file.bin" out+safe

and

s open "file:/pliant_data/my_org/my_file.bin" out+safe

TCP network connections

Server

In order to open TCP port 8080, and wait for a client, just use:

s open "tcp:/server/8080" in+out+safe

Client

In order to open to connect to TCP port 8080 on computer 'foo.fullpliant.org', just use:

s open "tcp://foo.fullpliant.org/client/8080" in+out+safe

When opening a Pliant connection, it is possible to specify a maximum connection time in seconds (the TCP default timeout is quite long):

s open "tcp://foo.fullpliant.org/client/8080" "timeout 15" in+out+safe

Querying

The most frequently used query is discover the IP address of the remote side of the connection:

var Str ip := s query "remote_ip_address"

Configuring

It is possible to adjust the timeout (in seconds) at any time using:

s configure "timeout 120"

There are a lot of other query and configuration options that are less frequently used. Please read the 'query' and 'configure' methods in /pliant/language/stream/tcp.pli since it's fairly simple.

Connection socket

A server side TCP connection uses two operating system connections

Dual threading

In the article explaining how to use Pliant stream, I stated that Pliant streams are not thread safe, but that it is possible to have one thread reading and one thread writing with no special protection, provided the stream has been open with 'noautopost' flag set.
If one of the two threads want's to close the stream, it must not call 'close' method while the other is still reading or writing, because 'close' method is not thread safe.
On the other hand, it can do:

s configure "shutdown"

and the result will be force read and write operations attempted by the other thread to fail. The application might have been writen so that when the thread notices that the read or write operations fail, it exists the handling loop, release some semaphore to specify that it's not dealing anymore with the stream, so that the other thread can safely close it.

UDP packets

In Pliant, UDP interface is very similar to TCP one.
Also, only a single packet, or one packet in each direction will be exchanged because UDP is session less.
In other words, either you open a server 'pseudo connection', read the packet content from it, then optionally write the answer, and finally close,
or you open a client 'pseudo connection', write the packet content to it, then optionally read the answer, and finally close.

module "/pliant/language/stream/udp.pli"
s open "udp:/server/53" in+out+safe

s open "udp://foo.fullpliant.org/client/53" in+out+safe

If you expect an answer from the remote, you should define a timeout:

s configure "timeout 15"

See 'query_provider' in /pliant/protocol/dns/common.pli for a full sample.

Loopback

A loopback connection is a bit like a TCP connection, but within a single Pliant process. The advantage is it cannot be accessed remotely or by another process on the same computer, so is secured.
The two sides connect together through providing the same string, 'abc' is the sample bellow, so the string shall be seen as a TCP port number equivalent.

module "/pliant/language/stream/loopback.pli"
s open "loopback:/server/abc" in+out+safe

s open "loopback:/client/abc" in+out+safe

On a loopback connection, if you do:

var Str ip := s query "remote_ip_address"

the 'ip' variable will receive 'loopback' value instead of a dotted IP address.

s configure "shutdown"

is also available on loopack connections in order to enable dual threading.

Setting a read timeout is also supported:

s configure "timeout 15"

also it is currently ignored when writing.

Secured connections

Pliant secured channels provide secured connections that are expected to protect from man in middle listening or content modification.
They are based on RSA to exchange a shared key from the public/private key pair, then RC4 to cipher the content. No secured mechanism is provided at the moment to securely exchange public keys over the network. Moreover, the code can hardly be trusted because not enough reviewed.

Server

Wait for a secured connection on TCP port 36:

module "/pliant/util/pml/channel.pli"
s open "zpml:36" in+out+safe

'zpml:' means Zlib compression + RC4 cyphering whereas 'pml:' would mean cyphering only.

Client

Securely connect to remote site:

s open "zpml:www.fullpliant.org" "service [dq]ui[dq] from_class [dq]user[dq] from_id [dq]me[dq] password [dq]my_key_password[dq] to_class [dq]site[dq]" in+out+safe

Supported classes are 'host' 'site' and 'user'. They are used to decide from which table to pick the keys corresponding to the provided ID. The 'to_id' parameter is provided just after 'pml:' or 'zpml:' instead of being provided in the options.

'service' option (if present) will tel the server what service to connect.

When the connection is established, any side can query the (local and) remote class and ID to check remote site identity:

var Str v
v := s query "local_class"
v := s query "local_id"
v := s query "remote_class"
v := s query "remote_id"

Once again, in case of troubles, use Pliant tracing mechanism to see what's going wrong.

Unix pipes and direct using operating system file handles

A Unix stream pipe is created through 'stream_pipe' function that returns the a name for each side. Each of them can used as the name of a stream, or passed to 'execute' to redirect the standard input or output.

module "/pliant/language/stream/pipe.pli"
if (stream_pipe (var Str in_name) (var Str out_name))=failure
  console "Failed to create the pipe !!!" eol
...
(var Stream s1) open in_name in+safe

Please study /pliant/appli/execute.ui applet for a usage example,
and also notice that Unix pipes are not available when running Pliant on Windows.

If you read the module code implementing Unix pipes, you will notice that the returned pipes names are 'handle:xxx'. Using an 'handle:xxx' name enables to use any operating system file handle for the stream input or output. As an example, the following code:

module "/pliant/language/stream/pipe.pli"
(var Stream s) open "handle:1" out+safe
s writeline "hello"

is equivalent to:

console "hello" eol

because operating system file handle 1 is always the standard output.

FTP

Accessing a remote FTP file (could be either 'in' or 'out'):

module "/pliant/protocol/ftp/client.pli"
s open "ftp://foo.fullpliant.org/my_directory/my_subdirectory/my_file.bin" "user [dq]my_account[dq] password[dq]my_password[dq]" in+safe

FTP client is implementing some extra filesystem functions, so you can also do:

module "/pliant/protocol/ftp/client.pli"
module "/pliant/admin/file.pli"
var Array:FileInfo := file_list "[dq]ftp://foo.fullpliant.org/my_directory/my_subdirectory/[dq] "user [dq]my_account[dq] password[dq]my_password[dq]"

or

module "/pliant/protocol/ftp/client.pli"
module "/pliant/admin/file.pli"
file_delete "[dq]ftp://foo.fullpliant.org/my_directory/my_subdirectory/my_file.bin[dq] "user [dq]my_account[dq] password[dq]my_password[dq]"

'file_tree_create', 'file_tree_delete' and 'file_move' are also expected to work.

Also the problem is that FTP files directory listing report is not standardized in the RFC, so you are not granted that Pliant will guess correctly the listing format on the remote server.
In such a case, adjusting the code in 'list' method in /pliant/protocol/ftp/client.pli should not be too hard.

Please also keep in mind that the right way to debug an FTP client connection is to use Pliant trace mechanism ('Configure' 'Trace' and 'FTP client').

HTTP

Accessing a remote HTTP file (could be either 'in' or 'out'):

module "/pliant/protocol/http/client.pli"
s open "http://foo.fullpliant.org/my_directory/my_subdirectory/my_file.bin" in+safe

TCP port to use can be specified for both HTTP and FTP:

s open "http://foo.fullpliant.org:8080/my_directory/my_subdirectory/my_file.bin" in+safe

As opposed to FTP, with HTTP, only 'file_delete' is supported.
Anyway, 'file_query' should enable to report the file date, and sometime the file size.

At any time, you can get the MIME type associated with the file through:

var Str mime := s query "mime"

Some features such as logging need to be added.

SMB

SMB is the network protocol used to share files under Windows.
Two wrappers are provided to enable reading and writing files from an SMB server without mounting the remote server at operating system level.
The problem is that 'libsmbclient' was far from robust when I tried using it for production.

module "/pliant/protocol/smb/client.pli"
s open "smb://foo.fullpliant.org/export_name/my_directory/my_subdirectory/my_file.bin" in+safe

Please notice that with SMB protocol, a server exports not a single true, but a set of subtrees, each associated with a name, and this name ('export_name' in the previous sample) has to be provided between the server name and the file path in the exported subtree.

Just like with FTP, all basic file operations are expected to work.

A second wrapper is provided in module '/pliant/protocol/smb/client2.pli' that use a different set of functions in 'libsmbclient'. Did not work any better for me. Make your own tests.

SMTP

SMTP is the protocol for sending mails, described in RFC 821.

module "/pliant/protocol/smtp/client.pli"
s open "smtp:somebody@fullpliant.org" "from [dq]my_mailbox@my_domain.org[dq] subject [dq]Some test mail for you[dq] server [dq]foo.fullpliant.org alternate.fullpliant.org[dq] timeout 120" out+safe

If 'server' option is not provided, it will be determined automatically using a DNS query on the mail target domain. There are two possible strategies for sending mails: either you provide no 'server' option so that Pliant SMTP client will try to post the mail directly in the recipient mailbox, or you specify a local or your Internet service provider SMTP server so that it will store then forward the mail. With the second solution, the SMTP server will do retry if the target mailbox SMTP server cannot be reached, but with the first solution, 'open' or 'close' method will return an error status and the application is responsible for resending at a later time. The spams invasion on the Internet led many mail servers to use gray lists that force to resend, so that the second strategy is now close to mandatory unless you are prepared to implement resend at application level.

When the mail has several recipents, you shoud use the options paramater:

s open "smtp:" "from [dq]my_maibox@my_domain.org[dq] to [dq]recipient1@domain1.org[dq] cc [dq]recipient2@domain2.org[dq] cc [dq]recipient3@domain3.org[dq] subject [dq]Some test mail for you[dq]" out+safe

Known bug: I have to do encoding on the mail content in order to allow lines with only a dot sign (a line with a single dot sign is the standard way to end a mail in SMTP protocol).

POP3

POP3 is a protocol for fetching mails, described in RFC 1939 :

module "/pliant/protocol/pop3/client.pli"
s open "pop3://foo.fullpliant.org/my_account/my_password" in+safe

Alternative syntax is:

s open "pop3://foo.fullpliant.org" "user [dq]my_account[dq] password[dq]my_password[dq]" in+safe

If you want to read the mail content, but not remove it from the server, use 'keep' command:

s configure "keep"

LPR

LPR is a protocol for print job spooling, described in RFC 1179.

module "/pliant/protocol/lpr/client.pli"
s open "lpr://foo.fullpliant.org/lp" out+safe

The file name is the name of the LPR queue on the server. On most servers, 'lp' is the name of default quere.

Several options can be used to provide a more complete job ticket set of informations:

s open "lpr://foo.fullpliant.org/lp/report.pdf" "title [dq]My report[dq] user [dq]me[dq] timeout 120" out+safe

where 'report.pdf' will be the file name of the file in the job ticket queue.

'lpng:' can be used instead of 'lpr:' with servers supporting zero file size as a convension for send files with unkown size. If not used, the file will be transparently stored in a temporary file before being sent.

Serial port

module "/pliant/language/stream/serial.pli"
s open "serial:0" "speed 9600 databits 7 parity [dq]odd[dq] stopbits 2 flowcontrol [dq]hardware[dq]" in+out+safe

the number provided in the name is the serial port number.
Possible values for the flow control are 'none' 'software' 'hardware' and 'both'.

Timeout is supported on serial ports:

s configure "timeout 15"

Zlib compression

A Zlib compressed stream can have three different effective encoding.

The compressed content, with a header, and tail CRC and size fields, as described in RFC 1952:

s open "gzip:file:/tmp/test.gz" "level 9 comment [dq]My comment[dq]" out+safe

The compressed content, with a header and a 32 bits CRC at end, as described in RFC 1950:

s open "zlib:file/tmp/test.zlib" out+safe

The compressed content, with nothing else, as described in RFC 1951:

s open "deflate:file:/tmp/test.deflate" out+safe

Of course, the compression 'level' option can be used with any of the three. It's value ranges from 1 (fastest) to 9 (best compression ration).

It is sometime difficult to know which one is really used because many documentation just call them all 'zlib'. The general rule is:
'gzip' is only used in files with .gz extension
'deflate' is mostly used by web browsers
'zlib' is used by most applications

Please notice that, as explained in the 'Advanced opening' section of the article explaining Pliant streams usage, you can also open a Zlib compression stream through passing the support stream as a parameter:

module "/pliant/language/stream/filesystembase.pli"
module "/pliant/language/stream/openmode.pli"
module "/pliant/language/stream.pli"
module "/pliant/language/stream/multi.pli"
ovar Stream support
var Stream s
...
var ExtendedStatus status := s open "zlib:" "level 1" out+safe pliant_default_file_system support

ZIP tarball

Pliant enables to easily scan a ZIP tarball:

(var Stream raw) open "file:/tmp/test.zip" in+safe
while { var ExtendedStatus s := (var Stream zip) open "zip:" "" in+safe pliant_default_file_system raw ; s=success }
  console "name " (zip query "name") "   "
  console "date " (zip query "datetime") "   "
  console "size " (zip query "size") eol
  while { zip read_available (var Address adr) (var Int size) ; size>0 }
    void
  zip close
raw close

In other words, a ZIP tarball is a set of compressed files one after the other. Each time you open with 'zip:', the next to come file header is parsed, so that you can get the name, date and size using 'query' method, then read the file content using standard reading commands. When closing, the file tail is consumed so that the file cursor is just at the beginning of the next file header.

There is no Pliant function to create a ZIP tarball.

Other transparent decoding

These decoding filters are documented here in order to avoid a restart from scratch if somebody need one of them, but they are not used much, so code quality has to be verified.

ASCII85

ASCII85 is one of the many encodings used in PDF files (a bit like base64).

module "/pliant/graphic/vfilter/ascii85.pli"
s open "ascii85:file:/tmp/file.bin" in+safe

CCITT

CCITT is mainly used to compress 1 bit per pixel black and white images. It's an Huffman encoding with fixed dictionary. Was very popular because fast, probably easy to implement in very limited hardware, and used by facsimile machines and TIFF file format.
I just don't remember exactly which instance of it is implemented here:

module "/pliant/graphic/vfilter/ccitt.pli"
s open "ccitt:file:/tmp/file.bin" in+safe

When decoding TIFF images, Pliant uses a libtiff wrapper, so effective decoding is handled by libtiff in facts.

Postscript Type1 fonts encoding

PFB is an encoding introduced by Adobe at a time it wanted Postscript type1 fonts to be secret:

module "/pliant/graphic/vector/font.pli"
s open "pfb:file:/tmp/font.pfb" in+safe

Chunking encapsulation

Chunking has been introduced in HTTP protocol to enable keep alive connections and proper error report on dynamically built answers (the size of the answer is not known at the beginning of the answer). Pliant supports both encoding and decoding:

module "/pliant/protocol/http/chunked.pli"
s open "chunked:file:/tmp/file.bin" out+safe

If 'pml' option is used, the chunking will use Pliant PML encapsulation instead of HTTP one.
Chunking also provides 'read_counter' and 'write_counter' query support just like 'count:' stream filter described bellow.

Adding seek capability through a temporary file

When reading a file through a network connection, or using a compression filter, seeking might not be available. Using a temporary stream filter will enable seeking through copying the content to a temporary file before reading, or storing the content in a temporary file when writing.

module "/pliant/language/stream/tmp.pli"
s open "tmp:gzip:/some_directory/some_file.gz" in+sage
s configure "seek 1000"

Counting, limiting size and speed

Counting bytes transmited on a stream:

module "/pliant/language/stream/count.pli"
s open "count:tcp://foo.fullpliant.org/client/8080" in+out+safe
...
console "red " (s query "read_counter") " bytes" eol
console "wrote " (s query "write_counter") " bytes" eol

Using 'size' option forces 'atend' method to return true after specified number of bytes have been red:

s open "count:file:/tmp/test.bin" "size 1000" in

Limiting transmission speed (speed is specified in characters per second):

s open "count:tcp://foo.fullpliant.org/client/8080" "read_cps 500000 write_cps 80000" in+out+safe

Speed limit is not applied accurately at all: basically, the calling program will be suspend enough time after each buffer read or write in order to grant the specified speed limit not to be exceeded on average. If the program takes time between reading and writing instructions to perform other things, this time will just be ignored so that the effective transmission speed will be lower than the specified one in the end.

Queuing

A filter is provided that enable to read a file, and instead of stopping as soon as the file end is reached, rather wait for the file content to extend. It is used by Pliant print spooler to enable sending the job content to the printer while receiving if over the network.
Wait for more data will stop as soon as the provided (through it's database path) database variable value is no more equal to the provided value.

module "/pliant/language/stream/flow.pli"
var Data:Str v
s open "flow:file:/tmp/some_file.bin" "flag_path "+(string pathof:v)+" flag_value "+string:v in+safe

Multiplexing

It is possible to write the stream content to several places at once:

module "/pliant/language/stream/flow.pli"
s open "multiplexer:[dq]file:/tmp/some_file.bin[dq] [dq]tcp://foo.fullpliant.org/client/3510[dq]" out+safe

The multiplexing code is not used, so poorly tested at the moment.

Null

A null stream just drops the content writen to, and returns zeros bytes when red from.

module "/pliant/language/stream/null.pli"
s open "null:" in+out+safe

Storage

This one does basic PML encapsulation. This is used to provide multiplexing (not that important) and tagging (user and timestamp, very important) in a Pliant storage file.

module "/pliant/storage/ground/filesystem.pli"
s open "storage:file:/tmp/log" "fiber [dq]abc[dq]" append+safe


thanks

Comment posted on 2009/01/08 16:52:41 by boris

Very useful article, thanks a lot!