How to write a transformation (channel)

Whenever a channel is created the core will not only get a reference to the structure containing the references to the driver procedures, i.e. the channel type, but a reference to a structure allocated by the caller as well. This reference is given to all driver procedures when called for that particular channel. The internals of this structure are known only to the channel driver; the core will just pass the reference around. This allows us to associate the specific state of the driver with the channel. In the case of a transformation the following information should be present, not necessarily under the name I gave them, you are free to choose your own identifiers: is a backlink from the instanceData to the channel. Without this link the transformation will be unable to access the downstream channel, which really makes no sense for it. Please note that due to the guarantee in this reference will always point to the top of the stack and not necessarily to the channel structure referring to this instanceData. This means that anytime the driver wants to access the downstream channel of the transformation it has to walk the chain of channels to find its own channel. Not doing this will cause an infinite recursion later on. This is best done in a utility function employing 'Tcl_GetStackedChannel'. See for example 'generic/giot.c:DownChannel'. This timer is necessary if the transformation is able to buffer processed data the generic I/O layer has no knowledge of. It will be used to flush out such data. See for more explanations. The transformation has to remember the current blocking mode to handle EOF on input right. See for more explanations. The transformation has to remember the current interest in events; see for more explanations.

As with every channel a transformation has to have a creation command at the tcl level and an equivalent procedure at the C-level. In contrast to other channel types this procedure must not use 'Tcl_CreateChannel' as that would create a new and separate channel, but 'Tcl_StackChannel' instead. This procedure takes as one of its parameters a reference to an existing channel and creates a new channel structure which is used to save the current state of the channel we are stacking the transformation onto. The old structure is then reinitialized for the transformation. The returned value refers to the saved state of the old channel. The advantage of this design is that all references to the channel we are stacking onto stay valid without opening backdoors into the whole stack. Other things, like the creation and initialization of the necessary clientData for the transformation, have to be done as usual. Please note that the backlink to the channel of the transformation must be initialized before the call to 'Tcl_StackChannel' as it is required for the WatchProc implementation and that one can be called from the inside of 'Tcl_StackChannel'. This restriction is no problem, as we know from that the channel structure of the transformation is the same as the old channel.

Creation procedure skeleton channel = old_channel; save_area = Tcl_StackChannel (interp, &trans_type, clientData, old_channel); interp->SetResult (old_channel->name); return TCL_OK; } ]]> The 'trans_type' in the code above is the structure containing the references to the driver procedures for the transformation.

The two ways of destroying a transformation are 'Tcl_CloseChannel' and 'Tcl_UnstackChannel'. The difference between them ? 'Tcl_CloseChannel' will not only destroy the transformation, but the underlying channel too, an action which then ripples through the whole stack until all channels in it are destroyed. 'Tcl_UnstackChannel' on the other hand will only destroy the topmost transformation in the stack. Of course, if the stack consisted only of one element both procedures are equivalent. As 'Tcl_CloseChannel' knows that the whole stack of channels is in destruction it does not deal with events anymore, except for destroying the internal data structures supposed to deal with them. But it does ask the various channels in the stack to flush buffered information down the stack so that nothing which is stuck is lost. This is not possible for information in the upward/read buffers, as there is no ultimate receiver for them, so these bytes are lost. 'Tcl_UnstackChannel' does the same as above, but also takes action to keep the event-system up and running (by redirecting existing scripts to the uncovered channel). In all cases the system will call the 'CloseProc' of the transformation to cleanup the data structures of the destroyed channels. A procedure to make 'Tcl_UnstackChannel' available at the script level is rather simple:

Unstack skeleton

It was said in that the transformation has to walk the stack to find its own channel structure, and that of the channel downstream, but not how this is done. The functionality for this is provided by the third new function in the C-API, 'Tcl_GetStackedChannel'. It takes a channel token and returns a token for the channel immediately downstream. A (Tcl_Channel) NULL indicates that the channel used as argument was at the bottom of the stack.

Now that the environment of the driver is a little more known we can explain the operations of the various driver procedures in detail. Every description will start with the general condition under which the procedure is called by the generic I/O layer of the tcl core and proceeds to the specialities a transformation has to take care of.

This procedure is called by the C-API function 'Tcl_GetChannelHandle' to retrieve the OS specific file handle associated to the queried channel. Transformations are not associated with OS specific handles, making this call a bit meaningless. So one possible implementation simply returns a NULL handle (erroring out is not possible). On the other hand the downstream channel could be OS specific and have such a handle. So the other implementation is to delegate that request to the underlying channel, via 'Tcl_GetChannelHandle'. If all transformations cooperate in this manner, the request will fall down to the channel at the bottom, like a file or socket, which is able to handle it and return something meaningful. To me the second implementation is the better policy of the two described here.

This procedure is called by the generic I/O layer whenever 'Tcl_SetChannelOption' is used (f.e. by 'fconfigure') and a non-standard option was specified as argument. The interesting case here is how to handle options unknown to the transformation. One could discard them, or delegate the call to the downstream channel. I recommend delegating the call.

SetOptionProc skeleton channel->supercedes, optionName, value); } ]]>

This procedure is called by the generic I/O layer whenever 'Tcl_GetChannelOption' is used (f.e. by 'fconfigure') to query the value of a non-standard (or all) option(s). Implement everything from the , probably more (a read-only option makes sense, write-only doesn't). Again it is possible to delegate unknown options to the underlying channel. In the case of a query for all options, delegation will generate a mighty long result. Pruning the unnecessary options values from the result of the underlying channel (-encoding, -buffering, -translation) is possible, but tedious (We have a DString, not Tcl_Obj'ects, especially no ListObj'ects). The named options have fixed and known values, due to the behaviour of 'Tcl_StackChannel' as described in .

GetOptionProc skeleton channel->supercedes optionName, dsPtr); } ]]>

This procedure is called by the generic I/O layer whenever the user asks the channel to move the 'file access point'. This is a hard problem for transformations. Should they seek using their own notion of access point ? Or should they use the notion of the underlying channel and then try to adapt their own state for fine-positioning? Depending on the transformation both can be impossible. A nice example are compressors (like zlib), with their completely non-linear and position-dependent relationship between the number of bytes coming in from the downstream channel and going out to its caller. Another reason could be that the transformation state is not reversible, i.e. cannot be rolled back in a simple way, without hogging memory. An example for this would be an encryption transformation using a cryptographically strong hash-function to go from the current state to the state for the encryption of the next byte (or block). This is not reversible. We can go forward from state to state, but not back to the old state, except for saving them all. Because of this my personal policy to deal with seeking is to propagate the request unchanged to the underlying channel and to discard all information in the internal buffers of the transformation. Data waiting to be written is converted as if they are the last block, in other words the special end of information processing is applied, and then flushed. The current state is abandoned too. The next call to InputProc or OutputProc will be handled as if it were the first call to the transformation. This is basically a 'The user knows best, is able to compute a place making sense and not creating garbage during recover'. In the end this means that the user of a certain transformation has to understand its properties and whether a seek on it makes sense at all. Oh, it is possible to deal even with non-reversible state, by recording all read/write calls and maintaining an exact image of the information read/written so far, but this is, ah, memory-extensive, to understate this a little.

SeekProc skeleton channel->supercedes, offset, mode ); *errorCodePtr = (result == -1) ? Tcl_GetErrno ():0; return result; } ]]>

This procedure is called by generic the I/O layer whenever the user (re)configures the value of option '-blocking'. For a normal channel this procedure has to take the necessary actions at OS level to switch the managed channel into (non-)blocking behaviour. For a transformation however the behaviour is pretty much fixed. It has to remember the current blocking-mode in the transformation state and to notify the downstream channel of the change. None of these can be left out because remembering the current mode is necessary for InputProc to deal correctly with empty reads on the downstream channel and the notification is required to synchronize all channels in the stack with respect to blocking. Failing to do this will result in mysterious lockups while trying to read from the stack.

BlockModeProc skeleton flags |= ASYNC; } else { clientdata->flags &= ~ASYNC; } return Tcl_SetChannelOption( NULL /* ignore interp arg */, clientdata->channel->supercedes, "-blocking", (mode == TCL_MODE_NONBLOCKING) ? "0" : "1"); } ]]>

This procedure is called by generic the I/O layer to tell a channel that it is about to be destroyed. It is the responsibility of the procedure to clean up any data structures held by the channel. A transformation has the additional responsibility to complete the conversion of all incomplete information sitting in its internal write buffers and to write the result into the downstream channel. Another task is the removal of all event related things, like ChannelHandlers and Timers, although this could be billeted under 'clean up of any data structures held by the channel' too.

CloseProc skeleton channel); /* (x) */ ... free data structures on the heap. return TCL_OK } ]]> Remark (x): At the time of the call the backlink to the transformation channel actually already refers to the downstream channel instead, which is now at the top.

This procedure is called by the generic I/O layer whenever some input is required. Instead of going to the OS for this a transformation has to ask the channel downstream for data to convert. Use 'Tcl_Read' to do this, not 'Tcl_Gets'. The channel downstream does no EOL-translation (See ) and so has basically has no notion of lines. 'Tcl_Gets' might trip on that, unnecessarily waiting for the end of a line, or EOF, although there is data which could be converted. Instead of a skeleton which would be overwhelming despite trimming down I list the rules my input procedures are based upon. See 'generic/giot.c:TransformInputProc'for a real-life example. If the request can be satisfied by the information in the internal read buffers of the transformation, use that. In case of not having enough data, use everything available in the internal buffers and then ask the underlying channel for more. In blocking mode this will wait until we get some data, or EOF. Convert the incoming bytes and check whether the result is enough to satisfy the initial request. See 1). Loop, if not. In case of EOF do all the necessary special handling of the incomplete data in the internal buffers, then return (a possibly partial result). Signal EOF upward only if the internal buffers are empty. In non-blocking mode we either get nothing, some data or EOF. Handle EOF and some data as in the previous item. In case of nothing retrieved return the partial (or even empty) result. If there is nothing in the internal buffers signal error EWOULDBLOCK. Other things to consider: The calling procedure may have an unannounced reference to the result of the interpreter we are working with. So take care to use 'Tcl_SaveResult' and 'Tcl_RestoreResult' (available since Tcl 8.1) to protect that reference if you are calling back into the tcl level from within the transformation. Not doing this may crash the interpreter with a defect list of free objects. Writing to the underlying channel is allowed! An example using this is the SSL/TLS transformation created by Matt Newman to handle the initial handshake between the two partners before going into transparent encryption mode. As long as the handshake is not complete nothing can be read from/written to the channel. The InputProc can have any type and number of side effects. Examples: Identity transformation collecting statistics (frequency of bytes, byte-pairs, triplets, etc.) Splitter: Identity transformation piping the information flowing through the channel to a second channel (different from the underlying one). Recursively reading from/writing to the transformation itself (maybe indirectly, see splitter) is not a good idea, it may lead to infinite looping.

This procedure is called by the generic I/O layer whenever something is written to the transformation and an I/O buffer is flushed. The transformation has to convert as much as possible of the data it got, and the result must be written to the channel downstream (Well, not really, but not writing it does not make much sense). Data which cannot be converted at once has to be buffered and converted later, together with the data written by the next calls to the transformation. The API to use here is 'Tcl_Write'. As with InputProc this procedure is free to read from the underlying channel too, or from some other channel, or ...

This procedure is called by the generic I/O layer whenever the user (or the system) announces its (dis)interest in events on the channel. The correct behaviour of the transformation here is essential for a smooth interoperation with the notifier and for working fileevents. Whenever it is called it has to add or delete a Channelhandler in the channel downstream, using the given mask. This propagates the interest in events down the stack until the channel at the bottom takes the necessary steps to actually generate these events. From we know that events generated by the channel at the bottom are delivered to all channels in the stack, with the topmost channel the first to be notified. This will call all of the installed ChannelHandlers. They have to check the internal read buffer for pending information and to set up a timer if there is some. The reason for this will be explained later in this section. To complete the action, the Eventhandler for the timer has to call 'Tcl_NotifyChannel' with the mask for 'READABLE', in other words it has to generate an artificial 'readable' event. This in turn will invoke any 'fileevent' script and all the ChannelHandlers, causing another check for the necessity of setting up more timers.

WatchProc skeleton mask) { return; /* no changes */ } if (clientdata->mask) { /* remove old channel handler */ Tcl_DeleteChannelHandler ( clientdata->channel->supercedes, ChannelHandler, clientdata ); } clientdata->mask = mask; if (clientdata->mask) { /* recreate channel handler */ Tcl_CreateChannelHandler ( clientdata->channel->supercedes, clientdata->mask, ChannelHandler, clientdata ); } } ]]>

ChannelHandler skeleton timer != NULL) { /* cancel outstanding timer */ Tcl_DeleteTimerHandler (clientdata->timer); clientdata->timer = NULL; } if ((mask & TCL_READABLE) && clientdata->ReadBufferContainsDeliverableInformation()) { clientdata->timer = Tcl_CreateTimerHandler (DELAY, ChannelHandlerTimer, clientdata ); } } ]]>

Timer handler skeleton timer = NULL; /* Handle like normal events */ Tcl_NotifyChannel (clientData->channel, TCL_READABLE); } ]]> Now the promised explanation about the necessity of timers. Consider this scenario: A transformation stacked upon a socket, its internal read buffer is empty. The transformation does not merge lines. A fileevent script is set up and waiting for calls. The socket has data available, say 400 bytes, in several lines (more than one), they are the last on the channel, i.e. followed by EOF. The notifier generates the appropriate 'readable' event. This event triggers the execution of the fileevent script in the top channel. The executed script uses 'gets' to read a single line. As the buffers are empty (s.a.) the transformation asks the socket for data to convert, using 'Tcl_Read' and a standard buffer of 4K size. Thus it gets all waiting bytes from the socket. These are converted, resulting in several lines (no merge). Some of them are delivered up into the generic layer and its buffering, but not all (small buffersize). At least one line remains in the buffers of the transformation. The script processes the one line it got and then goes back to sleep. Now what ? The generic I/O layer finds that its buffers are not empty and uses a timer to generate additional events to clear them. The generic I/O buffers are empty. Now what, again ? Nothing. No events, no processing of the remaining line(s) stored in the transformation. Why ? The socket has an EOF pending and will not generate events anymore. The generic layer has empty buffers and ceases to generate events too. It has no knowledge about the buffers inside the driver, i.e the transformation. So the script will not wake up again, neither ask for the line, nor detect the pending EOF. We are hung. The solution is the same one used by the generic layer, but from the inside of the transformation this time: The transformation has to check itself for data waiting to be read and then use a timer to generate the necessary 'readable' events. And that is what the construction of channel handlers and timers described above will do.