[RT] [eve] Streaming out responses, SEDA and Synchronization

[Alex Karasulu <ao...@bellsouth.net>]

Hi gang,

I've been sitting here contemplating how to whip together the encoder stage
and the output stage.  The fact remains that we would like to stream out
responses back to clients rather than maintaining ~2x the image of the
response in memory.  The ~2x comes from the fact that you need the response
in memory and the on the wire 'transfer' image of the PDU.  So we definitely
want to stream out responses.

FACT:
   Responses transfer images are to be streamed out in chunks.

If we're streaming out chunks of the response's transfer images we're
encoding in chunks.  We don't have to but then again this would defeat the
purpose of streaming wouldn't it?  

FACT:
   Encode the response creating the transfer image into chunks and emit
several output events from the encoder for a single response PDU.  

How do these facts fair with a SEDA design?  They bring about a slew of
problems dealing with synchronization.  Stages are designed to process
events in parallel, that's why they have a pool of worker threads to do
their bidding.  When multiple output events are generated for a single event
and those events need to be processed in order we have issues.  There are
two parts to this problem:

1). Synchronizing events at a single stage
    
Basically a stage can look at a queue and determine an ordering if priority
markers are available on events or there is some way to deduce these
priorities to make a choice when processing events.  Some initial research
into this area shows concepts revolving around event color's where the
events of a color are ordered.  Hmm sounds like an individual client
represents a color here.
   
2). Synchronizing events processing in multiple stages

An event at one stage can fragment into many other events to be processed at
another stage and so on in a cascade.  We have such a situation in the
server.  In the case of search, a RequestEvent may generate several
ResponseEvents.  These response events must be processed in order.  Down
stream each response event may be fragmented into several OutputEvents as
the encoding stage processes each ResponseEvent for a search request in
series.  The whole picture looks like so:

            +------------+           +------------+           +------------+
Request     |  Request   |  Response |  Encoding  |  Output   |  Output    |
  --------> | Processing | --------> |    Stage   | --------> |   Stage    |
 Events     |   Stage    |    Events |            |   Events  |            |
            +------------+           +------------+           +------------+

So RequestEvents can produce multiple ResponseEvents which can produce
multiple OutputEvents.  ResponseEvents for the same RequestEvent must be
ordered.  And likewise OuputEvents for the same ResponseEvent must be
ordered.

How do we elegantly and simply enable ordered processing of these events in
both these situations (1) and (2)?  Situation (1) is a no brainer and I
listed it anyway while thinking this as I write it.

No answer yet.  Perhaps its best if for the time being I make these two
stages operate synchronously just for the search operation for now until a
better solution is found.  Yes the code for these modules is negligible and
can be replaced with a better solution later.  

Alex