Question on C++ broker code and multithread

[Steve Huston <sh...@riverace.com>]

Hi folks,

I have a question for the C++ broker folks that has arisen while
debugging a problem in the Windows code.

In my case, there are two threads involved:

1. Processing a SessionDetach received from a client. This ends up
deleting the session state, and in so doing, ends up in
SemanticState::cancel() which removes a task from outputTasks.

2. Processing the completion of writing the SessionDetached back to
that client. This ends up calling AsynchIOHandler::idle() and a few
frames up, AggregateOutput::doOutput() which walks through outputTasks
- the same outputTasks that are being manipulated in thread 1.

There is no guard on the manipulation of outputTasks here... 

Is there some implicit ordering designed in here that I may be
violating? I can tell you one difference in the way the Windows output
handling is working now compared to the posix scheme:

In posix/epoll, when there's output to send, the output-possible state
is enabled in the poller and the block queued. On the next pass around
the poller events, the socket is writeable and the block is dequeued
and sent.

In Windows, when there's output to send, it's initiated right then and
the OS handles it in the background. The completion of the write is
detected in the poller loop and handled.

This difference may compress the time between "data ready to go" and
"done", but it doesn't seem like it should break any contracts.

Does it seem like I've run into a problem that's just never been hit,
or have I broken some fundamental ordering?

Thanks,
-Steve

Re: Question on C++ broker code and multithread

[Gordon Sim <gs...@redhat.com>]

Steve Huston wrote:
> Hi folks,
> 
> I have a question for the C++ broker folks that has arisen while
> debugging a problem in the Windows code.
> 
> In my case, there are two threads involved:
> 
> 1. Processing a SessionDetach received from a client. This ends up
> deleting the session state, and in so doing, ends up in
> SemanticState::cancel() which removes a task from outputTasks.
> 
> 2. Processing the completion of writing the SessionDetached back to
> that client. This ends up calling AsynchIOHandler::idle() and a few
> frames up, AggregateOutput::doOutput() which walks through outputTasks
> - the same outputTasks that are being manipulated in thread 1.
> 
> There is no guard on the manipulation of outputTasks here... 
> 
> Is there some implicit ordering designed in here that I may be
> violating? 

The code is designed under the assumption that a connection will be 
processed by a single thread at any given time. The only call that is 
allowed to be concurrent is the the OutputControl::actvateOutput() 
method (used to request that an otherwise potentially idle connection be 
given use of a thread through OutputTask::doOutput().

So calling AggregateOutput::doOutput() on one thread while processing an 
  incoming frame on another thread would (under the current assumptions) 
be invalid and would be expected to fail.