How should I implement multi-threaded clients and multi-threaded connections in C++?

[Ben Craig <be...@ni.com>]

Suppose I have a .thrift spec similar to the following:

service Slow
{
  void slowOperation(),
  void stopEverything(),
}

When my application launches, my code will establish a connection with the 
server, and hold on to that connection for the lifetime of the app.  When 
a user launches a dialog box from this application, I want to call 
slowOperation().  If the user hits "Cancel" on the dialog, I want to run 
stopEverything().  stopEverything should cause slowOperation to stop what 
it is doing and complete.

I have several problems with this setup right now.  First, the 
codegenerated SlowClient class isn't thread safe.  If multiple threads try 
to access it at the same time, bad things will happen.  Second, as far as 
I can tell, the existing servers only process messages on one thread per 
connection.  That means that even if I managed to send a stopEverything() 
call while a slowOperation() was in progress, the server wouldn't process 
that message until the slowOperation was complete.

Does the C++ library / compiler currently have anything that can satisfy 
this requirement?  I'm looking at some of the COB / Continuation OBject 
stuff, but it doesn't look like it makes the SlowClient thread safe (but 
maybe the AsyncChannel is supposed to be safe instead?).  It also looks 
like there isn't an easy way to get this behavior for any given transport.

I'm fine adding support for this kind of use case, but I want to make sure 
that nothing already exists that I've overlooked before I go down that 
path.

Re: How should I implement multi-threaded clients and multi-threaded connections in C++?

[Ben Craig <be...@ni.com>]

Jens,

> > Here is what I think you are suggesting:
> > service SlowRunner { ... }
> > service SlowStopper { ... }
> 
> I don't think that two different services are really necessary. In fact, 
I 
> would not even recommend it for your particular case.

Yep, you're right.  Two services don't make much sense there.  Put the 
stopEverything function in the SlowRunner service and I think stuff makes 
sense again.

> You didn't say that in your use case description. 
I apologize.  I'm trying to be helpful without pre-announcing anything 
from my company.  I think I've come up with some better explanations 
though.

Right now, I'm working with several cases where something that is 
currently in-process needs to move out-of-process.  The in-process code 
has a well defined API, and I want to keep something extremely similar to 
that API where possible.  The in-process code often has sessions and 
objects that can be operated on from multiple threads in a thread safe 
way.  Some of these operations are I/O operations with client specified 
time-outs.  These APIs provide a way to cancel a session's I/O from 
another thread.

Most (but not all) of these use cases already have session objects that 
would be a very reasonable place to put a thrift client.  The servers are 
likely to be resource constrained though.  If at all possible, I want to 
maintain the assumptions that calling code had before.  Basically, code 
that wasn't serialized in-process before shouldn't be serialized in the 
out-of-process case.

So I'll argue that I got separation of concerns down first :).  The API 
concern was "solved" before I even started researching Thrift.  I just 
want a server (and a client) that will guard the internal transport 
appropriately so that one blocked request doesn't stall other, newer 
requests on the same transport.

> > My desired interface is actually more like this:
> > service GenericService> {
> >   string doSomething(1: string cmd),
> > }
> 
> So you are adding one interface (your command line syntax) on top of a 
> different interface (Thrift)?    ;-)

Yep, Thrift is my golden hammer of choice right now. (
http://en.wikipedia.org/wiki/Law_of_the_instrument)

It seems a lot easier to use the servers, clients, and transports as 
provided in Thrift than to re-roll all of that, even if the end result 
might be a little simpler and smaller.

Re: How should I implement multi-threaded clients and multi-threaded connections in C++?

[Jens Geyer <je...@hotmail.com>]

Hi Ben,

> Here is what I think you are suggesting:
> service SlowRunner { ... }
> service SlowStopper { ... }

I don't think that two different services are really necessary. In fact, I 
would not even recommend it for your particular case.


> One of the commands would be "doSlowOperation" and another
> would be "stopEverything".  There would also be per connection
> state, to service hypothetical commands like "addToQueue" and
> "removeFromQueue".

You didn't say that in your use case description. You didn't even say that 
slowOperation() must be a blocking operation, I just assumed that from your 
description. Let's just assume, that slowOperation() could be executed 
asyncronously without problems. Then the client would not even need to wait 
for it leaving the connection open all the time!

The longer I think about it, the more I would like to suggest to sketch the 
entire scenario more closely to answer all those questions popping up.  How 
is the queue involved with slowOperation()? Maybe slowOperation() takes work 
items from a queue and processes them? If yes, why is it necessary to call 
slowOperation() at all? Couldn't processing be triggered automatically by 
addToQueue()? And by the way, how does the client keep track of the status 
of the running operations, e.g. to visualize their progress in the UI?


> However, that seems like it is adding one session tracking
> system (instanceIDs) on top of a different session tracking
> system (connections).

There are cases where open connections are a much more scarce resource than 
memory or disk space is, so this is not necessarily such a bad idea.

One should  carefully differentiate the sessions defined by some internal 
state from sessions defined by a connection. The connection is just the 
entry point used to access the internal state, but it does not necessarily 
have the same life time as that state. The lifetime of the internal state 
can easily span hundrets or thousands of established and closed connections, 
all used to access the very same internal state data. And that's the first 
and foremost reason why I would separate service implementation from 
implementation of the logic made accessible by means of the service: 
Separation of Concerns.


> My desired interface is actually more like this:
> service GenericService> {
>   string doSomething(1: string cmd),
> }

So you are adding one interface (your command line syntax) on top of a 
different interface (Thrift)?    ;-)


> I'm fine if Thrift doesn't currently support such a use case, and I will
> gladly develop and contribute the support.  I have several uses for this
> threading support in my company.  I just want to make sure that I'm not
> duplicating the work that already exists in some corner of Thrift I
> haven't become familiar with yet.

Agree. Maybe someone with more expertise regarding that particular area of 
the code joins the discussion, but I'm sure any contribution is welcome.


Best regards,
JensG


-----Ursprüngliche Nachricht----- 
From: Ben Craig
Sent: Tuesday, December 4, 2012 11:51 PM
To: dev@thrift.apache.org
Subject: Re: How should I implement multi-threaded clients and 
multi-threaded connections in C++?

Thanks for the reply Jens.  Here is what I think you are suggesting:

service SlowRunner
{
  i32 createOperation(),
  void slowOperation(1: i32 instanceID),
}
service SlowStopper
{
  void stopEverything(1: i32 instanceID),
}

So with my interface, a client would call SlowClient::slowOperation() in
the "main thread", and SlowClient::stopEverything() on the "stop thread",
both on the same SlowClient instance.

With this new interface, you are suggesting that the "main thread" call
SlowRunnerClient::createOperation(), then call
SlowRunnerClient::slowOperation().  If I need to stop things, then the
"stop thread" will need to get the value of the instanceID, then call
SlowStopper::stopEverything() on a SlowStopper instance.

This solves the problem as specified in my post, at the cost of extra
connections.  In reality, it doesn't solve my problem.  My desired
interface is actually more like this:

service GenericService
{
  string doSomething(1: string cmd),
}

One of the commands would be "doSlowOperation" and another would be
"stopEverything".  There would also be per connection state, to service
hypothetical commands like "addToQueue" and "removeFromQueue".

So even with this, it would be possible to add a bunch of instance ids to
the service, and force the client and server implementations to create
lots of connections and session tracking.  However, that seems like it is
adding one session tracking system (instanceIDs) on top of a different
session tracking system (connections).


I'm fine if Thrift doesn't currently support such a use case, and I will
gladly develop and contribute the support.  I have several uses for this
threading support in my company.  I just want to make sure that I'm not
duplicating the work that already exists in some corner of Thrift I
haven't become familiar with yet.


From:   "Jens Geyer" <je...@hotmail.com>
To:     <de...@thrift.apache.org>,
Date:   12/04/2012 04:00 PM
Subject:        Re: How should I implement multi-threaded clients and
multi-threaded connections in C++?



Hi Ben,

just as an idea what I would be trying:

1.  extract the implementation of slowOperation() and stopEverything()
from
the service into a dedicated class and make that one threadsafe.

2. if you need only one instance of this implementation class, make it a
singleton or even a static class. if you want to have more than one in
parallel, you need a (threadsafe) container for them. In the latter case
go
to step 3, otherwise continue with 4

3. Give each implemenation instance an ID and return this ID through an
additional createOperation() method to the client. This ID must be passed
in
when slowOperation() or stopEverything() are called to locate the correct
implementation instance. Since createOperation() creates a new
implementation instance, don't forget that this instance must be deleted
afterwards.

4. modify the service methods accordingly, they now just call the
implementation and create/free the instances.

5. the client is now free to open another connection to the server to call

stopEverything(). This call sets some kind of abort flag at the
implementation instance identified through the ID. The flag ist tested
repeatedly by slowOperation().

Could that work for you?
JensG

-----Ursprüngliche Nachricht----- 
From: Ben Craig
Sent: Tuesday, December 4, 2012 10:20 PM
To: dev@thrift.apache.org
Subject: How should I implement multi-threaded clients and multi-threaded
connections in C++?

Suppose I have a .thrift spec similar to the following:

service Slow
{
  void slowOperation(),
  void stopEverything(),
}

When my application launches, my code will establish a connection with the
server, and hold on to that connection for the lifetime of the app.  When
a user launches a dialog box from this application, I want to call
slowOperation().  If the user hits "Cancel" on the dialog, I want to run
stopEverything().  stopEverything should cause slowOperation to stop what
it is doing and complete.

I have several problems with this setup right now.  First, the
codegenerated SlowClient class isn't thread safe.  If multiple threads try
to access it at the same time, bad things will happen.  Second, as far as
I can tell, the existing servers only process messages on one thread per
connection.  That means that even if I managed to send a stopEverything()
call while a slowOperation() was in progress, the server wouldn't process
that message until the slowOperation was complete.

Does the C++ library / compiler currently have anything that can satisfy
this requirement?  I'm looking at some of the COB / Continuation OBject
stuff, but it doesn't look like it makes the SlowClient thread safe (but
maybe the AsyncChannel is supposed to be safe instead?).  It also looks
like there isn't an easy way to get this behavior for any given transport.

I'm fine adding support for this kind of use case, but I want to make sure
that nothing already exists that I've overlooked before I go down that
path.

Re: How should I implement multi-threaded clients and multi-threaded connections in C++?

[Ben Craig <be...@ni.com>]

Thanks for the reply Jens.  Here is what I think you are suggesting:

service SlowRunner
{
  i32 createOperation(),
  void slowOperation(1: i32 instanceID),
}
service SlowStopper
{
  void stopEverything(1: i32 instanceID),
}

So with my interface, a client would call SlowClient::slowOperation() in 
the "main thread", and SlowClient::stopEverything() on the "stop thread", 
both on the same SlowClient instance. 

With this new interface, you are suggesting that the "main thread" call 
SlowRunnerClient::createOperation(), then call 
SlowRunnerClient::slowOperation().  If I need to stop things, then the 
"stop thread" will need to get the value of the instanceID, then call 
SlowStopper::stopEverything() on a SlowStopper instance.

This solves the problem as specified in my post, at the cost of extra 
connections.  In reality, it doesn't solve my problem.  My desired 
interface is actually more like this:

service GenericService
{
  string doSomething(1: string cmd),
}

One of the commands would be "doSlowOperation" and another would be 
"stopEverything".  There would also be per connection state, to service 
hypothetical commands like "addToQueue" and "removeFromQueue".

So even with this, it would be possible to add a bunch of instance ids to 
the service, and force the client and server implementations to create 
lots of connections and session tracking.  However, that seems like it is 
adding one session tracking system (instanceIDs) on top of a different 
session tracking system (connections).


I'm fine if Thrift doesn't currently support such a use case, and I will 
gladly develop and contribute the support.  I have several uses for this 
threading support in my company.  I just want to make sure that I'm not 
duplicating the work that already exists in some corner of Thrift I 
haven't become familiar with yet.


From:   "Jens Geyer" <je...@hotmail.com>
To:     <de...@thrift.apache.org>, 
Date:   12/04/2012 04:00 PM
Subject:        Re: How should I implement multi-threaded clients and 
multi-threaded connections in C++?



Hi Ben,

just as an idea what I would be trying:

1.  extract the implementation of slowOperation() and stopEverything() 
from 
the service into a dedicated class and make that one threadsafe.

2. if you need only one instance of this implementation class, make it a 
singleton or even a static class. if you want to have more than one in 
parallel, you need a (threadsafe) container for them. In the latter case 
go 
to step 3, otherwise continue with 4

3. Give each implemenation instance an ID and return this ID through an 
additional createOperation() method to the client. This ID must be passed 
in 
when slowOperation() or stopEverything() are called to locate the correct 
implementation instance. Since createOperation() creates a new 
implementation instance, don't forget that this instance must be deleted 
afterwards.

4. modify the service methods accordingly, they now just call the 
implementation and create/free the instances.

5. the client is now free to open another connection to the server to call 

stopEverything(). This call sets some kind of abort flag at the 
implementation instance identified through the ID. The flag ist tested 
repeatedly by slowOperation().

Could that work for you?
JensG

-----Ursprüngliche Nachricht----- 
From: Ben Craig
Sent: Tuesday, December 4, 2012 10:20 PM
To: dev@thrift.apache.org
Subject: How should I implement multi-threaded clients and multi-threaded 
connections in C++?

Suppose I have a .thrift spec similar to the following:

service Slow
{
  void slowOperation(),
  void stopEverything(),
}

When my application launches, my code will establish a connection with the
server, and hold on to that connection for the lifetime of the app.  When
a user launches a dialog box from this application, I want to call
slowOperation().  If the user hits "Cancel" on the dialog, I want to run
stopEverything().  stopEverything should cause slowOperation to stop what
it is doing and complete.

I have several problems with this setup right now.  First, the
codegenerated SlowClient class isn't thread safe.  If multiple threads try
to access it at the same time, bad things will happen.  Second, as far as
I can tell, the existing servers only process messages on one thread per
connection.  That means that even if I managed to send a stopEverything()
call while a slowOperation() was in progress, the server wouldn't process
that message until the slowOperation was complete.

Does the C++ library / compiler currently have anything that can satisfy
this requirement?  I'm looking at some of the COB / Continuation OBject
stuff, but it doesn't look like it makes the SlowClient thread safe (but
maybe the AsyncChannel is supposed to be safe instead?).  It also looks
like there isn't an easy way to get this behavior for any given transport.

I'm fine adding support for this kind of use case, but I want to make sure
that nothing already exists that I've overlooked before I go down that
path. 

Re: How should I implement multi-threaded clients and multi-threaded connections in C++?

[Jens Geyer <je...@hotmail.com>]

Hi Ben,

just as an idea what I would be trying:

1.  extract the implementation of slowOperation() and stopEverything() from 
the service into a dedicated class and make that one threadsafe.

2. if you need only one instance of this implementation class, make it a 
singleton or even a static class. if you want to have more than one in 
parallel, you need a (threadsafe) container for them. In the latter case go 
to step 3, otherwise continue with 4

3. Give each implemenation instance an ID and return this ID through an 
additional createOperation() method to the client. This ID must be passed in 
when slowOperation() or stopEverything() are called to locate the correct 
implementation instance. Since createOperation() creates a new 
implementation instance, don't forget that this instance must be deleted 
afterwards.

4. modify the service methods accordingly, they now just call the 
implementation and create/free the instances.

5. the client is now free to open another connection to the server to call 
stopEverything(). This call sets some kind of abort flag at the 
implementation instance identified through the ID. The flag ist tested 
repeatedly by slowOperation().

Could that work for you?
JensG

-----Ursprüngliche Nachricht----- 
From: Ben Craig
Sent: Tuesday, December 4, 2012 10:20 PM
To: dev@thrift.apache.org
Subject: How should I implement multi-threaded clients and multi-threaded 
connections in C++?

Suppose I have a .thrift spec similar to the following:

service Slow
{
  void slowOperation(),
  void stopEverything(),
}

When my application launches, my code will establish a connection with the
server, and hold on to that connection for the lifetime of the app.  When
a user launches a dialog box from this application, I want to call
slowOperation().  If the user hits "Cancel" on the dialog, I want to run
stopEverything().  stopEverything should cause slowOperation to stop what
it is doing and complete.

I have several problems with this setup right now.  First, the
codegenerated SlowClient class isn't thread safe.  If multiple threads try
to access it at the same time, bad things will happen.  Second, as far as
I can tell, the existing servers only process messages on one thread per
connection.  That means that even if I managed to send a stopEverything()
call while a slowOperation() was in progress, the server wouldn't process
that message until the slowOperation was complete.

Does the C++ library / compiler currently have anything that can satisfy
this requirement?  I'm looking at some of the COB / Continuation OBject
stuff, but it doesn't look like it makes the SlowClient thread safe (but
maybe the AsyncChannel is supposed to be safe instead?).  It also looks
like there isn't an easy way to get this behavior for any given transport.

I'm fine adding support for this kind of use case, but I want to make sure
that nothing already exists that I've overlooked before I go down that
path.