You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by Charles Mason <ch...@gmail.com> on 2009/05/07 23:03:18 UTC
Bi-Directional RPC
>From reading the Thrift Paper and looking at the source it appears as
though Thrift doesn't support Bi-Directional RPC calls. I have an
application which functions as a server and 90% of the time the
clients will be calling RPC functions via Thrift like any other Thrift
app.
My problem is periodically the server needs to push a notification to
the client by effectively calling an RPC function on the client. My
app needs the clients to pick these notifications up fairly quickly
but they may only happen every few hours so its not practical to have
the clients poll the server every 10 seconds or so.
>From looking at the code and documentation it seems that I should be
able to effectively run the client and server methods on both ends of
the connection. Then I could implement my own transport to send the
traffic to the correct client or server methods depending on which the
traffic belongs to.
Is this feasible, or have a missed something about the design of Thrift?
Has anyone else done this before?
Thanks,
Charlie M
Re: Bi-Directional RPC
Posted by Charles Mason <ch...@gmail.com>.
On Tue, May 26, 2009 at 9:07 AM, Charles Mason<ch...@gmail.com> wrote:
> On Mon, May 25, 2009 at 10:45 PM, Chad Walters
> <Ch...@microsoft.com> wrote:
>> Charlie,
>>
>> Sounds great. -- looking forward to seeing the patch.
>>
>> What language are you working in?
>
> The first version will be in C#. If all goes according to plan I may
> need to make a Java port as well.
>
> I don't know any of the other Thrift languages well enough to make a
> port for them. It should be fairly straight forward for anyone to make
> a port from the C# version. I am trying to make the code clean and
> fairly well commented. I am also trying to use as many of the built in
> interfacees as possible, so I am not using the Socket directly but the
> TTransport interface. The user can specify which actual transport
> implementation they want.
I have now got an implementation of this working in C#. I haven't as
yet integrated it in to the Thrift build system, so at present its a
set of strand alone Classes which use the standard Thrift library. I
would like to eventually to get this into the thrift namespace and
build system once its mature enough. I would also like to make a Java
port.
You can download a sample project and the implementation from the address below.
http://www.cpsoftware.com/MultiplexedThriftTest.zip
The sample project is an extension of the common time server example
you see in several Thrift Tutorials. Basically the client periodically
calls a method that returns the servers current tick count every
second. In the mean time the server periodically calls a method on all
connected clients that provides the current tick count every 5
seconds. Its not exactly exciting but you can clearly see the
communication happening in both directions at the same time down a
single Thrift socket.
I have tried where possible to use the existing Thrift Transports and
Servers, however this has caused me some problems. The current version
is fairly inefficient using at least 2 extra threads per connection.
The big issue with the current Thrift library is you can't tell if
there's any data available to read from the socket so you have to
blindly read and wait for some data. If the read was none blocking or
simply provided the number of bytes currently available I could make
it much more efficient.
The current system allows up to 255 Virtual single direction
connections to be setup inside one Thrift connection. The system
assumes both ends know what each connection is uses for. So for
instance the server revives RDP calls on Virtual Connection 0 and
makes calls on Virtual Connection 1 and the reverse is true for the
clients. In theory you could set this up however you want not just
like the simple BI directional example above.
The actual multiplexing is done by spliting the traffic from each
Virtual Connection into frames of upto 256 bytes long. Which are
interleaved between each Virtual Connection. This shares the available
bandwidth between all the Virtual Connections, so one loaded
connection can't block all the others.
Let me know what you think.
Charlie M
Re: Bi-Directional RPC
Posted by Charles Mason <ch...@gmail.com>.
On Mon, May 25, 2009 at 10:45 PM, Chad Walters
<Ch...@microsoft.com> wrote:
> Charlie,
>
> Sounds great. -- looking forward to seeing the patch.
>
> What language are you working in?
The first version will be in C#. If all goes according to plan I may
need to make a Java port as well.
I don't know any of the other Thrift languages well enough to make a
port for them. It should be fairly straight forward for anyone to make
a port from the C# version. I am trying to make the code clean and
fairly well commented. I am also trying to use as many of the built in
interfacees as possible, so I am not using the Socket directly but the
TTransport interface. The user can specify which actual transport
implementation they want.
Charlie M
Re: Bi-Directional RPC
Posted by Chad Walters <Ch...@microsoft.com>.
Charlie,
Sounds great. -- looking forward to seeing the patch.
What language are you working in?
Chad
On 5/25/09 2:08 PM, "Charles Mason" <ch...@gmail.com> wrote:
On Fri, May 8, 2009 at 10:06 PM, Eric Anderson <an...@hpl.hp.com> wrote:
> Kevin Clark writes:
> > Almost identical question:
> > http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200905.mbox/browser
> >
> > The "game network protocol" thread in march and april:
> > http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200904.mbox/browser
> > http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200903.mbox/browser
> >
> > I've been hearing about this use case a lot lately. We may want to
> > consider providing support for it in the core lib . I'd personally be
> > happy to see patches towards that once people have an idea of how they
> > think it should work.
>
> We do this now by just passing marshalled structures back and forth
> across the wire; it gives us asynchronous send/reply in both
> directions. Somewhat simplified, the code looks like the below, it's not
> clear exactly what we would include in a patch to thrift since it's built
> outside and interacts with the event loop tha twe have.
Thanks for the code sample. This method has been suggested a few
times, the problem is it only allows async calls. For somethings this
is enough for others its not. I would like the flexibility for times
when a result would be useful, rather than trying to fudge it with two
aysnc calls.
I am currently half way through making a new Multiplexed transport. It
allows you to wrap any existing Thrift transport with this one. This
creates multiple Virtual transports which can be used like any other
transport however they don't actually send anything directly. What
happens is the Multiplexed transport picks up waiting data from each
Virtual Transport and transmits them to the other end. At the other
end there is another Multiplexed transport, which splits the data off
in to the relevant Virtual Transports which can then be processed as
normal. This can happen in both directions, so you may have a server
and client connection multiplexed together to have a bidirectional
RPC.
All this should allow as many single direction connections as you want
down the same single TCP connection. The best bit of this is you only
need one open socket on the server. Theres no need for the client to
be able to accept incoming connections.
I will post back when I have something working to share.
Charlie M
Re: Bi-Directional RPC
Posted by Charles Mason <ch...@gmail.com>.
On Fri, May 8, 2009 at 10:06 PM, Eric Anderson <an...@hpl.hp.com> wrote:
> Kevin Clark writes:
> > Almost identical question:
> > http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200905.mbox/browser
> >
> > The "game network protocol" thread in march and april:
> > http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200904.mbox/browser
> > http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200903.mbox/browser
> >
> > I've been hearing about this use case a lot lately. We may want to
> > consider providing support for it in the core lib . I'd personally be
> > happy to see patches towards that once people have an idea of how they
> > think it should work.
>
> We do this now by just passing marshalled structures back and forth
> across the wire; it gives us asynchronous send/reply in both
> directions. Somewhat simplified, the code looks like the below, it's not
> clear exactly what we would include in a patch to thrift since it's built
> outside and interacts with the event loop tha twe have.
Thanks for the code sample. This method has been suggested a few
times, the problem is it only allows async calls. For somethings this
is enough for others its not. I would like the flexibility for times
when a result would be useful, rather than trying to fudge it with two
aysnc calls.
I am currently half way through making a new Multiplexed transport. It
allows you to wrap any existing Thrift transport with this one. This
creates multiple Virtual transports which can be used like any other
transport however they don't actually send anything directly. What
happens is the Multiplexed transport picks up waiting data from each
Virtual Transport and transmits them to the other end. At the other
end there is another Multiplexed transport, which splits the data off
in to the relevant Virtual Transports which can then be processed as
normal. This can happen in both directions, so you may have a server
and client connection multiplexed together to have a bidirectional
RPC.
All this should allow as many single direction connections as you want
down the same single TCP connection. The best bit of this is you only
need one open socket on the server. Theres no need for the client to
be able to accept incoming connections.
I will post back when I have something working to share.
Charlie M
Re: Bi-Directional RPC
Posted by Eric Anderson <an...@hpl.hp.com>.
Kevin Clark writes:
> Almost identical question:
> http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200905.mbox/browser
>
> The "game network protocol" thread in march and april:
> http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200904.mbox/browser
> http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200903.mbox/browser
>
> I've been hearing about this use case a lot lately. We may want to
> consider providing support for it in the core lib . I'd personally be
> happy to see patches towards that once people have an idea of how they
> think it should work.
We do this now by just passing marshalled structures back and forth
across the wire; it gives us asynchronous send/reply in both
directions. Somewhat simplified, the code looks like the below, it's not
clear exactly what we would include in a patch to thrift since it's built
outside and interacts with the event loop tha twe have.
-Eric
uint32_t Message::write(facebook::thrift::protocol::TProtocol &oprot) const {
uint32_t bytes = 0;
bytes += oprot.writeMessageBegin(getType(),
facebook::thrift::protocol::T_CALL,
getSeqno());
bytes += data->write(&oprot);
bytes += oprot.writeMessageEnd();
return bytes;
}
MessageData *
MessageData::deMarshal(const std::string &from_name,
facebook::thrift::protocol::TProtocol &from,
int32_t &seqno, int64_t &path) {
MessageData *ret = NULL;
try {
string name;
facebook::thrift::protocol::TMessageType mtype;
from.readMessageBegin(name, mtype, seqno);
SINVARIANT(mtype == facebook::thrift::protocol::T_CALL);
MessageData::Reader *reader = getMap().lookup(name);
INVARIANT(reader != NULL, format("can not find reader for %s from %s")
% name % from_name);
ret = (*reader)(&from);
from.readMessageEnd();
} catch (...) {
...;
}
return ret;
}
void mainloop() {
while(true) {
(far_side, protocol) = getReadyFd();
uint32_t seqno = 0;
int64_t path = 0;
data = MessageData::deMarshal(far_side->getName(),
*binary_protocol, seqno, path);
Message::Ptr msg = Message::make(EventQueue::EQ().getMe(), data,
seqno, path);
EventQueue::recvMessage(msg);
}
}
Re: Bi-Directional RPC
Posted by Charles Mason <ch...@gmail.com>.
Hi Kevin,
Thanks for the info. If I get this working with Thrift I will
certainly be happy to submit patches for it.
It strikes me it should be possible to make this work by adding some
extra classes to the base library at the protocol layer. It does of
course it does of course mean the wire format would be different. I
also only have enough knowledege of Java and C# to write patches for
those languages. I suppose if anyone else is interested it should be
fairly easy to port it to a language of their choice.
I will keep looking in to this and post back if I make any progress.
Thanks again,
Charlie M
On Fri, May 8, 2009 at 12:54 AM, Kevin Clark <ke...@gmail.com> wrote:
> Hi Charlie,
> See:
>
> Almost identical question:
> http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200905.mbox/browser
>
> The "game network protocol" thread in march and april:
> http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200904.mbox/browser
> http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200903.mbox/browser
>
> I've been hearing about this use case a lot lately. We may want to
> consider providing support for it in the core lib . I'd personally be
> happy to see patches towards that once people have an idea of how they
> think it should work.
>
> --
> Kevin Clark
> http://glu.ttono.us
>
Re: Bi-Directional RPC
Posted by Kevin Clark <ke...@gmail.com>.
Hi Charlie,
See:
Almost identical question:
http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200905.mbox/browser
The "game network protocol" thread in march and april:
http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200904.mbox/browser
http://mail-archives.apache.org/mod_mbox/incubator-thrift-user/200903.mbox/browser
I've been hearing about this use case a lot lately. We may want to
consider providing support for it in the core lib . I'd personally be
happy to see patches towards that once people have an idea of how they
think it should work.
--
Kevin Clark
http://glu.ttono.us