You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Joel Meyer <jo...@gmail.com> on 2009/04/03 01:17:18 UTC
Re: Using thrift as part of a game network protocol
On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <da...@gmail.com>wrote:
> Ok I definitely plan on giving the Async RPC methods a try tonight, but I
> figured I'd just throw out some questions before I get home to start
> hacking
> on this stuff.
>
> The one-to-one message to RPC call Async solution will let a client send
> messages of any given type in my defined protocol, but how would a server
> respond to a client with a message that the client didn't request? For
> example say I was trying to write a FPS like Quake and I want to server to
> send position updates for all clients to all clients, how would i model
> that
> as a client RPC request for that. With the Async RPC solutions I could make
> a RPC call for Map<Integer, Position> getPositionUpdates(), Now say that
> the
> client needs to request 50 other messages to be notified of. I guess the
> solution would be to make an Async RPC call requesting those updates and
> respond to it when I receive it asynchronously and then reissue another
> Async RPC call for the next set of updates. It just seems inefficient to
> actively make the client request for data when the server could implicitly
> know that when connected on this game protocol I can just send these
> messages to the clients without them asking for it. Not to mention you'd
> have make sure you don't "miss" sending a client a message if they finished
> their Async call but haven't reestablished a new one.
>
I think I've done something similar to what you're trying to do, and as long
as you can commit to using only async messages it's possible to pull it off
without having to start a server on the client to accept RPCs from the
server.
When your RPC is marked as async the server doesn't send a response and the
client doesn't try to read one. So, if all your RPC calls from the client to
the server are async you have effectively freed up the inbound half of the
socket connection. That means that you can use it for receiving async
messages from the server - the only catch is that you have to start a new
thread to read and dispatch the incoming async RPC calls.
In a typical Thrift RPC system you'd create a MyService.Processor on your
server and a MyService.Client on your client. To do bidirectional async
message sending you'll need to go a step further and create a
MyService.Client on your server for each client that connects (this can be
accomplished by providing your own TProcessorFactory) and then on each
client you create a MyService.Processor. (This assumes that you've gone with
a generic MyService definition like you described above that has a bunch of
optional messages, another option would be to define separate service
definitions for the client and server.) With two clients connected the
objects in existence would look something like this:
Server:
MyService.Processor mainProcessor - handles incoming async RPCs
MyService.Client clientA - used to send outgoing async RPCs to ClientA
MyService.Client clientB - used to send outgoing async RPCs to ClientB
ClientA:
MyService.Client - used to send messages to Server
MyService.Processor clientProcessor - used (by a separate thread) to
process incoming async RPCs
ClientB:
MyService.Client - used to send messages to Server
MyService.Processor clientProcessor - used (by a separate thread) to
process incoming async RPCs
Hopefully that explains the concept. If you need example code I can try and
pull something together (it will be in Java). The nice thing about this
method is that you don't have to establish two connections, so you can get
around the firewall issues others have mentioned. I've been using this
method on a service in production and haven't had any problems. When you
have a separate thread in your client running a Processor you're basically
blocking on a read, waiting for a message from the server. The benefit of
this is that you're notified immediately when the server shuts down instead
of having to wait until you send a message and then finding out that the TCP
connection was reset.
Cheers,
Joel
>
> The biggest issue is that not all client request will result in a single
> response (like shooting a bullet, may blowup an entity, and damage all
> players in the area those events are seperate messages sent from the
> respective entities).
>
> At a game development studio I used to work at we developed a cross
> language
> IDL network protocol definition (C++, Java) very similiar to Protocol
> Buffers and Thrift (without some of the more mature features like being
> transport agnostic we explicitly built it for binary TCP socket transport,
> or protocol versioning), the stream of packets would contain as the first
> 32
> bits a message ID that would be a key to a map a Message class that would
> have methods to read in that message type from a byte[] stream.
>
> Looking through Thrift code in the TBinaryProtocol writeMessage it looks
> like it's including the name of the message being sent and it's type (is
> the
> concept of Message in thrift the same as RPC?), if so what's the
> corresponding code pathway for the client waiting for an RPC response
> because if I could just use this message name or type to key into what I
> need to serialize off the network from both client and server end then that
> would be perfect.
>
>
>
> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > I really think that using async service methods which are matched one to
> > one
> > with the message types that you want to send gives you exactly the
> > semantics
> > that are being requested with very simple implementation cost.
> >
> > It is important to not get toooo hung up on what RPC stands for. I use
> > async methods all the time to stream data structures for logging and it
> > works great. Moreover, it provides a really simple way of building
> > extractors and processors for this data since I have an interface sitting
> > there that will tell me about all of the methods (data types) that I need
> > to
> > handle or explicitly ignore.
> >
> > So the trick works and works really well. Give it a try!
> >
> > On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
> wrote:
> >
> > > Optional fields are not serialized onto the wire. There is a slight
> > > performance penalty at serialization time if you have a ton of unset
> > fields,
> > > but that's it.
> > >
> > > Am I over complicating things
> > >>
> > >
> > > Personally, sounds like it to me. Why do you need this streaming
> behavior
> > > or whatnot? Hotwiring the rpc stack to let you send any message you
> want
> > is
> > > going to be a ton of work and not really that much of a functionality
> > > improvement.
> > >
> > > -Bryan
> > >
> >
> >
> >
> > --
> > Ted Dunning, CTO
> > DeepDyve
> >
>
Re: Using thrift as part of a game network protocol
Posted by Terry Jones <te...@jon.es>.
>>>>> "Ted" == Ted Dunning <te...@gmail.com> writes:
Ted> That is, could there actually be a fully operational Thrift transport
Ted> that moves requests both ways on a single socket? You would give it
Ted> something that implements the server interface and it would give you
Ted> back something that also implements the interface. You could call it
Ted> and it could call you in horrifically complicated but very cool ways.
Right. I had the same thought, in this form:
You could take a normal Thrift service spec and essentially produce two
services from it. A server method that returned an X would become an async
server method, and a (client-side) server method that took an X as an
argument would be generated to run on the client to receive the method
result as originally intended. You would need something under the covers
to keep track of the outstanding calls and match them up with their
incoming async results (if any).
It would begin to look something like the Twisted/Thrift solution, which
conceptually already does this (with the help of Twisted's deferreds and a
client-side dictionary of outstanding requests).
The two are NOT exactly the same, just quite similar. As Doug said
originally:
>> you'd have make sure you don't "miss" sending a client a message if they
>> finished their Async call but haven't reestablished a new one.
But, to show a bit more similarity, Joel pointed out that his suggestion
allows the client to immediately know when the server has gone away. You
can do this already with the Twisted/Thrift combination: the client makes a
Thrift method call, say adviseOnShutdown, and immediately gets back a
deferred. That pending call just sits there client-side (the caller and
the Twisted/Thrift code both have a reference to the Deferred) until the
server either actively returns to indicate that it's going away, or
disappears, in which case the client gets the disconnect as a failure from
the pending deferred.
I hope the above all makes some sense!
Terry
Re: Using thrift as part of a game network protocol
Posted by Terry Jones <te...@jon.es>.
>>>>> "Ted" == Ted Dunning <te...@gmail.com> writes:
Ted> That is, could there actually be a fully operational Thrift transport
Ted> that moves requests both ways on a single socket? You would give it
Ted> something that implements the server interface and it would give you
Ted> back something that also implements the interface. You could call it
Ted> and it could call you in horrifically complicated but very cool ways.
Right. I had the same thought, in this form:
You could take a normal Thrift service spec and essentially produce two
services from it. A server method that returned an X would become an async
server method, and a (client-side) server method that took an X as an
argument would be generated to run on the client to receive the method
result as originally intended. You would need something under the covers
to keep track of the outstanding calls and match them up with their
incoming async results (if any).
It would begin to look something like the Twisted/Thrift solution, which
conceptually already does this (with the help of Twisted's deferreds and a
client-side dictionary of outstanding requests).
The two are NOT exactly the same, just quite similar. As Doug said
originally:
>> you'd have make sure you don't "miss" sending a client a message if they
>> finished their Async call but haven't reestablished a new one.
But, to show a bit more similarity, Joel pointed out that his suggestion
allows the client to immediately know when the server has gone away. You
can do this already with the Twisted/Thrift combination: the client makes a
Thrift method call, say adviseOnShutdown, and immediately gets back a
deferred. That pending call just sits there client-side (the caller and
the Twisted/Thrift code both have a reference to the Deferred) until the
server either actively returns to indicate that it's going away, or
disappears, in which case the client gets the disconnect as a failure from
the pending deferred.
I hope the above all makes some sense!
Terry
Re: Using thrift as part of a game network protocol
Posted by Ted Dunning <te...@gmail.com>.
Could this be made less dependent on the world supply of good-will and
optimism?
That is, could there actually be a fully operational Thrift transport that
moves requests both ways on a single socket? You would give it something
that implements the server interface and it would give you back something
that also implements the interface. You could call it and it could call you
in horrifically complicated but very cool ways.
*That* would be something that other RPC packages don't do.
On Thu, Apr 2, 2009 at 4:56 PM, David Reiss <dr...@facebook.com> wrote:
> What an ingenious hack! Cross-posting to thrift-dev in case some
> people don't read this list.
>
> --David
>
> Joel Meyer wrote:
> > On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <daniels.douglas@gmail.com
> >wrote:
> >
> >> Ok I definitely plan on giving the Async RPC methods a try tonight, but
> I
> >> figured I'd just throw out some questions before I get home to start
> >> hacking
> >> on this stuff.
> >>
> >> The one-to-one message to RPC call Async solution will let a client send
> >> messages of any given type in my defined protocol, but how would a
> server
> >> respond to a client with a message that the client didn't request? For
> >> example say I was trying to write a FPS like Quake and I want to server
> to
> >> send position updates for all clients to all clients, how would i model
> >> that
> >> as a client RPC request for that. With the Async RPC solutions I could
> make
> >> a RPC call for Map<Integer, Position> getPositionUpdates(), Now say that
> >> the
> >> client needs to request 50 other messages to be notified of. I guess the
> >> solution would be to make an Async RPC call requesting those updates and
> >> respond to it when I receive it asynchronously and then reissue another
> >> Async RPC call for the next set of updates. It just seems inefficient to
> >> actively make the client request for data when the server could
> implicitly
> >> know that when connected on this game protocol I can just send these
> >> messages to the clients without them asking for it. Not to mention you'd
> >> have make sure you don't "miss" sending a client a message if they
> finished
> >> their Async call but haven't reestablished a new one.
> >>
> >
> > I think I've done something similar to what you're trying to do, and as
> long
> > as you can commit to using only async messages it's possible to pull it
> off
> > without having to start a server on the client to accept RPCs from the
> > server.
> >
> > When your RPC is marked as async the server doesn't send a response and
> the
> > client doesn't try to read one. So, if all your RPC calls from the client
> to
> > the server are async you have effectively freed up the inbound half of
> the
> > socket connection. That means that you can use it for receiving async
> > messages from the server - the only catch is that you have to start a new
> > thread to read and dispatch the incoming async RPC calls.
> >
> > In a typical Thrift RPC system you'd create a MyService.Processor on your
> > server and a MyService.Client on your client. To do bidirectional async
> > message sending you'll need to go a step further and create a
> > MyService.Client on your server for each client that connects (this can
> be
> > accomplished by providing your own TProcessorFactory) and then on each
> > client you create a MyService.Processor. (This assumes that you've gone
> with
> > a generic MyService definition like you described above that has a bunch
> of
> > optional messages, another option would be to define separate service
> > definitions for the client and server.) With two clients connected the
> > objects in existence would look something like this:
> >
> > Server:
> > MyService.Processor mainProcessor - handles incoming async RPCs
> > MyService.Client clientA - used to send outgoing async RPCs to ClientA
> > MyService.Client clientB - used to send outgoing async RPCs to ClientB
> >
> > ClientA:
> > MyService.Client - used to send messages to Server
> > MyService.Processor clientProcessor - used (by a separate thread) to
> > process incoming async RPCs
> >
> > ClientB:
> > MyService.Client - used to send messages to Server
> > MyService.Processor clientProcessor - used (by a separate thread) to
> > process incoming async RPCs
> >
> > Hopefully that explains the concept. If you need example code I can try
> and
> > pull something together (it will be in Java). The nice thing about this
> > method is that you don't have to establish two connections, so you can
> get
> > around the firewall issues others have mentioned. I've been using this
> > method on a service in production and haven't had any problems. When you
> > have a separate thread in your client running a Processor you're
> basically
> > blocking on a read, waiting for a message from the server. The benefit of
> > this is that you're notified immediately when the server shuts down
> instead
> > of having to wait until you send a message and then finding out that the
> TCP
> > connection was reset.
> >
> > Cheers,
> > Joel
> >
> >
> >> The biggest issue is that not all client request will result in a single
> >> response (like shooting a bullet, may blowup an entity, and damage all
> >> players in the area those events are seperate messages sent from the
> >> respective entities).
> >>
> >> At a game development studio I used to work at we developed a cross
> >> language
> >> IDL network protocol definition (C++, Java) very similiar to Protocol
> >> Buffers and Thrift (without some of the more mature features like being
> >> transport agnostic we explicitly built it for binary TCP socket
> transport,
> >> or protocol versioning), the stream of packets would contain as the
> first
> >> 32
> >> bits a message ID that would be a key to a map a Message class that
> would
> >> have methods to read in that message type from a byte[] stream.
> >>
> >> Looking through Thrift code in the TBinaryProtocol writeMessage it looks
> >> like it's including the name of the message being sent and it's type (is
> >> the
> >> concept of Message in thrift the same as RPC?), if so what's the
> >> corresponding code pathway for the client waiting for an RPC response
> >> because if I could just use this message name or type to key into what I
> >> need to serialize off the network from both client and server end then
> that
> >> would be perfect.
> >>
> >>
> >>
> >> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <te...@gmail.com>
> >> wrote:
> >>
> >>> I really think that using async service methods which are matched one
> to
> >>> one
> >>> with the message types that you want to send gives you exactly the
> >>> semantics
> >>> that are being requested with very simple implementation cost.
> >>>
> >>> It is important to not get toooo hung up on what RPC stands for. I use
> >>> async methods all the time to stream data structures for logging and it
> >>> works great. Moreover, it provides a really simple way of building
> >>> extractors and processors for this data since I have an interface
> sitting
> >>> there that will tell me about all of the methods (data types) that I
> need
> >>> to
> >>> handle or explicitly ignore.
> >>>
> >>> So the trick works and works really well. Give it a try!
> >>>
> >>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
> >> wrote:
> >>>> Optional fields are not serialized onto the wire. There is a slight
> >>>> performance penalty at serialization time if you have a ton of unset
> >>> fields,
> >>>> but that's it.
> >>>>
> >>>> Am I over complicating things
> >>>> Personally, sounds like it to me. Why do you need this streaming
> >> behavior
> >>>> or whatnot? Hotwiring the rpc stack to let you send any message you
> >> want
> >>> is
> >>>> going to be a ton of work and not really that much of a functionality
> >>>> improvement.
> >>>>
> >>>> -Bryan
> >>>>
> >>>
> >>>
> >>> --
> >>> Ted Dunning, CTO
> >>> DeepDyve
> >>>
>
--
Ted Dunning, CTO
DeepDyve
111 West Evelyn Ave. Ste. 202
Sunnyvale, CA 94086
www.deepdyve.com
858-414-0013 (m)
408-773-0220 (fax)
Re: Using thrift as part of a game network protocol
Posted by Ted Dunning <te...@gmail.com>.
Could this be made less dependent on the world supply of good-will and
optimism?
That is, could there actually be a fully operational Thrift transport that
moves requests both ways on a single socket? You would give it something
that implements the server interface and it would give you back something
that also implements the interface. You could call it and it could call you
in horrifically complicated but very cool ways.
*That* would be something that other RPC packages don't do.
On Thu, Apr 2, 2009 at 4:56 PM, David Reiss <dr...@facebook.com> wrote:
> What an ingenious hack! Cross-posting to thrift-dev in case some
> people don't read this list.
>
> --David
>
> Joel Meyer wrote:
> > On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <daniels.douglas@gmail.com
> >wrote:
> >
> >> Ok I definitely plan on giving the Async RPC methods a try tonight, but
> I
> >> figured I'd just throw out some questions before I get home to start
> >> hacking
> >> on this stuff.
> >>
> >> The one-to-one message to RPC call Async solution will let a client send
> >> messages of any given type in my defined protocol, but how would a
> server
> >> respond to a client with a message that the client didn't request? For
> >> example say I was trying to write a FPS like Quake and I want to server
> to
> >> send position updates for all clients to all clients, how would i model
> >> that
> >> as a client RPC request for that. With the Async RPC solutions I could
> make
> >> a RPC call for Map<Integer, Position> getPositionUpdates(), Now say that
> >> the
> >> client needs to request 50 other messages to be notified of. I guess the
> >> solution would be to make an Async RPC call requesting those updates and
> >> respond to it when I receive it asynchronously and then reissue another
> >> Async RPC call for the next set of updates. It just seems inefficient to
> >> actively make the client request for data when the server could
> implicitly
> >> know that when connected on this game protocol I can just send these
> >> messages to the clients without them asking for it. Not to mention you'd
> >> have make sure you don't "miss" sending a client a message if they
> finished
> >> their Async call but haven't reestablished a new one.
> >>
> >
> > I think I've done something similar to what you're trying to do, and as
> long
> > as you can commit to using only async messages it's possible to pull it
> off
> > without having to start a server on the client to accept RPCs from the
> > server.
> >
> > When your RPC is marked as async the server doesn't send a response and
> the
> > client doesn't try to read one. So, if all your RPC calls from the client
> to
> > the server are async you have effectively freed up the inbound half of
> the
> > socket connection. That means that you can use it for receiving async
> > messages from the server - the only catch is that you have to start a new
> > thread to read and dispatch the incoming async RPC calls.
> >
> > In a typical Thrift RPC system you'd create a MyService.Processor on your
> > server and a MyService.Client on your client. To do bidirectional async
> > message sending you'll need to go a step further and create a
> > MyService.Client on your server for each client that connects (this can
> be
> > accomplished by providing your own TProcessorFactory) and then on each
> > client you create a MyService.Processor. (This assumes that you've gone
> with
> > a generic MyService definition like you described above that has a bunch
> of
> > optional messages, another option would be to define separate service
> > definitions for the client and server.) With two clients connected the
> > objects in existence would look something like this:
> >
> > Server:
> > MyService.Processor mainProcessor - handles incoming async RPCs
> > MyService.Client clientA - used to send outgoing async RPCs to ClientA
> > MyService.Client clientB - used to send outgoing async RPCs to ClientB
> >
> > ClientA:
> > MyService.Client - used to send messages to Server
> > MyService.Processor clientProcessor - used (by a separate thread) to
> > process incoming async RPCs
> >
> > ClientB:
> > MyService.Client - used to send messages to Server
> > MyService.Processor clientProcessor - used (by a separate thread) to
> > process incoming async RPCs
> >
> > Hopefully that explains the concept. If you need example code I can try
> and
> > pull something together (it will be in Java). The nice thing about this
> > method is that you don't have to establish two connections, so you can
> get
> > around the firewall issues others have mentioned. I've been using this
> > method on a service in production and haven't had any problems. When you
> > have a separate thread in your client running a Processor you're
> basically
> > blocking on a read, waiting for a message from the server. The benefit of
> > this is that you're notified immediately when the server shuts down
> instead
> > of having to wait until you send a message and then finding out that the
> TCP
> > connection was reset.
> >
> > Cheers,
> > Joel
> >
> >
> >> The biggest issue is that not all client request will result in a single
> >> response (like shooting a bullet, may blowup an entity, and damage all
> >> players in the area those events are seperate messages sent from the
> >> respective entities).
> >>
> >> At a game development studio I used to work at we developed a cross
> >> language
> >> IDL network protocol definition (C++, Java) very similiar to Protocol
> >> Buffers and Thrift (without some of the more mature features like being
> >> transport agnostic we explicitly built it for binary TCP socket
> transport,
> >> or protocol versioning), the stream of packets would contain as the
> first
> >> 32
> >> bits a message ID that would be a key to a map a Message class that
> would
> >> have methods to read in that message type from a byte[] stream.
> >>
> >> Looking through Thrift code in the TBinaryProtocol writeMessage it looks
> >> like it's including the name of the message being sent and it's type (is
> >> the
> >> concept of Message in thrift the same as RPC?), if so what's the
> >> corresponding code pathway for the client waiting for an RPC response
> >> because if I could just use this message name or type to key into what I
> >> need to serialize off the network from both client and server end then
> that
> >> would be perfect.
> >>
> >>
> >>
> >> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <te...@gmail.com>
> >> wrote:
> >>
> >>> I really think that using async service methods which are matched one
> to
> >>> one
> >>> with the message types that you want to send gives you exactly the
> >>> semantics
> >>> that are being requested with very simple implementation cost.
> >>>
> >>> It is important to not get toooo hung up on what RPC stands for. I use
> >>> async methods all the time to stream data structures for logging and it
> >>> works great. Moreover, it provides a really simple way of building
> >>> extractors and processors for this data since I have an interface
> sitting
> >>> there that will tell me about all of the methods (data types) that I
> need
> >>> to
> >>> handle or explicitly ignore.
> >>>
> >>> So the trick works and works really well. Give it a try!
> >>>
> >>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
> >> wrote:
> >>>> Optional fields are not serialized onto the wire. There is a slight
> >>>> performance penalty at serialization time if you have a ton of unset
> >>> fields,
> >>>> but that's it.
> >>>>
> >>>> Am I over complicating things
> >>>> Personally, sounds like it to me. Why do you need this streaming
> >> behavior
> >>>> or whatnot? Hotwiring the rpc stack to let you send any message you
> >> want
> >>> is
> >>>> going to be a ton of work and not really that much of a functionality
> >>>> improvement.
> >>>>
> >>>> -Bryan
> >>>>
> >>>
> >>>
> >>> --
> >>> Ted Dunning, CTO
> >>> DeepDyve
> >>>
>
--
Ted Dunning, CTO
DeepDyve
111 West Evelyn Ave. Ste. 202
Sunnyvale, CA 94086
www.deepdyve.com
858-414-0013 (m)
408-773-0220 (fax)
Re: Using thrift as part of a game network protocol
Posted by David Reiss <dr...@facebook.com>.
What an ingenious hack! Cross-posting to thrift-dev in case some
people don't read this list.
--David
Joel Meyer wrote:
> On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <da...@gmail.com>wrote:
>
>> Ok I definitely plan on giving the Async RPC methods a try tonight, but I
>> figured I'd just throw out some questions before I get home to start
>> hacking
>> on this stuff.
>>
>> The one-to-one message to RPC call Async solution will let a client send
>> messages of any given type in my defined protocol, but how would a server
>> respond to a client with a message that the client didn't request? For
>> example say I was trying to write a FPS like Quake and I want to server to
>> send position updates for all clients to all clients, how would i model
>> that
>> as a client RPC request for that. With the Async RPC solutions I could make
>> a RPC call for Map<Integer, Position> getPositionUpdates(), Now say that
>> the
>> client needs to request 50 other messages to be notified of. I guess the
>> solution would be to make an Async RPC call requesting those updates and
>> respond to it when I receive it asynchronously and then reissue another
>> Async RPC call for the next set of updates. It just seems inefficient to
>> actively make the client request for data when the server could implicitly
>> know that when connected on this game protocol I can just send these
>> messages to the clients without them asking for it. Not to mention you'd
>> have make sure you don't "miss" sending a client a message if they finished
>> their Async call but haven't reestablished a new one.
>>
>
> I think I've done something similar to what you're trying to do, and as long
> as you can commit to using only async messages it's possible to pull it off
> without having to start a server on the client to accept RPCs from the
> server.
>
> When your RPC is marked as async the server doesn't send a response and the
> client doesn't try to read one. So, if all your RPC calls from the client to
> the server are async you have effectively freed up the inbound half of the
> socket connection. That means that you can use it for receiving async
> messages from the server - the only catch is that you have to start a new
> thread to read and dispatch the incoming async RPC calls.
>
> In a typical Thrift RPC system you'd create a MyService.Processor on your
> server and a MyService.Client on your client. To do bidirectional async
> message sending you'll need to go a step further and create a
> MyService.Client on your server for each client that connects (this can be
> accomplished by providing your own TProcessorFactory) and then on each
> client you create a MyService.Processor. (This assumes that you've gone with
> a generic MyService definition like you described above that has a bunch of
> optional messages, another option would be to define separate service
> definitions for the client and server.) With two clients connected the
> objects in existence would look something like this:
>
> Server:
> MyService.Processor mainProcessor - handles incoming async RPCs
> MyService.Client clientA - used to send outgoing async RPCs to ClientA
> MyService.Client clientB - used to send outgoing async RPCs to ClientB
>
> ClientA:
> MyService.Client - used to send messages to Server
> MyService.Processor clientProcessor - used (by a separate thread) to
> process incoming async RPCs
>
> ClientB:
> MyService.Client - used to send messages to Server
> MyService.Processor clientProcessor - used (by a separate thread) to
> process incoming async RPCs
>
> Hopefully that explains the concept. If you need example code I can try and
> pull something together (it will be in Java). The nice thing about this
> method is that you don't have to establish two connections, so you can get
> around the firewall issues others have mentioned. I've been using this
> method on a service in production and haven't had any problems. When you
> have a separate thread in your client running a Processor you're basically
> blocking on a read, waiting for a message from the server. The benefit of
> this is that you're notified immediately when the server shuts down instead
> of having to wait until you send a message and then finding out that the TCP
> connection was reset.
>
> Cheers,
> Joel
>
>
>> The biggest issue is that not all client request will result in a single
>> response (like shooting a bullet, may blowup an entity, and damage all
>> players in the area those events are seperate messages sent from the
>> respective entities).
>>
>> At a game development studio I used to work at we developed a cross
>> language
>> IDL network protocol definition (C++, Java) very similiar to Protocol
>> Buffers and Thrift (without some of the more mature features like being
>> transport agnostic we explicitly built it for binary TCP socket transport,
>> or protocol versioning), the stream of packets would contain as the first
>> 32
>> bits a message ID that would be a key to a map a Message class that would
>> have methods to read in that message type from a byte[] stream.
>>
>> Looking through Thrift code in the TBinaryProtocol writeMessage it looks
>> like it's including the name of the message being sent and it's type (is
>> the
>> concept of Message in thrift the same as RPC?), if so what's the
>> corresponding code pathway for the client waiting for an RPC response
>> because if I could just use this message name or type to key into what I
>> need to serialize off the network from both client and server end then that
>> would be perfect.
>>
>>
>>
>> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>>
>>> I really think that using async service methods which are matched one to
>>> one
>>> with the message types that you want to send gives you exactly the
>>> semantics
>>> that are being requested with very simple implementation cost.
>>>
>>> It is important to not get toooo hung up on what RPC stands for. I use
>>> async methods all the time to stream data structures for logging and it
>>> works great. Moreover, it provides a really simple way of building
>>> extractors and processors for this data since I have an interface sitting
>>> there that will tell me about all of the methods (data types) that I need
>>> to
>>> handle or explicitly ignore.
>>>
>>> So the trick works and works really well. Give it a try!
>>>
>>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
>> wrote:
>>>> Optional fields are not serialized onto the wire. There is a slight
>>>> performance penalty at serialization time if you have a ton of unset
>>> fields,
>>>> but that's it.
>>>>
>>>> Am I over complicating things
>>>> Personally, sounds like it to me. Why do you need this streaming
>> behavior
>>>> or whatnot? Hotwiring the rpc stack to let you send any message you
>> want
>>> is
>>>> going to be a ton of work and not really that much of a functionality
>>>> improvement.
>>>>
>>>> -Bryan
>>>>
>>>
>>>
>>> --
>>> Ted Dunning, CTO
>>> DeepDyve
>>>
Re: Using thrift as part of a game network protocol
Posted by Ted Dunning <te...@gmail.com>.
Give the man the award for "Best Use of Optimism in a Professional Setting"!
Cool hack!
Scary, too!
On Thu, Apr 2, 2009 at 4:17 PM, Joel Meyer <jo...@gmail.com> wrote:
> and as long as you can commit to using only async messages it's possible to
> pull it off without having to start a server on the client to accept RPCs
> from the
> server.
>
> ... freed up the inbound half of the socket connection. That means that you
> can use it for receiving async messages from the server - the only catch is
> that you have to start a new
> thread to read and dispatch the incoming async RPC calls.
>
Re: Using thrift as part of a game network protocol
Posted by David Reiss <dr...@facebook.com>.
What an ingenious hack! Cross-posting to thrift-dev in case some
people don't read this list.
--David
Joel Meyer wrote:
> On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <da...@gmail.com>wrote:
>
>> Ok I definitely plan on giving the Async RPC methods a try tonight, but I
>> figured I'd just throw out some questions before I get home to start
>> hacking
>> on this stuff.
>>
>> The one-to-one message to RPC call Async solution will let a client send
>> messages of any given type in my defined protocol, but how would a server
>> respond to a client with a message that the client didn't request? For
>> example say I was trying to write a FPS like Quake and I want to server to
>> send position updates for all clients to all clients, how would i model
>> that
>> as a client RPC request for that. With the Async RPC solutions I could make
>> a RPC call for Map<Integer, Position> getPositionUpdates(), Now say that
>> the
>> client needs to request 50 other messages to be notified of. I guess the
>> solution would be to make an Async RPC call requesting those updates and
>> respond to it when I receive it asynchronously and then reissue another
>> Async RPC call for the next set of updates. It just seems inefficient to
>> actively make the client request for data when the server could implicitly
>> know that when connected on this game protocol I can just send these
>> messages to the clients without them asking for it. Not to mention you'd
>> have make sure you don't "miss" sending a client a message if they finished
>> their Async call but haven't reestablished a new one.
>>
>
> I think I've done something similar to what you're trying to do, and as long
> as you can commit to using only async messages it's possible to pull it off
> without having to start a server on the client to accept RPCs from the
> server.
>
> When your RPC is marked as async the server doesn't send a response and the
> client doesn't try to read one. So, if all your RPC calls from the client to
> the server are async you have effectively freed up the inbound half of the
> socket connection. That means that you can use it for receiving async
> messages from the server - the only catch is that you have to start a new
> thread to read and dispatch the incoming async RPC calls.
>
> In a typical Thrift RPC system you'd create a MyService.Processor on your
> server and a MyService.Client on your client. To do bidirectional async
> message sending you'll need to go a step further and create a
> MyService.Client on your server for each client that connects (this can be
> accomplished by providing your own TProcessorFactory) and then on each
> client you create a MyService.Processor. (This assumes that you've gone with
> a generic MyService definition like you described above that has a bunch of
> optional messages, another option would be to define separate service
> definitions for the client and server.) With two clients connected the
> objects in existence would look something like this:
>
> Server:
> MyService.Processor mainProcessor - handles incoming async RPCs
> MyService.Client clientA - used to send outgoing async RPCs to ClientA
> MyService.Client clientB - used to send outgoing async RPCs to ClientB
>
> ClientA:
> MyService.Client - used to send messages to Server
> MyService.Processor clientProcessor - used (by a separate thread) to
> process incoming async RPCs
>
> ClientB:
> MyService.Client - used to send messages to Server
> MyService.Processor clientProcessor - used (by a separate thread) to
> process incoming async RPCs
>
> Hopefully that explains the concept. If you need example code I can try and
> pull something together (it will be in Java). The nice thing about this
> method is that you don't have to establish two connections, so you can get
> around the firewall issues others have mentioned. I've been using this
> method on a service in production and haven't had any problems. When you
> have a separate thread in your client running a Processor you're basically
> blocking on a read, waiting for a message from the server. The benefit of
> this is that you're notified immediately when the server shuts down instead
> of having to wait until you send a message and then finding out that the TCP
> connection was reset.
>
> Cheers,
> Joel
>
>
>> The biggest issue is that not all client request will result in a single
>> response (like shooting a bullet, may blowup an entity, and damage all
>> players in the area those events are seperate messages sent from the
>> respective entities).
>>
>> At a game development studio I used to work at we developed a cross
>> language
>> IDL network protocol definition (C++, Java) very similiar to Protocol
>> Buffers and Thrift (without some of the more mature features like being
>> transport agnostic we explicitly built it for binary TCP socket transport,
>> or protocol versioning), the stream of packets would contain as the first
>> 32
>> bits a message ID that would be a key to a map a Message class that would
>> have methods to read in that message type from a byte[] stream.
>>
>> Looking through Thrift code in the TBinaryProtocol writeMessage it looks
>> like it's including the name of the message being sent and it's type (is
>> the
>> concept of Message in thrift the same as RPC?), if so what's the
>> corresponding code pathway for the client waiting for an RPC response
>> because if I could just use this message name or type to key into what I
>> need to serialize off the network from both client and server end then that
>> would be perfect.
>>
>>
>>
>> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>>
>>> I really think that using async service methods which are matched one to
>>> one
>>> with the message types that you want to send gives you exactly the
>>> semantics
>>> that are being requested with very simple implementation cost.
>>>
>>> It is important to not get toooo hung up on what RPC stands for. I use
>>> async methods all the time to stream data structures for logging and it
>>> works great. Moreover, it provides a really simple way of building
>>> extractors and processors for this data since I have an interface sitting
>>> there that will tell me about all of the methods (data types) that I need
>>> to
>>> handle or explicitly ignore.
>>>
>>> So the trick works and works really well. Give it a try!
>>>
>>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
>> wrote:
>>>> Optional fields are not serialized onto the wire. There is a slight
>>>> performance penalty at serialization time if you have a ton of unset
>>> fields,
>>>> but that's it.
>>>>
>>>> Am I over complicating things
>>>> Personally, sounds like it to me. Why do you need this streaming
>> behavior
>>>> or whatnot? Hotwiring the rpc stack to let you send any message you
>> want
>>> is
>>>> going to be a ton of work and not really that much of a functionality
>>>> improvement.
>>>>
>>>> -Bryan
>>>>
>>>
>>>
>>> --
>>> Ted Dunning, CTO
>>> DeepDyve
>>>
Re: Using thrift as part of a game network protocol
Posted by xu xiong <xi...@gmail.com>.
> In a typical Thrift RPC system you'd create a MyService.Processor on your
> server and a MyService.Client on your client. To do bidirectional async
> message sending you'll need to go a step further and create a
> MyService.Client on your server for each client that connects (this can be
> accomplished by providing your own TProcessorFactory) and then on each
> client you create a MyService.Processor. (This assumes that you've gone with
> a generic MyService definition like you described above that has a bunch of
> optional messages, another option would be to define separate service
> definitions for the client and server.) With two clients connected the
> objects in existence would look something like this:
I want to follow Joel's example to implement it in python, but there's
no TProcessorFactory in the thrift python library.
How can I write to the client connection?
-Xiong
Re: Using thrift as part of a game network protocol
Posted by Doug Daniels <da...@gmail.com>.
The bi-directional single socket RPC is exactly what I'm looking for! The
idea of using async messages is probably the most efficient way to just
deliver a message, and then wait on the client's RPC TProcessor receive side
for messages from the server. I also think it'd be beneficial to support
full synchronous RPC calls client to server and server to client not just
async ones.
At first I thought it would be great to send/receive any type of Thrift
struct over a socket and have application code sort out what type it's
handling, but in the end if you either package an "ALL" message with each
message as an optional field and a type indicator, or if you can
send/receive arbitrary message structs you still deal with the issue of
dispatching those messages once your appliction receives it. The dispatching
part is essentially what a well-defined RPC interface allows you to do
without having to write code to dispatch it you just write the code to
actually process it. If you wanted to support the send/receive "any" message
you could just declare a one-to-one message to async RPC service method.
For the client/server games I'm imagining I think being able to scale to 20k
concurrent would be fine. Low latency and fast response times is more
important than handling more concurrent connections in a "fair" manner for
most games, although a connect send/receive then disconnect might be
acceptable for some games and you could queue messages needing to be
delivered to "disconnected" clients until they poll again, very similiar to
the AJAX COMETD long-polling idea (http://cometdproject.dojotoolkit.org/),
COMETD can be used as AJAX style Sockets.
I began hacking some of the bi-directional RPC myself, my first thought was
to allow regular synchronous (and async) RPC client to server calls, but to
support server to client calls I'd do something like this:
I'd create a TProtocol that would wrap a TTransport, something like a
BidirectionalProtocolAdapter (it wouldn't directly implement TProtocol). The
BidirectionalProtocolAdapter would expose a TSendRPCProtocol and a
TReceiveRPCProtocol that would split the messages being read as follows:
- TClientSendProtocol
- would use the oproto to send RPC messages of type (TMessageType.CALL)
to the server
- readMessage() would only read messages of type (TMessageType.REPLY,
and TMessageType.EXCEPTION)
- TServerReceiveProtocol
- would use the oproto to send messages of type (REPLY and Exception)
- readMessage() would only read messages of type (CALL) and dispatch it to
a TProcessor implementing the appropriate service
Some issues/difficulties I see with this are:
- Only one of the client or server protocols could receive Exception
messages (it makes sense that the RPC Client receives exceptions is it
possible for a Thrift client to send an RPC server an Exception message?)
- The BidirectionalProtocolAdapter will probably need to use wait() and
notify() to allow the Client and Server protocols to read the correct
message type.
- Client and Server are tightly coupled to each other instead of just the
client knowing the server's RPC services the server must know the client's
RPC services
On Fri, Apr 3, 2009 at 10:48 PM, Will Lowe <wi...@gmail.com> wrote:
> I'm actually considering using Thrift in a similar way: as a fast,
> cross-language serialization-and-transport mechanism between a bunch of
> different apps in a pub/sub architecture.
>
> There are a number of possible message types -- 100? -- and it won't always
> be possible for each consumer to know if the other supports a given message
> type, so I'd like to avoid each app having an RPC service for each possible
> message type; I'd rather hand them objects and let them just ignore the
> ones they don't care about.
>
> I see a few different ways to do this:
>
> 1. Define only one struct with all possible fields for all possible
> messages, and a "type" field that lets you figure out what it is. It seems
> kinda stupid to do this, since one of the major reasons I'm interested in
> Thrift is type-awareness.
>
> 2. Modify the TService layer so that RPC arguments aren't statically typed:
> make it possible to declare an RPC call that accepts any struct. Feels
> like (void*), and probably also irritates purists who like the simple
> no-object-inheritance, no-function-overloading model Thrift uses today.
>
> 3. Build something custom at the TProcessor layer and skip TService
> altogether.
>
> Both #2 and #3 require changing some guts. I think that would go something
> like this:
>
> * Add a TMessageType (T_STRUCT?) that indicates "I'm sending you data, not
> calling a function!". Or is that what T_ONEWAY is for?
> * Modify TBinaryProtocol so that writeStructBegin() and writeStructEnd()
> aren't noops -- otherwise the receiver doesn't know what he's receiving!
> * Implement a TProcessor that can read the struct type, instantiate it,
> and do some sort of dispatch to the client app.
>
> Any thoughts on this? Has someone else already solved this?
>
> Will
>
>
>
> On Apr 3, 2009, at 6:22 PM, Brian Hammond wrote:
>
> That's neat Joel. However, does this scale? I mean, the underlying
>> assumption here is that clients are using persistent connections to the
>> service, and you [not so simply] are sending messages back to the client
>> over that same connection. Thus, your service now has to handle a
>> potentially large number of client connections. Unless you're using
>> something libev[ent] I don't see this scaling beyond say 20K connections.
>> Two things, I could be missing something here, and this level of
>> scalability is probably just fine for *many* types of services (perhaps not
>> for a chat server though)!
>>
>> I'm curious to hear other people's thoughts on this and how it could be
>> made scalable since, well, I'm planning on using polling in my project since
>> I am expecting potentially a very large number of simultaneous users of the
>> service and my servers can only handle so many connections.
>>
>> Thanks for sharing this.
>>
>> Brian
>>
>> On Apr 3, 2009, at 7:50 PM, Joel Meyer wrote:
>>
>> On Thu, Apr 2, 2009 at 4:17 PM, Joel Meyer <jo...@gmail.com> wrote:
>>>
>>> On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <
>>>> daniels.douglas@gmail.com>wrote:
>>>>
>>>> Ok I definitely plan on giving the Async RPC methods a try tonight, but
>>>>> I
>>>>> figured I'd just throw out some questions before I get home to start
>>>>> hacking
>>>>> on this stuff.
>>>>>
>>>>> The one-to-one message to RPC call Async solution will let a client
>>>>> send
>>>>> messages of any given type in my defined protocol, but how would a
>>>>> server
>>>>> respond to a client with a message that the client didn't request? For
>>>>> example say I was trying to write a FPS like Quake and I want to server
>>>>> to
>>>>> send position updates for all clients to all clients, how would i model
>>>>> that
>>>>> as a client RPC request for that. With the Async RPC solutions I could
>>>>> make
>>>>> a RPC call for Map<Integer, Position> getPositionUpdates(), Now say
>>>>> that
>>>>> the
>>>>> client needs to request 50 other messages to be notified of. I guess
>>>>> the
>>>>> solution would be to make an Async RPC call requesting those updates
>>>>> and
>>>>> respond to it when I receive it asynchronously and then reissue another
>>>>> Async RPC call for the next set of updates. It just seems inefficient
>>>>> to
>>>>> actively make the client request for data when the server could
>>>>> implicitly
>>>>> know that when connected on this game protocol I can just send these
>>>>> messages to the clients without them asking for it. Not to mention
>>>>> you'd
>>>>> have make sure you don't "miss" sending a client a message if they
>>>>> finished
>>>>> their Async call but haven't reestablished a new one.
>>>>>
>>>>>
>>>> I think I've done something similar to what you're trying to do, and as
>>>> long as you can commit to using only async messages it's possible to
>>>> pull it
>>>> off without having to start a server on the client to accept RPCs from
>>>> the
>>>> server.
>>>>
>>>> When your RPC is marked as async the server doesn't send a response and
>>>> the
>>>> client doesn't try to read one. So, if all your RPC calls from the
>>>> client to
>>>> the server are async you have effectively freed up the inbound half of
>>>> the
>>>> socket connection. That means that you can use it for receiving async
>>>> messages from the server - the only catch is that you have to start a
>>>> new
>>>> thread to read and dispatch the incoming async RPC calls.
>>>>
>>>> In a typical Thrift RPC system you'd create a MyService.Processor on
>>>> your
>>>> server and a MyService.Client on your client. To do bidirectional async
>>>> message sending you'll need to go a step further and create a
>>>> MyService.Client on your server for each client that connects (this can
>>>> be
>>>> accomplished by providing your own TProcessorFactory) and then on each
>>>> client you create a MyService.Processor. (This assumes that you've gone
>>>> with
>>>> a generic MyService definition like you described above that has a bunch
>>>> of
>>>> optional messages, another option would be to define separate service
>>>> definitions for the client and server.) With two clients connected the
>>>> objects in existence would look something like this:
>>>>
>>>> Server:
>>>> MyService.Processor mainProcessor - handles incoming async RPCs
>>>> MyService.Client clientA - used to send outgoing async RPCs to ClientA
>>>> MyService.Client clientB - used to send outgoing async RPCs to ClientB
>>>>
>>>> ClientA:
>>>> MyService.Client - used to send messages to Server
>>>> MyService.Processor clientProcessor - used (by a separate thread) to
>>>> process incoming async RPCs
>>>>
>>>> ClientB:
>>>> MyService.Client - used to send messages to Server
>>>> MyService.Processor clientProcessor - used (by a separate thread) to
>>>> process incoming async RPCs
>>>>
>>>> Hopefully that explains the concept. If you need example code I can try
>>>> and
>>>> pull something together (it will be in Java). The nice thing about this
>>>> method is that you don't have to establish two connections, so you can
>>>> get
>>>> around the firewall issues others have mentioned. I've been using this
>>>> method on a service in production and haven't had any problems. When you
>>>> have a separate thread in your client running a Processor you're
>>>> basically
>>>> blocking on a read, waiting for a message from the server. The benefit
>>>> of
>>>> this is that you're notified immediately when the server shuts down
>>>> instead
>>>> of having to wait until you send a message and then finding out that the
>>>> TCP
>>>> connection was reset.
>>>>
>>>> Cheers,
>>>> Joel
>>>>
>>>>
>>> Thanks for the feedback. I've created a simple example in Java
>>> demonstrating
>>> this in action:
>>> http://www.joelpm.com/wp-content/uploads/2009/04/bidimessages.tgz
>>>
>>> Post with a few details on the implementation:
>>> http://www.joelpm.com/2009/04/03/thrift-bidirectional-async-rpc/
>>>
>>> Please add me to the list of people who think there's value in a full
>>> async
>>> transport that provides (optional?) synchronization at the api level
>>> using
>>> futures/deferreds/etc.
>>>
>>> Cheers,
>>> Joel
>>>
>>>
>>>
>>>>
>>>>
>>>>> The biggest issue is that not all client request will result in a
>>>>> single
>>>>> response (like shooting a bullet, may blowup an entity, and damage all
>>>>> players in the area those events are seperate messages sent from the
>>>>> respective entities).
>>>>>
>>>>> At a game development studio I used to work at we developed a cross
>>>>> language
>>>>> IDL network protocol definition (C++, Java) very similiar to Protocol
>>>>> Buffers and Thrift (without some of the more mature features like being
>>>>> transport agnostic we explicitly built it for binary TCP socket
>>>>> transport,
>>>>> or protocol versioning), the stream of packets would contain as the
>>>>> first
>>>>> 32
>>>>> bits a message ID that would be a key to a map a Message class that
>>>>> would
>>>>> have methods to read in that message type from a byte[] stream.
>>>>>
>>>>> Looking through Thrift code in the TBinaryProtocol writeMessage it
>>>>> looks
>>>>> like it's including the name of the message being sent and it's type
>>>>> (is
>>>>> the
>>>>> concept of Message in thrift the same as RPC?), if so what's the
>>>>> corresponding code pathway for the client waiting for an RPC response
>>>>> because if I could just use this message name or type to key into what
>>>>> I
>>>>> need to serialize off the network from both client and server end then
>>>>> that
>>>>> would be perfect.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <te...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> I really think that using async service methods which are matched one
>>>>>> to
>>>>>> one
>>>>>> with the message types that you want to send gives you exactly the
>>>>>> semantics
>>>>>> that are being requested with very simple implementation cost.
>>>>>>
>>>>>> It is important to not get toooo hung up on what RPC stands for. I
>>>>>> use
>>>>>> async methods all the time to stream data structures for logging and
>>>>>> it
>>>>>> works great. Moreover, it provides a really simple way of building
>>>>>> extractors and processors for this data since I have an interface
>>>>>>
>>>>> sitting
>>>>>
>>>>>> there that will tell me about all of the methods (data types) that I
>>>>>>
>>>>> need
>>>>>
>>>>>> to
>>>>>> handle or explicitly ignore.
>>>>>>
>>>>>> So the trick works and works really well. Give it a try!
>>>>>>
>>>>>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
>>>>>>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Optional fields are not serialized onto the wire. There is a slight
>>>>>>> performance penalty at serialization time if you have a ton of unset
>>>>>>>
>>>>>> fields,
>>>>>>
>>>>>>> but that's it.
>>>>>>>
>>>>>>> Am I over complicating things
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Personally, sounds like it to me. Why do you need this streaming
>>>>>>>
>>>>>> behavior
>>>>>
>>>>>> or whatnot? Hotwiring the rpc stack to let you send any message you
>>>>>>>
>>>>>> want
>>>>>
>>>>>> is
>>>>>>
>>>>>>> going to be a ton of work and not really that much of a functionality
>>>>>>> improvement.
>>>>>>>
>>>>>>> -Bryan
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ted Dunning, CTO
>>>>>> DeepDyve
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>
>
Re: Using thrift as part of a game network protocol
Posted by Doug Daniels <da...@gmail.com>.
The bi-directional single socket RPC is exactly what I'm looking for! The
idea of using async messages is probably the most efficient way to just
deliver a message, and then wait on the client's RPC TProcessor receive side
for messages from the server. I also think it'd be beneficial to support
full synchronous RPC calls client to server and server to client not just
async ones.
At first I thought it would be great to send/receive any type of Thrift
struct over a socket and have application code sort out what type it's
handling, but in the end if you either package an "ALL" message with each
message as an optional field and a type indicator, or if you can
send/receive arbitrary message structs you still deal with the issue of
dispatching those messages once your appliction receives it. The dispatching
part is essentially what a well-defined RPC interface allows you to do
without having to write code to dispatch it you just write the code to
actually process it. If you wanted to support the send/receive "any" message
you could just declare a one-to-one message to async RPC service method.
For the client/server games I'm imagining I think being able to scale to 20k
concurrent would be fine. Low latency and fast response times is more
important than handling more concurrent connections in a "fair" manner for
most games, although a connect send/receive then disconnect might be
acceptable for some games and you could queue messages needing to be
delivered to "disconnected" clients until they poll again, very similiar to
the AJAX COMETD long-polling idea (http://cometdproject.dojotoolkit.org/),
COMETD can be used as AJAX style Sockets.
I began hacking some of the bi-directional RPC myself, my first thought was
to allow regular synchronous (and async) RPC client to server calls, but to
support server to client calls I'd do something like this:
I'd create a TProtocol that would wrap a TTransport, something like a
BidirectionalProtocolAdapter (it wouldn't directly implement TProtocol). The
BidirectionalProtocolAdapter would expose a TSendRPCProtocol and a
TReceiveRPCProtocol that would split the messages being read as follows:
- TClientSendProtocol
- would use the oproto to send RPC messages of type (TMessageType.CALL)
to the server
- readMessage() would only read messages of type (TMessageType.REPLY,
and TMessageType.EXCEPTION)
- TServerReceiveProtocol
- would use the oproto to send messages of type (REPLY and Exception)
- readMessage() would only read messages of type (CALL) and dispatch it to
a TProcessor implementing the appropriate service
Some issues/difficulties I see with this are:
- Only one of the client or server protocols could receive Exception
messages (it makes sense that the RPC Client receives exceptions is it
possible for a Thrift client to send an RPC server an Exception message?)
- The BidirectionalProtocolAdapter will probably need to use wait() and
notify() to allow the Client and Server protocols to read the correct
message type.
- Client and Server are tightly coupled to each other instead of just the
client knowing the server's RPC services the server must know the client's
RPC services
On Fri, Apr 3, 2009 at 10:48 PM, Will Lowe <wi...@gmail.com> wrote:
> I'm actually considering using Thrift in a similar way: as a fast,
> cross-language serialization-and-transport mechanism between a bunch of
> different apps in a pub/sub architecture.
>
> There are a number of possible message types -- 100? -- and it won't always
> be possible for each consumer to know if the other supports a given message
> type, so I'd like to avoid each app having an RPC service for each possible
> message type; I'd rather hand them objects and let them just ignore the
> ones they don't care about.
>
> I see a few different ways to do this:
>
> 1. Define only one struct with all possible fields for all possible
> messages, and a "type" field that lets you figure out what it is. It seems
> kinda stupid to do this, since one of the major reasons I'm interested in
> Thrift is type-awareness.
>
> 2. Modify the TService layer so that RPC arguments aren't statically typed:
> make it possible to declare an RPC call that accepts any struct. Feels
> like (void*), and probably also irritates purists who like the simple
> no-object-inheritance, no-function-overloading model Thrift uses today.
>
> 3. Build something custom at the TProcessor layer and skip TService
> altogether.
>
> Both #2 and #3 require changing some guts. I think that would go something
> like this:
>
> * Add a TMessageType (T_STRUCT?) that indicates "I'm sending you data, not
> calling a function!". Or is that what T_ONEWAY is for?
> * Modify TBinaryProtocol so that writeStructBegin() and writeStructEnd()
> aren't noops -- otherwise the receiver doesn't know what he's receiving!
> * Implement a TProcessor that can read the struct type, instantiate it,
> and do some sort of dispatch to the client app.
>
> Any thoughts on this? Has someone else already solved this?
>
> Will
>
>
>
> On Apr 3, 2009, at 6:22 PM, Brian Hammond wrote:
>
> That's neat Joel. However, does this scale? I mean, the underlying
>> assumption here is that clients are using persistent connections to the
>> service, and you [not so simply] are sending messages back to the client
>> over that same connection. Thus, your service now has to handle a
>> potentially large number of client connections. Unless you're using
>> something libev[ent] I don't see this scaling beyond say 20K connections.
>> Two things, I could be missing something here, and this level of
>> scalability is probably just fine for *many* types of services (perhaps not
>> for a chat server though)!
>>
>> I'm curious to hear other people's thoughts on this and how it could be
>> made scalable since, well, I'm planning on using polling in my project since
>> I am expecting potentially a very large number of simultaneous users of the
>> service and my servers can only handle so many connections.
>>
>> Thanks for sharing this.
>>
>> Brian
>>
>> On Apr 3, 2009, at 7:50 PM, Joel Meyer wrote:
>>
>> On Thu, Apr 2, 2009 at 4:17 PM, Joel Meyer <jo...@gmail.com> wrote:
>>>
>>> On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <
>>>> daniels.douglas@gmail.com>wrote:
>>>>
>>>> Ok I definitely plan on giving the Async RPC methods a try tonight, but
>>>>> I
>>>>> figured I'd just throw out some questions before I get home to start
>>>>> hacking
>>>>> on this stuff.
>>>>>
>>>>> The one-to-one message to RPC call Async solution will let a client
>>>>> send
>>>>> messages of any given type in my defined protocol, but how would a
>>>>> server
>>>>> respond to a client with a message that the client didn't request? For
>>>>> example say I was trying to write a FPS like Quake and I want to server
>>>>> to
>>>>> send position updates for all clients to all clients, how would i model
>>>>> that
>>>>> as a client RPC request for that. With the Async RPC solutions I could
>>>>> make
>>>>> a RPC call for Map<Integer, Position> getPositionUpdates(), Now say
>>>>> that
>>>>> the
>>>>> client needs to request 50 other messages to be notified of. I guess
>>>>> the
>>>>> solution would be to make an Async RPC call requesting those updates
>>>>> and
>>>>> respond to it when I receive it asynchronously and then reissue another
>>>>> Async RPC call for the next set of updates. It just seems inefficient
>>>>> to
>>>>> actively make the client request for data when the server could
>>>>> implicitly
>>>>> know that when connected on this game protocol I can just send these
>>>>> messages to the clients without them asking for it. Not to mention
>>>>> you'd
>>>>> have make sure you don't "miss" sending a client a message if they
>>>>> finished
>>>>> their Async call but haven't reestablished a new one.
>>>>>
>>>>>
>>>> I think I've done something similar to what you're trying to do, and as
>>>> long as you can commit to using only async messages it's possible to
>>>> pull it
>>>> off without having to start a server on the client to accept RPCs from
>>>> the
>>>> server.
>>>>
>>>> When your RPC is marked as async the server doesn't send a response and
>>>> the
>>>> client doesn't try to read one. So, if all your RPC calls from the
>>>> client to
>>>> the server are async you have effectively freed up the inbound half of
>>>> the
>>>> socket connection. That means that you can use it for receiving async
>>>> messages from the server - the only catch is that you have to start a
>>>> new
>>>> thread to read and dispatch the incoming async RPC calls.
>>>>
>>>> In a typical Thrift RPC system you'd create a MyService.Processor on
>>>> your
>>>> server and a MyService.Client on your client. To do bidirectional async
>>>> message sending you'll need to go a step further and create a
>>>> MyService.Client on your server for each client that connects (this can
>>>> be
>>>> accomplished by providing your own TProcessorFactory) and then on each
>>>> client you create a MyService.Processor. (This assumes that you've gone
>>>> with
>>>> a generic MyService definition like you described above that has a bunch
>>>> of
>>>> optional messages, another option would be to define separate service
>>>> definitions for the client and server.) With two clients connected the
>>>> objects in existence would look something like this:
>>>>
>>>> Server:
>>>> MyService.Processor mainProcessor - handles incoming async RPCs
>>>> MyService.Client clientA - used to send outgoing async RPCs to ClientA
>>>> MyService.Client clientB - used to send outgoing async RPCs to ClientB
>>>>
>>>> ClientA:
>>>> MyService.Client - used to send messages to Server
>>>> MyService.Processor clientProcessor - used (by a separate thread) to
>>>> process incoming async RPCs
>>>>
>>>> ClientB:
>>>> MyService.Client - used to send messages to Server
>>>> MyService.Processor clientProcessor - used (by a separate thread) to
>>>> process incoming async RPCs
>>>>
>>>> Hopefully that explains the concept. If you need example code I can try
>>>> and
>>>> pull something together (it will be in Java). The nice thing about this
>>>> method is that you don't have to establish two connections, so you can
>>>> get
>>>> around the firewall issues others have mentioned. I've been using this
>>>> method on a service in production and haven't had any problems. When you
>>>> have a separate thread in your client running a Processor you're
>>>> basically
>>>> blocking on a read, waiting for a message from the server. The benefit
>>>> of
>>>> this is that you're notified immediately when the server shuts down
>>>> instead
>>>> of having to wait until you send a message and then finding out that the
>>>> TCP
>>>> connection was reset.
>>>>
>>>> Cheers,
>>>> Joel
>>>>
>>>>
>>> Thanks for the feedback. I've created a simple example in Java
>>> demonstrating
>>> this in action:
>>> http://www.joelpm.com/wp-content/uploads/2009/04/bidimessages.tgz
>>>
>>> Post with a few details on the implementation:
>>> http://www.joelpm.com/2009/04/03/thrift-bidirectional-async-rpc/
>>>
>>> Please add me to the list of people who think there's value in a full
>>> async
>>> transport that provides (optional?) synchronization at the api level
>>> using
>>> futures/deferreds/etc.
>>>
>>> Cheers,
>>> Joel
>>>
>>>
>>>
>>>>
>>>>
>>>>> The biggest issue is that not all client request will result in a
>>>>> single
>>>>> response (like shooting a bullet, may blowup an entity, and damage all
>>>>> players in the area those events are seperate messages sent from the
>>>>> respective entities).
>>>>>
>>>>> At a game development studio I used to work at we developed a cross
>>>>> language
>>>>> IDL network protocol definition (C++, Java) very similiar to Protocol
>>>>> Buffers and Thrift (without some of the more mature features like being
>>>>> transport agnostic we explicitly built it for binary TCP socket
>>>>> transport,
>>>>> or protocol versioning), the stream of packets would contain as the
>>>>> first
>>>>> 32
>>>>> bits a message ID that would be a key to a map a Message class that
>>>>> would
>>>>> have methods to read in that message type from a byte[] stream.
>>>>>
>>>>> Looking through Thrift code in the TBinaryProtocol writeMessage it
>>>>> looks
>>>>> like it's including the name of the message being sent and it's type
>>>>> (is
>>>>> the
>>>>> concept of Message in thrift the same as RPC?), if so what's the
>>>>> corresponding code pathway for the client waiting for an RPC response
>>>>> because if I could just use this message name or type to key into what
>>>>> I
>>>>> need to serialize off the network from both client and server end then
>>>>> that
>>>>> would be perfect.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <te...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> I really think that using async service methods which are matched one
>>>>>> to
>>>>>> one
>>>>>> with the message types that you want to send gives you exactly the
>>>>>> semantics
>>>>>> that are being requested with very simple implementation cost.
>>>>>>
>>>>>> It is important to not get toooo hung up on what RPC stands for. I
>>>>>> use
>>>>>> async methods all the time to stream data structures for logging and
>>>>>> it
>>>>>> works great. Moreover, it provides a really simple way of building
>>>>>> extractors and processors for this data since I have an interface
>>>>>>
>>>>> sitting
>>>>>
>>>>>> there that will tell me about all of the methods (data types) that I
>>>>>>
>>>>> need
>>>>>
>>>>>> to
>>>>>> handle or explicitly ignore.
>>>>>>
>>>>>> So the trick works and works really well. Give it a try!
>>>>>>
>>>>>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
>>>>>>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Optional fields are not serialized onto the wire. There is a slight
>>>>>>> performance penalty at serialization time if you have a ton of unset
>>>>>>>
>>>>>> fields,
>>>>>>
>>>>>>> but that's it.
>>>>>>>
>>>>>>> Am I over complicating things
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Personally, sounds like it to me. Why do you need this streaming
>>>>>>>
>>>>>> behavior
>>>>>
>>>>>> or whatnot? Hotwiring the rpc stack to let you send any message you
>>>>>>>
>>>>>> want
>>>>>
>>>>>> is
>>>>>>
>>>>>>> going to be a ton of work and not really that much of a functionality
>>>>>>> improvement.
>>>>>>>
>>>>>>> -Bryan
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ted Dunning, CTO
>>>>>> DeepDyve
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>
>
Re: Using thrift as part of a game network protocol
Posted by Will Lowe <wi...@gmail.com>.
I'm actually considering using Thrift in a similar way: as a fast,
cross-language serialization-and-transport mechanism between a bunch
of different apps in a pub/sub architecture.
There are a number of possible message types -- 100? -- and it won't
always be possible for each consumer to know if the other supports a
given message type, so I'd like to avoid each app having an RPC
service for each possible message type; I'd rather hand them objects
and let them just ignore the ones they don't care about.
I see a few different ways to do this:
1. Define only one struct with all possible fields for all possible
messages, and a "type" field that lets you figure out what it is. It
seems kinda stupid to do this, since one of the major reasons I'm
interested in Thrift is type-awareness.
2. Modify the TService layer so that RPC arguments aren't statically
typed: make it possible to declare an RPC call that accepts any
struct. Feels like (void*), and probably also irritates purists who
like the simple no-object-inheritance, no-function-overloading model
Thrift uses today.
3. Build something custom at the TProcessor layer and skip TService
altogether.
Both #2 and #3 require changing some guts. I think that would go
something like this:
* Add a TMessageType (T_STRUCT?) that indicates "I'm sending you data,
not calling a function!". Or is that what T_ONEWAY is for?
* Modify TBinaryProtocol so that writeStructBegin() and
writeStructEnd() aren't noops -- otherwise the receiver doesn't know
what he's receiving!
* Implement a TProcessor that can read the struct type, instantiate
it, and do some sort of dispatch to the client app.
Any thoughts on this? Has someone else already solved this?
Will
On Apr 3, 2009, at 6:22 PM, Brian Hammond wrote:
> That's neat Joel. However, does this scale? I mean, the underlying
> assumption here is that clients are using persistent connections to
> the service, and you [not so simply] are sending messages back to
> the client over that same connection. Thus, your service now has to
> handle a potentially large number of client connections. Unless
> you're using something libev[ent] I don't see this scaling beyond
> say 20K connections. Two things, I could be missing something here,
> and this level of scalability is probably just fine for *many* types
> of services (perhaps not for a chat server though)!
>
> I'm curious to hear other people's thoughts on this and how it could
> be made scalable since, well, I'm planning on using polling in my
> project since I am expecting potentially a very large number of
> simultaneous users of the service and my servers can only handle so
> many connections.
>
> Thanks for sharing this.
>
> Brian
>
> On Apr 3, 2009, at 7:50 PM, Joel Meyer wrote:
>
>> On Thu, Apr 2, 2009 at 4:17 PM, Joel Meyer <jo...@gmail.com>
>> wrote:
>>
>>> On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <daniels.douglas@gmail.com
>>> >wrote:
>>>
>>>> Ok I definitely plan on giving the Async RPC methods a try
>>>> tonight, but I
>>>> figured I'd just throw out some questions before I get home to
>>>> start
>>>> hacking
>>>> on this stuff.
>>>>
>>>> The one-to-one message to RPC call Async solution will let a
>>>> client send
>>>> messages of any given type in my defined protocol, but how would
>>>> a server
>>>> respond to a client with a message that the client didn't
>>>> request? For
>>>> example say I was trying to write a FPS like Quake and I want to
>>>> server to
>>>> send position updates for all clients to all clients, how would i
>>>> model
>>>> that
>>>> as a client RPC request for that. With the Async RPC solutions I
>>>> could
>>>> make
>>>> a RPC call for Map<Integer, Position> getPositionUpdates(), Now
>>>> say that
>>>> the
>>>> client needs to request 50 other messages to be notified of. I
>>>> guess the
>>>> solution would be to make an Async RPC call requesting those
>>>> updates and
>>>> respond to it when I receive it asynchronously and then reissue
>>>> another
>>>> Async RPC call for the next set of updates. It just seems
>>>> inefficient to
>>>> actively make the client request for data when the server could
>>>> implicitly
>>>> know that when connected on this game protocol I can just send
>>>> these
>>>> messages to the clients without them asking for it. Not to
>>>> mention you'd
>>>> have make sure you don't "miss" sending a client a message if they
>>>> finished
>>>> their Async call but haven't reestablished a new one.
>>>>
>>>
>>> I think I've done something similar to what you're trying to do,
>>> and as
>>> long as you can commit to using only async messages it's possible
>>> to pull it
>>> off without having to start a server on the client to accept RPCs
>>> from the
>>> server.
>>>
>>> When your RPC is marked as async the server doesn't send a
>>> response and the
>>> client doesn't try to read one. So, if all your RPC calls from the
>>> client to
>>> the server are async you have effectively freed up the inbound
>>> half of the
>>> socket connection. That means that you can use it for receiving
>>> async
>>> messages from the server - the only catch is that you have to
>>> start a new
>>> thread to read and dispatch the incoming async RPC calls.
>>>
>>> In a typical Thrift RPC system you'd create a MyService.Processor
>>> on your
>>> server and a MyService.Client on your client. To do bidirectional
>>> async
>>> message sending you'll need to go a step further and create a
>>> MyService.Client on your server for each client that connects
>>> (this can be
>>> accomplished by providing your own TProcessorFactory) and then on
>>> each
>>> client you create a MyService.Processor. (This assumes that you've
>>> gone with
>>> a generic MyService definition like you described above that has a
>>> bunch of
>>> optional messages, another option would be to define separate
>>> service
>>> definitions for the client and server.) With two clients connected
>>> the
>>> objects in existence would look something like this:
>>>
>>> Server:
>>> MyService.Processor mainProcessor - handles incoming async RPCs
>>> MyService.Client clientA - used to send outgoing async RPCs to
>>> ClientA
>>> MyService.Client clientB - used to send outgoing async RPCs to
>>> ClientB
>>>
>>> ClientA:
>>> MyService.Client - used to send messages to Server
>>> MyService.Processor clientProcessor - used (by a separate thread) to
>>> process incoming async RPCs
>>>
>>> ClientB:
>>> MyService.Client - used to send messages to Server
>>> MyService.Processor clientProcessor - used (by a separate thread) to
>>> process incoming async RPCs
>>>
>>> Hopefully that explains the concept. If you need example code I
>>> can try and
>>> pull something together (it will be in Java). The nice thing about
>>> this
>>> method is that you don't have to establish two connections, so you
>>> can get
>>> around the firewall issues others have mentioned. I've been using
>>> this
>>> method on a service in production and haven't had any problems.
>>> When you
>>> have a separate thread in your client running a Processor you're
>>> basically
>>> blocking on a read, waiting for a message from the server. The
>>> benefit of
>>> this is that you're notified immediately when the server shuts
>>> down instead
>>> of having to wait until you send a message and then finding out
>>> that the TCP
>>> connection was reset.
>>>
>>> Cheers,
>>> Joel
>>>
>>
>> Thanks for the feedback. I've created a simple example in Java
>> demonstrating
>> this in action:
>> http://www.joelpm.com/wp-content/uploads/2009/04/bidimessages.tgz
>>
>> Post with a few details on the implementation:
>> http://www.joelpm.com/2009/04/03/thrift-bidirectional-async-rpc/
>>
>> Please add me to the list of people who think there's value in a
>> full async
>> transport that provides (optional?) synchronization at the api
>> level using
>> futures/deferreds/etc.
>>
>> Cheers,
>> Joel
>>
>>
>>>
>>>
>>>>
>>>> The biggest issue is that not all client request will result in a
>>>> single
>>>> response (like shooting a bullet, may blowup an entity, and
>>>> damage all
>>>> players in the area those events are seperate messages sent from
>>>> the
>>>> respective entities).
>>>>
>>>> At a game development studio I used to work at we developed a cross
>>>> language
>>>> IDL network protocol definition (C++, Java) very similiar to
>>>> Protocol
>>>> Buffers and Thrift (without some of the more mature features like
>>>> being
>>>> transport agnostic we explicitly built it for binary TCP socket
>>>> transport,
>>>> or protocol versioning), the stream of packets would contain as
>>>> the first
>>>> 32
>>>> bits a message ID that would be a key to a map a Message class
>>>> that would
>>>> have methods to read in that message type from a byte[] stream.
>>>>
>>>> Looking through Thrift code in the TBinaryProtocol writeMessage
>>>> it looks
>>>> like it's including the name of the message being sent and it's
>>>> type (is
>>>> the
>>>> concept of Message in thrift the same as RPC?), if so what's the
>>>> corresponding code pathway for the client waiting for an RPC
>>>> response
>>>> because if I could just use this message name or type to key into
>>>> what I
>>>> need to serialize off the network from both client and server end
>>>> then
>>>> that
>>>> would be perfect.
>>>>
>>>>
>>>>
>>>> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning
>>>> <te...@gmail.com>
>>>> wrote:
>>>>
>>>>> I really think that using async service methods which are
>>>>> matched one to
>>>>> one
>>>>> with the message types that you want to send gives you exactly the
>>>>> semantics
>>>>> that are being requested with very simple implementation cost.
>>>>>
>>>>> It is important to not get toooo hung up on what RPC stands
>>>>> for. I use
>>>>> async methods all the time to stream data structures for logging
>>>>> and it
>>>>> works great. Moreover, it provides a really simple way of
>>>>> building
>>>>> extractors and processors for this data since I have an interface
>>>> sitting
>>>>> there that will tell me about all of the methods (data types)
>>>>> that I
>>>> need
>>>>> to
>>>>> handle or explicitly ignore.
>>>>>
>>>>> So the trick works and works really well. Give it a try!
>>>>>
>>>>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
>>>> wrote:
>>>>>
>>>>>> Optional fields are not serialized onto the wire. There is a
>>>>>> slight
>>>>>> performance penalty at serialization time if you have a ton of
>>>>>> unset
>>>>> fields,
>>>>>> but that's it.
>>>>>>
>>>>>> Am I over complicating things
>>>>>>>
>>>>>>
>>>>>> Personally, sounds like it to me. Why do you need this streaming
>>>> behavior
>>>>>> or whatnot? Hotwiring the rpc stack to let you send any message
>>>>>> you
>>>> want
>>>>> is
>>>>>> going to be a ton of work and not really that much of a
>>>>>> functionality
>>>>>> improvement.
>>>>>>
>>>>>> -Bryan
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ted Dunning, CTO
>>>>> DeepDyve
>>>>>
>>>>
>>>
>>>
>
Re: Using thrift as part of a game network protocol
Posted by Joel Meyer <jo...@gmail.com>.
Hi Brian, responses inline below:
On Fri, Apr 3, 2009 at 6:22 PM, Brian Hammond <br...@brianhammond.com>wrote:
> That's neat Joel. However, does this scale? I mean, the underlying
> assumption here is that clients are using persistent connections to the
> service, and you [not so simply] are sending messages back to the client
> over that same connection. Thus, your service now has to handle a
> potentially large number of client connections. Unless you're using
> something libev[ent] I don't see this scaling beyond say 20K connections.
> Two things, I could be missing something here, and this level of
> scalability is probably just fine for *many* types of services (perhaps not
> for a chat server though)!
>
The code I've shared won't scale nearly as well as a kqueue/epoll/select
based implementation because it requires a thread on the server for every
client. Another problem with my example is that every client gets every
message, and when a message is sent the sending is done using the thread
handling requests from the client that sent the original message. Some of
this could be avoided by using the THaHsServer (I think that's the class
name, not positive) which is select/NIO based (in Java) and wouldn't require
a new thread for every client. Calling it a chat server was done more to
give a quick idea of what the code does - I'd definitely NOT recommend
designing your chat server this way :)
The concept that I was trying to demonstrate (sending messages both ways
using a single socket), however, should scale just fine - at least as well
as having socket connections in both directions. The number of connections a
single server can handle will be determined more by the workload required of
the server for each received message and how active each client is. If the
clients are dormant a majority of the time a libev[ent] based server should
scale quite well.
>
> I'm curious to hear other people's thoughts on this and how it could be
> made scalable since, well, I'm planning on using polling in my project since
> I am expecting potentially a very large number of simultaneous users of the
> service and my servers can only handle so many connections.
Me too, it'd be great to hear thoughts/experiences from other people who are
scaling Thrift based RPC servers.
Cheers,
Joel
>
>
> Thanks for sharing this.
>
> Brian
>
>
> On Apr 3, 2009, at 7:50 PM, Joel Meyer wrote:
>
> On Thu, Apr 2, 2009 at 4:17 PM, Joel Meyer <jo...@gmail.com> wrote:
>>
>> On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <daniels.douglas@gmail.com
>>> >wrote:
>>>
>>> Ok I definitely plan on giving the Async RPC methods a try tonight, but
>>>> I
>>>> figured I'd just throw out some questions before I get home to start
>>>> hacking
>>>> on this stuff.
>>>>
>>>> The one-to-one message to RPC call Async solution will let a client send
>>>> messages of any given type in my defined protocol, but how would a
>>>> server
>>>> respond to a client with a message that the client didn't request? For
>>>> example say I was trying to write a FPS like Quake and I want to server
>>>> to
>>>> send position updates for all clients to all clients, how would i model
>>>> that
>>>> as a client RPC request for that. With the Async RPC solutions I could
>>>> make
>>>> a RPC call for Map<Integer, Position> getPositionUpdates(), Now say that
>>>> the
>>>> client needs to request 50 other messages to be notified of. I guess the
>>>> solution would be to make an Async RPC call requesting those updates and
>>>> respond to it when I receive it asynchronously and then reissue another
>>>> Async RPC call for the next set of updates. It just seems inefficient to
>>>> actively make the client request for data when the server could
>>>> implicitly
>>>> know that when connected on this game protocol I can just send these
>>>> messages to the clients without them asking for it. Not to mention you'd
>>>> have make sure you don't "miss" sending a client a message if they
>>>> finished
>>>> their Async call but haven't reestablished a new one.
>>>>
>>>>
>>> I think I've done something similar to what you're trying to do, and as
>>> long as you can commit to using only async messages it's possible to pull
>>> it
>>> off without having to start a server on the client to accept RPCs from
>>> the
>>> server.
>>>
>>> When your RPC is marked as async the server doesn't send a response and
>>> the
>>> client doesn't try to read one. So, if all your RPC calls from the client
>>> to
>>> the server are async you have effectively freed up the inbound half of
>>> the
>>> socket connection. That means that you can use it for receiving async
>>> messages from the server - the only catch is that you have to start a new
>>> thread to read and dispatch the incoming async RPC calls.
>>>
>>> In a typical Thrift RPC system you'd create a MyService.Processor on your
>>> server and a MyService.Client on your client. To do bidirectional async
>>> message sending you'll need to go a step further and create a
>>> MyService.Client on your server for each client that connects (this can
>>> be
>>> accomplished by providing your own TProcessorFactory) and then on each
>>> client you create a MyService.Processor. (This assumes that you've gone
>>> with
>>> a generic MyService definition like you described above that has a bunch
>>> of
>>> optional messages, another option would be to define separate service
>>> definitions for the client and server.) With two clients connected the
>>> objects in existence would look something like this:
>>>
>>> Server:
>>> MyService.Processor mainProcessor - handles incoming async RPCs
>>> MyService.Client clientA - used to send outgoing async RPCs to ClientA
>>> MyService.Client clientB - used to send outgoing async RPCs to ClientB
>>>
>>> ClientA:
>>> MyService.Client - used to send messages to Server
>>> MyService.Processor clientProcessor - used (by a separate thread) to
>>> process incoming async RPCs
>>>
>>> ClientB:
>>> MyService.Client - used to send messages to Server
>>> MyService.Processor clientProcessor - used (by a separate thread) to
>>> process incoming async RPCs
>>>
>>> Hopefully that explains the concept. If you need example code I can try
>>> and
>>> pull something together (it will be in Java). The nice thing about this
>>> method is that you don't have to establish two connections, so you can
>>> get
>>> around the firewall issues others have mentioned. I've been using this
>>> method on a service in production and haven't had any problems. When you
>>> have a separate thread in your client running a Processor you're
>>> basically
>>> blocking on a read, waiting for a message from the server. The benefit of
>>> this is that you're notified immediately when the server shuts down
>>> instead
>>> of having to wait until you send a message and then finding out that the
>>> TCP
>>> connection was reset.
>>>
>>> Cheers,
>>> Joel
>>>
>>>
>> Thanks for the feedback. I've created a simple example in Java
>> demonstrating
>> this in action:
>> http://www.joelpm.com/wp-content/uploads/2009/04/bidimessages.tgz
>>
>> Post with a few details on the implementation:
>> http://www.joelpm.com/2009/04/03/thrift-bidirectional-async-rpc/
>>
>> Please add me to the list of people who think there's value in a full
>> async
>> transport that provides (optional?) synchronization at the api level using
>> futures/deferreds/etc.
>>
>> Cheers,
>> Joel
>>
>>
>>
>>>
>>>
>>>> The biggest issue is that not all client request will result in a single
>>>> response (like shooting a bullet, may blowup an entity, and damage all
>>>> players in the area those events are seperate messages sent from the
>>>> respective entities).
>>>>
>>>> At a game development studio I used to work at we developed a cross
>>>> language
>>>> IDL network protocol definition (C++, Java) very similiar to Protocol
>>>> Buffers and Thrift (without some of the more mature features like being
>>>> transport agnostic we explicitly built it for binary TCP socket
>>>> transport,
>>>> or protocol versioning), the stream of packets would contain as the
>>>> first
>>>> 32
>>>> bits a message ID that would be a key to a map a Message class that
>>>> would
>>>> have methods to read in that message type from a byte[] stream.
>>>>
>>>> Looking through Thrift code in the TBinaryProtocol writeMessage it looks
>>>> like it's including the name of the message being sent and it's type (is
>>>> the
>>>> concept of Message in thrift the same as RPC?), if so what's the
>>>> corresponding code pathway for the client waiting for an RPC response
>>>> because if I could just use this message name or type to key into what I
>>>> need to serialize off the network from both client and server end then
>>>> that
>>>> would be perfect.
>>>>
>>>>
>>>>
>>>> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <te...@gmail.com>
>>>> wrote:
>>>>
>>>> I really think that using async service methods which are matched one
>>>>> to
>>>>> one
>>>>> with the message types that you want to send gives you exactly the
>>>>> semantics
>>>>> that are being requested with very simple implementation cost.
>>>>>
>>>>> It is important to not get toooo hung up on what RPC stands for. I use
>>>>> async methods all the time to stream data structures for logging and it
>>>>> works great. Moreover, it provides a really simple way of building
>>>>> extractors and processors for this data since I have an interface
>>>>>
>>>> sitting
>>>>
>>>>> there that will tell me about all of the methods (data types) that I
>>>>>
>>>> need
>>>>
>>>>> to
>>>>> handle or explicitly ignore.
>>>>>
>>>>> So the trick works and works really well. Give it a try!
>>>>>
>>>>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
>>>>>
>>>> wrote:
>>>>
>>>>>
>>>>> Optional fields are not serialized onto the wire. There is a slight
>>>>>> performance penalty at serialization time if you have a ton of unset
>>>>>>
>>>>> fields,
>>>>>
>>>>>> but that's it.
>>>>>>
>>>>>> Am I over complicating things
>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Personally, sounds like it to me. Why do you need this streaming
>>>>>>
>>>>> behavior
>>>>
>>>>> or whatnot? Hotwiring the rpc stack to let you send any message you
>>>>>>
>>>>> want
>>>>
>>>>> is
>>>>>
>>>>>> going to be a ton of work and not really that much of a functionality
>>>>>> improvement.
>>>>>>
>>>>>> -Bryan
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ted Dunning, CTO
>>>>> DeepDyve
>>>>>
>>>>>
>>>>
>>>
>>>
>
Re: Using thrift as part of a game network protocol
Posted by Brian Hammond <br...@brianhammond.com>.
That's neat Joel. However, does this scale? I mean, the underlying
assumption here is that clients are using persistent connections to
the service, and you [not so simply] are sending messages back to the
client over that same connection. Thus, your service now has to
handle a potentially large number of client connections. Unless
you're using something libev[ent] I don't see this scaling beyond say
20K connections. Two things, I could be missing something here, and
this level of scalability is probably just fine for *many* types of
services (perhaps not for a chat server though)!
I'm curious to hear other people's thoughts on this and how it could
be made scalable since, well, I'm planning on using polling in my
project since I am expecting potentially a very large number of
simultaneous users of the service and my servers can only handle so
many connections.
Thanks for sharing this.
Brian
On Apr 3, 2009, at 7:50 PM, Joel Meyer wrote:
> On Thu, Apr 2, 2009 at 4:17 PM, Joel Meyer <jo...@gmail.com>
> wrote:
>
>> On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <daniels.douglas@gmail.com
>> >wrote:
>>
>>> Ok I definitely plan on giving the Async RPC methods a try
>>> tonight, but I
>>> figured I'd just throw out some questions before I get home to start
>>> hacking
>>> on this stuff.
>>>
>>> The one-to-one message to RPC call Async solution will let a
>>> client send
>>> messages of any given type in my defined protocol, but how would a
>>> server
>>> respond to a client with a message that the client didn't request?
>>> For
>>> example say I was trying to write a FPS like Quake and I want to
>>> server to
>>> send position updates for all clients to all clients, how would i
>>> model
>>> that
>>> as a client RPC request for that. With the Async RPC solutions I
>>> could
>>> make
>>> a RPC call for Map<Integer, Position> getPositionUpdates(), Now
>>> say that
>>> the
>>> client needs to request 50 other messages to be notified of. I
>>> guess the
>>> solution would be to make an Async RPC call requesting those
>>> updates and
>>> respond to it when I receive it asynchronously and then reissue
>>> another
>>> Async RPC call for the next set of updates. It just seems
>>> inefficient to
>>> actively make the client request for data when the server could
>>> implicitly
>>> know that when connected on this game protocol I can just send these
>>> messages to the clients without them asking for it. Not to mention
>>> you'd
>>> have make sure you don't "miss" sending a client a message if they
>>> finished
>>> their Async call but haven't reestablished a new one.
>>>
>>
>> I think I've done something similar to what you're trying to do,
>> and as
>> long as you can commit to using only async messages it's possible
>> to pull it
>> off without having to start a server on the client to accept RPCs
>> from the
>> server.
>>
>> When your RPC is marked as async the server doesn't send a response
>> and the
>> client doesn't try to read one. So, if all your RPC calls from the
>> client to
>> the server are async you have effectively freed up the inbound half
>> of the
>> socket connection. That means that you can use it for receiving async
>> messages from the server - the only catch is that you have to start
>> a new
>> thread to read and dispatch the incoming async RPC calls.
>>
>> In a typical Thrift RPC system you'd create a MyService.Processor
>> on your
>> server and a MyService.Client on your client. To do bidirectional
>> async
>> message sending you'll need to go a step further and create a
>> MyService.Client on your server for each client that connects (this
>> can be
>> accomplished by providing your own TProcessorFactory) and then on
>> each
>> client you create a MyService.Processor. (This assumes that you've
>> gone with
>> a generic MyService definition like you described above that has a
>> bunch of
>> optional messages, another option would be to define separate service
>> definitions for the client and server.) With two clients connected
>> the
>> objects in existence would look something like this:
>>
>> Server:
>> MyService.Processor mainProcessor - handles incoming async RPCs
>> MyService.Client clientA - used to send outgoing async RPCs to
>> ClientA
>> MyService.Client clientB - used to send outgoing async RPCs to
>> ClientB
>>
>> ClientA:
>> MyService.Client - used to send messages to Server
>> MyService.Processor clientProcessor - used (by a separate thread) to
>> process incoming async RPCs
>>
>> ClientB:
>> MyService.Client - used to send messages to Server
>> MyService.Processor clientProcessor - used (by a separate thread) to
>> process incoming async RPCs
>>
>> Hopefully that explains the concept. If you need example code I can
>> try and
>> pull something together (it will be in Java). The nice thing about
>> this
>> method is that you don't have to establish two connections, so you
>> can get
>> around the firewall issues others have mentioned. I've been using
>> this
>> method on a service in production and haven't had any problems.
>> When you
>> have a separate thread in your client running a Processor you're
>> basically
>> blocking on a read, waiting for a message from the server. The
>> benefit of
>> this is that you're notified immediately when the server shuts down
>> instead
>> of having to wait until you send a message and then finding out
>> that the TCP
>> connection was reset.
>>
>> Cheers,
>> Joel
>>
>
> Thanks for the feedback. I've created a simple example in Java
> demonstrating
> this in action:
> http://www.joelpm.com/wp-content/uploads/2009/04/bidimessages.tgz
>
> Post with a few details on the implementation:
> http://www.joelpm.com/2009/04/03/thrift-bidirectional-async-rpc/
>
> Please add me to the list of people who think there's value in a
> full async
> transport that provides (optional?) synchronization at the api level
> using
> futures/deferreds/etc.
>
> Cheers,
> Joel
>
>
>>
>>
>>>
>>> The biggest issue is that not all client request will result in a
>>> single
>>> response (like shooting a bullet, may blowup an entity, and damage
>>> all
>>> players in the area those events are seperate messages sent from the
>>> respective entities).
>>>
>>> At a game development studio I used to work at we developed a cross
>>> language
>>> IDL network protocol definition (C++, Java) very similiar to
>>> Protocol
>>> Buffers and Thrift (without some of the more mature features like
>>> being
>>> transport agnostic we explicitly built it for binary TCP socket
>>> transport,
>>> or protocol versioning), the stream of packets would contain as
>>> the first
>>> 32
>>> bits a message ID that would be a key to a map a Message class
>>> that would
>>> have methods to read in that message type from a byte[] stream.
>>>
>>> Looking through Thrift code in the TBinaryProtocol writeMessage it
>>> looks
>>> like it's including the name of the message being sent and it's
>>> type (is
>>> the
>>> concept of Message in thrift the same as RPC?), if so what's the
>>> corresponding code pathway for the client waiting for an RPC
>>> response
>>> because if I could just use this message name or type to key into
>>> what I
>>> need to serialize off the network from both client and server end
>>> then
>>> that
>>> would be perfect.
>>>
>>>
>>>
>>> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <te...@gmail.com>
>>> wrote:
>>>
>>>> I really think that using async service methods which are matched
>>>> one to
>>>> one
>>>> with the message types that you want to send gives you exactly the
>>>> semantics
>>>> that are being requested with very simple implementation cost.
>>>>
>>>> It is important to not get toooo hung up on what RPC stands for.
>>>> I use
>>>> async methods all the time to stream data structures for logging
>>>> and it
>>>> works great. Moreover, it provides a really simple way of building
>>>> extractors and processors for this data since I have an interface
>>> sitting
>>>> there that will tell me about all of the methods (data types)
>>>> that I
>>> need
>>>> to
>>>> handle or explicitly ignore.
>>>>
>>>> So the trick works and works really well. Give it a try!
>>>>
>>>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
>>> wrote:
>>>>
>>>>> Optional fields are not serialized onto the wire. There is a
>>>>> slight
>>>>> performance penalty at serialization time if you have a ton of
>>>>> unset
>>>> fields,
>>>>> but that's it.
>>>>>
>>>>> Am I over complicating things
>>>>>>
>>>>>
>>>>> Personally, sounds like it to me. Why do you need this streaming
>>> behavior
>>>>> or whatnot? Hotwiring the rpc stack to let you send any message
>>>>> you
>>> want
>>>> is
>>>>> going to be a ton of work and not really that much of a
>>>>> functionality
>>>>> improvement.
>>>>>
>>>>> -Bryan
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ted Dunning, CTO
>>>> DeepDyve
>>>>
>>>
>>
>>
Re: Using thrift as part of a game network protocol
Posted by Joel Meyer <jo...@gmail.com>.
On Thu, Apr 2, 2009 at 4:17 PM, Joel Meyer <jo...@gmail.com> wrote:
> On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <da...@gmail.com>wrote:
>
>> Ok I definitely plan on giving the Async RPC methods a try tonight, but I
>> figured I'd just throw out some questions before I get home to start
>> hacking
>> on this stuff.
>>
>> The one-to-one message to RPC call Async solution will let a client send
>> messages of any given type in my defined protocol, but how would a server
>> respond to a client with a message that the client didn't request? For
>> example say I was trying to write a FPS like Quake and I want to server to
>> send position updates for all clients to all clients, how would i model
>> that
>> as a client RPC request for that. With the Async RPC solutions I could
>> make
>> a RPC call for Map<Integer, Position> getPositionUpdates(), Now say that
>> the
>> client needs to request 50 other messages to be notified of. I guess the
>> solution would be to make an Async RPC call requesting those updates and
>> respond to it when I receive it asynchronously and then reissue another
>> Async RPC call for the next set of updates. It just seems inefficient to
>> actively make the client request for data when the server could implicitly
>> know that when connected on this game protocol I can just send these
>> messages to the clients without them asking for it. Not to mention you'd
>> have make sure you don't "miss" sending a client a message if they
>> finished
>> their Async call but haven't reestablished a new one.
>>
>
> I think I've done something similar to what you're trying to do, and as
> long as you can commit to using only async messages it's possible to pull it
> off without having to start a server on the client to accept RPCs from the
> server.
>
> When your RPC is marked as async the server doesn't send a response and the
> client doesn't try to read one. So, if all your RPC calls from the client to
> the server are async you have effectively freed up the inbound half of the
> socket connection. That means that you can use it for receiving async
> messages from the server - the only catch is that you have to start a new
> thread to read and dispatch the incoming async RPC calls.
>
> In a typical Thrift RPC system you'd create a MyService.Processor on your
> server and a MyService.Client on your client. To do bidirectional async
> message sending you'll need to go a step further and create a
> MyService.Client on your server for each client that connects (this can be
> accomplished by providing your own TProcessorFactory) and then on each
> client you create a MyService.Processor. (This assumes that you've gone with
> a generic MyService definition like you described above that has a bunch of
> optional messages, another option would be to define separate service
> definitions for the client and server.) With two clients connected the
> objects in existence would look something like this:
>
> Server:
> MyService.Processor mainProcessor - handles incoming async RPCs
> MyService.Client clientA - used to send outgoing async RPCs to ClientA
> MyService.Client clientB - used to send outgoing async RPCs to ClientB
>
> ClientA:
> MyService.Client - used to send messages to Server
> MyService.Processor clientProcessor - used (by a separate thread) to
> process incoming async RPCs
>
> ClientB:
> MyService.Client - used to send messages to Server
> MyService.Processor clientProcessor - used (by a separate thread) to
> process incoming async RPCs
>
> Hopefully that explains the concept. If you need example code I can try and
> pull something together (it will be in Java). The nice thing about this
> method is that you don't have to establish two connections, so you can get
> around the firewall issues others have mentioned. I've been using this
> method on a service in production and haven't had any problems. When you
> have a separate thread in your client running a Processor you're basically
> blocking on a read, waiting for a message from the server. The benefit of
> this is that you're notified immediately when the server shuts down instead
> of having to wait until you send a message and then finding out that the TCP
> connection was reset.
>
> Cheers,
> Joel
>
Thanks for the feedback. I've created a simple example in Java demonstrating
this in action:
http://www.joelpm.com/wp-content/uploads/2009/04/bidimessages.tgz
Post with a few details on the implementation:
http://www.joelpm.com/2009/04/03/thrift-bidirectional-async-rpc/
Please add me to the list of people who think there's value in a full async
transport that provides (optional?) synchronization at the api level using
futures/deferreds/etc.
Cheers,
Joel
>
>
>>
>> The biggest issue is that not all client request will result in a single
>> response (like shooting a bullet, may blowup an entity, and damage all
>> players in the area those events are seperate messages sent from the
>> respective entities).
>>
>> At a game development studio I used to work at we developed a cross
>> language
>> IDL network protocol definition (C++, Java) very similiar to Protocol
>> Buffers and Thrift (without some of the more mature features like being
>> transport agnostic we explicitly built it for binary TCP socket transport,
>> or protocol versioning), the stream of packets would contain as the first
>> 32
>> bits a message ID that would be a key to a map a Message class that would
>> have methods to read in that message type from a byte[] stream.
>>
>> Looking through Thrift code in the TBinaryProtocol writeMessage it looks
>> like it's including the name of the message being sent and it's type (is
>> the
>> concept of Message in thrift the same as RPC?), if so what's the
>> corresponding code pathway for the client waiting for an RPC response
>> because if I could just use this message name or type to key into what I
>> need to serialize off the network from both client and server end then
>> that
>> would be perfect.
>>
>>
>>
>> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>>
>> > I really think that using async service methods which are matched one to
>> > one
>> > with the message types that you want to send gives you exactly the
>> > semantics
>> > that are being requested with very simple implementation cost.
>> >
>> > It is important to not get toooo hung up on what RPC stands for. I use
>> > async methods all the time to stream data structures for logging and it
>> > works great. Moreover, it provides a really simple way of building
>> > extractors and processors for this data since I have an interface
>> sitting
>> > there that will tell me about all of the methods (data types) that I
>> need
>> > to
>> > handle or explicitly ignore.
>> >
>> > So the trick works and works really well. Give it a try!
>> >
>> > On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <br...@rapleaf.com>
>> wrote:
>> >
>> > > Optional fields are not serialized onto the wire. There is a slight
>> > > performance penalty at serialization time if you have a ton of unset
>> > fields,
>> > > but that's it.
>> > >
>> > > Am I over complicating things
>> > >>
>> > >
>> > > Personally, sounds like it to me. Why do you need this streaming
>> behavior
>> > > or whatnot? Hotwiring the rpc stack to let you send any message you
>> want
>> > is
>> > > going to be a ton of work and not really that much of a functionality
>> > > improvement.
>> > >
>> > > -Bryan
>> > >
>> >
>> >
>> >
>> > --
>> > Ted Dunning, CTO
>> > DeepDyve
>> >
>>
>
>
Re: Using thrift as part of a game network protocol
Posted by Rush Manbert <ru...@manbert.com>.
That really is awesomely cool. As Ted Dunning says, scary as anything,
but really, really clever!
Best regards and a tip of the hat,
Rush
On Apr 2, 2009, at 4:17 PM, Joel Meyer wrote:
> On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <daniels.douglas@gmail.com
> >wrote:
>
>
<snip>
> I think I've done something similar to what you're trying to do, and
> as long
> as you can commit to using only async messages it's possible to pull
> it off
> without having to start a server on the client to accept RPCs from the
> server.
>
> When your RPC is marked as async the server doesn't send a response
> and the
> client doesn't try to read one. So, if all your RPC calls from the
> client to
> the server are async you have effectively freed up the inbound half
> of the
> socket connection. That means that you can use it for receiving async
> messages from the server - the only catch is that you have to start
> a new
> thread to read and dispatch the incoming async RPC calls.
>
> In a typical Thrift RPC system you'd create a MyService.Processor on
> your
> server and a MyService.Client on your client. To do bidirectional
> async
> message sending you'll need to go a step further and create a
> MyService.Client on your server for each client that connects (this
> can be
> accomplished by providing your own TProcessorFactory) and then on each
> client you create a MyService.Processor. (This assumes that you've
> gone with
> a generic MyService definition like you described above that has a
> bunch of
> optional messages, another option would be to define separate service
> definitions for the client and server.) With two clients connected the
> objects in existence would look something like this:
>
> Server:
> MyService.Processor mainProcessor - handles incoming async RPCs
> MyService.Client clientA - used to send outgoing async RPCs to
> ClientA
> MyService.Client clientB - used to send outgoing async RPCs to
> ClientB
>
> ClientA:
> MyService.Client - used to send messages to Server
> MyService.Processor clientProcessor - used (by a separate thread) to
> process incoming async RPCs
>
> ClientB:
> MyService.Client - used to send messages to Server
> MyService.Processor clientProcessor - used (by a separate thread) to
> process incoming async RPCs
>
> Hopefully that explains the concept. If you need example code I can
> try and
> pull something together (it will be in Java). The nice thing about
> this
> method is that you don't have to establish two connections, so you
> can get
> around the firewall issues others have mentioned. I've been using this
> method on a service in production and haven't had any problems. When
> you
> have a separate thread in your client running a Processor you're
> basically
> blocking on a read, waiting for a message from the server. The
> benefit of
> this is that you're notified immediately when the server shuts down
> instead
> of having to wait until you send a message and then finding out that
> the TCP
> connection was reset.
<snip>