You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Shaun Williams <sh...@gmail.com> on 2011/12/01 20:43:14 UTC
Changes to NettyTransportCodec
Can anyone tell me what the implications are to chaining the transport
codec in the NettyTransportCodec, e.g. adding a additional flag? Do the
rpc implementations in any of the other languages support this codec
currently? Is there anything it might break?
Thanks,
Shaun
Re: Changes to NettyTransportCodec
Posted by Shaun Williams <sh...@apple.com>.
I think this would work for us. We would have to maintain two separate protocols, server and client, which is fine. I'm wondering, however, if we could make the relationship between the two explicit in the protocol definition, e.g. extend messages to support taking a protocol as a parameter (or otherwise attach a response protocol to a message). The getRequestor() method could then return a requestor within the context of a message.
Also, I may be mistaken, but the Netty implementation already supports AVRO-625. It writes a serial id before each request/response, and then stores callbacks to the requestor in a table by the serial id. Once a response is received, the callback is removed from the table.
-Shaun
On Dec 13, 2011, at 9:29 AM, Doug Cutting wrote:
> On 12/12/2011 05:11 PM, Shaun Williams wrote:
>> Our application needs to be able to be able to callback multiple times for a single request.
>
> That doesn't sounds like a traditional call/response pattern.
>
> Perhaps this would better be modeled as one-way messages. Host A would
> send a one-way message to host B indicating its interest in responses,
> then the B would send those responses as one-way messages, terminating
> these with a special end-of-response message. In Avro this would
> currently require two connections, one A->B and another B->A.
>
> But one might extend things so that a server could make requests back to
> the client over the same connection. For example, the metadata that
> precedes requests and responses might include a "Message-Type"
> parameter, with values "Request" and "Response". This might be combined
> with my proposal in AVRO-625 (http://s.apache.org/ASH), so these headers
> are only valid when both client and server also specify "Call-Id" in
> their metadata. In that case the client already needs to dispatch
> responses to callers and it might thus also be extended to dispatch
> requests.
>
> The server-side API might be a static method Responder.getRequestor()
> that can only be called while responding and returns a Requestor for
> calls back to the client. Client-side, one would construct one's
> Requestor with a Responder that it would use to respond to requests from
> the server.
>
> Or maybe its simpler to just use two connections!
>
> Does that make any sense?
>
> Doug
Re: Changes to NettyTransportCodec
Posted by Doug Cutting <cu...@apache.org>.
On 12/12/2011 05:11 PM, Shaun Williams wrote:
> Our application needs to be able to be able to callback multiple times for a single request.
That doesn't sounds like a traditional call/response pattern.
Perhaps this would better be modeled as one-way messages. Host A would
send a one-way message to host B indicating its interest in responses,
then the B would send those responses as one-way messages, terminating
these with a special end-of-response message. In Avro this would
currently require two connections, one A->B and another B->A.
But one might extend things so that a server could make requests back to
the client over the same connection. For example, the metadata that
precedes requests and responses might include a "Message-Type"
parameter, with values "Request" and "Response". This might be combined
with my proposal in AVRO-625 (http://s.apache.org/ASH), so these headers
are only valid when both client and server also specify "Call-Id" in
their metadata. In that case the client already needs to dispatch
responses to callers and it might thus also be extended to dispatch
requests.
The server-side API might be a static method Responder.getRequestor()
that can only be called while responding and returns a Requestor for
calls back to the client. Client-side, one would construct one's
Requestor with a Responder that it would use to respond to requests from
the server.
Or maybe its simpler to just use two connections!
Does that make any sense?
Doug
Re: Changes to NettyTransportCodec
Posted by Shaun Williams <sh...@apple.com>.
Unfortunately, no.
Our application needs to be able to be able to callback multiple times for a single request. To support this we need to 1) propagate callbacks all the way to the server. Currently, though the code generator creates two interfaces for each message (synchronous and asynchronous), the asynchronous version is never called by the Responder. And 2) the server can't remove request callbacks from the requests table (in the NettyTransceiver) until it has received the last response message.
So in our branch, we added 2 flags to each message: isBlockingRequest (i.e. call the blocking implementation instead of the non-blocking implementation) and isLastPacket.
To make the schema specification more intuitive we also added a new type: "callback<ReturnType>", which can be used to indicate that this message is meant to be asynchronous.
I realize this is a fairly big change, and I'm not sure if the community is interested. At any rate I'm preparing a patch now and will be submitting it soon.
I couldn't figure out a way to do this with plugins, but please let me know if you have any ideas.
-Shaun
On Dec 12, 2011, at 1:35 PM, Doug Cutting wrote:
> On 12/01/2011 11:43 AM, Shaun Williams wrote:
>> Can anyone tell me what the implications are to chaining the transport
>> codec in the NettyTransportCodec, e.g. adding a additional flag? Do the
>> rpc implementations in any of the other languages support this codec
>> currently? Is there anything it might break?
>
> I think the Netty transport is compatible with the (deprecated)
> SocketServer and SocketTransceiver. I think most languages include an
> HTTP-based implementation, not a socket-based implementation. The
> preferred socket implementation is SaslSocketTransceiver.
>
> It would be simple to convert the Netty implementation to be compatible
> with SaslSocketTransceiver for the anonymous mechanism:
>
> http://avro.apache.org/docs/current/sasl.html#anonymous
>
> Also, is there a way you might add the flag to the request or response
> metadata, perhaps with an RPC plugin?
>
> Doug
Re: Changes to NettyTransportCodec
Posted by Doug Cutting <cu...@apache.org>.
On 12/01/2011 11:43 AM, Shaun Williams wrote:
> Can anyone tell me what the implications are to chaining the transport
> codec in the NettyTransportCodec, e.g. adding a additional flag? Do the
> rpc implementations in any of the other languages support this codec
> currently? Is there anything it might break?
I think the Netty transport is compatible with the (deprecated)
SocketServer and SocketTransceiver. I think most languages include an
HTTP-based implementation, not a socket-based implementation. The
preferred socket implementation is SaslSocketTransceiver.
It would be simple to convert the Netty implementation to be compatible
with SaslSocketTransceiver for the anonymous mechanism:
http://avro.apache.org/docs/current/sasl.html#anonymous
Also, is there a way you might add the flag to the request or response
metadata, perhaps with an RPC plugin?
Doug