You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Pankaj Shroff <sh...@gmail.com> on 2013/03/12 22:01:38 UTC
Re: Bypassing "handshake" in Responder
Quick update on this issue. I think I finally figured out a way to bypass
Avro style "handshake" when employing "custom" format or content type
implementation but still trying to reuse or benefit from the
Serialize/Deserialize capabilities of Avro. Perhaps the following usage is
the "intended" REUSE use case in Avro. However, what I have below
completely bypasses Avro RPC classes in the avro-ipc package. Let me know
if this still mandates a patch proposal on Jira. Basically - I implemented
my own "Responder-Server" combination without relying on reflection based
method invocation of "classic" Avro. Link to source code below:
https://github.com/openrtb/openrtb2x/blob/2.0/demand-side/dsp-core/src/main/java/org/openrtb/dsp/core/DemandSideServer.java
Pankaj
On Wed, Feb 27, 2013 at 4:38 PM, Pankaj Shroff <sh...@gmail.com> wrote:
> Doug - thanks again.
>
> I agree with you. I have been looking into it for the past few days and it
> seems like this will require quite a bit of refactoring. I will try to
> follow up on Jira with more specific details.
>
> Thanks
> Pankaj
>
>
>
> On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <cu...@apache.org> wrote:
>
>> Pankaj,
>>
>> Avro RPC is currently specified to always uses the binary encoding
>> (like Avro data files). This might reasonably be extended to JSON.
>> First we'd need to specify the new wire format. Probably Avro's
>> framing would not make sense for JSON-encoded RPC over HTTP. Then
>> we'd need to figure what of the existing Java implementation might be
>> reused or adapted. At a glance, it doesn't look to me like a few
>> one-line changes would suffice, adding methods where things are
>> hardwired, that rather more substantial changes would be required, but
>> I might be missing something. If you're interested in pursuing this
>> then please file an issue in Jira where you can propose changes to the
>> specification and implementation.
>>
>> Cheers,
>>
>> Doug
>>
>> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <sh...@gmail.com>
>> wrote:
>> > I guess my question is more basic - given that this is somewhat
>> specific to
>> > my own use case:
>> >
>> > How does one use other forms of Encoder/Decoder implementations that are
>> > available in the Avro library along with the Avro-Ipc SDK.
>> >
>> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc
>> supports is
>> > the BinaryEncoding
>> >
>> > Pankaj
>> >
>> >
>> >
>> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <sh...@gmail.com>
>> wrote:
>> >>
>> >> Doug
>> >>
>> >> Perhaps you answered a portion of my conundrum in another thread
>> >> (permalink below) - but there is still the handshake and reuse of
>> invocation
>> >> logic question. Let me also think about this a little bit.
>> >>
>> >> Thanks in any case. Avro is a great tool in any case!
>> >>
>> >>
>> >>
>> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
>> >>
>> >>
>> >> Pankaj
>> >>
>> >> shroffg@gmail.com
>> >>
>> >>
>> >>
>> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cu...@apache.org>
>> wrote:
>> >>>
>> >>> This sounds like a different RPC wire format than Avro's. Avro's
>> >>> Requestor and Responder implement Avro's RPC wire format. Avro's
>> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
>> >>> implementation of other RPC wire formats that include Avro data.
>> >>> Avro's Transceiver API may or may not be reusable, since it assumes
>> >>> Avro-style framing. Parts of Requestor and Responder *might* be
>> >>> reusable and some refactoring of those classes *might* make such reuse
>> >>> easier, but there's not that much logic there that's not specific to
>> >>> Avro's wire format, so it might be just as easy to reimplement this
>> >>> layer for a different wire format. It's hard for me to say without
>> >>> seeing a patch with a proposed refactoring. Does that make sense?
>> >>>
>> >>> Doug
>> >>>
>> >>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <sh...@gmail.com>
>> wrote:
>> >>> > Hi
>> >>> >
>> >>> > We are using Avro for implementing an open source reference
>> >>> > implementation
>> >>> > of the OpenRTB protocol.
>> >>> >
>> >>> > We have made a design goal to model the protocol using Avro protocol
>> >>> > files
>> >>> > (.avpr) and generate types defined in the protocol schema using
>> Avro .
>> >>> > The
>> >>> > challenge is that the protocol does not necessarily require the use
>> of
>> >>> > Avro/
>> >>> > Binary wire encoding - or even the use of Avro/ RPC context. In fact
>> >>> > many
>> >>> > counter parties have proprietary implementations supporting either
>> >>> > Protobuf
>> >>> > or Json encoding.
>> >>> >
>> >>> > Now, there is a Json encoder/decoder in the Avro package but it
>> seems
>> >>> > that
>> >>> > the approach is a "schema-first" approach. The JsonEncoder assumes
>> that
>> >>> > the
>> >>> > encoding on the wire still follows the Avro Json encoding - which
>> >>> > includes a
>> >>> > handshake followed by schema confirmation on both sides (client and
>> >>> > server).
>> >>> >
>> >>> > For the protocol we are implementing - this presents 2 problems if
>> >>> > Avro/
>> >>> > binary is not the chose encoding type for both sides - and if
>> instead,
>> >>> > lets
>> >>> > say, raw Json encoding is being used
>> >>> >
>> >>> > 1) the handshake is rather Avro specific - and we would like to
>> >>> > completely
>> >>> > skip it if both sides have agreed on using raw json encoding - there
>> >>> > may be
>> >>> > a separate info-exchange phase in the protocol that is protocol
>> >>> > specific and
>> >>> > does not need to use Avro handshake. Is it possible to use Avro RPC
>> >>> > without
>> >>> > the handshake?
>> >>> >
>> >>> > 2) we would like to use the data binding and schema resolution as
>> >>> > implemented by the SpecificResponder class in Avro - but extend it
>> to
>> >>> > use
>> >>> > raw JSON - not Avro JSON - encodings.
>> >>> >
>> >>> > 3) We would prefer not to have to override the
>> "respond(List<buffers>)"
>> >>> > method of the base class Responder. This implementation always
>> performs
>> >>> > handshake and always uses BinaryEncoder/Decoder which removes any
>> >>> > flexibility of using a different encoder /decoder in a derived
>> class.
>> >>> > We
>> >>> > would prefer if the Responder or some derived class saves the chosen
>> >>> > Decoder/ encoder as a protected property of the Responder object.
>> >>> > Instead of
>> >>> > instantiating BinaryEncoder/ Decoder objects on the fly within the
>> >>> > respond
>> >>> > method, it would be great if this was made more extensible and if
>> the
>> >>> > Encoder/Decoder can be specified during construction.
>> >>> >
>> >>> > 4) For future flexibility it would be great to have the handshake
>> >>> > functionality available in sub-classes of Responder as an inherited
>> >>> > method
>> >>> > (instead of private scope right now).
>> >>> >
>> >>> > I would welcome any suggestions/corrections.
>> >>> >
>> >>> > Pankaj
>> >>> >
>> >>> > --
>> >>> > Pankaj Shroff
>> >>> > twitter: @chompi
>> >>> > http://github.com/chompi
>> >>> > http://github.com/openrtb/openrtb2x
>> >>> >
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Pankaj Shroff
>> >> shroffG@Gmail.com
>> >
>> >
>> >
>> >
>> > --
>> > Pankaj Shroff
>> > shroffG@Gmail.com
>>
>
>
>
> --
> Pankaj Shroff
> shroffG@Gmail.com
>
--
Pankaj Shroff
shroffG@Gmail.com
Re: Bypassing "handshake" in Responder
Posted by Doug Cutting <cu...@apache.org>.
This looks reasonable to me.
Cheers,
Doug
On Tue, Mar 12, 2013 at 2:01 PM, Pankaj Shroff <sh...@gmail.com> wrote:
> Quick update on this issue. I think I finally figured out a way to bypass
> Avro style "handshake" when employing "custom" format or content type
> implementation but still trying to reuse or benefit from the
> Serialize/Deserialize capabilities of Avro. Perhaps the following usage is
> the "intended" REUSE use case in Avro. However, what I have below completely
> bypasses Avro RPC classes in the avro-ipc package. Let me know if this still
> mandates a patch proposal on Jira. Basically - I implemented my own
> "Responder-Server" combination without relying on reflection based method
> invocation of "classic" Avro. Link to source code below:
>
> https://github.com/openrtb/openrtb2x/blob/2.0/demand-side/dsp-core/src/main/java/org/openrtb/dsp/core/DemandSideServer.java
>
> Pankaj
>
>
>
> On Wed, Feb 27, 2013 at 4:38 PM, Pankaj Shroff <sh...@gmail.com> wrote:
>>
>> Doug - thanks again.
>>
>> I agree with you. I have been looking into it for the past few days and it
>> seems like this will require quite a bit of refactoring. I will try to
>> follow up on Jira with more specific details.
>>
>> Thanks
>> Pankaj
>>
>>
>>
>> On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <cu...@apache.org> wrote:
>>>
>>> Pankaj,
>>>
>>> Avro RPC is currently specified to always uses the binary encoding
>>> (like Avro data files). This might reasonably be extended to JSON.
>>> First we'd need to specify the new wire format. Probably Avro's
>>> framing would not make sense for JSON-encoded RPC over HTTP. Then
>>> we'd need to figure what of the existing Java implementation might be
>>> reused or adapted. At a glance, it doesn't look to me like a few
>>> one-line changes would suffice, adding methods where things are
>>> hardwired, that rather more substantial changes would be required, but
>>> I might be missing something. If you're interested in pursuing this
>>> then please file an issue in Jira where you can propose changes to the
>>> specification and implementation.
>>>
>>> Cheers,
>>>
>>> Doug
>>>
>>> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <sh...@gmail.com>
>>> wrote:
>>> > I guess my question is more basic - given that this is somewhat
>>> > specific to
>>> > my own use case:
>>> >
>>> > How does one use other forms of Encoder/Decoder implementations that
>>> > are
>>> > available in the Avro library along with the Avro-Ipc SDK.
>>> >
>>> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc
>>> > supports is
>>> > the BinaryEncoding
>>> >
>>> > Pankaj
>>> >
>>> >
>>> >
>>> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <sh...@gmail.com>
>>> > wrote:
>>> >>
>>> >> Doug
>>> >>
>>> >> Perhaps you answered a portion of my conundrum in another thread
>>> >> (permalink below) - but there is still the handshake and reuse of
>>> >> invocation
>>> >> logic question. Let me also think about this a little bit.
>>> >>
>>> >> Thanks in any case. Avro is a great tool in any case!
>>> >>
>>> >>
>>> >>
>>> >> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
>>> >>
>>> >>
>>> >> Pankaj
>>> >>
>>> >> shroffg@gmail.com
>>> >>
>>> >>
>>> >>
>>> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cu...@apache.org>
>>> >> wrote:
>>> >>>
>>> >>> This sounds like a different RPC wire format than Avro's. Avro's
>>> >>> Requestor and Responder implement Avro's RPC wire format. Avro's
>>> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
>>> >>> implementation of other RPC wire formats that include Avro data.
>>> >>> Avro's Transceiver API may or may not be reusable, since it assumes
>>> >>> Avro-style framing. Parts of Requestor and Responder *might* be
>>> >>> reusable and some refactoring of those classes *might* make such
>>> >>> reuse
>>> >>> easier, but there's not that much logic there that's not specific to
>>> >>> Avro's wire format, so it might be just as easy to reimplement this
>>> >>> layer for a different wire format. It's hard for me to say without
>>> >>> seeing a patch with a proposed refactoring. Does that make sense?
>>> >>>
>>> >>> Doug
>>> >>>
>>> >>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <sh...@gmail.com>
>>> >>> wrote:
>>> >>> > Hi
>>> >>> >
>>> >>> > We are using Avro for implementing an open source reference
>>> >>> > implementation
>>> >>> > of the OpenRTB protocol.
>>> >>> >
>>> >>> > We have made a design goal to model the protocol using Avro
>>> >>> > protocol
>>> >>> > files
>>> >>> > (.avpr) and generate types defined in the protocol schema using
>>> >>> > Avro .
>>> >>> > The
>>> >>> > challenge is that the protocol does not necessarily require the use
>>> >>> > of
>>> >>> > Avro/
>>> >>> > Binary wire encoding - or even the use of Avro/ RPC context. In
>>> >>> > fact
>>> >>> > many
>>> >>> > counter parties have proprietary implementations supporting either
>>> >>> > Protobuf
>>> >>> > or Json encoding.
>>> >>> >
>>> >>> > Now, there is a Json encoder/decoder in the Avro package but it
>>> >>> > seems
>>> >>> > that
>>> >>> > the approach is a "schema-first" approach. The JsonEncoder assumes
>>> >>> > that
>>> >>> > the
>>> >>> > encoding on the wire still follows the Avro Json encoding - which
>>> >>> > includes a
>>> >>> > handshake followed by schema confirmation on both sides (client and
>>> >>> > server).
>>> >>> >
>>> >>> > For the protocol we are implementing - this presents 2 problems if
>>> >>> > Avro/
>>> >>> > binary is not the chose encoding type for both sides - and if
>>> >>> > instead,
>>> >>> > lets
>>> >>> > say, raw Json encoding is being used
>>> >>> >
>>> >>> > 1) the handshake is rather Avro specific - and we would like to
>>> >>> > completely
>>> >>> > skip it if both sides have agreed on using raw json encoding -
>>> >>> > there
>>> >>> > may be
>>> >>> > a separate info-exchange phase in the protocol that is protocol
>>> >>> > specific and
>>> >>> > does not need to use Avro handshake. Is it possible to use Avro RPC
>>> >>> > without
>>> >>> > the handshake?
>>> >>> >
>>> >>> > 2) we would like to use the data binding and schema resolution as
>>> >>> > implemented by the SpecificResponder class in Avro - but extend it
>>> >>> > to
>>> >>> > use
>>> >>> > raw JSON - not Avro JSON - encodings.
>>> >>> >
>>> >>> > 3) We would prefer not to have to override the
>>> >>> > "respond(List<buffers>)"
>>> >>> > method of the base class Responder. This implementation always
>>> >>> > performs
>>> >>> > handshake and always uses BinaryEncoder/Decoder which removes any
>>> >>> > flexibility of using a different encoder /decoder in a derived
>>> >>> > class.
>>> >>> > We
>>> >>> > would prefer if the Responder or some derived class saves the
>>> >>> > chosen
>>> >>> > Decoder/ encoder as a protected property of the Responder object.
>>> >>> > Instead of
>>> >>> > instantiating BinaryEncoder/ Decoder objects on the fly within the
>>> >>> > respond
>>> >>> > method, it would be great if this was made more extensible and if
>>> >>> > the
>>> >>> > Encoder/Decoder can be specified during construction.
>>> >>> >
>>> >>> > 4) For future flexibility it would be great to have the handshake
>>> >>> > functionality available in sub-classes of Responder as an inherited
>>> >>> > method
>>> >>> > (instead of private scope right now).
>>> >>> >
>>> >>> > I would welcome any suggestions/corrections.
>>> >>> >
>>> >>> > Pankaj
>>> >>> >
>>> >>> > --
>>> >>> > Pankaj Shroff
>>> >>> > twitter: @chompi
>>> >>> > http://github.com/chompi
>>> >>> > http://github.com/openrtb/openrtb2x
>>> >>> >
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Pankaj Shroff
>>> >> shroffG@Gmail.com
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Pankaj Shroff
>>> > shroffG@Gmail.com
>>
>>
>>
>>
>> --
>> Pankaj Shroff
>> shroffG@Gmail.com
>
>
>
>
> --
> Pankaj Shroff
> shroffG@Gmail.com