You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Pankaj Shroff <sh...@gmail.com> on 2013/02/25 18:08:52 UTC
Bypassing "handshake" in Responder
Hi
We are using Avro for implementing an open source reference implementation
of the OpenRTB protocol.
We have made a design goal to model the protocol using Avro protocol files
(.avpr) and generate types defined in the protocol schema using Avro . The
challenge is that the protocol does not necessarily require the use of
Avro/ Binary wire encoding - or even the use of Avro/ RPC context. In fact
many counter parties have proprietary implementations supporting either
Protobuf or Json encoding.
Now, there is a Json encoder/decoder in the Avro package but it seems that
the approach is a "schema-first" approach. The JsonEncoder assumes that the
encoding on the wire still follows the Avro Json encoding - which includes
a handshake followed by schema confirmation on both sides (client and
server).
For the protocol we are implementing - this presents 2 problems if Avro/
binary is not the chose encoding type for both sides - and if instead, lets
say, raw Json encoding is being used
1) the handshake is rather Avro specific - and we would like to completely
skip it if both sides have agreed on using raw json encoding - there may be
a separate info-exchange phase in the protocol that is protocol specific
and does not need to use Avro handshake. Is it possible to use Avro RPC
without the handshake?
2) we would like to use the data binding and schema resolution as
implemented by the SpecificResponder class in Avro - but extend it to use
raw JSON - not Avro JSON - encodings.
3) We would prefer not to have to override the "respond(List<buffers>)"
method of the base class Responder. This implementation always performs
handshake and always uses BinaryEncoder/Decoder which removes any
flexibility of using a different encoder /decoder in a derived class. We
would prefer if the Responder or some derived class saves the chosen
Decoder/ encoder as a protected property of the Responder object. Instead
of instantiating BinaryEncoder/ Decoder objects on the fly within the
respond method, it would be great if this was made more extensible and if
the Encoder/Decoder can be specified during construction.
4) For future flexibility it would be great to have the handshake
functionality available in sub-classes of Responder as an inherited method
(instead of private scope right now).
I would welcome any suggestions/corrections.
Pankaj
--
Pankaj Shroff
twitter: @chompi
http://github.com/chompi
http://github.com/openrtb/openrtb2x
Re: Bypassing "handshake" in Responder
Posted by Pankaj Shroff <sh...@gmail.com>.
Doug - thanks. I think I see what you are saying - but I was trying to
leverage Avro a bit more than perhaps currently designed.
There is a lot of great code in SpecificResponder that uses JsonDecoder
(for example) to do the data binding between a SpecificData derived object
and its Json representation. The great thing about this is that it
validates against the protocol schema specified (or installed upon
construction of a SpecificResponder object). To be clear, here is an
example:
1) I define Avro protocol file to describe my protocol (.avpr)
2) In the generate-sources phase, avro-maven plugin helps me to generate
Class types for my protocol objects and the interface for the message
exchange (my protocol API). These extend the SpecificRecord object in Avro
IO
3) I write an implementation of the protocol API by implementing the
interface generated by Avro
-- so far everything's great ----
4) Now I want to use the avro-ipc package to create client and server
classes (Requestor and Responder in avro-ipc)
5) My protocol does not "REQUIRE" Avro RPC framing - surely it CANNOT -
otherwise I would be forcing every client and server implementation to have
a dependency on Avro. They might just want to use direct Jackson or direct
Protobuf or direct Thrift support for encoding/decoding - or they may have
legacy code already in place which they would rather not refactor - just
because of the dependency I created.
6) I therefore need hooks to provide my own implementation of
SpecificDecoder and SpecificEncoder.
There is no way to do this.
If, as you suggest, I extend from the DatumReader/Writer classes directly,
I have to write all the code for codehaus.jackson's ObjectMapper and
JsonParser usage, which is currently nicely encapsulated in SpecificData. I
also have to write the invocation logic (that invokes the method
"respond()" in my implementation instance of my protocol API) which I would
have been just loved to reuse.
Also the hooks mentioned in (6) above would have allowed me to leverage in
the future any other encoder/decoder support Avro may provide (if at all) -
such as Protobuf, Thrift, etc.
In general what I was looking for is an "option" really - to tell Avro's
IPC API whether I am using Avro RPC framing or not! If not, it should just
let me handle the framing bits in a derived class or something and bypass
the handshake.
What do you think? :)
Cheers
Pankaj
--
On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cu...@apache.org> wrote:
> This sounds like a different RPC wire format than Avro's. Avro's
> Requestor and Responder implement Avro's RPC wire format. Avro's
> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
> implementation of other RPC wire formats that include Avro data.
> Avro's Transceiver API may or may not be reusable, since it assumes
> Avro-style framing. Parts of Requestor and Responder *might* be
> reusable and some refactoring of those classes *might* make such reuse
> easier, but there's not that much logic there that's not specific to
> Avro's wire format, so it might be just as easy to reimplement this
> layer for a different wire format. It's hard for me to say without
> seeing a patch with a proposed refactoring. Does that make sense?
>
> Doug
>
> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <sh...@gmail.com> wrote:
> > Hi
> >
> > We are using Avro for implementing an open source reference
> implementation
> > of the OpenRTB protocol.
> >
> > We have made a design goal to model the protocol using Avro protocol
> files
> > (.avpr) and generate types defined in the protocol schema using Avro .
> The
> > challenge is that the protocol does not necessarily require the use of
> Avro/
> > Binary wire encoding - or even the use of Avro/ RPC context. In fact many
> > counter parties have proprietary implementations supporting either
> Protobuf
> > or Json encoding.
> >
> > Now, there is a Json encoder/decoder in the Avro package but it seems
> that
> > the approach is a "schema-first" approach. The JsonEncoder assumes that
> the
> > encoding on the wire still follows the Avro Json encoding - which
> includes a
> > handshake followed by schema confirmation on both sides (client and
> server).
> >
> > For the protocol we are implementing - this presents 2 problems if Avro/
> > binary is not the chose encoding type for both sides - and if instead,
> lets
> > say, raw Json encoding is being used
> >
> > 1) the handshake is rather Avro specific - and we would like to
> completely
> > skip it if both sides have agreed on using raw json encoding - there may
> be
> > a separate info-exchange phase in the protocol that is protocol specific
> and
> > does not need to use Avro handshake. Is it possible to use Avro RPC
> without
> > the handshake?
> >
> > 2) we would like to use the data binding and schema resolution as
> > implemented by the SpecificResponder class in Avro - but extend it to use
> > raw JSON - not Avro JSON - encodings.
> >
> > 3) We would prefer not to have to override the "respond(List<buffers>)"
> > method of the base class Responder. This implementation always performs
> > handshake and always uses BinaryEncoder/Decoder which removes any
> > flexibility of using a different encoder /decoder in a derived class. We
> > would prefer if the Responder or some derived class saves the chosen
> > Decoder/ encoder as a protected property of the Responder object.
> Instead of
> > instantiating BinaryEncoder/ Decoder objects on the fly within the
> respond
> > method, it would be great if this was made more extensible and if the
> > Encoder/Decoder can be specified during construction.
> >
> > 4) For future flexibility it would be great to have the handshake
> > functionality available in sub-classes of Responder as an inherited
> method
> > (instead of private scope right now).
> >
> > I would welcome any suggestions/corrections.
> >
> > Pankaj
> >
> > --
> > Pankaj Shroff
> > twitter: @chompi
> > http://github.com/chompi
> > http://github.com/openrtb/openrtb2x
> >
>
--
Pankaj Shroff
shroffG@Gmail.com
Re: Bypassing "handshake" in Responder
Posted by Doug Cutting <cu...@apache.org>.
This looks reasonable to me.
Cheers,
Doug
On Tue, Mar 12, 2013 at 2:01 PM, Pankaj Shroff <sh...@gmail.com> wrote:
> Quick update on this issue. I think I finally figured out a way to bypass
> Avro style "handshake" when employing "custom" format or content type
> implementation but still trying to reuse or benefit from the
> Serialize/Deserialize capabilities of Avro. Perhaps the following usage is
> the "intended" REUSE use case in Avro. However, what I have below completely
> bypasses Avro RPC classes in the avro-ipc package. Let me know if this still
> mandates a patch proposal on Jira. Basically - I implemented my own
> "Responder-Server" combination without relying on reflection based method
> invocation of "classic" Avro. Link to source code below:
>
> https://github.com/openrtb/openrtb2x/blob/2.0/demand-side/dsp-core/src/main/java/org/openrtb/dsp/core/DemandSideServer.java
>
> Pankaj
>
>
>
> On Wed, Feb 27, 2013 at 4:38 PM, Pankaj Shroff <sh...@gmail.com> wrote:
>>
>> Doug - thanks again.
>>
>> I agree with you. I have been looking into it for the past few days and it
>> seems like this will require quite a bit of refactoring. I will try to
>> follow up on Jira with more specific details.
>>
>> Thanks
>> Pankaj
>>
>>
>>
>> On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <cu...@apache.org> wrote:
>>>
>>> Pankaj,
>>>
>>> Avro RPC is currently specified to always uses the binary encoding
>>> (like Avro data files). This might reasonably be extended to JSON.
>>> First we'd need to specify the new wire format. Probably Avro's
>>> framing would not make sense for JSON-encoded RPC over HTTP. Then
>>> we'd need to figure what of the existing Java implementation might be
>>> reused or adapted. At a glance, it doesn't look to me like a few
>>> one-line changes would suffice, adding methods where things are
>>> hardwired, that rather more substantial changes would be required, but
>>> I might be missing something. If you're interested in pursuing this
>>> then please file an issue in Jira where you can propose changes to the
>>> specification and implementation.
>>>
>>> Cheers,
>>>
>>> Doug
>>>
>>> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <sh...@gmail.com>
>>> wrote:
>>> > I guess my question is more basic - given that this is somewhat
>>> > specific to
>>> > my own use case:
>>> >
>>> > How does one use other forms of Encoder/Decoder implementations that
>>> > are
>>> > available in the Avro library along with the Avro-Ipc SDK.
>>> >
>>> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc
>>> > supports is
>>> > the BinaryEncoding
>>> >
>>> > Pankaj
>>> >
>>> >
>>> >
>>> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <sh...@gmail.com>
>>> > wrote:
>>> >>
>>> >> Doug
>>> >>
>>> >> Perhaps you answered a portion of my conundrum in another thread
>>> >> (permalink below) - but there is still the handshake and reuse of
>>> >> invocation
>>> >> logic question. Let me also think about this a little bit.
>>> >>
>>> >> Thanks in any case. Avro is a great tool in any case!
>>> >>
>>> >>
>>> >>
>>> >> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
>>> >>
>>> >>
>>> >> Pankaj
>>> >>
>>> >> shroffg@gmail.com
>>> >>
>>> >>
>>> >>
>>> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cu...@apache.org>
>>> >> wrote:
>>> >>>
>>> >>> This sounds like a different RPC wire format than Avro's. Avro's
>>> >>> Requestor and Responder implement Avro's RPC wire format. Avro's
>>> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
>>> >>> implementation of other RPC wire formats that include Avro data.
>>> >>> Avro's Transceiver API may or may not be reusable, since it assumes
>>> >>> Avro-style framing. Parts of Requestor and Responder *might* be
>>> >>> reusable and some refactoring of those classes *might* make such
>>> >>> reuse
>>> >>> easier, but there's not that much logic there that's not specific to
>>> >>> Avro's wire format, so it might be just as easy to reimplement this
>>> >>> layer for a different wire format. It's hard for me to say without
>>> >>> seeing a patch with a proposed refactoring. Does that make sense?
>>> >>>
>>> >>> Doug
>>> >>>
>>> >>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <sh...@gmail.com>
>>> >>> wrote:
>>> >>> > Hi
>>> >>> >
>>> >>> > We are using Avro for implementing an open source reference
>>> >>> > implementation
>>> >>> > of the OpenRTB protocol.
>>> >>> >
>>> >>> > We have made a design goal to model the protocol using Avro
>>> >>> > protocol
>>> >>> > files
>>> >>> > (.avpr) and generate types defined in the protocol schema using
>>> >>> > Avro .
>>> >>> > The
>>> >>> > challenge is that the protocol does not necessarily require the use
>>> >>> > of
>>> >>> > Avro/
>>> >>> > Binary wire encoding - or even the use of Avro/ RPC context. In
>>> >>> > fact
>>> >>> > many
>>> >>> > counter parties have proprietary implementations supporting either
>>> >>> > Protobuf
>>> >>> > or Json encoding.
>>> >>> >
>>> >>> > Now, there is a Json encoder/decoder in the Avro package but it
>>> >>> > seems
>>> >>> > that
>>> >>> > the approach is a "schema-first" approach. The JsonEncoder assumes
>>> >>> > that
>>> >>> > the
>>> >>> > encoding on the wire still follows the Avro Json encoding - which
>>> >>> > includes a
>>> >>> > handshake followed by schema confirmation on both sides (client and
>>> >>> > server).
>>> >>> >
>>> >>> > For the protocol we are implementing - this presents 2 problems if
>>> >>> > Avro/
>>> >>> > binary is not the chose encoding type for both sides - and if
>>> >>> > instead,
>>> >>> > lets
>>> >>> > say, raw Json encoding is being used
>>> >>> >
>>> >>> > 1) the handshake is rather Avro specific - and we would like to
>>> >>> > completely
>>> >>> > skip it if both sides have agreed on using raw json encoding -
>>> >>> > there
>>> >>> > may be
>>> >>> > a separate info-exchange phase in the protocol that is protocol
>>> >>> > specific and
>>> >>> > does not need to use Avro handshake. Is it possible to use Avro RPC
>>> >>> > without
>>> >>> > the handshake?
>>> >>> >
>>> >>> > 2) we would like to use the data binding and schema resolution as
>>> >>> > implemented by the SpecificResponder class in Avro - but extend it
>>> >>> > to
>>> >>> > use
>>> >>> > raw JSON - not Avro JSON - encodings.
>>> >>> >
>>> >>> > 3) We would prefer not to have to override the
>>> >>> > "respond(List<buffers>)"
>>> >>> > method of the base class Responder. This implementation always
>>> >>> > performs
>>> >>> > handshake and always uses BinaryEncoder/Decoder which removes any
>>> >>> > flexibility of using a different encoder /decoder in a derived
>>> >>> > class.
>>> >>> > We
>>> >>> > would prefer if the Responder or some derived class saves the
>>> >>> > chosen
>>> >>> > Decoder/ encoder as a protected property of the Responder object.
>>> >>> > Instead of
>>> >>> > instantiating BinaryEncoder/ Decoder objects on the fly within the
>>> >>> > respond
>>> >>> > method, it would be great if this was made more extensible and if
>>> >>> > the
>>> >>> > Encoder/Decoder can be specified during construction.
>>> >>> >
>>> >>> > 4) For future flexibility it would be great to have the handshake
>>> >>> > functionality available in sub-classes of Responder as an inherited
>>> >>> > method
>>> >>> > (instead of private scope right now).
>>> >>> >
>>> >>> > I would welcome any suggestions/corrections.
>>> >>> >
>>> >>> > Pankaj
>>> >>> >
>>> >>> > --
>>> >>> > Pankaj Shroff
>>> >>> > twitter: @chompi
>>> >>> > http://github.com/chompi
>>> >>> > http://github.com/openrtb/openrtb2x
>>> >>> >
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Pankaj Shroff
>>> >> shroffG@Gmail.com
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Pankaj Shroff
>>> > shroffG@Gmail.com
>>
>>
>>
>>
>> --
>> Pankaj Shroff
>> shroffG@Gmail.com
>
>
>
>
> --
> Pankaj Shroff
> shroffG@Gmail.com
Re: Bypassing "handshake" in Responder
Posted by Pankaj Shroff <sh...@gmail.com>.
Quick update on this issue. I think I finally figured out a way to bypass
Avro style "handshake" when employing "custom" format or content type
implementation but still trying to reuse or benefit from the
Serialize/Deserialize capabilities of Avro. Perhaps the following usage is
the "intended" REUSE use case in Avro. However, what I have below
completely bypasses Avro RPC classes in the avro-ipc package. Let me know
if this still mandates a patch proposal on Jira. Basically - I implemented
my own "Responder-Server" combination without relying on reflection based
method invocation of "classic" Avro. Link to source code below:
https://github.com/openrtb/openrtb2x/blob/2.0/demand-side/dsp-core/src/main/java/org/openrtb/dsp/core/DemandSideServer.java
Pankaj
On Wed, Feb 27, 2013 at 4:38 PM, Pankaj Shroff <sh...@gmail.com> wrote:
> Doug - thanks again.
>
> I agree with you. I have been looking into it for the past few days and it
> seems like this will require quite a bit of refactoring. I will try to
> follow up on Jira with more specific details.
>
> Thanks
> Pankaj
>
>
>
> On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <cu...@apache.org> wrote:
>
>> Pankaj,
>>
>> Avro RPC is currently specified to always uses the binary encoding
>> (like Avro data files). This might reasonably be extended to JSON.
>> First we'd need to specify the new wire format. Probably Avro's
>> framing would not make sense for JSON-encoded RPC over HTTP. Then
>> we'd need to figure what of the existing Java implementation might be
>> reused or adapted. At a glance, it doesn't look to me like a few
>> one-line changes would suffice, adding methods where things are
>> hardwired, that rather more substantial changes would be required, but
>> I might be missing something. If you're interested in pursuing this
>> then please file an issue in Jira where you can propose changes to the
>> specification and implementation.
>>
>> Cheers,
>>
>> Doug
>>
>> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <sh...@gmail.com>
>> wrote:
>> > I guess my question is more basic - given that this is somewhat
>> specific to
>> > my own use case:
>> >
>> > How does one use other forms of Encoder/Decoder implementations that are
>> > available in the Avro library along with the Avro-Ipc SDK.
>> >
>> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc
>> supports is
>> > the BinaryEncoding
>> >
>> > Pankaj
>> >
>> >
>> >
>> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <sh...@gmail.com>
>> wrote:
>> >>
>> >> Doug
>> >>
>> >> Perhaps you answered a portion of my conundrum in another thread
>> >> (permalink below) - but there is still the handshake and reuse of
>> invocation
>> >> logic question. Let me also think about this a little bit.
>> >>
>> >> Thanks in any case. Avro is a great tool in any case!
>> >>
>> >>
>> >>
>> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
>> >>
>> >>
>> >> Pankaj
>> >>
>> >> shroffg@gmail.com
>> >>
>> >>
>> >>
>> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cu...@apache.org>
>> wrote:
>> >>>
>> >>> This sounds like a different RPC wire format than Avro's. Avro's
>> >>> Requestor and Responder implement Avro's RPC wire format. Avro's
>> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
>> >>> implementation of other RPC wire formats that include Avro data.
>> >>> Avro's Transceiver API may or may not be reusable, since it assumes
>> >>> Avro-style framing. Parts of Requestor and Responder *might* be
>> >>> reusable and some refactoring of those classes *might* make such reuse
>> >>> easier, but there's not that much logic there that's not specific to
>> >>> Avro's wire format, so it might be just as easy to reimplement this
>> >>> layer for a different wire format. It's hard for me to say without
>> >>> seeing a patch with a proposed refactoring. Does that make sense?
>> >>>
>> >>> Doug
>> >>>
>> >>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <sh...@gmail.com>
>> wrote:
>> >>> > Hi
>> >>> >
>> >>> > We are using Avro for implementing an open source reference
>> >>> > implementation
>> >>> > of the OpenRTB protocol.
>> >>> >
>> >>> > We have made a design goal to model the protocol using Avro protocol
>> >>> > files
>> >>> > (.avpr) and generate types defined in the protocol schema using
>> Avro .
>> >>> > The
>> >>> > challenge is that the protocol does not necessarily require the use
>> of
>> >>> > Avro/
>> >>> > Binary wire encoding - or even the use of Avro/ RPC context. In fact
>> >>> > many
>> >>> > counter parties have proprietary implementations supporting either
>> >>> > Protobuf
>> >>> > or Json encoding.
>> >>> >
>> >>> > Now, there is a Json encoder/decoder in the Avro package but it
>> seems
>> >>> > that
>> >>> > the approach is a "schema-first" approach. The JsonEncoder assumes
>> that
>> >>> > the
>> >>> > encoding on the wire still follows the Avro Json encoding - which
>> >>> > includes a
>> >>> > handshake followed by schema confirmation on both sides (client and
>> >>> > server).
>> >>> >
>> >>> > For the protocol we are implementing - this presents 2 problems if
>> >>> > Avro/
>> >>> > binary is not the chose encoding type for both sides - and if
>> instead,
>> >>> > lets
>> >>> > say, raw Json encoding is being used
>> >>> >
>> >>> > 1) the handshake is rather Avro specific - and we would like to
>> >>> > completely
>> >>> > skip it if both sides have agreed on using raw json encoding - there
>> >>> > may be
>> >>> > a separate info-exchange phase in the protocol that is protocol
>> >>> > specific and
>> >>> > does not need to use Avro handshake. Is it possible to use Avro RPC
>> >>> > without
>> >>> > the handshake?
>> >>> >
>> >>> > 2) we would like to use the data binding and schema resolution as
>> >>> > implemented by the SpecificResponder class in Avro - but extend it
>> to
>> >>> > use
>> >>> > raw JSON - not Avro JSON - encodings.
>> >>> >
>> >>> > 3) We would prefer not to have to override the
>> "respond(List<buffers>)"
>> >>> > method of the base class Responder. This implementation always
>> performs
>> >>> > handshake and always uses BinaryEncoder/Decoder which removes any
>> >>> > flexibility of using a different encoder /decoder in a derived
>> class.
>> >>> > We
>> >>> > would prefer if the Responder or some derived class saves the chosen
>> >>> > Decoder/ encoder as a protected property of the Responder object.
>> >>> > Instead of
>> >>> > instantiating BinaryEncoder/ Decoder objects on the fly within the
>> >>> > respond
>> >>> > method, it would be great if this was made more extensible and if
>> the
>> >>> > Encoder/Decoder can be specified during construction.
>> >>> >
>> >>> > 4) For future flexibility it would be great to have the handshake
>> >>> > functionality available in sub-classes of Responder as an inherited
>> >>> > method
>> >>> > (instead of private scope right now).
>> >>> >
>> >>> > I would welcome any suggestions/corrections.
>> >>> >
>> >>> > Pankaj
>> >>> >
>> >>> > --
>> >>> > Pankaj Shroff
>> >>> > twitter: @chompi
>> >>> > http://github.com/chompi
>> >>> > http://github.com/openrtb/openrtb2x
>> >>> >
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Pankaj Shroff
>> >> shroffG@Gmail.com
>> >
>> >
>> >
>> >
>> > --
>> > Pankaj Shroff
>> > shroffG@Gmail.com
>>
>
>
>
> --
> Pankaj Shroff
> shroffG@Gmail.com
>
--
Pankaj Shroff
shroffG@Gmail.com
Re: Bypassing "handshake" in Responder
Posted by Pankaj Shroff <sh...@gmail.com>.
Doug - thanks again.
I agree with you. I have been looking into it for the past few days and it
seems like this will require quite a bit of refactoring. I will try to
follow up on Jira with more specific details.
Thanks
Pankaj
On Wed, Feb 27, 2013 at 2:50 PM, Doug Cutting <cu...@apache.org> wrote:
> Pankaj,
>
> Avro RPC is currently specified to always uses the binary encoding
> (like Avro data files). This might reasonably be extended to JSON.
> First we'd need to specify the new wire format. Probably Avro's
> framing would not make sense for JSON-encoded RPC over HTTP. Then
> we'd need to figure what of the existing Java implementation might be
> reused or adapted. At a glance, it doesn't look to me like a few
> one-line changes would suffice, adding methods where things are
> hardwired, that rather more substantial changes would be required, but
> I might be missing something. If you're interested in pursuing this
> then please file an issue in Jira where you can propose changes to the
> specification and implementation.
>
> Cheers,
>
> Doug
>
> On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <sh...@gmail.com> wrote:
> > I guess my question is more basic - given that this is somewhat specific
> to
> > my own use case:
> >
> > How does one use other forms of Encoder/Decoder implementations that are
> > available in the Avro library along with the Avro-Ipc SDK.
> >
> > As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc
> supports is
> > the BinaryEncoding
> >
> > Pankaj
> >
> >
> >
> > On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <sh...@gmail.com>
> wrote:
> >>
> >> Doug
> >>
> >> Perhaps you answered a portion of my conundrum in another thread
> >> (permalink below) - but there is still the handshake and reuse of
> invocation
> >> logic question. Let me also think about this a little bit.
> >>
> >> Thanks in any case. Avro is a great tool in any case!
> >>
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
> >>
> >>
> >> Pankaj
> >>
> >> shroffg@gmail.com
> >>
> >>
> >>
> >> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cu...@apache.org>
> wrote:
> >>>
> >>> This sounds like a different RPC wire format than Avro's. Avro's
> >>> Requestor and Responder implement Avro's RPC wire format. Avro's
> >>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
> >>> implementation of other RPC wire formats that include Avro data.
> >>> Avro's Transceiver API may or may not be reusable, since it assumes
> >>> Avro-style framing. Parts of Requestor and Responder *might* be
> >>> reusable and some refactoring of those classes *might* make such reuse
> >>> easier, but there's not that much logic there that's not specific to
> >>> Avro's wire format, so it might be just as easy to reimplement this
> >>> layer for a different wire format. It's hard for me to say without
> >>> seeing a patch with a proposed refactoring. Does that make sense?
> >>>
> >>> Doug
> >>>
> >>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <sh...@gmail.com>
> wrote:
> >>> > Hi
> >>> >
> >>> > We are using Avro for implementing an open source reference
> >>> > implementation
> >>> > of the OpenRTB protocol.
> >>> >
> >>> > We have made a design goal to model the protocol using Avro protocol
> >>> > files
> >>> > (.avpr) and generate types defined in the protocol schema using Avro
> .
> >>> > The
> >>> > challenge is that the protocol does not necessarily require the use
> of
> >>> > Avro/
> >>> > Binary wire encoding - or even the use of Avro/ RPC context. In fact
> >>> > many
> >>> > counter parties have proprietary implementations supporting either
> >>> > Protobuf
> >>> > or Json encoding.
> >>> >
> >>> > Now, there is a Json encoder/decoder in the Avro package but it seems
> >>> > that
> >>> > the approach is a "schema-first" approach. The JsonEncoder assumes
> that
> >>> > the
> >>> > encoding on the wire still follows the Avro Json encoding - which
> >>> > includes a
> >>> > handshake followed by schema confirmation on both sides (client and
> >>> > server).
> >>> >
> >>> > For the protocol we are implementing - this presents 2 problems if
> >>> > Avro/
> >>> > binary is not the chose encoding type for both sides - and if
> instead,
> >>> > lets
> >>> > say, raw Json encoding is being used
> >>> >
> >>> > 1) the handshake is rather Avro specific - and we would like to
> >>> > completely
> >>> > skip it if both sides have agreed on using raw json encoding - there
> >>> > may be
> >>> > a separate info-exchange phase in the protocol that is protocol
> >>> > specific and
> >>> > does not need to use Avro handshake. Is it possible to use Avro RPC
> >>> > without
> >>> > the handshake?
> >>> >
> >>> > 2) we would like to use the data binding and schema resolution as
> >>> > implemented by the SpecificResponder class in Avro - but extend it to
> >>> > use
> >>> > raw JSON - not Avro JSON - encodings.
> >>> >
> >>> > 3) We would prefer not to have to override the
> "respond(List<buffers>)"
> >>> > method of the base class Responder. This implementation always
> performs
> >>> > handshake and always uses BinaryEncoder/Decoder which removes any
> >>> > flexibility of using a different encoder /decoder in a derived class.
> >>> > We
> >>> > would prefer if the Responder or some derived class saves the chosen
> >>> > Decoder/ encoder as a protected property of the Responder object.
> >>> > Instead of
> >>> > instantiating BinaryEncoder/ Decoder objects on the fly within the
> >>> > respond
> >>> > method, it would be great if this was made more extensible and if the
> >>> > Encoder/Decoder can be specified during construction.
> >>> >
> >>> > 4) For future flexibility it would be great to have the handshake
> >>> > functionality available in sub-classes of Responder as an inherited
> >>> > method
> >>> > (instead of private scope right now).
> >>> >
> >>> > I would welcome any suggestions/corrections.
> >>> >
> >>> > Pankaj
> >>> >
> >>> > --
> >>> > Pankaj Shroff
> >>> > twitter: @chompi
> >>> > http://github.com/chompi
> >>> > http://github.com/openrtb/openrtb2x
> >>> >
> >>
> >>
> >>
> >>
> >> --
> >> Pankaj Shroff
> >> shroffG@Gmail.com
> >
> >
> >
> >
> > --
> > Pankaj Shroff
> > shroffG@Gmail.com
>
--
Pankaj Shroff
shroffG@Gmail.com
Re: Bypassing "handshake" in Responder
Posted by Doug Cutting <cu...@apache.org>.
Pankaj,
Avro RPC is currently specified to always uses the binary encoding
(like Avro data files). This might reasonably be extended to JSON.
First we'd need to specify the new wire format. Probably Avro's
framing would not make sense for JSON-encoded RPC over HTTP. Then
we'd need to figure what of the existing Java implementation might be
reused or adapted. At a glance, it doesn't look to me like a few
one-line changes would suffice, adding methods where things are
hardwired, that rather more substantial changes would be required, but
I might be missing something. If you're interested in pursuing this
then please file an issue in Jira where you can propose changes to the
specification and implementation.
Cheers,
Doug
On Wed, Feb 27, 2013 at 11:16 AM, Pankaj Shroff <sh...@gmail.com> wrote:
> I guess my question is more basic - given that this is somewhat specific to
> my own use case:
>
> How does one use other forms of Encoder/Decoder implementations that are
> available in the Avro library along with the Avro-Ipc SDK.
>
> As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc supports is
> the BinaryEncoding
>
> Pankaj
>
>
>
> On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <sh...@gmail.com> wrote:
>>
>> Doug
>>
>> Perhaps you answered a portion of my conundrum in another thread
>> (permalink below) - but there is still the handshake and reuse of invocation
>> logic question. Let me also think about this a little bit.
>>
>> Thanks in any case. Avro is a great tool in any case!
>>
>>
>> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
>>
>>
>> Pankaj
>>
>> shroffg@gmail.com
>>
>>
>>
>> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cu...@apache.org> wrote:
>>>
>>> This sounds like a different RPC wire format than Avro's. Avro's
>>> Requestor and Responder implement Avro's RPC wire format. Avro's
>>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
>>> implementation of other RPC wire formats that include Avro data.
>>> Avro's Transceiver API may or may not be reusable, since it assumes
>>> Avro-style framing. Parts of Requestor and Responder *might* be
>>> reusable and some refactoring of those classes *might* make such reuse
>>> easier, but there's not that much logic there that's not specific to
>>> Avro's wire format, so it might be just as easy to reimplement this
>>> layer for a different wire format. It's hard for me to say without
>>> seeing a patch with a proposed refactoring. Does that make sense?
>>>
>>> Doug
>>>
>>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <sh...@gmail.com> wrote:
>>> > Hi
>>> >
>>> > We are using Avro for implementing an open source reference
>>> > implementation
>>> > of the OpenRTB protocol.
>>> >
>>> > We have made a design goal to model the protocol using Avro protocol
>>> > files
>>> > (.avpr) and generate types defined in the protocol schema using Avro .
>>> > The
>>> > challenge is that the protocol does not necessarily require the use of
>>> > Avro/
>>> > Binary wire encoding - or even the use of Avro/ RPC context. In fact
>>> > many
>>> > counter parties have proprietary implementations supporting either
>>> > Protobuf
>>> > or Json encoding.
>>> >
>>> > Now, there is a Json encoder/decoder in the Avro package but it seems
>>> > that
>>> > the approach is a "schema-first" approach. The JsonEncoder assumes that
>>> > the
>>> > encoding on the wire still follows the Avro Json encoding - which
>>> > includes a
>>> > handshake followed by schema confirmation on both sides (client and
>>> > server).
>>> >
>>> > For the protocol we are implementing - this presents 2 problems if
>>> > Avro/
>>> > binary is not the chose encoding type for both sides - and if instead,
>>> > lets
>>> > say, raw Json encoding is being used
>>> >
>>> > 1) the handshake is rather Avro specific - and we would like to
>>> > completely
>>> > skip it if both sides have agreed on using raw json encoding - there
>>> > may be
>>> > a separate info-exchange phase in the protocol that is protocol
>>> > specific and
>>> > does not need to use Avro handshake. Is it possible to use Avro RPC
>>> > without
>>> > the handshake?
>>> >
>>> > 2) we would like to use the data binding and schema resolution as
>>> > implemented by the SpecificResponder class in Avro - but extend it to
>>> > use
>>> > raw JSON - not Avro JSON - encodings.
>>> >
>>> > 3) We would prefer not to have to override the "respond(List<buffers>)"
>>> > method of the base class Responder. This implementation always performs
>>> > handshake and always uses BinaryEncoder/Decoder which removes any
>>> > flexibility of using a different encoder /decoder in a derived class.
>>> > We
>>> > would prefer if the Responder or some derived class saves the chosen
>>> > Decoder/ encoder as a protected property of the Responder object.
>>> > Instead of
>>> > instantiating BinaryEncoder/ Decoder objects on the fly within the
>>> > respond
>>> > method, it would be great if this was made more extensible and if the
>>> > Encoder/Decoder can be specified during construction.
>>> >
>>> > 4) For future flexibility it would be great to have the handshake
>>> > functionality available in sub-classes of Responder as an inherited
>>> > method
>>> > (instead of private scope right now).
>>> >
>>> > I would welcome any suggestions/corrections.
>>> >
>>> > Pankaj
>>> >
>>> > --
>>> > Pankaj Shroff
>>> > twitter: @chompi
>>> > http://github.com/chompi
>>> > http://github.com/openrtb/openrtb2x
>>> >
>>
>>
>>
>>
>> --
>> Pankaj Shroff
>> shroffG@Gmail.com
>
>
>
>
> --
> Pankaj Shroff
> shroffG@Gmail.com
Re: Bypassing "handshake" in Responder
Posted by Pankaj Shroff <sh...@gmail.com>.
I guess my question is more basic - given that this is somewhat specific to
my own use case:
How does one use other forms of Encoder/Decoder implementations that are
available in the Avro library along with the Avro-Ipc SDK.
As of 1.7.3, I see that the only Encoding/Decoding that Avro-ipc supports
is the BinaryEncoding
Pankaj
On Mon, Feb 25, 2013 at 2:25 PM, Pankaj Shroff <sh...@gmail.com> wrote:
> Doug
>
> Perhaps you answered a portion of my conundrum in another thread
> (permalink below) - but there is still the handshake and reuse of
> invocation logic question. Let me also think about this a little bit.
>
> Thanks in any case. Avro is a great tool in any case!
>
>
> http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
>
>
> Pankaj
>
> shroffg@gmail.com
>
>
>
> On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cu...@apache.org> wrote:
>
>> This sounds like a different RPC wire format than Avro's. Avro's
>> Requestor and Responder implement Avro's RPC wire format. Avro's
>> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
>> implementation of other RPC wire formats that include Avro data.
>> Avro's Transceiver API may or may not be reusable, since it assumes
>> Avro-style framing. Parts of Requestor and Responder *might* be
>> reusable and some refactoring of those classes *might* make such reuse
>> easier, but there's not that much logic there that's not specific to
>> Avro's wire format, so it might be just as easy to reimplement this
>> layer for a different wire format. It's hard for me to say without
>> seeing a patch with a proposed refactoring. Does that make sense?
>>
>> Doug
>>
>> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <sh...@gmail.com> wrote:
>> > Hi
>> >
>> > We are using Avro for implementing an open source reference
>> implementation
>> > of the OpenRTB protocol.
>> >
>> > We have made a design goal to model the protocol using Avro protocol
>> files
>> > (.avpr) and generate types defined in the protocol schema using Avro .
>> The
>> > challenge is that the protocol does not necessarily require the use of
>> Avro/
>> > Binary wire encoding - or even the use of Avro/ RPC context. In fact
>> many
>> > counter parties have proprietary implementations supporting either
>> Protobuf
>> > or Json encoding.
>> >
>> > Now, there is a Json encoder/decoder in the Avro package but it seems
>> that
>> > the approach is a "schema-first" approach. The JsonEncoder assumes that
>> the
>> > encoding on the wire still follows the Avro Json encoding - which
>> includes a
>> > handshake followed by schema confirmation on both sides (client and
>> server).
>> >
>> > For the protocol we are implementing - this presents 2 problems if Avro/
>> > binary is not the chose encoding type for both sides - and if instead,
>> lets
>> > say, raw Json encoding is being used
>> >
>> > 1) the handshake is rather Avro specific - and we would like to
>> completely
>> > skip it if both sides have agreed on using raw json encoding - there
>> may be
>> > a separate info-exchange phase in the protocol that is protocol
>> specific and
>> > does not need to use Avro handshake. Is it possible to use Avro RPC
>> without
>> > the handshake?
>> >
>> > 2) we would like to use the data binding and schema resolution as
>> > implemented by the SpecificResponder class in Avro - but extend it to
>> use
>> > raw JSON - not Avro JSON - encodings.
>> >
>> > 3) We would prefer not to have to override the "respond(List<buffers>)"
>> > method of the base class Responder. This implementation always performs
>> > handshake and always uses BinaryEncoder/Decoder which removes any
>> > flexibility of using a different encoder /decoder in a derived class. We
>> > would prefer if the Responder or some derived class saves the chosen
>> > Decoder/ encoder as a protected property of the Responder object.
>> Instead of
>> > instantiating BinaryEncoder/ Decoder objects on the fly within the
>> respond
>> > method, it would be great if this was made more extensible and if the
>> > Encoder/Decoder can be specified during construction.
>> >
>> > 4) For future flexibility it would be great to have the handshake
>> > functionality available in sub-classes of Responder as an inherited
>> method
>> > (instead of private scope right now).
>> >
>> > I would welcome any suggestions/corrections.
>> >
>> > Pankaj
>> >
>> > --
>> > Pankaj Shroff
>> > twitter: @chompi
>> > http://github.com/chompi
>> > http://github.com/openrtb/openrtb2x
>> >
>>
>
>
>
> --
> Pankaj Shroff
> shroffG@Gmail.com
>
--
Pankaj Shroff
shroffG@Gmail.com
Re: Bypassing "handshake" in Responder
Posted by Pankaj Shroff <sh...@gmail.com>.
Doug
Perhaps you answered a portion of my conundrum in another thread (permalink
below) - but there is still the handshake and reuse of invocation logic
question. Let me also think about this a little bit.
Thanks in any case. Avro is a great tool in any case!
http://mail-archives.apache.org/mod_mbox/avro-user/201302.mbox/%3CCALEq1Z_rt8FasjSR%2B%2BOOgE3ogrAh0Y%2BtL3z47hznuiBAtfvWmw%40mail.gmail.com%3E
Pankaj
shroffg@gmail.com
On Mon, Feb 25, 2013 at 1:38 PM, Doug Cutting <cu...@apache.org> wrote:
> This sounds like a different RPC wire format than Avro's. Avro's
> Requestor and Responder implement Avro's RPC wire format. Avro's
> Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
> implementation of other RPC wire formats that include Avro data.
> Avro's Transceiver API may or may not be reusable, since it assumes
> Avro-style framing. Parts of Requestor and Responder *might* be
> reusable and some refactoring of those classes *might* make such reuse
> easier, but there's not that much logic there that's not specific to
> Avro's wire format, so it might be just as easy to reimplement this
> layer for a different wire format. It's hard for me to say without
> seeing a patch with a proposed refactoring. Does that make sense?
>
> Doug
>
> On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <sh...@gmail.com> wrote:
> > Hi
> >
> > We are using Avro for implementing an open source reference
> implementation
> > of the OpenRTB protocol.
> >
> > We have made a design goal to model the protocol using Avro protocol
> files
> > (.avpr) and generate types defined in the protocol schema using Avro .
> The
> > challenge is that the protocol does not necessarily require the use of
> Avro/
> > Binary wire encoding - or even the use of Avro/ RPC context. In fact many
> > counter parties have proprietary implementations supporting either
> Protobuf
> > or Json encoding.
> >
> > Now, there is a Json encoder/decoder in the Avro package but it seems
> that
> > the approach is a "schema-first" approach. The JsonEncoder assumes that
> the
> > encoding on the wire still follows the Avro Json encoding - which
> includes a
> > handshake followed by schema confirmation on both sides (client and
> server).
> >
> > For the protocol we are implementing - this presents 2 problems if Avro/
> > binary is not the chose encoding type for both sides - and if instead,
> lets
> > say, raw Json encoding is being used
> >
> > 1) the handshake is rather Avro specific - and we would like to
> completely
> > skip it if both sides have agreed on using raw json encoding - there may
> be
> > a separate info-exchange phase in the protocol that is protocol specific
> and
> > does not need to use Avro handshake. Is it possible to use Avro RPC
> without
> > the handshake?
> >
> > 2) we would like to use the data binding and schema resolution as
> > implemented by the SpecificResponder class in Avro - but extend it to use
> > raw JSON - not Avro JSON - encodings.
> >
> > 3) We would prefer not to have to override the "respond(List<buffers>)"
> > method of the base class Responder. This implementation always performs
> > handshake and always uses BinaryEncoder/Decoder which removes any
> > flexibility of using a different encoder /decoder in a derived class. We
> > would prefer if the Responder or some derived class saves the chosen
> > Decoder/ encoder as a protected property of the Responder object.
> Instead of
> > instantiating BinaryEncoder/ Decoder objects on the fly within the
> respond
> > method, it would be great if this was made more extensible and if the
> > Encoder/Decoder can be specified during construction.
> >
> > 4) For future flexibility it would be great to have the handshake
> > functionality available in sub-classes of Responder as an inherited
> method
> > (instead of private scope right now).
> >
> > I would welcome any suggestions/corrections.
> >
> > Pankaj
> >
> > --
> > Pankaj Shroff
> > twitter: @chompi
> > http://github.com/chompi
> > http://github.com/openrtb/openrtb2x
> >
>
--
Pankaj Shroff
shroffG@Gmail.com
Re: Bypassing "handshake" in Responder
Posted by Doug Cutting <cu...@apache.org>.
This sounds like a different RPC wire format than Avro's. Avro's
Requestor and Responder implement Avro's RPC wire format. Avro's
Encode/Decoder and DatumReader/DatumWriter APIs should facilitate
implementation of other RPC wire formats that include Avro data.
Avro's Transceiver API may or may not be reusable, since it assumes
Avro-style framing. Parts of Requestor and Responder *might* be
reusable and some refactoring of those classes *might* make such reuse
easier, but there's not that much logic there that's not specific to
Avro's wire format, so it might be just as easy to reimplement this
layer for a different wire format. It's hard for me to say without
seeing a patch with a proposed refactoring. Does that make sense?
Doug
On Mon, Feb 25, 2013 at 9:08 AM, Pankaj Shroff <sh...@gmail.com> wrote:
> Hi
>
> We are using Avro for implementing an open source reference implementation
> of the OpenRTB protocol.
>
> We have made a design goal to model the protocol using Avro protocol files
> (.avpr) and generate types defined in the protocol schema using Avro . The
> challenge is that the protocol does not necessarily require the use of Avro/
> Binary wire encoding - or even the use of Avro/ RPC context. In fact many
> counter parties have proprietary implementations supporting either Protobuf
> or Json encoding.
>
> Now, there is a Json encoder/decoder in the Avro package but it seems that
> the approach is a "schema-first" approach. The JsonEncoder assumes that the
> encoding on the wire still follows the Avro Json encoding - which includes a
> handshake followed by schema confirmation on both sides (client and server).
>
> For the protocol we are implementing - this presents 2 problems if Avro/
> binary is not the chose encoding type for both sides - and if instead, lets
> say, raw Json encoding is being used
>
> 1) the handshake is rather Avro specific - and we would like to completely
> skip it if both sides have agreed on using raw json encoding - there may be
> a separate info-exchange phase in the protocol that is protocol specific and
> does not need to use Avro handshake. Is it possible to use Avro RPC without
> the handshake?
>
> 2) we would like to use the data binding and schema resolution as
> implemented by the SpecificResponder class in Avro - but extend it to use
> raw JSON - not Avro JSON - encodings.
>
> 3) We would prefer not to have to override the "respond(List<buffers>)"
> method of the base class Responder. This implementation always performs
> handshake and always uses BinaryEncoder/Decoder which removes any
> flexibility of using a different encoder /decoder in a derived class. We
> would prefer if the Responder or some derived class saves the chosen
> Decoder/ encoder as a protected property of the Responder object. Instead of
> instantiating BinaryEncoder/ Decoder objects on the fly within the respond
> method, it would be great if this was made more extensible and if the
> Encoder/Decoder can be specified during construction.
>
> 4) For future flexibility it would be great to have the handshake
> functionality available in sub-classes of Responder as an inherited method
> (instead of private scope right now).
>
> I would welcome any suggestions/corrections.
>
> Pankaj
>
> --
> Pankaj Shroff
> twitter: @chompi
> http://github.com/chompi
> http://github.com/openrtb/openrtb2x
>