[Avro C] Interoperability with Java

[Gilles Gaillard <ga...@asperasoft.com>]

Hi,

we'd like to use Avro for inter process communication between C and Java 
using regular sockets and we noted that there is not yet an 
implementation ready for RPC with Avro-C.

Our requirements do not include to have a full 'block-mode' RPC and 
therefore we would be happy with an asynchronous mode where no return 
value is expected when a message is sent. So using the IDL, that's 
equivalent to defining messages as methods with a void return value.
With the current implementation of Avro-C, we see the following 
questions and would like to know if anybody has answers or has already 
thought about a solution - or even better code that is ready to go ;-)

- On August 25, there were commits reverted in preparation for 1.4.0
   branch (949400, 947962, 947389, 947385, and 947299) for AVRO-441,
   AVRO-464, AVRO-466, AVRO-549, and AVRO-552. Do you plan to plan to
   incorporate them back in a near future - since the branch for 1.4.x
   now exists ?
   Specifically:
   AVRO-551 / 552  - Build for Win32
   AVRO-549 - Route memory management through an allocator interface

- We would like to avoid any unnecessary data copies when transferring 
byte arrays (defined with "type" : "bytes" in the schema).

   In java the binary encoder takes care of encoding the messages in a 
list of buffers - this is called 'Message Framing' in the Avro 
specification - which can then be transferred as such on the wire. For 
arrays of bytes this often mean they're simply passed through and sent 
separately which means they're nicely read from the wire and processed 
at no cost when decoding.

   My understanding is that such framing does not exists in Avro-C. Has 
anybody already written this or have stub for it (and similar for 
decoding) ?

   Using this message framing would then allow to use the 
avro_reader_memory (a modified version of it).
   Another way to go would be to write an avro_socket_reader (to be 
added in the avro_io_type_t enum)

   Any hint, what's the best way to go ?

Thanks,
-- Gilles

Re: [Avro C] Interoperability with Java

[Doug Cutting <cu...@apache.org>]

On 01/04/2011 02:53 AM, Gilles Gaillard wrote:
> Our requirements do not include to have a full 'block-mode' RPC and
> therefore we would be happy with an asynchronous mode where no return
> value is expected when a message is sent. So using the IDL, that's
> equivalent to defining messages as methods with a void return value.

For fire-and-forget semantics, in addition to returning void and 
declaring no errors, you should add '"one-way": true' to message 
declarations.  A handshake request is still required before the first 
request on a connection.  The simplest handshake implementation would 
always send the client's protocol (clientProtocol!=null).  The server 
should then respond with either match=BOTH or match=NONE.  If you'll 
never receive responses, you can ignore the rest of the handshake 
response.  After the handshake, if all messages are one-way, the server 
will never write any more data over the connection.

   http://avro.apache.org/docs/current/spec.html#handshake

Also, implementing the anonymous SASL mechanism is easy and 
future-proofs your wire format should you ever wish to add 
authentication or encryption:

   http://avro.apache.org/docs/current/sasl.html#anonymous

> - On August 25, there were commits reverted in preparation for 1.4.0
> branch (949400, 947962, 947389, 947385, and 947299) for AVRO-441,
> AVRO-464, AVRO-466, AVRO-549, and AVRO-552. Do you plan to plan to
> incorporate them back in a near future - since the branch for 1.4.x
> now exists ?
> Specifically:
> AVRO-551 / 552 - Build for Win32
> AVRO-549 - Route memory management through an allocator interface

Bruce told me then that these changes were incomplete and not ready for 
release.  We're now preparing for a 1.5 release.  Bruce, should we 
re-introduce these changes now?

> My understanding is that such framing does not exists in Avro-C. Has
> anybody already written this or have stub for it (and similar for
> decoding) ?

Not to my knowledge.  Bruce?  Matt?

Doug