is avro production ready?

[Ran Tavory <ra...@gmail.com>]

ok, I know this question has many answers, but try to be honest ;)

I'd like to use avro in several use cases:
1. RPC. Both using json over HTTP and binary over TCP (will use both modes).
Async RPC clients/servers isn't a showstopper but could also be nice
2. As a serialization mechanism to send messages over AMQ or save object on
memcached
3. Perhaps use for saving space in logs (rather than plain text log files)
and in combination with flume. I haven't thought that out yet.

My main codebase is java (and scala), but there may be some small
non critical parts in python as well.

Are all the above mentioned use cases considered as stable and "production
ready"? If not, is there an eta?

Thanks

Re: is avro production ready?

[Scott Carey <sc...@richrelevance.com>]

On Jul 19, 2010, at 11:43 AM, Ran Tavory wrote:

> ok, I know this question has many answers, but try to be honest ;)
> 
> I'd like to use avro in several use cases:
> 1. RPC. Both using json over HTTP and binary over TCP (will use both modes). Async RPC clients/servers isn't a showstopper but could also be nice
See Doug's notes.
> 2. As a serialization mechanism to send messages over AMQ or save object on memcached
> 3. Perhaps use for saving space in logs (rather than plain text log files) and in combination with flume. I haven't thought that out yet.

The above two are production ready use cases. I do both right now (in Java), in addition to processing Avro log files in hadoop (as inputs and outputs, but not intermediate data since Pig/Hive control that).
I expect the Java APIs for #2 - type use cases will evolve over time to be more user-friendly and improve performance, but what is there now gets the job done.


> 
> My main codebase is java (and scala), but there may be some small non critical parts in python as well.
> 
> Are all the above mentioned use cases considered as stable and "production ready"? If not, is there an eta?
> 
> Thanks

Re: is avro production ready?

[Ran Tavory <ra...@gmail.com>]

On Mon, Jul 19, 2010 at 10:09 PM, Doug Cutting <cu...@apache.org> wrote:

> On 07/19/2010 11:43 AM, Ran Tavory wrote:
>
>> I'd like to use avro in several use cases:
>> 1. RPC. Both using json over HTTP and binary over TCP (will use both
>> modes). Async RPC clients/servers isn't a showstopper but could also be
>> nice
>>
>
> Binary RPC over HTTP is the supported mode today.  We'd eventually like to
> standardize on a second, non-HTTP, TCP-based high-performance, async,
> secure, RPC transport.  There are some TCP-based transports implemented that
> support request-only messages and async i/o, but these are non-standard and
> might not provide a good basis for long-term interoperability.
>
> I'd really like to have json over HTTP as well, even if not as the core
service transport (binary over http is fine), then at least for debugging a
remote service. So it doesn't have to be performant, just work, is there
something ready? (I guess I can write a silly one myself...)

>
>  2. As a serialization mechanism to send messages over AMQ or save object
>> on memcached
>>
>
> This should work well today.
>
>
>  3. Perhaps use for saving space in logs (rather than plain text log
>> files) and in combination with flume. I haven't thought that out yet.
>>
>
> Log files are a good use of Avro today.  Avro 1.4.0 will be released soon
> and contains APIs to write MapReduce programs that process Avro data files.
>  Support for Avro in Flume is expected soon.
>
> Doug
>

Re: is avro production ready?

[Doug Cutting <cu...@apache.org>]

On 07/19/2010 11:43 AM, Ran Tavory wrote:
> I'd like to use avro in several use cases:
> 1. RPC. Both using json over HTTP and binary over TCP (will use both
> modes). Async RPC clients/servers isn't a showstopper but could also be nice

Binary RPC over HTTP is the supported mode today.  We'd eventually like 
to standardize on a second, non-HTTP, TCP-based high-performance, async, 
secure, RPC transport.  There are some TCP-based transports implemented 
that support request-only messages and async i/o, but these are 
non-standard and might not provide a good basis for long-term 
interoperability.

> 2. As a serialization mechanism to send messages over AMQ or save object
> on memcached

This should work well today.

> 3. Perhaps use for saving space in logs (rather than plain text log
> files) and in combination with flume. I haven't thought that out yet.

Log files are a good use of Avro today.  Avro 1.4.0 will be released 
soon and contains APIs to write MapReduce programs that process Avro 
data files.  Support for Avro in Flume is expected soon.

Doug