You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Doug Cutting <cu...@apache.org> on 2009/04/03 00:05:08 UTC

[PROPOSAL] new subproject: Avro

I propose we add a new Hadoop subproject for Avro, a serialization 
system.  My ambition is for Avro to replace both Hadoop's RPC and to be 
used for most Hadoop data files, e.g., by Pig, Hive, etc.

Initial committers would be Sharad Agarwal and me, both existing Hadoop 
committers.  We are the sole authors of this software to date.

The code is currently at:

http://people.apache.org/~cutting/avro.git/

To learn more:

git clone http://people.apache.org/~cutting/avro.git/ avro
cat avro/README.txt

Comments?  Questions?

Doug

Re: [PROPOSAL] new subproject: Avro

Posted by Bryan Duxbury <br...@rapleaf.com>.
> Field ids are not present in Avro data except in the schema.  A  
> record's fields are serialized in the order that the fields occur  
> in the records schema, with no per-field annotations whatsoever.   
> For example, a record that contains a string and an int is  
> serialized simply as a string followed by an int, nothing before,  
> nothing between and nothing after. So, yes, it is a different data  
> format.

So you can't serialize nulls? It also seems like this would make  
forward/backward compatibility a little more complex. Thrift solves  
this problem by using tags to indicate what kind of field you're  
working with.

-Bryan

Re: [PROPOSAL] new subproject: Avro

Posted by George Porter <Ge...@Sun.COM>.
On Apr 3, 2009, at 1:02 PM, Doug Cutting wrote:

> George Porter wrote:
>> While this representation would certainly be as compact as  
>> possible, wouldn't it prevent evolving the data structure over  
>> time?  One of the nice features of Google Protocol Buffers and  
>> Thrift is that you can evolve the set of fields over time, and  
>> older/newer clients can talk to older/newer services.  If the  
>> proposed Avro is evolvable, then perhaps I'm misunderstanding your  
>> statement about the lack of IDs in the serialized data.
>
> Avro supports schema evolution.  In Avro, the schema used to write  
> the data must be available when the data is read.  (In files, it is  
> typically stored in the file metadata.)
>
> If you have the schema that was used to write the data, and you're  
> expecting a slightly different schema, then you simply keep those  
> fields that are in both schemas and skip those not.  This is  
> equivalent to Thrift and Protocol Buffer's support for schema  
> evolution, but does not require manually assigning numeric field ids.
>
> This feature can also be used to support projection.  If you have  
> records with many large fields, but only need a single field in a  
> particular computation, then you can specify an expected schema with  
> only that field, and the runtime will efficiently skip all of the  
> other fields, returning a record with just the single, expected field.

Thanks for the clarification--I better understand the schema  
relationship.  The projection feature is a nice feature, especially  
since it seems like it would be able to support "sparse files" where  
you want to just peek at large structs without invoking a lot of disk- 
io (for data serialized on-disk).

>
>
>> I also agree with Bryan, in that it would be unfortunate to have  
>> two different Apache projects with overlapping goals.
>
> We already have both Thrift and Etch in the incubator, which have  
> similar goals.  Apache does not attempt to mandate that projects  
> have disjoint goals.  There are many ways to slice things, and  
> Apache prefers to rely on survival of the fittest rather than  
> forcing things together.
>
>> Regardless of features, both protocol buffers and thrift have the  
>> advantage of being debugged in mission-critical production  
>> environments.
>
> Yes, but, as I've argued in other messages in this thread, they do  
> not support the dynamic features we need.  Adding those features  
> would add new code that would share little with existing code in  
> those projects. So, while the projects are conceptually similar, the  
> implementations are necessarily different, and, without significant  
> code overlap, separate projects seem more natural.
>
> Doug

Makes sense.  Thanks,
George

Re: [PROPOSAL] new subproject: Avro

Posted by Doug Cutting <cu...@apache.org>.
George Porter wrote:
> While this representation would certainly be as compact as possible, 
> wouldn't it prevent evolving the data structure over time?  One of the 
> nice features of Google Protocol Buffers and Thrift is that you can 
> evolve the set of fields over time, and older/newer clients can talk to 
> older/newer services.  If the proposed Avro is evolvable, then perhaps 
> I'm misunderstanding your statement about the lack of IDs in the 
> serialized data.

Avro supports schema evolution.  In Avro, the schema used to write the 
data must be available when the data is read.  (In files, it is 
typically stored in the file metadata.)

If you have the schema that was used to write the data, and you're 
expecting a slightly different schema, then you simply keep those fields 
that are in both schemas and skip those not.  This is equivalent to 
Thrift and Protocol Buffer's support for schema evolution, but does not 
require manually assigning numeric field ids.

This feature can also be used to support projection.  If you have 
records with many large fields, but only need a single field in a 
particular computation, then you can specify an expected schema with 
only that field, and the runtime will efficiently skip all of the other 
fields, returning a record with just the single, expected field.

> I also agree with Bryan, in that it would be unfortunate to have two 
> different Apache projects with overlapping goals.

We already have both Thrift and Etch in the incubator, which have 
similar goals.  Apache does not attempt to mandate that projects have 
disjoint goals.  There are many ways to slice things, and Apache prefers 
to rely on survival of the fittest rather than forcing things together.

> Regardless of 
> features, both protocol buffers and thrift have the advantage of being 
> debugged in mission-critical production environments.

Yes, but, as I've argued in other messages in this thread, they do not 
support the dynamic features we need.  Adding those features would add 
new code that would share little with existing code in those projects. 
So, while the projects are conceptually similar, the implementations are 
necessarily different, and, without significant code overlap, separate 
projects seem more natural.

Doug

Re: [PROPOSAL] new subproject: Avro

Posted by Scott Carey <sc...@richrelevance.com>.
On 4/3/09 12:03 PM, "George Porter" <Ge...@Sun.COM> wrote:

> 
> 
> On Apr 3, 2009, at 11:37 AM, Doug Cutting wrote:
>>> 
>> 
>> Field ids are not present in Avro data except in the schema.  A
>> record's fields are serialized in the order that the fields occur in
>> the records schema, with no per-field annotations whatsoever.  For
>> example, a record that contains a string and an int is serialized
>> simply as a string followed by an int, nothing before, nothing
>> between and nothing after. So, yes, it is a different data format.
> 
> While this representation would certainly be as compact as possible,
> wouldn't it prevent evolving the data structure over time?  One of the
> nice features of Google Protocol Buffers and Thrift is that you can
> evolve the set of fields over time, and older/newer clients can talk
> to older/newer services.  If the proposed Avro is evolvable, then
> perhaps I'm misunderstanding your statement about the lack of IDs in
> the serialized data.

>From a quick perusal of the serialization format -- it contains headers with
type/schema information, and other metadata blocks.  The types can be
inferred from this, and if this is done right then older/newer clients will
be able to read things just fine.  What can't be done is mixing two
different formats in the same stream if headers define the format of the
whole stream.

I have not looked much deeper than that, but it looks like schema evolution
is feasible.

> 
> I also agree with Bryan, in that it would be unfortunate to have two
> different Apache projects with overlapping goals.  Regardless of
> features, both protocol buffers and thrift have the advantage of being
> debugged in mission-critical production environments.
> 
> -George
> 


Re: [PROPOSAL] new subproject: Avro

Posted by George Porter <Ge...@Sun.COM>.
On Apr 3, 2009, at 11:37 AM, Doug Cutting wrote:
>>
>
> Field ids are not present in Avro data except in the schema.  A  
> record's fields are serialized in the order that the fields occur in  
> the records schema, with no per-field annotations whatsoever.  For  
> example, a record that contains a string and an int is serialized  
> simply as a string followed by an int, nothing before, nothing  
> between and nothing after. So, yes, it is a different data format.

While this representation would certainly be as compact as possible,  
wouldn't it prevent evolving the data structure over time?  One of the  
nice features of Google Protocol Buffers and Thrift is that you can  
evolve the set of fields over time, and older/newer clients can talk  
to older/newer services.  If the proposed Avro is evolvable, then  
perhaps I'm misunderstanding your statement about the lack of IDs in  
the serialized data.

I also agree with Bryan, in that it would be unfortunate to have two  
different Apache projects with overlapping goals.  Regardless of  
features, both protocol buffers and thrift have the advantage of being  
debugged in mission-critical production environments.

-George

Re: [PROPOSAL] new subproject: Avro

Posted by Doug Cutting <cu...@apache.org>.
Bryan Duxbury wrote:
> It's not actually a different data format, is it? You're saying that the 
> user wouldn't specify the field IDs, but you'd fundamentally still use 
> field ids for compactness and the like.

Field ids are not present in Avro data except in the schema.  A record's 
fields are serialized in the order that the fields occur in the records 
schema, with no per-field annotations whatsoever.  For example, a record 
that contains a string and an int is serialized simply as a string 
followed by an int, nothing before, nothing between and nothing after. 
So, yes, it is a different data format.

> The bottom line is that I would love to see greater cooperation between 
> Hadoop and Thrift. Unless it's impossible or impractical for Thrift to 
> be useful here, I think we'd be willing to work towards Hadoop's needs.

Perhaps Thrift could be augmented to support Avro's JSON schemas and 
serialization.  Then it could interoperate with other Avro-based 
systems.  But then Thrift would have yet another serialization format, 
that every language would need to implement for it to be useful...

Avro will only ever have one serialization format.  Thrift fundamentally 
standardizes an API, not a data format.  Avro fundamentally is a data 
format specification, like XML.  Thrift could implement this 
specification.  The Avro project includes reference implementations, but 
the format is intended to be simple enough and the specification stable 
enough that others might reasonably develop alternate, independent 
implementations.

Doug

Re: [PROPOSAL] new subproject: Avro

Posted by Bryan Duxbury <br...@rapleaf.com>.
> With the schema in hand, you don't need to tag data with field  
> numbers or types, since that's all there in the schema.  So, having  
> the schema, you can use a simpler data format.

To a degree, we already have that in Thrift - we call it the  
DenseProtocol.

> Would you write parsers for Thrift's IDL in every language?  Or  
> would you use JSON, as Avro does, to avoid that?

When it comes to having a code-usable IDL for the schema, I'm totally  
pro-JSON.

> Once you're using a different IDL and a different data format,  
> what's shared with Thrift?  Fundamentally, those two things define  
> a serialization system, no?

It's not actually a different data format, is it? You're saying that  
the user wouldn't specify the field IDs, but you'd fundamentally  
still use field ids for compactness and the like. You may not use  
actual Thrift generated objects, but you could certainly use Binary  
or Compact protocol from Thrift to do all the writing to the wire.

You might also be able to use (or contribute to) Thrift's RPC-level  
stuff like server implementations. We have some respectable Java  
servers written, and if those aren't enough for your uses, I'd  
actually be really interested in seeing if we could generalize some  
of the Hadoop stuff to be useful within Thrift.

The bottom line is that I would love to see greater cooperation  
between Hadoop and Thrift. Unless it's impossible or impractical for  
Thrift to be useful here, I think we'd be willing to work towards  
Hadoop's needs.

-Bryan


Re: [PROPOSAL] new subproject: Avro

Posted by Doug Cutting <cu...@apache.org>.
Bryan Duxbury wrote:
> It sounds like what you want is the option avoid pre-generated classes.

That's part of it.  But, once you have the schema, you might as well 
take advantage of it.

With the schema in hand, you don't need to tag data with field numbers 
or types, since that's all there in the schema.  So, having the schema, 
you can use a simpler data format.

Also, with the schema, resolving version differences is simplified. 
Developers don't need to assign field numbers, but can just use names. 
For performance, one can internally use field numbers while reading, to 
avoid string comparisons, but developers need no longer specify these, 
but can use names, as in most software.  Here having the schema means we 
can simplify the IDL and its versioning semantics.

> If that's the only thing you need, it seems like we could bolt that on 
> to Thrift with almost no work.

Would you write parsers for Thrift's IDL in every language?  Or would 
you use JSON, as Avro does, to avoid that?

Once you're using a different IDL and a different data format, what's 
shared with Thrift?  Fundamentally, those two things define a 
serialization system, no?

> I assume you'd have the schema stored in 
> metadata or file header or something, right? (You wouldn't want to store 
> the field names in the binary encoding as strings, since that would 
> probably very quickly dwarf the size of the actual data in a lot of cases.)

Yes, in data files the schema is typically stored in the metadata.

> If my assumptions are correct, it seems like it'd be a lot smarter to 
> leverage existing Thrift infrastructure and encoding work rather than 
> duplicating it for this lone feature.

What specific shared infrastructure would be leveraged?  For Hadoop's 
RPC, I hope to adapt Hadoop's client and server implementations as a 
transport, as these have been highly tuned for Hadoop's performance 
requirements.

Doug

Re: [PROPOSAL] new subproject: Avro

Posted by Doug Cutting <cu...@apache.org>.
I have responded to this on general at hadoop.apache.org.  Let's not 
cross-post, but rather keep this discussion on a single list.  I think 
general@ is the best list for this discussion.

http://mail-archives.apache.org/mod_mbox/hadoop-general/200904.mbox/browser

Thanks,

Doug

Re: [PROPOSAL] new subproject: Avro

Posted by Bryan Duxbury <br...@rapleaf.com>.
It sounds like what you want is the option avoid pre-generated  
classes. If that's the only thing you need, it seems like we could  
bolt that on to Thrift with almost no work. I assume you'd have the  
schema stored in metadata or file header or something, right? (You  
wouldn't want to store the field names in the binary encoding as  
strings, since that would probably very quickly dwarf the size of the  
actual data in a lot of cases.)

If my assumptions are correct, it seems like it'd be a lot smarter to  
leverage existing Thrift infrastructure and encoding work rather than  
duplicating it for this lone feature.

-Bryan

On Apr 3, 2009, at 9:06 AM, Doug Cutting wrote:

> Owen O'Malley wrote:
>> 2. Protocol buffers (and thrift) encode the field names as id  
>> numbers. That means that if you read them into dynamic language  
>> like Python that it has to use the field numbers instead of the  
>> field names. In Avro, the field names are saved and there are no  
>> field ids.
>
> This hints at a related problem with Thrift and Protocol Buffers,  
> which is that they require one to generate code for each datatype  
> one processes.  This is awkward in dynamic environments, where one  
> would like to write a script (Pig, Python, Perl, Hive, whatever) to  
> process input data and generate output data, without having to  
> locate the IDL for each input file, run an IDL compiler, load the  
> generated code, generate an IDL file for the output, run the  
> compiler again, load the output code and finally write your  
> output.  Avro rather lets you simply open your inputs, examine  
> their datatypes, specify output types and write them.
>
> Avro's Java implementation currently includes three different data  
> representations:
>
>  - a "generic" representation uses a standard set of datastructures  
> for all datatypes: records are represented as Map<String,Object>,  
> arrays as List<Object>, longs as Long, etc.
>
>  - a "reflect" representation uses Java reflection to permit one to  
> read and write existing Java classes with Avro.
>
>  - a "specific" representation generates Java classes that are  
> compiled and loaded, much like Thrift and Protocol Buffers.
>
> We don't expect most scripting languages to use more than a single  
> representation.  Implementing Avro is quite simple, by design.  We  
> have a Python implementation, and hope to add more soon.
>
> Doug


Re: [PROPOSAL] new subproject: Avro

Posted by Bryan Duxbury <br...@rapleaf.com>.
It sounds like what you want is the option avoid pre-generated  
classes. If that's the only thing you need, it seems like we could  
bolt that on to Thrift with almost no work. I assume you'd have the  
schema stored in metadata or file header or something, right? (You  
wouldn't want to store the field names in the binary encoding as  
strings, since that would probably very quickly dwarf the size of the  
actual data in a lot of cases.)

If my assumptions are correct, it seems like it'd be a lot smarter to  
leverage existing Thrift infrastructure and encoding work rather than  
duplicating it for this lone feature.

-Bryan

On Apr 3, 2009, at 9:06 AM, Doug Cutting wrote:

> Owen O'Malley wrote:
>> 2. Protocol buffers (and thrift) encode the field names as id  
>> numbers. That means that if you read them into dynamic language  
>> like Python that it has to use the field numbers instead of the  
>> field names. In Avro, the field names are saved and there are no  
>> field ids.
>
> This hints at a related problem with Thrift and Protocol Buffers,  
> which is that they require one to generate code for each datatype  
> one processes.  This is awkward in dynamic environments, where one  
> would like to write a script (Pig, Python, Perl, Hive, whatever) to  
> process input data and generate output data, without having to  
> locate the IDL for each input file, run an IDL compiler, load the  
> generated code, generate an IDL file for the output, run the  
> compiler again, load the output code and finally write your  
> output.  Avro rather lets you simply open your inputs, examine  
> their datatypes, specify output types and write them.
>
> Avro's Java implementation currently includes three different data  
> representations:
>
>  - a "generic" representation uses a standard set of datastructures  
> for all datatypes: records are represented as Map<String,Object>,  
> arrays as List<Object>, longs as Long, etc.
>
>  - a "reflect" representation uses Java reflection to permit one to  
> read and write existing Java classes with Avro.
>
>  - a "specific" representation generates Java classes that are  
> compiled and loaded, much like Thrift and Protocol Buffers.
>
> We don't expect most scripting languages to use more than a single  
> representation.  Implementing Avro is quite simple, by design.  We  
> have a Python implementation, and hope to add more soon.
>
> Doug


Re: [PROPOSAL] new subproject: Avro

Posted by Bryan Duxbury <br...@rapleaf.com>.
It sounds like what you want is the option avoid pre-generated  
classes. If that's the only thing you need, it seems like we could  
bolt that on to Thrift with almost no work. I assume you'd have the  
schema stored in metadata or file header or something, right? (You  
wouldn't want to store the field names in the binary encoding as  
strings, since that would probably very quickly dwarf the size of the  
actual data in a lot of cases.)

If my assumptions are correct, it seems like it'd be a lot smarter to  
leverage existing Thrift infrastructure and encoding work rather than  
duplicating it for this lone feature.

-Bryan

On Apr 3, 2009, at 9:06 AM, Doug Cutting wrote:

> Owen O'Malley wrote:
>> 2. Protocol buffers (and thrift) encode the field names as id  
>> numbers. That means that if you read them into dynamic language  
>> like Python that it has to use the field numbers instead of the  
>> field names. In Avro, the field names are saved and there are no  
>> field ids.
>
> This hints at a related problem with Thrift and Protocol Buffers,  
> which is that they require one to generate code for each datatype  
> one processes.  This is awkward in dynamic environments, where one  
> would like to write a script (Pig, Python, Perl, Hive, whatever) to  
> process input data and generate output data, without having to  
> locate the IDL for each input file, run an IDL compiler, load the  
> generated code, generate an IDL file for the output, run the  
> compiler again, load the output code and finally write your  
> output.  Avro rather lets you simply open your inputs, examine  
> their datatypes, specify output types and write them.
>
> Avro's Java implementation currently includes three different data  
> representations:
>
>  - a "generic" representation uses a standard set of datastructures  
> for all datatypes: records are represented as Map<String,Object>,  
> arrays as List<Object>, longs as Long, etc.
>
>  - a "reflect" representation uses Java reflection to permit one to  
> read and write existing Java classes with Avro.
>
>  - a "specific" representation generates Java classes that are  
> compiled and loaded, much like Thrift and Protocol Buffers.
>
> We don't expect most scripting languages to use more than a single  
> representation.  Implementing Avro is quite simple, by design.  We  
> have a Python implementation, and hope to add more soon.
>
> Doug


Re: [PROPOSAL] new subproject: Avro

Posted by Doug Cutting <cu...@apache.org>.
Owen O'Malley wrote:
> 2. Protocol buffers (and thrift) encode the field names as id numbers. 
> That means that if you read them into dynamic language like Python that 
> it has to use the field numbers instead of the field names. In Avro, the 
> field names are saved and there are no field ids.

This hints at a related problem with Thrift and Protocol Buffers, which 
is that they require one to generate code for each datatype one 
processes.  This is awkward in dynamic environments, where one would 
like to write a script (Pig, Python, Perl, Hive, whatever) to process 
input data and generate output data, without having to locate the IDL 
for each input file, run an IDL compiler, load the generated code, 
generate an IDL file for the output, run the compiler again, load the 
output code and finally write your output.  Avro rather lets you simply 
open your inputs, examine their datatypes, specify output types and 
write them.

Avro's Java implementation currently includes three different data 
representations:

  - a "generic" representation uses a standard set of datastructures for 
all datatypes: records are represented as Map<String,Object>, arrays as 
List<Object>, longs as Long, etc.

  - a "reflect" representation uses Java reflection to permit one to 
read and write existing Java classes with Avro.

  - a "specific" representation generates Java classes that are compiled 
and loaded, much like Thrift and Protocol Buffers.

We don't expect most scripting languages to use more than a single 
representation.  Implementing Avro is quite simple, by design.  We have 
a Python implementation, and hope to add more soon.

Doug

Re: [PROPOSAL] new subproject: Avro

Posted by Owen O'Malley <om...@apache.org>.
On Apr 2, 2009, at 5:11 PM, Abhishek Verma wrote:

> I am a newbie here. Why not use something existing like protocol  
> buffers :
> http://code.google.com/p/protobuf/ which is open source and works  
> amazingly
> well.

There are two blockers for protocol buffers that make them suboptimal  
for Hadoop. They are:

1. Protocol buffers are open source, but the community isn't open.  
Google doesn't seem interested in getting patches from outside of  
itself. If we needed something changed in protocol buffers, we'd end  
up needing to fork the project to make any progress.

2. Protocol buffers (and thrift) encode the field names as id numbers.  
That means that if you read them into dynamic language like Python  
that it has to use the field numbers instead of the field names. In  
Avro, the field names are saved and there are no field ids.

A final point is that since the schema isn't inlined in Avro, the  
binary representation is much tighter than protocol buffers.

-- Owen

Re: [PROPOSAL] new subproject: Avro

Posted by Abhishek Verma <ve...@gmail.com>.
I am a newbie here. Why not use something existing like protocol buffers :
http://code.google.com/p/protobuf/ which is open source and works amazingly
well.

On Thu, Apr 2, 2009 at 5:20 PM, Owen O'Malley <om...@apache.org> wrote:

>
> On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
>
>  I propose we add a new Hadoop subproject for Avro, a serialization system.
>>
>
> +1. Even if Justin comes after me for allowing another Jakarta. *smile*
>
> -- Owen
>

Re: [PROPOSAL] new subproject: Avro

Posted by Owen O'Malley <om...@apache.org>.
On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:

> I propose we add a new Hadoop subproject for Avro, a serialization  
> system.

+1. Even if Justin comes after me for allowing another Jakarta. *smile*

-- Owen

Re: [PROPOSAL] new subproject: Avro

Posted by Tom White <to...@gmail.com>.
On Thu, Apr 2, 2009 at 11:05 PM, Doug Cutting <cu...@apache.org> wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization system.

+1

Tom

>  My ambition is for Avro to replace both Hadoop's RPC and to be used for
> most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing Hadoop
> committers.  We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments?  Questions?
>
> Doug
>

Re: [PROPOSAL] new subproject: Avro

Posted by Dhruba Borthakur <dh...@gmail.com>.
+1. Awesome!

-dhruba

On Sun, Apr 5, 2009 at 10:33 PM, Sanjay Radia <sr...@yahoo-inc.com> wrote:

>
> On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
>
>  I propose we add a new Hadoop subproject for Avro, a serialization
>> system.  My ambition is for Avro to replace both Hadoop's RPC and to be
>> used for most Hadoop data files, e.g., by Pig, Hive, etc.
>>
>> Initial committers would be Sharad Agarwal and me, both existing Hadoop
>> committers.  We are the sole authors of this software to date.
>>
>> The code is currently at:
>>
>> http://people.apache.org/~cutting/avro.git/<http://people.apache.org/%7Ecutting/avro.git/>
>>
>> To learn more:
>>
>> git clone http://people.apache.org/~cutting/avro.git/<http://people.apache.org/%7Ecutting/avro.git/>avro
>> cat avro/README.txt
>>
>> Comments?  Questions?
>>
>> Doug
>>
>>  +1
>
> sanjay

Re: [PROPOSAL] new subproject: Avro

Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:

> I propose we add a new Hadoop subproject for Avro, a serialization
> system.  My ambition is for Avro to replace both Hadoop's RPC and to  
> be
> used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing  
> Hadoop
> committers.  We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments?  Questions?
>
> Doug
>
+1

sanjay

Re: [PROPOSAL] new subproject: Avro

Posted by Doug Cutting <cu...@apache.org>.
Doug Cutting wrote:
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt

I've posted the generated documentation.  The specification is at:

http://people.apache.org/~cutting/avro/spec.html

and the javadoc is at:

http://people.apache.org/~cutting/avro/api/index.html

Doug

Re: [PROPOSAL] new subproject: Avro

Posted by Doug Cutting <cu...@apache.org>.
Jim Kellerman (POWERSET) wrote:
> It is also my understanding (based on the email thread) that Avro only
> supports Java and python. That is a step backwards from Thrift.

We intend to add support for more languages.  Avro is not complete.

> It appears that Avro uses introspection heavily, which is expensive in
> applications that require a high message rate.

It only uses introspection if you wish to use your existing Java classes 
to represent Avro data.  There are three representations in Java: 
generic (uses Map<String,Object> for records, List<Object> for arrays), 
specific (generates a java class for each Avro record, like Thrift) and 
reflect (uses reflection to access existing classes).  So introspection 
is optional.  And, while introspection is indeed slow for processing 
file-based data, it would probably not a bottleneck for most RPC 
protocols and might be a useful tool to migrate existing code to Avro.

> So I guess my question is why Avro?

The compelling case is dynamic data types.  Pig, Hive, Python, Perl etc. 
scripts should not have to generate a Thrift IDL file each time they 
wish to write a data file with a new schema, nor should they need to run 
the Thrift compiler for each data file they wish to read.  For 
production applications, code-generation is not an imposition and may 
offer increased opportunities for optimization and error checking, but 
for exploration and experimentation, a very common use case for Hadoop, 
one would like to be able to browse datasets and build mapreduce 
programs more interactively.

Doug

RE: [PROPOSAL] new subproject: Avro

Posted by "Jim Kellerman (POWERSET)" <Ji...@microsoft.com>.
> -----Original Message-----
> On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
> 
> I propose we add a new Hadoop subproject for Avro, a serialization
> system.  My ambition is for Avro to replace both Hadoop's RPC and to
> be used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing
> Hadoop committers.  We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments?  Questions?
>
> Doug

After reading all the messages about Avro, I'm still not sure I understand
why we should invent "yet another wheel". There are a number of people in
the community who have significant investments in Thrift, and I have yet
to see a compelling argument for Avro over Thrift.

My understanding is that Thrift already supports multi-language bindings,
something the HBase community has been asking for, for some time.

It is also my understanding (based on the email thread) that Avro only
supports Java and python. That is a step backwards from Thrift.

It appears that Avro uses introspection heavily, which is expensive in
applications that require a high message rate.

So I guess my question is why Avro? I may be thick, but it seems to me
as if it is just another wheel of a different color. If I could see a
point by point comparison between Avro and Thrift I could be convinced
that Avro is the way to go. So far, I have not seen any compelling reason
to re-invent the wheel.

+-0

---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)




Re: [PROPOSAL] new subproject: Avro

Posted by Nigel Daley <nd...@yahoo-inc.com>.
+1.

On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:

> I propose we add a new Hadoop subproject for Avro, a serialization  
> system.  My ambition is for Avro to replace both Hadoop's RPC and to  
> be used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing  
> Hadoop committers.  We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments?  Questions?
>
> Doug


Re: [PROPOSAL] new subproject: Avro

Posted by Alan Gates <ga...@yahoo-inc.com>.
+1.  Pig would be happy to use a cross language serialization package  
that did not require pre-compilation to read and write.

Alan.

On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:

> I propose we add a new Hadoop subproject for Avro, a serialization  
> system.  My ambition is for Avro to replace both Hadoop's RPC and to  
> be used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing  
> Hadoop committers.  We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments?  Questions?
>
> Doug


Re: [RESULT] new subproject: Avro

Posted by Doug Cutting <cu...@apache.org>.
The mailing lists are now created.

If you are interested you may subscribe by sending messages to:
   avro-dev-subscribe at hadoop.apache.org
   avro-user-subscribe at hadoop.apache.org
   avro-commits-subscribe at hadoop.apache.org

Please join one or more of these lists if you are interested in Avro.  I 
do not intend to send more messages to general@ about Avro in the near 
future.

I've also uploaded the code to subversion, at:

   http://svn.apache.org/repos/asf/hadoop/avro/trunk/

A Jira bug database is at:

   https://issues.apache.org/jira/browse/AVRO

I will create a website and Wiki ASAP.

Doug

Doug Cutting wrote:
> Doug Cutting wrote:
>> Three days have now passed since I initially proposed this, the 
>> nominal time for an Apache vote.  Is there anyone who strongly opposes 
>> taking the development of Avro public as a Hadoop subproject?
> 
> With 6 +1 votes from the PMC (Alan, Dhruba, Nigel, Owen, Tom & me) and 
> no -1 votes, this has passed.  I will now ask infrastructure to create 
> the mailing lists, and create the Jira instance, svn repo, website & 
> wiki myself.
> 
> I'll send a message to this list once this is complete and the project 
> is open for business.
> 
> Thanks,
> 
> Doug

[RESULT] new subproject: Avro

Posted by Doug Cutting <cu...@apache.org>.
Doug Cutting wrote:
> Three days have now passed since I initially proposed this, the nominal 
> time for an Apache vote.  Is there anyone who strongly opposes taking 
> the development of Avro public as a Hadoop subproject?

With 6 +1 votes from the PMC (Alan, Dhruba, Nigel, Owen, Tom & me) and 
no -1 votes, this has passed.  I will now ask infrastructure to create 
the mailing lists, and create the Jira instance, svn repo, website & 
wiki myself.

I'll send a message to this list once this is complete and the project 
is open for business.

Thanks,

Doug

Re: [PROPOSAL] new subproject: Avro

Posted by Chad Walters <ch...@yahoo.com>.
Doug,

After our off-list chat, and given that you have indicated that the design is still in flux and that you are open to discussing changes that would permit interoperability, I am not as concerned as I was.

My urgency came from concern that once the design was put in place as part of an Apache subproject, rather than open sourced in some other less prominent forum, it would increase the barrier to interoperability; in particular, I was concerned that people would assume the design of the data format was fully-baked and start persisting large amounts of data in some early version of the format, potentially prematurely ossifying the design in a state unsuited for compatibility with Thrift. Given your clarifications around this, my fears clearly were not well-founded.

Please accept my apology if I came across as obstructionist. I was honestly advocating on behalf of what I believe is in the best interest our shared user base.

Clearly we have some disagreements about the value of some of Thrift's design choices and what those mean for various use cases. I think we also have some differences of opinion about the relative difficulty of implementation versus the value of interoperability. Hopefully, the next few months will afford an opportunity to examine the sources of those disagreements and see if they can be resolved.

Sincerely,

Chad



----- Original Message ----
From: Doug Cutting <cu...@apache.org>
To: general@hadoop.apache.org
Sent: Tuesday, April 7, 2009 8:33:32 PM
Subject: Re: [PROPOSAL] new subproject: Avro

To be clear, since a few folks have missed this point: Avro is not complete.  At some point in the future, before people start using it as a format for persistent data, we'll need to stop altering its specification, or at least do so much more cautiously.  But before then, my immediate goal to move development from private to open so that we have a chance to incorporate feedback before we lock down the specification.

For example, several folks have raised the issue of compatibility with Thrift.  We certainly want to avoid gratuitous incompatibilities.  There are also features clearly missing from Avro that we expect to add before we make a release, like default values, a more efficient RPC handshake, etc.  And some features that we might consider removing, if they're not broadly useful and inhibit interoperability, like single-float, which isn't in Thrift, Python, etc.  And I expect there will be more such issues raised in the coming weeks and months.

But before we can discuss and resolve such issues we need a forum in which to do so.  That's all I am after at this point: mailing lists, a bug database, a public source code repository, etc., so that we can start accepting patches, adding committers, etc.

Three days have now passed since I initially proposed this, the nominal time for an Apache vote.  Is there anyone who strongly opposes taking the development of Avro public as a Hadoop subproject?  Only PMC votes are binding, but I would vastly prefer that the broader community also supports this step in the process.

Thanks,

Doug

Doug Cutting wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization system.  My ambition is for Avro to replace both Hadoop's RPC and to be used for most Hadoop data files, e.g., by Pig, Hive, etc.
> 
> Initial committers would be Sharad Agarwal and me, both existing Hadoop committers.  We are the sole authors of this software to date.
> 
> The code is currently at:
> 
> http://people.apache.org/~cutting/avro.git/
> 
> To learn more:
> 
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
> 
> Comments?  Questions?
> 
> Doug


Re: [PROPOSAL] new subproject: Avro

Posted by Doug Cutting <cu...@apache.org>.
To be clear, since a few folks have missed this point: Avro is not 
complete.  At some point in the future, before people start using it as 
a format for persistent data, we'll need to stop altering its 
specification, or at least do so much more cautiously.  But before then, 
my immediate goal to move development from private to open so that we 
have a chance to incorporate feedback before we lock down the specification.

For example, several folks have raised the issue of compatibility with 
Thrift.  We certainly want to avoid gratuitous incompatibilities.  There 
are also features clearly missing from Avro that we expect to add before 
we make a release, like default values, a more efficient RPC handshake, 
etc.  And some features that we might consider removing, if they're not 
broadly useful and inhibit interoperability, like single-float, which 
isn't in Thrift, Python, etc.  And I expect there will be more such 
issues raised in the coming weeks and months.

But before we can discuss and resolve such issues we need a forum in 
which to do so.  That's all I am after at this point: mailing lists, a 
bug database, a public source code repository, etc., so that we can 
start accepting patches, adding committers, etc.

Three days have now passed since I initially proposed this, the nominal 
time for an Apache vote.  Is there anyone who strongly opposes taking 
the development of Avro public as a Hadoop subproject?  Only PMC votes 
are binding, but I would vastly prefer that the broader community also 
supports this step in the process.

Thanks,

Doug

Doug Cutting wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization 
> system.  My ambition is for Avro to replace both Hadoop's RPC and to be 
> used for most Hadoop data files, e.g., by Pig, Hive, etc.
> 
> Initial committers would be Sharad Agarwal and me, both existing Hadoop 
> committers.  We are the sole authors of this software to date.
> 
> The code is currently at:
> 
> http://people.apache.org/~cutting/avro.git/
> 
> To learn more:
> 
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
> 
> Comments?  Questions?
> 
> Doug

Re: [PROPOSAL] new subproject: Avro

Posted by Sameer Paranjpye <sp...@yahoo.com>.
+1

While protocol buffers and thrift have similar goals. Avro takes a different approach to schema evolution and reconciliation. I feel that Avros tighter layout of data and schema management is better suited for many of Hadoop and Pigs use cases for large data sets/tables on HDFS. Field ids start to matter when milions of objects have hundreds of fields each. There is, of course, the storage overhead. Schema management becomes hard especially if there are cases where field ids need to be assigned manually.


----- Original Message ----
From: Doug Cutting <cu...@apache.org>
To: general@hadoop.apache.org
Sent: Thursday, April 2, 2009 3:05:08 PM
Subject: [PROPOSAL] new subproject: Avro

I propose we add a new Hadoop subproject for Avro, a serialization system.  My ambition is for Avro to replace both Hadoop's RPC and to be used for most Hadoop data files, e.g., by Pig, Hive, etc.

Initial committers would be Sharad Agarwal and me, both existing Hadoop committers.  We are the sole authors of this software to date.

The code is currently at:

http://people.apache.org/~cutting/avro.git/

To learn more:

git clone http://people.apache.org/~cutting/avro.git/ avro
cat avro/README.txt

Comments?  Questions?

Doug



      

Re: [PROPOSAL] new subproject: Avro

Posted by Jerome Boulon <jb...@yahoo-inc.com>.
Chukwa needs a cross language serialization/RPC framework.
During demux, being able to create dynamically the schema based on the recordType will be a big plus instead of using an hardcoded one,
The projection feature will also be useful.

+1

/Jerome.


On 4/2/09 3:05 PM, "Doug Cutting" <cu...@apache.org> wrote:

I propose we add a new Hadoop subproject for Avro, a serialization
system.  My ambition is for Avro to replace both Hadoop's RPC and to be
used for most Hadoop data files, e.g., by Pig, Hive, etc.

Initial committers would be Sharad Agarwal and me, both existing Hadoop
committers.  We are the sole authors of this software to date.

The code is currently at:

http://people.apache.org/~cutting/avro.git/

To learn more:

git clone http://people.apache.org/~cutting/avro.git/ avro
cat avro/README.txt

Comments?  Questions?

Doug


Re: [PROPOSAL] new subproject: Avro

Posted by Brian Forney <bf...@integral7.com>.
On 4/2/09 5:05 PM, "Doug Cutting" <cu...@apache.org> wrote:

> I propose we add a new Hadoop subproject for Avro, a serialization
> system.  My ambition is for Avro to replace both Hadoop's RPC and to be
> used for most Hadoop data files, e.g., by Pig, Hive, etc.
> 
> Initial committers would be Sharad Agarwal and me, both existing Hadoop
> committers.  We are the sole authors of this software to date.
> 
> The code is currently at:
> 
> http://people.apache.org/~cutting/avro.git/
> 
> To learn more:
> 
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
> 
> Comments?  Questions?
> 
> Doug
> 

+1