You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Doug Cutting <cu...@apache.org> on 2009/04/03 00:05:08 UTC
[PROPOSAL] new subproject: Avro
I propose we add a new Hadoop subproject for Avro, a serialization
system. My ambition is for Avro to replace both Hadoop's RPC and to be
used for most Hadoop data files, e.g., by Pig, Hive, etc.
Initial committers would be Sharad Agarwal and me, both existing Hadoop
committers. We are the sole authors of this software to date.
The code is currently at:
http://people.apache.org/~cutting/avro.git/
To learn more:
git clone http://people.apache.org/~cutting/avro.git/ avro
cat avro/README.txt
Comments? Questions?
Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Bryan Duxbury <br...@rapleaf.com>.
> Field ids are not present in Avro data except in the schema. A
> record's fields are serialized in the order that the fields occur
> in the records schema, with no per-field annotations whatsoever.
> For example, a record that contains a string and an int is
> serialized simply as a string followed by an int, nothing before,
> nothing between and nothing after. So, yes, it is a different data
> format.
So you can't serialize nulls? It also seems like this would make
forward/backward compatibility a little more complex. Thrift solves
this problem by using tags to indicate what kind of field you're
working with.
-Bryan
Re: [PROPOSAL] new subproject: Avro
Posted by George Porter <Ge...@Sun.COM>.
On Apr 3, 2009, at 1:02 PM, Doug Cutting wrote:
> George Porter wrote:
>> While this representation would certainly be as compact as
>> possible, wouldn't it prevent evolving the data structure over
>> time? One of the nice features of Google Protocol Buffers and
>> Thrift is that you can evolve the set of fields over time, and
>> older/newer clients can talk to older/newer services. If the
>> proposed Avro is evolvable, then perhaps I'm misunderstanding your
>> statement about the lack of IDs in the serialized data.
>
> Avro supports schema evolution. In Avro, the schema used to write
> the data must be available when the data is read. (In files, it is
> typically stored in the file metadata.)
>
> If you have the schema that was used to write the data, and you're
> expecting a slightly different schema, then you simply keep those
> fields that are in both schemas and skip those not. This is
> equivalent to Thrift and Protocol Buffer's support for schema
> evolution, but does not require manually assigning numeric field ids.
>
> This feature can also be used to support projection. If you have
> records with many large fields, but only need a single field in a
> particular computation, then you can specify an expected schema with
> only that field, and the runtime will efficiently skip all of the
> other fields, returning a record with just the single, expected field.
Thanks for the clarification--I better understand the schema
relationship. The projection feature is a nice feature, especially
since it seems like it would be able to support "sparse files" where
you want to just peek at large structs without invoking a lot of disk-
io (for data serialized on-disk).
>
>
>> I also agree with Bryan, in that it would be unfortunate to have
>> two different Apache projects with overlapping goals.
>
> We already have both Thrift and Etch in the incubator, which have
> similar goals. Apache does not attempt to mandate that projects
> have disjoint goals. There are many ways to slice things, and
> Apache prefers to rely on survival of the fittest rather than
> forcing things together.
>
>> Regardless of features, both protocol buffers and thrift have the
>> advantage of being debugged in mission-critical production
>> environments.
>
> Yes, but, as I've argued in other messages in this thread, they do
> not support the dynamic features we need. Adding those features
> would add new code that would share little with existing code in
> those projects. So, while the projects are conceptually similar, the
> implementations are necessarily different, and, without significant
> code overlap, separate projects seem more natural.
>
> Doug
Makes sense. Thanks,
George
Re: [PROPOSAL] new subproject: Avro
Posted by Doug Cutting <cu...@apache.org>.
George Porter wrote:
> While this representation would certainly be as compact as possible,
> wouldn't it prevent evolving the data structure over time? One of the
> nice features of Google Protocol Buffers and Thrift is that you can
> evolve the set of fields over time, and older/newer clients can talk to
> older/newer services. If the proposed Avro is evolvable, then perhaps
> I'm misunderstanding your statement about the lack of IDs in the
> serialized data.
Avro supports schema evolution. In Avro, the schema used to write the
data must be available when the data is read. (In files, it is
typically stored in the file metadata.)
If you have the schema that was used to write the data, and you're
expecting a slightly different schema, then you simply keep those fields
that are in both schemas and skip those not. This is equivalent to
Thrift and Protocol Buffer's support for schema evolution, but does not
require manually assigning numeric field ids.
This feature can also be used to support projection. If you have
records with many large fields, but only need a single field in a
particular computation, then you can specify an expected schema with
only that field, and the runtime will efficiently skip all of the other
fields, returning a record with just the single, expected field.
> I also agree with Bryan, in that it would be unfortunate to have two
> different Apache projects with overlapping goals.
We already have both Thrift and Etch in the incubator, which have
similar goals. Apache does not attempt to mandate that projects have
disjoint goals. There are many ways to slice things, and Apache prefers
to rely on survival of the fittest rather than forcing things together.
> Regardless of
> features, both protocol buffers and thrift have the advantage of being
> debugged in mission-critical production environments.
Yes, but, as I've argued in other messages in this thread, they do not
support the dynamic features we need. Adding those features would add
new code that would share little with existing code in those projects.
So, while the projects are conceptually similar, the implementations are
necessarily different, and, without significant code overlap, separate
projects seem more natural.
Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Scott Carey <sc...@richrelevance.com>.
On 4/3/09 12:03 PM, "George Porter" <Ge...@Sun.COM> wrote:
>
>
> On Apr 3, 2009, at 11:37 AM, Doug Cutting wrote:
>>>
>>
>> Field ids are not present in Avro data except in the schema. A
>> record's fields are serialized in the order that the fields occur in
>> the records schema, with no per-field annotations whatsoever. For
>> example, a record that contains a string and an int is serialized
>> simply as a string followed by an int, nothing before, nothing
>> between and nothing after. So, yes, it is a different data format.
>
> While this representation would certainly be as compact as possible,
> wouldn't it prevent evolving the data structure over time? One of the
> nice features of Google Protocol Buffers and Thrift is that you can
> evolve the set of fields over time, and older/newer clients can talk
> to older/newer services. If the proposed Avro is evolvable, then
> perhaps I'm misunderstanding your statement about the lack of IDs in
> the serialized data.
>From a quick perusal of the serialization format -- it contains headers with
type/schema information, and other metadata blocks. The types can be
inferred from this, and if this is done right then older/newer clients will
be able to read things just fine. What can't be done is mixing two
different formats in the same stream if headers define the format of the
whole stream.
I have not looked much deeper than that, but it looks like schema evolution
is feasible.
>
> I also agree with Bryan, in that it would be unfortunate to have two
> different Apache projects with overlapping goals. Regardless of
> features, both protocol buffers and thrift have the advantage of being
> debugged in mission-critical production environments.
>
> -George
>
Re: [PROPOSAL] new subproject: Avro
Posted by George Porter <Ge...@Sun.COM>.
On Apr 3, 2009, at 11:37 AM, Doug Cutting wrote:
>>
>
> Field ids are not present in Avro data except in the schema. A
> record's fields are serialized in the order that the fields occur in
> the records schema, with no per-field annotations whatsoever. For
> example, a record that contains a string and an int is serialized
> simply as a string followed by an int, nothing before, nothing
> between and nothing after. So, yes, it is a different data format.
While this representation would certainly be as compact as possible,
wouldn't it prevent evolving the data structure over time? One of the
nice features of Google Protocol Buffers and Thrift is that you can
evolve the set of fields over time, and older/newer clients can talk
to older/newer services. If the proposed Avro is evolvable, then
perhaps I'm misunderstanding your statement about the lack of IDs in
the serialized data.
I also agree with Bryan, in that it would be unfortunate to have two
different Apache projects with overlapping goals. Regardless of
features, both protocol buffers and thrift have the advantage of being
debugged in mission-critical production environments.
-George
Re: [PROPOSAL] new subproject: Avro
Posted by Doug Cutting <cu...@apache.org>.
Bryan Duxbury wrote:
> It's not actually a different data format, is it? You're saying that the
> user wouldn't specify the field IDs, but you'd fundamentally still use
> field ids for compactness and the like.
Field ids are not present in Avro data except in the schema. A record's
fields are serialized in the order that the fields occur in the records
schema, with no per-field annotations whatsoever. For example, a record
that contains a string and an int is serialized simply as a string
followed by an int, nothing before, nothing between and nothing after.
So, yes, it is a different data format.
> The bottom line is that I would love to see greater cooperation between
> Hadoop and Thrift. Unless it's impossible or impractical for Thrift to
> be useful here, I think we'd be willing to work towards Hadoop's needs.
Perhaps Thrift could be augmented to support Avro's JSON schemas and
serialization. Then it could interoperate with other Avro-based
systems. But then Thrift would have yet another serialization format,
that every language would need to implement for it to be useful...
Avro will only ever have one serialization format. Thrift fundamentally
standardizes an API, not a data format. Avro fundamentally is a data
format specification, like XML. Thrift could implement this
specification. The Avro project includes reference implementations, but
the format is intended to be simple enough and the specification stable
enough that others might reasonably develop alternate, independent
implementations.
Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Bryan Duxbury <br...@rapleaf.com>.
> With the schema in hand, you don't need to tag data with field
> numbers or types, since that's all there in the schema. So, having
> the schema, you can use a simpler data format.
To a degree, we already have that in Thrift - we call it the
DenseProtocol.
> Would you write parsers for Thrift's IDL in every language? Or
> would you use JSON, as Avro does, to avoid that?
When it comes to having a code-usable IDL for the schema, I'm totally
pro-JSON.
> Once you're using a different IDL and a different data format,
> what's shared with Thrift? Fundamentally, those two things define
> a serialization system, no?
It's not actually a different data format, is it? You're saying that
the user wouldn't specify the field IDs, but you'd fundamentally
still use field ids for compactness and the like. You may not use
actual Thrift generated objects, but you could certainly use Binary
or Compact protocol from Thrift to do all the writing to the wire.
You might also be able to use (or contribute to) Thrift's RPC-level
stuff like server implementations. We have some respectable Java
servers written, and if those aren't enough for your uses, I'd
actually be really interested in seeing if we could generalize some
of the Hadoop stuff to be useful within Thrift.
The bottom line is that I would love to see greater cooperation
between Hadoop and Thrift. Unless it's impossible or impractical for
Thrift to be useful here, I think we'd be willing to work towards
Hadoop's needs.
-Bryan
Re: [PROPOSAL] new subproject: Avro
Posted by Doug Cutting <cu...@apache.org>.
Bryan Duxbury wrote:
> It sounds like what you want is the option avoid pre-generated classes.
That's part of it. But, once you have the schema, you might as well
take advantage of it.
With the schema in hand, you don't need to tag data with field numbers
or types, since that's all there in the schema. So, having the schema,
you can use a simpler data format.
Also, with the schema, resolving version differences is simplified.
Developers don't need to assign field numbers, but can just use names.
For performance, one can internally use field numbers while reading, to
avoid string comparisons, but developers need no longer specify these,
but can use names, as in most software. Here having the schema means we
can simplify the IDL and its versioning semantics.
> If that's the only thing you need, it seems like we could bolt that on
> to Thrift with almost no work.
Would you write parsers for Thrift's IDL in every language? Or would
you use JSON, as Avro does, to avoid that?
Once you're using a different IDL and a different data format, what's
shared with Thrift? Fundamentally, those two things define a
serialization system, no?
> I assume you'd have the schema stored in
> metadata or file header or something, right? (You wouldn't want to store
> the field names in the binary encoding as strings, since that would
> probably very quickly dwarf the size of the actual data in a lot of cases.)
Yes, in data files the schema is typically stored in the metadata.
> If my assumptions are correct, it seems like it'd be a lot smarter to
> leverage existing Thrift infrastructure and encoding work rather than
> duplicating it for this lone feature.
What specific shared infrastructure would be leveraged? For Hadoop's
RPC, I hope to adapt Hadoop's client and server implementations as a
transport, as these have been highly tuned for Hadoop's performance
requirements.
Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Doug Cutting <cu...@apache.org>.
I have responded to this on general at hadoop.apache.org. Let's not
cross-post, but rather keep this discussion on a single list. I think
general@ is the best list for this discussion.
http://mail-archives.apache.org/mod_mbox/hadoop-general/200904.mbox/browser
Thanks,
Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Bryan Duxbury <br...@rapleaf.com>.
It sounds like what you want is the option avoid pre-generated
classes. If that's the only thing you need, it seems like we could
bolt that on to Thrift with almost no work. I assume you'd have the
schema stored in metadata or file header or something, right? (You
wouldn't want to store the field names in the binary encoding as
strings, since that would probably very quickly dwarf the size of the
actual data in a lot of cases.)
If my assumptions are correct, it seems like it'd be a lot smarter to
leverage existing Thrift infrastructure and encoding work rather than
duplicating it for this lone feature.
-Bryan
On Apr 3, 2009, at 9:06 AM, Doug Cutting wrote:
> Owen O'Malley wrote:
>> 2. Protocol buffers (and thrift) encode the field names as id
>> numbers. That means that if you read them into dynamic language
>> like Python that it has to use the field numbers instead of the
>> field names. In Avro, the field names are saved and there are no
>> field ids.
>
> This hints at a related problem with Thrift and Protocol Buffers,
> which is that they require one to generate code for each datatype
> one processes. This is awkward in dynamic environments, where one
> would like to write a script (Pig, Python, Perl, Hive, whatever) to
> process input data and generate output data, without having to
> locate the IDL for each input file, run an IDL compiler, load the
> generated code, generate an IDL file for the output, run the
> compiler again, load the output code and finally write your
> output. Avro rather lets you simply open your inputs, examine
> their datatypes, specify output types and write them.
>
> Avro's Java implementation currently includes three different data
> representations:
>
> - a "generic" representation uses a standard set of datastructures
> for all datatypes: records are represented as Map<String,Object>,
> arrays as List<Object>, longs as Long, etc.
>
> - a "reflect" representation uses Java reflection to permit one to
> read and write existing Java classes with Avro.
>
> - a "specific" representation generates Java classes that are
> compiled and loaded, much like Thrift and Protocol Buffers.
>
> We don't expect most scripting languages to use more than a single
> representation. Implementing Avro is quite simple, by design. We
> have a Python implementation, and hope to add more soon.
>
> Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Bryan Duxbury <br...@rapleaf.com>.
It sounds like what you want is the option avoid pre-generated
classes. If that's the only thing you need, it seems like we could
bolt that on to Thrift with almost no work. I assume you'd have the
schema stored in metadata or file header or something, right? (You
wouldn't want to store the field names in the binary encoding as
strings, since that would probably very quickly dwarf the size of the
actual data in a lot of cases.)
If my assumptions are correct, it seems like it'd be a lot smarter to
leverage existing Thrift infrastructure and encoding work rather than
duplicating it for this lone feature.
-Bryan
On Apr 3, 2009, at 9:06 AM, Doug Cutting wrote:
> Owen O'Malley wrote:
>> 2. Protocol buffers (and thrift) encode the field names as id
>> numbers. That means that if you read them into dynamic language
>> like Python that it has to use the field numbers instead of the
>> field names. In Avro, the field names are saved and there are no
>> field ids.
>
> This hints at a related problem with Thrift and Protocol Buffers,
> which is that they require one to generate code for each datatype
> one processes. This is awkward in dynamic environments, where one
> would like to write a script (Pig, Python, Perl, Hive, whatever) to
> process input data and generate output data, without having to
> locate the IDL for each input file, run an IDL compiler, load the
> generated code, generate an IDL file for the output, run the
> compiler again, load the output code and finally write your
> output. Avro rather lets you simply open your inputs, examine
> their datatypes, specify output types and write them.
>
> Avro's Java implementation currently includes three different data
> representations:
>
> - a "generic" representation uses a standard set of datastructures
> for all datatypes: records are represented as Map<String,Object>,
> arrays as List<Object>, longs as Long, etc.
>
> - a "reflect" representation uses Java reflection to permit one to
> read and write existing Java classes with Avro.
>
> - a "specific" representation generates Java classes that are
> compiled and loaded, much like Thrift and Protocol Buffers.
>
> We don't expect most scripting languages to use more than a single
> representation. Implementing Avro is quite simple, by design. We
> have a Python implementation, and hope to add more soon.
>
> Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Bryan Duxbury <br...@rapleaf.com>.
It sounds like what you want is the option avoid pre-generated
classes. If that's the only thing you need, it seems like we could
bolt that on to Thrift with almost no work. I assume you'd have the
schema stored in metadata or file header or something, right? (You
wouldn't want to store the field names in the binary encoding as
strings, since that would probably very quickly dwarf the size of the
actual data in a lot of cases.)
If my assumptions are correct, it seems like it'd be a lot smarter to
leverage existing Thrift infrastructure and encoding work rather than
duplicating it for this lone feature.
-Bryan
On Apr 3, 2009, at 9:06 AM, Doug Cutting wrote:
> Owen O'Malley wrote:
>> 2. Protocol buffers (and thrift) encode the field names as id
>> numbers. That means that if you read them into dynamic language
>> like Python that it has to use the field numbers instead of the
>> field names. In Avro, the field names are saved and there are no
>> field ids.
>
> This hints at a related problem with Thrift and Protocol Buffers,
> which is that they require one to generate code for each datatype
> one processes. This is awkward in dynamic environments, where one
> would like to write a script (Pig, Python, Perl, Hive, whatever) to
> process input data and generate output data, without having to
> locate the IDL for each input file, run an IDL compiler, load the
> generated code, generate an IDL file for the output, run the
> compiler again, load the output code and finally write your
> output. Avro rather lets you simply open your inputs, examine
> their datatypes, specify output types and write them.
>
> Avro's Java implementation currently includes three different data
> representations:
>
> - a "generic" representation uses a standard set of datastructures
> for all datatypes: records are represented as Map<String,Object>,
> arrays as List<Object>, longs as Long, etc.
>
> - a "reflect" representation uses Java reflection to permit one to
> read and write existing Java classes with Avro.
>
> - a "specific" representation generates Java classes that are
> compiled and loaded, much like Thrift and Protocol Buffers.
>
> We don't expect most scripting languages to use more than a single
> representation. Implementing Avro is quite simple, by design. We
> have a Python implementation, and hope to add more soon.
>
> Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Doug Cutting <cu...@apache.org>.
Owen O'Malley wrote:
> 2. Protocol buffers (and thrift) encode the field names as id numbers.
> That means that if you read them into dynamic language like Python that
> it has to use the field numbers instead of the field names. In Avro, the
> field names are saved and there are no field ids.
This hints at a related problem with Thrift and Protocol Buffers, which
is that they require one to generate code for each datatype one
processes. This is awkward in dynamic environments, where one would
like to write a script (Pig, Python, Perl, Hive, whatever) to process
input data and generate output data, without having to locate the IDL
for each input file, run an IDL compiler, load the generated code,
generate an IDL file for the output, run the compiler again, load the
output code and finally write your output. Avro rather lets you simply
open your inputs, examine their datatypes, specify output types and
write them.
Avro's Java implementation currently includes three different data
representations:
- a "generic" representation uses a standard set of datastructures for
all datatypes: records are represented as Map<String,Object>, arrays as
List<Object>, longs as Long, etc.
- a "reflect" representation uses Java reflection to permit one to
read and write existing Java classes with Avro.
- a "specific" representation generates Java classes that are compiled
and loaded, much like Thrift and Protocol Buffers.
We don't expect most scripting languages to use more than a single
representation. Implementing Avro is quite simple, by design. We have
a Python implementation, and hope to add more soon.
Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Owen O'Malley <om...@apache.org>.
On Apr 2, 2009, at 5:11 PM, Abhishek Verma wrote:
> I am a newbie here. Why not use something existing like protocol
> buffers :
> http://code.google.com/p/protobuf/ which is open source and works
> amazingly
> well.
There are two blockers for protocol buffers that make them suboptimal
for Hadoop. They are:
1. Protocol buffers are open source, but the community isn't open.
Google doesn't seem interested in getting patches from outside of
itself. If we needed something changed in protocol buffers, we'd end
up needing to fork the project to make any progress.
2. Protocol buffers (and thrift) encode the field names as id numbers.
That means that if you read them into dynamic language like Python
that it has to use the field numbers instead of the field names. In
Avro, the field names are saved and there are no field ids.
A final point is that since the schema isn't inlined in Avro, the
binary representation is much tighter than protocol buffers.
-- Owen
Re: [PROPOSAL] new subproject: Avro
Posted by Abhishek Verma <ve...@gmail.com>.
I am a newbie here. Why not use something existing like protocol buffers :
http://code.google.com/p/protobuf/ which is open source and works amazingly
well.
On Thu, Apr 2, 2009 at 5:20 PM, Owen O'Malley <om...@apache.org> wrote:
>
> On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
>
> I propose we add a new Hadoop subproject for Avro, a serialization system.
>>
>
> +1. Even if Justin comes after me for allowing another Jakarta. *smile*
>
> -- Owen
>
Re: [PROPOSAL] new subproject: Avro
Posted by Owen O'Malley <om...@apache.org>.
On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization
> system.
+1. Even if Justin comes after me for allowing another Jakarta. *smile*
-- Owen
Re: [PROPOSAL] new subproject: Avro
Posted by Tom White <to...@gmail.com>.
On Thu, Apr 2, 2009 at 11:05 PM, Doug Cutting <cu...@apache.org> wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization system.
+1
Tom
> My ambition is for Avro to replace both Hadoop's RPC and to be used for
> most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing Hadoop
> committers. We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments? Questions?
>
> Doug
>
Re: [PROPOSAL] new subproject: Avro
Posted by Dhruba Borthakur <dh...@gmail.com>.
+1. Awesome!
-dhruba
On Sun, Apr 5, 2009 at 10:33 PM, Sanjay Radia <sr...@yahoo-inc.com> wrote:
>
> On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
>
> I propose we add a new Hadoop subproject for Avro, a serialization
>> system. My ambition is for Avro to replace both Hadoop's RPC and to be
>> used for most Hadoop data files, e.g., by Pig, Hive, etc.
>>
>> Initial committers would be Sharad Agarwal and me, both existing Hadoop
>> committers. We are the sole authors of this software to date.
>>
>> The code is currently at:
>>
>> http://people.apache.org/~cutting/avro.git/<http://people.apache.org/%7Ecutting/avro.git/>
>>
>> To learn more:
>>
>> git clone http://people.apache.org/~cutting/avro.git/<http://people.apache.org/%7Ecutting/avro.git/>avro
>> cat avro/README.txt
>>
>> Comments? Questions?
>>
>> Doug
>>
>> +1
>
> sanjay
Re: [PROPOSAL] new subproject: Avro
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization
> system. My ambition is for Avro to replace both Hadoop's RPC and to
> be
> used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing
> Hadoop
> committers. We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments? Questions?
>
> Doug
>
+1
sanjay
Re: [PROPOSAL] new subproject: Avro
Posted by Doug Cutting <cu...@apache.org>.
Doug Cutting wrote:
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
I've posted the generated documentation. The specification is at:
http://people.apache.org/~cutting/avro/spec.html
and the javadoc is at:
http://people.apache.org/~cutting/avro/api/index.html
Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Doug Cutting <cu...@apache.org>.
Jim Kellerman (POWERSET) wrote:
> It is also my understanding (based on the email thread) that Avro only
> supports Java and python. That is a step backwards from Thrift.
We intend to add support for more languages. Avro is not complete.
> It appears that Avro uses introspection heavily, which is expensive in
> applications that require a high message rate.
It only uses introspection if you wish to use your existing Java classes
to represent Avro data. There are three representations in Java:
generic (uses Map<String,Object> for records, List<Object> for arrays),
specific (generates a java class for each Avro record, like Thrift) and
reflect (uses reflection to access existing classes). So introspection
is optional. And, while introspection is indeed slow for processing
file-based data, it would probably not a bottleneck for most RPC
protocols and might be a useful tool to migrate existing code to Avro.
> So I guess my question is why Avro?
The compelling case is dynamic data types. Pig, Hive, Python, Perl etc.
scripts should not have to generate a Thrift IDL file each time they
wish to write a data file with a new schema, nor should they need to run
the Thrift compiler for each data file they wish to read. For
production applications, code-generation is not an imposition and may
offer increased opportunities for optimization and error checking, but
for exploration and experimentation, a very common use case for Hadoop,
one would like to be able to browse datasets and build mapreduce
programs more interactively.
Doug
RE: [PROPOSAL] new subproject: Avro
Posted by "Jim Kellerman (POWERSET)" <Ji...@microsoft.com>.
> -----Original Message-----
> On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
>
> I propose we add a new Hadoop subproject for Avro, a serialization
> system. My ambition is for Avro to replace both Hadoop's RPC and to
> be used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing
> Hadoop committers. We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments? Questions?
>
> Doug
After reading all the messages about Avro, I'm still not sure I understand
why we should invent "yet another wheel". There are a number of people in
the community who have significant investments in Thrift, and I have yet
to see a compelling argument for Avro over Thrift.
My understanding is that Thrift already supports multi-language bindings,
something the HBase community has been asking for, for some time.
It is also my understanding (based on the email thread) that Avro only
supports Java and python. That is a step backwards from Thrift.
It appears that Avro uses introspection heavily, which is expensive in
applications that require a high message rate.
So I guess my question is why Avro? I may be thick, but it seems to me
as if it is just another wheel of a different color. If I could see a
point by point comparison between Avro and Thrift I could be convinced
that Avro is the way to go. So far, I have not seen any compelling reason
to re-invent the wheel.
+-0
---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
Re: [PROPOSAL] new subproject: Avro
Posted by Nigel Daley <nd...@yahoo-inc.com>.
+1.
On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization
> system. My ambition is for Avro to replace both Hadoop's RPC and to
> be used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing
> Hadoop committers. We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments? Questions?
>
> Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Alan Gates <ga...@yahoo-inc.com>.
+1. Pig would be happy to use a cross language serialization package
that did not require pre-compilation to read and write.
Alan.
On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization
> system. My ambition is for Avro to replace both Hadoop's RPC and to
> be used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing
> Hadoop committers. We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments? Questions?
>
> Doug
Re: [RESULT] new subproject: Avro
Posted by Doug Cutting <cu...@apache.org>.
The mailing lists are now created.
If you are interested you may subscribe by sending messages to:
avro-dev-subscribe at hadoop.apache.org
avro-user-subscribe at hadoop.apache.org
avro-commits-subscribe at hadoop.apache.org
Please join one or more of these lists if you are interested in Avro. I
do not intend to send more messages to general@ about Avro in the near
future.
I've also uploaded the code to subversion, at:
http://svn.apache.org/repos/asf/hadoop/avro/trunk/
A Jira bug database is at:
https://issues.apache.org/jira/browse/AVRO
I will create a website and Wiki ASAP.
Doug
Doug Cutting wrote:
> Doug Cutting wrote:
>> Three days have now passed since I initially proposed this, the
>> nominal time for an Apache vote. Is there anyone who strongly opposes
>> taking the development of Avro public as a Hadoop subproject?
>
> With 6 +1 votes from the PMC (Alan, Dhruba, Nigel, Owen, Tom & me) and
> no -1 votes, this has passed. I will now ask infrastructure to create
> the mailing lists, and create the Jira instance, svn repo, website &
> wiki myself.
>
> I'll send a message to this list once this is complete and the project
> is open for business.
>
> Thanks,
>
> Doug
[RESULT] new subproject: Avro
Posted by Doug Cutting <cu...@apache.org>.
Doug Cutting wrote:
> Three days have now passed since I initially proposed this, the nominal
> time for an Apache vote. Is there anyone who strongly opposes taking
> the development of Avro public as a Hadoop subproject?
With 6 +1 votes from the PMC (Alan, Dhruba, Nigel, Owen, Tom & me) and
no -1 votes, this has passed. I will now ask infrastructure to create
the mailing lists, and create the Jira instance, svn repo, website &
wiki myself.
I'll send a message to this list once this is complete and the project
is open for business.
Thanks,
Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Chad Walters <ch...@yahoo.com>.
Doug,
After our off-list chat, and given that you have indicated that the design is still in flux and that you are open to discussing changes that would permit interoperability, I am not as concerned as I was.
My urgency came from concern that once the design was put in place as part of an Apache subproject, rather than open sourced in some other less prominent forum, it would increase the barrier to interoperability; in particular, I was concerned that people would assume the design of the data format was fully-baked and start persisting large amounts of data in some early version of the format, potentially prematurely ossifying the design in a state unsuited for compatibility with Thrift. Given your clarifications around this, my fears clearly were not well-founded.
Please accept my apology if I came across as obstructionist. I was honestly advocating on behalf of what I believe is in the best interest our shared user base.
Clearly we have some disagreements about the value of some of Thrift's design choices and what those mean for various use cases. I think we also have some differences of opinion about the relative difficulty of implementation versus the value of interoperability. Hopefully, the next few months will afford an opportunity to examine the sources of those disagreements and see if they can be resolved.
Sincerely,
Chad
----- Original Message ----
From: Doug Cutting <cu...@apache.org>
To: general@hadoop.apache.org
Sent: Tuesday, April 7, 2009 8:33:32 PM
Subject: Re: [PROPOSAL] new subproject: Avro
To be clear, since a few folks have missed this point: Avro is not complete. At some point in the future, before people start using it as a format for persistent data, we'll need to stop altering its specification, or at least do so much more cautiously. But before then, my immediate goal to move development from private to open so that we have a chance to incorporate feedback before we lock down the specification.
For example, several folks have raised the issue of compatibility with Thrift. We certainly want to avoid gratuitous incompatibilities. There are also features clearly missing from Avro that we expect to add before we make a release, like default values, a more efficient RPC handshake, etc. And some features that we might consider removing, if they're not broadly useful and inhibit interoperability, like single-float, which isn't in Thrift, Python, etc. And I expect there will be more such issues raised in the coming weeks and months.
But before we can discuss and resolve such issues we need a forum in which to do so. That's all I am after at this point: mailing lists, a bug database, a public source code repository, etc., so that we can start accepting patches, adding committers, etc.
Three days have now passed since I initially proposed this, the nominal time for an Apache vote. Is there anyone who strongly opposes taking the development of Avro public as a Hadoop subproject? Only PMC votes are binding, but I would vastly prefer that the broader community also supports this step in the process.
Thanks,
Doug
Doug Cutting wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization system. My ambition is for Avro to replace both Hadoop's RPC and to be used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing Hadoop committers. We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments? Questions?
>
> Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Doug Cutting <cu...@apache.org>.
To be clear, since a few folks have missed this point: Avro is not
complete. At some point in the future, before people start using it as
a format for persistent data, we'll need to stop altering its
specification, or at least do so much more cautiously. But before then,
my immediate goal to move development from private to open so that we
have a chance to incorporate feedback before we lock down the specification.
For example, several folks have raised the issue of compatibility with
Thrift. We certainly want to avoid gratuitous incompatibilities. There
are also features clearly missing from Avro that we expect to add before
we make a release, like default values, a more efficient RPC handshake,
etc. And some features that we might consider removing, if they're not
broadly useful and inhibit interoperability, like single-float, which
isn't in Thrift, Python, etc. And I expect there will be more such
issues raised in the coming weeks and months.
But before we can discuss and resolve such issues we need a forum in
which to do so. That's all I am after at this point: mailing lists, a
bug database, a public source code repository, etc., so that we can
start accepting patches, adding committers, etc.
Three days have now passed since I initially proposed this, the nominal
time for an Apache vote. Is there anyone who strongly opposes taking
the development of Avro public as a Hadoop subproject? Only PMC votes
are binding, but I would vastly prefer that the broader community also
supports this step in the process.
Thanks,
Doug
Doug Cutting wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization
> system. My ambition is for Avro to replace both Hadoop's RPC and to be
> used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing Hadoop
> committers. We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments? Questions?
>
> Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Sameer Paranjpye <sp...@yahoo.com>.
+1
While protocol buffers and thrift have similar goals. Avro takes a different approach to schema evolution and reconciliation. I feel that Avros tighter layout of data and schema management is better suited for many of Hadoop and Pigs use cases for large data sets/tables on HDFS. Field ids start to matter when milions of objects have hundreds of fields each. There is, of course, the storage overhead. Schema management becomes hard especially if there are cases where field ids need to be assigned manually.
----- Original Message ----
From: Doug Cutting <cu...@apache.org>
To: general@hadoop.apache.org
Sent: Thursday, April 2, 2009 3:05:08 PM
Subject: [PROPOSAL] new subproject: Avro
I propose we add a new Hadoop subproject for Avro, a serialization system. My ambition is for Avro to replace both Hadoop's RPC and to be used for most Hadoop data files, e.g., by Pig, Hive, etc.
Initial committers would be Sharad Agarwal and me, both existing Hadoop committers. We are the sole authors of this software to date.
The code is currently at:
http://people.apache.org/~cutting/avro.git/
To learn more:
git clone http://people.apache.org/~cutting/avro.git/ avro
cat avro/README.txt
Comments? Questions?
Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Jerome Boulon <jb...@yahoo-inc.com>.
Chukwa needs a cross language serialization/RPC framework.
During demux, being able to create dynamically the schema based on the recordType will be a big plus instead of using an hardcoded one,
The projection feature will also be useful.
+1
/Jerome.
On 4/2/09 3:05 PM, "Doug Cutting" <cu...@apache.org> wrote:
I propose we add a new Hadoop subproject for Avro, a serialization
system. My ambition is for Avro to replace both Hadoop's RPC and to be
used for most Hadoop data files, e.g., by Pig, Hive, etc.
Initial committers would be Sharad Agarwal and me, both existing Hadoop
committers. We are the sole authors of this software to date.
The code is currently at:
http://people.apache.org/~cutting/avro.git/
To learn more:
git clone http://people.apache.org/~cutting/avro.git/ avro
cat avro/README.txt
Comments? Questions?
Doug
Re: [PROPOSAL] new subproject: Avro
Posted by Brian Forney <bf...@integral7.com>.
On 4/2/09 5:05 PM, "Doug Cutting" <cu...@apache.org> wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization
> system. My ambition is for Avro to replace both Hadoop's RPC and to be
> used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing Hadoop
> committers. We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments? Questions?
>
> Doug
>
+1