You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Selina Tech <sw...@gmail.com> on 2015/11/19 00:52:19 UTC
Avro vs protocol buffer for internal type to Parquet
Dear All:
I need to generate some data by Samza to Kafka and then write to
Parquet formate file.
I was asked why I choose Avro type as my Samza output to Kafka
instead of Protocol Buffer. Since currently our data on Kafka are all
Protocol buffer type message.
I explained that Avro encoded message has advantages such as, the
encoded size smaller, no extra code compile, implementation easier. fast
to serialize/deserialize and supporting a lot language.
However some people believe when encoded the Avro message take as
much space as Protocol buffer, but with schema, the size could be much
bigger.
I am wondering if there are any other advantages make you choose
Avro as your message type How you consider the data size for Avro vs
Protocol buffer?
Sincerely,
Selina
Reference:
1. https://issues.apache.org/jira/browse/SAMZA-317
2.
http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
3. https://avro.apache.org/docs/1.7.7/gettingstartedjava.html
4.
https://www.igvita.com/2011/08/01/protocol-buffers-avro-thrift-messagepack/
5. http://tech.puredanger.com/2011/05/27/serialization-comparison/
Re: Avro vs protocol buffer for internal type to Parquet
Posted by Selina Tech <sw...@gmail.com>.
Hi, All:
I like to update some information about Avro message on Kafka. Avro
message include schema ID instead of Schema at each message
http://stackoverflow.com/questions/31204201/apache-kafka-with-avro-and-schema-repo-where-in-the-message-does-the-schema-id
sincerely,
Selina
On Wed, Nov 18, 2015 at 3:52 PM, Selina Tech <sw...@gmail.com> wrote:
> Dear All:
>
> I need to generate some data by Samza to Kafka and then write to
> Parquet formate file.
>
> I was asked why I choose Avro type as my Samza output to Kafka
> instead of Protocol Buffer. Since currently our data on Kafka are all
> Protocol buffer type message.
>
> I explained that Avro encoded message has advantages such as, the
> encoded size smaller, no extra code compile, implementation easier. fast
> to serialize/deserialize and supporting a lot language.
>
> However some people believe when encoded the Avro message take as
> much space as Protocol buffer, but with schema, the size could be much
> bigger.
>
> I am wondering if there are any other advantages make you choose
> Avro as your message type How you consider the data size for Avro vs
> Protocol buffer?
>
> Sincerely,
> Selina
>
>
> Reference:
> 1. https://issues.apache.org/jira/browse/SAMZA-317
> 2.
> http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
>
> 3. https://avro.apache.org/docs/1.7.7/gettingstartedjava.html
> 4.
> https://www.igvita.com/2011/08/01/protocol-buffers-avro-thrift-messagepack/
> 5. http://tech.puredanger.com/2011/05/27/serialization-comparison/
>
Re: Avro vs protocol buffer for internal type to Parquet
Posted by Cheng Lian <li...@gmail.com>.
In general parquet-avro is much more standard than parquet-protobuf if
you care about nested types (LISTs and MAPs) and interoperability with
other systems. It implemented most of the compatibility rules defined in
parquet-format spec.
Cheng
On 11/19/15 7:52 AM, Selina Tech wrote:
> Dear All:
>
> I need to generate some data by Samza to Kafka and then write to
> Parquet formate file.
>
> I was asked why I choose Avro type as my Samza output to Kafka
> instead of Protocol Buffer. Since currently our data on Kafka are all
> Protocol buffer type message.
>
> I explained that Avro encoded message has advantages such as, the
> encoded size smaller, no extra code compile, implementation easier. fast
> to serialize/deserialize and supporting a lot language.
>
> However some people believe when encoded the Avro message take as
> much space as Protocol buffer, but with schema, the size could be much
> bigger.
>
> I am wondering if there are any other advantages make you choose
> Avro as your message type How you consider the data size for Avro vs
> Protocol buffer?
>
> Sincerely,
> Selina
>
>
> Reference:
> 1. https://issues.apache.org/jira/browse/SAMZA-317
> 2.
> http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
>
> 3. https://avro.apache.org/docs/1.7.7/gettingstartedjava.html
> 4.
> https://www.igvita.com/2011/08/01/protocol-buffers-avro-thrift-messagepack/
> 5. http://tech.puredanger.com/2011/05/27/serialization-comparison/
>