You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Selina Tech <sw...@gmail.com> on 2015/11/19 00:52:19 UTC

Avro vs protocol buffer for internal type to Parquet

Dear All:

       I need to generate some data by Samza to Kafka and then write to
Parquet formate file.

      I was asked why I choose Avro type as my Samza output to Kafka
instead of Protocol Buffer. Since currently our data on Kafka are all
Protocol buffer type message.

      I explained that Avro encoded message has advantages such as, the
encoded size smaller, no extra code compile, implementation easier.  fast
to serialize/deserialize and supporting a lot language.

      However some people believe when encoded the Avro message take as
much space as Protocol buffer, but with schema, the size could be much
bigger.

        I am wondering if there are any other advantages make you choose
Avro as your message type  How you consider the data size for Avro vs
Protocol buffer?

 Sincerely,
Selina


Reference:
1. https://issues.apache.org/jira/browse/SAMZA-317
2.
http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

3. https://avro.apache.org/docs/1.7.7/gettingstartedjava.html
4.
https://www.igvita.com/2011/08/01/protocol-buffers-avro-thrift-messagepack/
5. http://tech.puredanger.com/2011/05/27/serialization-comparison/

Re: Avro vs protocol buffer for internal type to Parquet

Posted by Selina Tech <sw...@gmail.com>.
Hi, All:

    I like to update some information about Avro message on Kafka. Avro
message include schema ID instead of Schema at each message

http://stackoverflow.com/questions/31204201/apache-kafka-with-avro-and-schema-repo-where-in-the-message-does-the-schema-id

sincerely,
Selina

On Wed, Nov 18, 2015 at 3:52 PM, Selina Tech <sw...@gmail.com> wrote:

> Dear All:
>
>        I need to generate some data by Samza to Kafka and then write to
> Parquet formate file.
>
>       I was asked why I choose Avro type as my Samza output to Kafka
> instead of Protocol Buffer. Since currently our data on Kafka are all
> Protocol buffer type message.
>
>       I explained that Avro encoded message has advantages such as, the
> encoded size smaller, no extra code compile, implementation easier.  fast
> to serialize/deserialize and supporting a lot language.
>
>       However some people believe when encoded the Avro message take as
> much space as Protocol buffer, but with schema, the size could be much
> bigger.
>
>         I am wondering if there are any other advantages make you choose
> Avro as your message type  How you consider the data size for Avro vs
> Protocol buffer?
>
>  Sincerely,
> Selina
>
>
> Reference:
> 1. https://issues.apache.org/jira/browse/SAMZA-317
> 2.
> http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
>
> 3. https://avro.apache.org/docs/1.7.7/gettingstartedjava.html
> 4.
> https://www.igvita.com/2011/08/01/protocol-buffers-avro-thrift-messagepack/
> 5. http://tech.puredanger.com/2011/05/27/serialization-comparison/
>

Re: Avro vs protocol buffer for internal type to Parquet

Posted by Cheng Lian <li...@gmail.com>.
In general parquet-avro is much more standard than parquet-protobuf if 
you care about nested types (LISTs and MAPs) and interoperability with 
other systems. It implemented most of the compatibility rules defined in 
parquet-format spec.

Cheng

On 11/19/15 7:52 AM, Selina Tech wrote:
> Dear All:
>
>         I need to generate some data by Samza to Kafka and then write to
> Parquet formate file.
>
>        I was asked why I choose Avro type as my Samza output to Kafka
> instead of Protocol Buffer. Since currently our data on Kafka are all
> Protocol buffer type message.
>
>        I explained that Avro encoded message has advantages such as, the
> encoded size smaller, no extra code compile, implementation easier.  fast
> to serialize/deserialize and supporting a lot language.
>
>        However some people believe when encoded the Avro message take as
> much space as Protocol buffer, but with schema, the size could be much
> bigger.
>
>          I am wondering if there are any other advantages make you choose
> Avro as your message type  How you consider the data size for Avro vs
> Protocol buffer?
>
>   Sincerely,
> Selina
>
>
> Reference:
> 1. https://issues.apache.org/jira/browse/SAMZA-317
> 2.
> http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
>
> 3. https://avro.apache.org/docs/1.7.7/gettingstartedjava.html
> 4.
> https://www.igvita.com/2011/08/01/protocol-buffers-avro-thrift-messagepack/
> 5. http://tech.puredanger.com/2011/05/27/serialization-comparison/
>