You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@crunch.apache.org by Kristoffer Sjögren <st...@gmail.com> on 2014/06/10 10:13:42 UTC

crunch+protobuf+parquet

Hi

Are there source and target implementations available for using
protobuf (instead of avro) with parquet in crunch?

Cheers,
-Kristoffer

Re: crunch+protobuf+parquet

Posted by Kristoffer Sjögren <st...@gmail.com>.

Ok, I will have a look at the utility first, to see if it fit my
needs, or otherwise log an enhancement.

Thanks for the info!

On Tue, Jun 10, 2014 at 5:06 PM, Micah Whitacre <mk...@gmail.com> wrote:
> Currently there is no support for writing out a Parquet file using protobuf
> only Avro.  Seems like something we could support so feel free to long an
> enhancement.[1]  Are you looking for some functionality specific to protobuf
> with Parquet?  I haven't done it but I believe there are utilities to store
> protobuf as Avro[2] so that is an option unless you have a specific
> requirement to need to store in the Parquet Protobuf format.
>
> [1] - https://issues.apache.org/jira/browse/CRUNCH
> [2] -
> http://avro.apache.org/docs/1.6.1/api/java/org/apache/avro/protobuf/ProtobufData.html
>
>
> On Tue, Jun 10, 2014 at 3:13 AM, Kristoffer Sjögren <st...@gmail.com>
> wrote:
>>
>> Hi
>>
>> Are there source and target implementations available for using
>> protobuf (instead of avro) with parquet in crunch?
>>
>> Cheers,
>> -Kristoffer
>
>

Re: crunch+protobuf+parquet

Posted by Micah Whitacre <mk...@gmail.com>.

Currently there is no support for writing out a Parquet file using protobuf
only Avro.  Seems like something we could support so feel free to long an
enhancement.[1]  Are you looking for some functionality specific to
protobuf with Parquet?  I haven't done it but I believe there are utilities
to store protobuf as Avro[2] so that is an option unless you have a
specific requirement to need to store in the Parquet Protobuf format.

[1] - https://issues.apache.org/jira/browse/CRUNCH
[2] -
http://avro.apache.org/docs/1.6.1/api/java/org/apache/avro/protobuf/ProtobufData.html

On Tue, Jun 10, 2014 at 3:13 AM, Kristoffer Sjögren <st...@gmail.com>
wrote:

> Hi
>
> Are there source and target implementations available for using
> protobuf (instead of avro) with parquet in crunch?
>
> Cheers,
> -Kristoffer
>