You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by ebru <b2...@cs.hacettepe.edu.tr> on 2017/11/22 14:21:44 UTC
How to write dataset as parquet format
Hello all,
We are trying to write dataset as parquet format, we use AvroParquetOutputFormat but it is not compatible with Flink’s FileOutputFormat.
Is there a way to write dataset as parquet?
-Ebru
Re: How to write dataset as parquet format
Posted by ebru <b2...@cs.hacettepe.edu.tr>.
Flavio and Fabian thanks for your quick answers, it was very helpful.
-Ebru
> On 22 Nov 2017, at 20:47, Flavio Pompermaier <po...@okkam.it> wrote:
>
> I usually refer to this:
>
> https://github.com/FelixNeutatz/parquet-flinktacular <https://github.com/FelixNeutatz/parquet-flinktacular>
>
> On 22 Nov 2017 18:29, "Fabian Hueske" <fhueske@gmail.com <ma...@gmail.com>> wrote:
> Hi Ebru,
>
> AvroParquetOutputFormat seems to implement Hadoop's OutputFormat interface.
> Flink provides a wrapper for Hadoop's OutputFormat [1], so you can try to wrap AvroParquetOutputFormat in Flink's HadoopOutputFormat.
>
> Hope this helps,
> Fabian
>
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/hadoop_compatibility.html#using-hadoop-outputformats <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/hadoop_compatibility.html#using-hadoop-outputformats>
>
> 2017-11-22 15:21 GMT+01:00 ebru <b20926247@cs.hacettepe.edu.tr <ma...@cs.hacettepe.edu.tr>>:
> Hello all,
>
> We are trying to write dataset as parquet format, we use AvroParquetOutputFormat but it is not compatible with Flink’s FileOutputFormat.
>
> Is there a way to write dataset as parquet?
>
> -Ebru
>
Re: How to write dataset as parquet format
Posted by Flavio Pompermaier <po...@okkam.it>.
I usually refer to this:
https://github.com/FelixNeutatz/parquet-flinktacular
On 22 Nov 2017 18:29, "Fabian Hueske" <fh...@gmail.com> wrote:
> Hi Ebru,
>
> AvroParquetOutputFormat seems to implement Hadoop's OutputFormat interface.
> Flink provides a wrapper for Hadoop's OutputFormat [1], so you can try to
> wrap AvroParquetOutputFormat in Flink's HadoopOutputFormat.
>
> Hope this helps,
> Fabian
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/dev/batch/hadoop_compatibility.html#using-hadoop-outputformats
>
> 2017-11-22 15:21 GMT+01:00 ebru <b2...@cs.hacettepe.edu.tr>:
>
>> Hello all,
>>
>> We are trying to write dataset as parquet format, we use
>> AvroParquetOutputFormat but it is not compatible with Flink’s
>> FileOutputFormat.
>>
>> Is there a way to write dataset as parquet?
>>
>> -Ebru
>
>
>
Re: How to write dataset as parquet format
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Ebru,
AvroParquetOutputFormat seems to implement Hadoop's OutputFormat interface.
Flink provides a wrapper for Hadoop's OutputFormat [1], so you can try to
wrap AvroParquetOutputFormat in Flink's HadoopOutputFormat.
Hope this helps,
Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/hadoop_compatibility.html#using-hadoop-outputformats
2017-11-22 15:21 GMT+01:00 ebru <b2...@cs.hacettepe.edu.tr>:
> Hello all,
>
> We are trying to write dataset as parquet format, we use
> AvroParquetOutputFormat but it is not compatible with Flink’s
> FileOutputFormat.
>
> Is there a way to write dataset as parquet?
>
> -Ebru