You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Flavio Pompermaier <po...@okkam.it> on 2020/07/17 15:59:06 UTC

Parquet batch table sink in Flink 1.11

Hi to all,
is there a way to write out Parquet-Avro data using BatchTableEnvironment
with Flink 1.11?
At the moment I'm using the hadoop ParquetOutputFormat but I hope to be
able to get rid of it sooner or later..I saw that there's the
AvroOutputFormat but no support for it using Parquet.

Best,
Flavio

Re: Parquet batch table sink in Flink 1.11

Posted by Flavio Pompermaier <po...@okkam.it>.
I think that's not true when you need to integrate Flink into an existing
data-lake..I think it should be very straightforward (in my opinion) to
read/ write Parquet data with objects serialized with
avro/thrift/protobuf...or at least reuse hadoop input/output formats with
table API. At the moment I have to pass through a lot of custom code that
uses the Hadoop formats and is a lto of code just to read and write thrift
or avro serialized objects in parquet folders.

On Wed, Jul 22, 2020 at 3:35 AM Jingsong Li <ji...@gmail.com> wrote:

> In table/SQL,
>
> I think we don't need a source/sink for `AvroParquetOutputFormat`, because
> the data structure is always Row or RowData, should not be a avro object.
>
> Best,
> Jingsong
>
> On Tue, Jul 21, 2020 at 8:09 PM Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> This is what I actually do but I was hoping to be able to get rid of the
>> HadoopOutputForma and be able to use a  more comfortable Source/Sink
>> implementation.
>>
>> On Tue, Jul 21, 2020 at 12:38 PM Jingsong Li <ji...@gmail.com>
>> wrote:
>>
>>> Hi Flavio,
>>>
>>> AvroOutputFormat only supports writing Avro files.
>>> I think you can use `AvroParquetOutputFormat` as a hadoop output format,
>>> and wrap it through Flink `HadoopOutputFormat`.
>>>
>>> Best,
>>> Jingsong
>>>
>>> On Fri, Jul 17, 2020 at 11:59 PM Flavio Pompermaier <
>>> pompermaier@okkam.it> wrote:
>>>
>>>> Hi to all,
>>>> is there a way to write out Parquet-Avro data using
>>>> BatchTableEnvironment with Flink 1.11?
>>>> At the moment I'm using the hadoop ParquetOutputFormat but I hope to be
>>>> able to get rid of it sooner or later..I saw that there's the
>>>> AvroOutputFormat but no support for it using Parquet.
>>>>
>>>> Best,
>>>> Flavio
>>>>
>>>
>>>
>>> --
>>> Best, Jingsong Lee
>>>
>>
>
> --
> Best, Jingsong Lee
>

Re: Parquet batch table sink in Flink 1.11

Posted by Jingsong Li <ji...@gmail.com>.
In table/SQL,

I think we don't need a source/sink for `AvroParquetOutputFormat`, because
the data structure is always Row or RowData, should not be a avro object.

Best,
Jingsong

On Tue, Jul 21, 2020 at 8:09 PM Flavio Pompermaier <po...@okkam.it>
wrote:

> This is what I actually do but I was hoping to be able to get rid of the
> HadoopOutputForma and be able to use a  more comfortable Source/Sink
> implementation.
>
> On Tue, Jul 21, 2020 at 12:38 PM Jingsong Li <ji...@gmail.com>
> wrote:
>
>> Hi Flavio,
>>
>> AvroOutputFormat only supports writing Avro files.
>> I think you can use `AvroParquetOutputFormat` as a hadoop output format,
>> and wrap it through Flink `HadoopOutputFormat`.
>>
>> Best,
>> Jingsong
>>
>> On Fri, Jul 17, 2020 at 11:59 PM Flavio Pompermaier <po...@okkam.it>
>> wrote:
>>
>>> Hi to all,
>>> is there a way to write out Parquet-Avro data using
>>> BatchTableEnvironment with Flink 1.11?
>>> At the moment I'm using the hadoop ParquetOutputFormat but I hope to be
>>> able to get rid of it sooner or later..I saw that there's the
>>> AvroOutputFormat but no support for it using Parquet.
>>>
>>> Best,
>>> Flavio
>>>
>>
>>
>> --
>> Best, Jingsong Lee
>>
>

-- 
Best, Jingsong Lee

Re: Parquet batch table sink in Flink 1.11

Posted by Flavio Pompermaier <po...@okkam.it>.
This is what I actually do but I was hoping to be able to get rid of the
HadoopOutputForma and be able to use a  more comfortable Source/Sink
implementation.

On Tue, Jul 21, 2020 at 12:38 PM Jingsong Li <ji...@gmail.com> wrote:

> Hi Flavio,
>
> AvroOutputFormat only supports writing Avro files.
> I think you can use `AvroParquetOutputFormat` as a hadoop output format,
> and wrap it through Flink `HadoopOutputFormat`.
>
> Best,
> Jingsong
>
> On Fri, Jul 17, 2020 at 11:59 PM Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> Hi to all,
>> is there a way to write out Parquet-Avro data using BatchTableEnvironment
>> with Flink 1.11?
>> At the moment I'm using the hadoop ParquetOutputFormat but I hope to be
>> able to get rid of it sooner or later..I saw that there's the
>> AvroOutputFormat but no support for it using Parquet.
>>
>> Best,
>> Flavio
>>
>
>
> --
> Best, Jingsong Lee
>

Re: Parquet batch table sink in Flink 1.11

Posted by Jingsong Li <ji...@gmail.com>.
Hi Flavio,

AvroOutputFormat only supports writing Avro files.
I think you can use `AvroParquetOutputFormat` as a hadoop output format,
and wrap it through Flink `HadoopOutputFormat`.

Best,
Jingsong

On Fri, Jul 17, 2020 at 11:59 PM Flavio Pompermaier <po...@okkam.it>
wrote:

> Hi to all,
> is there a way to write out Parquet-Avro data using BatchTableEnvironment
> with Flink 1.11?
> At the moment I'm using the hadoop ParquetOutputFormat but I hope to be
> able to get rid of it sooner or later..I saw that there's the
> AvroOutputFormat but no support for it using Parquet.
>
> Best,
> Flavio
>


-- 
Best, Jingsong Lee