You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vijayendra Yadav <co...@gmail.com> on 2020/07/28 18:01:43 UTC
Compression Streamingfilesink ROW-encoded format
Hi Team,
Is there a way to enable compression in StreamingFileSink API for
Row-encoded formats ?.
val sink: StreamingFileSink[String] = StreamingFileSink
.forRowFormat(new Path(outputPath), new
SimpleStringEncoder[String]("UTF-8"))
Regards,
Vijay
Re: Compression Streamingfilesink ROW-encoded format
Posted by Vijayendra Yadav <co...@gmail.com>.
Thank You.
On Wed, Jul 29, 2020 at 2:07 AM Ravi Bhushan Ratnakar <
ravibhushanratnakar@gmail.com> wrote:
> Yes, flink-compress module is supported from 1.10.0 and onward.
>
> Regards,
> Ravi
>
> On Tue 28 Jul, 2020, 23:11 Vijayendra Yadav, <co...@gmail.com>
> wrote:
>
>> Thank You Ravi for Quick help. One Last Question is this compression
>> supported with Flink Version 1.10.0 ?
>>
>> Regards,
>> Vijay
>>
>> On Tue, Jul 28, 2020 at 1:20 PM Ravi Bhushan Ratnakar <
>> ravibhushanratnakar@gmail.com> wrote:
>>
>>> Hi Vijayendra,
>>>
>>> As far as rowFormat is concerned, it doesn't support compression.
>>>
>>>
>>> Regards,
>>> Ravi
>>>
>>> On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav, <co...@gmail.com>
>>> wrote:
>>>
>>>> Hi Ravi,
>>>>
>>>> Thanks for your response. But your example is for *forBulkForma**t*.
>>>> How about *forRowFormat* ?.
>>>>
>>>> Regards,
>>>> Vijay
>>>>
>>>> On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <
>>>> ravibhushanratnakar@gmail.com> wrote:
>>>>
>>>>> Hi Vijayendra,
>>>>>
>>>>> You could achieve row encoded with like this as well
>>>>>
>>>>> codecName = "org.apache.hadoop.io.compress.GzipCodec"
>>>>>
>>>>> val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()
>>>>>
>>>>> Regards,
>>>>> Ravi
>>>>>
>>>>> On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <
>>>>> contact.vsy@gmail.com> wrote:
>>>>>
>>>>>> Hi Team,
>>>>>>
>>>>>> Is there a way to enable compression in StreamingFileSink API for
>>>>>> Row-encoded formats ?.
>>>>>>
>>>>>> val sink: StreamingFileSink[String] = StreamingFileSink
>>>>>> .forRowFormat(new Path(outputPath), new
>>>>>> SimpleStringEncoder[String]("UTF-8"))
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Vijay
>>>>>>
>>>>>
Re: Compression Streamingfilesink ROW-encoded format
Posted by Ravi Bhushan Ratnakar <ra...@gmail.com>.
Yes, flink-compress module is supported from 1.10.0 and onward.
Regards,
Ravi
On Tue 28 Jul, 2020, 23:11 Vijayendra Yadav, <co...@gmail.com> wrote:
> Thank You Ravi for Quick help. One Last Question is this compression
> supported with Flink Version 1.10.0 ?
>
> Regards,
> Vijay
>
> On Tue, Jul 28, 2020 at 1:20 PM Ravi Bhushan Ratnakar <
> ravibhushanratnakar@gmail.com> wrote:
>
>> Hi Vijayendra,
>>
>> As far as rowFormat is concerned, it doesn't support compression.
>>
>>
>> Regards,
>> Ravi
>>
>> On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav, <co...@gmail.com>
>> wrote:
>>
>>> Hi Ravi,
>>>
>>> Thanks for your response. But your example is for *forBulkForma**t*.
>>> How about *forRowFormat* ?.
>>>
>>> Regards,
>>> Vijay
>>>
>>> On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <
>>> ravibhushanratnakar@gmail.com> wrote:
>>>
>>>> Hi Vijayendra,
>>>>
>>>> You could achieve row encoded with like this as well
>>>>
>>>> codecName = "org.apache.hadoop.io.compress.GzipCodec"
>>>>
>>>> val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()
>>>>
>>>> Regards,
>>>> Ravi
>>>>
>>>> On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <co...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> Is there a way to enable compression in StreamingFileSink API for
>>>>> Row-encoded formats ?.
>>>>>
>>>>> val sink: StreamingFileSink[String] = StreamingFileSink
>>>>> .forRowFormat(new Path(outputPath), new
>>>>> SimpleStringEncoder[String]("UTF-8"))
>>>>>
>>>>>
>>>>> Regards,
>>>>> Vijay
>>>>>
>>>>
Re: Compression Streamingfilesink ROW-encoded format
Posted by Vijayendra Yadav <co...@gmail.com>.
Thank You Ravi for Quick help. One Last Question is this compression
supported with Flink Version 1.10.0 ?
Regards,
Vijay
On Tue, Jul 28, 2020 at 1:20 PM Ravi Bhushan Ratnakar <
ravibhushanratnakar@gmail.com> wrote:
> Hi Vijayendra,
>
> As far as rowFormat is concerned, it doesn't support compression.
>
>
> Regards,
> Ravi
>
> On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav, <co...@gmail.com>
> wrote:
>
>> Hi Ravi,
>>
>> Thanks for your response. But your example is for *forBulkForma**t*. How
>> about *forRowFormat* ?.
>>
>> Regards,
>> Vijay
>>
>> On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <
>> ravibhushanratnakar@gmail.com> wrote:
>>
>>> Hi Vijayendra,
>>>
>>> You could achieve row encoded with like this as well
>>>
>>> codecName = "org.apache.hadoop.io.compress.GzipCodec"
>>>
>>> val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()
>>>
>>> Regards,
>>> Ravi
>>>
>>> On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <co...@gmail.com>
>>> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> Is there a way to enable compression in StreamingFileSink API for
>>>> Row-encoded formats ?.
>>>>
>>>> val sink: StreamingFileSink[String] = StreamingFileSink
>>>> .forRowFormat(new Path(outputPath), new
>>>> SimpleStringEncoder[String]("UTF-8"))
>>>>
>>>>
>>>> Regards,
>>>> Vijay
>>>>
>>>
Re: Compression Streamingfilesink ROW-encoded format
Posted by Ravi Bhushan Ratnakar <ra...@gmail.com>.
Hi Vijayendra,
As far as rowFormat is concerned, it doesn't support compression.
Regards,
Ravi
On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav, <co...@gmail.com> wrote:
> Hi Ravi,
>
> Thanks for your response. But your example is for *forBulkForma**t*. How
> about *forRowFormat* ?.
>
> Regards,
> Vijay
>
> On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <
> ravibhushanratnakar@gmail.com> wrote:
>
>> Hi Vijayendra,
>>
>> You could achieve row encoded with like this as well
>>
>> codecName = "org.apache.hadoop.io.compress.GzipCodec"
>>
>> val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()
>>
>> Regards,
>> Ravi
>>
>> On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <co...@gmail.com>
>> wrote:
>>
>>> Hi Team,
>>>
>>> Is there a way to enable compression in StreamingFileSink API for
>>> Row-encoded formats ?.
>>>
>>> val sink: StreamingFileSink[String] = StreamingFileSink
>>> .forRowFormat(new Path(outputPath), new
>>> SimpleStringEncoder[String]("UTF-8"))
>>>
>>>
>>> Regards,
>>> Vijay
>>>
>>
Re: Compression Streamingfilesink ROW-encoded format
Posted by Vijayendra Yadav <co...@gmail.com>.
Hi Ravi,
Thanks for your response. But your example is for *forBulkForma**t*. How
about *forRowFormat* ?.
Regards,
Vijay
On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <
ravibhushanratnakar@gmail.com> wrote:
> Hi Vijayendra,
>
> You could achieve row encoded with like this as well
>
> codecName = "org.apache.hadoop.io.compress.GzipCodec"
>
> val streamingFileSink:StreamingFileSink[String] = StreamingFileSink.forBulkFormat(new Path(outputPath),CompressWriters.forExtractor(new DefaultExtractor[String]).withHadoopCompression(codecName)).build()
>
> Regards,
> Ravi
>
> On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <co...@gmail.com>
> wrote:
>
>> Hi Team,
>>
>> Is there a way to enable compression in StreamingFileSink API for
>> Row-encoded formats ?.
>>
>> val sink: StreamingFileSink[String] = StreamingFileSink
>> .forRowFormat(new Path(outputPath), new
>> SimpleStringEncoder[String]("UTF-8"))
>>
>>
>> Regards,
>> Vijay
>>
>
Re: Compression Streamingfilesink ROW-encoded format
Posted by Ravi Bhushan Ratnakar <ra...@gmail.com>.
Hi Vijayendra,
You could achieve row encoded with like this as well
codecName = "org.apache.hadoop.io.compress.GzipCodec"
val streamingFileSink:StreamingFileSink[String] =
StreamingFileSink.forBulkFormat(new
Path(outputPath),CompressWriters.forExtractor(new
DefaultExtractor[String]).withHadoopCompression(codecName)).build()
Regards,
Ravi
On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <co...@gmail.com>
wrote:
> Hi Team,
>
> Is there a way to enable compression in StreamingFileSink API for
> Row-encoded formats ?.
>
> val sink: StreamingFileSink[String] = StreamingFileSink
> .forRowFormat(new Path(outputPath), new
> SimpleStringEncoder[String]("UTF-8"))
>
>
> Regards,
> Vijay
>