You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Kamil ty <ka...@gmail.com> on 2021/11/22 09:32:23 UTC

Table API Filesystem connector - disable interval rolling policy

Hey all,

I wanted to know if there is a way to disable the interval rolling policy
in the Table API filesystem connector.
From flink docs: FileSystem | Apache Flink
<https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/filesystem/#rolling-policy>
The key to change the interval: sink.rolling-policy.rollover-interval
Is it possible to fully disable this rolling policy or the only solution is
to set a very big duration?

Best Regards
Kamil

Re: Table API Filesystem connector - disable interval rolling policy

Posted by Kamil ty <ka...@gmail.com>.

Hi,

The end result that I'm expecting is to only have a few files that are
filled up to a specific file size and not multiple small files.

So if I understand you correctly, when I enable the auto compaction, all
the small files that are being created will be merged into files that are
up to the specified size?

eg. (with the default 128MB max filesize)
Without autocompaction:
output/
    file1.avro - 28MB
    file2.avro - 100MB
    file3.avro - 30MB

With autocompaction:
output/
    file1.avro - 128MB
    file2.avro - 30MB

Am I understanding this correctly? If so, I think autocompaction is the
exact solution to this issue.

Best Regards
Kamil

On Mon, 22 Nov 2021 at 16:43, Francesco Guardiani <fr...@ververica.com>
wrote:

> Hi,
> Looking at the code, there is no ability to disable the rollover-interval.
>
> But I'm wondering, what are you trying to do? Write a file up to the
> configured file-size? Note that if you're using auto compaction, on every
> checkpoint you'll have a rollover, regardless of the rollover-interval.
>
> I cc'ed Fabian in the discussion which has a better knowledge than me on
> file sink relates topics.
>
> FG
>
> On Mon, Nov 22, 2021 at 3:51 PM Matthias Pohl <ma...@ververica.com>
> wrote:
>
>> Hi Kamil,
>> by looking at the code I'd say that the only option you have is to
>> increase the parameter you already mentioned to a very high number. But I'm
>> not sure about the side effects. I'm gonna add Francesco to this thread.
>> Maybe he has better ideas on how to answer your question.
>>
>> Best,
>> Matthias
>>
>> On Mon, Nov 22, 2021 at 10:32 AM Kamil ty <ka...@gmail.com> wrote:
>>
>>> Hey all,
>>>
>>> I wanted to know if there is a way to disable the interval rolling
>>> policy in the Table API filesystem connector.
>>> From flink docs: FileSystem | Apache Flink
>>> <https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/filesystem/#rolling-policy>
>>> The key to change the interval: sink.rolling-policy.rollover-interval
>>> Is it possible to fully disable this rolling policy or the only solution
>>> is to set a very big duration?
>>>
>>> Best Regards
>>> Kamil
>>>
>>

Re: Table API Filesystem connector - disable interval rolling policy

Posted by Francesco Guardiani <fr...@ververica.com>.

Hi,
Looking at the code, there is no ability to disable the rollover-interval.

But I'm wondering, what are you trying to do? Write a file up to the
configured file-size? Note that if you're using auto compaction, on every
checkpoint you'll have a rollover, regardless of the rollover-interval.

I cc'ed Fabian in the discussion which has a better knowledge than me on
file sink relates topics.

FG

On Mon, Nov 22, 2021 at 3:51 PM Matthias Pohl <ma...@ververica.com>
wrote:

> Hi Kamil,
> by looking at the code I'd say that the only option you have is to
> increase the parameter you already mentioned to a very high number. But I'm
> not sure about the side effects. I'm gonna add Francesco to this thread.
> Maybe he has better ideas on how to answer your question.
>
> Best,
> Matthias
>
> On Mon, Nov 22, 2021 at 10:32 AM Kamil ty <ka...@gmail.com> wrote:
>
>> Hey all,
>>
>> I wanted to know if there is a way to disable the interval rolling policy
>> in the Table API filesystem connector.
>> From flink docs: FileSystem | Apache Flink
>> <https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/filesystem/#rolling-policy>
>> The key to change the interval: sink.rolling-policy.rollover-interval
>> Is it possible to fully disable this rolling policy or the only solution
>> is to set a very big duration?
>>
>> Best Regards
>> Kamil
>>
>

Re: Table API Filesystem connector - disable interval rolling policy

Posted by Matthias Pohl <ma...@ververica.com>.

Hi Kamil,
by looking at the code I'd say that the only option you have is to increase
the parameter you already mentioned to a very high number. But I'm not sure
about the side effects. I'm gonna add Francesco to this thread. Maybe he
has better ideas on how to answer your question.

Best,
Matthias

On Mon, Nov 22, 2021 at 10:32 AM Kamil ty <ka...@gmail.com> wrote:

> Hey all,
>
> I wanted to know if there is a way to disable the interval rolling policy
> in the Table API filesystem connector.
> From flink docs: FileSystem | Apache Flink
> <https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/filesystem/#rolling-policy>
> The key to change the interval: sink.rolling-policy.rollover-interval
> Is it possible to fully disable this rolling policy or the only solution
> is to set a very big duration?
>
> Best Regards
> Kamil
>