You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishal Santoshi <vi...@gmail.com> on 2022/04/03 22:07:16 UTC
Source with S3 bucket with millions ( billions ) of object ( small files )
Folks,
I am doing a simple batch job that uses readFile() with
"s3a://[bucket_name]" as the path with setNestedFileEnumeration(true). I am
a little curious about a few things.
In batch mode which I think is turned on by FileProcessingMode.PROCESS_ONCE
mode does the source list all the S3 objects in the bucket to create input
splits *before* it calls downstream operators ?
Thanks.
Re: Source with S3 bucket with millions ( billions ) of object ( small files )
Posted by Vishal Santoshi <vi...@gmail.com>.
Thanks for the clarification.
My experiments have been in line with what you have suggested.
Regards.
On Mon, Apr 4, 2022 at 5:30 AM Arvid Heise <ar...@apache.org> wrote:
> Hi Vishal,
>
> with readFile, files are first collected and then sorted [1]. The same is
> true for the new FileSource. Here, you could plugin your own Enumerator to
> output files in chunks but then you need to continuously pull more and
> can't use batch mode.
>
> We are happy to receive any patch for that behavior (for the new source).
>
> [1]
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/ContinuousFileMonitoringFunction.java#L259-L261
>
> On Mon, Apr 4, 2022 at 12:07 AM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
>> Folks,
>> I am doing a simple batch job that uses readFile() with
>> "s3a://[bucket_name]" as the path with setNestedFileEnumeration(true). I am
>> a little curious about a few things.
>>
>> In batch mode which I think is turned on by
>> FileProcessingMode.PROCESS_ONCE mode does the source list all the S3
>> objects in the bucket to create input splits *before* it calls
>> downstream operators ?
>>
>>
>>
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
Re: Source with S3 bucket with millions ( billions ) of object ( small files )
Posted by Arvid Heise <ar...@apache.org>.
Hi Vishal,
with readFile, files are first collected and then sorted [1]. The same is
true for the new FileSource. Here, you could plugin your own Enumerator to
output files in chunks but then you need to continuously pull more and
can't use batch mode.
We are happy to receive any patch for that behavior (for the new source).
[1]
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/ContinuousFileMonitoringFunction.java#L259-L261
On Mon, Apr 4, 2022 at 12:07 AM Vishal Santoshi <vi...@gmail.com>
wrote:
> Folks,
> I am doing a simple batch job that uses readFile() with
> "s3a://[bucket_name]" as the path with setNestedFileEnumeration(true). I am
> a little curious about a few things.
>
> In batch mode which I think is turned on by
> FileProcessingMode.PROCESS_ONCE mode does the source list all the S3
> objects in the bucket to create input splits *before* it calls downstream
> operators ?
>
>
>
>
> Thanks.
>
>
>
>
>
>
Re: Source with S3 bucket with millions ( billions ) of object ( small files )
Posted by Roman Grebennikov <gr...@dfdx.me>.
Hi,
in a unified stream/batch FileSource there is a processStaticFileSet() method to enumerate all the splits only once,
and make Source complete when it's finished.
As for my own experience using the processStaticFileSet with large s3 buckets, the enumeration seems to happen on the jobmanager, and listing bucket with a 1B items will probably block it for a long time - this happens on 1.14, not sure how it's going on an upcoming 1.15.
with best regards,
Roman Grebennikov | grv@dfdx.me
On Sun, Apr 3, 2022, at 22:07, Vishal Santoshi wrote:
> Folks,
> I am doing a simple batch job that uses readFile() with "s3a://[bucket_name]" as the path with setNestedFileEnumeration(true). I am a little curious about a few things.
>
> In batch mode which I think is turned on by FileProcessingMode.PROCESS_ONCE mode does the source list all the S3 objects in the bucket to create input splits *before* it calls downstream operators ?
>
>
>
>
> Thanks.
>
>
>
>
>