You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Piotr Nowojski <pn...@apache.org> on 2021/02/01 10:24:20 UTC

Re: Question about setNestedFileEnumeration()

Hi Billy,

Could you maybe share some minimal code reproducing the problem? For
example I would suggest to start with reading from local files with some
trivial application.

Best Piotrek

pt., 22 sty 2021 o 00:21 Billy Bain <bi...@gmail.com> napisał(a):

> I have a Streaming process where new directories are added daily in S3.
>
> s3://foo/bar/2021-01-18/data.gz
> s3://foo/bar/2021-01-19/data.gz
> s3://foo/bar/2021-01-20/data.gz
>
> If I start the process, it won't pick up anything other than the
> directories visible when the process was started.
>
> The textInput has this applied:
> textInputFormat.setNestedFileEnumeration(true);
>
> DataStreamSource<String> lines = env.readFile(textInputFormat, inputPath,
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1000);
> --
> Wayne D. Young
> aka Billy Bob Bain
> billybobbain@gmail.com
>