You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Anchit Jatana <de...@gmail.com> on 2016/09/28 20:54:13 UTC
Create stream from multiple files using "env.readFile(format, input1,
FileProcessingMode.PROCESS_CONTINUOUSLY, 1000, FilePathFilter.createDefaultFilter())"
?
Hi All,
I have a use case where in need to create multiple source streams from
multiple files and monitor the files for any changes using the "
FileProcessingMode.PROCESS_CONTINUOUSLY"
Intention is to achieve something like this(have a monitored stream for
each of the multiple files), something like this:
DataStream<String> stream1 = env.readFile(format, input1,
FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
FilePathFilter.createDefaultFilter());
DataStream<String> stream2 = env.readFile(format, input2,
FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
FilePathFilter.createDefaultFilter());
DataStream<String> stream3= env.readFile(format, input3, FileProcessingMode.
PROCESS_CONTINUOUSLY, 1000, FilePathFilter.createDefaultFilter());
.
.
.
.
DataStream<String> streamN = env.readFile(format, inputN,
FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
FilePathFilter.createDefaultFilter());
Since, this implementation doesn't work, can someone suggest a way how this
thing can be achieved?
PS: Main intention is to '*monitor'* all the files and stream the updated
content if any change has been made to it.
Thank you!
Regards,
Anchit
Re: Create stream from multiple files using "env.readFile(format,
input1, FileProcessingMode.PROCESS_CONTINUOUSLY, 1000, FilePathFilter.createDefaultFilter())"
?
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
how does "doesn't work" manifest?
Cheers,
Aljoscha
On Wed, 28 Sep 2016 at 22:54 Anchit Jatana <de...@gmail.com>
wrote:
> Hi All,
>
> I have a use case where in need to create multiple source streams from
> multiple files and monitor the files for any changes using the "
> FileProcessingMode.PROCESS_CONTINUOUSLY"
>
> Intention is to achieve something like this(have a monitored stream for
> each of the multiple files), something like this:
>
> DataStream<String> stream1 = env.readFile(format, input1,
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
> FilePathFilter.createDefaultFilter());
>
> DataStream<String> stream2 = env.readFile(format, input2,
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
> FilePathFilter.createDefaultFilter());
>
> DataStream<String> stream3= env.readFile(format, input3,
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
> FilePathFilter.createDefaultFilter());
>
> .
>
> .
>
> .
>
> .
>
> DataStream<String> streamN = env.readFile(format, inputN,
> FileProcessingMode.PROCESS_CONTINUOUSLY, 1000,
> FilePathFilter.createDefaultFilter());
>
> Since, this implementation doesn't work, can someone suggest a way how
> this thing can be achieved?
>
>
> PS: Main intention is to '*monitor'* all the files and stream the updated
> content if any change has been made to it.
>
>
> Thank you!
>
> Regards,
>
> Anchit
>