You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Niels Basjes <Ni...@basjes.nl> on 2017/01/20 10:20:25 UTC

Streaming file source?

Hi,

For testing and optimizing a streaming application I want to have a "100%
accurate repeatable" substitute for a Kafka source.
I was thinking of creating a streaming source class that simply reads the
records from a (static unchanging) set of files.
Each file would then produce the data which (in the live situation) come
from a single Kafka partition.

I hate reinventing the wheel so I'm wondering is something like this
already been built by someone?
If so, where can I find it?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Streaming file source?

Posted by Niels Basjes <Ni...@basjes.nl>.

Thanks!

This sounds really close to what I had in mind.
I'll use this first and see how far I get.

Niels

On Fri, Jan 20, 2017 at 11:27 AM, Stephan Ewen <se...@apache.org> wrote:

> Hi Niels!
>
> There is the Continuous File Monitoring Source, used via
>
> StreamExecutionEnvironment.readFile(FileInputFormat<OUT>
> inputFormat, String filePath, FileProcessingMode watchType, long interval);
>
> This can be used to both continuously ingest from files, or to read files
> once.
>
> Kostas can probably comment more about whether and how you can make the
> file order deterministic.
>
> Stephan
>
>
> On Fri, Jan 20, 2017 at 11:20 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>
>> Hi,
>>
>> For testing and optimizing a streaming application I want to have a "100%
>> accurate repeatable" substitute for a Kafka source.
>> I was thinking of creating a streaming source class that simply reads the
>> records from a (static unchanging) set of files.
>> Each file would then produce the data which (in the live situation) come
>> from a single Kafka partition.
>>
>> I hate reinventing the wheel so I'm wondering is something like this
>> already been built by someone?
>> If so, where can I find it?
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Re: Streaming file source?

Posted by Stephan Ewen <se...@apache.org>.

Hi Niels!

There is the Continuous File Monitoring Source, used via

StreamExecutionEnvironment.readFile(FileInputFormat<OUT>
inputFormat, String filePath, FileProcessingMode watchType, long interval);

This can be used to both continuously ingest from files, or to read files
once.

Kostas can probably comment more about whether and how you can make the
file order deterministic.

Stephan


On Fri, Jan 20, 2017 at 11:20 AM, Niels Basjes <Ni...@basjes.nl> wrote:

> Hi,
>
> For testing and optimizing a streaming application I want to have a "100%
> accurate repeatable" substitute for a Kafka source.
> I was thinking of creating a streaming source class that simply reads the
> records from a (static unchanging) set of files.
> Each file would then produce the data which (in the live situation) come
> from a single Kafka partition.
>
> I hate reinventing the wheel so I'm wondering is something like this
> already been built by someone?
> If so, where can I find it?
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>