S3 as streaming source

[Soumya Simanta <so...@gmail.com>]

Is there a standard Flink S3 source yet?

Thanks
-Soumya

Re: S3 as streaming source

[Ufuk Celebi <uc...@apache.org>]

You can check this docs page out for S3/AWS support:
https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/aws.html

On Fri, Jun 3, 2016 at 8:55 AM, Tzu-Li (Gordon) Tai <tz...@gmail.com> wrote:
> Hi,
>
> I've gave it a quick test and Chiwan is right. The methods `readFile`,
> `readFileStream`, `readTextFile` on StreamExecutionEnvironment works with
> the S3 scheme to stream from S3 objects.
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/S3-as-streaming-source-tp7357p7361.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: S3 as streaming source

["Tzu-Li (Gordon) Tai" <tz...@gmail.com>]

Hi,

I've gave it a quick test and Chiwan is right. The methods `readFile`,
`readFileStream`, `readTextFile` on StreamExecutionEnvironment works with
the S3 scheme to stream from S3 objects.



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/S3-as-streaming-source-tp7357p7361.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: S3 as streaming source

[Chiwan Park <ch...@apache.org>]

Hi all,

I think we can use `readFile`, `readFileStream` methods in `StreamExecutionEnvironment` to create streaming source from S3 because data are stored as file in S3. But I haven’t test it.

Regards,
Chiwan Park

> On Jun 3, 2016, at 2:37 PM, Tzu-Li (Gordon) Tai <tz...@gmail.com> wrote:
> 
> Hi Soumya,
> 
> No, currently there is no Flink standard supported S3 streaming source. As
> far as I know, there isn't one out in the public yet either. The community
> is open to submissions for new connectors, so if you happen to be working on
> one for S3, you can file up a JIRA to let us know.
> 
> Also, are you looking for a S3 streaming source that fetches S3 event
> notifications (ref:
> http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html), or
> streaming files / objects from S3 for a data stream program? I assume the
> first one, since otherwise writing Flink batch jobs will suit you more (the
> batch DataSet API already supports this).
> 
> 
> 
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/S3-as-streaming-source-tp7357p7358.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: S3 as streaming source

["Tzu-Li (Gordon) Tai" <tz...@gmail.com>]

Hi Soumya,

No, currently there is no Flink standard supported S3 streaming source. As
far as I know, there isn't one out in the public yet either. The community
is open to submissions for new connectors, so if you happen to be working on
one for S3, you can file up a JIRA to let us know.

Also, are you looking for a S3 streaming source that fetches S3 event
notifications (ref:
http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html), or
streaming files / objects from S3 for a data stream program? I assume the
first one, since otherwise writing Flink batch jobs will suit you more (the
batch DataSet API already supports this).



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/S3-as-streaming-source-tp7357p7358.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.