You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by anurag <an...@gmail.com> on 2019/06/04 02:57:06 UTC

Read file from S3 and write to kafka

Hi All,
I am searched a lot on google but could not find how I can achieve writing
a flink  function which reads a file in S3 and for each line in the file
write a message to kafka.
Thanks a lot , much appreciated. I am sorry if I did not searched properly.
Thanks,
Anurag

Re: Read file from S3 and write to kafka

Posted by miki haiat <mi...@gmail.com>.

You can use the DataSet API to parse files from S3.

https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/batch/#data-sources
https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html#s3-simple-storage-service


And then  parsed it and send it to kafka.



On Tue, Jun 4, 2019 at 5:57 AM anurag <an...@gmail.com> wrote:

> Hi All,
> I am searched a lot on google but could not find how I can achieve writing
> a flink  function which reads a file in S3 and for each line in the file
> write a message to kafka.
> Thanks a lot , much appreciated. I am sorry if I did not searched properly.
> Thanks,
> Anurag
>