You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sathi Chowdhury <Sa...@elliemae.com> on 2017/03/31 04:12:00 UTC

s3 read while writing inside sink function

Hi fellow Flink enthusiasts,

I am trying figure out a recommended way to read s3 data  while  I am trying to write to s3 using BucketingSink.



BucketingSink<String> s3Sink = new BucketingSink<String>("s3://" + entityBucket + "/")
        .setBucketer(new EntityCustomBucketer())
        .setWriter(new StringWriter<String>())
……………….


The issue is that I want to write to S3 based on a condition check on the filename already existing there.
I wanted to bake this check within the sink itself, any clue on what is needed for this?
Thanks
Sathi


=============Notice to Recipient: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain information that is confidential or legally privileged, and intended for the use of the individual or entity named above. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately notify the sender by telephone or return e-mail and delete the original transmission and its attachments without reading or saving in any manner. Thank you. =============