You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by "Raja.Aravapalli" <Ra...@target.com> on 2017/09/07 20:00:37 UTC

Bucketing HDFS Sink Failing randomly after fews days it runs successfully

Hi Team,

I have a Bucketing Sink writing to HDFS files…. Which is running successfully for 4days failing suddenly with below exception:

Caused by: java.io.IOException: Cannot find required BLOB at /tmp/blobStore


Code below:

BucketingSink<String> HdfsSink = new BucketingSink<String> (hdfsOutputPath);

HdfsSink.setBucketer(new BasePathBucketer<String>());
HdfsSink.setBatchSize(1024 * 1024 * hdfsOutputBatchSizeInMB); // this means 'hdfsOutputBatchSizeInMB' MB
HdfsSink.setPartPrefix("PART-FILE-" + Long.toString(System.currentTimeMillis()));


Can someone please share thoughts on how I  can fix this.

Thanks.


Regards,
Raja.