You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Elkhan Dadashov <el...@gmail.com> on 2019/07/22 19:52:29 UTC

Will StreamingFileSink.forBulkFormat(...) support overriding OnCheckpointRollingPolicy?

Hi folks,

Will StreamingFileSink.forBulkFormat(...) support overriding
OnCheckpointRollingPolicy?

Does anyone use StreamingFileSink *with checkpoint disabled *for writing
Parquet output files?

The output parquet files are generated, but they are empty, and stay in
*inprogress* state, even when the job completes:

.part-0-0.inprogress.3e31ba42-588c-48cc-ad6d-d0ebcf1d8632
.part-1-0.inprogress.78e1f1dc-3c1c-417b-8270-2bf0298f985a
.part-2-0.inprogress.087cf3f1-7e2d-4a03-a518-62f576ed7eea

Exactly-once semantics is not important for my case, would then using
*BucketingSink* the only option ?

Thanks.