You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Lakshmi Rao (JIRA)" <ji...@apache.org> on 2018/03/30 23:48:00 UTC
[jira] [Created] (FLINK-9118) Support a time based rollover of part
file in Bucketing Sink
Lakshmi Rao created FLINK-9118:
----------------------------------
Summary: Support a time based rollover of part file in Bucketing Sink
Key: FLINK-9118
URL: https://issues.apache.org/jira/browse/FLINK-9118
Project: Flink
Issue Type: Improvement
Components: filesystem-connector
Reporter: Lakshmi Rao
In the current implementation, the BucketingSink rolls over a part file based on a _batchSize_ ([here|https://github.com/apache/flink/blob/release-1.4.0/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L459]). Can we also support a roll over based on a constant time interval? This is not the same as the _inactiveBucketCheckInterval_ as this bucket is not truly inactive, it's still being written to but just has to be flushed every X minutes, where X is a user-specified time interval.
The change would involve tracking a _bucketCreationTime_ in the BucketState (much like the _lastWrittenToTime_) whenever a new part file is opened and would include a condition to check _currentProcessingTime_ - _bucketCreationTime_ > X in the _shouldRoll_ method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)