You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Lakshmi Rao (JIRA)" <ji...@apache.org> on 2018/04/05 16:35:00 UTC
[jira] [Resolved] (FLINK-9118) Support a time based rollover of
part file in Bucketing Sink
[ https://issues.apache.org/jira/browse/FLINK-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lakshmi Rao resolved FLINK-9118.
--------------------------------
Resolution: Duplicate
> Support a time based rollover of part file in Bucketing Sink
> ------------------------------------------------------------
>
> Key: FLINK-9118
> URL: https://issues.apache.org/jira/browse/FLINK-9118
> Project: Flink
> Issue Type: Improvement
> Components: filesystem-connector
> Reporter: Lakshmi Rao
> Priority: Major
>
> In the current implementation, the BucketingSink rolls over a part file based on a _batchSize_ ([here|https://github.com/apache/flink/blob/release-1.4.0/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L459]). Can we also support a roll over based on a constant time interval? This is not the same as the _inactiveBucketCheckInterval_ as this bucket is not truly inactive, it's still being written to but just has to be flushed every X minutes, where X is a user-specified time interval.
> The change would involve tracking a _bucketCreationTime_ in the BucketState (much like the _lastWrittenToTime_) whenever a new part file is opened and would include a condition to check _currentProcessingTime_ - _bucketCreationTime_ > X in the _shouldRoll_ method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)