You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Ewen Cheslack-Postava (JIRA)" <ji...@apache.org> on 2017/01/02 20:45:58 UTC
[jira] [Updated] (KAFKA-4161) Decouple flush and offset commits
[ https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ewen Cheslack-Postava updated KAFKA-4161:
-----------------------------------------
Labels: kip needs-kip (was: needs-kip)
> Decouple flush and offset commits
> ---------------------------------
>
> Key: KAFKA-4161
> URL: https://issues.apache.org/jira/browse/KAFKA-4161
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Shikhar Bhushan
> Assignee: Shikhar Bhushan
> Labels: kip, needs-kip
> Fix For: 0.10.2.0
>
>
> It is desirable to have, in addition to the time-based flush interval, volume or size-based commits. E.g. a sink connector which is buffering in terms of number of records may want to request a flush when the buffer is full, or when sufficient amount of data has been buffered in a file.
> Having a method like say {{requestFlush()}} on the {{SinkTaskContext}} would allow for connectors to have flexible policies around flushes. This would be in addition to the time interval based flushes that are controlled with {{offset.flush.interval.ms}}, for which the clock should be reset when any kind of flush happens.
> We should probably also support requesting flushes via the {{SourceTaskContext}} for consistency though a use-case doesn't come to mind off the bat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)