[jira] [Created] (KAFKA-4161) Allow connectors to request flush via the context

["Shikhar Bhushan (JIRA)" <ji...@apache.org>]

Shikhar Bhushan created KAFKA-4161:
--------------------------------------

             Summary: Allow connectors to request flush via the context
                 Key: KAFKA-4161
                 URL: https://issues.apache.org/jira/browse/KAFKA-4161
             Project: Kafka
          Issue Type: New Feature
          Components: KafkaConnect
            Reporter: Shikhar Bhushan
            Assignee: Ewen Cheslack-Postava


It is desirable to have, in addition to the time-based flush interval, volume or size-based commits. E.g. a sink connector which is buffering in terms of number of records may want to request a flush when the buffer is full, or when sufficient amount of data has been buffered in a file.

Having a method like say {{requestFlush()}} on the {{SinkTaskContext}} would allow for connectors to have flexible policies around flushes. This would be in addition to the time interval based flushes that are controlled with {{offset.flush.interval.ms}}, for which the clock should be reset when any kind of flush happens.

We should probably also support requesting flushes via the {{SourceTaskContext}} for consistency though a use-case doesn't come to mind off the bat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)