[jira] [Created] (KAFKA-519) Allow commiting the state of single KafkaStream

["Esko Suomi (JIRA)" <ji...@apache.org>]

Esko Suomi created KAFKA-519:
--------------------------------

             Summary: Allow commiting the state of single KafkaStream
                 Key: KAFKA-519
                 URL: https://issues.apache.org/jira/browse/KAFKA-519
             Project: Kafka
          Issue Type: Improvement
    Affects Versions: 0.7.1, 0.7
            Reporter: Esko Suomi


Currently consuming multiple topics through ZK by first acquiring ConsumerConnector and then fetching message streams for wanted topics. And when the messages have been consumed, the current consuming state is commited with the method ConsumerConnector#commitOffsets().

This scheme has a flaw when the consuming application is used as sort of a data piping proxy instead of final consuming sink. In our case we read data from Kafka, repackage it and only then move it to persistent storage. The repackaging step is relatively long running and may span several hours (usually a few minutes) which in addition is mixed with highly asymmetric topic throughputs; one of our topics gets about 80% of total throughput. We have about 20 topics in total. As an unwanted side effect of all this, commiting the offset whenever the per-topic persistence step has been taken means commiting offsets for other topics too which may eventually manifest as loss of data if the consuming application or the machine it is running on crashes.

So, while this loss of data can be alleviated to some extent with for example local temp storage, it would be cleaner if KafkaStream itself would allow for partition level offset commiting.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira