[jira] [Commented] (STORM-2357) add At-Most-Once guarantee in KafkaSpout

["Jungtaek Lim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864815#comment-15864815 ] 

Jungtaek Lim commented on STORM-2357:
-------------------------------------

As I commented from https://github.com/apache/storm/pull/1919, below code comes from blog post on Confluent:

{code}
try {
  while (running) {
  ConsumerRecords<String, String> records = consumer.poll(1000);

  try {
    consumer.commitSync();
    for (ConsumerRecord<String, String> record : records)
      System.out.println(record.offset() + ": " + record.value());
    } catch (CommitFailedException e) {
      // application specific failure handling
    }
  }
} finally {
  consumer.close();
}
{code}

which we can exactly do the same: poll the data, call commitSync(), and emit tuples only when commitSync() succeeds. It will guarantee at-most-once.

> add At-Most-Once guarantee in KafkaSpout
> ----------------------------------------
>
>                 Key: STORM-2357
>                 URL: https://issues.apache.org/jira/browse/STORM-2357
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-kafka-client
>            Reporter: Xu Mingmin
>
> KafkaSpout cannot guarantee exactly at-most-once semantic, as it commits offset periodically, and KafkaSpout.close() is not guarantee to call when a crash happens. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)