You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by sweetest <gi...@git.apache.org> on 2015/01/09 02:10:06 UTC
[GitHub] storm pull request: STORM-618 : Add spoutconfig option to make kaf...
GitHub user sweetest opened a pull request:
https://github.com/apache/storm/pull/376
STORM-618 : Add spoutconfig option to make kafka spout process messages at most once.
Closes [STORM-618](https://issues.apache.org/jira/browse/STORM-618)
While it's nice for kafka spout to push failed tuple back into a sorted set and try to process it again, this way of guaranteed message processing sometimes makes situation pretty bad when a failed tuple repeatedly fails in downstream bolts since PartitionManager#fill method tries to fetch from that offset repeatedly.
This is a corresponding code snippet.
private void fill() {
...
if (had_failed) {
offset = failed.first();
} else {
offset = _emittedToOffset;
}
...
msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer, _partition, offset);
...
So there should be an option for a developer to decide if he wants to process failed tuple again or just skip failed tuple. One of the best thing of Storm is that spout together with trident can be implemented to guarantee at-least-once,exactly-once and at-most-once message processing.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sweetest/storm master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/376.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #376
----
commit 07a0106878896266c9f97f3622bcf708425ee15a
Author: SEUNGJIN LEE <sw...@navercorp.com>
Date: 2015-01-09T00:58:47Z
STORM-618 : Add spoutconfig option to make kafka spout process messages at most once.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-618 : Add spoutconfig option to make kaf...
Posted by sweetest <gi...@git.apache.org>.
Github user sweetest closed the pull request at:
https://github.com/apache/storm/pull/376
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---