You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@storm.apache.org by XuMingmin <gi...@git.apache.org> on 2017/01/05 21:25:10 UTC

[GitHub] storm pull request #1863: fix: KafkaSpout is blocked in AutoCommitMode

GitHub user XuMingmin opened a pull request:

    https://github.com/apache/storm/pull/1863

    fix: KafkaSpout is blocked in AutoCommitMode

    What's the issue?
    When Storm topology is run on 'At-Most-Once' mode, with ack executor=0, KafkaSpout cannot emit records after some time.
    
    What's the cause?
    Function poll() controls whether it need to pull more data from Kafka cluster. With the condition `!waitingToEmit() && numUncommittedOffsets < maxUncommittedOffsets`, it's always `FALSE` after reaching threshold, as `numUncommittedOffsets` is creasing and never reset on 'At-Most-Once' mode.  
    
    What's the solution?
    I change the condition to `!waitingToEmit() && (numUncommittedOffsets < maxUncommittedOffsets || consumerAutoCommitMode)`, that return `TRUE` with AutoCommitMode regarding of UnCommittedOffset, see the comments inline.
    Also, it's not required to track emitted(msgId) and numUncommittedOffsets with AutoCommitMode .

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/XuMingmin/storm master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/storm/pull/1863.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1863
    
----
commit f3c9985e26868fa52cee34335f70550c4f7744c4
Author: XuMingmin <mm...@gmail.com>
Date:   2017-01-05T19:31:42Z

    fix: KafkaSpout is blocked in AutoCommitMode
    
    What's the issue?
    When Storm topology is run on 'At-Most-Once' mode, with ack executor=0, KafkaSpout cannot emit records after some time.
    
    What's the cause?
    Function poll() controls whether it need to pull more data from Kafka cluster. With the condition `!waitingToEmit() && numUncommittedOffsets < maxUncommittedOffsets`, it's always `FALSE` after reaching threshold, as `numUncommittedOffsets` is creasing and never reset on 'At-Most-Once' mode.  
    
    What's the solution?
    I change the condition to `!waitingToEmit() && (numUncommittedOffsets < maxUncommittedOffsets || consumerAutoCommitMode)`, that return `TRUE` with AutoCommitMode regarding of UnCommittedOffset, see the comments inline.
    Also, it's not required to track emitted(msgId) and numUncommittedOffsets with AutoCommitMode .

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1863: fix: KafkaSpout is blocked in AutoCommitMode

Posted by XuMingmin <gi...@git.apache.org>.

Github user XuMingmin commented on the issue:

    https://github.com/apache/storm/pull/1863
  
    @srdo I create https://issues.apache.org/jira/browse/STORM-2340, 
    
    @HeartSaVioR , let me check with the latest code, and open a new pull request


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1863: fix: KafkaSpout is blocked in AutoCommitMode

Posted by HeartSaVioR <gi...@git.apache.org>.

Github user HeartSaVioR commented on the issue:

    https://github.com/apache/storm/pull/1863
  
    @XuMingmin Could you rebase this? STORM-2225 touches many places on storm-kafka-client so you might need to check it's still valid.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1863: fix: KafkaSpout is blocked in AutoCommitMode

Posted by XuMingmin <gi...@git.apache.org>.

Github user XuMingmin commented on the issue:

    https://github.com/apache/storm/pull/1863
  
    close this one, use https://github.com/apache/storm/pull/1919 instead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1863: fix: KafkaSpout is blocked in AutoCommitMode

Posted by XuMingmin <gi...@git.apache.org>.

Github user XuMingmin closed the pull request at:

    https://github.com/apache/storm/pull/1863


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1863: fix: KafkaSpout is blocked in AutoCommitMode

Posted by srdo <gi...@git.apache.org>.

Github user srdo commented on the issue:

    https://github.com/apache/storm/pull/1863
  
    Nice find :) You should go to https://issues.apache.org/jira/ and create an issue for this. Then rename this PR to mention the jira issue number (see the other open PRs for an example). The commit message should probably also mention the issue number.
    
    The fix itself looks fine to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1863: fix: KafkaSpout is blocked in AutoCommitMode

Posted by XuMingmin <gi...@git.apache.org>.

Github user XuMingmin commented on the issue:

    https://github.com/apache/storm/pull/1863
  
    @HeartSaVioR The issue if still there, so I create a new pull request https://github.com/apache/storm/pull/1919 after rebase, 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---