You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by moesol <gi...@git.apache.org> on 2016/04/01 01:19:10 UTC

[GitHub] storm pull request: Addresses network flood from KafkaSpout to kaf...

GitHub user moesol opened a pull request:

    https://github.com/apache/storm/pull/1287

    Addresses network flood from KafkaSpout to kafka server.

    * Allows minBytes in fetch request to be configured
      from KafkaConfig.fetchMinBytes.
    * Defaults new configuration KafkaConfig.fetchMinBytes to 1.
    * Defaults fetchMaxWait to 100ms instead of 10000ms.
    
    Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
    and our kafka servers even though no Kafka messages were moving.
    Using the wireshark kafka dissector, we were able to see that
    each FetchRequest had maxWait set to 10000
    and minBytes set to 0. When binBytes is set to 0 the kafka server
    responds immediately when there are no messages. In turn the KafkaSpout
    polls without any delay causing a constant stream of FetchRequest/
    FetchResponse messages. Using a non-KafkaSpout client had a similar
    traffic pattern with two key differences
    1) minBytes was 1
    2) maxWait was 100
    With these FetchRequest parameters and no messages flowing,
    the kafka server delays the FetchResponse by 100 ms. This reduces
    the network traffic from megabits to the low kilobits. It also
    reduced the CPU utilization of our kafka server from 140% to 2%.
    
    Hopefully the risk of this change is low because
    the old behavior can be restored using the API by setting
    KafkaConfig.fetchMaxWait to 10000
    KafkaConfig.fetchMinBytes to 0

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/MoebiusSolutions/storm 0.9.3-branch

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/storm/pull/1287.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1287
    
----
commit 66c38ad28ce597a4df6c05db3264ff94960d6764
Author: Robert Hastings <rh...@moesol.com>
Date:   2016-03-31T23:14:47Z

    Addresses network flood from KafkaSpout to kafka server.
    
    * Allows minBytes in fetch request to be configured
      from KafkaConfig.fetchMinBytes.
    * Defaults new configuration KafkaConfig.fetchMinBytes to 1.
    * Defaults fetchMaxWait to 100ms instead of 10000ms.
    
    Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
    and our kafka servers even though no Kafka messages were moving.
    Using the wireshark kafka dissector, we were able to see that
    each FetchRequest had maxWait set to 10000
    and minBytes set to 0. When binBytes is set to 0 the kafka server
    responds immediately when there are no messages. In turn the KafkaSpout
    polls without any delay causing a constant stream of FetchRequest/
    FetchResponse messages. Using a non-KafkaSpout client had a similar
    traffic pattern with two key differences
    1) minBytes was 1
    2) maxWait was 100
    With these FetchRequest parameters and no messages flowing,
    the kafka server delays the FetchResponse by 100 ms. This reduces
    the network traffic from megabits to the low kilobits. It also
    reduced the CPU utilization of our kafka server from 140% to 2%.
    
    Hopefully the risk of this change is low because
    the old behavior can be restored using the API by setting
    KafkaConfig.fetchMaxWait to 10000
    KafkaConfig.fetchMinBytes to 0

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: Addresses network flood from KafkaSpout to kaf...

Posted by knusbaum <gi...@git.apache.org>.
Github user knusbaum commented on the pull request:

    https://github.com/apache/storm/pull/1287#issuecomment-204207397
  
    Please file a [Jira](http://issues.apache.org/jira/) for this and update the title of the PR. I think it should be considered a bug, as this could potentially be detrimental to the network bandwidth.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1674: Addresses network flood from Kafka...

Posted by moesol <gi...@git.apache.org>.
Github user moesol commented on the pull request:

    https://github.com/apache/storm/pull/1287#issuecomment-204239216
  
    KafkaConfig has socketTimeoutMs which I assume maps to socket.timeout.ms. It's passed along to the SimpleConsumer's constructor.
    
    I'm not sure yet where rebalance.max.retries etc. would need to get applied. They are not a part of the FetchRequest. My vote is to keep this pull request focused on fixing the issue at hand and have a different pull request to implement the comprehensive list of configurations. 
    
    Also, I thought I saw some work to create a KafkaSpout that uses the Kafka 0.9 client. Perhaps the comprehensive list of configurations should be coupled with that work.
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: Addresses network flood from KafkaSpout to kaf...

Posted by sachingsachin <gi...@git.apache.org>.
Github user sachingsachin commented on the pull request:

    https://github.com/apache/storm/pull/1287#issuecomment-204216287
  
    Very useful observation @moesol 
    Comparing the [KafkaConfig.java](https://github.com/apache/storm/blob/a4f9f8bc5b4ca85de487a0a868e519ddcb94e852/external/storm-kafka/src/jvm/org/apache/storm/kafka/KafkaConfig.java) and [Kafka Configuration](https://kafka.apache.org/08/configuration.html), it seems several other useful configurations like socket.timeout.ms, rebalance.max.retries etc. are also absent from KafkaConfig.java
    
    We should probably add most (if not all) of these options to our spout-configuration.
    
    If we agree that it's useful, then we can add the remaining configuration options to this pull request OR I can submit a new one for the same.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1674: Addresses network flood from Kafka...

Posted by moesol <gi...@git.apache.org>.
Github user moesol commented on the pull request:

    https://github.com/apache/storm/pull/1287#issuecomment-210955517
  
    @knusbaum I added the JIRA ticket and updated the title a while ago. Are you the person who will actually perform the merge?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1287: STORM-1674: Addresses network flood from KafkaSpou...

Posted by moesol <gi...@git.apache.org>.
Github user moesol closed the pull request at:

    https://github.com/apache/storm/pull/1287


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1287: STORM-1674: Addresses network flood from KafkaSpout to ka...

Posted by moesol <gi...@git.apache.org>.
Github user moesol commented on the issue:

    https://github.com/apache/storm/pull/1287
  
    Merged via different PR. Closing this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1287: STORM-1674: Addresses network flood from KafkaSpout to ka...

Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:

    https://github.com/apache/storm/pull/1287
  
    @moesol can you please close this PR . Its already merged here https://github.com/apache/storm/commit/f069cfb08099a54448bba3f96f20c5b929ea6af3


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1674: Addresses network flood from Kafka...

Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the pull request:

    https://github.com/apache/storm/pull/1287#issuecomment-216483408
  
    @moesol Sorry to participate too lately.
    Could you craft the pull request based on master, 1.x-branch, and 0.10.x-branch? If you mind to craft several pull requests for the same, you can craft only one PR which is against the master branch.
    
    FYI, IMO, since we release Apache Storm 1.0.0, we can't handle 4 main version lines concurrently so fixing this at 0.9.x-branch doesn't make sense for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1674: Addresses network flood from Kafka...

Posted by abhishekagarwal87 <gi...@git.apache.org>.
Github user abhishekagarwal87 commented on the pull request:

    https://github.com/apache/storm/pull/1287#issuecomment-204266196
  
    This should also go to 1.x-branch given that it is bug and can improve the network bandwidth for lower load topologies


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---