[jira] [Commented] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268443#comment-15268443 ] 

ASF GitHub Bot commented on STORM-1674:
---------------------------------------

Github user HeartSaVioR commented on the pull request:

    https://github.com/apache/storm/pull/1287#issuecomment-216483408
  
    @moesol Sorry to participate too lately.
    Could you craft the pull request based on master, 1.x-branch, and 0.10.x-branch? If you mind to craft several pull requests for the same, you can craft only one PR which is against the master branch.
    
    FYI, IMO, since we release Apache Storm 1.0.0, we can't handle 4 main version lines concurrently so fixing this at 0.9.x-branch doesn't make sense for now.


> Idle KafkaSpout consumes more bandwidth than needed
> ---------------------------------------------------
>
>                 Key: STORM-1674
>                 URL: https://issues.apache.org/jira/browse/STORM-1674
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-kafka
>    Affects Versions: 0.9.3
>            Reporter: Robert Hastings
>
> Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
> and our kafka servers even though no Kafka messages were moving.
> Using the wireshark kafka dissector, we were able to see that
> each FetchRequest had maxWait set to 10000
> and minBytes set to 0. When binBytes is set to 0 the kafka server
> responds immediately when there are no messages. In turn the KafkaSpout
> polls without any delay causing a constant stream of FetchRequest/
> FetchResponse messages. Using a non-KafkaSpout client had a similar
> traffic pattern with two key differences
> 1) minBytes was 1
> 2) maxWait was 100
> With these FetchRequest parameters and no messages flowing,
> the kafka server delays the FetchResponse by 100 ms. This reduces
> the network traffic from megabits to the low kilobits. It also
> reduced the CPU utilization of our kafka server from 140% to 2%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)