[jira] [Updated] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

["P. Taylor Goetz (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

P. Taylor Goetz updated STORM-1674:
-----------------------------------
    Affects Version/s: 1.0.1
                       0.10.1

> Idle KafkaSpout consumes more bandwidth than needed
> ---------------------------------------------------
>
>                 Key: STORM-1674
>                 URL: https://issues.apache.org/jira/browse/STORM-1674
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-kafka
>    Affects Versions: 0.9.3, 0.10.1, 1.0.1
>            Reporter: Robert Hastings
>             Fix For: 2.0.0, 1.0.2, 1.1.0
>
>
> Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
> and our kafka servers even though no Kafka messages were moving.
> Using the wireshark kafka dissector, we were able to see that
> each FetchRequest had maxWait set to 10000
> and minBytes set to 0. When binBytes is set to 0 the kafka server
> responds immediately when there are no messages. In turn the KafkaSpout
> polls without any delay causing a constant stream of FetchRequest/
> FetchResponse messages. Using a non-KafkaSpout client had a similar
> traffic pattern with two key differences
> 1) minBytes was 1
> 2) maxWait was 100
> With these FetchRequest parameters and no messages flowing,
> the kafka server delays the FetchResponse by 100 ms. This reduces
> the network traffic from megabits to the low kilobits. It also
> reduced the CPU utilization of our kafka server from 140% to 2%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)