You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 杨珏吉 <ju...@gmail.com> on 2018/11/01 03:38:39 UTC
kafka 0.9.0 is busy waiting when no message available
Hi Kafka experts
I'm using kafka 0.9.0. I made sure that
Fetch.MinBytes(fetch.min.bytes) is set to be 1 and Fetch.MaxWaitTime
(fetch.wait.max.ms) is set to be 250 ms. There are no messages in the
topic but client (I tried golang client `sarama-cluster` and
`bin/kafka-console-consumer.sh`) keeps fetching and broker responses
immediately in 1 ms. This spinning cause huge network and cpu
consumption.
More details:
1. No client side error logs or outputs.
2. For empty topic, the broker will wait 250 ms as expected.
3. After a new message produced and consumed normally, the broker will
be waiting 250 ms as no more message are produced. But once the broker
performs "Deleting index", the broker will again
response immediately with empty message.
the `server.log` at the time when broker change from waiting 250 ms to
responding immediately
[2018-10-26 21:14:52,788] INFO [Group Metadata Manager on Broker 1]:
Removed 0 expired offsets in 0 milliseconds.
(kafka.coordinator.GroupMetadataManager)
[2018-10-26 21:15:21,679] INFO Rolled new log segment for 'sync-0' in
1 ms. (kafka.log.Log)
[2018-10-26 21:15:21,679] INFO Scheduling log segment 0 for log sync-0
for deletion. (kafka.log.Log)
[2018-10-26 21:15:21,680] INFO Rolled new log segment for 'Sync-0' in
0 ms. (kafka.log.Log)
[2018-10-26 21:15:21,680] INFO Scheduling log segment 83 for log
Sync-0 for deletion. (kafka.log.Log)
[2018-10-26 21:16:21,680] INFO Deleting segment 0 from log sync-0.
(kafka.log.Log)
[2018-10-26 21:16:21,680] INFO Deleting index
/data/log/kafka-logs/sync-0/00000000000000000000.index.deleted
(kafka.log.OffsetIndex)
[2018-10-26 21:16:21,680] INFO Deleting segment 83 from log Sync-0.
(kafka.log.Log)
[2018-10-26 21:16:21,680] INFO Deleting index
/data/log/kafka-logs/Sync-0/00000000000000000083.index.deleted
(kafka.log.OffsetIndex)
Does anyone have ever faced this problem?
--
Best Regards.
Yang Jueji.
Re: kafka 0.9.0 is busy waiting when no message available
Posted by 杨珏吉 <ju...@gmail.com>.
It was reported as KAFKA-3159 fixed in 0.9.0.1. Upgrading solves my problem.
https://issues.apache.org/jira/browse/KAFKA-3159?attachmentSortBy=fileName
杨珏吉 <ju...@gmail.com> 于2018年11月1日周四 上午11:38写道：
>
> Hi Kafka experts
>
> I'm using kafka 0.9.0. I made sure that
> Fetch.MinBytes(fetch.min.bytes) is set to be 1 and Fetch.MaxWaitTime
> (fetch.wait.max.ms) is set to be 250 ms. There are no messages in the
> topic but client (I tried golang client `sarama-cluster` and
> `bin/kafka-console-consumer.sh`) keeps fetching and broker responses
> immediately in 1 ms. This spinning cause huge network and cpu
> consumption.
>
> More details:
> 1. No client side error logs or outputs.
> 2. For empty topic, the broker will wait 250 ms as expected.
> 3. After a new message produced and consumed normally, the broker will
> be waiting 250 ms as no more message are produced. But once the broker
> performs "Deleting index", the broker will again
> response immediately with empty message.
> the `server.log` at the time when broker change from waiting 250 ms to
> responding immediately
>
> [2018-10-26 21:14:52,788] INFO [Group Metadata Manager on Broker 1]:
> Removed 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.GroupMetadataManager)
> [2018-10-26 21:15:21,679] INFO Rolled new log segment for 'sync-0' in
> 1 ms. (kafka.log.Log)
> [2018-10-26 21:15:21,679] INFO Scheduling log segment 0 for log sync-0
> for deletion. (kafka.log.Log)
> [2018-10-26 21:15:21,680] INFO Rolled new log segment for 'Sync-0' in
> 0 ms. (kafka.log.Log)
> [2018-10-26 21:15:21,680] INFO Scheduling log segment 83 for log
> Sync-0 for deletion. (kafka.log.Log)
> [2018-10-26 21:16:21,680] INFO Deleting segment 0 from log sync-0.
> (kafka.log.Log)
> [2018-10-26 21:16:21,680] INFO Deleting index
> /data/log/kafka-logs/sync-0/00000000000000000000.index.deleted
> (kafka.log.OffsetIndex)
> [2018-10-26 21:16:21,680] INFO Deleting segment 83 from log Sync-0.
> (kafka.log.Log)
> [2018-10-26 21:16:21,680] INFO Deleting index
> /data/log/kafka-logs/Sync-0/00000000000000000083.index.deleted
> (kafka.log.OffsetIndex)
>
> Does anyone have ever faced this problem?
>
> --
> Best Regards.
>
> Yang Jueji.
--
Best Regards.
Yang Jueji.