You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Saladi Naidu <na...@yahoo.com.INVALID> on 2015/03/20 21:41:00 UTC
No broker partitions consumed by consumer thread
We are running 0.8.1.1, am experiencing following issue, any clue
I have 7 topics in my Kafka cluster of 5 nodes. I pushed 1 million messages to each topic using perf producer tool (no compression, no acknowledgement) . I saw old messages about a bug in 0.8 Consumer code and it is fixed but I am still experiencing this issue
/opt/kafka/kafka-0.8.1.1-src/bin/kafka-producer-perf-test.sh --topics ${dc}-perf-topic${topic} --messages 142857 --message-size 2000 --broker-list ${listBroker} --threads ${threads} --request-num-acks ${acks} --compression-codec ${codec} > ${logFileProd}.${dc}.topic${topic}.log 2>&1
After Producers populated topics, I run 7 consumers to consume from each above topic (1 consumer from 1 topic only) using Consumer Perf tool using below command
/opt/kafka/kafka-0.8.1.1-src/bin/kafka-consumer-perf-test.sh --zookeeper ${listZoo}/kafka --from-latest --topic ${dc}-perf-topic${topic} --compression-codec ${codec} > ${logFileConsume}.${dc}.topic${topic}.log 2>&1
All my consumers end up with following message
start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.secSLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.[2015-03-20 18:51:36,890] WARN [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No broker partitions consumed by consumer thread perf-consumer-68009_gckcnsys090001.1426877494243-2472feec-8 for topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,903] WARN [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No broker partitions consumed by consumer thread perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-7 for topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,904] WARN [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No broker partitions consumed by consumer thread perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-4 for topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,906] WARN [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No broker partitions consumed by consumer thread perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-1 for topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,906] WARN [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No broker partitions consumed by consumer thread perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-5 for topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No broker partitions consumed by consumer thread perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-2 for topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No broker partitions consumed by consumer thread perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-9 for topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No broker partitions consumed by consumer thread perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-3 for topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No broker partitions consumed by consumer thread perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-6 for topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)2015-03-20 18:51:37:939, 2015-03-20 18:51:42:960, 1048576, 0.0000, 0.0000, 0, 0.0000
Naidu Saladi
Re: No broker partitions consumed by consumer thread
Posted by Gwen Shapira <gs...@cloudera.com>.
I think the issue is with " --from-latest" - this means consumers will
consume only data that arrives AFTER they start.
If you do that, first start consumers, leave them running, and then start
producers.
If you want to run producers first and only start consuming when producers
are done, remove this option.
Gwen
On Fri, Mar 20, 2015 at 1:41 PM, Saladi Naidu <naidusp2002@yahoo.com.invalid
> wrote:
> We are running 0.8.1.1, am experiencing following issue, any clue
> I have 7 topics in my Kafka cluster of 5 nodes. I pushed 1 million
> messages to each topic using perf producer tool (no compression, no
> acknowledgement) . I saw old messages about a bug in 0.8 Consumer code and
> it is fixed but I am still experiencing this issue
> /opt/kafka/kafka-0.8.1.1-src/bin/kafka-producer-perf-test.sh --topics
> ${dc}-perf-topic${topic} --messages 142857 --message-size 2000
> --broker-list ${listBroker} --threads ${threads} --request-num-acks ${acks}
> --compression-codec ${codec} > ${logFileProd}.${dc}.topic${topic}.log 2>&1
>
> After Producers populated topics, I run 7 consumers to consume from each
> above topic (1 consumer from 1 topic only) using Consumer Perf tool using
> below command
> /opt/kafka/kafka-0.8.1.1-src/bin/kafka-consumer-perf-test.sh --zookeeper
> ${listZoo}/kafka --from-latest --topic ${dc}-perf-topic${topic}
> --compression-codec ${codec} > ${logFileConsume}.${dc}.topic${topic}.log
> 2>&1
>
> All my consumers end up with following message
> start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec,
> data.consumed.in.nMsg, nMsg.secSLF4J: Failed to load class
> "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP)
> logger implementationSLF4J: See
> http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.[2015-03-20 18:51:36,890] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.1426877494243-2472feec-8 for topic
> dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20
> 18:51:36,903] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-7 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,904] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-4 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,906] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-1 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,906] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-5 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-2 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-9 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-3 for
> topic dc7-perf-topic7
> (kafka.consumer.ZookeeperConsumerConnector)[2015-03-20 18:51:36,907] WARN
> [perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec], No
> broker partitions consumed by consumer thread
> perf-consumer-68009_gckcnsys090001.c090.net-1426877494243-2472feec-6 for
> topic dc7-perf-topic7 (kafka.consumer.ZookeeperConsumerConnector)2015-03-20
> 18:51:37:939, 2015-03-20 18:51:42:960, 1048576, 0.0000, 0.0000, 0, 0.0000
>
> Naidu Saladi
>