You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by pravin kumar <pk...@gmail.com> on 2018/03/27 12:41:55 UTC
Consumer slowness issue
i have two topics with 5 partitions each
wikifeedInputT10
KafkaProducer produces 100000 elements and wikifeedInputT10 have
received these elements.
[admin@nms-181 bin]$ sh kafka-run-class.sh kafka.tools.GetOffsetShell
--broker-list localhost:9092 --topic wikifeedInputT10 --time -1
wikifeedInputT10:2:20000
wikifeedInputT10:4:20000
wikifeedInputT10:1:20000
wikifeedInputT10:3:20000
wikifeedInputT10:0:20000
but after processing reading from my outputTopic: wikifeedOutputT15
i have received
[admin@nms-181 bin]$ sh kafka-run-class.sh kafka.tools.GetOffsetShell
--broker-list localhost:9092 --topic wikifeedOutputT15 --time -1
wikifeedOutputT15:2:1
wikifeedOutputT15:4:1
wikifeedOutputT15:1:3
wikifeedOutputT15:3:3
wikifeedOutputT15:0:3
I have received the output in my console as
[2018-03-27 17:55:32,359] INFO Kafka version : 1.0.1
(org.apache.kafka.common.utils.AppInfoParser)
[2018-03-27 17:55:32,359] INFO Kafka commitId : c0518aa65f25317e
(org.apache.kafka.common.utils.AppInfoParser)
[2018-03-27 17:55:32,600] INFO [Consumer clientId=C2,
groupId=ConsumerWikiFeedLambda4] Discovered group coordinator
nms-181.nmsworks.co.in:9092 (id: 2147483647 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-03-27 17:55:32,602] INFO [Consumer clientId=C2,
groupId=ConsumerWikiFeedLambda4] Revoking previously assigned
partitions [] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2018-03-27 17:55:32,602] INFO [Consumer clientId=C2,
groupId=ConsumerWikiFeedLambda4] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-03-27 17:55:32,610] INFO [Consumer clientId=C2,
groupId=ConsumerWikiFeedLambda4] Successfully joined group with
generation 5 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-03-27 17:55:32,611] INFO [Consumer clientId=C2,
groupId=ConsumerWikiFeedLambda4] Setting newly assigned partitions
[wikifeedOutputT15-2, wikifeedOutputT15-1, wikifeedOutputT15-0,
wikifeedOutputT15-4, wikifeedOutputT15-3]
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
Topic :::::::wikifeedOutputT15 Partition:::::0 Key::::joe = Value:::::: 18263
Topic :::::::wikifeedOutputT15 Partition:::::0 Key::::phil = Value:::::: 18230
Topic :::::::wikifeedOutputT15 Partition:::::0 Key::::tania =
Value:::::: 18344
Topic :::::::wikifeedOutputT15 Partition:::::1 Key::::pravin =
Value:::::: 18140
Topic :::::::wikifeedOutputT15 Partition:::::1 Key::::kumar =
Value:::::: 18248
Topic :::::::wikifeedOutputT15 Partition:::::1 Key::::joseph =
Value:::::: 18116
Topic :::::::wikifeedOutputT15 Partition:::::2 Key::::lauren =
Value:::::: 18150
Topic :::::::wikifeedOutputT15 Partition:::::3 Key::::bob = Value:::::: 18131
Topic :::::::wikifeedOutputT15 Partition:::::3 Key::::erica =
Value:::::: 18084
Topic :::::::wikifeedOutputT15 Partition:::::3 Key::::damian =
Value:::::: 18126
Topic :::::::wikifeedOutputT15 Partition:::::4 Key::::sam = Value:::::: 18168
it stops here and im not getting any msgs
i have attached my code below
Fwd: Consumer slowness issue
Posted by pravin kumar <pk...@gmail.com>.
---------- Forwarded message ----------
From: "pravin kumar" <pk...@gmail.com>
Date: 27-Mar-2018 6:11 PM
Subject: Consumer slowness issue
To: <us...@kafka.apache.org>
Cc:
i have two topics with 5 partitions each
wikifeedInputT10
KafkaProducer produces 100000 elements and wikifeedInputT10 have
received these elements.
[admin@nms-181 bin]$ sh kafka-run-class.sh kafka.tools.GetOffsetShell
--broker-list localhost:9092 --topic wikifeedInputT10 --time -1
wikifeedInputT10:2:20000
wikifeedInputT10:4:20000
wikifeedInputT10:1:20000
wikifeedInputT10:3:20000
wikifeedInputT10:0:20000
but after processing reading from my outputTopic: wikifeedOutputT15
i have received
[admin@nms-181 bin]$ sh kafka-run-class.sh kafka.tools.GetOffsetShell
--broker-list localhost:9092 --topic wikifeedOutputT15 --time -1
wikifeedOutputT15:2:1
wikifeedOutputT15:4:1
wikifeedOutputT15:1:3
wikifeedOutputT15:3:3
wikifeedOutputT15:0:3
I have received the output in my console as
[2018-03-27 17:55:32,359] INFO Kafka version : 1.0.1
(org.apache.kafka.common.utils.AppInfoParser)
[2018-03-27 17:55:32,359] INFO Kafka commitId : c0518aa65f25317e
(org.apache.kafka.common.utils.AppInfoParser)
[2018-03-27 17:55:32,600] INFO [Consumer clientId=C2,
groupId=ConsumerWikiFeedLambda4] Discovered group coordinator
nms-181.nmsworks.co.in:9092 (id: 2147483647 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-03-27 17:55:32,602] INFO [Consumer clientId=C2,
groupId=ConsumerWikiFeedLambda4] Revoking previously assigned
partitions [] (org.apache.kafka.clients.consumer.internals.
ConsumerCoordinator)
[2018-03-27 17:55:32,602] INFO [Consumer clientId=C2,
groupId=ConsumerWikiFeedLambda4] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-03-27 17:55:32,610] INFO [Consumer clientId=C2,
groupId=ConsumerWikiFeedLambda4] Successfully joined group with
generation 5 (org.apache.kafka.clients.consumer.internals.
AbstractCoordinator)
[2018-03-27 17:55:32,611] INFO [Consumer clientId=C2,
groupId=ConsumerWikiFeedLambda4] Setting newly assigned partitions
[wikifeedOutputT15-2, wikifeedOutputT15-1, wikifeedOutputT15-0,
wikifeedOutputT15-4, wikifeedOutputT15-3]
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
Topic :::::::wikifeedOutputT15 Partition:::::0 Key::::joe = Value::::::
18263
Topic :::::::wikifeedOutputT15 Partition:::::0 Key::::phil = Value::::::
18230
Topic :::::::wikifeedOutputT15 Partition:::::0 Key::::tania =
Value:::::: 18344
Topic :::::::wikifeedOutputT15 Partition:::::1 Key::::pravin =
Value:::::: 18140
Topic :::::::wikifeedOutputT15 Partition:::::1 Key::::kumar =
Value:::::: 18248
Topic :::::::wikifeedOutputT15 Partition:::::1 Key::::joseph =
Value:::::: 18116
Topic :::::::wikifeedOutputT15 Partition:::::2 Key::::lauren =
Value:::::: 18150
Topic :::::::wikifeedOutputT15 Partition:::::3 Key::::bob = Value::::::
18131
Topic :::::::wikifeedOutputT15 Partition:::::3 Key::::erica =
Value:::::: 18084
Topic :::::::wikifeedOutputT15 Partition:::::3 Key::::damian =
Value:::::: 18126
Topic :::::::wikifeedOutputT15 Partition:::::4 Key::::sam = Value::::::
18168
it stops here and im not getting any msgs
i have attached my code below