You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Matthias J. Sax (JIRA)" <ji...@apache.org> on 2017/11/16 06:37:00 UTC
[jira] [Commented] (KAFKA-6212) Kafka Streams - Incorrect partition
rebalancing
[ https://issues.apache.org/jira/browse/KAFKA-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254825#comment-16254825 ]
Matthias J. Sax commented on KAFKA-6212:
----------------------------------------
Thanks for report this. It's hard to say what happening without any log output. It's too bad you are stuck with {{0.10.0.x}} -- Kafka Streams is backward compatible to older clients starting from {{0.10.1.x}}...
> Kafka Streams - Incorrect partition rebalancing
> -----------------------------------------------
>
> Key: KAFKA-6212
> URL: https://issues.apache.org/jira/browse/KAFKA-6212
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.10.0.1
> Environment: Linux
> Reporter: Ivan Atanasov
>
> Trying to use streaming with version 0.10.0.1 of kafka but it is not working how I'd expect. I realize that this is a fairly old version now but it is what we are running and are not in a position to upgrade right now.
> The particular problem I am having is when an extra instance of the streaming app is run using the same application ID. What seems to happen is the newly introduced instance takes half of the partitions available, which is expected but the original instance drops all the partitions it was reading from. therefore from then on, data is only read from half the partitions.
> Strangely offsets are still being committed for the other partitions but the data from them is not consumed as expected.
> My topology is very simple for now, all it does is a print of the message. Also I have tried making both instances use different client IDs and state directories.
> Is this a known bug in 0.10.0.1?
> *Logs Below:*
> Instance 1:
> {quote}[2017-11-15 10:41:41,597] INFO [StreamThread-1] Setting newly assigned partitions [rawEvents-5, rawEvents-6, rawEvents-3, rawEvents-4, rawEvents-9, rawEvents-7, rawEvents-8, rawEvents-1, rawEvents-2, rawEvents-0] for group kafka-stream-test (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-11-15 10:41:41,616] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_0 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,645] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_1 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,645] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_2 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_3 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_4 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_5 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_6 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_7 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_8 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,648] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_9 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,682] INFO [StreamThread-1] Revoking previously assigned partitions [rawEvents-5, rawEvents-6, rawEvents-3, rawEvents-4, rawEvents-9, rawEvents-7, rawEvents-8, rawEvents-1, rawEvents-2, rawEvents-0] for group kafka-stream-test (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-11-15 10:42:08,682] INFO [StreamThread-1] Removing a task 0_0 (org.apache.kafka.streams.processor.internals.StreamThread){quote}
> Instance 2:
> {quote}[2017-11-15 10:42:08,827] INFO [StreamThread-1] Successfully joined group kafka-stream-test with generation 2 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2017-11-15 10:42:08,829] INFO [StreamThread-1] Setting newly assigned partitions [rawEvents-5, rawEvents-3, rawEvents-1, rawEvents-2, rawEvents-0] for group kafka-stream-test (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-11-15 10:42:08,840] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_0 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,869] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_1 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,870] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_2 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,871] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_3 (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,872] INFO [StreamThread-1] Creating restoration consumer client for stream task #0_5 (org.apache.kafka.streams.processor.internals.StreamTask){quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)