Re: Consumer offsets reset for _all_ topics after increasing partitions for one topic

[Juho Autio <ju...@rovio.com>]

Does anyone know about this? Altering topic partitions seems to reset
consumer offsets.

On Tue, Sep 27, 2016 at 1:28 PM, Juho Autio <ju...@rovio.com> wrote:

> I increased partitions for one existing topic (2->10), but was surprised
> to see that it entirely reset the committed offsets of my consumer group.
>
> All topics & partitions were reset to the earliest offset available, and
> the consumer read everything again.
>
> Documentation doesn't mention anything like this. Is this how it's
> supposed to work, or a bug?
>
> I would've expected the consumer offsets to not decrease at all,
> especially for the topics that I didn't even touch.
>
> For the altered topic I would've expected that consuming the previously
> existing partitions 0 and 1 would've continued from the position where they
> were, and naturally starting to read the new added partitions from 0.
>
> I added partitions according to the "Modifying topics" section of Kafka
> 0.10.0 Documentation:
>
> "To add partitions you can do
>
>  > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic
> altered_topic --partitions 10
> "
>
> Previously this topic had 2 partitions.
>
> For the consumer I'm using:
> kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()
>
> And version is:
>
> <groupId>org.apache.kafka</groupId>
> <artifactId>kafka_2.11</artifactId>
> <version>0.10.0.1</version>
>
> Kafka cluster itself is kafka_2.11-0.10.0.1.
>

Re: Consumer offsets reset for _all_ topics after increasing partitions for one topic

[Juho Autio <ju...@rovio.com>]

Created a bug: https://issues.apache.org/jira/browse/KAFKA-4299

On Wed, Oct 5, 2016 at 12:34 PM, Juho Autio <ju...@rovio.com> wrote:

> Does anyone know about this? Altering topic partitions seems to reset
> consumer offsets.
>
> On Tue, Sep 27, 2016 at 1:28 PM, Juho Autio <ju...@rovio.com> wrote:
>
>> I increased partitions for one existing topic (2->10), but was surprised
>> to see that it entirely reset the committed offsets of my consumer group.
>>
>> All topics & partitions were reset to the earliest offset available, and
>> the consumer read everything again.
>>
>> Documentation doesn't mention anything like this. Is this how it's
>> supposed to work, or a bug?
>>
>> I would've expected the consumer offsets to not decrease at all,
>> especially for the topics that I didn't even touch.
>>
>> For the altered topic I would've expected that consuming the previously
>> existing partitions 0 and 1 would've continued from the position where they
>> were, and naturally starting to read the new added partitions from 0.
>>
>> I added partitions according to the "Modifying topics" section of Kafka
>> 0.10.0 Documentation:
>>
>> "To add partitions you can do
>>
>>  > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic
>> altered_topic --partitions 10
>> "
>>
>> Previously this topic had 2 partitions.
>>
>> For the consumer I'm using:
>> kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()
>>
>> And version is:
>>
>> <groupId>org.apache.kafka</groupId>
>> <artifactId>kafka_2.11</artifactId>
>> <version>0.10.0.1</version>
>>
>> Kafka cluster itself is kafka_2.11-0.10.0.1.
>>
>
>


-- 
*Juho Autio*
Senior Analytics Developer

Data Ops, Games
Rovio Entertainment Ltd
Mobile: + 358 (0)45 313 0122
juho.autio@rovio.com
www.rovio.com

*This message and its attachments may contain confidential information and
is intended solely for the attention and use of the named addressee(s). If
you are not the intended recipient and / or you have received this message
in error, please contact the sender immediately and delete all material you
have received in this message. You are hereby notified that any use of the
information, which you have received in error in whatsoever form, is
strictly prohibited. Thank you for your co-operation.*