You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Raghu Angadi <an...@gmail.com> on 2017/07/26 04:43:35 UTC

Consumer group metadata retention

I am writing an exactly-once Kafka sink for Apache Beam.
In order to avoid duplicates due to retries, it stores a sequential id and
producer signature in consumer group metadata, and commits it atomically
with messages (using sendOffsetsToTransaction()).

I have a couple of clarification questions on partition metadata associated
with a consumer group.

   - Looks like a partition number could be larger than number of
   partitions for a topic. Is this formally supported (which is great!)?
      - E.g. I was able to set and read partition metadata for partition
      #20 for a topic with 4 partitions (though `console-consumer-group` didn't
      quite work.).
   - This is critical information for a sink. But looks like the metadata
   gets purged in 24 hours (server config) if it is inactive. Is there a way
   to set a longer TTL programatically?

Thanks.
Raghu.

Re: Consumer group metadata retention

Posted by James Cheng <wu...@gmail.com>.
The offsets.retention.minutes value (1440 = 24 hours = 1 day) is a broker level configuration, and can't be changed dynamically during runtime. You would have to modify the broker configurations, and restart the brokers.

-James

> On Jul 25, 2017, at 9:43 PM, Raghu Angadi <an...@gmail.com> wrote:
> 
> I am writing an exactly-once Kafka sink for Apache Beam.
> In order to avoid duplicates due to retries, it stores a sequential id and
> producer signature in consumer group metadata, and commits it atomically
> with messages (using sendOffsetsToTransaction()).
> 
> I have a couple of clarification questions on partition metadata associated
> with a consumer group.
> 
>   - Looks like a partition number could be larger than number of
>   partitions for a topic. Is this formally supported (which is great!)?
>      - E.g. I was able to set and read partition metadata for partition
>      #20 for a topic with 4 partitions (though `console-consumer-group` didn't
>      quite work.).
>   - This is critical information for a sink. But looks like the metadata
>   gets purged in 24 hours (server config) if it is inactive. Is there a way
>   to set a longer TTL programatically?
> 
> Thanks.
> Raghu.