You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Abhimanyu Nagrath <ab...@gmail.com> on 2017/05/19 07:14:59 UTC
Messages are repeating in kafka
I am using single node Kafka broker (v 0.10.2) Hardware (8 core, 16 GB RAM,
1 TD HardDisk) and zookeeper (v 3.4.8). I have a topic with 200 partition
in which messages contains the total of 3 Million messages. It took 5 days
to completely process all the messages and as soon as message got processed
i.e. Kafka-consumer-groups.sh showed 0 lag in all the partition of the
topic I stopped the consumer .but after 6 hrs again it was showing the lag
of 2 million message which I found that were duplicate messages. This thing
is happening very frequently. My offsets are stored on Kafka broker itself.
My server configuration is:
broker.id=1
delete.topic.enable=true
#listeners=PLAINTEXT://:9092
#advertised.listeners=PLAINTEXT://your.host.name:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/kafka/data/logs
num.partitions=1
num.recovery.threads.per.data.dir=5
log.flush.interval.messages=10000
#log.flush.interval.ms=1000
log.retention.hours=480
log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=<zkIp>:2181
zookeeper.connection.timeout.ms=6000
Is there in the configuration that I am missing?
Any help is appreciated
Re: Messages are repeating in kafka
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
@Kant
Actually, after setting that configuration I deleted a topic having 100
partitions. After that, I checked through ./kafka-topic.sh --list command.
It was not showing that topic. Again I started producers and ran the
following command
*./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe
--group <groupName- for topic marked for deletion>*
and got the following output:
*Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).*
*[2017-05-24 07:17:48,444] WARN Error while fetching metadata with
correlation id 1 : {topicName=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient)*
The same describe groups command is working fine for other groups.
So I am not able to get what can be the reason for such error.and how to
fix it.
Regards,
Abhimanyu
On Wed, May 24, 2017 at 12:27 PM, kant kodali <ka...@gmail.com> wrote:
> @Abhimanyu
>
> 1) My guess is that topic offsets will remain for 30 days since that is the
> configuration you are explicitly setting and Kafka should respect that
> although I don't know for sure.
>
> 2) same as #1 offsets should remain to whatever time you specify.
>
> What is the problem with setting offsets.retention.minutes ==
> log.retention.hours
> ? Did it fix the problem you were facing before?
>
>
> On Tue, May 23, 2017 at 11:40 PM, Abhimanyu Nagrath <
> abhimanyunagrath@gmail.com> wrote:
>
> > Hi Kant,
> >
> > After setting this configuration offsets.retention.minutes . I am in
> doubt
> > about the two things
> > 1. If I am deleting a topic will that topic offsets would also get
> deleted
> > or will they present for 30 days?
> > 2. What will happen if for some topics my log.retention.hours = 168 and
> > offsets.retention.minutes= 1440 * 30 ?
> >
> > Regards,
> > Abhimanyu
> >
> > On Mon, May 22, 2017 at 4:09 PM, Abhimanyu Nagrath <
> > abhimanyunagrath@gmail.com> wrote:
> >
> > > @Kant I was going through the offset related configurations before
> > setting
> > > offsets.retention.minutes so came accross this configuration and
> thought
> > to
> > > ask whether this should also be tuned or not.
> > >
> > >
> > > Regards,
> > > Abhimanyu
> > >
> > >
> > >
> > >
> > > On Mon, May 22, 2017 at 2:24 PM, kant kodali <ka...@gmail.com>
> wrote:
> > >
> > >> @Abhimanyu Why do you think you need to set that? Did you try setting
> > >> offsets.retention.minutes
> > >> = 1440 * 30 and still seeing duplicates?
> > >>
> > >> On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath <
> > >> abhimanyunagrath@gmail.com> wrote:
> > >>
> > >> > Hi Girish ,
> > >> >
> > >> > Do I need to tune this configuration offsets.retention.check.interv
> > >> al.ms
> > >> > also . Please let me know if I need to tune any other configuration.
> > >> >
> > >> >
> > >> > Regards,
> > >> > Abhimanyu
> > >> >
> > >> > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <gi...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Yup, exactly as Kant said.
> > >> > > Also make sure that the retention of the offsets topic is an upper
> > >> bound
> > >> > > across all topics. So in this case, don't create any other topics
> in
> > >> the
> > >> > > future with retention of more than 30 days or otherwise they may
> > have
> > >> the
> > >> > > same problem too.
> > >> > >
> > >> > > On May 21, 2017 03:25, "Abhimanyu Nagrath" <
> > >> abhimanyunagrath@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> Hi Kant,
> > >> > >>
> > >> > >> Thanks for the suggestion.
> > >> > >>
> > >> > >>
> > >> > >> Regards,
> > >> > >> Abhimanyu
> > >> > >>
> > >> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth909@gmail.com
> >
> > >> > wrote:
> > >> > >>
> > >> > >>> @Abhimanyu You can try setting offset.retention = 30
> > >> (log.retention).
> > >> > At
> > >> > >>> most, you will have a storage overhead of 5 million msgs per
> day *
> > >> 30
> > >> > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since
> > you
> > >> > have
> > >> > >>> a
> > >> > >>> TB of hard disk)
> > >> > >>>
> > >> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <
> kanth909@gmail.com>
> > >> > wrote:
> > >> > >>>
> > >> > >>> > Looking at that ticket and reading the comments it looks like
> > one
> > >> of
> > >> > >>> the
> > >> > >>> > concern is as follows.
> > >> > >>> >
> > >> > >>> > "offsets.retention.minutes is designed to handle the case
> that a
> > >> > >>> consumer
> > >> > >>> > group goes away forever. In that case, we don't want to store
> > the
> > >> > >>> offsets
> > >> > >>> > for that group forever."
> > >> > >>> >
> > >> > >>> > This can simply be addressed by setting offset.retention ==
> > >> > >>> log.retention
> > >> > >>> > by default right? In which case offset wont be stored forever
> > even
> > >> > when
> > >> > >>> > consumer group goes away forever. When the consumer group goes
> > >> away
> > >> > >>> forever
> > >> > >>> > the upper bound to clean up offsets would be equal to
> > >> log.retention.
> > >> > >>> >
> > >> > >>> >
> > >> > >>> >
> > >> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <
> > kanth909@gmail.com>
> > >> > >>> wrote:
> > >> > >>> >
> > >> > >>> >> What is your average message size and network speed?
> > >> > >>> >>
> > >> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> > >> > >>> >> abhimanyunagrath@gmail.com> wrote:
> > >> > >>> >>
> > >> > >>> >>> Hi Girish,
> > >> > >>> >>>
> > >> > >>> >>> I did not set any value for offsets.retention.minutes so
> > >> therefore
> > >> > >>> what I
> > >> > >>> >>> think is picking its default value i.e 1440 minutes so what
> do
> > >> you
> > >> > >>> think
> > >> > >>> >>> what should I set if I am keeping my data for 30 days?
> > >> > >>> >>>
> > >> > >>> >>> Regards,
> > >> > >>> >>> Abhimanyu
> > >> > >>> >>>
> > >> > >>> >>
> > >> > >>> >>
> > >> > >>> >
> > >> > >>>
> > >> > >>
> > >> > >>
> > >> >
> > >>
> > >
> > >
> >
>
Re: Messages are repeating in kafka
Posted by kant kodali <ka...@gmail.com>.
@Abhimanyu
1) My guess is that topic offsets will remain for 30 days since that is the
configuration you are explicitly setting and Kafka should respect that
although I don't know for sure.
2) same as #1 offsets should remain to whatever time you specify.
What is the problem with setting offsets.retention.minutes ==
log.retention.hours
? Did it fix the problem you were facing before?
On Tue, May 23, 2017 at 11:40 PM, Abhimanyu Nagrath <
abhimanyunagrath@gmail.com> wrote:
> Hi Kant,
>
> After setting this configuration offsets.retention.minutes . I am in doubt
> about the two things
> 1. If I am deleting a topic will that topic offsets would also get deleted
> or will they present for 30 days?
> 2. What will happen if for some topics my log.retention.hours = 168 and
> offsets.retention.minutes= 1440 * 30 ?
>
> Regards,
> Abhimanyu
>
> On Mon, May 22, 2017 at 4:09 PM, Abhimanyu Nagrath <
> abhimanyunagrath@gmail.com> wrote:
>
> > @Kant I was going through the offset related configurations before
> setting
> > offsets.retention.minutes so came accross this configuration and thought
> to
> > ask whether this should also be tuned or not.
> >
> >
> > Regards,
> > Abhimanyu
> >
> >
> >
> >
> > On Mon, May 22, 2017 at 2:24 PM, kant kodali <ka...@gmail.com> wrote:
> >
> >> @Abhimanyu Why do you think you need to set that? Did you try setting
> >> offsets.retention.minutes
> >> = 1440 * 30 and still seeing duplicates?
> >>
> >> On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath <
> >> abhimanyunagrath@gmail.com> wrote:
> >>
> >> > Hi Girish ,
> >> >
> >> > Do I need to tune this configuration offsets.retention.check.interv
> >> al.ms
> >> > also . Please let me know if I need to tune any other configuration.
> >> >
> >> >
> >> > Regards,
> >> > Abhimanyu
> >> >
> >> > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <gi...@gmail.com>
> >> wrote:
> >> >
> >> > > Yup, exactly as Kant said.
> >> > > Also make sure that the retention of the offsets topic is an upper
> >> bound
> >> > > across all topics. So in this case, don't create any other topics in
> >> the
> >> > > future with retention of more than 30 days or otherwise they may
> have
> >> the
> >> > > same problem too.
> >> > >
> >> > > On May 21, 2017 03:25, "Abhimanyu Nagrath" <
> >> abhimanyunagrath@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> Hi Kant,
> >> > >>
> >> > >> Thanks for the suggestion.
> >> > >>
> >> > >>
> >> > >> Regards,
> >> > >> Abhimanyu
> >> > >>
> >> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <ka...@gmail.com>
> >> > wrote:
> >> > >>
> >> > >>> @Abhimanyu You can try setting offset.retention = 30
> >> (log.retention).
> >> > At
> >> > >>> most, you will have a storage overhead of 5 million msgs per day *
> >> 30
> >> > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since
> you
> >> > have
> >> > >>> a
> >> > >>> TB of hard disk)
> >> > >>>
> >> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <ka...@gmail.com>
> >> > wrote:
> >> > >>>
> >> > >>> > Looking at that ticket and reading the comments it looks like
> one
> >> of
> >> > >>> the
> >> > >>> > concern is as follows.
> >> > >>> >
> >> > >>> > "offsets.retention.minutes is designed to handle the case that a
> >> > >>> consumer
> >> > >>> > group goes away forever. In that case, we don't want to store
> the
> >> > >>> offsets
> >> > >>> > for that group forever."
> >> > >>> >
> >> > >>> > This can simply be addressed by setting offset.retention ==
> >> > >>> log.retention
> >> > >>> > by default right? In which case offset wont be stored forever
> even
> >> > when
> >> > >>> > consumer group goes away forever. When the consumer group goes
> >> away
> >> > >>> forever
> >> > >>> > the upper bound to clean up offsets would be equal to
> >> log.retention.
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <
> kanth909@gmail.com>
> >> > >>> wrote:
> >> > >>> >
> >> > >>> >> What is your average message size and network speed?
> >> > >>> >>
> >> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> >> > >>> >> abhimanyunagrath@gmail.com> wrote:
> >> > >>> >>
> >> > >>> >>> Hi Girish,
> >> > >>> >>>
> >> > >>> >>> I did not set any value for offsets.retention.minutes so
> >> therefore
> >> > >>> what I
> >> > >>> >>> think is picking its default value i.e 1440 minutes so what do
> >> you
> >> > >>> think
> >> > >>> >>> what should I set if I am keeping my data for 30 days?
> >> > >>> >>>
> >> > >>> >>> Regards,
> >> > >>> >>> Abhimanyu
> >> > >>> >>>
> >> > >>> >>
> >> > >>> >>
> >> > >>> >
> >> > >>>
> >> > >>
> >> > >>
> >> >
> >>
> >
> >
>
Re: Messages are repeating in kafka
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
Hi Kant,
After setting this configuration offsets.retention.minutes . I am in doubt
about the two things
1. If I am deleting a topic will that topic offsets would also get deleted
or will they present for 30 days?
2. What will happen if for some topics my log.retention.hours = 168 and
offsets.retention.minutes= 1440 * 30 ?
Regards,
Abhimanyu
On Mon, May 22, 2017 at 4:09 PM, Abhimanyu Nagrath <
abhimanyunagrath@gmail.com> wrote:
> @Kant I was going through the offset related configurations before setting
> offsets.retention.minutes so came accross this configuration and thought to
> ask whether this should also be tuned or not.
>
>
> Regards,
> Abhimanyu
>
>
>
>
> On Mon, May 22, 2017 at 2:24 PM, kant kodali <ka...@gmail.com> wrote:
>
>> @Abhimanyu Why do you think you need to set that? Did you try setting
>> offsets.retention.minutes
>> = 1440 * 30 and still seeing duplicates?
>>
>> On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath <
>> abhimanyunagrath@gmail.com> wrote:
>>
>> > Hi Girish ,
>> >
>> > Do I need to tune this configuration offsets.retention.check.interv
>> al.ms
>> > also . Please let me know if I need to tune any other configuration.
>> >
>> >
>> > Regards,
>> > Abhimanyu
>> >
>> > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <gi...@gmail.com>
>> wrote:
>> >
>> > > Yup, exactly as Kant said.
>> > > Also make sure that the retention of the offsets topic is an upper
>> bound
>> > > across all topics. So in this case, don't create any other topics in
>> the
>> > > future with retention of more than 30 days or otherwise they may have
>> the
>> > > same problem too.
>> > >
>> > > On May 21, 2017 03:25, "Abhimanyu Nagrath" <
>> abhimanyunagrath@gmail.com>
>> > > wrote:
>> > >
>> > >> Hi Kant,
>> > >>
>> > >> Thanks for the suggestion.
>> > >>
>> > >>
>> > >> Regards,
>> > >> Abhimanyu
>> > >>
>> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <ka...@gmail.com>
>> > wrote:
>> > >>
>> > >>> @Abhimanyu You can try setting offset.retention = 30
>> (log.retention).
>> > At
>> > >>> most, you will have a storage overhead of 5 million msgs per day *
>> 30
>> > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you
>> > have
>> > >>> a
>> > >>> TB of hard disk)
>> > >>>
>> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <ka...@gmail.com>
>> > wrote:
>> > >>>
>> > >>> > Looking at that ticket and reading the comments it looks like one
>> of
>> > >>> the
>> > >>> > concern is as follows.
>> > >>> >
>> > >>> > "offsets.retention.minutes is designed to handle the case that a
>> > >>> consumer
>> > >>> > group goes away forever. In that case, we don't want to store the
>> > >>> offsets
>> > >>> > for that group forever."
>> > >>> >
>> > >>> > This can simply be addressed by setting offset.retention ==
>> > >>> log.retention
>> > >>> > by default right? In which case offset wont be stored forever even
>> > when
>> > >>> > consumer group goes away forever. When the consumer group goes
>> away
>> > >>> forever
>> > >>> > the upper bound to clean up offsets would be equal to
>> log.retention.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <ka...@gmail.com>
>> > >>> wrote:
>> > >>> >
>> > >>> >> What is your average message size and network speed?
>> > >>> >>
>> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
>> > >>> >> abhimanyunagrath@gmail.com> wrote:
>> > >>> >>
>> > >>> >>> Hi Girish,
>> > >>> >>>
>> > >>> >>> I did not set any value for offsets.retention.minutes so
>> therefore
>> > >>> what I
>> > >>> >>> think is picking its default value i.e 1440 minutes so what do
>> you
>> > >>> think
>> > >>> >>> what should I set if I am keeping my data for 30 days?
>> > >>> >>>
>> > >>> >>> Regards,
>> > >>> >>> Abhimanyu
>> > >>> >>>
>> > >>> >>
>> > >>> >>
>> > >>> >
>> > >>>
>> > >>
>> > >>
>> >
>>
>
>
Re: Messages are repeating in kafka
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
@Kant I was going through the offset related configurations before setting
offsets.retention.minutes so came accross this configuration and thought to
ask whether this should also be tuned or not.
Regards,
Abhimanyu
On Mon, May 22, 2017 at 2:24 PM, kant kodali <ka...@gmail.com> wrote:
> @Abhimanyu Why do you think you need to set that? Did you try setting
> offsets.retention.minutes
> = 1440 * 30 and still seeing duplicates?
>
> On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath <
> abhimanyunagrath@gmail.com> wrote:
>
> > Hi Girish ,
> >
> > Do I need to tune this configuration offsets.retention.check.interval.ms
> > also . Please let me know if I need to tune any other configuration.
> >
> >
> > Regards,
> > Abhimanyu
> >
> > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <gi...@gmail.com>
> wrote:
> >
> > > Yup, exactly as Kant said.
> > > Also make sure that the retention of the offsets topic is an upper
> bound
> > > across all topics. So in this case, don't create any other topics in
> the
> > > future with retention of more than 30 days or otherwise they may have
> the
> > > same problem too.
> > >
> > > On May 21, 2017 03:25, "Abhimanyu Nagrath" <abhimanyunagrath@gmail.com
> >
> > > wrote:
> > >
> > >> Hi Kant,
> > >>
> > >> Thanks for the suggestion.
> > >>
> > >>
> > >> Regards,
> > >> Abhimanyu
> > >>
> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <ka...@gmail.com>
> > wrote:
> > >>
> > >>> @Abhimanyu You can try setting offset.retention = 30 (log.retention).
> > At
> > >>> most, you will have a storage overhead of 5 million msgs per day * 30
> > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you
> > have
> > >>> a
> > >>> TB of hard disk)
> > >>>
> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <ka...@gmail.com>
> > wrote:
> > >>>
> > >>> > Looking at that ticket and reading the comments it looks like one
> of
> > >>> the
> > >>> > concern is as follows.
> > >>> >
> > >>> > "offsets.retention.minutes is designed to handle the case that a
> > >>> consumer
> > >>> > group goes away forever. In that case, we don't want to store the
> > >>> offsets
> > >>> > for that group forever."
> > >>> >
> > >>> > This can simply be addressed by setting offset.retention ==
> > >>> log.retention
> > >>> > by default right? In which case offset wont be stored forever even
> > when
> > >>> > consumer group goes away forever. When the consumer group goes away
> > >>> forever
> > >>> > the upper bound to clean up offsets would be equal to
> log.retention.
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <ka...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> >> What is your average message size and network speed?
> > >>> >>
> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> > >>> >> abhimanyunagrath@gmail.com> wrote:
> > >>> >>
> > >>> >>> Hi Girish,
> > >>> >>>
> > >>> >>> I did not set any value for offsets.retention.minutes so
> therefore
> > >>> what I
> > >>> >>> think is picking its default value i.e 1440 minutes so what do
> you
> > >>> think
> > >>> >>> what should I set if I am keeping my data for 30 days?
> > >>> >>>
> > >>> >>> Regards,
> > >>> >>> Abhimanyu
> > >>> >>>
> > >>> >>
> > >>> >>
> > >>> >
> > >>>
> > >>
> > >>
> >
>
Re: Messages are repeating in kafka
Posted by kant kodali <ka...@gmail.com>.
@Abhimanyu Why do you think you need to set that? Did you try setting
offsets.retention.minutes
= 1440 * 30 and still seeing duplicates?
On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath <
abhimanyunagrath@gmail.com> wrote:
> Hi Girish ,
>
> Do I need to tune this configuration offsets.retention.check.interval.ms
> also . Please let me know if I need to tune any other configuration.
>
>
> Regards,
> Abhimanyu
>
> On Sun, May 21, 2017 at 8:01 PM, Girish Aher <gi...@gmail.com> wrote:
>
> > Yup, exactly as Kant said.
> > Also make sure that the retention of the offsets topic is an upper bound
> > across all topics. So in this case, don't create any other topics in the
> > future with retention of more than 30 days or otherwise they may have the
> > same problem too.
> >
> > On May 21, 2017 03:25, "Abhimanyu Nagrath" <ab...@gmail.com>
> > wrote:
> >
> >> Hi Kant,
> >>
> >> Thanks for the suggestion.
> >>
> >>
> >> Regards,
> >> Abhimanyu
> >>
> >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <ka...@gmail.com>
> wrote:
> >>
> >>> @Abhimanyu You can try setting offset.retention = 30 (log.retention).
> At
> >>> most, you will have a storage overhead of 5 million msgs per day * 30
> >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you
> have
> >>> a
> >>> TB of hard disk)
> >>>
> >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <ka...@gmail.com>
> wrote:
> >>>
> >>> > Looking at that ticket and reading the comments it looks like one of
> >>> the
> >>> > concern is as follows.
> >>> >
> >>> > "offsets.retention.minutes is designed to handle the case that a
> >>> consumer
> >>> > group goes away forever. In that case, we don't want to store the
> >>> offsets
> >>> > for that group forever."
> >>> >
> >>> > This can simply be addressed by setting offset.retention ==
> >>> log.retention
> >>> > by default right? In which case offset wont be stored forever even
> when
> >>> > consumer group goes away forever. When the consumer group goes away
> >>> forever
> >>> > the upper bound to clean up offsets would be equal to log.retention.
> >>> >
> >>> >
> >>> >
> >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <ka...@gmail.com>
> >>> wrote:
> >>> >
> >>> >> What is your average message size and network speed?
> >>> >>
> >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> >>> >> abhimanyunagrath@gmail.com> wrote:
> >>> >>
> >>> >>> Hi Girish,
> >>> >>>
> >>> >>> I did not set any value for offsets.retention.minutes so therefore
> >>> what I
> >>> >>> think is picking its default value i.e 1440 minutes so what do you
> >>> think
> >>> >>> what should I set if I am keeping my data for 30 days?
> >>> >>>
> >>> >>> Regards,
> >>> >>> Abhimanyu
> >>> >>>
> >>> >>
> >>> >>
> >>> >
> >>>
> >>
> >>
>
Re: Messages are repeating in kafka
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
Hi Girish ,
Do I need to tune this configuration offsets.retention.check.interval.ms
also . Please let me know if I need to tune any other configuration.
Regards,
Abhimanyu
On Sun, May 21, 2017 at 8:01 PM, Girish Aher <gi...@gmail.com> wrote:
> Yup, exactly as Kant said.
> Also make sure that the retention of the offsets topic is an upper bound
> across all topics. So in this case, don't create any other topics in the
> future with retention of more than 30 days or otherwise they may have the
> same problem too.
>
> On May 21, 2017 03:25, "Abhimanyu Nagrath" <ab...@gmail.com>
> wrote:
>
>> Hi Kant,
>>
>> Thanks for the suggestion.
>>
>>
>> Regards,
>> Abhimanyu
>>
>> On Sun, May 21, 2017 at 3:44 PM, kant kodali <ka...@gmail.com> wrote:
>>
>>> @Abhimanyu You can try setting offset.retention = 30 (log.retention). At
>>> most, you will have a storage overhead of 5 million msgs per day * 30
>>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you have
>>> a
>>> TB of hard disk)
>>>
>>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <ka...@gmail.com> wrote:
>>>
>>> > Looking at that ticket and reading the comments it looks like one of
>>> the
>>> > concern is as follows.
>>> >
>>> > "offsets.retention.minutes is designed to handle the case that a
>>> consumer
>>> > group goes away forever. In that case, we don't want to store the
>>> offsets
>>> > for that group forever."
>>> >
>>> > This can simply be addressed by setting offset.retention ==
>>> log.retention
>>> > by default right? In which case offset wont be stored forever even when
>>> > consumer group goes away forever. When the consumer group goes away
>>> forever
>>> > the upper bound to clean up offsets would be equal to log.retention.
>>> >
>>> >
>>> >
>>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <ka...@gmail.com>
>>> wrote:
>>> >
>>> >> What is your average message size and network speed?
>>> >>
>>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
>>> >> abhimanyunagrath@gmail.com> wrote:
>>> >>
>>> >>> Hi Girish,
>>> >>>
>>> >>> I did not set any value for offsets.retention.minutes so therefore
>>> what I
>>> >>> think is picking its default value i.e 1440 minutes so what do you
>>> think
>>> >>> what should I set if I am keeping my data for 30 days?
>>> >>>
>>> >>> Regards,
>>> >>> Abhimanyu
>>> >>>
>>> >>
>>> >>
>>> >
>>>
>>
>>
Re: Messages are repeating in kafka
Posted by Girish Aher <gi...@gmail.com>.
Yup, exactly as Kant said.
Also make sure that the retention of the offsets topic is an upper bound
across all topics. So in this case, don't create any other topics in the
future with retention of more than 30 days or otherwise they may have the
same problem too.
On May 21, 2017 03:25, "Abhimanyu Nagrath" <ab...@gmail.com>
wrote:
> Hi Kant,
>
> Thanks for the suggestion.
>
>
> Regards,
> Abhimanyu
>
> On Sun, May 21, 2017 at 3:44 PM, kant kodali <ka...@gmail.com> wrote:
>
>> @Abhimanyu You can try setting offset.retention = 30 (log.retention). At
>> most, you will have a storage overhead of 5 million msgs per day * 30
>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you have a
>> TB of hard disk)
>>
>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <ka...@gmail.com> wrote:
>>
>> > Looking at that ticket and reading the comments it looks like one of the
>> > concern is as follows.
>> >
>> > "offsets.retention.minutes is designed to handle the case that a
>> consumer
>> > group goes away forever. In that case, we don't want to store the
>> offsets
>> > for that group forever."
>> >
>> > This can simply be addressed by setting offset.retention ==
>> log.retention
>> > by default right? In which case offset wont be stored forever even when
>> > consumer group goes away forever. When the consumer group goes away
>> forever
>> > the upper bound to clean up offsets would be equal to log.retention.
>> >
>> >
>> >
>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <ka...@gmail.com>
>> wrote:
>> >
>> >> What is your average message size and network speed?
>> >>
>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
>> >> abhimanyunagrath@gmail.com> wrote:
>> >>
>> >>> Hi Girish,
>> >>>
>> >>> I did not set any value for offsets.retention.minutes so therefore
>> what I
>> >>> think is picking its default value i.e 1440 minutes so what do you
>> think
>> >>> what should I set if I am keeping my data for 30 days?
>> >>>
>> >>> Regards,
>> >>> Abhimanyu
>> >>>
>> >>
>> >>
>> >
>>
>
>
Re: Messages are repeating in kafka
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
Hi Kant,
Thanks for the suggestion.
Regards,
Abhimanyu
On Sun, May 21, 2017 at 3:44 PM, kant kodali <ka...@gmail.com> wrote:
> @Abhimanyu You can try setting offset.retention = 30 (log.retention). At
> most, you will have a storage overhead of 5 million msgs per day * 30
> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you have a
> TB of hard disk)
>
> On Sun, May 21, 2017 at 3:05 AM, kant kodali <ka...@gmail.com> wrote:
>
> > Looking at that ticket and reading the comments it looks like one of the
> > concern is as follows.
> >
> > "offsets.retention.minutes is designed to handle the case that a consumer
> > group goes away forever. In that case, we don't want to store the offsets
> > for that group forever."
> >
> > This can simply be addressed by setting offset.retention == log.retention
> > by default right? In which case offset wont be stored forever even when
> > consumer group goes away forever. When the consumer group goes away
> forever
> > the upper bound to clean up offsets would be equal to log.retention.
> >
> >
> >
> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <ka...@gmail.com> wrote:
> >
> >> What is your average message size and network speed?
> >>
> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> >> abhimanyunagrath@gmail.com> wrote:
> >>
> >>> Hi Girish,
> >>>
> >>> I did not set any value for offsets.retention.minutes so therefore
> what I
> >>> think is picking its default value i.e 1440 minutes so what do you
> think
> >>> what should I set if I am keeping my data for 30 days?
> >>>
> >>> Regards,
> >>> Abhimanyu
> >>>
> >>
> >>
> >
>
Re: Messages are repeating in kafka
Posted by kant kodali <ka...@gmail.com>.
@Abhimanyu You can try setting offset.retention = 30 (log.retention). At
most, you will have a storage overhead of 5 million msgs per day * 30
(days) * 8 bytes (for each offset) = 1.2GB (not that much since you have a
TB of hard disk)
On Sun, May 21, 2017 at 3:05 AM, kant kodali <ka...@gmail.com> wrote:
> Looking at that ticket and reading the comments it looks like one of the
> concern is as follows.
>
> "offsets.retention.minutes is designed to handle the case that a consumer
> group goes away forever. In that case, we don't want to store the offsets
> for that group forever."
>
> This can simply be addressed by setting offset.retention == log.retention
> by default right? In which case offset wont be stored forever even when
> consumer group goes away forever. When the consumer group goes away forever
> the upper bound to clean up offsets would be equal to log.retention.
>
>
>
> On Sun, May 21, 2017 at 2:19 AM, kant kodali <ka...@gmail.com> wrote:
>
>> What is your average message size and network speed?
>>
>> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
>> abhimanyunagrath@gmail.com> wrote:
>>
>>> Hi Girish,
>>>
>>> I did not set any value for offsets.retention.minutes so therefore what I
>>> think is picking its default value i.e 1440 minutes so what do you think
>>> what should I set if I am keeping my data for 30 days?
>>>
>>> Regards,
>>> Abhimanyu
>>>
>>
>>
>
Re: Messages are repeating in kafka
Posted by kant kodali <ka...@gmail.com>.
Looking at that ticket and reading the comments it looks like one of the
concern is as follows.
"offsets.retention.minutes is designed to handle the case that a consumer
group goes away forever. In that case, we don't want to store the offsets
for that group forever."
This can simply be addressed by setting offset.retention == log.retention
by default right? In which case offset wont be stored forever even when
consumer group goes away forever. When the consumer group goes away forever
the upper bound to clean up offsets would be equal to log.retention.
On Sun, May 21, 2017 at 2:19 AM, kant kodali <ka...@gmail.com> wrote:
> What is your average message size and network speed?
>
> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> abhimanyunagrath@gmail.com> wrote:
>
>> Hi Girish,
>>
>> I did not set any value for offsets.retention.minutes so therefore what I
>> think is picking its default value i.e 1440 minutes so what do you think
>> what should I set if I am keeping my data for 30 days?
>>
>> Regards,
>> Abhimanyu
>>
>
>
Re: Messages are repeating in kafka
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
Hi,
My average message size is less 30 Kb and I get 1 near about 5 Million
message per day
On Sun, May 21, 2017 at 2:49 PM, kant kodali <ka...@gmail.com> wrote:
> What is your average message size and network speed?
>
> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> abhimanyunagrath@gmail.com> wrote:
>
> > Hi Girish,
> >
> > I did not set any value for offsets.retention.minutes so therefore what I
> > think is picking its default value i.e 1440 minutes so what do you think
> > what should I set if I am keeping my data for 30 days?
> >
> > Regards,
> > Abhimanyu
> >
>
Re: Messages are repeating in kafka
Posted by kant kodali <ka...@gmail.com>.
What is your average message size and network speed?
On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
abhimanyunagrath@gmail.com> wrote:
> Hi Girish,
>
> I did not set any value for offsets.retention.minutes so therefore what I
> think is picking its default value i.e 1440 minutes so what do you think
> what should I set if I am keeping my data for 30 days?
>
> Regards,
> Abhimanyu
>
Re: Messages are repeating in kafka
Posted by Abhimanyu Nagrath <ab...@gmail.com>.
Hi Girish,
I did not set any value for offsets.retention.minutes so therefore what I
think is picking its default value i.e 1440 minutes so what do you think
what should I set if I am keeping my data for 30 days?
Regards,
Abhimanyu
Re: Messages are repeating in kafka
Posted by Girish Aher <gi...@gmail.com>.
Hi Abhimanyu,
What is your offsets.retention.minutes on brokers set to? It could have
happened that the offsets in __consumer_offsets topic got wiped out after
the configured time and so when the consumer restarted, it did not find
where to start with. If auto.offset.reset is set to earliest on consumers,
you will end up re-consuming your messages.
This is something on the same lines:
https://issues.apache.org/jira/browse/KAFKA-3806
HTH.
On Fri, May 19, 2017 at 12:14 AM, Abhimanyu Nagrath <
abhimanyunagrath@gmail.com> wrote:
> I am using single node Kafka broker (v 0.10.2) Hardware (8 core, 16 GB RAM,
> 1 TD HardDisk) and zookeeper (v 3.4.8). I have a topic with 200 partition
> in which messages contains the total of 3 Million messages. It took 5 days
> to completely process all the messages and as soon as message got processed
> i.e. Kafka-consumer-groups.sh showed 0 lag in all the partition of the
> topic I stopped the consumer .but after 6 hrs again it was showing the lag
> of 2 million message which I found that were duplicate messages. This thing
> is happening very frequently. My offsets are stored on Kafka broker itself.
> My server configuration is:
>
> broker.id=1
> delete.topic.enable=true
> #listeners=PLAINTEXT://:9092
> #advertised.listeners=PLAINTEXT://your.host.name:9092
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/kafka/data/logs
> num.partitions=1
> num.recovery.threads.per.data.dir=5
> log.flush.interval.messages=10000
> #log.flush.interval.ms=1000
> log.retention.hours=480
> log.retention.bytes=1073741824
> log.segment.bytes=1073741824
> log.retention.check.interval.ms=300000
> zookeeper.connect=<zkIp>:2181
> zookeeper.connection.timeout.ms=6000
>
> Is there in the configuration that I am missing?
> Any help is appreciated
>