You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by A A <ek...@hotmail.com> on 2014/02/10 10:21:46 UTC
Dropping messages ?
Hi
We have been using Kafka(0.8) for the past few months with the following setup
Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
Yesterday, while running Stress tests in one of the QA machines , we observed that a few messages which were produced within a couple of milliseconds of each other did not reach the Kafka consumer. ie There was no trace of that message at the consumer end.
We decided to check whether we had any errors at our side or there was a network issue. We did not find any issue. We then decided to check whether we can find that message in one of the Kafka partitions. The message was found in one of the topic partitions.
We are not sure why Kafka did not notify any consumers about the message. Are there any special cases where Kafka silently drops a message ?
We also found a delay in the notifications/watches triggered from zookeeper. We are not sure whether these are related ? It will be difficult to reproduce as the test probably took a few days to complete. But surely we did lose approximately 5% of the messages. We have logs of messages being produced at the producer side and corresponding entries in Kafka partitions logs. But nothing at the consumer side. The only repeating pattern was that the messages were probably produced within the same millisecond. So if you have a sequence of messages which was produced in the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably have M0,M1,M3 but not M2. This is puzzling as to how only message is dropped out of the given 4.
We use the High Level Kafka Producer and Consumer. Both are single threaded(at our end).
Does kafka need its own dedicated zookeeper ensemble ? We also use the same zookeeper ensemble as our configuration service.
Unfortunately, we did not have DEBUG messages at the server enabled during the setup. Although, NO error messages were observed during the same time period.
Before we try running the same Tests again, can someone please shed more light as to the reasons why kafka dropped a few messages ?
Kat
RE: Dropping messages ?
Posted by Kat Walker <ek...@hotmail.com>.
Hi Guozhang
The dropped messages were found in one of the topic partitions as indicated in my first email. It is clear that the messages were persisted in Kafka logs.
Can you answer my other questions regarding Kafka's consistency and durability.
Thanks
Kat
> Date: Tue, 11 Feb 2014 00:24:27 -0800
> Subject: Re: Dropping messages ?
> From: wangguoz@gmail.com
> To: users@kafka.apache.org
>
> Can you verify if the missing messages exist in the broker or not by ways
> Jun have said, you can just create another consumer group which does no
> processing but just count messages.
>
> Guozhang
>
>
>
> On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek...@hotmail.com> wrote:
>
> > Hi Jun/Guozhang
> >
> > We might have to retry our QA tests in its entirety. We simply cannot
> > reset consumer offset as there is a lot of processing involved after
> > consuming those messages. This might take almost a week. The Kafka message
> > also contains `volatile` data which is fetched from a database and
> > destroyed after consuming that Kafka message.
> >
> > What is puzzling is that we have been running Kafka in Production for over
> > 4 months now and we have never faced this issue. Our peak volume is <= 1000
> > messages / second. On an average, it is less than 100.
> >
> > Our zookeeper version is 3.3.6. What I suspect is that we managed to `roll
> > over` a few messages due to inconsistencies in zookeeper offsets. Recall
> > that we only had only 1 Kafka broker. Once again, this is just a `wild`
> > speculation.
> >
> > The topics were created before we started our tests.
> >
> > Has someone ever lost a Kafka message before ? Are there any guarantees ?
> > We are okay with all kinds of delays provided, the messages arrive in order
> > and are delivered regardless.
> >
> > Thanks
> > Kat
> >
> > > Date: Mon, 10 Feb 2014 07:38:01 -0800
> > > Subject: Re: Dropping messages ?
> > > From: junrao@gmail.com
> > > To: users@kafka.apache.org
> > >
> > > If you reset the consumer offset and try to consume those messages again,
> > > do you see the same drop?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek...@hotmail.com> wrote:
> > >
> > > > Hi
> > > >
> > > > We have been using Kafka(0.8) for the past few months with the
> > following
> > > > setup
> > > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> > > >
> > > > Yesterday, while running Stress tests in one of the QA machines , we
> > > > observed that a few messages which were produced within a couple of
> > > > milliseconds of each other did not reach the Kafka consumer. ie There
> > was
> > > > no trace of that message at the consumer end.
> > > >
> > > > We decided to check whether we had any errors at our side or there was
> > a
> > > > network issue. We did not find any issue. We then decided to check
> > whether
> > > > we can find that message in one of the Kafka partitions. The message
> > was
> > > > found in one of the topic partitions.
> > > >
> > > > We are not sure why Kafka did not notify any consumers about the
> > message.
> > > > Are there any special cases where Kafka silently drops a message ?
> > > >
> > > > We also found a delay in the notifications/watches triggered from
> > > > zookeeper. We are not sure whether these are related ? It will be
> > difficult
> > > > to reproduce as the test probably took a few days to complete. But
> > surely
> > > > we did lose approximately 5% of the messages. We have logs of messages
> > > > being produced at the producer side and corresponding entries in Kafka
> > > > partitions logs. But nothing at the consumer side. The only repeating
> > > > pattern was that the messages were probably produced within the same
> > > > millisecond. So if you have a sequence of messages which was produced
> > in
> > > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably
> > have
> > > > M0,M1,M3 but not M2. This is puzzling as to how only message is
> > dropped out
> > > > of the given 4.
> > > >
> > > >
> > > > We use the High Level Kafka Producer and Consumer. Both are single
> > > > threaded(at our end).
> > > >
> > > > Does kafka need its own dedicated zookeeper ensemble ? We also use the
> > > > same zookeeper ensemble as our configuration service.
> > > >
> > > > Unfortunately, we did not have DEBUG messages at the server enabled
> > during
> > > > the setup. Although, NO error messages were observed during the same
> > time
> > > > period.
> > > >
> > > >
> > > > Before we try running the same Tests again, can someone please shed
> > more
> > > > light as to the reasons why kafka dropped a few messages ?
> > > >
> > > > Kat
> > > >
> >
> >
>
>
>
> --
> -- Guozhang
Re: Dropping messages ?
Posted by Guozhang Wang <wa...@gmail.com>.
Can you verify if the missing messages exist in the broker or not by ways
Jun have said, you can just create another consumer group which does no
processing but just count messages.
Guozhang
On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek...@hotmail.com> wrote:
> Hi Jun/Guozhang
>
> We might have to retry our QA tests in its entirety. We simply cannot
> reset consumer offset as there is a lot of processing involved after
> consuming those messages. This might take almost a week. The Kafka message
> also contains `volatile` data which is fetched from a database and
> destroyed after consuming that Kafka message.
>
> What is puzzling is that we have been running Kafka in Production for over
> 4 months now and we have never faced this issue. Our peak volume is <= 1000
> messages / second. On an average, it is less than 100.
>
> Our zookeeper version is 3.3.6. What I suspect is that we managed to `roll
> over` a few messages due to inconsistencies in zookeeper offsets. Recall
> that we only had only 1 Kafka broker. Once again, this is just a `wild`
> speculation.
>
> The topics were created before we started our tests.
>
> Has someone ever lost a Kafka message before ? Are there any guarantees ?
> We are okay with all kinds of delays provided, the messages arrive in order
> and are delivered regardless.
>
> Thanks
> Kat
>
> > Date: Mon, 10 Feb 2014 07:38:01 -0800
> > Subject: Re: Dropping messages ?
> > From: junrao@gmail.com
> > To: users@kafka.apache.org
> >
> > If you reset the consumer offset and try to consume those messages again,
> > do you see the same drop?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek...@hotmail.com> wrote:
> >
> > > Hi
> > >
> > > We have been using Kafka(0.8) for the past few months with the
> following
> > > setup
> > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> > >
> > > Yesterday, while running Stress tests in one of the QA machines , we
> > > observed that a few messages which were produced within a couple of
> > > milliseconds of each other did not reach the Kafka consumer. ie There
> was
> > > no trace of that message at the consumer end.
> > >
> > > We decided to check whether we had any errors at our side or there was
> a
> > > network issue. We did not find any issue. We then decided to check
> whether
> > > we can find that message in one of the Kafka partitions. The message
> was
> > > found in one of the topic partitions.
> > >
> > > We are not sure why Kafka did not notify any consumers about the
> message.
> > > Are there any special cases where Kafka silently drops a message ?
> > >
> > > We also found a delay in the notifications/watches triggered from
> > > zookeeper. We are not sure whether these are related ? It will be
> difficult
> > > to reproduce as the test probably took a few days to complete. But
> surely
> > > we did lose approximately 5% of the messages. We have logs of messages
> > > being produced at the producer side and corresponding entries in Kafka
> > > partitions logs. But nothing at the consumer side. The only repeating
> > > pattern was that the messages were probably produced within the same
> > > millisecond. So if you have a sequence of messages which was produced
> in
> > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably
> have
> > > M0,M1,M3 but not M2. This is puzzling as to how only message is
> dropped out
> > > of the given 4.
> > >
> > >
> > > We use the High Level Kafka Producer and Consumer. Both are single
> > > threaded(at our end).
> > >
> > > Does kafka need its own dedicated zookeeper ensemble ? We also use the
> > > same zookeeper ensemble as our configuration service.
> > >
> > > Unfortunately, we did not have DEBUG messages at the server enabled
> during
> > > the setup. Although, NO error messages were observed during the same
> time
> > > period.
> > >
> > >
> > > Before we try running the same Tests again, can someone please shed
> more
> > > light as to the reasons why kafka dropped a few messages ?
> > >
> > > Kat
> > >
>
>
--
-- Guozhang
Re: Dropping messages ?
Posted by Jun Rao <ju...@gmail.com>.
Kat,
The answer to all your questions are similar. Basically, either we will
discover there is a real bug in Kafka or there is an issue in the
application (mis-config, network issue, etc). So far, given the info you
provided, it's a bit hard to draw any conclusion. For issues like this, I
will typically do the following: (1) see if the problem is reproducible,
(2) if so, see if the problem can be reproduced easier, potentially in a
different testing environment. All my previous questions are trying to lead
us to a easily reproducible problem. If other people can reproduce the
problem that you encountered, finding out the cause will be easier.
Thanks,
Jun
On Thu, Feb 13, 2014 at 2:17 AM, Kat Walker <ek...@hotmail.com> wrote:
> We almost have a week before we get back the results. Meanwhile, I would
> like to ask a few questions or reiterate those questions again.
>
> 1) What if we find the same messages again this time ? ie they were
> successfully consumed ? What should we conclude ?
>
> 2) What if we don't find the same messages again ? ie. they were not
> consumed. What do we conclude ?
>
> 3) What if we find new `dropped` messages ?
>
> A definite clear answer will be appreciated.
>
> Thanks
> Kat
>
> > Date: Tue, 11 Feb 2014 22:08:00 -0800
> > Subject: Re: Dropping messages ?
> > From: junrao@gmail.com
> > To: users@kafka.apache.org
> >
> > I haven't seen an issue like this before: a consumer doesn't see some
> > messages existing in a broker. So, some more debugging is needed. If you
> > start a new consumer consuming from the beginning, do you see those
> missing
> > messages?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 11, 2014 at 9:43 PM, Kat Walker <ek...@hotmail.com>
> wrote:
> >
> > > > Are you committing offsets manually?
> > >
> > > We are not committing offsets manually. As indicated in my first
> email, we
> > > are using the high level consumer API.
> > >
> > > >How do you realize that some messages
> > > >are lost? Do you log every message returned to Kafka consumer client?
> > >
> > > Yes we log every message received and sent. We can track every
> message. We
> > > have a unique identifier for every message.
> > >
> > > >Is it
> > > > possible that a message is returned to the consumer, but is lost in
> the
> > > > application logic?
> > >
> > > No. That is not possible. As soon as we receive a message, we print our
> > > unique identifier. And most of the contents of the message which is
> about
> > > to be processed. Recall that we have only 1 broker and a zookeeper
> ensemble
> > > consisting of 3 zookeeper servers. We did not observe any issue during
> the
> > > testing time period.
> > >
> > > It will be nice if you answer my previous questions about Kafka's
> message
> > > guarantees. As of now, it seems you are asking questions which I have
> > > already answered in my previous emails. Please refer the entire email
> > > thread.
> > >
> > >
> > > > > Has someone ever lost a Kafka message before ? Are there any
> > > guarantees ?
> > > > > We are okay with all kinds of delays provided, the messages arrive
> in
> > > order
> > > > > and are delivered regardless.
> > >
> > >
> > > Thanks
> > > Kat
> > >
> > > > Date: Tue, 11 Feb 2014 08:02:16 -0800
> > > > Subject: Re: Dropping messages ?
> > > > From: junrao@gmail.com
> > > > To: users@kafka.apache.org
> > > >
> > > > Are you committing offsets manually? How do you realize that some
> > > messages
> > > > are lost? Do you log every message returned to Kafka consumer
> client? Is
> > > it
> > > > possible that a message is returned to the consumer, but is lost in
> the
> > > > application logic?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek.tha.cat@hotmail.com
> >
> > > wrote:
> > > >
> > > > > Hi Jun/Guozhang
> > > > >
> > > > > We might have to retry our QA tests in its entirety. We simply
> cannot
> > > > > reset consumer offset as there is a lot of processing involved
> after
> > > > > consuming those messages. This might take almost a week. The Kafka
> > > message
> > > > > also contains `volatile` data which is fetched from a database and
> > > > > destroyed after consuming that Kafka message.
> > > > >
> > > > > What is puzzling is that we have been running Kafka in Production
> for
> > > over
> > > > > 4 months now and we have never faced this issue. Our peak volume
> is <=
> > > 1000
> > > > > messages / second. On an average, it is less than 100.
> > > > >
> > > > > Our zookeeper version is 3.3.6. What I suspect is that we managed
> to
> > > `roll
> > > > > over` a few messages due to inconsistencies in zookeeper offsets.
> > > Recall
> > > > > that we only had only 1 Kafka broker. Once again, this is just a
> `wild`
> > > > > speculation.
> > > > >
> > > > > The topics were created before we started our tests.
> > > > >
> > > > > Has someone ever lost a Kafka message before ? Are there any
> > > guarantees ?
> > > > > We are okay with all kinds of delays provided, the messages arrive
> in
> > > order
> > > > > and are delivered regardless.
> > > > >
> > > > > Thanks
> > > > > Kat
> > > > >
> > > > > > Date: Mon, 10 Feb 2014 07:38:01 -0800
> > > > > > Subject: Re: Dropping messages ?
> > > > > > From: junrao@gmail.com
> > > > > > To: users@kafka.apache.org
> > > > > >
> > > > > > If you reset the consumer offset and try to consume those
> messages
> > > again,
> > > > > > do you see the same drop?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek...@hotmail.com>
> wrote:
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > We have been using Kafka(0.8) for the past few months with the
> > > > > following
> > > > > > > setup
> > > > > > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> > > > > > >
> > > > > > > Yesterday, while running Stress tests in one of the QA
> machines ,
> > > we
> > > > > > > observed that a few messages which were produced within a
> couple of
> > > > > > > milliseconds of each other did not reach the Kafka consumer. ie
> > > There
> > > > > was
> > > > > > > no trace of that message at the consumer end.
> > > > > > >
> > > > > > > We decided to check whether we had any errors at our side or
> there
> > > was
> > > > > a
> > > > > > > network issue. We did not find any issue. We then decided to
> check
> > > > > whether
> > > > > > > we can find that message in one of the Kafka partitions. The
> > > message
> > > > > was
> > > > > > > found in one of the topic partitions.
> > > > > > >
> > > > > > > We are not sure why Kafka did not notify any consumers about
> the
> > > > > message.
> > > > > > > Are there any special cases where Kafka silently drops a
> message ?
> > > > > > >
> > > > > > > We also found a delay in the notifications/watches triggered
> from
> > > > > > > zookeeper. We are not sure whether these are related ? It will
> be
> > > > > difficult
> > > > > > > to reproduce as the test probably took a few days to complete.
> But
> > > > > surely
> > > > > > > we did lose approximately 5% of the messages. We have logs of
> > > messages
> > > > > > > being produced at the producer side and corresponding entries
> in
> > > Kafka
> > > > > > > partitions logs. But nothing at the consumer side. The only
> > > repeating
> > > > > > > pattern was that the messages were probably produced within the
> > > same
> > > > > > > millisecond. So if you have a sequence of messages which was
> > > produced
> > > > > in
> > > > > > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We
> probably
> > > > > have
> > > > > > > M0,M1,M3 but not M2. This is puzzling as to how only message is
> > > > > dropped out
> > > > > > > of the given 4.
> > > > > > >
> > > > > > >
> > > > > > > We use the High Level Kafka Producer and Consumer. Both are
> single
> > > > > > > threaded(at our end).
> > > > > > >
> > > > > > > Does kafka need its own dedicated zookeeper ensemble ? We also
> use
> > > the
> > > > > > > same zookeeper ensemble as our configuration service.
> > > > > > >
> > > > > > > Unfortunately, we did not have DEBUG messages at the server
> enabled
> > > > > during
> > > > > > > the setup. Although, NO error messages were observed during the
> > > same
> > > > > time
> > > > > > > period.
> > > > > > >
> > > > > > >
> > > > > > > Before we try running the same Tests again, can someone please
> shed
> > > > > more
> > > > > > > light as to the reasons why kafka dropped a few messages ?
> > > > > > >
> > > > > > > Kat
> > > > > > >
> > > > >
> > > > >
> > >
> > >
>
>
RE: Dropping messages ?
Posted by Kat Walker <ek...@hotmail.com>.
We almost have a week before we get back the results. Meanwhile, I would like to ask a few questions or reiterate those questions again.
1) What if we find the same messages again this time ? ie they were successfully consumed ? What should we conclude ?
2) What if we don't find the same messages again ? ie. they were not consumed. What do we conclude ?
3) What if we find new `dropped` messages ?
A definite clear answer will be appreciated.
Thanks
Kat
> Date: Tue, 11 Feb 2014 22:08:00 -0800
> Subject: Re: Dropping messages ?
> From: junrao@gmail.com
> To: users@kafka.apache.org
>
> I haven't seen an issue like this before: a consumer doesn't see some
> messages existing in a broker. So, some more debugging is needed. If you
> start a new consumer consuming from the beginning, do you see those missing
> messages?
>
> Thanks,
>
> Jun
>
>
> On Tue, Feb 11, 2014 at 9:43 PM, Kat Walker <ek...@hotmail.com> wrote:
>
> > > Are you committing offsets manually?
> >
> > We are not committing offsets manually. As indicated in my first email, we
> > are using the high level consumer API.
> >
> > >How do you realize that some messages
> > >are lost? Do you log every message returned to Kafka consumer client?
> >
> > Yes we log every message received and sent. We can track every message. We
> > have a unique identifier for every message.
> >
> > >Is it
> > > possible that a message is returned to the consumer, but is lost in the
> > > application logic?
> >
> > No. That is not possible. As soon as we receive a message, we print our
> > unique identifier. And most of the contents of the message which is about
> > to be processed. Recall that we have only 1 broker and a zookeeper ensemble
> > consisting of 3 zookeeper servers. We did not observe any issue during the
> > testing time period.
> >
> > It will be nice if you answer my previous questions about Kafka's message
> > guarantees. As of now, it seems you are asking questions which I have
> > already answered in my previous emails. Please refer the entire email
> > thread.
> >
> >
> > > > Has someone ever lost a Kafka message before ? Are there any
> > guarantees ?
> > > > We are okay with all kinds of delays provided, the messages arrive in
> > order
> > > > and are delivered regardless.
> >
> >
> > Thanks
> > Kat
> >
> > > Date: Tue, 11 Feb 2014 08:02:16 -0800
> > > Subject: Re: Dropping messages ?
> > > From: junrao@gmail.com
> > > To: users@kafka.apache.org
> > >
> > > Are you committing offsets manually? How do you realize that some
> > messages
> > > are lost? Do you log every message returned to Kafka consumer client? Is
> > it
> > > possible that a message is returned to the consumer, but is lost in the
> > > application logic?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek...@hotmail.com>
> > wrote:
> > >
> > > > Hi Jun/Guozhang
> > > >
> > > > We might have to retry our QA tests in its entirety. We simply cannot
> > > > reset consumer offset as there is a lot of processing involved after
> > > > consuming those messages. This might take almost a week. The Kafka
> > message
> > > > also contains `volatile` data which is fetched from a database and
> > > > destroyed after consuming that Kafka message.
> > > >
> > > > What is puzzling is that we have been running Kafka in Production for
> > over
> > > > 4 months now and we have never faced this issue. Our peak volume is <=
> > 1000
> > > > messages / second. On an average, it is less than 100.
> > > >
> > > > Our zookeeper version is 3.3.6. What I suspect is that we managed to
> > `roll
> > > > over` a few messages due to inconsistencies in zookeeper offsets.
> > Recall
> > > > that we only had only 1 Kafka broker. Once again, this is just a `wild`
> > > > speculation.
> > > >
> > > > The topics were created before we started our tests.
> > > >
> > > > Has someone ever lost a Kafka message before ? Are there any
> > guarantees ?
> > > > We are okay with all kinds of delays provided, the messages arrive in
> > order
> > > > and are delivered regardless.
> > > >
> > > > Thanks
> > > > Kat
> > > >
> > > > > Date: Mon, 10 Feb 2014 07:38:01 -0800
> > > > > Subject: Re: Dropping messages ?
> > > > > From: junrao@gmail.com
> > > > > To: users@kafka.apache.org
> > > > >
> > > > > If you reset the consumer offset and try to consume those messages
> > again,
> > > > > do you see the same drop?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek...@hotmail.com> wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > We have been using Kafka(0.8) for the past few months with the
> > > > following
> > > > > > setup
> > > > > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> > > > > >
> > > > > > Yesterday, while running Stress tests in one of the QA machines ,
> > we
> > > > > > observed that a few messages which were produced within a couple of
> > > > > > milliseconds of each other did not reach the Kafka consumer. ie
> > There
> > > > was
> > > > > > no trace of that message at the consumer end.
> > > > > >
> > > > > > We decided to check whether we had any errors at our side or there
> > was
> > > > a
> > > > > > network issue. We did not find any issue. We then decided to check
> > > > whether
> > > > > > we can find that message in one of the Kafka partitions. The
> > message
> > > > was
> > > > > > found in one of the topic partitions.
> > > > > >
> > > > > > We are not sure why Kafka did not notify any consumers about the
> > > > message.
> > > > > > Are there any special cases where Kafka silently drops a message ?
> > > > > >
> > > > > > We also found a delay in the notifications/watches triggered from
> > > > > > zookeeper. We are not sure whether these are related ? It will be
> > > > difficult
> > > > > > to reproduce as the test probably took a few days to complete. But
> > > > surely
> > > > > > we did lose approximately 5% of the messages. We have logs of
> > messages
> > > > > > being produced at the producer side and corresponding entries in
> > Kafka
> > > > > > partitions logs. But nothing at the consumer side. The only
> > repeating
> > > > > > pattern was that the messages were probably produced within the
> > same
> > > > > > millisecond. So if you have a sequence of messages which was
> > produced
> > > > in
> > > > > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably
> > > > have
> > > > > > M0,M1,M3 but not M2. This is puzzling as to how only message is
> > > > dropped out
> > > > > > of the given 4.
> > > > > >
> > > > > >
> > > > > > We use the High Level Kafka Producer and Consumer. Both are single
> > > > > > threaded(at our end).
> > > > > >
> > > > > > Does kafka need its own dedicated zookeeper ensemble ? We also use
> > the
> > > > > > same zookeeper ensemble as our configuration service.
> > > > > >
> > > > > > Unfortunately, we did not have DEBUG messages at the server enabled
> > > > during
> > > > > > the setup. Although, NO error messages were observed during the
> > same
> > > > time
> > > > > > period.
> > > > > >
> > > > > >
> > > > > > Before we try running the same Tests again, can someone please shed
> > > > more
> > > > > > light as to the reasons why kafka dropped a few messages ?
> > > > > >
> > > > > > Kat
> > > > > >
> > > >
> > > >
> >
> >
Re: Dropping messages ?
Posted by Jun Rao <ju...@gmail.com>.
I haven't seen an issue like this before: a consumer doesn't see some
messages existing in a broker. So, some more debugging is needed. If you
start a new consumer consuming from the beginning, do you see those missing
messages?
Thanks,
Jun
On Tue, Feb 11, 2014 at 9:43 PM, Kat Walker <ek...@hotmail.com> wrote:
> > Are you committing offsets manually?
>
> We are not committing offsets manually. As indicated in my first email, we
> are using the high level consumer API.
>
> >How do you realize that some messages
> >are lost? Do you log every message returned to Kafka consumer client?
>
> Yes we log every message received and sent. We can track every message. We
> have a unique identifier for every message.
>
> >Is it
> > possible that a message is returned to the consumer, but is lost in the
> > application logic?
>
> No. That is not possible. As soon as we receive a message, we print our
> unique identifier. And most of the contents of the message which is about
> to be processed. Recall that we have only 1 broker and a zookeeper ensemble
> consisting of 3 zookeeper servers. We did not observe any issue during the
> testing time period.
>
> It will be nice if you answer my previous questions about Kafka's message
> guarantees. As of now, it seems you are asking questions which I have
> already answered in my previous emails. Please refer the entire email
> thread.
>
>
> > > Has someone ever lost a Kafka message before ? Are there any
> guarantees ?
> > > We are okay with all kinds of delays provided, the messages arrive in
> order
> > > and are delivered regardless.
>
>
> Thanks
> Kat
>
> > Date: Tue, 11 Feb 2014 08:02:16 -0800
> > Subject: Re: Dropping messages ?
> > From: junrao@gmail.com
> > To: users@kafka.apache.org
> >
> > Are you committing offsets manually? How do you realize that some
> messages
> > are lost? Do you log every message returned to Kafka consumer client? Is
> it
> > possible that a message is returned to the consumer, but is lost in the
> > application logic?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek...@hotmail.com>
> wrote:
> >
> > > Hi Jun/Guozhang
> > >
> > > We might have to retry our QA tests in its entirety. We simply cannot
> > > reset consumer offset as there is a lot of processing involved after
> > > consuming those messages. This might take almost a week. The Kafka
> message
> > > also contains `volatile` data which is fetched from a database and
> > > destroyed after consuming that Kafka message.
> > >
> > > What is puzzling is that we have been running Kafka in Production for
> over
> > > 4 months now and we have never faced this issue. Our peak volume is <=
> 1000
> > > messages / second. On an average, it is less than 100.
> > >
> > > Our zookeeper version is 3.3.6. What I suspect is that we managed to
> `roll
> > > over` a few messages due to inconsistencies in zookeeper offsets.
> Recall
> > > that we only had only 1 Kafka broker. Once again, this is just a `wild`
> > > speculation.
> > >
> > > The topics were created before we started our tests.
> > >
> > > Has someone ever lost a Kafka message before ? Are there any
> guarantees ?
> > > We are okay with all kinds of delays provided, the messages arrive in
> order
> > > and are delivered regardless.
> > >
> > > Thanks
> > > Kat
> > >
> > > > Date: Mon, 10 Feb 2014 07:38:01 -0800
> > > > Subject: Re: Dropping messages ?
> > > > From: junrao@gmail.com
> > > > To: users@kafka.apache.org
> > > >
> > > > If you reset the consumer offset and try to consume those messages
> again,
> > > > do you see the same drop?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek...@hotmail.com> wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > We have been using Kafka(0.8) for the past few months with the
> > > following
> > > > > setup
> > > > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> > > > >
> > > > > Yesterday, while running Stress tests in one of the QA machines ,
> we
> > > > > observed that a few messages which were produced within a couple of
> > > > > milliseconds of each other did not reach the Kafka consumer. ie
> There
> > > was
> > > > > no trace of that message at the consumer end.
> > > > >
> > > > > We decided to check whether we had any errors at our side or there
> was
> > > a
> > > > > network issue. We did not find any issue. We then decided to check
> > > whether
> > > > > we can find that message in one of the Kafka partitions. The
> message
> > > was
> > > > > found in one of the topic partitions.
> > > > >
> > > > > We are not sure why Kafka did not notify any consumers about the
> > > message.
> > > > > Are there any special cases where Kafka silently drops a message ?
> > > > >
> > > > > We also found a delay in the notifications/watches triggered from
> > > > > zookeeper. We are not sure whether these are related ? It will be
> > > difficult
> > > > > to reproduce as the test probably took a few days to complete. But
> > > surely
> > > > > we did lose approximately 5% of the messages. We have logs of
> messages
> > > > > being produced at the producer side and corresponding entries in
> Kafka
> > > > > partitions logs. But nothing at the consumer side. The only
> repeating
> > > > > pattern was that the messages were probably produced within the
> same
> > > > > millisecond. So if you have a sequence of messages which was
> produced
> > > in
> > > > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably
> > > have
> > > > > M0,M1,M3 but not M2. This is puzzling as to how only message is
> > > dropped out
> > > > > of the given 4.
> > > > >
> > > > >
> > > > > We use the High Level Kafka Producer and Consumer. Both are single
> > > > > threaded(at our end).
> > > > >
> > > > > Does kafka need its own dedicated zookeeper ensemble ? We also use
> the
> > > > > same zookeeper ensemble as our configuration service.
> > > > >
> > > > > Unfortunately, we did not have DEBUG messages at the server enabled
> > > during
> > > > > the setup. Although, NO error messages were observed during the
> same
> > > time
> > > > > period.
> > > > >
> > > > >
> > > > > Before we try running the same Tests again, can someone please shed
> > > more
> > > > > light as to the reasons why kafka dropped a few messages ?
> > > > >
> > > > > Kat
> > > > >
> > >
> > >
>
>
RE: Dropping messages ?
Posted by Kat Walker <ek...@hotmail.com>.
> Are you committing offsets manually?
We are not committing offsets manually. As indicated in my first email, we are using the high level consumer API.
>How do you realize that some messages
>are lost? Do you log every message returned to Kafka consumer client?
Yes we log every message received and sent. We can track every message. We have a unique identifier for every message.
>Is it
> possible that a message is returned to the consumer, but is lost in the
> application logic?
No. That is not possible. As soon as we receive a message, we print our unique identifier. And most of the contents of the message which is about to be processed. Recall that we have only 1 broker and a zookeeper ensemble consisting of 3 zookeeper servers. We did not observe any issue during the testing time period.
It will be nice if you answer my previous questions about Kafka's message guarantees. As of now, it seems you are asking questions which I have already answered in my previous emails. Please refer the entire email thread.
> > Has someone ever lost a Kafka message before ? Are there any guarantees ?
> > We are okay with all kinds of delays provided, the messages arrive in order
> > and are delivered regardless.
Thanks
Kat
> Date: Tue, 11 Feb 2014 08:02:16 -0800
> Subject: Re: Dropping messages ?
> From: junrao@gmail.com
> To: users@kafka.apache.org
>
> Are you committing offsets manually? How do you realize that some messages
> are lost? Do you log every message returned to Kafka consumer client? Is it
> possible that a message is returned to the consumer, but is lost in the
> application logic?
>
> Thanks,
>
> Jun
>
>
> On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek...@hotmail.com> wrote:
>
> > Hi Jun/Guozhang
> >
> > We might have to retry our QA tests in its entirety. We simply cannot
> > reset consumer offset as there is a lot of processing involved after
> > consuming those messages. This might take almost a week. The Kafka message
> > also contains `volatile` data which is fetched from a database and
> > destroyed after consuming that Kafka message.
> >
> > What is puzzling is that we have been running Kafka in Production for over
> > 4 months now and we have never faced this issue. Our peak volume is <= 1000
> > messages / second. On an average, it is less than 100.
> >
> > Our zookeeper version is 3.3.6. What I suspect is that we managed to `roll
> > over` a few messages due to inconsistencies in zookeeper offsets. Recall
> > that we only had only 1 Kafka broker. Once again, this is just a `wild`
> > speculation.
> >
> > The topics were created before we started our tests.
> >
> > Has someone ever lost a Kafka message before ? Are there any guarantees ?
> > We are okay with all kinds of delays provided, the messages arrive in order
> > and are delivered regardless.
> >
> > Thanks
> > Kat
> >
> > > Date: Mon, 10 Feb 2014 07:38:01 -0800
> > > Subject: Re: Dropping messages ?
> > > From: junrao@gmail.com
> > > To: users@kafka.apache.org
> > >
> > > If you reset the consumer offset and try to consume those messages again,
> > > do you see the same drop?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek...@hotmail.com> wrote:
> > >
> > > > Hi
> > > >
> > > > We have been using Kafka(0.8) for the past few months with the
> > following
> > > > setup
> > > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> > > >
> > > > Yesterday, while running Stress tests in one of the QA machines , we
> > > > observed that a few messages which were produced within a couple of
> > > > milliseconds of each other did not reach the Kafka consumer. ie There
> > was
> > > > no trace of that message at the consumer end.
> > > >
> > > > We decided to check whether we had any errors at our side or there was
> > a
> > > > network issue. We did not find any issue. We then decided to check
> > whether
> > > > we can find that message in one of the Kafka partitions. The message
> > was
> > > > found in one of the topic partitions.
> > > >
> > > > We are not sure why Kafka did not notify any consumers about the
> > message.
> > > > Are there any special cases where Kafka silently drops a message ?
> > > >
> > > > We also found a delay in the notifications/watches triggered from
> > > > zookeeper. We are not sure whether these are related ? It will be
> > difficult
> > > > to reproduce as the test probably took a few days to complete. But
> > surely
> > > > we did lose approximately 5% of the messages. We have logs of messages
> > > > being produced at the producer side and corresponding entries in Kafka
> > > > partitions logs. But nothing at the consumer side. The only repeating
> > > > pattern was that the messages were probably produced within the same
> > > > millisecond. So if you have a sequence of messages which was produced
> > in
> > > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably
> > have
> > > > M0,M1,M3 but not M2. This is puzzling as to how only message is
> > dropped out
> > > > of the given 4.
> > > >
> > > >
> > > > We use the High Level Kafka Producer and Consumer. Both are single
> > > > threaded(at our end).
> > > >
> > > > Does kafka need its own dedicated zookeeper ensemble ? We also use the
> > > > same zookeeper ensemble as our configuration service.
> > > >
> > > > Unfortunately, we did not have DEBUG messages at the server enabled
> > during
> > > > the setup. Although, NO error messages were observed during the same
> > time
> > > > period.
> > > >
> > > >
> > > > Before we try running the same Tests again, can someone please shed
> > more
> > > > light as to the reasons why kafka dropped a few messages ?
> > > >
> > > > Kat
> > > >
> >
> >
Re: Dropping messages ?
Posted by Jun Rao <ju...@gmail.com>.
Are you committing offsets manually? How do you realize that some messages
are lost? Do you log every message returned to Kafka consumer client? Is it
possible that a message is returned to the consumer, but is lost in the
application logic?
Thanks,
Jun
On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek...@hotmail.com> wrote:
> Hi Jun/Guozhang
>
> We might have to retry our QA tests in its entirety. We simply cannot
> reset consumer offset as there is a lot of processing involved after
> consuming those messages. This might take almost a week. The Kafka message
> also contains `volatile` data which is fetched from a database and
> destroyed after consuming that Kafka message.
>
> What is puzzling is that we have been running Kafka in Production for over
> 4 months now and we have never faced this issue. Our peak volume is <= 1000
> messages / second. On an average, it is less than 100.
>
> Our zookeeper version is 3.3.6. What I suspect is that we managed to `roll
> over` a few messages due to inconsistencies in zookeeper offsets. Recall
> that we only had only 1 Kafka broker. Once again, this is just a `wild`
> speculation.
>
> The topics were created before we started our tests.
>
> Has someone ever lost a Kafka message before ? Are there any guarantees ?
> We are okay with all kinds of delays provided, the messages arrive in order
> and are delivered regardless.
>
> Thanks
> Kat
>
> > Date: Mon, 10 Feb 2014 07:38:01 -0800
> > Subject: Re: Dropping messages ?
> > From: junrao@gmail.com
> > To: users@kafka.apache.org
> >
> > If you reset the consumer offset and try to consume those messages again,
> > do you see the same drop?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek...@hotmail.com> wrote:
> >
> > > Hi
> > >
> > > We have been using Kafka(0.8) for the past few months with the
> following
> > > setup
> > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> > >
> > > Yesterday, while running Stress tests in one of the QA machines , we
> > > observed that a few messages which were produced within a couple of
> > > milliseconds of each other did not reach the Kafka consumer. ie There
> was
> > > no trace of that message at the consumer end.
> > >
> > > We decided to check whether we had any errors at our side or there was
> a
> > > network issue. We did not find any issue. We then decided to check
> whether
> > > we can find that message in one of the Kafka partitions. The message
> was
> > > found in one of the topic partitions.
> > >
> > > We are not sure why Kafka did not notify any consumers about the
> message.
> > > Are there any special cases where Kafka silently drops a message ?
> > >
> > > We also found a delay in the notifications/watches triggered from
> > > zookeeper. We are not sure whether these are related ? It will be
> difficult
> > > to reproduce as the test probably took a few days to complete. But
> surely
> > > we did lose approximately 5% of the messages. We have logs of messages
> > > being produced at the producer side and corresponding entries in Kafka
> > > partitions logs. But nothing at the consumer side. The only repeating
> > > pattern was that the messages were probably produced within the same
> > > millisecond. So if you have a sequence of messages which was produced
> in
> > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably
> have
> > > M0,M1,M3 but not M2. This is puzzling as to how only message is
> dropped out
> > > of the given 4.
> > >
> > >
> > > We use the High Level Kafka Producer and Consumer. Both are single
> > > threaded(at our end).
> > >
> > > Does kafka need its own dedicated zookeeper ensemble ? We also use the
> > > same zookeeper ensemble as our configuration service.
> > >
> > > Unfortunately, we did not have DEBUG messages at the server enabled
> during
> > > the setup. Although, NO error messages were observed during the same
> time
> > > period.
> > >
> > >
> > > Before we try running the same Tests again, can someone please shed
> more
> > > light as to the reasons why kafka dropped a few messages ?
> > >
> > > Kat
> > >
>
>
RE: Dropping messages ?
Posted by Kat Walker <ek...@hotmail.com>.
Hi Jun/Guozhang
We might have to retry our QA tests in its entirety. We simply cannot reset consumer offset as there is a lot of processing involved after consuming those messages. This might take almost a week. The Kafka message also contains `volatile` data which is fetched from a database and destroyed after consuming that Kafka message.
What is puzzling is that we have been running Kafka in Production for over 4 months now and we have never faced this issue. Our peak volume is <= 1000 messages / second. On an average, it is less than 100.
Our zookeeper version is 3.3.6. What I suspect is that we managed to `roll over` a few messages due to inconsistencies in zookeeper offsets. Recall that we only had only 1 Kafka broker. Once again, this is just a `wild` speculation.
The topics were created before we started our tests.
Has someone ever lost a Kafka message before ? Are there any guarantees ? We are okay with all kinds of delays provided, the messages arrive in order and are delivered regardless.
Thanks
Kat
> Date: Mon, 10 Feb 2014 07:38:01 -0800
> Subject: Re: Dropping messages ?
> From: junrao@gmail.com
> To: users@kafka.apache.org
>
> If you reset the consumer offset and try to consume those messages again,
> do you see the same drop?
>
> Thanks,
>
> Jun
>
>
> On Mon, Feb 10, 2014 at 1:21 AM, A A <ek...@hotmail.com> wrote:
>
> > Hi
> >
> > We have been using Kafka(0.8) for the past few months with the following
> > setup
> > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> >
> > Yesterday, while running Stress tests in one of the QA machines , we
> > observed that a few messages which were produced within a couple of
> > milliseconds of each other did not reach the Kafka consumer. ie There was
> > no trace of that message at the consumer end.
> >
> > We decided to check whether we had any errors at our side or there was a
> > network issue. We did not find any issue. We then decided to check whether
> > we can find that message in one of the Kafka partitions. The message was
> > found in one of the topic partitions.
> >
> > We are not sure why Kafka did not notify any consumers about the message.
> > Are there any special cases where Kafka silently drops a message ?
> >
> > We also found a delay in the notifications/watches triggered from
> > zookeeper. We are not sure whether these are related ? It will be difficult
> > to reproduce as the test probably took a few days to complete. But surely
> > we did lose approximately 5% of the messages. We have logs of messages
> > being produced at the producer side and corresponding entries in Kafka
> > partitions logs. But nothing at the consumer side. The only repeating
> > pattern was that the messages were probably produced within the same
> > millisecond. So if you have a sequence of messages which was produced in
> > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably have
> > M0,M1,M3 but not M2. This is puzzling as to how only message is dropped out
> > of the given 4.
> >
> >
> > We use the High Level Kafka Producer and Consumer. Both are single
> > threaded(at our end).
> >
> > Does kafka need its own dedicated zookeeper ensemble ? We also use the
> > same zookeeper ensemble as our configuration service.
> >
> > Unfortunately, we did not have DEBUG messages at the server enabled during
> > the setup. Although, NO error messages were observed during the same time
> > period.
> >
> >
> > Before we try running the same Tests again, can someone please shed more
> > light as to the reasons why kafka dropped a few messages ?
> >
> > Kat
> >
Re: Dropping messages ?
Posted by Guozhang Wang <wa...@gmail.com>.
Are these messages sent to a newly created topic?
Guozhang
On Mon, Feb 10, 2014 at 7:38 AM, Jun Rao <ju...@gmail.com> wrote:
> If you reset the consumer offset and try to consume those messages again,
> do you see the same drop?
>
> Thanks,
>
> Jun
>
>
> On Mon, Feb 10, 2014 at 1:21 AM, A A <ek...@hotmail.com> wrote:
>
> > Hi
> >
> > We have been using Kafka(0.8) for the past few months with the following
> > setup
> > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> >
> > Yesterday, while running Stress tests in one of the QA machines , we
> > observed that a few messages which were produced within a couple of
> > milliseconds of each other did not reach the Kafka consumer. ie There was
> > no trace of that message at the consumer end.
> >
> > We decided to check whether we had any errors at our side or there was a
> > network issue. We did not find any issue. We then decided to check
> whether
> > we can find that message in one of the Kafka partitions. The message was
> > found in one of the topic partitions.
> >
> > We are not sure why Kafka did not notify any consumers about the message.
> > Are there any special cases where Kafka silently drops a message ?
> >
> > We also found a delay in the notifications/watches triggered from
> > zookeeper. We are not sure whether these are related ? It will be
> difficult
> > to reproduce as the test probably took a few days to complete. But surely
> > we did lose approximately 5% of the messages. We have logs of messages
> > being produced at the producer side and corresponding entries in Kafka
> > partitions logs. But nothing at the consumer side. The only repeating
> > pattern was that the messages were probably produced within the same
> > millisecond. So if you have a sequence of messages which was produced in
> > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably have
> > M0,M1,M3 but not M2. This is puzzling as to how only message is dropped
> out
> > of the given 4.
> >
> >
> > We use the High Level Kafka Producer and Consumer. Both are single
> > threaded(at our end).
> >
> > Does kafka need its own dedicated zookeeper ensemble ? We also use the
> > same zookeeper ensemble as our configuration service.
> >
> > Unfortunately, we did not have DEBUG messages at the server enabled
> during
> > the setup. Although, NO error messages were observed during the same time
> > period.
> >
> >
> > Before we try running the same Tests again, can someone please shed more
> > light as to the reasons why kafka dropped a few messages ?
> >
> > Kat
> >
>
--
-- Guozhang
Re: Dropping messages ?
Posted by Jun Rao <ju...@gmail.com>.
If you reset the consumer offset and try to consume those messages again,
do you see the same drop?
Thanks,
Jun
On Mon, Feb 10, 2014 at 1:21 AM, A A <ek...@hotmail.com> wrote:
> Hi
>
> We have been using Kafka(0.8) for the past few months with the following
> setup
> Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
>
> Yesterday, while running Stress tests in one of the QA machines , we
> observed that a few messages which were produced within a couple of
> milliseconds of each other did not reach the Kafka consumer. ie There was
> no trace of that message at the consumer end.
>
> We decided to check whether we had any errors at our side or there was a
> network issue. We did not find any issue. We then decided to check whether
> we can find that message in one of the Kafka partitions. The message was
> found in one of the topic partitions.
>
> We are not sure why Kafka did not notify any consumers about the message.
> Are there any special cases where Kafka silently drops a message ?
>
> We also found a delay in the notifications/watches triggered from
> zookeeper. We are not sure whether these are related ? It will be difficult
> to reproduce as the test probably took a few days to complete. But surely
> we did lose approximately 5% of the messages. We have logs of messages
> being produced at the producer side and corresponding entries in Kafka
> partitions logs. But nothing at the consumer side. The only repeating
> pattern was that the messages were probably produced within the same
> millisecond. So if you have a sequence of messages which was produced in
> the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably have
> M0,M1,M3 but not M2. This is puzzling as to how only message is dropped out
> of the given 4.
>
>
> We use the High Level Kafka Producer and Consumer. Both are single
> threaded(at our end).
>
> Does kafka need its own dedicated zookeeper ensemble ? We also use the
> same zookeeper ensemble as our configuration service.
>
> Unfortunately, we did not have DEBUG messages at the server enabled during
> the setup. Although, NO error messages were observed during the same time
> period.
>
>
> Before we try running the same Tests again, can someone please shed more
> light as to the reasons why kafka dropped a few messages ?
>
> Kat
>