You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ross Black <ro...@gmail.com> on 2013/05/22 02:42:08 UTC

message ordering guarantees

Hi,

I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with a
single broker service process.

I am occasionally seeing messages (from a *single* partition) being
processed out of order to what I expect and I am trying to find where the
problem lies.  The problem may well be in my code - I just would like to
eliminate Kafka as a potential cause.

Messages are being sent sequentially from the producer process, using a
single SyncProducer.
Does Kafka provide any guarantees for message ordering in this case?

eg.  If The sync producer sends messages A then B then C, does the Kafka
broker guarantee that messages will be persisted with the order A,B,C?
If not, is there any way to ensure this ordering?
Has anything changed in 0.8 that could be used to ensure that ordering?


Thanks,
Ross

Re: message ordering guarantees

Posted by Ross Black <ro...@gmail.com>.
Thanks for the explanation.

Ross



On 23 May 2013 07:19, Neha Narkhede <ne...@gmail.com> wrote:

> Thanks,
> Neha
> On May 21, 2013 5:42 PM, "Ross Black" <ro...@gmail.com> wrote:
>
> > Hi,
> >
> > I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with a
> > single broker service process.
> >
> > I am occasionally seeing messages (from a *single* partition) being
> > processed out of order to what I expect and I am trying to find where the
> > problem lies.  The problem may well be in my code - I just would like to
> > eliminate Kafka as a potential cause.
> >
> > Messages are being sent sequentially from the producer process, using a
> > single SyncProducer.
> > Does Kafka provide any guarantees for message ordering in this case?
> >
> > eg.  If The sync producer sends messages A then B then C, does the Kafka
> > broker guarantee that messages will be persisted with the order A,B,C?
> > If not, is there any way to ensure this ordering?
> > Has anything changed in 0.8 that could be used to ensure that ordering?
> >
> >
> > Thanks,
> > Ross
> >
>

Re: message ordering guarantees

Posted by Neha Narkhede <ne...@gmail.com>.
Thanks,
Neha
On May 21, 2013 5:42 PM, "Ross Black" <ro...@gmail.com> wrote:

> Hi,
>
> I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with a
> single broker service process.
>
> I am occasionally seeing messages (from a *single* partition) being
> processed out of order to what I expect and I am trying to find where the
> problem lies.  The problem may well be in my code - I just would like to
> eliminate Kafka as a potential cause.
>
> Messages are being sent sequentially from the producer process, using a
> single SyncProducer.
> Does Kafka provide any guarantees for message ordering in this case?
>
> eg.  If The sync producer sends messages A then B then C, does the Kafka
> broker guarantee that messages will be persisted with the order A,B,C?
> If not, is there any way to ensure this ordering?
> Has anything changed in 0.8 that could be used to ensure that ordering?
>
>
> Thanks,
> Ross
>

Re: message ordering guarantees

Posted by Jun Rao <ju...@gmail.com>.
Yes, reconnect.time.interval.ms is a new config that we added to 0.7. Will
update the website.

VIP is a virtual IP on a load balancer. You can associate multiple brokers
with a VIP and the VIP will randomly select one of the brokers to connect
to during socket establish time. The issue is that if a broker is brought
up, initially there is no connection to it. The reconnect.interval property
gives the producer a chance to pick up the new broker periodically.

Thanks,

Jun


On Tue, May 21, 2013 at 9:48 PM, Ross Black <ro...@gmail.com> wrote:

> Hi Jun,
>
> Thanks for the quick response.
> (I just discovered that you had answered a similar question on the dev list
> -
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201304.mbox/%3CCAFbh0Q1XSjG2B9iO%2BpDeM8p9RsDDqKA9E8LY5ScXzAncaQ_mVg%40mail.gmail.com%3E
> )
>
> I set the following properties on the SyncProducer:
>
>         syncProperties.setProperty("reconnect.interval",
> Integer.toString(Integer.MAX_VALUE));
>         syncProperties.setProperty("reconnect.time.interval.ms", "-1");
>
> which resolved the out-of-order problem.
>
> reconnect.time.interval.ms is not documented on the page you specified,
> but
> it was also necessary to stop the periodic reconnect.
>
> We only use a direct connection to the broker using SyncProducer, and don't
> make use of zookeeper.
> Could you please explain what the reconnect is used for (I don't understand
> what VIP is for producers)?
>
> Thanks,
> Ross
>
>
>
> On 22 May 2013 14:19, Jun Rao <ju...@gmail.com> wrote:
>
> > In 0.7, each producer reconnects to the broker periodically. Messages
> sent
> > in different socket connections could arrive out of order. Take a look at
> > reconnect.interval in http://kafka.apache.org/07/configuration.html. If
> > you
> > don't use VIP for the producers, you can actually set reconnect.interval
> to
> > a really large value. In 0.8, there is no reconnect.interval any more.
> > Messages sent from a single producer are expected to arrive in order at
> the
> > broker, as long as they sent to the same partition in a topic.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, May 21, 2013 at 5:42 PM, Ross Black <ro...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with
> a
> > > single broker service process.
> > >
> > > I am occasionally seeing messages (from a *single* partition) being
> > > processed out of order to what I expect and I am trying to find where
> the
> > > problem lies.  The problem may well be in my code - I just would like
> to
> > > eliminate Kafka as a potential cause.
> > >
> > > Messages are being sent sequentially from the producer process, using a
> > > single SyncProducer.
> > > Does Kafka provide any guarantees for message ordering in this case?
> > >
> > > eg.  If The sync producer sends messages A then B then C, does the
> Kafka
> > > broker guarantee that messages will be persisted with the order A,B,C?
> > > If not, is there any way to ensure this ordering?
> > > Has anything changed in 0.8 that could be used to ensure that ordering?
> > >
> > >
> > > Thanks,
> > > Ross
> > >
> >
>

Re: message ordering guarantees

Posted by Ross Black <ro...@gmail.com>.
Hi Jun,

Thanks for the quick response.
(I just discovered that you had answered a similar question on the dev list
-
http://mail-archives.apache.org/mod_mbox/kafka-dev/201304.mbox/%3CCAFbh0Q1XSjG2B9iO%2BpDeM8p9RsDDqKA9E8LY5ScXzAncaQ_mVg%40mail.gmail.com%3E
)

I set the following properties on the SyncProducer:

        syncProperties.setProperty("reconnect.interval",
Integer.toString(Integer.MAX_VALUE));
        syncProperties.setProperty("reconnect.time.interval.ms", "-1");

which resolved the out-of-order problem.

reconnect.time.interval.ms is not documented on the page you specified, but
it was also necessary to stop the periodic reconnect.

We only use a direct connection to the broker using SyncProducer, and don't
make use of zookeeper.
Could you please explain what the reconnect is used for (I don't understand
what VIP is for producers)?

Thanks,
Ross



On 22 May 2013 14:19, Jun Rao <ju...@gmail.com> wrote:

> In 0.7, each producer reconnects to the broker periodically. Messages sent
> in different socket connections could arrive out of order. Take a look at
> reconnect.interval in http://kafka.apache.org/07/configuration.html. If
> you
> don't use VIP for the producers, you can actually set reconnect.interval to
> a really large value. In 0.8, there is no reconnect.interval any more.
> Messages sent from a single producer are expected to arrive in order at the
> broker, as long as they sent to the same partition in a topic.
>
> Thanks,
>
> Jun
>
>
> On Tue, May 21, 2013 at 5:42 PM, Ross Black <ro...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with a
> > single broker service process.
> >
> > I am occasionally seeing messages (from a *single* partition) being
> > processed out of order to what I expect and I am trying to find where the
> > problem lies.  The problem may well be in my code - I just would like to
> > eliminate Kafka as a potential cause.
> >
> > Messages are being sent sequentially from the producer process, using a
> > single SyncProducer.
> > Does Kafka provide any guarantees for message ordering in this case?
> >
> > eg.  If The sync producer sends messages A then B then C, does the Kafka
> > broker guarantee that messages will be persisted with the order A,B,C?
> > If not, is there any way to ensure this ordering?
> > Has anything changed in 0.8 that could be used to ensure that ordering?
> >
> >
> > Thanks,
> > Ross
> >
>

Re: message ordering guarantees

Posted by Jun Rao <ju...@gmail.com>.
In 0.7, each producer reconnects to the broker periodically. Messages sent
in different socket connections could arrive out of order. Take a look at
reconnect.interval in http://kafka.apache.org/07/configuration.html. If you
don't use VIP for the producers, you can actually set reconnect.interval to
a really large value. In 0.8, there is no reconnect.interval any more.
Messages sent from a single producer are expected to arrive in order at the
broker, as long as they sent to the same partition in a topic.

Thanks,

Jun


On Tue, May 21, 2013 at 5:42 PM, Ross Black <ro...@gmail.com> wrote:

> Hi,
>
> I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with a
> single broker service process.
>
> I am occasionally seeing messages (from a *single* partition) being
> processed out of order to what I expect and I am trying to find where the
> problem lies.  The problem may well be in my code - I just would like to
> eliminate Kafka as a potential cause.
>
> Messages are being sent sequentially from the producer process, using a
> single SyncProducer.
> Does Kafka provide any guarantees for message ordering in this case?
>
> eg.  If The sync producer sends messages A then B then C, does the Kafka
> broker guarantee that messages will be persisted with the order A,B,C?
> If not, is there any way to ensure this ordering?
> Has anything changed in 0.8 that could be used to ensure that ordering?
>
>
> Thanks,
> Ross
>