You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jeffrey Damick <je...@gmail.com> on 2011/07/20 21:47:30 UTC
kafka-49
Is there any current thought around KAFKA-49, for acknowledgement of
producers?
Will this be optional, a new message type(s)?
Will the ack be synchronous or asynchronous or depending on request type?
It would be fantastic if the ack contained the starting offset of the
message published, and not just the ending.
This is quickly becoming an issue for us, so we may be able to provide some
help in this area..
thanks
-jeff
Re: kafka-49
Posted by Jeffrey Damick <je...@gmail.com>.
That would seem sub-optimal.. Perhaps you could have a separate vip for producing that was configured by some process watching zk to determine which is leader or some custom health check maybe via restful api..just a thought.
Sent from my iPhone
On Jul 21, 2011, at 6:33 PM, Jun Rao <ju...@gmail.com> wrote:
> If we have ACK, the producer can catch any exception and resend.
>
> Jun
>
> On Thu, Jul 21, 2011 at 12:47 PM, Jay Kreps <ja...@gmail.com> wrote:
>
>> Yeah, that makes sense. How are we going to handle production through a vip
>> in that case?
>>
>> -Jay
>>
>> On Thu, Jul 21, 2011 at 7:44 AM, Jun Rao <ju...@gmail.com> wrote:
>>
>>> Without replication, we can make ACK optional. With replication, a
>> producer
>>> can only write to the leader replica. Without ACK, there is no way that
>> the
>>> broker can inform the producer that it's trying to write to the wrong
>>> broker.
>>>
>>> Jun
>>>
>>> On Wed, Jul 20, 2011 at 8:58 PM, Jay Kreps <ja...@gmail.com> wrote:
>>>
>>>> It might be nice to consider making the ack optional and part of the
>>>> request. The current behavior is good for many uses, the request is
>>>> instantaneously written to the socket buffer but sent asynchronously. I
>>>> think that is a valuable use case where you care about throughput. I
>>> guess
>>>> the question is whether the asynchronous api already covers that well
>>>> enough
>>>> and how much complexity exposing that causes.
>>>>
>>>> -Jay
>>>>
>>>> On Wed, Jul 20, 2011 at 5:51 PM, Jun Rao <ju...@gmail.com> wrote:
>>>>
>>>>> Jeff,
>>>>>
>>>>> I was thinking of making the ACK mandatory for the producer. The ACK
>>> can
>>>> be
>>>>> sent when the message either hits 1 replicas or multiple replicas,
>>>>> depending
>>>>> on the setting.
>>>>>
>>>>> Having the ACK include the starting offset of the message seems
>>>> reasonable.
>>>>> It will be a bit complicated for multisend since multiple offsets
>> have
>>> to
>>>>> be
>>>>> returned. What do you need the offset for?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Wed, Jul 20, 2011 at 12:47 PM, Jeffrey Damick <
>>>> jeffreydamick@gmail.com
>>>>>> wrote:
>>>>>
>>>>>> Is there any current thought around KAFKA-49, for acknowledgement
>> of
>>>>>> producers?
>>>>>> Will this be optional, a new message type(s)?
>>>>>> Will the ack be synchronous or asynchronous or depending on request
>>>> type?
>>>>>>
>>>>>> It would be fantastic if the ack contained the starting offset of
>> the
>>>>>> message published, and not just the ending.
>>>>>>
>>>>>> This is quickly becoming an issue for us, so we may be able to
>>> provide
>>>>> some
>>>>>> help in this area..
>>>>>>
>>>>>>
>>>>>> thanks
>>>>>> -jeff
>>>>>>
>>>>>
>>>>
>>>
>>
Re: kafka-49
Posted by Jun Rao <ju...@gmail.com>.
If we have ACK, the producer can catch any exception and resend.
Jun
On Thu, Jul 21, 2011 at 12:47 PM, Jay Kreps <ja...@gmail.com> wrote:
> Yeah, that makes sense. How are we going to handle production through a vip
> in that case?
>
> -Jay
>
> On Thu, Jul 21, 2011 at 7:44 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Without replication, we can make ACK optional. With replication, a
> producer
> > can only write to the leader replica. Without ACK, there is no way that
> the
> > broker can inform the producer that it's trying to write to the wrong
> > broker.
> >
> > Jun
> >
> > On Wed, Jul 20, 2011 at 8:58 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > It might be nice to consider making the ack optional and part of the
> > > request. The current behavior is good for many uses, the request is
> > > instantaneously written to the socket buffer but sent asynchronously. I
> > > think that is a valuable use case where you care about throughput. I
> > guess
> > > the question is whether the asynchronous api already covers that well
> > > enough
> > > and how much complexity exposing that causes.
> > >
> > > -Jay
> > >
> > > On Wed, Jul 20, 2011 at 5:51 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Jeff,
> > > >
> > > > I was thinking of making the ACK mandatory for the producer. The ACK
> > can
> > > be
> > > > sent when the message either hits 1 replicas or multiple replicas,
> > > > depending
> > > > on the setting.
> > > >
> > > > Having the ACK include the starting offset of the message seems
> > > reasonable.
> > > > It will be a bit complicated for multisend since multiple offsets
> have
> > to
> > > > be
> > > > returned. What do you need the offset for?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Jul 20, 2011 at 12:47 PM, Jeffrey Damick <
> > > jeffreydamick@gmail.com
> > > > >wrote:
> > > >
> > > > > Is there any current thought around KAFKA-49, for acknowledgement
> of
> > > > > producers?
> > > > > Will this be optional, a new message type(s)?
> > > > > Will the ack be synchronous or asynchronous or depending on request
> > > type?
> > > > >
> > > > > It would be fantastic if the ack contained the starting offset of
> the
> > > > > message published, and not just the ending.
> > > > >
> > > > > This is quickly becoming an issue for us, so we may be able to
> > provide
> > > > some
> > > > > help in this area..
> > > > >
> > > > >
> > > > > thanks
> > > > > -jeff
> > > > >
> > > >
> > >
> >
>
Re: kafka-49
Posted by Jay Kreps <ja...@gmail.com>.
Yeah, that makes sense. How are we going to handle production through a vip
in that case?
-Jay
On Thu, Jul 21, 2011 at 7:44 AM, Jun Rao <ju...@gmail.com> wrote:
> Without replication, we can make ACK optional. With replication, a producer
> can only write to the leader replica. Without ACK, there is no way that the
> broker can inform the producer that it's trying to write to the wrong
> broker.
>
> Jun
>
> On Wed, Jul 20, 2011 at 8:58 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > It might be nice to consider making the ack optional and part of the
> > request. The current behavior is good for many uses, the request is
> > instantaneously written to the socket buffer but sent asynchronously. I
> > think that is a valuable use case where you care about throughput. I
> guess
> > the question is whether the asynchronous api already covers that well
> > enough
> > and how much complexity exposing that causes.
> >
> > -Jay
> >
> > On Wed, Jul 20, 2011 at 5:51 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Jeff,
> > >
> > > I was thinking of making the ACK mandatory for the producer. The ACK
> can
> > be
> > > sent when the message either hits 1 replicas or multiple replicas,
> > > depending
> > > on the setting.
> > >
> > > Having the ACK include the starting offset of the message seems
> > reasonable.
> > > It will be a bit complicated for multisend since multiple offsets have
> to
> > > be
> > > returned. What do you need the offset for?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Jul 20, 2011 at 12:47 PM, Jeffrey Damick <
> > jeffreydamick@gmail.com
> > > >wrote:
> > >
> > > > Is there any current thought around KAFKA-49, for acknowledgement of
> > > > producers?
> > > > Will this be optional, a new message type(s)?
> > > > Will the ack be synchronous or asynchronous or depending on request
> > type?
> > > >
> > > > It would be fantastic if the ack contained the starting offset of the
> > > > message published, and not just the ending.
> > > >
> > > > This is quickly becoming an issue for us, so we may be able to
> provide
> > > some
> > > > help in this area..
> > > >
> > > >
> > > > thanks
> > > > -jeff
> > > >
> > >
> >
>
Re: kafka-49
Posted by Jun Rao <ju...@gmail.com>.
Without replication, we can make ACK optional. With replication, a producer
can only write to the leader replica. Without ACK, there is no way that the
broker can inform the producer that it's trying to write to the wrong
broker.
Jun
On Wed, Jul 20, 2011 at 8:58 PM, Jay Kreps <ja...@gmail.com> wrote:
> It might be nice to consider making the ack optional and part of the
> request. The current behavior is good for many uses, the request is
> instantaneously written to the socket buffer but sent asynchronously. I
> think that is a valuable use case where you care about throughput. I guess
> the question is whether the asynchronous api already covers that well
> enough
> and how much complexity exposing that causes.
>
> -Jay
>
> On Wed, Jul 20, 2011 at 5:51 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Jeff,
> >
> > I was thinking of making the ACK mandatory for the producer. The ACK can
> be
> > sent when the message either hits 1 replicas or multiple replicas,
> > depending
> > on the setting.
> >
> > Having the ACK include the starting offset of the message seems
> reasonable.
> > It will be a bit complicated for multisend since multiple offsets have to
> > be
> > returned. What do you need the offset for?
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Jul 20, 2011 at 12:47 PM, Jeffrey Damick <
> jeffreydamick@gmail.com
> > >wrote:
> >
> > > Is there any current thought around KAFKA-49, for acknowledgement of
> > > producers?
> > > Will this be optional, a new message type(s)?
> > > Will the ack be synchronous or asynchronous or depending on request
> type?
> > >
> > > It would be fantastic if the ack contained the starting offset of the
> > > message published, and not just the ending.
> > >
> > > This is quickly becoming an issue for us, so we may be able to provide
> > some
> > > help in this area..
> > >
> > >
> > > thanks
> > > -jeff
> > >
> >
>
Re: kafka-49
Posted by Jay Kreps <ja...@gmail.com>.
It might be nice to consider making the ack optional and part of the
request. The current behavior is good for many uses, the request is
instantaneously written to the socket buffer but sent asynchronously. I
think that is a valuable use case where you care about throughput. I guess
the question is whether the asynchronous api already covers that well enough
and how much complexity exposing that causes.
-Jay
On Wed, Jul 20, 2011 at 5:51 PM, Jun Rao <ju...@gmail.com> wrote:
> Jeff,
>
> I was thinking of making the ACK mandatory for the producer. The ACK can be
> sent when the message either hits 1 replicas or multiple replicas,
> depending
> on the setting.
>
> Having the ACK include the starting offset of the message seems reasonable.
> It will be a bit complicated for multisend since multiple offsets have to
> be
> returned. What do you need the offset for?
>
> Thanks,
>
> Jun
>
> On Wed, Jul 20, 2011 at 12:47 PM, Jeffrey Damick <jeffreydamick@gmail.com
> >wrote:
>
> > Is there any current thought around KAFKA-49, for acknowledgement of
> > producers?
> > Will this be optional, a new message type(s)?
> > Will the ack be synchronous or asynchronous or depending on request type?
> >
> > It would be fantastic if the ack contained the starting offset of the
> > message published, and not just the ending.
> >
> > This is quickly becoming an issue for us, so we may be able to provide
> some
> > help in this area..
> >
> >
> > thanks
> > -jeff
> >
>
Re: kafka-49
Posted by Jeffrey Damick <je...@gmail.com>.
But if there are no replicas, a produce with an ACK requested would at least
force a flush to disk I assume? That would be enough for me right now.
And I agree with Jay's comment, the tradeoff of durability vs. throughput
could be chosen by the producer, either with a different message type or
flag in the request.
As for the offset of the message, I'd really like to be able to know what it
is after I produce so that it can be logged and monitored. Then if needed
something or someone can quickly grab the latest message produced on a topic
for metrics, monitoring, and debugging if something goes awry, otherwise I
need to have the producer also consume the topic to figure it out.. (the
data I'm transporting is a little more important than logs)
So back to the ACK'ing, I'd willing to submit a patch for at least the
optional flag or new message type to force a flush to disk if it would
help..
thanks,
-jeff
On Wed, Jul 20, 2011 at 8:51 PM, Jun Rao <ju...@gmail.com> wrote:
> Jeff,
>
> I was thinking of making the ACK mandatory for the producer. The ACK can be
> sent when the message either hits 1 replicas or multiple replicas,
> depending
> on the setting.
>
> Having the ACK include the starting offset of the message seems reasonable.
> It will be a bit complicated for multisend since multiple offsets have to
> be
> returned. What do you need the offset for?
>
> Thanks,
>
> Jun
>
> On Wed, Jul 20, 2011 at 12:47 PM, Jeffrey Damick <jeffreydamick@gmail.com
> >wrote:
>
> > Is there any current thought around KAFKA-49, for acknowledgement of
> > producers?
> > Will this be optional, a new message type(s)?
> > Will the ack be synchronous or asynchronous or depending on request type?
> >
> > It would be fantastic if the ack contained the starting offset of the
> > message published, and not just the ending.
> >
> > This is quickly becoming an issue for us, so we may be able to provide
> some
> > help in this area..
> >
> >
> > thanks
> > -jeff
> >
>
Re: kafka-49
Posted by Jun Rao <ju...@gmail.com>.
Jeff,
I was thinking of making the ACK mandatory for the producer. The ACK can be
sent when the message either hits 1 replicas or multiple replicas, depending
on the setting.
Having the ACK include the starting offset of the message seems reasonable.
It will be a bit complicated for multisend since multiple offsets have to be
returned. What do you need the offset for?
Thanks,
Jun
On Wed, Jul 20, 2011 at 12:47 PM, Jeffrey Damick <je...@gmail.com>wrote:
> Is there any current thought around KAFKA-49, for acknowledgement of
> producers?
> Will this be optional, a new message type(s)?
> Will the ack be synchronous or asynchronous or depending on request type?
>
> It would be fantastic if the ack contained the starting offset of the
> message published, and not just the ending.
>
> This is quickly becoming an issue for us, so we may be able to provide some
> help in this area..
>
>
> thanks
> -jeff
>