You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jason Gustafson <ja...@confluent.io> on 2017/09/05 18:17:22 UTC
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
The proposal looks good. Two minor comments:
1. Can we call out how we handle the duplicate case? This is a change in
behavior since we currently raise OutOfOrderSequence in this case.
2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I
wonder if we should have a field for the minimum required message format.
When using enable.idempotence=required, we could set the minimum required
version to v2. For enable.idempotence=requested, we could use v0. The
advantage is that we may find other uses for a more general field in the
future. Alternatively, maybe we really should be returning the message
format version of each topic in the TopicMetadata response. A nice bonus of
doing so is that it gives the producer the ability to craft the right
format version ahead of time and avoid the need for conversion on the
broker.
Thanks,
Jason
On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta <ap...@confluent.io> wrote:
> Hi,
>
> In the discussion of KIP-185 (enable idempotence by default), we discovered
> some shortcomings of the existing idempotent producer implementation.
> Fixing these issues requires changes to the ProduceRequest and
> ProduceResponse protocols as well as changes to the values of the
> 'enable.idempotence' producer config.
>
> Hence, I split off those changes into another KIP so as to decouple the two
> issues. Please have a look at my follow up KIP below:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
>
> KIP-185 depends on KIP-192, and I hope to make progress on the latter
> independently.
>
> Thanks,
> Apurva
>
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
Posted by Ismael Juma <is...@juma.me.uk>.
Agreed, we should make it consistent across the various pages.
On Tue, Sep 5, 2017 at 10:06 PM, Ted Yu <yu...@gmail.com> wrote:
> Sounds good.
>
> Please update the KIP and:
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> Kafka+Exactly+Once+-+Dealing+with+older+message+formats+
> when+idempotence+is+enabled
>
> On Tue, Sep 5, 2017 at 2:03 PM, Ismael Juma <is...@gmail.com> wrote:
>
> > Hi Ted,
> >
> > The current proposal has 3 options: requested, required, off ("safe" was
> in
> > an earlier proposal). I think these convey the meaning more clearly IMO.
> >
> > Ismael
> >
> > On 5 Sep 2017 9:22 pm, "Ted Yu" <yu...@gmail.com> wrote:
> >
> > > For enable.idempotence=safe, it seems giving user impression that
> > > idempotence
> > > would be safe.
> > >
> > > However, since it really means best effort, the 'safety' is debatable.
> > >
> > > Why not just call the new mode besteffort ?
> > >
> > > Cheers
> > >
> > > On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > If we add the message format version (a topic config) in the response
> > of
> > > > TopicMetadata, we should consider adding the max message bytes as
> well.
> > > > That would allow us to later improve the implementation of KIP-126 to
> > > split
> > > > the batch _before_ sending.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson <ja...@confluent.io>
> > > > wrote:
> > > >
> > > > > The proposal looks good. Two minor comments:
> > > > >
> > > > > 1. Can we call out how we handle the duplicate case? This is a
> change
> > > in
> > > > > behavior since we currently raise OutOfOrderSequence in this case.
> > > > >
> > > > > 2. Instead of passing through `idempotenceLevel` in the
> > > ProduceRequest, I
> > > > > wonder if we should have a field for the minimum required message
> > > format.
> > > > > When using enable.idempotence=required, we could set the minimum
> > > required
> > > > > version to v2. For enable.idempotence=requested, we could use v0.
> The
> > > > > advantage is that we may find other uses for a more general field
> in
> > > the
> > > > > future. Alternatively, maybe we really should be returning the
> > message
> > > > > format version of each topic in the TopicMetadata response. A nice
> > > bonus
> > > > of
> > > > > doing so is that it gives the producer the ability to craft the
> right
> > > > > format version ahead of time and avoid the need for conversion on
> the
> > > > > broker.
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta <apurva@confluent.io
> >
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > In the discussion of KIP-185 (enable idempotence by default), we
> > > > > discovered
> > > > > > some shortcomings of the existing idempotent producer
> > implementation.
> > > > > > Fixing these issues requires changes to the ProduceRequest and
> > > > > > ProduceResponse protocols as well as changes to the values of the
> > > > > > 'enable.idempotence' producer config.
> > > > > >
> > > > > > Hence, I split off those changes into another KIP so as to
> decouple
> > > the
> > > > > two
> > > > > > issues. Please have a look at my follow up KIP below:
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> > > > > >
> > > > > > KIP-185 depends on KIP-192, and I hope to make progress on the
> > latter
> > > > > > independently.
> > > > > >
> > > > > > Thanks,
> > > > > > Apurva
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
Posted by Ted Yu <yu...@gmail.com>.
Sounds good.
Please update the KIP and:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Exactly+Once+-+Dealing+with+older+message+formats+when+idempotence+is+enabled
On Tue, Sep 5, 2017 at 2:03 PM, Ismael Juma <is...@gmail.com> wrote:
> Hi Ted,
>
> The current proposal has 3 options: requested, required, off ("safe" was in
> an earlier proposal). I think these convey the meaning more clearly IMO.
>
> Ismael
>
> On 5 Sep 2017 9:22 pm, "Ted Yu" <yu...@gmail.com> wrote:
>
> > For enable.idempotence=safe, it seems giving user impression that
> > idempotence
> > would be safe.
> >
> > However, since it really means best effort, the 'safety' is debatable.
> >
> > Why not just call the new mode besteffort ?
> >
> > Cheers
> >
> > On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > If we add the message format version (a topic config) in the response
> of
> > > TopicMetadata, we should consider adding the max message bytes as well.
> > > That would allow us to later improve the implementation of KIP-126 to
> > split
> > > the batch _before_ sending.
> > >
> > > Ismael
> > >
> > > On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > > > The proposal looks good. Two minor comments:
> > > >
> > > > 1. Can we call out how we handle the duplicate case? This is a change
> > in
> > > > behavior since we currently raise OutOfOrderSequence in this case.
> > > >
> > > > 2. Instead of passing through `idempotenceLevel` in the
> > ProduceRequest, I
> > > > wonder if we should have a field for the minimum required message
> > format.
> > > > When using enable.idempotence=required, we could set the minimum
> > required
> > > > version to v2. For enable.idempotence=requested, we could use v0. The
> > > > advantage is that we may find other uses for a more general field in
> > the
> > > > future. Alternatively, maybe we really should be returning the
> message
> > > > format version of each topic in the TopicMetadata response. A nice
> > bonus
> > > of
> > > > doing so is that it gives the producer the ability to craft the right
> > > > format version ahead of time and avoid the need for conversion on the
> > > > broker.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta <ap...@confluent.io>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > In the discussion of KIP-185 (enable idempotence by default), we
> > > > discovered
> > > > > some shortcomings of the existing idempotent producer
> implementation.
> > > > > Fixing these issues requires changes to the ProduceRequest and
> > > > > ProduceResponse protocols as well as changes to the values of the
> > > > > 'enable.idempotence' producer config.
> > > > >
> > > > > Hence, I split off those changes into another KIP so as to decouple
> > the
> > > > two
> > > > > issues. Please have a look at my follow up KIP below:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> > > > >
> > > > > KIP-185 depends on KIP-192, and I hope to make progress on the
> latter
> > > > > independently.
> > > > >
> > > > > Thanks,
> > > > > Apurva
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
Posted by Ismael Juma <is...@gmail.com>.
Hi Ted,
The current proposal has 3 options: requested, required, off ("safe" was in
an earlier proposal). I think these convey the meaning more clearly IMO.
Ismael
On 5 Sep 2017 9:22 pm, "Ted Yu" <yu...@gmail.com> wrote:
> For enable.idempotence=safe, it seems giving user impression that
> idempotence
> would be safe.
>
> However, since it really means best effort, the 'safety' is debatable.
>
> Why not just call the new mode besteffort ?
>
> Cheers
>
> On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > If we add the message format version (a topic config) in the response of
> > TopicMetadata, we should consider adding the max message bytes as well.
> > That would allow us to later improve the implementation of KIP-126 to
> split
> > the batch _before_ sending.
> >
> > Ismael
> >
> > On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson <ja...@confluent.io>
> > wrote:
> >
> > > The proposal looks good. Two minor comments:
> > >
> > > 1. Can we call out how we handle the duplicate case? This is a change
> in
> > > behavior since we currently raise OutOfOrderSequence in this case.
> > >
> > > 2. Instead of passing through `idempotenceLevel` in the
> ProduceRequest, I
> > > wonder if we should have a field for the minimum required message
> format.
> > > When using enable.idempotence=required, we could set the minimum
> required
> > > version to v2. For enable.idempotence=requested, we could use v0. The
> > > advantage is that we may find other uses for a more general field in
> the
> > > future. Alternatively, maybe we really should be returning the message
> > > format version of each topic in the TopicMetadata response. A nice
> bonus
> > of
> > > doing so is that it gives the producer the ability to craft the right
> > > format version ahead of time and avoid the need for conversion on the
> > > broker.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta <ap...@confluent.io>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > In the discussion of KIP-185 (enable idempotence by default), we
> > > discovered
> > > > some shortcomings of the existing idempotent producer implementation.
> > > > Fixing these issues requires changes to the ProduceRequest and
> > > > ProduceResponse protocols as well as changes to the values of the
> > > > 'enable.idempotence' producer config.
> > > >
> > > > Hence, I split off those changes into another KIP so as to decouple
> the
> > > two
> > > > issues. Please have a look at my follow up KIP below:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> > > >
> > > > KIP-185 depends on KIP-192, and I hope to make progress on the latter
> > > > independently.
> > > >
> > > > Thanks,
> > > > Apurva
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
Posted by Ted Yu <yu...@gmail.com>.
For enable.idempotence=safe, it seems giving user impression that idempotence
would be safe.
However, since it really means best effort, the 'safety' is debatable.
Why not just call the new mode besteffort ?
Cheers
On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma <is...@juma.me.uk> wrote:
> If we add the message format version (a topic config) in the response of
> TopicMetadata, we should consider adding the max message bytes as well.
> That would allow us to later improve the implementation of KIP-126 to split
> the batch _before_ sending.
>
> Ismael
>
> On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > The proposal looks good. Two minor comments:
> >
> > 1. Can we call out how we handle the duplicate case? This is a change in
> > behavior since we currently raise OutOfOrderSequence in this case.
> >
> > 2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I
> > wonder if we should have a field for the minimum required message format.
> > When using enable.idempotence=required, we could set the minimum required
> > version to v2. For enable.idempotence=requested, we could use v0. The
> > advantage is that we may find other uses for a more general field in the
> > future. Alternatively, maybe we really should be returning the message
> > format version of each topic in the TopicMetadata response. A nice bonus
> of
> > doing so is that it gives the producer the ability to craft the right
> > format version ahead of time and avoid the need for conversion on the
> > broker.
> >
> > Thanks,
> > Jason
> >
> > On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta <ap...@confluent.io>
> wrote:
> >
> > > Hi,
> > >
> > > In the discussion of KIP-185 (enable idempotence by default), we
> > discovered
> > > some shortcomings of the existing idempotent producer implementation.
> > > Fixing these issues requires changes to the ProduceRequest and
> > > ProduceResponse protocols as well as changes to the values of the
> > > 'enable.idempotence' producer config.
> > >
> > > Hence, I split off those changes into another KIP so as to decouple the
> > two
> > > issues. Please have a look at my follow up KIP below:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> > >
> > > KIP-185 depends on KIP-192, and I hope to make progress on the latter
> > > independently.
> > >
> > > Thanks,
> > > Apurva
> > >
> >
>
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
Posted by Ismael Juma <is...@juma.me.uk>.
If we add the message format version (a topic config) in the response of
TopicMetadata, we should consider adding the max message bytes as well.
That would allow us to later improve the implementation of KIP-126 to split
the batch _before_ sending.
Ismael
On Tue, Sep 5, 2017 at 7:17 PM, Jason Gustafson <ja...@confluent.io> wrote:
> The proposal looks good. Two minor comments:
>
> 1. Can we call out how we handle the duplicate case? This is a change in
> behavior since we currently raise OutOfOrderSequence in this case.
>
> 2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I
> wonder if we should have a field for the minimum required message format.
> When using enable.idempotence=required, we could set the minimum required
> version to v2. For enable.idempotence=requested, we could use v0. The
> advantage is that we may find other uses for a more general field in the
> future. Alternatively, maybe we really should be returning the message
> format version of each topic in the TopicMetadata response. A nice bonus of
> doing so is that it gives the producer the ability to craft the right
> format version ahead of time and avoid the need for conversion on the
> broker.
>
> Thanks,
> Jason
>
> On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta <ap...@confluent.io> wrote:
>
> > Hi,
> >
> > In the discussion of KIP-185 (enable idempotence by default), we
> discovered
> > some shortcomings of the existing idempotent producer implementation.
> > Fixing these issues requires changes to the ProduceRequest and
> > ProduceResponse protocols as well as changes to the values of the
> > 'enable.idempotence' producer config.
> >
> > Hence, I split off those changes into another KIP so as to decouple the
> two
> > issues. Please have a look at my follow up KIP below:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> >
> > KIP-185 depends on KIP-192, and I hope to make progress on the latter
> > independently.
> >
> > Thanks,
> > Apurva
> >
>
Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled
Posted by Apurva Mehta <ap...@confluent.io>.
Hi Jason,
Thanks for the comments:
1. I have also updated the KIP to indicate that the
DuplicateSequenceException could be the new error code returned in the
producer callback as a result of tightening up the semantics of the
OutOfOrderSequenceException.
2. I think returning the message format version in the TopicMetadata of
the MetadatResponse is a great idea. It is general enough, and could be put
to other uses in the future. Also, it is more efficient to send the right
message format to the broker to begin with, so that's another thing going
for this solution. I have updated the KIP to reflect this change.
Tom: I have update the KIP so that the 'safe' value of 'enable.idempotence'
is converted to 'requested' everywhere.
Thanks,
Apurva
On Tue, Sep 5, 2017 at 11:17 AM, Jason Gustafson <ja...@confluent.io> wrote:
> The proposal looks good. Two minor comments:
>
> 1. Can we call out how we handle the duplicate case? This is a change in
> behavior since we currently raise OutOfOrderSequence in this case.
>
> 2. Instead of passing through `idempotenceLevel` in the ProduceRequest, I
> wonder if we should have a field for the minimum required message format.
> When using enable.idempotence=required, we could set the minimum required
> version to v2. For enable.idempotence=requested, we could use v0. The
> advantage is that we may find other uses for a more general field in the
> future. Alternatively, maybe we really should be returning the message
> format version of each topic in the TopicMetadata response. A nice bonus of
> doing so is that it gives the producer the ability to craft the right
> format version ahead of time and avoid the need for conversion on the
> broker.
>
> Thanks,
> Jason
>
> On Tue, Aug 29, 2017 at 4:32 PM, Apurva Mehta <ap...@confluent.io> wrote:
>
> > Hi,
> >
> > In the discussion of KIP-185 (enable idempotence by default), we
> discovered
> > some shortcomings of the existing idempotent producer implementation.
> > Fixing these issues requires changes to the ProduceRequest and
> > ProduceResponse protocols as well as changes to the values of the
> > 'enable.idempotence' producer config.
> >
> > Hence, I split off those changes into another KIP so as to decouple the
> two
> > issues. Please have a look at my follow up KIP below:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 192+%3A+Provide+cleaner+semantics+when+idempotence+is+enabled
> >
> > KIP-185 depends on KIP-192, and I hope to make progress on the latter
> > independently.
> >
> > Thanks,
> > Apurva
> >
>