You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Mickael Maison <mi...@gmail.com> on 2019/02/11 11:51:53 UTC
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Bumping this thread as it's been a couple of weeks.
On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <mi...@gmail.com> wrote:
>
> Thanks Ismael for the feedback. I think your point has 2 parts:
> - Having the reset functionality in the AdminClient:
> The fact we have a command line tool illustrate that this operation is
> relatively common. I seems valuable to be able to perform this
> operation directly via a proper API in addition of the CLI tool.
>
> - Sending an OffsetCommit directly instead of relying on KafkaConsumer:
> The KafkaConsumer requires a lot of stuff to commit offsets. Its group
> cannot change so you need to start a new Consumer every time, that
> creates new connections and overal sends more requests. Also there are
> already a bunch of AdminClient APIs that have logic very close to
> what needs to be done to send a commit request, keeping the code small
> and consistent.
>
> I've updated the KIP with these details and moved the 2nd part to
> "Proposed changes" as it's more an implementation detail.
>
> I hope this answers your question
>
> On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com> wrote:
> >
> > The KIP doesn't discuss the option of using KafkaConsumer directly as far
> > as I can tell. We have tried to avoid having the same functionality in
> > multiple clients so it would be good to explain why this is necessary here
> > (not saying it isn't).
> >
> > Ismael
> >
> > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <mickael.maison@gmail.com
> > wrote:
> >
> > > Thanks Ryanne for the feedback, all suggestions sounded good, I've
> > > updated the KIP accordingly.
> > >
> > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <ry...@gmail.com>
> > > wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > But I suggest:
> > > >
> > > > - drop "get" from getOffset, getTimestamp.
> > > >
> > > > - add to the motivation section why this is better than constructing a
> > > > KafkaConsumer and using seek(), commit() etc.
> > > >
> > > > - add some rejected alternatives.
> > > >
> > > > Ryanne
> > > >
> > > >
> > > >
> > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <dongjin@apache.org wrote:
> > > >
> > > > > We have +4 non-binding for this vote. Is there any committer who is
> > > > > interested in this issue?
> > > > >
> > > > > Thanks,
> > > > > Dongjin
> > > > >
> > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > andrew_schofield@live.com>
> > > > > wrote:
> > > > >
> > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > >
> > > > > > On 21/01/2019, 12:45, "Eno Thereska" <en...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > +1 (non binding). Thanks.
> > > > > >
> > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > mickael.maison@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Bumping this thread. Considering this KIP is relatively straigh
> > > > > > > forward, can we get some votes or feedback if you think it's
> > > not?
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > edocomar@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > > Thanks Mickael!
> > > > > > > >
> > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > pkleindl@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding)
> > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > Best regards
> > > > > > > > > Patrik
> > > > > > > > >
> > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael Maison <
> > > > > > > mickael.maison@gmail.com
> > > > > > > > > >:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > "When the people fear their government, there is tyranny;
> > > when
> > > > > the
> > > > > > > > government fears the people, there is liberty." [Thomas
> > > > > Jefferson]
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > *Dongjin Lee*
> > > > >
> > > > > *A hitchhiker in the mathematical world.*
> > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > <https://github.com/dongjinleekr>linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > speakerdeck.com/dongjin
> > > > > <https://speakerdeck.com/dongjin>*
> > > > >
> > > > >
> > >
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Mickael Maison <mi...@gmail.com>.
Hi Jason,
4. Yes, I've added that in the KIP
5. I've updated the KIP to use `alterConsumerGroupOffsets` so it's
consistent with other APIs
Thank you for the feedback
On Wed, Aug 7, 2019 at 7:08 PM Jason Gustafson <ja...@confluent.io> wrote:
>
> Thanks Mickael, +1 from me.
>
> Just a couple more comments:
>
> > Yes mapping it to a friendlier error would be nice but I'm not sure
> if we can do that because UNKNOWN_MEMBER_ID is also returned if the
> group is dead.
>
> I think the logic was chanced recently to return
> `COORDINATOR_NOT_AVAILABLE` if the group is Dead, so I don't think this is
> a problem, at least not for new versions. One thing to keep in mind is that
> the Dead state is just a transient state while we're in the process of
> unloading the group. It should be rare that we hit it. For old brokers
> which still return UNKNOWN_MEMBER_ID in this state, I don't think it would
> be so bad to return a NotEmptyGroup error. Basically we have to throw
> something in this case, so we may as well throw something which indicates
> the most likely problem.
>
> 4. I assume the default isolation level for listOffsets would be
> read_uncommitted?
>
> 5. I don't feel too strongly about it, but other admin APIs have tended to
> use "alter" for updates (e.g. `alterConfigs` and `alterReplicaLogDirs`).
> You might consider `alterConsumerGroupOffsets` over
> `resetConsumerGroupOffsets`.
>
> -Jason
>
> On Wed, Aug 7, 2019 at 10:15 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi Jason,
> >
> > Thanks for the feedback
> >
> > 1. Yes listOffsets() should be able to retrieve earliest, latest and
> > by-timestamp offsets.
> > I agree, it's better to avoid exposing magic values. I've updated the
> > KIP to use an OffsetSpec object as suggested.
> >
> > 2. Yes let's expose the leader epoch in ListOffsetsResultInfo.
> >
> > 3. Yes mapping it to a friendlier error would be nice but I'm not sure
> > if we can do that because UNKNOWN_MEMBER_ID is also returned if the
> > group is dead.
> >
> > On Tue, Aug 6, 2019 at 6:38 PM Jason Gustafson <ja...@confluent.io> wrote:
> > >
> > > Thanks for the KIP. This makes sense to me. Just a couple small comments:
> > >
> > > 1. Can the listOffsets API be used to get the start and end offsets? In
> > the
> > > consumer, we use separate APIs for this: `beginningOffsets` and
> > > `endOffsets` to avoid the need for sentinels. An alternative would be to
> > > introduce an `OffsetSpec` (or maybe `OffsetQuery`) object to customize
> > the
> > > query. For example:
> > >
> > > public ListOffsetsResult listOffsets(Map<TopicPartition, OffsetSpec>
> > > partitionOffsetSpecs)
> > >
> > > The benefit is that we can avoid sentinel values and we have an extension
> > > point for additional query options in the future. What do you think?
> > >
> > > 2. The ListOffset response includes the leader epoch corresponding to the
> > > offset that was found. This is useful for finer-grained reasoning about
> > the
> > > log. We expose this in the consumer in the OffsetAndTimestamp object
> > which
> > > is returned from `offsetsForTimes`. Does it make sense to add this to
> > > `ListOffsetsResultInfo` as well?
> > >
> > > 3. If the group is still active, the call to reset offsets will fail.
> > > Currently this would result in an UNKNOWN_MEMBER_ID error. I think it
> > would
> > > make sense to map this exception to a friendlier error before raising to
> > > the user. For example, `NonEmptyGroupException` or something like that.
> > >
> > > -Jason
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Aug 6, 2019 at 9:33 AM Mickael Maison <mi...@gmail.com>
> > > wrote:
> > >
> > > > Hi Colin,
> > > >
> > > > Thank you for taking a look!
> > > > I agree, being able to set consumer group offsets via the AdminClient
> > > > would be really useful, hence I created this KIP.
> > > >
> > > > With the total absence of binding votes, I guessed I needed to make
> > > > some changes. Do you mean you preferred the previous naming
> > > > (commitConsumerGroupOffsets) over "resetConsumerGroupOffsets"?
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Aug 5, 2019 at 8:26 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > >
> > > > > I think it would be useful to have this in AdminClient. Especially
> > if
> > > > we implement KIP-496: Administrative API to delete consumer offsets.
> > It
> > > > would be odd to have a way to delete consumer offsets in AdminClient,
> > but
> > > > not to create them. What do you think?
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Sun, Aug 4, 2019, at 09:27, Mickael Maison wrote:
> > > > > > Hi,
> > > > > >
> > > > > > In an attempt to unblock this KIP, I've made some adjustments:
> > > > > > I've renamed the commitConsumerGroupOffsets() methods to
> > > > > > resetConsumerGroupOffsets() to reduce confusion. That should better
> > > > > > highlight the differences with the regular commit() operation from
> > the
> > > > > > Consumer API. I've also added some details to the motivation
> > section.
> > > > > >
> > > > > > So we have +5 non binding votes and 0 binding votes
> > > > > >
> > > > > > On Mon, Mar 25, 2019 at 1:10 PM Mickael Maison <
> > > > mickael.maison@gmail.com> wrote:
> > > > > > >
> > > > > > > Bumping this thread once again
> > > > > > >
> > > > > > > Ismael, have I answered your questions?
> > > > > > > While this has received a few non-binding +1s, no committers have
> > > > > > > voted yet. If you have concerns or questions, please let me know.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > > > > <mi...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Bumping this thread as it's been a couple of weeks.
> > > > > > > >
> > > > > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > > mickael.maison@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Thanks Ismael for the feedback. I think your point has 2
> > parts:
> > > > > > > > > - Having the reset functionality in the AdminClient:
> > > > > > > > > The fact we have a command line tool illustrate that this
> > > > operation is
> > > > > > > > > relatively common. I seems valuable to be able to perform
> > this
> > > > > > > > > operation directly via a proper API in addition of the CLI
> > tool.
> > > > > > > > >
> > > > > > > > > - Sending an OffsetCommit directly instead of relying on
> > > > KafkaConsumer:
> > > > > > > > > The KafkaConsumer requires a lot of stuff to commit offsets.
> > Its
> > > > group
> > > > > > > > > cannot change so you need to start a new Consumer every time,
> > > > that
> > > > > > > > > creates new connections and overal sends more requests. Also
> > > > there are
> > > > > > > > > already a bunch of AdminClient APIs that have logic very
> > close
> > > > to
> > > > > > > > > what needs to be done to send a commit request, keeping the
> > code
> > > > small
> > > > > > > > > and consistent.
> > > > > > > > >
> > > > > > > > > I've updated the KIP with these details and moved the 2nd
> > part to
> > > > > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > > > > >
> > > > > > > > > I hope this answers your question
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
> > ismaelj@gmail.com>
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > The KIP doesn't discuss the option of using KafkaConsumer
> > > > directly as far
> > > > > > > > > > as I can tell. We have tried to avoid having the same
> > > > functionality in
> > > > > > > > > > multiple clients so it would be good to explain why this is
> > > > necessary here
> > > > > > > > > > (not saying it isn't).
> > > > > > > > > >
> > > > > > > > > > Ismael
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > > > mickael.maison@gmail.com
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks Ryanne for the feedback, all suggestions sounded
> > > > good, I've
> > > > > > > > > > > updated the KIP accordingly.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > > > ryannedolan@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > >
> > > > > > > > > > > > But I suggest:
> > > > > > > > > > > >
> > > > > > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > > > > > >
> > > > > > > > > > > > - add to the motivation section why this is better than
> > > > constructing a
> > > > > > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > > > > > >
> > > > > > > > > > > > - add some rejected alternatives.
> > > > > > > > > > > >
> > > > > > > > > > > > Ryanne
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> > > > dongjin@apache.org wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We have +4 non-binding for this vote. Is there any
> > > > committer who is
> > > > > > > > > > > > > interested in this issue?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Dongjin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> > > > eno.thereska@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael
> > Maison <
> > > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Bumping this thread. Considering this KIP is
> > > > relatively straigh
> > > > > > > > > > > > > > > forward, can we get some votes or feedback if
> > > > you think it's
> > > > > > > > > > > not?
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar
> > <
> > > > > > > > > > > edocomar@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik
> > Kleindl <
> > > > > > > > > > > pkleindl@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > > > > > > Best regards
> > > > > > > > > > > > > > > > > Patrik
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael
> > > > Maison <
> > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > >
> > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > "When the people fear their government,
> > there
> > > > is tyranny;
> > > > > > > > > > > when
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > government fears the people, there is
> > > > liberty." [Thomas
> > > > > > > > > > > > > Jefferson]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > *Dongjin Lee*
> > > > > > > > > > > > >
> > > > > > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > > > > *github: <http://goog_969573159/>
> > > > github.com/dongjinleekr
> > > > > > > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr
> > >speakerdeck:
> > > > > > > > > > > > > speakerdeck.com/dongjin
> > > > > > > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > >
> > > >
> >
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Jason Gustafson <ja...@confluent.io>.
Thanks Mickael, +1 from me.
Just a couple more comments:
> Yes mapping it to a friendlier error would be nice but I'm not sure
if we can do that because UNKNOWN_MEMBER_ID is also returned if the
group is dead.
I think the logic was chanced recently to return
`COORDINATOR_NOT_AVAILABLE` if the group is Dead, so I don't think this is
a problem, at least not for new versions. One thing to keep in mind is that
the Dead state is just a transient state while we're in the process of
unloading the group. It should be rare that we hit it. For old brokers
which still return UNKNOWN_MEMBER_ID in this state, I don't think it would
be so bad to return a NotEmptyGroup error. Basically we have to throw
something in this case, so we may as well throw something which indicates
the most likely problem.
4. I assume the default isolation level for listOffsets would be
read_uncommitted?
5. I don't feel too strongly about it, but other admin APIs have tended to
use "alter" for updates (e.g. `alterConfigs` and `alterReplicaLogDirs`).
You might consider `alterConsumerGroupOffsets` over
`resetConsumerGroupOffsets`.
-Jason
On Wed, Aug 7, 2019 at 10:15 AM Mickael Maison <mi...@gmail.com>
wrote:
> Hi Jason,
>
> Thanks for the feedback
>
> 1. Yes listOffsets() should be able to retrieve earliest, latest and
> by-timestamp offsets.
> I agree, it's better to avoid exposing magic values. I've updated the
> KIP to use an OffsetSpec object as suggested.
>
> 2. Yes let's expose the leader epoch in ListOffsetsResultInfo.
>
> 3. Yes mapping it to a friendlier error would be nice but I'm not sure
> if we can do that because UNKNOWN_MEMBER_ID is also returned if the
> group is dead.
>
> On Tue, Aug 6, 2019 at 6:38 PM Jason Gustafson <ja...@confluent.io> wrote:
> >
> > Thanks for the KIP. This makes sense to me. Just a couple small comments:
> >
> > 1. Can the listOffsets API be used to get the start and end offsets? In
> the
> > consumer, we use separate APIs for this: `beginningOffsets` and
> > `endOffsets` to avoid the need for sentinels. An alternative would be to
> > introduce an `OffsetSpec` (or maybe `OffsetQuery`) object to customize
> the
> > query. For example:
> >
> > public ListOffsetsResult listOffsets(Map<TopicPartition, OffsetSpec>
> > partitionOffsetSpecs)
> >
> > The benefit is that we can avoid sentinel values and we have an extension
> > point for additional query options in the future. What do you think?
> >
> > 2. The ListOffset response includes the leader epoch corresponding to the
> > offset that was found. This is useful for finer-grained reasoning about
> the
> > log. We expose this in the consumer in the OffsetAndTimestamp object
> which
> > is returned from `offsetsForTimes`. Does it make sense to add this to
> > `ListOffsetsResultInfo` as well?
> >
> > 3. If the group is still active, the call to reset offsets will fail.
> > Currently this would result in an UNKNOWN_MEMBER_ID error. I think it
> would
> > make sense to map this exception to a friendlier error before raising to
> > the user. For example, `NonEmptyGroupException` or something like that.
> >
> > -Jason
> >
> >
> >
> >
> >
> > On Tue, Aug 6, 2019 at 9:33 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> >
> > > Hi Colin,
> > >
> > > Thank you for taking a look!
> > > I agree, being able to set consumer group offsets via the AdminClient
> > > would be really useful, hence I created this KIP.
> > >
> > > With the total absence of binding votes, I guessed I needed to make
> > > some changes. Do you mean you preferred the previous naming
> > > (commitConsumerGroupOffsets) over "resetConsumerGroupOffsets"?
> > >
> > > Thanks
> > >
> > > On Mon, Aug 5, 2019 at 8:26 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > >
> > > > I think it would be useful to have this in AdminClient. Especially
> if
> > > we implement KIP-496: Administrative API to delete consumer offsets.
> It
> > > would be odd to have a way to delete consumer offsets in AdminClient,
> but
> > > not to create them. What do you think?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Sun, Aug 4, 2019, at 09:27, Mickael Maison wrote:
> > > > > Hi,
> > > > >
> > > > > In an attempt to unblock this KIP, I've made some adjustments:
> > > > > I've renamed the commitConsumerGroupOffsets() methods to
> > > > > resetConsumerGroupOffsets() to reduce confusion. That should better
> > > > > highlight the differences with the regular commit() operation from
> the
> > > > > Consumer API. I've also added some details to the motivation
> section.
> > > > >
> > > > > So we have +5 non binding votes and 0 binding votes
> > > > >
> > > > > On Mon, Mar 25, 2019 at 1:10 PM Mickael Maison <
> > > mickael.maison@gmail.com> wrote:
> > > > > >
> > > > > > Bumping this thread once again
> > > > > >
> > > > > > Ismael, have I answered your questions?
> > > > > > While this has received a few non-binding +1s, no committers have
> > > > > > voted yet. If you have concerns or questions, please let me know.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > > > <mi...@gmail.com> wrote:
> > > > > > >
> > > > > > > Bumping this thread as it's been a couple of weeks.
> > > > > > >
> > > > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > mickael.maison@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Thanks Ismael for the feedback. I think your point has 2
> parts:
> > > > > > > > - Having the reset functionality in the AdminClient:
> > > > > > > > The fact we have a command line tool illustrate that this
> > > operation is
> > > > > > > > relatively common. I seems valuable to be able to perform
> this
> > > > > > > > operation directly via a proper API in addition of the CLI
> tool.
> > > > > > > >
> > > > > > > > - Sending an OffsetCommit directly instead of relying on
> > > KafkaConsumer:
> > > > > > > > The KafkaConsumer requires a lot of stuff to commit offsets.
> Its
> > > group
> > > > > > > > cannot change so you need to start a new Consumer every time,
> > > that
> > > > > > > > creates new connections and overal sends more requests. Also
> > > there are
> > > > > > > > already a bunch of AdminClient APIs that have logic very
> close
> > > to
> > > > > > > > what needs to be done to send a commit request, keeping the
> code
> > > small
> > > > > > > > and consistent.
> > > > > > > >
> > > > > > > > I've updated the KIP with these details and moved the 2nd
> part to
> > > > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > > > >
> > > > > > > > I hope this answers your question
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
> ismaelj@gmail.com>
> > > wrote:
> > > > > > > > >
> > > > > > > > > The KIP doesn't discuss the option of using KafkaConsumer
> > > directly as far
> > > > > > > > > as I can tell. We have tried to avoid having the same
> > > functionality in
> > > > > > > > > multiple clients so it would be good to explain why this is
> > > necessary here
> > > > > > > > > (not saying it isn't).
> > > > > > > > >
> > > > > > > > > Ismael
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > > mickael.maison@gmail.com
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks Ryanne for the feedback, all suggestions sounded
> > > good, I've
> > > > > > > > > > updated the KIP accordingly.
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > > ryannedolan@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > >
> > > > > > > > > > > But I suggest:
> > > > > > > > > > >
> > > > > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > > > > >
> > > > > > > > > > > - add to the motivation section why this is better than
> > > constructing a
> > > > > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > > > > >
> > > > > > > > > > > - add some rejected alternatives.
> > > > > > > > > > >
> > > > > > > > > > > Ryanne
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> > > dongjin@apache.org wrote:
> > > > > > > > > > >
> > > > > > > > > > > > We have +4 non-binding for this vote. Is there any
> > > committer who is
> > > > > > > > > > > > interested in this issue?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Dongjin
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> > > eno.thereska@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael
> Maison <
> > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Bumping this thread. Considering this KIP is
> > > relatively straigh
> > > > > > > > > > > > > > forward, can we get some votes or feedback if
> > > you think it's
> > > > > > > > > > not?
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar
> <
> > > > > > > > > > edocomar@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik
> Kleindl <
> > > > > > > > > > pkleindl@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > > > > > Best regards
> > > > > > > > > > > > > > > > Patrik
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael
> > > Maison <
> > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > >
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > "When the people fear their government,
> there
> > > is tyranny;
> > > > > > > > > > when
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > government fears the people, there is
> > > liberty." [Thomas
> > > > > > > > > > > > Jefferson]
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > *Dongjin Lee*
> > > > > > > > > > > >
> > > > > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > > > *github: <http://goog_969573159/>
> > > github.com/dongjinleekr
> > > > > > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr
> >speakerdeck:
> > > > > > > > > > > > speakerdeck.com/dongjin
> > > > > > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > >
> > >
>
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Mickael Maison <mi...@gmail.com>.
Hi Jason,
Thanks for the feedback
1. Yes listOffsets() should be able to retrieve earliest, latest and
by-timestamp offsets.
I agree, it's better to avoid exposing magic values. I've updated the
KIP to use an OffsetSpec object as suggested.
2. Yes let's expose the leader epoch in ListOffsetsResultInfo.
3. Yes mapping it to a friendlier error would be nice but I'm not sure
if we can do that because UNKNOWN_MEMBER_ID is also returned if the
group is dead.
On Tue, Aug 6, 2019 at 6:38 PM Jason Gustafson <ja...@confluent.io> wrote:
>
> Thanks for the KIP. This makes sense to me. Just a couple small comments:
>
> 1. Can the listOffsets API be used to get the start and end offsets? In the
> consumer, we use separate APIs for this: `beginningOffsets` and
> `endOffsets` to avoid the need for sentinels. An alternative would be to
> introduce an `OffsetSpec` (or maybe `OffsetQuery`) object to customize the
> query. For example:
>
> public ListOffsetsResult listOffsets(Map<TopicPartition, OffsetSpec>
> partitionOffsetSpecs)
>
> The benefit is that we can avoid sentinel values and we have an extension
> point for additional query options in the future. What do you think?
>
> 2. The ListOffset response includes the leader epoch corresponding to the
> offset that was found. This is useful for finer-grained reasoning about the
> log. We expose this in the consumer in the OffsetAndTimestamp object which
> is returned from `offsetsForTimes`. Does it make sense to add this to
> `ListOffsetsResultInfo` as well?
>
> 3. If the group is still active, the call to reset offsets will fail.
> Currently this would result in an UNKNOWN_MEMBER_ID error. I think it would
> make sense to map this exception to a friendlier error before raising to
> the user. For example, `NonEmptyGroupException` or something like that.
>
> -Jason
>
>
>
>
>
> On Tue, Aug 6, 2019 at 9:33 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi Colin,
> >
> > Thank you for taking a look!
> > I agree, being able to set consumer group offsets via the AdminClient
> > would be really useful, hence I created this KIP.
> >
> > With the total absence of binding votes, I guessed I needed to make
> > some changes. Do you mean you preferred the previous naming
> > (commitConsumerGroupOffsets) over "resetConsumerGroupOffsets"?
> >
> > Thanks
> >
> > On Mon, Aug 5, 2019 at 8:26 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > I think it would be useful to have this in AdminClient. Especially if
> > we implement KIP-496: Administrative API to delete consumer offsets. It
> > would be odd to have a way to delete consumer offsets in AdminClient, but
> > not to create them. What do you think?
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Sun, Aug 4, 2019, at 09:27, Mickael Maison wrote:
> > > > Hi,
> > > >
> > > > In an attempt to unblock this KIP, I've made some adjustments:
> > > > I've renamed the commitConsumerGroupOffsets() methods to
> > > > resetConsumerGroupOffsets() to reduce confusion. That should better
> > > > highlight the differences with the regular commit() operation from the
> > > > Consumer API. I've also added some details to the motivation section.
> > > >
> > > > So we have +5 non binding votes and 0 binding votes
> > > >
> > > > On Mon, Mar 25, 2019 at 1:10 PM Mickael Maison <
> > mickael.maison@gmail.com> wrote:
> > > > >
> > > > > Bumping this thread once again
> > > > >
> > > > > Ismael, have I answered your questions?
> > > > > While this has received a few non-binding +1s, no committers have
> > > > > voted yet. If you have concerns or questions, please let me know.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > > <mi...@gmail.com> wrote:
> > > > > >
> > > > > > Bumping this thread as it's been a couple of weeks.
> > > > > >
> > > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > mickael.maison@gmail.com> wrote:
> > > > > > >
> > > > > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > > > > - Having the reset functionality in the AdminClient:
> > > > > > > The fact we have a command line tool illustrate that this
> > operation is
> > > > > > > relatively common. I seems valuable to be able to perform this
> > > > > > > operation directly via a proper API in addition of the CLI tool.
> > > > > > >
> > > > > > > - Sending an OffsetCommit directly instead of relying on
> > KafkaConsumer:
> > > > > > > The KafkaConsumer requires a lot of stuff to commit offsets. Its
> > group
> > > > > > > cannot change so you need to start a new Consumer every time,
> > that
> > > > > > > creates new connections and overal sends more requests. Also
> > there are
> > > > > > > already a bunch of AdminClient APIs that have logic very close
> > to
> > > > > > > what needs to be done to send a commit request, keeping the code
> > small
> > > > > > > and consistent.
> > > > > > >
> > > > > > > I've updated the KIP with these details and moved the 2nd part to
> > > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > > >
> > > > > > > I hope this answers your question
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com>
> > wrote:
> > > > > > > >
> > > > > > > > The KIP doesn't discuss the option of using KafkaConsumer
> > directly as far
> > > > > > > > as I can tell. We have tried to avoid having the same
> > functionality in
> > > > > > > > multiple clients so it would be good to explain why this is
> > necessary here
> > > > > > > > (not saying it isn't).
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > mickael.maison@gmail.com
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Ryanne for the feedback, all suggestions sounded
> > good, I've
> > > > > > > > > updated the KIP accordingly.
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > ryannedolan@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1 (non-binding)
> > > > > > > > > >
> > > > > > > > > > But I suggest:
> > > > > > > > > >
> > > > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > > > >
> > > > > > > > > > - add to the motivation section why this is better than
> > constructing a
> > > > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > > > >
> > > > > > > > > > - add some rejected alternatives.
> > > > > > > > > >
> > > > > > > > > > Ryanne
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> > dongjin@apache.org wrote:
> > > > > > > > > >
> > > > > > > > > > > We have +4 non-binding for this vote. Is there any
> > committer who is
> > > > > > > > > > > interested in this issue?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Dongjin
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> > eno.thereska@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Bumping this thread. Considering this KIP is
> > relatively straigh
> > > > > > > > > > > > > forward, can we get some votes or feedback if
> > you think it's
> > > > > > > > > not?
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > > > > > edocomar@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > > > > > pkleindl@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > > > > Best regards
> > > > > > > > > > > > > > > Patrik
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael
> > Maison <
> > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > "When the people fear their government, there
> > is tyranny;
> > > > > > > > > when
> > > > > > > > > > > the
> > > > > > > > > > > > > > government fears the people, there is
> > liberty." [Thomas
> > > > > > > > > > > Jefferson]
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *Dongjin Lee*
> > > > > > > > > > >
> > > > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > > *github: <http://goog_969573159/>
> > github.com/dongjinleekr
> > > > > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > > > > > > speakerdeck.com/dongjin
> > > > > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > >
> >
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Jason Gustafson <ja...@confluent.io>.
Thanks for the KIP. This makes sense to me. Just a couple small comments:
1. Can the listOffsets API be used to get the start and end offsets? In the
consumer, we use separate APIs for this: `beginningOffsets` and
`endOffsets` to avoid the need for sentinels. An alternative would be to
introduce an `OffsetSpec` (or maybe `OffsetQuery`) object to customize the
query. For example:
public ListOffsetsResult listOffsets(Map<TopicPartition, OffsetSpec>
partitionOffsetSpecs)
The benefit is that we can avoid sentinel values and we have an extension
point for additional query options in the future. What do you think?
2. The ListOffset response includes the leader epoch corresponding to the
offset that was found. This is useful for finer-grained reasoning about the
log. We expose this in the consumer in the OffsetAndTimestamp object which
is returned from `offsetsForTimes`. Does it make sense to add this to
`ListOffsetsResultInfo` as well?
3. If the group is still active, the call to reset offsets will fail.
Currently this would result in an UNKNOWN_MEMBER_ID error. I think it would
make sense to map this exception to a friendlier error before raising to
the user. For example, `NonEmptyGroupException` or something like that.
-Jason
On Tue, Aug 6, 2019 at 9:33 AM Mickael Maison <mi...@gmail.com>
wrote:
> Hi Colin,
>
> Thank you for taking a look!
> I agree, being able to set consumer group offsets via the AdminClient
> would be really useful, hence I created this KIP.
>
> With the total absence of binding votes, I guessed I needed to make
> some changes. Do you mean you preferred the previous naming
> (commitConsumerGroupOffsets) over "resetConsumerGroupOffsets"?
>
> Thanks
>
> On Mon, Aug 5, 2019 at 8:26 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > I think it would be useful to have this in AdminClient. Especially if
> we implement KIP-496: Administrative API to delete consumer offsets. It
> would be odd to have a way to delete consumer offsets in AdminClient, but
> not to create them. What do you think?
> >
> > best,
> > Colin
> >
> >
> > On Sun, Aug 4, 2019, at 09:27, Mickael Maison wrote:
> > > Hi,
> > >
> > > In an attempt to unblock this KIP, I've made some adjustments:
> > > I've renamed the commitConsumerGroupOffsets() methods to
> > > resetConsumerGroupOffsets() to reduce confusion. That should better
> > > highlight the differences with the regular commit() operation from the
> > > Consumer API. I've also added some details to the motivation section.
> > >
> > > So we have +5 non binding votes and 0 binding votes
> > >
> > > On Mon, Mar 25, 2019 at 1:10 PM Mickael Maison <
> mickael.maison@gmail.com> wrote:
> > > >
> > > > Bumping this thread once again
> > > >
> > > > Ismael, have I answered your questions?
> > > > While this has received a few non-binding +1s, no committers have
> > > > voted yet. If you have concerns or questions, please let me know.
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > <mi...@gmail.com> wrote:
> > > > >
> > > > > Bumping this thread as it's been a couple of weeks.
> > > > >
> > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> mickael.maison@gmail.com> wrote:
> > > > > >
> > > > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > > > - Having the reset functionality in the AdminClient:
> > > > > > The fact we have a command line tool illustrate that this
> operation is
> > > > > > relatively common. I seems valuable to be able to perform this
> > > > > > operation directly via a proper API in addition of the CLI tool.
> > > > > >
> > > > > > - Sending an OffsetCommit directly instead of relying on
> KafkaConsumer:
> > > > > > The KafkaConsumer requires a lot of stuff to commit offsets. Its
> group
> > > > > > cannot change so you need to start a new Consumer every time,
> that
> > > > > > creates new connections and overal sends more requests. Also
> there are
> > > > > > already a bunch of AdminClient APIs that have logic very close
> to
> > > > > > what needs to be done to send a commit request, keeping the code
> small
> > > > > > and consistent.
> > > > > >
> > > > > > I've updated the KIP with these details and moved the 2nd part to
> > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > >
> > > > > > I hope this answers your question
> > > > > >
> > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com>
> wrote:
> > > > > > >
> > > > > > > The KIP doesn't discuss the option of using KafkaConsumer
> directly as far
> > > > > > > as I can tell. We have tried to avoid having the same
> functionality in
> > > > > > > multiple clients so it would be good to explain why this is
> necessary here
> > > > > > > (not saying it isn't).
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> mickael.maison@gmail.com
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Ryanne for the feedback, all suggestions sounded
> good, I've
> > > > > > > > updated the KIP accordingly.
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> ryannedolan@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > +1 (non-binding)
> > > > > > > > >
> > > > > > > > > But I suggest:
> > > > > > > > >
> > > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > > >
> > > > > > > > > - add to the motivation section why this is better than
> constructing a
> > > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > > >
> > > > > > > > > - add some rejected alternatives.
> > > > > > > > >
> > > > > > > > > Ryanne
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> dongjin@apache.org wrote:
> > > > > > > > >
> > > > > > > > > > We have +4 non-binding for this vote. Is there any
> committer who is
> > > > > > > > > > interested in this issue?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Dongjin
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > > > >
> > > > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> eno.thereska@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Bumping this thread. Considering this KIP is
> relatively straigh
> > > > > > > > > > > > forward, can we get some votes or feedback if
> you think it's
> > > > > > > > not?
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > > > > edocomar@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > > > > pkleindl@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > > > Best regards
> > > > > > > > > > > > > > Patrik
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael
> Maison <
> > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > "When the people fear their government, there
> is tyranny;
> > > > > > > > when
> > > > > > > > > > the
> > > > > > > > > > > > > government fears the people, there is
> liberty." [Thomas
> > > > > > > > > > Jefferson]
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Dongjin Lee*
> > > > > > > > > >
> > > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > *github: <http://goog_969573159/>
> github.com/dongjinleekr
> > > > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > > > > > speakerdeck.com/dongjin
> > > > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > >
>
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Mickael Maison <mi...@gmail.com>.
Hi Colin,
Thank you for taking a look!
I agree, being able to set consumer group offsets via the AdminClient
would be really useful, hence I created this KIP.
With the total absence of binding votes, I guessed I needed to make
some changes. Do you mean you preferred the previous naming
(commitConsumerGroupOffsets) over "resetConsumerGroupOffsets"?
Thanks
On Mon, Aug 5, 2019 at 8:26 PM Colin McCabe <cm...@apache.org> wrote:
>
> I think it would be useful to have this in AdminClient. Especially if we implement KIP-496: Administrative API to delete consumer offsets. It would be odd to have a way to delete consumer offsets in AdminClient, but not to create them. What do you think?
>
> best,
> Colin
>
>
> On Sun, Aug 4, 2019, at 09:27, Mickael Maison wrote:
> > Hi,
> >
> > In an attempt to unblock this KIP, I've made some adjustments:
> > I've renamed the commitConsumerGroupOffsets() methods to
> > resetConsumerGroupOffsets() to reduce confusion. That should better
> > highlight the differences with the regular commit() operation from the
> > Consumer API. I've also added some details to the motivation section.
> >
> > So we have +5 non binding votes and 0 binding votes
> >
> > On Mon, Mar 25, 2019 at 1:10 PM Mickael Maison <mi...@gmail.com> wrote:
> > >
> > > Bumping this thread once again
> > >
> > > Ismael, have I answered your questions?
> > > While this has received a few non-binding +1s, no committers have
> > > voted yet. If you have concerns or questions, please let me know.
> > >
> > > Thanks
> > >
> > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > <mi...@gmail.com> wrote:
> > > >
> > > > Bumping this thread as it's been a couple of weeks.
> > > >
> > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <mi...@gmail.com> wrote:
> > > > >
> > > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > > - Having the reset functionality in the AdminClient:
> > > > > The fact we have a command line tool illustrate that this operation is
> > > > > relatively common. I seems valuable to be able to perform this
> > > > > operation directly via a proper API in addition of the CLI tool.
> > > > >
> > > > > - Sending an OffsetCommit directly instead of relying on KafkaConsumer:
> > > > > The KafkaConsumer requires a lot of stuff to commit offsets. Its group
> > > > > cannot change so you need to start a new Consumer every time, that
> > > > > creates new connections and overal sends more requests. Also there are
> > > > > already a bunch of AdminClient APIs that have logic very close to
> > > > > what needs to be done to send a commit request, keeping the code small
> > > > > and consistent.
> > > > >
> > > > > I've updated the KIP with these details and moved the 2nd part to
> > > > > "Proposed changes" as it's more an implementation detail.
> > > > >
> > > > > I hope this answers your question
> > > > >
> > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com> wrote:
> > > > > >
> > > > > > The KIP doesn't discuss the option of using KafkaConsumer directly as far
> > > > > > as I can tell. We have tried to avoid having the same functionality in
> > > > > > multiple clients so it would be good to explain why this is necessary here
> > > > > > (not saying it isn't).
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <mickael.maison@gmail.com
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Ryanne for the feedback, all suggestions sounded good, I've
> > > > > > > updated the KIP accordingly.
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <ry...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > But I suggest:
> > > > > > > >
> > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > >
> > > > > > > > - add to the motivation section why this is better than constructing a
> > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > >
> > > > > > > > - add some rejected alternatives.
> > > > > > > >
> > > > > > > > Ryanne
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <dongjin@apache.org wrote:
> > > > > > > >
> > > > > > > > > We have +4 non-binding for this vote. Is there any committer who is
> > > > > > > > > interested in this issue?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dongjin
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > > >
> > > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <en...@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Bumping this thread. Considering this KIP is relatively straigh
> > > > > > > > > > > forward, can we get some votes or feedback if you think it's
> > > > > > > not?
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > > > edocomar@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > > > pkleindl@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > > Best regards
> > > > > > > > > > > > > Patrik
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > "When the people fear their government, there is tyranny;
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > > > government fears the people, there is liberty." [Thomas
> > > > > > > > > Jefferson]
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Dongjin Lee*
> > > > > > > > >
> > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > > > > speakerdeck.com/dongjin
> > > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> >
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Colin McCabe <cm...@apache.org>.
I think it would be useful to have this in AdminClient. Especially if we implement KIP-496: Administrative API to delete consumer offsets. It would be odd to have a way to delete consumer offsets in AdminClient, but not to create them. What do you think?
best,
Colin
On Sun, Aug 4, 2019, at 09:27, Mickael Maison wrote:
> Hi,
>
> In an attempt to unblock this KIP, I've made some adjustments:
> I've renamed the commitConsumerGroupOffsets() methods to
> resetConsumerGroupOffsets() to reduce confusion. That should better
> highlight the differences with the regular commit() operation from the
> Consumer API. I've also added some details to the motivation section.
>
> So we have +5 non binding votes and 0 binding votes
>
> On Mon, Mar 25, 2019 at 1:10 PM Mickael Maison <mi...@gmail.com> wrote:
> >
> > Bumping this thread once again
> >
> > Ismael, have I answered your questions?
> > While this has received a few non-binding +1s, no committers have
> > voted yet. If you have concerns or questions, please let me know.
> >
> > Thanks
> >
> > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > <mi...@gmail.com> wrote:
> > >
> > > Bumping this thread as it's been a couple of weeks.
> > >
> > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <mi...@gmail.com> wrote:
> > > >
> > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > - Having the reset functionality in the AdminClient:
> > > > The fact we have a command line tool illustrate that this operation is
> > > > relatively common. I seems valuable to be able to perform this
> > > > operation directly via a proper API in addition of the CLI tool.
> > > >
> > > > - Sending an OffsetCommit directly instead of relying on KafkaConsumer:
> > > > The KafkaConsumer requires a lot of stuff to commit offsets. Its group
> > > > cannot change so you need to start a new Consumer every time, that
> > > > creates new connections and overal sends more requests. Also there are
> > > > already a bunch of AdminClient APIs that have logic very close to
> > > > what needs to be done to send a commit request, keeping the code small
> > > > and consistent.
> > > >
> > > > I've updated the KIP with these details and moved the 2nd part to
> > > > "Proposed changes" as it's more an implementation detail.
> > > >
> > > > I hope this answers your question
> > > >
> > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com> wrote:
> > > > >
> > > > > The KIP doesn't discuss the option of using KafkaConsumer directly as far
> > > > > as I can tell. We have tried to avoid having the same functionality in
> > > > > multiple clients so it would be good to explain why this is necessary here
> > > > > (not saying it isn't).
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <mickael.maison@gmail.com
> > > > > wrote:
> > > > >
> > > > > > Thanks Ryanne for the feedback, all suggestions sounded good, I've
> > > > > > updated the KIP accordingly.
> > > > > >
> > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <ry...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > But I suggest:
> > > > > > >
> > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > >
> > > > > > > - add to the motivation section why this is better than constructing a
> > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > >
> > > > > > > - add some rejected alternatives.
> > > > > > >
> > > > > > > Ryanne
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <dongjin@apache.org wrote:
> > > > > > >
> > > > > > > > We have +4 non-binding for this vote. Is there any committer who is
> > > > > > > > interested in this issue?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dongjin
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > andrew_schofield@live.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > >
> > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <en...@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Bumping this thread. Considering this KIP is relatively straigh
> > > > > > > > > > forward, can we get some votes or feedback if you think it's
> > > > > > not?
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > > edocomar@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > >
> > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > > pkleindl@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > Best regards
> > > > > > > > > > > > Patrik
> > > > > > > > > > > >
> > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > "When the people fear their government, there is tyranny;
> > > > > > when
> > > > > > > > the
> > > > > > > > > > > government fears the people, there is liberty." [Thomas
> > > > > > > > Jefferson]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Dongjin Lee*
> > > > > > > >
> > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > > > speakerdeck.com/dongjin
> > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > >
> > > > > > > >
> > > > > >
>
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Mickael Maison <mi...@gmail.com>.
Hi,
In an attempt to unblock this KIP, I've made some adjustments:
I've renamed the commitConsumerGroupOffsets() methods to
resetConsumerGroupOffsets() to reduce confusion. That should better
highlight the differences with the regular commit() operation from the
Consumer API. I've also added some details to the motivation section.
So we have +5 non binding votes and 0 binding votes
On Mon, Mar 25, 2019 at 1:10 PM Mickael Maison <mi...@gmail.com> wrote:
>
> Bumping this thread once again
>
> Ismael, have I answered your questions?
> While this has received a few non-binding +1s, no committers have
> voted yet. If you have concerns or questions, please let me know.
>
> Thanks
>
> On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> <mi...@gmail.com> wrote:
> >
> > Bumping this thread as it's been a couple of weeks.
> >
> > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <mi...@gmail.com> wrote:
> > >
> > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > - Having the reset functionality in the AdminClient:
> > > The fact we have a command line tool illustrate that this operation is
> > > relatively common. I seems valuable to be able to perform this
> > > operation directly via a proper API in addition of the CLI tool.
> > >
> > > - Sending an OffsetCommit directly instead of relying on KafkaConsumer:
> > > The KafkaConsumer requires a lot of stuff to commit offsets. Its group
> > > cannot change so you need to start a new Consumer every time, that
> > > creates new connections and overal sends more requests. Also there are
> > > already a bunch of AdminClient APIs that have logic very close to
> > > what needs to be done to send a commit request, keeping the code small
> > > and consistent.
> > >
> > > I've updated the KIP with these details and moved the 2nd part to
> > > "Proposed changes" as it's more an implementation detail.
> > >
> > > I hope this answers your question
> > >
> > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com> wrote:
> > > >
> > > > The KIP doesn't discuss the option of using KafkaConsumer directly as far
> > > > as I can tell. We have tried to avoid having the same functionality in
> > > > multiple clients so it would be good to explain why this is necessary here
> > > > (not saying it isn't).
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <mickael.maison@gmail.com
> > > > wrote:
> > > >
> > > > > Thanks Ryanne for the feedback, all suggestions sounded good, I've
> > > > > updated the KIP accordingly.
> > > > >
> > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <ry...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > But I suggest:
> > > > > >
> > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > >
> > > > > > - add to the motivation section why this is better than constructing a
> > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > >
> > > > > > - add some rejected alternatives.
> > > > > >
> > > > > > Ryanne
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <dongjin@apache.org wrote:
> > > > > >
> > > > > > > We have +4 non-binding for this vote. Is there any committer who is
> > > > > > > interested in this issue?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dongjin
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > andrew_schofield@live.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > >
> > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <en...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > +1 (non binding). Thanks.
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Bumping this thread. Considering this KIP is relatively straigh
> > > > > > > > > forward, can we get some votes or feedback if you think it's
> > > > > not?
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > edocomar@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1 (non-binding)
> > > > > > > > > > Thanks Mickael!
> > > > > > > > > >
> > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > pkleindl@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > Best regards
> > > > > > > > > > > Patrik
> > > > > > > > > > >
> > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > "When the people fear their government, there is tyranny;
> > > > > when
> > > > > > > the
> > > > > > > > > > government fears the people, there is liberty." [Thomas
> > > > > > > Jefferson]
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Dongjin Lee*
> > > > > > >
> > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > > speakerdeck.com/dongjin
> > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > >
> > > > > > >
> > > > >
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Mickael Maison <mi...@gmail.com>.
Closing this vote as we are well over 72 hours :)
The vote has passed with +6 binding votes (Guozhang, Vahid, Bill,
Harsha, Colin and Jason) and +7 non binding votes (Gabor, Jungtaek,
Ryanne, Andrew, Eno, Edo and Patrik).
Thanks to everyone that reviewed and helped improve this proposal.
On Thu, Aug 15, 2019 at 9:18 PM Guozhang Wang <wa...@gmail.com> wrote:
>
> +1 (binding).
>
> Thanks!
>
>
> Guozhang
>
> On Wed, Aug 14, 2019 at 5:18 PM Vahid Hashemian <va...@gmail.com>
> wrote:
>
> > +1 (binding)
> >
> > Thanks Michael for the suggestion of simplifying offset
> > retrieval/alteration operations.
> >
> > --Vahid
> >
> > On Wed, Aug 14, 2019 at 4:42 PM Bill Bejeck <bb...@gmail.com> wrote:
> >
> > > Thanks for the KIP Mickael, looks very useful.
> > > +1 (binding)
> > >
> > > -Bill
> > >
> > > On Wed, Aug 14, 2019 at 6:14 PM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > > Thanks for the KIP Mickael. LGTM +1 (binding).
> > > > -Harsha
> > > >
> > > >
> > > > On Wed, Aug 14, 2019 at 1:10 PM, Colin McCabe <cm...@apache.org>
> > > wrote:
> > > >
> > > > > Thanks, Mickael. +1 (binding)
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > > I've read it through in depth and as Jungtaek said Spark can make
> > good
> > > > use
> > > > > of it.
> > > > >
> > > > > On Wed, 14 Aug 2019, 17:06 Jungtaek Lim, <ka...@gmail.com> wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > I found it very useful for Spark's case. (Discussion on KIP-505
> > > described
> > > > > it.)
> > > > >
> > > > > Thanks for driving the effort!
> > > > >
> > > > > 2019년 8월 14일 (수) 오후 8:49, Mickael Maison <mickael.maison@gmail.com
> > >님이
> > > > 작성:
> > > > >
> > > > > Hi Guozhang,
> > > > >
> > > > > Thanks for taking a look.
> > > > >
> > > > > 1. Right, I updated the titles of the code blocks
> > > > >
> > > > > 2. Yes that's a good idea. I've updated the KIP
> > > > >
> > > > > Thank you
> > > > >
> > > > > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> > > > > <mi...@gmail.com> wrote:
> > > > >
> > > > > Hi Colin,
> > > > >
> > > > > Thanks for raising these 2 valid points. I've updated the KIP
> > > > >
> > > > > accordingly.
> > > > >
> > > > > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang <wa...@gmail.com>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for the KIP!
> > > > >
> > > > > Just some minor comments.
> > > > >
> > > > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java
> > > > > <http://commitoffsetsoptions.java/>"
> > > > >
> > > > > should
> > > > >
> > > > > be
> > > > >
> > > > > "AlterOffsetsOptions".
> > > > >
> > > > > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> > > > >
> > > > > to
> > > > >
> > > > > *KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
> > > > >
> > > > > This is because we will have a hierarchy of two-layers of errors
> > > > >
> > > > > since
> > > > >
> > > > > we
> > > > >
> > > > > need to find out the group coordinator first and then issue the
> > > > >
> > > > > commit
> > > > >
> > > > > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> > > > >
> > > > > exclude
> > > > >
> > > > > partitions that have errors, or the DeleteMembersResult as part of
> > > > >
> > > > > KIP-345).
> > > > >
> > > > > If the discover-coordinator returns non-triable error, we would set
> > > > >
> > > > > it
> > > > >
> > > > > on
> > > > >
> > > > > the first layer of the KafkaFuture, and the per-partition error would
> > > > >
> > > > > be
> > > > >
> > > > > set on the second layer of the KafkaFuture.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org>
> > > > >
> > > > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Considering that KIP-496, which adds a way of deleting consumer
> > > > >
> > > > > offsets
> > > > >
> > > > > from AdminClient, looks like it is going to get in, this seems like
> > > > > functionality we should definitely have.
> > > > >
> > > > > For alterConsumerGroupOffsets, is the intention to ignore
> > > > >
> > > > > partitions
> > > > >
> > > > > that
> > > > >
> > > > > are not specified in the map? If so, we should specify that in the
> > > > >
> > > > > JavaDoc.
> > > > >
> > > > > isolationLevel seems like it should be an enum rather than a
> > > > >
> > > > > string. The
> > > > >
> > > > > existing enum is in org.apache.kafka.common.requests, so we should
> > > > >
> > > > > probably
> > > > >
> > > > > create a new one which is public in org.apache.kafka.clients.admin.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > > >
> > > > > Bumping this thread once again
> > > > >
> > > > > Ismael, have I answered your questions?
> > > > > While this has received a few non-binding +1s, no committers have
> > voted
> > > > > yet. If you have concerns or questions, please let me know.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > > <mi...@gmail.com> wrote:
> > > > >
> > > > > Bumping this thread as it's been a couple of weeks.
> > > > >
> > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > > >
> > > > > mickael.maison@gmail.com> wrote:
> > > > >
> > > > > Thanks Ismael for the feedback. I think your point has 2
> > > > >
> > > > > parts:
> > > > >
> > > > > - Having the reset functionality in the AdminClient: The fact we
> > have a
> > > > > command line tool illustrate that this
> > > > >
> > > > > operation
> > > > >
> > > > > is
> > > > >
> > > > > relatively common. I seems valuable to be able to perform
> > > > >
> > > > > this
> > > > >
> > > > > operation directly via a proper API in addition of the CLI
> > > > >
> > > > > tool.
> > > > >
> > > > > - Sending an OffsetCommit directly instead of relying on
> > > > >
> > > > > KafkaConsumer:
> > > > >
> > > > > The KafkaConsumer requires a lot of stuff to commit offsets.
> > > > >
> > > > > Its
> > > > >
> > > > > group
> > > > >
> > > > > cannot change so you need to start a new Consumer every time,
> > > > >
> > > > > that
> > > > >
> > > > > creates new connections and overal sends more requests. Also
> > > > >
> > > > > there
> > > > >
> > > > > are
> > > > >
> > > > > already a bunch of AdminClient APIs that have logic very
> > > > >
> > > > > close to
> > > > >
> > > > > what needs to be done to send a commit request, keeping the
> > > > >
> > > > > code
> > > > >
> > > > > small
> > > > >
> > > > > and consistent.
> > > > >
> > > > > I've updated the KIP with these details and moved the 2nd
> > > > >
> > > > > part
> > > > >
> > > > > to
> > > > >
> > > > > "Proposed changes" as it's more an implementation detail.
> > > > >
> > > > > I hope this answers your question
> > > > >
> > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
> > > > >
> > > > > ismaelj@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > The KIP doesn't discuss the option of using KafkaConsumer
> > > > >
> > > > > directly
> > > > >
> > > > > as far
> > > > >
> > > > > as I can tell. We have tried to avoid having the same
> > > > >
> > > > > functionality in
> > > > >
> > > > > multiple clients so it would be good to explain why this is
> > > > >
> > > > > necessary here
> > > > >
> > > > > (not saying it isn't).
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > > > >
> > > > > mickael.maison@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > Thanks Ryanne for the feedback, all suggestions sounded
> > > > >
> > > > > good,
> > > > >
> > > > > I've
> > > > >
> > > > > updated the KIP accordingly.
> > > > >
> > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > > > >
> > > > > ryannedolan@gmail.com>
> > > > >
> > > > > wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > But I suggest:
> > > > >
> > > > > - drop "get" from getOffset, getTimestamp.
> > > > >
> > > > > - add to the motivation section why this is better than
> > > > >
> > > > > constructing a
> > > > >
> > > > > KafkaConsumer and using seek(), commit() etc.
> > > > >
> > > > > - add some rejected alternatives.
> > > > >
> > > > > Ryanne
> > > > >
> > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> > > > >
> > > > > dongjin@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > We have +4 non-binding for this vote. Is there any
> > > > >
> > > > > committer
> > > > >
> > > > > who is
> > > > >
> > > > > interested in this issue?
> > > > >
> > > > > Thanks,
> > > > > Dongjin
> > > > >
> > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > andrew_schofield@live.
> > > > > com>
> > > > > wrote:
> > > > >
> > > > > +1 (non-binding). Thanks for the KIP.
> > > > >
> > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> > > > >
> > > > > eno.thereska@gmail.com>
> > > > >
> > > > > wrote:
> > > > >
> > > > > +1 (non binding). Thanks.
> > > > >
> > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael
> > > > >
> > > > > Maison <
> > > > >
> > > > > mickael.maison@gmail.com>
> > > > > wrote:
> > > > >
> > > > > Bumping this thread. Considering this KIP is
> > > > >
> > > > > relatively straigh
> > > > >
> > > > > forward, can we get some votes or feedback if
> > > > >
> > > > > you
> > > > >
> > > > > think it's
> > > > >
> > > > > not?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar
> > > > >
> > > > > <
> > > > >
> > > > > edocomar@gmail.com>
> > > > >
> > > > > wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > > Thanks Mickael!
> > > > >
> > > > > On Tue, 8 Jan 2019 at 17:39, Patrik
> > > > >
> > > > > Kleindl <
> > > > >
> > > > > pkleindl@gmail.com>
> > > > >
> > > > > wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > > Thanks, sounds very helpful
> > > > > Best regards
> > > > > Patrik
> > > > >
> > > > > Am 08.01.2019 um 18:10 schrieb Mickael
> > > > >
> > > > > Maison <
> > > > >
> > > > > mickael.maison@gmail.com
> > > > >
> > > > > :
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start the vote on KIP-396:
> > > > >
> > > > >
> > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.
> > > > > apache.org%2Fconfluence%2Fpages%2Fviewpage.
> > > > >
> > > >
> > >
> > action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > >
> > > > > Thanks
> > > > >
> > > > > --
> > > > > "When the people fear their government,
> > > > >
> > > > > there is
> > > > >
> > > > > tyranny;
> > > > >
> > > > > when
> > > > >
> > > > > the
> > > > >
> > > > > government fears the people, there is
> > > > >
> > > > > liberty."
> > > > >
> > > > > [Thomas
> > > > >
> > > > > Jefferson]
> > > > >
> > > > > --
> > > > > *Dongjin Lee*
> > > > >
> > > > > *A hitchhiker in the mathematical world.*
> > > > > *github: <http://goog_969573159/>
> > > > >
> > > > > github.com/dongjinleekr
> > > > >
> > > > > <https://github.com/dongjinleekr>linkedin:
> > > > >
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > >
> > > > > <https://kr.linkedin.com/in/dongjinleekr
> > > > >
> > > > > speakerdeck:
> > > > >
> > > > > speakerdeck.com/dongjin
> > > > > <https://speakerdeck.com/dongjin>*
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Thanks!
> > --Vahid
> >
>
>
> --
> -- Guozhang
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Guozhang Wang <wa...@gmail.com>.
+1 (binding).
Thanks!
Guozhang
On Wed, Aug 14, 2019 at 5:18 PM Vahid Hashemian <va...@gmail.com>
wrote:
> +1 (binding)
>
> Thanks Michael for the suggestion of simplifying offset
> retrieval/alteration operations.
>
> --Vahid
>
> On Wed, Aug 14, 2019 at 4:42 PM Bill Bejeck <bb...@gmail.com> wrote:
>
> > Thanks for the KIP Mickael, looks very useful.
> > +1 (binding)
> >
> > -Bill
> >
> > On Wed, Aug 14, 2019 at 6:14 PM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Thanks for the KIP Mickael. LGTM +1 (binding).
> > > -Harsha
> > >
> > >
> > > On Wed, Aug 14, 2019 at 1:10 PM, Colin McCabe <cm...@apache.org>
> > wrote:
> > >
> > > > Thanks, Mickael. +1 (binding)
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote:
> > > >
> > > > +1 (non-binding)
> > > > I've read it through in depth and as Jungtaek said Spark can make
> good
> > > use
> > > > of it.
> > > >
> > > > On Wed, 14 Aug 2019, 17:06 Jungtaek Lim, <ka...@gmail.com> wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > I found it very useful for Spark's case. (Discussion on KIP-505
> > described
> > > > it.)
> > > >
> > > > Thanks for driving the effort!
> > > >
> > > > 2019년 8월 14일 (수) 오후 8:49, Mickael Maison <mickael.maison@gmail.com
> >님이
> > > 작성:
> > > >
> > > > Hi Guozhang,
> > > >
> > > > Thanks for taking a look.
> > > >
> > > > 1. Right, I updated the titles of the code blocks
> > > >
> > > > 2. Yes that's a good idea. I've updated the KIP
> > > >
> > > > Thank you
> > > >
> > > > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> > > > <mi...@gmail.com> wrote:
> > > >
> > > > Hi Colin,
> > > >
> > > > Thanks for raising these 2 valid points. I've updated the KIP
> > > >
> > > > accordingly.
> > > >
> > > > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang <wa...@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > Just some minor comments.
> > > >
> > > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java
> > > > <http://commitoffsetsoptions.java/>"
> > > >
> > > > should
> > > >
> > > > be
> > > >
> > > > "AlterOffsetsOptions".
> > > >
> > > > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> > > >
> > > > to
> > > >
> > > > *KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
> > > >
> > > > This is because we will have a hierarchy of two-layers of errors
> > > >
> > > > since
> > > >
> > > > we
> > > >
> > > > need to find out the group coordinator first and then issue the
> > > >
> > > > commit
> > > >
> > > > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> > > >
> > > > exclude
> > > >
> > > > partitions that have errors, or the DeleteMembersResult as part of
> > > >
> > > > KIP-345).
> > > >
> > > > If the discover-coordinator returns non-triable error, we would set
> > > >
> > > > it
> > > >
> > > > on
> > > >
> > > > the first layer of the KafkaFuture, and the per-partition error would
> > > >
> > > > be
> > > >
> > > > set on the second layer of the KafkaFuture.
> > > >
> > > > Guozhang
> > > >
> > > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org>
> > > >
> > > > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > Considering that KIP-496, which adds a way of deleting consumer
> > > >
> > > > offsets
> > > >
> > > > from AdminClient, looks like it is going to get in, this seems like
> > > > functionality we should definitely have.
> > > >
> > > > For alterConsumerGroupOffsets, is the intention to ignore
> > > >
> > > > partitions
> > > >
> > > > that
> > > >
> > > > are not specified in the map? If so, we should specify that in the
> > > >
> > > > JavaDoc.
> > > >
> > > > isolationLevel seems like it should be an enum rather than a
> > > >
> > > > string. The
> > > >
> > > > existing enum is in org.apache.kafka.common.requests, so we should
> > > >
> > > > probably
> > > >
> > > > create a new one which is public in org.apache.kafka.clients.admin.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > >
> > > > Bumping this thread once again
> > > >
> > > > Ismael, have I answered your questions?
> > > > While this has received a few non-binding +1s, no committers have
> voted
> > > > yet. If you have concerns or questions, please let me know.
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > <mi...@gmail.com> wrote:
> > > >
> > > > Bumping this thread as it's been a couple of weeks.
> > > >
> > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > >
> > > > mickael.maison@gmail.com> wrote:
> > > >
> > > > Thanks Ismael for the feedback. I think your point has 2
> > > >
> > > > parts:
> > > >
> > > > - Having the reset functionality in the AdminClient: The fact we
> have a
> > > > command line tool illustrate that this
> > > >
> > > > operation
> > > >
> > > > is
> > > >
> > > > relatively common. I seems valuable to be able to perform
> > > >
> > > > this
> > > >
> > > > operation directly via a proper API in addition of the CLI
> > > >
> > > > tool.
> > > >
> > > > - Sending an OffsetCommit directly instead of relying on
> > > >
> > > > KafkaConsumer:
> > > >
> > > > The KafkaConsumer requires a lot of stuff to commit offsets.
> > > >
> > > > Its
> > > >
> > > > group
> > > >
> > > > cannot change so you need to start a new Consumer every time,
> > > >
> > > > that
> > > >
> > > > creates new connections and overal sends more requests. Also
> > > >
> > > > there
> > > >
> > > > are
> > > >
> > > > already a bunch of AdminClient APIs that have logic very
> > > >
> > > > close to
> > > >
> > > > what needs to be done to send a commit request, keeping the
> > > >
> > > > code
> > > >
> > > > small
> > > >
> > > > and consistent.
> > > >
> > > > I've updated the KIP with these details and moved the 2nd
> > > >
> > > > part
> > > >
> > > > to
> > > >
> > > > "Proposed changes" as it's more an implementation detail.
> > > >
> > > > I hope this answers your question
> > > >
> > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
> > > >
> > > > ismaelj@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > The KIP doesn't discuss the option of using KafkaConsumer
> > > >
> > > > directly
> > > >
> > > > as far
> > > >
> > > > as I can tell. We have tried to avoid having the same
> > > >
> > > > functionality in
> > > >
> > > > multiple clients so it would be good to explain why this is
> > > >
> > > > necessary here
> > > >
> > > > (not saying it isn't).
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > > >
> > > > mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > Thanks Ryanne for the feedback, all suggestions sounded
> > > >
> > > > good,
> > > >
> > > > I've
> > > >
> > > > updated the KIP accordingly.
> > > >
> > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > > >
> > > > ryannedolan@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > But I suggest:
> > > >
> > > > - drop "get" from getOffset, getTimestamp.
> > > >
> > > > - add to the motivation section why this is better than
> > > >
> > > > constructing a
> > > >
> > > > KafkaConsumer and using seek(), commit() etc.
> > > >
> > > > - add some rejected alternatives.
> > > >
> > > > Ryanne
> > > >
> > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> > > >
> > > > dongjin@apache.org
> > > >
> > > > wrote:
> > > >
> > > > We have +4 non-binding for this vote. Is there any
> > > >
> > > > committer
> > > >
> > > > who is
> > > >
> > > > interested in this issue?
> > > >
> > > > Thanks,
> > > > Dongjin
> > > >
> > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > andrew_schofield@live.
> > > > com>
> > > > wrote:
> > > >
> > > > +1 (non-binding). Thanks for the KIP.
> > > >
> > > > On 21/01/2019, 12:45, "Eno Thereska" <
> > > >
> > > > eno.thereska@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > +1 (non binding). Thanks.
> > > >
> > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael
> > > >
> > > > Maison <
> > > >
> > > > mickael.maison@gmail.com>
> > > > wrote:
> > > >
> > > > Bumping this thread. Considering this KIP is
> > > >
> > > > relatively straigh
> > > >
> > > > forward, can we get some votes or feedback if
> > > >
> > > > you
> > > >
> > > > think it's
> > > >
> > > > not?
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar
> > > >
> > > > <
> > > >
> > > > edocomar@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > +1 (non-binding)
> > > > Thanks Mickael!
> > > >
> > > > On Tue, 8 Jan 2019 at 17:39, Patrik
> > > >
> > > > Kleindl <
> > > >
> > > > pkleindl@gmail.com>
> > > >
> > > > wrote:
> > > >
> > > > +1 (non-binding)
> > > > Thanks, sounds very helpful
> > > > Best regards
> > > > Patrik
> > > >
> > > > Am 08.01.2019 um 18:10 schrieb Mickael
> > > >
> > > > Maison <
> > > >
> > > > mickael.maison@gmail.com
> > > >
> > > > :
> > > >
> > > > Hi all,
> > > >
> > > > I'd like to start the vote on KIP-396:
> > > >
> > > >
> > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.
> > > > apache.org%2Fconfluence%2Fpages%2Fviewpage.
> > > >
> > >
> >
> action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > >
> > > > Thanks
> > > >
> > > > --
> > > > "When the people fear their government,
> > > >
> > > > there is
> > > >
> > > > tyranny;
> > > >
> > > > when
> > > >
> > > > the
> > > >
> > > > government fears the people, there is
> > > >
> > > > liberty."
> > > >
> > > > [Thomas
> > > >
> > > > Jefferson]
> > > >
> > > > --
> > > > *Dongjin Lee*
> > > >
> > > > *A hitchhiker in the mathematical world.*
> > > > *github: <http://goog_969573159/>
> > > >
> > > > github.com/dongjinleekr
> > > >
> > > > <https://github.com/dongjinleekr>linkedin:
> > > >
> > > > kr.linkedin.com/in/dongjinleekr
> > > >
> > > > <https://kr.linkedin.com/in/dongjinleekr
> > > >
> > > > speakerdeck:
> > > >
> > > > speakerdeck.com/dongjin
> > > > <https://speakerdeck.com/dongjin>*
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > > >
> > >
> >
>
>
> --
>
> Thanks!
> --Vahid
>
--
-- Guozhang
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Vahid Hashemian <va...@gmail.com>.
+1 (binding)
Thanks Michael for the suggestion of simplifying offset
retrieval/alteration operations.
--Vahid
On Wed, Aug 14, 2019 at 4:42 PM Bill Bejeck <bb...@gmail.com> wrote:
> Thanks for the KIP Mickael, looks very useful.
> +1 (binding)
>
> -Bill
>
> On Wed, Aug 14, 2019 at 6:14 PM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > Thanks for the KIP Mickael. LGTM +1 (binding).
> > -Harsha
> >
> >
> > On Wed, Aug 14, 2019 at 1:10 PM, Colin McCabe <cm...@apache.org>
> wrote:
> >
> > > Thanks, Mickael. +1 (binding)
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote:
> > >
> > > +1 (non-binding)
> > > I've read it through in depth and as Jungtaek said Spark can make good
> > use
> > > of it.
> > >
> > > On Wed, 14 Aug 2019, 17:06 Jungtaek Lim, <ka...@gmail.com> wrote:
> > >
> > > +1 (non-binding)
> > >
> > > I found it very useful for Spark's case. (Discussion on KIP-505
> described
> > > it.)
> > >
> > > Thanks for driving the effort!
> > >
> > > 2019년 8월 14일 (수) 오후 8:49, Mickael Maison <mi...@gmail.com>님이
> > 작성:
> > >
> > > Hi Guozhang,
> > >
> > > Thanks for taking a look.
> > >
> > > 1. Right, I updated the titles of the code blocks
> > >
> > > 2. Yes that's a good idea. I've updated the KIP
> > >
> > > Thank you
> > >
> > > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> > > <mi...@gmail.com> wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for raising these 2 valid points. I've updated the KIP
> > >
> > > accordingly.
> > >
> > > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang <wa...@gmail.com>
> > >
> > > wrote:
> > >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP!
> > >
> > > Just some minor comments.
> > >
> > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java
> > > <http://commitoffsetsoptions.java/>"
> > >
> > > should
> > >
> > > be
> > >
> > > "AlterOffsetsOptions".
> > >
> > > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> > >
> > > to
> > >
> > > *KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
> > >
> > > This is because we will have a hierarchy of two-layers of errors
> > >
> > > since
> > >
> > > we
> > >
> > > need to find out the group coordinator first and then issue the
> > >
> > > commit
> > >
> > > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> > >
> > > exclude
> > >
> > > partitions that have errors, or the DeleteMembersResult as part of
> > >
> > > KIP-345).
> > >
> > > If the discover-coordinator returns non-triable error, we would set
> > >
> > > it
> > >
> > > on
> > >
> > > the first layer of the KafkaFuture, and the per-partition error would
> > >
> > > be
> > >
> > > set on the second layer of the KafkaFuture.
> > >
> > > Guozhang
> > >
> > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org>
> > >
> > > wrote:
> > >
> > > Hi Mickael,
> > >
> > > Considering that KIP-496, which adds a way of deleting consumer
> > >
> > > offsets
> > >
> > > from AdminClient, looks like it is going to get in, this seems like
> > > functionality we should definitely have.
> > >
> > > For alterConsumerGroupOffsets, is the intention to ignore
> > >
> > > partitions
> > >
> > > that
> > >
> > > are not specified in the map? If so, we should specify that in the
> > >
> > > JavaDoc.
> > >
> > > isolationLevel seems like it should be an enum rather than a
> > >
> > > string. The
> > >
> > > existing enum is in org.apache.kafka.common.requests, so we should
> > >
> > > probably
> > >
> > > create a new one which is public in org.apache.kafka.clients.admin.
> > >
> > > best,
> > > Colin
> > >
> > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > >
> > > Bumping this thread once again
> > >
> > > Ismael, have I answered your questions?
> > > While this has received a few non-binding +1s, no committers have voted
> > > yet. If you have concerns or questions, please let me know.
> > >
> > > Thanks
> > >
> > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > <mi...@gmail.com> wrote:
> > >
> > > Bumping this thread as it's been a couple of weeks.
> > >
> > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > >
> > > mickael.maison@gmail.com> wrote:
> > >
> > > Thanks Ismael for the feedback. I think your point has 2
> > >
> > > parts:
> > >
> > > - Having the reset functionality in the AdminClient: The fact we have a
> > > command line tool illustrate that this
> > >
> > > operation
> > >
> > > is
> > >
> > > relatively common. I seems valuable to be able to perform
> > >
> > > this
> > >
> > > operation directly via a proper API in addition of the CLI
> > >
> > > tool.
> > >
> > > - Sending an OffsetCommit directly instead of relying on
> > >
> > > KafkaConsumer:
> > >
> > > The KafkaConsumer requires a lot of stuff to commit offsets.
> > >
> > > Its
> > >
> > > group
> > >
> > > cannot change so you need to start a new Consumer every time,
> > >
> > > that
> > >
> > > creates new connections and overal sends more requests. Also
> > >
> > > there
> > >
> > > are
> > >
> > > already a bunch of AdminClient APIs that have logic very
> > >
> > > close to
> > >
> > > what needs to be done to send a commit request, keeping the
> > >
> > > code
> > >
> > > small
> > >
> > > and consistent.
> > >
> > > I've updated the KIP with these details and moved the 2nd
> > >
> > > part
> > >
> > > to
> > >
> > > "Proposed changes" as it's more an implementation detail.
> > >
> > > I hope this answers your question
> > >
> > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
> > >
> > > ismaelj@gmail.com
> > >
> > > wrote:
> > >
> > > The KIP doesn't discuss the option of using KafkaConsumer
> > >
> > > directly
> > >
> > > as far
> > >
> > > as I can tell. We have tried to avoid having the same
> > >
> > > functionality in
> > >
> > > multiple clients so it would be good to explain why this is
> > >
> > > necessary here
> > >
> > > (not saying it isn't).
> > >
> > > Ismael
> > >
> > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > >
> > > mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > Thanks Ryanne for the feedback, all suggestions sounded
> > >
> > > good,
> > >
> > > I've
> > >
> > > updated the KIP accordingly.
> > >
> > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > >
> > > ryannedolan@gmail.com>
> > >
> > > wrote:
> > >
> > > +1 (non-binding)
> > >
> > > But I suggest:
> > >
> > > - drop "get" from getOffset, getTimestamp.
> > >
> > > - add to the motivation section why this is better than
> > >
> > > constructing a
> > >
> > > KafkaConsumer and using seek(), commit() etc.
> > >
> > > - add some rejected alternatives.
> > >
> > > Ryanne
> > >
> > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> > >
> > > dongjin@apache.org
> > >
> > > wrote:
> > >
> > > We have +4 non-binding for this vote. Is there any
> > >
> > > committer
> > >
> > > who is
> > >
> > > interested in this issue?
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > andrew_schofield@live.
> > > com>
> > > wrote:
> > >
> > > +1 (non-binding). Thanks for the KIP.
> > >
> > > On 21/01/2019, 12:45, "Eno Thereska" <
> > >
> > > eno.thereska@gmail.com>
> > >
> > > wrote:
> > >
> > > +1 (non binding). Thanks.
> > >
> > > On Mon, Jan 21, 2019 at 12:30 PM Mickael
> > >
> > > Maison <
> > >
> > > mickael.maison@gmail.com>
> > > wrote:
> > >
> > > Bumping this thread. Considering this KIP is
> > >
> > > relatively straigh
> > >
> > > forward, can we get some votes or feedback if
> > >
> > > you
> > >
> > > think it's
> > >
> > > not?
> > >
> > > Thanks
> > >
> > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar
> > >
> > > <
> > >
> > > edocomar@gmail.com>
> > >
> > > wrote:
> > >
> > > +1 (non-binding)
> > > Thanks Mickael!
> > >
> > > On Tue, 8 Jan 2019 at 17:39, Patrik
> > >
> > > Kleindl <
> > >
> > > pkleindl@gmail.com>
> > >
> > > wrote:
> > >
> > > +1 (non-binding)
> > > Thanks, sounds very helpful
> > > Best regards
> > > Patrik
> > >
> > > Am 08.01.2019 um 18:10 schrieb Mickael
> > >
> > > Maison <
> > >
> > > mickael.maison@gmail.com
> > >
> > > :
> > >
> > > Hi all,
> > >
> > > I'd like to start the vote on KIP-396:
> > >
> > >
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.
> > > apache.org%2Fconfluence%2Fpages%2Fviewpage.
> > >
> >
> action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > >
> > > Thanks
> > >
> > > --
> > > "When the people fear their government,
> > >
> > > there is
> > >
> > > tyranny;
> > >
> > > when
> > >
> > > the
> > >
> > > government fears the people, there is
> > >
> > > liberty."
> > >
> > > [Thomas
> > >
> > > Jefferson]
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > > *github: <http://goog_969573159/>
> > >
> > > github.com/dongjinleekr
> > >
> > > <https://github.com/dongjinleekr>linkedin:
> > >
> > > kr.linkedin.com/in/dongjinleekr
> > >
> > > <https://kr.linkedin.com/in/dongjinleekr
> > >
> > > speakerdeck:
> > >
> > > speakerdeck.com/dongjin
> > > <https://speakerdeck.com/dongjin>*
> > >
> > > --
> > > -- Guozhang
> > >
> > >
> >
>
--
Thanks!
--Vahid
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Bill Bejeck <bb...@gmail.com>.
Thanks for the KIP Mickael, looks very useful.
+1 (binding)
-Bill
On Wed, Aug 14, 2019 at 6:14 PM Harsha Chintalapani <ka...@harsha.io> wrote:
> Thanks for the KIP Mickael. LGTM +1 (binding).
> -Harsha
>
>
> On Wed, Aug 14, 2019 at 1:10 PM, Colin McCabe <cm...@apache.org> wrote:
>
> > Thanks, Mickael. +1 (binding)
> >
> > best,
> > Colin
> >
> > On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote:
> >
> > +1 (non-binding)
> > I've read it through in depth and as Jungtaek said Spark can make good
> use
> > of it.
> >
> > On Wed, 14 Aug 2019, 17:06 Jungtaek Lim, <ka...@gmail.com> wrote:
> >
> > +1 (non-binding)
> >
> > I found it very useful for Spark's case. (Discussion on KIP-505 described
> > it.)
> >
> > Thanks for driving the effort!
> >
> > 2019년 8월 14일 (수) 오후 8:49, Mickael Maison <mi...@gmail.com>님이
> 작성:
> >
> > Hi Guozhang,
> >
> > Thanks for taking a look.
> >
> > 1. Right, I updated the titles of the code blocks
> >
> > 2. Yes that's a good idea. I've updated the KIP
> >
> > Thank you
> >
> > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> > <mi...@gmail.com> wrote:
> >
> > Hi Colin,
> >
> > Thanks for raising these 2 valid points. I've updated the KIP
> >
> > accordingly.
> >
> > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang <wa...@gmail.com>
> >
> > wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP!
> >
> > Just some minor comments.
> >
> > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java
> > <http://commitoffsetsoptions.java/>"
> >
> > should
> >
> > be
> >
> > "AlterOffsetsOptions".
> >
> > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> >
> > to
> >
> > *KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
> >
> > This is because we will have a hierarchy of two-layers of errors
> >
> > since
> >
> > we
> >
> > need to find out the group coordinator first and then issue the
> >
> > commit
> >
> > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> >
> > exclude
> >
> > partitions that have errors, or the DeleteMembersResult as part of
> >
> > KIP-345).
> >
> > If the discover-coordinator returns non-triable error, we would set
> >
> > it
> >
> > on
> >
> > the first layer of the KafkaFuture, and the per-partition error would
> >
> > be
> >
> > set on the second layer of the KafkaFuture.
> >
> > Guozhang
> >
> > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org>
> >
> > wrote:
> >
> > Hi Mickael,
> >
> > Considering that KIP-496, which adds a way of deleting consumer
> >
> > offsets
> >
> > from AdminClient, looks like it is going to get in, this seems like
> > functionality we should definitely have.
> >
> > For alterConsumerGroupOffsets, is the intention to ignore
> >
> > partitions
> >
> > that
> >
> > are not specified in the map? If so, we should specify that in the
> >
> > JavaDoc.
> >
> > isolationLevel seems like it should be an enum rather than a
> >
> > string. The
> >
> > existing enum is in org.apache.kafka.common.requests, so we should
> >
> > probably
> >
> > create a new one which is public in org.apache.kafka.clients.admin.
> >
> > best,
> > Colin
> >
> > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> >
> > Bumping this thread once again
> >
> > Ismael, have I answered your questions?
> > While this has received a few non-binding +1s, no committers have voted
> > yet. If you have concerns or questions, please let me know.
> >
> > Thanks
> >
> > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > <mi...@gmail.com> wrote:
> >
> > Bumping this thread as it's been a couple of weeks.
> >
> > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> >
> > mickael.maison@gmail.com> wrote:
> >
> > Thanks Ismael for the feedback. I think your point has 2
> >
> > parts:
> >
> > - Having the reset functionality in the AdminClient: The fact we have a
> > command line tool illustrate that this
> >
> > operation
> >
> > is
> >
> > relatively common. I seems valuable to be able to perform
> >
> > this
> >
> > operation directly via a proper API in addition of the CLI
> >
> > tool.
> >
> > - Sending an OffsetCommit directly instead of relying on
> >
> > KafkaConsumer:
> >
> > The KafkaConsumer requires a lot of stuff to commit offsets.
> >
> > Its
> >
> > group
> >
> > cannot change so you need to start a new Consumer every time,
> >
> > that
> >
> > creates new connections and overal sends more requests. Also
> >
> > there
> >
> > are
> >
> > already a bunch of AdminClient APIs that have logic very
> >
> > close to
> >
> > what needs to be done to send a commit request, keeping the
> >
> > code
> >
> > small
> >
> > and consistent.
> >
> > I've updated the KIP with these details and moved the 2nd
> >
> > part
> >
> > to
> >
> > "Proposed changes" as it's more an implementation detail.
> >
> > I hope this answers your question
> >
> > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
> >
> > ismaelj@gmail.com
> >
> > wrote:
> >
> > The KIP doesn't discuss the option of using KafkaConsumer
> >
> > directly
> >
> > as far
> >
> > as I can tell. We have tried to avoid having the same
> >
> > functionality in
> >
> > multiple clients so it would be good to explain why this is
> >
> > necessary here
> >
> > (not saying it isn't).
> >
> > Ismael
> >
> > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> >
> > mickael.maison@gmail.com
> >
> > wrote:
> >
> > Thanks Ryanne for the feedback, all suggestions sounded
> >
> > good,
> >
> > I've
> >
> > updated the KIP accordingly.
> >
> > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> >
> > ryannedolan@gmail.com>
> >
> > wrote:
> >
> > +1 (non-binding)
> >
> > But I suggest:
> >
> > - drop "get" from getOffset, getTimestamp.
> >
> > - add to the motivation section why this is better than
> >
> > constructing a
> >
> > KafkaConsumer and using seek(), commit() etc.
> >
> > - add some rejected alternatives.
> >
> > Ryanne
> >
> > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> >
> > dongjin@apache.org
> >
> > wrote:
> >
> > We have +4 non-binding for this vote. Is there any
> >
> > committer
> >
> > who is
> >
> > interested in this issue?
> >
> > Thanks,
> > Dongjin
> >
> > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> andrew_schofield@live.
> > com>
> > wrote:
> >
> > +1 (non-binding). Thanks for the KIP.
> >
> > On 21/01/2019, 12:45, "Eno Thereska" <
> >
> > eno.thereska@gmail.com>
> >
> > wrote:
> >
> > +1 (non binding). Thanks.
> >
> > On Mon, Jan 21, 2019 at 12:30 PM Mickael
> >
> > Maison <
> >
> > mickael.maison@gmail.com>
> > wrote:
> >
> > Bumping this thread. Considering this KIP is
> >
> > relatively straigh
> >
> > forward, can we get some votes or feedback if
> >
> > you
> >
> > think it's
> >
> > not?
> >
> > Thanks
> >
> > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar
> >
> > <
> >
> > edocomar@gmail.com>
> >
> > wrote:
> >
> > +1 (non-binding)
> > Thanks Mickael!
> >
> > On Tue, 8 Jan 2019 at 17:39, Patrik
> >
> > Kleindl <
> >
> > pkleindl@gmail.com>
> >
> > wrote:
> >
> > +1 (non-binding)
> > Thanks, sounds very helpful
> > Best regards
> > Patrik
> >
> > Am 08.01.2019 um 18:10 schrieb Mickael
> >
> > Maison <
> >
> > mickael.maison@gmail.com
> >
> > :
> >
> > Hi all,
> >
> > I'd like to start the vote on KIP-396:
> >
> > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.
> > apache.org%2Fconfluence%2Fpages%2Fviewpage.
> >
> action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> >
> > Thanks
> >
> > --
> > "When the people fear their government,
> >
> > there is
> >
> > tyranny;
> >
> > when
> >
> > the
> >
> > government fears the people, there is
> >
> > liberty."
> >
> > [Thomas
> >
> > Jefferson]
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> > *github: <http://goog_969573159/>
> >
> > github.com/dongjinleekr
> >
> > <https://github.com/dongjinleekr>linkedin:
> >
> > kr.linkedin.com/in/dongjinleekr
> >
> > <https://kr.linkedin.com/in/dongjinleekr
> >
> > speakerdeck:
> >
> > speakerdeck.com/dongjin
> > <https://speakerdeck.com/dongjin>*
> >
> > --
> > -- Guozhang
> >
> >
>
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Harsha Chintalapani <ka...@harsha.io>.
Thanks for the KIP Mickael. LGTM +1 (binding).
-Harsha
On Wed, Aug 14, 2019 at 1:10 PM, Colin McCabe <cm...@apache.org> wrote:
> Thanks, Mickael. +1 (binding)
>
> best,
> Colin
>
> On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote:
>
> +1 (non-binding)
> I've read it through in depth and as Jungtaek said Spark can make good use
> of it.
>
> On Wed, 14 Aug 2019, 17:06 Jungtaek Lim, <ka...@gmail.com> wrote:
>
> +1 (non-binding)
>
> I found it very useful for Spark's case. (Discussion on KIP-505 described
> it.)
>
> Thanks for driving the effort!
>
> 2019년 8월 14일 (수) 오후 8:49, Mickael Maison <mi...@gmail.com>님이 작성:
>
> Hi Guozhang,
>
> Thanks for taking a look.
>
> 1. Right, I updated the titles of the code blocks
>
> 2. Yes that's a good idea. I've updated the KIP
>
> Thank you
>
> On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> <mi...@gmail.com> wrote:
>
> Hi Colin,
>
> Thanks for raising these 2 valid points. I've updated the KIP
>
> accordingly.
>
> On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang <wa...@gmail.com>
>
> wrote:
>
> Hi Mickael,
>
> Thanks for the KIP!
>
> Just some minor comments.
>
> 1. Java class names are stale, e.g. "CommitOffsetsOptions.java
> <http://commitoffsetsoptions.java/>"
>
> should
>
> be
>
> "AlterOffsetsOptions".
>
> 2. I'd suggest we change the future structure of "AlterOffsetsResult"
>
> to
>
> *KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
>
> This is because we will have a hierarchy of two-layers of errors
>
> since
>
> we
>
> need to find out the group coordinator first and then issue the
>
> commit
>
> offset request (see e.g. the ListConsumerGroupOffsetsResult which
>
> exclude
>
> partitions that have errors, or the DeleteMembersResult as part of
>
> KIP-345).
>
> If the discover-coordinator returns non-triable error, we would set
>
> it
>
> on
>
> the first layer of the KafkaFuture, and the per-partition error would
>
> be
>
> set on the second layer of the KafkaFuture.
>
> Guozhang
>
> On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org>
>
> wrote:
>
> Hi Mickael,
>
> Considering that KIP-496, which adds a way of deleting consumer
>
> offsets
>
> from AdminClient, looks like it is going to get in, this seems like
> functionality we should definitely have.
>
> For alterConsumerGroupOffsets, is the intention to ignore
>
> partitions
>
> that
>
> are not specified in the map? If so, we should specify that in the
>
> JavaDoc.
>
> isolationLevel seems like it should be an enum rather than a
>
> string. The
>
> existing enum is in org.apache.kafka.common.requests, so we should
>
> probably
>
> create a new one which is public in org.apache.kafka.clients.admin.
>
> best,
> Colin
>
> On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
>
> Bumping this thread once again
>
> Ismael, have I answered your questions?
> While this has received a few non-binding +1s, no committers have voted
> yet. If you have concerns or questions, please let me know.
>
> Thanks
>
> On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> <mi...@gmail.com> wrote:
>
> Bumping this thread as it's been a couple of weeks.
>
> On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
>
> mickael.maison@gmail.com> wrote:
>
> Thanks Ismael for the feedback. I think your point has 2
>
> parts:
>
> - Having the reset functionality in the AdminClient: The fact we have a
> command line tool illustrate that this
>
> operation
>
> is
>
> relatively common. I seems valuable to be able to perform
>
> this
>
> operation directly via a proper API in addition of the CLI
>
> tool.
>
> - Sending an OffsetCommit directly instead of relying on
>
> KafkaConsumer:
>
> The KafkaConsumer requires a lot of stuff to commit offsets.
>
> Its
>
> group
>
> cannot change so you need to start a new Consumer every time,
>
> that
>
> creates new connections and overal sends more requests. Also
>
> there
>
> are
>
> already a bunch of AdminClient APIs that have logic very
>
> close to
>
> what needs to be done to send a commit request, keeping the
>
> code
>
> small
>
> and consistent.
>
> I've updated the KIP with these details and moved the 2nd
>
> part
>
> to
>
> "Proposed changes" as it's more an implementation detail.
>
> I hope this answers your question
>
> On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
>
> ismaelj@gmail.com
>
> wrote:
>
> The KIP doesn't discuss the option of using KafkaConsumer
>
> directly
>
> as far
>
> as I can tell. We have tried to avoid having the same
>
> functionality in
>
> multiple clients so it would be good to explain why this is
>
> necessary here
>
> (not saying it isn't).
>
> Ismael
>
> On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
>
> mickael.maison@gmail.com
>
> wrote:
>
> Thanks Ryanne for the feedback, all suggestions sounded
>
> good,
>
> I've
>
> updated the KIP accordingly.
>
> On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
>
> ryannedolan@gmail.com>
>
> wrote:
>
> +1 (non-binding)
>
> But I suggest:
>
> - drop "get" from getOffset, getTimestamp.
>
> - add to the motivation section why this is better than
>
> constructing a
>
> KafkaConsumer and using seek(), commit() etc.
>
> - add some rejected alternatives.
>
> Ryanne
>
> On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
>
> dongjin@apache.org
>
> wrote:
>
> We have +4 non-binding for this vote. Is there any
>
> committer
>
> who is
>
> interested in this issue?
>
> Thanks,
> Dongjin
>
> On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield < andrew_schofield@live.
> com>
> wrote:
>
> +1 (non-binding). Thanks for the KIP.
>
> On 21/01/2019, 12:45, "Eno Thereska" <
>
> eno.thereska@gmail.com>
>
> wrote:
>
> +1 (non binding). Thanks.
>
> On Mon, Jan 21, 2019 at 12:30 PM Mickael
>
> Maison <
>
> mickael.maison@gmail.com>
> wrote:
>
> Bumping this thread. Considering this KIP is
>
> relatively straigh
>
> forward, can we get some votes or feedback if
>
> you
>
> think it's
>
> not?
>
> Thanks
>
> On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar
>
> <
>
> edocomar@gmail.com>
>
> wrote:
>
> +1 (non-binding)
> Thanks Mickael!
>
> On Tue, 8 Jan 2019 at 17:39, Patrik
>
> Kleindl <
>
> pkleindl@gmail.com>
>
> wrote:
>
> +1 (non-binding)
> Thanks, sounds very helpful
> Best regards
> Patrik
>
> Am 08.01.2019 um 18:10 schrieb Mickael
>
> Maison <
>
> mickael.maison@gmail.com
>
> :
>
> Hi all,
>
> I'd like to start the vote on KIP-396:
>
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.
> apache.org%2Fconfluence%2Fpages%2Fviewpage.
> action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
>
> Thanks
>
> --
> "When the people fear their government,
>
> there is
>
> tyranny;
>
> when
>
> the
>
> government fears the people, there is
>
> liberty."
>
> [Thomas
>
> Jefferson]
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
> *github: <http://goog_969573159/>
>
> github.com/dongjinleekr
>
> <https://github.com/dongjinleekr>linkedin:
>
> kr.linkedin.com/in/dongjinleekr
>
> <https://kr.linkedin.com/in/dongjinleekr
>
> speakerdeck:
>
> speakerdeck.com/dongjin
> <https://speakerdeck.com/dongjin>*
>
> --
> -- Guozhang
>
>
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Colin McCabe <cm...@apache.org>.
Thanks, Mickael. +1 (binding)
best,
Colin
On Wed, Aug 14, 2019, at 12:07, Gabor Somogyi wrote:
> +1 (non-binding)
> I've read it through in depth and as Jungtaek said Spark can make good use
> of it.
>
> On Wed, 14 Aug 2019, 17:06 Jungtaek Lim, <ka...@gmail.com> wrote:
>
> > +1 (non-binding)
> >
> > I found it very useful for Spark's case. (Discussion on KIP-505 described
> > it.)
> >
> > Thanks for driving the effort!
> >
> > 2019년 8월 14일 (수) 오후 8:49, Mickael Maison <mi...@gmail.com>님이 작성:
> >
> > > Hi Guozhang,
> > >
> > > Thanks for taking a look.
> > >
> > > 1. Right, I updated the titles of the code blocks
> > >
> > > 2. Yes that's a good idea. I've updated the KIP
> > >
> > > Thank you
> > >
> > > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> > > <mi...@gmail.com> wrote:
> > > >
> > > > Hi Colin,
> > > >
> > > > Thanks for raising these 2 valid points. I've updated the KIP
> > > accordingly.
> > > >
> > > > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for the KIP!
> > > > >
> > > > > Just some minor comments.
> > > > >
> > > > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java"
> > should
> > > be
> > > > > "AlterOffsetsOptions".
> > > > >
> > > > > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> > > to
> > > > >
> > > > > *KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
> > > > >
> > > > > This is because we will have a hierarchy of two-layers of errors
> > since
> > > we
> > > > > need to find out the group coordinator first and then issue the
> > commit
> > > > > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> > > exclude
> > > > > partitions that have errors, or the DeleteMembersResult as part of
> > > KIP-345).
> > > > >
> > > > > If the discover-coordinator returns non-triable error, we would set
> > it
> > > on
> > > > > the first layer of the KafkaFuture, and the per-partition error would
> > > be
> > > > > set on the second layer of the KafkaFuture.
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > Considering that KIP-496, which adds a way of deleting consumer
> > > offsets
> > > > > > from AdminClient, looks like it is going to get in, this seems like
> > > > > > functionality we should definitely have.
> > > > > >
> > > > > > For alterConsumerGroupOffsets, is the intention to ignore
> > partitions
> > > that
> > > > > > are not specified in the map? If so, we should specify that in the
> > > JavaDoc.
> > > > > >
> > > > > > isolationLevel seems like it should be an enum rather than a
> > > string. The
> > > > > > existing enum is in org.apache.kafka.common.requests, so we should
> > > probably
> > > > > > create a new one which is public in org.apache.kafka.clients.admin.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > > > > > Bumping this thread once again
> > > > > > >
> > > > > > > Ismael, have I answered your questions?
> > > > > > > While this has received a few non-binding +1s, no committers have
> > > > > > > voted yet. If you have concerns or questions, please let me know.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > > > > <mi...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Bumping this thread as it's been a couple of weeks.
> > > > > > > >
> > > > > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > > > > mickael.maison@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Thanks Ismael for the feedback. I think your point has 2
> > parts:
> > > > > > > > > - Having the reset functionality in the AdminClient:
> > > > > > > > > The fact we have a command line tool illustrate that this
> > > operation
> > > > > > is
> > > > > > > > > relatively common. I seems valuable to be able to perform
> > this
> > > > > > > > > operation directly via a proper API in addition of the CLI
> > > tool.
> > > > > > > > >
> > > > > > > > > - Sending an OffsetCommit directly instead of relying on
> > > > > > KafkaConsumer:
> > > > > > > > > The KafkaConsumer requires a lot of stuff to commit offsets.
> > > Its
> > > > > > group
> > > > > > > > > cannot change so you need to start a new Consumer every time,
> > > that
> > > > > > > > > creates new connections and overal sends more requests. Also
> > > there
> > > > > > are
> > > > > > > > > already a bunch of AdminClient APIs that have logic very
> > > close to
> > > > > > > > > what needs to be done to send a commit request, keeping the
> > > code
> > > > > > small
> > > > > > > > > and consistent.
> > > > > > > > >
> > > > > > > > > I've updated the KIP with these details and moved the 2nd
> > part
> > > to
> > > > > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > > > > >
> > > > > > > > > I hope this answers your question
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
> > ismaelj@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > The KIP doesn't discuss the option of using KafkaConsumer
> > > directly
> > > > > > as far
> > > > > > > > > > as I can tell. We have tried to avoid having the same
> > > > > > functionality in
> > > > > > > > > > multiple clients so it would be good to explain why this is
> > > > > > necessary here
> > > > > > > > > > (not saying it isn't).
> > > > > > > > > >
> > > > > > > > > > Ismael
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > > > > > mickael.maison@gmail.com
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks Ryanne for the feedback, all suggestions sounded
> > > good,
> > > > > > I've
> > > > > > > > > > > updated the KIP accordingly.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > >
> > > > > > > > > > > > But I suggest:
> > > > > > > > > > > >
> > > > > > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > > > > > >
> > > > > > > > > > > > - add to the motivation section why this is better than
> > > > > > constructing a
> > > > > > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > > > > > >
> > > > > > > > > > > > - add some rejected alternatives.
> > > > > > > > > > > >
> > > > > > > > > > > > Ryanne
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> > > dongjin@apache.org
> > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We have +4 non-binding for this vote. Is there any
> > > committer
> > > > > > who is
> > > > > > > > > > > > > interested in this issue?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Dongjin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> > > > > > eno.thereska@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael
> > Maison <
> > > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Bumping this thread. Considering this KIP is
> > > > > > relatively straigh
> > > > > > > > > > > > > > > forward, can we get some votes or feedback if
> > > you
> > > > > > think it's
> > > > > > > > > > > not?
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar
> > <
> > > > > > > > > > > edocomar@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik
> > Kleindl <
> > > > > > > > > > > pkleindl@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > > > > > > Best regards
> > > > > > > > > > > > > > > > > Patrik
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael
> > > Maison <
> > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > >
> > >
> > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > "When the people fear their government,
> > > there is
> > > > > > tyranny;
> > > > > > > > > > > when
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > government fears the people, there is
> > > liberty."
> > > > > > [Thomas
> > > > > > > > > > > > > Jefferson]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > *Dongjin Lee*
> > > > > > > > > > > > >
> > > > > > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > > > > *github: <http://goog_969573159/>
> > > github.com/dongjinleekr
> > > > > > > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr
> > >speakerdeck:
> > > > > > > > > > > > > speakerdeck.com/dongjin
> > > > > > > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > >
> >
>
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Gabor Somogyi <ga...@gmail.com>.
+1 (non-binding)
I've read it through in depth and as Jungtaek said Spark can make good use
of it.
On Wed, 14 Aug 2019, 17:06 Jungtaek Lim, <ka...@gmail.com> wrote:
> +1 (non-binding)
>
> I found it very useful for Spark's case. (Discussion on KIP-505 described
> it.)
>
> Thanks for driving the effort!
>
> 2019년 8월 14일 (수) 오후 8:49, Mickael Maison <mi...@gmail.com>님이 작성:
>
> > Hi Guozhang,
> >
> > Thanks for taking a look.
> >
> > 1. Right, I updated the titles of the code blocks
> >
> > 2. Yes that's a good idea. I've updated the KIP
> >
> > Thank you
> >
> > On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> > <mi...@gmail.com> wrote:
> > >
> > > Hi Colin,
> > >
> > > Thanks for raising these 2 valid points. I've updated the KIP
> > accordingly.
> > >
> > > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang <wa...@gmail.com>
> > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > Just some minor comments.
> > > >
> > > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java"
> should
> > be
> > > > "AlterOffsetsOptions".
> > > >
> > > > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> > to
> > > >
> > > > *KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
> > > >
> > > > This is because we will have a hierarchy of two-layers of errors
> since
> > we
> > > > need to find out the group coordinator first and then issue the
> commit
> > > > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> > exclude
> > > > partitions that have errors, or the DeleteMembersResult as part of
> > KIP-345).
> > > >
> > > > If the discover-coordinator returns non-triable error, we would set
> it
> > on
> > > > the first layer of the KafkaFuture, and the per-partition error would
> > be
> > > > set on the second layer of the KafkaFuture.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org>
> > wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Considering that KIP-496, which adds a way of deleting consumer
> > offsets
> > > > > from AdminClient, looks like it is going to get in, this seems like
> > > > > functionality we should definitely have.
> > > > >
> > > > > For alterConsumerGroupOffsets, is the intention to ignore
> partitions
> > that
> > > > > are not specified in the map? If so, we should specify that in the
> > JavaDoc.
> > > > >
> > > > > isolationLevel seems like it should be an enum rather than a
> > string. The
> > > > > existing enum is in org.apache.kafka.common.requests, so we should
> > probably
> > > > > create a new one which is public in org.apache.kafka.clients.admin.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > > > > Bumping this thread once again
> > > > > >
> > > > > > Ismael, have I answered your questions?
> > > > > > While this has received a few non-binding +1s, no committers have
> > > > > > voted yet. If you have concerns or questions, please let me know.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > > > <mi...@gmail.com> wrote:
> > > > > > >
> > > > > > > Bumping this thread as it's been a couple of weeks.
> > > > > > >
> > > > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > > > mickael.maison@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Thanks Ismael for the feedback. I think your point has 2
> parts:
> > > > > > > > - Having the reset functionality in the AdminClient:
> > > > > > > > The fact we have a command line tool illustrate that this
> > operation
> > > > > is
> > > > > > > > relatively common. I seems valuable to be able to perform
> this
> > > > > > > > operation directly via a proper API in addition of the CLI
> > tool.
> > > > > > > >
> > > > > > > > - Sending an OffsetCommit directly instead of relying on
> > > > > KafkaConsumer:
> > > > > > > > The KafkaConsumer requires a lot of stuff to commit offsets.
> > Its
> > > > > group
> > > > > > > > cannot change so you need to start a new Consumer every time,
> > that
> > > > > > > > creates new connections and overal sends more requests. Also
> > there
> > > > > are
> > > > > > > > already a bunch of AdminClient APIs that have logic very
> > close to
> > > > > > > > what needs to be done to send a commit request, keeping the
> > code
> > > > > small
> > > > > > > > and consistent.
> > > > > > > >
> > > > > > > > I've updated the KIP with these details and moved the 2nd
> part
> > to
> > > > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > > > >
> > > > > > > > I hope this answers your question
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <
> ismaelj@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > The KIP doesn't discuss the option of using KafkaConsumer
> > directly
> > > > > as far
> > > > > > > > > as I can tell. We have tried to avoid having the same
> > > > > functionality in
> > > > > > > > > multiple clients so it would be good to explain why this is
> > > > > necessary here
> > > > > > > > > (not saying it isn't).
> > > > > > > > >
> > > > > > > > > Ismael
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > > > > mickael.maison@gmail.com
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks Ryanne for the feedback, all suggestions sounded
> > good,
> > > > > I've
> > > > > > > > > > updated the KIP accordingly.
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > > > > ryannedolan@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > >
> > > > > > > > > > > But I suggest:
> > > > > > > > > > >
> > > > > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > > > > >
> > > > > > > > > > > - add to the motivation section why this is better than
> > > > > constructing a
> > > > > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > > > > >
> > > > > > > > > > > - add some rejected alternatives.
> > > > > > > > > > >
> > > > > > > > > > > Ryanne
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> > dongjin@apache.org
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > We have +4 non-binding for this vote. Is there any
> > committer
> > > > > who is
> > > > > > > > > > > > interested in this issue?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Dongjin
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> > > > > eno.thereska@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael
> Maison <
> > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Bumping this thread. Considering this KIP is
> > > > > relatively straigh
> > > > > > > > > > > > > > forward, can we get some votes or feedback if
> > you
> > > > > think it's
> > > > > > > > > > not?
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar
> <
> > > > > > > > > > edocomar@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik
> Kleindl <
> > > > > > > > > > pkleindl@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > > > > > Best regards
> > > > > > > > > > > > > > > > Patrik
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael
> > Maison <
> > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > >
> >
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > "When the people fear their government,
> > there is
> > > > > tyranny;
> > > > > > > > > > when
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > government fears the people, there is
> > liberty."
> > > > > [Thomas
> > > > > > > > > > > > Jefferson]
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > *Dongjin Lee*
> > > > > > > > > > > >
> > > > > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > > > *github: <http://goog_969573159/>
> > github.com/dongjinleekr
> > > > > > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr
> >speakerdeck:
> > > > > > > > > > > > speakerdeck.com/dongjin
> > > > > > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> >
>
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Jungtaek Lim <ka...@gmail.com>.
+1 (non-binding)
I found it very useful for Spark's case. (Discussion on KIP-505 described
it.)
Thanks for driving the effort!
2019년 8월 14일 (수) 오후 8:49, Mickael Maison <mi...@gmail.com>님이 작성:
> Hi Guozhang,
>
> Thanks for taking a look.
>
> 1. Right, I updated the titles of the code blocks
>
> 2. Yes that's a good idea. I've updated the KIP
>
> Thank you
>
> On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
> <mi...@gmail.com> wrote:
> >
> > Hi Colin,
> >
> > Thanks for raising these 2 valid points. I've updated the KIP
> accordingly.
> >
> > On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang <wa...@gmail.com>
> wrote:
> > >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP!
> > >
> > > Just some minor comments.
> > >
> > > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java" should
> be
> > > "AlterOffsetsOptions".
> > >
> > > 2. I'd suggest we change the future structure of "AlterOffsetsResult"
> to
> > >
> > > *KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
> > >
> > > This is because we will have a hierarchy of two-layers of errors since
> we
> > > need to find out the group coordinator first and then issue the commit
> > > offset request (see e.g. the ListConsumerGroupOffsetsResult which
> exclude
> > > partitions that have errors, or the DeleteMembersResult as part of
> KIP-345).
> > >
> > > If the discover-coordinator returns non-triable error, we would set it
> on
> > > the first layer of the KafkaFuture, and the per-partition error would
> be
> > > set on the second layer of the KafkaFuture.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org>
> wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > Considering that KIP-496, which adds a way of deleting consumer
> offsets
> > > > from AdminClient, looks like it is going to get in, this seems like
> > > > functionality we should definitely have.
> > > >
> > > > For alterConsumerGroupOffsets, is the intention to ignore partitions
> that
> > > > are not specified in the map? If so, we should specify that in the
> JavaDoc.
> > > >
> > > > isolationLevel seems like it should be an enum rather than a
> string. The
> > > > existing enum is in org.apache.kafka.common.requests, so we should
> probably
> > > > create a new one which is public in org.apache.kafka.clients.admin.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > > > Bumping this thread once again
> > > > >
> > > > > Ismael, have I answered your questions?
> > > > > While this has received a few non-binding +1s, no committers have
> > > > > voted yet. If you have concerns or questions, please let me know.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > > <mi...@gmail.com> wrote:
> > > > > >
> > > > > > Bumping this thread as it's been a couple of weeks.
> > > > > >
> > > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > > mickael.maison@gmail.com> wrote:
> > > > > > >
> > > > > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > > > > - Having the reset functionality in the AdminClient:
> > > > > > > The fact we have a command line tool illustrate that this
> operation
> > > > is
> > > > > > > relatively common. I seems valuable to be able to perform this
> > > > > > > operation directly via a proper API in addition of the CLI
> tool.
> > > > > > >
> > > > > > > - Sending an OffsetCommit directly instead of relying on
> > > > KafkaConsumer:
> > > > > > > The KafkaConsumer requires a lot of stuff to commit offsets.
> Its
> > > > group
> > > > > > > cannot change so you need to start a new Consumer every time,
> that
> > > > > > > creates new connections and overal sends more requests. Also
> there
> > > > are
> > > > > > > already a bunch of AdminClient APIs that have logic very
> close to
> > > > > > > what needs to be done to send a commit request, keeping the
> code
> > > > small
> > > > > > > and consistent.
> > > > > > >
> > > > > > > I've updated the KIP with these details and moved the 2nd part
> to
> > > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > > >
> > > > > > > I hope this answers your question
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <ismaelj@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > The KIP doesn't discuss the option of using KafkaConsumer
> directly
> > > > as far
> > > > > > > > as I can tell. We have tried to avoid having the same
> > > > functionality in
> > > > > > > > multiple clients so it would be good to explain why this is
> > > > necessary here
> > > > > > > > (not saying it isn't).
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > > > mickael.maison@gmail.com
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Ryanne for the feedback, all suggestions sounded
> good,
> > > > I've
> > > > > > > > > updated the KIP accordingly.
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > > > ryannedolan@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1 (non-binding)
> > > > > > > > > >
> > > > > > > > > > But I suggest:
> > > > > > > > > >
> > > > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > > > >
> > > > > > > > > > - add to the motivation section why this is better than
> > > > constructing a
> > > > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > > > >
> > > > > > > > > > - add some rejected alternatives.
> > > > > > > > > >
> > > > > > > > > > Ryanne
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <
> dongjin@apache.org
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > We have +4 non-binding for this vote. Is there any
> committer
> > > > who is
> > > > > > > > > > > interested in this issue?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Dongjin
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> > > > eno.thereska@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Bumping this thread. Considering this KIP is
> > > > relatively straigh
> > > > > > > > > > > > > forward, can we get some votes or feedback if
> you
> > > > think it's
> > > > > > > > > not?
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > > > > > edocomar@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > > > > > pkleindl@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > > > > Best regards
> > > > > > > > > > > > > > > Patrik
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael
> Maison <
> > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > >
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > "When the people fear their government,
> there is
> > > > tyranny;
> > > > > > > > > when
> > > > > > > > > > > the
> > > > > > > > > > > > > > government fears the people, there is
> liberty."
> > > > [Thomas
> > > > > > > > > > > Jefferson]
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *Dongjin Lee*
> > > > > > > > > > >
> > > > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > > *github: <http://goog_969573159/>
> github.com/dongjinleekr
> > > > > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > > > > > > speakerdeck.com/dongjin
> > > > > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
>
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Mickael Maison <mi...@gmail.com>.
Hi Guozhang,
Thanks for taking a look.
1. Right, I updated the titles of the code blocks
2. Yes that's a good idea. I've updated the KIP
Thank you
On Wed, Aug 14, 2019 at 11:05 AM Mickael Maison
<mi...@gmail.com> wrote:
>
> Hi Colin,
>
> Thanks for raising these 2 valid points. I've updated the KIP accordingly.
>
> On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang <wa...@gmail.com> wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP!
> >
> > Just some minor comments.
> >
> > 1. Java class names are stale, e.g. "CommitOffsetsOptions.java" should be
> > "AlterOffsetsOptions".
> >
> > 2. I'd suggest we change the future structure of "AlterOffsetsResult" to
> >
> > *KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
> >
> > This is because we will have a hierarchy of two-layers of errors since we
> > need to find out the group coordinator first and then issue the commit
> > offset request (see e.g. the ListConsumerGroupOffsetsResult which exclude
> > partitions that have errors, or the DeleteMembersResult as part of KIP-345).
> >
> > If the discover-coordinator returns non-triable error, we would set it on
> > the first layer of the KafkaFuture, and the per-partition error would be
> > set on the second layer of the KafkaFuture.
> >
> >
> > Guozhang
> >
> >
> > On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Hi Mickael,
> > >
> > > Considering that KIP-496, which adds a way of deleting consumer offsets
> > > from AdminClient, looks like it is going to get in, this seems like
> > > functionality we should definitely have.
> > >
> > > For alterConsumerGroupOffsets, is the intention to ignore partitions that
> > > are not specified in the map? If so, we should specify that in the JavaDoc.
> > >
> > > isolationLevel seems like it should be an enum rather than a string. The
> > > existing enum is in org.apache.kafka.common.requests, so we should probably
> > > create a new one which is public in org.apache.kafka.clients.admin.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > > Bumping this thread once again
> > > >
> > > > Ismael, have I answered your questions?
> > > > While this has received a few non-binding +1s, no committers have
> > > > voted yet. If you have concerns or questions, please let me know.
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > > <mi...@gmail.com> wrote:
> > > > >
> > > > > Bumping this thread as it's been a couple of weeks.
> > > > >
> > > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > > mickael.maison@gmail.com> wrote:
> > > > > >
> > > > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > > > - Having the reset functionality in the AdminClient:
> > > > > > The fact we have a command line tool illustrate that this operation
> > > is
> > > > > > relatively common. I seems valuable to be able to perform this
> > > > > > operation directly via a proper API in addition of the CLI tool.
> > > > > >
> > > > > > - Sending an OffsetCommit directly instead of relying on
> > > KafkaConsumer:
> > > > > > The KafkaConsumer requires a lot of stuff to commit offsets. Its
> > > group
> > > > > > cannot change so you need to start a new Consumer every time, that
> > > > > > creates new connections and overal sends more requests. Also there
> > > are
> > > > > > already a bunch of AdminClient APIs that have logic very close to
> > > > > > what needs to be done to send a commit request, keeping the code
> > > small
> > > > > > and consistent.
> > > > > >
> > > > > > I've updated the KIP with these details and moved the 2nd part to
> > > > > > "Proposed changes" as it's more an implementation detail.
> > > > > >
> > > > > > I hope this answers your question
> > > > > >
> > > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > The KIP doesn't discuss the option of using KafkaConsumer directly
> > > as far
> > > > > > > as I can tell. We have tried to avoid having the same
> > > functionality in
> > > > > > > multiple clients so it would be good to explain why this is
> > > necessary here
> > > > > > > (not saying it isn't).
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > > mickael.maison@gmail.com
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Ryanne for the feedback, all suggestions sounded good,
> > > I've
> > > > > > > > updated the KIP accordingly.
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > > ryannedolan@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > +1 (non-binding)
> > > > > > > > >
> > > > > > > > > But I suggest:
> > > > > > > > >
> > > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > > >
> > > > > > > > > - add to the motivation section why this is better than
> > > constructing a
> > > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > > >
> > > > > > > > > - add some rejected alternatives.
> > > > > > > > >
> > > > > > > > > Ryanne
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <dongjin@apache.org
> > > wrote:
> > > > > > > > >
> > > > > > > > > > We have +4 non-binding for this vote. Is there any committer
> > > who is
> > > > > > > > > > interested in this issue?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Dongjin
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > > > >
> > > > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> > > eno.thereska@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Bumping this thread. Considering this KIP is
> > > relatively straigh
> > > > > > > > > > > > forward, can we get some votes or feedback if you
> > > think it's
> > > > > > > > not?
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > > > > edocomar@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > > > > pkleindl@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > > > Best regards
> > > > > > > > > > > > > > Patrik
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael Maison <
> > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > "When the people fear their government, there is
> > > tyranny;
> > > > > > > > when
> > > > > > > > > > the
> > > > > > > > > > > > > government fears the people, there is liberty."
> > > [Thomas
> > > > > > > > > > Jefferson]
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Dongjin Lee*
> > > > > > > > > >
> > > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > > > > > speakerdeck.com/dongjin
> > > > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Mickael Maison <mi...@gmail.com>.
Hi Colin,
Thanks for raising these 2 valid points. I've updated the KIP accordingly.
On Tue, Aug 13, 2019 at 9:50 PM Guozhang Wang <wa...@gmail.com> wrote:
>
> Hi Mickael,
>
> Thanks for the KIP!
>
> Just some minor comments.
>
> 1. Java class names are stale, e.g. "CommitOffsetsOptions.java" should be
> "AlterOffsetsOptions".
>
> 2. I'd suggest we change the future structure of "AlterOffsetsResult" to
>
> *KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
>
> This is because we will have a hierarchy of two-layers of errors since we
> need to find out the group coordinator first and then issue the commit
> offset request (see e.g. the ListConsumerGroupOffsetsResult which exclude
> partitions that have errors, or the DeleteMembersResult as part of KIP-345).
>
> If the discover-coordinator returns non-triable error, we would set it on
> the first layer of the KafkaFuture, and the per-partition error would be
> set on the second layer of the KafkaFuture.
>
>
> Guozhang
>
>
> On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Mickael,
> >
> > Considering that KIP-496, which adds a way of deleting consumer offsets
> > from AdminClient, looks like it is going to get in, this seems like
> > functionality we should definitely have.
> >
> > For alterConsumerGroupOffsets, is the intention to ignore partitions that
> > are not specified in the map? If so, we should specify that in the JavaDoc.
> >
> > isolationLevel seems like it should be an enum rather than a string. The
> > existing enum is in org.apache.kafka.common.requests, so we should probably
> > create a new one which is public in org.apache.kafka.clients.admin.
> >
> > best,
> > Colin
> >
> >
> > On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > > Bumping this thread once again
> > >
> > > Ismael, have I answered your questions?
> > > While this has received a few non-binding +1s, no committers have
> > > voted yet. If you have concerns or questions, please let me know.
> > >
> > > Thanks
> > >
> > > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > > <mi...@gmail.com> wrote:
> > > >
> > > > Bumping this thread as it's been a couple of weeks.
> > > >
> > > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> > mickael.maison@gmail.com> wrote:
> > > > >
> > > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > > - Having the reset functionality in the AdminClient:
> > > > > The fact we have a command line tool illustrate that this operation
> > is
> > > > > relatively common. I seems valuable to be able to perform this
> > > > > operation directly via a proper API in addition of the CLI tool.
> > > > >
> > > > > - Sending an OffsetCommit directly instead of relying on
> > KafkaConsumer:
> > > > > The KafkaConsumer requires a lot of stuff to commit offsets. Its
> > group
> > > > > cannot change so you need to start a new Consumer every time, that
> > > > > creates new connections and overal sends more requests. Also there
> > are
> > > > > already a bunch of AdminClient APIs that have logic very close to
> > > > > what needs to be done to send a commit request, keeping the code
> > small
> > > > > and consistent.
> > > > >
> > > > > I've updated the KIP with these details and moved the 2nd part to
> > > > > "Proposed changes" as it's more an implementation detail.
> > > > >
> > > > > I hope this answers your question
> > > > >
> > > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com>
> > wrote:
> > > > > >
> > > > > > The KIP doesn't discuss the option of using KafkaConsumer directly
> > as far
> > > > > > as I can tell. We have tried to avoid having the same
> > functionality in
> > > > > > multiple clients so it would be good to explain why this is
> > necessary here
> > > > > > (not saying it isn't).
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> > mickael.maison@gmail.com
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Ryanne for the feedback, all suggestions sounded good,
> > I've
> > > > > > > updated the KIP accordingly.
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> > ryannedolan@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > But I suggest:
> > > > > > > >
> > > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > > >
> > > > > > > > - add to the motivation section why this is better than
> > constructing a
> > > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > > >
> > > > > > > > - add some rejected alternatives.
> > > > > > > >
> > > > > > > > Ryanne
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <dongjin@apache.org
> > wrote:
> > > > > > > >
> > > > > > > > > We have +4 non-binding for this vote. Is there any committer
> > who is
> > > > > > > > > interested in this issue?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dongjin
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > > andrew_schofield@live.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > > >
> > > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> > eno.thereska@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Bumping this thread. Considering this KIP is
> > relatively straigh
> > > > > > > > > > > forward, can we get some votes or feedback if you
> > think it's
> > > > > > > not?
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > > > edocomar@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > > > pkleindl@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > > Best regards
> > > > > > > > > > > > > Patrik
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > "When the people fear their government, there is
> > tyranny;
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > > > government fears the people, there is liberty."
> > [Thomas
> > > > > > > > > Jefferson]
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Dongjin Lee*
> > > > > > > > >
> > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > > > > speakerdeck.com/dongjin
> > > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > >
> >
>
>
> --
> -- Guozhang
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Mickael,
Thanks for the KIP!
Just some minor comments.
1. Java class names are stale, e.g. "CommitOffsetsOptions.java" should be
"AlterOffsetsOptions".
2. I'd suggest we change the future structure of "AlterOffsetsResult" to
*KafkaFuture<Map<TopicPartition, KafkaFuture<Void>>>*
This is because we will have a hierarchy of two-layers of errors since we
need to find out the group coordinator first and then issue the commit
offset request (see e.g. the ListConsumerGroupOffsetsResult which exclude
partitions that have errors, or the DeleteMembersResult as part of KIP-345).
If the discover-coordinator returns non-triable error, we would set it on
the first layer of the KafkaFuture, and the per-partition error would be
set on the second layer of the KafkaFuture.
Guozhang
On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe <cm...@apache.org> wrote:
> Hi Mickael,
>
> Considering that KIP-496, which adds a way of deleting consumer offsets
> from AdminClient, looks like it is going to get in, this seems like
> functionality we should definitely have.
>
> For alterConsumerGroupOffsets, is the intention to ignore partitions that
> are not specified in the map? If so, we should specify that in the JavaDoc.
>
> isolationLevel seems like it should be an enum rather than a string. The
> existing enum is in org.apache.kafka.common.requests, so we should probably
> create a new one which is public in org.apache.kafka.clients.admin.
>
> best,
> Colin
>
>
> On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > Bumping this thread once again
> >
> > Ismael, have I answered your questions?
> > While this has received a few non-binding +1s, no committers have
> > voted yet. If you have concerns or questions, please let me know.
> >
> > Thanks
> >
> > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> > <mi...@gmail.com> wrote:
> > >
> > > Bumping this thread as it's been a couple of weeks.
> > >
> > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> mickael.maison@gmail.com> wrote:
> > > >
> > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > - Having the reset functionality in the AdminClient:
> > > > The fact we have a command line tool illustrate that this operation
> is
> > > > relatively common. I seems valuable to be able to perform this
> > > > operation directly via a proper API in addition of the CLI tool.
> > > >
> > > > - Sending an OffsetCommit directly instead of relying on
> KafkaConsumer:
> > > > The KafkaConsumer requires a lot of stuff to commit offsets. Its
> group
> > > > cannot change so you need to start a new Consumer every time, that
> > > > creates new connections and overal sends more requests. Also there
> are
> > > > already a bunch of AdminClient APIs that have logic very close to
> > > > what needs to be done to send a commit request, keeping the code
> small
> > > > and consistent.
> > > >
> > > > I've updated the KIP with these details and moved the 2nd part to
> > > > "Proposed changes" as it's more an implementation detail.
> > > >
> > > > I hope this answers your question
> > > >
> > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com>
> wrote:
> > > > >
> > > > > The KIP doesn't discuss the option of using KafkaConsumer directly
> as far
> > > > > as I can tell. We have tried to avoid having the same
> functionality in
> > > > > multiple clients so it would be good to explain why this is
> necessary here
> > > > > (not saying it isn't).
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> mickael.maison@gmail.com
> > > > > wrote:
> > > > >
> > > > > > Thanks Ryanne for the feedback, all suggestions sounded good,
> I've
> > > > > > updated the KIP accordingly.
> > > > > >
> > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> ryannedolan@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > But I suggest:
> > > > > > >
> > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > >
> > > > > > > - add to the motivation section why this is better than
> constructing a
> > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > >
> > > > > > > - add some rejected alternatives.
> > > > > > >
> > > > > > > Ryanne
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <dongjin@apache.org
> wrote:
> > > > > > >
> > > > > > > > We have +4 non-binding for this vote. Is there any committer
> who is
> > > > > > > > interested in this issue?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dongjin
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > andrew_schofield@live.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > >
> > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> eno.thereska@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Bumping this thread. Considering this KIP is
> relatively straigh
> > > > > > > > > > forward, can we get some votes or feedback if you
> think it's
> > > > > > not?
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > > edocomar@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > Thanks Mickael!
> > > > > > > > > > >
> > > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > > pkleindl@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > > Best regards
> > > > > > > > > > > > Patrik
> > > > > > > > > > > >
> > > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > "When the people fear their government, there is
> tyranny;
> > > > > > when
> > > > > > > > the
> > > > > > > > > > > government fears the people, there is liberty."
> [Thomas
> > > > > > > > Jefferson]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Dongjin Lee*
> > > > > > > >
> > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > > > speakerdeck.com/dongjin
> > > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > > >
> > > > > > > >
> > > > > >
> >
>
--
-- Guozhang
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Colin McCabe <cm...@apache.org>.
Hi Mickael,
Considering that KIP-496, which adds a way of deleting consumer offsets from AdminClient, looks like it is going to get in, this seems like functionality we should definitely have.
For alterConsumerGroupOffsets, is the intention to ignore partitions that are not specified in the map? If so, we should specify that in the JavaDoc.
isolationLevel seems like it should be an enum rather than a string. The existing enum is in org.apache.kafka.common.requests, so we should probably create a new one which is public in org.apache.kafka.clients.admin.
best,
Colin
On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> Bumping this thread once again
>
> Ismael, have I answered your questions?
> While this has received a few non-binding +1s, no committers have
> voted yet. If you have concerns or questions, please let me know.
>
> Thanks
>
> On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> <mi...@gmail.com> wrote:
> >
> > Bumping this thread as it's been a couple of weeks.
> >
> > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <mi...@gmail.com> wrote:
> > >
> > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > - Having the reset functionality in the AdminClient:
> > > The fact we have a command line tool illustrate that this operation is
> > > relatively common. I seems valuable to be able to perform this
> > > operation directly via a proper API in addition of the CLI tool.
> > >
> > > - Sending an OffsetCommit directly instead of relying on KafkaConsumer:
> > > The KafkaConsumer requires a lot of stuff to commit offsets. Its group
> > > cannot change so you need to start a new Consumer every time, that
> > > creates new connections and overal sends more requests. Also there are
> > > already a bunch of AdminClient APIs that have logic very close to
> > > what needs to be done to send a commit request, keeping the code small
> > > and consistent.
> > >
> > > I've updated the KIP with these details and moved the 2nd part to
> > > "Proposed changes" as it's more an implementation detail.
> > >
> > > I hope this answers your question
> > >
> > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com> wrote:
> > > >
> > > > The KIP doesn't discuss the option of using KafkaConsumer directly as far
> > > > as I can tell. We have tried to avoid having the same functionality in
> > > > multiple clients so it would be good to explain why this is necessary here
> > > > (not saying it isn't).
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <mickael.maison@gmail.com
> > > > wrote:
> > > >
> > > > > Thanks Ryanne for the feedback, all suggestions sounded good, I've
> > > > > updated the KIP accordingly.
> > > > >
> > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <ry...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > But I suggest:
> > > > > >
> > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > >
> > > > > > - add to the motivation section why this is better than constructing a
> > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > >
> > > > > > - add some rejected alternatives.
> > > > > >
> > > > > > Ryanne
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <dongjin@apache.org wrote:
> > > > > >
> > > > > > > We have +4 non-binding for this vote. Is there any committer who is
> > > > > > > interested in this issue?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dongjin
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > andrew_schofield@live.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > >
> > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <en...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > +1 (non binding). Thanks.
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Bumping this thread. Considering this KIP is relatively straigh
> > > > > > > > > forward, can we get some votes or feedback if you think it's
> > > > > not?
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > edocomar@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1 (non-binding)
> > > > > > > > > > Thanks Mickael!
> > > > > > > > > >
> > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > pkleindl@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > Best regards
> > > > > > > > > > > Patrik
> > > > > > > > > > >
> > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > "When the people fear their government, there is tyranny;
> > > > > when
> > > > > > > the
> > > > > > > > > > government fears the people, there is liberty." [Thomas
> > > > > > > Jefferson]
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Dongjin Lee*
> > > > > > >
> > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > > speakerdeck.com/dongjin
> > > > > > > <https://speakerdeck.com/dongjin>*
> > > > > > >
> > > > > > >
> > > > >
>
Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient
Posted by Mickael Maison <mi...@gmail.com>.
Bumping this thread once again
Ismael, have I answered your questions?
While this has received a few non-binding +1s, no committers have
voted yet. If you have concerns or questions, please let me know.
Thanks
On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
<mi...@gmail.com> wrote:
>
> Bumping this thread as it's been a couple of weeks.
>
> On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <mi...@gmail.com> wrote:
> >
> > Thanks Ismael for the feedback. I think your point has 2 parts:
> > - Having the reset functionality in the AdminClient:
> > The fact we have a command line tool illustrate that this operation is
> > relatively common. I seems valuable to be able to perform this
> > operation directly via a proper API in addition of the CLI tool.
> >
> > - Sending an OffsetCommit directly instead of relying on KafkaConsumer:
> > The KafkaConsumer requires a lot of stuff to commit offsets. Its group
> > cannot change so you need to start a new Consumer every time, that
> > creates new connections and overal sends more requests. Also there are
> > already a bunch of AdminClient APIs that have logic very close to
> > what needs to be done to send a commit request, keeping the code small
> > and consistent.
> >
> > I've updated the KIP with these details and moved the 2nd part to
> > "Proposed changes" as it's more an implementation detail.
> >
> > I hope this answers your question
> >
> > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma <is...@gmail.com> wrote:
> > >
> > > The KIP doesn't discuss the option of using KafkaConsumer directly as far
> > > as I can tell. We have tried to avoid having the same functionality in
> > > multiple clients so it would be good to explain why this is necessary here
> > > (not saying it isn't).
> > >
> > > Ismael
> > >
> > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <mickael.maison@gmail.com
> > > wrote:
> > >
> > > > Thanks Ryanne for the feedback, all suggestions sounded good, I've
> > > > updated the KIP accordingly.
> > > >
> > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <ry...@gmail.com>
> > > > wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > But I suggest:
> > > > >
> > > > > - drop "get" from getOffset, getTimestamp.
> > > > >
> > > > > - add to the motivation section why this is better than constructing a
> > > > > KafkaConsumer and using seek(), commit() etc.
> > > > >
> > > > > - add some rejected alternatives.
> > > > >
> > > > > Ryanne
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee <dongjin@apache.org wrote:
> > > > >
> > > > > > We have +4 non-binding for this vote. Is there any committer who is
> > > > > > interested in this issue?
> > > > > >
> > > > > > Thanks,
> > > > > > Dongjin
> > > > > >
> > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > andrew_schofield@live.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > >
> > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <en...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > +1 (non binding). Thanks.
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > mickael.maison@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Bumping this thread. Considering this KIP is relatively straigh
> > > > > > > > forward, can we get some votes or feedback if you think it's
> > > > not?
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > edocomar@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > +1 (non-binding)
> > > > > > > > > Thanks Mickael!
> > > > > > > > >
> > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > pkleindl@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 (non-binding)
> > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > Best regards
> > > > > > > > > > Patrik
> > > > > > > > > >
> > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael Maison <
> > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I'd like to start the vote on KIP-396:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D97551484&data=02%7C01%7C%7C47c103e1919142c35d7c08d67f9e4c5d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636836715187389495&sdata=ihLaSXvB8C%2BK%2F%2BWjVDqKXgUJoRDmwfIi7FvFLRzmFe4%3D&reserved=0
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > "When the people fear their government, there is tyranny;
> > > > when
> > > > > > the
> > > > > > > > > government fears the people, there is liberty." [Thomas
> > > > > > Jefferson]
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > *Dongjin Lee*
> > > > > >
> > > > > > *A hitchhiker in the mathematical world.*
> > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
> > > > > > <https://github.com/dongjinleekr>linkedin:
> > > > kr.linkedin.com/in/dongjinleekr
> > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck:
> > > > > > speakerdeck.com/dongjin
> > > > > > <https://speakerdeck.com/dongjin>*
> > > > > >
> > > > > >
> > > >