You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Vahid S Hashemian <va...@us.ibm.com> on 2017/01/05 20:09:50 UTC
[DISCUSS] KIP-109: Old Consumer Deprecation
Hi all,
There was some discussion recently on deprecating the old consumer (
https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
Ismael suggested to cover the discussion and voting of major deprecations
like this under a KIP.
So I started KIP-109 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-109%3A+Old+Consumer+Deprecation
) and look forward to your feedback and comments.
We'd like to implement this deprecation in the upcoming 0.10.2.0 release.
Thanks.
--Vahid
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Tom Crayford <tc...@heroku.com>.
It's not time for voting, but a huge +1 from me.
On Thu, Jan 5, 2017 at 8:25 PM, Vahid S Hashemian <vahidhashemian@us.ibm.com
> wrote:
> Thanks Ismael. I added that to the KIP.
>
>
>
>
> From: Ismael Juma <is...@juma.me.uk>
> To: dev@kafka.apache.org
> Date: 01/05/2017 12:16 PM
> Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> Sent by: ismaelj@gmail.com
>
>
>
> Thanks Vahid, +1 (predictably). Worth mentioning in the KIP that
> compatibility with older brokers (0.10.0 and later) is another feature
> that
> will only be supported by the Java consumer.
>
> Ismael
>
> On 5 Jan 2017 8:10 pm, "Vahid S Hashemian" <va...@us.ibm.com>
> wrote:
>
> > Hi all,
> >
> > There was some discussion recently on deprecating the old consumer (
> > https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
> > Ismael suggested to cover the discussion and voting of major
> deprecations
> > like this under a KIP.
> >
> > So I started KIP-109 (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-109%3A+Old+Consumer+
> > Deprecation
> > ) and look forward to your feedback and comments.
> >
> > We'd like to implement this deprecation in the upcoming 0.10.2.0
> release.
> >
> > Thanks.
> > --Vahid
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Thanks Ismael. I added that to the KIP.
From: Ismael Juma <is...@juma.me.uk>
To: dev@kafka.apache.org
Date: 01/05/2017 12:16 PM
Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Sent by: ismaelj@gmail.com
Thanks Vahid, +1 (predictably). Worth mentioning in the KIP that
compatibility with older brokers (0.10.0 and later) is another feature
that
will only be supported by the Java consumer.
Ismael
On 5 Jan 2017 8:10 pm, "Vahid S Hashemian" <va...@us.ibm.com>
wrote:
> Hi all,
>
> There was some discussion recently on deprecating the old consumer (
> https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
> Ismael suggested to cover the discussion and voting of major
deprecations
> like this under a KIP.
>
> So I started KIP-109 (
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-109%3A+Old+Consumer+
> Deprecation
> ) and look forward to your feedback and comments.
>
> We'd like to implement this deprecation in the upcoming 0.10.2.0
release.
>
> Thanks.
> --Vahid
>
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks Vahid, +1 (predictably). Worth mentioning in the KIP that
compatibility with older brokers (0.10.0 and later) is another feature that
will only be supported by the Java consumer.
Ismael
On 5 Jan 2017 8:10 pm, "Vahid S Hashemian" <va...@us.ibm.com>
wrote:
> Hi all,
>
> There was some discussion recently on deprecating the old consumer (
> https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
> Ismael suggested to cover the discussion and voting of major deprecations
> like this under a KIP.
>
> So I started KIP-109 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-109%3A+Old+Consumer+
> Deprecation
> ) and look forward to your feedback and comments.
>
> We'd like to implement this deprecation in the upcoming 0.10.2.0 release.
>
> Thanks.
> --Vahid
>
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Joel Koshy <jj...@gmail.com>.
On Tue, Jan 10, 2017 at 12:53 PM, Renu Tewari <te...@gmail.com> wrote:
> Hi Ismael,
> What are the expected timelines we are talking about between the major
> releases? At LI we are expecting to have atleast 1 year between the old
> consumer deprecation and removal so we have enough time to upgrade all
> applications. The rollout to new consumer has hit many hurdles so hasn't
> proceeded at the expected pace. What impact would an official deprecation
> have on applications? Any warnings would be disruptive and would prefer
> that happens when there is a migration plan in place so we have a bound on
> how long it will take. There are too many unknowns at this time.
>
> Thoughts on timelines?
>
> regards
> Renu
>
> On Mon, Jan 9, 2017 at 6:34 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Joel,
> >
> > Great to hear that LinkedIn is likely to implement KAFKA-4513. :)
> >
> > Yes, the KIP as it stands is a compromise given the situation. We could
> > push the deprecation to the subsequent release: likely to be 0.11.0.0
> since
> > there are a number of KIPs that require message format changes. This
> would
> > mean that the old consumers would not be removed before 0.12.0.0 (the
> major
> > release after 0.11.0.0). Would that work better for you all?
>
It helps, but the main concern is deprecating before implementing the
migration path. So this means merging in the deprecation PR right after
cutting 0.10.2 is also going to be problematic since we release off trunk.
So we can prioritize working on KAFKA-4513.
@Ewen: good question on message format changes - I agree with Ismael that
for features such as a new compression scheme we can do without a format
change. I don't think we have any formal guidance on the scenarios that you
highlighted at this point so it may help to have a discussion on a separate
thread and codify that in our docs under a new "Kafka message and protocol
versioning" section.
Thanks,
Joel
> >
> > On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy <jj...@gmail.com>
> wrote:
> >
> > > >
> > > >
> > > > The ideal scenario would be for us to provide a tool for no downtime
> > > > migration as discussed in the original thread (I filed
> > > > https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
> > > > discussion). There are a few issues, however:
> > > >
> > > > - There doesn't seem to be much demand for it (outside of
> LinkedIn,
> > at
> > > > least)
> > > > - No-one is working on it or has indicated that they are planning
> to
> > > > work on it
> > > > - It's a non-trivial change and it requires a good amount of
> testing
> > > to
> > > > ensure it works as expected
> > > >
> > >
> > > For LinkedIn: while there are a few consuming applications for which
> the
> > > current shut-down and restart approach to migration will suffice, I
> doubt
> > > we will be able to do this for majority of services that are outside
> our
> > > direct control. Given that a seamless migration is a pre-req for us to
> > > switch to the new consumer widely (there are a few use-cases already on
> > it)
> > > it is something that we (LinkedIn) will likely implement although we
> > > haven't done further brainstorming/improvements beyond what was
> proposed
> > in
> > > the other deprecation thread.
> > >
> > >
> > > > In the meantime, we have this suboptimal situation where the old
> > > consumers
> > > > are close to unmaintained even though we don't say it outright: they
> > > don't
> > >
> > > get new features (basic things like security are missing) and bug fixes
> > are
> > > > rare. In practice, the old clients have been deprecated a while back,
> > we
> > > >
> > >
> > > Agreed that it is suboptimal, but the reality is that LI and I think a
> > few
> > > other companies are still to a large extent on the old consumer and
> will
> > be
> > > for at least a good part of this year. This does mean that we have the
> > > overhead of maintaining some internal workarounds for the old consumer.
> > >
> > >
> > > > just haven't made it official. This proposal is about rectifying that
> > so
> > > > that we communicate our intentions to our users more clearly. As
> Vahid
> > > > said, this KIP is not about changing how we maintain the existing
> code.
> > > >
> > > > The KIP that proposes the removal of all the old clients will be more
> > > > interesting, but it doesn't exist yet. :)
> > > >
> > >
> > > Deprecating *after* providing a sound migration path still seems to be
> > the
> > > right thing to do but if there isn't any demand for it then maybe
> that's
> > a
> > > reasonable compromise. I'm still surprised that more users aren't as
> > > impacted by this and as mentioned earlier, it could be an issue of
> > > awareness but I'm not sure that deprecating before a migration path is
> in
> > > place would be considered best-practice in raising awareness.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > >
> > >
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com
> > > > > wrote:
> > > >
> > > > > One thing that probably needs some clarification is what is implied
> > by
> > > > > "deprecated" in the Kafka project.
> > > > > I googled it a bit and it doesn't seem that deprecation
> > conventionally
> > > > > implies termination of support (or anything that could negatively
> > > impact
> > > > > existing users). That's my interpretation too.
> > > > > It would be good to know if Kafka follows a different
> interpretation
> > of
> > > > > the term.
> > > > >
> > > > > If my understanding of the term is correct, since we are not yet
> > > > targeting
> > > > > a certain major release in which the old consumer will be removed,
> I
> > > > don't
> > > > > see any harm in marking it as deprecated.
> > > > > There will be enough time to plan and implement the migration, if
> the
> > > > > community decides that's the way to go, before phasing it out.
> > > > >
> > > > > At the minimum new Kafka users will pick the Java consumer without
> > any
> > > > > confusion. And existing users will know that Kafka is preparing for
> > the
> > > > > old consumer's retirement.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Joel Koshy <jj...@gmail.com>
> > > > > To: "dev@kafka.apache.org" <de...@kafka.apache.org>
> > > > > Date: 01/05/2017 06:55 PM
> > > > > Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > > >
> > > > >
> > > > >
> > > > > While I realize this only marks the old consumer as deprecated and
> > not
> > > a
> > > > > complete removal, I agree that it is somewhat premature to do this
> > > prior
> > > > > to
> > > > > having a migration process implemented. Onur has described this in
> > > detail
> > > > > in the earlier thread: http://markmail.org/message/ek
> v352zy7xttco5s
> > > and
> > > > > I'm
> > > > > surprised that more companies aren't affected by (or aware of?) the
> > > > issue.
> > > > >
> > > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <ra...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I cant speak for anyone else, but a rolling upgrade is definitely
> > how
> > > > we
> > > > > > (LinkedIn) will do the migration.
> > > > > >
> > > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > > it sounds good to have
> > > > > > > it, but that's probably not how people will end up migrati
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
It might also be helpful for anyone who has done a bit of thinking around
the migration story to dump their thoughts into the JIRA re: the plan.
There are also questions I would have around what the exact requirements
are. For example, if supporting auto commit is required, things get a lot
hairier. If commits are explicit and standard partition assignors are used,
then we may be able to get away with something relatively cheap
(implementation-wise), e.g. you deploy with both clients but can start
accepting data from the new consumer as soon as assignments match between
the old & new (with some coordination to deal with timing differences
between groups).
It's really hard to judge how aggressive we might want to be with
deprecating without any idea what the LOE is to provide the transition
tool, and difficult to plan for building the transition tool as a prereq
for deprecation if we don't have any rough LOE. If the LOE is small enough,
maybe deprecating now is fine as long as we still provide a big enough
window for transition after the tool is built.
I also think it's worth pointing out that the other important impact of
deprecation is that folks should stop writing *new* code with the
deprecated consumer. If we can encourage folks to start using the new
consumer now, it makes their transition experience better since they have
fewer apps to move. For this reason alone, I'd like to deprecate the old
consumer before having the full migration story (with the docs/release note
that we'll provide something with plenty of time to make the transition
before removing the client entirely).
-Ewen
On Tue, Jan 10, 2017 at 2:11 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Renu,
>
> 0.10 was released in May. 2016. 0.11 will be not be before May 2017 (it
> could be later if the next release turns out to be 0.10.3). So the most
> recent data indicates a minimum of 1 year between major releases, but no
> decision has been made on future major release yet.
>
> The impact would indeed be build warnings, but those can be suppressed
> easily in the build config or via @SupressWarnings annotations. Do you use
> the consumer API directly or do you have a company wrapper?
>
> Ismael
>
> On Tue, Jan 10, 2017 at 8:53 PM, Renu Tewari <te...@gmail.com> wrote:
>
> > Hi Ismael,
> > What are the expected timelines we are talking about between the major
> > releases? At LI we are expecting to have atleast 1 year between the old
> > consumer deprecation and removal so we have enough time to upgrade all
> > applications. The rollout to new consumer has hit many hurdles so hasn't
> > proceeded at the expected pace. What impact would an official deprecation
> > have on applications? Any warnings would be disruptive and would prefer
> > that happens when there is a migration plan in place so we have a bound
> on
> > how long it will take. There are too many unknowns at this time.
> >
> > Thoughts on timelines?
> >
> > regards
> > Renu
> >
> > On Mon, Jan 9, 2017 at 6:34 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Joel,
> > >
> > > Great to hear that LinkedIn is likely to implement KAFKA-4513. :)
> > >
> > > Yes, the KIP as it stands is a compromise given the situation. We could
> > > push the deprecation to the subsequent release: likely to be 0.11.0.0
> > since
> > > there are a number of KIPs that require message format changes. This
> > would
> > > mean that the old consumers would not be removed before 0.12.0.0 (the
> > major
> > > release after 0.11.0.0). Would that work better for you all?
> > >
> > > Ismael
> > >
> > > On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy <jj...@gmail.com>
> > wrote:
> > >
> > > > >
> > > > >
> > > > > The ideal scenario would be for us to provide a tool for no
> downtime
> > > > > migration as discussed in the original thread (I filed
> > > > > https://issues.apache.org/jira/browse/KAFKA-4513 in response to
> that
> > > > > discussion). There are a few issues, however:
> > > > >
> > > > > - There doesn't seem to be much demand for it (outside of
> > LinkedIn,
> > > at
> > > > > least)
> > > > > - No-one is working on it or has indicated that they are
> planning
> > to
> > > > > work on it
> > > > > - It's a non-trivial change and it requires a good amount of
> > testing
> > > > to
> > > > > ensure it works as expected
> > > > >
> > > >
> > > > For LinkedIn: while there are a few consuming applications for which
> > the
> > > > current shut-down and restart approach to migration will suffice, I
> > doubt
> > > > we will be able to do this for majority of services that are outside
> > our
> > > > direct control. Given that a seamless migration is a pre-req for us
> to
> > > > switch to the new consumer widely (there are a few use-cases already
> on
> > > it)
> > > > it is something that we (LinkedIn) will likely implement although we
> > > > haven't done further brainstorming/improvements beyond what was
> > proposed
> > > in
> > > > the other deprecation thread.
> > > >
> > > >
> > > > > In the meantime, we have this suboptimal situation where the old
> > > > consumers
> > > > > are close to unmaintained even though we don't say it outright:
> they
> > > > don't
> > > >
> > > > get new features (basic things like security are missing) and bug
> fixes
> > > are
> > > > > rare. In practice, the old clients have been deprecated a while
> back,
> > > we
> > > > >
> > > >
> > > > Agreed that it is suboptimal, but the reality is that LI and I think
> a
> > > few
> > > > other companies are still to a large extent on the old consumer and
> > will
> > > be
> > > > for at least a good part of this year. This does mean that we have
> the
> > > > overhead of maintaining some internal workarounds for the old
> consumer.
> > > >
> > > >
> > > > > just haven't made it official. This proposal is about rectifying
> that
> > > so
> > > > > that we communicate our intentions to our users more clearly. As
> > Vahid
> > > > > said, this KIP is not about changing how we maintain the existing
> > code.
> > > > >
> > > > > The KIP that proposes the removal of all the old clients will be
> more
> > > > > interesting, but it doesn't exist yet. :)
> > > > >
> > > >
> > > > Deprecating *after* providing a sound migration path still seems to
> be
> > > the
> > > > right thing to do but if there isn't any demand for it then maybe
> > that's
> > > a
> > > > reasonable compromise. I'm still surprised that more users aren't as
> > > > impacted by this and as mentioned earlier, it could be an issue of
> > > > awareness but I'm not sure that deprecating before a migration path
> is
> > in
> > > > place would be considered best-practice in raising awareness.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > > >
> > > >
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > > > vahidhashemian@us.ibm.com
> > > > > > wrote:
> > > > >
> > > > > > One thing that probably needs some clarification is what is
> implied
> > > by
> > > > > > "deprecated" in the Kafka project.
> > > > > > I googled it a bit and it doesn't seem that deprecation
> > > conventionally
> > > > > > implies termination of support (or anything that could negatively
> > > > impact
> > > > > > existing users). That's my interpretation too.
> > > > > > It would be good to know if Kafka follows a different
> > interpretation
> > > of
> > > > > > the term.
> > > > > >
> > > > > > If my understanding of the term is correct, since we are not yet
> > > > > targeting
> > > > > > a certain major release in which the old consumer will be
> removed,
> > I
> > > > > don't
> > > > > > see any harm in marking it as deprecated.
> > > > > > There will be enough time to plan and implement the migration, if
> > the
> > > > > > community decides that's the way to go, before phasing it out.
> > > > > >
> > > > > > At the minimum new Kafka users will pick the Java consumer
> without
> > > any
> > > > > > confusion. And existing users will know that Kafka is preparing
> for
> > > the
> > > > > > old consumer's retirement.
> > > > > >
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Joel Koshy <jj...@gmail.com>
> > > > > > To: "dev@kafka.apache.org" <de...@kafka.apache.org>
> > > > > > Date: 01/05/2017 06:55 PM
> > > > > > Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > > > >
> > > > > >
> > > > > >
> > > > > > While I realize this only marks the old consumer as deprecated
> and
> > > not
> > > > a
> > > > > > complete removal, I agree that it is somewhat premature to do
> this
> > > > prior
> > > > > > to
> > > > > > having a migration process implemented. Onur has described this
> in
> > > > detail
> > > > > > in the earlier thread: http://markmail.org/message/
> > ekv352zy7xttco5s
> > > > and
> > > > > > I'm
> > > > > > surprised that more companies aren't affected by (or aware of?)
> the
> > > > > issue.
> > > > > >
> > > > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <
> radai.rosenblatt@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > I cant speak for anyone else, but a rolling upgrade is
> definitely
> > > how
> > > > > we
> > > > > > > (LinkedIn) will do the migration.
> > > > > > >
> > > > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <
> gwen@confluent.io>
> > > > > wrote:
> > > > > > >
> > > > > > > > it sounds good to have
> > > > > > > > it, but that's probably not how people will end up migrati
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Renu,
0.10 was released in May. 2016. 0.11 will be not be before May 2017 (it
could be later if the next release turns out to be 0.10.3). So the most
recent data indicates a minimum of 1 year between major releases, but no
decision has been made on future major release yet.
The impact would indeed be build warnings, but those can be suppressed
easily in the build config or via @SupressWarnings annotations. Do you use
the consumer API directly or do you have a company wrapper?
Ismael
On Tue, Jan 10, 2017 at 8:53 PM, Renu Tewari <te...@gmail.com> wrote:
> Hi Ismael,
> What are the expected timelines we are talking about between the major
> releases? At LI we are expecting to have atleast 1 year between the old
> consumer deprecation and removal so we have enough time to upgrade all
> applications. The rollout to new consumer has hit many hurdles so hasn't
> proceeded at the expected pace. What impact would an official deprecation
> have on applications? Any warnings would be disruptive and would prefer
> that happens when there is a migration plan in place so we have a bound on
> how long it will take. There are too many unknowns at this time.
>
> Thoughts on timelines?
>
> regards
> Renu
>
> On Mon, Jan 9, 2017 at 6:34 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Joel,
> >
> > Great to hear that LinkedIn is likely to implement KAFKA-4513. :)
> >
> > Yes, the KIP as it stands is a compromise given the situation. We could
> > push the deprecation to the subsequent release: likely to be 0.11.0.0
> since
> > there are a number of KIPs that require message format changes. This
> would
> > mean that the old consumers would not be removed before 0.12.0.0 (the
> major
> > release after 0.11.0.0). Would that work better for you all?
> >
> > Ismael
> >
> > On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy <jj...@gmail.com>
> wrote:
> >
> > > >
> > > >
> > > > The ideal scenario would be for us to provide a tool for no downtime
> > > > migration as discussed in the original thread (I filed
> > > > https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
> > > > discussion). There are a few issues, however:
> > > >
> > > > - There doesn't seem to be much demand for it (outside of
> LinkedIn,
> > at
> > > > least)
> > > > - No-one is working on it or has indicated that they are planning
> to
> > > > work on it
> > > > - It's a non-trivial change and it requires a good amount of
> testing
> > > to
> > > > ensure it works as expected
> > > >
> > >
> > > For LinkedIn: while there are a few consuming applications for which
> the
> > > current shut-down and restart approach to migration will suffice, I
> doubt
> > > we will be able to do this for majority of services that are outside
> our
> > > direct control. Given that a seamless migration is a pre-req for us to
> > > switch to the new consumer widely (there are a few use-cases already on
> > it)
> > > it is something that we (LinkedIn) will likely implement although we
> > > haven't done further brainstorming/improvements beyond what was
> proposed
> > in
> > > the other deprecation thread.
> > >
> > >
> > > > In the meantime, we have this suboptimal situation where the old
> > > consumers
> > > > are close to unmaintained even though we don't say it outright: they
> > > don't
> > >
> > > get new features (basic things like security are missing) and bug fixes
> > are
> > > > rare. In practice, the old clients have been deprecated a while back,
> > we
> > > >
> > >
> > > Agreed that it is suboptimal, but the reality is that LI and I think a
> > few
> > > other companies are still to a large extent on the old consumer and
> will
> > be
> > > for at least a good part of this year. This does mean that we have the
> > > overhead of maintaining some internal workarounds for the old consumer.
> > >
> > >
> > > > just haven't made it official. This proposal is about rectifying that
> > so
> > > > that we communicate our intentions to our users more clearly. As
> Vahid
> > > > said, this KIP is not about changing how we maintain the existing
> code.
> > > >
> > > > The KIP that proposes the removal of all the old clients will be more
> > > > interesting, but it doesn't exist yet. :)
> > > >
> > >
> > > Deprecating *after* providing a sound migration path still seems to be
> > the
> > > right thing to do but if there isn't any demand for it then maybe
> that's
> > a
> > > reasonable compromise. I'm still surprised that more users aren't as
> > > impacted by this and as mentioned earlier, it could be an issue of
> > > awareness but I'm not sure that deprecating before a migration path is
> in
> > > place would be considered best-practice in raising awareness.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > >
> > >
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com
> > > > > wrote:
> > > >
> > > > > One thing that probably needs some clarification is what is implied
> > by
> > > > > "deprecated" in the Kafka project.
> > > > > I googled it a bit and it doesn't seem that deprecation
> > conventionally
> > > > > implies termination of support (or anything that could negatively
> > > impact
> > > > > existing users). That's my interpretation too.
> > > > > It would be good to know if Kafka follows a different
> interpretation
> > of
> > > > > the term.
> > > > >
> > > > > If my understanding of the term is correct, since we are not yet
> > > > targeting
> > > > > a certain major release in which the old consumer will be removed,
> I
> > > > don't
> > > > > see any harm in marking it as deprecated.
> > > > > There will be enough time to plan and implement the migration, if
> the
> > > > > community decides that's the way to go, before phasing it out.
> > > > >
> > > > > At the minimum new Kafka users will pick the Java consumer without
> > any
> > > > > confusion. And existing users will know that Kafka is preparing for
> > the
> > > > > old consumer's retirement.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Joel Koshy <jj...@gmail.com>
> > > > > To: "dev@kafka.apache.org" <de...@kafka.apache.org>
> > > > > Date: 01/05/2017 06:55 PM
> > > > > Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > > >
> > > > >
> > > > >
> > > > > While I realize this only marks the old consumer as deprecated and
> > not
> > > a
> > > > > complete removal, I agree that it is somewhat premature to do this
> > > prior
> > > > > to
> > > > > having a migration process implemented. Onur has described this in
> > > detail
> > > > > in the earlier thread: http://markmail.org/message/
> ekv352zy7xttco5s
> > > and
> > > > > I'm
> > > > > surprised that more companies aren't affected by (or aware of?) the
> > > > issue.
> > > > >
> > > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <ra...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I cant speak for anyone else, but a rolling upgrade is definitely
> > how
> > > > we
> > > > > > (LinkedIn) will do the migration.
> > > > > >
> > > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > > it sounds good to have
> > > > > > > it, but that's probably not how people will end up migrati
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Renu Tewari <te...@gmail.com>.
Hi Ismael,
What are the expected timelines we are talking about between the major
releases? At LI we are expecting to have atleast 1 year between the old
consumer deprecation and removal so we have enough time to upgrade all
applications. The rollout to new consumer has hit many hurdles so hasn't
proceeded at the expected pace. What impact would an official deprecation
have on applications? Any warnings would be disruptive and would prefer
that happens when there is a migration plan in place so we have a bound on
how long it will take. There are too many unknowns at this time.
Thoughts on timelines?
regards
Renu
On Mon, Jan 9, 2017 at 6:34 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Joel,
>
> Great to hear that LinkedIn is likely to implement KAFKA-4513. :)
>
> Yes, the KIP as it stands is a compromise given the situation. We could
> push the deprecation to the subsequent release: likely to be 0.11.0.0 since
> there are a number of KIPs that require message format changes. This would
> mean that the old consumers would not be removed before 0.12.0.0 (the major
> release after 0.11.0.0). Would that work better for you all?
>
> Ismael
>
> On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy <jj...@gmail.com> wrote:
>
> > >
> > >
> > > The ideal scenario would be for us to provide a tool for no downtime
> > > migration as discussed in the original thread (I filed
> > > https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
> > > discussion). There are a few issues, however:
> > >
> > > - There doesn't seem to be much demand for it (outside of LinkedIn,
> at
> > > least)
> > > - No-one is working on it or has indicated that they are planning to
> > > work on it
> > > - It's a non-trivial change and it requires a good amount of testing
> > to
> > > ensure it works as expected
> > >
> >
> > For LinkedIn: while there are a few consuming applications for which the
> > current shut-down and restart approach to migration will suffice, I doubt
> > we will be able to do this for majority of services that are outside our
> > direct control. Given that a seamless migration is a pre-req for us to
> > switch to the new consumer widely (there are a few use-cases already on
> it)
> > it is something that we (LinkedIn) will likely implement although we
> > haven't done further brainstorming/improvements beyond what was proposed
> in
> > the other deprecation thread.
> >
> >
> > > In the meantime, we have this suboptimal situation where the old
> > consumers
> > > are close to unmaintained even though we don't say it outright: they
> > don't
> >
> > get new features (basic things like security are missing) and bug fixes
> are
> > > rare. In practice, the old clients have been deprecated a while back,
> we
> > >
> >
> > Agreed that it is suboptimal, but the reality is that LI and I think a
> few
> > other companies are still to a large extent on the old consumer and will
> be
> > for at least a good part of this year. This does mean that we have the
> > overhead of maintaining some internal workarounds for the old consumer.
> >
> >
> > > just haven't made it official. This proposal is about rectifying that
> so
> > > that we communicate our intentions to our users more clearly. As Vahid
> > > said, this KIP is not about changing how we maintain the existing code.
> > >
> > > The KIP that proposes the removal of all the old clients will be more
> > > interesting, but it doesn't exist yet. :)
> > >
> >
> > Deprecating *after* providing a sound migration path still seems to be
> the
> > right thing to do but if there isn't any demand for it then maybe that's
> a
> > reasonable compromise. I'm still surprised that more users aren't as
> > impacted by this and as mentioned earlier, it could be an issue of
> > awareness but I'm not sure that deprecating before a migration path is in
> > place would be considered best-practice in raising awareness.
> >
> > Thanks,
> >
> > Joel
> >
> >
> >
> > >
> > > Ismael
> > >
> > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com
> > > > wrote:
> > >
> > > > One thing that probably needs some clarification is what is implied
> by
> > > > "deprecated" in the Kafka project.
> > > > I googled it a bit and it doesn't seem that deprecation
> conventionally
> > > > implies termination of support (or anything that could negatively
> > impact
> > > > existing users). That's my interpretation too.
> > > > It would be good to know if Kafka follows a different interpretation
> of
> > > > the term.
> > > >
> > > > If my understanding of the term is correct, since we are not yet
> > > targeting
> > > > a certain major release in which the old consumer will be removed, I
> > > don't
> > > > see any harm in marking it as deprecated.
> > > > There will be enough time to plan and implement the migration, if the
> > > > community decides that's the way to go, before phasing it out.
> > > >
> > > > At the minimum new Kafka users will pick the Java consumer without
> any
> > > > confusion. And existing users will know that Kafka is preparing for
> the
> > > > old consumer's retirement.
> > > >
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Joel Koshy <jj...@gmail.com>
> > > > To: "dev@kafka.apache.org" <de...@kafka.apache.org>
> > > > Date: 01/05/2017 06:55 PM
> > > > Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > >
> > > >
> > > >
> > > > While I realize this only marks the old consumer as deprecated and
> not
> > a
> > > > complete removal, I agree that it is somewhat premature to do this
> > prior
> > > > to
> > > > having a migration process implemented. Onur has described this in
> > detail
> > > > in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s
> > and
> > > > I'm
> > > > surprised that more companies aren't affected by (or aware of?) the
> > > issue.
> > > >
> > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <ra...@gmail.com>
> > > wrote:
> > > >
> > > > > I cant speak for anyone else, but a rolling upgrade is definitely
> how
> > > we
> > > > > (LinkedIn) will do the migration.
> > > > >
> > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > > > >
> > > > > > it sounds good to have
> > > > > > it, but that's probably not how people will end up migrati
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Ewen,
I agree that it would be good to clarify what we consider a message format
change.
The compaction tombstone was a message format change because it would mean
that `null` no longer meant delete for compacted topics, so there is a
semantics change as well (and the cleaner would have to be aware of this).
For something like a new compression algorithm, it could potentially be
done without a message format change because you have to opt-in (i..e don't
create topics with the new compression algorithm until your brokers and
clients support it). As you said though, bumping both message format
version and fetch/produce request/response versions does give a little more
flexibility and safety and it's nice to be able to specify that for message
format version X, a certain compression algorithm must be supported. Still,
down-conversion is costly enough that it may be worth avoiding unless we
can bundle the compression format addition with other message format
changes (it may be worth bundling in this case).
Ismael
On Tue, Jan 10, 2017 at 6:04 AM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> I don't mean to derail this at all, but given Ismael's question I'm
> wondering what exactly we consider a message format change? Are we assuming
> that additions like a new compression format (i.e. semantic changes where
> previously undefined bits, but bits which existed) don't require a format
> version change? I ask because we went the other way with protocol requests,
> but there are a number of outstanding KIPs (zstd compression and compaction
> tombstones off the top of my head) which people presumably were targeting
> for earlier than 0.11.0.0 but are, strictly speaking, "new" formats.
>
> I think the different definitions might make sense, but it is probably
> worth clarifying. And this would have important implications. We have a
> bunch of proposed structural changes upcoming in the message format, but
> obviously we'd like to keep the number of times folks have to think about
> this to a minimum. On the other hand, this means that semantic changes like
> supporting a specific compression format might force folks to stay on an
> earlier broker version if they want to avoid, e.g., producers accidentally
> sending data in a format consumers won't understand (i.e. in this case they
> may want all clients upgraded before all brokers, which is where separating
> log format and request/response versions comes in handy). The deletion
> tombstones might be an even better example here since the details are
> probably handled automatically by the library and it only takes one
> upgraded client to do the wrong thing and break lots of downstream systems.
>
> In other words, we potentially tie quite a few improvements that we might
> otherwise think of as incremental improvements only to major version
> releases. This could in turn force/encourage us to do major releases more
> frequently and we lose the benefit of infrequent message format changes, or
> we still do them at the same cadence and it takes forever for new
> incremental features to land.
>
> -Ewen
>
> On Mon, Jan 9, 2017 at 6:34 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Joel,
> >
> > Great to hear that LinkedIn is likely to implement KAFKA-4513. :)
> >
> > Yes, the KIP as it stands is a compromise given the situation. We could
> > push the deprecation to the subsequent release: likely to be 0.11.0.0
> since
> > there are a number of KIPs that require message format changes. This
> would
> > mean that the old consumers would not be removed before 0.12.0.0 (the
> major
> > release after 0.11.0.0). Would that work better for you all?
> >
> > Ismael
> >
> > On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy <jj...@gmail.com>
> wrote:
> >
> > > >
> > > >
> > > > The ideal scenario would be for us to provide a tool for no downtime
> > > > migration as discussed in the original thread (I filed
> > > > https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
> > > > discussion). There are a few issues, however:
> > > >
> > > > - There doesn't seem to be much demand for it (outside of
> LinkedIn,
> > at
> > > > least)
> > > > - No-one is working on it or has indicated that they are planning
> to
> > > > work on it
> > > > - It's a non-trivial change and it requires a good amount of
> testing
> > > to
> > > > ensure it works as expected
> > > >
> > >
> > > For LinkedIn: while there are a few consuming applications for which
> the
> > > current shut-down and restart approach to migration will suffice, I
> doubt
> > > we will be able to do this for majority of services that are outside
> our
> > > direct control. Given that a seamless migration is a pre-req for us to
> > > switch to the new consumer widely (there are a few use-cases already on
> > it)
> > > it is something that we (LinkedIn) will likely implement although we
> > > haven't done further brainstorming/improvements beyond what was
> proposed
> > in
> > > the other deprecation thread.
> > >
> > >
> > > > In the meantime, we have this suboptimal situation where the old
> > > consumers
> > > > are close to unmaintained even though we don't say it outright: they
> > > don't
> > >
> > > get new features (basic things like security are missing) and bug fixes
> > are
> > > > rare. In practice, the old clients have been deprecated a while back,
> > we
> > > >
> > >
> > > Agreed that it is suboptimal, but the reality is that LI and I think a
> > few
> > > other companies are still to a large extent on the old consumer and
> will
> > be
> > > for at least a good part of this year. This does mean that we have the
> > > overhead of maintaining some internal workarounds for the old consumer.
> > >
> > >
> > > > just haven't made it official. This proposal is about rectifying that
> > so
> > > > that we communicate our intentions to our users more clearly. As
> Vahid
> > > > said, this KIP is not about changing how we maintain the existing
> code.
> > > >
> > > > The KIP that proposes the removal of all the old clients will be more
> > > > interesting, but it doesn't exist yet. :)
> > > >
> > >
> > > Deprecating *after* providing a sound migration path still seems to be
> > the
> > > right thing to do but if there isn't any demand for it then maybe
> that's
> > a
> > > reasonable compromise. I'm still surprised that more users aren't as
> > > impacted by this and as mentioned earlier, it could be an issue of
> > > awareness but I'm not sure that deprecating before a migration path is
> in
> > > place would be considered best-practice in raising awareness.
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > >
> > >
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > > vahidhashemian@us.ibm.com
> > > > > wrote:
> > > >
> > > > > One thing that probably needs some clarification is what is implied
> > by
> > > > > "deprecated" in the Kafka project.
> > > > > I googled it a bit and it doesn't seem that deprecation
> > conventionally
> > > > > implies termination of support (or anything that could negatively
> > > impact
> > > > > existing users). That's my interpretation too.
> > > > > It would be good to know if Kafka follows a different
> interpretation
> > of
> > > > > the term.
> > > > >
> > > > > If my understanding of the term is correct, since we are not yet
> > > > targeting
> > > > > a certain major release in which the old consumer will be removed,
> I
> > > > don't
> > > > > see any harm in marking it as deprecated.
> > > > > There will be enough time to plan and implement the migration, if
> the
> > > > > community decides that's the way to go, before phasing it out.
> > > > >
> > > > > At the minimum new Kafka users will pick the Java consumer without
> > any
> > > > > confusion. And existing users will know that Kafka is preparing for
> > the
> > > > > old consumer's retirement.
> > > > >
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: Joel Koshy <jj...@gmail.com>
> > > > > To: "dev@kafka.apache.org" <de...@kafka.apache.org>
> > > > > Date: 01/05/2017 06:55 PM
> > > > > Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > > >
> > > > >
> > > > >
> > > > > While I realize this only marks the old consumer as deprecated and
> > not
> > > a
> > > > > complete removal, I agree that it is somewhat premature to do this
> > > prior
> > > > > to
> > > > > having a migration process implemented. Onur has described this in
> > > detail
> > > > > in the earlier thread: http://markmail.org/message/ek
> v352zy7xttco5s
> > > and
> > > > > I'm
> > > > > surprised that more companies aren't affected by (or aware of?) the
> > > > issue.
> > > > >
> > > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <ra...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I cant speak for anyone else, but a rolling upgrade is definitely
> > how
> > > > we
> > > > > > (LinkedIn) will do the migration.
> > > > > >
> > > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > > it sounds good to have
> > > > > > > it, but that's probably not how people will end up migrati
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
I don't mean to derail this at all, but given Ismael's question I'm
wondering what exactly we consider a message format change? Are we assuming
that additions like a new compression format (i.e. semantic changes where
previously undefined bits, but bits which existed) don't require a format
version change? I ask because we went the other way with protocol requests,
but there are a number of outstanding KIPs (zstd compression and compaction
tombstones off the top of my head) which people presumably were targeting
for earlier than 0.11.0.0 but are, strictly speaking, "new" formats.
I think the different definitions might make sense, but it is probably
worth clarifying. And this would have important implications. We have a
bunch of proposed structural changes upcoming in the message format, but
obviously we'd like to keep the number of times folks have to think about
this to a minimum. On the other hand, this means that semantic changes like
supporting a specific compression format might force folks to stay on an
earlier broker version if they want to avoid, e.g., producers accidentally
sending data in a format consumers won't understand (i.e. in this case they
may want all clients upgraded before all brokers, which is where separating
log format and request/response versions comes in handy). The deletion
tombstones might be an even better example here since the details are
probably handled automatically by the library and it only takes one
upgraded client to do the wrong thing and break lots of downstream systems.
In other words, we potentially tie quite a few improvements that we might
otherwise think of as incremental improvements only to major version
releases. This could in turn force/encourage us to do major releases more
frequently and we lose the benefit of infrequent message format changes, or
we still do them at the same cadence and it takes forever for new
incremental features to land.
-Ewen
On Mon, Jan 9, 2017 at 6:34 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Joel,
>
> Great to hear that LinkedIn is likely to implement KAFKA-4513. :)
>
> Yes, the KIP as it stands is a compromise given the situation. We could
> push the deprecation to the subsequent release: likely to be 0.11.0.0 since
> there are a number of KIPs that require message format changes. This would
> mean that the old consumers would not be removed before 0.12.0.0 (the major
> release after 0.11.0.0). Would that work better for you all?
>
> Ismael
>
> On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy <jj...@gmail.com> wrote:
>
> > >
> > >
> > > The ideal scenario would be for us to provide a tool for no downtime
> > > migration as discussed in the original thread (I filed
> > > https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
> > > discussion). There are a few issues, however:
> > >
> > > - There doesn't seem to be much demand for it (outside of LinkedIn,
> at
> > > least)
> > > - No-one is working on it or has indicated that they are planning to
> > > work on it
> > > - It's a non-trivial change and it requires a good amount of testing
> > to
> > > ensure it works as expected
> > >
> >
> > For LinkedIn: while there are a few consuming applications for which the
> > current shut-down and restart approach to migration will suffice, I doubt
> > we will be able to do this for majority of services that are outside our
> > direct control. Given that a seamless migration is a pre-req for us to
> > switch to the new consumer widely (there are a few use-cases already on
> it)
> > it is something that we (LinkedIn) will likely implement although we
> > haven't done further brainstorming/improvements beyond what was proposed
> in
> > the other deprecation thread.
> >
> >
> > > In the meantime, we have this suboptimal situation where the old
> > consumers
> > > are close to unmaintained even though we don't say it outright: they
> > don't
> >
> > get new features (basic things like security are missing) and bug fixes
> are
> > > rare. In practice, the old clients have been deprecated a while back,
> we
> > >
> >
> > Agreed that it is suboptimal, but the reality is that LI and I think a
> few
> > other companies are still to a large extent on the old consumer and will
> be
> > for at least a good part of this year. This does mean that we have the
> > overhead of maintaining some internal workarounds for the old consumer.
> >
> >
> > > just haven't made it official. This proposal is about rectifying that
> so
> > > that we communicate our intentions to our users more clearly. As Vahid
> > > said, this KIP is not about changing how we maintain the existing code.
> > >
> > > The KIP that proposes the removal of all the old clients will be more
> > > interesting, but it doesn't exist yet. :)
> > >
> >
> > Deprecating *after* providing a sound migration path still seems to be
> the
> > right thing to do but if there isn't any demand for it then maybe that's
> a
> > reasonable compromise. I'm still surprised that more users aren't as
> > impacted by this and as mentioned earlier, it could be an issue of
> > awareness but I'm not sure that deprecating before a migration path is in
> > place would be considered best-practice in raising awareness.
> >
> > Thanks,
> >
> > Joel
> >
> >
> >
> > >
> > > Ismael
> > >
> > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > vahidhashemian@us.ibm.com
> > > > wrote:
> > >
> > > > One thing that probably needs some clarification is what is implied
> by
> > > > "deprecated" in the Kafka project.
> > > > I googled it a bit and it doesn't seem that deprecation
> conventionally
> > > > implies termination of support (or anything that could negatively
> > impact
> > > > existing users). That's my interpretation too.
> > > > It would be good to know if Kafka follows a different interpretation
> of
> > > > the term.
> > > >
> > > > If my understanding of the term is correct, since we are not yet
> > > targeting
> > > > a certain major release in which the old consumer will be removed, I
> > > don't
> > > > see any harm in marking it as deprecated.
> > > > There will be enough time to plan and implement the migration, if the
> > > > community decides that's the way to go, before phasing it out.
> > > >
> > > > At the minimum new Kafka users will pick the Java consumer without
> any
> > > > confusion. And existing users will know that Kafka is preparing for
> the
> > > > old consumer's retirement.
> > > >
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From: Joel Koshy <jj...@gmail.com>
> > > > To: "dev@kafka.apache.org" <de...@kafka.apache.org>
> > > > Date: 01/05/2017 06:55 PM
> > > > Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > > >
> > > >
> > > >
> > > > While I realize this only marks the old consumer as deprecated and
> not
> > a
> > > > complete removal, I agree that it is somewhat premature to do this
> > prior
> > > > to
> > > > having a migration process implemented. Onur has described this in
> > detail
> > > > in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s
> > and
> > > > I'm
> > > > surprised that more companies aren't affected by (or aware of?) the
> > > issue.
> > > >
> > > > On Thu, Jan 5, 2017 at 4:40 PM, radai <ra...@gmail.com>
> > > wrote:
> > > >
> > > > > I cant speak for anyone else, but a rolling upgrade is definitely
> how
> > > we
> > > > > (LinkedIn) will do the migration.
> > > > >
> > > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > > > >
> > > > > > it sounds good to have
> > > > > > it, but that's probably not how people will end up migrati
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Joel,
Great to hear that LinkedIn is likely to implement KAFKA-4513. :)
Yes, the KIP as it stands is a compromise given the situation. We could
push the deprecation to the subsequent release: likely to be 0.11.0.0 since
there are a number of KIPs that require message format changes. This would
mean that the old consumers would not be removed before 0.12.0.0 (the major
release after 0.11.0.0). Would that work better for you all?
Ismael
On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy <jj...@gmail.com> wrote:
> >
> >
> > The ideal scenario would be for us to provide a tool for no downtime
> > migration as discussed in the original thread (I filed
> > https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
> > discussion). There are a few issues, however:
> >
> > - There doesn't seem to be much demand for it (outside of LinkedIn, at
> > least)
> > - No-one is working on it or has indicated that they are planning to
> > work on it
> > - It's a non-trivial change and it requires a good amount of testing
> to
> > ensure it works as expected
> >
>
> For LinkedIn: while there are a few consuming applications for which the
> current shut-down and restart approach to migration will suffice, I doubt
> we will be able to do this for majority of services that are outside our
> direct control. Given that a seamless migration is a pre-req for us to
> switch to the new consumer widely (there are a few use-cases already on it)
> it is something that we (LinkedIn) will likely implement although we
> haven't done further brainstorming/improvements beyond what was proposed in
> the other deprecation thread.
>
>
> > In the meantime, we have this suboptimal situation where the old
> consumers
> > are close to unmaintained even though we don't say it outright: they
> don't
>
> get new features (basic things like security are missing) and bug fixes are
> > rare. In practice, the old clients have been deprecated a while back, we
> >
>
> Agreed that it is suboptimal, but the reality is that LI and I think a few
> other companies are still to a large extent on the old consumer and will be
> for at least a good part of this year. This does mean that we have the
> overhead of maintaining some internal workarounds for the old consumer.
>
>
> > just haven't made it official. This proposal is about rectifying that so
> > that we communicate our intentions to our users more clearly. As Vahid
> > said, this KIP is not about changing how we maintain the existing code.
> >
> > The KIP that proposes the removal of all the old clients will be more
> > interesting, but it doesn't exist yet. :)
> >
>
> Deprecating *after* providing a sound migration path still seems to be the
> right thing to do but if there isn't any demand for it then maybe that's a
> reasonable compromise. I'm still surprised that more users aren't as
> impacted by this and as mentioned earlier, it could be an issue of
> awareness but I'm not sure that deprecating before a migration path is in
> place would be considered best-practice in raising awareness.
>
> Thanks,
>
> Joel
>
>
>
> >
> > Ismael
> >
> > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com
> > > wrote:
> >
> > > One thing that probably needs some clarification is what is implied by
> > > "deprecated" in the Kafka project.
> > > I googled it a bit and it doesn't seem that deprecation conventionally
> > > implies termination of support (or anything that could negatively
> impact
> > > existing users). That's my interpretation too.
> > > It would be good to know if Kafka follows a different interpretation of
> > > the term.
> > >
> > > If my understanding of the term is correct, since we are not yet
> > targeting
> > > a certain major release in which the old consumer will be removed, I
> > don't
> > > see any harm in marking it as deprecated.
> > > There will be enough time to plan and implement the migration, if the
> > > community decides that's the way to go, before phasing it out.
> > >
> > > At the minimum new Kafka users will pick the Java consumer without any
> > > confusion. And existing users will know that Kafka is preparing for the
> > > old consumer's retirement.
> > >
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Joel Koshy <jj...@gmail.com>
> > > To: "dev@kafka.apache.org" <de...@kafka.apache.org>
> > > Date: 01/05/2017 06:55 PM
> > > Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> > >
> > >
> > >
> > > While I realize this only marks the old consumer as deprecated and not
> a
> > > complete removal, I agree that it is somewhat premature to do this
> prior
> > > to
> > > having a migration process implemented. Onur has described this in
> detail
> > > in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s
> and
> > > I'm
> > > surprised that more companies aren't affected by (or aware of?) the
> > issue.
> > >
> > > On Thu, Jan 5, 2017 at 4:40 PM, radai <ra...@gmail.com>
> > wrote:
> > >
> > > > I cant speak for anyone else, but a rolling upgrade is definitely how
> > we
> > > > (LinkedIn) will do the migration.
> > > >
> > > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io>
> > wrote:
> > > >
> > > > > it sounds good to have
> > > > > it, but that's probably not how people will end up migrati
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Joel Koshy <jj...@gmail.com>.
>
>
> The ideal scenario would be for us to provide a tool for no downtime
> migration as discussed in the original thread (I filed
> https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
> discussion). There are a few issues, however:
>
> - There doesn't seem to be much demand for it (outside of LinkedIn, at
> least)
> - No-one is working on it or has indicated that they are planning to
> work on it
> - It's a non-trivial change and it requires a good amount of testing to
> ensure it works as expected
>
For LinkedIn: while there are a few consuming applications for which the
current shut-down and restart approach to migration will suffice, I doubt
we will be able to do this for majority of services that are outside our
direct control. Given that a seamless migration is a pre-req for us to
switch to the new consumer widely (there are a few use-cases already on it)
it is something that we (LinkedIn) will likely implement although we
haven't done further brainstorming/improvements beyond what was proposed in
the other deprecation thread.
> In the meantime, we have this suboptimal situation where the old consumers
> are close to unmaintained even though we don't say it outright: they don't
get new features (basic things like security are missing) and bug fixes are
> rare. In practice, the old clients have been deprecated a while back, we
>
Agreed that it is suboptimal, but the reality is that LI and I think a few
other companies are still to a large extent on the old consumer and will be
for at least a good part of this year. This does mean that we have the
overhead of maintaining some internal workarounds for the old consumer.
> just haven't made it official. This proposal is about rectifying that so
> that we communicate our intentions to our users more clearly. As Vahid
> said, this KIP is not about changing how we maintain the existing code.
>
> The KIP that proposes the removal of all the old clients will be more
> interesting, but it doesn't exist yet. :)
>
Deprecating *after* providing a sound migration path still seems to be the
right thing to do but if there isn't any demand for it then maybe that's a
reasonable compromise. I'm still surprised that more users aren't as
impacted by this and as mentioned earlier, it could be an issue of
awareness but I'm not sure that deprecating before a migration path is in
place would be considered best-practice in raising awareness.
Thanks,
Joel
>
> Ismael
>
> On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com
> > wrote:
>
> > One thing that probably needs some clarification is what is implied by
> > "deprecated" in the Kafka project.
> > I googled it a bit and it doesn't seem that deprecation conventionally
> > implies termination of support (or anything that could negatively impact
> > existing users). That's my interpretation too.
> > It would be good to know if Kafka follows a different interpretation of
> > the term.
> >
> > If my understanding of the term is correct, since we are not yet
> targeting
> > a certain major release in which the old consumer will be removed, I
> don't
> > see any harm in marking it as deprecated.
> > There will be enough time to plan and implement the migration, if the
> > community decides that's the way to go, before phasing it out.
> >
> > At the minimum new Kafka users will pick the Java consumer without any
> > confusion. And existing users will know that Kafka is preparing for the
> > old consumer's retirement.
> >
> > --Vahid
> >
> >
> >
> >
> > From: Joel Koshy <jj...@gmail.com>
> > To: "dev@kafka.apache.org" <de...@kafka.apache.org>
> > Date: 01/05/2017 06:55 PM
> > Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
> >
> >
> >
> > While I realize this only marks the old consumer as deprecated and not a
> > complete removal, I agree that it is somewhat premature to do this prior
> > to
> > having a migration process implemented. Onur has described this in detail
> > in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s and
> > I'm
> > surprised that more companies aren't affected by (or aware of?) the
> issue.
> >
> > On Thu, Jan 5, 2017 at 4:40 PM, radai <ra...@gmail.com>
> wrote:
> >
> > > I cant speak for anyone else, but a rolling upgrade is definitely how
> we
> > > (LinkedIn) will do the migration.
> > >
> > > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io>
> wrote:
> > >
> > > > it sounds good to have
> > > > it, but that's probably not how people will end up migrati
> > > >
> > >
> >
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Ismael Juma <is...@juma.me.uk>.
Hi all,
Thanks for the feedback. A few thoughts follow.
The ideal scenario would be for us to provide a tool for no downtime
migration as discussed in the original thread (I filed
https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
discussion). There are a few issues, however:
- There doesn't seem to be much demand for it (outside of LinkedIn, at
least)
- No-one is working on it or has indicated that they are planning to
work on it
- It's a non-trivial change and it requires a good amount of testing to
ensure it works as expected
I suggested the KIP to raise awareness. Maybe there's more demand than we
think and/or someone is planning to work on it. The latter, in particular,
would be great news.
In the meantime, we have this suboptimal situation where the old consumers
are close to unmaintained even though we don't say it outright: they don't
get new features (basic things like security are missing) and bug fixes are
rare. In practice, the old clients have been deprecated a while back, we
just haven't made it official. This proposal is about rectifying that so
that we communicate our intentions to our users more clearly. As Vahid
said, this KIP is not about changing how we maintain the existing code.
The KIP that proposes the removal of all the old clients will be more
interesting, but it doesn't exist yet. :)
Ismael
On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <vahidhashemian@us.ibm.com
> wrote:
> One thing that probably needs some clarification is what is implied by
> "deprecated" in the Kafka project.
> I googled it a bit and it doesn't seem that deprecation conventionally
> implies termination of support (or anything that could negatively impact
> existing users). That's my interpretation too.
> It would be good to know if Kafka follows a different interpretation of
> the term.
>
> If my understanding of the term is correct, since we are not yet targeting
> a certain major release in which the old consumer will be removed, I don't
> see any harm in marking it as deprecated.
> There will be enough time to plan and implement the migration, if the
> community decides that's the way to go, before phasing it out.
>
> At the minimum new Kafka users will pick the Java consumer without any
> confusion. And existing users will know that Kafka is preparing for the
> old consumer's retirement.
>
> --Vahid
>
>
>
>
> From: Joel Koshy <jj...@gmail.com>
> To: "dev@kafka.apache.org" <de...@kafka.apache.org>
> Date: 01/05/2017 06:55 PM
> Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
>
>
>
> While I realize this only marks the old consumer as deprecated and not a
> complete removal, I agree that it is somewhat premature to do this prior
> to
> having a migration process implemented. Onur has described this in detail
> in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s and
> I'm
> surprised that more companies aren't affected by (or aware of?) the issue.
>
> On Thu, Jan 5, 2017 at 4:40 PM, radai <ra...@gmail.com> wrote:
>
> > I cant speak for anyone else, but a rolling upgrade is definitely how we
> > (LinkedIn) will do the migration.
> >
> > On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io> wrote:
> >
> > > it sounds good to have
> > > it, but that's probably not how people will end up migrati
> > >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Vahid S Hashemian <va...@us.ibm.com>.
One thing that probably needs some clarification is what is implied by
"deprecated" in the Kafka project.
I googled it a bit and it doesn't seem that deprecation conventionally
implies termination of support (or anything that could negatively impact
existing users). That's my interpretation too.
It would be good to know if Kafka follows a different interpretation of
the term.
If my understanding of the term is correct, since we are not yet targeting
a certain major release in which the old consumer will be removed, I don't
see any harm in marking it as deprecated.
There will be enough time to plan and implement the migration, if the
community decides that's the way to go, before phasing it out.
At the minimum new Kafka users will pick the Java consumer without any
confusion. And existing users will know that Kafka is preparing for the
old consumer's retirement.
--Vahid
From: Joel Koshy <jj...@gmail.com>
To: "dev@kafka.apache.org" <de...@kafka.apache.org>
Date: 01/05/2017 06:55 PM
Subject: Re: [DISCUSS] KIP-109: Old Consumer Deprecation
While I realize this only marks the old consumer as deprecated and not a
complete removal, I agree that it is somewhat premature to do this prior
to
having a migration process implemented. Onur has described this in detail
in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s and
I'm
surprised that more companies aren't affected by (or aware of?) the issue.
On Thu, Jan 5, 2017 at 4:40 PM, radai <ra...@gmail.com> wrote:
> I cant speak for anyone else, but a rolling upgrade is definitely how we
> (LinkedIn) will do the migration.
>
> On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > it sounds good to have
> > it, but that's probably not how people will end up migrati
> >
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Joel Koshy <jj...@gmail.com>.
While I realize this only marks the old consumer as deprecated and not a
complete removal, I agree that it is somewhat premature to do this prior to
having a migration process implemented. Onur has described this in detail
in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s and I'm
surprised that more companies aren't affected by (or aware of?) the issue.
On Thu, Jan 5, 2017 at 4:40 PM, radai <ra...@gmail.com> wrote:
> I cant speak for anyone else, but a rolling upgrade is definitely how we
> (LinkedIn) will do the migration.
>
> On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > it sounds good to have
> > it, but that's probably not how people will end up migrati
> >
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by radai <ra...@gmail.com>.
I cant speak for anyone else, but a rolling upgrade is definitely how we
(LinkedIn) will do the migration.
On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira <gw...@confluent.io> wrote:
> it sounds good to have
> it, but that's probably not how people will end up migrati
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Gwen Shapira <gw...@confluent.io>.
Since the APIs are super different, I expect migrating from the old to
the new consumer will involve some re-write of the app that does the
consuming. In most such cases, the upgrade path involves running both
versions side-by-side for a while, validating results and then
retiring the old version. Sometimes migration of offsets is needed and
Grant published a tool for that a while back.
Having a rolling upgrade plan between both APIs is pretty involved and
I'm not sure there's a real demand for it (i.e. it sounds good to have
it, but that's probably not how people will end up migrating).
Gwen
On Thu, Jan 5, 2017 at 3:42 PM, radai <ra...@gmail.com> wrote:
> im all for (working towards) getting rid of old code, but there's still no
> solid migration path - you'll be "stranding" users on deprecated, no longer
> maintained code with no "safe" way out that does not involve downtime
> (specifically old and new consumers cannot correctly divide up partitions
> between themselves if both operate within the same group on the same topic).
>
> On Thu, Jan 5, 2017 at 3:10 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
>> Very strong support from me too :)
>>
>> On Thu, Jan 5, 2017 at 12:09 PM, Vahid S Hashemian
>> <va...@us.ibm.com> wrote:
>> > Hi all,
>> >
>> > There was some discussion recently on deprecating the old consumer (
>> > https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
>> > Ismael suggested to cover the discussion and voting of major deprecations
>> > like this under a KIP.
>> >
>> > So I started KIP-109 (
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 109%3A+Old+Consumer+Deprecation
>> > ) and look forward to your feedback and comments.
>> >
>> > We'd like to implement this deprecation in the upcoming 0.10.2.0 release.
>> >
>> > Thanks.
>> > --Vahid
>> >
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by radai <ra...@gmail.com>.
im all for (working towards) getting rid of old code, but there's still no
solid migration path - you'll be "stranding" users on deprecated, no longer
maintained code with no "safe" way out that does not involve downtime
(specifically old and new consumers cannot correctly divide up partitions
between themselves if both operate within the same group on the same topic).
On Thu, Jan 5, 2017 at 3:10 PM, Gwen Shapira <gw...@confluent.io> wrote:
> Very strong support from me too :)
>
> On Thu, Jan 5, 2017 at 12:09 PM, Vahid S Hashemian
> <va...@us.ibm.com> wrote:
> > Hi all,
> >
> > There was some discussion recently on deprecating the old consumer (
> > https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
> > Ismael suggested to cover the discussion and voting of major deprecations
> > like this under a KIP.
> >
> > So I started KIP-109 (
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 109%3A+Old+Consumer+Deprecation
> > ) and look forward to your feedback and comments.
> >
> > We'd like to implement this deprecation in the upcoming 0.10.2.0 release.
> >
> > Thanks.
> > --Vahid
> >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
Re: [DISCUSS] KIP-109: Old Consumer Deprecation
Posted by Gwen Shapira <gw...@confluent.io>.
Very strong support from me too :)
On Thu, Jan 5, 2017 at 12:09 PM, Vahid S Hashemian
<va...@us.ibm.com> wrote:
> Hi all,
>
> There was some discussion recently on deprecating the old consumer (
> https://www.mail-archive.com/dev@kafka.apache.org/msg59084.html).
> Ismael suggested to cover the discussion and voting of major deprecations
> like this under a KIP.
>
> So I started KIP-109 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-109%3A+Old+Consumer+Deprecation
> ) and look forward to your feedback and comments.
>
> We'd like to implement this deprecation in the upcoming 0.10.2.0 release.
>
> Thanks.
> --Vahid
>
--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog