You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "brandboat (via GitHub)" <gi...@apache.org> on 2024/12/15 07:52:25 UTC
[PR] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
brandboat opened a new pull request, #18193:
URL: https://github.com/apache/kafka/pull/18193
related to https://issues.apache.org/jira/browse/KAFKA-18082
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2564488560
Yes, that would be simplest. That said, we should be clear that Connect is more of a server component (i.e. it doesn't run alongside customer applications) even though it communicates with the brokers as regular clients do.
Perhaps we should start that discussion with the Streams folks again. I want to check the specifics of the Streams upgrade test as it seems pretty complicated (https://github.com/apache/kafka/pull/17876/files).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2563174847
> which includes the underlying clients used by Kafka Streams
Users might be confused when they see that "Kafka Streams compatibility ranges includes 0.11 to 4.0 and Kafka Consumer requires 0.11 -> 3.9 and 3.9 -> 4.0". Although we can explain that Kafka Streams internally handles consumer compatibility. However, if there is indeed no known compatibility issue - it seems to me providing a clear and straightforward compatibility matrix #18091 along with comprehensive compatibility tests would better encourage users to upgrade to version 4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2573003033
Thanks for @chia7712 conclusion,
> review issue https://github.com/apache/kafka/pull/18091 to ensure we have comprehensive documentation for both the protocol and upgrade procedures. (Thank you, @m1a2st.
From the user’s perspective, I completely agree with presenting this complex information to them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2571837813
> I thought KS would follow client compatibility since KS relies on the client. But later I realize KS has its own client APIs, so we can't guarantee that KS will always align with the same compatibility matrix as Kafka client.
Of course, KS uses client internally, and thus it inherits the broker-client compatibility. Thus, KS 2.0 cannot work against a 4.0 broker...
> Otherwise, users might upgrade their client to a "recommended" bridge version only to find that it lacks support for necessary protocols...
Again. It seems we are mixing client-broker vs client/KS side upgrade compatibility... Of course, brokers 4.0 require KS 2.1...
> I want to emphasize that version 4.x occupies the central position,
I disagree... You still argue in term of client-broker compatibility, but that's not my point.
> Consequently, including version 2.0 in the bridge version could potentially result in compatibility issues with the 4.0 broker.
Again: I am not arguing client-broker upgrades, and yes, broker 4.0 require KS 2.1. But we need to keep both compatibility dimensions independent. The notion of a "bridge" version is for client-broker and yes, also applies to KS. But if I am on an older broker, I can still use KS 2.0.
Plain clients only focus on client-broker compatibility, because their Java API is rather simple (even if there is also compatibility guarantees here), and because the Java API is stateless. Kafka Streams is stateful and has much higher dependency also to brokers (ie, changelog/repartition topics), so KS compatibility by itself is a big deal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "brandboat (via GitHub)" <gi...@apache.org>.
brandboat commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2562793563
> @chia7712 Kafka Streams is different - it has to handle its own state during upgrades (kind of like the broker). The same is not true for regular clients. Maintaining tests like this one is pretty painful. Things will eventually break in odd ways (since the 0.11 code is so old). Also, it's too risky to go from 0.11 to 4.x+ in one go given multiple cycles of deprecations, removals, etc. (including non intersecting supported Java versions).
Sorry to interrupt the discussion and thanks for the insightful comment, I 100% agree we don't encourage user to do major bumps like 0.11 to 4.x.
I have a dumb question here. If we can't guarantee Kafka Consumer can successfully upgrade from 0.11 to 4.x, how can Kafka Streams guarantee that? Especially this kind of upgrade is still valid until Kafka Streams 5.0.0 comes out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2589683436
@dajac Does that mean the whole discussion was not about this test?
The discussion was about testing the upgrades of the consumer with the classic protocol, because currently Streams only works with the classic protocol. However, the test is about upgrading from classic to consumer protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2571324280
> We should add a document recommending the best way to upgrade from a very old version to the latest one. As you suggested above, users should first upgrade to 3.9 and then to 4.x. In other words, 3.9 will serve as the bridge version.
I think the upgrade path should be included by #18091 @m1a2st WDYT？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2569960386
I agree that is a larger discussion here for future releases and we should do that via a new KIP. I am also suggesting that we piggyback on KIP-896 to make 2.1 the baseline for clients/streams upgrades as well.
It's a simple message and It's pretty conservative (which is good for the first time we do something like this) and means we don't have to debug/diagnose issues related to super old versions. I tested many of these versions in the context of KIP-896 and it's painful - so much has changed since then (even things like basic CLI flags and configurations are not the same).
Also, I think it is actually a bad idea to upgrade directly from such an old version to 4.0. You can't even use the same Java version before/after the upgrade, there are many breaking changes, etc. it's very risky.
If people think it makes sense, we can take the discussion to the mailing list. If not, perhaps we can debate a bit more and see if we can find some common ground.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "brandboat (via GitHub)" <gi...@apache.org>.
brandboat commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2570375989
Have we reached a conclusion? It seems like we all agree that it's unlikely for users to upgrade from 0.11 to 4.x. Does this mean that Streams will follow the same pattern? If so, should we remove the end-to-end upgrade test (upgrading from 0.11, 1.0, 1.1, and 2.0 to the latest version) in Streams again?
> The main problem I see is, that we don't have any official statement what upgrades we support
@mjsax, +1 on this,
We should add a document recommending the best way to upgrade from a very old version to the latest one. As you suggested above, users should first upgrade to 3.9 and then to 4.x. In other words, 3.9 will serve as the bridge version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "brandboat (via GitHub)" <gi...@apache.org>.
brandboat commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2571558473
> Yes, I agree. Should we ensure that all clients (Consumer, Streams, Connect) follow a consistent upgrade path? It might be difficult for users to determine the correct sequence.
Hi all, sorry I want to go back to @chia7712's question. Assume that user have Streams and Kafka client on the same node. And Kafka Streams and the Kafka client have divergent upgrade paths. Which upgrade path should they choose? Or this kind of scenario doesn't exist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "brandboat (via GitHub)" <gi...@apache.org>.
brandboat commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2572303685
Thanks for the comment @chia7712,
> add 2.0 to consumer e2e upgrade test (this PR)
remove 0.11, 1.0 and 1.1 from streams e2e upgrade test ( this can be addressed in this PR too)
I can help with these two items in this PR if there is no objections, thank you.
> (optional) file a KIP to document the upgrade path (perhaps @brandboat could assist) and involve a wider range of kafka users
Sure thing, I believe document the upgrade path benefit lots of users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2563197985
@chia7712 the number of users looking to upgrade from 0.11 to 4.0 is likely 0. Such users are going to take years before they upgrade to 4.0. So, I think we'd be optimizing for the wrong thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2571326634
I will address it to the document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2570298714
> I am also suggesting that we piggyback on KIP-896 to make 2.1 the baseline for clients/streams upgrades as well.
Additionally, KIP-896 has explicitly defined version 2.1 as the baseline version. Based on this, it is reasonable to require users to first upgrade from version [ 0.7 ~ 2.0 ] to version [ 2.1 ~ 3.9 ], and then proceed to version 4.0.
If each module has its own different baseline, I believe it will cause maintenance and development difficulties, similar to the situation where each module has its own supported Java version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2589236563
Hi all. I am sorry but I am late to the party. What are we trying to achieve here? I see this jira marked as a blocker for 4.0 and at the same time it says that it needs a KIP. We have a dead lock here. Can we remove it as a blocker?
For the context, @dongnuo123 and I added this test as part of KIP-848 to test the the online migration path from the old protocol to the new protocol. This test was not meant to be a generic upgrade path verification for the consumer. At the beginning, we included all the versions but the test took to much time to run. So we ended up cherry-picking that versions that we were interested in (the one with major changes in the rebalance protocol).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2557957350
> Thanks for the PR. My take is that we do not want to do this. We want to take advantage of KIP-896 and have 2.1 as the new baseline. The value over benefit for these is not worth it
The benefit of this change is to ensure that rolling upgrades of the consumer work when the server is running a compatible version (e.g., 3.9). Since `streams_upgrade_test.py` already includes similar tests, adding tests for the consumer makes sense.
However, I agree that the benefit is minimal, as the use case is rare (e.g., upgrading from 0.11 to 0.4 is highly unusual). Therefore, I prefer to add a new test case (same file) that runs the 3.9 server to cover these rare scenarios. This approach would eliminate the numerous if-else conditions in the environment setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "brandboat (via GitHub)" <gi...@apache.org>.
brandboat commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2557994338
Thank you for the comments, and a special thanks to Chia-Ping for helping with the answer!
> However, I agree that the benefit is minimal, as the use case is rare (e.g., upgrading from 0.11 to 0.4 is highly unusual). Therefore, I prefer to add a new test case (same file) that runs the 3.9 server to cover these rare scenarios. This approach would eliminate the numerous if-else conditions in the environment setup.
Thanks for the suggestion! I’ll go ahead and extract those conditions into new tests, and if either of you have any further concerns, please feel free to let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2561940300
On a related note, I do think Kafka Streams should also trim their upgrade matrix, but I think they'll have to wait to 5.0 for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2564598755
@mjsax Apologies for bringing up this issue again. I believe that having clear and consistent rolling-upgrade rules can encourage users to adopt version 4.0, which is why I am persistent about this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2571564107
> I don't really think that KIP-896 is in the center, as it defines broker-client cross-version compatibility, while we are discussing KS upgrade compatibility...
I totally agree that it is a different issue. However, it would be beneficial to have a consistent matrix. Otherwise, users might upgrade their client to a "recommended" bridge version only to find that it lacks support for necessary protocols...
> [0.11.x - 1.x] -> [2.0 - 3.9] -> 4.x
I want to emphasize that version 4.x occupies the central position, and therefore, either an upgrade path or protocol compatibility is crucial for mitigating the anxiety associated with adopting 4.x.
Consequently, including version 2.0 in the bridge version could potentially result in compatibility issues with the 4.0 broker. This effectively renders the bridge unusable, as users would be unable to even conduct basic testing.
> It's confusing why 2.0 would be in the first bucket, but not in the second one...
This makes sense to me. I haven't investigated this thoroughly, but perhaps another approach would be to include version 2.0 within the scope of protocol changes. This would enable us to suggest a consistent upgrade/testing path for users: [0.11.x - 1.x] -> [2.0 - 3.9] -> 4.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2677396507
I am going to close this - given the discussion in KIP-1124, it is clear we won't be merging this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "brandboat (via GitHub)" <gi...@apache.org>.
brandboat commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2547378632
client e2e all passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "brandboat (via GitHub)" <gi...@apache.org>.
brandboat commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2571491577
> I don't really think that KIP-896 is in the center, as it defines broker-client cross-version compatibility, while we are discussing KS upgrade compatibility... This is two different things IMHO. If I have a 2.1+ broker, I can run a 2.0, 1.x, or 0.11.x KS application and upgrade it to 4.0 directly.
Thanks for clarifying! Initially, I thought KS would follow client compatibility since KS relies on the client. But later I realize KS has its own client APIs, so we can't guarantee that KS will always align with the same compatibility matrix as Kafka client.
> I would rather go with [0.11.x - 1.x] -> [2.0 - 3.9] -> 4.x
Sounds like this is something worth to document, should we write it down?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2569954461
Was just catching up on the discussion. Don't think it should be only me, but also @cadonna and @ableegoldman.
The main problem I see is, that we don't have any official statement what upgrades we support. So while I agree that is it very unlikely that somebody upgrades from 0.11 to 4.0, our policy so far was that we do support it... Of course, given that 4.0 is a major release, we could start to make a cut. The question is of course where to make the cut.
In the past, ie, 0.x and 1.x release cycles, we did push out major versions more frequently, and it was reasonable to have a longer window with regard to major versions. With 2.x and 3.x however, we did 9 and 10 releases (ie, 3 years!) each, so it might be sufficient to only support a single major version... Ie, we would only support upgrading from 3.x to 4.x (what is still 3 years worth of releases...), but even drop 2.8.x and older. Not sure if we want to make such a hard cut right away (or still include 2.x for now), but overall it would make sense to me (and we might want to do this at least with 5.x release...).
However, we should make this an official policy, and thus we need to discuss this on the dev mailing list IMHO, but not on a PR... Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2572283545
> Again: I am not arguing client-broker upgrades, and yes, broker 4.0 require KS 2.1. But we need to keep both compatibility dimensions independent. The notion of a "bridge" version is for client-broker and yes, also applies to KS. But if I am on an older broker, I can still use KS 2.0.
I understand. We can include a scenario in the documentation that covers upgrading the client to test the 4.0 broker. My primary concern is to prevent user confusion when selecting the appropriate bridge version. @m1a2st, please consider noting this special case in issue #18091? "If you plan to upgrade your broker to version 4.0, you may need to carefully consider avoiding the use of version 2.0 as a bridge version for your clients. This is because version 2.0 may include protocols that are no longer supported in broker version 4.0."
### conclusion
1. protocol matrix -> 2.1 is base line
2. (client and streams) upgrade path -> [0.11.x - 1.x] -> [2.0 - 3.9] -> 4.x
### follow-up
1. add 2.0 to consumer e2e upgrade test (this PR)
2. remove 0.11, 1.0 and 1.1 from streams e2e upgrade test ( this can be addressed in this PR too)
3. (optional) file a KIP to document the upgrade path (perhaps @brandboat could assist) and involve a wider range of kafka users
4. review issue #18091 to ensure we have comprehensive documentation for both the protocol and upgrade procedures. (Thank you, @m1a2st.
@ijuma @mjsax @cadonna @brandboat @m1a2st @ableegoldman WDYT? I believe this is a valuable discussion, as it allows for the establishment of an officially supported compatibility matrix. This matrix will be a valuable resource for users during upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2573651781
SGTM. Guess it was kinda random to make 2.1 the bridge version for client-broker compatibility -- not sure why we did not use 2.0 for it... but well. Water down the bridge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2574486228
To document the client upgrade path, I've opened a KIP at https://issues.apache.org/jira/browse/KAFKA-18422. Gathering community feedback and votes will likely take some time. Therefore, @m1a2st, perhaps you could focus on finalizing the protocol matrix in issue #18091. Then, @brandboat, you could enhance that table by adding the upgrade path information.
This PR should remain pending until KAFKA-18422 KIP is voted on :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2570125768
> In the past, ie, 0.x and 1.x release cycles, we did push out major versions more frequently, and it was reasonable to have a longer window with regard to major versions. With 2.x and 3.x however, we did 9 and 10 releases (ie, 3 years!) each, so it might be sufficient to only support a single major version... Ie, we would only support upgrading from 3.x to 4.x (what is still 3 years worth of releases...), but even drop 2.8.x and older. Not sure if we want to make such a hard cut right away (or still include 2.x for now), but overall it would make sense to me (and we might want to do this at least with 5.x release...).
Following the KIP-896 baseline provides a straightforward approach to defining the upgrade path. For example, for clients and streams: [0.7 - 2.0] -> [2.1 - 3.9] -> 4.0+ Given that the release date of 2.1.0 was "Released Nov 20, 2018," this approach offers a substantial window for client upgrades. I suggest including upgrade support for 2.x clients/streams in the 4.0 release and then moving the baseline to 3.x in the 5.0 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2571390278
> KIP-896 has explicitly defined version 2.1 as the baseline version
I don't really think that KIP-896 is in the center, as it defines broker-client cross-version compatibility, while we are discussing KS upgrade compatibility... This is two different things IMHO. If I have a 2.1+ broker, I can run a 2.0, 1.x, or 0.11.x KS application and upgrade it to 4.0 directly.
KIP-896 only defines 2.1 as baseline _client version_ for a broker upgrade to 4.0+. It does not say anything about client/KS upgrades. Thus, we should not mix both dimension (broker upgrade vs client upgrade) but discuss them independently IMHO.
Thus, following `[0.7 - 2.0] -> [2.1 - 3.9] -> 4.0+` proposal is somewhat weird from a KS upgrade POV... It's confusing why 2.0 would be in the first bucket, but not in the second one... I would rather go with `[0.11.x - 1.x] -> [2.0 - 3.9] -> 4.x`. Note that I changed lowest version to 0.11.x as we did have some KS breaking changes in 0.10.x releases (and there was no KS before 0.10.0.0 anyway), and also changed from `4.0+` to `4.x` to exclude `5.x` on purpose, so we can prepare to cut down upgrade guarantees to two major versions only. I think supporting upgrade between 3 major versions is something we should drop mid/long term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2557944364
Thanks for the PR. My take is that we do not want to do this. We want to take advantage of KIP-896 and have 2.1 as the new baseline. The value over benefit for these is not worth it. cc @chia7712.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2561939527
@chia7712 Kafka Streams is different - it has to handle its own state during upgrades (kind of like the broker). The same is not true for regular clients. Maintaining tests like this one is pretty painful. Things will eventually break in odd ways (since the 0.11 code is so old). Also, it's so risky to go from 0.11 to 4.x+ in one go given multiple cycles of deprecations, removals, etc.
It's simpler to say that such users should first upgrade to 3.9 and then do a second upgrade to 4.x+. This way you avoid maintaining (and running) these tests forever.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2562925466
@brandboat The Kafka Streams upgrade test ensures the end to end behavior is fine (which includes the underlying clients used by Kafka Streams).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2563219994
> the number of users looking to upgrade from 0.11 to 4.0 is likely 0. Such users are going to take years before they upgrade to 4.0. So, I think we'd be optimizing for the wrong thing.
Yes, I agree. Should we ensure that all clients (Consumer, Streams, Connect) follow a consistent upgrade path? It might be difficult for users to determine the correct sequence. For example, users with applications utilizing both Streams and Consumers may be unsure whether they should first upgrade both jars to version 3.9 and then proceed to 4.x+, or follow the Streams compatibility matrix to upgrade directly to 4.x+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "cadonna (via GitHub)" <gi...@apache.org>.
cadonna commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2574904402
I agree with the conclusion and the follow-up steps.
Looking forward to the KIP for the upgrade path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "brandboat (via GitHub)" <gi...@apache.org>.
brandboat commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2592912317
Hi all, I just wanted to let you know that I’ve filed a KIP to provide a clear Kafka client upgrade path: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1124%3A+Providing+a+clear+Kafka+Client+upgrade+path. I’d appreciate it if you could take a look when you have some time. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18193:
URL: https://github.com/apache/kafka/pull/18193#issuecomment-2592880931
> I see this jira marked as a blocker for 4.0 and at the same time it says that it needs a KIP. We have a dead lock here. Can we remove it as a blocker?
Could we consider this KIP as an exception? The primary changes introduced by this KIP are the addition of related end-to-end tests and the update of upgrade paths in the documentation. These changes primarily pertain to testing and documentation, and do not directly impact production code. Given that we have nearly reached a consensus on the associated PR, perhaps we could proceed with direct voting for this KIP.
Alternatively, we could defer the KIP (along with the related upgrade/matrix documentation) to the 4.1 release. The streams end-to-end tests already include support for 2.0, suggesting that there are no immediate critical upgrade issues at this time. If Kafka users inquire about client upgrade paths for 4.0, we can direct them to this PR.
> The discussion was about testing the upgrades of the consumer with the classic protocol, because currently Streams only works with the classic protocol. However, the test is about upgrading from classic to consumer protocol.
While `consumer_protocol_migration_test.py` currently lacks a dedicated test for the classic protocol, the file provides numerous helpful utilities that could facilitate the addition of such a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma closed pull request #18193: [WIP] KAFKA-18082: Add 0.11, 1.0, 1.1 and 2.0 to consumer upgrade e2e
URL: https://github.com/apache/kafka/pull/18193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org