You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mimaison (via GitHub)" <gi...@apache.org> on 2023/06/15 13:28:05 UTC
[GitHub] [kafka] mimaison opened a new pull request, #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
mimaison opened a new pull request, #13860:
URL: https://github.com/apache/kafka/pull/13860
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1773771925
@mjsax Good call, I've split this PR in 2:
- 3.4: https://github.com/apache/kafka/pull/14601
- 3.5: https://github.com/apache/kafka/pull/14602
Closing this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1760562882
> while it was backported to 3.3 branch, too, we never did a bug fix release afterwards and thus the system test that pull release binaries only cannot pickup the cherry-pick
I could not reproduce the issue any longer and did more digging -- it think I hit this issue previously because we did not bump the `Dockerfile` to use the latest bug fix versions. Looking into the fix for this, it's actually shipped with 3.3.2 release, so I think we are fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1748136807
I identified one fundamental problem in 3.3 release -- a bug was introduced that fundamentally breaks KS systems tests using that version. This PR (https://github.com/apache/kafka/commit/fa0324485b1e76388bb15ea98170cfb9202aab1f) changes `ProcessorParameter` and allows `this.processorSupplier` to be `null` what breaks `toString()` method on the same class.
It was fixed via https://github.com/apache/kafka/commit/0de037423bef416abd654bd8b90b642d56eb33ea but it's only included in 3.4.1 release -- while it was backported to `3.3` branch, too, we never did a bug fix release afterwards and thus the system test that pull release binaries only cannot pickup the cherry-pick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1652102807
With KIP-904 we changes some serialization format inside an internal repartition topic (cf https://cwiki.apache.org/confluence/display/KAFKA/KIP-904%3A+Kafka+Streams+-+Guarantee+subtractor+is+called+before+adder+if+key+has+not+changed)
To upgrade KS correctly, you need to do two rolling bounces (first one with `upgrade.from = 3.4` -- or whatever version you are using), and second one with the config set back to `null`.
Thus, it seems, while the upgrade path for KP-904 itself works, the `AssignorConfiguration` you mentioned would not recognize `3.4.0` an thus crash Kafka Streams.
We did not get to finish the system test for this upgrade (it's a WIP PR), that I assume would have caught this issue...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "vvcephei (via GitHub)" <gi...@apache.org>.
vvcephei commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1651883248
Good catch, @mimaison .
It looks like we could keep adding "upgrade from" versions on each release, but unless there's a compatibility issue, we don't need to. If you look at `git blame`, you can see that, in the past, we didn't add those cases one at a time, but in chunks when we had a compatibility problem to manage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1651991427
Thanks for confirming @vvcephei -- as a matter of fact, we released KP-914 with 3.5, so it seems to be an issue. Let me do a PR to address this and also add a test to get a guard in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1653760139
System tests look better with https://github.com/apache/kafka/pull/14103. In addition to a few flaky tests, I got 3 failures:
1.
```
Module: kafkatest.tests.streams.streams_broker_bounce_test
Class: StreamsBrokerBounceTest
Method: test_broker_type_bounce
Arguments:
{
"broker_type": "controller",
"failure_mode": "hard_shutdown",
"metadata_quorum": "ZK",
"num_threads": 1,
"sleep_time_secs": 120
}
```
2.
```
Module: kafkatest.tests.streams.streams_broker_bounce_test
Class: StreamsBrokerBounceTest
Method: test_broker_type_bounce
Arguments:
{
"broker_type": "leader",
"failure_mode": "hard_shutdown",
"metadata_quorum": "ZK",
"num_threads": 1,
"sleep_time_secs": 120
}
```
3.
```
Module: kafkatest.tests.streams.streams_upgrade_test
Class: StreamsUpgradeTest
Method: test_rolling_upgrade_with_2_bounces
Arguments:
{
"from_version": "3.5.0",
"to_version": "3.6.0-SNAPSHOT"
}
```
I still need to look at the logs of all failures but at first sight 1. and 2. don't seem directly related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1757751832
Thanks @mjsax for following up. So what's the path forward? Are we able to merge this or do we need to release new bugfix versions for 3.3 and 3.4 (and 3.5?)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1716805879
We also observed failing tests -- it's on our TODO list but might take some time until we get to it :( -- I'll keep you posted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] divijvaidya commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1653352409
> can we add 3.5.1 in this PR right away
I have added system tests to use 3.5.1 in https://github.com/apache/kafka/pull/14069
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1767572941
Summary about your test results:
- `streams_upgrade_test` expected to still fail because 14539 was not merged yet
- `streams_cooperative_rebalance_upgrade_test` -- seems to be flaky -- after re-running them multiple times, I got each one to pass at least once locally (we should still try to stabilize this test, but I don't think it's a broker to move forward with this PR)
- `streams_broker_down_resilience_test` -- this test was broken by a recent commit (https://github.com/apache/kafka/commit/fcac880fd54efbec3fe385000cf990a19972dafa) which changes a log message the test expects -- I'll add a fix to 14539 for it
- `streams_broker_compatibility_test` -- all test passed locally on the first run; not sure -- if we indeed see issues on Jenkins we need to figure out why it's flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1767576747
Updated https://github.com/apache/kafka/pull/14539 and re-triggered a system test run. Let's hope for the best :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1684358121
@mimaison Merged the hotfix PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1744647216
@mjsax These missing tests are starting to pile up. We're already missing them for 3.4 and 3.5, and 3.6 will be out shortly. If we were to break compatibility in Streams how would we notice it if we don't have these tests? Actually do we know whether the failures are caused by the tests or by issues in Streams? Figuring this out seems relatively high priority for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1649631445
@mjsax I also noticed https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/AssignorConfiguration.java#L180
This was not updated for 3.4, does this need updating too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1632088720
I copied what was done in previous releases. I'm unsure if any APIs have changed or what to really look for.
@mjsax @vvcephei Can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1688603919
I rebased on top of trunk and reran the system tests in our CI. Many of the Streams upgrade system tests are flaky but I've seen all of them pass at least once apart from `kafkatest.tests.streams.streams_upgrade_test.test_rolling_upgrade_with_2_bounces_from_version=3_5_1_to_version=3_6_0-SNAPSHOT`. This one consistently failed across ~10 runs.
From my limited Streams understanding I don't see any obvious issues in the logs. @mjsax can you take a look and see if it's just really flaky or if there's a real issue. Is is passing in your CI? I've uploaded the logs from the last run to my Apache home directory: https://home.apache.org/~mimaison/streams-upgrade-failure.zip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1669737299
This depends on https://github.com/apache/kafka/pull/14103 so we need to get that merged first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1767260687
> I rebased this PR on trunk
https://github.com/apache/kafka/pull/14539 is not merged yet... So that's expected I guess. -- We actually also just found another bug that was fixed in the meantime: https://github.com/apache/kafka/pull/14555
Let's see if we can push 14539 over the finish line first... Will ping here again after that's done; sorry that it take so long, but we are on it. Promised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison closed pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
URL: https://github.com/apache/kafka/pull/13860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1773536152
Merged the other PR. Btw: should we split this PR into two, one to add 3.4 and one to add 3.5? So it's easier to cherry-pick? (Adding 3.5 should only go into `3.6` branch, but adding 3.4 should got into `3.6` and `3.5` branches.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1651991541
Are there compatibility issues with 3.4 or 3.5?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1760532227
I just opened a PR to fix `trunk` -- if this PR is merged, I can cherry-pick it to older branches, and you can rebase this PR so we can verify everything is in order and we can merge this one.
https://github.com/apache/kafka/pull/14539
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mimaison (via GitHub)" <gi...@apache.org>.
mimaison commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1766096286
I rebased this PR on trunk and rerun the Streams system tests and still got a bunch of failures:
```
kafkatest.tests.streams.streams_upgrade_test.test_rolling_upgrade_with_2_bounces_from_version=0_10_0_1_to_version=3_7_0-SNAPSHOT
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.test_upgrade_to_cooperative_rebalance_upgrade_from_version=0_10_0_1
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.test_upgrade_to_cooperative_rebalance_upgrade_from_version=0_10_1_1
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.test_upgrade_to_cooperative_rebalance_upgrade_from_version=0_10_2_2
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.test_upgrade_to_cooperative_rebalance_upgrade_from_version=0_11_0_3
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.test_upgrade_to_cooperative_rebalance_upgrade_from_version=1_0_2
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.test_upgrade_to_cooperative_rebalance_upgrade_from_version=1_1_1
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.test_upgrade_to_cooperative_rebalance_upgrade_from_version=2_0_1
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.test_upgrade_to_cooperative_rebalance_upgrade_from_version=2_1_1
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.test_upgrade_to_cooperative_rebalance_upgrade_from_version=2_2_2
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.test_upgrade_to_cooperative_rebalance_upgrade_from_version=2_3_1
kafkatest.tests.streams.streams_broker_down_resilience_test.test_streams_runs_with_broker_down_initially_metadata_quorum=ISOLATED_KRAFT
kafkatest.tests.streams.streams_upgrade_test.test_rolling_upgrade_with_2_bounces_from_version=0_10_1_1_to_version=3_7_0-SNAPSHOT
kafkatest.tests.streams.streams_upgrade_test.test_rolling_upgrade_with_2_bounces_from_version=0_11_0_3_to_version=3_7_0-SNAPSHOT
kafkatest.tests.streams.streams_upgrade_test.test_rolling_upgrade_with_2_bounces_from_version=0_10_2_2_to_version=3_7_0-SNAPSHOT
kafkatest.tests.streams.streams_upgrade_test.test_rolling_upgrade_with_2_bounces_from_version=1_0_2_to_version=3_7_0-SNAPSHOT
kafkatest.tests.streams.streams_upgrade_test.test_rolling_upgrade_with_2_bounces_from_version=1_1_1_to_version=3_7_0-SNAPSHOT
kafkatest.tests.streams.streams_upgrade_test.test_rolling_upgrade_with_2_bounces_from_version=3_5_1_to_version=3_7_0-SNAPSHOT
kafkatest.tests.streams.streams_broker_compatibility_test.test_compatible_brokers_eos_alpha_enabled_broker_version=0_11_0_3
kafkatest.tests.streams.streams_broker_compatibility_test.test_compatible_brokers_eos_disabled_broker_version=0_11_0_3
kafkatest.tests.streams.streams_broker_compatibility_test.test_fail_fast_on_incompatible_brokers_broker_version=0_10_1_1
kafkatest.tests.streams.streams_broker_compatibility_test.test_fail_fast_on_incompatible_brokers_broker_version=0_10_2_2
kafkatest.tests.streams.streams_broker_compatibility_test.test_fail_fast_on_incompatible_brokers_if_eos_v2_enabled_broker_version=0_11_0_3
```
Tests can be a bit flaky in our CI but many of these seem to fail consistently. @mjsax did you get a clean run or do you see these tests failing in your CI as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1748162304
Another issue just introduced in `trunk` -> https://github.com/apache/kafka/pull/14490
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a diff in pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #13860:
URL: https://github.com/apache/kafka/pull/13860#discussion_r1263001020
##########
streams/upgrade-system-tests-34/src/test/java/org/apache/kafka/streams/tests/StreamsUpgradeTest.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.tests;
+
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Produced;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.ProcessorSupplier;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Properties;
+
+import static org.apache.kafka.streams.tests.SmokeTestUtil.intSerde;
+import static org.apache.kafka.streams.tests.SmokeTestUtil.stringSerde;
+
+
+public class StreamsUpgradeTest {
+
+ @SuppressWarnings("unchecked")
+ public static void main(final String[] args) throws Exception {
+ if (args.length < 1) {
+ System.err.println("StreamsUpgradeTest requires one argument (properties-file) but provided none");
+ }
+ final String propFileName = args[0];
+
+ final Properties streamsProperties = Utils.loadProps(propFileName);
+
+ System.out.println("StreamsTest instance started (StreamsUpgradeTest v3.3)");
Review Comment:
```suggestion
System.out.println("StreamsTest instance started (StreamsUpgradeTest v3.4)");
```
##########
streams/upgrade-system-tests-34/src/test/java/org/apache/kafka/streams/tests/SmokeTestUtil.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.tests;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KeyValueMapper;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.ProcessorSupplier;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.time.Instant;
+
+public class SmokeTestUtil {
+
+ final static int END = Integer.MAX_VALUE;
+
+ static ProcessorSupplier<Object, Object, Void, Void> printProcessorSupplier(final String topic) {
+ return printProcessorSupplier(topic, "");
+ }
+
+ static ProcessorSupplier<Object, Object, Void, Void> printProcessorSupplier(final String topic, final String name) {
+ return () -> new ContextualProcessor<Object, Object, Void, Void>() {
+ private int numRecordsProcessed = 0;
+ private long smallestOffset = Long.MAX_VALUE;
+ private long largestOffset = Long.MIN_VALUE;
+
+ @Override
+ public void init(final ProcessorContext<Void, Void> context) {
+ super.init(context);
+ System.out.println("[3.3] initializing processor: topic=" + topic + " taskId=" + context.taskId());
Review Comment:
```suggestion
System.out.println("[3.4] initializing processor: topic=" + topic + " taskId=" + context.taskId());
```
##########
streams/upgrade-system-tests-35/src/test/java/org/apache/kafka/streams/tests/SmokeTestUtil.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.tests;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KeyValueMapper;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.ProcessorSupplier;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.time.Instant;
+
+public class SmokeTestUtil {
+
+ final static int END = Integer.MAX_VALUE;
+
+ static ProcessorSupplier<Object, Object, Void, Void> printProcessorSupplier(final String topic) {
+ return printProcessorSupplier(topic, "");
+ }
+
+ static ProcessorSupplier<Object, Object, Void, Void> printProcessorSupplier(final String topic, final String name) {
+ return () -> new ContextualProcessor<Object, Object, Void, Void>() {
+ private int numRecordsProcessed = 0;
+ private long smallestOffset = Long.MAX_VALUE;
+ private long largestOffset = Long.MIN_VALUE;
+
+ @Override
+ public void init(final ProcessorContext<Void, Void> context) {
+ super.init(context);
+ System.out.println("[3.3] initializing processor: topic=" + topic + " taskId=" + context.taskId());
Review Comment:
```suggestion
System.out.println("[3.5] initializing processor: topic=" + topic + " taskId=" + context.taskId());
```
##########
streams/upgrade-system-tests-35/src/test/java/org/apache/kafka/streams/tests/StreamsUpgradeTest.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.tests;
+
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Produced;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.ProcessorSupplier;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Properties;
+
+import static org.apache.kafka.streams.tests.SmokeTestUtil.intSerde;
+import static org.apache.kafka.streams.tests.SmokeTestUtil.stringSerde;
+
+
+public class StreamsUpgradeTest {
+
+ @SuppressWarnings("unchecked")
+ public static void main(final String[] args) throws Exception {
+ if (args.length < 1) {
+ System.err.println("StreamsUpgradeTest requires one argument (properties-file) but provided none");
+ }
+ final String propFileName = args[0];
+
+ final Properties streamsProperties = Utils.loadProps(propFileName);
+
+ System.out.println("StreamsTest instance started (StreamsUpgradeTest v3.3)");
+ System.out.println("props=" + streamsProperties);
+
+ final StreamsBuilder builder = new StreamsBuilder();
+ final KTable<String, Integer> dataTable = builder.table(
+ "data", Consumed.with(stringSerde, intSerde));
+ final KStream<String, Integer> dataStream = dataTable.toStream();
+ dataStream.process(printProcessorSupplier("data"));
+ dataStream.to("echo");
+
+ final boolean runFkJoin = Boolean.parseBoolean(streamsProperties.getProperty(
+ "test.run_fk_join",
+ "false"));
+ if (runFkJoin) {
+ try {
+ final KTable<Integer, String> fkTable = builder.table(
+ "fk", Consumed.with(intSerde, stringSerde));
+ buildFKTable(dataStream, fkTable);
+ } catch (final Exception e) {
+ System.err.println("Caught " + e.getMessage());
+ }
+ }
+
+ final Properties config = new Properties();
+ config.setProperty(
+ StreamsConfig.APPLICATION_ID_CONFIG,
+ "StreamsUpgradeTest");
+ config.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000);
+ config.putAll(streamsProperties);
+
+ final KafkaStreams streams = new KafkaStreams(builder.build(), config);
+ streams.start();
+
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ streams.close();
+ System.out.println("UPGRADE-TEST-CLIENT-CLOSED");
+ System.out.flush();
+ }));
+ }
+
+ private static void buildFKTable(final KStream<String, Integer> primaryTable,
+ final KTable<Integer, String> otherTable) {
+ final KStream<String, String> kStream = primaryTable.toTable()
+ .join(otherTable, v -> v, (k0, v0) -> v0)
+ .toStream();
+ kStream.process(printProcessorSupplier("fk"));
+ kStream.to("fk-result", Produced.with(stringSerde, stringSerde));
+ }
+
+ private static <KIn, VIn, KOut, VOut> ProcessorSupplier<KIn, VIn, KOut, VOut> printProcessorSupplier(final String topic) {
+ return () -> new ContextualProcessor<KIn, VIn, KOut, VOut>() {
+ private int numRecordsProcessed = 0;
+
+ @Override
+ public void init(final ProcessorContext<KOut, VOut> context) {
+ System.out.println("[3.3] initializing processor: topic=" + topic + "taskId=" + context.taskId());
Review Comment:
```suggestion
System.out.println("[3.5] initializing processor: topic=" + topic + "taskId=" + context.taskId());
```
##########
streams/upgrade-system-tests-35/src/test/java/org/apache/kafka/streams/tests/StreamsUpgradeTest.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.tests;
+
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Produced;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.ProcessorSupplier;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Properties;
+
+import static org.apache.kafka.streams.tests.SmokeTestUtil.intSerde;
+import static org.apache.kafka.streams.tests.SmokeTestUtil.stringSerde;
+
+
+public class StreamsUpgradeTest {
+
+ @SuppressWarnings("unchecked")
+ public static void main(final String[] args) throws Exception {
+ if (args.length < 1) {
+ System.err.println("StreamsUpgradeTest requires one argument (properties-file) but provided none");
+ }
+ final String propFileName = args[0];
+
+ final Properties streamsProperties = Utils.loadProps(propFileName);
+
+ System.out.println("StreamsTest instance started (StreamsUpgradeTest v3.3)");
Review Comment:
```suggestion
System.out.println("StreamsTest instance started (StreamsUpgradeTest v3.5)");
```
##########
streams/upgrade-system-tests-34/src/test/java/org/apache/kafka/streams/tests/StreamsUpgradeTest.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.tests;
+
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Produced;
+import org.apache.kafka.streams.processor.api.ContextualProcessor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.ProcessorSupplier;
+import org.apache.kafka.streams.processor.api.Record;
+
+import java.util.Properties;
+
+import static org.apache.kafka.streams.tests.SmokeTestUtil.intSerde;
+import static org.apache.kafka.streams.tests.SmokeTestUtil.stringSerde;
+
+
+public class StreamsUpgradeTest {
+
+ @SuppressWarnings("unchecked")
+ public static void main(final String[] args) throws Exception {
+ if (args.length < 1) {
+ System.err.println("StreamsUpgradeTest requires one argument (properties-file) but provided none");
+ }
+ final String propFileName = args[0];
+
+ final Properties streamsProperties = Utils.loadProps(propFileName);
+
+ System.out.println("StreamsTest instance started (StreamsUpgradeTest v3.3)");
+ System.out.println("props=" + streamsProperties);
+
+ final StreamsBuilder builder = new StreamsBuilder();
+ final KTable<String, Integer> dataTable = builder.table(
+ "data", Consumed.with(stringSerde, intSerde));
+ final KStream<String, Integer> dataStream = dataTable.toStream();
+ dataStream.process(printProcessorSupplier("data"));
+ dataStream.to("echo");
+
+ final boolean runFkJoin = Boolean.parseBoolean(streamsProperties.getProperty(
+ "test.run_fk_join",
+ "false"));
+ if (runFkJoin) {
+ try {
+ final KTable<Integer, String> fkTable = builder.table(
+ "fk", Consumed.with(intSerde, stringSerde));
+ buildFKTable(dataStream, fkTable);
+ } catch (final Exception e) {
+ System.err.println("Caught " + e.getMessage());
+ }
+ }
+
+ final Properties config = new Properties();
+ config.setProperty(
+ StreamsConfig.APPLICATION_ID_CONFIG,
+ "StreamsUpgradeTest");
+ config.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000);
+ config.putAll(streamsProperties);
+
+ final KafkaStreams streams = new KafkaStreams(builder.build(), config);
+ streams.start();
+
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ streams.close();
+ System.out.println("UPGRADE-TEST-CLIENT-CLOSED");
+ System.out.flush();
+ }));
+ }
+
+ private static void buildFKTable(final KStream<String, Integer> primaryTable,
+ final KTable<Integer, String> otherTable) {
+ final KStream<String, String> kStream = primaryTable.toTable()
+ .join(otherTable, v -> v, (k0, v0) -> v0)
+ .toStream();
+ kStream.process(printProcessorSupplier("fk"));
+ kStream.to("fk-result", Produced.with(stringSerde, stringSerde));
+ }
+
+ private static <KIn, VIn, KOut, VOut> ProcessorSupplier<KIn, VIn, KOut, VOut> printProcessorSupplier(final String topic) {
+ return () -> new ContextualProcessor<KIn, VIn, KOut, VOut>() {
+ private int numRecordsProcessed = 0;
+
+ @Override
+ public void init(final ProcessorContext<KOut, VOut> context) {
+ System.out.println("[3.3] initializing processor: topic=" + topic + "taskId=" + context.taskId());
Review Comment:
```suggestion
System.out.println("[3.4] initializing processor: topic=" + topic + "taskId=" + context.taskId());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1652126476
Did a PR: https://github.com/apache/kafka/pull/14103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #13860: KAFKA-15093: Add 3.4 and 3.5 Streams upgrade system tests
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13860:
URL: https://github.com/apache/kafka/pull/13860#issuecomment-1652268988
I just uploaded the artifact for 3.5.1 to the S3 bucket we use for system tests -- can we add 3.5.1 in this PR right away (or even wait for 3.5.2 -- we might need a hotfix for this upgrade issue...?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org