You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by gu...@apache.org on 2019/01/11 19:35:45 UTC
[kafka] branch trunk updated: MINOR: Need to have same wait as
verify timeout broker upgrade downgrade (#6127)
This is an automated email from the ASF dual-hosted git repository.
guozhang pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new 3746bf2 MINOR: Need to have same wait as verify timeout broker upgrade downgrade (#6127)
3746bf2 is described below
commit 3746bf2c848e28ad5551a1adcf7516da1d61dbc5
Author: Bill Bejeck <bb...@gmail.com>
AuthorDate: Fri Jan 11 14:35:33 2019 -0500
MINOR: Need to have same wait as verify timeout broker upgrade downgrade (#6127)
When I originally refactored the streams_upgrade_test#upgrade_downgrade_brokers test I removed the wait call which would wait for up 24 minutes for the SmokeTestDriver class to publish and verify all of its records.
Since most of the tests run in two minutes or less I set the monitor_log timeout to three minutes. However, the SmokeTestDriver#verify method allows up to six minutes to consume all records before verifying the monitor_log timeout needs to be greater than 6 minutes. I've set the timeout to 8 minutes.
Additionally, the steps needed to update the streams_upgrade_test should be documented as there are several components that need to get updated. So I've documented those steps here on the test as a giant comment.
Reviewers: Guozhang Wang <wa...@gmail.com>
---
.../tests/streams/streams_upgrade_test.py | 41 +++++++++++++++++++++-
1 file changed, 40 insertions(+), 1 deletion(-)
diff --git a/tests/kafkatest/tests/streams/streams_upgrade_test.py b/tests/kafkatest/tests/streams/streams_upgrade_test.py
index 885aefe..2810be2 100644
--- a/tests/kafkatest/tests/streams/streams_upgrade_test.py
+++ b/tests/kafkatest/tests/streams/streams_upgrade_test.py
@@ -36,6 +36,40 @@ metadata_2_versions = [str(LATEST_0_10_1), str(LATEST_0_10_2), str(LATEST_0_11_0
backward_compatible_metadata_2_versions = [str(LATEST_0_10_2), str(LATEST_0_11_0), str(LATEST_1_0), str(LATEST_1_1)]
metadata_3_or_higher_versions = [str(LATEST_2_0), str(LATEST_2_1), str(DEV_VERSION)]
+"""
+After each release one should first check that the released version has been uploaded to
+https://s3-us-west-2.amazonaws.com/kafka-packages/ which is the url used by system test to download jars;
+anyone can verify that by calling
+curl https://s3-us-west-2.amazonaws.com/kafka-packages/kafka_$scala_version-$version.tgz to download the jar
+and if it is not uploaded yet, ping the dev@kafka mailing list to request it being uploaded.
+
+This test needs to get updated, but this requires several steps
+which are outlined here:
+
+1. Update all relevant versions in tests/kafkatest/version.py this will include adding a new version for the new
+ release and bumping all relevant already released versions.
+
+2. Add the new version to the "kafkatest.version" import above and include the version in the
+ broker_upgrade_versions list above. You'll also need to add the new version to the
+ "StreamsUpgradeTestJobRunnerService" on line 484 to make sure the correct arguments are passed
+ during the system test run.
+
+3. Update the vagrant/bash.sh file to include all new versions, including the newly released version
+ and all point releases for existing releases. You only need to list the latest version in
+ this file.
+
+4. Then update all relevant versions in the tests/docker/Dockerfile
+
+5. Add a new "upgrade-system-tests-XXXX module under streams. You can probably just copy the
+ latest system test module from the last release. Just make sure to update the systout print
+ statement in StreamsUpgradeTest to the version for the release. After you add the new module
+ you'll need to update settings.gradle file to include the name of the module you just created
+ for gradle to recognize the newly added module
+
+6. Then you'll need to update any version changes in gradle/dependencies.gradle
+
+"""
+
class StreamsUpgradeTest(Test):
"""
Test upgrading Kafka Streams (all version combination)
@@ -139,8 +173,13 @@ class StreamsUpgradeTest(Test):
timeout_sec=60,
err_msg="Never saw output '%s' on" % self.processed_msg + str(processor.node.account))
+ # SmokeTestDriver allows up to 6 minutes to consume all
+ # records for the verification step so this timeout is set to
+ # 6 minutes (360 seconds) for consuming of verification records
+ # and a very conservative additional 2 minutes (120 seconds) to process
+ # the records in the verification step
driver_monitor.wait_until('ALL-RECORDS-DELIVERED\|PROCESSED-MORE-THAN-GENERATED',
- timeout_sec=180,
+ timeout_sec=480,
err_msg="Never saw output '%s' on" % 'ALL-RECORDS-DELIVERED|PROCESSED-MORE-THAN-GENERATED' + str(self.driver.node.account))
self.driver.stop()