[jira] [Commented] (KAFKA-9323) Refactor Streams' upgrade system tests

["Nikolay Izhikov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016167#comment-17016167 ] 

Nikolay Izhikov commented on KAFKA-9323:
----------------------------------------

Hello, [~ableegoldman]

Can you, please, confirm my analysis of the current source code of stream system tests:

We have 4 python scripts with the upgrade tests:
 * streams_broker_compatibility_test.py
 * streams_cooperative_rebalance_upgrade_test.py
 * streams_statis_memebership_test.py
 * streams_upgrade_test.py

> a) only the version probing test tests "forwards compatibility" (upgrade from latest to future version)

streams_upgrade_test.py#test_version_probing_upgrade

> b) nothing tests version probing "backwards compatibility" (upgrade from older version to latest), except:

a new test that should be created.

> c) the cooperative rebalancing test actually happens to involve a version probing step, and so coincidentally DOES test VP (but only starting with 2.4)

stream_cooperative_rebalance_upgrade_test.py#test_upgrade_to_cooperative_rebalance

> we should have a single matrix that we update with each new version that specifies which upgrade path that version combination actually requires

Do we have this matrix written somewhere?

As I can understand your proposal we should refactor all script from the mentioned files.
 Is it correct?

> Refactor Streams'  upgrade system tests
> ---------------------------------------
>
>                 Key: KAFKA-9323
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9323
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Sophie Blee-Goldman
>            Assignee: Nikolay Izhikov
>            Priority: Major
>
> With the introduction of version probing in 2.0 and cooperative rebalancing in 2.4, the specific upgrade path depends heavily on the to & from version. This can be a complex operation, and we should make sure to test a realistic upgrade scenario across all possible combinations. The current system tests have gaps however, which have allowed bugs in the upgrade path to slip by unnoticed for several versions. 
> Our current system tests include a metadata upgrade test, a version probing test, and a cooperative upgrade test. This has a few drawbacks:
> a) only the version probing test tests "forwards compatibility" (upgrade from latest to future version)
> b) nothing tests version probing "backwards compatibility" (upgrade from older version to latest), except:
> c) the cooperative rebalancing test actually happens to involve a version probing step, and so coincidentally DOES test VP (but only starting with 2.4)
> d) each test itself tries to test the upgrade across different versions, meaning there may be overlap and/or unnecessary tests 
> e) as new versions are released, it is unclear to those not directly involved in these tests and/or projects whether and what needs to be updated (eg should this new version be added to the cooperative test? should the old version be aded to the metadata test?)
> We should definitely fill in the testing gap here, but how to do so is of course up for discussion.
> I would propose to refactor the upgrade tests, and rather than maintain different lists of versions to pass as input to each different test, we should have a single matrix that we update with each new version that specifies which upgrade path that version combination actually requires. We can then loop through each version combination and test only the actual upgrade path that users would actually need to follow. This way we can be sure we are not missing anything, as each and every possible upgrade would be tested.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)