You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ruslan Fomkin (Jira)" <ji...@apache.org> on 2021/09/13 12:06:00 UTC
[jira] [Created] (CASSANDRA-16946) Find and remove duplicate
execution of dtests enabled by inheritance
Ruslan Fomkin created CASSANDRA-16946:
-----------------------------------------
Summary: Find and remove duplicate execution of dtests enabled by inheritance
Key: CASSANDRA-16946
URL: https://issues.apache.org/jira/browse/CASSANDRA-16946
Project: Cassandra
Issue Type: Improvement
Components: Test/dtest/python
Reporter: Ruslan Fomkin
Patch to CASSANDRA-16841 helped to reveal that there are tests, which are selected multiple times for execution. The discussion starts from this comment. The reason for double test execution is having a test class inheriting another test class, which means that test methods in the parent test class will be selected for the execution two times: one as part of their class, i.e., the parent class, and one as part of the child class, since it inherits them.
In particular I noticed two different cases:
1. A base class is a test class according to PyTest convention and it contains test methods in it. It is imported in different dtest file, e.g., in upgrade folder to be executed as upgrade test, where it is inherited in a child class, which is also a test class. Just in the second file the test methods from the base class will be present two times: once due to import and once due to inheritance.
2. A base test class is inherited inside the same file and it is assumed in the file that the tests need to be executed only once. However, the tests will be selected two times, since the base test class will be selected to execution, and the child test class will include them and thus the test will be selected for the execution.
I illustrate this on a [commit before my fix|https://github.com/apache/cassandra-dtest/commit/efb82f574cd99ea598713035bbc9a95e568d73ce].
1. [upgrade_tests/bootstrap_upgrade_test.py|https://github.com/apache/cassandra-dtest/blob/efb82f574cd99ea598713035bbc9a95e568d73ce/upgrade_tests/bootstrap_upgrade_test.py] imports a test class {{TestBootstrap}} and then a test class {{TestBootstrapUpgrade}} inherits from it. This results in executing tests from {{TestBootstrap}} two times:
{code}
pytest --cassandra-dir=../cassandra/ upgrade_tests/bootstrap_upgrade_test.py --collect-only --execute-upgrade-tests-only
...
collected 57 items
<Module 'upgrade_tests/bootstrap_upgrade_test.py'>
<Class 'TestBootstrap'>
<Instance '()'>
<Function 'test_simple_bootstrap_with_ssl'>
<Function 'test_simple_bootstrap'>
<Function 'test_bootstrap_on_write_survey'>
<Function 'test_simple_bootstrap_small_keepalive_period'>
<Function 'test_simple_bootstrap_nodata'>
<Function 'test_schema_removed_nodes'>
<Function 'test_read_from_bootstrapped_node'>
<Function 'test_bootstrap_waits_for_streaming_to_finish'>
<Function 'test_consistent_range_movement_true_with_replica_down_should_fail'>
<Function 'test_consistent_range_movement_false_with_replica_down_should_succeed'>
<Function 'test_consistent_range_movement_true_with_rf1_should_fail'>
<Function 'test_consistent_range_movement_false_with_rf1_should_succeed'>
<Function 'test_rf_gt_nodes_multidc_should_succeed'>
<Function 'test_resumable_bootstrap'>
<Function 'test_bootstrap_with_reset_bootstrap_state'>
<Function 'test_manual_bootstrap'>
<Function 'test_local_quorum_bootstrap'>
<Function 'test_shutdown_wiped_node_cannot_join'>
<Function 'test_killed_wiped_node_cannot_join'>
<Function 'test_decommissioned_wiped_node_can_join'>
<Function 'test_decommissioned_wiped_node_can_gossip_to_single_seed'>
<Function 'test_failed_bootstrap_wiped_node_can_join'>
<Function 'test_node_cannot_join_as_hibernating_node_without_replace_address'>
<Function 'test_simultaneous_bootstrap'>
<Function 'test_cleanup'>
<Function 'test_bootstrap_binary_disabled'>
<Function 'test_invalid_host_id'>
<Function 'test_host_id_override'>
<Class 'TestBootstrapUpgrade'>
<Instance '()'>
<Function 'test_simple_bootstrap_with_ssl'>
<Function 'test_simple_bootstrap'>
<Function 'test_bootstrap_on_write_survey'>
<Function 'test_simple_bootstrap_small_keepalive_period'>
<Function 'test_simple_bootstrap_nodata'>
<Function 'test_schema_removed_nodes'>
<Function 'test_read_from_bootstrapped_node'>
<Function 'test_bootstrap_waits_for_streaming_to_finish'>
<Function 'test_consistent_range_movement_true_with_replica_down_should_fail'>
<Function 'test_consistent_range_movement_false_with_replica_down_should_succeed'>
<Function 'test_consistent_range_movement_true_with_rf1_should_fail'>
<Function 'test_consistent_range_movement_false_with_rf1_should_succeed'>
<Function 'test_rf_gt_nodes_multidc_should_succeed'>
<Function 'test_resumable_bootstrap'>
<Function 'test_bootstrap_with_reset_bootstrap_state'>
<Function 'test_manual_bootstrap'>
<Function 'test_local_quorum_bootstrap'>
<Function 'test_shutdown_wiped_node_cannot_join'>
<Function 'test_killed_wiped_node_cannot_join'>
<Function 'test_decommissioned_wiped_node_can_join'>
<Function 'test_decommissioned_wiped_node_can_gossip_to_single_seed'>
<Function 'test_failed_bootstrap_wiped_node_can_join'>
<Function 'test_node_cannot_join_as_hibernating_node_without_replace_address'>
<Function 'test_simultaneous_bootstrap'>
<Function 'test_cleanup'>
<Function 'test_bootstrap_binary_disabled'>
<Function 'test_invalid_host_id'>
<Function 'test_host_id_override'>
<Function 'test_simple_bootstrap_mixed_versions'>
{code}
2. [/sstable_generation_loading_test.py|https://github.com/apache/cassandra-dtest/blob/efb82f574cd99ea598713035bbc9a95e568d73ce/sstable_generation_loading_test.py] defines a base class {{TestBaseSStableLoader}}, which is inherited by {{TestSSTableGenerationAndLoading}} without doing any changes to test configurations. As result the test methods from the base class are selected for execution two times:
{code}
pytest --cassandra-dir=../cassandra/ sstable_generation_loading_test.py --collect-only
...
collected 24 items
<Module 'sstable_generation_loading_test.py'>
<Class 'TestBaseSStableLoader'>
<Instance '()'>
<Function 'test_sstableloader_compression_none_to_none'>
<Function 'test_sstableloader_compression_none_to_snappy'>
<Function 'test_sstableloader_compression_none_to_deflate'>
<Function 'test_sstableloader_compression_snappy_to_none'>
<Function 'test_sstableloader_compression_snappy_to_snappy'>
<Function 'test_sstableloader_compression_snappy_to_deflate'>
<Function 'test_sstableloader_compression_deflate_to_none'>
<Function 'test_sstableloader_compression_deflate_to_snappy'>
<Function 'test_sstableloader_compression_deflate_to_deflate'>
<Function 'test_sstableloader_with_mv'>
<Class 'TestSSTableGenerationAndLoading'>
<Instance '()'>
<Function 'test_sstableloader_compression_none_to_none'>
<Function 'test_sstableloader_compression_none_to_snappy'>
<Function 'test_sstableloader_compression_none_to_deflate'>
<Function 'test_sstableloader_compression_snappy_to_none'>
<Function 'test_sstableloader_compression_snappy_to_snappy'>
<Function 'test_sstableloader_compression_snappy_to_deflate'>
<Function 'test_sstableloader_compression_deflate_to_none'>
<Function 'test_sstableloader_compression_deflate_to_snappy'>
<Function 'test_sstableloader_compression_deflate_to_deflate'>
<Function 'test_sstableloader_uppercase_keyspace_name'>
<Function 'test_incompressible_data_in_compressed_table'>
<Function 'test_remove_index_file'>
<Function 'test_sstableloader_with_mv'>
<Function 'test_sstableloader_with_failing_2i'>
{code}
[This commit|https://github.com/apache/cassandra-dtest/pull/152/commits/4952c6392a4cc5735a6c8488312defab930ac543] fixes the above cases, however it is likely to be more of them in dtest repo. It is possible to discover them in two ways:
1. Run pytest with collect only and see if there are duplications. It will require to use additional options to enable marked tests.
2. Search for test classes inheriting other test classes, i.e., according the pattern {{class Test*(Test*)}}
Note that there will be false positives, since there are cases to inherit on purpose with different configuration. See [this example|https://github.com/apache/cassandra-dtest/blob/efb82f574cd99ea598713035bbc9a95e568d73ce/upgrade_tests/storage_engine_upgrade_test.py#L514]
It will be good to implement a way to prevent such issue in future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org