You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by ad...@apache.org on 2021/05/27 15:40:47 UTC
[cassandra] branch cassandra-4.0 updated (cfc8ef3 -> 0d3ead9)
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a change to branch cassandra-4.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git.
from cfc8ef3 ninja fix six typo to be six >=1.12.0
new 269bc5c Add CircleCI jobs to repeat upgrade tests n times
new 7694cc5 Merge branch 'cassandra-3.0' into cassandra-3.11
new 0d3ead9 Merge branch 'cassandra-3.11' into cassandra-4.0
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.circleci/config-2_1.yml | 224 +++--
.circleci/config.yml | 1836 +++++++++++++++++++++++------------
.circleci/config.yml.HIGHRES | 1836 +++++++++++++++++++++++------------
.circleci/config.yml.LOWRES | 1836 +++++++++++++++++++++++------------
.circleci/config.yml.MIDRES | 1852 +++++++++++++++++++++++-------------
doc/source/development/testing.rst | 113 ++-
6 files changed, 5069 insertions(+), 2628 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.11' into cassandra-4.0
Posted by ad...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch cassandra-4.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 0d3ead9f556cd3b04830a996518596182e78c3d4
Merge: cfc8ef3 7694cc5
Author: Andrés de la Peña <a....@gmail.com>
AuthorDate: Thu May 27 16:33:44 2021 +0100
Merge branch 'cassandra-3.11' into cassandra-4.0
.circleci/config-2_1.yml | 224 +++--
.circleci/config.yml | 1836 +++++++++++++++++++++++------------
.circleci/config.yml.HIGHRES | 1836 +++++++++++++++++++++++------------
.circleci/config.yml.LOWRES | 1836 +++++++++++++++++++++++------------
.circleci/config.yml.MIDRES | 1852 +++++++++++++++++++++++-------------
doc/source/development/testing.rst | 113 ++-
6 files changed, 5069 insertions(+), 2628 deletions(-)
diff --cc .circleci/config-2_1.yml
index 429ca79,efe9052..54eb3e4
--- a/.circleci/config-2_1.yml
+++ b/.circleci/config-2_1.yml
@@@ -97,284 -113,107 +120,301 @@@ j8_seq_executor: &j8_seq_executo
#exec_resource_class: xlarge
parallelism: 1 # sequential, single container tests: no parallelism benefits
-with_dtests_jobs: &with_dtest_jobs
- jobs:
- - build
- # Java 8 unit tests will be run automatically
- - j8_unit_tests:
- requires:
- - build
- - j8_jvm_dtests:
- requires:
- - build
- # specialized unit tests (all run on request using Java 8)
- - start_utests_long:
- type: approval
- requires:
- - build
- - utests_long:
- requires:
- - start_utests_long
- - start_utests_compression:
- type: approval
- requires:
- - build
- - utests_compression:
- requires:
- - start_utests_compression
- - start_utests_stress:
- type: approval
- requires:
- - build
- - utests_stress:
- requires:
- - start_utests_stress
- - start_j8_dtest_jars_build:
- type: approval
- - j8_dtest_jars_build:
- requires:
- - build
- - start_j8_dtest_jars_build
- - start_jvm_upgrade_dtest:
- type: approval
- - j8_jvm_upgrade_dtests:
- requires:
- - start_jvm_upgrade_dtest
- - j8_dtest_jars_build
- # Java 8 dtests (on request)
- - start_j8_dtests:
- type: approval
- requires:
- - build
- - j8_dtests-with-vnodes:
- requires:
- - start_j8_dtests
- - j8_dtests-no-vnodes:
- requires:
- - start_j8_dtests
- # Java 8 upgrade tests
- - start_upgrade_tests:
- type: approval
- requires:
- - build
- - j8_upgradetests-no-vnodes:
- requires:
- - start_upgrade_tests
- # Java 8 repeated utest (on request)
- - start_j8_repeated_utest:
- type: approval
- - j8_repeated_utest:
- requires:
- - start_j8_repeated_utest
- - build
- # Java 8 repeated dtest (on request)
- - start_j8_repeated_dtest:
- type: approval
- - j8_repeated_dtest:
- requires:
- - start_j8_repeated_dtest
- - build
- # Repeated Python upgrade dtest (on request)
- - start_repeated_upgrade_dtest:
- type: approval
- - repeated_upgrade_dtest:
- requires:
- - start_repeated_upgrade_dtest
- - build
- # Repeated JVM upgrade dtest (on request)
- - start_repeated_jvm_upgrade_dtest:
- type: approval
- - repeated_jvm_upgrade_dtest:
- requires:
- - start_repeated_jvm_upgrade_dtest
- - j8_dtest_jars_build
-
-with_dtest_jobs_only: &with_dtest_jobs_only
- jobs:
- - build
- - j8_dtests-with-vnodes:
- requires:
- - build
- - j8_dtests-no-vnodes:
- requires:
- - build
+j11_par_executor: &j11_par_executor
+ executor:
+ name: java11-executor
+ #exec_resource_class: xlarge
+ parallelism: 4
+
+j11_small_par_executor: &j11_small_par_executor
+ executor:
+ name: java11-executor
+ #exec_resource_class: xlarge
+ parallelism: 1
+
+j11_small_executor: &j11_small_executor
+ executor:
+ name: java11-executor
+ #exec_resource_class: medium
+ parallelism: 1
+
+j8_with_dtests_jobs: &j8_with_dtests_jobs
+ jobs:
+ - j8_build
+ # Java 8 unit tests will be run automatically
+ - j8_unit_tests:
+ requires:
+ - j8_build
+ - j8_jvm_dtests:
+ requires:
+ - j8_build
+ # Java 11 unit tests (on request, currently not working)
+ - start_j11_unit_tests:
+ type: approval
+ - j11_unit_tests:
+ requires:
+ - start_j11_unit_tests
+ - j8_build
+ # specialized unit tests (all run on request using Java 8)
+ - j8_cqlshlib_tests:
+ requires:
+ - j8_build
+ - start_utests_long:
+ type: approval
+ - utests_long:
+ requires:
+ - start_utests_long
+ - j8_build
+ - start_utests_compression:
+ type: approval
+ - utests_compression:
+ requires:
+ - start_utests_compression
+ - j8_build
+ - start_utests_stress:
+ type: approval
+ - utests_stress:
+ requires:
+ - start_utests_stress
+ - j8_build
+ - start_utests_fqltool:
+ type: approval
+ - utests_fqltool:
+ requires:
+ - start_utests_fqltool
+ - j8_build
+ - start_utests_system_keyspace_directory:
+ type: approval
+ - utests_system_keyspace_directory:
+ requires:
+ - start_utests_system_keyspace_directory
+ - j8_build
- - start_jvm_upgrade_dtest:
++ - start_j8_dtest_jars_build:
+ type: approval
+ - j8_dtest_jars_build:
+ requires:
+ - j8_build
- - start_jvm_upgrade_dtest
++ - start_j8_dtest_jars_build
++ - start_jvm_upgrade_dtest:
++ type: approval
+ - j8_jvm_upgrade_dtests:
+ requires:
++ - start_jvm_upgrade_dtest
+ - j8_dtest_jars_build
+ # Java 8 dtests (on request)
+ - start_j8_dtests:
+ type: approval
+ - j8_dtests-with-vnodes:
+ requires:
+ - start_j8_dtests
+ - j8_build
+ - j8_dtests-no-vnodes:
+ requires:
+ - start_j8_dtests
+ - j8_build
+ # Java 11 dtests (on request)
+ - start_j11_dtests:
+ type: approval
+ - j11_dtests-with-vnodes:
+ requires:
+ - start_j11_dtests
+ - j8_build
+ - j11_dtests-no-vnodes:
+ requires:
+ - start_j11_dtests
+ - j8_build
+ # Java 8 upgrade tests
+ - start_upgrade_tests:
+ type: approval
+ - j8_upgradetests-no-vnodes:
+ requires:
+ - start_upgrade_tests
+ - j8_build
+ - start_j8_cqlsh_tests-with-vnodes:
+ type: approval
+ - j8_cqlsh-dtests-py2-with-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-with-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py3-with-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-with-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py38-with-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-with-vnodes
+ - j8_build
+ - start_j8_cqlsh_tests-no-vnodes:
+ type: approval
+ - j8_cqlsh-dtests-py2-no-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-no-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py3-no-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-no-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py38-no-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-no-vnodes
+ - j8_build
+ - start_j11_cqlsh_tests-with-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py3-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py38-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j8_build
+ - start_j11_cqlsh_tests-no-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py3-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py38-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j8_build
+ # Java 8 repeated utest (on request)
- - start_j8_repeated-utest:
++ - start_j8_repeated_utest:
+ type: approval
- - j8_repeated-utest:
++ - j8_repeated_utest:
+ requires:
- - start_j8_repeated-utest
++ - start_j8_repeated_utest
+ - j8_build
+ # Java 11 repeated utest (on request)
- - start_j11_repeated-utest:
++ - start_j11_repeated_utest:
+ type: approval
- - j11_repeated-utest:
++ - j11_repeated_utest:
+ requires:
- - start_j11_repeated-utest
++ - start_j11_repeated_utest
+ - j8_build
+ # Java 8 repeated dtest (on request)
- - start_j8_repeated-dtest:
++ - start_j8_repeated_dtest:
+ type: approval
- - j8_repeated-dtest:
++ - j8_repeated_dtest:
+ requires:
- - start_j8_repeated-dtest
++ - start_j8_repeated_dtest
+ - j8_build
+ # Java 11 repeated dtest (on request)
- - start_j11_repeated-dtest:
++ - start_j11_repeated_dtest:
+ type: approval
- - j11_repeated-dtest:
++ - j11_repeated_dtest:
+ requires:
- - start_j11_repeated-dtest
++ - start_j11_repeated_dtest
+ - j8_build
++ # Repeated Python upgrade dtest (on request)
++ - start_repeated_upgrade_dtest:
++ type: approval
++ - repeated_upgrade_dtest:
++ requires:
++ - start_repeated_upgrade_dtest
++ - j8_build
++ # Repeated JVM upgrade dtest (on request)
++ - start_repeated_jvm_upgrade_dtest:
++ type: approval
++ - repeated_jvm_upgrade_dtest:
++ requires:
++ - start_repeated_jvm_upgrade_dtest
++ - j8_dtest_jars_build
+
+j11_with_dtests_jobs: &j11_with_dtests_jobs
+ jobs:
+ - j11_build
+ # Java 11 unit tests
+ - j11_unit_tests:
+ requires:
+ - j11_build
+ - j11_jvm_dtests:
+ requires:
+ - j11_build
+ - j11_cqlshlib_tests:
+ requires:
+ - j11_build
+ # Java 11 dtests (on request)
+ - start_j11_dtests:
+ type: approval
+ - j11_dtests-with-vnodes:
+ requires:
+ - start_j11_dtests
+ - j11_build
+ - j11_dtests-no-vnodes:
+ requires:
+ - start_j11_dtests
+ - j11_build
+ - start_j11_cqlsh_tests-with-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py3-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py38-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j11_build
+ - start_j11_cqlsh_tests-no-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py3-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py38-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j11_build
+ # Java 11 repeated utest (on request)
- - start_j11_repeated-utest:
++ - start_j11_repeated_utest:
+ type: approval
- - j11_repeated-utest:
++ - j11_repeated_utest:
+ requires:
- - start_j11_repeated-utest
++ - start_j11_repeated_utest
+ - j11_build
+ # Java 11 repeated dtest (on request)
- - start_j11_repeated-dtest:
++ - start_j11_repeated_dtest:
+ type: approval
- - j11_repeated-dtest:
++ - j11_repeated_dtest:
+ requires:
- - start_j11_repeated-dtest
++ - start_j11_repeated_dtest
+ - j11_build
+
+j8_with_dtest_jobs_only: &j8_with_dtest_jobs_only
+ jobs:
+ - j8_build
+ - j8_dtests-with-vnodes:
+ requires:
+ - j8_build
+ - j8_dtests-no-vnodes:
+ requires:
+ - j8_build
+
+j11_with_dtest_jobs_only: &j11_with_dtest_jobs_only
+ jobs:
+ - build
+ - j11-with-vnodes:
+ requires:
+ - j11_build
+ - j11_dtests-no-vnodes:
+ requires:
+ - j11_build
workflows:
version: 2
@@@ -631,237 -359,67 +671,284 @@@ jobs
- create_venv
- create_dtest_containers:
file_tag: j8_upgradetests_without_vnodes
- run_dtests_extra_args: '--execute-upgrade-tests'
- extra_env_args: 'RUN_STATIC_UPGRADE_MATRIX=true'
- tests_filter_pattern: '^upgrade_tests'
+ run_dtests_extra_args: '--execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all'
- run_dtests:
file_tag: j8_upgradetests_without_vnodes
- extra_env_args: 'RUN_STATIC_UPGRADE_MATRIX=true'
- pytest_extra_args: '--execute-upgrade-tests'
+ pytest_extra_args: '--execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all'
+
+ j8_cqlsh-dtests-py2-with-vnodes:
+ <<: *j8_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv
+ - create_dtest_containers:
+ file_tag: j8_with_vnodes
+ run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'"
+ - run_dtests:
+ file_tag: j8_with_vnodes
+ pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7'
+
+ j8_cqlsh-dtests-py3-with-vnodes:
+ <<: *j8_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv
+ - create_dtest_containers:
+ file_tag: j8_with_vnodes
+ run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'"
+ - run_dtests:
+ file_tag: j8_with_vnodes
+ pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6'
+
+ j8_cqlsh-dtests-py38-with-vnodes:
+ <<: *j8_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv:
+ python_version: '3.8'
+ - create_dtest_containers:
+ file_tag: j8_with_vnodes
+ run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'"
+ python_version: '3.8'
+ - run_dtests:
+ file_tag: j8_with_vnodes
+ pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8'
+ python_version: '3.8'
+
+ j8_cqlsh-dtests-py2-no-vnodes:
+ <<: *j8_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv
+ - create_dtest_containers:
+ file_tag: j8_without_vnodes
+ run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'"
+ - run_dtests:
+ file_tag: j8_without_vnodes
+ pytest_extra_args: '--skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7'
+
+ j8_cqlsh-dtests-py3-no-vnodes:
+ <<: *j8_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv
+ - create_dtest_containers:
+ file_tag: j8_without_vnodes
+ run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'"
+ - run_dtests:
+ file_tag: j8_without_vnodes
+ pytest_extra_args: '--skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6'
+
+ j8_cqlsh-dtests-py38-no-vnodes:
+ <<: *j8_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv:
+ python_version: '3.8'
+ - create_dtest_containers:
+ file_tag: j8_without_vnodes
+ run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'"
+ python_version: '3.8'
+ - run_dtests:
+ file_tag: j8_without_vnodes
+ pytest_extra_args: '--skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8'
+ python_version: '3.8'
+
+ j11_cqlsh-dtests-py2-with-vnodes:
+ <<: *j11_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv
+ - create_dtest_containers:
+ file_tag: j11_with_vnodes
+ run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'"
+ - run_dtests:
+ file_tag: j11_with_vnodes
+ pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7'
+
+ j11_cqlsh-dtests-py3-with-vnodes:
+ <<: *j11_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv
+ - create_dtest_containers:
+ file_tag: j11_with_vnodes
+ run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'"
+ - run_dtests:
+ file_tag: j11_with_vnodes
+ pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6'
+
+ j11_cqlsh-dtests-py38-with-vnodes:
+ <<: *j11_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv:
+ python_version: '3.8'
+ - create_dtest_containers:
+ file_tag: j11_with_vnodes
+ run_dtests_extra_args: "--use-vnodes --skip-resource-intensive-tests --pytest-options '-k cql'"
+ python_version: '3.8'
+ - run_dtests:
+ file_tag: j11_with_vnodes
+ pytest_extra_args: '--use-vnodes --num-tokens=16 --skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8'
+ python_version: '3.8'
+
+ j11_cqlsh-dtests-py2-no-vnodes:
+ <<: *j11_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv
+ - create_dtest_containers:
+ file_tag: j11_without_vnodes
+ run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'"
+ - run_dtests:
+ file_tag: j11_without_vnodes
+ pytest_extra_args: '--skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python2.7'
+
+ j11_cqlsh-dtests-py3-no-vnodes:
+ <<: *j11_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv
+ - create_dtest_containers:
+ file_tag: j11_without_vnodes
+ run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'"
+ - run_dtests:
+ file_tag: j11_without_vnodes
+ pytest_extra_args: '--skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.6'
+
+ j11_cqlsh-dtests-py38-no-vnodes:
+ <<: *j11_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv:
+ python_version: '3.8'
+ - create_dtest_containers:
+ file_tag: j11_without_vnodes
+ run_dtests_extra_args: "--skip-resource-intensive-tests --pytest-options '-k cql'"
+ python_version: '3.8'
+ - run_dtests:
+ file_tag: j11_without_vnodes
+ pytest_extra_args: '--skip-resource-intensive-tests'
+ extra_env_args: 'CQLSH_PYTHON=/usr/bin/python3.8'
+ python_version: '3.8'
- j8_repeated-utest:
+ j8_repeated_utest:
<<: *j8_par_executor
steps:
- attach_workspace:
at: /home/cassandra
- log_environment
- - run_repeated_utest
-
- j11_repeated-utest:
+ - run_repeated_utest:
+ target: ${REPEATED_UTEST_TARGET}
+ class: ${REPEATED_UTEST_CLASS}
+ methods: ${REPEATED_UTEST_METHODS}
+ count: ${REPEATED_UTEST_COUNT}
+ stop_on_failure: ${REPEATED_UTEST_STOP_ON_FAILURE}
+
++ j11_repeated_utest:
+ <<: *j11_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - log_environment
- - run_repeated_utest
-
- j8_repeated-dtest:
++ - run_repeated_utest:
++ target: ${REPEATED_UTEST_TARGET}
++ class: ${REPEATED_UTEST_CLASS}
++ methods: ${REPEATED_UTEST_METHODS}
++ count: ${REPEATED_UTEST_COUNT}
++ stop_on_failure: ${REPEATED_UTEST_STOP_ON_FAILURE}
++
+ j8_repeated_dtest:
<<: *j8_par_executor
steps:
- attach_workspace:
at: /home/cassandra
- clone_dtest
- create_venv
- - run_repeated_dtest
-
- j11_repeated-dtest:
+ - run_repeated_dtest:
+ tests: ${REPEATED_DTEST_NAME}
+ vnodes: ${REPEATED_DTEST_VNODES}
+ upgrade: "false"
+ count: ${REPEATED_DTEST_COUNT}
+ stop_on_failure: ${REPEATED_DTEST_STOP_ON_FAILURE}
+
++ j11_repeated_dtest:
+ <<: *j11_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - log_environment
+ - clone_dtest
+ - create_venv
- - run_repeated_dtest
++ - run_repeated_dtest:
++ tests: ${REPEATED_DTEST_NAME}
++ vnodes: ${REPEATED_DTEST_VNODES}
++ upgrade: "false"
++ count: ${REPEATED_DTEST_COUNT}
++ stop_on_failure: ${REPEATED_DTEST_STOP_ON_FAILURE}
++
+ repeated_jvm_upgrade_dtest:
+ <<: *j8_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - log_environment
+ - run_repeated_utest:
+ target: test-jvm-dtest-some
+ class: ${REPEATED_JVM_UPGRADE_DTEST_CLASS}
+ methods: ${REPEATED_JVM_UPGRADE_DTEST_METHODS}
+ count: ${REPEATED_JVM_UPGRADE_DTEST_COUNT}
+ stop_on_failure: ${REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE}
+
+ repeated_upgrade_dtest:
+ <<: *j8_par_executor
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - clone_dtest
+ - create_venv
+ - run_repeated_dtest:
+ tests: ${REPEATED_UPGRADE_DTEST_NAME}
+ vnodes: "false"
+ upgrade: "true"
+ stop_on_failure: ${REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE}
+ count: ${REPEATED_UPGRADE_DTEST_COUNT}
commands:
log_environment:
@@@ -1209,23 -743,34 +1296,34 @@@
destination: dtest_<<parameters.file_tag>>_logs
run_repeated_utest:
+ parameters:
+ target:
+ type: string
+ class:
+ type: string
+ methods:
+ type: string
+ count:
+ type: string
+ stop_on_failure:
+ type: string
steps:
- run:
- name: Run repeated utest
+ name: Run repeated JUnit test
no_output_timeout: 15m
command: |
- if [ "$REPEATED_UTEST_CLASS" == "<nil>" ]; then
+ if [ "<<parameters.class>>" == "<nil>" ]; then
echo "Repeated utest class name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" == "<nil>" ]; then
+ elif [ "<<parameters.count>>" == "<nil>" ]; then
echo "Repeated utest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" -le 0 ]; then
+ elif [ "<<parameters.count>>" -le 0 ]; then
echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
else
-
+
# Calculate the number of test iterations to be run by the current parallel runner.
# Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_UTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_UTEST_COUNT % CIRCLE_NODE_TOTAL))); then
+ count=$((<<parameters.count>> / CIRCLE_NODE_TOTAL))
+ if (($CIRCLE_NODE_INDEX < (<<parameters.count>> % CIRCLE_NODE_TOTAL))); then
count=$((count+1))
fi
@@@ -1344,8 -900,8 +1453,8 @@@
if (($count <= 0)); then
echo "No tests to run in this runner"
else
- echo "Running $REPEATED_DTEST_NAME $count times"
+ echo "Running <<parameters.tests>> $count times"
-
+
source ~/env3.6/bin/activate
export PATH=$JAVA_HOME/bin:$PATH
diff --cc .circleci/config.yml
index 822f246,1451eea..17a4855
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@@ -118,73 -117,159 +118,255 @@@ jobs
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ repeated_jvm_upgrade_dtest:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run repeated JUnit test
+ no_output_timeout: 15m
+ command: |
+ if [ "${REPEATED_JVM_UPGRADE_DTEST_CLASS}" == "<nil>" ]; then
+ echo "Repeated utest class name hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" == "<nil>" ]; then
+ echo "Repeated utest count hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" -le 0 ]; then
+ echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
+ else
+
+ # Calculate the number of test iterations to be run by the current parallel runner.
+ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
+ count=$((${REPEATED_JVM_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
+ if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
+ count=$((count+1))
+ fi
+
+ if (($count <= 0)); then
+ echo "No tests to run in this runner"
+ else
+ echo "Running test-jvm-dtest-some ${REPEATED_JVM_UPGRADE_DTEST_CLASS} ${REPEATED_JVM_UPGRADE_DTEST_METHODS} ${REPEATED_JVM_UPGRADE_DTEST_COUNT} times"
+
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+
+ target=test-jvm-dtest-some
+ class_path=${REPEATED_JVM_UPGRADE_DTEST_CLASS}
+ class_name="${class_path##*.}"
+
+ # Prepare the -Dtest.name argument.
+ # It can be the fully qualified class name or the short class name, depending on the target.
+ if [[ $target == "test" || \
+ $target == "test-cdc" || \
+ $target == "test-compression" || \
+ $target == "test-system-keyspace-directory" ]]; then
+ name="-Dtest.name=$class_name"
+ else
+ name="-Dtest.name=$class_path"
+ fi
+
+ # Prepare the -Dtest.methods argument, which is optional
+ if [ "${REPEATED_JVM_UPGRADE_DTEST_METHODS}" == "<nil>" ]; then
+ methods=""
+ else
+ methods="-Dtest.methods=${REPEATED_JVM_UPGRADE_DTEST_METHODS}"
+ fi
+
+ # Run the test target as many times as requested collecting the exit code,
+ # stopping the iteration only if stop_on_failure is set.
+ exit_code="$?"
+ for i in $(seq -w 1 $count); do
+
+ echo "Running test iteration $i of $count"
+
+ # run the test
+ status="passes"
+ if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
+ status="fails"
+ exit_code=1
+ fi
+
+ # move the stdout output file
+ dest=/tmp/results/repeated_utest/stdout/${status}/${i}
+ mkdir -p $dest
+ mv stdout.txt $dest/test-jvm-dtest-some-${REPEATED_JVM_UPGRADE_DTEST_CLASS}.txt
+
+ # move the XML output files
+ source=build/test/output
+ dest=/tmp/results/repeated_utest/output/${status}/${i}
+ mkdir -p $dest
+ if [[ -d $source && -n "$(ls $source)" ]]; then
+ mv $source/* $dest/
+ fi
+
+ # move the log files
+ source=build/test/logs
+ dest=/tmp/results/repeated_utest/logs/${status}/${i}
+ mkdir -p $dest
+ if [[ -d $source && -n "$(ls $source)" ]]; then
+ mv $source/* $dest/
+ fi
+
+ # maybe stop iterations on test failure
+ if [[ ${REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then
+ break
+ fi
+ done
+
+ (exit ${exit_code})
+ fi
+ fi
+ - store_test_results:
+ path: /tmp/results/repeated_utest/output
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/stdout
+ destination: stdout
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/logs
+ destination: logs
+ environment:
++ - ANT_HOME: /usr/share/ant
++ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ - LANG: en_US.UTF-8
++ - KEEP_TEST_DIR: true
++ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
++ - PYTHONIOENCODING: utf-8
++ - PYTHONUNBUFFERED: true
++ - CASS_DRIVER_NO_EXTENSIONS: true
++ - CASS_DRIVER_NO_CYTHON: true
++ - CASSANDRA_SKIP_SYNC: true
++ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
++ - DTEST_BRANCH: trunk
++ - CCM_MAX_HEAP_SIZE: 1024M
++ - CCM_HEAP_NEWSIZE: 256M
++ - REPEATED_UTEST_TARGET: testsome
++ - REPEATED_UTEST_CLASS: null
++ - REPEATED_UTEST_METHODS: null
++ - REPEATED_UTEST_COUNT: 100
++ - REPEATED_UTEST_STOP_ON_FAILURE: false
++ - REPEATED_DTEST_NAME: null
++ - REPEATED_DTEST_VNODES: false
++ - REPEATED_DTEST_COUNT: 100
++ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j8_cqlsh-dtests-py2-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j8_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip- [...]
+ - run:
+ name: Run dtests (j8_with_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
+ export CQLSH_PYTHON=/usr/bin/python2.7
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j8_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_with_vnodes_logs
+ environment:
- ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- LANG: en_US.UTF-8
- KEEP_TEST_DIR: true
- DEFAULT_DIR: /home/cassandra/cassandra-dtest
@@@ -206,112 -291,136 +388,248 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_unit_tests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Determine unit Tests to Run
+ command: |
+ # reminder: this code (along with all the steps) is independently executed on every circle container
+ # so the goal here is to get the circleci script to return the tests *this* container will run
+ # which we do via the `circleci` cli tool.
+
+ rm -fr ~/cassandra-dtest/upgrade_tests
+ echo "***java tests***"
+
+ # get all of our unit test filenames
+ set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt
+
+ # split up the unit tests into groups based on the number of containers we have
+ set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt
+ set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt"
+ cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ no_output_timeout: 15m
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run Unit Tests (testclasslist)
+ command: |
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true)
+ if [ -z "$test_timeout" ]; then
+ test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
+ fi
+ ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ repeated_upgrade_dtest:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
- pip3 install --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Run repeated Python dtest
+ no_output_timeout: 15m
+ command: |
+ if [ "${REPEATED_UPGRADE_DTEST_NAME}" == "<nil>" ]; then
+ echo "Repeated dtest name hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" == "<nil>" ]; then
+ echo "Repeated dtest count hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" -le 0 ]; then
+ echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
+ else
+
+ # Calculate the number of test iterations to be run by the current parallel runner.
+ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
+ count=$((${REPEATED_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
+ if (($CIRCLE_NODE_INDEX < (${REPEATED_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
+ count=$((count+1))
+ fi
+
+ if (($count <= 0)); then
+ echo "No tests to run in this runner"
+ else
+ echo "Running ${REPEATED_UPGRADE_DTEST_NAME} $count times"
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+
+ stop_on_failure_arg=""
+ if ${REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE}; then
+ stop_on_failure_arg="-x"
+ fi
+
+ vnodes_args=""
+ if false; then
+ vnodes_args="--use-vnodes --num-tokens=16"
+ fi
+
+ upgrade_arg=""
+ if true; then
+ upgrade_arg="--execute-upgrade-tests"
+ fi
+
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_UPGRADE_DTEST_NAME} | tee /tmp/dtest/stdout.txt
+ fi
+ fi
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_logs
+ environment:
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - ANT_HOME: /usr/share/ant
++ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_repeated_dtest:
+ j8_cqlsh-dtests-py38-no-vnodes:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: medium
@@@ -368,1938 -503,13 +686,2095 @@@
path: /tmp/results
- store_artifacts:
path: /tmp/dtest
- destination: dtest
+ destination: dtest_j8_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_cqlsh-dtests-py3-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
+ - run:
+ name: Run dtests (j11_with_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
+ export CQLSH_PYTHON=/usr/bin/python3.6
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_cqlsh-dtests-py3-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource- [...]
+ - run:
+ name: Run dtests (j11_without_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
+ export CQLSH_PYTHON=/usr/bin/python3.6
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
- j11_repeated-dtest:
++ j11_cqlsh-dtests-py38-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
- name: Log Environment Information
- command: |
- echo '*** id ***'
- id
- echo '*** cat /proc/cpuinfo ***'
- cat /proc/cpuinfo
- echo '*** free -m ***'
- free -m
- echo '*** df -m ***'
- df -m
- echo '*** ifconfig -a ***'
- ifconfig -a
- echo '*** uname -a ***'
- uname -a
- echo '*** mount ***'
- mount
- echo '*** env ***'
- env
- echo '*** java ***'
- which java
- java -version
- - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
- source ~/env3.6/bin/activate
++ source ~/env3.8/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Run repeated dtest
++ name: Determine Tests to Run (j11_with_vnodes)
++ no_output_timeout: 5m
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
++ - run:
++ name: Run dtests (j11_with_vnodes)
+ no_output_timeout: 15m
+ command: |
- if [ "$REPEATED_DTEST_NAME" == "<nil>" ]; then
- echo "Repeated dtest name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_DTEST_COUNT" == "<nil>" ]; then
- echo "Repeated dtest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_DTEST_COUNT" -le 0 ]; then
- echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
- else
-
- # Calculate the number of test iterations to be run by the current parallel runner.
- # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_DTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_DTEST_COUNT % CIRCLE_NODE_TOTAL))); then
- count=$((count+1))
- fi
-
- if (($count <= 0)); then
- echo "No tests to run in this runner"
- else
- echo "Running $REPEATED_DTEST_NAME $count times"
-
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
-
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
-
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
++ echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
+
- stop_on_failure_arg=""
- if $REPEATED_UTEST_STOP_ON_FAILURE; then
- stop_on_failure_arg="-x"
- fi
++ source ~/env3.8/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then
++ export CQLSH_PYTHON=/usr/bin/python3.8
++ fi
+
- vnodes_args=""
- if $REPEATED_DTEST_VNODES; then
- vnodes_args="--use-vnodes --num-tokens=16"
- fi
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
+
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $REPEATED_DTEST_NAME | tee /tmp/dtest/stdout.txt
- fi
- fi
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
- destination: dtest
++ destination: dtest_j11_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
- destination: dtest_logs
++ destination: dtest_j11_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
- j11_cqlsh-dtests-py38-with-vnodes:
++ j8_cqlsh-dtests-py3-with-vnodes:
+ docker:
- - image: apache/cassandra-testing-ubuntu2004-java11:20210304
++ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
- source ~/env3.8/bin/activate
++ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Determine Tests to Run (j11_with_vnodes)
++ name: Determine Tests to Run (j8_with_vnodes)
+ no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip- [...]
+ - run:
- name: Run dtests (j11_with_vnodes)
++ name: Run dtests (j8_with_vnodes)
+ no_output_timeout: 15m
+ command: |
- echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
- cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
++ echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt
+
- source ~/env3.8/bin/activate
++ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
- if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then
- export CQLSH_PYTHON=/usr/bin/python3.8
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
++ export CQLSH_PYTHON=/usr/bin/python3.6
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
- set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
- destination: dtest_j11_with_vnodes
++ destination: dtest_j8_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
- destination: dtest_j11_with_vnodes_logs
++ destination: dtest_j8_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - CASSANDRA_USE_JDK11: true
- j11_repeated-utest:
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ j8_cqlsh-dtests-py2-no-vnodes:
+ docker:
- - image: apache/cassandra-testing-ubuntu2004-java11:20210304
++ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
- name: Log Environment Information
++ name: Clone Cassandra dtest Repository (via git)
+ command: |
- echo '*** id ***'
- id
- echo '*** cat /proc/cpuinfo ***'
- cat /proc/cpuinfo
- echo '*** free -m ***'
- free -m
- echo '*** df -m ***'
- df -m
- echo '*** ifconfig -a ***'
- ifconfig -a
- echo '*** uname -a ***'
- uname -a
- echo '*** mount ***'
- mount
- echo '*** env ***'
- env
- echo '*** java ***'
- which java
- java -version
++ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
- name: Run repeated utest
++ name: Configure virtualenv and python Dependencies
++ command: |
++ # note, this should be super quick as all dependencies should be pre-installed in the docker image
++ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
++ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
++ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 uninstall -y cqlsh
++ pip3 freeze
++ - run:
++ name: Determine Tests to Run (j8_without_vnodes)
++ no_output_timeout: 5m
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
++ - run:
++ name: Run dtests (j8_without_vnodes)
+ no_output_timeout: 15m
+ command: |
- if [ "$REPEATED_UTEST_CLASS" == "<nil>" ]; then
- echo "Repeated utest class name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" == "<nil>" ]; then
- echo "Repeated utest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" -le 0 ]; then
- echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
- else
-
- # Calculate the number of test iterations to be run by the current parallel runner.
- # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_UTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_UTEST_COUNT % CIRCLE_NODE_TOTAL))); then
- count=$((count+1))
- fi
-
- if (($count <= 0)); then
- echo "No tests to run in this runner"
- else
- echo "Running $REPEATED_UTEST_TARGET $REPEATED_UTEST_CLASS $REPEATED_UTEST_METHODS $count times"
-
- set -x
- export PATH=$JAVA_HOME/bin:$PATH
- time mv ~/cassandra /tmp
- cd /tmp/cassandra
- if [ -d ~/dtest_jars ]; then
- cp ~/dtest_jars/dtest* /tmp/cassandra/build/
- fi
-
- target=$REPEATED_UTEST_TARGET
- class_path=$REPEATED_UTEST_CLASS
- class_name="${class_path##*.}"
-
- # Prepare the -Dtest.name argument.
- # It can be the fully qualified class name or the short class name, depending on the target.
- if [[ $target == "test" || \
- $target == "test-cdc" || \
- $target == "test-compression" || \
- $target == "test-system-keyspace-directory" ]]; then
- name="-Dtest.name=$class_name"
- else
- name="-Dtest.name=$class_path"
- fi
-
- # Prepare the -Dtest.methods argument, which is optional
- if [ "$REPEATED_UTEST_METHODS" == "<nil>" ]; then
- methods=""
- else
- methods="-Dtest.methods=$REPEATED_UTEST_METHODS"
- fi
-
- # Run the test target as many times as requested collecting the exit code,
- # stopping the iteration only if REPEATED_UTEST_STOP_ON_FAILURE is set.
- exit_code="$?"
- for i in $(seq -w 1 $count); do
-
- echo "Running test iteration $i of $count"
-
- # run the test
- status="passes"
- if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
- status="fails"
- exit_code=1
- fi
-
- # move the stdout output file
- dest=/tmp/results/repeated_utest/stdout/${status}/${i}
- mkdir -p $dest
- mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt
-
- # move the XML output files
- source=build/test/output
- dest=/tmp/results/repeated_utest/output/${status}/${i}
- mkdir -p $dest
- if [[ -d $source && -n "$(ls $source)" ]]; then
- mv $source/* $dest/
- fi
++ echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt
+
- # move the log files
- source=build/test/logs
- dest=/tmp/results/repeated_utest/logs/${status}/${i}
- mkdir -p $dest
- if [[ -d $source && -n "$(ls $source)" ]]; then
- mv $source/* $dest/
- fi
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
++ export CQLSH_PYTHON=/usr/bin/python2.7
++ fi
+
- # maybe stop iterations on test failure
- if [[ $REPEATED_UTEST_STOP_ON_FAILURE = true ]] && (( $exit_code > 0 )); then
- break
- fi
- done
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
+
- (exit ${exit_code})
- fi
- fi
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
- path: /tmp/results/repeated_utest/output
- - store_artifacts:
- path: /tmp/results/repeated_utest/stdout
- destination: stdout
++ path: /tmp/results
+ - store_artifacts:
- path: /tmp/results/repeated_utest/output
- destination: junitxml
++ path: /tmp/dtest
++ destination: dtest_j8_without_vnodes
+ - store_artifacts:
- path: /tmp/results/repeated_utest/logs
- destination: logs
++ path: ~/cassandra-dtest/logs
++ destination: dtest_j8_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - CASSANDRA_USE_JDK11: true
- j8_cqlsh-dtests-py3-with-vnodes:
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ j11_repeated_dtest:
+ docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
++ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
++ name: Log Environment Information
++ command: |
++ echo '*** id ***'
++ id
++ echo '*** cat /proc/cpuinfo ***'
++ cat /proc/cpuinfo
++ echo '*** free -m ***'
++ free -m
++ echo '*** df -m ***'
++ df -m
++ echo '*** ifconfig -a ***'
++ ifconfig -a
++ echo '*** uname -a ***'
++ uname -a
++ echo '*** mount ***'
++ mount
++ echo '*** env ***'
++ env
++ echo '*** java ***'
++ which java
++ java -version
++ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Determine Tests to Run (j8_with_vnodes)
- no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip- [...]
- - run:
- name: Run dtests (j8_with_vnodes)
++ name: Run repeated Python dtest
+ no_output_timeout: 15m
+ command: |
- echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt"
- cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt
++ if [ "${REPEATED_DTEST_NAME}" == "<nil>" ]; then
++ echo "Repeated dtest name hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_DTEST_COUNT}" == "<nil>" ]; then
++ echo "Repeated dtest count hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then
++ echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
++ else
+
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
- export CQLSH_PYTHON=/usr/bin/python3.6
- fi
++ # Calculate the number of test iterations to be run by the current parallel runner.
++ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
++ count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
++ if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
++ count=$((count+1))
++ fi
+
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
++ if (($count <= 0)); then
++ echo "No tests to run in this runner"
++ else
++ echo "Running ${REPEATED_DTEST_NAME} $count times"
+
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`
- set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
++
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++
++ stop_on_failure_arg=""
++ if ${REPEATED_DTEST_STOP_ON_FAILURE}; then
++ stop_on_failure_arg="-x"
++ fi
++
++ vnodes_args=""
++ if ${REPEATED_DTEST_VNODES}; then
++ vnodes_args="--use-vnodes --num-tokens=16"
++ fi
++
++ upgrade_arg=""
++ if false; then
++ upgrade_arg="--execute-upgrade-tests"
++ fi
++
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt
++ fi
++ fi
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
- destination: dtest_j8_with_vnodes
++ destination: dtest
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
- destination: dtest_j8_with_vnodes_logs
++ destination: dtest_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_cqlsh-dtests-py2-no-vnodes:
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - CASSANDRA_USE_JDK11: true
++ j8_repeated_dtest:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Determine Tests to Run (j8_without_vnodes)
- no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
- - run:
- name: Run dtests (j8_without_vnodes)
++ name: Run repeated Python dtest
+ no_output_timeout: 15m
+ command: |
- echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt"
- cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt
++ if [ "${REPEATED_DTEST_NAME}" == "<nil>" ]; then
++ echo "Repeated dtest name hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_DTEST_COUNT}" == "<nil>" ]; then
++ echo "Repeated dtest count hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then
++ echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
++ else
+
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
- export CQLSH_PYTHON=/usr/bin/python2.7
- fi
++ # Calculate the number of test iterations to be run by the current parallel runner.
++ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
++ count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
++ if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
++ count=$((count+1))
++ fi
+
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
++ if (($count <= 0)); then
++ echo "No tests to run in this runner"
++ else
++ echo "Running ${REPEATED_DTEST_NAME} $count times"
+
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`
- set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
++
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++
++ stop_on_failure_arg=""
++ if ${REPEATED_DTEST_STOP_ON_FAILURE}; then
++ stop_on_failure_arg="-x"
++ fi
++
++ vnodes_args=""
++ if ${REPEATED_DTEST_VNODES}; then
++ vnodes_args="--use-vnodes --num-tokens=16"
++ fi
++
++ upgrade_arg=""
++ if false; then
++ upgrade_arg="--execute-upgrade-tests"
++ fi
++
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt
++ fi
++ fi
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
- destination: dtest_j8_without_vnodes
++ destination: dtest
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
- destination: dtest_j8_without_vnodes_logs
++ destination: dtest_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_cqlsh-dtests-py2-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
+ - run:
+ name: Run dtests (j11_with_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
+ export CQLSH_PYTHON=/usr/bin/python2.7
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_dtests-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
+ - run:
+ name: Run dtests (j11_with_vnodes)
+ no_output_timeout: 15m
+ command: "echo \"cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the [...]
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j8_dtests-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j8_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
+ - run:
+ name: Run dtests (j8_without_vnodes)
+ no_output_timeout: 15m
+ command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not [...]
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j8_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j8_upgradetests-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j8_upgradetests_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_upgradetests_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --ex [...]
+ - run:
+ name: Run dtests (j8_upgradetests_without_vnodes)
+ no_output_timeout: 15m
+ command: "echo \"cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually [...]
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j8_upgradetests_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_upgradetests_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_repeated-dtest:
- docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
- resource_class: medium
- working_directory: ~/
- shell: /bin/bash -eo pipefail -l
- parallelism: 4
- steps:
- - attach_workspace:
- at: /home/cassandra
- - run:
- name: Clone Cassandra dtest Repository (via git)
- command: |
- git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
- - run:
- name: Configure virtualenv and python Dependencies
- command: |
- # note, this should be super quick as all dependencies should be pre-installed in the docker image
- # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
- # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
- # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
- pip3 uninstall -y cqlsh
- pip3 freeze
- - run:
- name: Run repeated dtest
- no_output_timeout: 15m
- command: |
- if [ "$REPEATED_DTEST_NAME" == "<nil>" ]; then
- echo "Repeated dtest name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_DTEST_COUNT" == "<nil>" ]; then
- echo "Repeated dtest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_DTEST_COUNT" -le 0 ]; then
- echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
- else
-
- # Calculate the number of test iterations to be run by the current parallel runner.
- # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_DTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_DTEST_COUNT % CIRCLE_NODE_TOTAL))); then
- count=$((count+1))
- fi
-
- if (($count <= 0)); then
- echo "No tests to run in this runner"
- else
- echo "Running $REPEATED_DTEST_NAME $count times"
-
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
-
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
-
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
-
- stop_on_failure_arg=""
- if $REPEATED_UTEST_STOP_ON_FAILURE; then
- stop_on_failure_arg="-x"
- fi
-
- vnodes_args=""
- if $REPEATED_DTEST_VNODES; then
- vnodes_args="--use-vnodes --num-tokens=16"
- fi
-
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $REPEATED_DTEST_NAME | tee /tmp/dtest/stdout.txt
- fi
- fi
- - store_test_results:
- path: /tmp/results
- - store_artifacts:
- path: /tmp/dtest
- destination: dtest
- - store_artifacts:
- path: ~/cassandra-dtest/logs
- destination: dtest_logs
- environment:
- - ANT_HOME: /usr/share/ant
- - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - LANG: en_US.UTF-8
- - KEEP_TEST_DIR: true
- - DEFAULT_DIR: /home/cassandra/cassandra-dtest
- - PYTHONIOENCODING: utf-8
- - PYTHONUNBUFFERED: true
- - CASS_DRIVER_NO_EXTENSIONS: true
- - CASS_DRIVER_NO_CYTHON: true
- - CASSANDRA_SKIP_SYNC: true
- - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
- - DTEST_BRANCH: trunk
- - CCM_MAX_HEAP_SIZE: 1024M
- - CCM_HEAP_NEWSIZE: 256M
- - REPEATED_UTEST_TARGET: testsome
- - REPEATED_UTEST_CLASS: null
- - REPEATED_UTEST_METHODS: null
- - REPEATED_UTEST_COUNT: 100
- - REPEATED_UTEST_STOP_ON_FAILURE: false
- - REPEATED_DTEST_NAME: null
- - REPEATED_DTEST_VNODES: false
- - REPEATED_DTEST_COUNT: 100
- - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ utests_stress:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 1
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Run Unit Tests (stress-test)
+ command: |
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ ant stress-test
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j8_unit_tests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Determine unit Tests to Run
+ command: |
+ # reminder: this code (along with all the steps) is independently executed on every circle container
+ # so the goal here is to get the circleci script to return the tests *this* container will run
+ # which we do via the `circleci` cli tool.
+
+ rm -fr ~/cassandra-dtest/upgrade_tests
+ echo "***java tests***"
+
+ # get all of our unit test filenames
+ set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt
+
+ # split up the unit tests into groups based on the number of containers we have
+ set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt
+ set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt"
+ cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ no_output_timeout: 15m
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run Unit Tests (testclasslist)
+ command: |
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true)
+ if [ -z "$test_timeout" ]; then
+ test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
+ fi
+ ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_jvm_dtests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 1
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Determine distributed Tests to Run
+ command: |
+ # reminder: this code (along with all the steps) is independently executed on every circle container
+ # so the goal here is to get the circleci script to return the tests *this* container will run
+ # which we do via the `circleci` cli tool.
+
+ rm -fr ~/cassandra-dtest/upgrade_tests
+ echo "***java tests***"
+
+ # get all of our unit test filenames
+ set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt
+
+ # split up the unit tests into groups based on the number of containers we have
+ set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt
+ set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt"
+ cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ no_output_timeout: 15m
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run Unit Tests (testclasslist)
+ command: |
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true)
+ if [ -z "$test_timeout" ]; then
+ test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
+ fi
+ ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_build:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 1
+ steps:
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Clone Cassandra Repository (via git)
+ command: |
+ git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH git://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra
+ - run:
+ name: Build Cassandra
+ command: |
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd ~/cassandra
+ # Loop to prevent failure due to maven-ant-tasks not downloading a jar..
+ for x in $(seq 1 3); do
+ ${ANT_HOME}/bin/ant clean realclean jar
+ RETURN="$?"
+ if [ "${RETURN}" -eq "0" ]; then
+ break
+ fi
+ done
+ # Exit, if we didn't build successfully
+ if [ "${RETURN}" -ne "0" ]; then
+ echo "Build failed with exit code: ${RETURN}"
+ exit ${RETURN}
+ fi
+ no_output_timeout: 15m
+ - run:
+ name: Run eclipse-warnings
+ command: |
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd ~/cassandra
+ ant eclipse-warnings
+ - persist_to_workspace:
+ root: /home/cassandra
+ paths:
+ - cassandra
+ - .m2
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_cqlsh-dtests-py2-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource- [...]
+ - run:
+ name: Run dtests (j11_without_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
+ export CQLSH_PYTHON=/usr/bin/python2.7
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_cqlshlib_tests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 1
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Run cqlshlib Unit Tests
+ command: |
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra/pylib
+ ./cassandra-cqlsh-tests.sh ..
+ no_output_timeout: 15m
+ - store_test_results:
- path: /tmp/cassandra/pylib
++ path: /tmp/cassandra/pylib
++ environment:
++ - ANT_HOME: /usr/share/ant
++ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ - LANG: en_US.UTF-8
++ - KEEP_TEST_DIR: true
++ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
++ - PYTHONIOENCODING: utf-8
++ - PYTHONUNBUFFERED: true
++ - CASS_DRIVER_NO_EXTENSIONS: true
++ - CASS_DRIVER_NO_CYTHON: true
++ - CASSANDRA_SKIP_SYNC: true
++ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
++ - DTEST_BRANCH: trunk
++ - CCM_MAX_HEAP_SIZE: 1024M
++ - CCM_HEAP_NEWSIZE: 256M
++ - REPEATED_UTEST_TARGET: testsome
++ - REPEATED_UTEST_CLASS: null
++ - REPEATED_UTEST_METHODS: null
++ - REPEATED_UTEST_COUNT: 100
++ - REPEATED_UTEST_STOP_ON_FAILURE: false
++ - REPEATED_DTEST_NAME: null
++ - REPEATED_DTEST_VNODES: false
++ - REPEATED_DTEST_COUNT: 100
++ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - CASSANDRA_USE_JDK11: true
++ j11_repeated_utest:
++ docker:
++ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
++ resource_class: medium
++ working_directory: ~/
++ shell: /bin/bash -eo pipefail -l
++ parallelism: 4
++ steps:
++ - attach_workspace:
++ at: /home/cassandra
++ - run:
++ name: Log Environment Information
++ command: |
++ echo '*** id ***'
++ id
++ echo '*** cat /proc/cpuinfo ***'
++ cat /proc/cpuinfo
++ echo '*** free -m ***'
++ free -m
++ echo '*** df -m ***'
++ df -m
++ echo '*** ifconfig -a ***'
++ ifconfig -a
++ echo '*** uname -a ***'
++ uname -a
++ echo '*** mount ***'
++ mount
++ echo '*** env ***'
++ env
++ echo '*** java ***'
++ which java
++ java -version
++ - run:
++ name: Run repeated JUnit test
++ no_output_timeout: 15m
++ command: |
++ if [ "${REPEATED_UTEST_CLASS}" == "<nil>" ]; then
++ echo "Repeated utest class name hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_UTEST_COUNT}" == "<nil>" ]; then
++ echo "Repeated utest count hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then
++ echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
++ else
++
++ # Calculate the number of test iterations to be run by the current parallel runner.
++ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
++ count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL))
++ if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
++ count=$((count+1))
++ fi
++
++ if (($count <= 0)); then
++ echo "No tests to run in this runner"
++ else
++ echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times"
++
++ set -x
++ export PATH=$JAVA_HOME/bin:$PATH
++ time mv ~/cassandra /tmp
++ cd /tmp/cassandra
++ if [ -d ~/dtest_jars ]; then
++ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
++ fi
++
++ target=${REPEATED_UTEST_TARGET}
++ class_path=${REPEATED_UTEST_CLASS}
++ class_name="${class_path##*.}"
++
++ # Prepare the -Dtest.name argument.
++ # It can be the fully qualified class name or the short class name, depending on the target.
++ if [[ $target == "test" || \
++ $target == "test-cdc" || \
++ $target == "test-compression" || \
++ $target == "test-system-keyspace-directory" ]]; then
++ name="-Dtest.name=$class_name"
++ else
++ name="-Dtest.name=$class_path"
++ fi
++
++ # Prepare the -Dtest.methods argument, which is optional
++ if [ "${REPEATED_UTEST_METHODS}" == "<nil>" ]; then
++ methods=""
++ else
++ methods="-Dtest.methods=${REPEATED_UTEST_METHODS}"
++ fi
++
++ # Run the test target as many times as requested collecting the exit code,
++ # stopping the iteration only if stop_on_failure is set.
++ exit_code="$?"
++ for i in $(seq -w 1 $count); do
++
++ echo "Running test iteration $i of $count"
++
++ # run the test
++ status="passes"
++ if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
++ status="fails"
++ exit_code=1
++ fi
++
++ # move the stdout output file
++ dest=/tmp/results/repeated_utest/stdout/${status}/${i}
++ mkdir -p $dest
++ mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt
++
++ # move the XML output files
++ source=build/test/output
++ dest=/tmp/results/repeated_utest/output/${status}/${i}
++ mkdir -p $dest
++ if [[ -d $source && -n "$(ls $source)" ]]; then
++ mv $source/* $dest/
++ fi
++
++ # move the log files
++ source=build/test/logs
++ dest=/tmp/results/repeated_utest/logs/${status}/${i}
++ mkdir -p $dest
++ if [[ -d $source && -n "$(ls $source)" ]]; then
++ mv $source/* $dest/
++ fi
++
++ # maybe stop iterations on test failure
++ if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then
++ break
++ fi
++ done
++
++ (exit ${exit_code})
++ fi
++ fi
++ - store_test_results:
++ path: /tmp/results/repeated_utest/output
++ - store_artifacts:
++ path: /tmp/results/repeated_utest/stdout
++ destination: stdout
++ - store_artifacts:
++ path: /tmp/results/repeated_utest/output
++ destination: junitxml
++ - store_artifacts:
++ path: /tmp/results/repeated_utest/logs
++ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j8_dtests-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j8_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip- [...]
+ - run:
+ name: Run dtests (j8_with_vnodes)
+ no_output_timeout: 15m
+ command: "echo \"cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the e [...]
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j8_with_vnodes
- store_artifacts:
path: ~/cassandra-dtest/logs
- destination: dtest_logs
+ destination: dtest_j8_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_cqlsh-dtests-py38-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.8/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource- [...]
+ - run:
+ name: Run dtests (j11_without_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt
+
+ source ~/env3.8/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then
+ export CQLSH_PYTHON=/usr/bin/python3.8
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j8_jvm_dtests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 1
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Determine distributed Tests to Run
+ command: |
+ # reminder: this code (along with all the steps) is independently executed on every circle container
+ # so the goal here is to get the circleci script to return the tests *this* container will run
+ # which we do via the `circleci` cli tool.
+
+ rm -fr ~/cassandra-dtest/upgrade_tests
+ echo "***java tests***"
+
+ # get all of our unit test filenames
+ set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt
+
+ # split up the unit tests into groups based on the number of containers we have
+ set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt
+ set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt"
+ cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ no_output_timeout: 15m
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run Unit Tests (testclasslist)
+ command: |
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true)
+ if [ -z "$test_timeout" ]; then
+ test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
+ fi
+ ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
environment:
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- LANG: en_US.UTF-8
- KEEP_TEST_DIR: true
- DEFAULT_DIR: /home/cassandra/cassandra-dtest
@@@ -2321,9 -531,16 +2796,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- build:
+ j8_build:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: medium
@@@ -2400,253 -616,71 +2882,92 @@@
- CASS_DRIVER_NO_CYTHON: true
- CASSANDRA_SKIP_SYNC: true
- DTEST_REPO: git://github.com/apache/cassandra-dtest.git
- - DTEST_BRANCH: trunk
- - CCM_MAX_HEAP_SIZE: 1024M
- - CCM_HEAP_NEWSIZE: 256M
- - REPEATED_UTEST_TARGET: testsome
- - REPEATED_UTEST_CLASS: null
- - REPEATED_UTEST_METHODS: null
- - REPEATED_UTEST_COUNT: 100
- - REPEATED_UTEST_STOP_ON_FAILURE: false
- - REPEATED_DTEST_NAME: null
- - REPEATED_DTEST_VNODES: false
- - REPEATED_DTEST_COUNT: 100
- - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_cqlsh-dtests-py3-no-vnodes:
- docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
- resource_class: medium
- working_directory: ~/
- shell: /bin/bash -eo pipefail -l
- parallelism: 4
- steps:
- - attach_workspace:
- at: /home/cassandra
- - run:
- name: Clone Cassandra dtest Repository (via git)
- command: |
- git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
- - run:
- name: Configure virtualenv and python Dependencies
- command: |
- # note, this should be super quick as all dependencies should be pre-installed in the docker image
- # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
- # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
- # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
- pip3 uninstall -y cqlsh
- pip3 freeze
- - run:
- name: Determine Tests to Run (j8_without_vnodes)
- no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
- - run:
- name: Run dtests (j8_without_vnodes)
- no_output_timeout: 15m
- command: |
- echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt"
- cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt
-
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
- export CQLSH_PYTHON=/usr/bin/python3.6
- fi
-
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
-
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`
- set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
- - store_test_results:
- path: /tmp/results
- - store_artifacts:
- path: /tmp/dtest
- destination: dtest_j8_without_vnodes
- - store_artifacts:
- path: ~/cassandra-dtest/logs
- destination: dtest_j8_without_vnodes_logs
- environment:
- - ANT_HOME: /usr/share/ant
- - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - LANG: en_US.UTF-8
- - KEEP_TEST_DIR: true
- - DEFAULT_DIR: /home/cassandra/cassandra-dtest
- - PYTHONIOENCODING: utf-8
- - PYTHONUNBUFFERED: true
- - CASS_DRIVER_NO_EXTENSIONS: true
- - CASS_DRIVER_NO_CYTHON: true
- - CASSANDRA_SKIP_SYNC: true
- - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
- - DTEST_BRANCH: trunk
- - CCM_MAX_HEAP_SIZE: 1024M
- - CCM_HEAP_NEWSIZE: 256M
- - REPEATED_UTEST_TARGET: testsome
- - REPEATED_UTEST_CLASS: null
- - REPEATED_UTEST_METHODS: null
- - REPEATED_UTEST_COUNT: 100
- - REPEATED_UTEST_STOP_ON_FAILURE: false
- - REPEATED_DTEST_NAME: null
- - REPEATED_DTEST_VNODES: false
- - REPEATED_DTEST_COUNT: 100
- - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_repeated-utest:
- docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
- resource_class: medium
- working_directory: ~/
- shell: /bin/bash -eo pipefail -l
- parallelism: 4
- steps:
- - attach_workspace:
- at: /home/cassandra
- - run:
- name: Log Environment Information
- command: |
- echo '*** id ***'
- id
- echo '*** cat /proc/cpuinfo ***'
- cat /proc/cpuinfo
- echo '*** free -m ***'
- free -m
- echo '*** df -m ***'
- df -m
- echo '*** ifconfig -a ***'
- ifconfig -a
- echo '*** uname -a ***'
- uname -a
- echo '*** mount ***'
- mount
- echo '*** env ***'
- env
- echo '*** java ***'
- which java
- java -version
- - run:
- name: Run repeated utest
- no_output_timeout: 15m
- command: |
- if [ "$REPEATED_UTEST_CLASS" == "<nil>" ]; then
- echo "Repeated utest class name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" == "<nil>" ]; then
- echo "Repeated utest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" -le 0 ]; then
- echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
- else
-
- # Calculate the number of test iterations to be run by the current parallel runner.
- # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_UTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_UTEST_COUNT % CIRCLE_NODE_TOTAL))); then
- count=$((count+1))
- fi
-
- if (($count <= 0)); then
- echo "No tests to run in this runner"
- else
- echo "Running $REPEATED_UTEST_TARGET $REPEATED_UTEST_CLASS $REPEATED_UTEST_METHODS $count times"
-
- set -x
- export PATH=$JAVA_HOME/bin:$PATH
- time mv ~/cassandra /tmp
- cd /tmp/cassandra
- if [ -d ~/dtest_jars ]; then
- cp ~/dtest_jars/dtest* /tmp/cassandra/build/
- fi
-
- target=$REPEATED_UTEST_TARGET
- class_path=$REPEATED_UTEST_CLASS
- class_name="${class_path##*.}"
-
- # Prepare the -Dtest.name argument.
- # It can be the fully qualified class name or the short class name, depending on the target.
- if [[ $target == "test" || \
- $target == "test-cdc" || \
- $target == "test-compression" || \
- $target == "test-system-keyspace-directory" ]]; then
- name="-Dtest.name=$class_name"
- else
- name="-Dtest.name=$class_path"
- fi
-
- # Prepare the -Dtest.methods argument, which is optional
- if [ "$REPEATED_UTEST_METHODS" == "<nil>" ]; then
- methods=""
- else
- methods="-Dtest.methods=$REPEATED_UTEST_METHODS"
- fi
-
- # Run the test target as many times as requested collecting the exit code,
- # stopping the iteration only if REPEATED_UTEST_STOP_ON_FAILURE is set.
- exit_code="$?"
- for i in $(seq -w 1 $count); do
-
- echo "Running test iteration $i of $count"
-
- # run the test
- status="passes"
- if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
- status="fails"
- exit_code=1
- fi
-
- # move the stdout output file
- dest=/tmp/results/repeated_utest/stdout/${status}/${i}
- mkdir -p $dest
- mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt
-
- # move the XML output files
- source=build/test/output
- dest=/tmp/results/repeated_utest/output/${status}/${i}
- mkdir -p $dest
- if [[ -d $source && -n "$(ls $source)" ]]; then
- mv $source/* $dest/
- fi
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_dtests-no-vnodes:
++ j8_cqlsh-dtests-py3-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
- pip3 install --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j8_without_vnodes)
+ no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
+ - run:
+ name: Run dtests (j8_without_vnodes)
+ no_output_timeout: 15m
- command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not [...]
++ command: |
++ echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt
+
- # move the log files
- source=build/test/logs
- dest=/tmp/results/repeated_utest/logs/${status}/${i}
- mkdir -p $dest
- if [[ -d $source && -n "$(ls $source)" ]]; then
- mv $source/* $dest/
- fi
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
++ export CQLSH_PYTHON=/usr/bin/python3.6
++ fi
+
- # maybe stop iterations on test failure
- if [[ $REPEATED_UTEST_STOP_ON_FAILURE = true ]] && (( $exit_code > 0 )); then
- break
- fi
- done
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
+
- (exit ${exit_code})
- fi
- fi
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
- store_test_results:
- path: /tmp/results/repeated_utest/output
- - store_artifacts:
- path: /tmp/results/repeated_utest/stdout
- destination: stdout
+ path: /tmp/results
- store_artifacts:
- path: /tmp/results/repeated_utest/output
- destination: junitxml
+ path: /tmp/dtest
+ destination: dtest_j8_without_vnodes
- store_artifacts:
- path: /tmp/results/repeated_utest/logs
- destination: logs
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_without_vnodes_logs
environment:
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- LANG: en_US.UTF-8
- KEEP_TEST_DIR: true
- DEFAULT_DIR: /home/cassandra/cassandra-dtest
@@@ -2668,9 -702,16 +2989,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_upgradetests-no-vnodes:
+ j8_cqlsh-dtests-py38-with-vnodes:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: medium
@@@ -2756,9 -817,16 +3084,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- utests_stress:
+ utests_long:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: medium
@@@ -2812,9 -879,16 +3147,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_unit_tests:
+ utests_system_keyspace_directory:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: medium
@@@ -2914,9 -987,16 +3256,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_dtests-with-vnodes:
+ j8_cqlshlib_tests:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: medium
@@@ -2961,9 -1061,16 +3310,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_jvm_dtests:
+ utests_fqltool:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: medium
@@@ -3017,11 -1169,18 +3373,18 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- utests_long:
+ j11_dtests-no-vnodes:
docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
resource_class: medium
working_directory: ~/
shell: /bin/bash -eo pipefail -l
@@@ -3108,9 -1231,15 +3471,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
utests_compression:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
@@@ -3211,6 -1339,187 +3581,188 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j8_repeated_utest:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run repeated JUnit test
+ no_output_timeout: 15m
+ command: |
+ if [ "${REPEATED_UTEST_CLASS}" == "<nil>" ]; then
+ echo "Repeated utest class name hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_UTEST_COUNT}" == "<nil>" ]; then
+ echo "Repeated utest count hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then
+ echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
+ else
+
+ # Calculate the number of test iterations to be run by the current parallel runner.
+ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
+ count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL))
+ if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
+ count=$((count+1))
+ fi
+
+ if (($count <= 0)); then
+ echo "No tests to run in this runner"
+ else
+ echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times"
+
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+
+ target=${REPEATED_UTEST_TARGET}
+ class_path=${REPEATED_UTEST_CLASS}
+ class_name="${class_path##*.}"
+
+ # Prepare the -Dtest.name argument.
+ # It can be the fully qualified class name or the short class name, depending on the target.
+ if [[ $target == "test" || \
+ $target == "test-cdc" || \
+ $target == "test-compression" || \
+ $target == "test-system-keyspace-directory" ]]; then
+ name="-Dtest.name=$class_name"
+ else
+ name="-Dtest.name=$class_path"
+ fi
+
+ # Prepare the -Dtest.methods argument, which is optional
+ if [ "${REPEATED_UTEST_METHODS}" == "<nil>" ]; then
+ methods=""
+ else
+ methods="-Dtest.methods=${REPEATED_UTEST_METHODS}"
+ fi
+
+ # Run the test target as many times as requested collecting the exit code,
+ # stopping the iteration only if stop_on_failure is set.
+ exit_code="$?"
+ for i in $(seq -w 1 $count); do
+
+ echo "Running test iteration $i of $count"
+
+ # run the test
+ status="passes"
+ if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
+ status="fails"
+ exit_code=1
+ fi
+
+ # move the stdout output file
+ dest=/tmp/results/repeated_utest/stdout/${status}/${i}
+ mkdir -p $dest
+ mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt
+
+ # move the XML output files
+ source=build/test/output
+ dest=/tmp/results/repeated_utest/output/${status}/${i}
+ mkdir -p $dest
+ if [[ -d $source && -n "$(ls $source)" ]]; then
+ mv $source/* $dest/
+ fi
+
+ # move the log files
+ source=build/test/logs
+ dest=/tmp/results/repeated_utest/logs/${status}/${i}
+ mkdir -p $dest
+ if [[ -d $source && -n "$(ls $source)" ]]; then
+ mv $source/* $dest/
+ fi
+
+ # maybe stop iterations on test failure
+ if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then
+ break
+ fi
+ done
+
+ (exit ${exit_code})
+ fi
+ fi
+ - store_test_results:
+ path: /tmp/results/repeated_utest/output
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/stdout
+ destination: stdout
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/logs
+ destination: logs
+ environment:
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - ANT_HOME: /usr/share/ant
++ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
j8_dtest_jars_build:
@@@ -3338,27 -1646,19 +3897,30 @@@ workflows
- utests_stress:
requires:
- start_utests_stress
+ - j8_build
+ - start_utests_fqltool:
+ type: approval
+ - utests_fqltool:
+ requires:
+ - start_utests_fqltool
+ - j8_build
+ - start_utests_system_keyspace_directory:
+ type: approval
+ - utests_system_keyspace_directory:
+ requires:
+ - start_utests_system_keyspace_directory
+ - j8_build
- - start_jvm_upgrade_dtest:
+ - start_j8_dtest_jars_build:
type: approval
- j8_dtest_jars_build:
requires:
- - build
+ - j8_build
- - start_jvm_upgrade_dtest
+ - start_j8_dtest_jars_build
+ - start_jvm_upgrade_dtest:
+ type: approval
- j8_jvm_upgrade_dtests:
requires:
+ - start_jvm_upgrade_dtest
- j8_dtest_jars_build
- start_j8_dtests:
type: approval
@@@ -3385,146 -1677,27 +3947,158 @@@
- j8_upgradetests-no-vnodes:
requires:
- start_upgrade_tests
+ - j8_build
+ - start_j8_cqlsh_tests-with-vnodes:
+ type: approval
+ - j8_cqlsh-dtests-py2-with-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-with-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py3-with-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-with-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py38-with-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-with-vnodes
+ - j8_build
+ - start_j8_cqlsh_tests-no-vnodes:
+ type: approval
+ - j8_cqlsh-dtests-py2-no-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-no-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py3-no-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-no-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py38-no-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-no-vnodes
+ - j8_build
+ - start_j11_cqlsh_tests-with-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py3-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py38-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j8_build
+ - start_j11_cqlsh_tests-no-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py3-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py38-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j8_build
- - start_j8_repeated-utest:
+ - start_j8_repeated_utest:
+ type: approval
+ - j8_repeated_utest:
+ requires:
+ - start_j8_repeated_utest
- - build
++ - j8_build
++ - start_j11_repeated_utest:
+ type: approval
- - j8_repeated-utest:
++ - j11_repeated_utest:
+ requires:
- - start_j8_repeated-utest
++ - start_j11_repeated_utest
+ - j8_build
- - start_j11_repeated-utest:
+ - start_j8_repeated_dtest:
type: approval
- - j11_repeated-utest:
+ - j8_repeated_dtest:
requires:
- - start_j11_repeated-utest
+ - start_j8_repeated_dtest
- - build
+ - j8_build
- - start_j8_repeated-dtest:
++ - start_j11_repeated_dtest:
+ type: approval
- - j8_repeated-dtest:
++ - j11_repeated_dtest:
+ requires:
- - start_j8_repeated-dtest
++ - start_j11_repeated_dtest
+ - j8_build
- - start_j11_repeated-dtest:
+ - start_repeated_upgrade_dtest:
type: approval
- - j11_repeated-dtest:
+ - repeated_upgrade_dtest:
requires:
- - start_j11_repeated-dtest
+ - start_repeated_upgrade_dtest
- - build
+ - j8_build
+ - start_repeated_jvm_upgrade_dtest:
+ type: approval
+ - repeated_jvm_upgrade_dtest:
+ requires:
+ - start_repeated_jvm_upgrade_dtest
+ - j8_dtest_jars_build
+ java11_build_and_run_tests:
+ jobs:
+ - j11_build
+ - j11_unit_tests:
+ requires:
+ - j11_build
+ - j11_jvm_dtests:
+ requires:
+ - j11_build
+ - j11_cqlshlib_tests:
+ requires:
+ - j11_build
+ - start_j11_dtests:
+ type: approval
+ - j11_dtests-with-vnodes:
+ requires:
+ - start_j11_dtests
+ - j11_build
+ - j11_dtests-no-vnodes:
+ requires:
+ - start_j11_dtests
+ - j11_build
+ - start_j11_cqlsh_tests-with-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py3-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py38-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j11_build
+ - start_j11_cqlsh_tests-no-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py3-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py38-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j11_build
- - start_j11_repeated-utest:
++ - start_j11_repeated_utest:
+ type: approval
- - j11_repeated-utest:
++ - j11_repeated_utest:
+ requires:
- - start_j11_repeated-utest
++ - start_j11_repeated_utest
+ - j11_build
- - start_j11_repeated-dtest:
++ - start_j11_repeated_dtest:
+ type: approval
- - j11_repeated-dtest:
++ - j11_repeated_dtest:
+ requires:
- - start_j11_repeated-dtest
++ - start_j11_repeated_dtest
+ - j11_build
diff --cc .circleci/config.yml.HIGHRES
index ecab98d,19e9ff8..cd8b5e0
--- a/.circleci/config.yml.HIGHRES
+++ b/.circleci/config.yml.HIGHRES
@@@ -118,9 -117,190 +118,191 @@@ jobs
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ repeated_jvm_upgrade_dtest:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run repeated JUnit test
+ no_output_timeout: 15m
+ command: |
+ if [ "${REPEATED_JVM_UPGRADE_DTEST_CLASS}" == "<nil>" ]; then
+ echo "Repeated utest class name hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" == "<nil>" ]; then
+ echo "Repeated utest count hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" -le 0 ]; then
+ echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
+ else
+
+ # Calculate the number of test iterations to be run by the current parallel runner.
+ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
+ count=$((${REPEATED_JVM_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
+ if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
+ count=$((count+1))
+ fi
+
+ if (($count <= 0)); then
+ echo "No tests to run in this runner"
+ else
+ echo "Running test-jvm-dtest-some ${REPEATED_JVM_UPGRADE_DTEST_CLASS} ${REPEATED_JVM_UPGRADE_DTEST_METHODS} ${REPEATED_JVM_UPGRADE_DTEST_COUNT} times"
+
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+
+ target=test-jvm-dtest-some
+ class_path=${REPEATED_JVM_UPGRADE_DTEST_CLASS}
+ class_name="${class_path##*.}"
+
+ # Prepare the -Dtest.name argument.
+ # It can be the fully qualified class name or the short class name, depending on the target.
+ if [[ $target == "test" || \
+ $target == "test-cdc" || \
+ $target == "test-compression" || \
+ $target == "test-system-keyspace-directory" ]]; then
+ name="-Dtest.name=$class_name"
+ else
+ name="-Dtest.name=$class_path"
+ fi
+
+ # Prepare the -Dtest.methods argument, which is optional
+ if [ "${REPEATED_JVM_UPGRADE_DTEST_METHODS}" == "<nil>" ]; then
+ methods=""
+ else
+ methods="-Dtest.methods=${REPEATED_JVM_UPGRADE_DTEST_METHODS}"
+ fi
+
+ # Run the test target as many times as requested collecting the exit code,
+ # stopping the iteration only if stop_on_failure is set.
+ exit_code="$?"
+ for i in $(seq -w 1 $count); do
+
+ echo "Running test iteration $i of $count"
+
+ # run the test
+ status="passes"
+ if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
+ status="fails"
+ exit_code=1
+ fi
+
+ # move the stdout output file
+ dest=/tmp/results/repeated_utest/stdout/${status}/${i}
+ mkdir -p $dest
+ mv stdout.txt $dest/test-jvm-dtest-some-${REPEATED_JVM_UPGRADE_DTEST_CLASS}.txt
+
+ # move the XML output files
+ source=build/test/output
+ dest=/tmp/results/repeated_utest/output/${status}/${i}
+ mkdir -p $dest
+ if [[ -d $source && -n "$(ls $source)" ]]; then
+ mv $source/* $dest/
+ fi
+
+ # move the log files
+ source=build/test/logs
+ dest=/tmp/results/repeated_utest/logs/${status}/${i}
+ mkdir -p $dest
+ if [[ -d $source && -n "$(ls $source)" ]]; then
+ mv $source/* $dest/
+ fi
+
+ # maybe stop iterations on test failure
+ if [[ ${REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then
+ break
+ fi
+ done
+
+ (exit ${exit_code})
+ fi
+ fi
+ - store_test_results:
+ path: /tmp/results/repeated_utest/output
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/stdout
+ destination: stdout
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/logs
+ destination: logs
+ environment:
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - ANT_HOME: /usr/share/ant
++ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
- - CCM_MAX_HEAP_SIZE: 2048M
- - CCM_HEAP_NEWSIZE: 512M
++ - CCM_MAX_HEAP_SIZE: 1024M
++ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- repeated_upgrade_dtest:
+ j8_cqlsh-dtests-py2-with-vnodes:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: xlarge
@@@ -206,2176 -411,188 +388,2476 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_repeated_dtest:
+ j11_unit_tests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Determine unit Tests to Run
+ command: |
+ # reminder: this code (along with all the steps) is independently executed on every circle container
+ # so the goal here is to get the circleci script to return the tests *this* container will run
+ # which we do via the `circleci` cli tool.
+
+ rm -fr ~/cassandra-dtest/upgrade_tests
+ echo "***java tests***"
+
+ # get all of our unit test filenames
+ set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt
+
+ # split up the unit tests into groups based on the number of containers we have
+ set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt
+ set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt"
+ cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ no_output_timeout: 15m
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run Unit Tests (testclasslist)
+ command: |
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true)
+ if [ -z "$test_timeout" ]; then
+ test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
+ fi
+ ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
++ repeated_upgrade_dtest:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
- pip3 install --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Run repeated Python dtest
+ no_output_timeout: 15m
+ command: |
- if [ "${REPEATED_DTEST_NAME}" == "<nil>" ]; then
++ if [ "${REPEATED_UPGRADE_DTEST_NAME}" == "<nil>" ]; then
+ echo "Repeated dtest name hasn't been defined, exiting without running any test"
- elif [ "${REPEATED_DTEST_COUNT}" == "<nil>" ]; then
++ elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" == "<nil>" ]; then
+ echo "Repeated dtest count hasn't been defined, exiting without running any test"
- elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then
++ elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" -le 0 ]; then
+ echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
+ else
+
+ # Calculate the number of test iterations to be run by the current parallel runner.
+ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
++ count=$((${REPEATED_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
++ if (($CIRCLE_NODE_INDEX < (${REPEATED_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
+ count=$((count+1))
+ fi
+
+ if (($count <= 0)); then
+ echo "No tests to run in this runner"
+ else
- echo "Running ${REPEATED_DTEST_NAME} $count times"
++ echo "Running ${REPEATED_UPGRADE_DTEST_NAME} $count times"
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+
+ stop_on_failure_arg=""
- if ${REPEATED_DTEST_STOP_ON_FAILURE}; then
++ if ${REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE}; then
+ stop_on_failure_arg="-x"
+ fi
+
+ vnodes_args=""
- if ${REPEATED_DTEST_VNODES}; then
++ if false; then
+ vnodes_args="--use-vnodes --num-tokens=16"
+ fi
+
+ upgrade_arg=""
- if false; then
++ if true; then
+ upgrade_arg="--execute-upgrade-tests"
+ fi
+
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt
++ set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_UPGRADE_DTEST_NAME} | tee /tmp/dtest/stdout.txt
+ fi
+ fi
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_logs
+ environment:
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - ANT_HOME: /usr/share/ant
++ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
- - CCM_MAX_HEAP_SIZE: 2048M
- - CCM_HEAP_NEWSIZE: 512M
++ - CCM_MAX_HEAP_SIZE: 1024M
++ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- build:
+ j8_cqlsh-dtests-py38-no-vnodes:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
- resource_class: medium
+ resource_class: xlarge
working_directory: ~/
shell: /bin/bash -eo pipefail -l
- parallelism: 1
+ parallelism: 100
steps:
+ - attach_workspace:
+ at: /home/cassandra
- run:
- name: Log Environment Information
+ name: Clone Cassandra dtest Repository (via git)
command: |
- echo '*** id ***'
- id
- echo '*** cat /proc/cpuinfo ***'
- cat /proc/cpuinfo
- echo '*** free -m ***'
- free -m
- echo '*** df -m ***'
- df -m
- echo '*** ifconfig -a ***'
- ifconfig -a
- echo '*** uname -a ***'
- uname -a
- echo '*** mount ***'
- mount
- echo '*** env ***'
- env
- echo '*** java ***'
- which java
- java -version
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
- run:
- name: Clone Cassandra Repository (via git)
+ name: Configure virtualenv and python Dependencies
command: |
- git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH git://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.8/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
- run:
- name: Build Cassandra
+ name: Determine Tests to Run (j8_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
+ - run:
+ name: Run dtests (j8_without_vnodes)
+ no_output_timeout: 15m
command: |
+ echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt
+
+ source ~/env3.8/bin/activate
export PATH=$JAVA_HOME/bin:$PATH
- cd ~/cassandra
- # Loop to prevent failure due to maven-ant-tasks not downloading a jar..
- for x in $(seq 1 3); do
- ${ANT_HOME}/bin/ant clean jar
- RETURN="$?"
- if [ "${RETURN}" -eq "0" ]; then
- break
- fi
- done
- # Exit, if we didn't build successfully
- if [ "${RETURN}" -ne "0" ]; then
- echo "Build failed with exit code: ${RETURN}"
- exit ${RETURN}
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then
+ export CQLSH_PYTHON=/usr/bin/python3.8
fi
- no_output_timeout: 15m
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j8_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_cqlsh-dtests-py3-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
+ - run:
+ name: Run dtests (j11_with_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
+ export CQLSH_PYTHON=/usr/bin/python3.6
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_cqlsh-dtests-py3-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource- [...]
+ - run:
+ name: Run dtests (j11_without_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
+ export CQLSH_PYTHON=/usr/bin/python3.6
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
- j11_repeated-dtest:
++ j11_cqlsh-dtests-py38-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
- name: Log Environment Information
- command: |
- echo '*** id ***'
- id
- echo '*** cat /proc/cpuinfo ***'
- cat /proc/cpuinfo
- echo '*** free -m ***'
- free -m
- echo '*** df -m ***'
- df -m
- echo '*** ifconfig -a ***'
- ifconfig -a
- echo '*** uname -a ***'
- uname -a
- echo '*** mount ***'
- mount
- echo '*** env ***'
- env
- echo '*** java ***'
- which java
- java -version
- - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
- source ~/env3.6/bin/activate
++ source ~/env3.8/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Run repeated dtest
++ name: Determine Tests to Run (j11_with_vnodes)
++ no_output_timeout: 5m
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
++ - run:
++ name: Run dtests (j11_with_vnodes)
+ no_output_timeout: 15m
+ command: |
- if [ "$REPEATED_DTEST_NAME" == "<nil>" ]; then
- echo "Repeated dtest name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_DTEST_COUNT" == "<nil>" ]; then
- echo "Repeated dtest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_DTEST_COUNT" -le 0 ]; then
- echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
- else
-
- # Calculate the number of test iterations to be run by the current parallel runner.
- # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_DTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_DTEST_COUNT % CIRCLE_NODE_TOTAL))); then
- count=$((count+1))
- fi
-
- if (($count <= 0)); then
- echo "No tests to run in this runner"
- else
- echo "Running $REPEATED_DTEST_NAME $count times"
-
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
-
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
-
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
++ echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
+
- stop_on_failure_arg=""
- if $REPEATED_UTEST_STOP_ON_FAILURE; then
- stop_on_failure_arg="-x"
- fi
++ source ~/env3.8/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then
++ export CQLSH_PYTHON=/usr/bin/python3.8
++ fi
+
- vnodes_args=""
- if $REPEATED_DTEST_VNODES; then
- vnodes_args="--use-vnodes --num-tokens=16"
- fi
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
+
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $REPEATED_DTEST_NAME | tee /tmp/dtest/stdout.txt
- fi
- fi
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
- destination: dtest
++ destination: dtest_j11_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
- destination: dtest_logs
++ destination: dtest_j11_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
- j11_cqlsh-dtests-py38-with-vnodes:
++ j8_cqlsh-dtests-py3-with-vnodes:
+ docker:
- - image: apache/cassandra-testing-ubuntu2004-java11:20210304
++ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
- source ~/env3.8/bin/activate
++ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Determine Tests to Run (j11_with_vnodes)
++ name: Determine Tests to Run (j8_with_vnodes)
+ no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip- [...]
+ - run:
- name: Run dtests (j11_with_vnodes)
++ name: Run dtests (j8_with_vnodes)
+ no_output_timeout: 15m
+ command: |
- echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
- cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
++ echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt
+
- source ~/env3.8/bin/activate
++ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
- if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then
- export CQLSH_PYTHON=/usr/bin/python3.8
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
++ export CQLSH_PYTHON=/usr/bin/python3.6
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
- set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
- destination: dtest_j11_with_vnodes
++ destination: dtest_j8_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
- destination: dtest_j11_with_vnodes_logs
++ destination: dtest_j8_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - CASSANDRA_USE_JDK11: true
- j11_repeated-utest:
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ j8_cqlsh-dtests-py2-no-vnodes:
+ docker:
- - image: apache/cassandra-testing-ubuntu2004-java11:20210304
++ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
- name: Log Environment Information
++ name: Clone Cassandra dtest Repository (via git)
+ command: |
- echo '*** id ***'
- id
- echo '*** cat /proc/cpuinfo ***'
- cat /proc/cpuinfo
- echo '*** free -m ***'
- free -m
- echo '*** df -m ***'
- df -m
- echo '*** ifconfig -a ***'
- ifconfig -a
- echo '*** uname -a ***'
- uname -a
- echo '*** mount ***'
- mount
- echo '*** env ***'
- env
- echo '*** java ***'
- which java
- java -version
++ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
- name: Run repeated utest
++ name: Configure virtualenv and python Dependencies
++ command: |
++ # note, this should be super quick as all dependencies should be pre-installed in the docker image
++ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
++ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
++ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 uninstall -y cqlsh
++ pip3 freeze
++ - run:
++ name: Determine Tests to Run (j8_without_vnodes)
++ no_output_timeout: 5m
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
++ - run:
++ name: Run dtests (j8_without_vnodes)
+ no_output_timeout: 15m
+ command: |
- if [ "$REPEATED_UTEST_CLASS" == "<nil>" ]; then
- echo "Repeated utest class name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" == "<nil>" ]; then
- echo "Repeated utest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" -le 0 ]; then
- echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
- else
-
- # Calculate the number of test iterations to be run by the current parallel runner.
- # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_UTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_UTEST_COUNT % CIRCLE_NODE_TOTAL))); then
- count=$((count+1))
- fi
-
- if (($count <= 0)); then
- echo "No tests to run in this runner"
- else
- echo "Running $REPEATED_UTEST_TARGET $REPEATED_UTEST_CLASS $REPEATED_UTEST_METHODS $count times"
-
- set -x
- export PATH=$JAVA_HOME/bin:$PATH
- time mv ~/cassandra /tmp
- cd /tmp/cassandra
- if [ -d ~/dtest_jars ]; then
- cp ~/dtest_jars/dtest* /tmp/cassandra/build/
- fi
-
- target=$REPEATED_UTEST_TARGET
- class_path=$REPEATED_UTEST_CLASS
- class_name="${class_path##*.}"
-
- # Prepare the -Dtest.name argument.
- # It can be the fully qualified class name or the short class name, depending on the target.
- if [[ $target == "test" || \
- $target == "test-cdc" || \
- $target == "test-compression" || \
- $target == "test-system-keyspace-directory" ]]; then
- name="-Dtest.name=$class_name"
- else
- name="-Dtest.name=$class_path"
- fi
-
- # Prepare the -Dtest.methods argument, which is optional
- if [ "$REPEATED_UTEST_METHODS" == "<nil>" ]; then
- methods=""
- else
- methods="-Dtest.methods=$REPEATED_UTEST_METHODS"
- fi
-
- # Run the test target as many times as requested collecting the exit code,
- # stopping the iteration only if REPEATED_UTEST_STOP_ON_FAILURE is set.
- exit_code="$?"
- for i in $(seq -w 1 $count); do
-
- echo "Running test iteration $i of $count"
-
- # run the test
- status="passes"
- if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
- status="fails"
- exit_code=1
- fi
-
- # move the stdout output file
- dest=/tmp/results/repeated_utest/stdout/${status}/${i}
- mkdir -p $dest
- mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt
-
- # move the XML output files
- source=build/test/output
- dest=/tmp/results/repeated_utest/output/${status}/${i}
- mkdir -p $dest
- if [[ -d $source && -n "$(ls $source)" ]]; then
- mv $source/* $dest/
- fi
++ echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt
+
- # move the log files
- source=build/test/logs
- dest=/tmp/results/repeated_utest/logs/${status}/${i}
- mkdir -p $dest
- if [[ -d $source && -n "$(ls $source)" ]]; then
- mv $source/* $dest/
- fi
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
++ export CQLSH_PYTHON=/usr/bin/python2.7
++ fi
+
- # maybe stop iterations on test failure
- if [[ $REPEATED_UTEST_STOP_ON_FAILURE = true ]] && (( $exit_code > 0 )); then
- break
- fi
- done
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
+
- (exit ${exit_code})
- fi
- fi
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
- path: /tmp/results/repeated_utest/output
- - store_artifacts:
- path: /tmp/results/repeated_utest/stdout
- destination: stdout
++ path: /tmp/results
+ - store_artifacts:
- path: /tmp/results/repeated_utest/output
- destination: junitxml
++ path: /tmp/dtest
++ destination: dtest_j8_without_vnodes
+ - store_artifacts:
- path: /tmp/results/repeated_utest/logs
- destination: logs
++ path: ~/cassandra-dtest/logs
++ destination: dtest_j8_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - CASSANDRA_USE_JDK11: true
- j8_cqlsh-dtests-py3-with-vnodes:
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ j11_repeated_dtest:
+ docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
++ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
++ name: Log Environment Information
++ command: |
++ echo '*** id ***'
++ id
++ echo '*** cat /proc/cpuinfo ***'
++ cat /proc/cpuinfo
++ echo '*** free -m ***'
++ free -m
++ echo '*** df -m ***'
++ df -m
++ echo '*** ifconfig -a ***'
++ ifconfig -a
++ echo '*** uname -a ***'
++ uname -a
++ echo '*** mount ***'
++ mount
++ echo '*** env ***'
++ env
++ echo '*** java ***'
++ which java
++ java -version
++ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Determine Tests to Run (j8_with_vnodes)
- no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip- [...]
- - run:
- name: Run dtests (j8_with_vnodes)
++ name: Run repeated Python dtest
+ no_output_timeout: 15m
+ command: |
- echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt"
- cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt
++ if [ "${REPEATED_DTEST_NAME}" == "<nil>" ]; then
++ echo "Repeated dtest name hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_DTEST_COUNT}" == "<nil>" ]; then
++ echo "Repeated dtest count hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then
++ echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
++ else
+
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
- export CQLSH_PYTHON=/usr/bin/python3.6
- fi
++ # Calculate the number of test iterations to be run by the current parallel runner.
++ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
++ count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
++ if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
++ count=$((count+1))
++ fi
+
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
++ if (($count <= 0)); then
++ echo "No tests to run in this runner"
++ else
++ echo "Running ${REPEATED_DTEST_NAME} $count times"
+
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`
- set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
++
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++
++ stop_on_failure_arg=""
++ if ${REPEATED_DTEST_STOP_ON_FAILURE}; then
++ stop_on_failure_arg="-x"
++ fi
++
++ vnodes_args=""
++ if ${REPEATED_DTEST_VNODES}; then
++ vnodes_args="--use-vnodes --num-tokens=16"
++ fi
++
++ upgrade_arg=""
++ if false; then
++ upgrade_arg="--execute-upgrade-tests"
++ fi
++
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt
++ fi
++ fi
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
- destination: dtest_j8_with_vnodes
++ destination: dtest
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
- destination: dtest_j8_with_vnodes_logs
++ destination: dtest_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_cqlsh-dtests-py2-no-vnodes:
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - CASSANDRA_USE_JDK11: true
++ j8_repeated_dtest:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Determine Tests to Run (j8_without_vnodes)
- no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
- - run:
- name: Run dtests (j8_without_vnodes)
++ name: Run repeated Python dtest
+ no_output_timeout: 15m
+ command: |
- echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt"
- cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt
++ if [ "${REPEATED_DTEST_NAME}" == "<nil>" ]; then
++ echo "Repeated dtest name hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_DTEST_COUNT}" == "<nil>" ]; then
++ echo "Repeated dtest count hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then
++ echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
++ else
+
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
- export CQLSH_PYTHON=/usr/bin/python2.7
- fi
++ # Calculate the number of test iterations to be run by the current parallel runner.
++ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
++ count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
++ if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
++ count=$((count+1))
++ fi
+
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
++ if (($count <= 0)); then
++ echo "No tests to run in this runner"
++ else
++ echo "Running ${REPEATED_DTEST_NAME} $count times"
+
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`
- set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
++
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++
++ stop_on_failure_arg=""
++ if ${REPEATED_DTEST_STOP_ON_FAILURE}; then
++ stop_on_failure_arg="-x"
++ fi
++
++ vnodes_args=""
++ if ${REPEATED_DTEST_VNODES}; then
++ vnodes_args="--use-vnodes --num-tokens=16"
++ fi
++
++ upgrade_arg=""
++ if false; then
++ upgrade_arg="--execute-upgrade-tests"
++ fi
++
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_DTEST_NAME} | tee /tmp/dtest/stdout.txt
++ fi
++ fi
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
- destination: dtest_j8_without_vnodes
++ destination: dtest
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
- destination: dtest_j8_without_vnodes_logs
++ destination: dtest_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_cqlsh-dtests-py2-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
+ - run:
+ name: Run dtests (j11_with_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
+ export CQLSH_PYTHON=/usr/bin/python2.7
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_dtests-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
+ - run:
+ name: Run dtests (j11_with_vnodes)
+ no_output_timeout: 15m
+ command: "echo \"cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j11_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the [...]
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j8_dtests-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j8_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
+ - run:
+ name: Run dtests (j8_without_vnodes)
+ no_output_timeout: 15m
+ command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not [...]
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j8_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j8_upgradetests-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j8_upgradetests_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_upgradetests_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --ex [...]
+ - run:
+ name: Run dtests (j8_upgradetests_without_vnodes)
+ no_output_timeout: 15m
+ command: "echo \"cat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_upgradetests_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually [...]
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j8_upgradetests_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_upgradetests_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_repeated-dtest:
- docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
- resource_class: xlarge
- working_directory: ~/
- shell: /bin/bash -eo pipefail -l
- parallelism: 100
- steps:
- - attach_workspace:
- at: /home/cassandra
- - run:
- name: Clone Cassandra dtest Repository (via git)
- command: |
- git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
- - run:
- name: Configure virtualenv and python Dependencies
- command: |
- # note, this should be super quick as all dependencies should be pre-installed in the docker image
- # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
- # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
- # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
- pip3 uninstall -y cqlsh
- pip3 freeze
- - run:
- name: Run repeated dtest
- no_output_timeout: 15m
- command: |
- if [ "$REPEATED_DTEST_NAME" == "<nil>" ]; then
- echo "Repeated dtest name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_DTEST_COUNT" == "<nil>" ]; then
- echo "Repeated dtest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_DTEST_COUNT" -le 0 ]; then
- echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
- else
-
- # Calculate the number of test iterations to be run by the current parallel runner.
- # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_DTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_DTEST_COUNT % CIRCLE_NODE_TOTAL))); then
- count=$((count+1))
- fi
-
- if (($count <= 0)); then
- echo "No tests to run in this runner"
- else
- echo "Running $REPEATED_DTEST_NAME $count times"
-
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
-
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
-
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
-
- stop_on_failure_arg=""
- if $REPEATED_UTEST_STOP_ON_FAILURE; then
- stop_on_failure_arg="-x"
- fi
-
- vnodes_args=""
- if $REPEATED_DTEST_VNODES; then
- vnodes_args="--use-vnodes --num-tokens=16"
- fi
-
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $REPEATED_DTEST_NAME | tee /tmp/dtest/stdout.txt
- fi
- fi
- - store_test_results:
- path: /tmp/results
- - store_artifacts:
- path: /tmp/dtest
- destination: dtest
- - store_artifacts:
- path: ~/cassandra-dtest/logs
- destination: dtest_logs
- environment:
- - ANT_HOME: /usr/share/ant
- - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - LANG: en_US.UTF-8
- - KEEP_TEST_DIR: true
- - DEFAULT_DIR: /home/cassandra/cassandra-dtest
- - PYTHONIOENCODING: utf-8
- - PYTHONUNBUFFERED: true
- - CASS_DRIVER_NO_EXTENSIONS: true
- - CASS_DRIVER_NO_CYTHON: true
- - CASSANDRA_SKIP_SYNC: true
- - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
- - DTEST_BRANCH: trunk
- - CCM_MAX_HEAP_SIZE: 1024M
- - CCM_HEAP_NEWSIZE: 256M
- - REPEATED_UTEST_TARGET: testsome
- - REPEATED_UTEST_CLASS: null
- - REPEATED_UTEST_METHODS: null
- - REPEATED_UTEST_COUNT: 100
- - REPEATED_UTEST_STOP_ON_FAILURE: false
- - REPEATED_DTEST_NAME: null
- - REPEATED_DTEST_VNODES: false
- - REPEATED_DTEST_COUNT: 100
- - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ utests_stress:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 1
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Run Unit Tests (stress-test)
+ command: |
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ ant stress-test
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j8_unit_tests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Determine unit Tests to Run
+ command: |
+ # reminder: this code (along with all the steps) is independently executed on every circle container
+ # so the goal here is to get the circleci script to return the tests *this* container will run
+ # which we do via the `circleci` cli tool.
+
+ rm -fr ~/cassandra-dtest/upgrade_tests
+ echo "***java tests***"
+
+ # get all of our unit test filenames
+ set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt
+
+ # split up the unit tests into groups based on the number of containers we have
+ set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt
+ set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt"
+ cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ no_output_timeout: 15m
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run Unit Tests (testclasslist)
+ command: |
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true)
+ if [ -z "$test_timeout" ]; then
+ test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
+ fi
+ ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_jvm_dtests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 2
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Determine distributed Tests to Run
+ command: |
+ # reminder: this code (along with all the steps) is independently executed on every circle container
+ # so the goal here is to get the circleci script to return the tests *this* container will run
+ # which we do via the `circleci` cli tool.
+
+ rm -fr ~/cassandra-dtest/upgrade_tests
+ echo "***java tests***"
+
+ # get all of our unit test filenames
+ set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt
+
+ # split up the unit tests into groups based on the number of containers we have
+ set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt
+ set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt"
+ cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ no_output_timeout: 15m
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run Unit Tests (testclasslist)
+ command: |
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true)
+ if [ -z "$test_timeout" ]; then
+ test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
+ fi
+ ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_build:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 1
+ steps:
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Clone Cassandra Repository (via git)
+ command: |
+ git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH git://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra
+ - run:
+ name: Build Cassandra
+ command: |
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd ~/cassandra
+ # Loop to prevent failure due to maven-ant-tasks not downloading a jar..
+ for x in $(seq 1 3); do
+ ${ANT_HOME}/bin/ant clean realclean jar
+ RETURN="$?"
+ if [ "${RETURN}" -eq "0" ]; then
+ break
+ fi
+ done
+ # Exit, if we didn't build successfully
+ if [ "${RETURN}" -ne "0" ]; then
+ echo "Build failed with exit code: ${RETURN}"
+ exit ${RETURN}
+ fi
+ no_output_timeout: 15m
+ - run:
+ name: Run eclipse-warnings
+ command: |
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd ~/cassandra
+ ant eclipse-warnings
+ - persist_to_workspace:
+ root: /home/cassandra
+ paths:
+ - cassandra
+ - .m2
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_cqlsh-dtests-py2-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource- [...]
+ - run:
+ name: Run dtests (j11_without_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
+ export CQLSH_PYTHON=/usr/bin/python2.7
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_cqlshlib_tests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 1
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Run cqlshlib Unit Tests
+ command: |
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra/pylib
+ ./cassandra-cqlsh-tests.sh ..
+ no_output_timeout: 15m
+ - store_test_results:
- path: /tmp/cassandra/pylib
++ path: /tmp/cassandra/pylib
++ environment:
++ - ANT_HOME: /usr/share/ant
++ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ - LANG: en_US.UTF-8
++ - KEEP_TEST_DIR: true
++ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
++ - PYTHONIOENCODING: utf-8
++ - PYTHONUNBUFFERED: true
++ - CASS_DRIVER_NO_EXTENSIONS: true
++ - CASS_DRIVER_NO_CYTHON: true
++ - CASSANDRA_SKIP_SYNC: true
++ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
++ - DTEST_BRANCH: trunk
++ - CCM_MAX_HEAP_SIZE: 1024M
++ - CCM_HEAP_NEWSIZE: 256M
++ - REPEATED_UTEST_TARGET: testsome
++ - REPEATED_UTEST_CLASS: null
++ - REPEATED_UTEST_METHODS: null
++ - REPEATED_UTEST_COUNT: 100
++ - REPEATED_UTEST_STOP_ON_FAILURE: false
++ - REPEATED_DTEST_NAME: null
++ - REPEATED_DTEST_VNODES: false
++ - REPEATED_DTEST_COUNT: 100
++ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - CASSANDRA_USE_JDK11: true
++ j11_repeated_utest:
++ docker:
++ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
++ resource_class: xlarge
++ working_directory: ~/
++ shell: /bin/bash -eo pipefail -l
++ parallelism: 100
++ steps:
++ - attach_workspace:
++ at: /home/cassandra
++ - run:
++ name: Log Environment Information
++ command: |
++ echo '*** id ***'
++ id
++ echo '*** cat /proc/cpuinfo ***'
++ cat /proc/cpuinfo
++ echo '*** free -m ***'
++ free -m
++ echo '*** df -m ***'
++ df -m
++ echo '*** ifconfig -a ***'
++ ifconfig -a
++ echo '*** uname -a ***'
++ uname -a
++ echo '*** mount ***'
++ mount
++ echo '*** env ***'
++ env
++ echo '*** java ***'
++ which java
++ java -version
++ - run:
++ name: Run repeated JUnit test
++ no_output_timeout: 15m
++ command: |
++ if [ "${REPEATED_UTEST_CLASS}" == "<nil>" ]; then
++ echo "Repeated utest class name hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_UTEST_COUNT}" == "<nil>" ]; then
++ echo "Repeated utest count hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then
++ echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
++ else
++
++ # Calculate the number of test iterations to be run by the current parallel runner.
++ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
++ count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL))
++ if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
++ count=$((count+1))
++ fi
++
++ if (($count <= 0)); then
++ echo "No tests to run in this runner"
++ else
++ echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times"
++
++ set -x
++ export PATH=$JAVA_HOME/bin:$PATH
++ time mv ~/cassandra /tmp
++ cd /tmp/cassandra
++ if [ -d ~/dtest_jars ]; then
++ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
++ fi
++
++ target=${REPEATED_UTEST_TARGET}
++ class_path=${REPEATED_UTEST_CLASS}
++ class_name="${class_path##*.}"
++
++ # Prepare the -Dtest.name argument.
++ # It can be the fully qualified class name or the short class name, depending on the target.
++ if [[ $target == "test" || \
++ $target == "test-cdc" || \
++ $target == "test-compression" || \
++ $target == "test-system-keyspace-directory" ]]; then
++ name="-Dtest.name=$class_name"
++ else
++ name="-Dtest.name=$class_path"
++ fi
++
++ # Prepare the -Dtest.methods argument, which is optional
++ if [ "${REPEATED_UTEST_METHODS}" == "<nil>" ]; then
++ methods=""
++ else
++ methods="-Dtest.methods=${REPEATED_UTEST_METHODS}"
++ fi
++
++ # Run the test target as many times as requested collecting the exit code,
++ # stopping the iteration only if stop_on_failure is set.
++ exit_code="$?"
++ for i in $(seq -w 1 $count); do
++
++ echo "Running test iteration $i of $count"
++
++ # run the test
++ status="passes"
++ if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
++ status="fails"
++ exit_code=1
++ fi
++
++ # move the stdout output file
++ dest=/tmp/results/repeated_utest/stdout/${status}/${i}
++ mkdir -p $dest
++ mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt
++
++ # move the XML output files
++ source=build/test/output
++ dest=/tmp/results/repeated_utest/output/${status}/${i}
++ mkdir -p $dest
++ if [[ -d $source && -n "$(ls $source)" ]]; then
++ mv $source/* $dest/
++ fi
++
++ # move the log files
++ source=build/test/logs
++ dest=/tmp/results/repeated_utest/logs/${status}/${i}
++ mkdir -p $dest
++ if [[ -d $source && -n "$(ls $source)" ]]; then
++ mv $source/* $dest/
++ fi
++
++ # maybe stop iterations on test failure
++ if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then
++ break
++ fi
++ done
++
++ (exit ${exit_code})
++ fi
++ fi
++ - store_test_results:
++ path: /tmp/results/repeated_utest/output
++ - store_artifacts:
++ path: /tmp/results/repeated_utest/stdout
++ destination: stdout
++ - store_artifacts:
++ path: /tmp/results/repeated_utest/output
++ destination: junitxml
++ - store_artifacts:
++ path: /tmp/results/repeated_utest/logs
++ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j8_dtests-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j8_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip- [...]
+ - run:
+ name: Run dtests (j8_with_vnodes)
+ no_output_timeout: 15m
+ command: "echo \"cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_with_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not the e [...]
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j8_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_cqlsh-dtests-py38-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.8/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource- [...]
+ - run:
+ name: Run dtests (j11_without_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt
+
+ source ~/env3.8/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then
+ export CQLSH_PYTHON=/usr/bin/python3.8
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j8_jvm_dtests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 5
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Determine distributed Tests to Run
+ command: |
+ # reminder: this code (along with all the steps) is independently executed on every circle container
+ # so the goal here is to get the circleci script to return the tests *this* container will run
+ # which we do via the `circleci` cli tool.
+
+ rm -fr ~/cassandra-dtest/upgrade_tests
+ echo "***java tests***"
+
+ # get all of our unit test filenames
+ set -eo pipefail && circleci tests glob "$HOME/cassandra/test/distributed/**/*.java" > /tmp/all_java_unit_tests.txt
+
+ # split up the unit tests into groups based on the number of containers we have
+ set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt
+ set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/distributed/;;g" | grep "Test\.java$" | grep -v upgrade > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt"
+ cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ no_output_timeout: 15m
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run Unit Tests (testclasslist)
+ command: |
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ test_timeout=$(grep 'name="test.distributed.timeout"' build.xml | awk -F'"' '{print $4}' || true)
+ if [ -z "$test_timeout" ]; then
+ test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
+ fi
+ ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=distributed
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j8_build:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 1
+ steps:
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Clone Cassandra Repository (via git)
+ command: |
+ git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH git://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git ~/cassandra
+ - run:
+ name: Build Cassandra
+ command: |
+ export PATH=$JAVA_HOME/bin:$PATH
+ cd ~/cassandra
+ # Loop to prevent failure due to maven-ant-tasks not downloading a jar..
+ for x in $(seq 1 3); do
+ ${ANT_HOME}/bin/ant clean realclean jar
+ RETURN="$?"
+ if [ "${RETURN}" -eq "0" ]; then
+ break
+ fi
+ done
+ # Exit, if we didn't build successfully
+ if [ "${RETURN}" -ne "0" ]; then
+ echo "Build failed with exit code: ${RETURN}"
+ exit ${RETURN}
+ fi
+ no_output_timeout: 15m
- run:
name: Run eclipse-warnings
command: |
@@@ -2400,253 -616,71 +2882,92 @@@
- CASS_DRIVER_NO_CYTHON: true
- CASSANDRA_SKIP_SYNC: true
- DTEST_REPO: git://github.com/apache/cassandra-dtest.git
- - DTEST_BRANCH: trunk
- - CCM_MAX_HEAP_SIZE: 1024M
- - CCM_HEAP_NEWSIZE: 256M
- - REPEATED_UTEST_TARGET: testsome
- - REPEATED_UTEST_CLASS: null
- - REPEATED_UTEST_METHODS: null
- - REPEATED_UTEST_COUNT: 100
- - REPEATED_UTEST_STOP_ON_FAILURE: false
- - REPEATED_DTEST_NAME: null
- - REPEATED_DTEST_VNODES: false
- - REPEATED_DTEST_COUNT: 100
- - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_cqlsh-dtests-py3-no-vnodes:
- docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
- resource_class: xlarge
- working_directory: ~/
- shell: /bin/bash -eo pipefail -l
- parallelism: 100
- steps:
- - attach_workspace:
- at: /home/cassandra
- - run:
- name: Clone Cassandra dtest Repository (via git)
- command: |
- git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
- - run:
- name: Configure virtualenv and python Dependencies
- command: |
- # note, this should be super quick as all dependencies should be pre-installed in the docker image
- # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
- # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
- # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
- pip3 uninstall -y cqlsh
- pip3 freeze
- - run:
- name: Determine Tests to Run (j8_without_vnodes)
- no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
- - run:
- name: Run dtests (j8_without_vnodes)
- no_output_timeout: 15m
- command: |
- echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt"
- cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt
-
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
- export CQLSH_PYTHON=/usr/bin/python3.6
- fi
-
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
-
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`
- set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
- - store_test_results:
- path: /tmp/results
- - store_artifacts:
- path: /tmp/dtest
- destination: dtest_j8_without_vnodes
- - store_artifacts:
- path: ~/cassandra-dtest/logs
- destination: dtest_j8_without_vnodes_logs
- environment:
- - ANT_HOME: /usr/share/ant
- - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - LANG: en_US.UTF-8
- - KEEP_TEST_DIR: true
- - DEFAULT_DIR: /home/cassandra/cassandra-dtest
- - PYTHONIOENCODING: utf-8
- - PYTHONUNBUFFERED: true
- - CASS_DRIVER_NO_EXTENSIONS: true
- - CASS_DRIVER_NO_CYTHON: true
- - CASSANDRA_SKIP_SYNC: true
- - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
- - DTEST_BRANCH: trunk
- - CCM_MAX_HEAP_SIZE: 1024M
- - CCM_HEAP_NEWSIZE: 256M
- - REPEATED_UTEST_TARGET: testsome
- - REPEATED_UTEST_CLASS: null
- - REPEATED_UTEST_METHODS: null
- - REPEATED_UTEST_COUNT: 100
- - REPEATED_UTEST_STOP_ON_FAILURE: false
- - REPEATED_DTEST_NAME: null
- - REPEATED_DTEST_VNODES: false
- - REPEATED_DTEST_COUNT: 100
- - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_repeated-utest:
- docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
- resource_class: xlarge
- working_directory: ~/
- shell: /bin/bash -eo pipefail -l
- parallelism: 100
- steps:
- - attach_workspace:
- at: /home/cassandra
- - run:
- name: Log Environment Information
- command: |
- echo '*** id ***'
- id
- echo '*** cat /proc/cpuinfo ***'
- cat /proc/cpuinfo
- echo '*** free -m ***'
- free -m
- echo '*** df -m ***'
- df -m
- echo '*** ifconfig -a ***'
- ifconfig -a
- echo '*** uname -a ***'
- uname -a
- echo '*** mount ***'
- mount
- echo '*** env ***'
- env
- echo '*** java ***'
- which java
- java -version
- - run:
- name: Run repeated utest
- no_output_timeout: 15m
- command: |
- if [ "$REPEATED_UTEST_CLASS" == "<nil>" ]; then
- echo "Repeated utest class name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" == "<nil>" ]; then
- echo "Repeated utest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" -le 0 ]; then
- echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
- else
-
- # Calculate the number of test iterations to be run by the current parallel runner.
- # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_UTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_UTEST_COUNT % CIRCLE_NODE_TOTAL))); then
- count=$((count+1))
- fi
-
- if (($count <= 0)); then
- echo "No tests to run in this runner"
- else
- echo "Running $REPEATED_UTEST_TARGET $REPEATED_UTEST_CLASS $REPEATED_UTEST_METHODS $count times"
-
- set -x
- export PATH=$JAVA_HOME/bin:$PATH
- time mv ~/cassandra /tmp
- cd /tmp/cassandra
- if [ -d ~/dtest_jars ]; then
- cp ~/dtest_jars/dtest* /tmp/cassandra/build/
- fi
-
- target=$REPEATED_UTEST_TARGET
- class_path=$REPEATED_UTEST_CLASS
- class_name="${class_path##*.}"
-
- # Prepare the -Dtest.name argument.
- # It can be the fully qualified class name or the short class name, depending on the target.
- if [[ $target == "test" || \
- $target == "test-cdc" || \
- $target == "test-compression" || \
- $target == "test-system-keyspace-directory" ]]; then
- name="-Dtest.name=$class_name"
- else
- name="-Dtest.name=$class_path"
- fi
-
- # Prepare the -Dtest.methods argument, which is optional
- if [ "$REPEATED_UTEST_METHODS" == "<nil>" ]; then
- methods=""
- else
- methods="-Dtest.methods=$REPEATED_UTEST_METHODS"
- fi
-
- # Run the test target as many times as requested collecting the exit code,
- # stopping the iteration only if REPEATED_UTEST_STOP_ON_FAILURE is set.
- exit_code="$?"
- for i in $(seq -w 1 $count); do
-
- echo "Running test iteration $i of $count"
-
- # run the test
- status="passes"
- if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
- status="fails"
- exit_code=1
- fi
-
- # move the stdout output file
- dest=/tmp/results/repeated_utest/stdout/${status}/${i}
- mkdir -p $dest
- mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt
-
- # move the XML output files
- source=build/test/output
- dest=/tmp/results/repeated_utest/output/${status}/${i}
- mkdir -p $dest
- if [[ -d $source && -n "$(ls $source)" ]]; then
- mv $source/* $dest/
- fi
+ - DTEST_BRANCH: trunk
- - CCM_MAX_HEAP_SIZE: 2048M
- - CCM_HEAP_NEWSIZE: 512M
++ - CCM_MAX_HEAP_SIZE: 1024M
++ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_dtests-no-vnodes:
++ j8_cqlsh-dtests-py3-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
- pip3 install --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j8_without_vnodes)
+ no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
+ - run:
+ name: Run dtests (j8_without_vnodes)
+ no_output_timeout: 15m
- command: "echo \"cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\"\ncat /tmp/split_dtest_tests_j8_without_vnodes_final.txt\n\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\nif [ -n '' ]; then\n export \nfi\n\njava -version\ncd ~/cassandra-dtest\nmkdir -p /tmp/dtest\n\necho \"env: $(env)\"\necho \"** done env\"\nmkdir -p /tmp/results/dtests\n# we need the \"set -o pipefail\" here so that the exit code that circleci will actually use is from pytest and not [...]
++ command: |
++ echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt
+
- # move the log files
- source=build/test/logs
- dest=/tmp/results/repeated_utest/logs/${status}/${i}
- mkdir -p $dest
- if [[ -d $source && -n "$(ls $source)" ]]; then
- mv $source/* $dest/
- fi
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
++ export CQLSH_PYTHON=/usr/bin/python3.6
++ fi
+
- # maybe stop iterations on test failure
- if [[ $REPEATED_UTEST_STOP_ON_FAILURE = true ]] && (( $exit_code > 0 )); then
- break
- fi
- done
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
+
- (exit ${exit_code})
- fi
- fi
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
- store_test_results:
- path: /tmp/results/repeated_utest/output
- - store_artifacts:
- path: /tmp/results/repeated_utest/stdout
- destination: stdout
+ path: /tmp/results
- store_artifacts:
- path: /tmp/results/repeated_utest/output
- destination: junitxml
+ path: /tmp/dtest
+ destination: dtest_j8_without_vnodes
- store_artifacts:
- path: /tmp/results/repeated_utest/logs
- destination: logs
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_without_vnodes_logs
environment:
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- LANG: en_US.UTF-8
- KEEP_TEST_DIR: true
- DEFAULT_DIR: /home/cassandra/cassandra-dtest
@@@ -2668,9 -702,16 +2989,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_upgradetests-no-vnodes:
+ j8_cqlsh-dtests-py38-with-vnodes:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: xlarge
@@@ -2756,9 -817,16 +3084,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- utests_stress:
+ utests_long:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: xlarge
@@@ -2812,9 -879,16 +3147,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_unit_tests:
+ utests_system_keyspace_directory:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: xlarge
@@@ -2914,15 -987,22 +3256,22 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_dtests-with-vnodes:
+ j8_cqlshlib_tests:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
- resource_class: xlarge
+ resource_class: medium
working_directory: ~/
shell: /bin/bash -eo pipefail -l
- parallelism: 100
+ parallelism: 1
steps:
- attach_workspace:
at: /home/cassandra
@@@ -2961,9 -1061,16 +3310,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_jvm_dtests:
+ utests_fqltool:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: xlarge
@@@ -3017,11 -1169,18 +3373,18 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- utests_long:
+ j11_dtests-no-vnodes:
docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
resource_class: xlarge
working_directory: ~/
shell: /bin/bash -eo pipefail -l
@@@ -3108,9 -1231,15 +3471,16 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
utests_compression:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
@@@ -3211,6 -1339,187 +3581,188 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j8_repeated_utest:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: xlarge
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 100
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run repeated JUnit test
+ no_output_timeout: 15m
+ command: |
+ if [ "${REPEATED_UTEST_CLASS}" == "<nil>" ]; then
+ echo "Repeated utest class name hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_UTEST_COUNT}" == "<nil>" ]; then
+ echo "Repeated utest count hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_UTEST_COUNT}" -le 0 ]; then
+ echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
+ else
+
+ # Calculate the number of test iterations to be run by the current parallel runner.
+ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
+ count=$((${REPEATED_UTEST_COUNT} / CIRCLE_NODE_TOTAL))
+ if (($CIRCLE_NODE_INDEX < (${REPEATED_UTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
+ count=$((count+1))
+ fi
+
+ if (($count <= 0)); then
+ echo "No tests to run in this runner"
+ else
+ echo "Running ${REPEATED_UTEST_TARGET} ${REPEATED_UTEST_CLASS} ${REPEATED_UTEST_METHODS} ${REPEATED_UTEST_COUNT} times"
+
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+
+ target=${REPEATED_UTEST_TARGET}
+ class_path=${REPEATED_UTEST_CLASS}
+ class_name="${class_path##*.}"
+
+ # Prepare the -Dtest.name argument.
+ # It can be the fully qualified class name or the short class name, depending on the target.
+ if [[ $target == "test" || \
+ $target == "test-cdc" || \
+ $target == "test-compression" || \
+ $target == "test-system-keyspace-directory" ]]; then
+ name="-Dtest.name=$class_name"
+ else
+ name="-Dtest.name=$class_path"
+ fi
+
+ # Prepare the -Dtest.methods argument, which is optional
+ if [ "${REPEATED_UTEST_METHODS}" == "<nil>" ]; then
+ methods=""
+ else
+ methods="-Dtest.methods=${REPEATED_UTEST_METHODS}"
+ fi
+
+ # Run the test target as many times as requested collecting the exit code,
+ # stopping the iteration only if stop_on_failure is set.
+ exit_code="$?"
+ for i in $(seq -w 1 $count); do
+
+ echo "Running test iteration $i of $count"
+
+ # run the test
+ status="passes"
+ if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
+ status="fails"
+ exit_code=1
+ fi
+
+ # move the stdout output file
+ dest=/tmp/results/repeated_utest/stdout/${status}/${i}
+ mkdir -p $dest
+ mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt
+
+ # move the XML output files
+ source=build/test/output
+ dest=/tmp/results/repeated_utest/output/${status}/${i}
+ mkdir -p $dest
+ if [[ -d $source && -n "$(ls $source)" ]]; then
+ mv $source/* $dest/
+ fi
+
+ # move the log files
+ source=build/test/logs
+ dest=/tmp/results/repeated_utest/logs/${status}/${i}
+ mkdir -p $dest
+ if [[ -d $source && -n "$(ls $source)" ]]; then
+ mv $source/* $dest/
+ fi
+
+ # maybe stop iterations on test failure
+ if [[ ${REPEATED_UTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then
+ break
+ fi
+ done
+
+ (exit ${exit_code})
+ fi
+ fi
+ - store_test_results:
+ path: /tmp/results/repeated_utest/output
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/stdout
+ destination: stdout
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/logs
+ destination: logs
+ environment:
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - ANT_HOME: /usr/share/ant
++ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
- - CCM_MAX_HEAP_SIZE: 2048M
- - CCM_HEAP_NEWSIZE: 512M
++ - CCM_MAX_HEAP_SIZE: 1024M
++ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
j8_dtest_jars_build:
@@@ -3338,27 -1646,19 +3897,30 @@@ workflows
- utests_stress:
requires:
- start_utests_stress
+ - j8_build
+ - start_utests_fqltool:
+ type: approval
+ - utests_fqltool:
+ requires:
+ - start_utests_fqltool
+ - j8_build
+ - start_utests_system_keyspace_directory:
+ type: approval
+ - utests_system_keyspace_directory:
+ requires:
+ - start_utests_system_keyspace_directory
+ - j8_build
- - start_jvm_upgrade_dtest:
+ - start_j8_dtest_jars_build:
type: approval
- j8_dtest_jars_build:
requires:
- - build
+ - j8_build
- - start_jvm_upgrade_dtest
+ - start_j8_dtest_jars_build
+ - start_jvm_upgrade_dtest:
+ type: approval
- j8_jvm_upgrade_dtests:
requires:
+ - start_jvm_upgrade_dtest
- j8_dtest_jars_build
- start_j8_dtests:
type: approval
@@@ -3385,146 -1677,27 +3947,158 @@@
- j8_upgradetests-no-vnodes:
requires:
- start_upgrade_tests
+ - j8_build
+ - start_j8_cqlsh_tests-with-vnodes:
+ type: approval
+ - j8_cqlsh-dtests-py2-with-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-with-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py3-with-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-with-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py38-with-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-with-vnodes
+ - j8_build
+ - start_j8_cqlsh_tests-no-vnodes:
+ type: approval
+ - j8_cqlsh-dtests-py2-no-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-no-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py3-no-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-no-vnodes
+ - j8_build
+ - j8_cqlsh-dtests-py38-no-vnodes:
+ requires:
+ - start_j8_cqlsh_tests-no-vnodes
+ - j8_build
+ - start_j11_cqlsh_tests-with-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py3-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py38-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j8_build
+ - start_j11_cqlsh_tests-no-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py3-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j8_build
+ - j11_cqlsh-dtests-py38-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j8_build
- - start_j8_repeated-utest:
+ - start_j8_repeated_utest:
+ type: approval
+ - j8_repeated_utest:
+ requires:
+ - start_j8_repeated_utest
- - build
++ - j8_build
++ - start_j11_repeated_utest:
+ type: approval
- - j8_repeated-utest:
++ - j11_repeated_utest:
+ requires:
- - start_j8_repeated-utest
++ - start_j11_repeated_utest
+ - j8_build
- - start_j11_repeated-utest:
+ - start_j8_repeated_dtest:
type: approval
- - j11_repeated-utest:
+ - j8_repeated_dtest:
requires:
- - start_j11_repeated-utest
+ - start_j8_repeated_dtest
- - build
+ - j8_build
- - start_j8_repeated-dtest:
++ - start_j11_repeated_dtest:
+ type: approval
- - j8_repeated-dtest:
++ - j11_repeated_dtest:
+ requires:
- - start_j8_repeated-dtest
++ - start_j11_repeated_dtest
+ - j8_build
- - start_j11_repeated-dtest:
+ - start_repeated_upgrade_dtest:
type: approval
- - j11_repeated-dtest:
+ - repeated_upgrade_dtest:
requires:
- - start_j11_repeated-dtest
+ - start_repeated_upgrade_dtest
- - build
+ - j8_build
+ - start_repeated_jvm_upgrade_dtest:
+ type: approval
+ - repeated_jvm_upgrade_dtest:
+ requires:
+ - start_repeated_jvm_upgrade_dtest
+ - j8_dtest_jars_build
+ java11_build_and_run_tests:
+ jobs:
+ - j11_build
+ - j11_unit_tests:
+ requires:
+ - j11_build
+ - j11_jvm_dtests:
+ requires:
+ - j11_build
+ - j11_cqlshlib_tests:
+ requires:
+ - j11_build
+ - start_j11_dtests:
+ type: approval
+ - j11_dtests-with-vnodes:
+ requires:
+ - start_j11_dtests
+ - j11_build
+ - j11_dtests-no-vnodes:
+ requires:
+ - start_j11_dtests
+ - j11_build
+ - start_j11_cqlsh_tests-with-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py3-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py38-with-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-with-vnodes
+ - j11_build
+ - start_j11_cqlsh_tests-no-vnodes:
+ type: approval
+ - j11_cqlsh-dtests-py2-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py3-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j11_build
+ - j11_cqlsh-dtests-py38-no-vnodes:
+ requires:
+ - start_j11_cqlsh_tests-no-vnodes
+ - j11_build
- - start_j11_repeated-utest:
++ - start_j11_repeated_utest:
+ type: approval
- - j11_repeated-utest:
++ - j11_repeated_utest:
+ requires:
- - start_j11_repeated-utest
++ - start_j11_repeated_utest
+ - j11_build
- - start_j11_repeated-dtest:
++ - start_j11_repeated_dtest:
+ type: approval
- - j11_repeated-dtest:
++ - j11_repeated_dtest:
+ requires:
- - start_j11_repeated-dtest
++ - start_j11_repeated_dtest
+ - j11_build
diff --cc .circleci/config.yml.LOWRES
index 822f246,1451eea..17a4855
--- a/.circleci/config.yml.LOWRES
+++ b/.circleci/config.yml.LOWRES
@@@ -118,73 -117,159 +118,255 @@@ jobs
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ repeated_jvm_upgrade_dtest:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run repeated JUnit test
+ no_output_timeout: 15m
+ command: |
+ if [ "${REPEATED_JVM_UPGRADE_DTEST_CLASS}" == "<nil>" ]; then
+ echo "Repeated utest class name hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" == "<nil>" ]; then
+ echo "Repeated utest count hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_JVM_UPGRADE_DTEST_COUNT}" -le 0 ]; then
+ echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
+ else
+
+ # Calculate the number of test iterations to be run by the current parallel runner.
+ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
+ count=$((${REPEATED_JVM_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
+ if (($CIRCLE_NODE_INDEX < (${REPEATED_JVM_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
+ count=$((count+1))
+ fi
+
+ if (($count <= 0)); then
+ echo "No tests to run in this runner"
+ else
+ echo "Running test-jvm-dtest-some ${REPEATED_JVM_UPGRADE_DTEST_CLASS} ${REPEATED_JVM_UPGRADE_DTEST_METHODS} ${REPEATED_JVM_UPGRADE_DTEST_COUNT} times"
+
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+
+ target=test-jvm-dtest-some
+ class_path=${REPEATED_JVM_UPGRADE_DTEST_CLASS}
+ class_name="${class_path##*.}"
+
+ # Prepare the -Dtest.name argument.
+ # It can be the fully qualified class name or the short class name, depending on the target.
+ if [[ $target == "test" || \
+ $target == "test-cdc" || \
+ $target == "test-compression" || \
+ $target == "test-system-keyspace-directory" ]]; then
+ name="-Dtest.name=$class_name"
+ else
+ name="-Dtest.name=$class_path"
+ fi
+
+ # Prepare the -Dtest.methods argument, which is optional
+ if [ "${REPEATED_JVM_UPGRADE_DTEST_METHODS}" == "<nil>" ]; then
+ methods=""
+ else
+ methods="-Dtest.methods=${REPEATED_JVM_UPGRADE_DTEST_METHODS}"
+ fi
+
+ # Run the test target as many times as requested collecting the exit code,
+ # stopping the iteration only if stop_on_failure is set.
+ exit_code="$?"
+ for i in $(seq -w 1 $count); do
+
+ echo "Running test iteration $i of $count"
+
+ # run the test
+ status="passes"
+ if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
+ status="fails"
+ exit_code=1
+ fi
+
+ # move the stdout output file
+ dest=/tmp/results/repeated_utest/stdout/${status}/${i}
+ mkdir -p $dest
+ mv stdout.txt $dest/test-jvm-dtest-some-${REPEATED_JVM_UPGRADE_DTEST_CLASS}.txt
+
+ # move the XML output files
+ source=build/test/output
+ dest=/tmp/results/repeated_utest/output/${status}/${i}
+ mkdir -p $dest
+ if [[ -d $source && -n "$(ls $source)" ]]; then
+ mv $source/* $dest/
+ fi
+
+ # move the log files
+ source=build/test/logs
+ dest=/tmp/results/repeated_utest/logs/${status}/${i}
+ mkdir -p $dest
+ if [[ -d $source && -n "$(ls $source)" ]]; then
+ mv $source/* $dest/
+ fi
+
+ # maybe stop iterations on test failure
+ if [[ ${REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE} = true ]] && (( $exit_code > 0 )); then
+ break
+ fi
+ done
+
+ (exit ${exit_code})
+ fi
+ fi
+ - store_test_results:
+ path: /tmp/results/repeated_utest/output
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/stdout
+ destination: stdout
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/results/repeated_utest/logs
+ destination: logs
+ environment:
++ - ANT_HOME: /usr/share/ant
++ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ - LANG: en_US.UTF-8
++ - KEEP_TEST_DIR: true
++ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
++ - PYTHONIOENCODING: utf-8
++ - PYTHONUNBUFFERED: true
++ - CASS_DRIVER_NO_EXTENSIONS: true
++ - CASS_DRIVER_NO_CYTHON: true
++ - CASSANDRA_SKIP_SYNC: true
++ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
++ - DTEST_BRANCH: trunk
++ - CCM_MAX_HEAP_SIZE: 1024M
++ - CCM_HEAP_NEWSIZE: 256M
++ - REPEATED_UTEST_TARGET: testsome
++ - REPEATED_UTEST_CLASS: null
++ - REPEATED_UTEST_METHODS: null
++ - REPEATED_UTEST_COUNT: 100
++ - REPEATED_UTEST_STOP_ON_FAILURE: false
++ - REPEATED_DTEST_NAME: null
++ - REPEATED_DTEST_VNODES: false
++ - REPEATED_DTEST_COUNT: 100
++ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j8_cqlsh-dtests-py2-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j8_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip- [...]
+ - run:
+ name: Run dtests (j8_with_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
+ export CQLSH_PYTHON=/usr/bin/python2.7
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j8_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_with_vnodes_logs
+ environment:
- ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- LANG: en_US.UTF-8
- KEEP_TEST_DIR: true
- DEFAULT_DIR: /home/cassandra/cassandra-dtest
@@@ -206,112 -291,136 +388,248 @@@
- REPEATED_DTEST_VNODES: false
- REPEATED_DTEST_COUNT: 100
- REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_unit_tests:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Determine unit Tests to Run
+ command: |
+ # reminder: this code (along with all the steps) is independently executed on every circle container
+ # so the goal here is to get the circleci script to return the tests *this* container will run
+ # which we do via the `circleci` cli tool.
+
+ rm -fr ~/cassandra-dtest/upgrade_tests
+ echo "***java tests***"
+
+ # get all of our unit test filenames
+ set -eo pipefail && circleci tests glob "$HOME/cassandra/test/unit/**/*.java" > /tmp/all_java_unit_tests.txt
+
+ # split up the unit tests into groups based on the number of containers we have
+ set -eo pipefail && circleci tests split --split-by=timings --timings-type=filename --index=${CIRCLE_NODE_INDEX} --total=${CIRCLE_NODE_TOTAL} /tmp/all_java_unit_tests.txt > /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt
+ set -eo pipefail && cat /tmp/java_tests_${CIRCLE_NODE_INDEX}.txt | sed "s;^/home/cassandra/cassandra/test/unit/;;g" | grep "Test\.java$" > /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ echo "** /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt"
+ cat /tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt
+ no_output_timeout: 15m
+ - run:
+ name: Log Environment Information
+ command: |
+ echo '*** id ***'
+ id
+ echo '*** cat /proc/cpuinfo ***'
+ cat /proc/cpuinfo
+ echo '*** free -m ***'
+ free -m
+ echo '*** df -m ***'
+ df -m
+ echo '*** ifconfig -a ***'
+ ifconfig -a
+ echo '*** uname -a ***'
+ uname -a
+ echo '*** mount ***'
+ mount
+ echo '*** env ***'
+ env
+ echo '*** java ***'
+ which java
+ java -version
+ - run:
+ name: Run Unit Tests (testclasslist)
+ command: |
+ set -x
+ export PATH=$JAVA_HOME/bin:$PATH
+ time mv ~/cassandra /tmp
+ cd /tmp/cassandra
+ if [ -d ~/dtest_jars ]; then
+ cp ~/dtest_jars/dtest* /tmp/cassandra/build/
+ fi
+ test_timeout=$(grep 'name="test.unit.timeout"' build.xml | awk -F'"' '{print $4}' || true)
+ if [ -z "$test_timeout" ]; then
+ test_timeout=$(grep 'name="test.timeout"' build.xml | awk -F'"' '{print $4}')
+ fi
+ ant testclasslist -Dtest.timeout="$test_timeout" -Dtest.classlistfile=/tmp/java_tests_${CIRCLE_NODE_INDEX}_final.txt -Dtest.classlistprefix=unit
+ no_output_timeout: 15m
+ - store_test_results:
+ path: /tmp/cassandra/build/test/output/
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/output
+ destination: junitxml
+ - store_artifacts:
+ path: /tmp/cassandra/build/test/logs
+ destination: logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ repeated_upgrade_dtest:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
- pip3 install --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Run repeated Python dtest
+ no_output_timeout: 15m
+ command: |
+ if [ "${REPEATED_UPGRADE_DTEST_NAME}" == "<nil>" ]; then
+ echo "Repeated dtest name hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" == "<nil>" ]; then
+ echo "Repeated dtest count hasn't been defined, exiting without running any test"
+ elif [ "${REPEATED_UPGRADE_DTEST_COUNT}" -le 0 ]; then
+ echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
+ else
+
+ # Calculate the number of test iterations to be run by the current parallel runner.
+ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
+ count=$((${REPEATED_UPGRADE_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
+ if (($CIRCLE_NODE_INDEX < (${REPEATED_UPGRADE_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
+ count=$((count+1))
+ fi
+
+ if (($count <= 0)); then
+ echo "No tests to run in this runner"
+ else
+ echo "Running ${REPEATED_UPGRADE_DTEST_NAME} $count times"
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+
+ stop_on_failure_arg=""
+ if ${REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE}; then
+ stop_on_failure_arg="-x"
+ fi
+
+ vnodes_args=""
+ if false; then
+ vnodes_args="--use-vnodes --num-tokens=16"
+ fi
+
+ upgrade_arg=""
+ if true; then
+ upgrade_arg="--execute-upgrade-tests"
+ fi
+
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg $upgrade_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir ${REPEATED_UPGRADE_DTEST_NAME} | tee /tmp/dtest/stdout.txt
+ fi
+ fi
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_logs
+ environment:
- - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - ANT_HOME: /usr/share/ant
++ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
++ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_UPGRADE_DTEST_NAME: null
+ - REPEATED_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
+ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
+ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
+ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
- j8_repeated_dtest:
+ j8_cqlsh-dtests-py38-no-vnodes:
docker:
- image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
resource_class: medium
@@@ -368,1938 -503,13 +686,2095 @@@
path: /tmp/results
- store_artifacts:
path: /tmp/dtest
- destination: dtest
+ destination: dtest_j8_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j8_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ j11_cqlsh-dtests-py3-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_with_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
+ - run:
+ name: Run dtests (j11_with_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
+ export CQLSH_PYTHON=/usr/bin/python3.6
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
+ j11_cqlsh-dtests-py3-no-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
+ name: Determine Tests to Run (j11_without_vnodes)
+ no_output_timeout: 5m
+ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource- [...]
+ - run:
+ name: Run dtests (j11_without_vnodes)
+ no_output_timeout: 15m
+ command: |
+ echo "cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt"
+ cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt
+
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
+ export CQLSH_PYTHON=/usr/bin/python3.6
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
+ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_without_vnodes_final.txt`
+ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
+ destination: dtest_j11_without_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
+ destination: dtest_j11_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
- j11_repeated-dtest:
++ j11_cqlsh-dtests-py38-with-vnodes:
+ docker:
+ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
- name: Log Environment Information
- command: |
- echo '*** id ***'
- id
- echo '*** cat /proc/cpuinfo ***'
- cat /proc/cpuinfo
- echo '*** free -m ***'
- free -m
- echo '*** df -m ***'
- df -m
- echo '*** ifconfig -a ***'
- ifconfig -a
- echo '*** uname -a ***'
- uname -a
- echo '*** mount ***'
- mount
- echo '*** env ***'
- env
- echo '*** java ***'
- which java
- java -version
- - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
- source ~/env3.6/bin/activate
++ source ~/env3.8/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Run repeated dtest
++ name: Determine Tests to Run (j11_with_vnodes)
++ no_output_timeout: 5m
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
++ - run:
++ name: Run dtests (j11_with_vnodes)
+ no_output_timeout: 15m
+ command: |
- if [ "$REPEATED_DTEST_NAME" == "<nil>" ]; then
- echo "Repeated dtest name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_DTEST_COUNT" == "<nil>" ]; then
- echo "Repeated dtest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_DTEST_COUNT" -le 0 ]; then
- echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
- else
-
- # Calculate the number of test iterations to be run by the current parallel runner.
- # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_DTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_DTEST_COUNT % CIRCLE_NODE_TOTAL))); then
- count=$((count+1))
- fi
-
- if (($count <= 0)); then
- echo "No tests to run in this runner"
- else
- echo "Running $REPEATED_DTEST_NAME $count times"
-
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
-
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
-
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
++ echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
+
- stop_on_failure_arg=""
- if $REPEATED_UTEST_STOP_ON_FAILURE; then
- stop_on_failure_arg="-x"
- fi
++ source ~/env3.8/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then
++ export CQLSH_PYTHON=/usr/bin/python3.8
++ fi
+
- vnodes_args=""
- if $REPEATED_DTEST_VNODES; then
- vnodes_args="--use-vnodes --num-tokens=16"
- fi
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
+
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- set -o pipefail && cd ~/cassandra-dtest && pytest $vnodes_args --count=$count $stop_on_failure_arg --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $REPEATED_DTEST_NAME | tee /tmp/dtest/stdout.txt
- fi
- fi
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
- destination: dtest
++ destination: dtest_j11_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
- destination: dtest_logs
++ destination: dtest_j11_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
+ - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - CASSANDRA_USE_JDK11: true
- j11_cqlsh-dtests-py38-with-vnodes:
++ j8_cqlsh-dtests-py3-with-vnodes:
+ docker:
- - image: apache/cassandra-testing-ubuntu2004-java11:20210304
++ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
- source ~/env3.8/bin/activate
++ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Determine Tests to Run (j11_with_vnodes)
++ name: Determine Tests to Run (j8_with_vnodes)
+ no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.8/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j11_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip [...]
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip- [...]
+ - run:
- name: Run dtests (j11_with_vnodes)
++ name: Run dtests (j8_with_vnodes)
+ no_output_timeout: 15m
+ command: |
- echo "cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt"
- cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt
++ echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt
+
- source ~/env3.8/bin/activate
++ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
- if [ -n 'CQLSH_PYTHON=/usr/bin/python3.8' ]; then
- export CQLSH_PYTHON=/usr/bin/python3.8
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
++ export CQLSH_PYTHON=/usr/bin/python3.6
+ fi
+
+ java -version
+ cd ~/cassandra-dtest
+ mkdir -p /tmp/dtest
+
+ echo "env: $(env)"
+ echo "** done env"
+ mkdir -p /tmp/results/dtests
+ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j11_with_vnodes_final.txt`
- set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j11_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
+ path: /tmp/results
+ - store_artifacts:
+ path: /tmp/dtest
- destination: dtest_j11_with_vnodes
++ destination: dtest_j8_with_vnodes
+ - store_artifacts:
+ path: ~/cassandra-dtest/logs
- destination: dtest_j11_with_vnodes_logs
++ destination: dtest_j8_with_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - CASSANDRA_USE_JDK11: true
- j11_repeated-utest:
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ j8_cqlsh-dtests-py2-no-vnodes:
+ docker:
- - image: apache/cassandra-testing-ubuntu2004-java11:20210304
++ - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
- name: Log Environment Information
++ name: Clone Cassandra dtest Repository (via git)
+ command: |
- echo '*** id ***'
- id
- echo '*** cat /proc/cpuinfo ***'
- cat /proc/cpuinfo
- echo '*** free -m ***'
- free -m
- echo '*** df -m ***'
- df -m
- echo '*** ifconfig -a ***'
- ifconfig -a
- echo '*** uname -a ***'
- uname -a
- echo '*** mount ***'
- mount
- echo '*** env ***'
- env
- echo '*** java ***'
- which java
- java -version
++ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
- name: Run repeated utest
++ name: Configure virtualenv and python Dependencies
++ command: |
++ # note, this should be super quick as all dependencies should be pre-installed in the docker image
++ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
++ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
++ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
++ pip3 uninstall -y cqlsh
++ pip3 freeze
++ - run:
++ name: Determine Tests to Run (j8_without_vnodes)
++ no_output_timeout: 5m
++ command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_without_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --skip-resource-i [...]
++ - run:
++ name: Run dtests (j8_without_vnodes)
+ no_output_timeout: 15m
+ command: |
- if [ "$REPEATED_UTEST_CLASS" == "<nil>" ]; then
- echo "Repeated utest class name hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" == "<nil>" ]; then
- echo "Repeated utest count hasn't been defined, exiting without running any test"
- elif [ "$REPEATED_UTEST_COUNT" -le 0 ]; then
- echo "Repeated utest count is lesser or equals than zero, exiting without running any test"
- else
-
- # Calculate the number of test iterations to be run by the current parallel runner.
- # Since we are running the same test multiple times there is no need to use `circleci tests split`.
- count=$((REPEATED_UTEST_COUNT / CIRCLE_NODE_TOTAL))
- if (($CIRCLE_NODE_INDEX < (REPEATED_UTEST_COUNT % CIRCLE_NODE_TOTAL))); then
- count=$((count+1))
- fi
-
- if (($count <= 0)); then
- echo "No tests to run in this runner"
- else
- echo "Running $REPEATED_UTEST_TARGET $REPEATED_UTEST_CLASS $REPEATED_UTEST_METHODS $count times"
-
- set -x
- export PATH=$JAVA_HOME/bin:$PATH
- time mv ~/cassandra /tmp
- cd /tmp/cassandra
- if [ -d ~/dtest_jars ]; then
- cp ~/dtest_jars/dtest* /tmp/cassandra/build/
- fi
-
- target=$REPEATED_UTEST_TARGET
- class_path=$REPEATED_UTEST_CLASS
- class_name="${class_path##*.}"
-
- # Prepare the -Dtest.name argument.
- # It can be the fully qualified class name or the short class name, depending on the target.
- if [[ $target == "test" || \
- $target == "test-cdc" || \
- $target == "test-compression" || \
- $target == "test-system-keyspace-directory" ]]; then
- name="-Dtest.name=$class_name"
- else
- name="-Dtest.name=$class_path"
- fi
-
- # Prepare the -Dtest.methods argument, which is optional
- if [ "$REPEATED_UTEST_METHODS" == "<nil>" ]; then
- methods=""
- else
- methods="-Dtest.methods=$REPEATED_UTEST_METHODS"
- fi
-
- # Run the test target as many times as requested collecting the exit code,
- # stopping the iteration only if REPEATED_UTEST_STOP_ON_FAILURE is set.
- exit_code="$?"
- for i in $(seq -w 1 $count); do
-
- echo "Running test iteration $i of $count"
-
- # run the test
- status="passes"
- if !( set -o pipefail && ant $target $name $methods -Dno-build-test=true | tee stdout.txt ); then
- status="fails"
- exit_code=1
- fi
-
- # move the stdout output file
- dest=/tmp/results/repeated_utest/stdout/${status}/${i}
- mkdir -p $dest
- mv stdout.txt $dest/${REPEATED_UTEST_TARGET}-${REPEATED_UTEST_CLASS}.txt
-
- # move the XML output files
- source=build/test/output
- dest=/tmp/results/repeated_utest/output/${status}/${i}
- mkdir -p $dest
- if [[ -d $source && -n "$(ls $source)" ]]; then
- mv $source/* $dest/
- fi
++ echo "cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt"
++ cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt
+
- # move the log files
- source=build/test/logs
- dest=/tmp/results/repeated_utest/logs/${status}/${i}
- mkdir -p $dest
- if [[ -d $source && -n "$(ls $source)" ]]; then
- mv $source/* $dest/
- fi
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++ if [ -n 'CQLSH_PYTHON=/usr/bin/python2.7' ]; then
++ export CQLSH_PYTHON=/usr/bin/python2.7
++ fi
+
- # maybe stop iterations on test failure
- if [[ $REPEATED_UTEST_STOP_ON_FAILURE = true ]] && (( $exit_code > 0 )); then
- break
- fi
- done
++ java -version
++ cd ~/cassandra-dtest
++ mkdir -p /tmp/dtest
+
- (exit ${exit_code})
- fi
- fi
++ echo "env: $(env)"
++ echo "** done env"
++ mkdir -p /tmp/results/dtests
++ # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
++ export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_without_vnodes_final.txt`
++ set -o pipefail && cd ~/cassandra-dtest && pytest --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_without_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
+ - store_test_results:
- path: /tmp/results/repeated_utest/output
- - store_artifacts:
- path: /tmp/results/repeated_utest/stdout
- destination: stdout
++ path: /tmp/results
+ - store_artifacts:
- path: /tmp/results/repeated_utest/output
- destination: junitxml
++ path: /tmp/dtest
++ destination: dtest_j8_without_vnodes
+ - store_artifacts:
- path: /tmp/results/repeated_utest/logs
- destination: logs
++ path: ~/cassandra-dtest/logs
++ destination: dtest_j8_without_vnodes_logs
+ environment:
+ - ANT_HOME: /usr/share/ant
+ - JAVA11_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+ - JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64
+ - LANG: en_US.UTF-8
+ - KEEP_TEST_DIR: true
+ - DEFAULT_DIR: /home/cassandra/cassandra-dtest
+ - PYTHONIOENCODING: utf-8
+ - PYTHONUNBUFFERED: true
+ - CASS_DRIVER_NO_EXTENSIONS: true
+ - CASS_DRIVER_NO_CYTHON: true
+ - CASSANDRA_SKIP_SYNC: true
+ - DTEST_REPO: git://github.com/apache/cassandra-dtest.git
+ - DTEST_BRANCH: trunk
+ - CCM_MAX_HEAP_SIZE: 1024M
+ - CCM_HEAP_NEWSIZE: 256M
+ - REPEATED_UTEST_TARGET: testsome
+ - REPEATED_UTEST_CLASS: null
+ - REPEATED_UTEST_METHODS: null
+ - REPEATED_UTEST_COUNT: 100
+ - REPEATED_UTEST_STOP_ON_FAILURE: false
+ - REPEATED_DTEST_NAME: null
+ - REPEATED_DTEST_VNODES: false
+ - REPEATED_DTEST_COUNT: 100
+ - REPEATED_DTEST_STOP_ON_FAILURE: false
- - JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - JDK_HOME: /usr/lib/jvm/java-11-openjdk-amd64
- - CASSANDRA_USE_JDK11: true
- j8_cqlsh-dtests-py3-with-vnodes:
++ - REPEATED_UPGRADE_DTEST_NAME: null
++ - REPEATED_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - REPEATED_JVM_UPGRADE_DTEST_CLASS: null
++ - REPEATED_JVM_UPGRADE_DTEST_METHODS: null
++ - REPEATED_JVM_UPGRADE_DTEST_COUNT: 100
++ - REPEATED_JVM_UPGRADE_DTEST_STOP_ON_FAILURE: false
++ - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64
++ j11_repeated_dtest:
+ docker:
- - image: apache/cassandra-testing-ubuntu2004-java11-w-dependencies:20210304
++ - image: apache/cassandra-testing-ubuntu2004-java11:20210304
+ resource_class: medium
+ working_directory: ~/
+ shell: /bin/bash -eo pipefail -l
+ parallelism: 4
+ steps:
+ - attach_workspace:
+ at: /home/cassandra
+ - run:
++ name: Log Environment Information
++ command: |
++ echo '*** id ***'
++ id
++ echo '*** cat /proc/cpuinfo ***'
++ cat /proc/cpuinfo
++ echo '*** free -m ***'
++ free -m
++ echo '*** df -m ***'
++ df -m
++ echo '*** ifconfig -a ***'
++ ifconfig -a
++ echo '*** uname -a ***'
++ uname -a
++ echo '*** mount ***'
++ mount
++ echo '*** env ***'
++ env
++ echo '*** java ***'
++ which java
++ java -version
++ - run:
+ name: Clone Cassandra dtest Repository (via git)
+ command: |
+ git clone --single-branch --branch $DTEST_BRANCH --depth 1 $DTEST_REPO ~/cassandra-dtest
+ - run:
+ name: Configure virtualenv and python Dependencies
+ command: |
+ # note, this should be super quick as all dependencies should be pre-installed in the docker image
+ # if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated
+ # we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and
+ # rebuild the docker image! (it automatically pulls the latest requirements.txt on build)
+ source ~/env3.6/bin/activate
+ export PATH=$JAVA_HOME/bin:$PATH
+ pip3 install --exists-action w --upgrade -r ~/cassandra-dtest/requirements.txt
+ pip3 uninstall -y cqlsh
+ pip3 freeze
+ - run:
- name: Determine Tests to Run (j8_with_vnodes)
- no_output_timeout: 5m
- command: "# reminder: this code (along with all the steps) is independently executed on every circle container\n# so the goal here is to get the circleci script to return the tests *this* container will run\n# which we do via the `circleci` cli tool.\n\ncd cassandra-dtest\nsource ~/env3.6/bin/activate\nexport PATH=$JAVA_HOME/bin:$PATH\n\nif [ -n '' ]; then\n export \nfi\n\necho \"***Collected DTests (j8_with_vnodes)***\"\nset -eo pipefail && ./run_dtests.py --use-vnodes --skip- [...]
- - run:
- name: Run dtests (j8_with_vnodes)
++ name: Run repeated Python dtest
+ no_output_timeout: 15m
+ command: |
- echo "cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt"
- cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt
++ if [ "${REPEATED_DTEST_NAME}" == "<nil>" ]; then
++ echo "Repeated dtest name hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_DTEST_COUNT}" == "<nil>" ]; then
++ echo "Repeated dtest count hasn't been defined, exiting without running any test"
++ elif [ "${REPEATED_DTEST_COUNT}" -le 0 ]; then
++ echo "Repeated dtest count is lesser or equals than zero, exiting without running any test"
++ else
+
- source ~/env3.6/bin/activate
- export PATH=$JAVA_HOME/bin:$PATH
- if [ -n 'CQLSH_PYTHON=/usr/bin/python3.6' ]; then
- export CQLSH_PYTHON=/usr/bin/python3.6
- fi
++ # Calculate the number of test iterations to be run by the current parallel runner.
++ # Since we are running the same test multiple times there is no need to use `circleci tests split`.
++ count=$((${REPEATED_DTEST_COUNT} / CIRCLE_NODE_TOTAL))
++ if (($CIRCLE_NODE_INDEX < (${REPEATED_DTEST_COUNT} % CIRCLE_NODE_TOTAL))); then
++ count=$((count+1))
++ fi
+
- java -version
- cd ~/cassandra-dtest
- mkdir -p /tmp/dtest
++ if (($count <= 0)); then
++ echo "No tests to run in this runner"
++ else
++ echo "Running ${REPEATED_DTEST_NAME} $count times"
+
- echo "env: $(env)"
- echo "** done env"
- mkdir -p /tmp/results/dtests
- # we need the "set -o pipefail" here so that the exit code that circleci will actually use is from pytest and not the exit code from tee
- export SPLIT_TESTS=`cat /tmp/split_dtest_tests_j8_with_vnodes_final.txt`
- set -o pipefail && cd ~/cassandra-dtest && pytest --use-vnodes --num-tokens=16 --skip-resource-intensive-tests --log-cli-level=DEBUG --junit-xml=/tmp/results/dtests/pytest_result_j8_with_vnodes.xml -s --cassandra-dir=/home/cassandra/cassandra --keep-test-dir $SPLIT_TESTS 2>&1 | tee /tmp/dtest/stdout.txt
++ source ~/env3.6/bin/activate
++ export PATH=$JAVA_HOME/bin:$PATH
++
++ java -version
++ cd ~/cassandra-dtest
... 7276 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org