You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/07/28 19:30:05 UTC

[GitHub] [tvm] Lunderberg commented on a change in pull request #8576: Enable pytest-xdist for TVM CI

Lunderberg commented on a change in pull request #8576:
URL: https://github.com/apache/tvm/pull/8576#discussion_r678589727



##########
File path: tests/scripts/setup-pytest-env.sh
##########
@@ -32,7 +32,33 @@ export PYTHONPATH="${TVM_PATH}/python"
 export TVM_PYTEST_RESULT_DIR="${TVM_PATH}/build/pytest-results"
 mkdir -p "${TVM_PYTEST_RESULT_DIR}"
 
+if [ -n "${CI_PYTEST_NUM_CPUS-}" ]; then
+    PYTEST_NUM_CPUS=${CI_PYTEST_NUM_CPUS}
+else
+    PYTEST_NUM_CPUS=$(nproc)
+    if [ -z "${PYTEST_NUM_CPUS}" ]; then
+        PYTEST_NUM_CPUS=1
+    elif [ ${PYTEST_NUM_CPUS} -gt 1 ]; then
+        PYTEST_NUM_CPUS=$(expr ${PYTEST_NUM_CPUS} - 1)  # Don't nuke interactive work.
+    fi
+fi
+
+if [ ${PYTEST_NUM_CPUS} -gt 8 ]; then
+    PYTEST_NUM_CPUS=8  # It usually doesn't make sense to launch > 8 workers
+fi
+
+# DNS: remove after we actually fix up CI_PYTEST_NUM_CPUS in Jenkinsfile
+if [ ${PYTEST_NUM_CPUS} -gt 2 ]; then
+    PYTEST_NUM_CPUS=2  # Fix to 2 CPUs for Jenkins
+fi
+
+
 function run_pytest() {
+    local extra_args=( )
+    if [ "$1" == "--parallel" ]; then

Review comment:
       Since the documentation [currently recommends](https://github.com/apache/tvm/blob/main/docs/contribute/pull_request.rst#L121) using `task_python_unittest.sh` to run the unit tests, we should either make sure that location also has `pytest-xdist` in the list of packages to install in order to run the tests, or check whether `pytest-xdist` is installed by wrapping this in `if python3 -c "import xdist" > /dev/null 2>&1; then`.

##########
File path: tests/scripts/task_python_unittest.sh
##########
@@ -31,9 +31,9 @@ if [ -z "${TVM_UNITTEST_TESTSUITE_NAME:-}" ]; then
 fi
 
 # First run minimal test on both ctypes and cython.
-run_pytest ctypes ${TVM_UNITTEST_TESTSUITE_NAME}-platform-minimal-test tests/python/all-platform-minimal-test
-run_pytest cython ${TVM_UNITTEST_TESTSUITE_NAME}-platform-minimal-test tests/python/all-platform-minimal-test
+run_pytest --parallel ctypes ${TVM_UNITTEST_TESTSUITE_NAME}-platform-minimal-test tests/python/all-platform-minimal-test

Review comment:
       Looks like this applies to everything in the unittest directory, including `test_tvm_testing_features.py`.  I don't see the [explicit ordering](https://github.com/apache/tvm/pull/8343#discussion_r660038825) that will be required for some of those tests, so we should add it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org