You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/07/11 17:23:36 UTC

[GitHub] [tvm] driazati opened a new pull request, #12055: [ci] Re-run failed tests on failure

driazati opened a new pull request, #12055:
URL: https://github.com/apache/tvm/pull/12055

   This uses [pytest-rerunfailures](https://github.com/pytest-dev/pytest-rerunfailures) to retry failed tests. This should help alleviate flakiness for issues like segfaults in ethosu tests or flaky numerics. This obviously isn't as good as fixing the tests themselves, but it's an easy way to fix the most pressing issue (the cost of developer time to wait for, check, and re-run CI) while keeping the signal from the times the tests do fail consistently.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] areusch merged pull request #12055: [ci] Re-run failed tests on failure

Posted by GitBox <gi...@apache.org>.
areusch merged PR #12055:
URL: https://github.com/apache/tvm/pull/12055


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] driazati commented on a diff in pull request #12055: [ci] Re-run failed tests on failure

Posted by GitBox <gi...@apache.org>.
driazati commented on code in PR #12055:
URL: https://github.com/apache/tvm/pull/12055#discussion_r919310220


##########
tests/scripts/setup-pytest-env.sh:
##########
@@ -51,19 +54,24 @@ function run_pytest() {
     shift
     local test_suite_name="$1"
     shift
+    extra_args=( "$@" )
     if [ -z "${ffi_type}" -o -z "${test_suite_name}" ]; then
-        echo "error: run_pytest called incorrectly: run_pytest ${ffi_type} ${test_suite_name} $@"
+        echo "error: run_pytest called incorrectly: run_pytest ${ffi_type} ${test_suite_name} $extra_args"
         echo "usage: run_pytest <FFI_TYPE> <TEST_SUITE_NAME> [pytest args...]"
         exit 2
     fi
+    has_help=$(python3 -m pytest --help 2>&1 | grep 'reruns=' || true)
+    if [ -n "$has_help" ]; then
+        extra_args+=('--reruns=3')

Review Comment:
   the number is just arbitrary, though given that the flaky tests we do have come from a pretty consistent set of places I think 2 failures happening in a row is not that unlikely



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] github-actions[bot] commented on pull request #12055: [ci] Re-run failed tests on failure

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12055:
URL: https://github.com/apache/tvm/pull/12055#issuecomment-1181216548

   <!---docs-bot-comment-->
   
   Built docs for commit df4e21720a541a9ee9c93fd0945fee4bcade8197 can be found [here](https://pr-docs.tlcpack.ai/PR-12055/4/docs/index.html).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] driazati commented on a diff in pull request #12055: [ci] Re-run failed tests on failure

Posted by GitBox <gi...@apache.org>.
driazati commented on code in PR #12055:
URL: https://github.com/apache/tvm/pull/12055#discussion_r919309242


##########
tests/scripts/setup-pytest-env.sh:
##########
@@ -24,11 +24,14 @@ if [[ ! -z $CI_PYTEST_ADD_OPTIONS ]]; then
 else
     export PYTEST_ADDOPTS="-s -vv $PYTEST_ADDOPTS"
 fi
-set -u
+set -ux
 
 export TVM_PATH=`pwd`
 export PYTHONPATH="${TVM_PATH}/python"
 
+# TODO: Remove this line once https://github.com/apache/tvm/pull/12055 is merged into the Docker images
+python3 -m pip install --user pytest-rerunfailures==10.2

Review Comment:
   it's not a huge difference either way, but we've done this kind of thing before (with Jinja and `generate.py` and it turned out fine



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] areusch commented on a diff in pull request #12055: [ci] Re-run failed tests on failure

Posted by GitBox <gi...@apache.org>.
areusch commented on code in PR #12055:
URL: https://github.com/apache/tvm/pull/12055#discussion_r919307370


##########
tests/scripts/setup-pytest-env.sh:
##########
@@ -51,19 +54,24 @@ function run_pytest() {
     shift
     local test_suite_name="$1"
     shift
+    extra_args=( "$@" )
     if [ -z "${ffi_type}" -o -z "${test_suite_name}" ]; then
-        echo "error: run_pytest called incorrectly: run_pytest ${ffi_type} ${test_suite_name} $@"
+        echo "error: run_pytest called incorrectly: run_pytest ${ffi_type} ${test_suite_name} $extra_args"
         echo "usage: run_pytest <FFI_TYPE> <TEST_SUITE_NAME> [pytest args...]"
         exit 2
     fi
+    has_help=$(python3 -m pytest --help 2>&1 | grep 'reruns=' || true)
+    if [ -n "$has_help" ]; then
+        extra_args+=('--reruns=3')

Review Comment:
   should we just do 1? why 3?



##########
tests/scripts/setup-pytest-env.sh:
##########
@@ -24,11 +24,14 @@ if [[ ! -z $CI_PYTEST_ADD_OPTIONS ]]; then
 else
     export PYTEST_ADDOPTS="-s -vv $PYTEST_ADDOPTS"
 fi
-set -u
+set -ux
 
 export TVM_PATH=`pwd`
 export PYTHONPATH="${TVM_PATH}/python"
 
+# TODO: Remove this line once https://github.com/apache/tvm/pull/12055 is merged into the Docker images
+python3 -m pip install --user pytest-rerunfailures==10.2

Review Comment:
   maybe to reduce risk we should just update the ci images and turn this on?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org