You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/06/27 19:55:09 UTC
[spark] branch branch-3.1 updated: [SPARK-39621][PYTHON][TESTS] Make `run-tests.py` robust by avoiding `rmtree` on MacOS
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new b324b610354 [SPARK-39621][PYTHON][TESTS] Make `run-tests.py` robust by avoiding `rmtree` on MacOS
b324b610354 is described below
commit b324b610354bc2909a2a93431706cd714da6065a
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Mon Jun 27 12:52:11 2022 -0700
[SPARK-39621][PYTHON][TESTS] Make `run-tests.py` robust by avoiding `rmtree` on MacOS
This PR aims to make `run-tests.py` robust by avoiding `rmtree` on MacOS.
There exists a race condition in Python and it causes flakiness in MacOS
- https://bugs.python.org/issue29699
- https://github.com/python/cpython/issues/73885
No.
After passing CIs, this should be tested on MacOS.
Closes #37010 from dongjoon-hyun/SPARK-39621.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 432945db743965f1beb59e3a001605335ca2df83)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
python/run-tests.py | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/python/run-tests.py b/python/run-tests.py
index a13828d81f0..a322aef055b 100755
--- a/python/run-tests.py
+++ b/python/run-tests.py
@@ -20,6 +20,7 @@
import logging
from argparse import ArgumentParser
import os
+import platform
import re
import shutil
import subprocess
@@ -105,7 +106,12 @@ def run_individual_python_test(target_dir, test_name, pyspark_python):
retcode = subprocess.Popen(
[os.path.join(SPARK_HOME, "bin/pyspark")] + test_name.split(),
stderr=per_test_output, stdout=per_test_output, env=env).wait()
- shutil.rmtree(tmp_dir, ignore_errors=True)
+ # There exists a race condition in Python and it causes flakiness in MacOS
+ # https://github.com/python/cpython/issues/73885
+ if platform.system() == "Darwin":
+ os.system("rm -rf " + tmp_dir)
+ else:
+ shutil.rmtree(tmp_dir, ignore_errors=True)
except:
LOGGER.exception("Got exception while running %s with %s", test_name, pyspark_python)
# Here, we use os._exit() instead of sys.exit() in order to force Python to exit even if
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org