[spark] branch master updated: [SPARK-39621][PYTHON][TESTS] Make `run-tests.py` robust by avoiding `rmtree` on MacOS

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 432945db743 [SPARK-39621][PYTHON][TESTS] Make `run-tests.py` robust by avoiding `rmtree` on MacOS
432945db743 is described below

commit 432945db743965f1beb59e3a001605335ca2df83
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Mon Jun 27 12:52:11 2022 -0700

    [SPARK-39621][PYTHON][TESTS] Make `run-tests.py` robust by avoiding `rmtree` on MacOS
    
    ### What changes were proposed in this pull request?
    
    This PR aims to make `run-tests.py` robust by avoiding `rmtree` on MacOS.
    
    ### Why are the changes needed?
    
    There exists a race condition in Python and it causes flakiness in MacOS
    - https://bugs.python.org/issue29699
    - https://github.com/python/cpython/issues/73885
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    After passing CIs, this should be tested on MacOS.
    
    Closes #37010 from dongjoon-hyun/SPARK-39621.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 python/run-tests.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/python/run-tests.py b/python/run-tests.py
index 6fce3f9a1ce..1e3c1e8544d 100755
--- a/python/run-tests.py
+++ b/python/run-tests.py
@@ -20,6 +20,7 @@
 import logging
 from argparse import ArgumentParser
 import os
+import platform
 import re
 import shutil
 import subprocess
@@ -113,7 +114,12 @@ def run_individual_python_test(target_dir, test_name, pyspark_python):
         retcode = subprocess.Popen(
             [os.path.join(SPARK_HOME, "bin/pyspark")] + test_name.split(),
             stderr=per_test_output, stdout=per_test_output, env=env).wait()
-        shutil.rmtree(tmp_dir, ignore_errors=True)
+        # There exists a race condition in Python and it causes flakiness in MacOS
+        # https://github.com/python/cpython/issues/73885
+        if platform.system() == "Darwin":
+            os.system("rm -rf " + tmp_dir)
+        else:
+            shutil.rmtree(tmp_dir, ignore_errors=True)
     except BaseException:
         LOGGER.exception("Got exception while running %s with %s", test_name, pyspark_python)
         # Here, we use os._exit() instead of sys.exit() in order to force Python to exit even if


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org