You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2024/11/20 04:40:36 UTC

(spark) branch master updated: [SPARK-50331][INFRA][FOLLOW-UP] Skip Torch/DeepSpeed tests in MacOS PySpark Daily test

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 19b8250588bf [SPARK-50331][INFRA][FOLLOW-UP] Skip Torch/DeepSpeed tests in MacOS PySpark Daily test
19b8250588bf is described below

commit 19b8250588bf5a4be39a53c4c60fcb5f85a8ec1e
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Wed Nov 20 12:40:16 2024 +0800

    [SPARK-50331][INFRA][FOLLOW-UP] Skip Torch/DeepSpeed tests in MacOS PySpark Daily test
    
    ### What changes were proposed in this pull request?
    Skip Torch/DeepSpeed tests in MacOS PySpark Daily test
    
    https://github.com/apache/spark/actions/runs/11921746968/job/33226552068
    
    ### Why are the changes needed?
    we don't need to test them on MacOS
    
    ### Does this PR introduce _any_ user-facing change?
    no, test only
    
    ### How was this patch tested?
    They should be skipped due to no installation:
    
    https://github.com/apache/spark/blob/b724955bb7c2995fbbd9c7fe550e44f16397cb5b/python/pyspark/ml/torch/tests/test_data_loader.py#L36
    
    manually test in my local MacOS:
    ```
    (spark_312) ➜  spark git:(master) python/run-tests -k --python-executables python3 --testnames 'pyspark.ml.torch.tests.test_data_loader'
    Running PySpark tests. Output is in /Users/ruifeng.zheng/Dev/spark/python/unit-tests.log
    Will test against the following Python executables: ['python3']
    Will test the following Python tests: ['pyspark.ml.torch.tests.test_data_loader']
    python3 python_implementation is CPython
    python3 version is: Python 3.12.7
    Starting test(python3): pyspark.ml.torch.tests.test_data_loader (temp output: /Users/ruifeng.zheng/Dev/spark/python/target/c7bf947a-a746-4519-b475-aed24a1f8cec/python3__pyspark.ml.torch.tests.test_data_loader__1yq21_l5.log)
    Finished test(python3): pyspark.ml.torch.tests.test_data_loader (0s) ... 1 tests were skipped
    Tests passed in 0 seconds
    
    Skipped tests in pyspark.ml.torch.tests.test_data_loader with python3:
          test_data_loader (pyspark.ml.torch.tests.test_data_loader.TorchDistributorDataLoaderUnitTests.test_data_loader) ... skip (0.001s)
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #48900 from zhengruifeng/macos_py_lib.
    
    Authored-by: Ruifeng Zheng <ru...@apache.org>
    Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
 .github/workflows/python_macos_test.yml | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/.github/workflows/python_macos_test.yml b/.github/workflows/python_macos_test.yml
index 32a8e21801db..cca133dab541 100644
--- a/.github/workflows/python_macos_test.yml
+++ b/.github/workflows/python_macos_test.yml
@@ -133,11 +133,8 @@ jobs:
         run: |
           python${{matrix.python}} -m pip install --ignore-installed 'blinker>=1.6.2'
           python${{matrix.python}} -m pip install --ignore-installed 'six==1.16.0'
-          python${{matrix.python}} -m pip install py-cpuinfo && \
           python${{matrix.python}} -m pip install numpy 'pyarrow>=15.0.0' 'six==1.16.0' 'pandas==2.2.3' scipy 'plotly>=4.8' 'mlflow>=2.8.1' coverage matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2' unittest-xml-reporting && \
           python${{matrix.python}} -m pip install 'grpcio==1.67.0' 'grpcio-status==1.67.0' 'protobuf==5.28.3' 'googleapis-common-protos==1.65.0' 'graphviz==0.20.3' && \
-          python${{matrix.python}} -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu && \
-          python${{matrix.python}} -m pip install deepspeed torcheval && \
           python${{matrix.python}} -m pip cache purge && \
           python${{matrix.python}} -m pip list
       # Run the tests.


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org