You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2023/09/04 00:46:02 UTC

[spark] branch master updated: [SPARK-44667][INFRA][FOLLOWUP] Uninstall `deepspeed` libraries for non-ML jobs

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 6d1be3a389e [SPARK-44667][INFRA][FOLLOWUP] Uninstall `deepspeed` libraries for non-ML jobs
6d1be3a389e is described below

commit 6d1be3a389e407cb1422fa50e4b29270415d8fe5
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Sep 4 09:45:49 2023 +0900

    [SPARK-44667][INFRA][FOLLOWUP] Uninstall `deepspeed` libraries for non-ML jobs
    
    ### What changes were proposed in this pull request?
    Uninstall `deepspeed` libraries for non-ML jobs
    
    ### Why are the changes needed?
    after https://github.com/apache/spark/pull/42334, `deepspeed` was introduced as a dependency, which is not needed for non-ML jobs
    
    ### Does this PR introduce _any_ user-facing change?
    no, dev-only
    
    ### How was this patch tested?
    CI
    
    ### Was this patch authored or co-authored using generative AI tooling?
    NO
    
    Closes #42768 from zhengruifeng/uninstall_deepspeed.
    
    Authored-by: Ruifeng Zheng <ru...@apache.org>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 .github/workflows/build_and_test.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index beb5a7772b7..f0bd65bcf41 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -425,7 +425,7 @@ jobs:
       run: |
         if [[ "$MODULES_TO_TEST" != *"pyspark-ml"* ]] && [[ "$BRANCH" != "branch-3.5" ]]; then
           # uninstall libraries dedicated for ML testing
-          python3.9 -m pip uninstall -y torch torchvision torcheval torchtnt tensorboard mlflow
+          python3.9 -m pip uninstall -y torch torchvision torcheval torchtnt tensorboard mlflow deepspeed
         fi
         if [ -f ./dev/free_disk_space_container ]; then
           ./dev/free_disk_space_container


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org