You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2025/02/17 06:28:49 UTC

(spark) branch master updated: [SPARK-51232][PYTHON][DOCS] Remove PySpark 3.3 and older logic from `binder/postBuild`

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 8423d74a76f4 [SPARK-51232][PYTHON][DOCS] Remove PySpark 3.3 and older logic from `binder/postBuild`
8423d74a76f4 is described below

commit 8423d74a76f4a5105d4480e8691cd2ac11dc99ba
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun Feb 16 22:28:08 2025 -0800

    [SPARK-51232][PYTHON][DOCS] Remove PySpark 3.3 and older logic from `binder/postBuild`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to remove PySpark 3.3 and older logic from `binder/postBuild` because they are the end of life already and didn't have `PySpark Connect` module.
    
    ### Why are the changes needed?
    
    After simplifying by removing legacy PySpark without `PySpark Connect` module, we are going to add additional logic to handle `PySpark Connect 4.0+` because they are different from `PySpark Connect 3.4 ~ 3.5`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, this is a kind of documentation via Binder service.
    
    ### How was this patch tested?
    
    Manual review.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #49973 from dongjoon-hyun/SPARK-51232.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 binder/postBuild | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/binder/postBuild b/binder/postBuild
index 0b326f62e8f5..92f4c70ffff5 100755
--- a/binder/postBuild
+++ b/binder/postBuild
@@ -37,11 +37,7 @@ else
   SPECIFIER="<="
 fi
 
-if [[ ! $VERSION < "3.4.0" ]]; then
-  pip install plotly "pandas<2.0.0" "pyspark[sql,ml,mllib,pandas_on_spark,connect]$SPECIFIER$VERSION"
-else
-  pip install plotly "pandas<2.0.0" "pyspark[sql,ml,mllib,pandas_on_spark]$SPECIFIER$VERSION"
-fi
+pip install plotly "pandas<2.0.0" "pyspark[sql,ml,mllib,pandas_on_spark,connect]$SPECIFIER$VERSION"
 
 # Set 'PYARROW_IGNORE_TIMEZONE' to suppress warnings from PyArrow.
 echo "export PYARROW_IGNORE_TIMEZONE=1" >> ~/.profile


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org