(spark) branch branch-4.x updated: [SPARK-57393] Build: PySpark and SparkR source distributions are missing LICENSE and NOTICE files

[hu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

asf-gitbox-commits pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 49689e847514 [SPARK-57393] Build: PySpark and SparkR source distributions are missing LICENSE and NOTICE files
49689e847514 is described below

commit 49689e8475142ac761adc3944e460ef6577df9bb
Author: Huaxin Gao <hu...@gmail.com>
AuthorDate: Thu Jun 11 15:59:10 2026 -0700

    [SPARK-57393] Build: PySpark and SparkR source distributions are missing LICENSE and NOTICE files
    
    ### What changes were proposed in this pull request?
    
    Make the PySpark and SparkR source distributions include top-level LICENSE and NOTICE files:
    
    - Add include LICENSE / include NOTICE to python/MANIFEST.in.
    - In dev/make-distribution.sh, copy LICENSE/NOTICE into python/ and R/pkg/ before building the sdists (with an EXIT trap to clean them up).
    - The classic pyspark sdist bundles the assembly jars, so it ships the binary license variants (LICENSE-binary/NOTICE-binary plus the full licenses-binary set), mirroring the binary distribution. The pyspark_connect, pyspark_client, and SparkR artifacts bundle no jars, so they ship the plain source LICENSE/NOTICE. packaging/classic/setup.py falls back to licenses/ when licenses-binary/ is absent (RELEASE-mode builds).
    - During the SparkR build, + file LICENSE is added to DESCRIPTION temporarily (restored after the build) so R CMD check --as-cran does not warn that the bundled LICENSE is not mentioned. The committed DESCRIPTION is unchanged, so SparkR CI is unaffected. The "Non-standard file/directory found at top level: 'NOTICE'" NOTE cannot be silenced this way and is expected in release-build logs.
    - Add tar-listing regression guards after the sdist and SparkR builds that fail the release build if LICENSE or NOTICE ever goes missing again, instead of relying on an RC vote to catch it.
    
    ### Why are the changes needed?
    
    ASF release policy requires every distributed artifact to include LICENSE and NOTICE. The pyspark, pyspark_connect, pyspark_client, and SparkR source tarballs currently don't, which was raised as a -1 during the Spark 4.2.0 RC1 vote.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No
    
    ### How was this patch tested?
    
    Built the source distributions locally and confirmed each contains LICENSE and NOTICE at the package root
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Yes, drafted with assistance from Cursor.
    
    Closes #56453 from huaxingao/fix_license.
    
    Authored-by: Huaxin Gao <hu...@gmail.com>
    Signed-off-by: huaxin-gao_snow <hu...@snowflake.com>
    (cherry picked from commit b88952cabfff54973957cdc21d3a548acf951ac1)
    Signed-off-by: huaxin-gao_snow <hu...@snowflake.com>
---
 R/.gitignore                      |  4 +++
 dev/make-distribution.sh          | 69 +++++++++++++++++++++++++++++++++++++++
 python/.gitignore                 |  3 ++
 python/MANIFEST.in                |  4 ++-
 python/packaging/classic/setup.py | 11 +++++--
 5 files changed, 88 insertions(+), 3 deletions(-)

diff --git a/R/.gitignore b/R/.gitignore
index c98504ab0778..693b2b7c6917 100644
--- a/R/.gitignore
+++ b/R/.gitignore
@@ -6,3 +6,7 @@ pkg/man
 pkg/html
 SparkR.Rcheck/
 SparkR_*.tar.gz
+# Transient copies made by dev/make-distribution.sh while building the package.
+pkg/LICENSE
+pkg/NOTICE
+DESCRIPTION.orig
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index 7cd9ea7889e8..e3bedf15e565 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -32,6 +32,21 @@ set -x
 SPARK_HOME="$(cd "`dirname "$0"`/.."; pwd)"
 DISTDIR="$SPARK_HOME/dist"
 
+# The Apache LICENSE and NOTICE are copied into the Python and R package
+# directories below so they are bundled into the source distributions. Remove
+# them on exit so a failed build does not leave stray files behind.
+function cleanup_dist_license_files {
+  rm -f "$SPARK_HOME/python/LICENSE" "$SPARK_HOME/python/NOTICE" \
+        "$SPARK_HOME/R/pkg/LICENSE" "$SPARK_HOME/R/pkg/NOTICE"
+  # Restore the SparkR DESCRIPTION if a release build patched it in place (see
+  # the R packaging section). Guards against an interrupted build leaving the
+  # tracked DESCRIPTION modified.
+  if [ -f "$SPARK_HOME/R/DESCRIPTION.orig" ]; then
+    mv -f "$SPARK_HOME/R/DESCRIPTION.orig" "$SPARK_HOME/R/pkg/DESCRIPTION"
+  fi
+}
+trap cleanup_dist_license_files EXIT
+
 MAKE_TGZ=false
 MAKE_PIP=false
 MAKE_R=false
@@ -259,9 +274,39 @@ if [ "$MAKE_PIP" == "true" ]; then
   pushd "$SPARK_HOME/python" > /dev/null
   # Delete the egg info file if it exists, this can cache older setup files.
   rm -rf pyspark.egg-info || echo "No existing egg info file, skipping deletion"
+  # Ship the Apache LICENSE and NOTICE inside the PySpark source distributions
+  # (see MANIFEST.in). These are removed again after the sdists are built.
+  #
+  # The classic pyspark sdist bundles the assembly jars (packaging/classic/setup.py
+  # builds a deps/jars symlink farm), so it ships the binary LICENSE/NOTICE that
+  # enumerate the bundled third-party jars' licenses, mirroring the binary
+  # distribution above. The connect and client sdists bundle no jars and ship the
+  # plain source LICENSE/NOTICE.
+  if [ -e "$SPARK_HOME/LICENSE-binary" ]; then
+    cp "$SPARK_HOME/LICENSE-binary" LICENSE
+    cp "$SPARK_HOME/NOTICE-binary" NOTICE
+  else
+    cp "$SPARK_HOME/LICENSE" LICENSE
+    cp "$SPARK_HOME/NOTICE" NOTICE
+  fi
   python3 packaging/classic/setup.py sdist
+
+  cp "$SPARK_HOME/LICENSE" LICENSE
+  cp "$SPARK_HOME/NOTICE" NOTICE
   python3 packaging/connect/setup.py sdist
   python3 packaging/client/setup.py sdist
+  rm -f LICENSE NOTICE
+
+  # Guard against regressions: every PySpark sdist must contain LICENSE and NOTICE
+  # at the package root. The missing files were only caught by a Spark 4.2.0 RC1
+  # vote -1 (SPARK-57393); fail the release build here instead of at vote time.
+  for f in dist/pyspark*.tar.gz; do
+    listing=$(tar tzf "$f")
+    for required in LICENSE NOTICE; do
+      grep -qE "^[^/]+/$required\$" <<< "$listing" || \
+        { echo "ERROR: $f is missing $required at the package root"; exit 1; }
+    done
+  done
   popd > /dev/null
 else
   echo "Skipping building python distribution package"
@@ -272,9 +317,33 @@ if [ "$MAKE_R" == "true" ]; then
   echo "Building R source package"
   R_PACKAGE_VERSION=`grep Version "$SPARK_HOME/R/pkg/DESCRIPTION" | awk '{print $NF}'`
   pushd "$SPARK_HOME/R" > /dev/null
+  # Ship the Apache LICENSE and NOTICE inside the SparkR source package. These
+  # are removed again after the package is built.
+  cp "$SPARK_HOME/LICENSE" pkg/LICENSE
+  cp "$SPARK_HOME/NOTICE" pkg/NOTICE
+  # Reference the bundled LICENSE from DESCRIPTION so `R CMD check --as-cran` does
+  # not emit "File LICENSE is not mentioned in the DESCRIPTION file". The committed
+  # DESCRIPTION is left untouched because SparkR CI runs check-cran.sh without the
+  # LICENSE file present; this edit is transient and restored after the build (and
+  # by the EXIT trap on failure). The backup lives outside pkg/ so R CMD check does
+  # not flag it as a non-standard file. NOTE: the "Non-standard file 'NOTICE'" note
+  # cannot be silenced this way and is expected.
+  cp pkg/DESCRIPTION "$SPARK_HOME/R/DESCRIPTION.orig"
+  sed 's/^License: Apache License (== 2.0)$/License: Apache License (== 2.0) + file LICENSE/' \
+    "$SPARK_HOME/R/DESCRIPTION.orig" > pkg/DESCRIPTION
   # Build source package and run full checks
   # Do not source the check-cran.sh - it should be run from where it is for it to set SPARK_HOME
   NO_TESTS=1 "$SPARK_HOME/R/check-cran.sh"
+  mv -f "$SPARK_HOME/R/DESCRIPTION.orig" pkg/DESCRIPTION
+  rm -f pkg/LICENSE pkg/NOTICE
+
+  # Guard against regressions: the SparkR source package must contain LICENSE and
+  # NOTICE at the package root (SPARK-57393).
+  listing=$(tar tzf "SparkR_$R_PACKAGE_VERSION.tar.gz")
+  for required in LICENSE NOTICE; do
+    grep -qE "^[^/]+/$required\$" <<< "$listing" || \
+      { echo "ERROR: SparkR source package is missing $required"; exit 1; }
+  done
 
   # Move R source package to match the Spark release version if the versions are not the same.
   # NOTE(shivaram): `mv` throws an error on Linux if source and destination are same file
diff --git a/python/.gitignore b/python/.gitignore
index 6d95b545fd56..b865a73038ce 100644
--- a/python/.gitignore
+++ b/python/.gitignore
@@ -7,3 +7,6 @@ build/
 dist/
 ./setup.py
 ./setup.cfg
+# Transient copies made by dev/make-distribution.sh while building the sdists.
+/LICENSE
+/NOTICE
diff --git a/python/MANIFEST.in b/python/MANIFEST.in
index 45c9dca8b474..82979a0344c3 100644
--- a/python/MANIFEST.in
+++ b/python/MANIFEST.in
@@ -21,10 +21,12 @@ recursive-include deps/jars *.jar
 graft deps/bin
 recursive-include deps/sbin spark-config.sh spark-daemon.sh start-history-server.sh stop-history-server.sh
 recursive-include deps/data *.data *.txt
-recursive-include deps/licenses *.txt
+graft deps/licenses
 recursive-include deps/examples *.py
 recursive-include lib *.zip
 include README.md
+include LICENSE
+include NOTICE
 
 # Note that these commands are processed in the order they appear, so keep
 # this exclude at the end.
diff --git a/python/packaging/classic/setup.py b/python/packaging/classic/setup.py
index 54b7707dbc44..53d11a917f55 100755
--- a/python/packaging/classic/setup.py
+++ b/python/packaging/classic/setup.py
@@ -104,7 +104,14 @@ EXAMPLES_PATH = os.path.join(SPARK_HOME, "examples/src/main/python")
 SCRIPTS_PATH = os.path.join(SPARK_HOME, "bin")
 USER_SCRIPTS_PATH = os.path.join(SPARK_HOME, "sbin")
 DATA_PATH = os.path.join(SPARK_HOME, "data")
-LICENSES_PATH = os.path.join(SPARK_HOME, "licenses")
+# The classic PySpark package bundles the assembly jars, so it ships the binary
+# license texts (licenses-binary), which enumerate those jars' licenses, mirroring
+# the binary distribution. The connect/client packages bundle no jars.
+LICENSES_PATH = os.path.join(SPARK_HOME, "licenses-binary")
+if not os.path.isdir(LICENSES_PATH):
+    # In a binary release dist (the RELEASE mode below), the binary license texts
+    # were already copied to licenses/ (see dev/make-distribution.sh).
+    LICENSES_PATH = os.path.join(SPARK_HOME, "licenses")
 
 SCRIPTS_TARGET = os.path.join(TEMP_PATH, "bin")
 USER_SCRIPTS_TARGET = os.path.join(TEMP_PATH, "sbin")
@@ -343,7 +350,7 @@ try:
             ],
             "pyspark.python.lib": ["*.zip"],
             "pyspark.data": ["*.txt", "*.data"],
-            "pyspark.licenses": ["*.txt"],
+            "pyspark.licenses": ["*"],
             "pyspark.examples.src.main.python": ["*.py", "*/*.py"],
         },
         scripts=scripts,


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org