You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2025/05/02 04:49:43 UTC
(spark) branch master updated: [SPARK-51980][PYTHON][TESTS] Enable `--use-pep517` in `dev/run-pip-tests`
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 31f7278bc00b [SPARK-51980][PYTHON][TESTS] Enable `--use-pep517` in `dev/run-pip-tests`
31f7278bc00b is described below
commit 31f7278bc00b0c3f4457b819398b9315b1ff7f22
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Thu May 1 21:49:31 2025 -0700
[SPARK-51980][PYTHON][TESTS] Enable `--use-pep517` in `dev/run-pip-tests`
### What changes were proposed in this pull request?
This pr enable `--use-pep517` option in `dev/run-pip-tests` refer to:
```
25l DEPRECATION: Legacy editable install of pyspark==4.1.0.dev0 from file:///__w/spark/spark/python/packaging/classic (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can b [...]
```
### Why are the changes needed?
To restore Github Action test failed:
- https://github.com/apache/spark/actions/runs/14778042174/job/41496865818
```
Installing collected packages: py4j, pyspark
25l DEPRECATION: Legacy editable install of pyspark==4.1.0.dev0 from file:///__w/spark/spark/python/packaging/classic (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can b [...]
Running setup.py develop for pyspark
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark] error: subprocess-exited-with-error
× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [74 lines of output]
/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
!!
********************************************************************************
Please consider removing the following classifiers in favor of a SPDX license expression:
License :: OSI Approved :: Apache Software License
See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
********************************************************************************
!!
self._finalize_license_expression()
running develop
/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:90: DevelopDeprecationWarning: develop command is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` and ``develop``.
Instead, use standards-based tools like pip or uv.
By 2025-Oct-31, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
See https://github.com/pypa/setuptools/issues/917 for details.
********************************************************************************
!!
self.initialize_options()
Obtaining file:///__w/spark/spark/python
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Checking if build backend supports build_editable: started
Checking if build backend supports build_editable: finished with status 'done'
Getting requirements to build editable: started
Getting requirements to build editable: finished with status 'error'
error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully.
│ exit code: 255
╰─> [1 lines of output]
Temp path for symlink to parent already exists deps
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully.
│ exit code: 255
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 35, in <module>
File "/__w/spark/spark/python/packaging/classic/setup.py", line 254, in <module>
setup(
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
return distutils.core.setup(**attrs)
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 186, in setup
return run_commands(dist)
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
dist.run_commands()
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
self.run_command(cmd)
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/command/develop.py", line 39, in run
subprocess.check_call(cmd)
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/tmp/tmp.zGt6QETKsu/3.9/bin/python3.9', '-m', 'pip', 'install', '-e', '.', '--use-pep517', '--no-deps']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [74 lines of output]
/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
!!
********************************************************************************
Please consider removing the following classifiers in favor of a SPDX license expression:
License :: OSI Approved :: Apache Software License
See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
********************************************************************************
!!
self._finalize_license_expression()
running develop
/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:90: DevelopDeprecationWarning: develop command is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` and ``develop``.
Instead, use standards-based tools like pip or uv.
By 2025-Oct-31, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
See https://github.com/pypa/setuptools/issues/917 for details.
********************************************************************************
!!
self.initialize_options()
Obtaining file:///__w/spark/spark/python
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Checking if build backend supports build_editable: started
Checking if build backend supports build_editable: finished with status 'done'
Getting requirements to build editable: started
Getting requirements to build editable: finished with status 'error'
error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully.
│ exit code: 255
╰─> [1 lines of output]
Temp path for symlink to parent already exists deps
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully.
│ exit code: 255
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 35, in <module>
File "/__w/spark/spark/python/packaging/classic/setup.py", line 254, in <module>
setup(
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
return distutils.core.setup(**attrs)
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 186, in setup
return run_commands(dist)
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
dist.run_commands()
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
self.run_command(cmd)
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/site-packages/setuptools/command/develop.py", line 39, in run
subprocess.check_call(cmd)
File "/tmp/tmp.zGt6QETKsu/3.9/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/tmp/tmp.zGt6QETKsu/3.9/bin/python3.9', '-m', 'pip', 'install', '-e', '.', '--use-pep517', '--no-deps']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [pyspark]
25h
Cleaning up temporary directory - /tmp/tmp.zGt6QETKsu
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #50778 from LuciferYang/use-pep517.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
dev/run-pip-tests | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev/run-pip-tests b/dev/run-pip-tests
index f8a547b0c917..7740481e4d83 100755
--- a/dev/run-pip-tests
+++ b/dev/run-pip-tests
@@ -70,7 +70,7 @@ fi
PYSPARK_VERSION=$(python3 -c "exec(open('python/pyspark/version.py').read());print(__version__)")
PYSPARK_DIST="$FWDIR/python/dist/pyspark-$PYSPARK_VERSION.tar.gz"
# The pip install options we use for all the pip commands
-PIP_OPTIONS="--upgrade --no-cache-dir --force-reinstall"
+PIP_OPTIONS="--upgrade --no-cache-dir --force-reinstall --use-pep517"
# Test both regular user and edit/dev install modes.
PIP_COMMANDS=("pip install $PIP_OPTIONS $PYSPARK_DIST"
"pip install $PIP_OPTIONS -e python/packaging/classic")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org