You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/04/13 22:49:29 UTC
(spark) branch master updated: [SPARK-56452][PYTHON] Fix pip install failure for prerelease versions
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 790cad762424 [SPARK-56452][PYTHON] Fix pip install failure for prerelease versions
790cad762424 is described below
commit 790cad762424bda9222b318ed917a121499ca54f
Author: Kiyeon Jeon <ki...@gmail.com>
AuthorDate: Tue Apr 14 07:49:15 2026 +0900
[SPARK-56452][PYTHON] Fix pip install failure for prerelease versions
### What changes were proposed in this pull request?
Updated the version validation regexes in `python/pyspark/install.py` to accept prerelease version strings (e.g. `4.2.0.dev4`).
Two regexes were updated:
- `checked_versions()` (line 71): the regex that detects bare version strings and prepends `spark-`
- `convert_old_hadoop_version()` (line 109): the regex that extracts major/minor version parts
Both now include an optional `(?:\.dev[0-9]+)?` suffix.
### Why are the changes needed?
When installing a PySpark prerelease via pip (e.g. `pip install pyspark==4.2.0.dev4`), the version string `4.2.0.dev4` does not match the existing regex `^[0-9]+\.[0-9]+\.[0-9]+$`. This causes the `spark-` prefix to not be added, which then triggers a `RuntimeError`:
```
RuntimeError: Spark version should start with 'spark-' prefix; however, got 4.2.0.dev4
```
GitHub issue: https://github.com/apache/spark/issues/55289
### Does this PR introduce _any_ user-facing change?
Yes. Previously, `pip install pyspark==4.2.0.dev4` would fail with a `RuntimeError`. After this fix, prerelease versions install correctly.
### How was this patch tested?
Added two test cases in `test_checked_versions` for prerelease versions:
- `checked_versions("4.2.0.dev4", "3", "2.3")` — bare version with dev suffix
- `checked_versions("spark-4.2.0.dev4", "hadoop3", "hive2.3")` — already prefixed version with dev suffix
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #55311 from kiyeonjeon21/SPARK-56452.
Authored-by: Kiyeon Jeon <ki...@gmail.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/install.py | 6 ++++--
python/pyspark/tests/test_install_spark.py | 11 +++++++++++
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/install.py b/python/pyspark/install.py
index f7d44d3b942a..55f3b0c2d8bc 100644
--- a/python/pyspark/install.py
+++ b/python/pyspark/install.py
@@ -68,7 +68,7 @@ def checked_versions(
fully-qualified versions of Spark, Hadoop and Hive in a tuple.
For example, spark-3.2.0, hadoop3 and hive2.3.
"""
- if re.match("^[0-9]+\\.[0-9]+\\.[0-9]+$", spark_version):
+ if re.match("^[0-9]+\\.[0-9]+\\.[0-9]+(?:\\.dev[0-9]+)?$", spark_version):
spark_version = "spark-%s" % spark_version
if not spark_version.startswith("spark-"):
raise RuntimeError(
@@ -106,7 +106,9 @@ def convert_old_hadoop_version(spark_version: str, hadoop_version: str) -> str:
"without": "without",
"without-hadoop": "without-hadoop",
}
- spark_version_parts = re.search("^spark-([0-9]+)\\.([0-9]+)\\.[0-9]+$", spark_version)
+ spark_version_parts = re.search(
+ "^spark-([0-9]+)\\.([0-9]+)\\.[0-9]+(?:\\.dev[0-9]+)?$", spark_version
+ )
assert spark_version_parts is not None
spark_major_version = int(spark_version_parts.group(1))
spark_minor_version = int(spark_version_parts.group(2))
diff --git a/python/pyspark/tests/test_install_spark.py b/python/pyspark/tests/test_install_spark.py
index b977fdf51024..ca4ea60fb114 100644
--- a/python/pyspark/tests/test_install_spark.py
+++ b/python/pyspark/tests/test_install_spark.py
@@ -106,6 +106,17 @@ class SparkInstallationTestCase(unittest.TestCase):
checked_versions("spark-3.3.0", "hadoop3", "hive2.3"),
)
+ # Prerelease version (e.g. pip dev builds)
+ self.assertEqual(
+ ("spark-4.2.0.dev4", "hadoop3", "hive2.3"),
+ checked_versions("4.2.0.dev4", "3", "2.3"),
+ )
+
+ self.assertEqual(
+ ("spark-4.2.0.dev4", "hadoop3", "hive2.3"),
+ checked_versions("spark-4.2.0.dev4", "hadoop3", "hive2.3"),
+ )
+
# Negative test cases
for hadoop_version, hive_version in UNSUPPORTED_COMBINATIONS:
with self.assertRaisesRegex(RuntimeError, "Hive.*should.*Hadoop"):
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org