You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/10/20 08:30:50 UTC
(spark) branch branch-4.x updated: [SPARK-60279][CONNECT][PYTHON] Fix Spark Connect pivot rejecting None, date and Decimal values
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 6f4dd87dcfad [SPARK-60279][CONNECT][PYTHON] Fix Spark Connect pivot rejecting None, date and Decimal values
6f4dd87dcfad is described below
commit 6f4dd87dcfad390d1b0a0cc2b173e85ca5dc8d36
Author: Shuheng (Markus) Ma <35...@users.noreply.github.com>
AuthorDate: Tue Oct 20 16:30:13 2026 +0800
[SPARK-60279][CONNECT][PYTHON] Fix Spark Connect pivot rejecting None, date and Decimal values
### What changes were proposed in this pull request?
Fixes [SPARK-60279](https://issues.apache.org/jira/browse/SPARK-60279).
Fix:
- `GroupedData.pivot` in Spark Connect checks each pivot value with `isinstance(v, (bool, float, int, str))`. This PR also accepts `None`, `datetime.date` (which covers `datetime.datetime`) and `decimal.Decimal` there.
- The values are then passed to `F.lit` (in `GroupedData.__init__`), and `F.lit` already supports these types, so nothing else needs to change. The `expected_type` in the error message is updated to match.
- Classic is not changed: it passes the values to the JVM, which already accepts them.
### Why are the changes needed?
The same code works in classic and fails in Connect. A common way to hit it is to collect the pivot values with `distinct()`, which gives `None` when the column has a NULL.
```python
# spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect, .master("local[2]") for classic
import datetime, decimal
df = spark.createDataFrame(
[(1, "a", datetime.date(2020, 1, 1), decimal.Decimal("1.5"), 10),
(2, None, datetime.date(2020, 1, 2), decimal.Decimal("2.5"), 20)],
"k int, s string, d date, dec decimal(10,1), v int",
)
vals = [r[0] for r in df.select("s").distinct().collect()] # contains None
df.groupBy("k").pivot("s", vals).sum("v").columns
# classic: ['k', 'a', 'null']
# Connect on master and 4.2.0: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be bool, float, int or str, got NoneType.
```
`datetime.date` and `decimal.Decimal` values fail the same way (`got date`, `got Decimal`), while classic gives columns `2020-01-01` and `1.5`.
When it started:
- The check has been there since the Connect pivot was added in [SPARK-41693](https://issues.apache.org/jira/browse/SPARK-41693) (3.4.0), so every Connect version is affected. The allowed types match the `LiteralType` type hint (`Union[bool, float, int, str]`); `F.lit` already handled `None`, `date` and `Decimal` at that time.
Reports from users:
- A feature pipeline on Databricks serverless (Spark Connect) hit the `None` case: the first error in its error catalog is `NOT_EXPECTED_TYPE: got NoneType` in `.pivot()`. The workaround drops `None` from the values, which loses the `null` column that classic would produce ([WenjieAtBayer/kerendia_nba_3_1_develop#3](https://github.com/WenjieAtBayer/kerendia_nba_3_1_develop/pull/3)).
What goes wrong:
- In [`connect/group.py`](https://github.com/apache/spark/blob/d18b51873dbcb8f18677b9a5860b3131f16b1d8e/python/pyspark/sql/connect/group.py#L248-L257), `pivot` raises `NOT_EXPECTED_TYPE` for any value that is not a bool, float, int or str, before the values reach `F.lit`.
### Does this PR introduce _any_ user-facing change?
Yes, relative to 4.2.0 as well as master.
Behavior change:
- In Spark Connect, `pivot` now accepts `None`, `date`, `datetime` and `Decimal` values and returns the same columns and rows as classic. Before, it raised `NOT_EXPECTED_TYPE`.
- The error message for other values changes from "should be bool, float, int or str" to "should be bool, date, Decimal, float, int, None or str".
- `pivot` without `values` and values of the types accepted before behave as before.
Not covered:
- A tuple of values (`values=("a",)`) is still rejected in Connect, though classic accepts it at runtime. The `values` type hint does not allow tuples in classic either; [SPARK-37116](https://issues.apache.org/jira/browse/SPARK-37116) (allow tuples as pivot values in PySpark) has been open since 2021.
- `bytes`, `bytearray`, `datetime.time`, `datetime.timedelta` and NumPy scalars such as `np.int64(1)`, `np.bool_(True)` and `np.float32(1.5)` are still rejected in Connect, though classic accepts them. Only `np.float64` and `np.str_` values already work, because they subclass `float` and `str`. They are left out to keep this change small; `SparkConnectStatTests.test_grouped_data` still uses `bytes` as its invalid example.
- The `values` type hint (`List[LiteralType]`) is unchanged in both classic and Connect, so type checkers still flag `None`, `date` and `Decimal` values.
### How was this patch tested?
Tests:
- Added `GroupTestsMixin.test_pivot_values_with_null_date_and_decimal`, which runs in classic (`test_group`) and Connect (`test_parity_group`). It checks the column names and rows for pivot values from `distinct()` that include `None` (columns `k`, `null`, `a`), a `date` value (column `2020-01-01`) and a `Decimal` value (column `10.0`).
- Updated the expected message in `SparkConnectStatTests.test_grouped_data`.
- Without the fix, the new test passes in classic and fails in Connect with `NOT_EXPECTED_TYPE ... got NoneType`.
- With the fix, all tests in the three modules pass (9 in `test_group`, 9 in `test_parity_group`, 18 in `test_connect_stat`).
- The repro above was also run against PySpark 4.2.0 from PyPI.
- Commands:
```
python/run-tests --python-executables=python3 --testnames 'pyspark.sql.tests.test_group,pyspark.sql.tests.connect.test_parity_group,pyspark.sql.tests.connect.test_connect_stat'
dev/lint-python
```
### Was this patch authored or co-authored using generative AI tooling?
Co-developed with Claude Code (Claude Opus 5.5): the fix, the tests and this description. The checks above were run locally.
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59541 from hzmsh/connect-pivot-values.
Authored-by: Shuheng (Markus) Ma <35...@users.noreply.github.com>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
(cherry picked from commit d81fe92f797291f279574519aa92c85b25d06ca7)
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
python/pyspark/sql/connect/group.py | 8 ++++++--
.../pyspark/sql/tests/connect/test_connect_stat.py | 2 +-
python/pyspark/sql/tests/test_group.py | 22 ++++++++++++++++++++++
3 files changed, 29 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/connect/group.py b/python/pyspark/sql/connect/group.py
index 8c21b8154e39..f16da3626c11 100644
--- a/python/pyspark/sql/connect/group.py
+++ b/python/pyspark/sql/connect/group.py
@@ -15,6 +15,8 @@
# limitations under the License.
#
+import datetime
+import decimal
import warnings
from typing import (
TYPE_CHECKING,
@@ -246,11 +248,13 @@ class GroupedData:
},
)
for v in values:
- if not isinstance(v, (bool, float, int, str)):
+ if v is not None and not isinstance(
+ v, (bool, float, int, str, datetime.date, decimal.Decimal)
+ ):
raise PySparkTypeError(
errorClass="NOT_EXPECTED_TYPE",
messageParameters={
- "expected_type": "bool, float, int or str",
+ "expected_type": "bool, date, Decimal, float, int, None or str",
"arg_name": "value",
"arg_type": type(v).__name__,
},
diff --git a/python/pyspark/sql/tests/connect/test_connect_stat.py b/python/pyspark/sql/tests/connect/test_connect_stat.py
index 827059653dc1..51d89e5051f1 100644
--- a/python/pyspark/sql/tests/connect/test_connect_stat.py
+++ b/python/pyspark/sql/tests/connect/test_connect_stat.py
@@ -550,7 +550,7 @@ class SparkConnectStatTests(SparkConnectSQLTestCase):
exception=pe.exception,
errorClass="NOT_EXPECTED_TYPE",
messageParameters={
- "expected_type": "bool, float, int or str",
+ "expected_type": "bool, date, Decimal, float, int, None or str",
"arg_name": "value",
"arg_type": "bytes",
},
diff --git a/python/pyspark/sql/tests/test_group.py b/python/pyspark/sql/tests/test_group.py
index 534caf0928b7..4d122e5a6209 100644
--- a/python/pyspark/sql/tests/test_group.py
+++ b/python/pyspark/sql/tests/test_group.py
@@ -14,6 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+import datetime
+import decimal
import unittest
from pyspark.errors import AnalysisException
@@ -192,6 +194,26 @@ class GroupTestsMixin:
with self.assertRaises(IndexError):
df.orderBy(-3)
+ def test_pivot_values_with_null_date_and_decimal(self):
+ # SPARK-60279: pivot values can be None, date or Decimal
+ df = self.spark.createDataFrame(
+ [(1, "a", datetime.date(2020, 1, 1), 10), (2, None, datetime.date(2020, 1, 2), 20)],
+ "k int, s string, d date, v int",
+ ).withColumn("dec", sf.col("v").cast("decimal(10,1)"))
+
+ values = [r.s for r in df.select("s").distinct().sort("s").collect()]
+ res = df.groupBy("k").pivot("s", values).sum("v")
+ self.assertEqual(res.columns, ["k", "null", "a"])
+ assertDataFrameEqual(res, [Row(k=1, null=None, a=10), Row(k=2, null=20, a=None)])
+
+ res = df.groupBy("k").pivot("d", [datetime.date(2020, 1, 1)]).sum("v")
+ self.assertEqual(res.columns, ["k", "2020-01-01"])
+ assertDataFrameEqual(res, [Row(1, 10), Row(2, None)])
+
+ res = df.groupBy("k").pivot("dec", [decimal.Decimal("10.0")]).sum("v")
+ self.assertEqual(res.columns, ["k", "10.0"])
+ assertDataFrameEqual(res, [Row(1, 10), Row(2, None)])
+
def test_pivot_exceed_max_values(self):
with self.assertRaises(AnalysisException):
self.spark.range(100001).groupBy(sf.lit(1)).pivot("id").count().show()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org