You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "hzmsh (via GitHub)" <gi...@apache.org> on 2026/10/16 07:18:18 UTC
[PR] [SPARK-60279][CONNECT][PYTHON] Fix Spark Connect pivot rejecting None, date and Decimal values [spark]
hzmsh opened a new pull request, #59541:
URL: https://github.com/apache/spark/pull/59541
### What changes were proposed in this pull request?
Fixes [SPARK-60279](https://issues.apache.org/jira/browse/SPARK-60279).
Fix:
- `GroupedData.pivot` in Spark Connect checks each pivot value with `isinstance(v, (bool, float, int, str))`. This PR also accepts `None`, `datetime.date` (which covers `datetime.datetime`) and `decimal.Decimal` there.
- The values are then passed to `F.lit` (in `GroupedData.__init__`), and `F.lit` already supports these types, so nothing else needs to change. The `expected_type` in the error message is updated to match.
- Classic is not changed: it passes the values to the JVM, which already accepts them.
### Why are the changes needed?
The same code works in classic and fails in Connect. A common way to hit it is to collect the pivot values with `distinct()`, which gives `None` when the column has a NULL.
```python
# spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect, .master("local[2]") for classic
import datetime, decimal
df = spark.createDataFrame(
[(1, "a", datetime.date(2020, 1, 1), decimal.Decimal("1.5"), 10),
(2, None, datetime.date(2020, 1, 2), decimal.Decimal("2.5"), 20)],
"k int, s string, d date, dec decimal(10,1), v int",
)
vals = [r[0] for r in df.select("s").distinct().collect()] # contains None
df.groupBy("k").pivot("s", vals).sum("v").columns
# classic: ['k', 'a', 'null']
# Connect on master and 4.2.0: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be bool, float, int or str, got NoneType.
```
`datetime.date` and `decimal.Decimal` values fail the same way (`got date`, `got Decimal`), while classic gives columns `2020-01-01` and `1.5`.
When it started:
- The check has been there since the Connect pivot was added in [SPARK-41693](https://issues.apache.org/jira/browse/SPARK-41693) (3.4.0), so every Connect version is affected. The allowed types match the `LiteralType` type hint (`Union[bool, float, int, str]`); `F.lit` already handled `None`, `date` and `Decimal` at that time.
Reports from users:
- A feature pipeline on Databricks serverless (Spark Connect) hit the `None` case: the first error in its error catalog is `NOT_EXPECTED_TYPE: got NoneType` in `.pivot()`. The workaround drops `None` from the values, which loses the `null` column that classic would produce ([WenjieAtBayer/kerendia_nba_3_1_develop#3](https://github.com/WenjieAtBayer/kerendia_nba_3_1_develop/pull/3)).
What goes wrong:
- In [`connect/group.py`](https://github.com/apache/spark/blob/d18b51873dbcb8f18677b9a5860b3131f16b1d8e/python/pyspark/sql/connect/group.py#L248-L257), `pivot` raises `NOT_EXPECTED_TYPE` for any value that is not a bool, float, int or str, before the values reach `F.lit`.
### Does this PR introduce _any_ user-facing change?
Yes, relative to 4.2.0 as well as master.
Behavior change:
- In Spark Connect, `pivot` now accepts `None`, `date`, `datetime` and `Decimal` values and returns the same columns and rows as classic. Before, it raised `NOT_EXPECTED_TYPE`.
- The error message for other values changes from "should be bool, float, int or str" to "should be bool, date, Decimal, float, int, None or str".
- `pivot` without `values` and values of the types accepted before behave as before.
Not covered:
- A tuple of values (`values=("a",)`) is still rejected in Connect, though classic accepts it at runtime. The `values` type hint does not allow tuples in classic either; [SPARK-37116](https://issues.apache.org/jira/browse/SPARK-37116) (allow tuples as pivot values in PySpark) has been open since 2021.
- `bytes`, `bytearray`, `datetime.time`, `datetime.timedelta` and NumPy scalars such as `np.int64(1)`, `np.bool_(True)` and `np.float32(1.5)` are still rejected in Connect, though classic accepts them. Only `np.float64` and `np.str_` values already work, because they subclass `float` and `str`. They are left out to keep this change small; `SparkConnectStatTests.test_grouped_data` still uses `bytes` as its invalid example.
- The `values` type hint (`List[LiteralType]`) is unchanged in both classic and Connect, so type checkers still flag `None`, `date` and `Decimal` values.
### How was this patch tested?
Tests:
- Added `GroupTestsMixin.test_pivot_values_with_null_date_and_decimal`, which runs in classic (`test_group`) and Connect (`test_parity_group`). It checks the column names and rows for pivot values from `distinct()` that include `None` (columns `k`, `null`, `a`), a `date` value (column `2020-01-01`) and a `Decimal` value (column `10.0`).
- Updated the expected message in `SparkConnectStatTests.test_grouped_data`.
- Without the fix, the new test passes in classic and fails in Connect with `NOT_EXPECTED_TYPE ... got NoneType`.
- With the fix, all tests in the three modules pass (9 in `test_group`, 9 in `test_parity_group`, 18 in `test_connect_stat`).
- The repro above was also run against PySpark 4.2.0 from PyPI.
- Commands:
```
python/run-tests --python-executables=python3 --testnames 'pyspark.sql.tests.test_group,pyspark.sql.tests.connect.test_parity_group,pyspark.sql.tests.connect.test_connect_stat'
dev/lint-python
```
### Was this patch authored or co-authored using generative AI tooling?
Co-developed with Claude Code (Claude Opus 5.5): the fix, the tests and this description. The checks above were run locally.
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60279][CONNECT][PYTHON] Fix Spark Connect pivot rejecting None, date and Decimal values [spark]
Posted by "hzmsh (via GitHub)" <gi...@apache.org>.
hzmsh commented on PR #59541:
URL: https://github.com/apache/spark/pull/59541#issuecomment-6210282255
Hi @zhengruifeng, could you take a look when you have a moment? This widens the pivot value check you added in [#39191](https://github.com/apache/spark/pull/39191) ([SPARK-41693](https://issues.apache.org/jira/browse/SPARK-41693)), so that None, date and Decimal work in Connect as they do in classic.
I kept a client-side check so that a Column or a nested list still gets a clear error before reaching the server. If you'd prefer to drop the per-value check and let `F.lit` validate, as classic leaves it to the JVM, I'm happy to change it; that would also make bytes, time, timedelta and the NumPy scalars that `F.lit` supports (for example `np.int64`, `np.bool_`, `np.float32`) work.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60279][CONNECT][PYTHON] Fix Spark Connect pivot rejecting None, date and Decimal values [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #59541: [SPARK-60279][CONNECT][PYTHON] Fix Spark Connect pivot rejecting None, date and Decimal values
URL: https://github.com/apache/spark/pull/59541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60279][CONNECT][PYTHON] Fix Spark Connect pivot rejecting None, date and Decimal values [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #59541:
URL: https://github.com/apache/spark/pull/59541#issuecomment-6286835842
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/d81fe92f797291f279574519aa92c85b25d06ca7
- merged into branch-4.x https://github.com/apache/spark/commit/6f4dd87dcfad390d1b0a0cc2b173e85ca5dc8d36
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org