You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anxkhn (via GitHub)" <gi...@apache.org> on 2026/07/05 17:43:21 UTC
[PR] [PYTHON][SQL] Fix DataFrame.dropna to raise PySparkValueError on invalid how [spark]
anxkhn opened a new pull request, #57012:
URL: https://github.com/apache/spark/pull/57012
### What changes were proposed in this pull request?
`DataFrame.dropna` on the classic path validates the `how` argument and, on an
invalid value, raises `PySparkValueError` with error class
`VALUE_NOT_ANY_OR_ALL`. The `messageParameters` passed a key named `arg_type`,
but the `VALUE_NOT_ANY_OR_ALL` template interpolates `<arg_value>`:
```
"Value for `<arg_name>` must be 'any' or 'all', got '<arg_value>'."
```
So the template placeholder set is `{arg_name, arg_value}` while the code passed
`{arg_name, arg_type}`. `ErrorClassesReader.get_error_message`
(`python/pyspark/errors/utils.py`) asserts the two sets are equal, so the
mismatch raised an opaque `AssertionError` from inside the exception constructor
and masked the intended `PySparkValueError`.
This PR changes that one `messageParameters` key from `arg_type` to `arg_value`
(the value, `how`, is unchanged) so it matches the template, and adds a
regression test. `VALUE_NOT_ANY_OR_ALL` is used at exactly this one site, so no
other call site is affected.
### Why are the changes needed?
`df.dropna(how="foo")` is a common public DataFrame API call. Before this change
it surfaced an internal `AssertionError`:
```python
>>> df.dropna(how="foo")
AssertionError: Undefined error message parameter for error class:
VALUE_NOT_ANY_OR_ALL. Parameters: {'arg_name': 'how', 'arg_type': 'foo'}
```
instead of the intended, user-facing error:
```python
>>> df.dropna(how="foo")
pyspark.errors.exceptions.captured.PySparkValueError:
[VALUE_NOT_ANY_OR_ALL] Value for `how` must be 'any' or 'all', got 'foo'.
```
The invalid-`how` branch was never exercised by a test, so the broken error
contract went unnoticed.
### Does this PR introduce _any_ user-facing change?
Yes. Passing an invalid `how` to `DataFrame.dropna` (equivalently
`DataFrame.na.drop`) now raises the documented `PySparkValueError`
(`VALUE_NOT_ANY_OR_ALL`) with a clear message, instead of an internal
`AssertionError`. Both are error paths, so valid usage is unchanged; only the
exception raised for invalid input differs.
### How was this patch tested?
Added a regression case to `DataFrameStatTestsMixin.test_dropna` in
`python/pyspark/sql/tests/test_stat.py` that asserts
`dropna(how="foo")` raises `PySparkValueError` with condition
`VALUE_NOT_ANY_OR_ALL` and parameters `{arg_name: "how", arg_value: "foo"}`,
using the same `check_error` idiom as the adjacent `NOT_EXPECTED_TYPE` case.
Ran the suite locally against a real Spark session:
```
python/run-tests --testnames "pyspark.sql.tests.test_stat DataFrameStatTests.test_dropna"
```
It fails on the unpatched code (the masked `AssertionError`) and passes with the
fix. `dev/lint-python` (ruff) is clean on both changed files.
### Was this patch authored or co-authored using generative AI tooling?
<!-- Anas: fill this per ASF policy before opening. If AI tooling was used in
authoring, this must read e.g. `Generated-by: <tool name and version>`;
otherwise `No`. See CONTEXT below. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [PYTHON][SQL] Fix DataFrame.dropna to raise PySparkValueError on invalid how [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57012:
URL: https://github.com/apache/spark/pull/57012#issuecomment-4889492355
@anxkhn thanks for the contribution, can you enable the CI referring to https://spark.apache.org/contributing.html and add a JIRA id in the PR title?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57964][PYTHON][SQL] Fix DataFrame.dropna to raise PySparkValueError on invalid how [spark]
Posted by "anxkhn (via GitHub)" <gi...@apache.org>.
anxkhn closed pull request #57012: [SPARK-57964][PYTHON][SQL] Fix DataFrame.dropna to raise PySparkValueError on invalid how
URL: https://github.com/apache/spark/pull/57012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org