You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Alwaysgaurav1 (via GitHub)" <gi...@apache.org> on 2026/07/11 18:21:15 UTC
[PR] [SPARK-XXXXX][PYTHON] Standardize PySpark row count mismatch error class to use RESULT_ROWS_MISMATCH [spark]
Alwaysgaurav1 opened a new pull request, #57203:
URL: https://github.com/apache/spark/pull/57203
### What changes were proposed in this pull request?
This PR resolves a TODO in `python/pyspark/worker.py` to standardize row count mismatch error reporting by using the unified `RESULT_ROWS_MISMATCH` error class instead of the legacy `SCHEMA_MISMATCH_FOR_PANDAS_UDF` class.
Specifically:
- Refactored the local `verify_result_length` helper in `wrap_scalar_pandas_udf` to raise `RESULT_ROWS_MISMATCH` and passed matching `messageParameters` (`output_length` and `input_length`).
- Refactored the second local `verify_result_length` helper in `wrap_arrow_batch_udf` to also raise `RESULT_ROWS_MISMATCH`.
- Removed the deprecated `"SCHEMA_MISMATCH_FOR_PANDAS_UDF"` entry from the python error conditions registry (`python/pyspark/errors/error-conditions.json`).
### Why are the changes needed?
Previously, PySpark raised a custom `SCHEMA_MISMATCH_FOR_PANDAS_UDF` error class during UDF row count mismatches. Standardizing this error class to `RESULT_ROWS_MISMATCH` makes error codes and message patterns uniform across PySpark.
### Does this PR introduce _any_ user-facing change?
Yes, the error class raised on a Pandas/Arrow UDF row count mismatch changes:
- **Previous Error**: `[SCHEMA_MISMATCH_FOR_PANDAS_UDF] Result vector from pandas_udf was not the required length: expected 1, got 0.`
- **New Error**: `[RESULT_ROWS_MISMATCH] The number of output rows (0) must match the number of input rows (1).`
### How was this patch tested?
1. Ran unit tests targeting Pandas UDF execution:
```bash
python/run-tests.py --python-executables venv/bin/python --testnames pyspark.sql.tests.pandas.test_pandas_udf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-XXXXX][PYTHON] Standardize PySpark row count mismatch error class to use RESULT_ROWS_MISMATCH [spark]
Posted by "Alwaysgaurav1 (via GitHub)" <gi...@apache.org>.
Alwaysgaurav1 closed pull request #57203: [SPARK-XXXXX][PYTHON] Standardize PySpark row count mismatch error class to use RESULT_ROWS_MISMATCH
URL: https://github.com/apache/spark/pull/57203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org