You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/03/10 19:06:41 UTC
[PR] [SPARK-55723][PYTHON] Generalize enforce_schema error to PySparkTypeError [spark]
Yicong-Huang opened a new pull request, #54736:
URL: https://github.com/apache/spark/pull/54736
### What changes were proposed in this pull request?
Replace `PySparkRuntimeError` with `RESULT_COLUMNS_MISMATCH_FOR_ARROW_UDTF` error class in `enforce_schema` and `ArrowStreamArrowUDTFSerializer` with a general `PySparkTypeError` that reports column name, expected type, and actual type without being specific to any UDF type.
### Why are the changes needed?
The `RESULT_COLUMNS_MISMATCH_FOR_ARROW_UDTF` error class was UDTF-specific, but `enforce_schema` is a general utility used across UDF types. The error message ("Column names ... do not match specified schema") was also misleading -- the actual failure is a type cast error, not a column name mismatch.
Part of SPARK-55724.
### Does this PR introduce _any_ user-facing change?
Yes. The error type changes from `PySparkRuntimeError` to `PySparkTypeError`, and the message now accurately describes the type mismatch:
```
Result type of column 'id' does not match the expected type. Expected: int32, got: string.
```
### How was this patch tested?
Updated existing test in `test_arrow_udtf.py`.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55723][PYTHON] Generalize enforce_schema error to PySparkTypeError [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #54736:
URL: https://github.com/apache/spark/pull/54736#issuecomment-4087548880
> I'm okay with this in general, but what's our policy to remove an error class? Can we just do it? @HyukjinKwon and @zhengruifeng
we don't have a strict policy on removing error class, and I think normally it won't be treated as behavior change (it should be rare case that a job relies on the error class)
I think it is in general fine to remove duplicated ones and reuse existing ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55723][PYTHON] Generalize enforce_schema error to PySparkTypeError [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #54736: [SPARK-55723][PYTHON] Generalize enforce_schema error to PySparkTypeError
URL: https://github.com/apache/spark/pull/54736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55723][PYTHON] Generalize enforce_schema error to PySparkTypeError [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #54736:
URL: https://github.com/apache/spark/pull/54736#issuecomment-4089138028
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55723][PYTHON] Generalize enforce_schema error to PySparkTypeError [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #54736:
URL: https://github.com/apache/spark/pull/54736#issuecomment-4033798028
cc @gaogaotiantian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55723][PYTHON] Generalize enforce_schema error to PySparkTypeError [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on PR #54736:
URL: https://github.com/apache/spark/pull/54736#issuecomment-4050097969
I'm okay with this in general, but what's our policy to remove an error class? Can we just do it? @HyukjinKwon and @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org