You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by al...@apache.org on 2025/08/22 22:35:16 UTC
(spark) branch master updated: [SPARK-53358] Improve arrow Python UDTF output type mismatch error message
This is an automated email from the ASF dual-hosted git repository.
allisonwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c86371783102 [SPARK-53358] Improve arrow Python UDTF output type mismatch error message
c86371783102 is described below
commit c863717831029c44069ec17dcf8091147e14a7b3
Author: Allison Wang <al...@databricks.com>
AuthorDate: Fri Aug 22 15:35:03 2025 -0700
[SPARK-53358] Improve arrow Python UDTF output type mismatch error message
### What changes were proposed in this pull request?
This PR updates the error message when output type mismatch required type for arrow python UDTFs to make it more user friendly.
### Why are the changes needed?
Improve error message to make it more actionable.
Before this change:
```pyspark.errors.exceptions.base.PySparkRuntimeError: [UDTF_ARROW_TYPE_CONVERSION_ERROR] Cannot convert the output value of the input '[
0
]' with type 'struct<x:int>' to the specified return type of the column: 'struct<x: int32>'. Please check if the data types match and try again.
```
After this change:
```
PyArrow UDTF must return an iterator of pyarrow.Table or pyarrow.RecordBatch objects.
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing UTs
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #52103 from allisonwang-db/spark-53358-arrow-udtf-err-msg.
Authored-by: Allison Wang <al...@databricks.com>
Signed-off-by: Allison Wang <al...@databricks.com>
---
python/pyspark/errors/error-conditions.json | 2 +-
python/pyspark/worker.py | 6 +-----
2 files changed, 2 insertions(+), 6 deletions(-)
diff --git a/python/pyspark/errors/error-conditions.json b/python/pyspark/errors/error-conditions.json
index ba682a65a9f4..1da6b8e15478 100644
--- a/python/pyspark/errors/error-conditions.json
+++ b/python/pyspark/errors/error-conditions.json
@@ -1146,7 +1146,7 @@
},
"UDTF_ARROW_TYPE_CONVERSION_ERROR": {
"message": [
- "Cannot convert the output value of the input '<data>' with type '<schema>' to the specified return type of the column: '<arrow_schema>'. Please check if the data types match and try again."
+ "PyArrow UDTF must return an iterator of pyarrow.Table or pyarrow.RecordBatch objects."
]
},
"UDTF_CONSTRUCTOR_INVALID_IMPLEMENTS_ANALYZE_METHOD": {
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index abdc956188e4..b8a9976699be 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -2066,11 +2066,7 @@ def read_udtf(pickleSer, infile, eval_type):
# Arrow UDTF should only return Arrow types (RecordBatch/Table)
raise PySparkRuntimeError(
errorClass="UDTF_ARROW_TYPE_CONVERSION_ERROR",
- messageParameters={
- "data": str(item),
- "schema": return_type.simpleString(),
- "arrow_schema": str(arrow_return_type),
- },
+ messageParameters={},
)
return batches
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org