You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/05/21 23:48:22 UTC
[PR] [SPARK-56999][PYSPARK] Fix mapInArrow opaque getInt error by coercing output to declared schema [spark]
Yicong-Huang opened a new pull request, #56049:
URL: https://github.com/apache/spark/pull/56049
### What changes were proposed in this pull request?
Make the `SQL_MAP_ARROW_ITER_UDF` branch in `python/pyspark/worker.py` coerce each
output `pa.RecordBatch` against the UDF's declared output schema via
`ArrowBatchTransformer.enforce_schema`, mirroring every sibling Arrow eval-type
branch. `reorder_by_name` is wired to the existing
`spark.sql.legacy.execution.pandas.groupedMap.assignColumnsByName` config, matching
`SQL_GROUPED_MAP_ARROW_UDF` / `applyInArrow`.
### Why are the changes needed?
Without coercion, an Arrow type mismatch between the UDF output and the declared
output schema surfaces in the JVM as an opaque accessor error:
[UNSUPPORTED_CALL.WITHOUT_SUGGESTION] Cannot call the method "getInt" of the
class "org.apache.spark.sql.vectorized.ArrowColumnVector\$ArrowVectorAccessor".
SQLSTATE: 0A000
Repro:
\`\`\`python
import pyarrow as pa
from pyspark.sql.types import StructType, StructField, IntegerType
def double_x(iter_batches):
for batch in iter_batches:
df = batch.to_pandas()
df["x"] = df["x"] * 2
yield pa.RecordBatch.from_pandas(df[["x"]])
src = spark.createDataFrame([(1,), (2,), (3,)], ["x"]) # x is long (int64)
out = src.mapInArrow(double_x, schema=StructType([StructField("x", IntegerType())]))
out.show()
\`\`\`
\`createDataFrame\` infers \`x\` as \`LongType\`; the pandas roundtrip preserves int64;
the declared schema is int32; the JVM picks \`LongAccessor\` for a \`BigIntVector\`
and the outer scan calls \`getInt\` on it, hitting the no-op base accessor.
### Does this PR introduce _any_ user-facing change?
Yes. \`mapInArrow\` output batches that previously caused an opaque JVM accessor
failure are now coerced in Python:
- Compatible Arrow type mismatches (e.g. int64 -> int32 from a pandas roundtrip)
are cast with \`safecheck=True\`, so overflows still raise.
- Column ordering is reconciled by name under
\`spark.sql.legacy.execution.pandas.groupedMap.assignColumnsByName\` (default
true), consistent with \`applyInArrow\`.
- Incompatible mismatches raise a Python \`RESULT_COLUMN_NAMES_MISMATCH\` /
\`RESULT_COLUMN_TYPES_MISMATCH\` error instead of failing in the JVM.
### How was this patch tested?
\`python/run-tests --testnames pyspark.sql.tests.arrow.test_arrow_map\`
(81 tests pass). Added regression test \`test_coerce_output_type_to_declared_schema\`
covering the int64 -> int32 case; updated \`test_top_level_wrong_order\` and
\`test_nested_extraneous_field\` to reflect the new coerce-by-name behavior.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56999][PYTHON] Fix mapInArrow opaque getInt error by coercing output to declared schema [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56049:
URL: https://github.com/apache/spark/pull/56049#issuecomment-5737934110
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56999][PYSPARK] Fix mapInArrow opaque getInt error by coercing output to declared schema [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #56049:
URL: https://github.com/apache/spark/pull/56049#discussion_r3337877522
##########
python/pyspark/sql/tests/arrow/test_arrow_map.py:
##########
@@ -79,6 +79,26 @@ def func(iterator):
expected = df.collect()
self.assertEqual(actual, expected)
+ def test_coerce_output_type_to_declared_schema(self):
+ # Regression test: when the user yields a batch whose Arrow type does
+ # not match the declared output schema, the worker should coerce it
+ # rather than letting the JVM fail later with an opaque getInt error
+ # on the wrong ArrowColumnVector accessor.
+ from pyspark.sql.types import IntegerType, StructField, StructType
+
+ def double_x(iter_batches):
+ for batch in iter_batches:
+ # The input column is long (int64); produce int64 output even
Review Comment:
Does the input column matter? I saw 3 types here - `type=pa.int64()`, inferred long type for `createDataFrame` and `IntegerType()`. Do all 3 matter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56999][PYTHON] Fix mapInArrow opaque getInt error by coercing output to declared schema [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #56049: [SPARK-56999][PYTHON] Fix mapInArrow opaque getInt error by coercing output to declared schema
URL: https://github.com/apache/spark/pull/56049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org