You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/07/27 17:53:28 UTC
[PR] [SPARK-58381][SQL] Gate the Arrow cache zero-copy write path on physical congruence with the cache schema [spark]
viirya opened a new pull request, #57577:
URL: https://github.com/apache/spark/pull/57577
### What changes were proposed in this pull request?
Replace the blocklist that decides zero-copy eligibility on the Arrow cache write path (`ColumnarBatchToArrowCachedBatchIterator`) with a per-column physical-congruence check against the cache schema, using `ArrowUtils.isCompatibleWithDeclaredField` (introduced by SPARK-58258 for the columnar Python UDF pass-through). Input vectors whose field trees are not physically congruent with the corresponding cache schema field now take the row-based conversion path, which rewrites the values through `ArrowWriter` under the cache schema. The `containsCacheSchemaMismatch` blocklist is removed.
### Why are the changes needed?
The zero-copy path serializes the input vectors' buffers verbatim under the cache's own schema: `serializeBatch` writes only the record batch, and the read path reconstructs the schema from `cacheSchema` and loads the buffers positionally into it. The blocklist covered only large var-width vectors and interchange-shaped nanosecond timestamp / `CalendarInterval` vectors, so every other physical divergence was forwarded verbatim and silently reinterpreted under the canonical layout when the cached batch is read back -- the same defect class SPARK-58258 eliminated on the UDF read side, one lifecycle stage earlier. Shapes the blocklist missed include view and dictionary encodings, list offset widths (`LargeList`, `ListView`), tagged structs carrying extra children, map entry children in the wrong order, and timestamp units.
The corruption is concrete: an Arrow map vector whose entry-struct children sit in `[value, key]` order is tolerated by Arrow and read correctly by `ArrowColumnVector` (key/value are addressed by name), but its buffers are positional, so a cached `{123: 7}` comes back as `{7: 123}` -- silently, and for the lifetime of the cached relation. The new test demonstrates this: it fails on master with `Array((7, 123)) did not equal Array((123, 7))` and passes with the gate.
In-tree columnar producers happened to be covered by the blocklist (the cache scan itself produces canonical vectors; Python UDF output's large var-width and interchange nanos/interval shapes were the blocked entries), so this is reachable today only from external Arrow-backed DSv2 columnar sources -- `supportsColumnarInput` accepts any of them. The congruence check closes the whole class rather than the enumerated instances, and keeps the write gate symmetric with the read-side gate from SPARK-58258.
A false reject is safe by construction: the row-based fallback re-encodes values under the cache schema (with its value-domain guards), so eligible batches lose only the zero-copy optimization, never correctness.
### Does this PR introduce _any_ user-facing change?
No. The Arrow cache serializer (SPARK-57268) is unreleased.
### How was this patch tested?
New test in `ArrowCachedBatchSerializerSuite` that builds a spec-tolerated but non-canonical map vector (entry children `[value, key]`), verifies the source reads correctly, round-trips it through the serializer's columnar write and read paths, and asserts the values survive. Without the fix it fails with the keys and values swapped. Full `ArrowCachedBatchSerializerSuite` passes (72 tests), confirming canonical shapes remain zero-copy eligible.
### Was this patch authored or co-authored using generative AI tooling?
Yes, this pull request and its description were written by Claude Code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58381][SQL] Gate the Arrow cache zero-copy write path on physical congruence with the cache schema [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57577:
URL: https://github.com/apache/spark/pull/57577#issuecomment-5110722487
Thanks @uros-b @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58381][SQL] Gate the Arrow cache zero-copy write path on physical congruence with the cache schema [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57577:
URL: https://github.com/apache/spark/pull/57577#issuecomment-5110719057
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/ece8eacb8f44742395f39e67a0714eca2efc4a3e
- merged into branch-4.x https://github.com/apache/spark/commit/dbea368bbc6b9400fcec70a07c9a8584a799ad6b
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58381][SQL] Gate the Arrow cache zero-copy write path on physical congruence with the cache schema [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #57577: [SPARK-58381][SQL] Gate the Arrow cache zero-copy write path on physical congruence with the cache schema
URL: https://github.com/apache/spark/pull/57577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58381][SQL] Gate the Arrow cache zero-copy write path on physical congruence with the cache schema [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57577:
URL: https://github.com/apache/spark/pull/57577#issuecomment-5106955202
Thank you @viirya!
Please resolve conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58381][SQL] Gate the Arrow cache zero-copy write path on physical congruence with the cache schema [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #57577:
URL: https://github.com/apache/spark/pull/57577#issuecomment-5107005978
Thank you @uros-b. I resolved the conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org