You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "lyne7-sc (via GitHub)" <gi...@apache.org> on 2026/08/18 08:20:07 UTC
[PR] [AURON #2472] Fix nullable struct field null propagation [auron]
lyne7-sc opened a new pull request, #2473:
URL: https://github.com/apache/auron/pull/2473
# Which issue does this PR close?
Closes #2472
# Rationale for this change
Auron does not propagate the parent struct's null bitmap when accessing a child field. As a result, a field of a null struct may incorrectly return a physical value such as `0` instead of `null`.
# What changes are included in this PR?
- Combine the parent struct and child field null bitmaps in `GetIndexedFieldExpr`.
- Preserve the zero-copy path when the parent struct contains no nulls.
# Are there any user-facing changes?
Bug fix only, accessing a field of a null struct now returns null, consistent with vanilla Spark.
# How was this patch tested?
- Add Rust tests covering parent and child null combinations.
- Add a Spark Parquet E2E test for nullable struct field access.
# Was this patch authored or co-authored using generative AI tooling?
- [x] Yes
- [ ] No
If yes, include: `Generated-by: gpt-5.6`
ASF guidance: https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2472] Fix nullable struct field null propagation [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #2473:
URL: https://github.com/apache/auron/pull/2473#issuecomment-5364056176
@lyne7-sc Thanks for the contribution! Merged into trunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2472] Fix nullable struct field null propagation [auron]
Posted by "lyne7-sc (via GitHub)" <gi...@apache.org>.
lyne7-sc commented on code in PR #2473:
URL: https://github.com/apache/auron/pull/2473#discussion_r3829286926
##########
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronQuerySuite.scala:
##########
@@ -542,6 +541,18 @@ class AuronQuerySuite extends AuronQueryTest with BaseAuronSQLSuite with AuronSQ
}
}
+ test("get field from nullable struct propagates parent null") {
+ withTable("t_nullable_struct") {
+ sql("""
+ |create table t_nullable_struct using parquet as
Review Comment:
Thanks for verifying this. I rechecked the orc-rust implementation, and I think the orc path may not trigger this issue.
`orc-rust` passes the struct’s present bitmap to its child decoders, so parent-null positions are already marked null in the child arrow array. Therefore, the old `GetIndexedFieldExpr` also returns the correct result for orc because null propagation has already happened during decoding.
however, the parquet reader can keep the parent validity only on the `StructArray`, which exposes the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2472] Fix nullable struct field null propagation [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 merged PR #2473:
URL: https://github.com/apache/auron/pull/2473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2472] Fix nullable struct field null propagation [auron]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #2473:
URL: https://github.com/apache/auron/pull/2473#discussion_r3827114552
##########
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronQuerySuite.scala:
##########
@@ -542,6 +541,18 @@ class AuronQuerySuite extends AuronQueryTest with BaseAuronSQLSuite with AuronSQ
}
}
+ test("get field from nullable struct propagates parent null") {
+ withTable("t_nullable_struct") {
+ sql("""
+ |create table t_nullable_struct using parquet as
Review Comment:
Thank you for your fix. I verified that this problem only exists using parquet. If it is orc or stored as parquet, this problem will not be triggered. I don’t know if I understand it correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org