[PR] fix: handle NullType struct fields in JVM columnar shuffle [datafusion-comet]

["jadenfix (via GitHub)" <gi...@apache.org>]

jadenfix opened a new pull request, #7099:
URL: https://github.com/apache/datafusion-comet/pull/7099

   ## Which issue does this PR close?
   
   Closes #7062.
   
   ## Rationale for this change
   
   A shuffle on the JVM columnar shuffle path fails the query when a column is a struct with a `NullType` field, such as `named_struct('x', a, 'n', null)`. The two field-major struct functions in `native/shuffle/src/spark_unsafe/row.rs` have no `DataType::Null` arm and fall through to `unreachable!`, while `append_field` in the same file already handles it. With the default configuration this affects every round-robin repartition of such a column.
   
   ## What changes are included in this PR?
   
   - `append_struct_fields_field_major` (top-level struct column) and `append_nested_struct_fields_field_major` (struct inside a struct) get a `DataType::Null` arm that appends one null per row to the field's `NullBuilder`, the same as `append_field`.
   - Rust unit tests that lay out Spark `UnsafeRow`s for a top-level and a nested struct with a `NullType` field, including null structs. Both panic with the message from the issue without the fix.
   - A regression test in `CometColumnarShuffleSuite` that shuffles both shapes with round-robin and hash partitioning and compares against Spark.
   
   ## How are these changes tested?
   
   - `cd native && cargo test -p datafusion-comet-shuffle`: 189 passed, including the two new tests (which fail on `main`).
   - `cargo fmt --all -- --check` and `cargo clippy --all-targets -p datafusion-comet-shuffle -- -D warnings` are clean.
   - The new Scala test has not been run locally (no JDK on this machine); it relies on CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] fix: handle NullType struct fields in JVM columnar shuffle [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove merged PR #7099:
URL: https://github.com/apache/datafusion-comet/pull/7099


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org