You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/30 19:04:41 UTC
[PR] test: cover booleans in native explode output past the first batch [datafusion-comet]
andygrove opened a new pull request, #6473:
URL: https://github.com/apache/datafusion-comet/pull/6473
## Which issue does this PR close?
Closes #6464.
## Rationale for this change
#6464 was found in 1.1.0-rc1. Since #5362 and #5667, the native explode slices each output batch out of the exploded child instead of gathering it with `take`. So the booleans in an exploded struct leave native at a non-zero bit offset, and Arrow Java ignores that offset when it imports the batch (#6288). From the second output batch on, the JVM reads those booleans and their nulls from the wrong bits. An exploded top-level boolean goes wrong the same way when a native `named_struct` wraps it or a boolean Scala UDF reads it.
#6339 already fixed this on `main` by zeroing boolean offsets at every level before export, and the reproducer in #6464 passes on `main`. None of #6339's tests go through explode, though, so this PR adds that coverage. The fix for `branch-1.1` is #6449, the backport of #6339.
## What changes are included in this PR?
Tests only. `CometGenerateExecSuite` gets seven tests, each over a single Parquet file so that one input batch explodes into several output batches:
- `explode`, `explode_outer`, `posexplode` and `posexplode_outer` of 3000 rows of 13 structs, each with a non-null and a nullable boolean field, at the default batch size. At 13 elements a row the output batches are 8190 rows, so the later ones start both on and off a byte boundary.
- One row of 250 structs with `spark.comet.batchSize=100`. The row is unnested in one build, which is then split at rows 100 and 200.
- A native `named_struct` over an exploded boolean.
- A boolean Scala UDF over an exploded boolean. Spark wraps a primitive-argument UDF in `IF(isnull(v), NULL, ...)`, and native evaluates that branch on a filtered copy whenever a batch has a NULL, which would hide the slice. So this array has no NULLs, and a comment on the fixture says why.
## How are these changes tested?
All seven pass on `main` on Spark 4.1 and on Spark 3.4 with Scala 2.12, along with the rest of `CometGenerateExecSuite`. To check that they catch #6464, I rebuilt native with `zero_offsets` limited to the top level, which is all that rc1's `take` covered, and with the UDF bridge inputs left unnormalized. All seven fail there with wrong results, as do #6339's three `CometCodegenSuite` bridge tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: cover booleans in native explode output past the first batch [datafusion-comet]
Posted by "0lai0 (via GitHub)" <gi...@apache.org>.
0lai0 commented on code in PR #6473:
URL: https://github.com/apache/datafusion-comet/pull/6473#discussion_r4149612392
##########
spark/src/test/scala/org/apache/comet/exec/CometGenerateExecSuite.scala:
##########
@@ -659,4 +659,67 @@ class CometGenerateExecSuite extends CometTestBase {
}
}
+ // The native explode slices each output batch out of the exploded child instead of gathering
+ // it, so an exploded boolean, or a boolean field of an exploded struct, leaves native at a
+ // non-zero bit offset. Arrow Java ignores that offset on import, so native has to zero it at
+ // every level before export, including in a struct built over the booleans and in the input to
+ // a Scala UDF, or they come back wrong after the first output batch.
+ // https://github.com/apache/datafusion-comet/issues/6464
+ private def withBooleanArrays(numRows: Int, arrayLength: Int)(f: => Unit): Unit = {
+ withTempPath { dir =>
+ // One file, so a single input batch explodes into several output batches.
+ withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+ spark
+ .range(0, numRows, 1, 1)
Review Comment:
Strict Scala warnings (Spark 3.5, JDK 17) fails to compile this line with implicit numeric widening: `Dataset.range` widens the first three `Int` arguments to `Long`. Could you change the fixture to `withBooleanArrays(numRows: Long, arrayLength: Long)` and call `spark.range(0L, numRows, 1L, 1)`?
The three call sites already pass integer literals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: cover booleans in native explode output past the first batch [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6473:
URL: https://github.com/apache/datafusion-comet/pull/6473#discussion_r4150045926
##########
spark/src/test/scala/org/apache/comet/exec/CometGenerateExecSuite.scala:
##########
@@ -659,4 +659,67 @@ class CometGenerateExecSuite extends CometTestBase {
}
}
+ // The native explode slices each output batch out of the exploded child instead of gathering
+ // it, so an exploded boolean, or a boolean field of an exploded struct, leaves native at a
+ // non-zero bit offset. Arrow Java ignores that offset on import, so native has to zero it at
+ // every level before export, including in a struct built over the booleans and in the input to
+ // a Scala UDF, or they come back wrong after the first output batch.
+ // https://github.com/apache/datafusion-comet/issues/6464
+ private def withBooleanArrays(numRows: Int, arrayLength: Int)(f: => Unit): Unit = {
+ withTempPath { dir =>
+ // One file, so a single input batch explodes into several output batches.
+ withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+ spark
+ .range(0, numRows, 1, 1)
Review Comment:
Done in 310a7eacf9. The fixture now takes `Long` arguments and calls `spark.range(0L, numRows, 1L, 1)`.
The full reactor Spark 3.5/JDK 17 strict-warning compilation passes locally: `./mvnw -o -B test-compile -Pspark-3.5 -Pstrict-warnings -DskipTests`. CI is still running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: cover booleans in native explode output past the first batch [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6473:
URL: https://github.com/apache/datafusion-comet/pull/6473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org