You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/26 20:41:00 UTC
[I] arrays_zip with two same-named inputs fails with "ArrowArray struct has 2 children (expected 1)" [datafusion-comet]
andygrove opened a new issue, #6251:
URL: https://github.com/apache/datafusion-comet/issues/6251
### Describe the bug
`arrays_zip` over two inputs with the same name fails the task when Comet runs it natively. Spark names each struct field after its input, so `arrays_zip(a, a)` has type `array<struct<a: string, a: string>>`, and Comet's native projection returns that struct fine. Importing the result on the JVM then fails, because Java Arrow keys struct children by name and the two `a` children collapse into one:
```
java.lang.IllegalStateException: ArrowArray struct has 2 children (expected 1)
at org.apache.arrow.util.Preconditions.checkState(Preconditions.java:562)
at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:92)
at org.apache.arrow.c.ArrayImporter.importChild(ArrayImporter.java:83)
at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:101)
at org.apache.arrow.c.ArrayImporter.importArray(ArrayImporter.java:68)
at org.apache.arrow.c.ArrowImporter.importVector(ArrowImporter.java:62)
at org.apache.comet.vector.NativeUtil.importVector(NativeUtil.scala:264)
at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:211)
at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:237)
```
`arrays_zip(a, b)` over the same table works. Spark returns the zipped rows for all three queries below.
This is the same Java Arrow limitation as #1015 and #5605. `CometCreateNamedStruct.getSupportLevel` already returns `Unsupported` when `names` has duplicates, and `DataTypeSupport` rejects such structs in operator schemas, but `CometArraysZip.getSupportLevel` only checks the input types and never looks at `expr.names`. Two same-named inputs come up naturally after a join, for example `arrays_zip(t1.tags, t2.tags)`.
### Steps to reproduce
Default configs, reproduced on `main` at `f7952de73` with Spark 4.1.3:
```scala
spark.range(4)
.selectExpr("id", "array(cast(id as string), 'x') as a", "array(id, id + 1) as b")
.write.parquet(path)
val df = spark.read.parquet(path)
df.selectExpr("id", "arrays_zip(a, a) AS r").collect() // IllegalStateException
df.selectExpr("id", "arrays_zip(b, b) AS r").collect() // IllegalStateException
df.selectExpr("id", "arrays_zip(a, b) AS r").collect() // matches Spark
```
The plan is `CometProject` over `CometNativeScan`.
### Expected behavior
Either match Spark or fall back. The smallest fix is probably for `CometArraysZip.getSupportLevel` to return `Unsupported` when `expr.names` has duplicates, the same way `CometCreateNamedStruct` does.
### Additional context
Found while reviewing #6036, but unrelated to that PR's change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] arrays_zip with two same-named inputs fails with "ArrowArray struct has 2 children (expected 1)" [datafusion-comet]
Posted by "mohitgurav20 (via GitHub)" <gi...@apache.org>.
mohitgurav20 commented on issue #6251:
URL: https://github.com/apache/datafusion-comet/issues/6251#issuecomment-5852759035
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] arrays_zip with two same-named inputs fails with "ArrowArray struct has 2 children (expected 1)" [datafusion-comet]
Posted by "mohitgurav20 (via GitHub)" <gi...@apache.org>.
mohitgurav20 commented on issue #6251:
URL: https://github.com/apache/datafusion-comet/issues/6251#issuecomment-5852758497
Hi @andygrove, I would like to take this up and implement the fix!
### Root Cause & Proposed Fix
The issue occurs because Java Arrow's `ArrayImporter` indexes struct children by field name during JVM vector import, causing duplicate struct field names (e.g., from `arrays_zip(a, a)`) to collapse and throw `java.lang.IllegalStateException: ArrowArray struct has 2 children (expected 1)`.
Following the established pattern in `CometCreateNamedStruct`:
1. **SerDe Validation**: Update `CometArraysZip.getSupportLevel` in `spark/src/main/scala/org/apache/comet/serde/arrays.scala` to evaluate `expr.names` and check for duplicate field names:
```scala
val names = expr.names.map(_.eval(EmptyRow))
if (names.length != names.distinct.length) {
return Unsupported(Some("`arrays_zip` with duplicate field names is not supported"))
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] arrays_zip with two same-named inputs fails with "ArrowArray struct has 2 children (expected 1)" [datafusion-comet]
Posted by "mohitgurav20 (via GitHub)" <gi...@apache.org>.
mohitgurav20 commented on issue #6251:
URL: https://github.com/apache/datafusion-comet/issues/6251#issuecomment-5873853732
thanks for assigning me here will soon raise a pr , feel free to review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] arrays_zip with two same-named inputs fails with "ArrowArray struct has 2 children (expected 1)" [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #6251:
URL: https://github.com/apache/datafusion-comet/issues/6251#issuecomment-5873464690
Thanks @mohitgurav20
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org