You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/08/31 14:31:40 UTC
[I] `named_struct` with duplicate field names falls back to Spark [datafusion-comet]
andygrove opened a new issue, #5586:
URL: https://github.com/apache/datafusion-comet/issues/5586
Part of #5572.
`named_struct` with duplicate field names falls the whole projection back to Spark:
```scala
if (expr.names.length != expr.names.distinct.length) {
Unsupported(Some(duplicateNamesReason))
}
```
`spark/src/main/scala/org/apache/comet/serde/structs.scala:42-43`
`docs/source/user-guide/latest/expressions.md` records it as "Duplicate field names fall back".
Spark permits duplicate field names in a struct and `CreateNamedStruct.doGenCode` builds the row positionally, so the dispatcher reproduces it exactly. `StructType` is supported by `CometBatchKernelCodegen.isSupportedDataType` recursively, and duplicate names are not a problem on the Arrow side — the kernel's output field construction is positional too. Worth confirming with a test that the resulting Arrow schema round-trips, since duplicate field names in an Arrow struct are legal but unusual.
Fix: mix `CodegenDispatchFallback` into `CometCreateNamedStruct`.
`CometCreateNamedStruct` also declines inside `convert` at `:66` when a child fails to serialize, which the dispatcher cannot see; see the prerequisite issue on `convert`-side declines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `named_struct` with duplicate field names falls back to Spark [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #5586:
URL: https://github.com/apache/datafusion-comet/issues/5586#issuecomment-5981492343
Closing this as not planned. Since it was filed, #5766 made the codegen dispatcher decline structs with repeated field names, so mixing `CodegenDispatchFallback` into `CometCreateNamedStruct` would no longer route these through the dispatcher. Main now declines duplicate-name structs wherever they would reach Java Arrow, and the enclosing operator falls back to Spark. Running them in Comet would need the Arrow boundary changes from #5603, which we decided not to take on (see the closing comment there). The cases that still fail are tracked in #6251, #6591 and #6592.
Thanks @RRXXZZYY for picking this up and for all the work in #5603.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `named_struct` with duplicate field names falls back to Spark [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #5586: `named_struct` with duplicate field names falls back to Spark
URL: https://github.com/apache/datafusion-comet/issues/5586
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `named_struct` with duplicate field names falls back to Spark [datafusion-comet]
Posted by "RRXXZZYY (via GitHub)" <gi...@apache.org>.
RRXXZZYY commented on issue #5586:
URL: https://github.com/apache/datafusion-comet/issues/5586#issuecomment-5482269377
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org