You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rich7420 (via GitHub)" <gi...@apache.org> on 2026/09/22 08:05:23 UTC
[PR] feat: support map inputs for explode [datafusion-comet]
rich7420 opened a new pull request, #6107:
URL: https://github.com/apache/datafusion-comet/pull/6107
## Which issue does this PR close?
Closes #2837.
## Rationale for this change
Map inputs to `explode` and `posexplode` currently fall back to Spark.
## What changes are included in this PR?
Support maps in all four explode variants by sharing their entry buffers with the existing unnest path. Preserve field metadata, entry order and outer NULL rows. Add regression tests and update support documentation.
## How are these changes tested?
18 Rust tests and 52 targeted tests on each of Spark 3.5.9 and 4.1.3 pass, covering native execution, NULL/empty maps, nested inputs, sliced arrays and batch boundaries. [Fork CI](https://github.com/rich7420/datafusion-comet/pull/32/checks) passes across Spark profiles and the Spark 4.1 SQL suites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support map inputs for explode [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6107:
URL: https://github.com/apache/datafusion-comet/pull/6107#discussion_r4393666379
##########
spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:
##########
@@ -1928,12 +1928,8 @@ object CometExplodeExec extends CometOperatorSerde[GenerateExec] {
return Unsupported(Some(s"Unsupported generator: ${op.generator.nodeName}"))
}
op.generator.children.head.dataType match {
- case _: ArrayType =>
+ case _: ArrayType | _: MapType =>
Review Comment:
[P2] Bound one map's expansion before repeating carried strings
Could we keep this shape on Spark until the native operator can split an individual map before repeating its carried columns? A single input row with a 2,049-entry `map<int,int>` and a 1 MiB `STRING` payload is enough to fail. `PendingInput::next_chunk_rows` never splits one input row, so `repeat_arrs` calls Arrow `take` for the entire map before `BatchSplitStream` runs. The repeated string reaches the 32-bit offset limit even with a configured batch size of 256.
I reproduced `Arrow error: Offset overflow error: 2147483648` on the first output poll using this head's unchanged `MapEntriesExpr` and `ExplodeExec`, Arrow 59.3.1 and DataFusion 55.2.0. A small control passes. With the same input stored in Parquet, plain Spark 4.1.3 successfully computes `SELECT sum(CAST(length(concat(payload, CAST(key AS STRING))) AS BIGINT)) FROM wide_map LATERAL VIEW explode(m) e AS key, value`. Its executed plan keeps `payload` through `Generate` and consumes the rows in the aggregate.
The base and `branch-1.2` leave map generators on Spark, so this admission change exposes a query failure even though array explode already has the underlying limitation. Please add a wide carried-column regression and split within a map before materialization, or retain a safe fallback for this shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support map inputs for explode [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6107:
URL: https://github.com/apache/datafusion-comet/pull/6107#issuecomment-5958309299
I filed #6549 for the float map key issue from my first comment, so no need to open one. It reproduces on `branch-1.1` with `map_from_entries`, and with duplicate zero keys in `map_from_arrays`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support map inputs for explode [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6107:
URL: https://github.com/apache/datafusion-comet/pull/6107#issuecomment-5955530309
I would like to do a deeper review of this PR before it is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support map inputs for explode [datafusion-comet]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on PR #6107:
URL: https://github.com/apache/datafusion-comet/pull/6107#issuecomment-5988836983
@andygrove could you take a look , please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org