You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jeffw13 (via GitHub)" <gi...@apache.org> on 2026/09/09 21:09:33 UTC
[I] Support Spark 4 EmptyRelationExec as a native input [datafusion-comet]
jeffw13 opened a new issue, #5819:
URL: https://github.com/apache/datafusion-comet/issues/5819
### What is the problem the feature request solves?
Spark 4.x can replace a query stage that produces no rows with `EmptyRelationExec` during adaptive execution. Comet does not currently recognize this operator as a native input, so supported operators above it can fall back to Spark.
For example, with AQE enabled, an aggregate over an empty grouped result can acquire an `EmptyRelationExec` input after the shuffle completes:
```sql
SELECT count(*), sum(v)
FROM (
SELECT id % 2 AS k, sum(id) AS v
FROM range(0, 10, 1, 2)
WHERE id < 0
GROUP BY id % 2
)
```
The result must remain one row containing `0` and `NULL`. A grouped aggregate over the same empty input must produce no rows.
### Describe the potential solution
Add a columnar `CometEmptyRelationExec` using the existing native Arrow-input path. Preserve the output attributes and zero partitions of Spark's operator, without executing the eliminated logical subtree. Register it on Spark 4.x through a version shim and add an operator enable/disable setting.
Test both the results and the executed plans for aggregates, supported joins, AQE and reused exchanges. Preserve existing type and operator fallback rules.
Parquet writes need an additional guard: admitting this zero-partition input must not route a write through the native writer when that would omit the schema-bearing empty file. Keep Spark's writer for those inputs and add an overwrite/readback regression test.
### Additional context
Related to #5303, which tracks native Parquet writes producing no output file for a zero-partition RDD. The writer fallback here would protect the newly supported input; it would not resolve all cases in that issue.
This adds support for consuming Spark's empty relation. It does not change Spark's rules for discovering or propagating empty relations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support Spark 4 EmptyRelationExec as a native input [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #5819: Support Spark 4 EmptyRelationExec as a native input
URL: https://github.com/apache/datafusion-comet/issues/5819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support Spark 4 EmptyRelationExec as a native input [datafusion-comet]
Posted by "jeffw13 (via GitHub)" <gi...@apache.org>.
jeffw13 commented on issue #5819:
URL: https://github.com/apache/datafusion-comet/issues/5819#issuecomment-5608794151
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org