You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/07/07 21:13:01 UTC
[I] Add support for SortAggregateExec [datafusion-comet]
andygrove opened a new issue, #1994:
URL: https://github.com/apache/datafusion-comet/issues/1994
### What is the problem the feature request solves?
I was experimenting with some benchmarking and noticed that we do not yet have a native version of `SortAggregateExec`. This resulted in poor performance due to `CometColumnarToRow` being inserted into the plan.
```
Execute InsertIntoHadoopFsRelationCommand [COMET: Execute InsertIntoHadoopFsRelationCommand is not supported]
+- WriteFiles [COMET: WriteFiles is not supported]
+- Window [COMET: Partitioning and sorting specifications must be the same.]
+- CometSort
+- AQEShuffleRead
+- CometSinkPlaceHolder
+- CometColumnarExchange
+- SortAggregate [COMET: SortAggregate is not supported]
+- CometColumnarToRow
+- CometSort
+- AQEShuffleRead
+- CometColumnarExchange
+- SortAggregate [COMET: SortAggregate is not supported]
+- CometColumnarToRow
+- CometSort
+- CometScan parquet
```
### Describe the potential solution
We should see what is involved in implementing `SortAggregateExec` natively.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for SortAggregateExec [datafusion-comet]
Posted by "rishvin (via GitHub)" <gi...@apache.org>.
rishvin commented on issue #1994:
URL: https://github.com/apache/datafusion-comet/issues/1994#issuecomment-3226301493
I would like to explore this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for SortAggregateExec [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on issue #1994:
URL: https://github.com/apache/datafusion-comet/issues/1994#issuecomment-3057482879
i can try implementing this @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for SortAggregateExec [datafusion-comet]
Posted by "rishvin (via GitHub)" <gi...@apache.org>.
rishvin commented on issue #1994:
URL: https://github.com/apache/datafusion-comet/issues/1994#issuecomment-3316214799
Did some initial investigation on this.
Unlike Spark, Datafusion does not have a separate operator to perform sort aggregation. However, Datafusion's `AggregateExec` internally can mimic Spark's `SortAggregateExec` behaviour.
Found [this related blog](https://datafusion.apache.org/blog/2025/03/11/ordering-analysis/?utm_source=chatgpt.com) post from Datafusion which provides details on how Datafusion detects the sortedness of data and leverages it avoid sorting. This mechanism is used by `AggregateExec` to detect if the incoming data is sorted and leverage it to perform aggregation. If the data is sorted, the `InputOrderMode` is set to `Sorted` ([code link](https://github.com/apache/datafusion/blob/1629420162815e1a725b9be5344fafd1fbe2e5ff/datafusion/physical-plan/src/aggregates/mod.rs#L531)). The `InputOrderMode` helps derive the grouping order, [here](https://github.com/apache/datafusion/blob/1629420162815e1a725b9be5344fafd1fbe2e5ff/datafusion/physical-plan/src/aggregates/order/mod.rs#L44). For sorted grouping data, the `AggregateExec` will not wait to emit until `EOF` is found, it will emit once it sees the next group. This emitting logic is handle [here](https://github.com/apache/datafusion/blob/1629
420162815e1a725b9be5344fafd1fbe2e5ff/datafusion/physical-plan/src/aggregates/order/mod.rs#L56).
Based on this, the initial impression is that we already have native mechanism is place to achieve this sort aggregation. However, Comet does not serialize `SortAggregateExec`, it only serializes `HashAggregateExec` [here](https://github.com/apache/datafusion-comet/blob/f1fb980518342e91fae8358a4de3d76b7bbf9d4d/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala#L1286). We may serialize the sort aggregate plan and leverage the existing `AggregateExec` to do all the heavy lifting.
**Next Steps**
- Do spike to tangibly understand the actual work needed.
- If possible, do benchmarking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org