You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2026/07/31 10:16:46 UTC
[PR] Add per-aggregate argument evaluation metrics for grouped hash aggregation [datafusion]
kosiew opened a new pull request, #24024:
URL: https://github.com/apache/datafusion/pull/24024
#### Which issue does this PR close?
* Part of #23570
---
## Rationale for this change
Grouped hash aggregation currently exposes a single `aggregate_arguments_time` metric for all aggregate argument evaluation. This makes it difficult to determine which aggregate expression is responsible for argument evaluation cost when multiple aggregates are present (for example, `SUM(a)` versus `SUM(b)`).
This change adds per-aggregate argument evaluation metrics while preserving the existing operator-level metric for compatibility.
---
## What changes are included in this PR?
* Add an `AggregateArgumentMetrics` helper that registers one timer per aggregate expression during operator construction.
* Label per-aggregate metrics using the aggregate expression display text (or alias), allowing otherwise identical aggregate functions on different inputs (for example, `SUM(a)` and `SUM(b)`) to be distinguished.
* Wrap aggregate argument evaluation with the corresponding per-aggregate timer in:
* grouped hash aggregation stream
* migrated hash table aggregation paths
* ordered aggregation table paths
* Introduce `OrderedAggregateTableMetrics` to carry both group-by and aggregate argument metrics through ordered aggregation replay/spill paths.
* Factor out a reusable `aggregate_metric_label` helper for consistent metric labels.
* Preserve the existing `aggregate_arguments_time` metric so whole-phase timing continues to be reported.
---
## Are these changes tested?
Yes.
This PR adds and updates tests including:
* `test_groupby_aggregate_argument_metrics_distinguish_inputs`, which verifies that separate metrics are created for `SUM(a)` and `SUM(b)` with distinct metric names and aggregate labels.
* Existing group-by metrics tests updated to use shared aggregate construction helpers.
* Grouped hash aggregation stream tests updated to verify both:
* the existing `aggregate_arguments_time` metric remains present and non-zero, and
* the new per-aggregate metric (`agg_expr_0_arguments_time`) is emitted and records time.
---
## Are there any user-facing changes?
Yes.
`EXPLAIN ANALYZE` and execution plan metrics for the covered grouped hash aggregation paths now include per-aggregate argument evaluation timers (for example, `agg_expr_0_arguments_time`) with aggregate labels identifying the corresponding aggregate expression, while retaining the existing `aggregate_arguments_time` metric for compatibility.
---
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add per-aggregate argument evaluation metrics for grouped hash aggregation [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #24024:
URL: https://github.com/apache/datafusion/pull/24024#issuecomment-5275073452
@rluvaton
Thanks for your review and feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add per-aggregate argument evaluation metrics for grouped hash aggregation [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew merged PR #24024:
URL: https://github.com/apache/datafusion/pull/24024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add per-aggregate argument evaluation metrics for grouped hash aggregation [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #24024:
URL: https://github.com/apache/datafusion/pull/24024#discussion_r3727423946
##########
datafusion/physical-plan/src/aggregates/group_values/metrics.rs:
##########
@@ -19,6 +19,43 @@
use crate::metrics::{ExecutionPlanMetricsSet, MetricBuilder, Time};
+#[derive(Clone)]
+pub(crate) struct AggregateArgumentMetrics {
+ argument_times: Vec<Time>,
+}
+
+impl AggregateArgumentMetrics {
+ pub(crate) fn new<T>(
+ metrics: &ExecutionPlanMetricsSet,
+ partition: usize,
+ aggregate_labels: impl IntoIterator<Item = T>,
+ ) -> Self
+ where
+ T: Into<String>,
+ {
+ let argument_times = aggregate_labels
+ .into_iter()
+ .enumerate()
+ .map(|(idx, label)| {
+ MetricBuilder::new(metrics)
+ .with_new_label("aggregate", label.into())
+ .subset_time(format!("agg_expr_{idx}_arguments_time"), partition)
Review Comment:
Or the unique field name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add per-aggregate argument evaluation metrics for grouped hash aggregation [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #24024:
URL: https://github.com/apache/datafusion/pull/24024#discussion_r3747151790
##########
datafusion/physical-plan/src/aggregates/group_values/metrics.rs:
##########
@@ -19,6 +19,43 @@
use crate::metrics::{ExecutionPlanMetricsSet, MetricBuilder, Time};
+#[derive(Clone)]
+pub(crate) struct AggregateArgumentMetrics {
+ argument_times: Vec<Time>,
+}
+
+impl AggregateArgumentMetrics {
+ pub(crate) fn new<T>(
+ metrics: &ExecutionPlanMetricsSet,
+ partition: usize,
+ aggregate_labels: impl IntoIterator<Item = T>,
+ ) -> Self
+ where
+ T: Into<String>,
+ {
+ let argument_times = aggregate_labels
+ .into_iter()
+ .enumerate()
+ .map(|(idx, label)| {
+ MetricBuilder::new(metrics)
+ .with_new_label("aggregate", label.into())
+ .subset_time(format!("agg_expr_{idx}_arguments_time"), partition)
Review Comment:
@rluvaton
The stable indexed metric name is paired with an `aggregate` label built from the aggregate display/alias (`aggregate_metric_label`). The regression asserts `agg_expr_0_arguments_time, aggregate=SUM(a)` and `agg_expr_1_arguments_time, aggregate=SUM(b)`.
The default view keeps compact indexed names, whose order matches the adjacent `aggr=[...]` list.
`EXPLAIN ANALYZE VERBOSE` renders those labels.
Example:
```
cat >/tmp/aggregate_metrics.sql <<'SQL'
CREATE TABLE t AS
SELECT
CAST(value % 100 AS INT) AS k,
CAST(value AS BIGINT) AS a,
CAST(value * 2 AS BIGINT) AS b,
CAST(value % 3 AS BIGINT) AS c
FROM range(1000000);
EXPLAIN ANALYZE VERBOSE
SELECT k, SUM(a), SUM(b), COUNT(c)
FROM t
GROUP BY k;
SQL
cargo run -p datafusion-cli -- -q -f /tmp/aggregate_metrics.sql
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add per-aggregate argument evaluation metrics for grouped hash aggregation [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #24024:
URL: https://github.com/apache/datafusion/pull/24024#discussion_r3727411574
##########
datafusion/physical-plan/src/aggregates/group_values/metrics.rs:
##########
@@ -19,6 +19,43 @@
use crate::metrics::{ExecutionPlanMetricsSet, MetricBuilder, Time};
+#[derive(Clone)]
+pub(crate) struct AggregateArgumentMetrics {
+ argument_times: Vec<Time>,
+}
+
+impl AggregateArgumentMetrics {
+ pub(crate) fn new<T>(
+ metrics: &ExecutionPlanMetricsSet,
+ partition: usize,
+ aggregate_labels: impl IntoIterator<Item = T>,
+ ) -> Self
+ where
+ T: Into<String>,
+ {
+ let argument_times = aggregate_labels
+ .into_iter()
+ .enumerate()
+ .map(|(idx, label)| {
+ MetricBuilder::new(metrics)
+ .with_new_label("aggregate", label.into())
+ .subset_time(format!("agg_expr_{idx}_arguments_time"), partition)
Review Comment:
Having only the id will make it very annoying for user experience, maybe we can also include the expression name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add per-aggregate argument evaluation metrics for grouped hash aggregation [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24024:
URL: https://github.com/apache/datafusion/pull/24024#issuecomment-5142019866
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24024?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `92.34234%` with `17 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.86%. Comparing base ([`f8b9ed8`](https://app.codecov.io/gh/apache/datafusion/commit/f8b9ed81f0dc8c19eea0c0f73faf1c24c840bab7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a43dd63`](https://app.codecov.io/gh/apache/datafusion/commit/a43dd637067e7dc80ca90b669ff18381665556a5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24024?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ysical-plan/src/aggregates/group\_values/metrics.rs](https://app.codecov.io/gh/apache/datafusion/pull/24024?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fgroup_values%2Fmetrics.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2dyb3VwX3ZhbHVlcy9tZXRyaWNzLnJz) | 88.54% | [1 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24024?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...hysical-plan/src/aggregates/grouped\_hash\_stream.rs](https://app.codecov.io/gh/apache/datafusion/pull/24024?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fgrouped_hash_stream.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2dyb3VwZWRfaGFzaF9zdHJlYW0ucnM=) | 94.73% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24024?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24024 +/- ##
========================================
Coverage 80.86% 80.86%
========================================
Files 1099 1099
Lines 374720 374904 +184
Branches 374720 374904 +184
========================================
+ Hits 303015 303183 +168
- Misses 53614 53623 +9
- Partials 18091 18098 +7
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24024?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add per-aggregate argument evaluation metrics for grouped hash aggregation [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #24024:
URL: https://github.com/apache/datafusion/pull/24024#discussion_r3747151790
##########
datafusion/physical-plan/src/aggregates/group_values/metrics.rs:
##########
@@ -19,6 +19,43 @@
use crate::metrics::{ExecutionPlanMetricsSet, MetricBuilder, Time};
+#[derive(Clone)]
+pub(crate) struct AggregateArgumentMetrics {
+ argument_times: Vec<Time>,
+}
+
+impl AggregateArgumentMetrics {
+ pub(crate) fn new<T>(
+ metrics: &ExecutionPlanMetricsSet,
+ partition: usize,
+ aggregate_labels: impl IntoIterator<Item = T>,
+ ) -> Self
+ where
+ T: Into<String>,
+ {
+ let argument_times = aggregate_labels
+ .into_iter()
+ .enumerate()
+ .map(|(idx, label)| {
+ MetricBuilder::new(metrics)
+ .with_new_label("aggregate", label.into())
+ .subset_time(format!("agg_expr_{idx}_arguments_time"), partition)
Review Comment:
@rluvaton
The stable indexed metric name is paired with an `aggregate` label built from the aggregate display/alias (`aggregate_metric_label`). The regression asserts `agg_expr_0_arguments_time, aggregate=SUM(a)` and `agg_expr_1_arguments_time, aggregate=SUM(b)`.
The default view keeps compact indexed names, whose order matches the adjacent `aggr=[...]` list.
`EXPLAIN ANALYZE VERBOSE` renders those labels.
Example:
```
cat >/tmp/aggregate_metrics.sql <<'SQL'
CREATE TABLE t AS
SELECT
CAST(value % 100 AS INT) AS k,
CAST(value AS BIGINT) AS a,
CAST(value * 2 AS BIGINT) AS b,
CAST(value % 3 AS BIGINT) AS c
FROM range(1000000);
EXPLAIN ANALYZE VERBOSE
SELECT k, SUM(a), SUM(b), COUNT(c)
FROM t
GROUP BY k;
SQL
cargo run -p datafusion-cli -- -q -f /tmp/aggregate_metrics.sql
```
The partial `AggregateExec` per-partition metrics include distinct stable keys and expression labels, e.g.:
```text
agg_expr_0_arguments_time{partition=3, aggregate=sum(t.a)}=...
agg_expr_1_arguments_time{partition=3, aggregate=sum(t.b)}=...
agg_expr_2_arguments_time{partition=3, aggregate=count(t.c)}=...
aggregate_arguments_time{partition=3}=...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org