You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2026/04/08 13:55:18 UTC
[PR] Hybrid eager/deferred accumulation for string_agg GroupsAccumulator to reduce copying and memory usage [datafusion]
kosiew opened a new pull request, #21469:
URL: https://github.com/apache/datafusion/pull/21469
## Which issue does this PR close?
* Part of #21156
---
## Rationale for this change
The current `StringAggGroupsAccumulator` eagerly copies every input string into per-group `String` buffers during `update_batch`. This approach is simple but can lead to significant memory overhead and unnecessary copying, especially for large payloads or high-cardinality groupings.
This PR introduces a hybrid strategy that defers copying when it is likely to be beneficial. Instead of immediately materializing strings, the accumulator can retain references to input batches and store lightweight `(group_idx, row_idx)` entries. Actual string concatenation is deferred until `evaluate()`.
This approach aims to:
* Reduce memory duplication for large strings
* Improve performance for workloads with many groups and large payloads
* Maintain efficiency for small inputs via an eager fast path
---
## What changes are included in this PR?
* Introduced a hybrid accumulation model:
* **Eager path**: existing behavior for small inputs
* **Deferred path**: stores references to input batches and row indices
* Added new internal structures:
* `batches: Vec<ArrayRef>` to retain input arrays
* `batch_entries: Vec<Vec<(u32, u32)>>` to track `(group_idx, row_idx)` pairs
* `num_groups` to track total group count
* Introduced `StringInputArray` abstraction to unify handling of:
* `Utf8`
* `LargeUtf8`
* `Utf8View`
* Implemented heuristics to decide when to defer:
* `DEFER_GROUP_THRESHOLD`
* `DEFER_PAYLOAD_LEN_THRESHOLD`
* Refactored append logic:
* `append_rows_typed` for deferred indexing
* `append_batch_typed` for eager materialization
* `append_batch_values_typed` for reconstruction during evaluation
* Updated `evaluate()` to:
* Materialize deferred data into output
* Support partial emits (`EmitTo::First`)
* Retain un-emitted state correctly
* Added state management improvements:
* `clear_state()` to fully reset buffers
* `retain_after_emit()` to compact deferred state after partial emits
* Extended memory accounting in `size()` to include:
* retained batches
* batch entry metadata
* Added test:
* `groups_mixed_eager_and_deferred_batches` to validate correctness of hybrid behavior
---
## Are these changes tested?
Yes.
* Added a new test covering mixed eager and deferred batches
* Existing tests continue to pass
* The new test verifies:
* Correct concatenation across eager + deferred inputs
* Correct handling of partial emits
* Proper retention of remaining groups after emission
---
## Are there any user-facing changes?
No.
* This change is internal to the execution engine
* No API or SQL behavior changes
* Expected improvements are limited to performance and memory usage characteristics
---
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] string_agg: introduce deferred copying for GroupsAccumulator with mixed eager/deferred paths [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21469:
URL: https://github.com/apache/datafusion/pull/21469#issuecomment-4655399177
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] string_agg: introduce deferred copying for GroupsAccumulator with mixed eager/deferred paths [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew closed pull request #21469: string_agg: introduce deferred copying for GroupsAccumulator with mixed eager/deferred paths
URL: https://github.com/apache/datafusion/pull/21469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Hybrid eager/deferred accumulation for string_agg GroupsAccumulator to reduce copying and memory usage [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #21469:
URL: https://github.com/apache/datafusion/pull/21469#issuecomment-4206785485
Benchmark (#21437)
```
Criterion Benchmark Summary (Statistically Significant Changes)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark ┃ Mean Change ┃ P-value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ aggregate_query_approx_percentile_cont_on_f32 │ -4.93% │ 0.000000 │
│ aggregate_query_approx_percentile_cont_on_u64 │ -4.68% │ 0.000000 │
│ aggregate_query_distinct_median │ -1.59% │ 0.000000 │
│ aggregate_query_group_by │ -4.48% │ 0.000000 │
│ aggregate_query_group_by_u64 15 12 │ -1.34% │ 0.000000 │
│ aggregate_query_group_by_u64_multiple_keys │ -3.66% │ 0.000000 │
│ aggregate_query_group_by_wide_u64_and_f32_without_aggregate_expressions │ -5.07% │ 0.000000 │
│ (aggregate_query_group_by_wide_u64_and_f32_without_aggregate_expr) │ │ │
│ aggregate_query_group_by_wide_u64_and_string_without_aggregate_expressions │ -4.94% │ 0.000000 │
│ (aggregate_query_group_by_wide_u64_and_string_without_aggregate_e) │ │ │
│ aggregate_query_group_by_with_filter │ -3.93% │ 0.000000 │
│ aggregate_query_no_group_by_count_distinct_wide │ -4.73% │ 0.000000 │
│ aggregate_query_no_group_by_min_max_f64 │ -3.28% │ 0.000000 │
│ array_agg_query_group_by_few_groups │ -3.85% │ 0.000000 │
│ array_agg_query_group_by_many_groups │ -3.06% │ 0.000000 │
│ array_agg_query_group_by_mid_groups │ -5.56% │ 0.000000 │
│ array_agg_struct_query_group_by_mid_groups │ -3.55% │ 0.000000 │
│ first_last_ignore_nulls │ -1.65% │ 0.000000 │
│ first_last_many_columns │ -1.76% │ 0.000000 │
│ first_last_one_column │ -4.00% │ 0.000000 │
│ string_agg_payloads/few_groups/large_1024b (large_1024b) │ -4.48% │ 0.000000 │
│ string_agg_payloads/few_groups/medium_64b (medium_64b) │ -2.78% │ 0.000000 │
│ string_agg_payloads/few_groups/small_3b (small_3b) │ -4.02% │ 0.000000 │
│ string_agg_payloads/many_groups/large_1024b (large_1024b) │ -2.12% │ 0.000000 │
│ string_agg_payloads/many_groups/medium_64b (medium_64b) │ -5.19% │ 0.000000 │
│ string_agg_payloads/many_groups/small_3b (small_3b) │ -4.37% │ 0.000000 │
│ string_agg_payloads/mid_groups/large_1024b (large_1024b) │ -4.01% │ 0.000000 │
│ string_agg_payloads/mid_groups/medium_64b (medium_64b) │ -12.10% │ 0.000000 │
└───────────────────────────────────────────────────────────────────────────────────────────────┴─────────────┴──────────┘
Summary: 26 improvements, 0 regressions (p < 0.05)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org