You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2025/03/24 10:22:03 UTC
[I] Support zero copy hash repartitioning inside Hash Aggregate [datafusion]
Dandandan opened a new issue, #15383:
URL: https://github.com/apache/datafusion/issues/15383
### Is your feature request related to a problem or challenge?
### Is your feature request related to a problem or challenge?
Currently `RepartitionExec: partitioning=Hash` will be added whenever for aggregates in `FinalPartitioned` and `SinglePartitioned`
The benefit is increased parallelism, but at the cost of copying the entire table (in a not-so efficient way).
We should consider lowering the cost of repartitioning by not having to copy the input.
### Describe the solution you'd like
Instead of repartitioning the input in `RepartitionExec`, support repartitioning the inputs based on a selection vector.
Instead of `taking` the `RecordBatch`, we can consider doing the following:
* Add a (boolean) selection vector as output column for each output partition. I.e. `true` means the row is selected for the partition.
* The rest of the `RecordBatch` remains unchanged (i.e. no copy).
* CoalesceBatchesExec is no longer needed for the output (reducing another copy)
* In the hash aggregate code handle the selection vector.
### Describe alternatives you've considered
The partitioning could be done inside the hash aggregate (at the cost of more complexity inside it).
### Additional context
_No response_
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2751304311
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]
Posted by "zebsme (via GitHub)" <gi...@apache.org>.
zebsme commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2766784298
> I'm considering another approach. Maybe I shouldn't use `filter_record_batch` ๐ค. It filters the all column iteratly. I should filter the row when the accumulator `merge_batch` ๐ค I'll draft another PR for the approach.
Agree with you, we should try to avoid directly operate on the record batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2766551662
Based on https://github.com/goldmedal/datafusion/pull/3, I did the some benchmarks(`clieckbench_1`, `h2o_medium`) for it.
`feat_zero-copy-hash-agg-false` is the branch that disables the configuration.
`feat_zero-copy-hash-agg` is the branch enabling the configuration.
In conclusion, HashAggregate is slower in the selection vector mode.
```
Comparing feat_zero-copy-hash-agg-false and feat_zero-copy-hash-agg
--------------------
Benchmark clickbench_1.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ feat_zero-copy-hash-agg-false โ feat_zero-copy-hash-agg โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 0.24ms โ 0.32ms โ 1.33x slower โ
โ QQuery 1 โ 26.98ms โ 24.56ms โ +1.10x faster โ
โ QQuery 2 โ 55.89ms โ 52.55ms โ +1.06x faster โ
โ QQuery 3 โ 48.20ms โ 45.62ms โ +1.06x faster โ
โ QQuery 4 โ 313.79ms โ 347.08ms โ 1.11x slower โ
โ QQuery 5 โ 490.80ms โ 471.41ms โ no change โ
โ QQuery 6 โ 25.06ms โ 25.46ms โ no change โ
โ QQuery 7 โ 28.17ms โ 27.29ms โ no change โ
โ QQuery 8 โ 353.53ms โ 406.58ms โ 1.15x slower โ
โ QQuery 9 โ 514.71ms โ 478.99ms โ +1.07x faster โ
โ QQuery 10 โ 132.73ms โ 130.81ms โ no change โ
โ QQuery 11 โ 142.59ms โ 143.29ms โ no change โ
โ QQuery 12 โ 475.75ms โ 493.83ms โ no change โ
โ QQuery 13 โ 569.90ms โ 630.60ms โ 1.11x slower โ
โ QQuery 14 โ 435.30ms โ 444.02ms โ no change โ
โ QQuery 15 โ 361.60ms โ 406.62ms โ 1.12x slower โ
โ QQuery 16 โ 825.41ms โ 856.13ms โ no change โ
โ QQuery 17 โ 752.13ms โ 766.95ms โ no change โ
โ QQuery 18 โ 1813.04ms โ 1934.07ms โ 1.07x slower โ
โ QQuery 19 โ 40.67ms โ 41.49ms โ no change โ
โ QQuery 20 โ 621.14ms โ 625.89ms โ no change โ
โ QQuery 21 โ 769.98ms โ 749.81ms โ no change โ
โ QQuery 22 โ 1544.70ms โ 1560.61ms โ no change โ
โ QQuery 23 โ 4471.51ms โ 4356.12ms โ no change โ
โ QQuery 24 โ 257.77ms โ 265.81ms โ no change โ
โ QQuery 25 โ 268.53ms โ 273.24ms โ no change โ
โ QQuery 26 โ 294.19ms โ 307.36ms โ no change โ
โ QQuery 27 โ 983.41ms โ 987.90ms โ no change โ
โ QQuery 28 โ 7514.46ms โ 7533.94ms โ no change โ
โ QQuery 29 โ 346.70ms โ 344.54ms โ no change โ
โ QQuery 30 โ 387.65ms โ 405.92ms โ no change โ
โ QQuery 31 โ 390.81ms โ 427.40ms โ 1.09x slower โ
โ QQuery 32 โ 1597.45ms โ 1987.50ms โ 1.24x slower โ
โ QQuery 33 โ 1753.56ms โ 1863.63ms โ 1.06x slower โ
โ QQuery 34 โ 1950.84ms โ 1945.21ms โ no change โ
โ QQuery 35 โ 510.78ms โ 560.47ms โ 1.10x slower โ
โ QQuery 36 โ 105.22ms โ 110.02ms โ no change โ
โ QQuery 37 โ 56.69ms โ 53.63ms โ +1.06x faster โ
โ QQuery 38 โ 74.69ms โ 77.84ms โ no change โ
โ QQuery 39 โ 189.59ms โ 193.83ms โ no change โ
โ QQuery 40 โ 24.37ms โ 24.50ms โ no change โ
โ QQuery 41 โ 23.01ms โ 23.45ms โ no change โ
โ QQuery 42 โ 27.48ms โ 27.82ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (feat_zero-copy-hash-agg-false) โ 31570.98ms โ
โ Total Time (feat_zero-copy-hash-agg) โ 32434.14ms โ
โ Average Time (feat_zero-copy-hash-agg-false) โ 734.21ms โ
โ Average Time (feat_zero-copy-hash-agg) โ 754.28ms โ
โ Queries Faster โ 5 โ
โ Queries Slower โ 10 โ
โ Queries with No Change โ 28 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
--------------------
Benchmark h2o.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโ
โ Query โ feat_zero-copy-hash-agg-false โ feat_zero-copy-hash-agg โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 1053.42ms โ 1044.36ms โ no change โ
โ QQuery 2 โ 2155.01ms โ 2317.51ms โ 1.08x slower โ
โ QQuery 3 โ 2275.93ms โ 2611.06ms โ 1.15x slower โ
โ QQuery 4 โ 1236.64ms โ 1256.03ms โ no change โ
โ QQuery 5 โ 1608.77ms โ 1892.13ms โ 1.18x slower โ
โ QQuery 6 โ 1369.68ms โ 1382.62ms โ no change โ
โ QQuery 7 โ 2258.63ms โ 2548.21ms โ 1.13x slower โ
โ QQuery 8 โ 3876.11ms โ 3985.41ms โ no change โ
โ QQuery 9 โ 5989.38ms โ 6721.88ms โ 1.12x slower โ
โ QQuery 10 โ 3064.89ms โ 3677.05ms โ 1.20x slower โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (feat_zero-copy-hash-agg-false) โ 24888.45ms โ
โ Total Time (feat_zero-copy-hash-agg) โ 27436.27ms โ
โ Average Time (feat_zero-copy-hash-agg-false) โ 2488.85ms โ
โ Average Time (feat_zero-copy-hash-agg) โ 2743.63ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 6 โ
โ Queries with No Change โ 4 โ
```
I tried to profile Clickbench QQuery 4:
When selection vector enable:

When selection vector disabled:

In the current implementation, the CPU time of `filter_record_batch` is significantly greater than `take_arrays`.
Does arrow have a more efficient way to filter a record batch by a boolean array?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2767253437
> I'm considering another approach. Maybe I shouldn't use filter_record_batch ๐ค. It filters the all column iteratly. I should filter the row when the accumulator merge_batch ๐ค
Yes, doing so will copy the entire batch (which is what we try to avoid) and will be slower than `take` (in the end it will do the same).
I think what we probably want is to get the indices via https://docs.rs/arrow/latest/arrow/buffer/struct.BooleanBuffer.html#method.set_indices so it only will aggregate the values for those indices.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2817168799
I have another implementation for this issue https://github.com/goldmedal/datafusion/pull/4
The concept is that getting the row according to indices in the selection vector instead of going through all the rows in the batch.
Because it may involve many changes, I want to check if the implementations make sense.
Currently, I only implement `GroupValuesPrimitive::intern` for the group-by values. For the aggregation, I only implement `count` and some aggregations that use `GroupsAccumulator`.
I also did some optimization for the sv-mode repartition https://github.com/apache/datafusion/pull/15423/files#r2051721176.
However, I found the performance won't be better for Clickbench queries 4 and 7.
```
Query 4: SELECT COUNT(DISTINCT "UserID") FROM hits;
Query 7: SELECT "AdvEngineID", COUNT(*) FROM hits WHERE "AdvEngineID" <> 0 GROUP BY "AdvEngineID" ORDER BY COUNT(*) DESC;
--------------------
Benchmark clickbench_1.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Query โ feat_hash-agg-sv-disable โ feat_hash-agg-sv โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 4 โ 311.74ms โ 320.72ms โ no change โ
โ QQuery 7 โ 30.28ms โ 29.01ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
```
I'm not sure if I'm on the right way ๐ค
@Dandandan @Rachelint Do you have any suggestions for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2763293892
@Dandandan
I have a draft https://github.com/goldmedal/datafusion/pull/3 based on #15423 for `HashAggregate`. Could you check if it's heading in the right direction?
When the selection vector mode is enabled:
- `CoalesceBatchesExec` is not added for `FinalPartitioned`.
- The selection vector is used to filter the required rows before merging batches.
The plan looks like this:
```
> create table t(c int) as values (1), (1), (1), (1), (2), (2), (3), (3)
> explain select count(distinct c) from t;
+---------------+--------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+--------------------------------------------------------------------------------------------------+
| logical_plan | Projection: count(alias1) AS count(DISTINCT t.c) |
| | Aggregate: groupBy=[[]], aggr=[[count(alias1)]] |
| | Aggregate: groupBy=[[t.c AS alias1]], aggr=[[]] |
| | TableScan: t projection=[c] |
| physical_plan | ProjectionExec: expr=[count(alias1)@0 as count(DISTINCT t.c)] |
| | AggregateExec: mode=Final, gby=[], aggr=[count(alias1)] |
| | CoalescePartitionsExec |
| | AggregateExec: mode=Partial, gby=[], aggr=[count(alias1)] |
| | AggregateExec: mode=FinalPartitioned, gby=[alias1@0 as alias1], aggr=[] |
| | RepartitionExec: partitioning=HashSelectionVector([alias1@0], 12), input_partitions=12 |
| | RepartitionExec: partitioning=RoundRobinBatch(12), input_partitions=1 |
| | AggregateExec: mode=Partial, gby=[c@0 as alias1], aggr=[] |
| | DataSourceExec: partitions=1, partition_sizes=[1] |
| | |
+---------------+--------------------------------------------------------------------------------------------------+
```
I'll review more aggregation patterns and add additional tests.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2766751553
I'm considering another approach. Maybe I shouldn't use `filter_record_batch` ๐ค. It filters the all column iteratly. I should filter the row when the accumulator `update_batch` ๐ค
I'll draft another PR for the approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2770996279
> I'm considering another approach. Maybe I shouldn't use filter_record_batch ๐ค. It filters the all column iteratly. I should filter the row when the accumulator merge_batch ๐ค
I think also need to filter rows in `GroupValues::intern`, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2817566333
> However, I found the performance won't be better for Clickbench queries 4 and 7.
I think it may be possible that the test queries can't reflect the improvement well.
I may try to make some cases this evening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org