You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2025/03/24 10:22:03 UTC

[I] Support zero copy hash repartitioning inside Hash Aggregate [datafusion]

Dandandan opened a new issue, #15383:
URL: https://github.com/apache/datafusion/issues/15383

   ### Is your feature request related to a problem or challenge?
   
   ### Is your feature request related to a problem or challenge?
   
   Currently `RepartitionExec: partitioning=Hash` will be added whenever for aggregates in `FinalPartitioned` and `SinglePartitioned`
    
   The benefit is increased parallelism, but at the cost of copying the entire table (in a not-so efficient way).
   
   We should consider lowering the cost of repartitioning by not having to copy the input.
   
   ### Describe the solution you'd like
   
   Instead of repartitioning the input in `RepartitionExec`, support repartitioning the inputs based on a selection vector.
   
   Instead of `taking` the `RecordBatch`, we can consider doing the following:
   
   * Add a (boolean) selection vector as output column for each output partition. I.e. `true` means the row is selected for the partition.
   * The rest of the `RecordBatch` remains unchanged (i.e. no copy).
   * CoalesceBatchesExec is no longer needed for the output (reducing another copy) 
   * In the hash aggregate code handle the selection vector.
   
   ### Describe alternatives you've considered
   
   The partitioning could be done inside the hash aggregate (at the cost of more complexity inside it).
    
   ### Additional context
   
   _No response_
   
   ### Describe the solution you'd like
   
   _No response_
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2751304311

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]

Posted by "zebsme (via GitHub)" <gi...@apache.org>.
zebsme commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2766784298

   > I'm considering another approach. Maybe I shouldn't use `filter_record_batch` ๐Ÿค”. It filters the all column iteratly. I should filter the row when the accumulator `merge_batch` ๐Ÿค” I'll draft another PR for the approach.
   
   Agree with you, we should try to avoid directly operate on the record batch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2766551662

   Based on https://github.com/goldmedal/datafusion/pull/3, I did the some benchmarks(`clieckbench_1`, `h2o_medium`) for it.
   `feat_zero-copy-hash-agg-false` is the branch that disables the configuration.
   `feat_zero-copy-hash-agg` is the branch enabling the configuration.
   
   In conclusion, HashAggregate is slower in the selection vector mode.
   ```
   Comparing feat_zero-copy-hash-agg-false and feat_zero-copy-hash-agg
   --------------------
   Benchmark clickbench_1.json
   --------------------
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Query        โ”ƒ feat_zero-copy-hash-agg-false โ”ƒ feat_zero-copy-hash-agg โ”ƒ        Change โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ QQuery 0     โ”‚                        0.24ms โ”‚                  0.32ms โ”‚  1.33x slower โ”‚
   โ”‚ QQuery 1     โ”‚                       26.98ms โ”‚                 24.56ms โ”‚ +1.10x faster โ”‚
   โ”‚ QQuery 2     โ”‚                       55.89ms โ”‚                 52.55ms โ”‚ +1.06x faster โ”‚
   โ”‚ QQuery 3     โ”‚                       48.20ms โ”‚                 45.62ms โ”‚ +1.06x faster โ”‚
   โ”‚ QQuery 4     โ”‚                      313.79ms โ”‚                347.08ms โ”‚  1.11x slower โ”‚
   โ”‚ QQuery 5     โ”‚                      490.80ms โ”‚                471.41ms โ”‚     no change โ”‚
   โ”‚ QQuery 6     โ”‚                       25.06ms โ”‚                 25.46ms โ”‚     no change โ”‚
   โ”‚ QQuery 7     โ”‚                       28.17ms โ”‚                 27.29ms โ”‚     no change โ”‚
   โ”‚ QQuery 8     โ”‚                      353.53ms โ”‚                406.58ms โ”‚  1.15x slower โ”‚
   โ”‚ QQuery 9     โ”‚                      514.71ms โ”‚                478.99ms โ”‚ +1.07x faster โ”‚
   โ”‚ QQuery 10    โ”‚                      132.73ms โ”‚                130.81ms โ”‚     no change โ”‚
   โ”‚ QQuery 11    โ”‚                      142.59ms โ”‚                143.29ms โ”‚     no change โ”‚
   โ”‚ QQuery 12    โ”‚                      475.75ms โ”‚                493.83ms โ”‚     no change โ”‚
   โ”‚ QQuery 13    โ”‚                      569.90ms โ”‚                630.60ms โ”‚  1.11x slower โ”‚
   โ”‚ QQuery 14    โ”‚                      435.30ms โ”‚                444.02ms โ”‚     no change โ”‚
   โ”‚ QQuery 15    โ”‚                      361.60ms โ”‚                406.62ms โ”‚  1.12x slower โ”‚
   โ”‚ QQuery 16    โ”‚                      825.41ms โ”‚                856.13ms โ”‚     no change โ”‚
   โ”‚ QQuery 17    โ”‚                      752.13ms โ”‚                766.95ms โ”‚     no change โ”‚
   โ”‚ QQuery 18    โ”‚                     1813.04ms โ”‚               1934.07ms โ”‚  1.07x slower โ”‚
   โ”‚ QQuery 19    โ”‚                       40.67ms โ”‚                 41.49ms โ”‚     no change โ”‚
   โ”‚ QQuery 20    โ”‚                      621.14ms โ”‚                625.89ms โ”‚     no change โ”‚
   โ”‚ QQuery 21    โ”‚                      769.98ms โ”‚                749.81ms โ”‚     no change โ”‚
   โ”‚ QQuery 22    โ”‚                     1544.70ms โ”‚               1560.61ms โ”‚     no change โ”‚
   โ”‚ QQuery 23    โ”‚                     4471.51ms โ”‚               4356.12ms โ”‚     no change โ”‚
   โ”‚ QQuery 24    โ”‚                      257.77ms โ”‚                265.81ms โ”‚     no change โ”‚
   โ”‚ QQuery 25    โ”‚                      268.53ms โ”‚                273.24ms โ”‚     no change โ”‚
   โ”‚ QQuery 26    โ”‚                      294.19ms โ”‚                307.36ms โ”‚     no change โ”‚
   โ”‚ QQuery 27    โ”‚                      983.41ms โ”‚                987.90ms โ”‚     no change โ”‚
   โ”‚ QQuery 28    โ”‚                     7514.46ms โ”‚               7533.94ms โ”‚     no change โ”‚
   โ”‚ QQuery 29    โ”‚                      346.70ms โ”‚                344.54ms โ”‚     no change โ”‚
   โ”‚ QQuery 30    โ”‚                      387.65ms โ”‚                405.92ms โ”‚     no change โ”‚
   โ”‚ QQuery 31    โ”‚                      390.81ms โ”‚                427.40ms โ”‚  1.09x slower โ”‚
   โ”‚ QQuery 32    โ”‚                     1597.45ms โ”‚               1987.50ms โ”‚  1.24x slower โ”‚
   โ”‚ QQuery 33    โ”‚                     1753.56ms โ”‚               1863.63ms โ”‚  1.06x slower โ”‚
   โ”‚ QQuery 34    โ”‚                     1950.84ms โ”‚               1945.21ms โ”‚     no change โ”‚
   โ”‚ QQuery 35    โ”‚                      510.78ms โ”‚                560.47ms โ”‚  1.10x slower โ”‚
   โ”‚ QQuery 36    โ”‚                      105.22ms โ”‚                110.02ms โ”‚     no change โ”‚
   โ”‚ QQuery 37    โ”‚                       56.69ms โ”‚                 53.63ms โ”‚ +1.06x faster โ”‚
   โ”‚ QQuery 38    โ”‚                       74.69ms โ”‚                 77.84ms โ”‚     no change โ”‚
   โ”‚ QQuery 39    โ”‚                      189.59ms โ”‚                193.83ms โ”‚     no change โ”‚
   โ”‚ QQuery 40    โ”‚                       24.37ms โ”‚                 24.50ms โ”‚     no change โ”‚
   โ”‚ QQuery 41    โ”‚                       23.01ms โ”‚                 23.45ms โ”‚     no change โ”‚
   โ”‚ QQuery 42    โ”‚                       27.48ms โ”‚                 27.82ms โ”‚     no change โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Benchmark Summary                            โ”ƒ            โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ Total Time (feat_zero-copy-hash-agg-false)   โ”‚ 31570.98ms โ”‚
   โ”‚ Total Time (feat_zero-copy-hash-agg)         โ”‚ 32434.14ms โ”‚
   โ”‚ Average Time (feat_zero-copy-hash-agg-false) โ”‚   734.21ms โ”‚
   โ”‚ Average Time (feat_zero-copy-hash-agg)       โ”‚   754.28ms โ”‚
   โ”‚ Queries Faster                               โ”‚          5 โ”‚
   โ”‚ Queries Slower                               โ”‚         10 โ”‚
   โ”‚ Queries with No Change                       โ”‚         28 โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   --------------------
   Benchmark h2o.json
   --------------------
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Query        โ”ƒ feat_zero-copy-hash-agg-false โ”ƒ feat_zero-copy-hash-agg โ”ƒ       Change โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ QQuery 1     โ”‚                     1053.42ms โ”‚               1044.36ms โ”‚    no change โ”‚
   โ”‚ QQuery 2     โ”‚                     2155.01ms โ”‚               2317.51ms โ”‚ 1.08x slower โ”‚
   โ”‚ QQuery 3     โ”‚                     2275.93ms โ”‚               2611.06ms โ”‚ 1.15x slower โ”‚
   โ”‚ QQuery 4     โ”‚                     1236.64ms โ”‚               1256.03ms โ”‚    no change โ”‚
   โ”‚ QQuery 5     โ”‚                     1608.77ms โ”‚               1892.13ms โ”‚ 1.18x slower โ”‚
   โ”‚ QQuery 6     โ”‚                     1369.68ms โ”‚               1382.62ms โ”‚    no change โ”‚
   โ”‚ QQuery 7     โ”‚                     2258.63ms โ”‚               2548.21ms โ”‚ 1.13x slower โ”‚
   โ”‚ QQuery 8     โ”‚                     3876.11ms โ”‚               3985.41ms โ”‚    no change โ”‚
   โ”‚ QQuery 9     โ”‚                     5989.38ms โ”‚               6721.88ms โ”‚ 1.12x slower โ”‚
   โ”‚ QQuery 10    โ”‚                     3064.89ms โ”‚               3677.05ms โ”‚ 1.20x slower โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Benchmark Summary                            โ”ƒ            โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ Total Time (feat_zero-copy-hash-agg-false)   โ”‚ 24888.45ms โ”‚
   โ”‚ Total Time (feat_zero-copy-hash-agg)         โ”‚ 27436.27ms โ”‚
   โ”‚ Average Time (feat_zero-copy-hash-agg-false) โ”‚  2488.85ms โ”‚
   โ”‚ Average Time (feat_zero-copy-hash-agg)       โ”‚  2743.63ms โ”‚
   โ”‚ Queries Faster                               โ”‚          0 โ”‚
   โ”‚ Queries Slower                               โ”‚          6 โ”‚
   โ”‚ Queries with No Change                       โ”‚          4 โ”‚
   ```
   
   I tried to profile Clickbench QQuery 4:
   When selection vector enable:
   ![Image](https://github.com/user-attachments/assets/62059df9-f49d-416c-a416-42a37fa2d716)
   
   When selection vector disabled:
   ![Image](https://github.com/user-attachments/assets/64600993-bf66-465e-9133-d0468a99ebe6)
   
   In the current implementation, the CPU time of `filter_record_batch` is significantly greater than `take_arrays`.
   Does arrow have a more efficient way to filter a record batch by a boolean array?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2767253437

   > I'm considering another approach. Maybe I shouldn't use filter_record_batch ๐Ÿค”. It filters the all column iteratly. I should filter the row when the accumulator merge_batch ๐Ÿค”
   
   Yes, doing so will copy the entire batch (which is what we try to avoid) and will be slower than `take` (in the end it will do the same).
   I think what we probably want is to get the indices via https://docs.rs/arrow/latest/arrow/buffer/struct.BooleanBuffer.html#method.set_indices so it only will aggregate the values for those indices.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2817168799

   I have another implementation for this issue https://github.com/goldmedal/datafusion/pull/4 
   The concept is that getting the row according to indices in the selection vector instead of going through all the rows in the batch.
   
   Because it may involve many changes, I want to check if the implementations make sense.
   Currently, I only implement `GroupValuesPrimitive::intern` for the group-by values. For the aggregation, I only implement `count` and some aggregations that use `GroupsAccumulator`.
   
   I also did some optimization for the sv-mode repartition https://github.com/apache/datafusion/pull/15423/files#r2051721176.
   
   However, I found the performance won't be better for Clickbench queries 4 and 7. 
   ```
   Query 4: SELECT COUNT(DISTINCT "UserID") FROM hits;
   Query 7: SELECT "AdvEngineID", COUNT(*) FROM hits WHERE "AdvEngineID" <> 0 GROUP BY "AdvEngineID" ORDER BY COUNT(*) DESC;
   
   --------------------
   Benchmark clickbench_1.json
   --------------------
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Query        โ”ƒ feat_hash-agg-sv-disable โ”ƒ feat_hash-agg-sv โ”ƒ    Change โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ QQuery 4     โ”‚                 311.74ms โ”‚         320.72ms โ”‚ no change โ”‚
   โ”‚ QQuery 7     โ”‚                  30.28ms โ”‚          29.01ms โ”‚ no change โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   ```
   I'm not sure if I'm on the right way ๐Ÿค” 
   
   @Dandandan @Rachelint Do you have any suggestions for it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2763293892

   @Dandandan 
   I have a draft https://github.com/goldmedal/datafusion/pull/3 based on #15423 for `HashAggregate`. Could you check if it's heading in the right direction?  
   
   When the selection vector mode is enabled:  
   - `CoalesceBatchesExec` is not added for `FinalPartitioned`.  
   - The selection vector is used to filter the required rows before merging batches.  
   
   The plan looks like this:
   ```
   > create table t(c int) as values (1), (1), (1), (1), (2), (2), (3), (3)
   > explain select count(distinct c) from t;
   +---------------+--------------------------------------------------------------------------------------------------+
   | plan_type     | plan                                                                                             |
   +---------------+--------------------------------------------------------------------------------------------------+
   | logical_plan  | Projection: count(alias1) AS count(DISTINCT t.c)                                                 |
   |               |   Aggregate: groupBy=[[]], aggr=[[count(alias1)]]                                                |
   |               |     Aggregate: groupBy=[[t.c AS alias1]], aggr=[[]]                                              |
   |               |       TableScan: t projection=[c]                                                                |
   | physical_plan | ProjectionExec: expr=[count(alias1)@0 as count(DISTINCT t.c)]                                    |
   |               |   AggregateExec: mode=Final, gby=[], aggr=[count(alias1)]                                        |
   |               |     CoalescePartitionsExec                                                                       |
   |               |       AggregateExec: mode=Partial, gby=[], aggr=[count(alias1)]                                  |
   |               |         AggregateExec: mode=FinalPartitioned, gby=[alias1@0 as alias1], aggr=[]                  |
   |               |           RepartitionExec: partitioning=HashSelectionVector([alias1@0], 12), input_partitions=12 |
   |               |             RepartitionExec: partitioning=RoundRobinBatch(12), input_partitions=1                |
   |               |               AggregateExec: mode=Partial, gby=[c@0 as alias1], aggr=[]                          |
   |               |                 DataSourceExec: partitions=1, partition_sizes=[1]                                |
   |               |                                                                                                  |
   +---------------+--------------------------------------------------------------------------------------------------+
   ```
   
   I'll review more aggregation patterns and add additional tests.
   Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2766751553

   I'm considering another approach. Maybe I shouldn't use `filter_record_batch` ๐Ÿค”. It filters the all column iteratly. I should filter the row when the accumulator `update_batch` ๐Ÿค” 
   I'll draft another PR for the approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]

Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2770996279

   > I'm considering another approach. Maybe I shouldn't use filter_record_batch ๐Ÿค”. It filters the all column iteratly. I should filter the row when the accumulator merge_batch ๐Ÿค”
   
   I think also need to filter rows in `GroupValues::intern`, too.
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Support zero copy hash repartitioning for Hash Aggregate [datafusion]

Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #15383:
URL: https://github.com/apache/datafusion/issues/15383#issuecomment-2817566333

   > However, I found the performance won't be better for Clickbench queries 4 and 7.
   
   I think it may be possible that the test queries can't reflect the improvement well.
   I may try to make some cases this evening.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org