Re: [I] Dictionary-encoded group keys pay 27-37% aggregation tax vs Utf8View in TPC-H (Q2/Q19/Q20), DF 54 & 55 both [datafusion]

["lxc512157407 (via GitHub)" <gi...@apache.org>]

lxc512157407 commented on issue #24822:
URL: https://github.com/apache/datafusion/issues/24822#issuecomment-5535114049

   Hi @Rich-T-kid, thanks for taking a look.
   
   Yes — I have a WIP branch: https://github.com/lxc512157407/datafusion/tree/fix/dictionary-regression
   
   What it currently addresses is a **hash-layer** regression on the dictionary path: generalizing `hash_utils` over `BuildHasher` split the `hash_dictionary` hot loop into separate monomorphizations (RandomState path vs original), costing ~15% on `dictionary_utf8_int32`. The fix separates "hash the dictionary values" (per-hasher) from "scatter hashes to the key codes" (shared, non-generic hot loop) — commit `3a3bfc57`.
   
   To set expectations honestly: that is one component of the tax in this issue, not all of it. The 27–37% numbers here point at the aggregation path itself — group keys appear to be upcast/normalized per batch rather than grouping on the codes directly (for single-batch aggregation the dictionary codes are already perfect group ids). So the branch should remove part of the gap, and the per-batch normalization is the bigger remaining suspect.
   
   The branch is from April and is ~1500 commits behind current main; I'll rebase it and re-run the Q2/Q19/Q20 numbers from the issue on top. Happy to share the standalone bench (same batches, only the group-key column encoding differs) if you want to poke at it yourself — and glad to collaborate on the group-by-on-codes direction.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org