You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/04/29 16:10:53 UTC

[PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

adriangb opened a new pull request, #21931:
URL: https://github.com/apache/datafusion/pull/21931

   ## Which issue does this PR close?
   
   - Refs #19858 — "Pushing down HashJoinExec build side dynamic filters makes tpch queries slower". This PR addresses the structural side of the problem; the adaptive selectivity work (#19639) is complementary.
   
   ## Rationale for this change
   
   Today the `Partitioned`-mode `HashJoinExec` builds a dynamic filter that's structured around the *repartition* layout:
   
   ```
   CASE hash_repartition % N
     WHEN 0 THEN p0_bounds AND (p0_inlist | p0_hash_lookup)
     WHEN 1 THEN p1_bounds AND (p1_inlist | p1_hash_lookup)
     ...
     ELSE false
   END
   ```
   
   Two problems with this:
   
   1. **Per-row routing-hash cost.** Every probe row pays a `hash_repartition % N` even though the partition's hash table will be probed anyway with a different seed (`HASH_JOIN_SEED`).
   2. **Coupling.** The dynamic filter's shape depends on how the build side was repartitioned, even though semantically the filter is just "is this key somewhere in the build side?"
   
   This PR replaces the routing CASE with a structure that depends only on the *content* of the build side, not its layout. It also adds a small cross-partition merged `IN (SET)` fast path so small joins can participate in parquet stats / bloom-filter pruning at the scan side.
   
   ## What changes are included in this PR?
   
   Four commits:
   
   1. **`perf: collapse all-Map dynamic filter into MultiMapLookupExpr`** — new `MultiMapLookupExpr` that hashes the join keys once with `HASH_JOIN_SEED` and ORs `contain_hashes()` across every reported partition's hash table. Replaces CASE for the all-Map shape; ~halves the regression on TPC-H SF=1 (+3.0% → +1.6% TOTAL min vs. dynamic-filter-off).
   
   2. **`perf: collapse small all-InList dynamic filter into one cross-partition IN (SET)`** — when every reported partition contributed an InList array and the cross-partition union is small enough, concatenate them into a single global `IN (SET)`. The point isn't runtime selectivity but parquet stats / bloom-filter pruning at the scan, which a `multi_hash_lookup` can't do. A cap sweep at SF=1 picked 20–50 as the runtime sweet spot (past ~200 the larger `static_filter` hash set blows out of L1).
   
   3. **`refactor: drop CASE routing from Partitioned dynamic filters`** — `PushdownStrategy` is now a struct that always carries the `Map` (the join's hash table is built unconditionally) plus an optional InList array. With the map always available, every reported partition can contribute to `multi_hash_lookup`, and the routing CASE can go away entirely. Removes `repartition_random_state` from `SharedBuildAccumulator`, drops the `REPARTITION_RANDOM_STATE` import in `exec.rs`, and unifies the `force_hash_collisions` snapshot — both branches now produce the same shape.
   
   4. **`refactor: dedup cross-partition InList, reuse per-partition cap`** — combine path explicitly deduplicates by `ScalarValue` (HashSet first-seen walk + one `arrow::compute::take`) and re-gates on distinct count. The same `optimizer.hash_join_inlist_pushdown_max_distinct_values` knob now caps both per-partition InList eligibility and the cross-partition merged set. Default lowered from 150 → 20 to align with parquet stats / bloom-filter pruning practicality.
   
   ### Final filter-shape matrix (after the PR)
   
   | input | shape |
   |-------|-------|
   | `enable_dynamic_filter_pushdown=false` | no filter installed |
   | `CollectLeft`, build empty | filter stays at `lit(true)` |
   | `CollectLeft`, build small (under inlist caps) | `bounds AND IN (SET)` |
   | `CollectLeft`, build large | `bounds AND hash_lookup` |
   | `Partitioned`, all reported partitions empty | `lit(false)` |
   | `Partitioned`, any partition canceled | `lit(true)` |
   | `Partitioned`, every partition InList AND combined ≤ cap | `bounds AND IN (SET)` (parquet-prunable) |
   | `Partitioned`, anything else | `bounds AND multi_hash_lookup` |
   
   No more `CASE`, no more `hash_repartition`, no more `REPARTITION_RANDOM_STATE` in the dynamic-filter path.
   
   ## Are these changes tested?
   
   - Existing `joins::hash_join` lib tests pass (380 tests).
   - `physical_optimizer::filter_pushdown` integration tests pass (51 tests). One snapshot updated:
     - `test_hashjoin_dynamic_filter_pushdown_partitioned-2` (the no-`force_hash_collisions` snapshot) now matches the merged `IN (SET)` shape that the `force_hash_collisions` snapshot already had — the size-gated path subsumes that special case.
     - `test_hashjoin_hash_table_pushdown_partitioned` now positively asserts `multi_hash_lookup` (and the absence of `hash_repartition`) instead of just `hash_lookup`.
   - `push_down_filter_parquet.slt` passes.
   - `cargo clippy -p datafusion-physical-plan --all-targets -- -D warnings` is clean.
   
   ### Bench (TPC-H SF=1, 7 iters back-to-back)
   
   Reduction in dynamic-filter-pushdown regression vs. the legacy CASE design:
   
   ```
   TOTAL min vs no_DF:
     legacy CASE:        +3.0%
     this PR (final):    +1.6%   (~halves the regression)
   ```
   
   Per-query wins (this PR vs legacy CASE) include Q4 (-6.0%), Q5 (-3.3%), Q7 (-6.0%), Q8 (-4.0%), Q9 (-3.5%), Q12 (-3.2%), Q17 (-1.6%), Q21 (-3.8%). A handful of small queries (Q11, Q14) regress 1–4 % because the moderate-InList shape they used to take through CASE now uses `multi_hash_lookup`; net is still a clear win.
   
   The bigger win that won't show in TPC-H runtime is parquet I/O: the small merged `IN (SET)` path produces a predicate that scans can use for stats/bloom-filter pruning, which the routing CASE never could.
   
   ## Are there any user-facing changes?
   
   - **Default lowered:** `optimizer.hash_join_inlist_pushdown_max_distinct_values` 150 → 20. Affects which build-side shapes choose InList vs. hash-table pushdown per partition, and now also gates the cross-partition merged InList. Users who depend on the old per-partition behavior can set it back to 150 in their config. Doc string in `config.rs` and `docs/source/user-guide/configs.md` are updated.
   - **Plan output:** `EXPLAIN ANALYZE` for `Partitioned` hash joins no longer shows `CASE hash_repartition % N WHEN ...`. Instead it shows either `bounds AND struct(...) IN (SET) ([...])` (small joins) or `bounds AND multi_hash_lookup` (everything else).
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345926628

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4345772857)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                              HEAD ┃ worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    52.05 / 53.47 ±1.27 / 55.54 ms │      52.46 / 53.78 ±1.30 / 55.95 ms │     no change │
   │ QQuery 2  │    43.53 / 45.21 ±1.02 / 46.38 ms │      44.19 / 47.68 ±1.96 / 49.75 ms │  1.05x slower │
   │ QQuery 3  │ 150.12 / 158.75 ±5.93 / 167.47 ms │   162.14 / 166.32 ±3.24 / 169.49 ms │     no change │
   │ QQuery 4  │    37.45 / 38.42 ±0.65 / 39.41 ms │      37.47 / 39.01 ±1.57 / 41.51 ms │     no change │
   │ QQuery 5  │ 250.10 / 260.92 ±8.26 / 270.85 ms │   482.17 / 490.53 ±6.89 / 498.80 ms │  1.88x slower │
   │ QQuery 6  │    37.75 / 38.66 ±1.18 / 40.91 ms │      37.37 / 38.63 ±0.86 / 39.76 ms │     no change │
   │ QQuery 7  │ 355.41 / 367.58 ±9.23 / 383.28 ms │   299.28 / 302.02 ±2.32 / 306.23 ms │ +1.22x faster │
   │ QQuery 8  │ 275.92 / 292.06 ±8.97 / 301.66 ms │   397.50 / 400.91 ±2.51 / 404.69 ms │  1.37x slower │
   │ QQuery 9  │ 307.64 / 313.54 ±3.44 / 317.16 ms │   503.08 / 513.16 ±8.07 / 526.26 ms │  1.64x slower │
   │ QQuery 10 │    90.62 / 92.01 ±1.37 / 94.33 ms │      92.26 / 93.08 ±0.94 / 94.78 ms │     no change │
   │ QQuery 11 │    38.05 / 39.69 ±1.26 / 41.91 ms │      41.18 / 42.07 ±0.71 / 43.00 ms │  1.06x slower │
   │ QQuery 12 │ 136.20 / 148.87 ±7.23 / 156.97 ms │   156.43 / 160.31 ±2.80 / 164.44 ms │  1.08x slower │
   │ QQuery 13 │ 118.32 / 125.06 ±8.53 / 140.29 ms │   138.64 / 142.22 ±1.91 / 143.88 ms │  1.14x slower │
   │ QQuery 14 │    79.33 / 80.29 ±1.00 / 82.11 ms │   118.86 / 120.90 ±1.58 / 122.93 ms │  1.51x slower │
   │ QQuery 15 │    70.79 / 71.76 ±0.78 / 72.92 ms │      68.56 / 72.13 ±2.14 / 74.64 ms │     no change │
   │ QQuery 16 │    52.72 / 54.60 ±1.38 / 56.70 ms │      51.29 / 54.30 ±1.66 / 56.01 ms │     no change │
   │ QQuery 17 │ 289.38 / 294.78 ±5.76 / 302.41 ms │   155.59 / 157.78 ±1.75 / 160.34 ms │ +1.87x faster │
   │ QQuery 18 │ 228.93 / 243.10 ±8.59 / 255.82 ms │   171.78 / 175.22 ±2.07 / 178.18 ms │ +1.39x faster │
   │ QQuery 19 │    44.33 / 45.59 ±0.80 / 46.55 ms │      44.85 / 45.29 ±0.41 / 45.81 ms │     no change │
   │ QQuery 20 │ 151.21 / 157.23 ±5.36 / 166.85 ms │      87.82 / 89.43 ±1.11 / 90.61 ms │ +1.76x faster │
   │ QQuery 21 │ 217.93 / 233.27 ±9.60 / 247.64 ms │   375.11 / 377.42 ±1.62 / 379.55 ms │  1.62x slower │
   │ QQuery 22 │    52.66 / 54.19 ±1.64 / 57.34 ms │      51.94 / 53.94 ±1.32 / 56.07 ms │     no change │
   └───────────┴───────────────────────────────────┴─────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 3209.06ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 3636.15ms │
   │ Average Time (HEAD)                                │  145.87ms │
   │ Average Time (worktree-dynamic-filter-restructure) │  165.28ms │
   │ Queries Faster                                     │         4 │
   │ Queries Slower                                     │         9 │
   │ Queries with No Change                             │         9 │
   │ Queries with Failure                               │         0 │
   └────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 20.0s |
   | Peak memory | 6.2 GiB |
   | Avg memory | 5.4 GiB |
   | CPU user | 140.0s |
   | CPU sys | 3.4s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 20.0s |
   | Peak memory | 5.9 GiB |
   | Avg memory | 5.3 GiB |
   | CPU user | 149.8s |
   | CPU sys | 3.7s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347957955

   🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347919711-1933-d77d2 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d)
   BENCH_NAME=hj
   BENCH_COMMAND=cargo bench --features=parquet --bench hj
   BENCH_FILTER=
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347936422

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347919215-1924-x6mr2 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: clickbench_partitioned
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348068790

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347920152-1937-8km22 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpcds
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4402437992

   @gene-bordegaray thank you for the review
   
   @Dandandan based on https://github.com/apache/datafusion/pull/21931#issuecomment-4364636257 it sounds like you're on board with this direction. Once I rebase and address Gene's review, would you be able to review this as a committer?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4466526429

   > I dont know if it can happen in practice but that flatten bug is still there
   
   the regression should eliminate the first column and will see a shift of the second column going to the first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348137499

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃   worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.46 / 4.92 ±6.80 / 18.52 ms │          1.46 / 4.98 ±6.88 / 18.74 ms │     no change │
   │ QQuery 1  │        17.16 / 18.23 ±0.58 / 18.86 ms │        17.91 / 18.45 ±0.53 / 19.40 ms │     no change │
   │ QQuery 2  │        41.08 / 41.72 ±0.54 / 42.45 ms │        41.84 / 42.73 ±0.68 / 43.93 ms │     no change │
   │ QQuery 3  │        33.43 / 35.03 ±1.27 / 36.32 ms │        33.57 / 34.81 ±1.19 / 36.92 ms │     no change │
   │ QQuery 4  │     266.98 / 269.12 ±1.76 / 272.14 ms │     269.28 / 271.39 ±1.57 / 273.68 ms │     no change │
   │ QQuery 5  │     320.04 / 322.75 ±1.91 / 325.32 ms │     324.25 / 326.62 ±1.66 / 329.40 ms │     no change │
   │ QQuery 6  │        17.66 / 20.10 ±2.79 / 25.34 ms │        18.10 / 19.02 ±0.81 / 20.46 ms │ +1.06x faster │
   │ QQuery 7  │        27.48 / 27.62 ±0.10 / 27.76 ms │        28.22 / 30.08 ±3.27 / 36.61 ms │  1.09x slower │
   │ QQuery 8  │     380.54 / 381.36 ±0.55 / 381.97 ms │     377.47 / 381.91 ±3.41 / 387.81 ms │     no change │
   │ QQuery 9  │     367.47 / 380.12 ±8.94 / 392.81 ms │    368.14 / 385.94 ±15.28 / 406.02 ms │     no change │
   │ QQuery 10 │     116.05 / 117.98 ±1.71 / 120.57 ms │     115.74 / 118.69 ±2.26 / 121.63 ms │     no change │
   │ QQuery 11 │     128.07 / 129.72 ±1.84 / 132.90 ms │     130.39 / 131.63 ±1.11 / 132.98 ms │     no change │
   │ QQuery 12 │    346.07 / 381.57 ±39.41 / 439.10 ms │     348.82 / 355.31 ±9.14 / 373.31 ms │ +1.07x faster │
   │ QQuery 13 │     599.60 / 607.03 ±5.44 / 613.10 ms │     598.78 / 602.52 ±2.65 / 606.82 ms │     no change │
   │ QQuery 14 │    350.55 / 365.53 ±10.90 / 382.61 ms │     359.03 / 362.00 ±1.92 / 363.86 ms │     no change │
   │ QQuery 15 │     326.84 / 329.09 ±2.17 / 332.72 ms │     320.38 / 325.03 ±3.29 / 329.22 ms │     no change │
   │ QQuery 16 │     698.47 / 704.56 ±3.38 / 707.72 ms │     704.18 / 712.40 ±8.62 / 722.99 ms │     no change │
   │ QQuery 17 │    691.16 / 717.97 ±21.80 / 755.41 ms │     696.32 / 699.85 ±3.16 / 705.24 ms │     no change │
   │ QQuery 18 │ 1422.14 / 1446.23 ±22.66 / 1479.92 ms │ 1431.57 / 1447.11 ±12.40 / 1464.21 ms │     no change │
   │ QQuery 19 │        35.25 / 36.01 ±0.93 / 37.84 ms │       35.61 / 51.89 ±15.97 / 71.95 ms │  1.44x slower │
   │ QQuery 20 │    501.20 / 520.45 ±10.67 / 533.65 ms │     497.03 / 501.28 ±3.18 / 504.50 ms │     no change │
   │ QQuery 21 │    567.02 / 589.49 ±15.91 / 615.57 ms │    583.74 / 599.57 ±13.72 / 621.71 ms │     no change │
   │ QQuery 22 │    927.62 / 959.43 ±25.00 / 994.04 ms │    935.22 / 959.27 ±18.30 / 984.56 ms │     no change │
   │ QQuery 23 │   364.83 / 570.63 ±131.08 / 702.82 ms │    443.69 / 551.66 ±64.20 / 630.98 ms │     no change │
   │ QQuery 24 │      81.74 / 99.25 ±15.24 / 125.43 ms │       87.40 / 97.67 ±8.27 / 108.29 ms │     no change │
   │ QQuery 25 │     149.77 / 159.36 ±6.87 / 168.44 ms │     149.01 / 158.55 ±7.90 / 170.64 ms │     no change │
   │ QQuery 26 │    109.55 / 132.73 ±14.21 / 147.86 ms │     133.68 / 141.73 ±6.16 / 149.09 ms │  1.07x slower │
   │ QQuery 27 │    695.92 / 725.08 ±34.00 / 790.41 ms │    693.34 / 706.03 ±11.91 / 723.29 ms │     no change │
   │ QQuery 28 │ 2907.85 / 2944.28 ±39.63 / 3009.98 ms │ 2911.39 / 2948.20 ±28.84 / 2995.55 ms │     no change │
   │ QQuery 29 │       48.55 / 54.89 ±10.38 / 75.56 ms │       49.48 / 60.94 ±15.38 / 89.13 ms │  1.11x slower │
   │ QQuery 30 │     343.70 / 350.09 ±4.06 / 355.39 ms │     348.63 / 354.00 ±4.54 / 360.84 ms │     no change │
   │ QQuery 31 │     405.59 / 410.65 ±3.18 / 414.72 ms │     398.26 / 408.08 ±5.89 / 414.34 ms │     no change │
   │ QQuery 32 │ 1615.67 / 1669.11 ±29.16 / 1699.77 ms │ 1577.99 / 1621.18 ±38.73 / 1692.39 ms │     no change │
   │ QQuery 33 │ 1557.19 / 1576.05 ±12.02 / 1589.39 ms │ 1514.27 / 1568.68 ±50.92 / 1655.35 ms │     no change │
   │ QQuery 34 │ 1542.25 / 1576.50 ±18.64 / 1596.51 ms │ 1541.75 / 1571.13 ±15.01 / 1583.27 ms │     no change │
   │ QQuery 35 │    302.42 / 329.78 ±42.36 / 414.10 ms │    293.69 / 314.64 ±14.80 / 333.79 ms │     no change │
   │ QQuery 36 │        80.21 / 85.03 ±3.24 / 89.07 ms │        76.98 / 80.59 ±2.59 / 84.31 ms │ +1.05x faster │
   │ QQuery 37 │        48.77 / 55.32 ±7.27 / 68.42 ms │        48.19 / 56.93 ±6.30 / 65.37 ms │     no change │
   │ QQuery 38 │        46.07 / 50.26 ±5.40 / 60.42 ms │        46.60 / 50.48 ±4.45 / 59.15 ms │     no change │
   │ QQuery 39 │     144.79 / 150.37 ±4.54 / 154.88 ms │     147.73 / 148.83 ±0.96 / 150.38 ms │     no change │
   │ QQuery 40 │        29.45 / 32.12 ±1.95 / 35.39 ms │        29.40 / 31.72 ±2.45 / 35.37 ms │     no change │
   │ QQuery 41 │        27.80 / 31.04 ±3.12 / 35.71 ms │        26.90 / 27.29 ±0.33 / 27.69 ms │ +1.14x faster │
   │ QQuery 42 │        23.73 / 27.81 ±4.61 / 35.72 ms │        23.83 / 25.59 ±2.22 / 29.96 ms │ +1.09x faster │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 19436.39ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 19306.40ms │
   │ Average Time (HEAD)                                │   452.01ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   448.99ms │
   │ Queries Faster                                     │          5 │
   │ Queries Slower                                     │          4 │
   │ Queries with No Change                             │         34 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 32.3 GiB |
   | Avg memory | 26.3 GiB |
   | CPU user | 1041.2s |
   | CPU sys | 64.0s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 31.9 GiB |
   | Avg memory | 26.0 GiB |
   | CPU user | 1033.4s |
   | CPU sys | 65.5s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345772857

   run benchmark tpch
   
   ```yaml
   baseline:
       ref: main
       env:
          DATAFUSION_EXECUTION_TARGET_PARTITIONS: 128
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: true
   changed:
       ref: HEAD
       env:
          DATAFUSION_EXECUTION_TARGET_PARTITIONS: 128
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: true
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471519872

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃   worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.19 / 4.51 ±6.59 / 17.70 ms │          1.18 / 4.58 ±6.73 / 18.04 ms │     no change │
   │ QQuery 1  │        12.55 / 12.67 ±0.09 / 12.80 ms │        12.50 / 12.75 ±0.15 / 12.89 ms │     no change │
   │ QQuery 2  │        35.28 / 35.48 ±0.12 / 35.66 ms │        35.08 / 35.38 ±0.20 / 35.69 ms │     no change │
   │ QQuery 3  │        30.32 / 30.65 ±0.59 / 31.83 ms │        29.99 / 30.50 ±0.48 / 31.31 ms │     no change │
   │ QQuery 4  │     225.06 / 229.66 ±2.83 / 232.87 ms │     227.80 / 230.20 ±1.29 / 231.65 ms │     no change │
   │ QQuery 5  │     273.45 / 275.19 ±1.62 / 277.67 ms │     273.62 / 274.68 ±1.27 / 277.15 ms │     no change │
   │ QQuery 6  │           5.46 / 5.88 ±0.26 / 6.15 ms │           5.59 / 6.10 ±0.43 / 6.87 ms │     no change │
   │ QQuery 7  │        16.39 / 16.47 ±0.10 / 16.66 ms │        16.11 / 16.16 ±0.03 / 16.21 ms │     no change │
   │ QQuery 8  │     319.86 / 321.42 ±1.27 / 323.71 ms │     306.72 / 311.70 ±2.88 / 314.67 ms │     no change │
   │ QQuery 9  │     441.39 / 446.04 ±3.73 / 452.08 ms │     440.78 / 446.44 ±4.91 / 452.09 ms │     no change │
   │ QQuery 10 │        92.31 / 93.13 ±0.44 / 93.51 ms │        92.39 / 93.02 ±0.50 / 93.60 ms │     no change │
   │ QQuery 11 │     102.56 / 103.60 ±0.75 / 104.59 ms │     102.65 / 106.15 ±4.23 / 114.31 ms │     no change │
   │ QQuery 12 │     299.84 / 303.23 ±2.59 / 307.11 ms │     301.28 / 305.16 ±3.79 / 311.32 ms │     no change │
   │ QQuery 13 │     414.14 / 417.14 ±4.15 / 425.31 ms │     412.82 / 416.38 ±2.38 / 419.17 ms │     no change │
   │ QQuery 14 │     314.00 / 320.66 ±5.73 / 329.17 ms │     309.58 / 315.76 ±4.94 / 321.62 ms │     no change │
   │ QQuery 15 │     273.19 / 278.98 ±9.57 / 298.05 ms │     272.92 / 275.24 ±2.35 / 278.99 ms │     no change │
   │ QQuery 16 │     612.59 / 618.84 ±8.45 / 635.41 ms │     590.06 / 597.87 ±5.18 / 605.96 ms │     no change │
   │ QQuery 17 │     612.80 / 621.98 ±8.66 / 638.16 ms │     598.17 / 601.22 ±2.15 / 604.45 ms │     no change │
   │ QQuery 18 │ 1214.41 / 1244.18 ±17.84 / 1265.62 ms │ 1176.40 / 1198.50 ±14.51 / 1221.19 ms │     no change │
   │ QQuery 19 │        28.65 / 31.20 ±4.35 / 39.86 ms │        28.95 / 31.23 ±3.42 / 38.00 ms │     no change │
   │ QQuery 20 │     507.20 / 515.71 ±7.22 / 528.23 ms │     514.24 / 522.83 ±4.39 / 526.73 ms │     no change │
   │ QQuery 21 │     560.04 / 569.23 ±7.67 / 581.62 ms │     563.55 / 569.78 ±4.56 / 576.29 ms │     no change │
   │ QQuery 22 │     911.34 / 915.05 ±2.80 / 919.06 ms │     916.07 / 922.46 ±5.91 / 931.90 ms │     no change │
   │ QQuery 23 │    114.77 / 126.86 ±13.26 / 143.45 ms │     113.41 / 119.51 ±4.20 / 124.63 ms │ +1.06x faster │
   │ QQuery 24 │        39.69 / 42.76 ±2.96 / 48.24 ms │        39.84 / 41.09 ±1.26 / 43.40 ms │     no change │
   │ QQuery 25 │     144.20 / 148.17 ±5.62 / 159.34 ms │     145.02 / 148.73 ±5.91 / 160.42 ms │     no change │
   │ QQuery 26 │        50.62 / 51.62 ±0.55 / 52.22 ms │        61.78 / 65.74 ±3.80 / 72.52 ms │  1.27x slower │
   │ QQuery 27 │    697.12 / 710.67 ±11.05 / 730.64 ms │     715.07 / 727.18 ±6.84 / 735.26 ms │     no change │
   │ QQuery 28 │  3000.80 / 3015.23 ±8.42 / 3025.70 ms │ 3020.49 / 3043.37 ±17.54 / 3073.56 ms │     no change │
   │ QQuery 29 │        41.10 / 43.38 ±3.54 / 50.43 ms │        41.05 / 45.25 ±5.45 / 55.17 ms │     no change │
   │ QQuery 30 │     300.00 / 309.58 ±5.82 / 318.01 ms │     303.10 / 308.38 ±3.58 / 313.89 ms │     no change │
   │ QQuery 31 │     292.92 / 298.37 ±2.76 / 300.39 ms │     290.63 / 296.15 ±3.16 / 299.73 ms │     no change │
   │ QQuery 32 │   947.59 / 969.28 ±32.67 / 1033.59 ms │    884.35 / 904.34 ±16.63 / 932.10 ms │ +1.07x faster │
   │ QQuery 33 │ 1373.07 / 1385.94 ±12.42 / 1402.79 ms │ 1380.30 / 1408.13 ±16.46 / 1428.18 ms │     no change │
   │ QQuery 34 │ 1397.68 / 1411.47 ±14.11 / 1435.12 ms │ 1403.22 / 1423.08 ±16.46 / 1452.78 ms │     no change │
   │ QQuery 35 │    275.95 / 312.52 ±45.11 / 392.46 ms │    277.77 / 293.82 ±14.66 / 317.26 ms │ +1.06x faster │
   │ QQuery 36 │       61.57 / 71.14 ±10.34 / 90.53 ms │        62.52 / 70.25 ±6.57 / 81.33 ms │     no change │
   │ QQuery 37 │        35.97 / 36.67 ±0.51 / 37.47 ms │        36.52 / 38.58 ±2.20 / 42.09 ms │  1.05x slower │
   │ QQuery 38 │        34.56 / 38.28 ±4.58 / 45.63 ms │        34.23 / 38.63 ±5.64 / 49.06 ms │     no change │
   │ QQuery 39 │     113.80 / 126.29 ±8.42 / 136.69 ms │     115.82 / 126.37 ±5.41 / 131.03 ms │     no change │
   │ QQuery 40 │        18.37 / 18.79 ±0.31 / 19.12 ms │        17.94 / 18.33 ±0.51 / 19.33 ms │     no change │
   │ QQuery 41 │        16.89 / 17.20 ±0.27 / 17.68 ms │        16.75 / 16.84 ±0.07 / 16.92 ms │     no change │
   │ QQuery 42 │        14.18 / 14.40 ±0.20 / 14.77 ms │        14.06 / 18.74 ±8.52 / 35.77 ms │  1.30x slower │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 16559.50ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 16476.63ms │
   │ Average Time (HEAD)                                │   385.10ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   383.18ms │
   │ Queries Faster                                     │          3 │
   │ Queries Slower                                     │          3 │
   │ Queries with No Change                             │         37 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 85.0s |
   | Peak memory | 30.5 GiB |
   | Avg memory | 23.3 GiB |
   | CPU user | 875.8s |
   | CPU sys | 50.0s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 85.0s |
   | Peak memory | 29.3 GiB |
   | Avg memory | 22.9 GiB |
   | CPU user | 872.1s |
   | CPU sys | 49.3s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471517415

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpch_sf10.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                 HEAD ┃   worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    315.15 / 317.51 ±1.61 / 319.58 ms │     317.81 / 319.35 ±1.87 / 322.85 ms │     no change │
   │ QQuery 2  │    171.00 / 172.87 ±2.08 / 176.43 ms │     230.07 / 235.31 ±4.66 / 241.48 ms │  1.36x slower │
   │ QQuery 3  │    366.16 / 369.55 ±2.77 / 373.99 ms │     425.61 / 432.73 ±5.36 / 442.22 ms │  1.17x slower │
   │ QQuery 4  │    314.26 / 316.36 ±1.84 / 319.59 ms │     393.79 / 396.80 ±2.66 / 399.98 ms │  1.25x slower │
   │ QQuery 5  │    578.31 / 593.38 ±8.42 / 602.38 ms │     781.28 / 795.52 ±9.20 / 804.71 ms │  1.34x slower │
   │ QQuery 6  │    290.23 / 292.81 ±2.14 / 296.70 ms │     289.70 / 291.97 ±1.89 / 295.31 ms │     no change │
   │ QQuery 7  │    423.99 / 425.85 ±1.76 / 428.45 ms │     486.88 / 488.01 ±0.82 / 488.96 ms │  1.15x slower │
   │ QQuery 8  │    610.72 / 614.01 ±3.09 / 618.69 ms │     686.98 / 690.28 ±3.20 / 694.25 ms │  1.12x slower │
   │ QQuery 9  │ 1337.58 / 1347.25 ±5.52 / 1352.57 ms │ 1700.75 / 1718.82 ±12.85 / 1738.23 ms │  1.28x slower │
   │ QQuery 10 │    396.58 / 400.48 ±3.06 / 404.15 ms │     433.52 / 440.01 ±3.46 / 443.02 ms │  1.10x slower │
   │ QQuery 11 │    136.45 / 139.24 ±4.42 / 147.97 ms │     139.90 / 142.55 ±3.30 / 148.88 ms │     no change │
   │ QQuery 12 │    335.67 / 339.87 ±4.49 / 347.99 ms │     388.59 / 393.28 ±5.14 / 399.65 ms │  1.16x slower │
   │ QQuery 13 │    377.10 / 389.58 ±9.81 / 404.80 ms │     428.66 / 434.28 ±6.76 / 446.01 ms │  1.11x slower │
   │ QQuery 14 │    201.09 / 202.49 ±1.53 / 205.20 ms │     203.44 / 204.70 ±1.12 / 206.76 ms │     no change │
   │ QQuery 15 │    396.60 / 400.91 ±3.03 / 404.03 ms │     398.21 / 401.38 ±2.06 / 404.67 ms │     no change │
   │ QQuery 16 │    100.97 / 104.96 ±5.54 / 115.78 ms │     111.94 / 113.32 ±1.33 / 115.61 ms │  1.08x slower │
   │ QQuery 17 │    755.12 / 761.65 ±4.87 / 768.22 ms │     662.50 / 667.97 ±5.03 / 677.44 ms │ +1.14x faster │
   │ QQuery 18 │    664.78 / 676.35 ±7.80 / 685.13 ms │     620.18 / 630.24 ±7.41 / 641.90 ms │ +1.07x faster │
   │ QQuery 19 │    281.54 / 287.59 ±8.32 / 303.99 ms │     286.33 / 293.19 ±8.12 / 305.21 ms │     no change │
   │ QQuery 20 │    318.05 / 321.19 ±1.64 / 322.82 ms │     303.56 / 307.98 ±3.83 / 314.78 ms │     no change │
   │ QQuery 21 │ 1153.91 / 1162.40 ±6.29 / 1171.19 ms │ 1341.00 / 1357.49 ±12.25 / 1373.93 ms │  1.17x slower │
   │ QQuery 22 │    128.25 / 132.09 ±2.77 / 136.09 ms │     121.66 / 127.85 ±4.16 / 132.25 ms │     no change │
   └───────────┴──────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │  9768.40ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 10883.03ms │
   │ Average Time (HEAD)                                │   444.02ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   494.68ms │
   │ Queries Faster                                     │          2 │
   │ Queries Slower                                     │         12 │
   │ Queries with No Change                             │          8 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch10 — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 9.9 GiB |
   | Avg memory | 8.1 GiB |
   | CPU user | 537.5s |
   | CPU sys | 20.2s |
   | Peak spill | 0 B |
   
   **tpch10 — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 9.7 GiB |
   | Avg memory | 8.2 GiB |
   | CPU user | 598.3s |
   | CPU sys | 20.3s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471505648

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                              HEAD ┃ worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    38.53 / 40.46 ±2.41 / 44.62 ms │      38.81 / 39.62 ±1.08 / 41.54 ms │     no change │
   │ QQuery 2  │    23.88 / 24.18 ±0.43 / 25.04 ms │      24.44 / 24.87 ±0.48 / 25.56 ms │     no change │
   │ QQuery 3  │    46.61 / 47.18 ±0.59 / 48.28 ms │      51.03 / 51.41 ±0.30 / 51.89 ms │  1.09x slower │
   │ QQuery 4  │    21.58 / 21.73 ±0.09 / 21.83 ms │      21.54 / 21.87 ±0.51 / 22.88 ms │     no change │
   │ QQuery 5  │    65.70 / 66.60 ±0.67 / 67.63 ms │      69.10 / 69.54 ±0.41 / 70.30 ms │     no change │
   │ QQuery 6  │    34.91 / 35.28 ±0.32 / 35.67 ms │      34.36 / 35.22 ±0.82 / 36.45 ms │     no change │
   │ QQuery 7  │    56.15 / 56.45 ±0.31 / 57.02 ms │      52.78 / 52.99 ±0.24 / 53.43 ms │ +1.07x faster │
   │ QQuery 8  │    77.75 / 79.18 ±1.18 / 81.27 ms │      73.40 / 74.85 ±0.92 / 76.26 ms │ +1.06x faster │
   │ QQuery 9  │ 102.01 / 103.65 ±2.01 / 107.59 ms │   113.77 / 118.63 ±4.58 / 124.44 ms │  1.14x slower │
   │ QQuery 10 │    73.92 / 74.42 ±0.58 / 75.52 ms │      73.12 / 73.39 ±0.37 / 74.10 ms │     no change │
   │ QQuery 11 │    14.94 / 15.14 ±0.13 / 15.27 ms │      14.97 / 15.24 ±0.27 / 15.75 ms │     no change │
   │ QQuery 12 │    44.06 / 44.70 ±0.43 / 45.19 ms │      43.30 / 43.70 ±0.41 / 44.48 ms │     no change │
   │ QQuery 13 │    45.46 / 45.70 ±0.14 / 45.86 ms │      61.05 / 61.29 ±0.15 / 61.50 ms │  1.34x slower │
   │ QQuery 14 │    38.48 / 38.71 ±0.13 / 38.85 ms │      42.91 / 43.32 ±0.33 / 43.86 ms │  1.12x slower │
   │ QQuery 15 │    40.52 / 41.20 ±0.63 / 42.38 ms │      40.95 / 41.60 ±0.79 / 43.07 ms │     no change │
   │ QQuery 16 │    22.54 / 22.67 ±0.11 / 22.85 ms │      22.58 / 22.73 ±0.17 / 23.04 ms │     no change │
   │ QQuery 17 │ 158.58 / 161.33 ±3.25 / 167.29 ms │      53.82 / 54.62 ±0.53 / 55.43 ms │ +2.95x faster │
   │ QQuery 18 │    70.35 / 71.03 ±0.43 / 71.63 ms │      51.62 / 52.42 ±0.79 / 53.89 ms │ +1.36x faster │
   │ QQuery 19 │    39.85 / 40.51 ±0.52 / 41.09 ms │      39.52 / 39.74 ±0.19 / 40.07 ms │     no change │
   │ QQuery 20 │    44.26 / 44.83 ±0.80 / 46.39 ms │      40.67 / 41.47 ±1.24 / 43.94 ms │ +1.08x faster │
   │ QQuery 21 │    79.76 / 80.36 ±0.47 / 81.01 ms │      82.28 / 83.64 ±0.95 / 84.81 ms │     no change │
   │ QQuery 22 │    34.57 / 34.85 ±0.30 / 35.39 ms │      34.62 / 35.00 ±0.31 / 35.52 ms │     no change │
   └───────────┴───────────────────────────────────┴─────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 1190.17ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 1097.17ms │
   │ Average Time (HEAD)                                │   54.10ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   49.87ms │
   │ Queries Faster                                     │         5 │
   │ Queries Slower                                     │         4 │
   │ Queries with No Change                             │        13 │
   │ Queries with Failure                               │         0 │
   └────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 10.0s |
   | Peak memory | 5.2 GiB |
   | Avg memory | 4.5 GiB |
   | CPU user | 45.2s |
   | CPU sys | 2.0s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 10.0s |
   | Peak memory | 5.2 GiB |
   | Avg memory | 4.4 GiB |
   | CPU user | 39.7s |
   | CPU sys | 2.0s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345503296

   run benchmark tcph
   
   ```yaml
   baseline:
       ref: main
       env:
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: false
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: false
   changed:
       ref: HEAD
       env:
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: false
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: false
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345661976

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4345545637)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                              HEAD ┃ worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    43.41 / 44.20 ±0.95 / 46.07 ms │      43.37 / 44.58 ±1.47 / 47.38 ms │     no change │
   │ QQuery 2  │    25.18 / 25.59 ±0.35 / 26.10 ms │      25.09 / 25.23 ±0.24 / 25.72 ms │     no change │
   │ QQuery 3  │    48.02 / 48.76 ±0.41 / 49.09 ms │      52.88 / 53.66 ±0.85 / 55.19 ms │  1.10x slower │
   │ QQuery 4  │    22.73 / 22.86 ±0.11 / 23.02 ms │      22.47 / 22.68 ±0.13 / 22.88 ms │     no change │
   │ QQuery 5  │    70.47 / 72.83 ±2.26 / 76.28 ms │      73.07 / 76.76 ±3.21 / 81.56 ms │  1.05x slower │
   │ QQuery 6  │    35.98 / 36.50 ±0.54 / 37.55 ms │      36.12 / 37.55 ±1.15 / 38.84 ms │     no change │
   │ QQuery 7  │    57.86 / 58.04 ±0.21 / 58.45 ms │      55.25 / 56.19 ±1.09 / 58.34 ms │     no change │
   │ QQuery 8  │    79.29 / 80.19 ±0.67 / 81.15 ms │      76.28 / 76.74 ±0.51 / 77.67 ms │     no change │
   │ QQuery 9  │ 110.20 / 112.39 ±2.16 / 115.28 ms │   120.81 / 124.17 ±2.52 / 126.74 ms │  1.10x slower │
   │ QQuery 10 │    76.60 / 77.36 ±0.60 / 78.04 ms │      76.50 / 76.70 ±0.31 / 77.31 ms │     no change │
   │ QQuery 11 │    16.63 / 17.50 ±0.98 / 19.19 ms │      16.15 / 16.94 ±0.76 / 18.03 ms │     no change │
   │ QQuery 12 │    46.65 / 47.21 ±0.60 / 48.34 ms │      45.96 / 47.22 ±2.09 / 51.39 ms │     no change │
   │ QQuery 13 │    48.67 / 49.58 ±0.56 / 50.27 ms │      63.30 / 63.75 ±0.36 / 64.23 ms │  1.29x slower │
   │ QQuery 14 │    40.73 / 41.21 ±0.61 / 42.29 ms │      44.72 / 45.58 ±0.69 / 46.50 ms │  1.11x slower │
   │ QQuery 15 │    45.55 / 46.45 ±0.75 / 47.39 ms │      45.01 / 46.74 ±1.97 / 50.52 ms │     no change │
   │ QQuery 16 │    23.72 / 24.27 ±0.57 / 25.18 ms │      23.46 / 23.72 ±0.18 / 23.98 ms │     no change │
   │ QQuery 17 │ 161.67 / 165.40 ±3.60 / 171.03 ms │      55.41 / 55.87 ±0.49 / 56.52 ms │ +2.96x faster │
   │ QQuery 18 │    75.97 / 76.74 ±0.55 / 77.28 ms │      57.68 / 58.81 ±0.99 / 59.97 ms │ +1.30x faster │
   │ QQuery 19 │    42.01 / 42.27 ±0.20 / 42.49 ms │      42.38 / 42.80 ±0.29 / 43.14 ms │     no change │
   │ QQuery 20 │    45.92 / 46.56 ±0.53 / 47.48 ms │      42.72 / 43.95 ±2.31 / 48.58 ms │ +1.06x faster │
   │ QQuery 21 │    84.97 / 86.79 ±2.22 / 91.01 ms │      89.70 / 90.48 ±0.67 / 91.45 ms │     no change │
   │ QQuery 22 │    35.83 / 36.41 ±0.32 / 36.77 ms │      36.07 / 36.24 ±0.13 / 36.46 ms │     no change │
   └───────────┴───────────────────────────────────┴─────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 1259.10ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 1166.39ms │
   │ Average Time (HEAD)                                │   57.23ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   53.02ms │
   │ Queries Faster                                     │         3 │
   │ Queries Slower                                     │         5 │
   │ Queries with No Change                             │        14 │
   │ Queries with Failure                               │         0 │
   └────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 10.0s |
   | Peak memory | 5.3 GiB |
   | Avg memory | 4.6 GiB |
   | CPU user | 47.7s |
   | CPU sys | 2.6s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 10.0s |
   | Peak memory | 5.3 GiB |
   | Avg memory | 4.5 GiB |
   | CPU user | 42.2s |
   | CPU sys | 2.7s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345731808

   Hi @adriangb, your benchmark configuration could not be parsed (https://github.com/apache/datafusion/pull/21931#issuecomment-4345731555).
   
   **Error:** `invalid configuration: baseline.env: invalid type: string "DATAFUSION_EXECUTION_TARGET_PARTITIONS=128 DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS", expected a map at line 5 column 8`
   
   Supported benchmarks:
   - Standard: clickbench_1, clickbench_extended, clickbench_partitioned, clickbench_pushdown, external_aggr, smj, sort_pushdown, sort_pushdown_inexact, sort_pushdown_inexact_overlap, sort_pushdown_inexact_unsorted, sort_pushdown_sorted, topk_tpch, tpcds, tpch, tpch10, tpch_mem, tpch_mem10
   - Criterion: (any)
   
   Usage:
   ```
   run benchmark <name>           # run specific benchmark(s)
   run benchmarks                 # run default suite
   run benchmarks <name1> <name2> # run specific benchmarks
   ```
   
   Per-side configuration (`run benchmark tpch` followed by):
   ```yaml
   env:
   SHARED_SETTING: enabled
   baseline:
   ref: v45.0.0
   env:
   DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
   changed:
   ref: v46.0.0
   env:
   DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G
   ```
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345732634

   Hi @adriangb, your benchmark configuration could not be parsed (https://github.com/apache/datafusion/pull/21931#issuecomment-4345732500).
   
   **Error:** `invalid configuration: baseline.env: invalid type: string "DATAFUSION_EXECUTION_TARGET_PARTITIONS=128 DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS", expected a map at line 5 column 8`
   
   Supported benchmarks:
   - Standard: clickbench_1, clickbench_extended, clickbench_partitioned, clickbench_pushdown, external_aggr, smj, sort_pushdown, sort_pushdown_inexact, sort_pushdown_inexact_overlap, sort_pushdown_inexact_unsorted, sort_pushdown_sorted, topk_tpch, tpcds, tpch, tpch10, tpch_mem, tpch_mem10
   - Criterion: (any)
   
   Usage:
   ```
   run benchmark <name>           # run specific benchmark(s)
   run benchmarks                 # run default suite
   run benchmarks <name1> <name2> # run specific benchmarks
   ```
   
   Per-side configuration (`run benchmark tpch` followed by):
   ```yaml
   env:
   SHARED_SETTING: enabled
   baseline:
   ref: v45.0.0
   env:
   DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
   changed:
   ref: v46.0.0
   env:
   DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G
   ```
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345833923

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4345772857)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4345772857-1921-hxt5c 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (18129fe9a705988f6805567a2aeec9d7ce509b3e) to main [diff](https://github.com/apache/datafusion/compare/5fda21683ed73f89b06431fac94777d6b1540b02..18129fe9a705988f6805567a2aeec9d7ce509b3e) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711

   run benchmarks clickbench_partitioned tpch tpch10 tpcds hj
   
   ```yaml
   env:
     DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
     DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: "true"
   baseline:
     ref: main
   changed:
     ref: HEAD
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4358527270

   > > I'm quite in favor of this change. It also avoids blowing up the expression based on number of partitions, which can happen when partition count is high.
   > 
   > Well it doesn't avoid it completely, and in some ways it makes it worse. We still have 1 hash map per partition (cannot be avoided unless we pay the memory and build time cost of combining them). And we now scale our probes with the number of partitions, they used to be constant with number of partitions. But probes are much faster than hashes which is why I think unless the partition count is high this will likely be faster.
   
   I am not 100% following this.
   
   > A possible structural follow-up — re-introducing partition routing inside `MultiMapLookupExpr` (1 routing hash + 1 probe, so per-row cost matches legacy CASE) — would close the regression at any N, with or without https://github.com/apache/datafusion/pull/20363.
   
   I think this would be worthwhile to add so we avoid both the expression blow up as not having to probe each of them?
   
   Perhaps we can as well use https://github.com/apache/datafusion/pull/21900 here as for primitive columns the cost of `%` is _much_ higher than hashing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4361813807

   > > > I'm quite in favor of this change. It also avoids blowing up the expression based on number of partitions, which can happen when partition count is high.
   > > 
   > > 
   > > Well it doesn't avoid it completely, and in some ways it makes it worse. We still have 1 hash map per partition (cannot be avoided unless we pay the memory and build time cost of combining them). And we now scale our probes with the number of partitions, they used to be constant with number of partitions. But probes are much faster than hashes which is why I think unless the partition count is high this will likely be faster.
   > 
   > I am not 100% following this.
   
   The point is: it is probably still slower at some very high partition count. But it seems to not matter in reasonable workloads.
   
   > 
   > > A possible structural follow-up — re-introducing partition routing inside `MultiMapLookupExpr` (1 routing hash + 1 probe, so per-row cost matches legacy CASE) — would close the regression at any N, with or without #20363.
   > 
   > I think this would be worthwhile to add so we avoid both the expression blow up as not having to probe each of them?
   > 
   > Perhaps we can as well use #21900 here as for primitive columns the cost of `%` is _much_ higher than hashing.
   
   That brings back coupling to hash routing, which IMO would be nice to avoid. I will try incorporating 
   #21900


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471640723

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471617137)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471617137-180-ctztd 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed) using: tpch10
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144

   run benchmarks clickbench_partitioned tpch tpch10 tpcds hj
   
   ```yaml
   env:
     DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
     DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: "true"
   baseline:
     ref: main
   changed:
     ref: HEAD
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348163682

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
    Downloading crates ...
     Downloaded prost-derive v0.14.3
     Downloaded prost v0.14.3
     Downloaded anyhow v1.0.102
      Compiling anyhow v1.0.102
      Compiling either v1.15.0
      Compiling datafusion-benchmarks v53.1.0 (/workspace/datafusion-base/benchmarks)
      Compiling itertools v0.14.0
      Compiling prost-derive v0.14.3
      Compiling prost v0.14.3
      Compiling datafusion-proto-common v53.1.0 (/workspace/datafusion-base/datafusion/proto-common)
      Compiling datafusion-proto v53.1.0 (/workspace/datafusion-base/datafusion/proto)
       Finished `bench` profile [optimized + debuginfo] target(s) in 8m 50s
        Running benches/sql.rs (/workspace/datafusion-base/target/release/deps/sql-2c9f51cbfcaf27f8)
   Gnuplot not found, using plotters backend
   
   thread 'main' (21062) panicked at benchmarks/benches/sql.rs:131:46:
   assertion failed: Execution("Error in result on row 1, column 1 running query \"\nSELECT COUNT(*) > 0 from lineitem;\": expected value \"true\" but got value \"false\" in row: [\"false\"]")
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   error: bench failed, to rerun pass `--bench sql`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471443372

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpch_sf10.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃                               HEAD ┃ worktree-dynamic-filter-restructure ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │  315.30 / 316.47 ±0.90 / 317.51 ms │   313.84 / 314.99 ±0.84 / 316.37 ms │    no change │
   │ QQuery 2  │  119.70 / 121.09 ±1.10 / 122.42 ms │   122.03 / 124.78 ±2.33 / 128.46 ms │    no change │
   │ QQuery 3  │  257.68 / 260.56 ±2.14 / 263.63 ms │   261.54 / 266.11 ±3.74 / 271.63 ms │    no change │
   │ QQuery 4  │  125.85 / 128.20 ±1.25 / 129.58 ms │   126.89 / 131.13 ±3.32 / 135.19 ms │    no change │
   │ QQuery 5  │  368.10 / 377.48 ±4.79 / 381.45 ms │   383.62 / 387.77 ±4.51 / 395.68 ms │    no change │
   │ QQuery 6  │  127.36 / 128.57 ±1.03 / 130.05 ms │   126.68 / 127.57 ±0.67 / 128.64 ms │    no change │
   │ QQuery 7  │  478.95 / 481.96 ±2.86 / 487.17 ms │   488.56 / 496.58 ±4.16 / 499.43 ms │    no change │
   │ QQuery 8  │  398.09 / 407.52 ±6.42 / 416.39 ms │   427.29 / 431.89 ±4.88 / 440.77 ms │ 1.06x slower │
   │ QQuery 9  │  565.52 / 567.96 ±1.54 / 570.16 ms │   603.06 / 609.17 ±3.24 / 612.70 ms │ 1.07x slower │
   │ QQuery 10 │ 303.61 / 314.62 ±11.41 / 336.55 ms │   306.28 / 314.87 ±5.65 / 321.87 ms │    no change │
   │ QQuery 11 │     87.48 / 91.96 ±4.16 / 99.78 ms │      94.59 / 95.78 ±1.12 / 97.69 ms │    no change │
   │ QQuery 12 │  180.41 / 185.33 ±4.93 / 194.69 ms │   182.70 / 188.22 ±8.32 / 204.68 ms │    no change │
   │ QQuery 13 │ 287.92 / 301.73 ±12.51 / 319.83 ms │   293.04 / 298.22 ±3.46 / 302.95 ms │    no change │
   │ QQuery 14 │  176.84 / 180.27 ±3.15 / 184.67 ms │   176.25 / 179.12 ±3.02 / 184.90 ms │    no change │
   │ QQuery 15 │  317.74 / 319.56 ±2.23 / 323.85 ms │   316.12 / 319.68 ±4.80 / 328.41 ms │    no change │
   │ QQuery 16 │     71.88 / 75.72 ±3.87 / 82.97 ms │      72.57 / 74.68 ±1.32 / 76.73 ms │    no change │
   │ QQuery 17 │  660.64 / 666.68 ±7.50 / 680.83 ms │  672.37 / 683.72 ±10.52 / 698.34 ms │    no change │
   │ QQuery 18 │  695.85 / 702.47 ±5.64 / 712.58 ms │   703.46 / 716.77 ±7.42 / 723.96 ms │    no change │
   │ QQuery 19 │ 254.69 / 264.91 ±12.22 / 280.32 ms │  253.83 / 265.63 ±13.54 / 292.02 ms │    no change │
   │ QQuery 20 │  287.81 / 295.43 ±4.78 / 300.14 ms │   293.67 / 300.74 ±7.12 / 314.12 ms │    no change │
   │ QQuery 21 │  696.77 / 704.36 ±6.23 / 715.51 ms │   721.52 / 726.95 ±5.08 / 735.14 ms │    no change │
   │ QQuery 22 │     64.34 / 67.15 ±3.52 / 74.02 ms │      63.72 / 65.52 ±1.76 / 68.89 ms │    no change │
   └───────────┴────────────────────────────────────┴─────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 6959.98ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 7119.89ms │
   │ Average Time (HEAD)                                │  316.36ms │
   │ Average Time (worktree-dynamic-filter-restructure) │  323.63ms │
   │ Queries Faster                                     │         0 │
   │ Queries Slower                                     │         2 │
   │ Queries with No Change                             │        20 │
   │ Queries with Failure                               │         0 │
   └────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch10 — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 35.0s |
   | Peak memory | 11.9 GiB |
   | Avg memory | 8.6 GiB |
   | CPU user | 369.4s |
   | CPU sys | 22.3s |
   | Peak spill | 0 B |
   
   **tpch10 — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 40.0s |
   | Peak memory | 11.9 GiB |
   | Avg memory | 8.4 GiB |
   | CPU user | 378.1s |
   | CPU sys | 22.5s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471453679

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471353144-178-62lgl 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed) using: tpcds
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347937800

   🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347919215-1928-m9g9g 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d)
   BENCH_NAME=hj
   BENCH_COMMAND=cargo bench --features=parquet --bench hj
   BENCH_FILTER=
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215

   run benchmarks clickbench_partitioned tpch tpch10 tpcds hj
   
   ```yaml
   env:
     DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "false"
     DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: "false"
   baseline:
     ref: main
   changed:
     ref: HEAD
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345547022

   run benchmark tpch
   
   ```yaml
   baseline:
       ref: main
       env:
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: false
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: false
   changed:
       ref: HEAD
       env:
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: false
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: false
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348087236

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347920499-1942-d72cl 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpcds
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348044625

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                     HEAD ┃      worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │              6.82 / 7.35 ±0.88 / 9.12 ms │              7.50 / 7.97 ±0.82 / 9.60 ms │  1.08x slower │
   │ QQuery 2  │        144.11 / 145.80 ±1.37 / 147.82 ms │        145.05 / 145.44 ±0.52 / 146.44 ms │     no change │
   │ QQuery 3  │        112.89 / 113.97 ±1.00 / 115.83 ms │        113.71 / 114.46 ±0.57 / 115.18 ms │     no change │
   │ QQuery 4  │    1301.91 / 1371.80 ±45.41 / 1441.97 ms │     1303.73 / 1313.81 ±7.28 / 1323.52 ms │     no change │
   │ QQuery 5  │        177.12 / 177.94 ±0.91 / 179.58 ms │        174.95 / 175.92 ±0.81 / 177.10 ms │     no change │
   │ QQuery 6  │        139.63 / 143.46 ±4.86 / 152.80 ms │        164.45 / 167.94 ±3.69 / 175.09 ms │  1.17x slower │
   │ QQuery 7  │        334.60 / 339.91 ±5.31 / 347.55 ms │        334.15 / 338.60 ±2.42 / 341.38 ms │     no change │
   │ QQuery 8  │        113.10 / 114.83 ±2.19 / 118.94 ms │        112.18 / 112.91 ±1.00 / 114.88 ms │     no change │
   │ QQuery 9  │          96.99 / 99.78 ±2.83 / 103.34 ms │         97.64 / 103.06 ±5.20 / 112.06 ms │     no change │
   │ QQuery 10 │        103.73 / 105.98 ±2.79 / 111.40 ms │        102.66 / 103.45 ±0.63 / 104.59 ms │     no change │
   │ QQuery 11 │       927.86 / 949.55 ±25.50 / 999.47 ms │        918.48 / 925.31 ±6.40 / 934.93 ms │     no change │
   │ QQuery 12 │           44.69 / 46.59 ±1.95 / 50.36 ms │           43.17 / 43.61 ±0.52 / 44.59 ms │ +1.07x faster │
   │ QQuery 13 │        386.43 / 395.32 ±6.96 / 404.82 ms │        386.48 / 396.05 ±6.21 / 404.15 ms │     no change │
   │ QQuery 14 │     1039.18 / 1044.28 ±4.00 / 1051.24 ms │     1029.61 / 1041.36 ±6.59 / 1049.80 ms │     no change │
   │ QQuery 15 │           15.93 / 16.23 ±0.28 / 16.74 ms │           14.57 / 14.90 ±0.20 / 15.15 ms │ +1.09x faster │
   │ QQuery 16 │              7.94 / 8.04 ±0.10 / 8.23 ms │              7.51 / 7.64 ±0.21 / 8.06 ms │     no change │
   │ QQuery 17 │        221.98 / 225.79 ±2.86 / 230.18 ms │        219.58 / 221.83 ±1.67 / 224.18 ms │     no change │
   │ QQuery 18 │        125.86 / 127.19 ±0.84 / 128.15 ms │        119.60 / 120.04 ±0.30 / 120.46 ms │ +1.06x faster │
   │ QQuery 19 │        151.50 / 155.70 ±4.02 / 162.64 ms │        158.08 / 161.32 ±2.57 / 165.37 ms │     no change │
   │ QQuery 20 │           12.91 / 13.05 ±0.11 / 13.22 ms │           13.79 / 13.98 ±0.17 / 14.27 ms │  1.07x slower │
   │ QQuery 21 │           18.78 / 19.29 ±0.60 / 20.46 ms │           20.02 / 20.46 ±0.26 / 20.71 ms │  1.06x slower │
   │ QQuery 22 │        479.25 / 486.16 ±4.70 / 491.56 ms │        509.71 / 515.38 ±3.50 / 519.88 ms │  1.06x slower │
   │ QQuery 23 │    1114.56 / 1130.80 ±12.48 / 1146.58 ms │    1146.90 / 1158.05 ±10.23 / 1176.95 ms │     no change │
   │ QQuery 24 │       714.53 / 734.14 ±10.64 / 744.39 ms │        717.36 / 722.28 ±4.71 / 730.46 ms │     no change │
   │ QQuery 25 │        331.28 / 337.60 ±7.62 / 350.35 ms │        328.83 / 335.73 ±7.19 / 346.10 ms │     no change │
   │ QQuery 26 │           76.90 / 77.79 ±0.91 / 79.50 ms │           77.54 / 77.73 ±0.16 / 77.99 ms │     no change │
   │ QQuery 27 │              7.44 / 7.53 ±0.11 / 7.75 ms │              7.14 / 7.28 ±0.15 / 7.57 ms │     no change │
   │ QQuery 28 │        146.48 / 149.03 ±2.92 / 154.73 ms │        147.48 / 150.56 ±3.57 / 156.86 ms │     no change │
   │ QQuery 29 │        273.31 / 278.78 ±4.64 / 284.26 ms │        270.01 / 274.68 ±8.89 / 292.46 ms │     no change │
   │ QQuery 30 │           41.33 / 42.41 ±0.67 / 43.26 ms │           41.17 / 41.82 ±0.37 / 42.12 ms │     no change │
   │ QQuery 31 │        163.99 / 168.25 ±4.30 / 175.57 ms │        163.01 / 167.58 ±5.34 / 177.82 ms │     no change │
   │ QQuery 32 │           13.25 / 13.42 ±0.22 / 13.84 ms │           13.38 / 13.59 ±0.15 / 13.78 ms │     no change │
   │ QQuery 33 │        137.88 / 139.59 ±1.55 / 142.49 ms │        139.47 / 142.76 ±3.32 / 147.97 ms │     no change │
   │ QQuery 34 │              7.59 / 7.81 ±0.15 / 7.99 ms │              7.05 / 7.19 ±0.19 / 7.57 ms │ +1.09x faster │
   │ QQuery 35 │        100.48 / 103.12 ±2.78 / 106.90 ms │        100.47 / 103.89 ±3.95 / 109.71 ms │     no change │
   │ QQuery 36 │              6.53 / 6.85 ±0.23 / 7.21 ms │              6.92 / 7.33 ±0.25 / 7.64 ms │  1.07x slower │
   │ QQuery 37 │              8.08 / 8.22 ±0.16 / 8.47 ms │              8.12 / 8.30 ±0.16 / 8.57 ms │     no change │
   │ QQuery 38 │           86.19 / 88.68 ±1.88 / 92.02 ms │           84.97 / 87.51 ±1.83 / 89.92 ms │     no change │
   │ QQuery 39 │       116.15 / 122.37 ±10.00 / 142.29 ms │        117.31 / 125.38 ±8.20 / 136.11 ms │     no change │
   │ QQuery 40 │        102.16 / 104.58 ±2.39 / 108.81 ms │        102.86 / 105.70 ±2.95 / 111.14 ms │     no change │
   │ QQuery 41 │           14.38 / 14.72 ±0.30 / 15.29 ms │           14.30 / 14.54 ±0.22 / 14.89 ms │     no change │
   │ QQuery 42 │        106.06 / 107.63 ±0.95 / 108.79 ms │        107.00 / 109.68 ±1.98 / 112.46 ms │     no change │
   │ QQuery 43 │              6.21 / 6.27 ±0.06 / 6.37 ms │              5.87 / 6.00 ±0.13 / 6.25 ms │     no change │
   │ QQuery 44 │           11.49 / 11.77 ±0.15 / 11.90 ms │           10.93 / 11.00 ±0.09 / 11.16 ms │ +1.07x faster │
   │ QQuery 45 │           49.68 / 50.45 ±0.61 / 51.52 ms │           47.85 / 48.12 ±0.17 / 48.33 ms │     no change │
   │ QQuery 46 │            9.11 / 10.82 ±3.20 / 17.22 ms │              8.47 / 8.70 ±0.19 / 9.05 ms │ +1.24x faster │
   │ QQuery 47 │       696.76 / 751.57 ±44.45 / 829.17 ms │       665.06 / 714.64 ±26.70 / 739.29 ms │     no change │
   │ QQuery 48 │        273.16 / 281.92 ±5.22 / 288.01 ms │        283.17 / 287.58 ±2.68 / 290.89 ms │     no change │
   │ QQuery 49 │        251.09 / 254.08 ±2.18 / 256.62 ms │        253.34 / 256.62 ±2.31 / 260.32 ms │     no change │
   │ QQuery 50 │        200.20 / 211.98 ±7.45 / 220.34 ms │        206.80 / 219.25 ±7.16 / 228.10 ms │     no change │
   │ QQuery 51 │        177.49 / 182.70 ±6.46 / 194.47 ms │        177.41 / 182.21 ±3.60 / 187.13 ms │     no change │
   │ QQuery 52 │        106.63 / 107.57 ±1.11 / 109.70 ms │        109.04 / 110.26 ±0.86 / 111.33 ms │     no change │
   │ QQuery 53 │        101.89 / 104.61 ±1.53 / 106.31 ms │        104.76 / 106.58 ±1.83 / 109.95 ms │     no change │
   │ QQuery 54 │        143.24 / 146.05 ±2.98 / 150.76 ms │        148.16 / 151.10 ±1.94 / 153.92 ms │     no change │
   │ QQuery 55 │        106.14 / 108.06 ±1.52 / 109.77 ms │        108.77 / 110.73 ±1.75 / 113.95 ms │     no change │
   │ QQuery 56 │        140.31 / 142.93 ±1.91 / 145.99 ms │        139.87 / 144.69 ±2.74 / 148.33 ms │     no change │
   │ QQuery 57 │        166.06 / 169.64 ±3.18 / 175.06 ms │        167.71 / 170.50 ±2.58 / 174.67 ms │     no change │
   │ QQuery 58 │        266.13 / 267.37 ±1.05 / 269.16 ms │        267.31 / 270.50 ±2.93 / 274.87 ms │     no change │
   │ QQuery 59 │        192.47 / 197.29 ±3.20 / 201.69 ms │        195.60 / 199.25 ±3.03 / 204.53 ms │     no change │
   │ QQuery 60 │        141.83 / 145.98 ±3.12 / 151.49 ms │        139.77 / 142.85 ±2.73 / 146.35 ms │     no change │
   │ QQuery 61 │           14.12 / 14.24 ±0.15 / 14.52 ms │           13.69 / 13.87 ±0.18 / 14.19 ms │     no change │
   │ QQuery 62 │       873.91 / 901.50 ±25.98 / 936.97 ms │       874.60 / 891.39 ±16.98 / 922.72 ms │     no change │
   │ QQuery 63 │        104.89 / 107.59 ±3.71 / 114.86 ms │        101.79 / 104.10 ±2.61 / 108.49 ms │     no change │
   │ QQuery 64 │       669.72 / 687.22 ±15.53 / 712.15 ms │        659.83 / 674.80 ±8.85 / 684.35 ms │     no change │
   │ QQuery 65 │        257.34 / 265.40 ±6.27 / 272.21 ms │       242.62 / 252.34 ±11.16 / 270.52 ms │     no change │
   │ QQuery 66 │       218.47 / 236.56 ±20.58 / 274.82 ms │        214.91 / 226.47 ±7.73 / 236.15 ms │     no change │
   │ QQuery 67 │        294.23 / 306.12 ±8.49 / 316.83 ms │       293.43 / 311.62 ±19.42 / 344.59 ms │     no change │
   │ QQuery 68 │              9.14 / 9.32 ±0.12 / 9.46 ms │              8.59 / 9.13 ±0.37 / 9.66 ms │     no change │
   │ QQuery 69 │         98.65 / 100.25 ±1.99 / 104.04 ms │           98.07 / 98.54 ±0.44 / 99.36 ms │     no change │
   │ QQuery 70 │        315.08 / 324.35 ±7.99 / 338.78 ms │        311.84 / 325.92 ±8.41 / 334.04 ms │     no change │
   │ QQuery 71 │        135.16 / 140.05 ±4.70 / 147.80 ms │        132.65 / 133.89 ±1.39 / 136.55 ms │     no change │
   │ QQuery 72 │        588.29 / 597.97 ±9.76 / 616.13 ms │       580.85 / 611.75 ±16.57 / 629.48 ms │     no change │
   │ QQuery 73 │              6.55 / 6.76 ±0.24 / 7.23 ms │              7.22 / 7.38 ±0.10 / 7.53 ms │  1.09x slower │
   │ QQuery 74 │       544.40 / 576.79 ±21.13 / 604.05 ms │       562.14 / 606.18 ±39.38 / 654.58 ms │  1.05x slower │
   │ QQuery 75 │        266.10 / 270.82 ±3.75 / 276.09 ms │        268.67 / 273.50 ±3.55 / 278.32 ms │     no change │
   │ QQuery 76 │        129.76 / 134.62 ±3.67 / 140.63 ms │        134.23 / 136.46 ±2.64 / 141.49 ms │     no change │
   │ QQuery 77 │        185.79 / 189.78 ±2.75 / 193.25 ms │        191.24 / 194.12 ±1.60 / 195.80 ms │     no change │
   │ QQuery 78 │        326.79 / 332.33 ±3.96 / 338.85 ms │        330.98 / 337.57 ±6.87 / 349.86 ms │     no change │
   │ QQuery 79 │       231.08 / 241.52 ±11.22 / 257.17 ms │       229.59 / 237.43 ±10.31 / 257.49 ms │     no change │
   │ QQuery 80 │        319.41 / 322.98 ±3.98 / 329.95 ms │        322.12 / 324.38 ±1.46 / 326.47 ms │     no change │
   │ QQuery 81 │           26.50 / 27.52 ±1.62 / 30.75 ms │           26.71 / 27.11 ±0.39 / 27.65 ms │     no change │
   │ QQuery 82 │           39.25 / 40.29 ±0.84 / 41.55 ms │           40.19 / 40.71 ±0.31 / 41.13 ms │     no change │
   │ QQuery 83 │           38.53 / 39.12 ±0.40 / 39.57 ms │           37.63 / 38.27 ±0.39 / 38.71 ms │     no change │
   │ QQuery 84 │           46.05 / 46.52 ±0.43 / 47.25 ms │           46.29 / 46.68 ±0.37 / 47.35 ms │     no change │
   │ QQuery 85 │        140.36 / 143.84 ±3.11 / 148.47 ms │        141.58 / 142.92 ±1.78 / 146.10 ms │     no change │
   │ QQuery 86 │           38.33 / 38.82 ±0.32 / 39.18 ms │           37.45 / 38.07 ±0.52 / 38.96 ms │     no change │
   │ QQuery 87 │              3.53 / 3.65 ±0.16 / 3.97 ms │              3.62 / 3.73 ±0.16 / 4.04 ms │     no change │
   │ QQuery 88 │         99.53 / 101.98 ±3.68 / 109.29 ms │         99.77 / 102.06 ±1.86 / 105.05 ms │     no change │
   │ QQuery 89 │        118.71 / 119.97 ±0.87 / 121.18 ms │        116.25 / 119.97 ±4.20 / 127.47 ms │     no change │
   │ QQuery 90 │           22.21 / 22.69 ±0.28 / 22.95 ms │           24.17 / 24.54 ±0.26 / 24.85 ms │  1.08x slower │
   │ QQuery 91 │           58.07 / 59.93 ±2.08 / 63.87 ms │           58.49 / 59.65 ±0.63 / 60.24 ms │     no change │
   │ QQuery 92 │           56.58 / 57.69 ±0.64 / 58.30 ms │           56.48 / 57.48 ±0.98 / 59.30 ms │     no change │
   │ QQuery 93 │        179.59 / 186.69 ±3.83 / 191.20 ms │        179.88 / 183.01 ±3.60 / 189.86 ms │     no change │
   │ QQuery 94 │           62.66 / 63.21 ±0.38 / 63.65 ms │           61.07 / 62.62 ±1.86 / 66.12 ms │     no change │
   │ QQuery 95 │        129.20 / 130.54 ±1.27 / 132.72 ms │        125.63 / 127.68 ±2.48 / 132.17 ms │     no change │
   │ QQuery 96 │           68.34 / 69.77 ±1.59 / 72.75 ms │           70.14 / 71.10 ±0.78 / 72.14 ms │     no change │
   │ QQuery 97 │        119.16 / 123.44 ±2.55 / 126.43 ms │        120.24 / 123.73 ±2.04 / 125.85 ms │     no change │
   │ QQuery 98 │        149.95 / 155.15 ±3.90 / 159.96 ms │        153.41 / 155.80 ±3.18 / 162.01 ms │     no change │
   │ QQuery 99 │ 10830.00 / 10947.63 ±60.60 / 10993.78 ms │ 10813.72 / 10880.17 ±57.60 / 10965.09 ms │     no change │
   └───────────┴──────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 31047.06ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 30943.71ms │
   │ Average Time (HEAD)                                │   313.61ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   312.56ms │
   │ Queries Faster                                     │          6 │
   │ Queries Slower                                     │          9 │
   │ Queries with No Change                             │         84 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 160.0s |
   | Peak memory | 6.4 GiB |
   | Avg memory | 5.7 GiB |
   | CPU user | 246.0s |
   | CPU sys | 8.6s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 155.0s |
   | Peak memory | 6.3 GiB |
   | Avg memory | 5.7 GiB |
   | CPU user | 245.7s |
   | CPU sys | 7.8s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348060844

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
    Downloading crates ...
     Downloaded prost v0.14.3
     Downloaded anyhow v1.0.102
     Downloaded prost-derive v0.14.3
      Compiling anyhow v1.0.102
      Compiling either v1.15.0
      Compiling datafusion-benchmarks v53.1.0 (/workspace/datafusion-base/benchmarks)
      Compiling itertools v0.14.0
      Compiling prost-derive v0.14.3
      Compiling prost v0.14.3
      Compiling datafusion-proto-common v53.1.0 (/workspace/datafusion-base/datafusion/proto-common)
      Compiling datafusion-proto v53.1.0 (/workspace/datafusion-base/datafusion/proto)
       Finished `bench` profile [optimized + debuginfo] target(s) in 9m 18s
        Running benches/sql.rs (/workspace/datafusion-base/target/release/deps/sql-2c9f51cbfcaf27f8)
   Gnuplot not found, using plotters backend
   
   thread 'main' (21351) panicked at benchmarks/benches/sql.rs:131:46:
   assertion failed: Execution("Error in result on row 1, column 1 running query \"\nSELECT COUNT(*) > 0 from lineitem;\": expected value \"true\" but got value \"false\" in row: [\"false\"]")
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   error: bench failed, to rerun pass `--bench sql`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348071047

   🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347920152-1938-4xqr8 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d)
   BENCH_NAME=hj
   BENCH_COMMAND=cargo bench --features=parquet --bench hj
   BENCH_FILTER=
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4470329535

   > thanks for the ping @gene-bordegaray, I had [42adf29](https://github.com/apache/datafusion/commit/42adf29b6bd78b10156cec3bc8838570fe666aed) locally but had not pushed it (it's been crazy trying to squeeze work into breaks at PyCon). The test seems exactly like what you suggested 😃
   
   @adriangb 
   
   Hahaha squeezing in dyn filter work at breaks, pretty hectic 😅 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4364567871

   Yes agreed.
   
   My thought is that this a good default for now that is less brittle to things messing with partitioning.
   
   And down the line once we've got a good story for range partitioning we can use https://github.com/apache/datafusion/pull/21900 for hash partitioned cases (assuming performance looks good) and some other system for range partitioning.
   
   @Dandandan I looked into https://github.com/apache/datafusion/pull/21900 but immediately realized the point is that would make the partition routing faster, but part of the goal here is to not have any partition routing because it introduces brittleness and is slower than just probing more hash tables (although https://github.com/apache/datafusion/pull/21900 could flip the performance story, we'd have to confirm).
   
   In any case - we can always merge this now and follow up with a version of `MultiMapLookupExpr` that uses https://github.com/apache/datafusion/pull/21900 if perf looks good?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4579990252

   i am but i’m just strapped way too thin atm, if you can take a look would really appreciate it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471454119

   🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471353144-179-kkrvn 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed)
   BENCH_NAME=hj
   BENCH_COMMAND=cargo bench --features=parquet --bench hj
   BENCH_FILTER=
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471433196

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.28 / 38.99 ±0.95 / 40.86 ms │      38.19 / 38.99 ±0.96 / 40.89 ms │     no change │
   │ QQuery 2  │ 20.02 / 20.44 ±0.31 / 20.97 ms │      20.44 / 20.60 ±0.17 / 20.91 ms │     no change │
   │ QQuery 3  │ 34.49 / 37.00 ±1.56 / 39.21 ms │      34.48 / 34.91 ±0.37 / 35.42 ms │ +1.06x faster │
   │ QQuery 4  │ 17.26 / 17.45 ±0.14 / 17.64 ms │      17.48 / 17.60 ±0.10 / 17.73 ms │     no change │
   │ QQuery 5  │ 40.55 / 42.62 ±1.05 / 43.35 ms │      42.61 / 43.73 ±1.26 / 45.49 ms │     no change │
   │ QQuery 6  │ 16.01 / 16.59 ±0.88 / 18.33 ms │      16.35 / 16.76 ±0.50 / 17.69 ms │     no change │
   │ QQuery 7  │ 48.44 / 49.18 ±0.79 / 50.65 ms │      47.52 / 49.99 ±1.79 / 52.79 ms │     no change │
   │ QQuery 8  │ 44.62 / 45.36 ±0.75 / 46.78 ms │      44.50 / 44.58 ±0.06 / 44.66 ms │     no change │
   │ QQuery 9  │ 49.55 / 51.64 ±1.49 / 53.51 ms │      48.82 / 50.33 ±1.32 / 52.74 ms │     no change │
   │ QQuery 10 │ 63.31 / 64.62 ±0.89 / 66.10 ms │      63.33 / 63.54 ±0.19 / 63.87 ms │     no change │
   │ QQuery 11 │ 13.08 / 13.55 ±0.41 / 14.12 ms │      13.24 / 13.43 ±0.13 / 13.63 ms │     no change │
   │ QQuery 12 │ 24.39 / 25.16 ±1.14 / 27.41 ms │      24.52 / 25.17 ±0.61 / 26.30 ms │     no change │
   │ QQuery 13 │ 35.60 / 36.08 ±0.47 / 36.98 ms │      35.83 / 36.43 ±0.81 / 38.02 ms │     no change │
   │ QQuery 14 │ 25.34 / 25.48 ±0.08 / 25.57 ms │      25.35 / 25.73 ±0.30 / 26.12 ms │     no change │
   │ QQuery 15 │ 31.11 / 31.68 ±0.65 / 32.92 ms │      31.14 / 31.35 ±0.19 / 31.60 ms │     no change │
   │ QQuery 16 │ 14.71 / 14.90 ±0.14 / 15.06 ms │      14.88 / 15.03 ±0.15 / 15.31 ms │     no change │
   │ QQuery 17 │ 73.74 / 74.08 ±0.29 / 74.52 ms │      66.37 / 67.21 ±0.66 / 68.03 ms │ +1.10x faster │
   │ QQuery 18 │ 64.84 / 65.80 ±0.56 / 66.48 ms │      68.21 / 69.94 ±1.57 / 72.74 ms │  1.06x slower │
   │ QQuery 19 │ 34.91 / 35.27 ±0.53 / 36.28 ms │      35.19 / 35.41 ±0.17 / 35.66 ms │     no change │
   │ QQuery 20 │ 37.40 / 37.75 ±0.25 / 38.15 ms │      37.59 / 37.66 ±0.06 / 37.76 ms │     no change │
   │ QQuery 21 │ 56.97 / 58.57 ±1.80 / 61.67 ms │      56.14 / 58.68 ±1.52 / 60.92 ms │     no change │
   │ QQuery 22 │ 22.92 / 23.53 ±0.45 / 24.11 ms │      23.61 / 25.71 ±2.88 / 31.40 ms │  1.09x slower │
   └───────────┴────────────────────────────────┴─────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 825.73ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 822.78ms │
   │ Average Time (HEAD)                                │  37.53ms │
   │ Average Time (worktree-dynamic-filter-restructure) │  37.40ms │
   │ Queries Faster                                     │        2 │
   │ Queries Slower                                     │        2 │
   │ Queries with No Change                             │       18 │
   │ Queries with Failure                               │        0 │
   └────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 5.5 GiB |
   | Avg memory | 5.0 GiB |
   | CPU user | 30.1s |
   | CPU sys | 2.2s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 5.6 GiB |
   | Avg memory | 5.0 GiB |
   | CPU user | 30.2s |
   | CPU sys | 2.2s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471363345

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471352777-171-k955k 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471776218

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471633434)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpch_sf10.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                               HEAD ┃   worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │  313.69 / 316.71 ±3.99 / 324.52 ms │     317.79 / 319.81 ±2.75 / 325.27 ms │     no change │
   │ QQuery 2  │  120.80 / 122.50 ±1.66 / 125.39 ms │     172.62 / 174.62 ±1.30 / 175.99 ms │  1.43x slower │
   │ QQuery 3  │  252.94 / 255.75 ±2.18 / 258.85 ms │     364.39 / 366.89 ±2.02 / 370.38 ms │  1.43x slower │
   │ QQuery 4  │  127.66 / 129.75 ±2.31 / 134.19 ms │     315.90 / 320.12 ±3.76 / 326.84 ms │  2.47x slower │
   │ QQuery 5  │  383.09 / 391.29 ±6.62 / 399.51 ms │     589.24 / 598.78 ±8.76 / 613.68 ms │  1.53x slower │
   │ QQuery 6  │  129.43 / 130.34 ±0.82 / 131.53 ms │     290.07 / 291.85 ±1.92 / 295.58 ms │  2.24x slower │
   │ QQuery 7  │  508.07 / 514.76 ±6.35 / 526.52 ms │     435.58 / 438.26 ±2.34 / 441.66 ms │ +1.17x faster │
   │ QQuery 8  │  408.40 / 416.09 ±6.18 / 425.45 ms │     621.61 / 624.81 ±2.03 / 627.92 ms │  1.50x slower │
   │ QQuery 9  │  564.97 / 580.60 ±8.16 / 588.51 ms │ 1347.46 / 1361.81 ±11.04 / 1373.81 ms │  2.35x slower │
   │ QQuery 10 │  310.11 / 321.14 ±6.81 / 328.39 ms │     397.12 / 403.50 ±3.90 / 408.62 ms │  1.26x slower │
   │ QQuery 11 │    88.55 / 94.57 ±5.03 / 103.50 ms │    141.29 / 148.45 ±10.12 / 167.29 ms │  1.57x slower │
   │ QQuery 12 │  178.26 / 184.80 ±7.06 / 194.59 ms │     339.41 / 342.50 ±2.04 / 344.52 ms │  1.85x slower │
   │ QQuery 13 │  297.96 / 306.97 ±7.40 / 320.27 ms │    400.72 / 413.28 ±11.97 / 434.98 ms │  1.35x slower │
   │ QQuery 14 │  175.27 / 177.23 ±2.25 / 181.63 ms │     199.93 / 203.72 ±2.53 / 206.61 ms │  1.15x slower │
   │ QQuery 15 │  316.41 / 324.34 ±4.93 / 331.62 ms │     395.26 / 401.50 ±3.68 / 405.32 ms │  1.24x slower │
   │ QQuery 16 │     74.74 / 77.56 ±1.91 / 79.93 ms │     102.27 / 103.73 ±1.12 / 105.41 ms │  1.34x slower │
   │ QQuery 17 │  683.03 / 696.23 ±7.97 / 704.35 ms │    757.66 / 771.08 ±17.04 / 804.72 ms │  1.11x slower │
   │ QQuery 18 │ 690.26 / 707.63 ±10.44 / 718.66 ms │    679.79 / 693.25 ±11.13 / 710.98 ms │     no change │
   │ QQuery 19 │ 253.93 / 267.66 ±10.27 / 281.32 ms │    283.10 / 291.69 ±10.03 / 310.93 ms │  1.09x slower │
   │ QQuery 20 │  287.04 / 291.81 ±3.65 / 297.45 ms │     325.34 / 331.13 ±5.97 / 341.37 ms │  1.13x slower │
   │ QQuery 21 │ 702.28 / 724.77 ±19.04 / 747.34 ms │  1179.06 / 1183.21 ±3.51 / 1189.51 ms │  1.63x slower │
   │ QQuery 22 │     65.44 / 67.93 ±1.55 / 70.08 ms │     128.58 / 134.66 ±3.24 / 137.90 ms │  1.98x slower │
   └───────────┴────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 7100.43ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 9918.66ms │
   │ Average Time (HEAD)                                │  322.75ms │
   │ Average Time (worktree-dynamic-filter-restructure) │  450.85ms │
   │ Queries Faster                                     │         1 │
   │ Queries Slower                                     │        19 │
   │ Queries with No Change                             │         2 │
   │ Queries with Failure                               │         0 │
   └────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch10 — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 40.0s |
   | Peak memory | 12.0 GiB |
   | Avg memory | 8.5 GiB |
   | CPU user | 378.0s |
   | CPU sys | 20.6s |
   | Peak spill | 0 B |
   
   **tpch10 — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 9.8 GiB |
   | Avg memory | 8.1 GiB |
   | CPU user | 546.5s |
   | CPU sys | 18.5s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345732500

   run benchmark tpch
   
   ```yaml
   baseline:
       ref: main
       env:
          DATAFUSION_EXECUTION_TARGET_PARTITIONS=128
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: true
   changed:
       ref: HEAD
       env:
          DATAFUSION_EXECUTION_TARGET_PARTITIONS=128
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: true
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348078491

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347920499-1939-7shw7 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: clickbench_partitioned
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348067486

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347920152-1935-57b9c 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348156232

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                      HEAD ┃       worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │            24.37 / 24.75 ±0.66 / 26.07 ms │            23.96 / 24.69 ±0.71 / 26.04 ms │     no change │
   │ QQuery 2  │         206.29 / 208.06 ±2.30 / 212.60 ms │         203.51 / 204.09 ±0.50 / 204.96 ms │     no change │
   │ QQuery 3  │         123.31 / 124.78 ±0.99 / 126.00 ms │         124.95 / 125.44 ±0.44 / 126.06 ms │     no change │
   │ QQuery 4  │      1136.06 / 1144.67 ±4.70 / 1148.27 ms │      1136.91 / 1146.00 ±7.25 / 1154.03 ms │     no change │
   │ QQuery 5  │        294.21 / 315.17 ±12.93 / 328.57 ms │         285.84 / 292.61 ±3.78 / 296.60 ms │ +1.08x faster │
   │ QQuery 6  │         121.99 / 124.89 ±2.21 / 128.27 ms │         125.62 / 130.57 ±4.80 / 138.90 ms │     no change │
   │ QQuery 7  │        441.27 / 453.68 ±10.80 / 469.36 ms │         619.35 / 623.28 ±4.85 / 632.62 ms │  1.37x slower │
   │ QQuery 8  │         177.11 / 181.95 ±5.15 / 191.89 ms │         179.71 / 182.67 ±3.07 / 188.46 ms │     no change │
   │ QQuery 9  │         289.74 / 298.57 ±5.61 / 304.82 ms │         289.57 / 298.06 ±5.38 / 305.76 ms │     no change │
   │ QQuery 10 │        174.78 / 185.88 ±11.06 / 206.20 ms │         177.70 / 190.51 ±8.48 / 204.16 ms │     no change │
   │ QQuery 11 │         724.99 / 738.55 ±8.84 / 752.13 ms │         734.19 / 742.60 ±4.64 / 748.41 ms │     no change │
   │ QQuery 12 │            58.76 / 61.37 ±2.51 / 66.06 ms │            60.02 / 61.07 ±0.71 / 61.94 ms │     no change │
   │ QQuery 13 │        542.67 / 554.99 ±10.36 / 573.70 ms │         552.51 / 566.03 ±8.94 / 577.87 ms │     no change │
   │ QQuery 14 │     1214.14 / 1245.82 ±24.48 / 1288.46 ms │     1217.20 / 1256.01 ±30.21 / 1292.24 ms │     no change │
   │ QQuery 15 │          98.04 / 104.30 ±5.11 / 109.94 ms │            72.25 / 77.62 ±6.80 / 91.05 ms │ +1.34x faster │
   │ QQuery 16 │            23.08 / 23.47 ±0.27 / 23.73 ms │            22.87 / 23.91 ±0.76 / 25.23 ms │     no change │
   │ QQuery 17 │        295.72 / 310.91 ±10.05 / 327.08 ms │         244.05 / 253.44 ±6.68 / 261.15 ms │ +1.23x faster │
   │ QQuery 18 │         819.77 / 826.83 ±5.17 / 835.24 ms │         793.38 / 798.50 ±4.42 / 805.03 ms │     no change │
   │ QQuery 19 │         149.78 / 155.34 ±3.19 / 158.54 ms │         154.11 / 156.32 ±2.17 / 159.88 ms │     no change │
   │ QQuery 20 │            39.60 / 41.54 ±1.89 / 45.13 ms │            40.68 / 41.53 ±0.72 / 42.57 ms │     no change │
   │ QQuery 21 │            43.99 / 45.20 ±1.44 / 48.00 ms │            42.19 / 42.46 ±0.28 / 42.95 ms │ +1.06x faster │
   │ QQuery 22 │         519.90 / 529.44 ±5.29 / 535.06 ms │         514.12 / 524.52 ±7.75 / 535.21 ms │     no change │
   │ QQuery 23 │     2564.38 / 2592.90 ±24.81 / 2630.63 ms │     4532.57 / 4559.10 ±20.97 / 4590.18 ms │  1.76x slower │
   │ QQuery 24 │        589.06 / 618.06 ±20.06 / 644.94 ms │        362.80 / 374.85 ±10.26 / 391.15 ms │ +1.65x faster │
   │ QQuery 25 │         564.64 / 574.50 ±6.78 / 582.98 ms │         634.55 / 645.13 ±8.88 / 659.74 ms │  1.12x slower │
   │ QQuery 26 │        237.80 / 255.80 ±11.32 / 267.92 ms │         308.27 / 312.99 ±4.29 / 320.44 ms │  1.22x slower │
   │ QQuery 27 │            25.20 / 25.80 ±0.51 / 26.44 ms │            25.05 / 25.47 ±0.23 / 25.65 ms │     no change │
   │ QQuery 28 │         236.83 / 240.46 ±3.43 / 245.72 ms │        232.95 / 242.66 ±10.02 / 261.00 ms │     no change │
   │ QQuery 29 │        379.63 / 399.01 ±15.46 / 420.74 ms │         339.60 / 350.15 ±9.99 / 367.14 ms │ +1.14x faster │
   │ QQuery 30 │            85.96 / 87.07 ±1.10 / 88.88 ms │            85.84 / 86.72 ±0.71 / 87.63 ms │     no change │
   │ QQuery 31 │         216.27 / 226.62 ±7.13 / 235.32 ms │         212.89 / 222.12 ±7.32 / 232.85 ms │     no change │
   │ QQuery 32 │            33.81 / 36.60 ±1.70 / 38.56 ms │           33.52 / 41.64 ±11.82 / 64.99 ms │  1.14x slower │
   │ QQuery 33 │         151.72 / 156.27 ±2.69 / 159.78 ms │         159.48 / 161.57 ±2.06 / 164.34 ms │     no change │
   │ QQuery 34 │            22.99 / 23.47 ±0.45 / 24.27 ms │            22.55 / 25.51 ±3.33 / 32.00 ms │  1.09x slower │
   │ QQuery 35 │         163.45 / 165.82 ±2.22 / 169.51 ms │         166.46 / 170.59 ±3.91 / 177.63 ms │     no change │
   │ QQuery 36 │            30.50 / 31.55 ±0.68 / 32.46 ms │            29.87 / 35.29 ±8.86 / 52.98 ms │  1.12x slower │
   │ QQuery 37 │            15.55 / 18.37 ±4.55 / 27.43 ms │            15.13 / 15.78 ±0.43 / 16.25 ms │ +1.16x faster │
   │ QQuery 38 │         145.37 / 149.88 ±3.75 / 154.19 ms │         143.83 / 148.43 ±4.90 / 157.90 ms │     no change │
   │ QQuery 39 │         244.70 / 246.82 ±1.32 / 248.34 ms │        243.09 / 250.34 ±10.18 / 270.32 ms │     no change │
   │ QQuery 40 │         206.57 / 215.95 ±6.22 / 224.87 ms │         197.84 / 207.43 ±8.10 / 217.81 ms │     no change │
   │ QQuery 41 │            35.18 / 35.87 ±0.56 / 36.74 ms │            34.68 / 35.03 ±0.32 / 35.53 ms │     no change │
   │ QQuery 42 │         118.31 / 122.18 ±4.71 / 129.49 ms │         120.27 / 122.64 ±1.32 / 124.01 ms │     no change │
   │ QQuery 43 │            19.68 / 19.98 ±0.25 / 20.30 ms │            19.25 / 19.58 ±0.29 / 20.07 ms │     no change │
   │ QQuery 44 │            54.15 / 59.76 ±6.35 / 71.95 ms │            54.48 / 59.46 ±7.76 / 74.93 ms │     no change │
   │ QQuery 45 │         119.76 / 124.56 ±4.66 / 133.42 ms │            81.18 / 84.12 ±1.78 / 85.86 ms │ +1.48x faster │
   │ QQuery 46 │            24.85 / 25.98 ±0.86 / 26.96 ms │            23.85 / 24.59 ±0.91 / 26.23 ms │ +1.06x faster │
   │ QQuery 47 │         876.71 / 882.60 ±3.83 / 886.45 ms │         875.75 / 886.62 ±6.03 / 893.33 ms │     no change │
   │ QQuery 48 │        458.56 / 474.73 ±18.90 / 510.70 ms │         464.62 / 478.25 ±7.87 / 488.22 ms │     no change │
   │ QQuery 49 │         348.63 / 363.03 ±8.08 / 369.83 ms │         351.63 / 358.30 ±6.43 / 369.73 ms │     no change │
   │ QQuery 50 │      1323.65 / 1341.34 ±8.89 / 1347.36 ms │     1653.53 / 1667.33 ±10.15 / 1682.98 ms │  1.24x slower │
   │ QQuery 51 │         291.39 / 303.12 ±6.32 / 310.20 ms │         301.55 / 304.46 ±2.89 / 309.65 ms │     no change │
   │ QQuery 52 │         122.11 / 126.52 ±2.67 / 129.70 ms │         124.62 / 128.32 ±3.26 / 134.22 ms │     no change │
   │ QQuery 53 │         155.57 / 163.33 ±7.05 / 175.17 ms │         157.26 / 161.73 ±4.72 / 170.68 ms │     no change │
   │ QQuery 54 │         180.60 / 187.38 ±4.83 / 194.96 ms │         179.07 / 183.09 ±3.58 / 189.80 ms │     no change │
   │ QQuery 55 │         119.73 / 125.36 ±5.86 / 135.18 ms │         119.86 / 123.37 ±3.18 / 128.44 ms │     no change │
   │ QQuery 56 │         155.70 / 159.96 ±2.64 / 163.69 ms │         158.57 / 161.42 ±2.01 / 163.71 ms │     no change │
   │ QQuery 57 │         275.36 / 282.85 ±5.45 / 290.44 ms │         276.09 / 282.33 ±3.24 / 285.35 ms │     no change │
   │ QQuery 58 │         265.15 / 268.35 ±1.94 / 270.22 ms │         217.11 / 222.69 ±3.95 / 228.13 ms │ +1.20x faster │
   │ QQuery 59 │         285.92 / 290.86 ±3.41 / 295.49 ms │         287.95 / 294.31 ±4.47 / 299.63 ms │     no change │
   │ QQuery 60 │         158.76 / 163.74 ±3.50 / 168.24 ms │         160.58 / 165.43 ±4.18 / 171.37 ms │     no change │
   │ QQuery 61 │            23.37 / 23.63 ±0.25 / 24.05 ms │            24.12 / 24.77 ±0.48 / 25.31 ms │     no change │
   │ QQuery 62 │         914.20 / 923.63 ±7.90 / 938.13 ms │         919.75 / 932.25 ±8.77 / 946.07 ms │     no change │
   │ QQuery 63 │         155.69 / 166.92 ±8.28 / 181.03 ms │         156.33 / 163.46 ±6.91 / 174.62 ms │     no change │
   │ QQuery 64 │     1469.61 / 1494.53 ±16.25 / 1517.09 ms │     1743.60 / 1760.01 ±14.79 / 1782.86 ms │  1.18x slower │
   │ QQuery 65 │        379.49 / 387.66 ±10.17 / 407.54 ms │        380.45 / 397.45 ±16.70 / 419.44 ms │     no change │
   │ QQuery 66 │         209.37 / 218.97 ±9.43 / 234.72 ms │         211.51 / 218.76 ±7.41 / 229.73 ms │     no change │
   │ QQuery 67 │        524.78 / 547.38 ±16.52 / 568.30 ms │        529.89 / 548.56 ±12.05 / 564.04 ms │     no change │
   │ QQuery 68 │            25.00 / 25.31 ±0.36 / 26.00 ms │            24.28 / 27.66 ±4.88 / 37.29 ms │  1.09x slower │
   │ QQuery 69 │         173.69 / 182.32 ±5.40 / 188.37 ms │        172.78 / 184.78 ±10.10 / 199.96 ms │     no change │
   │ QQuery 70 │        449.53 / 459.95 ±12.99 / 485.41 ms │         454.35 / 460.46 ±4.18 / 466.30 ms │     no change │
   │ QQuery 71 │         145.97 / 149.40 ±3.13 / 155.03 ms │         150.98 / 158.60 ±9.97 / 177.89 ms │  1.06x slower │
   │ QQuery 72 │     3999.94 / 4060.72 ±44.12 / 4128.30 ms │     4412.21 / 4425.75 ±13.12 / 4445.80 ms │  1.09x slower │
   │ QQuery 73 │            23.10 / 23.59 ±0.39 / 24.17 ms │            23.29 / 27.09 ±6.34 / 39.75 ms │  1.15x slower │
   │ QQuery 74 │        531.71 / 543.21 ±13.34 / 568.82 ms │         542.77 / 547.06 ±4.47 / 554.98 ms │     no change │
   │ QQuery 75 │        377.54 / 393.12 ±10.67 / 405.60 ms │         376.66 / 388.68 ±9.03 / 399.38 ms │     no change │
   │ QQuery 76 │        663.72 / 695.97 ±25.17 / 740.60 ms │      1106.22 / 1110.87 ±4.02 / 1116.57 ms │  1.60x slower │
   │ QQuery 77 │         266.17 / 270.17 ±3.32 / 274.10 ms │         269.75 / 274.36 ±3.33 / 278.42 ms │     no change │
   │ QQuery 78 │        401.97 / 416.16 ±10.48 / 433.10 ms │         407.45 / 413.85 ±5.94 / 423.27 ms │     no change │
   │ QQuery 79 │         263.98 / 274.08 ±6.58 / 281.53 ms │         262.43 / 273.07 ±8.38 / 283.86 ms │     no change │
   │ QQuery 80 │        347.91 / 359.78 ±10.41 / 378.76 ms │         338.28 / 349.05 ±8.40 / 360.12 ms │     no change │
   │ QQuery 81 │            61.65 / 62.91 ±0.80 / 63.84 ms │            61.84 / 62.49 ±0.51 / 63.28 ms │     no change │
   │ QQuery 82 │            57.91 / 59.35 ±0.97 / 60.88 ms │            57.73 / 61.45 ±5.77 / 72.87 ms │     no change │
   │ QQuery 83 │         100.12 / 105.56 ±5.15 / 115.02 ms │          93.86 / 100.05 ±6.29 / 111.96 ms │ +1.06x faster │
   │ QQuery 84 │            72.69 / 74.06 ±1.36 / 75.89 ms │            72.69 / 76.23 ±3.12 / 82.00 ms │     no change │
   │ QQuery 85 │        358.31 / 379.79 ±15.69 / 402.90 ms │        372.64 / 381.07 ±11.24 / 402.57 ms │     no change │
   │ QQuery 86 │            65.33 / 68.72 ±5.14 / 78.92 ms │            66.00 / 67.37 ±1.18 / 68.96 ms │     no change │
   │ QQuery 87 │               3.53 / 3.66 ±0.23 / 4.11 ms │               3.59 / 3.73 ±0.27 / 4.27 ms │     no change │
   │ QQuery 88 │        158.40 / 166.56 ±10.03 / 185.82 ms │        156.79 / 169.49 ±10.97 / 187.69 ms │     no change │
   │ QQuery 89 │        175.81 / 191.41 ±22.86 / 236.22 ms │        176.34 / 187.00 ±11.28 / 201.15 ms │     no change │
   │ QQuery 90 │            38.28 / 39.61 ±0.68 / 40.13 ms │            38.23 / 39.40 ±0.91 / 40.42 ms │     no change │
   │ QQuery 91 │         107.11 / 113.88 ±7.45 / 127.56 ms │         102.87 / 109.14 ±4.19 / 115.02 ms │     no change │
   │ QQuery 92 │            67.05 / 69.81 ±1.78 / 72.49 ms │            67.98 / 71.68 ±3.83 / 78.63 ms │     no change │
   │ QQuery 93 │         322.41 / 326.68 ±4.03 / 333.56 ms │         313.52 / 320.56 ±4.79 / 326.04 ms │     no change │
   │ QQuery 94 │            90.52 / 91.37 ±0.70 / 92.24 ms │            93.01 / 94.56 ±1.39 / 96.66 ms │     no change │
   │ QQuery 95 │         336.58 / 342.91 ±4.20 / 349.32 ms │         293.79 / 304.41 ±7.78 / 312.61 ms │ +1.13x faster │
   │ QQuery 96 │            83.03 / 88.23 ±3.96 / 94.58 ms │            84.28 / 86.96 ±2.29 / 89.92 ms │     no change │
   │ QQuery 97 │         187.70 / 195.14 ±6.36 / 204.37 ms │         188.22 / 191.97 ±3.19 / 196.15 ms │     no change │
   │ QQuery 98 │         138.22 / 141.16 ±2.94 / 145.70 ms │         137.57 / 143.64 ±3.32 / 146.28 ms │     no change │
   │ QQuery 99 │ 10864.81 / 11171.83 ±165.92 / 11336.69 ms │ 10842.69 / 11059.94 ±119.52 / 11185.66 ms │     no change │
   └───────────┴───────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 44595.75ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 47618.44ms │
   │ Average Time (HEAD)                                │   450.46ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   480.99ms │
   │ Queries Faster                                     │         12 │
   │ Queries Slower                                     │         14 │
   │ Queries with No Change                             │         73 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 225.1s |
   | Peak memory | 6.5 GiB |
   | Avg memory | 5.7 GiB |
   | CPU user | 479.8s |
   | CPU sys | 25.6s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 240.1s |
   | Peak memory | 6.3 GiB |
   | Avg memory | 5.7 GiB |
   | CPU user | 412.5s |
   | CPU sys | 26.9s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348131705

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                   HEAD ┃    worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │           1.47 / 4.85 ±6.72 / 18.28 ms │           1.54 / 5.13 ±7.01 / 19.15 ms │  1.06x slower │
   │ QQuery 1  │         16.96 / 18.02 ±0.68 / 18.91 ms │         16.76 / 18.93 ±1.11 / 19.83 ms │  1.05x slower │
   │ QQuery 2  │         42.34 / 42.78 ±0.34 / 43.28 ms │         41.09 / 42.35 ±1.04 / 43.80 ms │     no change │
   │ QQuery 3  │         34.30 / 35.46 ±0.95 / 37.05 ms │         34.71 / 35.24 ±0.40 / 35.93 ms │     no change │
   │ QQuery 4  │      268.57 / 270.71 ±1.15 / 271.88 ms │      274.18 / 276.51 ±1.44 / 278.04 ms │     no change │
   │ QQuery 5  │      321.04 / 323.13 ±1.79 / 326.07 ms │      323.92 / 331.05 ±4.20 / 336.29 ms │     no change │
   │ QQuery 6  │         12.26 / 13.18 ±0.60 / 14.13 ms │         14.05 / 17.09 ±3.90 / 24.78 ms │  1.30x slower │
   │ QQuery 7  │         25.35 / 27.02 ±2.34 / 31.65 ms │         25.30 / 26.04 ±0.44 / 26.50 ms │     no change │
   │ QQuery 8  │      376.79 / 378.80 ±1.65 / 380.96 ms │      378.01 / 382.34 ±2.29 / 384.49 ms │     no change │
   │ QQuery 9  │     358.92 / 374.49 ±12.75 / 397.06 ms │     374.19 / 393.09 ±15.27 / 419.74 ms │     no change │
   │ QQuery 10 │        96.24 / 98.00 ±1.82 / 100.42 ms │         96.12 / 96.90 ±0.66 / 97.77 ms │     no change │
   │ QQuery 11 │      109.27 / 109.90 ±0.74 / 110.87 ms │      109.34 / 110.93 ±1.65 / 113.89 ms │     no change │
   │ QQuery 12 │      315.46 / 318.96 ±3.30 / 324.59 ms │      319.54 / 327.53 ±6.33 / 335.23 ms │     no change │
   │ QQuery 13 │      564.68 / 575.67 ±7.25 / 587.47 ms │     588.65 / 611.89 ±26.45 / 658.94 ms │  1.06x slower │
   │ QQuery 14 │      319.15 / 320.42 ±1.21 / 322.32 ms │     331.87 / 343.39 ±10.01 / 356.11 ms │  1.07x slower │
   │ QQuery 15 │      329.25 / 334.56 ±5.81 / 344.54 ms │     328.88 / 366.27 ±51.64 / 466.46 ms │  1.09x slower │
   │ QQuery 16 │      694.46 / 707.11 ±8.97 / 718.09 ms │     712.83 / 790.25 ±78.52 / 903.54 ms │  1.12x slower │
   │ QQuery 17 │     692.05 / 707.24 ±13.54 / 726.59 ms │     722.29 / 731.34 ±12.28 / 755.26 ms │     no change │
   │ QQuery 18 │  1419.04 / 1447.33 ±19.91 / 1478.85 ms │   1466.87 / 1475.15 ±7.77 / 1488.55 ms │     no change │
   │ QQuery 19 │        35.57 / 47.77 ±15.79 / 75.28 ms │        35.37 / 51.97 ±15.70 / 77.51 ms │  1.09x slower │
   │ QQuery 20 │      525.41 / 534.23 ±7.23 / 543.80 ms │     500.15 / 515.68 ±11.08 / 534.28 ms │     no change │
   │ QQuery 21 │      597.84 / 606.78 ±9.32 / 619.38 ms │     594.43 / 631.03 ±35.55 / 697.58 ms │     no change │
   │ QQuery 22 │  1050.82 / 1107.46 ±56.54 / 1196.88 ms │  1044.45 / 1083.74 ±24.23 / 1111.39 ms │     no change │
   │ QQuery 23 │ 2126.49 / 2324.05 ±125.54 / 2512.03 ms │ 2191.48 / 2429.89 ±150.55 / 2603.46 ms │     no change │
   │ QQuery 24 │       51.98 / 84.09 ±35.63 / 146.01 ms │         52.57 / 60.28 ±4.40 / 65.25 ms │ +1.39x faster │
   │ QQuery 25 │      116.94 / 120.16 ±2.75 / 124.28 ms │      117.84 / 124.34 ±7.54 / 138.87 ms │     no change │
   │ QQuery 26 │         51.68 / 56.05 ±3.39 / 60.53 ms │         48.66 / 54.67 ±4.45 / 62.32 ms │     no change │
   │ QQuery 27 │     659.45 / 678.59 ±17.43 / 708.88 ms │      672.73 / 683.82 ±6.52 / 692.06 ms │     no change │
   │ QQuery 28 │  2877.34 / 2904.64 ±27.88 / 2951.69 ms │  2961.97 / 3003.72 ±74.72 / 3152.91 ms │     no change │
   │ QQuery 29 │        48.45 / 63.35 ±16.83 / 94.57 ms │         50.22 / 51.60 ±1.25 / 53.78 ms │ +1.23x faster │
   │ QQuery 30 │      341.21 / 344.90 ±3.42 / 349.32 ms │      348.60 / 352.90 ±3.60 / 359.56 ms │     no change │
   │ QQuery 31 │     404.38 / 415.38 ±10.36 / 433.78 ms │      417.65 / 433.42 ±7.96 / 439.35 ms │     no change │
   │ QQuery 32 │   1592.42 / 1601.33 ±5.36 / 1607.61 ms │  1694.04 / 1718.48 ±23.93 / 1755.49 ms │  1.07x slower │
   │ QQuery 33 │  1532.68 / 1588.53 ±32.48 / 1623.87 ms │  1571.61 / 1597.73 ±17.25 / 1620.14 ms │     no change │
   │ QQuery 34 │  1558.23 / 1580.45 ±25.50 / 1622.95 ms │  1604.66 / 1643.55 ±43.21 / 1720.54 ms │     no change │
   │ QQuery 35 │      309.73 / 322.95 ±8.56 / 334.01 ms │     310.42 / 327.25 ±16.36 / 357.31 ms │     no change │
   │ QQuery 36 │         79.81 / 82.99 ±3.62 / 89.46 ms │         79.21 / 88.43 ±6.90 / 96.60 ms │  1.07x slower │
   │ QQuery 37 │         51.02 / 57.02 ±4.53 / 64.97 ms │         47.45 / 53.38 ±6.13 / 64.34 ms │ +1.07x faster │
   │ QQuery 38 │         52.85 / 57.32 ±5.70 / 67.51 ms │         52.46 / 58.59 ±4.34 / 64.88 ms │     no change │
   │ QQuery 39 │      150.33 / 158.66 ±5.19 / 164.43 ms │      150.22 / 161.73 ±7.63 / 169.47 ms │     no change │
   │ QQuery 40 │         28.20 / 29.56 ±1.31 / 31.96 ms │         26.76 / 28.63 ±2.64 / 33.86 ms │     no change │
   │ QQuery 41 │         26.94 / 28.51 ±1.51 / 31.35 ms │         25.45 / 29.83 ±4.15 / 36.50 ms │     no change │
   │ QQuery 42 │         23.97 / 24.77 ±0.89 / 26.40 ms │         22.49 / 26.59 ±6.46 / 39.40 ms │  1.07x slower │
   └───────────┴────────────────────────────────────────┴────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 20927.17ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 21592.67ms │
   │ Average Time (HEAD)                                │   486.68ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   502.16ms │
   │ Queries Faster                                     │          3 │
   │ Queries Slower                                     │         11 │
   │ Queries with No Change                             │         29 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 110.0s |
   | Peak memory | 33.6 GiB |
   | Avg memory | 26.6 GiB |
   | CPU user | 1117.1s |
   | CPU sys | 70.1s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 110.0s |
   | Peak memory | 32.8 GiB |
   | Avg memory | 26.3 GiB |
   | CPU user | 1149.3s |
   | CPU sys | 75.5s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345545637

   run benchmark tpch
   
   ```yaml
   baseline:
       ref: main
       env:
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: true
   changed:
       ref: HEAD
       env:
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: true
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347945716

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347919711-1929-xs9r5 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: clickbench_partitioned
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348087506

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
     Downloaded dunce v1.0.5
     Downloaded anyhow v1.0.102
     Downloaded chacha20 v0.10.0
     Downloaded byteorder v1.5.0
     Downloaded aws-smithy-json v0.62.5
     Downloaded form_urlencoded v1.2.2
     Downloaded allocator-api2 v0.2.21
     Downloaded arrow-data v58.1.0
     Downloaded anstyle-query v1.1.5
     Downloaded anstyle v1.0.14
     Downloaded bzip2 v0.6.1
     Downloaded aws-smithy-query v0.60.15
     Downloaded atoi v2.0.0
     Downloaded aws-smithy-runtime-api-macros v1.0.0
     Downloaded ahash v0.8.12
     Downloaded anstream v1.0.0
       Blocking waiting for file lock on package cache
   error: no bench target named `hj` in default-run packages
   
   help: a target with a similar name exists: `chr`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345770652

   run benchmark tpch
   
   ```yaml
   baseline:
       ref: main
       env:
          DATAFUSION_EXECUTION_TARGET_PARTITIONS: 128
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: false
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: false
   changed:
       ref: HEAD
       env:
          DATAFUSION_EXECUTION_TARGET_PARTITIONS: 128
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: false
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: false
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345555819

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4345545637)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4345545637-1915-vxjfd 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (7a8272fed3de221096a523aec1c5273122c7f201) to main [diff](https://github.com/apache/datafusion/compare/66980e26df75740006229125acbf6815ee04a9f4..7a8272fed3de221096a523aec1c5273122c7f201) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345925249

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4345770652)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                               HEAD ┃ worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │     50.39 / 51.37 ±0.72 / 52.18 ms │      50.98 / 51.49 ±0.41 / 51.93 ms │     no change │
   │ QQuery 2  │     41.24 / 42.35 ±1.13 / 44.39 ms │      42.22 / 43.18 ±1.52 / 46.19 ms │     no change │
   │ QQuery 3  │ 121.16 / 137.14 ±10.30 / 149.26 ms │      80.57 / 81.85 ±1.46 / 84.12 ms │ +1.68x faster │
   │ QQuery 4  │     35.18 / 37.11 ±1.11 / 38.15 ms │      35.39 / 36.62 ±1.11 / 38.35 ms │     no change │
   │ QQuery 5  │ 202.96 / 225.56 ±12.70 / 237.39 ms │   121.17 / 124.14 ±2.37 / 127.42 ms │ +1.82x faster │
   │ QQuery 6  │     21.61 / 22.62 ±0.68 / 23.24 ms │      22.16 / 22.79 ±0.61 / 23.75 ms │     no change │
   │ QQuery 7  │  209.79 / 218.28 ±4.51 / 222.71 ms │   120.22 / 124.67 ±2.41 / 127.04 ms │ +1.75x faster │
   │ QQuery 8  │  207.20 / 212.46 ±3.23 / 216.63 ms │     95.98 / 98.63 ±1.75 / 100.98 ms │ +2.15x faster │
   │ QQuery 9  │  196.07 / 198.42 ±2.40 / 202.39 ms │   118.49 / 121.18 ±2.19 / 124.98 ms │ +1.64x faster │
   │ QQuery 10 │     83.06 / 84.09 ±0.83 / 85.33 ms │      82.62 / 84.10 ±1.05 / 85.83 ms │     no change │
   │ QQuery 11 │     40.02 / 41.99 ±1.88 / 44.95 ms │      40.26 / 41.90 ±1.06 / 43.37 ms │     no change │
   │ QQuery 12 │  113.38 / 118.16 ±4.21 / 123.64 ms │      62.57 / 65.21 ±1.62 / 67.20 ms │ +1.81x faster │
   │ QQuery 13 │  105.37 / 111.78 ±4.20 / 118.07 ms │      61.98 / 63.48 ±1.16 / 64.82 ms │ +1.76x faster │
   │ QQuery 14 │     44.86 / 45.82 ±0.69 / 46.79 ms │      46.48 / 47.54 ±1.38 / 49.97 ms │     no change │
   │ QQuery 15 │     60.09 / 62.55 ±1.75 / 64.45 ms │      62.15 / 63.99 ±1.88 / 67.34 ms │     no change │
   │ QQuery 16 │     41.66 / 43.44 ±1.00 / 44.29 ms │      41.97 / 44.42 ±1.39 / 45.76 ms │     no change │
   │ QQuery 17 │  201.30 / 202.76 ±1.35 / 204.66 ms │   125.61 / 127.81 ±2.07 / 131.26 ms │ +1.59x faster │
   │ QQuery 18 │  287.19 / 295.98 ±5.08 / 301.13 ms │   226.72 / 230.02 ±2.75 / 234.29 ms │ +1.29x faster │
   │ QQuery 19 │     43.90 / 45.02 ±0.61 / 45.65 ms │      44.09 / 44.77 ±0.51 / 45.66 ms │     no change │
   │ QQuery 20 │  127.40 / 139.73 ±6.29 / 145.14 ms │      65.72 / 68.09 ±1.50 / 69.87 ms │ +2.05x faster │
   │ QQuery 21 │  181.94 / 191.00 ±8.25 / 205.46 ms │   130.00 / 133.81 ±3.16 / 139.60 ms │ +1.43x faster │
   │ QQuery 22 │     37.94 / 38.71 ±0.62 / 39.46 ms │      34.81 / 37.13 ±1.58 / 39.13 ms │     no change │
   └───────────┴────────────────────────────────────┴─────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 2566.31ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 1756.80ms │
   │ Average Time (HEAD)                                │  116.65ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   79.85ms │
   │ Queries Faster                                     │        11 │
   │ Queries Slower                                     │         0 │
   │ Queries with No Change                             │        11 │
   │ Queries with Failure                               │         0 │
   └────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 15.0s |
   | Peak memory | 6.4 GiB |
   | Avg memory | 5.5 GiB |
   | CPU user | 107.0s |
   | CPU sys | 3.7s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 10.0s |
   | Peak memory | 6.5 GiB |
   | Avg memory | 5.6 GiB |
   | CPU user | 58.1s |
   | CPU sys | 3.9s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347957086

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347919711-1932-2ghzk 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpcds
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347952748

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347919711-1930-8c972 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347958398

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
     Downloaded dirs-sys v0.5.0
     Downloaded cpufeatures v0.2.17
     Downloaded block-buffer v0.10.4
     Downloaded atomic-waker v1.1.2
     Downloaded arrayref v0.3.9
     Downloaded anstyle-query v1.1.5
     Downloaded base64-simd v0.8.0
     Downloaded axum-core v0.5.6
     Downloaded autocfg v1.5.0
     Downloaded ciborium-io v0.2.2
     Downloaded bytes-utils v0.1.4
     Downloaded block-buffer v0.12.0
     Downloaded aws-smithy-async v1.2.14
     Downloaded async-stream-impl v0.3.6
     Downloaded cmov v0.5.3
     Downloaded aws-lc-sys v0.40.0
       Blocking waiting for file lock on package cache
   error: no bench target named `hj` in default-run packages
   
   help: a target with a similar name exists: `chr`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348087142

   🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347920499-1943-q2589 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d)
   BENCH_NAME=hj
   BENCH_COMMAND=cargo bench --features=parquet --bench hj
   BENCH_FILTER=
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348138069

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                      HEAD ┃       worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │            24.31 / 24.91 ±0.73 / 26.33 ms │            26.93 / 27.20 ±0.32 / 27.81 ms │  1.09x slower │
   │ QQuery 2  │         186.34 / 193.78 ±6.07 / 203.54 ms │         182.96 / 189.32 ±5.39 / 197.01 ms │     no change │
   │ QQuery 3  │         126.53 / 128.86 ±1.28 / 130.19 ms │         125.23 / 127.18 ±1.92 / 130.50 ms │     no change │
   │ QQuery 4  │     1439.55 / 1465.41 ±26.50 / 1499.88 ms │     1394.82 / 1436.34 ±42.07 / 1517.41 ms │     no change │
   │ QQuery 5  │        254.81 / 271.42 ±10.22 / 282.30 ms │         241.09 / 250.64 ±7.99 / 261.17 ms │ +1.08x faster │
   │ QQuery 6  │         175.15 / 185.03 ±8.77 / 197.32 ms │        168.09 / 190.00 ±19.93 / 215.35 ms │     no change │
   │ QQuery 7  │        547.51 / 564.99 ±12.10 / 577.17 ms │         510.54 / 514.86 ±3.29 / 520.71 ms │ +1.10x faster │
   │ QQuery 8  │         145.86 / 147.49 ±0.84 / 148.21 ms │         141.80 / 144.47 ±2.42 / 148.87 ms │     no change │
   │ QQuery 9  │         169.07 / 175.35 ±5.77 / 182.55 ms │         170.21 / 177.58 ±4.88 / 184.10 ms │     no change │
   │ QQuery 10 │         120.96 / 127.82 ±3.63 / 130.99 ms │         121.95 / 126.71 ±3.69 / 130.92 ms │     no change │
   │ QQuery 11 │       998.69 / 1009.89 ±8.61 / 1019.80 ms │        944.91 / 965.19 ±12.00 / 980.75 ms │     no change │
   │ QQuery 12 │            63.62 / 64.39 ±0.60 / 65.37 ms │            62.72 / 64.37 ±1.46 / 66.42 ms │     no change │
   │ QQuery 13 │         399.96 / 409.78 ±9.97 / 427.13 ms │         398.79 / 404.94 ±4.11 / 410.58 ms │     no change │
   │ QQuery 14 │     1246.70 / 1265.78 ±13.33 / 1285.34 ms │     1237.65 / 1252.11 ±10.87 / 1269.15 ms │     no change │
   │ QQuery 15 │           89.37 / 95.37 ±4.65 / 101.25 ms │            51.17 / 52.27 ±1.08 / 54.21 ms │ +1.82x faster │
   │ QQuery 16 │            22.90 / 23.54 ±0.57 / 24.30 ms │            24.97 / 25.60 ±0.63 / 26.71 ms │  1.09x slower │
   │ QQuery 17 │         517.83 / 528.73 ±8.11 / 540.81 ms │         401.39 / 413.24 ±8.52 / 423.14 ms │ +1.28x faster │
   │ QQuery 18 │         245.33 / 248.61 ±3.94 / 256.37 ms │         185.55 / 192.20 ±9.64 / 211.21 ms │ +1.29x faster │
   │ QQuery 19 │         169.78 / 175.54 ±6.44 / 186.63 ms │         170.71 / 175.04 ±3.47 / 179.81 ms │     no change │
   │ QQuery 20 │            33.48 / 34.30 ±0.88 / 35.61 ms │            34.04 / 34.79 ±0.88 / 36.29 ms │     no change │
   │ QQuery 21 │            34.88 / 35.26 ±0.35 / 35.86 ms │            37.38 / 39.15 ±1.35 / 41.56 ms │  1.11x slower │
   │ QQuery 22 │        491.17 / 509.84 ±13.04 / 525.64 ms │         494.79 / 500.73 ±3.83 / 504.80 ms │     no change │
   │ QQuery 23 │     1938.85 / 1993.92 ±36.71 / 2024.51 ms │      1774.40 / 1785.15 ±7.75 / 1796.09 ms │ +1.12x faster │
   │ QQuery 24 │     1484.18 / 1500.06 ±17.11 / 1531.07 ms │     1264.65 / 1281.00 ±11.94 / 1293.96 ms │ +1.17x faster │
   │ QQuery 25 │        661.16 / 685.06 ±12.71 / 698.51 ms │         552.91 / 561.08 ±6.08 / 571.82 ms │ +1.22x faster │
   │ QQuery 26 │         167.48 / 178.98 ±9.15 / 195.37 ms │         129.59 / 130.71 ±1.01 / 132.10 ms │ +1.37x faster │
   │ QQuery 27 │            24.81 / 25.75 ±0.63 / 26.69 ms │            27.01 / 30.30 ±4.36 / 38.92 ms │  1.18x slower │
   │ QQuery 28 │         174.57 / 178.23 ±4.18 / 186.39 ms │         169.73 / 173.34 ±3.11 / 178.75 ms │     no change │
   │ QQuery 29 │         613.11 / 623.05 ±5.45 / 628.68 ms │         486.39 / 501.07 ±8.52 / 512.40 ms │ +1.24x faster │
   │ QQuery 30 │            80.87 / 83.14 ±1.28 / 84.24 ms │            82.69 / 83.84 ±0.61 / 84.48 ms │     no change │
   │ QQuery 31 │         221.20 / 226.31 ±4.67 / 233.70 ms │         209.59 / 223.13 ±9.45 / 235.04 ms │     no change │
   │ QQuery 32 │            35.10 / 37.97 ±1.48 / 39.15 ms │            35.99 / 42.81 ±8.65 / 59.59 ms │  1.13x slower │
   │ QQuery 33 │         172.08 / 174.95 ±4.43 / 183.78 ms │         169.16 / 170.82 ±1.57 / 173.29 ms │     no change │
   │ QQuery 34 │            23.60 / 24.25 ±0.47 / 24.70 ms │            23.32 / 26.63 ±3.10 / 32.34 ms │  1.10x slower │
   │ QQuery 35 │         127.12 / 133.25 ±4.15 / 139.70 ms │         127.21 / 130.58 ±3.59 / 137.40 ms │     no change │
   │ QQuery 36 │            30.14 / 32.20 ±1.78 / 35.43 ms │            29.02 / 30.96 ±1.38 / 33.20 ms │     no change │
   │ QQuery 37 │            17.45 / 18.11 ±0.69 / 19.35 ms │            17.56 / 18.03 ±0.50 / 18.96 ms │     no change │
   │ QQuery 38 │         116.90 / 123.54 ±7.38 / 135.53 ms │         118.46 / 125.18 ±5.26 / 132.24 ms │     no change │
   │ QQuery 39 │        234.02 / 244.73 ±12.40 / 267.68 ms │         228.74 / 246.71 ±9.18 / 253.78 ms │     no change │
   │ QQuery 40 │         180.26 / 189.31 ±8.76 / 205.10 ms │        178.13 / 189.92 ±12.44 / 212.57 ms │     no change │
   │ QQuery 41 │            34.56 / 35.31 ±0.44 / 35.89 ms │            34.12 / 34.78 ±0.41 / 35.29 ms │     no change │
   │ QQuery 42 │         118.97 / 121.36 ±1.97 / 124.64 ms │         119.47 / 125.09 ±8.92 / 142.76 ms │     no change │
   │ QQuery 43 │            21.63 / 22.31 ±0.38 / 22.69 ms │            19.66 / 19.99 ±0.19 / 20.20 ms │ +1.12x faster │
   │ QQuery 44 │            52.96 / 55.22 ±2.20 / 58.90 ms │            51.45 / 54.01 ±2.10 / 57.13 ms │     no change │
   │ QQuery 45 │         140.70 / 153.27 ±6.97 / 159.81 ms │         102.06 / 105.96 ±3.56 / 111.02 ms │ +1.45x faster │
   │ QQuery 46 │            24.25 / 25.16 ±1.16 / 27.39 ms │            23.97 / 24.64 ±0.62 / 25.41 ms │     no change │
   │ QQuery 47 │        840.85 / 871.13 ±29.39 / 920.10 ms │        795.55 / 811.51 ±13.84 / 830.60 ms │ +1.07x faster │
   │ QQuery 48 │         286.81 / 296.90 ±7.52 / 306.53 ms │        285.22 / 296.64 ±10.63 / 314.15 ms │     no change │
   │ QQuery 49 │         356.79 / 362.05 ±4.23 / 365.72 ms │         353.43 / 358.44 ±2.83 / 361.31 ms │     no change │
   │ QQuery 50 │         409.77 / 416.81 ±5.38 / 423.24 ms │        330.83 / 345.76 ±11.36 / 361.97 ms │ +1.21x faster │
   │ QQuery 51 │         268.26 / 275.30 ±5.54 / 281.52 ms │         267.84 / 274.38 ±5.05 / 282.31 ms │     no change │
   │ QQuery 52 │         119.96 / 122.97 ±2.85 / 127.96 ms │         118.60 / 120.24 ±2.92 / 126.08 ms │     no change │
   │ QQuery 53 │         123.48 / 126.54 ±2.62 / 129.79 ms │         122.01 / 125.15 ±2.55 / 127.75 ms │     no change │
   │ QQuery 54 │         208.94 / 213.91 ±4.08 / 220.04 ms │         205.67 / 208.17 ±2.41 / 212.57 ms │     no change │
   │ QQuery 55 │         118.28 / 121.11 ±2.84 / 125.80 ms │         116.81 / 119.80 ±2.41 / 122.72 ms │     no change │
   │ QQuery 56 │         166.84 / 170.99 ±3.00 / 175.79 ms │         166.78 / 170.15 ±3.09 / 174.80 ms │     no change │
   │ QQuery 57 │         261.82 / 270.39 ±5.12 / 277.48 ms │         256.61 / 263.77 ±5.37 / 271.13 ms │     no change │
   │ QQuery 58 │         611.63 / 625.77 ±9.85 / 638.50 ms │         420.56 / 426.80 ±6.69 / 439.65 ms │ +1.47x faster │
   │ QQuery 59 │         230.71 / 233.61 ±1.73 / 235.63 ms │         220.98 / 229.24 ±8.87 / 246.24 ms │     no change │
   │ QQuery 60 │         173.37 / 181.35 ±7.25 / 194.54 ms │         174.05 / 177.84 ±3.53 / 184.10 ms │     no change │
   │ QQuery 61 │            25.05 / 25.69 ±0.49 / 26.34 ms │            25.22 / 26.73 ±0.91 / 27.93 ms │     no change │
   │ QQuery 62 │         905.75 / 914.73 ±8.05 / 929.66 ms │        922.22 / 938.83 ±10.11 / 952.18 ms │     no change │
   │ QQuery 63 │         128.53 / 133.20 ±5.55 / 142.80 ms │         122.01 / 126.80 ±5.05 / 133.70 ms │     no change │
   │ QQuery 64 │     1164.43 / 1209.77 ±37.08 / 1259.08 ms │      1008.14 / 1012.53 ±4.39 / 1020.51 ms │ +1.19x faster │
   │ QQuery 65 │        280.83 / 303.74 ±17.44 / 331.96 ms │        280.25 / 296.51 ±10.84 / 313.83 ms │     no change │
   │ QQuery 66 │        247.43 / 278.82 ±20.39 / 301.74 ms │        250.42 / 276.12 ±17.58 / 297.15 ms │     no change │
   │ QQuery 67 │         321.37 / 326.34 ±7.99 / 342.26 ms │        326.08 / 341.02 ±13.69 / 364.34 ms │     no change │
   │ QQuery 68 │            24.61 / 25.62 ±0.90 / 26.74 ms │            25.04 / 30.39 ±7.31 / 44.86 ms │  1.19x slower │
   │ QQuery 69 │         117.80 / 123.20 ±4.33 / 130.97 ms │         119.37 / 123.84 ±2.26 / 125.53 ms │     no change │
   │ QQuery 70 │        362.78 / 375.85 ±10.41 / 393.57 ms │        358.39 / 374.01 ±12.98 / 395.00 ms │     no change │
   │ QQuery 71 │         157.50 / 161.63 ±3.33 / 166.36 ms │         155.49 / 162.12 ±5.75 / 170.75 ms │     no change │
   │ QQuery 72 │     2866.36 / 2882.86 ±13.54 / 2901.98 ms │        878.05 / 895.11 ±14.65 / 917.97 ms │ +3.22x faster │
   │ QQuery 73 │            23.29 / 23.78 ±0.48 / 24.39 ms │           23.34 / 30.77 ±11.35 / 53.37 ms │  1.29x slower │
   │ QQuery 74 │        647.65 / 666.92 ±10.94 / 681.40 ms │        601.00 / 641.16 ±30.30 / 677.12 ms │     no change │
   │ QQuery 75 │         389.62 / 396.12 ±4.89 / 403.33 ms │         383.68 / 393.76 ±5.99 / 399.76 ms │     no change │
   │ QQuery 76 │        253.98 / 277.99 ±16.45 / 304.86 ms │         209.29 / 211.97 ±1.92 / 214.44 ms │ +1.31x faster │
   │ QQuery 77 │         248.42 / 260.06 ±7.89 / 270.50 ms │         240.73 / 249.00 ±9.95 / 268.52 ms │     no change │
   │ QQuery 78 │        528.01 / 550.48 ±12.83 / 564.13 ms │        537.25 / 566.36 ±17.51 / 591.04 ms │     no change │
   │ QQuery 79 │         250.32 / 264.37 ±8.23 / 272.65 ms │         245.84 / 256.21 ±9.95 / 271.62 ms │     no change │
   │ QQuery 80 │         522.45 / 533.00 ±8.13 / 545.07 ms │         517.87 / 528.16 ±8.19 / 541.77 ms │     no change │
   │ QQuery 81 │            63.64 / 65.95 ±1.63 / 67.89 ms │            63.91 / 69.35 ±6.08 / 79.88 ms │  1.05x slower │
   │ QQuery 82 │            53.95 / 61.14 ±7.64 / 72.75 ms │            57.92 / 58.49 ±0.41 / 58.91 ms │     no change │
   │ QQuery 83 │         171.05 / 176.23 ±3.28 / 181.00 ms │         102.73 / 107.43 ±5.50 / 118.00 ms │ +1.64x faster │
   │ QQuery 84 │            58.60 / 59.96 ±1.06 / 61.31 ms │            58.91 / 60.32 ±0.92 / 61.53 ms │     no change │
   │ QQuery 85 │         219.62 / 231.24 ±6.21 / 237.45 ms │         165.51 / 171.07 ±3.25 / 174.39 ms │ +1.35x faster │
   │ QQuery 86 │            57.27 / 58.55 ±0.86 / 59.63 ms │            54.23 / 57.53 ±1.94 / 59.93 ms │     no change │
   │ QQuery 87 │               3.68 / 3.78 ±0.16 / 4.10 ms │               3.82 / 3.94 ±0.12 / 4.17 ms │     no change │
   │ QQuery 88 │         147.82 / 151.39 ±3.51 / 158.07 ms │         143.63 / 148.88 ±5.69 / 156.01 ms │     no change │
   │ QQuery 89 │         143.50 / 149.67 ±8.79 / 167.13 ms │         139.58 / 143.88 ±4.35 / 151.44 ms │     no change │
   │ QQuery 90 │            37.79 / 40.16 ±1.62 / 42.30 ms │            35.97 / 36.74 ±0.90 / 38.33 ms │ +1.09x faster │
   │ QQuery 91 │            76.13 / 78.09 ±2.03 / 81.80 ms │            74.44 / 76.35 ±1.36 / 78.54 ms │     no change │
   │ QQuery 92 │            77.71 / 78.95 ±1.47 / 81.18 ms │            73.22 / 78.38 ±3.33 / 82.17 ms │     no change │
   │ QQuery 93 │         323.58 / 330.71 ±5.45 / 336.29 ms │         311.53 / 316.99 ±5.17 / 325.62 ms │     no change │
   │ QQuery 94 │            89.13 / 90.44 ±1.06 / 91.74 ms │            85.48 / 91.14 ±3.43 / 94.23 ms │     no change │
   │ QQuery 95 │         334.96 / 341.71 ±6.07 / 352.33 ms │         245.32 / 252.61 ±6.79 / 262.38 ms │ +1.35x faster │
   │ QQuery 96 │            80.38 / 82.41 ±1.18 / 83.41 ms │            77.64 / 80.07 ±1.84 / 81.99 ms │     no change │
   │ QQuery 97 │         173.80 / 178.99 ±4.68 / 187.71 ms │         166.26 / 172.01 ±3.95 / 177.43 ms │     no change │
   │ QQuery 98 │         183.75 / 187.97 ±4.88 / 195.98 ms │         168.74 / 174.19 ±4.94 / 180.35 ms │ +1.08x faster │
   │ QQuery 99 │ 10957.84 / 11242.15 ±203.51 / 11483.42 ms │ 10887.94 / 11070.49 ±117.96 / 11226.72 ms │     no change │
   └───────────┴───────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 42204.00ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 37998.77ms │
   │ Average Time (HEAD)                                │   426.30ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   383.83ms │
   │ Queries Faster                                     │         23 │
   │ Queries Slower                                     │          9 │
   │ Queries with No Change                             │         67 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 215.0s |
   | Peak memory | 6.4 GiB |
   | Avg memory | 5.4 GiB |
   | CPU user | 685.2s |
   | CPU sys | 26.0s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 195.0s |
   | Peak memory | 6.4 GiB |
   | Avg memory | 5.3 GiB |
   | CPU user | 445.2s |
   | CPU sys | 26.5s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471444358

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃   worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │           6.30 / 6.79 ±0.90 / 8.60 ms │           6.35 / 6.87 ±0.82 / 8.50 ms │     no change │
   │ QQuery 2  │        81.92 / 82.35 ±0.34 / 82.86 ms │        80.19 / 81.09 ±0.59 / 81.74 ms │     no change │
   │ QQuery 3  │        28.86 / 29.30 ±0.41 / 29.82 ms │        28.48 / 29.16 ±0.40 / 29.73 ms │     no change │
   │ QQuery 4  │     509.00 / 513.11 ±3.32 / 517.26 ms │     516.51 / 521.28 ±3.39 / 526.97 ms │     no change │
   │ QQuery 5  │        52.71 / 53.27 ±0.51 / 54.11 ms │        52.53 / 52.93 ±0.42 / 53.55 ms │     no change │
   │ QQuery 6  │        35.64 / 36.13 ±0.44 / 36.84 ms │        35.27 / 35.52 ±0.18 / 35.79 ms │     no change │
   │ QQuery 7  │     105.71 / 106.79 ±1.35 / 109.41 ms │     109.52 / 110.93 ±2.02 / 114.87 ms │     no change │
   │ QQuery 8  │        38.82 / 39.46 ±0.57 / 40.46 ms │        38.52 / 39.02 ±0.40 / 39.53 ms │     no change │
   │ QQuery 9  │        52.60 / 55.38 ±2.80 / 59.89 ms │        53.54 / 54.76 ±1.12 / 56.23 ms │     no change │
   │ QQuery 10 │        80.63 / 81.08 ±0.23 / 81.26 ms │        79.98 / 81.17 ±1.73 / 84.61 ms │     no change │
   │ QQuery 11 │     310.38 / 315.15 ±4.13 / 322.16 ms │     319.32 / 324.37 ±4.62 / 331.74 ms │     no change │
   │ QQuery 12 │        28.35 / 28.84 ±0.39 / 29.54 ms │        28.36 / 29.07 ±0.49 / 29.77 ms │     no change │
   │ QQuery 13 │     128.18 / 129.17 ±1.14 / 131.33 ms │     128.33 / 129.57 ±1.22 / 131.72 ms │     no change │
   │ QQuery 14 │     505.04 / 507.33 ±2.15 / 510.06 ms │     509.09 / 512.61 ±3.29 / 517.01 ms │     no change │
   │ QQuery 15 │        60.13 / 61.36 ±0.87 / 62.60 ms │        59.95 / 61.07 ±0.73 / 62.04 ms │     no change │
   │ QQuery 16 │           6.89 / 7.05 ±0.12 / 7.22 ms │           6.98 / 7.12 ±0.23 / 7.59 ms │     no change │
   │ QQuery 17 │        80.63 / 82.41 ±1.50 / 84.69 ms │        82.25 / 82.71 ±0.44 / 83.47 ms │     no change │
   │ QQuery 18 │     150.26 / 150.96 ±0.71 / 152.16 ms │     155.00 / 155.78 ±0.96 / 157.62 ms │     no change │
   │ QQuery 19 │        41.27 / 41.62 ±0.30 / 42.02 ms │        41.50 / 41.79 ±0.24 / 42.16 ms │     no change │
   │ QQuery 20 │        34.90 / 35.31 ±0.35 / 35.92 ms │        34.91 / 35.58 ±0.60 / 36.52 ms │     no change │
   │ QQuery 21 │        17.77 / 17.99 ±0.15 / 18.23 ms │        17.89 / 18.16 ±0.22 / 18.54 ms │     no change │
   │ QQuery 22 │        62.20 / 62.72 ±0.36 / 63.22 ms │        61.91 / 62.25 ±0.23 / 62.55 ms │     no change │
   │ QQuery 23 │     473.40 / 476.52 ±2.83 / 480.64 ms │     478.74 / 482.66 ±3.82 / 489.29 ms │     no change │
   │ QQuery 24 │     232.58 / 236.64 ±4.91 / 246.30 ms │     239.44 / 242.55 ±2.55 / 246.69 ms │     no change │
   │ QQuery 25 │     112.58 / 113.96 ±0.94 / 115.41 ms │     115.99 / 116.81 ±1.23 / 119.25 ms │     no change │
   │ QQuery 26 │        70.13 / 70.24 ±0.14 / 70.51 ms │        70.76 / 71.20 ±0.45 / 72.02 ms │     no change │
   │ QQuery 27 │           7.00 / 7.12 ±0.15 / 7.42 ms │           7.12 / 7.23 ±0.15 / 7.54 ms │     no change │
   │ QQuery 28 │        59.04 / 63.22 ±2.35 / 66.26 ms │        58.46 / 61.76 ±1.81 / 63.50 ms │     no change │
   │ QQuery 29 │      99.20 / 102.49 ±2.22 / 106.08 ms │      99.78 / 101.49 ±2.30 / 105.98 ms │     no change │
   │ QQuery 30 │        30.37 / 30.72 ±0.47 / 31.66 ms │        30.39 / 30.74 ±0.32 / 31.19 ms │     no change │
   │ QQuery 31 │     112.20 / 113.88 ±2.46 / 118.73 ms │     111.55 / 113.03 ±0.86 / 113.92 ms │     no change │
   │ QQuery 32 │        20.32 / 20.79 ±0.29 / 21.15 ms │        20.83 / 21.12 ±0.23 / 21.52 ms │     no change │
   │ QQuery 33 │        39.38 / 39.57 ±0.22 / 39.97 ms │        40.24 / 40.63 ±0.50 / 41.61 ms │     no change │
   │ QQuery 34 │         9.77 / 10.28 ±0.35 / 10.75 ms │        10.41 / 10.78 ±0.33 / 11.18 ms │     no change │
   │ QQuery 35 │        80.74 / 81.35 ±0.60 / 82.17 ms │        81.78 / 83.91 ±1.63 / 85.99 ms │     no change │
   │ QQuery 36 │           6.43 / 6.56 ±0.18 / 6.90 ms │           6.57 / 6.71 ±0.19 / 7.09 ms │     no change │
   │ QQuery 37 │           7.05 / 7.28 ±0.16 / 7.54 ms │           7.33 / 7.61 ±0.27 / 8.00 ms │     no change │
   │ QQuery 38 │        68.13 / 69.32 ±0.80 / 70.22 ms │        69.09 / 69.57 ±0.50 / 70.54 ms │     no change │
   │ QQuery 39 │        97.12 / 98.38 ±0.80 / 99.32 ms │      99.46 / 100.44 ±0.98 / 102.04 ms │     no change │
   │ QQuery 40 │        22.59 / 22.97 ±0.38 / 23.62 ms │        24.03 / 24.16 ±0.16 / 24.48 ms │  1.05x slower │
   │ QQuery 41 │        13.94 / 14.12 ±0.21 / 14.50 ms │        14.05 / 14.27 ±0.18 / 14.60 ms │     no change │
   │ QQuery 42 │        24.27 / 24.60 ±0.40 / 25.36 ms │        24.33 / 24.67 ±0.26 / 25.00 ms │     no change │
   │ QQuery 43 │           5.34 / 5.43 ±0.14 / 5.71 ms │           5.47 / 5.57 ±0.12 / 5.81 ms │     no change │
   │ QQuery 44 │        10.91 / 11.07 ±0.11 / 11.19 ms │        11.15 / 11.29 ±0.10 / 11.43 ms │     no change │
   │ QQuery 45 │        39.70 / 40.42 ±0.42 / 40.89 ms │        40.64 / 41.21 ±0.31 / 41.46 ms │     no change │
   │ QQuery 46 │        13.16 / 13.40 ±0.21 / 13.78 ms │        13.70 / 13.94 ±0.13 / 14.10 ms │     no change │
   │ QQuery 47 │     228.07 / 229.99 ±1.34 / 231.96 ms │     233.53 / 237.12 ±3.70 / 244.19 ms │     no change │
   │ QQuery 48 │     103.78 / 104.31 ±0.59 / 105.33 ms │     103.79 / 106.15 ±1.84 / 108.46 ms │     no change │
   │ QQuery 49 │        80.93 / 81.24 ±0.31 / 81.81 ms │        82.00 / 82.78 ±0.42 / 83.30 ms │     no change │
   │ QQuery 50 │        60.03 / 60.64 ±0.56 / 61.50 ms │        61.61 / 64.86 ±3.42 / 70.08 ms │  1.07x slower │
   │ QQuery 51 │        91.91 / 94.46 ±2.31 / 98.05 ms │        94.14 / 94.84 ±0.46 / 95.33 ms │     no change │
   │ QQuery 52 │        24.14 / 24.64 ±0.37 / 25.12 ms │        24.84 / 25.20 ±0.49 / 26.17 ms │     no change │
   │ QQuery 53 │        30.44 / 30.87 ±0.38 / 31.55 ms │        31.14 / 32.32 ±1.59 / 35.43 ms │     no change │
   │ QQuery 54 │        53.96 / 54.70 ±0.47 / 55.32 ms │        54.03 / 55.80 ±1.87 / 59.38 ms │     no change │
   │ QQuery 55 │        23.87 / 25.00 ±1.89 / 28.75 ms │        23.84 / 24.16 ±0.26 / 24.61 ms │     no change │
   │ QQuery 56 │        39.92 / 40.08 ±0.15 / 40.36 ms │        39.84 / 40.16 ±0.23 / 40.47 ms │     no change │
   │ QQuery 57 │     175.91 / 177.86 ±1.53 / 180.07 ms │     179.18 / 181.60 ±2.01 / 185.03 ms │     no change │
   │ QQuery 58 │     115.95 / 116.40 ±0.35 / 116.87 ms │     115.92 / 116.32 ±0.23 / 116.53 ms │     no change │
   │ QQuery 59 │     118.44 / 119.57 ±1.00 / 121.43 ms │     117.86 / 118.54 ±0.71 / 119.42 ms │     no change │
   │ QQuery 60 │        40.31 / 40.72 ±0.23 / 40.95 ms │        40.42 / 40.56 ±0.25 / 41.06 ms │     no change │
   │ QQuery 61 │        14.08 / 14.18 ±0.11 / 14.37 ms │        13.86 / 13.98 ±0.17 / 14.31 ms │     no change │
   │ QQuery 62 │        46.43 / 47.25 ±0.98 / 49.03 ms │        47.21 / 48.23 ±0.91 / 49.54 ms │     no change │
   │ QQuery 63 │        30.50 / 31.06 ±0.71 / 32.41 ms │        30.55 / 30.79 ±0.19 / 30.99 ms │     no change │
   │ QQuery 64 │     452.89 / 456.71 ±4.36 / 462.42 ms │     467.57 / 473.83 ±4.94 / 481.12 ms │     no change │
   │ QQuery 65 │     143.68 / 145.94 ±2.73 / 151.29 ms │        69.05 / 70.35 ±1.24 / 72.68 ms │ +2.07x faster │
   │ QQuery 66 │        83.39 / 85.64 ±3.67 / 92.96 ms │        83.70 / 85.69 ±2.71 / 91.02 ms │     no change │
   │ QQuery 67 │     242.37 / 246.31 ±2.83 / 250.82 ms │     241.24 / 246.82 ±3.64 / 250.64 ms │     no change │
   │ QQuery 68 │        13.94 / 14.04 ±0.13 / 14.30 ms │        13.65 / 13.92 ±0.28 / 14.45 ms │     no change │
   │ QQuery 69 │        77.12 / 78.55 ±2.44 / 83.41 ms │        76.18 / 76.40 ±0.34 / 77.08 ms │     no change │
   │ QQuery 70 │     104.34 / 111.07 ±8.99 / 128.31 ms │     106.65 / 110.51 ±4.46 / 118.92 ms │     no change │
   │ QQuery 71 │        35.93 / 36.18 ±0.17 / 36.45 ms │        35.86 / 36.39 ±0.56 / 37.08 ms │     no change │
   │ QQuery 72 │ 2204.00 / 2278.47 ±69.05 / 2402.16 ms │ 2178.87 / 2244.60 ±55.46 / 2347.51 ms │     no change │
   │ QQuery 73 │          9.57 / 9.82 ±0.27 / 10.25 ms │          9.49 / 9.71 ±0.28 / 10.25 ms │     no change │
   │ QQuery 74 │     177.67 / 180.81 ±2.35 / 184.72 ms │     176.67 / 180.18 ±2.45 / 182.98 ms │     no change │
   │ QQuery 75 │     148.45 / 149.05 ±0.83 / 150.68 ms │     149.21 / 149.92 ±0.56 / 150.83 ms │     no change │
   │ QQuery 76 │        35.70 / 36.95 ±2.08 / 41.11 ms │        36.04 / 36.16 ±0.09 / 36.27 ms │     no change │
   │ QQuery 77 │        62.60 / 64.73 ±3.72 / 72.16 ms │        61.51 / 62.91 ±1.90 / 66.66 ms │     no change │
   │ QQuery 78 │     189.09 / 189.56 ±0.42 / 190.16 ms │     194.46 / 198.18 ±2.52 / 200.81 ms │     no change │
   │ QQuery 79 │        67.33 / 68.67 ±1.29 / 70.84 ms │        67.17 / 67.88 ±0.59 / 68.97 ms │     no change │
   │ QQuery 80 │     101.78 / 105.34 ±3.99 / 112.45 ms │     104.15 / 105.24 ±0.81 / 106.57 ms │     no change │
   │ QQuery 81 │        24.70 / 24.95 ±0.13 / 25.06 ms │        24.25 / 25.77 ±2.62 / 31.00 ms │     no change │
   │ QQuery 82 │        16.85 / 16.89 ±0.05 / 16.97 ms │        16.44 / 17.05 ±0.37 / 17.46 ms │     no change │
   │ QQuery 83 │        37.29 / 37.74 ±0.36 / 38.39 ms │        36.21 / 36.75 ±0.32 / 37.15 ms │     no change │
   │ QQuery 84 │        43.27 / 43.77 ±0.31 / 44.14 ms │        42.90 / 43.41 ±0.47 / 44.18 ms │     no change │
   │ QQuery 85 │     133.02 / 134.86 ±2.48 / 139.71 ms │     134.18 / 136.85 ±2.35 / 140.62 ms │     no change │
   │ QQuery 86 │        25.03 / 25.36 ±0.30 / 25.86 ms │        25.36 / 25.59 ±0.35 / 26.29 ms │     no change │
   │ QQuery 87 │        68.78 / 69.84 ±0.89 / 71.04 ms │        68.61 / 69.30 ±0.36 / 69.60 ms │     no change │
   │ QQuery 88 │        64.78 / 65.18 ±0.41 / 65.92 ms │        64.85 / 65.52 ±0.52 / 66.16 ms │     no change │
   │ QQuery 89 │        36.29 / 36.71 ±0.34 / 37.09 ms │        36.32 / 37.01 ±0.51 / 37.61 ms │     no change │
   │ QQuery 90 │        17.40 / 17.71 ±0.18 / 17.92 ms │        17.39 / 17.65 ±0.20 / 17.96 ms │     no change │
   │ QQuery 91 │        51.84 / 52.30 ±0.32 / 52.73 ms │        51.89 / 52.15 ±0.19 / 52.42 ms │     no change │
   │ QQuery 92 │        29.82 / 30.63 ±0.96 / 32.50 ms │        28.80 / 29.11 ±0.20 / 29.39 ms │     no change │
   │ QQuery 93 │        51.19 / 52.51 ±1.35 / 55.07 ms │        52.01 / 52.71 ±0.52 / 53.50 ms │     no change │
   │ QQuery 94 │        38.03 / 38.61 ±0.70 / 39.87 ms │        37.65 / 38.55 ±1.28 / 41.02 ms │     no change │
   │ QQuery 95 │        87.62 / 89.41 ±1.39 / 91.25 ms │        88.36 / 88.93 ±0.43 / 89.67 ms │     no change │
   │ QQuery 96 │        24.54 / 24.67 ±0.17 / 25.01 ms │        24.31 / 24.53 ±0.23 / 24.91 ms │     no change │
   │ QQuery 97 │        46.86 / 46.93 ±0.10 / 47.11 ms │        46.79 / 47.36 ±0.44 / 48.10 ms │     no change │
   │ QQuery 98 │        42.06 / 42.52 ±0.26 / 42.75 ms │        42.15 / 43.14 ±0.84 / 44.67 ms │     no change │
   │ QQuery 99 │        69.96 / 70.19 ±0.17 / 70.39 ms │        70.78 / 71.10 ±0.28 / 71.57 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 10595.50ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 10580.74ms │
   │ Average Time (HEAD)                                │   107.03ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   106.88ms │
   │ Queries Faster                                     │          1 │
   │ Queries Slower                                     │          2 │
   │ Queries with No Change                             │         96 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 6.9 GiB |
   | Avg memory | 6.2 GiB |
   | CPU user | 244.8s |
   | CPU sys | 6.1s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 6.8 GiB |
   | Avg memory | 6.1 GiB |
   | CPU user | 241.2s |
   | CPU sys | 6.5s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471451359

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471353144-177-5bflm 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed) using: tpch10
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471653022

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471633434)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471633434-181-nswg9 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing main (47655fd6c9ef060d73497987e6ccb98e57196508) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..47655fd6c9ef060d73497987e6ccb98e57196508) using: tpch10
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4579618907

   hey @adriangb are you still interested in this work. I would be willing to pick up and investigate this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4425167367

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion v53.1.0 (current)
          Built [  81.298s] (current)
        Parsing datafusion v53.1.0 (current)
         Parsed [   0.034s] (current)
       Building datafusion v53.1.0 (baseline)
          Built [  81.045s] (baseline)
        Parsing datafusion v53.1.0 (baseline)
         Parsed [   0.034s] (baseline)
       Checking datafusion v53.1.0 -> v53.1.0 (no change; assume patch)
        Checked [   0.636s] 222 checks: 222 pass, 30 skip
        Summary no semver update required
       Finished [ 164.946s] datafusion
       Building datafusion-common v53.1.0 (current)
          Built [  33.222s] (current)
        Parsing datafusion-common v53.1.0 (current)
         Parsed [   0.056s] (current)
       Building datafusion-common v53.1.0 (baseline)
          Built [  32.675s] (baseline)
        Parsing datafusion-common v53.1.0 (baseline)
         Parsed [   0.056s] (baseline)
       Checking datafusion-common v53.1.0 -> v53.1.0 (no change; assume patch)
        Checked [   0.642s] 222 checks: 222 pass, 30 skip
        Summary no semver update required
       Finished [  68.116s] datafusion-common
       Building datafusion-physical-plan v53.1.0 (current)
          Built [  31.826s] (current)
        Parsing datafusion-physical-plan v53.1.0 (current)
         Parsed [   0.123s] (current)
       Building datafusion-physical-plan v53.1.0 (baseline)
          Built [  32.131s] (baseline)
        Parsing datafusion-physical-plan v53.1.0 (baseline)
         Parsed [   0.130s] (baseline)
       Checking datafusion-physical-plan v53.1.0 -> v53.1.0 (no change; assume patch)
        Checked [   0.575s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip
   
   --- failure type_marked_deprecated: #[deprecated] added on type ---
   
   Description:
   A type is now #[deprecated]. Downstream crates will get a compiler warning when using this type.
           ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/type_marked_deprecated.ron
   
   Failed in:
     Struct HashExpr in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:79
   
        Summary semver requires new minor version: 0 major and 1 minor checks failed
       Finished [  66.032s] datafusion-physical-plan
       Building datafusion-proto v53.1.0 (current)
          Built [  53.042s] (current)
        Parsing datafusion-proto v53.1.0 (current)
         Parsed [   0.134s] (current)
       Building datafusion-proto v53.1.0 (baseline)
          Built [  52.839s] (baseline)
        Parsing datafusion-proto v53.1.0 (baseline)
         Parsed [   0.134s] (baseline)
       Checking datafusion-proto v53.1.0 -> v53.1.0 (no change; assume patch)
        Checked [   1.751s] 222 checks: 222 pass, 30 skip
        Summary no semver update required
       Finished [ 110.023s] datafusion-proto
       Building datafusion-sqllogictest v53.1.0 (current)
          Built [ 131.412s] (current)
        Parsing datafusion-sqllogictest v53.1.0 (current)
         Parsed [   0.021s] (current)
       Building datafusion-sqllogictest v53.1.0 (baseline)
          Built [ 131.980s] (baseline)
        Parsing datafusion-sqllogictest v53.1.0 (baseline)
         Parsed [   0.023s] (baseline)
       Checking datafusion-sqllogictest v53.1.0 -> v53.1.0 (no change; assume patch)
        Checked [   0.091s] 222 checks: 222 pass, 30 skip
        Summary no semver update required
       Finished [ 267.714s] datafusion-sqllogictest
   ```
   
   </details>
   <!-- semver-check-comment -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345528576

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4345503296) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
     Downloaded compression-codecs v0.4.38
     Downloaded cfg_aliases v0.2.1
     Downloaded aws-credential-types v1.2.14
     Downloaded async-stream v0.3.6
     Downloaded atoi v2.0.0
     Downloaded arrow-string v58.1.0
     Downloaded anstyle v1.0.14
     Downloaded axum-core v0.5.6
     Downloaded aws-smithy-xml v0.60.15
     Downloaded aws-smithy-observability v0.2.6
     Downloaded autocfg v1.5.0
     Downloaded ahash v0.8.12
     Downloaded anstream v1.0.0
     Downloaded async-recursion v1.1.1
     Downloaded async-ffi v0.5.0
     Downloaded anes v0.1.6
       Blocking waiting for file lock on package cache
   error: no bench target named `tcph` in default-run packages
   
   help: a target with a similar name exists: `chr`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345504799

   run benchmark tcph
   
   ```yaml
   baseline:
       ref: main
       env:
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: true
   changed:
       ref: HEAD
       env:
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: true
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4364636257

   > In any case - we can always merge this now and follow up with a version of MultiMapLookupExpr that uses https://github.com/apache/datafusion/pull/21900 if perf looks good?
   
   Sounds good


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on code in PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#discussion_r3188067905


##########
datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs:
##########
@@ -202,6 +195,64 @@ fn combine_membership_and_bounds(
     }
 }
 
+/// Compute the global (envelope) min/max bounds across a set of partition bounds.
+///
+/// For each column index, returns the smallest min seen and the largest max seen.
+/// Columns where any partition is missing bounds, or where bounds are not totally
+/// ordered (e.g. mixed-type comparisons), are dropped from the global envelope.
+fn compute_global_bounds(per_partition: &[&PartitionBounds]) -> Option<PartitionBounds> {
+    let mut iter = per_partition.iter();
+    let first = iter.next()?;
+    let mut acc: Vec<Option<ColumnBounds>> = first
+        .column_bounds
+        .iter()
+        .map(|cb| Some(cb.clone()))
+        .collect();
+
+    for partition in iter {
+        if partition.column_bounds.len() != acc.len() {
+            return None;
+        }
+        for (slot, cb) in acc.iter_mut().zip(partition.column_bounds.iter()) {
+            let Some(existing) = slot.as_mut() else {
+                continue;
+            };
+            match cb.min.partial_cmp(&existing.min) {
+                Some(std::cmp::Ordering::Less) => existing.min = cb.min.clone(),
+                Some(_) => {}
+                None => {
+                    *slot = None;
+                    continue;
+                }
+            }
+            match cb.max.partial_cmp(&existing.max) {
+                Some(std::cmp::Ordering::Greater) => existing.max = cb.max.clone(),
+                Some(_) => {}
+                None => *slot = None,
+            }
+        }
+    }
+
+    let merged: Vec<ColumnBounds> = acc.into_iter().flatten().collect();

Review Comment:
   I believe this can cause an issue here. If we insert `None` when bounds can't be computed for a column, then call `flatten()`, this will remove that `None` slot in the Vec.
   
   Thus the slot of a col that was not computed correctly would cause other slots to shift and get incrrect bounds when we do `create_bounds_predicate()`



##########
datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:
##########
@@ -343,6 +343,197 @@ impl PhysicalExpr for HashTableLookupExpr {
     }
 }
 
+/// Physical expression that probes the same join keys against multiple [`Map`]s
+/// and returns `true` for any row whose join keys match at least one map.
+///
+/// Equivalent to `OR`-ing several [`HashTableLookupExpr`]s, but
+/// `create_hashes` runs exactly once for the whole batch and every
+/// [`Map::HashMap`] probe shares the same hash buffer. All `HashMap`
+/// entries must therefore have been built with the same `RandomState`;
+/// [`Map::ArrayMap`] entries are queried via `contain_keys` and do not
+/// consume hashes.
+pub struct MultiMapLookupExpr {
+    /// Join-key expressions evaluated against each input batch.
+    on_columns: Vec<PhysicalExprRef>,
+    /// Hashing seed shared by every entry in `maps`.
+    random_state: SeededRandomState,
+    /// Build-side maps to OR over, one per partition.
+    maps: Vec<Arc<Map>>,

Review Comment:
   I noticed we aren't handling this is in the proto like we do with `HashTableLookupExpr`. For that we are handling it in `serialize_physical_expr_with_converter()` in `to_proto.rs` by passing `lit(true)` for now.



##########
datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs:
##########
@@ -649,47 +700,157 @@ impl SharedBuildAccumulator {
                     }
                 }
 
-                let filter_expr = if has_canceled_unknown {
-                    let mut when_then_branches = empty_partition_ids
-                        .into_iter()
-                        .map(|partition_id| {
-                            (
-                                lit(ScalarValue::UInt64(Some(partition_id as u64))),
-                                lit(false),
-                            )
-                        })
-                        .collect::<Vec<_>>();
-                    when_then_branches.extend(real_branches);
-
-                    if when_then_branches.is_empty() {
-                        lit(true)
-                    } else {
-                        Arc::new(CaseExpr::try_new(
-                            Some(modulo_expr),
-                            when_then_branches,
-                            Some(lit(true)),
-                        )?) as Arc<dyn PhysicalExpr>
-                    }
-                } else if real_branches.is_empty() {
-                    lit(false)
-                } else if real_branches.len() == 1
-                    && empty_partition_ids.len() + 1 == num_partitions
-                {
-                    Arc::clone(&real_branches[0].1)
-                } else {
-                    Arc::new(CaseExpr::try_new(
-                        Some(modulo_expr),
-                        real_branches,
-                        Some(lit(false)),
-                    )?) as Arc<dyn PhysicalExpr>
-                };
-
+                let filter_expr = self
+                    .build_partitioned_filter(&real_partitions, has_canceled_unknown)?;
                 self.dynamic_filter.update(filter_expr)?;
             }
         }
 
         Ok(())
     }
+
+    /// CollectLeft has a single shared build side, so we always have one
+    /// `Map`. We prefer the InList expression when it's available (the build
+    /// side fit under the InList caps) because it's directly representable in
+    /// parquet stats / bloom-filter pruning at the scan; otherwise fall back
+    /// to a single `hash_lookup` against the map.
+    fn collect_left_membership(
+        &self,
+        pushdown: &PushdownStrategy,
+    ) -> Result<Option<Arc<dyn PhysicalExpr>>> {
+        if let Some(arr) = &pushdown.inlist {
+            return Ok(Some(create_inlist_predicate(
+                &self.on_right,
+                Arc::clone(arr),
+                self.probe_schema.as_ref(),
+            )?));
+        }
+        Ok(pushdown.map.as_ref().map(|map| {
+            create_hash_lookup_predicate(&self.on_right, Arc::clone(map), &HASH_JOIN_SEED)
+        }))
+    }
+
+    /// Build the dynamic filter for `PartitionMode::Partitioned`. Emits
+    /// `global_minmax AND ([merged_in_list AND] multi_hash_lookup)` —
+    /// independent of how the build side was repartitioned.
+    ///
+    /// * `global_minmax` — envelope of every partition's per-column min/max.
+    ///   Cheap short-circuit and the only piece visible to scan-level
+    ///   `pruning_predicate` extraction.
+    /// * `merged_in_list` — concatenated, deduplicated build keys when every
+    ///   reported partition contributed an `InList` array and the
+    ///   cross-partition union fits under
+    ///   `optimizer.hash_join_inlist_pushdown_max_distinct_values`. A small
+    ///   `IN (SET)` participates in parquet stats / bloom-filter pruning,
+    ///   which `multi_hash_lookup` does not. When present it fully replaces
+    ///   the lookup.
+    /// * `multi_hash_lookup` — hashes the join keys once and ORs
+    ///   `contain_hashes()` across every partition's hash table.
+    ///
+    /// `has_canceled_unknown` partitions short-circuit to `lit(true)`: we
+    /// don't have their maps, so we cannot include them in the lookup, and
+    /// the query is being torn down anyway.
+    fn build_partitioned_filter(
+        &self,
+        real_partitions: &[&PartitionData],
+        has_canceled_unknown: bool,
+    ) -> Result<Arc<dyn PhysicalExpr>> {
+        if has_canceled_unknown {
+            return Ok(lit(true));
+        }
+        if real_partitions.is_empty() {
+            return Ok(lit(false));
+        }
+
+        let bounds_refs: Vec<&PartitionBounds> =
+            real_partitions.iter().map(|p| &p.bounds).collect();
+        let global_bounds_expr = compute_global_bounds(&bounds_refs)
+            .as_ref()
+            .and_then(|b| create_bounds_predicate(&self.on_right, b));
+
+        // The merged InList covers the union of every partition's
+        // build-side keys, so when it fires it stands alone — there is no
+        // need to also AND a `multi_hash_lookup` (which would just probe
+        // the same data via a different structure).
+        let membership_expr =
+            if let Some(merged) = self.try_build_merged_inlist(real_partitions)? {
+                Some(merged)
+            } else {
+                let maps: Vec<Arc<Map>> = real_partitions
+                    .iter()
+                    .filter_map(|p| p.pushdown.map.clone())
+                    .collect();
+                if maps.is_empty() {
+                    // Defensive: every reported (non-empty) partition is
+                    // supposed to carry a Map. Falling through to None means
+                    // we degrade to bounds-only filtering.
+                    None
+                } else {
+                    Some(Arc::new(MultiMapLookupExpr::new(
+                        self.on_right.clone(),
+                        HASH_JOIN_SEED.clone(),
+                        maps,
+                        "multi_hash_lookup".to_string(),
+                    )) as Arc<dyn PhysicalExpr>)
+                }
+            };
+
+        Ok(
+            combine_membership_and_bounds(membership_expr, global_bounds_expr)
+                .unwrap_or_else(|| lit(true)),
+        )
+    }
+
+    /// If every reported partition contributed an InList array, concatenate
+    /// them, deduplicate by scalar value, and gate on the
+    /// `inlist_max_distinct_values` cap. Returns the merged
+    /// `(struct(...))? IN (SET) ([…])` predicate built over the
+    /// deduplicated keys when the cap is satisfied; `None` otherwise.
+    ///
+    /// Per-partition arrays carry duplicates — each partition ships its raw
+    /// build-side join keys, dedup happens here. The dedup walk early-aborts
+    /// the moment we cross the cap, so the cost stays bounded by
+    /// `O(rows-until-cap+1-distinct-found)` rather than total input size.
+    fn try_build_merged_inlist(
+        &self,
+        real_partitions: &[&PartitionData],
+    ) -> Result<Option<Arc<dyn PhysicalExpr>>> {
+        let cap = self.inlist_max_distinct_values;
+        let mut arrays: Vec<ArrayRef> = Vec::with_capacity(real_partitions.len());
+        for p in real_partitions {
+            let Some(arr) = &p.pushdown.inlist else {
+                return Ok(None);
+            };
+            arrays.push(Arc::clone(arr));
+        }
+        let Some(merged) = merge_inlist_arrays(&arrays) else {

Review Comment:
   `merge_inlist_arrays()` concats which is allocating one large array with each partition `InList` value.
   
   Like:
   ```text
   p0: 20 vals
   p1: 20 vals
   ...
   pN: 20 vals
   ```
   
   So I don't think we are necessarily aborting at the cap as intended.
   
   I think if we traverse each partition array and track distinct values, then abort right when we hit the cap, then build the deduped array, that would have less allocation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471360455

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471352777-170-2whjd 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed) using: clickbench_partitioned
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471633434

   run benchmarks tpch10
   
   ```yaml
   baseline:
     ref: main
     env:
       DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "false"
       DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: "false"
   changed:
     ref: main
     env:
       DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
       DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: "true"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471637511

   control: https://github.com/apache/datafusion/pull/21931#issuecomment-4471633434
   test: https://github.com/apache/datafusion/pull/21931#issuecomment-4471617137
   
   the goal is to see if this change makes turning pushdown on closer to good or not


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471834179

   Hmm those SF10 results aren’t great…


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471717183

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471617137)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpch_sf10.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                               HEAD ┃   worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │  317.41 / 318.52 ±0.92 / 319.98 ms │     322.02 / 323.31 ±1.32 / 325.85 ms │     no change │
   │ QQuery 2  │  120.53 / 122.62 ±1.83 / 125.52 ms │     231.77 / 234.15 ±2.01 / 237.76 ms │  1.91x slower │
   │ QQuery 3  │  261.71 / 265.64 ±3.06 / 270.59 ms │     430.75 / 435.88 ±5.94 / 444.17 ms │  1.64x slower │
   │ QQuery 4  │  128.41 / 128.94 ±0.74 / 130.36 ms │     413.86 / 418.21 ±4.00 / 423.52 ms │  3.24x slower │
   │ QQuery 5  │  370.16 / 375.77 ±7.15 / 388.61 ms │     812.65 / 828.15 ±8.75 / 838.55 ms │  2.20x slower │
   │ QQuery 6  │  129.11 / 131.29 ±1.76 / 134.07 ms │     297.76 / 298.98 ±1.50 / 301.92 ms │  2.28x slower │
   │ QQuery 7  │  493.16 / 496.43 ±3.59 / 501.97 ms │     501.97 / 507.40 ±3.51 / 511.83 ms │     no change │
   │ QQuery 8  │  406.39 / 414.95 ±6.23 / 425.13 ms │     705.35 / 711.18 ±4.60 / 717.68 ms │  1.71x slower │
   │ QQuery 9  │  575.78 / 578.45 ±3.38 / 585.04 ms │ 1804.40 / 1818.76 ±14.40 / 1844.20 ms │  3.14x slower │
   │ QQuery 10 │  309.58 / 315.72 ±6.48 / 328.09 ms │     442.99 / 451.15 ±5.92 / 458.21 ms │  1.43x slower │
   │ QQuery 11 │    91.03 / 97.27 ±5.12 / 106.25 ms │     142.45 / 144.00 ±1.08 / 145.79 ms │  1.48x slower │
   │ QQuery 12 │  183.94 / 188.76 ±5.17 / 198.68 ms │     392.15 / 397.05 ±6.68 / 410.08 ms │  2.10x slower │
   │ QQuery 13 │  298.57 / 308.47 ±6.91 / 316.08 ms │    423.92 / 447.37 ±16.72 / 468.93 ms │  1.45x slower │
   │ QQuery 14 │  180.65 / 182.01 ±1.44 / 184.70 ms │     205.84 / 210.40 ±7.28 / 224.88 ms │  1.16x slower │
   │ QQuery 15 │  323.82 / 327.02 ±3.37 / 331.68 ms │     402.97 / 410.96 ±5.14 / 417.49 ms │  1.26x slower │
   │ QQuery 16 │     72.91 / 75.91 ±1.78 / 78.44 ms │     115.93 / 117.82 ±1.36 / 119.31 ms │  1.55x slower │
   │ QQuery 17 │ 666.37 / 680.16 ±11.66 / 694.40 ms │     668.92 / 674.00 ±5.82 / 685.15 ms │     no change │
   │ QQuery 18 │  691.55 / 704.23 ±9.34 / 720.54 ms │     633.15 / 640.51 ±4.87 / 648.18 ms │ +1.10x faster │
   │ QQuery 19 │ 256.09 / 276.49 ±15.91 / 296.16 ms │     289.66 / 292.17 ±2.65 / 297.31 ms │  1.06x slower │
   │ QQuery 20 │  292.17 / 299.60 ±6.33 / 309.52 ms │     310.60 / 312.11 ±1.83 / 315.69 ms │     no change │
   │ QQuery 21 │  710.24 / 712.33 ±1.76 / 714.82 ms │  1417.88 / 1420.43 ±2.42 / 1424.25 ms │  1.99x slower │
   │ QQuery 22 │     64.13 / 67.51 ±3.45 / 73.68 ms │     122.53 / 131.15 ±5.13 / 135.54 ms │  1.94x slower │
   └───────────┴────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │  7068.09ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 11225.15ms │
   │ Average Time (HEAD)                                │   321.28ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   510.23ms │
   │ Queries Faster                                     │          1 │
   │ Queries Slower                                     │         17 │
   │ Queries with No Change                             │          4 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch10 — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 40.0s |
   | Peak memory | 12.4 GiB |
   | Avg memory | 8.4 GiB |
   | CPU user | 375.6s |
   | CPU sys | 21.8s |
   | Peak spill | 0 B |
   
   **tpch10 — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 60.0s |
   | Peak memory | 10.0 GiB |
   | Avg memory | 8.0 GiB |
   | CPU user | 617.4s |
   | CPU sys | 20.4s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347936688

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347919215-1926-bsrmc 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpch10
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348042882

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃   worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.18 / 4.71 ±6.91 / 18.53 ms │          1.28 / 4.85 ±6.95 / 18.75 ms │     no change │
   │ QQuery 1  │        12.93 / 12.99 ±0.05 / 13.06 ms │        13.11 / 13.30 ±0.17 / 13.59 ms │     no change │
   │ QQuery 2  │        37.75 / 38.06 ±0.26 / 38.37 ms │        36.55 / 37.05 ±0.26 / 37.26 ms │     no change │
   │ QQuery 3  │        32.81 / 33.19 ±0.29 / 33.59 ms │        31.86 / 32.39 ±0.30 / 32.69 ms │     no change │
   │ QQuery 4  │     248.08 / 259.77 ±9.20 / 269.28 ms │     245.43 / 255.16 ±6.66 / 262.37 ms │     no change │
   │ QQuery 5  │    288.22 / 297.83 ±11.55 / 318.71 ms │    284.36 / 294.18 ±10.34 / 311.77 ms │     no change │
   │ QQuery 6  │           6.16 / 6.23 ±0.07 / 6.32 ms │           5.47 / 6.02 ±0.45 / 6.75 ms │     no change │
   │ QQuery 7  │        16.62 / 16.74 ±0.07 / 16.84 ms │        16.36 / 16.43 ±0.05 / 16.51 ms │     no change │
   │ QQuery 8  │     340.24 / 350.02 ±6.21 / 359.64 ms │    332.97 / 343.69 ±10.95 / 359.33 ms │     no change │
   │ QQuery 9  │    456.81 / 479.85 ±16.91 / 504.41 ms │    449.43 / 470.62 ±21.87 / 498.34 ms │     no change │
   │ QQuery 10 │      96.88 / 102.03 ±2.78 / 104.85 ms │      97.81 / 104.22 ±5.07 / 111.71 ms │     no change │
   │ QQuery 11 │     107.57 / 108.70 ±0.99 / 110.53 ms │     109.01 / 109.78 ±0.64 / 110.60 ms │     no change │
   │ QQuery 12 │    313.14 / 333.39 ±13.99 / 353.31 ms │    310.66 / 321.50 ±10.99 / 339.08 ms │     no change │
   │ QQuery 13 │    443.88 / 465.98 ±15.98 / 486.96 ms │    436.15 / 457.38 ±22.29 / 485.54 ms │     no change │
   │ QQuery 14 │     338.90 / 345.05 ±3.18 / 347.78 ms │    330.47 / 342.85 ±10.62 / 355.63 ms │     no change │
   │ QQuery 15 │     292.53 / 295.45 ±3.19 / 301.63 ms │    298.62 / 317.47 ±13.46 / 335.51 ms │  1.07x slower │
   │ QQuery 16 │    630.12 / 677.64 ±29.09 / 708.18 ms │    647.67 / 668.91 ±11.63 / 682.20 ms │     no change │
   │ QQuery 17 │    631.12 / 668.59 ±21.54 / 689.51 ms │    627.85 / 668.97 ±26.37 / 710.84 ms │     no change │
   │ QQuery 18 │ 1289.52 / 1319.10 ±17.50 / 1337.54 ms │ 1290.91 / 1316.38 ±21.61 / 1340.28 ms │     no change │
   │ QQuery 19 │        30.67 / 35.40 ±7.02 / 49.02 ms │        32.25 / 34.82 ±4.10 / 42.97 ms │     no change │
   │ QQuery 20 │     524.25 / 532.66 ±8.22 / 542.81 ms │    520.65 / 534.95 ±12.10 / 553.45 ms │     no change │
   │ QQuery 21 │     578.96 / 589.49 ±9.97 / 605.85 ms │    579.24 / 596.66 ±14.65 / 620.04 ms │     no change │
   │ QQuery 22 │    936.25 / 947.29 ±11.44 / 968.44 ms │   948.72 / 964.30 ±21.23 / 1005.46 ms │     no change │
   │ QQuery 23 │     114.64 / 121.96 ±4.95 / 130.13 ms │    116.11 / 126.49 ±10.17 / 144.71 ms │     no change │
   │ QQuery 24 │        41.46 / 45.98 ±6.48 / 58.59 ms │        43.07 / 48.23 ±8.97 / 66.05 ms │     no change │
   │ QQuery 25 │     148.61 / 150.74 ±1.63 / 153.20 ms │     148.75 / 152.73 ±2.17 / 154.92 ms │     no change │
   │ QQuery 26 │        61.68 / 63.03 ±1.10 / 64.87 ms │        62.78 / 65.08 ±2.27 / 69.02 ms │     no change │
   │ QQuery 27 │    726.21 / 741.80 ±12.87 / 762.62 ms │    725.54 / 749.12 ±16.06 / 771.17 ms │     no change │
   │ QQuery 28 │ 3077.37 / 3121.85 ±36.79 / 3164.20 ms │ 3106.95 / 3118.29 ±13.99 / 3145.86 ms │     no change │
   │ QQuery 29 │        42.48 / 43.78 ±2.10 / 47.97 ms │        42.46 / 42.83 ±0.36 / 43.40 ms │     no change │
   │ QQuery 30 │    322.29 / 336.96 ±10.65 / 352.37 ms │     324.13 / 336.97 ±9.96 / 345.90 ms │     no change │
   │ QQuery 31 │     318.90 / 328.72 ±7.75 / 339.11 ms │    325.58 / 338.58 ±12.80 / 360.73 ms │     no change │
   │ QQuery 32 │  1058.34 / 1067.62 ±8.03 / 1081.12 ms │ 1042.99 / 1053.56 ±13.23 / 1079.36 ms │     no change │
   │ QQuery 33 │ 1455.44 / 1503.45 ±40.06 / 1560.37 ms │ 1464.90 / 1524.56 ±37.61 / 1579.46 ms │     no change │
   │ QQuery 34 │ 1542.33 / 1591.49 ±36.06 / 1624.86 ms │ 1497.86 / 1562.64 ±39.75 / 1618.67 ms │     no change │
   │ QQuery 35 │    297.30 / 327.15 ±24.09 / 369.41 ms │    291.66 / 304.45 ±20.10 / 344.33 ms │ +1.07x faster │
   │ QQuery 36 │        65.31 / 69.25 ±4.76 / 78.52 ms │      65.74 / 77.14 ±13.34 / 101.54 ms │  1.11x slower │
   │ QQuery 37 │        38.42 / 42.11 ±3.96 / 49.55 ms │        38.33 / 42.02 ±3.36 / 47.52 ms │     no change │
   │ QQuery 38 │        35.49 / 39.38 ±4.90 / 48.81 ms │        35.81 / 38.41 ±1.61 / 40.20 ms │     no change │
   │ QQuery 39 │     118.51 / 127.45 ±5.99 / 135.47 ms │     131.91 / 137.80 ±6.53 / 149.49 ms │  1.08x slower │
   │ QQuery 40 │        19.59 / 21.29 ±2.68 / 26.60 ms │        19.57 / 19.91 ±0.23 / 20.27 ms │ +1.07x faster │
   │ QQuery 41 │        17.37 / 17.68 ±0.30 / 18.24 ms │        17.79 / 18.03 ±0.19 / 18.31 ms │     no change │
   │ QQuery 42 │        14.61 / 14.81 ±0.13 / 14.93 ms │        15.74 / 18.04 ±4.03 / 26.07 ms │  1.22x slower │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 17696.66ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 17686.75ms │
   │ Average Time (HEAD)                                │   411.55ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   411.32ms │
   │ Queries Faster                                     │          2 │
   │ Queries Slower                                     │          4 │
   │ Queries with No Change                             │         37 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 90.0s |
   | Peak memory | 30.2 GiB |
   | Avg memory | 22.8 GiB |
   | CPU user | 932.1s |
   | CPU sys | 61.3s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 90.0s |
   | Peak memory | 29.7 GiB |
   | Avg memory | 23.1 GiB |
   | CPU user | 929.3s |
   | CPU sys | 60.8s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348067398

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347920152-1934-xw5cp 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: clickbench_partitioned
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348087664

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347920499-1941-hx962 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpch10
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348144610

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
    Downloading crates ...
     Downloaded prost v0.14.3
     Downloaded prost-derive v0.14.3
     Downloaded anyhow v1.0.102
      Compiling anyhow v1.0.102
      Compiling either v1.15.0
      Compiling datafusion-benchmarks v53.1.0 (/workspace/datafusion-base/benchmarks)
      Compiling itertools v0.14.0
      Compiling prost-derive v0.14.3
      Compiling prost v0.14.3
      Compiling datafusion-proto-common v53.1.0 (/workspace/datafusion-base/datafusion/proto-common)
      Compiling datafusion-proto v53.1.0 (/workspace/datafusion-base/datafusion/proto)
       Finished `bench` profile [optimized + debuginfo] target(s) in 8m 43s
        Running benches/sql.rs (/workspace/datafusion-base/target/release/deps/sql-2c9f51cbfcaf27f8)
   Gnuplot not found, using plotters backend
   
   thread 'main' (21027) panicked at benchmarks/benches/sql.rs:131:46:
   assertion failed: Execution("Error in result on row 1, column 1 running query \"\nSELECT COUNT(*) > 0 from lineitem;\": expected value \"true\" but got value \"false\" in row: [\"false\"]")
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   error: bench failed, to rerun pass `--bench sql`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345537714

   🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4345504799)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4345504799-1913-ftp4k 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (7a8272fed3de221096a523aec1c5273122c7f201) to main [diff](https://github.com/apache/datafusion/compare/66980e26df75740006229125acbf6815ee04a9f4..7a8272fed3de221096a523aec1c5273122c7f201)
   BENCH_NAME=tcph
   BENCH_COMMAND=cargo bench --features=parquet --bench tcph
   BENCH_FILTER=
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347937393

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347919215-1927-zcqns 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpcds
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348147809

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
    Downloading crates ...
     Downloaded anyhow v1.0.102
     Downloaded prost-derive v0.14.3
     Downloaded prost v0.14.3
      Compiling anyhow v1.0.102
      Compiling either v1.15.0
      Compiling datafusion-benchmarks v53.1.0 (/workspace/datafusion-base/benchmarks)
      Compiling itertools v0.14.0
      Compiling prost-derive v0.14.3
      Compiling prost v0.14.3
      Compiling datafusion-proto-common v53.1.0 (/workspace/datafusion-base/datafusion/proto-common)
      Compiling datafusion-proto v53.1.0 (/workspace/datafusion-base/datafusion/proto)
       Finished `bench` profile [optimized + debuginfo] target(s) in 8m 57s
        Running benches/sql.rs (/workspace/datafusion-base/target/release/deps/sql-2c9f51cbfcaf27f8)
   Gnuplot not found, using plotters backend
   
   thread 'main' (21020) panicked at benchmarks/benches/sql.rs:131:46:
   assertion failed: Execution("Error in result on row 1, column 1 running query \"\nSELECT COUNT(*) > 0 from lineitem;\": expected value \"true\" but got value \"false\" in row: [\"false\"]")
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   error: bench failed, to rerun pass `--bench sql`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348162673

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
    Downloading crates ...
     Downloaded prost v0.14.3
     Downloaded prost-derive v0.14.3
     Downloaded anyhow v1.0.102
      Compiling anyhow v1.0.102
      Compiling either v1.15.0
      Compiling datafusion-benchmarks v53.1.0 (/workspace/datafusion-base/benchmarks)
      Compiling itertools v0.14.0
      Compiling prost-derive v0.14.3
      Compiling prost v0.14.3
      Compiling datafusion-proto-common v53.1.0 (/workspace/datafusion-base/datafusion/proto-common)
      Compiling datafusion-proto v53.1.0 (/workspace/datafusion-base/datafusion/proto)
       Finished `bench` profile [optimized + debuginfo] target(s) in 8m 59s
        Running benches/sql.rs (/workspace/datafusion-base/target/release/deps/sql-2c9f51cbfcaf27f8)
   Gnuplot not found, using plotters backend
   
   thread 'main' (21031) panicked at benchmarks/benches/sql.rs:131:46:
   assertion failed: Execution("Error in result on row 1, column 1 running query \"\nSELECT COUNT(*) > 0 from lineitem;\": expected value \"true\" but got value \"false\" in row: [\"false\"]")
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   error: bench failed, to rerun pass `--bench sql`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348064363

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                      HEAD ┃       worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │               6.47 / 7.01 ±0.92 / 8.86 ms │               6.72 / 7.23 ±0.96 / 9.15 ms │     no change │
   │ QQuery 2  │         110.78 / 112.15 ±1.61 / 115.23 ms │         112.11 / 112.48 ±0.37 / 113.15 ms │     no change │
   │ QQuery 3  │         108.04 / 108.65 ±0.47 / 109.30 ms │         106.86 / 107.51 ±0.65 / 108.58 ms │     no change │
   │ QQuery 4  │      1013.81 / 1024.01 ±6.46 / 1032.44 ms │      1019.12 / 1025.38 ±4.06 / 1031.83 ms │     no change │
   │ QQuery 5  │         182.15 / 185.43 ±4.69 / 194.61 ms │         181.62 / 183.65 ±2.67 / 188.86 ms │     no change │
   │ QQuery 6  │            85.98 / 87.64 ±1.31 / 89.96 ms │            88.18 / 89.44 ±1.38 / 92.08 ms │     no change │
   │ QQuery 7  │         310.02 / 316.46 ±4.93 / 324.02 ms │         307.87 / 314.64 ±5.81 / 324.43 ms │     no change │
   │ QQuery 8  │         142.60 / 145.06 ±2.00 / 147.89 ms │         144.21 / 145.19 ±0.80 / 146.39 ms │     no change │
   │ QQuery 9  │        200.21 / 218.61 ±11.63 / 229.88 ms │        209.63 / 226.68 ±11.20 / 244.40 ms │     no change │
   │ QQuery 10 │         152.57 / 158.22 ±7.34 / 172.70 ms │         153.60 / 156.70 ±3.15 / 162.60 ms │     no change │
   │ QQuery 11 │         639.07 / 645.78 ±5.70 / 653.81 ms │         644.95 / 647.48 ±1.81 / 649.41 ms │     no change │
   │ QQuery 12 │            34.57 / 36.26 ±2.27 / 40.76 ms │            34.94 / 35.18 ±0.22 / 35.48 ms │     no change │
   │ QQuery 13 │         512.72 / 518.75 ±3.88 / 523.58 ms │         513.07 / 517.09 ±3.47 / 522.78 ms │     no change │
   │ QQuery 14 │         970.38 / 977.53 ±4.11 / 982.06 ms │         976.37 / 979.29 ±2.66 / 983.30 ms │     no change │
   │ QQuery 15 │            16.83 / 17.10 ±0.23 / 17.52 ms │            16.20 / 16.46 ±0.31 / 16.98 ms │     no change │
   │ QQuery 16 │               6.95 / 7.04 ±0.11 / 7.25 ms │               6.90 / 7.06 ±0.20 / 7.45 ms │     no change │
   │ QQuery 17 │         175.15 / 179.16 ±6.74 / 192.59 ms │         172.41 / 175.45 ±4.41 / 184.13 ms │     no change │
   │ QQuery 18 │        363.66 / 401.19 ±20.10 / 418.91 ms │         223.76 / 225.41 ±1.41 / 227.49 ms │ +1.78x faster │
   │ QQuery 19 │         130.26 / 130.99 ±0.69 / 132.03 ms │         131.54 / 132.99 ±1.26 / 135.05 ms │     no change │
   │ QQuery 20 │            13.94 / 14.24 ±0.19 / 14.51 ms │            14.29 / 14.51 ±0.14 / 14.69 ms │     no change │
   │ QQuery 21 │            24.99 / 25.43 ±0.31 / 25.83 ms │            24.81 / 24.90 ±0.07 / 25.01 ms │     no change │
   │ QQuery 22 │         489.82 / 492.87 ±2.46 / 497.18 ms │         490.48 / 496.01 ±4.42 / 501.66 ms │     no change │
   │ QQuery 23 │      1363.25 / 1378.22 ±8.05 / 1384.52 ms │      1321.96 / 1334.25 ±9.13 / 1349.16 ms │     no change │
   │ QQuery 24 │         173.71 / 176.12 ±2.02 / 178.75 ms │         173.96 / 177.20 ±4.47 / 185.96 ms │     no change │
   │ QQuery 25 │         264.66 / 268.24 ±4.41 / 276.65 ms │         261.52 / 265.63 ±5.53 / 276.50 ms │     no change │
   │ QQuery 26 │         130.48 / 134.30 ±6.05 / 146.26 ms │         129.18 / 129.88 ±0.54 / 130.58 ms │     no change │
   │ QQuery 27 │               6.42 / 6.64 ±0.19 / 7.00 ms │               6.60 / 6.76 ±0.13 / 6.98 ms │     no change │
   │ QQuery 28 │         201.40 / 204.83 ±3.86 / 212.06 ms │         200.85 / 204.91 ±4.99 / 214.54 ms │     no change │
   │ QQuery 29 │        215.99 / 223.73 ±12.65 / 248.79 ms │        214.46 / 223.79 ±15.40 / 254.44 ms │     no change │
   │ QQuery 30 │            47.90 / 50.03 ±3.71 / 57.45 ms │            48.34 / 50.35 ±1.97 / 53.15 ms │     no change │
   │ QQuery 31 │         158.77 / 161.09 ±1.55 / 163.54 ms │         157.81 / 159.33 ±1.22 / 161.41 ms │     no change │
   │ QQuery 32 │            13.24 / 13.53 ±0.25 / 13.95 ms │            13.29 / 13.67 ±0.29 / 14.00 ms │     no change │
   │ QQuery 33 │         122.99 / 124.58 ±2.65 / 129.86 ms │         124.74 / 127.70 ±3.84 / 135.16 ms │     no change │
   │ QQuery 34 │               6.71 / 6.90 ±0.21 / 7.30 ms │               6.89 / 7.05 ±0.18 / 7.38 ms │     no change │
   │ QQuery 35 │         129.09 / 133.29 ±4.16 / 141.15 ms │         131.97 / 138.32 ±8.44 / 154.91 ms │     no change │
   │ QQuery 36 │               6.21 / 6.38 ±0.18 / 6.69 ms │               6.45 / 6.73 ±0.17 / 6.94 ms │  1.05x slower │
   │ QQuery 37 │               5.12 / 5.15 ±0.03 / 5.21 ms │               5.15 / 5.26 ±0.10 / 5.44 ms │     no change │
   │ QQuery 38 │         105.20 / 106.22 ±0.93 / 107.79 ms │         107.00 / 108.00 ±0.62 / 108.74 ms │     no change │
   │ QQuery 39 │         124.39 / 129.38 ±6.43 / 141.92 ms │         126.48 / 130.20 ±4.04 / 138.04 ms │     no change │
   │ QQuery 40 │         118.93 / 121.83 ±2.46 / 125.44 ms │         121.38 / 125.09 ±2.31 / 128.20 ms │     no change │
   │ QQuery 41 │            14.26 / 14.50 ±0.24 / 14.97 ms │            13.91 / 14.18 ±0.26 / 14.66 ms │     no change │
   │ QQuery 42 │         103.28 / 104.62 ±0.87 / 105.96 ms │         104.32 / 105.59 ±0.95 / 107.22 ms │     no change │
   │ QQuery 43 │               5.53 / 5.62 ±0.14 / 5.91 ms │               5.47 / 5.61 ±0.20 / 6.00 ms │     no change │
   │ QQuery 44 │            10.38 / 10.49 ±0.10 / 10.68 ms │            10.51 / 10.66 ±0.11 / 10.83 ms │     no change │
   │ QQuery 45 │            38.33 / 39.04 ±0.65 / 40.07 ms │            38.15 / 39.64 ±2.20 / 43.99 ms │     no change │
   │ QQuery 46 │               8.25 / 8.34 ±0.09 / 8.51 ms │               8.24 / 8.46 ±0.24 / 8.90 ms │     no change │
   │ QQuery 47 │         705.53 / 715.55 ±7.86 / 725.72 ms │         710.28 / 725.23 ±8.02 / 732.47 ms │     no change │
   │ QQuery 48 │        425.73 / 437.14 ±10.41 / 456.14 ms │        426.80 / 438.64 ±11.86 / 455.02 ms │     no change │
   │ QQuery 49 │         258.02 / 260.35 ±1.91 / 263.14 ms │         258.38 / 262.07 ±3.40 / 267.80 ms │     no change │
   │ QQuery 50 │         483.40 / 494.01 ±7.52 / 502.18 ms │         496.59 / 499.15 ±2.03 / 502.02 ms │     no change │
   │ QQuery 51 │         198.27 / 201.26 ±2.54 / 205.38 ms │         199.50 / 205.42 ±7.86 / 220.98 ms │     no change │
   │ QQuery 52 │         102.88 / 103.86 ±0.94 / 105.55 ms │         104.11 / 106.45 ±2.75 / 111.68 ms │     no change │
   │ QQuery 53 │         126.13 / 131.83 ±5.74 / 142.50 ms │         126.25 / 128.78 ±1.96 / 132.01 ms │     no change │
   │ QQuery 54 │         116.42 / 120.52 ±6.67 / 133.79 ms │         115.14 / 118.36 ±2.77 / 122.12 ms │     no change │
   │ QQuery 55 │         101.70 / 103.96 ±1.48 / 106.34 ms │         102.24 / 102.88 ±0.78 / 104.31 ms │     no change │
   │ QQuery 56 │         121.88 / 124.24 ±2.23 / 127.39 ms │         123.74 / 126.02 ±1.95 / 129.43 ms │     no change │
   │ QQuery 57 │         174.63 / 177.05 ±2.51 / 181.74 ms │         175.95 / 177.96 ±2.32 / 181.98 ms │     no change │
   │ QQuery 58 │         210.22 / 211.73 ±1.74 / 214.57 ms │         129.97 / 132.01 ±1.41 / 134.39 ms │ +1.60x faster │
   │ QQuery 59 │         249.87 / 253.74 ±2.11 / 256.34 ms │         250.12 / 255.92 ±3.97 / 260.23 ms │     no change │
   │ QQuery 60 │         126.22 / 127.81 ±1.61 / 130.91 ms │         127.57 / 127.96 ±0.26 / 128.37 ms │     no change │
   │ QQuery 61 │            12.65 / 12.78 ±0.13 / 13.02 ms │            12.59 / 14.57 ±3.66 / 21.88 ms │  1.14x slower │
   │ QQuery 62 │         871.85 / 879.27 ±8.76 / 895.73 ms │         861.14 / 868.52 ±5.40 / 875.57 ms │     no change │
   │ QQuery 63 │         130.66 / 135.71 ±6.10 / 147.46 ms │         128.04 / 130.28 ±2.43 / 134.92 ms │     no change │
   │ QQuery 64 │ 28570.71 / 29681.52 ±788.75 / 30730.98 ms │ 28199.92 / 29632.78 ±753.58 / 30377.84 ms │     no change │
   │ QQuery 65 │         328.52 / 330.90 ±2.34 / 335.00 ms │         329.50 / 337.65 ±4.70 / 343.36 ms │     no change │
   │ QQuery 66 │         174.53 / 179.88 ±3.69 / 184.04 ms │         176.99 / 179.61 ±2.83 / 185.08 ms │     no change │
   │ QQuery 67 │         460.22 / 469.76 ±7.37 / 479.75 ms │        466.23 / 475.65 ±11.34 / 496.74 ms │     no change │
   │ QQuery 68 │              8.29 / 9.00 ±1.06 / 11.11 ms │               8.60 / 8.87 ±0.20 / 9.20 ms │     no change │
   │ QQuery 69 │         145.12 / 154.22 ±7.39 / 163.07 ms │         146.41 / 158.04 ±7.04 / 164.29 ms │     no change │
   │ QQuery 70 │         394.37 / 400.25 ±5.19 / 407.45 ms │         398.00 / 404.28 ±4.79 / 412.29 ms │     no change │
   │ QQuery 71 │         117.40 / 121.75 ±3.80 / 128.24 ms │         119.99 / 124.22 ±3.13 / 129.73 ms │     no change │
   │ QQuery 72 │     1159.12 / 1172.54 ±22.99 / 1218.35 ms │     1054.99 / 1075.78 ±28.50 / 1132.23 ms │ +1.09x faster │
   │ QQuery 73 │               6.59 / 6.75 ±0.19 / 7.12 ms │               6.59 / 6.74 ±0.20 / 7.13 ms │     no change │
   │ QQuery 74 │         450.77 / 460.00 ±7.25 / 469.93 ms │         457.83 / 465.08 ±6.23 / 475.70 ms │     no change │
   │ QQuery 75 │         254.10 / 261.16 ±5.59 / 270.42 ms │         254.29 / 261.91 ±4.63 / 268.60 ms │     no change │
   │ QQuery 76 │         261.25 / 266.32 ±4.36 / 274.32 ms │         280.33 / 283.35 ±2.95 / 288.67 ms │  1.06x slower │
   │ QQuery 77 │         209.61 / 214.93 ±3.94 / 219.83 ms │         211.75 / 215.04 ±2.17 / 218.24 ms │     no change │
   │ QQuery 78 │         279.80 / 284.97 ±3.23 / 288.63 ms │         283.66 / 286.82 ±2.12 / 289.40 ms │     no change │
   │ QQuery 79 │         236.42 / 241.38 ±3.13 / 245.35 ms │         238.89 / 243.02 ±3.50 / 247.59 ms │     no change │
   │ QQuery 80 │         261.78 / 265.67 ±2.09 / 267.54 ms │         265.80 / 268.26 ±2.22 / 272.11 ms │     no change │
   │ QQuery 81 │            29.49 / 30.30 ±0.96 / 32.11 ms │            29.40 / 30.08 ±0.77 / 31.50 ms │     no change │
   │ QQuery 82 │            42.16 / 43.45 ±0.98 / 44.55 ms │            42.05 / 43.76 ±1.42 / 46.25 ms │     no change │
   │ QQuery 83 │            42.75 / 44.19 ±1.71 / 47.53 ms │            34.51 / 34.98 ±0.55 / 36.05 ms │ +1.26x faster │
   │ QQuery 84 │            60.47 / 62.19 ±2.32 / 66.76 ms │            60.27 / 62.04 ±1.20 / 63.52 ms │     no change │
   │ QQuery 85 │         243.48 / 246.55 ±2.86 / 251.73 ms │         248.34 / 253.33 ±3.79 / 259.04 ms │     no change │
   │ QQuery 86 │            45.51 / 46.71 ±0.98 / 48.46 ms │            46.28 / 47.05 ±0.68 / 48.28 ms │     no change │
   │ QQuery 87 │               3.56 / 3.65 ±0.14 / 3.93 ms │               3.58 / 3.70 ±0.17 / 4.03 ms │     no change │
   │ QQuery 88 │         118.48 / 124.35 ±4.08 / 129.08 ms │         113.10 / 122.28 ±5.26 / 129.21 ms │     no change │
   │ QQuery 89 │         146.53 / 154.57 ±6.42 / 162.35 ms │         151.06 / 157.69 ±5.04 / 163.10 ms │     no change │
   │ QQuery 90 │            22.97 / 23.86 ±0.64 / 24.70 ms │            22.95 / 24.74 ±1.57 / 27.02 ms │     no change │
   │ QQuery 91 │            88.86 / 94.23 ±3.67 / 99.89 ms │            86.68 / 92.03 ±3.90 / 96.46 ms │     no change │
   │ QQuery 92 │            49.36 / 49.96 ±0.41 / 50.59 ms │            48.65 / 49.71 ±1.06 / 51.60 ms │     no change │
   │ QQuery 93 │         177.16 / 180.91 ±2.55 / 185.07 ms │         179.35 / 181.37 ±1.83 / 183.77 ms │     no change │
   │ QQuery 94 │            63.55 / 65.48 ±1.45 / 67.72 ms │            62.66 / 64.75 ±1.24 / 66.16 ms │     no change │
   │ QQuery 95 │         141.49 / 143.80 ±1.89 / 146.89 ms │         120.94 / 121.31 ±0.20 / 121.49 ms │ +1.19x faster │
   │ QQuery 96 │            73.35 / 76.59 ±2.01 / 79.26 ms │            70.27 / 74.98 ±2.67 / 77.79 ms │     no change │
   │ QQuery 97 │         134.96 / 141.02 ±3.15 / 143.80 ms │         140.53 / 142.52 ±2.21 / 146.63 ms │     no change │
   │ QQuery 98 │         111.39 / 114.76 ±2.51 / 119.07 ms │         112.58 / 114.00 ±1.05 / 115.74 ms │     no change │
   │ QQuery 99 │  10689.14 / 10742.95 ±61.40 / 10861.64 ms │  10650.52 / 10704.57 ±34.66 / 10754.27 ms │     no change │
   └───────────┴───────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 60643.99ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 60211.18ms │
   │ Average Time (HEAD)                                │   612.57ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   608.19ms │
   │ Queries Faster                                     │          5 │
   │ Queries Slower                                     │          3 │
   │ Queries with No Change                             │         91 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 305.1s |
   | Peak memory | 18.5 GiB |
   | Avg memory | 10.0 GiB |
   | CPU user | 597.6s |
   | CPU sys | 12.4s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 305.1s |
   | Peak memory | 18.4 GiB |
   | Avg memory | 10.0 GiB |
   | CPU user | 590.3s |
   | CPU sys | 12.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348060287

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
    Downloading crates ...
     Downloaded prost-derive v0.14.3
     Downloaded anyhow v1.0.102
     Downloaded prost v0.14.3
      Compiling anyhow v1.0.102
      Compiling either v1.15.0
      Compiling datafusion-benchmarks v53.1.0 (/workspace/datafusion-base/benchmarks)
      Compiling itertools v0.14.0
      Compiling prost-derive v0.14.3
      Compiling prost v0.14.3
      Compiling datafusion-proto-common v53.1.0 (/workspace/datafusion-base/datafusion/proto-common)
      Compiling datafusion-proto v53.1.0 (/workspace/datafusion-base/datafusion/proto)
       Finished `bench` profile [optimized + debuginfo] target(s) in 9m 13s
        Running benches/sql.rs (/workspace/datafusion-base/target/release/deps/sql-2c9f51cbfcaf27f8)
   Gnuplot not found, using plotters backend
   
   thread 'main' (22082) panicked at benchmarks/benches/sql.rs:131:46:
   assertion failed: Execution("Error in result on row 1, column 1 running query \"\nSELECT COUNT(*) > 0 from lineitem;\": expected value \"true\" but got value \"false\" in row: [\"false\"]")
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   error: bench failed, to rerun pass `--bench sql`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347936630

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347919215-1925-8sn56 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499

   run benchmarks clickbench_partitioned tpch tpch10 tpcds hj
   
   ```yaml
   env:
     DATAFUSION_EXECUTION_TARGET_PARTITIONS: "128"
     DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
     DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: "true"
   baseline:
     ref: main
   changed:
     ref: HEAD
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348088667

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920499)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347920499-1940-4x7q5 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on code in PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#discussion_r3242278159


##########
datafusion/common/src/config.rs:
##########
@@ -1309,11 +1309,15 @@ config_namespace! {
         /// very large IN lists that might not provide much benefit over hash table lookups.
         ///
         /// This uses the deduplicated row count once the build side has been evaluated.
+        /// In `Partitioned` hash-join mode the same threshold also gates the

Review Comment:
   does this also affect CollectLeft?



##########
datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs:
##########
@@ -650,47 +701,157 @@ impl SharedBuildAccumulator {
                     }
                 }
 
-                let filter_expr = if has_canceled_unknown {
-                    let mut when_then_branches = empty_partition_ids
-                        .into_iter()
-                        .map(|partition_id| {
-                            (
-                                lit(ScalarValue::UInt64(Some(partition_id as u64))),
-                                lit(false),
-                            )
-                        })
-                        .collect::<Vec<_>>();
-                    when_then_branches.extend(real_branches);
-
-                    if when_then_branches.is_empty() {
-                        lit(true)
-                    } else {
-                        Arc::new(CaseExpr::try_new(
-                            Some(modulo_expr),
-                            when_then_branches,
-                            Some(lit(true)),
-                        )?) as Arc<dyn PhysicalExpr>
-                    }
-                } else if real_branches.is_empty() {
-                    lit(false)
-                } else if real_branches.len() == 1
-                    && empty_partition_ids.len() + 1 == num_partitions
-                {
-                    Arc::clone(&real_branches[0].1)
-                } else {
-                    Arc::new(CaseExpr::try_new(
-                        Some(modulo_expr),
-                        real_branches,
-                        Some(lit(false)),
-                    )?) as Arc<dyn PhysicalExpr>
-                };
-
+                let filter_expr = self
+                    .build_partitioned_filter(&real_partitions, has_canceled_unknown)?;
                 self.dynamic_filter.update(filter_expr)?;
             }
         }
 
         Ok(())
     }
+
+    /// CollectLeft has a single shared build side, so we always have one
+    /// `Map`. We prefer the InList expression when it's available (the build
+    /// side fit under the InList caps) because it's directly representable in
+    /// parquet stats / bloom-filter pruning at the scan; otherwise fall back
+    /// to a single `hash_lookup` against the map.
+    fn collect_left_membership(
+        &self,
+        pushdown: &PushdownStrategy,
+    ) -> Result<Option<Arc<dyn PhysicalExpr>>> {
+        if let Some(arr) = &pushdown.inlist {
+            return Ok(Some(create_inlist_predicate(
+                &self.on_right,
+                Arc::clone(arr),
+                self.probe_schema.as_ref(),
+            )?));
+        }
+        Ok(pushdown.map.as_ref().map(|map| {
+            create_hash_lookup_predicate(&self.on_right, Arc::clone(map), &HASH_JOIN_SEED)
+        }))
+    }
+
+    /// Build the dynamic filter for `PartitionMode::Partitioned`. Emits
+    /// `global_minmax AND ([merged_in_list AND] multi_hash_lookup)` —
+    /// independent of how the build side was repartitioned.
+    ///
+    /// * `global_minmax` — envelope of every partition's per-column min/max.
+    ///   Cheap short-circuit and the only piece visible to scan-level
+    ///   `pruning_predicate` extraction.
+    /// * `merged_in_list` — concatenated, deduplicated build keys when every
+    ///   reported partition contributed an `InList` array and the
+    ///   cross-partition union fits under
+    ///   `optimizer.hash_join_inlist_pushdown_max_distinct_values`. A small
+    ///   `IN (SET)` participates in parquet stats / bloom-filter pruning,
+    ///   which `multi_hash_lookup` does not. When present it fully replaces
+    ///   the lookup.
+    /// * `multi_hash_lookup` — hashes the join keys once and ORs
+    ///   `contain_hashes()` across every partition's hash table.
+    ///
+    /// `has_canceled_unknown` partitions short-circuit to `lit(true)`: we
+    /// don't have their maps, so we cannot include them in the lookup, and
+    /// the query is being torn down anyway.
+    fn build_partitioned_filter(
+        &self,
+        real_partitions: &[&PartitionData],
+        has_canceled_unknown: bool,
+    ) -> Result<Arc<dyn PhysicalExpr>> {
+        if has_canceled_unknown {
+            return Ok(lit(true));
+        }
+        if real_partitions.is_empty() {
+            return Ok(lit(false));
+        }
+
+        let bounds_refs: Vec<&PartitionBounds> =
+            real_partitions.iter().map(|p| &p.bounds).collect();
+        let global_bounds_expr = compute_global_bounds(&bounds_refs)
+            .as_ref()
+            .and_then(|b| create_bounds_predicate(&self.on_right, b));
+
+        // The merged InList covers the union of every partition's
+        // build-side keys, so when it fires it stands alone — there is no
+        // need to also AND a `multi_hash_lookup` (which would just probe
+        // the same data via a different structure).
+        let membership_expr =
+            if let Some(merged) = self.try_build_merged_inlist(real_partitions)? {
+                Some(merged)
+            } else {
+                let maps: Vec<Arc<Map>> = real_partitions
+                    .iter()
+                    .filter_map(|p| p.pushdown.map.clone())
+                    .collect();

Review Comment:
   This can become bounds only if non are present



##########
datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs:
##########
@@ -202,6 +195,64 @@ fn combine_membership_and_bounds(
     }
 }
 
+/// Compute the global (envelope) min/max bounds across a set of partition bounds.
+///
+/// For each column index, returns the smallest min seen and the largest max seen.
+/// Columns where any partition is missing bounds, or where bounds are not totally
+/// ordered (e.g. mixed-type comparisons), are dropped from the global envelope.
+fn compute_global_bounds(per_partition: &[&PartitionBounds]) -> Option<PartitionBounds> {
+    let mut iter = per_partition.iter();
+    let first = iter.next()?;
+    let mut acc: Vec<Option<ColumnBounds>> = first
+        .column_bounds
+        .iter()
+        .map(|cb| Some(cb.clone()))
+        .collect();
+
+    for partition in iter {
+        if partition.column_bounds.len() != acc.len() {
+            return None;
+        }
+        for (slot, cb) in acc.iter_mut().zip(partition.column_bounds.iter()) {
+            let Some(existing) = slot.as_mut() else {
+                continue;
+            };
+            match cb.min.partial_cmp(&existing.min) {
+                Some(std::cmp::Ordering::Less) => existing.min = cb.min.clone(),
+                Some(_) => {}
+                None => {
+                    *slot = None;
+                    continue;
+                }
+            }
+            match cb.max.partial_cmp(&existing.max) {
+                Some(std::cmp::Ordering::Greater) => existing.max = cb.max.clone(),
+                Some(_) => {}
+                None => *slot = None,
+            }
+        }
+    }
+
+    let merged: Vec<ColumnBounds> = acc.into_iter().flatten().collect();

Review Comment:
   still think this flatten can cuase a problem. would prefer regression test with this as well 👍 



##########
datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:
##########
@@ -343,6 +354,197 @@ impl PhysicalExpr for HashTableLookupExpr {
     }
 }
 
+/// Physical expression that probes the same join keys against multiple [`Map`]s
+/// and returns `true` for any row whose join keys match at least one map.
+///
+/// Equivalent to `OR`-ing several [`HashTableLookupExpr`]s, but
+/// `create_hashes` runs exactly once for the whole batch and every
+/// [`Map::HashMap`] probe shares the same hash buffer. All `HashMap`
+/// entries must therefore have been built with the same `RandomState`;
+/// [`Map::ArrayMap`] entries are queried via `contain_keys` and do not
+/// consume hashes.
+pub struct MultiMapLookupExpr {
+    /// Join-key expressions evaluated against each input batch.
+    on_columns: Vec<PhysicalExprRef>,
+    /// Hashing seed shared by every entry in `maps`.
+    random_state: SeededRandomState,
+    /// Build-side maps to OR over, one per partition.
+    maps: Vec<Arc<Map>>,
+    /// Display name used in `EXPLAIN` output (e.g. `"multi_hash_lookup"`).
+    description: String,
+}
+
+impl MultiMapLookupExpr {
+    pub fn new(
+        on_columns: Vec<PhysicalExprRef>,
+        random_state: SeededRandomState,
+        maps: Vec<Arc<Map>>,
+        description: String,
+    ) -> Self {
+        Self {
+            on_columns,
+            random_state,
+            maps,
+            description,
+        }
+    }
+}
+
+impl std::fmt::Debug for MultiMapLookupExpr {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        let cols = self
+            .on_columns
+            .iter()
+            .map(|e| e.to_string())
+            .collect::<Vec<_>>()
+            .join(", ");
+        write!(
+            f,
+            "{}({cols}, [{}], maps={})",
+            self.description,
+            self.random_state.seed(),
+            self.maps.len()
+        )
+    }
+}
+
+impl Hash for MultiMapLookupExpr {
+    fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+        self.on_columns.dyn_hash(state);
+        self.description.hash(state);
+        self.random_state.seed().hash(state);
+        // See HashTableLookupExpr — pointer identity is what we use for Maps.
+        for map in &self.maps {
+            Arc::as_ptr(map).hash(state);
+        }
+    }
+}
+
+impl PartialEq for MultiMapLookupExpr {
+    fn eq(&self, other: &Self) -> bool {
+        self.on_columns == other.on_columns
+            && self.description == other.description
+            && self.random_state.seed() == other.random_state.seed()
+            && self.maps.len() == other.maps.len()
+            && self
+                .maps
+                .iter()
+                .zip(other.maps.iter())
+                .all(|(a, b)| Arc::ptr_eq(a, b))
+    }
+}
+
+impl Eq for MultiMapLookupExpr {}
+
+impl Display for MultiMapLookupExpr {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", self.description)
+    }
+}
+
+impl PhysicalExpr for MultiMapLookupExpr {
+    fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+        self.on_columns.iter().collect()
+    }
+
+    fn with_new_children(
+        self: Arc<Self>,
+        children: Vec<Arc<dyn PhysicalExpr>>,
+    ) -> Result<Arc<dyn PhysicalExpr>> {
+        Ok(Arc::new(MultiMapLookupExpr::new(
+            children,
+            self.random_state.clone(),
+            self.maps.clone(),
+            self.description.clone(),
+        )))
+    }
+
+    fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+        Ok(DataType::Boolean)
+    }
+
+    fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+        Ok(false)
+    }
+
+    fn evaluate(&self, batch: &RecordBatch) -> Result<ColumnarValue> {
+        let num_rows = batch.num_rows();
+        let join_keys = evaluate_columns(&self.on_columns, batch)?;
+
+        if self.maps.is_empty() || num_rows == 0 {
+            // Empty `maps` would not be constructed by the dynamic-filter
+            // builder — guard anyway: an empty OR is `false` for every row.
+            let buffer = BooleanBufferBuilder::new(num_rows);
+            let mut buffer = buffer;

Review Comment:
   just "let mut buffer = ..."



##########
datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs:
##########
@@ -650,47 +701,157 @@ impl SharedBuildAccumulator {
                     }
                 }
 
-                let filter_expr = if has_canceled_unknown {
-                    let mut when_then_branches = empty_partition_ids
-                        .into_iter()
-                        .map(|partition_id| {
-                            (
-                                lit(ScalarValue::UInt64(Some(partition_id as u64))),
-                                lit(false),
-                            )
-                        })
-                        .collect::<Vec<_>>();
-                    when_then_branches.extend(real_branches);
-
-                    if when_then_branches.is_empty() {
-                        lit(true)
-                    } else {
-                        Arc::new(CaseExpr::try_new(
-                            Some(modulo_expr),
-                            when_then_branches,
-                            Some(lit(true)),
-                        )?) as Arc<dyn PhysicalExpr>
-                    }
-                } else if real_branches.is_empty() {
-                    lit(false)
-                } else if real_branches.len() == 1
-                    && empty_partition_ids.len() + 1 == num_partitions
-                {
-                    Arc::clone(&real_branches[0].1)
-                } else {
-                    Arc::new(CaseExpr::try_new(
-                        Some(modulo_expr),
-                        real_branches,
-                        Some(lit(false)),
-                    )?) as Arc<dyn PhysicalExpr>
-                };
-
+                let filter_expr = self
+                    .build_partitioned_filter(&real_partitions, has_canceled_unknown)?;
                 self.dynamic_filter.update(filter_expr)?;
             }
         }
 
         Ok(())
     }
+
+    /// CollectLeft has a single shared build side, so we always have one
+    /// `Map`. We prefer the InList expression when it's available (the build
+    /// side fit under the InList caps) because it's directly representable in
+    /// parquet stats / bloom-filter pruning at the scan; otherwise fall back
+    /// to a single `hash_lookup` against the map.
+    fn collect_left_membership(
+        &self,
+        pushdown: &PushdownStrategy,
+    ) -> Result<Option<Arc<dyn PhysicalExpr>>> {
+        if let Some(arr) = &pushdown.inlist {
+            return Ok(Some(create_inlist_predicate(
+                &self.on_right,
+                Arc::clone(arr),
+                self.probe_schema.as_ref(),
+            )?));
+        }
+        Ok(pushdown.map.as_ref().map(|map| {
+            create_hash_lookup_predicate(&self.on_right, Arc::clone(map), &HASH_JOIN_SEED)
+        }))
+    }
+
+    /// Build the dynamic filter for `PartitionMode::Partitioned`. Emits

Review Comment:
   I think more accurate is:
   ```text
   global_minmax AND merged_in_list
   
   or
   
   global_minmax AND multi_hash_lookup
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on code in PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#discussion_r3250295194


##########
datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs:
##########
@@ -202,6 +195,64 @@ fn combine_membership_and_bounds(
     }
 }
 
+/// Compute the global (envelope) min/max bounds across a set of partition bounds.
+///
+/// For each column index, returns the smallest min seen and the largest max seen.
+/// Columns where any partition is missing bounds, or where bounds are not totally
+/// ordered (e.g. mixed-type comparisons), are dropped from the global envelope.
+fn compute_global_bounds(per_partition: &[&PartitionBounds]) -> Option<PartitionBounds> {
+    let mut iter = per_partition.iter();
+    let first = iter.next()?;
+    let mut acc: Vec<Option<ColumnBounds>> = first
+        .column_bounds
+        .iter()
+        .map(|cb| Some(cb.clone()))
+        .collect();
+
+    for partition in iter {
+        if partition.column_bounds.len() != acc.len() {
+            return None;
+        }
+        for (slot, cb) in acc.iter_mut().zip(partition.column_bounds.iter()) {
+            let Some(existing) = slot.as_mut() else {
+                continue;
+            };
+            match cb.min.partial_cmp(&existing.min) {
+                Some(std::cmp::Ordering::Less) => existing.min = cb.min.clone(),
+                Some(_) => {}
+                None => {
+                    *slot = None;
+                    continue;
+                }
+            }
+            match cb.max.partial_cmp(&existing.max) {
+                Some(std::cmp::Ordering::Greater) => existing.max = cb.max.clone(),
+                Some(_) => {}
+                None => *slot = None,
+            }
+        }
+    }
+
+    let merged: Vec<ColumnBounds> = acc.into_iter().flatten().collect();

Review Comment:
   I noticed you pushed more commits but this is still here. The regression test does not replicate because it drops the second column



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4356101837

   I'm quite in favor of this change. It also avoids blowing up the expression based on number of partitions, which can happen when partition count is high.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4477213114

   The CPU utilization went up a good amount.
   
   Could it be better to have some threshold to only use the `global_minmax` and drop the `multi_hash_lookup` so we aren't probing every partition map when expensive?
   
   Or we could split and tune `datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values`. It is set to 20 right now which caps how large the arrays can be per partition and the union arrays across partitions. This seems pretty low especially for the global. Maybe split this into 2 configs and we can tune them a bit more?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471450093

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471353144-176-lmx9b 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471617137

   run benchmarks tpch10
   
   ```yaml
   baseline:
     ref: main
     env:
       DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "false"
       DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: "false"
   changed:
     ref: HEAD
     env:
       DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
       DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: "true"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-5125579898

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345663356

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4345547022)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 41.66 / 42.57 ±1.04 / 44.21 ms │      40.50 / 41.42 ±1.12 / 43.61 ms │     no change │
   │ QQuery 2  │ 21.92 / 22.31 ±0.47 / 23.20 ms │      20.98 / 21.26 ±0.16 / 21.46 ms │     no change │
   │ QQuery 3  │ 39.33 / 43.06 ±2.19 / 46.23 ms │      38.00 / 39.19 ±1.17 / 40.70 ms │ +1.10x faster │
   │ QQuery 4  │ 19.18 / 19.43 ±0.14 / 19.59 ms │      18.65 / 19.08 ±0.62 / 20.30 ms │     no change │
   │ QQuery 5  │ 48.30 / 50.23 ±1.65 / 52.60 ms │      49.54 / 50.89 ±1.39 / 53.45 ms │     no change │
   │ QQuery 6  │ 17.67 / 17.74 ±0.07 / 17.85 ms │      17.64 / 17.83 ±0.15 / 18.08 ms │     no change │
   │ QQuery 7  │ 55.53 / 56.45 ±0.75 / 57.57 ms │      54.68 / 56.55 ±1.58 / 58.30 ms │     no change │
   │ QQuery 8  │ 48.77 / 48.97 ±0.18 / 49.24 ms │      48.75 / 48.86 ±0.08 / 49.01 ms │     no change │
   │ QQuery 9  │ 54.11 / 55.41 ±0.91 / 56.61 ms │      54.78 / 56.06 ±1.70 / 59.40 ms │     no change │
   │ QQuery 10 │ 66.35 / 67.23 ±1.43 / 70.07 ms │      67.59 / 69.20 ±1.44 / 71.78 ms │     no change │
   │ QQuery 11 │ 14.35 / 14.72 ±0.47 / 15.60 ms │      14.69 / 15.15 ±0.30 / 15.50 ms │     no change │
   │ QQuery 12 │ 27.56 / 28.25 ±0.63 / 29.05 ms │      28.14 / 28.76 ±0.80 / 30.30 ms │     no change │
   │ QQuery 13 │ 38.35 / 38.83 ±0.29 / 39.13 ms │      39.07 / 39.66 ±0.37 / 40.25 ms │     no change │
   │ QQuery 14 │ 28.33 / 28.63 ±0.22 / 28.99 ms │      28.93 / 29.22 ±0.23 / 29.56 ms │     no change │
   │ QQuery 15 │ 34.09 / 34.42 ±0.31 / 35.01 ms │      34.76 / 35.24 ±0.37 / 35.86 ms │     no change │
   │ QQuery 16 │ 15.70 / 15.80 ±0.07 / 15.90 ms │      16.00 / 16.27 ±0.38 / 17.03 ms │     no change │
   │ QQuery 17 │ 79.29 / 80.63 ±0.82 / 81.84 ms │      75.69 / 77.38 ±0.90 / 78.18 ms │     no change │
   │ QQuery 18 │ 76.53 / 77.79 ±0.95 / 79.12 ms │      78.73 / 79.49 ±0.46 / 80.05 ms │     no change │
   │ QQuery 19 │ 38.12 / 38.39 ±0.16 / 38.58 ms │      38.75 / 39.13 ±0.24 / 39.35 ms │     no change │
   │ QQuery 20 │ 40.28 / 41.62 ±2.55 / 46.73 ms │      40.75 / 42.65 ±3.22 / 49.05 ms │     no change │
   │ QQuery 21 │ 64.08 / 65.63 ±1.07 / 66.75 ms │      66.91 / 67.85 ±0.80 / 69.22 ms │     no change │
   │ QQuery 22 │ 24.47 / 24.68 ±0.14 / 24.82 ms │      24.97 / 25.24 ±0.19 / 25.47 ms │     no change │
   └───────────┴────────────────────────────────┴─────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 912.79ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 916.39ms │
   │ Average Time (HEAD)                                │  41.49ms │
   │ Average Time (worktree-dynamic-filter-restructure) │  41.65ms │
   │ Queries Faster                                     │        1 │
   │ Queries Slower                                     │        0 │
   │ Queries with No Change                             │       21 │
   │ Queries with Failure                               │        0 │
   └────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 5.5 GiB |
   | Avg memory | 4.9 GiB |
   | CPU user | 34.2s |
   | CPU sys | 2.5s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 5.5 GiB |
   | Avg memory | 4.8 GiB |
   | CPU user | 34.3s |
   | CPU sys | 2.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345562875

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4345547022)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4345547022-1916-gt4f5 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (7a8272fed3de221096a523aec1c5273122c7f201) to main [diff](https://github.com/apache/datafusion/compare/5fda21683ed73f89b06431fac94777d6b1540b02..7a8272fed3de221096a523aec1c5273122c7f201) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345834007

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4345770652)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4345770652-1920-8dwhv 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (18129fe9a705988f6805567a2aeec9d7ce509b3e) to main [diff](https://github.com/apache/datafusion/compare/5fda21683ed73f89b06431fac94777d6b1540b02..18129fe9a705988f6805567a2aeec9d7ce509b3e) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348044656

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃   worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.19 / 4.69 ±6.92 / 18.53 ms │          1.19 / 4.51 ±6.59 / 17.69 ms │     no change │
   │ QQuery 1  │        12.16 / 12.51 ±0.23 / 12.85 ms │        12.31 / 12.46 ±0.08 / 12.55 ms │     no change │
   │ QQuery 2  │        36.60 / 36.92 ±0.31 / 37.33 ms │        36.60 / 36.87 ±0.30 / 37.39 ms │     no change │
   │ QQuery 3  │        32.65 / 33.39 ±0.45 / 33.99 ms │        33.20 / 33.71 ±0.33 / 34.13 ms │     no change │
   │ QQuery 4  │     244.21 / 252.08 ±6.54 / 260.20 ms │     244.81 / 250.39 ±9.24 / 268.83 ms │     no change │
   │ QQuery 5  │     284.85 / 297.23 ±9.25 / 307.82 ms │     282.32 / 296.01 ±9.14 / 305.21 ms │     no change │
   │ QQuery 6  │           6.72 / 7.71 ±0.51 / 8.16 ms │           6.49 / 7.11 ±0.51 / 7.73 ms │ +1.08x faster │
   │ QQuery 7  │        14.19 / 14.31 ±0.09 / 14.42 ms │        13.50 / 13.67 ±0.14 / 13.88 ms │     no change │
   │ QQuery 8  │    329.22 / 346.05 ±14.54 / 371.92 ms │    329.29 / 344.99 ±11.47 / 358.01 ms │     no change │
   │ QQuery 9  │    456.23 / 470.55 ±12.92 / 492.18 ms │    461.69 / 479.28 ±14.81 / 500.34 ms │     no change │
   │ QQuery 10 │        72.86 / 76.19 ±3.90 / 83.72 ms │        77.80 / 79.04 ±1.27 / 81.02 ms │     no change │
   │ QQuery 11 │        83.12 / 85.01 ±1.55 / 87.05 ms │       87.76 / 90.86 ±4.97 / 100.72 ms │  1.07x slower │
   │ QQuery 12 │    275.15 / 291.23 ±12.70 / 313.52 ms │     306.88 / 310.22 ±3.58 / 317.13 ms │  1.07x slower │
   │ QQuery 13 │    393.31 / 410.66 ±16.77 / 436.11 ms │    408.45 / 419.61 ±10.19 / 434.44 ms │     no change │
   │ QQuery 14 │     284.97 / 292.22 ±9.49 / 310.69 ms │    287.35 / 297.40 ±13.21 / 323.41 ms │     no change │
   │ QQuery 15 │     283.71 / 301.03 ±9.74 / 312.18 ms │    288.22 / 303.10 ±14.24 / 325.92 ms │     no change │
   │ QQuery 16 │    629.36 / 639.18 ±10.15 / 656.35 ms │    629.65 / 653.74 ±17.21 / 682.86 ms │     no change │
   │ QQuery 17 │    628.30 / 644.95 ±19.53 / 679.47 ms │    630.94 / 662.53 ±18.56 / 687.61 ms │     no change │
   │ QQuery 18 │ 1309.17 / 1347.42 ±31.01 / 1393.23 ms │ 1295.95 / 1312.42 ±13.21 / 1334.48 ms │     no change │
   │ QQuery 19 │        29.61 / 30.26 ±0.39 / 30.78 ms │       28.58 / 34.04 ±10.09 / 54.21 ms │  1.13x slower │
   │ QQuery 20 │     520.15 / 534.16 ±8.85 / 546.19 ms │     520.89 / 530.40 ±8.64 / 545.71 ms │     no change │
   │ QQuery 21 │     598.64 / 605.89 ±6.45 / 615.89 ms │     601.60 / 609.42 ±5.23 / 615.67 ms │     no change │
   │ QQuery 22 │  1075.26 / 1079.90 ±3.85 / 1085.85 ms │ 1078.63 / 1101.41 ±19.45 / 1130.00 ms │     no change │
   │ QQuery 23 │ 3367.48 / 3422.97 ±31.56 / 3460.65 ms │ 3378.99 / 3430.08 ±26.32 / 3451.13 ms │     no change │
   │ QQuery 24 │        42.35 / 45.76 ±5.09 / 55.89 ms │        44.01 / 47.25 ±5.48 / 58.18 ms │     no change │
   │ QQuery 25 │     113.60 / 118.17 ±4.56 / 126.53 ms │     114.53 / 115.64 ±1.06 / 117.55 ms │     no change │
   │ QQuery 26 │        42.86 / 44.41 ±1.57 / 47.16 ms │        43.12 / 44.27 ±1.31 / 46.40 ms │     no change │
   │ QQuery 27 │     676.24 / 680.93 ±4.73 / 689.23 ms │     677.88 / 689.59 ±7.01 / 698.47 ms │     no change │
   │ QQuery 28 │ 3037.14 / 3066.30 ±22.28 / 3097.38 ms │ 3040.47 / 3061.08 ±22.48 / 3100.68 ms │     no change │
   │ QQuery 29 │        42.23 / 42.59 ±0.42 / 43.37 ms │        42.18 / 47.11 ±6.01 / 56.25 ms │  1.11x slower │
   │ QQuery 30 │     305.82 / 321.98 ±8.88 / 330.80 ms │     332.08 / 336.95 ±4.97 / 344.57 ms │     no change │
   │ QQuery 31 │     307.29 / 317.12 ±7.00 / 325.54 ms │     313.03 / 322.31 ±9.16 / 334.34 ms │     no change │
   │ QQuery 32 │ 1048.69 / 1087.01 ±27.75 / 1132.01 ms │  1053.94 / 1062.14 ±4.38 / 1066.81 ms │     no change │
   │ QQuery 33 │ 1493.34 / 1545.83 ±31.11 / 1580.68 ms │ 1498.22 / 1520.79 ±18.23 / 1553.44 ms │     no change │
   │ QQuery 34 │ 1460.45 / 1489.21 ±22.28 / 1518.93 ms │ 1481.06 / 1539.00 ±51.50 / 1628.81 ms │     no change │
   │ QQuery 35 │    282.72 / 304.57 ±19.70 / 332.54 ms │     327.08 / 329.93 ±1.75 / 331.98 ms │  1.08x slower │
   │ QQuery 36 │        65.24 / 69.22 ±4.38 / 74.59 ms │        65.80 / 73.32 ±8.50 / 89.55 ms │  1.06x slower │
   │ QQuery 37 │        34.97 / 38.67 ±6.12 / 50.87 ms │        38.06 / 41.73 ±3.51 / 47.57 ms │  1.08x slower │
   │ QQuery 38 │        40.04 / 44.41 ±3.19 / 48.61 ms │        41.31 / 42.72 ±1.24 / 44.41 ms │     no change │
   │ QQuery 39 │    126.34 / 143.60 ±13.65 / 167.13 ms │     126.20 / 136.45 ±8.91 / 148.23 ms │     no change │
   │ QQuery 40 │        14.27 / 15.62 ±2.05 / 19.70 ms │        14.56 / 18.01 ±3.80 / 24.59 ms │  1.15x slower │
   │ QQuery 41 │        13.45 / 14.62 ±1.56 / 17.67 ms │        13.98 / 15.51 ±2.92 / 21.36 ms │  1.06x slower │
   │ QQuery 42 │        12.98 / 13.81 ±1.20 / 16.18 ms │        13.28 / 13.57 ±0.15 / 13.72 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 20640.34ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 20770.66ms │
   │ Average Time (HEAD)                                │   480.01ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   483.04ms │
   │ Queries Faster                                     │          1 │
   │ Queries Slower                                     │          9 │
   │ Queries with No Change                             │         33 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 105.0s |
   | Peak memory | 30.8 GiB |
   | Avg memory | 23.5 GiB |
   | CPU user | 1086.7s |
   | CPU sys | 70.6s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 105.0s |
   | Peak memory | 30.4 GiB |
   | Avg memory | 23.2 GiB |
   | CPU user | 1093.6s |
   | CPU sys | 72.9s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347956563

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347919711-1931-l59fw 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpch10
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152

   run benchmarks clickbench_partitioned tpch tpch10 tpcds hj
   
   ```yaml
   env:
     DATAFUSION_EXECUTION_TARGET_PARTITIONS: "128"
     DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "false"
     DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: "false"
   baseline:
     ref: main
   changed:
     ref: HEAD
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348068497

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4347920152-1936-qkcr7 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (f717a99adc3eee55d2b07676bba57e343b3baf4d) to main [diff](https://github.com/apache/datafusion/compare/42cd2fa90d18a8a54f433fc94917c20a4d1e7338..f717a99adc3eee55d2b07676bba57e343b3baf4d) using: tpch10
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471363318

   🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471352777-174-82gfl 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed)
   BENCH_NAME=hj
   BENCH_COMMAND=cargo bench --features=parquet --bench hj
   BENCH_FILTER=
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471360842

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471352777-172-ncz9d 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed) using: tpch10
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471363930

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471352777-173-w58v2 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed) using: tpcds
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471351985

   Thanks @gene-bordegaray ! @Dandandan would you mind taking a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471454401

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
    * [new ref]         refs/pull/21931/head -> worktree-dynamic-filter-restructure
    * branch            main       -> FETCH_HEAD
   Switched to branch 'worktree-dynamic-filter-restructure'
   6a57b220842f342e3765912f5e64c299f9756529
   From https://github.com/apache/datafusion
    * branch            refs/pull/21931/head -> FETCH_HEAD
   Cloning into '/workspace/datafusion-base'...
   From https://github.com/apache/datafusion
    * branch            refs/pull/21931/head -> FETCH_HEAD
   Your branch is up to date with 'origin/main'.
   Already on 'main'
   rustc 1.95.0 (59807616e 2026-04-14)
   42adf29b6bd78b10156cec3bc8838570fe666aed
   47655fd6c9ef060d73497987e6ccb98e57196508
       Blocking waiting for file lock on package cache
       Blocking waiting for file lock on package cache
       Blocking waiting for file lock on package cache
   error: no bench target named `hj` in default-run packages
   
   help: a target with a similar name exists: `chr`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471521216

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ worktree-dynamic-filter-restructure ┃         Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
   │ QQuery 1  │           6.28 / 6.75 ±0.79 / 8.33 ms │         6.09 / 6.53 ±0.83 / 8.18 ms │      no change │
   │ QQuery 2  │        46.88 / 47.51 ±0.43 / 48.15 ms │      44.62 / 44.97 ±0.44 / 45.82 ms │  +1.06x faster │
   │ QQuery 3  │        27.98 / 28.39 ±0.43 / 29.02 ms │      27.89 / 28.08 ±0.24 / 28.55 ms │      no change │
   │ QQuery 4  │     293.02 / 303.86 ±8.04 / 314.76 ms │   292.43 / 295.25 ±4.33 / 303.87 ms │      no change │
   │ QQuery 5  │        78.27 / 79.24 ±1.34 / 81.87 ms │      60.69 / 62.01 ±1.58 / 65.13 ms │  +1.28x faster │
   │ QQuery 6  │        29.94 / 30.38 ±0.55 / 31.44 ms │      29.21 / 29.74 ±0.28 / 29.98 ms │      no change │
   │ QQuery 7  │     139.56 / 143.60 ±3.42 / 147.51 ms │   142.06 / 144.87 ±3.32 / 151.40 ms │      no change │
   │ QQuery 8  │        20.25 / 20.57 ±0.30 / 20.94 ms │      21.30 / 21.42 ±0.12 / 21.60 ms │      no change │
   │ QQuery 9  │     130.20 / 132.97 ±3.29 / 137.66 ms │   136.95 / 142.55 ±4.33 / 149.24 ms │   1.07x slower │
   │ QQuery 10 │     114.82 / 116.74 ±2.09 / 119.62 ms │   113.09 / 118.08 ±5.39 / 127.70 ms │      no change │
   │ QQuery 11 │     198.16 / 201.82 ±2.92 / 207.05 ms │   207.39 / 209.60 ±1.27 / 211.30 ms │      no change │
   │ QQuery 12 │        21.81 / 22.81 ±0.81 / 24.00 ms │      22.72 / 22.85 ±0.11 / 23.03 ms │      no change │
   │ QQuery 13 │     144.47 / 147.95 ±5.27 / 158.34 ms │   147.74 / 149.01 ±1.98 / 152.94 ms │      no change │
   │ QQuery 14 │     520.15 / 524.60 ±4.12 / 531.60 ms │   511.82 / 521.57 ±5.04 / 525.67 ms │      no change │
   │ QQuery 15 │        20.26 / 20.94 ±1.03 / 22.97 ms │      17.16 / 18.26 ±1.10 / 20.37 ms │  +1.15x faster │
   │ QQuery 16 │           6.95 / 7.10 ±0.16 / 7.40 ms │         6.47 / 6.56 ±0.08 / 6.66 ms │  +1.08x faster │
   │ QQuery 17 │        76.71 / 77.33 ±0.53 / 78.17 ms │      70.67 / 72.89 ±1.68 / 74.44 ms │  +1.06x faster │
   │ QQuery 18 │     279.13 / 287.41 ±4.47 / 292.16 ms │   218.99 / 220.05 ±1.31 / 222.36 ms │  +1.31x faster │
   │ QQuery 19 │        47.78 / 48.25 ±0.50 / 49.02 ms │      48.00 / 48.48 ±0.35 / 48.88 ms │      no change │
   │ QQuery 20 │        20.07 / 20.21 ±0.09 / 20.30 ms │      20.19 / 20.72 ±0.75 / 22.20 ms │      no change │
   │ QQuery 21 │        16.84 / 17.10 ±0.19 / 17.38 ms │      17.11 / 17.22 ±0.08 / 17.37 ms │      no change │
   │ QQuery 22 │        61.51 / 64.82 ±2.64 / 67.97 ms │      63.95 / 66.95 ±3.36 / 73.52 ms │      no change │
   │ QQuery 23 │     378.78 / 381.22 ±1.90 / 384.55 ms │   371.81 / 382.13 ±6.73 / 393.01 ms │      no change │
   │ QQuery 24 │     118.86 / 121.77 ±2.93 / 126.15 ms │   116.49 / 118.99 ±2.47 / 122.18 ms │      no change │
   │ QQuery 25 │        76.84 / 78.27 ±1.10 / 79.98 ms │      75.41 / 77.10 ±2.08 / 80.99 ms │      no change │
   │ QQuery 26 │     118.69 / 123.80 ±4.89 / 131.87 ms │   117.71 / 120.26 ±3.18 / 126.14 ms │      no change │
   │ QQuery 27 │           7.47 / 7.62 ±0.12 / 7.82 ms │         6.83 / 6.96 ±0.15 / 7.21 ms │  +1.10x faster │
   │ QQuery 28 │        81.20 / 82.35 ±1.92 / 86.18 ms │      77.03 / 78.85 ±1.38 / 80.50 ms │      no change │
   │ QQuery 29 │       87.79 / 91.96 ±4.88 / 101.53 ms │      83.23 / 85.08 ±1.74 / 87.86 ms │  +1.08x faster │
   │ QQuery 30 │        33.45 / 33.87 ±0.32 / 34.26 ms │      33.04 / 33.37 ±0.27 / 33.66 ms │      no change │
   │ QQuery 31 │     134.38 / 137.28 ±2.44 / 141.52 ms │   123.24 / 125.66 ±2.14 / 128.69 ms │  +1.09x faster │
   │ QQuery 32 │        22.00 / 22.55 ±0.57 / 23.60 ms │      20.06 / 20.35 ±0.27 / 20.81 ms │  +1.11x faster │
   │ QQuery 33 │        41.56 / 42.09 ±0.44 / 42.85 ms │      39.78 / 39.99 ±0.13 / 40.13 ms │      no change │
   │ QQuery 34 │        10.18 / 10.71 ±0.51 / 11.45 ms │      10.21 / 10.35 ±0.24 / 10.82 ms │      no change │
   │ QQuery 35 │     116.74 / 119.14 ±4.00 / 127.09 ms │   118.42 / 120.27 ±2.83 / 125.76 ms │      no change │
   │ QQuery 36 │           6.40 / 6.71 ±0.30 / 7.26 ms │         6.33 / 6.47 ±0.16 / 6.76 ms │      no change │
   │ QQuery 37 │           7.53 / 7.67 ±0.09 / 7.80 ms │         7.51 / 7.60 ±0.09 / 7.76 ms │      no change │
   │ QQuery 38 │        85.61 / 86.89 ±0.89 / 88.29 ms │      84.71 / 85.14 ±0.33 / 85.58 ms │      no change │
   │ QQuery 39 │       91.15 / 94.11 ±4.05 / 101.97 ms │      91.73 / 94.59 ±2.91 / 99.58 ms │      no change │
   │ QQuery 40 │        20.59 / 21.38 ±0.67 / 22.24 ms │      20.01 / 20.34 ±0.20 / 20.64 ms │      no change │
   │ QQuery 41 │        13.36 / 13.52 ±0.13 / 13.69 ms │      13.52 / 13.67 ±0.18 / 14.01 ms │      no change │
   │ QQuery 42 │        23.85 / 24.19 ±0.32 / 24.76 ms │      22.47 / 22.68 ±0.15 / 22.84 ms │  +1.07x faster │
   │ QQuery 43 │           5.17 / 5.36 ±0.23 / 5.74 ms │         5.04 / 5.13 ±0.11 / 5.35 ms │      no change │
   │ QQuery 44 │        10.57 / 10.74 ±0.13 / 10.94 ms │      10.50 / 10.74 ±0.18 / 10.97 ms │      no change │
   │ QQuery 45 │        27.38 / 28.11 ±0.43 / 28.51 ms │      26.19 / 26.50 ±0.27 / 26.94 ms │  +1.06x faster │
   │ QQuery 46 │        14.32 / 14.60 ±0.24 / 15.03 ms │      14.07 / 14.21 ±0.16 / 14.48 ms │      no change │
   │ QQuery 47 │    245.69 / 253.25 ±10.77 / 274.62 ms │  248.54 / 269.87 ±11.11 / 280.32 ms │   1.07x slower │
   │ QQuery 48 │     134.80 / 139.15 ±3.34 / 144.34 ms │   138.60 / 139.82 ±1.38 / 142.43 ms │      no change │
   │ QQuery 49 │        75.10 / 76.09 ±0.65 / 76.81 ms │      73.68 / 76.56 ±2.18 / 80.06 ms │      no change │
   │ QQuery 50 │     146.15 / 149.71 ±2.46 / 153.76 ms │   105.15 / 105.69 ±0.45 / 106.34 ms │  +1.42x faster │
   │ QQuery 51 │     103.49 / 106.69 ±2.50 / 110.23 ms │   104.94 / 109.12 ±2.46 / 112.21 ms │      no change │
   │ QQuery 52 │        23.95 / 24.33 ±0.27 / 24.77 ms │      23.13 / 23.48 ±0.33 / 23.94 ms │      no change │
   │ QQuery 53 │        39.19 / 39.88 ±0.71 / 41.18 ms │      39.07 / 39.31 ±0.16 / 39.52 ms │      no change │
   │ QQuery 54 │        26.73 / 27.03 ±0.19 / 27.27 ms │      25.85 / 26.14 ±0.18 / 26.36 ms │      no change │
   │ QQuery 55 │        23.17 / 23.50 ±0.32 / 23.96 ms │      22.40 / 22.84 ±0.37 / 23.47 ms │      no change │
   │ QQuery 56 │        37.30 / 38.42 ±0.98 / 39.99 ms │      36.12 / 37.52 ±2.04 / 41.57 ms │      no change │
   │ QQuery 57 │     135.26 / 137.82 ±2.92 / 143.51 ms │   136.48 / 139.39 ±2.49 / 143.33 ms │      no change │
   │ QQuery 58 │        60.53 / 61.77 ±1.35 / 64.08 ms │      59.18 / 60.65 ±1.89 / 64.30 ms │      no change │
   │ QQuery 59 │        93.26 / 95.26 ±2.42 / 99.51 ms │      92.05 / 93.30 ±1.48 / 95.65 ms │      no change │
   │ QQuery 60 │        42.10 / 42.41 ±0.31 / 42.98 ms │      41.01 / 41.38 ±0.37 / 42.06 ms │      no change │
   │ QQuery 61 │        13.53 / 13.75 ±0.20 / 14.08 ms │      13.47 / 13.61 ±0.16 / 13.92 ms │      no change │
   │ QQuery 62 │        44.38 / 45.24 ±0.84 / 46.55 ms │      43.89 / 45.41 ±1.82 / 48.81 ms │      no change │
   │ QQuery 63 │        40.09 / 41.43 ±1.98 / 45.31 ms │      39.99 / 40.93 ±0.49 / 41.42 ms │      no change │
   │ QQuery 64 │     476.75 / 481.51 ±3.65 / 487.67 ms │   522.82 / 531.01 ±6.71 / 541.03 ms │   1.10x slower │
   │ QQuery 65 │ 1060.50 / 1092.54 ±28.79 / 1130.15 ms │    99.21 / 104.12 ±6.27 / 115.77 ms │ +10.49x faster │
   │ QQuery 66 │        70.12 / 71.64 ±1.53 / 74.58 ms │      67.77 / 68.93 ±1.17 / 71.05 ms │      no change │
   │ QQuery 67 │     283.22 / 288.87 ±6.03 / 298.00 ms │   269.30 / 276.45 ±6.85 / 288.73 ms │      no change │
   │ QQuery 68 │        15.22 / 15.38 ±0.10 / 15.53 ms │      14.31 / 14.50 ±0.22 / 14.92 ms │  +1.06x faster │
   │ QQuery 69 │     112.03 / 117.69 ±5.40 / 127.88 ms │   102.87 / 105.16 ±2.39 / 108.83 ms │  +1.12x faster │
   │ QQuery 70 │     131.43 / 137.39 ±3.95 / 141.48 ms │   128.38 / 134.57 ±8.09 / 150.45 ms │      no change │
   │ QQuery 71 │        33.47 / 33.96 ±0.30 / 34.37 ms │      31.48 / 32.56 ±0.82 / 33.56 ms │      no change │
   │ QQuery 72 │     199.22 / 201.13 ±1.98 / 204.92 ms │   182.27 / 184.65 ±1.59 / 186.75 ms │  +1.09x faster │
   │ QQuery 73 │        10.75 / 10.98 ±0.14 / 11.17 ms │      10.07 / 10.36 ±0.35 / 11.04 ms │  +1.06x faster │
   │ QQuery 74 │     139.86 / 144.03 ±5.80 / 155.51 ms │   135.79 / 138.51 ±3.16 / 143.99 ms │      no change │
   │ QQuery 75 │     167.02 / 168.13 ±1.07 / 169.91 ms │   167.26 / 170.64 ±2.30 / 173.05 ms │      no change │
   │ QQuery 76 │        46.90 / 50.06 ±4.03 / 57.62 ms │      45.88 / 46.25 ±0.23 / 46.45 ms │  +1.08x faster │
   │ QQuery 77 │        72.75 / 73.14 ±0.40 / 73.81 ms │      70.58 / 71.48 ±1.07 / 73.20 ms │      no change │
   │ QQuery 78 │     142.80 / 147.05 ±2.97 / 152.11 ms │   143.92 / 145.22 ±1.91 / 148.97 ms │      no change │
   │ QQuery 79 │        75.93 / 76.14 ±0.16 / 76.37 ms │      74.73 / 75.54 ±0.78 / 76.83 ms │      no change │
   │ QQuery 80 │        87.46 / 88.33 ±0.65 / 89.03 ms │      84.91 / 86.44 ±1.39 / 88.95 ms │      no change │
   │ QQuery 81 │        28.46 / 28.89 ±0.31 / 29.37 ms │      27.62 / 28.34 ±0.70 / 29.61 ms │      no change │
   │ QQuery 82 │        23.63 / 23.88 ±0.21 / 24.17 ms │      23.28 / 23.60 ±0.25 / 23.92 ms │      no change │
   │ QQuery 83 │        36.95 / 37.25 ±0.22 / 37.48 ms │      33.47 / 33.90 ±0.36 / 34.39 ms │  +1.10x faster │
   │ QQuery 84 │        56.70 / 56.93 ±0.16 / 57.13 ms │      56.21 / 57.41 ±2.01 / 61.41 ms │      no change │
   │ QQuery 85 │     234.40 / 239.31 ±3.47 / 243.91 ms │   168.96 / 172.71 ±2.85 / 175.91 ms │  +1.39x faster │
   │ QQuery 86 │        30.70 / 32.13 ±2.13 / 36.37 ms │      30.02 / 30.69 ±1.08 / 32.86 ms │      no change │
   │ QQuery 87 │        89.02 / 89.56 ±0.39 / 89.96 ms │      86.68 / 88.57 ±1.43 / 90.57 ms │      no change │
   │ QQuery 88 │        70.11 / 75.92 ±5.43 / 83.05 ms │      72.56 / 76.75 ±3.86 / 83.30 ms │      no change │
   │ QQuery 89 │        47.00 / 49.47 ±3.60 / 56.55 ms │      47.48 / 49.54 ±2.17 / 53.52 ms │      no change │
   │ QQuery 90 │        19.81 / 20.04 ±0.13 / 20.16 ms │      20.11 / 20.25 ±0.12 / 20.42 ms │      no change │
   │ QQuery 91 │        71.91 / 72.67 ±0.47 / 73.31 ms │      73.66 / 75.32 ±1.84 / 78.71 ms │      no change │
   │ QQuery 92 │        32.16 / 32.86 ±0.47 / 33.60 ms │      29.42 / 29.99 ±0.40 / 30.52 ms │  +1.10x faster │
   │ QQuery 93 │        53.02 / 53.62 ±0.71 / 54.68 ms │      53.43 / 55.20 ±1.56 / 57.13 ms │      no change │
   │ QQuery 94 │        39.28 / 39.81 ±0.54 / 40.86 ms │      38.61 / 38.96 ±0.25 / 39.36 ms │      no change │
   │ QQuery 95 │     114.56 / 116.24 ±0.95 / 117.07 ms │      91.02 / 91.90 ±0.55 / 92.72 ms │  +1.26x faster │
   │ QQuery 96 │        27.90 / 29.66 ±2.92 / 35.48 ms │      27.93 / 28.25 ±0.26 / 28.57 ms │      no change │
   │ QQuery 97 │        50.41 / 51.58 ±0.78 / 52.82 ms │      51.23 / 51.89 ±1.00 / 53.85 ms │      no change │
   │ QQuery 98 │        30.80 / 31.20 ±0.25 / 31.56 ms │      29.63 / 29.97 ±0.26 / 30.42 ms │      no change │
   │ QQuery 99 │        68.79 / 69.49 ±0.74 / 70.90 ms │      69.79 / 70.30 ±0.59 / 71.44 ms │      no change │
   └───────────┴───────────────────────────────────────┴─────────────────────────────────────┴────────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 9580.49ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 8329.09ms │
   │ Average Time (HEAD)                                │   96.77ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   84.13ms │
   │ Queries Faster                                     │        23 │
   │ Queries Slower                                     │         3 │
   │ Queries with No Change                             │        73 │
   │ Queries with Failure                               │         0 │
   └────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 6.3 GiB |
   | Avg memory | 5.8 GiB |
   | CPU user | 133.4s |
   | CPU sys | 5.1s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 45.0s |
   | Peak memory | 6.4 GiB |
   | Avg memory | 5.9 GiB |
   | CPU user | 115.0s |
   | CPU sys | 5.0s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471632005

   The only regressions are `tpch10` w/ filter pushdown on. I guess those are benefiting from the old `CASE` structure more than this new structure. I am re-running with pushdown on vs. off. I think that's the real comparison. My hypothesis is that the `CASE` structure is a bigger win when the hash join dynamic filters work but also a much bigger cost when they don't.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4356288696

   > I'm quite in favor of this change. It also avoids blowing up the expression based on number of partitions, which can happen when partition count is high.
   
   Well it doesn't avoid it completely, and in some ways it makes it worse. We still have 1 hash map per partition (cannot be avoided unless we pay the memory and build time cost of combining them). And we now scale our probes with the number of partitions, they used to be constant with number of partitions. But probes are much faster than hashes which is why I think unless the partition count is high this will likely be faster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4364552279

   I like this direction as a default, seems less brittle and cleaner baseline. This approach will definitely provide some value for `preserved_file_partitions` (#21207 ) 😄 
   
   I’d still like to discuss the possibility of partition-aware filters path when DataFusion can prove both sides share the same partition mapping and as discussed in #21207 the proper infrastructure for expressing more types of partitioning is in place. For use cases we are seeing the partition-specific filters selectivity is quite nice.
   
   I imagine that if we provided information from partitioning that two partitioning spaces (build and probe sides) are compatible (such as the same range partitioning)  we would not only choose from `IN (SET)` or `multi_hash_lookup`, but a partition routed lookup instead with the global as a fallback.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4582646270

   > i am but i’m just strapped way too thin atm, if you can take a look would really appreciate it!
   
   @adriangb cool I will play around with it a bit and post updates here for you. No rush but just had this on queue of PRs 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #21931: perf: simplify HashJoinExec dynamic filter, drop CASE routing
URL: https://github.com/apache/datafusion/pull/21931


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4469502614

   thanks for the ping @gene-bordegaray, I had 42adf29 locally but had not pushed it (it's been crazy trying to squeeze work into breaks at PyCon). The test seems exactly like what you suggested 😃 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471363549

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
     Downloaded ciborium-io v0.2.2
     Downloaded cast v0.3.0
     Downloaded block-buffer v0.10.4
     Downloaded aws-smithy-observability v0.2.6
     Downloaded anstyle v1.0.14
     Downloaded aws-smithy-runtime-api-macros v1.0.0
     Downloaded aws-smithy-async v1.2.14
     Downloaded autocfg v1.5.0
     Downloaded async-trait v0.1.89
     Downloaded arrayvec v0.7.6
     Downloaded anyhow v1.0.102
     Downloaded aws-smithy-http v0.63.6
     Downloaded atomic-waker v1.1.2
     Downloaded async-ffi v0.5.0
     Downloaded cmov v0.5.3
     Downloaded anes v0.1.6
       Blocking waiting for file lock on package cache
   error: no bench target named `hj` in default-run packages
   
   help: a target with a similar name exists: `chr`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777

   run benchmarks clickbench_partitioned tpch tpch10 tpcds hj
   
   ```yaml
   env:
     DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "false"
     DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: "false"
   baseline:
     ref: main
   changed:
     ref: HEAD
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471459579

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471352777)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and worktree-dynamic-filter-restructure
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃   worktree-dynamic-filter-restructure ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.23 / 4.75 ±6.89 / 18.53 ms │          1.26 / 4.86 ±7.03 / 18.92 ms │     no change │
   │ QQuery 1  │        12.79 / 13.25 ±0.23 / 13.43 ms │        12.63 / 13.15 ±0.34 / 13.54 ms │     no change │
   │ QQuery 2  │        36.14 / 36.45 ±0.30 / 36.91 ms │        36.63 / 36.84 ±0.32 / 37.48 ms │     no change │
   │ QQuery 3  │        30.89 / 31.54 ±0.84 / 33.14 ms │        31.34 / 31.60 ±0.29 / 32.15 ms │     no change │
   │ QQuery 4  │     232.90 / 236.75 ±2.79 / 240.43 ms │     241.99 / 246.97 ±4.20 / 252.04 ms │     no change │
   │ QQuery 5  │     274.50 / 276.21 ±2.00 / 280.07 ms │     286.37 / 287.71 ±1.41 / 290.13 ms │     no change │
   │ QQuery 6  │           7.00 / 7.22 ±0.22 / 7.64 ms │          6.92 / 8.90 ±2.00 / 12.72 ms │  1.23x slower │
   │ QQuery 7  │        14.13 / 14.44 ±0.48 / 15.39 ms │        13.85 / 14.25 ±0.36 / 14.89 ms │     no change │
   │ QQuery 8  │     324.78 / 327.01 ±1.64 / 329.29 ms │     321.14 / 324.63 ±2.44 / 327.76 ms │     no change │
   │ QQuery 9  │     446.08 / 459.48 ±7.90 / 469.34 ms │     449.67 / 460.94 ±7.08 / 470.48 ms │     no change │
   │ QQuery 10 │        68.90 / 70.03 ±1.05 / 72.01 ms │        68.15 / 69.17 ±1.10 / 71.31 ms │     no change │
   │ QQuery 11 │        79.58 / 81.48 ±1.92 / 84.97 ms │        78.56 / 79.81 ±0.89 / 81.11 ms │     no change │
   │ QQuery 12 │     268.86 / 272.43 ±3.59 / 278.80 ms │     272.40 / 275.86 ±2.97 / 280.04 ms │     no change │
   │ QQuery 13 │    396.43 / 413.16 ±12.72 / 429.28 ms │     378.17 / 388.69 ±9.78 / 404.34 ms │ +1.06x faster │
   │ QQuery 14 │     292.56 / 296.43 ±3.81 / 301.53 ms │     277.53 / 280.69 ±3.68 / 287.27 ms │ +1.06x faster │
   │ QQuery 15 │     279.93 / 285.62 ±4.06 / 290.15 ms │     275.60 / 281.26 ±4.11 / 287.26 ms │     no change │
   │ QQuery 16 │     611.31 / 622.16 ±8.26 / 636.42 ms │    593.04 / 615.11 ±15.20 / 633.47 ms │     no change │
   │ QQuery 17 │     618.95 / 623.27 ±3.91 / 629.77 ms │     622.10 / 627.81 ±3.27 / 631.65 ms │     no change │
   │ QQuery 18 │ 1226.25 / 1250.84 ±12.86 / 1262.67 ms │  1206.24 / 1212.93 ±5.20 / 1219.87 ms │     no change │
   │ QQuery 19 │       28.17 / 36.78 ±15.63 / 68.01 ms │        28.08 / 31.01 ±4.59 / 40.14 ms │ +1.19x faster │
   │ QQuery 20 │     513.19 / 524.03 ±9.06 / 540.29 ms │     520.93 / 525.06 ±3.24 / 530.22 ms │     no change │
   │ QQuery 21 │     594.67 / 597.70 ±1.86 / 600.12 ms │     605.07 / 609.98 ±2.60 / 612.03 ms │     no change │
   │ QQuery 22 │ 1051.84 / 1066.70 ±16.17 / 1094.49 ms │ 1054.73 / 1078.13 ±13.20 / 1092.97 ms │     no change │
   │ QQuery 23 │ 3166.28 / 3197.69 ±32.92 / 3259.13 ms │ 3158.53 / 3188.59 ±41.15 / 3267.56 ms │     no change │
   │ QQuery 24 │        42.18 / 44.07 ±1.49 / 46.57 ms │        41.71 / 49.04 ±5.84 / 54.93 ms │  1.11x slower │
   │ QQuery 25 │     111.21 / 112.40 ±1.00 / 113.99 ms │     111.51 / 113.40 ±2.00 / 117.00 ms │     no change │
   │ QQuery 26 │        42.30 / 42.91 ±0.97 / 44.82 ms │        42.25 / 43.77 ±2.09 / 47.76 ms │     no change │
   │ QQuery 27 │     673.87 / 679.26 ±8.39 / 695.92 ms │     665.75 / 670.11 ±3.38 / 674.03 ms │     no change │
   │ QQuery 28 │ 2973.99 / 3002.82 ±24.16 / 3041.98 ms │ 3003.19 / 3033.66 ±16.54 / 3048.59 ms │     no change │
   │ QQuery 29 │        41.95 / 50.71 ±7.47 / 60.40 ms │        41.52 / 43.69 ±3.63 / 50.92 ms │ +1.16x faster │
   │ QQuery 30 │     298.57 / 305.82 ±6.20 / 315.06 ms │     299.19 / 303.36 ±2.89 / 306.55 ms │     no change │
   │ QQuery 31 │     294.26 / 302.90 ±6.88 / 313.53 ms │     291.06 / 295.83 ±3.94 / 302.92 ms │     no change │
   │ QQuery 32 │     951.59 / 962.13 ±8.88 / 973.32 ms │    943.33 / 958.25 ±16.06 / 984.64 ms │     no change │
   │ QQuery 33 │ 1404.57 / 1439.81 ±28.37 / 1489.19 ms │ 1416.36 / 1435.15 ±18.95 / 1467.04 ms │     no change │
   │ QQuery 34 │ 1447.85 / 1495.67 ±28.99 / 1531.27 ms │ 1457.73 / 1496.01 ±22.49 / 1514.69 ms │     no change │
   │ QQuery 35 │    280.89 / 291.81 ±13.32 / 317.79 ms │    285.40 / 312.38 ±33.21 / 375.95 ms │  1.07x slower │
   │ QQuery 36 │        60.91 / 64.75 ±3.18 / 70.57 ms │        60.53 / 64.91 ±6.94 / 78.74 ms │     no change │
   │ QQuery 37 │        34.91 / 39.99 ±6.12 / 51.05 ms │        35.17 / 39.15 ±5.31 / 49.55 ms │     no change │
   │ QQuery 38 │        40.69 / 45.54 ±6.91 / 59.11 ms │        43.72 / 46.61 ±3.42 / 51.93 ms │     no change │
   │ QQuery 39 │     119.57 / 130.34 ±6.66 / 140.20 ms │     122.32 / 136.42 ±8.60 / 149.34 ms │     no change │
   │ QQuery 40 │        14.01 / 14.42 ±0.35 / 15.02 ms │        14.03 / 15.64 ±1.66 / 18.75 ms │  1.08x slower │
   │ QQuery 41 │        13.77 / 14.94 ±2.14 / 19.22 ms │        13.60 / 14.01 ±0.26 / 14.28 ms │ +1.07x faster │
   │ QQuery 42 │        13.64 / 19.71 ±7.19 / 28.80 ms │        12.87 / 14.86 ±3.29 / 21.43 ms │ +1.33x faster │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                  ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                  │ 19805.41ms │
   │ Total Time (worktree-dynamic-filter-restructure)   │ 19780.21ms │
   │ Average Time (HEAD)                                │   460.59ms │
   │ Average Time (worktree-dynamic-filter-restructure) │   460.00ms │
   │ Queries Faster                                     │          6 │
   │ Queries Slower                                     │          4 │
   │ Queries with No Change                             │         33 │
   │ Queries with Failure                               │          0 │
   └────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 29.4 GiB |
   | Avg memory | 23.0 GiB |
   | CPU user | 1043.2s |
   | CPU sys | 63.2s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 30.0 GiB |
   | Avg memory | 23.1 GiB |
   | CPU user | 1041.6s |
   | CPU sys | 63.4s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4471443827

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4471353144)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4471353144-175-5nqtd 6.12.68+ #1 SMP Wed Apr  1 02:23:28 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (42adf29b6bd78b10156cec3bc8838570fe666aed) to main [diff](https://github.com/apache/datafusion/compare/47655fd6c9ef060d73497987e6ccb98e57196508..42adf29b6bd78b10156cec3bc8838570fe666aed) using: clickbench_partitioned
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348062074

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
    Downloading crates ...
     Downloaded prost v0.14.3
     Downloaded anyhow v1.0.102
     Downloaded prost-derive v0.14.3
      Compiling anyhow v1.0.102
      Compiling either v1.15.0
      Compiling datafusion-benchmarks v53.1.0 (/workspace/datafusion-base/benchmarks)
      Compiling itertools v0.14.0
      Compiling prost-derive v0.14.3
      Compiling prost v0.14.3
      Compiling datafusion-proto-common v53.1.0 (/workspace/datafusion-base/datafusion/proto-common)
      Compiling datafusion-proto v53.1.0 (/workspace/datafusion-base/datafusion/proto)
       Finished `bench` profile [optimized + debuginfo] target(s) in 9m 16s
        Running benches/sql.rs (/workspace/datafusion-base/target/release/deps/sql-2c9f51cbfcaf27f8)
   Gnuplot not found, using plotters backend
   
   thread 'main' (21962) panicked at benchmarks/benches/sql.rs:131:46:
   assertion failed: Execution("Error in result on row 1, column 1 running query \"\nSELECT COUNT(*) > 0 from lineitem;\": expected value \"true\" but got value \"false\" in row: [\"false\"]")
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   error: bench failed, to rerun pass `--bench sql`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348062527

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919711) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
    Downloading crates ...
     Downloaded prost v0.14.3
     Downloaded prost-derive v0.14.3
     Downloaded anyhow v1.0.102
      Compiling anyhow v1.0.102
      Compiling either v1.15.0
      Compiling datafusion-benchmarks v53.1.0 (/workspace/datafusion-base/benchmarks)
      Compiling itertools v0.14.0
      Compiling prost-derive v0.14.3
      Compiling prost v0.14.3
      Compiling datafusion-proto-common v53.1.0 (/workspace/datafusion-base/datafusion/proto-common)
      Compiling datafusion-proto v53.1.0 (/workspace/datafusion-base/datafusion/proto)
       Finished `bench` profile [optimized + debuginfo] target(s) in 9m 19s
        Running benches/sql.rs (/workspace/datafusion-base/target/release/deps/sql-2c9f51cbfcaf27f8)
   Gnuplot not found, using plotters backend
   
   thread 'main' (21313) panicked at benchmarks/benches/sql.rs:131:46:
   assertion failed: Execution("Error in result on row 1, column 1 running query \"\nSELECT COUNT(*) > 0 from lineitem;\": expected value \"true\" but got value \"false\" in row: [\"false\"]")
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   error: bench failed, to rerun pass `--bench sql`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348071399

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347920152) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
    * [new ref]         refs/pull/21931/head -> worktree-dynamic-filter-restructure
    * branch            main       -> FETCH_HEAD
   Switched to branch 'worktree-dynamic-filter-restructure'
   42cd2fa90d18a8a54f433fc94917c20a4d1e7338
   From https://github.com/apache/datafusion
    * branch            refs/pull/21931/head -> FETCH_HEAD
   Cloning into '/workspace/datafusion-base'...
   From https://github.com/apache/datafusion
    * branch            refs/pull/21931/head -> FETCH_HEAD
   Your branch is up to date with 'origin/main'.
   Already on 'main'
   rustc 1.95.0 (59807616e 2026-04-14)
   f717a99adc3eee55d2b07676bba57e343b3baf4d
   42cd2fa90d18a8a54f433fc94917c20a4d1e7338
       Blocking waiting for file lock on package cache
       Blocking waiting for file lock on package cache
       Blocking waiting for file lock on package cache
   error: no bench target named `hj` in default-run packages
   
   help: a target with a similar name exists: `chr`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345731555

   run benchmark tpch
   
   ```yaml
   baseline:
       ref: main
       env:
          DATAFUSION_EXECUTION_TARGET_PARTITIONS=128
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: false
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: false
   changed:
       ref: HEAD
       env:
          DATAFUSION_EXECUTION_TARGET_PARTITIONS=128
          DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: false
          DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS: false
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4348579449

   @gene-bordegaray @Dandandan I'm curious what you think of this. It's a bit of a compromise:
   
   1. Faster or neutral at low partition counts for both filter pushdown on and off.
   2. Faster or neutral at high partition counts for pushdown off. Slower for pushdown on. I think this is something we can address later.
   
   What this gets us is:
   1. Better perf at settings most benchmarks (and laptops, etc.) run under.
   2. Resolves @gene-bordegaray 's range partitioning issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4347938216

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4347919215) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
     Downloaded aws-credential-types v1.2.14
     Downloaded arrow-ord v58.1.0
     Downloaded arrow-arith v58.1.0
     Downloaded arrayvec v0.7.6
     Downloaded anstyle v1.0.14
     Downloaded cfg-if v1.0.4
     Downloaded block-buffer v0.12.0
     Downloaded block-buffer v0.10.4
     Downloaded async-ffi v0.5.0
     Downloaded aws-smithy-query v0.60.15
     Downloaded aws-smithy-runtime-api-macros v1.0.0
     Downloaded arrayref v0.3.9
     Downloaded aws-smithy-async v1.2.14
     Downloaded autocfg v1.5.0
     Downloaded atomic-waker v1.1.2
     Downloaded ahash v0.8.12
       Blocking waiting for file lock on package cache
   error: no bench target named `hj` in default-run packages
   
   help: a target with a similar name exists: `chr`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4364635439

   > is slower than just probing more hash tables
   
   I am very skeptical of probing partition x hash tables for every row is very efficient, but I see it can still be faster than evaluating a long nested expression which grows based on number of partitions, as DF doesn't have special knowledge of the "routing" (+ if everything matches it will be pure overhead).
   
   I feel like doing the `% partition` in the physical expression and routing that to the correct expression / probe should be faster + more selective than just probing all tables (which as well doesn't scale well to many partitions)?
   
   That said - I like removing the big `CASE`-based routing as it seems hard to get that performant.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345538559

   Benchmark for [this request](https://github.com/apache/datafusion/pull/21931#issuecomment-4345504799) failed.
   
   Last 20 lines of output:
   <details><summary>Click to expand</summary>
   
   ```
     Downloaded clap v4.6.1
     Downloaded ciborium v0.2.2
     Downloaded aws-smithy-xml v0.60.15
     Downloaded async-stream-impl v0.3.6
     Downloaded arrow-csv v58.1.0
     Downloaded compression-core v0.4.32
     Downloaded bytes-utils v0.1.4
     Downloaded blake2 v0.10.6
     Downloaded bitflags v2.11.1
     Downloaded aws-types v1.3.15
     Downloaded aws-smithy-async v1.2.14
     Downloaded anstyle v1.0.14
     Downloaded clap_lex v1.1.0
     Downloaded aws-smithy-query v0.60.15
     Downloaded aws-smithy-observability v0.2.6
     Downloaded arrayref v0.3.9
       Blocking waiting for file lock on package cache
   error: no bench target named `tcph` in default-run packages
   
   help: a target with a similar name exists: `chr`
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4345527891

   🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21931#issuecomment-4345503296)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4345503296-1912-446v6 6.12.55+ #1 SMP Sun Feb  1 08:59:41 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing HEAD (7a8272fed3de221096a523aec1c5273122c7f201) to main [diff](https://github.com/apache/datafusion/compare/66980e26df75740006229125acbf6815ee04a9f4..7a8272fed3de221096a523aec1c5273122c7f201)
   BENCH_NAME=tcph
   BENCH_COMMAND=cargo bench --features=parquet --bench tcph
   BENCH_FILTER=
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#issuecomment-4361848931

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning origin/main
       Building datafusion v53.1.0 (current)
          Built [  85.427s] (current)
        Parsing datafusion v53.1.0 (current)
         Parsed [   0.037s] (current)
       Building datafusion v53.1.0 (baseline)
          Built [  83.381s] (baseline)
        Parsing datafusion v53.1.0 (baseline)
         Parsed [   0.037s] (baseline)
       Checking datafusion v53.1.0 -> v53.1.0 (no change; assume patch)
        Checked [   0.875s] 222 checks: 222 pass, 30 skip
        Summary no semver update required
       Finished [ 171.499s] datafusion
       Building datafusion-common v53.1.0 (current)
   error: running cargo-doc on crate 'datafusion-common' failed with output:
   -----
      Compiling proc-macro2 v1.0.106
      Compiling unicode-ident v1.0.24
      Compiling quote v1.0.45
      Compiling libc v0.2.186
      Compiling libm v0.2.16
      Compiling autocfg v1.5.0
       Checking cfg-if v1.0.4
      Compiling num-traits v0.2.19
      Compiling syn v2.0.117
       Checking memchr v2.8.0
      Compiling find-msvc-tools v0.1.9
      Compiling shlex v1.3.0
      Compiling zerocopy v0.8.48
      Compiling serde_core v1.0.228
       Checking itoa v1.0.18
       Checking bytes v1.11.1
      Compiling zmij v1.0.21
      Compiling jobserver v0.1.34
      Compiling cc v1.2.61
      Compiling serde_json v1.0.149
       Checking num-integer v0.1.46
       Checking iana-time-zone v0.1.65
      Compiling version_check v0.9.5
       Checking siphasher v1.0.2
       Checking stable_deref_trait v1.2.1
      Compiling getrandom v0.3.4
       Checking phf_shared v0.12.1
      Compiling ahash v0.8.12
       Checking chrono v0.4.44
       Checking num-bigint v0.4.6
      Compiling synstructure v0.13.2
      Compiling chrono-tz v0.10.4
       Checking arrow-schema v58.1.0
       Checking phf v0.12.1
       Checking once_cell v1.21.4
       Checking num-complex v0.4.6
       Checking hashbrown v0.16.1
       Checking litemap v0.8.2
       Checking lexical-util v1.0.7
       Checking writeable v0.6.3
      Compiling pkg-config v0.3.33
       Checking utf8_iter v1.0.4
      Compiling object v0.37.3
       Checking smallvec v1.15.1
      Compiling icu_properties_data v2.2.0
      Compiling zerocopy-derive v0.8.48
      Compiling zerofrom-derive v0.1.7
      Compiling yoke-derive v0.8.2
      Compiling zerovec-derive v0.11.3
       Checking zerofrom v0.1.7
       Checking yoke v0.8.2
      Compiling displaydoc v0.2.5
       Checking zerotrie v0.2.4
       Checking zerovec v0.11.6
      Compiling zstd-sys v2.0.16+zstd.1.5.7
      Compiling icu_normalizer_data v2.2.0
       Checking tinystr v0.8.3
       Checking icu_locale_core v2.2.0
       Checking potential_utf v0.1.5
       Checking icu_collections v2.2.0
       Checking icu_provider v2.2.0
      Compiling semver v1.0.28
      Compiling rustc_version v0.4.1
       Checking lexical-parse-integer v1.0.6
       Checking lexical-write-integer v1.0.6
      Compiling zstd-safe v7.2.4
       Checking lexical-write-float v1.0.6
       Checking half v2.7.1
       Checking lexical-parse-float v1.0.6
       Checking arrow-buffer v58.1.0
       Checking icu_normalizer v2.2.0
       Checking icu_properties v2.2.0
       Checking arrow-data v58.1.0
       Checking arrow-array v58.1.0
      Compiling flatbuffers v25.12.19
       Checking aho-corasick v1.1.4
       Checking regex-syntax v0.8.10
      Compiling ar_archive_writer v0.5.1
       Checking arrow-select v58.1.0
       Checking base64 v0.22.1
       Checking ryu v1.0.23
       Checking unicode-width v0.2.2
       Checking unicode-segmentation v1.13.2
       Checking pin-project-lite v0.2.17
      Compiling parking_lot_core v0.9.12
       Checking futures-sink v0.3.32
       Checking futures-core v0.3.32
       Checking regex-automata v0.4.14
       Checking futures-channel v0.3.32
       Checking comfy-table v7.2.2
       Checking arrow-ord v58.1.0
      Compiling psm v0.1.31
       Checking idna_adapter v1.2.2
       Checking lexical-core v1.0.6
      Compiling futures-macro v0.3.32
       Checking atoi v2.0.0
       Checking alloc-no-stdlib v2.0.4
       Checking foldhash v0.2.0
       Checking futures-task v0.3.32
       Checking twox-hash v2.1.2
       Checking slab v0.4.12
       Checking equivalent v1.0.2
       Checking bitflags v2.11.1
       Checking futures-io v0.3.32
       Checking scopeguard v1.2.0
       Checking percent-encoding v2.3.2
       Checking allocator-api2 v0.2.21
      Compiling thiserror v2.0.18
       Checking hashbrown v0.17.0
       Checking regex v1.12.3
       Checking form_urlencoded v1.2.2
       Checking lock_api v0.4.14
       Checking futures-util v0.3.32
       Checking lz4_flex v0.13.0
       Checking alloc-stdlib v0.2.2
       Checking arrow-cast v58.1.0
       Checking idna v1.1.0
      Compiling thiserror-impl v2.0.18
      Compiling ring v0.17.14
      Compiling stacker v0.1.24
       Checking csv-core v0.1.13
      Compiling getrandom v0.4.2
       Checking either v1.15.0
      Compiling paste v1.0.15
       Checking simdutf8 v0.1.5
      Compiling snap v1.1.1
       Checking itertools v0.14.0
       Checking csv v1.4.0
       Checking parking_lot v0.12.5
       Checking url v2.5.8
       Checking brotli-decompressor v5.0.0
       Checking indexmap v2.14.0
      Compiling async-trait v0.1.89
      Compiling tokio-macros v2.7.0
       Checking zstd v0.13.3
       Checking arrow-ipc v58.1.0
       Checking http v1.4.0
       Checking ordered-float v2.10.1
       Checking getrandom v0.2.17
       Checking humantime v2.3.0
       Checking integer-encoding v3.0.4
       Checking zlib-rs v0.6.3
       Checking untrusted v0.9.0
       Checking byteorder v1.5.0
       Checking thrift v0.17.0
       Checking object_store v0.13.2
       Checking tokio v1.52.1
       Checking flate2 v1.1.9
       Checking arrow-json v58.1.0
       Checking brotli v8.0.2
       Checking arrow-csv v58.1.0
       Checking futures v0.3.32
       Checking arrow-string v58.1.0
       Checking arrow-arith v58.1.0
       Checking arrow-row v58.1.0
      Compiling sqlparser_derive v0.5.0
      Compiling recursive-proc-macro-impl v0.1.1
       Checking log v0.4.29
      Compiling seq-macro v0.3.6
       Checking recursive v0.1.1
       Checking sqlparser v0.61.0
       Checking arrow v58.1.0
       Checking uuid v1.23.1
       Checking hex v0.4.3
       Checking parquet v58.1.0
   error[E0432]: unresolved import `object_store::buffered`
      --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/parquet-58.1.0/src/arrow/async_writer/store.rs:25:19
       |
    25 | use object_store::buffered::BufWriter;
       |                   ^^^^^^^^ could not find `buffered` in `object_store`
       |
   note: found an item that was configured out
      --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/object_store-0.13.2/src/lib.rs:545:9
       |
   544 | #[cfg(feature = "tokio")]
       |       ----------------- the item is gated behind the `tokio` feature
   545 | pub mod buffered;
       |         ^^^^^^^^
   
   error[E0282]: type annotations needed
      --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/parquet-58.1.0/src/arrow/async_writer/store.rs:98:13
       |
    98 | /             self.w
    99 | |                 .put(bs)
   100 | |                 .await
       | |______________________^ cannot infer type
   
   error[E0282]: type annotations needed
      --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/parquet-58.1.0/src/arrow/async_writer/store.rs:107:13
       |
   107 | /             self.w
   108 | |                 .shutdown()
   109 | |                 .await
       | |______________________^ cannot infer type
   
   Some errors have detailed explanations: E0282, E0432.
   For more information about an error, try `rustc --explain E0282`.
   error: could not compile `parquet` (lib) due to 3 previous errors
   warning: build failed, waiting for other jobs to finish...
   
   -----
   
   error: failed to build rustdoc for crate datafusion-common v53.1.0
   note: this is usually due to a compilation error in the crate,
         and is unlikely to be a bug in cargo-semver-checks
   note: the following command can be used to reproduce the error:
         cargo new --lib example &&
             cd example &&
             echo '[workspace]' >> Cargo.toml &&
             cargo add --path /home/runner/work/datafusion/datafusion/datafusion/common --features backtrace,force_hash_collisions,object_store,parquet,parquet_encryption,recursive_protection,sql,sqlparser &&
             cargo check &&
             cargo doc
   
       Building datafusion-physical-plan v53.1.0 (current)
          Built [  31.251s] (current)
        Parsing datafusion-physical-plan v53.1.0 (current)
         Parsed [   0.128s] (current)
       Building datafusion-physical-plan v53.1.0 (baseline)
          Built [  31.371s] (baseline)
        Parsing datafusion-physical-plan v53.1.0 (baseline)
         Parsed [   0.131s] (baseline)
       Checking datafusion-physical-plan v53.1.0 -> v53.1.0 (no change; assume patch)
        Checked [   0.775s] 222 checks: 222 pass, 30 skip
        Summary no semver update required
       Finished [  65.339s] datafusion-physical-plan
       Building datafusion-sqllogictest v53.1.0 (current)
          Built [ 135.703s] (current)
        Parsing datafusion-sqllogictest v53.1.0 (current)
         Parsed [   0.024s] (current)
       Building datafusion-sqllogictest v53.1.0 (baseline)
          Built [ 135.547s] (baseline)
        Parsing datafusion-sqllogictest v53.1.0 (baseline)
         Parsed [   0.024s] (baseline)
       Checking datafusion-sqllogictest v53.1.0 -> v53.1.0 (no change; assume patch)
        Checked [   0.113s] 222 checks: 222 pass, 30 skip
        Summary no semver update required
       Finished [ 274.368s] datafusion-sqllogictest
   error: aborting due to failure to build rustdoc for crate datafusion-common v53.1.0
   ```
   
   </details>
   <!-- semver-check-comment -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: simplify HashJoinExec dynamic filter, drop CASE routing [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #21931:
URL: https://github.com/apache/datafusion/pull/21931#discussion_r3222027073


##########
datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs:
##########
@@ -649,47 +700,157 @@ impl SharedBuildAccumulator {
                     }
                 }
 
-                let filter_expr = if has_canceled_unknown {
-                    let mut when_then_branches = empty_partition_ids
-                        .into_iter()
-                        .map(|partition_id| {
-                            (
-                                lit(ScalarValue::UInt64(Some(partition_id as u64))),
-                                lit(false),
-                            )
-                        })
-                        .collect::<Vec<_>>();
-                    when_then_branches.extend(real_branches);
-
-                    if when_then_branches.is_empty() {
-                        lit(true)
-                    } else {
-                        Arc::new(CaseExpr::try_new(
-                            Some(modulo_expr),
-                            when_then_branches,
-                            Some(lit(true)),
-                        )?) as Arc<dyn PhysicalExpr>
-                    }
-                } else if real_branches.is_empty() {
-                    lit(false)
-                } else if real_branches.len() == 1
-                    && empty_partition_ids.len() + 1 == num_partitions
-                {
-                    Arc::clone(&real_branches[0].1)
-                } else {
-                    Arc::new(CaseExpr::try_new(
-                        Some(modulo_expr),
-                        real_branches,
-                        Some(lit(false)),
-                    )?) as Arc<dyn PhysicalExpr>
-                };
-
+                let filter_expr = self
+                    .build_partitioned_filter(&real_partitions, has_canceled_unknown)?;
                 self.dynamic_filter.update(filter_expr)?;
             }
         }
 
         Ok(())
     }
+
+    /// CollectLeft has a single shared build side, so we always have one
+    /// `Map`. We prefer the InList expression when it's available (the build
+    /// side fit under the InList caps) because it's directly representable in
+    /// parquet stats / bloom-filter pruning at the scan; otherwise fall back
+    /// to a single `hash_lookup` against the map.
+    fn collect_left_membership(
+        &self,
+        pushdown: &PushdownStrategy,
+    ) -> Result<Option<Arc<dyn PhysicalExpr>>> {
+        if let Some(arr) = &pushdown.inlist {
+            return Ok(Some(create_inlist_predicate(
+                &self.on_right,
+                Arc::clone(arr),
+                self.probe_schema.as_ref(),
+            )?));
+        }
+        Ok(pushdown.map.as_ref().map(|map| {
+            create_hash_lookup_predicate(&self.on_right, Arc::clone(map), &HASH_JOIN_SEED)
+        }))
+    }
+
+    /// Build the dynamic filter for `PartitionMode::Partitioned`. Emits
+    /// `global_minmax AND ([merged_in_list AND] multi_hash_lookup)` —
+    /// independent of how the build side was repartitioned.
+    ///
+    /// * `global_minmax` — envelope of every partition's per-column min/max.
+    ///   Cheap short-circuit and the only piece visible to scan-level
+    ///   `pruning_predicate` extraction.
+    /// * `merged_in_list` — concatenated, deduplicated build keys when every
+    ///   reported partition contributed an `InList` array and the
+    ///   cross-partition union fits under
+    ///   `optimizer.hash_join_inlist_pushdown_max_distinct_values`. A small
+    ///   `IN (SET)` participates in parquet stats / bloom-filter pruning,
+    ///   which `multi_hash_lookup` does not. When present it fully replaces
+    ///   the lookup.
+    /// * `multi_hash_lookup` — hashes the join keys once and ORs
+    ///   `contain_hashes()` across every partition's hash table.
+    ///
+    /// `has_canceled_unknown` partitions short-circuit to `lit(true)`: we
+    /// don't have their maps, so we cannot include them in the lookup, and
+    /// the query is being torn down anyway.
+    fn build_partitioned_filter(
+        &self,
+        real_partitions: &[&PartitionData],
+        has_canceled_unknown: bool,
+    ) -> Result<Arc<dyn PhysicalExpr>> {
+        if has_canceled_unknown {
+            return Ok(lit(true));
+        }
+        if real_partitions.is_empty() {
+            return Ok(lit(false));
+        }
+
+        let bounds_refs: Vec<&PartitionBounds> =
+            real_partitions.iter().map(|p| &p.bounds).collect();
+        let global_bounds_expr = compute_global_bounds(&bounds_refs)
+            .as_ref()
+            .and_then(|b| create_bounds_predicate(&self.on_right, b));
+
+        // The merged InList covers the union of every partition's
+        // build-side keys, so when it fires it stands alone — there is no
+        // need to also AND a `multi_hash_lookup` (which would just probe
+        // the same data via a different structure).
+        let membership_expr =
+            if let Some(merged) = self.try_build_merged_inlist(real_partitions)? {
+                Some(merged)
+            } else {
+                let maps: Vec<Arc<Map>> = real_partitions
+                    .iter()
+                    .filter_map(|p| p.pushdown.map.clone())
+                    .collect();
+                if maps.is_empty() {
+                    // Defensive: every reported (non-empty) partition is
+                    // supposed to carry a Map. Falling through to None means
+                    // we degrade to bounds-only filtering.
+                    None
+                } else {
+                    Some(Arc::new(MultiMapLookupExpr::new(
+                        self.on_right.clone(),
+                        HASH_JOIN_SEED.clone(),
+                        maps,
+                        "multi_hash_lookup".to_string(),
+                    )) as Arc<dyn PhysicalExpr>)
+                }
+            };
+
+        Ok(
+            combine_membership_and_bounds(membership_expr, global_bounds_expr)
+                .unwrap_or_else(|| lit(true)),
+        )
+    }
+
+    /// If every reported partition contributed an InList array, concatenate
+    /// them, deduplicate by scalar value, and gate on the
+    /// `inlist_max_distinct_values` cap. Returns the merged
+    /// `(struct(...))? IN (SET) ([…])` predicate built over the
+    /// deduplicated keys when the cap is satisfied; `None` otherwise.
+    ///
+    /// Per-partition arrays carry duplicates — each partition ships its raw
+    /// build-side join keys, dedup happens here. The dedup walk early-aborts
+    /// the moment we cross the cap, so the cost stays bounded by
+    /// `O(rows-until-cap+1-distinct-found)` rather than total input size.
+    fn try_build_merged_inlist(
+        &self,
+        real_partitions: &[&PartitionData],
+    ) -> Result<Option<Arc<dyn PhysicalExpr>>> {
+        let cap = self.inlist_max_distinct_values;
+        let mut arrays: Vec<ArrayRef> = Vec::with_capacity(real_partitions.len());
+        for p in real_partitions {
+            let Some(arr) = &p.pushdown.inlist else {
+                return Ok(None);
+            };
+            arrays.push(Arc::clone(arr));
+        }
+        let Some(merged) = merge_inlist_arrays(&arrays) else {

Review Comment:
   It's a bit tricky. Per partition is caped by size (bytes) because that's cheap to calculate. So the size of the non-deduplicated merged InList is already capped (by bytes) so we only pay deduplication cost once.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org