[PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid opened a new pull request, #24227:
URL: https://github.com/apache/datafusion/pull/24227

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   - Closes #.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   
   Please explain the problem you are trying to solve in terms of the user-visible
   behavior, rather than the implementation.
   
   For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the
   implementation. "COUNT(DISTINCT) returns wrong results when the column contains
   nulls" is the user-visible problem.
   -->
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5296334997

   @adriangb When the flag is on, DataFusion promotes string and binary columns that are physically RLE_DICTIONARY encoded in the parquet file to Dictionary(Int32, Utf8) or Dictionary(Int32, Binary) at schema inference time. This applies to all parquet scans regardless of how the table was registered.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3878264513


##########
datafusion/datasource-parquet/src/file_format.rs:
##########
@@ -402,7 +426,12 @@ impl FileFormat for ParquetFormat {
         }
         drop(seen);
 
-        let schemas = schemas.into_iter().map(|(_, schema)| schema);
+        // Normalize dict-promoted schemas before merging so mixed dict/plain files merge cleanly.
+        let mut schemas: Vec<Schema> =
+            schemas.into_iter().map(|(_, schema)| schema).collect();

Review Comment:
   will remove the allowlist so this should no longer be an issue 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364831797

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364728282)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.63 ms │                              37.63 ms │     no change │
   │ QQuery 2  │ 18.98 ms │                              21.30 ms │  1.12x slower │
   │ QQuery 3  │ 30.23 ms │                              30.66 ms │     no change │
   │ QQuery 4  │ 17.25 ms │                              17.57 ms │     no change │
   │ QQuery 5  │ 39.08 ms │                              36.97 ms │ +1.06x faster │
   │ QQuery 6  │ 15.78 ms │                              15.64 ms │     no change │
   │ QQuery 7  │ 43.72 ms │                              43.27 ms │     no change │
   │ QQuery 8  │ 41.92 ms │                              42.20 ms │     no change │
   │ QQuery 9  │ 49.25 ms │                              49.21 ms │     no change │
   │ QQuery 10 │ 41.78 ms │                              59.95 ms │  1.43x slower │
   │ QQuery 11 │ 13.18 ms │                              13.15 ms │     no change │
   │ QQuery 12 │ 23.88 ms │                              23.35 ms │     no change │
   │ QQuery 13 │ 31.73 ms │                              43.57 ms │  1.37x slower │
   │ QQuery 14 │ 22.90 ms │                              23.00 ms │     no change │
   │ QQuery 15 │ 30.21 ms │                              30.08 ms │     no change │
   │ QQuery 16 │ 13.60 ms │                              13.77 ms │     no change │
   │ QQuery 17 │ 68.77 ms │                              70.08 ms │     no change │
   │ QQuery 18 │ 57.29 ms │                              94.52 ms │  1.65x slower │
   │ QQuery 19 │ 32.28 ms │                              31.22 ms │     no change │
   │ QQuery 20 │ 31.03 ms │                              35.57 ms │  1.15x slower │
   │ QQuery 21 │ 55.05 ms │                              54.08 ms │     no change │
   │ QQuery 22 │ 13.80 ms │                              15.36 ms │  1.11x slower │
   └───────────┴──────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 730.35ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 802.16ms │
   │ Average Time (HEAD)                                  │  33.20ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  36.46ms │
   │ Queries Faster                                       │        1 │
   │ Queries Slower                                       │        6 │
   │ Queries with No Change                               │       15 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.63 / 39.53 ±0.95 / 40.96 ms │        37.63 / 38.40 ±1.03 / 40.38 ms │    no change │
   │ QQuery 2  │ 18.98 / 19.31 ±0.32 / 19.91 ms │        21.30 / 21.60 ±0.25 / 21.99 ms │ 1.12x slower │
   │ QQuery 3  │ 30.23 / 30.92 ±1.05 / 33.01 ms │        30.66 / 32.31 ±1.23 / 33.48 ms │    no change │
   │ QQuery 4  │ 17.25 / 17.72 ±0.56 / 18.81 ms │        17.57 / 17.94 ±0.57 / 19.07 ms │    no change │
   │ QQuery 5  │ 39.08 / 40.34 ±1.21 / 41.97 ms │        36.97 / 40.24 ±1.95 / 43.12 ms │    no change │
   │ QQuery 6  │ 15.78 / 15.95 ±0.09 / 16.03 ms │        15.64 / 15.85 ±0.15 / 15.99 ms │    no change │
   │ QQuery 7  │ 43.72 / 45.87 ±1.80 / 48.49 ms │        43.27 / 44.91 ±1.15 / 46.54 ms │    no change │
   │ QQuery 8  │ 41.92 / 42.68 ±0.89 / 44.42 ms │        42.20 / 42.33 ±0.11 / 42.52 ms │    no change │
   │ QQuery 9  │ 49.25 / 50.21 ±0.97 / 51.78 ms │        49.21 / 50.77 ±1.12 / 52.40 ms │    no change │
   │ QQuery 10 │ 41.78 / 41.90 ±0.10 / 42.08 ms │        59.95 / 60.19 ±0.22 / 60.56 ms │ 1.44x slower │
   │ QQuery 11 │ 13.18 / 13.34 ±0.16 / 13.64 ms │        13.15 / 13.42 ±0.22 / 13.82 ms │    no change │
   │ QQuery 12 │ 23.88 / 24.27 ±0.27 / 24.55 ms │        23.35 / 23.76 ±0.49 / 24.71 ms │    no change │
   │ QQuery 13 │ 31.73 / 33.26 ±1.55 / 35.91 ms │        43.57 / 44.90 ±1.33 / 47.13 ms │ 1.35x slower │
   │ QQuery 14 │ 22.90 / 23.24 ±0.27 / 23.69 ms │        23.00 / 23.20 ±0.15 / 23.37 ms │    no change │
   │ QQuery 15 │ 30.21 / 30.76 ±0.58 / 31.79 ms │        30.08 / 30.80 ±0.73 / 32.12 ms │    no change │
   │ QQuery 16 │ 13.60 / 13.73 ±0.09 / 13.88 ms │        13.77 / 13.99 ±0.15 / 14.14 ms │    no change │
   │ QQuery 17 │ 68.77 / 70.35 ±1.10 / 71.88 ms │        70.08 / 71.44 ±1.37 / 73.95 ms │    no change │
   │ QQuery 18 │ 57.29 / 59.97 ±1.85 / 62.18 ms │        94.52 / 96.43 ±1.50 / 98.88 ms │ 1.61x slower │
   │ QQuery 19 │ 32.28 / 33.15 ±1.01 / 35.04 ms │        31.22 / 32.16 ±1.38 / 34.89 ms │    no change │
   │ QQuery 20 │ 31.03 / 31.60 ±0.65 / 32.83 ms │        35.57 / 36.11 ±0.41 / 36.59 ms │ 1.14x slower │
   │ QQuery 21 │ 55.05 / 57.33 ±2.05 / 60.62 ms │        54.08 / 55.03 ±0.70 / 56.14 ms │    no change │
   │ QQuery 22 │ 13.80 / 14.02 ±0.23 / 14.45 ms │        15.36 / 15.71 ±0.18 / 15.84 ms │ 1.12x slower │
   └───────────┴────────────────────────────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 749.45ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 821.49ms │
   │ Average Time (HEAD)                                  │  34.07ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  37.34ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        6 │
   │ Queries with No Change                               │       16 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.1 GiB |
   | Avg memory | 507.8 MiB |
   | CPU user | 21.3s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.0 GiB |
   | Avg memory | 644.1 MiB |
   | CPU user | 21.7s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5373795396

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5373757117)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5373757117-1846-n2q94 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (4865205f97c7ad3ef19d44e2e2275e14304c0d48) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..4865205f97c7ad3ef19d44e2e2275e14304c0d48)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675319962

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5675290885-2364-nvq2j 6.12.94+ #1 SMP Tue Aug  4 08:44:15 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f (cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f) to cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f [diff](https://github.com/apache/datafusion/compare/cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f..cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   env:
     DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
   baseline:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
   changed:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5988374620

   I can rebase this before we merge


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5483193779

   thank you for the review @kumarUjjawal, I implemented all of you suggestions in https://github.com/apache/datafusion/pull/24227/commits/6b75dcf3cc972c6f1a1c296bf781a7f28bd7c14f. everything related to the per-column allowlist has been removed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5345489826

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5345449022)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5345449022-1763-5nwjt 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (449355e7c5098104ce4868fa3ce347ca95a24ce3) to bbf739a (merge-base) [diff](https://github.com/apache/datafusion/compare/bbf739a3cfe1d5cc7646e37250c947c6a56b5b0e..449355e7c5098104ce4868fa3ce347ca95a24ce3)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3946482880


##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -135,6 +157,114 @@ pub fn apply_file_schema_type_coercions(
     ))
 }
 
+fn dictionary_value_type(data_type: &DataType) -> Option<&DataType> {
+    match data_type {
+        DataType::Dictionary(_, value_type) => Some(value_type.as_ref()),
+        _ => None,
+    }
+}
+
+// Find the value type that can represent both sides without narrowing offsets
+// or crossing string/binary families.
+fn common_dictionary_value_type(
+    field_type: &DataType,
+    dictionary_value_type: &DataType,
+) -> Option<DataType> {
+    let field_type = match field_type {
+        DataType::Dictionary(_, field_value_type) => field_value_type.as_ref(),
+        _ => field_type,
+    };
+
+    match (field_type, dictionary_value_type) {
+        (DataType::Utf8, DataType::Utf8) => Some(DataType::Utf8),
+        (DataType::Utf8 | DataType::LargeUtf8, DataType::Utf8 | DataType::LargeUtf8) => {
+            Some(DataType::LargeUtf8)
+        }
+        (DataType::Binary, DataType::Binary) => Some(DataType::Binary),
+        (
+            DataType::Binary | DataType::LargeBinary,
+            DataType::Binary | DataType::LargeBinary,
+        ) => Some(DataType::LargeBinary),
+        _ => None,
+    }
+}
+
+/// Allows safe widening into the table dictionary type.
+/// - `Utf8` to `Dictionary(Int32, LargeUtf8)`: allowed
+/// - `LargeBinary` to `Dictionary(Int32, Binary)`: rejected
+fn can_promote_to_dictionary_type(
+    file_field_type: &DataType,
+    table_dictionary_type: &DataType,
+) -> bool {
+    dictionary_value_type(table_dictionary_type).is_some_and(|dictionary_value_type| {
+        common_dictionary_value_type(file_field_type, dictionary_value_type)
+            .is_some_and(|common_type| &common_type == dictionary_value_type)
+    })
+}
+
+/// Normalize per-file schemas so that a column promoted to `Dictionary` in
+/// *any* file is promoted to the same `Dictionary` type in *all* files.
+///
+/// This lets [`Schema::try_merge`] accept directories that mix dictionary and
+/// plain encodings for the same column.
+pub(crate) fn uniform_dict_schemas(schemas: Vec<Schema>) -> Vec<Schema> {
+    // First pass: record the dictionary type for every column that is Dictionary in
+    // at least one schema.
+    let mut dict_types: HashMap<String, DataType> = HashMap::new();
+    for schema in &schemas {
+        for field in schema.fields() {
+            if matches!(field.data_type(), DataType::Dictionary(_, _)) {
+                dict_types
+                    .entry(field.name().clone())
+                    .or_insert_with(|| field.data_type().clone());
+            }
+        }
+    }
+    if dict_types.is_empty() {
+        return schemas;
+    }
+
+    // Widen the recorded dictionary value type before promoting plain fields.
+    for schema in &schemas {
+        for field in schema.fields() {
+            let Some(dict_type) = dict_types.get_mut(field.name()) else {
+                continue;
+            };
+            let DataType::Dictionary(key_type, value_type) = dict_type else {

Review Comment:
   https://github.com/apache/datafusion/pull/24227/changes/ffd6230868958af8ee2840bcb310a62dbabbcc12; `uniform_dict_schemas` now treats plain fields as having Int32 key capacity when computing the common key type (so Dict(Int8) + plain → Dict(Int32)), and `can_promote_to_dictionary_type` now rejects promotion of plain files to narrow-key dicts. Added `uniform_dict_schemas_plain_field_widens_key_to_int32` unit test and an execution-level test with 130 distinct values in the plain file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3750916670


##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -985,6 +992,91 @@ impl MetadataLoadedParquetOpen {
             metadata_dirty = true;
         }
 
+        // Detect dictionary-encoded string/binary columns and rewrite their Arrow types
+        // to Dictionary(Int32, ...) so the parquet reader produces dictionary arrays.
+        // All three schemas are updated together so the reader, expr-adapter, and
+        // downstream operators all agree on the output type.
+        if prepared.enable_rle_to_dictionary {

Review Comment:
   All three schemas are updated together so the reader, expr-adapter, and downstream operators all agree on the output type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5283980878

   > Why only RLE and not dictionaries as well? How does this compare to / relate to the schema_force_view_types option?
   
   my bad when I say RLE i'm referring to `RLE_DICTIONARY`
   
   > It also looks like this goes through infer_schema right? A lot of code paths never touch that (CREATE EXTERNAL TABLE (a VARCHAR), any custom table providers, etc.).
   
   > I'd be more interested in seeing something at the parquet scan level that was able to e.g. optimize how row filters are applied by applying them to the dictionary instead of expanding into Utf8View. That would be applicable to all DataFusion users.
   
   I agree, ill update the PR to target all parquet scans.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5365010866

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364868497)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f09f1555c30e2f0b88db32a80d7bcd87062bcc5d) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f09f1555c30e2f0b88db32a80d7bcd87062bcc5d)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 0  │    1.26 ms │                               1.21 ms │    no change │
   │ QQuery 1  │   11.73 ms │                              11.60 ms │    no change │
   │ QQuery 2  │   36.28 ms │                              35.80 ms │    no change │
   │ QQuery 3  │   31.35 ms │                              30.15 ms │    no change │
   │ QQuery 4  │  230.20 ms │                             221.98 ms │    no change │
   │ QQuery 5  │  284.02 ms │                            1454.07 ms │ 5.12x slower │
   │ QQuery 6  │    1.31 ms │                               1.42 ms │ 1.09x slower │
   │ QQuery 7  │   13.71 ms │                              14.00 ms │    no change │
   │ QQuery 8  │  329.16 ms │                             390.43 ms │ 1.19x slower │
   │ QQuery 9  │  451.49 ms │                             511.92 ms │ 1.13x slower │
   │ QQuery 10 │   70.40 ms │                             203.56 ms │ 2.89x slower │
   │ QQuery 11 │   81.16 ms │                             215.22 ms │ 2.65x slower │
   │ QQuery 12 │  267.62 ms │                            1414.18 ms │ 5.28x slower │
   │ QQuery 13 │  362.38 ms │                            2036.18 ms │ 5.62x slower │
   │ QQuery 14 │  280.61 ms │                            1600.63 ms │ 5.70x slower │
   │ QQuery 15 │  275.87 ms │                             284.36 ms │    no change │
   │ QQuery 16 │  612.29 ms │                            3425.81 ms │ 5.60x slower │
   │ QQuery 17 │  620.26 ms │                            3508.30 ms │ 5.66x slower │
   │ QQuery 18 │ 1244.06 ms │                            6519.44 ms │ 5.24x slower │
   │ QQuery 19 │   27.05 ms │                              27.24 ms │    no change │
   │ QQuery 20 │  515.80 ms │                            1040.32 ms │ 2.02x slower │
   │ QQuery 21 │  511.39 ms │                            1357.84 ms │ 2.66x slower │
   │ QQuery 22 │  986.03 ms │                            3281.49 ms │ 3.33x slower │
   │ QQuery 23 │ 3058.42 ms │                            9100.90 ms │ 2.98x slower │
   │ QQuery 24 │   43.22 ms │                             100.00 ms │ 2.31x slower │
   │ QQuery 25 │  113.20 ms │                             367.00 ms │ 3.24x slower │
   │ QQuery 26 │   41.40 ms │                              97.97 ms │ 2.37x slower │
   │ QQuery 27 │  510.71 ms │                            1126.05 ms │ 2.20x slower │
   │ QQuery 28 │ 2889.67 ms │                            3546.84 ms │ 1.23x slower │
   │ QQuery 29 │   40.65 ms │                              40.28 ms │    no change │
   │ QQuery 30 │  301.44 ms │                             535.87 ms │ 1.78x slower │
   │ QQuery 31 │  279.30 ms │                             524.52 ms │ 1.88x slower │
   │ QQuery 32 │  952.00 ms │                             989.87 ms │    no change │
   │ QQuery 33 │ 1415.89 ms │                           13736.54 ms │ 9.70x slower │
   │ QQuery 34 │ 1461.50 ms │                           13301.21 ms │ 9.10x slower │
   │ QQuery 35 │  282.11 ms │                             322.63 ms │ 1.14x slower │
   │ QQuery 36 │   66.08 ms │                             166.02 ms │ 2.51x slower │
   │ QQuery 37 │   36.05 ms │                              96.21 ms │ 2.67x slower │
   │ QQuery 38 │   43.34 ms │                              80.55 ms │ 1.86x slower │
   │ QQuery 39 │  139.63 ms │                             387.05 ms │ 2.77x slower │
   │ QQuery 40 │   13.99 ms │                              13.94 ms │    no change │
   │ QQuery 41 │   13.78 ms │                              13.65 ms │    no change │
   │ QQuery 42 │   12.94 ms │                              12.80 ms │    no change │
   └───────────┴────────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 18960.75ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 72147.05ms │
   │ Average Time (HEAD)                                  │   440.95ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  1677.84ms │
   │ Queries Faster                                       │          0 │
   │ Queries Slower                                       │         30 │
   │ Queries with No Change                               │         13 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                   HEAD ┃     rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │           1.26 / 3.93 ±5.31 / 14.54 ms │              1.21 / 3.88 ±5.28 / 14.45 ms │     no change │
   │ QQuery 1  │         11.73 / 11.98 ±0.17 / 12.21 ms │            11.60 / 11.72 ±0.10 / 11.90 ms │     no change │
   │ QQuery 2  │         36.28 / 36.78 ±0.44 / 37.51 ms │            35.80 / 36.05 ±0.26 / 36.53 ms │     no change │
   │ QQuery 3  │         31.35 / 31.78 ±0.58 / 32.91 ms │            30.15 / 31.46 ±1.59 / 34.58 ms │     no change │
   │ QQuery 4  │     230.20 / 240.56 ±11.98 / 262.85 ms │         221.98 / 226.02 ±2.55 / 229.76 ms │ +1.06x faster │
   │ QQuery 5  │      284.02 / 297.28 ±9.65 / 307.57 ms │    1454.07 / 1590.81 ±116.29 / 1784.17 ms │  5.35x slower │
   │ QQuery 6  │            1.31 / 1.45 ±0.22 / 1.88 ms │               1.42 / 1.58 ±0.20 / 1.96 ms │  1.09x slower │
   │ QQuery 7  │         13.71 / 13.88 ±0.19 / 14.23 ms │            14.00 / 14.19 ±0.13 / 14.34 ms │     no change │
   │ QQuery 8  │      329.16 / 331.04 ±2.48 / 335.94 ms │         390.43 / 402.10 ±9.89 / 418.58 ms │  1.21x slower │
   │ QQuery 9  │      451.49 / 458.79 ±4.63 / 463.77 ms │        511.92 / 556.61 ±28.71 / 593.71 ms │  1.21x slower │
   │ QQuery 10 │         70.40 / 71.45 ±1.25 / 73.83 ms │         203.56 / 207.61 ±4.84 / 217.12 ms │  2.91x slower │
   │ QQuery 11 │         81.16 / 81.50 ±0.24 / 81.85 ms │         215.22 / 216.04 ±0.83 / 217.52 ms │  2.65x slower │
   │ QQuery 12 │      267.62 / 275.60 ±6.27 / 283.21 ms │     1414.18 / 1519.73 ±73.01 / 1636.95 ms │  5.51x slower │
   │ QQuery 13 │     362.38 / 379.49 ±12.60 / 394.52 ms │    2036.18 / 2170.85 ±224.18 / 2618.37 ms │  5.72x slower │
   │ QQuery 14 │      280.61 / 288.97 ±6.96 / 296.41 ms │     1600.63 / 1640.45 ±45.14 / 1723.34 ms │  5.68x slower │
   │ QQuery 15 │      275.87 / 283.43 ±7.05 / 293.87 ms │         284.36 / 287.83 ±1.97 / 290.47 ms │     no change │
   │ QQuery 16 │     612.29 / 632.88 ±27.35 / 685.57 ms │    3425.81 / 3628.64 ±113.31 / 3748.77 ms │  5.73x slower │
   │ QQuery 17 │     620.26 / 644.25 ±19.58 / 665.97 ms │    3508.30 / 3740.74 ±190.48 / 4024.45 ms │  5.81x slower │
   │ QQuery 18 │  1244.06 / 1259.68 ±13.34 / 1282.62 ms │    6519.44 / 6842.11 ±254.08 / 7158.96 ms │  5.43x slower │
   │ QQuery 19 │        27.05 / 33.61 ±11.69 / 56.96 ms │            27.24 / 29.27 ±2.80 / 34.76 ms │ +1.15x faster │
   │ QQuery 20 │      515.80 / 524.42 ±8.88 / 535.31 ms │     1040.32 / 1062.00 ±15.55 / 1083.73 ms │  2.03x slower │
   │ QQuery 21 │     511.39 / 525.41 ±12.23 / 547.55 ms │     1357.84 / 1385.36 ±35.77 / 1455.65 ms │  2.64x slower │
   │ QQuery 22 │    986.03 / 999.21 ±10.53 / 1017.61 ms │     3281.49 / 3319.14 ±21.20 / 3341.81 ms │  3.32x slower │
   │ QQuery 23 │ 3058.42 / 3194.16 ±120.71 / 3348.73 ms │    9100.90 / 9431.05 ±288.15 / 9874.98 ms │  2.95x slower │
   │ QQuery 24 │         43.22 / 43.59 ±0.20 / 43.78 ms │        100.00 / 113.51 ±19.43 / 151.19 ms │  2.60x slower │
   │ QQuery 25 │      113.20 / 121.94 ±7.73 / 135.40 ms │         367.00 / 374.55 ±6.26 / 385.36 ms │  3.07x slower │
   │ QQuery 26 │         41.40 / 42.27 ±0.95 / 44.08 ms │            97.97 / 98.71 ±0.69 / 99.95 ms │  2.33x slower │
   │ QQuery 27 │      510.71 / 515.53 ±3.74 / 521.47 ms │      1126.05 / 1141.24 ±8.65 / 1150.14 ms │  2.21x slower │
   │ QQuery 28 │  2889.67 / 2920.32 ±33.53 / 2979.83 ms │     3546.84 / 3572.85 ±26.06 / 3616.59 ms │  1.22x slower │
   │ QQuery 29 │        40.65 / 49.15 ±12.88 / 74.15 ms │           40.28 / 49.70 ±16.55 / 82.75 ms │     no change │
   │ QQuery 30 │      301.44 / 310.40 ±8.43 / 322.92 ms │        535.87 / 556.93 ±14.30 / 578.79 ms │  1.79x slower │
   │ QQuery 31 │     279.30 / 293.35 ±10.72 / 310.81 ms │        524.52 / 562.68 ±28.78 / 610.23 ms │  1.92x slower │
   │ QQuery 32 │    952.00 / 999.43 ±43.39 / 1081.21 ms │      989.87 / 1023.65 ±20.33 / 1049.18 ms │     no change │
   │ QQuery 33 │  1415.89 / 1476.27 ±44.68 / 1549.90 ms │ 13736.54 / 14435.38 ±551.53 / 15374.57 ms │  9.78x slower │
   │ QQuery 34 │  1461.50 / 1522.67 ±68.92 / 1645.54 ms │ 13301.21 / 13876.53 ±379.62 / 14334.65 ms │  9.11x slower │
   │ QQuery 35 │     282.11 / 319.06 ±39.61 / 393.22 ms │        322.63 / 375.97 ±46.46 / 457.83 ms │  1.18x slower │
   │ QQuery 36 │         66.08 / 72.57 ±6.49 / 83.03 ms │        166.02 / 193.85 ±26.81 / 238.90 ms │  2.67x slower │
   │ QQuery 37 │         36.05 / 45.21 ±9.20 / 60.91 ms │          96.21 / 104.08 ±7.65 / 118.29 ms │  2.30x slower │
   │ QQuery 38 │         43.34 / 46.08 ±1.69 / 48.20 ms │            80.55 / 82.45 ±1.39 / 84.62 ms │  1.79x slower │
   │ QQuery 39 │      139.63 / 155.32 ±9.60 / 168.96 ms │         387.05 / 398.55 ±6.21 / 405.61 ms │  2.57x slower │
   │ QQuery 40 │         13.99 / 14.59 ±0.42 / 15.26 ms │            13.94 / 14.64 ±0.45 / 15.17 ms │     no change │
   │ QQuery 41 │         13.78 / 15.48 ±2.73 / 20.93 ms │            13.65 / 14.48 ±1.47 / 17.41 ms │ +1.07x faster │
   │ QQuery 42 │         12.94 / 13.24 ±0.26 / 13.69 ms │            12.80 / 14.52 ±2.76 / 20.03 ms │  1.10x slower │
   └───────────┴────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 19598.02ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 75359.49ms │
   │ Average Time (HEAD)                                  │   455.77ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  1752.55ms │
   │ Queries Faster                                       │          3 │
   │ Queries Slower                                       │         31 │
   │ Queries with No Change                               │          9 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 10.7 GiB |
   | Avg memory | 4.4 GiB |
   | CPU user | 1003.4s |
   | CPU sys | 70.0s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 380.1s |
   | Peak memory | 12.7 GiB |
   | Avg memory | 5.8 GiB |
   | CPU user | 3325.5s |
   | CPU sys | 200.0s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364948872

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364868497)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f09f1555c30e2f0b88db32a80d7bcd87062bcc5d) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f09f1555c30e2f0b88db32a80d7bcd87062bcc5d)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 40.20 ms │                              39.67 ms │    no change │
   │ QQuery 2  │ 20.60 ms │                              22.90 ms │ 1.11x slower │
   │ QQuery 3  │ 32.56 ms │                              32.44 ms │    no change │
   │ QQuery 4  │ 18.70 ms │                              18.42 ms │    no change │
   │ QQuery 5  │ 43.04 ms │                              41.07 ms │    no change │
   │ QQuery 6  │ 16.71 ms │                              16.72 ms │    no change │
   │ QQuery 7  │ 48.79 ms │                              47.90 ms │    no change │
   │ QQuery 8  │ 44.09 ms │                              44.57 ms │    no change │
   │ QQuery 9  │ 51.91 ms │                              53.77 ms │    no change │
   │ QQuery 10 │ 44.83 ms │                              65.37 ms │ 1.46x slower │
   │ QQuery 11 │ 14.52 ms │                              14.70 ms │    no change │
   │ QQuery 12 │ 25.07 ms │                              25.03 ms │    no change │
   │ QQuery 13 │ 33.43 ms │                              45.60 ms │ 1.36x slower │
   │ QQuery 14 │ 24.42 ms │                              24.63 ms │    no change │
   │ QQuery 15 │ 32.36 ms │                              33.02 ms │    no change │
   │ QQuery 16 │ 14.77 ms │                              14.76 ms │    no change │
   │ QQuery 17 │ 79.40 ms │                              78.68 ms │    no change │
   │ QQuery 18 │ 64.07 ms │                             106.61 ms │ 1.66x slower │
   │ QQuery 19 │ 34.02 ms │                              33.34 ms │    no change │
   │ QQuery 20 │ 32.75 ms │                              38.35 ms │ 1.17x slower │
   │ QQuery 21 │ 59.85 ms │                              59.55 ms │    no change │
   │ QQuery 22 │ 15.37 ms │                              16.60 ms │ 1.08x slower │
   └───────────┴──────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 791.45ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 873.72ms │
   │ Average Time (HEAD)                                  │  35.97ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  39.71ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        6 │
   │ Queries with No Change                               │       16 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 40.20 / 41.61 ±1.47 / 43.96 ms │        39.67 / 41.09 ±1.20 / 42.72 ms │    no change │
   │ QQuery 2  │ 20.60 / 20.82 ±0.13 / 20.97 ms │        22.90 / 23.29 ±0.58 / 24.43 ms │ 1.12x slower │
   │ QQuery 3  │ 32.56 / 33.46 ±0.97 / 35.18 ms │        32.44 / 33.71 ±1.32 / 36.21 ms │    no change │
   │ QQuery 4  │ 18.70 / 18.83 ±0.13 / 19.08 ms │        18.42 / 19.19 ±0.94 / 20.79 ms │    no change │
   │ QQuery 5  │ 43.04 / 43.27 ±0.19 / 43.51 ms │        41.07 / 43.01 ±1.14 / 44.37 ms │    no change │
   │ QQuery 6  │ 16.71 / 16.99 ±0.24 / 17.42 ms │        16.72 / 17.08 ±0.55 / 18.17 ms │    no change │
   │ QQuery 7  │ 48.79 / 49.70 ±0.71 / 50.91 ms │        47.90 / 50.12 ±1.88 / 53.01 ms │    no change │
   │ QQuery 8  │ 44.09 / 44.69 ±0.44 / 45.29 ms │        44.57 / 44.98 ±0.45 / 45.85 ms │    no change │
   │ QQuery 9  │ 51.91 / 52.64 ±0.41 / 53.02 ms │        53.77 / 54.91 ±1.27 / 57.31 ms │    no change │
   │ QQuery 10 │ 44.83 / 46.10 ±1.20 / 48.07 ms │        65.37 / 66.34 ±1.00 / 68.00 ms │ 1.44x slower │
   │ QQuery 11 │ 14.52 / 14.66 ±0.17 / 14.99 ms │        14.70 / 15.43 ±1.24 / 17.91 ms │ 1.05x slower │
   │ QQuery 12 │ 25.07 / 25.57 ±0.61 / 26.74 ms │        25.03 / 25.26 ±0.17 / 25.44 ms │    no change │
   │ QQuery 13 │ 33.43 / 35.37 ±1.61 / 37.34 ms │        45.60 / 46.52 ±1.09 / 48.66 ms │ 1.32x slower │
   │ QQuery 14 │ 24.42 / 25.12 ±1.01 / 27.10 ms │        24.63 / 24.71 ±0.07 / 24.83 ms │    no change │
   │ QQuery 15 │ 32.36 / 33.22 ±1.02 / 34.70 ms │        33.02 / 33.27 ±0.34 / 33.94 ms │    no change │
   │ QQuery 16 │ 14.77 / 14.90 ±0.12 / 15.11 ms │        14.76 / 14.95 ±0.14 / 15.16 ms │    no change │
   │ QQuery 17 │ 79.40 / 83.06 ±2.54 / 87.34 ms │        78.68 / 81.34 ±2.84 / 86.86 ms │    no change │
   │ QQuery 18 │ 64.07 / 65.77 ±0.93 / 66.91 ms │     106.61 / 108.48 ±1.54 / 110.92 ms │ 1.65x slower │
   │ QQuery 19 │ 34.02 / 35.24 ±1.37 / 37.20 ms │        33.34 / 33.81 ±0.42 / 34.59 ms │    no change │
   │ QQuery 20 │ 32.75 / 33.98 ±1.01 / 35.69 ms │        38.35 / 39.13 ±0.77 / 40.57 ms │ 1.15x slower │
   │ QQuery 21 │ 59.85 / 62.60 ±2.06 / 66.23 ms │        59.55 / 61.12 ±0.91 / 62.05 ms │    no change │
   │ QQuery 22 │ 15.37 / 16.98 ±1.82 / 20.19 ms │        16.60 / 16.94 ±0.20 / 17.22 ms │    no change │
   └───────────┴────────────────────────────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 814.58ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 894.68ms │
   │ Average Time (HEAD)                                  │  37.03ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  40.67ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        6 │
   │ Queries with No Change                               │       16 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.3 GiB |
   | Avg memory | 700.6 MiB |
   | CPU user | 23.3s |
   | CPU sys | 2.1s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.1 GiB |
   | Avg memory | 665.9 MiB |
   | CPU user | 21.8s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5400223262

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5400020690)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8c1ada76c5bf9da164594ac9ff6609c885) to 26b40dd (merge-base) [diff](https://github.com/apache/datafusion/compare/26b40dd781d2eeab1acc38370a7069a8b9ebe450..35dd8d8c1ada76c5bf9da164594ac9ff6609c885)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.19 ms │                              39.75 ms │    no change │
   │ QQuery 2  │ 18.34 ms │                              19.50 ms │ 1.06x slower │
   │ QQuery 3  │ 28.72 ms │                              30.42 ms │ 1.06x slower │
   │ QQuery 4  │ 17.31 ms │                              18.39 ms │ 1.06x slower │
   │ QQuery 5  │ 35.42 ms │                              39.11 ms │ 1.10x slower │
   │ QQuery 6  │ 15.89 ms │                              16.90 ms │ 1.06x slower │
   │ QQuery 7  │ 39.17 ms │                              45.43 ms │ 1.16x slower │
   │ QQuery 8  │ 40.63 ms │                              43.25 ms │ 1.06x slower │
   │ QQuery 9  │ 47.93 ms │                              53.46 ms │ 1.12x slower │
   │ QQuery 10 │ 41.34 ms │                              45.32 ms │ 1.10x slower │
   │ QQuery 11 │ 13.28 ms │                              14.68 ms │ 1.11x slower │
   │ QQuery 12 │ 23.96 ms │                              25.47 ms │ 1.06x slower │
   │ QQuery 13 │ 38.88 ms │                              46.54 ms │ 1.20x slower │
   │ QQuery 14 │ 24.10 ms │                              26.18 ms │ 1.09x slower │
   │ QQuery 15 │ 30.37 ms │                              33.09 ms │ 1.09x slower │
   │ QQuery 16 │ 13.64 ms │                              14.71 ms │ 1.08x slower │
   │ QQuery 17 │ 69.18 ms │                              76.18 ms │ 1.10x slower │
   │ QQuery 18 │ 58.52 ms │                              62.46 ms │ 1.07x slower │
   │ QQuery 19 │ 32.48 ms │                              33.57 ms │    no change │
   │ QQuery 20 │ 31.09 ms │                              32.46 ms │    no change │
   │ QQuery 21 │ 54.48 ms │                              56.53 ms │    no change │
   │ QQuery 22 │ 13.31 ms │                              14.78 ms │ 1.11x slower │
   └───────────┴──────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 726.21ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 788.19ms │
   │ Average Time (HEAD)                                  │  33.01ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  35.83ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │       18 │
   │ Queries with No Change                               │        4 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.19 / 39.54 ±1.90 / 43.30 ms │        39.75 / 40.42 ±1.03 / 42.48 ms │    no change │
   │ QQuery 2  │ 18.34 / 18.86 ±0.47 / 19.57 ms │        19.50 / 19.71 ±0.20 / 20.06 ms │    no change │
   │ QQuery 3  │ 28.72 / 29.76 ±0.89 / 31.18 ms │        30.42 / 30.70 ±0.33 / 31.33 ms │    no change │
   │ QQuery 4  │ 17.31 / 18.03 ±0.58 / 18.94 ms │        18.39 / 18.55 ±0.13 / 18.74 ms │    no change │
   │ QQuery 5  │ 35.42 / 35.77 ±0.36 / 36.44 ms │        39.11 / 39.35 ±0.25 / 39.75 ms │ 1.10x slower │
   │ QQuery 6  │ 15.89 / 16.27 ±0.43 / 16.96 ms │        16.90 / 17.32 ±0.40 / 18.02 ms │ 1.06x slower │
   │ QQuery 7  │ 39.17 / 41.98 ±1.67 / 43.86 ms │        45.43 / 47.43 ±1.19 / 48.80 ms │ 1.13x slower │
   │ QQuery 8  │ 40.63 / 41.46 ±1.25 / 43.94 ms │        43.25 / 43.76 ±0.47 / 44.61 ms │ 1.06x slower │
   │ QQuery 9  │ 47.93 / 49.02 ±1.15 / 51.11 ms │        53.46 / 53.99 ±0.44 / 54.72 ms │ 1.10x slower │
   │ QQuery 10 │ 41.34 / 41.59 ±0.17 / 41.84 ms │        45.32 / 45.82 ±0.42 / 46.45 ms │ 1.10x slower │
   │ QQuery 11 │ 13.28 / 13.87 ±0.62 / 14.93 ms │        14.68 / 14.80 ±0.07 / 14.87 ms │ 1.07x slower │
   │ QQuery 12 │ 23.96 / 24.29 ±0.23 / 24.54 ms │        25.47 / 25.84 ±0.22 / 26.07 ms │ 1.06x slower │
   │ QQuery 13 │ 38.88 / 40.52 ±1.57 / 42.86 ms │        46.54 / 46.89 ±0.33 / 47.50 ms │ 1.16x slower │
   │ QQuery 14 │ 24.10 / 24.36 ±0.16 / 24.57 ms │        26.18 / 26.31 ±0.10 / 26.43 ms │ 1.08x slower │
   │ QQuery 15 │ 30.37 / 31.77 ±1.22 / 33.77 ms │        33.09 / 33.55 ±0.35 / 34.00 ms │ 1.06x slower │
   │ QQuery 16 │ 13.64 / 13.86 ±0.18 / 14.15 ms │        14.71 / 14.90 ±0.10 / 14.99 ms │ 1.08x slower │
   │ QQuery 17 │ 69.18 / 70.82 ±2.21 / 75.19 ms │        76.18 / 81.17 ±2.94 / 84.30 ms │ 1.15x slower │
   │ QQuery 18 │ 58.52 / 60.48 ±2.46 / 65.29 ms │        62.46 / 64.00 ±1.03 / 65.66 ms │ 1.06x slower │
   │ QQuery 19 │ 32.48 / 32.76 ±0.23 / 33.08 ms │        33.57 / 33.93 ±0.31 / 34.49 ms │    no change │
   │ QQuery 20 │ 31.09 / 31.72 ±0.83 / 33.35 ms │        32.46 / 33.04 ±0.45 / 33.79 ms │    no change │
   │ QQuery 21 │ 54.48 / 55.10 ±0.46 / 55.78 ms │        56.53 / 57.14 ±0.64 / 58.06 ms │    no change │
   │ QQuery 22 │ 13.31 / 13.69 ±0.31 / 14.19 ms │        14.78 / 15.38 ±0.82 / 16.99 ms │ 1.12x slower │
   └───────────┴────────────────────────────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 745.52ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 803.99ms │
   │ Average Time (HEAD)                                  │  33.89ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  36.55ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │       15 │
   │ Queries with No Change                               │        7 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.3 GiB |
   | Avg memory | 514.7 MiB |
   | CPU user | 20.6s |
   | CPU sys | 1.7s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.0 GiB |
   | Avg memory | 622.4 MiB |
   | CPU user | 22.9s |
   | CPU sys | 1.9s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5698213418

   > These results reproduce the earlier performance direction. I think the implementation needs to avoid automatically promoting high-cardinality or dictionary-fallback columns.
   
   @kumarUjjawal  That is expected with this implementation. The larger epic #24111, specifically #24117, will handle deciding on a per-query basis whether to read the column in as Dictionary or as Utf8/Utf8View. The point of this feature flag is to let users enable direct RLE_DICTIONARY reads from Parquet files in DataFusion, for cases where they know the shape of their data and query well enough to make that decision themselves. By default, this flag will be turned off.
   
   This line of thinking is in line with what alamb mentioned here https://github.com/apache/datafusion/issues/24111#issuecomment-5206277128


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangb (via GitHub)" <gi...@apache.org>]

adriangb commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5280625720

   If I understand correctly the goal is to evaluate filters during filter pushdown against dictionary / RLE encoded columns? We can't propagate these dynamic type changes to the rest of the query plan / scan. Is that right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5400052065

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5400020690)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5400020690-1906-gcg7g 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8c1ada76c5bf9da164594ac9ff6609c885) to 26b40dd (merge-base) [diff](https://github.com/apache/datafusion/compare/26b40dd781d2eeab1acc38370a7069a8b9ebe450..35dd8d8c1ada76c5bf9da164594ac9ff6609c885)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5989697061

   > I can rebase this before we merge
   
   Please resolve the conflicts.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5300656379

   > How does this compare to / relate to the schema_force_view_types option?
   
   `utf8` columns become `dict<_,utf8` so the default string type for parquet columns `utf8View` is replaced.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5345487574

   Benchmark for [this request](https://github.com/apache/datafusion/pull/24227#issuecomment-5345449022) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
   
   Benchmarks requested: `clickbench_partitioned`
   
   <details><summary>Kubernetes message</summary>
   
   ```
   Job has reached the specified backoff limit
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5345449022

   run benchmarks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675325542

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5675290885-2367-l2clt 6.12.94+ #1 SMP Tue Aug  4 08:44:15 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f (cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f) to cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f [diff](https://github.com/apache/datafusion/compare/cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f..cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark wide_schema
   env:
     DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
   baseline:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
   changed:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675528213

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f (cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f) to cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f [diff](https://github.com/apache/datafusion/compare/cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f..cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   env:
     DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
   baseline:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
   changed:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │    1.24 ms │                               1.19 ms │     no change │
   │ QQuery 1  │   11.94 ms │                              11.29 ms │ +1.06x faster │
   │ QQuery 2  │   36.70 ms │                              35.80 ms │     no change │
   │ QQuery 3  │   31.04 ms │                              30.26 ms │     no change │
   │ QQuery 4  │  220.65 ms │                             221.82 ms │     no change │
   │ QQuery 5  │  270.95 ms │                            1425.44 ms │  5.26x slower │
   │ QQuery 6  │    1.27 ms │                               1.24 ms │     no change │
   │ QQuery 7  │   13.13 ms │                              12.64 ms │     no change │
   │ QQuery 8  │  327.66 ms │                             339.50 ms │     no change │
   │ QQuery 9  │  455.32 ms │                             465.80 ms │     no change │
   │ QQuery 10 │   68.78 ms │                             202.13 ms │  2.94x slower │
   │ QQuery 11 │   80.61 ms │                             214.29 ms │  2.66x slower │
   │ QQuery 12 │  262.65 ms │                            1417.02 ms │  5.40x slower │
   │ QQuery 13 │  366.14 ms │                            2019.15 ms │  5.51x slower │
   │ QQuery 14 │  285.97 ms │                            1550.45 ms │  5.42x slower │
   │ QQuery 15 │  269.41 ms │                             277.13 ms │     no change │
   │ QQuery 16 │  621.05 ms │                            3491.74 ms │  5.62x slower │
   │ QQuery 17 │  623.43 ms │                            3556.36 ms │  5.70x slower │
   │ QQuery 18 │ 1269.91 ms │                            5992.72 ms │  4.72x slower │
   │ QQuery 19 │   27.11 ms │                              26.65 ms │     no change │
   │ QQuery 20 │  513.47 ms │                            1033.26 ms │  2.01x slower │
   │ QQuery 21 │  511.27 ms │                            1364.27 ms │  2.67x slower │
   │ QQuery 22 │  981.59 ms │                            3169.13 ms │  3.23x slower │
   │ QQuery 23 │ 3029.35 ms │                            9163.52 ms │  3.02x slower │
   │ QQuery 24 │   40.70 ms │                              96.23 ms │  2.36x slower │
   │ QQuery 25 │  110.45 ms │                             355.91 ms │  3.22x slower │
   │ QQuery 26 │   41.34 ms │                              96.50 ms │  2.33x slower │
   │ QQuery 27 │  513.69 ms │                            1127.43 ms │  2.19x slower │
   │ QQuery 28 │ 2880.62 ms │                            3537.71 ms │  1.23x slower │
   │ QQuery 29 │   41.45 ms │                              40.72 ms │     no change │
   │ QQuery 30 │  310.83 ms │                             547.97 ms │  1.76x slower │
   │ QQuery 31 │  286.09 ms │                             543.16 ms │  1.90x slower │
   │ QQuery 32 │  936.76 ms │                            1044.47 ms │  1.11x slower │
   │ QQuery 33 │ 1459.69 ms │                           13430.75 ms │  9.20x slower │
   │ QQuery 34 │ 1485.67 ms │                           13273.73 ms │  8.93x slower │
   │ QQuery 35 │  275.82 ms │                             284.74 ms │     no change │
   │ QQuery 36 │   67.62 ms │                             167.07 ms │  2.47x slower │
   │ QQuery 37 │   35.52 ms │                              96.46 ms │  2.72x slower │
   │ QQuery 38 │   42.23 ms │                              80.22 ms │  1.90x slower │
   │ QQuery 39 │  149.85 ms │                             375.56 ms │  2.51x slower │
   │ QQuery 40 │   13.91 ms │                              14.12 ms │     no change │
   │ QQuery 41 │   13.58 ms │                              13.69 ms │     no change │
   │ QQuery 42 │   12.98 ms │                              13.09 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 18999.43ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 71162.33ms │
   │ Average Time (HEAD)                                  │   441.85ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  1654.94ms │
   │ Queries Faster                                       │          1 │
   │ Queries Slower                                       │         27 │
   │ Queries with No Change                               │         15 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃     rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.24 / 4.00 ±5.44 / 14.89 ms │              1.19 / 3.92 ±5.39 / 14.70 ms │     no change │
   │ QQuery 1  │        11.94 / 11.99 ±0.05 / 12.09 ms │            11.29 / 11.43 ±0.08 / 11.50 ms │     no change │
   │ QQuery 2  │        36.70 / 37.03 ±0.24 / 37.34 ms │            35.80 / 36.17 ±0.36 / 36.86 ms │     no change │
   │ QQuery 3  │        31.04 / 31.55 ±0.49 / 32.43 ms │            30.26 / 30.80 ±0.57 / 31.87 ms │     no change │
   │ QQuery 4  │     220.65 / 225.04 ±3.32 / 229.90 ms │         221.82 / 224.77 ±2.53 / 229.44 ms │     no change │
   │ QQuery 5  │     270.95 / 275.22 ±3.18 / 280.63 ms │     1425.44 / 1488.69 ±42.65 / 1539.99 ms │  5.41x slower │
   │ QQuery 6  │           1.27 / 1.42 ±0.21 / 1.82 ms │               1.24 / 1.39 ±0.23 / 1.83 ms │     no change │
   │ QQuery 7  │        13.13 / 13.45 ±0.35 / 14.00 ms │            12.64 / 12.67 ±0.03 / 12.71 ms │ +1.06x faster │
   │ QQuery 8  │     327.66 / 331.61 ±2.50 / 334.06 ms │         339.50 / 345.94 ±6.73 / 358.01 ms │     no change │
   │ QQuery 9  │     455.32 / 460.31 ±5.67 / 470.67 ms │        465.80 / 494.00 ±17.60 / 521.36 ms │  1.07x slower │
   │ QQuery 10 │        68.78 / 70.08 ±1.45 / 72.84 ms │         202.13 / 203.71 ±1.47 / 206.48 ms │  2.91x slower │
   │ QQuery 11 │        80.61 / 80.95 ±0.48 / 81.90 ms │         214.29 / 214.46 ±0.19 / 214.78 ms │  2.65x slower │
   │ QQuery 12 │     262.65 / 268.27 ±5.23 / 278.10 ms │     1417.02 / 1504.42 ±67.45 / 1599.47 ms │  5.61x slower │
   │ QQuery 13 │     366.14 / 373.33 ±4.05 / 377.47 ms │    2019.15 / 2148.83 ±188.35 / 2520.03 ms │  5.76x slower │
   │ QQuery 14 │     285.97 / 289.47 ±2.93 / 293.79 ms │     1550.45 / 1597.18 ±28.19 / 1633.28 ms │  5.52x slower │
   │ QQuery 15 │    269.41 / 282.72 ±11.78 / 304.61 ms │         277.13 / 285.27 ±8.19 / 297.78 ms │     no change │
   │ QQuery 16 │     621.05 / 626.45 ±4.17 / 633.49 ms │     3491.74 / 3559.90 ±80.16 / 3702.28 ms │  5.68x slower │
   │ QQuery 17 │     623.43 / 630.44 ±3.84 / 635.08 ms │     3556.36 / 3665.36 ±92.61 / 3813.58 ms │  5.81x slower │
   │ QQuery 18 │ 1269.91 / 1307.75 ±33.11 / 1360.19 ms │    5992.72 / 6539.67 ±379.19 / 7140.70 ms │  5.00x slower │
   │ QQuery 19 │        27.11 / 31.81 ±5.12 / 39.86 ms │            26.65 / 27.14 ±0.47 / 28.02 ms │ +1.17x faster │
   │ QQuery 20 │    513.47 / 530.50 ±23.66 / 576.04 ms │      1033.26 / 1042.76 ±5.23 / 1048.63 ms │  1.97x slower │
   │ QQuery 21 │     511.27 / 515.56 ±3.72 / 521.35 ms │      1364.27 / 1373.22 ±6.19 / 1381.60 ms │  2.66x slower │
   │ QQuery 22 │     981.59 / 987.94 ±6.19 / 997.70 ms │     3169.13 / 3228.96 ±33.79 / 3265.64 ms │  3.27x slower │
   │ QQuery 23 │ 3029.35 / 3098.47 ±38.90 / 3145.61 ms │    9163.52 / 9320.42 ±185.21 / 9672.74 ms │  3.01x slower │
   │ QQuery 24 │        40.70 / 43.04 ±1.31 / 44.58 ms │         96.23 / 140.17 ±68.94 / 274.85 ms │  3.26x slower │
   │ QQuery 25 │     110.45 / 111.00 ±0.51 / 111.77 ms │         355.91 / 361.55 ±4.19 / 367.96 ms │  3.26x slower │
   │ QQuery 26 │        41.34 / 42.55 ±1.75 / 45.97 ms │            96.50 / 97.73 ±0.82 / 98.59 ms │  2.30x slower │
   │ QQuery 27 │     513.69 / 522.98 ±8.02 / 537.77 ms │     1127.43 / 1149.56 ±15.26 / 1174.73 ms │  2.20x slower │
   │ QQuery 28 │ 2880.62 / 2922.14 ±24.29 / 2956.05 ms │     3537.71 / 3581.56 ±27.07 / 3617.84 ms │  1.23x slower │
   │ QQuery 29 │        41.45 / 42.26 ±0.81 / 43.41 ms │            40.72 / 41.13 ±0.37 / 41.80 ms │     no change │
   │ QQuery 30 │     310.83 / 317.22 ±7.06 / 330.27 ms │         547.97 / 556.28 ±8.75 / 572.99 ms │  1.75x slower │
   │ QQuery 31 │    286.09 / 299.78 ±11.55 / 319.08 ms │         543.16 / 545.89 ±2.74 / 550.54 ms │  1.82x slower │
   │ QQuery 32 │    936.76 / 965.19 ±23.64 / 996.70 ms │     1044.47 / 1100.67 ±47.54 / 1165.52 ms │  1.14x slower │
   │ QQuery 33 │ 1459.69 / 1482.61 ±20.62 / 1514.47 ms │ 13430.75 / 14018.27 ±608.62 / 15064.74 ms │  9.46x slower │
   │ QQuery 34 │ 1485.67 / 1508.44 ±28.22 / 1563.64 ms │ 13273.73 / 14131.52 ±811.84 / 15262.78 ms │  9.37x slower │
   │ QQuery 35 │    275.82 / 297.94 ±28.81 / 354.66 ms │        284.74 / 323.69 ±73.30 / 470.23 ms │  1.09x slower │
   │ QQuery 36 │        67.62 / 71.34 ±2.71 / 74.41 ms │        167.07 / 180.55 ±10.40 / 190.42 ms │  2.53x slower │
   │ QQuery 37 │        35.52 / 38.43 ±3.55 / 45.40 ms │            96.46 / 97.58 ±1.04 / 99.30 ms │  2.54x slower │
   │ QQuery 38 │        42.23 / 46.74 ±4.91 / 55.12 ms │            80.22 / 85.49 ±4.45 / 91.16 ms │  1.83x slower │
   │ QQuery 39 │     149.85 / 153.88 ±2.77 / 157.12 ms │        375.56 / 391.25 ±11.11 / 410.07 ms │  2.54x slower │
   │ QQuery 40 │        13.91 / 18.35 ±3.63 / 24.26 ms │            14.12 / 15.24 ±1.24 / 17.65 ms │ +1.20x faster │
   │ QQuery 41 │        13.58 / 18.95 ±5.12 / 25.24 ms │            13.69 / 13.92 ±0.28 / 14.48 ms │ +1.36x faster │
   │ QQuery 42 │        12.98 / 14.20 ±1.94 / 18.06 ms │            13.09 / 17.76 ±9.01 / 35.77 ms │  1.25x slower │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 19402.39ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 74210.94ms │
   │ Average Time (HEAD)                                  │   451.22ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  1725.84ms │
   │ Queries Faster                                       │          4 │
   │ Queries Slower                                       │         30 │
   │ Queries with No Change                               │          9 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Memory Pool Peaks</summary>
   
   Peak `MemoryPool` reservation per query — what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
   
   Base: `cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f` | Changed: `cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f`
   
   **`clickbench_partitioned`**
   
   | Query | Base | Changed | Change |
   | --- | --- | --- | --- |
   | Query 0 | 0 B | 0 B | 0.0% |
   | Query 1 | 104 B | 104 B | +0.0% |
   | Query 2 | 936 B | 936 B | +0.0% |
   | Query 3 | 312 B | 312 B | +0.0% |
   | Query 4 | 761.4 MiB | 755.7 MiB | -0.7% |
   | Query 5 | 1.2 GiB | 1.5 GiB | +21.9% |
   | Query 6 | 0 B | 0 B | 0.0% |
   | Query 7 | 60.1 MiB | 60.0 MiB | -0.1% |
   | Query 8 | 965.7 MiB | 942.0 MiB | -2.5% |
   | Query 9 | 593.5 MiB | 551.9 MiB | -7.0% |
   | Query 10 | 115.5 MiB | 70.2 MiB | -39.2% |
   | Query 11 | 122.1 MiB | 77.1 MiB | -36.9% |
   | Query 12 | 1.3 GiB | 1.6 GiB | +17.9% |
   | Query 13 | 1.6 GiB | 1.8 GiB | +14.6% |
   | Query 14 | 1.3 GiB | 1.9 GiB | +47.3% |
   | Query 15 | 1.1 GiB | 1.1 GiB | -3.7% |
   | Query 16 | 3.4 GiB | 3.3 GiB | -2.4% |
   | Query 17 | 3.4 GiB | 3.4 GiB | -0.8% |
   | Query 18 | 8.1 GiB | 7.4 GiB | -8.6% |
   | Query 19 | 0 B | 0 B | 0.0% |
   | Query 20 | 104 B | 104 B | +0.0% |
   | Query 21 | 3.3 MiB | 1.3 MiB | -60.6% |
   | Query 22 | 3.0 MiB | 2.5 MiB | -17.4% |
   | Query 23 | 32.2 MiB | 12.0 MiB | -62.7% |
   | Query 24 | 60.3 MiB | 6.2 MiB | -89.8% |
   | Query 25 | 173.5 MiB | 8.1 MiB | -95.3% |
   | Query 26 | 60.3 MiB | 7.3 MiB | -87.8% |
   | Query 27 | 2.3 MiB | 2.4 MiB | +3.8% |
   | Query 28 | 1.7 GiB | 1.7 GiB | -0.6% |
   | Query 29 | 624 B | 624 B | +0.0% |
   | Query 30 | 678.9 MiB | 648.4 MiB | -4.5% |
   | Query 31 | 1.5 GiB | 1.6 GiB | +12.8% |
   | Query 32 | 7.1 GiB | 7.1 GiB | +0.2% |
   | Query 33 | 8.1 GiB | 12.3 GiB | +52.3% |
   | Query 34 | 8.2 GiB | 12.7 GiB | +54.2% |
   | Query 35 | 603.9 MiB | 598.9 MiB | -0.8% |
   | Query 36 | 114.4 MiB | 473.3 MiB | +313.6% |
   | Query 37 | 6.3 MiB | 7.7 MiB | +23.6% |
   | Query 38 | 5.2 MiB | 5.3 MiB | +1.7% |
   | Query 39 | 297.8 MiB | 450.0 MiB | +51.1% |
   | Query 40 | 2.0 MiB | 2.3 MiB | +14.5% |
   | Query 41 | 3.1 MiB | 3.1 MiB | -0.0% |
   | Query 42 | 1.7 MiB | 1.7 MiB | +0.0% |
   
   **Pool accounting vs. process RSS**
   
   Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
   
   | Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
   | --- | --- | --- | --- | --- | --- |
   | clickbench_partitioned | base (`cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f`) | 8.2 GiB | 11.0 GiB | 2.7 GiB | 1.3× |
   | clickbench_partitioned | changed (`cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f`) | 12.7 GiB | 12.4 GiB | −323.7 MiB | 1.0× |
   
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 11.0 GiB |
   | Avg memory | 4.1 GiB |
   | CPU user | 993.1s |
   | CPU sys | 70.2s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 375.1s |
   | Peak memory | 12.4 GiB |
   | Avg memory | 5.8 GiB |
   | CPU user | 3264.4s |
   | CPU sys | 201.7s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3940452913


##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -135,6 +157,114 @@ pub fn apply_file_schema_type_coercions(
     ))
 }
 
+fn dictionary_value_type(data_type: &DataType) -> Option<&DataType> {
+    match data_type {
+        DataType::Dictionary(_, value_type) => Some(value_type.as_ref()),
+        _ => None,
+    }
+}
+
+// Find the value type that can represent both sides without narrowing offsets
+// or crossing string/binary families.
+fn common_dictionary_value_type(
+    field_type: &DataType,
+    dictionary_value_type: &DataType,
+) -> Option<DataType> {
+    let field_type = match field_type {
+        DataType::Dictionary(_, field_value_type) => field_value_type.as_ref(),
+        _ => field_type,
+    };
+
+    match (field_type, dictionary_value_type) {
+        (DataType::Utf8, DataType::Utf8) => Some(DataType::Utf8),
+        (DataType::Utf8 | DataType::LargeUtf8, DataType::Utf8 | DataType::LargeUtf8) => {
+            Some(DataType::LargeUtf8)
+        }
+        (DataType::Binary, DataType::Binary) => Some(DataType::Binary),
+        (
+            DataType::Binary | DataType::LargeBinary,
+            DataType::Binary | DataType::LargeBinary,
+        ) => Some(DataType::LargeBinary),
+        _ => None,
+    }
+}
+
+/// Allows safe widening into the table dictionary type.
+/// - `Utf8` to `Dictionary(Int32, LargeUtf8)`: allowed
+/// - `LargeBinary` to `Dictionary(Int32, Binary)`: rejected
+fn can_promote_to_dictionary_type(
+    file_field_type: &DataType,
+    table_dictionary_type: &DataType,
+) -> bool {
+    dictionary_value_type(table_dictionary_type).is_some_and(|dictionary_value_type| {
+        common_dictionary_value_type(file_field_type, dictionary_value_type)
+            .is_some_and(|common_type| &common_type == dictionary_value_type)
+    })
+}
+
+/// Normalize per-file schemas so that a column promoted to `Dictionary` in
+/// *any* file is promoted to the same `Dictionary` type in *all* files.
+///
+/// This lets [`Schema::try_merge`] accept directories that mix dictionary and
+/// plain encodings for the same column.
+pub(crate) fn uniform_dict_schemas(schemas: Vec<Schema>) -> Vec<Schema> {
+    // First pass: record the dictionary type for every column that is Dictionary in
+    // at least one schema.
+    let mut dict_types: HashMap<String, DataType> = HashMap::new();
+    for schema in &schemas {
+        for field in schema.fields() {
+            if matches!(field.data_type(), DataType::Dictionary(_, _)) {
+                dict_types
+                    .entry(field.name().clone())
+                    .or_insert_with(|| field.data_type().clone());
+            }
+        }
+    }
+    if dict_types.is_empty() {
+        return schemas;
+    }
+
+    // Widen the recorded dictionary value type before promoting plain fields.
+    for schema in &schemas {
+        for field in schema.fields() {
+            let Some(dict_type) = dict_types.get_mut(field.name()) else {
+                continue;
+            };
+            let DataType::Dictionary(key_type, value_type) = dict_type else {

Review Comment:
   the dictionary/dictionary case is now widened, but I think the plain-file issue remains. uniform_dict_schemas retains the existing dictionary key for a plain field, and can_promote_to_dictionary_type returns true without considering key capacity. The new test explicitly expects plain Utf8 to become Dictionary(Int8, Utf8).
   
   With one Dictionary(Int8, Utf8) file and one plain Utf8 file containing at least 129 distinct values, the merged table schema remains Dictionary(Int8, Utf8), so Arrow cannot represent the 129th key. Could plain fields participate as the feature’s default Int32 key when selecting the common type, with an execution-level mixed-file regression test containing more than 128 distinct values?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513719436

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513429687)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (896eebe1dee3bd2845c6f6ffec1ecda7be808125) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..896eebe1dee3bd2845c6f6ffec1ecda7be808125)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    6.09 ms │                               6.07 ms │     no change │
   │ QQuery 2  │   83.76 ms │                              84.73 ms │     no change │
   │ QQuery 3  │   30.21 ms │                              29.80 ms │     no change │
   │ QQuery 4  │  574.69 ms │                             751.80 ms │  1.31x slower │
   │ QQuery 5  │   54.61 ms │                              54.12 ms │     no change │
   │ QQuery 6  │   38.99 ms │                              36.74 ms │ +1.06x faster │
   │ QQuery 7  │   78.01 ms │                              79.34 ms │     no change │
   │ QQuery 8  │   38.70 ms │                              38.50 ms │     no change │
   │ QQuery 9  │   54.84 ms │                              53.07 ms │     no change │
   │ QQuery 10 │   64.90 ms │                              65.52 ms │     no change │
   │ QQuery 11 │  344.80 ms │                             506.95 ms │  1.47x slower │
   │ QQuery 12 │   30.21 ms │                              27.67 ms │ +1.09x faster │
   │ QQuery 13 │  124.42 ms │                             126.00 ms │     no change │
   │ QQuery 14 │  425.67 ms │                             428.38 ms │     no change │
   │ QQuery 15 │   66.06 ms │                              71.59 ms │  1.08x slower │
   │ QQuery 16 │    7.40 ms │                               7.41 ms │     no change │
   │ QQuery 17 │   84.58 ms │                              85.61 ms │     no change │
   │ QQuery 18 │  113.81 ms │                             111.92 ms │     no change │
   │ QQuery 19 │   43.46 ms │                              44.03 ms │     no change │
   │ QQuery 20 │   38.15 ms │                              33.76 ms │ +1.13x faster │
   │ QQuery 21 │   18.47 ms │                              18.30 ms │     no change │
   │ QQuery 22 │   68.05 ms │                              78.24 ms │  1.15x slower │
   │ QQuery 23 │  345.51 ms │                             341.64 ms │     no change │
   │ QQuery 24 │  215.60 ms │                             220.64 ms │     no change │
   │ QQuery 25 │  115.16 ms │                             115.40 ms │     no change │
   │ QQuery 26 │   50.64 ms │                              51.67 ms │     no change │
   │ QQuery 27 │    6.86 ms │                               6.88 ms │     no change │
   │ QQuery 28 │   58.02 ms │                              57.76 ms │     no change │
   │ QQuery 29 │  100.68 ms │                             102.92 ms │     no change │
   │ QQuery 30 │   34.61 ms │                              42.24 ms │  1.22x slower │
   │ QQuery 31 │  116.81 ms │                             121.24 ms │     no change │
   │ QQuery 32 │   21.76 ms │                              23.05 ms │  1.06x slower │
   │ QQuery 33 │   40.02 ms │                              40.45 ms │     no change │
   │ QQuery 34 │   10.82 ms │                              10.61 ms │     no change │
   │ QQuery 35 │   79.34 ms │                              80.90 ms │     no change │
   │ QQuery 36 │    6.36 ms │                               6.55 ms │     no change │
   │ QQuery 37 │    7.34 ms │                               8.34 ms │  1.14x slower │
   │ QQuery 38 │   67.18 ms │                              85.04 ms │  1.27x slower │
   │ QQuery 39 │   99.46 ms │                             100.15 ms │     no change │
   │ QQuery 40 │   25.85 ms │                              26.33 ms │     no change │
   │ QQuery 41 │   12.27 ms │                              13.26 ms │  1.08x slower │
   │ QQuery 42 │   25.17 ms │                              25.48 ms │     no change │
   │ QQuery 43 │    5.95 ms │                               6.01 ms │     no change │
   │ QQuery 44 │   10.21 ms │                               9.99 ms │     no change │
   │ QQuery 45 │   45.00 ms │                              43.45 ms │     no change │
   │ QQuery 46 │   12.53 ms │                              12.45 ms │     no change │
   │ QQuery 47 │  269.32 ms │                             237.18 ms │ +1.14x faster │
   │ QQuery 48 │  101.08 ms │                             100.48 ms │     no change │
   │ QQuery 49 │   74.70 ms │                              73.95 ms │     no change │
   │ QQuery 50 │   61.75 ms │                              62.22 ms │     no change │
   │ QQuery 51 │   94.93 ms │                              95.15 ms │     no change │
   │ QQuery 52 │   25.31 ms │                              25.66 ms │     no change │
   │ QQuery 53 │   30.22 ms │                              30.26 ms │     no change │
   │ QQuery 54 │   56.68 ms │                              57.11 ms │     no change │
   │ QQuery 55 │   24.94 ms │                              24.33 ms │     no change │
   │ QQuery 56 │   40.87 ms │                              41.19 ms │     no change │
   │ QQuery 57 │  189.20 ms │                             170.03 ms │ +1.11x faster │
   │ QQuery 58 │  117.62 ms │                             114.16 ms │     no change │
   │ QQuery 59 │  122.46 ms │                             118.09 ms │     no change │
   │ QQuery 60 │   41.95 ms │                              41.33 ms │     no change │
   │ QQuery 61 │   13.56 ms │                              13.34 ms │     no change │
   │ QQuery 62 │   48.34 ms │                              48.10 ms │     no change │
   │ QQuery 63 │   30.60 ms │                              30.68 ms │     no change │
   │ QQuery 64 │  387.93 ms │                             400.57 ms │     no change │
   │ QQuery 65 │  129.72 ms │                             130.52 ms │     no change │
   │ QQuery 66 │   87.85 ms │                              86.11 ms │     no change │
   │ QQuery 67 │  286.07 ms │                             280.16 ms │     no change │
   │ QQuery 68 │   12.66 ms │                              12.92 ms │     no change │
   │ QQuery 69 │   59.47 ms │                              59.55 ms │     no change │
   │ QQuery 70 │  114.56 ms │                             108.88 ms │     no change │
   │ QQuery 71 │   36.87 ms │                              37.34 ms │     no change │
   │ QQuery 72 │ 1952.36 ms │                            1974.52 ms │     no change │
   │ QQuery 73 │   10.99 ms │                              10.19 ms │ +1.08x faster │
   │ QQuery 74 │  202.38 ms │                             262.84 ms │  1.30x slower │
   │ QQuery 75 │  153.40 ms │                             153.19 ms │     no change │
   │ QQuery 76 │   37.63 ms │                              37.69 ms │     no change │
   │ QQuery 77 │   64.01 ms │                              63.64 ms │     no change │
   │ QQuery 78 │  239.78 ms │                             246.49 ms │     no change │
   │ QQuery 79 │   69.60 ms │                              70.98 ms │     no change │
   │ QQuery 80 │  104.40 ms │                             104.06 ms │     no change │
   │ QQuery 81 │   27.78 ms │                              31.50 ms │  1.13x slower │
   │ QQuery 82 │   17.56 ms │                              18.86 ms │  1.07x slower │
   │ QQuery 83 │   35.69 ms │                              35.33 ms │     no change │
   │ QQuery 84 │   30.57 ms │                              32.88 ms │  1.08x slower │
   │ QQuery 85 │  107.49 ms │                             108.13 ms │     no change │
   │ QQuery 86 │   26.50 ms │                              26.64 ms │     no change │
   │ QQuery 87 │   65.74 ms │                              83.35 ms │  1.27x slower │
   │ QQuery 88 │   65.87 ms │                              65.06 ms │     no change │
   │ QQuery 89 │   36.44 ms │                              38.19 ms │     no change │
   │ QQuery 90 │   18.10 ms │                              17.87 ms │     no change │
   │ QQuery 91 │   46.62 ms │                              47.31 ms │     no change │
   │ QQuery 92 │   31.49 ms │                              31.59 ms │     no change │
   │ QQuery 93 │   51.80 ms │                              52.59 ms │     no change │
   │ QQuery 94 │   40.23 ms │                              39.67 ms │     no change │
   │ QQuery 95 │   83.54 ms │                              84.40 ms │     no change │
   │ QQuery 96 │   24.87 ms │                              24.82 ms │     no change │
   │ QQuery 97 │   54.33 ms │                              55.03 ms │     no change │
   │ QQuery 98 │   44.71 ms │                              40.06 ms │ +1.12x faster │
   │ QQuery 99 │   71.47 ms │                              71.84 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 10077.07ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 10520.00ms │
   │ Average Time (HEAD)                                  │   101.79ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   106.26ms │
   │ Queries Faster                                       │          7 │
   │ Queries Slower                                       │         14 │
   │ Queries with No Change                               │         78 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │           6.09 / 6.66 ±0.96 / 8.58 ms │           6.07 / 6.60 ±0.88 / 8.35 ms │     no change │
   │ QQuery 2  │        83.76 / 86.04 ±3.38 / 92.72 ms │        84.73 / 85.15 ±0.32 / 85.61 ms │     no change │
   │ QQuery 3  │        30.21 / 30.55 ±0.24 / 30.77 ms │        29.80 / 30.10 ±0.21 / 30.43 ms │     no change │
   │ QQuery 4  │     574.69 / 582.89 ±5.89 / 591.18 ms │    751.80 / 768.68 ±12.86 / 789.08 ms │  1.32x slower │
   │ QQuery 5  │        54.61 / 55.15 ±0.39 / 55.80 ms │        54.12 / 55.24 ±0.64 / 55.84 ms │     no change │
   │ QQuery 6  │        38.99 / 40.04 ±0.54 / 40.41 ms │        36.74 / 37.15 ±0.44 / 37.95 ms │ +1.08x faster │
   │ QQuery 7  │        78.01 / 79.57 ±1.21 / 81.09 ms │        79.34 / 81.70 ±3.77 / 89.23 ms │     no change │
   │ QQuery 8  │        38.70 / 39.22 ±0.38 / 39.77 ms │        38.50 / 39.09 ±0.33 / 39.47 ms │     no change │
   │ QQuery 9  │        54.84 / 56.00 ±0.98 / 57.54 ms │        53.07 / 56.40 ±2.19 / 59.64 ms │     no change │
   │ QQuery 10 │        64.90 / 66.76 ±1.87 / 70.32 ms │        65.52 / 68.46 ±3.81 / 75.91 ms │     no change │
   │ QQuery 11 │     344.80 / 351.15 ±8.18 / 366.20 ms │     506.95 / 519.77 ±8.01 / 531.35 ms │  1.48x slower │
   │ QQuery 12 │        30.21 / 32.00 ±1.82 / 35.46 ms │        27.67 / 28.15 ±0.25 / 28.33 ms │ +1.14x faster │
   │ QQuery 13 │     124.42 / 124.72 ±0.29 / 125.11 ms │     126.00 / 129.71 ±4.77 / 139.10 ms │     no change │
   │ QQuery 14 │     425.67 / 434.59 ±6.76 / 446.23 ms │     428.38 / 434.52 ±5.18 / 442.37 ms │     no change │
   │ QQuery 15 │        66.06 / 69.99 ±2.81 / 74.23 ms │        71.59 / 74.63 ±3.89 / 82.27 ms │  1.07x slower │
   │ QQuery 16 │           7.40 / 7.52 ±0.10 / 7.70 ms │           7.41 / 7.66 ±0.31 / 8.28 ms │     no change │
   │ QQuery 17 │        84.58 / 86.00 ±1.87 / 89.68 ms │        85.61 / 87.46 ±1.60 / 89.78 ms │     no change │
   │ QQuery 18 │     113.81 / 114.92 ±1.08 / 116.79 ms │     111.92 / 114.25 ±2.69 / 119.36 ms │     no change │
   │ QQuery 19 │        43.46 / 44.09 ±0.48 / 44.71 ms │        44.03 / 44.53 ±0.41 / 44.94 ms │     no change │
   │ QQuery 20 │        38.15 / 38.43 ±0.26 / 38.88 ms │        33.76 / 34.19 ±0.56 / 35.28 ms │ +1.12x faster │
   │ QQuery 21 │        18.47 / 19.05 ±0.38 / 19.68 ms │        18.30 / 18.61 ±0.21 / 18.93 ms │     no change │
   │ QQuery 22 │        68.05 / 69.88 ±1.00 / 70.92 ms │        78.24 / 79.74 ±0.94 / 80.94 ms │  1.14x slower │
   │ QQuery 23 │     345.51 / 352.02 ±6.42 / 362.49 ms │     341.64 / 349.00 ±5.44 / 356.56 ms │     no change │
   │ QQuery 24 │     215.60 / 218.70 ±3.51 / 225.36 ms │     220.64 / 225.50 ±4.18 / 232.38 ms │     no change │
   │ QQuery 25 │     115.16 / 117.54 ±1.64 / 119.93 ms │     115.40 / 117.97 ±2.01 / 120.37 ms │     no change │
   │ QQuery 26 │        50.64 / 51.31 ±0.58 / 52.33 ms │        51.67 / 52.83 ±1.50 / 55.78 ms │     no change │
   │ QQuery 27 │           6.86 / 7.01 ±0.09 / 7.13 ms │           6.88 / 7.05 ±0.13 / 7.29 ms │     no change │
   │ QQuery 28 │        58.02 / 60.18 ±1.93 / 62.70 ms │        57.76 / 60.90 ±2.49 / 63.47 ms │     no change │
   │ QQuery 29 │     100.68 / 105.48 ±7.68 / 120.67 ms │     102.92 / 105.56 ±2.56 / 109.07 ms │     no change │
   │ QQuery 30 │        34.61 / 35.18 ±0.30 / 35.42 ms │        42.24 / 42.69 ±0.42 / 43.34 ms │  1.21x slower │
   │ QQuery 31 │     116.81 / 118.76 ±2.10 / 122.61 ms │     121.24 / 122.60 ±1.58 / 125.03 ms │     no change │
   │ QQuery 32 │        21.76 / 22.06 ±0.18 / 22.28 ms │        23.05 / 24.07 ±1.70 / 27.44 ms │  1.09x slower │
   │ QQuery 33 │        40.02 / 40.37 ±0.32 / 40.89 ms │        40.45 / 41.23 ±0.48 / 41.86 ms │     no change │
   │ QQuery 34 │        10.82 / 11.12 ±0.27 / 11.62 ms │        10.61 / 10.82 ±0.25 / 11.28 ms │     no change │
   │ QQuery 35 │        79.34 / 79.83 ±0.37 / 80.41 ms │        80.90 / 83.47 ±3.05 / 89.36 ms │     no change │
   │ QQuery 36 │           6.36 / 6.48 ±0.19 / 6.86 ms │           6.55 / 6.70 ±0.15 / 6.99 ms │     no change │
   │ QQuery 37 │          7.34 / 8.35 ±1.86 / 12.08 ms │           8.34 / 8.59 ±0.37 / 9.32 ms │     no change │
   │ QQuery 38 │        67.18 / 68.35 ±1.62 / 71.56 ms │        85.04 / 85.44 ±0.37 / 86.05 ms │  1.25x slower │
   │ QQuery 39 │      99.46 / 100.55 ±0.78 / 101.43 ms │     100.15 / 101.52 ±1.38 / 103.40 ms │     no change │
   │ QQuery 40 │        25.85 / 26.41 ±0.32 / 26.77 ms │        26.33 / 26.80 ±0.27 / 27.12 ms │     no change │
   │ QQuery 41 │        12.27 / 13.86 ±2.93 / 19.72 ms │        13.26 / 13.51 ±0.16 / 13.69 ms │     no change │
   │ QQuery 42 │        25.17 / 25.68 ±0.56 / 26.70 ms │        25.48 / 26.26 ±1.10 / 28.44 ms │     no change │
   │ QQuery 43 │           5.95 / 6.01 ±0.05 / 6.08 ms │           6.01 / 6.06 ±0.06 / 6.18 ms │     no change │
   │ QQuery 44 │        10.21 / 10.34 ±0.08 / 10.47 ms │         9.99 / 10.12 ±0.07 / 10.21 ms │     no change │
   │ QQuery 45 │        45.00 / 46.47 ±1.03 / 48.07 ms │        43.45 / 45.05 ±1.06 / 46.40 ms │     no change │
   │ QQuery 46 │        12.53 / 13.15 ±0.43 / 13.80 ms │        12.45 / 12.74 ±0.20 / 12.97 ms │     no change │
   │ QQuery 47 │     269.32 / 271.49 ±1.45 / 273.72 ms │     237.18 / 240.85 ±3.52 / 246.91 ms │ +1.13x faster │
   │ QQuery 48 │     101.08 / 104.16 ±3.40 / 110.77 ms │     100.48 / 102.27 ±1.91 / 105.98 ms │     no change │
   │ QQuery 49 │        74.70 / 75.67 ±0.87 / 77.08 ms │        73.95 / 74.84 ±0.66 / 75.68 ms │     no change │
   │ QQuery 50 │        61.75 / 64.02 ±4.26 / 72.54 ms │        62.22 / 63.66 ±2.13 / 67.87 ms │     no change │
   │ QQuery 51 │       94.93 / 97.22 ±2.24 / 101.10 ms │        95.15 / 97.11 ±1.06 / 98.30 ms │     no change │
   │ QQuery 52 │        25.31 / 25.80 ±0.44 / 26.46 ms │        25.66 / 25.73 ±0.08 / 25.84 ms │     no change │
   │ QQuery 53 │        30.22 / 30.37 ±0.13 / 30.58 ms │        30.26 / 30.60 ±0.27 / 30.92 ms │     no change │
   │ QQuery 54 │        56.68 / 59.45 ±3.12 / 65.37 ms │        57.11 / 58.80 ±3.28 / 65.36 ms │     no change │
   │ QQuery 55 │        24.94 / 25.55 ±0.54 / 26.55 ms │        24.33 / 24.85 ±0.37 / 25.42 ms │     no change │
   │ QQuery 56 │        40.87 / 42.09 ±0.73 / 43.15 ms │        41.19 / 41.83 ±0.40 / 42.36 ms │     no change │
   │ QQuery 57 │     189.20 / 191.71 ±1.71 / 194.54 ms │     170.03 / 171.80 ±1.41 / 174.31 ms │ +1.12x faster │
   │ QQuery 58 │     117.62 / 120.20 ±3.08 / 125.98 ms │     114.16 / 116.14 ±1.60 / 118.61 ms │     no change │
   │ QQuery 59 │     122.46 / 123.13 ±0.52 / 123.99 ms │     118.09 / 119.31 ±0.95 / 120.97 ms │     no change │
   │ QQuery 60 │        41.95 / 42.58 ±0.52 / 43.52 ms │        41.33 / 41.82 ±0.40 / 42.52 ms │     no change │
   │ QQuery 61 │        13.56 / 13.68 ±0.09 / 13.81 ms │        13.34 / 13.42 ±0.09 / 13.58 ms │     no change │
   │ QQuery 62 │        48.34 / 51.08 ±3.60 / 58.16 ms │        48.10 / 49.96 ±2.22 / 54.26 ms │     no change │
   │ QQuery 63 │        30.60 / 31.35 ±0.42 / 31.78 ms │        30.68 / 31.03 ±0.43 / 31.88 ms │     no change │
   │ QQuery 64 │     387.93 / 392.00 ±3.09 / 396.71 ms │     400.57 / 403.36 ±2.12 / 406.44 ms │     no change │
   │ QQuery 65 │     129.72 / 131.43 ±1.99 / 134.26 ms │     130.52 / 134.80 ±3.20 / 140.44 ms │     no change │
   │ QQuery 66 │        87.85 / 89.69 ±2.08 / 93.75 ms │        86.11 / 88.35 ±3.50 / 95.33 ms │     no change │
   │ QQuery 67 │     286.07 / 292.04 ±4.97 / 298.60 ms │     280.16 / 284.70 ±4.22 / 291.74 ms │     no change │
   │ QQuery 68 │        12.66 / 13.20 ±0.38 / 13.64 ms │        12.92 / 12.97 ±0.03 / 13.01 ms │     no change │
   │ QQuery 69 │        59.47 / 60.54 ±0.93 / 61.80 ms │        59.55 / 60.61 ±0.54 / 61.05 ms │     no change │
   │ QQuery 70 │     114.56 / 116.03 ±0.89 / 117.30 ms │     108.88 / 115.32 ±9.88 / 134.79 ms │     no change │
   │ QQuery 71 │        36.87 / 39.47 ±3.19 / 45.73 ms │        37.34 / 37.95 ±0.38 / 38.38 ms │     no change │
   │ QQuery 72 │ 1952.36 / 2016.53 ±55.29 / 2118.65 ms │ 1974.52 / 2018.69 ±53.69 / 2121.39 ms │     no change │
   │ QQuery 73 │        10.99 / 11.88 ±0.85 / 13.25 ms │        10.19 / 10.40 ±0.25 / 10.89 ms │ +1.14x faster │
   │ QQuery 74 │     202.38 / 205.52 ±2.46 / 209.37 ms │     262.84 / 267.02 ±2.78 / 270.01 ms │  1.30x slower │
   │ QQuery 75 │     153.40 / 154.52 ±0.87 / 155.64 ms │     153.19 / 157.30 ±5.17 / 167.22 ms │     no change │
   │ QQuery 76 │        37.63 / 39.86 ±3.96 / 47.75 ms │        37.69 / 38.06 ±0.43 / 38.88 ms │     no change │
   │ QQuery 77 │        64.01 / 65.29 ±1.05 / 67.13 ms │        63.64 / 64.87 ±0.82 / 65.98 ms │     no change │
   │ QQuery 78 │     239.78 / 245.99 ±4.52 / 251.96 ms │     246.49 / 253.11 ±8.38 / 267.59 ms │     no change │
   │ QQuery 79 │        69.60 / 70.60 ±0.91 / 72.30 ms │        70.98 / 73.89 ±3.13 / 79.96 ms │     no change │
   │ QQuery 80 │     104.40 / 106.71 ±1.45 / 108.95 ms │     104.06 / 106.57 ±1.73 / 109.35 ms │     no change │
   │ QQuery 81 │        27.78 / 29.36 ±1.45 / 31.18 ms │        31.50 / 34.58 ±5.18 / 44.93 ms │  1.18x slower │
   │ QQuery 82 │        17.56 / 17.69 ±0.12 / 17.88 ms │        18.86 / 19.60 ±0.63 / 20.70 ms │  1.11x slower │
   │ QQuery 83 │        35.69 / 35.94 ±0.24 / 36.36 ms │        35.33 / 35.77 ±0.38 / 36.29 ms │     no change │
   │ QQuery 84 │        30.57 / 32.34 ±2.84 / 37.97 ms │        32.88 / 33.16 ±0.25 / 33.59 ms │     no change │
   │ QQuery 85 │     107.49 / 109.00 ±1.27 / 110.68 ms │     108.13 / 110.01 ±1.40 / 112.39 ms │     no change │
   │ QQuery 86 │        26.50 / 27.68 ±1.18 / 29.86 ms │        26.64 / 29.22 ±3.85 / 36.81 ms │  1.06x slower │
   │ QQuery 87 │        65.74 / 68.56 ±2.88 / 74.08 ms │        83.35 / 83.93 ±0.65 / 84.88 ms │  1.22x slower │
   │ QQuery 88 │        65.87 / 66.44 ±0.36 / 66.97 ms │        65.06 / 66.53 ±1.02 / 67.69 ms │     no change │
   │ QQuery 89 │        36.44 / 36.76 ±0.27 / 37.21 ms │        38.19 / 39.98 ±1.86 / 43.10 ms │  1.09x slower │
   │ QQuery 90 │        18.10 / 18.87 ±0.97 / 20.72 ms │        17.87 / 18.12 ±0.21 / 18.49 ms │     no change │
   │ QQuery 91 │        46.62 / 47.07 ±0.39 / 47.79 ms │        47.31 / 47.85 ±0.32 / 48.20 ms │     no change │
   │ QQuery 92 │        31.49 / 33.54 ±2.62 / 38.51 ms │        31.59 / 31.92 ±0.31 / 32.33 ms │     no change │
   │ QQuery 93 │        51.80 / 52.72 ±0.63 / 53.45 ms │        52.59 / 53.69 ±0.66 / 54.62 ms │     no change │
   │ QQuery 94 │        40.23 / 40.75 ±0.47 / 41.42 ms │        39.67 / 42.20 ±2.62 / 46.10 ms │     no change │
   │ QQuery 95 │        83.54 / 84.39 ±0.60 / 85.11 ms │        84.40 / 86.36 ±1.75 / 89.41 ms │     no change │
   │ QQuery 96 │        24.87 / 26.63 ±3.34 / 33.31 ms │        24.82 / 24.98 ±0.11 / 25.16 ms │ +1.07x faster │
   │ QQuery 97 │        54.33 / 55.56 ±1.92 / 59.36 ms │        55.03 / 56.29 ±0.96 / 57.72 ms │     no change │
   │ QQuery 98 │        44.71 / 46.20 ±1.53 / 49.08 ms │        40.06 / 40.72 ±0.53 / 41.48 ms │ +1.13x faster │
   │ QQuery 99 │        71.47 / 72.20 ±0.77 / 73.59 ms │        71.84 / 72.24 ±0.36 / 72.90 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 10302.95ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 10747.44ms │
   │ Average Time (HEAD)                                  │   104.07ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   108.56ms │
   │ Queries Faster                                       │          8 │
   │ Queries Slower                                       │         13 │
   │ Queries with No Change                               │         78 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 2.1 GiB |
   | Avg memory | 1.4 GiB |
   | CPU user | 224.5s |
   | CPU sys | 6.5s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 1.6 GiB |
   | Avg memory | 1.2 GiB |
   | CPU user | 223.7s |
   | CPU sys | 6.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5866303182

   Lets keep this open for few days so others can comment if interested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5564867498

   > > im a little confused by what you mean by this
   > 
   > ParquetOptions is a public struct and is not #[non_exhaustive], so downstream crates can construct it with an exhaustive ParquetOptions { ... } literal. After this PR adds enable_rle_to_dictionary, those existing literals fail to compile with E0063: missing field enable_rle_to_dictionary.
   > 
   > The docs/source/library-user-guide/upgrading/56.0.0.md should mention that affected callers must add enable_rle_to_dictionary: false or migrate to constructing from ParquetOptions::default().
   
   @kumarUjjawal ah makes sense,  Added a migration note to `library-user-guide/upgrading/56.0.0.md` documenting the new field and the compile break for exhaustive ParquetOptions { .. } literals.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3962339698


##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -135,6 +157,169 @@ pub fn apply_file_schema_type_coercions(
     ))
 }
 
+// Find the value type that can represent both sides without narrowing offsets
+// or crossing string/binary families.
+fn common_dictionary_value_type(
+    field_type: &DataType,
+    dictionary_value_type: &DataType,
+) -> Option<DataType> {
+    let field_type = match field_type {
+        DataType::Dictionary(_, field_value_type) => field_value_type.as_ref(),
+        _ => field_type,
+    };
+
+    match (field_type, dictionary_value_type) {
+        (DataType::Utf8, DataType::Utf8) => Some(DataType::Utf8),
+        (DataType::Utf8 | DataType::LargeUtf8, DataType::Utf8 | DataType::LargeUtf8) => {
+            Some(DataType::LargeUtf8)
+        }
+        (DataType::Binary, DataType::Binary) => Some(DataType::Binary),
+        (
+            DataType::Binary | DataType::LargeBinary,
+            DataType::Binary | DataType::LargeBinary,
+        ) => Some(DataType::LargeBinary),
+        _ => None,
+    }
+}
+
+// Same family (both signed or both unsigned): return the wider member.

Review Comment:
   > Could this select UInt64 for that case, or reject the normalization instead of silently narrowing it?
   
   with https://github.com/apache/datafusion/pull/24227/changes/4d607b74af918f514fb4f521b9b307a3458a5546 the type now widens instead of narrowing silently 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364746138

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364728282)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5364728282-1829-llknq 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5373793190

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5373757117)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5373757117-1847-tn7nd 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (4865205f97c7ad3ef19d44e2e2275e14304c0d48) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..4865205f97c7ad3ef19d44e2e2275e14304c0d48)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5373757117

   run benchmarks 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5373951377

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5373757117)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (4865205f97c7ad3ef19d44e2e2275e14304c0d48) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..4865205f97c7ad3ef19d44e2e2275e14304c0d48)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.34 ms │                              38.30 ms │    no change │
   │ QQuery 2  │ 19.14 ms │                              19.09 ms │    no change │
   │ QQuery 3  │ 30.87 ms │                              32.55 ms │ 1.05x slower │
   │ QQuery 4  │ 17.58 ms │                              17.33 ms │    no change │
   │ QQuery 5  │ 37.86 ms │                              37.95 ms │    no change │
   │ QQuery 6  │ 16.19 ms │                              15.73 ms │    no change │
   │ QQuery 7  │ 43.91 ms │                              42.53 ms │    no change │
   │ QQuery 8  │ 42.08 ms │                              41.76 ms │    no change │
   │ QQuery 9  │ 49.31 ms │                              48.91 ms │    no change │
   │ QQuery 10 │ 41.61 ms │                              41.66 ms │    no change │
   │ QQuery 11 │ 13.40 ms │                              13.17 ms │    no change │
   │ QQuery 12 │ 23.65 ms │                              23.56 ms │    no change │
   │ QQuery 13 │ 33.50 ms │                              32.19 ms │    no change │
   │ QQuery 14 │ 22.95 ms │                              22.88 ms │    no change │
   │ QQuery 15 │ 30.47 ms │                              30.27 ms │    no change │
   │ QQuery 16 │ 13.82 ms │                              13.68 ms │    no change │
   │ QQuery 17 │ 69.99 ms │                              69.98 ms │    no change │
   │ QQuery 18 │ 59.26 ms │                              58.59 ms │    no change │
   │ QQuery 19 │ 32.08 ms │                              32.28 ms │    no change │
   │ QQuery 20 │ 31.34 ms │                              31.11 ms │    no change │
   │ QQuery 21 │ 53.69 ms │                              55.53 ms │    no change │
   │ QQuery 22 │ 13.99 ms │                              13.90 ms │    no change │
   └───────────┴──────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 735.03ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 732.96ms │
   │ Average Time (HEAD)                                  │  33.41ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  33.32ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        1 │
   │ Queries with No Change                               │       21 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃    Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ QQuery 1  │ 38.34 / 40.62 ±3.07 / 46.51 ms │        38.30 / 39.66 ±1.04 / 40.81 ms │ no change │
   │ QQuery 2  │ 19.14 / 19.45 ±0.19 / 19.73 ms │        19.09 / 19.73 ±0.65 / 20.92 ms │ no change │
   │ QQuery 3  │ 30.87 / 32.16 ±1.28 / 34.00 ms │        32.55 / 33.25 ±0.54 / 33.95 ms │ no change │
   │ QQuery 4  │ 17.58 / 17.77 ±0.23 / 18.18 ms │        17.33 / 17.50 ±0.18 / 17.80 ms │ no change │
   │ QQuery 5  │ 37.86 / 40.01 ±1.12 / 41.09 ms │        37.95 / 39.66 ±0.93 / 40.64 ms │ no change │
   │ QQuery 6  │ 16.19 / 16.39 ±0.22 / 16.82 ms │        15.73 / 15.93 ±0.12 / 16.10 ms │ no change │
   │ QQuery 7  │ 43.91 / 44.41 ±0.37 / 44.84 ms │        42.53 / 43.63 ±0.64 / 44.48 ms │ no change │
   │ QQuery 8  │ 42.08 / 42.45 ±0.25 / 42.84 ms │        41.76 / 42.09 ±0.18 / 42.29 ms │ no change │
   │ QQuery 9  │ 49.31 / 50.25 ±0.92 / 51.90 ms │        48.91 / 49.17 ±0.29 / 49.73 ms │ no change │
   │ QQuery 10 │ 41.61 / 42.36 ±1.08 / 44.50 ms │        41.66 / 42.56 ±0.72 / 43.78 ms │ no change │
   │ QQuery 11 │ 13.40 / 13.51 ±0.14 / 13.79 ms │        13.17 / 13.40 ±0.16 / 13.67 ms │ no change │
   │ QQuery 12 │ 23.65 / 24.12 ±0.29 / 24.46 ms │        23.56 / 23.92 ±0.28 / 24.36 ms │ no change │
   │ QQuery 13 │ 33.50 / 34.26 ±1.24 / 36.73 ms │        32.19 / 34.17 ±1.65 / 36.52 ms │ no change │
   │ QQuery 14 │ 22.95 / 23.11 ±0.19 / 23.49 ms │        22.88 / 23.05 ±0.13 / 23.23 ms │ no change │
   │ QQuery 15 │ 30.47 / 31.12 ±0.45 / 31.88 ms │        30.27 / 30.58 ±0.33 / 31.16 ms │ no change │
   │ QQuery 16 │ 13.82 / 13.93 ±0.12 / 14.10 ms │        13.68 / 13.85 ±0.14 / 14.09 ms │ no change │
   │ QQuery 17 │ 69.99 / 70.23 ±0.22 / 70.57 ms │        69.98 / 71.43 ±1.30 / 73.19 ms │ no change │
   │ QQuery 18 │ 59.26 / 60.44 ±0.88 / 61.88 ms │        58.59 / 59.21 ±0.52 / 60.03 ms │ no change │
   │ QQuery 19 │ 32.08 / 33.02 ±1.53 / 36.07 ms │        32.28 / 33.15 ±1.00 / 34.68 ms │ no change │
   │ QQuery 20 │ 31.34 / 31.62 ±0.23 / 31.88 ms │        31.11 / 31.60 ±0.41 / 32.28 ms │ no change │
   │ QQuery 21 │ 53.69 / 55.06 ±1.25 / 57.24 ms │        55.53 / 56.85 ±0.77 / 57.65 ms │ no change │
   │ QQuery 22 │ 13.99 / 14.12 ±0.07 / 14.21 ms │        13.90 / 14.09 ±0.17 / 14.37 ms │ no change │
   └───────────┴────────────────────────────────┴───────────────────────────────────────┴───────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 750.39ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 748.47ms │
   │ Average Time (HEAD)                                  │  34.11ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  34.02ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        0 │
   │ Queries with No Change                               │       22 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.1 GiB |
   | Avg memory | 495.3 MiB |
   | CPU user | 21.4s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.2 GiB |
   | Avg memory | 530.1 MiB |
   | CPU user | 21.3s |
   | CPU sys | 1.7s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675107604

   > Hi @Rich-T-kid This looks good to me at the moment. Let's resolve the conflicts and re-run the benchmarks.
   
   @kumarUjjawal just fixed the merge conflicts. Im not sure which benchmark exactly to run 🤔 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675306413

   > Im not sure which benchmark exactly to run
   
   Started benchmakr. Let's see if it works 🤞


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["yinli-systems (via GitHub)" <gi...@apache.org>]

yinli-systems commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5676104867

   The new paired benchmark is useful because it compares the same `cb3a578` head with only `DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY` changed. Correctness CI is green, but flag-on performance needs a narrower target: TPC-H total 732.41 -> 810.78 ms (~11% slower), TPC-DS 9354.95 -> 9732.14 ms (~4% slower), and ClickBench partitioned 18999.43 -> 71162.33 ms (~3.75x slower). ClickBench Q33/Q34 are `GROUP BY URL`; they become 9.20x/8.93x slower, with peak MemoryPool reservation rising 8.1 -> 12.3 GiB and 8.2 -> 12.7 GiB respectively.
   
   A concrete hypothesis, not yet a proven attribution: `DFParquetMetadata::fetch_schema` currently promotes a top-level string/binary column if *any* row group has a dictionary page, without an NDV or repetition gate. That can select an expensive dictionary path for URL-like high-cardinality group keys. Could we capture flag-off/flag-on `EXPLAIN ANALYZE`, the actual scan schema for URL, and row-group dictionary cardinalities for Q33/Q34? Those would distinguish reader/materialization cost from grouping/hash cost before choosing a fix. The default-off flag keeps existing workloads unchanged, but these A/B results argue against enabling it broadly without a cost gate.
   
   For #24117 I am keeping the staged optimizer opt-in and fail-closed: exact NDV and row-count, physical page evidence, and a high repetition threshold are required before selectively requesting Dictionary on a direct group key. #25185 may help reused dictionary-value hashing, but its standalone benchmark is not yet end-to-end evidence that it offsets this regression.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5274190748

   @adriangb  pinging you since you seem interested in parquet related speed ups 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangb (via GitHub)" <gi...@apache.org>]

adriangb commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5282764688

   > no not exactly. The goal of this PR is to keep RLE parquet columns in their compacted form by materializing them as dictionary arrays instead of regular strings.
   
   Why only RLE and not dictionaries as well? How does this compare to / relate to the `schema_force_view_types` option?
   
   It also looks like this goes through `infer_schema` right? A lot of code paths never touch that (`CREATE EXTERNAL TABLE (a VARCHAR)`, any custom table providers, etc.).
   
   I'd be more interested in seeing something at the parquet scan level that was able to e.g. optimize how row filters are applied by applying them to the dictionary instead of expanding into `Utf8View`. That would be applicable to all DataFusion users.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5310758833

   it would be interesting to see the benchmark results of turning the flag on. im not sure if @adriangbot can run with specific parquet flags 🤔 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5242428575

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion-common v54.1.0 (current)
          Built [  35.295s] (current)
        Parsing datafusion-common v54.1.0 (current)
         Parsed [   0.064s] (current)
       Building datafusion-common v54.1.0 (baseline)
          Built [  32.944s] (baseline)
        Parsing datafusion-common v54.1.0 (baseline)
         Parsed [   0.066s] (baseline)
       Checking datafusion-common v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.921s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
   
   --- failure constructible_struct_adds_field: externally-constructible struct adds field ---
   
   Description:
   A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1103
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [  70.664s] datafusion-common
       Building datafusion-datasource-parquet v54.1.0 (current)
   error: running cargo-doc on crate 'datafusion-datasource-parquet' failed with output:
   -----
      Compiling proc-macro2 v1.0.107
      Compiling quote v1.0.47
      Compiling unicode-ident v1.0.24
      Compiling libc v0.2.189
       Checking cfg-if v1.0.4
       Checking memchr v2.8.3
      Compiling autocfg v1.5.1
      Compiling libm v0.2.16
      Compiling num-traits v0.2.19
      Compiling syn v2.0.119
      Compiling syn v3.0.3
      Compiling zerocopy v0.8.56
       Checking bytes v1.12.1
       Checking once_cell v1.21.4
       Checking equivalent v1.0.2
       Checking itoa v1.0.18
       Checking foldhash v0.2.0
       Checking allocator-api2 v0.2.21
      Compiling serde_core v1.0.229
      Compiling getrandom v0.3.4
       Checking hashbrown v0.17.1
      Compiling zmij v1.0.23
       Checking indexmap v2.14.0
      Compiling serde_json v1.0.151
       Checking num-integer v0.1.46
      Compiling version_check v0.9.5
       Checking siphasher v1.0.3
       Checking stable_deref_trait v1.2.1
       Checking iana-time-zone v0.1.65
       Checking chrono v0.4.45
      Compiling synstructure v0.13.2
       Checking phf_shared v0.12.1
      Compiling ahash v0.8.12
       Checking num-bigint v0.5.1
      Compiling chrono-tz v0.10.4
       Checking arrow-schema v59.2.0
      Compiling zerocopy-derive v0.8.56
      Compiling zerofrom-derive v0.1.7
      Compiling yoke-derive v0.8.2
      Compiling jobserver v0.1.35
      Compiling zerovec-derive v0.11.3
       Checking phf v0.12.1
      Compiling find-msvc-tools v0.1.10
       Checking zerofrom v0.1.8
       Checking yoke v0.8.3
      Compiling shlex v2.0.1
       Checking zerovec v0.11.6
      Compiling cc v1.4.2
      Compiling displaydoc v0.2.7
       Checking num-complex v0.4.6
       Checking tinystr v0.8.3
       Checking litemap v0.8.2
       Checking lexical-util v1.0.7
       Checking smallvec v1.15.2
      Compiling pkg-config v0.3.33
       Checking pin-project-lite v0.2.17
       Checking writeable v0.6.3
       Checking icu_locale_core v2.2.0
      Compiling zstd-sys v2.0.16+zstd.1.5.7
       Checking zerotrie v0.2.4
       Checking potential_utf v0.1.5
      Compiling icu_properties_data v2.2.0
       Checking futures-core v0.3.33
      Compiling icu_normalizer_data v2.2.0
       Checking futures-sink v0.3.33
       Checking utf8_iter v1.0.4
       Checking icu_provider v2.2.0
       Checking icu_collections v2.2.0
      Compiling semver v1.0.28
      Compiling rustc_version v0.4.1
       Checking futures-channel v0.3.33
       Checking lexical-write-integer v1.0.6
       Checking half v2.7.1
       Checking lexical-parse-integer v1.0.6
      Compiling futures-macro v0.3.33
       Checking arrow-buffer v59.2.0
      Compiling zstd-safe v7.2.4
       Checking arrow-data v59.2.0
       Checking slab v0.4.12
       Checking futures-task v0.3.33
       Checking futures-io v0.3.33
       Checking bitflags v2.13.1
      Compiling parking_lot_core v0.9.12
       Checking futures-util v0.3.33
       Checking arrow-array v59.2.0
       Checking lexical-parse-float v1.0.6
       Checking lexical-write-float v1.0.6
       Checking icu_normalizer v2.2.0
       Checking icu_properties v2.2.0
       Checking arrow-select v59.2.0
      Compiling flatbuffers v25.12.19
      Compiling tokio-macros v2.7.2
       Checking aho-corasick v1.1.5
       Checking scopeguard v1.2.0
       Checking regex-syntax v0.8.11
       Checking ryu v1.0.23
       Checking unicode-segmentation v1.13.3
       Checking unicode-width v0.2.2
       Checking base64 v0.23.1
       Checking comfy-table v7.2.2
       Checking arrow-ord v59.2.0
       Checking lock_api v0.4.14
       Checking tokio v1.53.1
       Checking idna_adapter v1.2.2
       Checking lexical-core v1.0.6
       Checking regex-automata v0.4.18
       Checking atoi v2.0.0
       Checking either v1.17.0
      Compiling thiserror v2.0.20
       Checking percent-encoding v2.3.2
      Compiling getrandom v0.4.3
       Checking alloc-no-stdlib v2.0.4
       Checking twox-hash v2.1.3
       Checking lz4_flex v0.14.0
       Checking alloc-stdlib v0.2.4
       Checking form_urlencoded v1.2.2
       Checking arrow-cast v59.2.0
       Checking regex v1.13.1
       Checking idna v1.1.0
       Checking futures-executor v0.3.33
      Compiling ring v0.17.14
      Compiling tracing-attributes v0.1.31
      Compiling thiserror-impl v2.0.20
       Checking tracing-core v0.1.36
       Checking csv-core v0.1.13
       Checking same-file v1.0.6
      Compiling snap v1.1.2
       Checking simdutf8 v0.1.5
       Checking walkdir v2.5.0
       Checking tracing v0.1.44
       Checking csv v1.4.0
       Checking futures v0.3.33
       Checking url v2.5.8
       Checking brotli-decompressor v5.0.3
       Checking itertools v0.14.0
       Checking parking_lot v0.12.5
      Compiling async-trait v0.1.92
       Checking http v1.5.0
       Checking getrandom v0.2.17
       Checking untrusted v0.9.0
       Checking zlib-rs v0.6.7
       Checking humantime v2.4.0
      Compiling anyhow v1.0.104
       Checking object_store v0.13.2
       Checking flate2 v1.1.9
       Checking brotli v8.0.4
       Checking arrow-csv v59.2.0
       Checking zstd v0.13.3
       Checking arrow-ipc v59.2.0
       Checking arrow-json v59.2.0
       Checking arrow-string v59.2.0
       Checking arrow-arith v59.2.0
       Checking arrow-row v59.2.0
      Compiling seq-macro v0.3.6
       Checking log v0.4.33
       Checking arrow v59.2.0
       Checking uuid v1.24.0
       Checking itertools v0.15.0
       Checking hex v0.4.3
      Compiling pin-project-internal v1.1.13
       Checking parquet v59.2.0
       Checking pin-project v1.1.13
      Compiling prost-derive v0.14.4
      Compiling rustix v1.1.4
      Compiling crossbeam-utils v0.8.22
       Checking ppv-lite86 v0.2.21
       Checking rand_core v0.9.5
       Checking linux-raw-sys v0.12.1
       Checking datafusion-doc v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
       Checking rand_chacha v0.9.0
       Checking fastrand v2.5.0
       Checking foldhash v0.1.5
       Checking hashbrown v0.14.5
       Checking prost v0.14.4
       Checking hashbrown v0.15.5
       Checking dashmap v6.2.1
       Checking rand v0.9.5
       Checking tokio-util v0.7.19
       Checking tempfile v3.27.0
       Checking fixedbitset v0.5.7
       Checking datafusion-common-runtime v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/common-runtime)
       Checking glob v0.3.4
       Checking petgraph v0.8.3
      Compiling datafusion-macros v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/macros)
       Checking datafusion-common v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
       Checking datafusion-proto-common v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/proto-common)
       Checking datafusion-expr-common v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/expr-common)
   error[E0063]: missing field `enable_rle_to_dictionary` in initializer of `datafusion_common::config::ParquetOptions`
       --> /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/from_proto/mod.rs:1053:12
        |
   1053 |         Ok(ParquetOptions {
        |            ^^^^^^^^^^^^^^ missing `enable_rle_to_dictionary`
   
   For more information about this error, try `rustc --explain E0063`.
   error: could not compile `datafusion-proto-common` (lib) due to 1 previous error
   
   -----
   
   error: failed to build rustdoc for crate datafusion-datasource-parquet v54.1.0
   note: this is usually due to a compilation error in the crate,
         and is unlikely to be a bug in cargo-semver-checks
   note: the following command can be used to reproduce the error:
         cargo new --lib example &&
             cd example &&
             echo '[workspace]' >> Cargo.toml &&
             cargo add --path /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet --features parquet_encryption,proto &&
             cargo check &&
             cargo doc
   
       Building datafusion-sqllogictest v54.1.0 (current)
          Built [ 187.612s] (current)
        Parsing datafusion-sqllogictest v54.1.0 (current)
         Parsed [   0.024s] (current)
       Building datafusion-sqllogictest v54.1.0 (baseline)
          Built [ 187.178s] (baseline)
        Parsing datafusion-sqllogictest v54.1.0 (baseline)
         Parsed [   0.025s] (baseline)
       Checking datafusion-sqllogictest v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.109s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [ 378.611s] datafusion-sqllogictest
   error: aborting due to failure to build rustdoc for crate datafusion-datasource-parquet v54.1.0
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5374480016

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5374297430)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    5.56 ms │                               5.78 ms │     no change │
   │ QQuery 2  │   79.84 ms │                              79.86 ms │     no change │
   │ QQuery 3  │   28.80 ms │                              28.74 ms │     no change │
   │ QQuery 4  │  477.24 ms │                             497.34 ms │     no change │
   │ QQuery 5  │   50.65 ms │                              51.57 ms │     no change │
   │ QQuery 6  │   35.37 ms │                              35.61 ms │     no change │
   │ QQuery 7  │   92.26 ms │                              92.98 ms │     no change │
   │ QQuery 8  │   36.09 ms │                              36.17 ms │     no change │
   │ QQuery 9  │   49.80 ms │                              51.23 ms │     no change │
   │ QQuery 10 │   62.10 ms │                              62.37 ms │     no change │
   │ QQuery 11 │  305.34 ms │                             309.76 ms │     no change │
   │ QQuery 12 │   28.29 ms │                              28.62 ms │     no change │
   │ QQuery 13 │  115.99 ms │                             116.91 ms │     no change │
   │ QQuery 14 │  410.27 ms │                             413.00 ms │     no change │
   │ QQuery 15 │   56.14 ms │                              56.40 ms │     no change │
   │ QQuery 16 │    6.60 ms │                               6.61 ms │     no change │
   │ QQuery 17 │   78.40 ms │                              79.44 ms │     no change │
   │ QQuery 18 │  120.65 ms │                             120.36 ms │     no change │
   │ QQuery 19 │   40.32 ms │                              40.35 ms │     no change │
   │ QQuery 20 │   34.65 ms │                              35.33 ms │     no change │
   │ QQuery 21 │   17.12 ms │                              17.17 ms │     no change │
   │ QQuery 22 │   62.38 ms │                              61.28 ms │     no change │
   │ QQuery 23 │  333.65 ms │                             335.75 ms │     no change │
   │ QQuery 24 │  222.35 ms │                             217.72 ms │     no change │
   │ QQuery 25 │  108.75 ms │                             107.53 ms │     no change │
   │ QQuery 26 │   57.57 ms │                              56.49 ms │     no change │
   │ QQuery 27 │    6.28 ms │                               6.17 ms │     no change │
   │ QQuery 28 │   59.39 ms │                              59.11 ms │     no change │
   │ QQuery 29 │   96.26 ms │                              94.90 ms │     no change │
   │ QQuery 30 │   32.33 ms │                              31.76 ms │     no change │
   │ QQuery 31 │  109.96 ms │                             108.42 ms │     no change │
   │ QQuery 32 │   20.33 ms │                              19.63 ms │     no change │
   │ QQuery 33 │   37.78 ms │                              37.13 ms │     no change │
   │ QQuery 34 │   10.08 ms │                               9.68 ms │     no change │
   │ QQuery 35 │   73.01 ms │                              72.01 ms │     no change │
   │ QQuery 36 │    5.71 ms │                               5.79 ms │     no change │
   │ QQuery 37 │    6.83 ms │                               6.71 ms │     no change │
   │ QQuery 38 │   60.99 ms │                              60.83 ms │     no change │
   │ QQuery 39 │   89.69 ms │                              88.46 ms │     no change │
   │ QQuery 40 │   23.29 ms │                              22.97 ms │     no change │
   │ QQuery 41 │   11.59 ms │                              10.88 ms │ +1.06x faster │
   │ QQuery 42 │   23.50 ms │                              23.31 ms │     no change │
   │ QQuery 43 │    5.05 ms │                               5.10 ms │     no change │
   │ QQuery 44 │    9.42 ms │                               9.13 ms │     no change │
   │ QQuery 45 │   38.19 ms │                              37.36 ms │     no change │
   │ QQuery 46 │   12.08 ms │                              11.61 ms │     no change │
   │ QQuery 47 │  231.84 ms │                             219.46 ms │ +1.06x faster │
   │ QQuery 48 │   94.79 ms │                              94.07 ms │     no change │
   │ QQuery 49 │   75.34 ms │                              74.70 ms │     no change │
   │ QQuery 50 │   58.18 ms │                              58.16 ms │     no change │
   │ QQuery 51 │   91.99 ms │                              91.62 ms │     no change │
   │ QQuery 52 │   23.86 ms │                              23.47 ms │     no change │
   │ QQuery 53 │   29.17 ms │                              28.41 ms │     no change │
   │ QQuery 54 │   54.37 ms │                              54.02 ms │     no change │
   │ QQuery 55 │   23.66 ms │                              22.98 ms │     no change │
   │ QQuery 56 │   38.59 ms │                              37.79 ms │     no change │
   │ QQuery 57 │  173.33 ms │                             173.80 ms │     no change │
   │ QQuery 58 │  111.69 ms │                             110.28 ms │     no change │
   │ QQuery 59 │  117.92 ms │                             117.62 ms │     no change │
   │ QQuery 60 │   38.66 ms │                              38.99 ms │     no change │
   │ QQuery 61 │   12.25 ms │                              12.15 ms │     no change │
   │ QQuery 62 │   46.36 ms │                              45.58 ms │     no change │
   │ QQuery 63 │   29.47 ms │                              29.04 ms │     no change │
   │ QQuery 64 │  401.64 ms │                             404.29 ms │     no change │
   │ QQuery 65 │  121.31 ms │                             122.20 ms │     no change │
   │ QQuery 66 │   80.58 ms │                              79.57 ms │     no change │
   │ QQuery 67 │  241.97 ms │                             238.00 ms │     no change │
   │ QQuery 68 │   12.01 ms │                              11.88 ms │     no change │
   │ QQuery 69 │   56.60 ms │                              56.32 ms │     no change │
   │ QQuery 70 │  105.47 ms │                             104.66 ms │     no change │
   │ QQuery 71 │   34.46 ms │                              34.29 ms │     no change │
   │ QQuery 72 │ 2033.09 ms │                            1999.15 ms │     no change │
   │ QQuery 73 │   10.11 ms │                               9.91 ms │     no change │
   │ QQuery 74 │  177.93 ms │                             184.55 ms │     no change │
   │ QQuery 75 │  147.09 ms │                             149.15 ms │     no change │
   │ QQuery 76 │   34.91 ms │                              35.40 ms │     no change │
   │ QQuery 77 │   60.35 ms │                              61.80 ms │     no change │
   │ QQuery 78 │  193.99 ms │                             197.97 ms │     no change │
   │ QQuery 79 │   66.98 ms │                              66.87 ms │     no change │
   │ QQuery 80 │   97.60 ms │                              98.42 ms │     no change │
   │ QQuery 81 │   25.61 ms │                              26.41 ms │     no change │
   │ QQuery 82 │   15.93 ms │                              16.54 ms │     no change │
   │ QQuery 83 │   40.24 ms │                              40.51 ms │     no change │
   │ QQuery 84 │   30.20 ms │                              30.04 ms │     no change │
   │ QQuery 85 │  106.08 ms │                             105.69 ms │     no change │
   │ QQuery 86 │   25.38 ms │                              25.65 ms │     no change │
   │ QQuery 87 │   63.24 ms │                              61.92 ms │     no change │
   │ QQuery 88 │   62.53 ms │                              63.40 ms │     no change │
   │ QQuery 89 │   35.03 ms │                              35.28 ms │     no change │
   │ QQuery 90 │   17.09 ms │                              17.09 ms │     no change │
   │ QQuery 91 │   45.80 ms │                              46.24 ms │     no change │
   │ QQuery 92 │   29.62 ms │                              29.44 ms │     no change │
   │ QQuery 93 │   49.33 ms │                              49.56 ms │     no change │
   │ QQuery 94 │   37.44 ms │                              38.00 ms │     no change │
   │ QQuery 95 │   79.35 ms │                              80.35 ms │     no change │
   │ QQuery 96 │   23.99 ms │                              23.99 ms │     no change │
   │ QQuery 97 │   46.41 ms │                              46.79 ms │     no change │
   │ QQuery 98 │   41.75 ms │                              42.32 ms │     no change │
   │ QQuery 99 │   69.72 ms │                              69.72 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 9620.22ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 9600.85ms │
   │ Average Time (HEAD)                                  │   97.17ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   96.98ms │
   │ Queries Faster                                       │         2 │
   │ Queries Slower                                       │         0 │
   │ Queries with No Change                               │        97 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │           5.56 / 6.09 ±0.88 / 7.86 ms │           5.78 / 6.28 ±0.85 / 7.98 ms │     no change │
   │ QQuery 2  │        79.84 / 80.27 ±0.31 / 80.65 ms │        79.86 / 80.10 ±0.20 / 80.44 ms │     no change │
   │ QQuery 3  │        28.80 / 28.93 ±0.09 / 29.05 ms │        28.74 / 28.85 ±0.11 / 29.00 ms │     no change │
   │ QQuery 4  │     477.24 / 482.76 ±3.75 / 488.42 ms │     497.34 / 505.58 ±6.75 / 514.21 ms │     no change │
   │ QQuery 5  │        50.65 / 51.77 ±0.68 / 52.61 ms │        51.57 / 51.96 ±0.43 / 52.79 ms │     no change │
   │ QQuery 6  │        35.37 / 35.71 ±0.36 / 36.27 ms │        35.61 / 36.12 ±0.29 / 36.43 ms │     no change │
   │ QQuery 7  │        92.26 / 92.90 ±0.58 / 93.87 ms │       92.98 / 94.88 ±3.14 / 101.14 ms │     no change │
   │ QQuery 8  │        36.09 / 37.73 ±2.65 / 43.02 ms │        36.17 / 37.25 ±0.89 / 38.74 ms │     no change │
   │ QQuery 9  │        49.80 / 52.77 ±2.03 / 55.86 ms │        51.23 / 53.24 ±1.20 / 54.76 ms │     no change │
   │ QQuery 10 │        62.10 / 62.43 ±0.28 / 62.90 ms │        62.37 / 62.80 ±0.35 / 63.39 ms │     no change │
   │ QQuery 11 │     305.34 / 310.31 ±4.20 / 316.01 ms │     309.76 / 313.75 ±3.20 / 317.52 ms │     no change │
   │ QQuery 12 │        28.29 / 28.70 ±0.31 / 29.16 ms │        28.62 / 28.95 ±0.30 / 29.46 ms │     no change │
   │ QQuery 13 │     115.99 / 116.69 ±0.44 / 117.15 ms │     116.91 / 117.36 ±0.30 / 117.70 ms │     no change │
   │ QQuery 14 │     410.27 / 414.25 ±3.43 / 419.35 ms │     413.00 / 415.76 ±2.82 / 421.07 ms │     no change │
   │ QQuery 15 │        56.14 / 58.00 ±1.05 / 59.06 ms │        56.40 / 57.53 ±0.91 / 58.87 ms │     no change │
   │ QQuery 16 │           6.60 / 6.75 ±0.17 / 7.08 ms │           6.61 / 6.84 ±0.23 / 7.27 ms │     no change │
   │ QQuery 17 │        78.40 / 79.73 ±1.70 / 83.05 ms │        79.44 / 80.41 ±1.12 / 82.55 ms │     no change │
   │ QQuery 18 │     120.65 / 121.89 ±1.34 / 124.47 ms │     120.36 / 121.99 ±2.12 / 126.17 ms │     no change │
   │ QQuery 19 │        40.32 / 40.72 ±0.41 / 41.36 ms │        40.35 / 40.57 ±0.12 / 40.68 ms │     no change │
   │ QQuery 20 │        34.65 / 34.97 ±0.37 / 35.69 ms │        35.33 / 35.74 ±0.23 / 36.01 ms │     no change │
   │ QQuery 21 │        17.12 / 17.43 ±0.33 / 18.01 ms │        17.17 / 17.39 ±0.33 / 18.04 ms │     no change │
   │ QQuery 22 │        62.38 / 63.08 ±0.70 / 64.15 ms │        61.28 / 62.64 ±0.93 / 64.09 ms │     no change │
   │ QQuery 23 │     333.65 / 342.42 ±8.00 / 355.48 ms │     335.75 / 339.95 ±3.15 / 344.50 ms │     no change │
   │ QQuery 24 │     222.35 / 225.26 ±3.98 / 233.15 ms │     217.72 / 222.35 ±3.24 / 227.71 ms │     no change │
   │ QQuery 25 │     108.75 / 110.89 ±1.95 / 114.22 ms │     107.53 / 108.98 ±1.28 / 111.14 ms │     no change │
   │ QQuery 26 │        57.57 / 59.14 ±2.44 / 63.95 ms │        56.49 / 58.24 ±2.81 / 63.80 ms │     no change │
   │ QQuery 27 │           6.28 / 6.52 ±0.19 / 6.85 ms │           6.17 / 6.28 ±0.15 / 6.57 ms │     no change │
   │ QQuery 28 │        59.39 / 60.19 ±0.52 / 60.89 ms │        59.11 / 59.84 ±0.47 / 60.50 ms │     no change │
   │ QQuery 29 │       96.26 / 97.90 ±1.40 / 100.36 ms │        94.90 / 96.13 ±1.48 / 99.00 ms │     no change │
   │ QQuery 30 │        32.33 / 33.45 ±1.75 / 36.93 ms │        31.76 / 33.99 ±2.93 / 39.76 ms │     no change │
   │ QQuery 31 │     109.96 / 112.18 ±1.74 / 115.18 ms │     108.42 / 110.26 ±1.40 / 112.62 ms │     no change │
   │ QQuery 32 │        20.33 / 20.88 ±0.32 / 21.25 ms │        19.63 / 20.20 ±0.29 / 20.43 ms │     no change │
   │ QQuery 33 │        37.78 / 37.98 ±0.25 / 38.46 ms │        37.13 / 37.25 ±0.11 / 37.43 ms │     no change │
   │ QQuery 34 │        10.08 / 10.29 ±0.17 / 10.55 ms │         9.68 / 10.92 ±1.90 / 14.68 ms │  1.06x slower │
   │ QQuery 35 │        73.01 / 75.89 ±2.19 / 79.45 ms │        72.01 / 72.63 ±1.00 / 74.63 ms │     no change │
   │ QQuery 36 │           5.71 / 5.90 ±0.21 / 6.30 ms │           5.79 / 5.95 ±0.19 / 6.32 ms │     no change │
   │ QQuery 37 │           6.83 / 6.93 ±0.08 / 7.04 ms │           6.71 / 6.87 ±0.09 / 6.98 ms │     no change │
   │ QQuery 38 │        60.99 / 62.87 ±1.13 / 63.90 ms │        60.83 / 61.54 ±0.78 / 62.97 ms │     no change │
   │ QQuery 39 │        89.69 / 91.20 ±0.82 / 92.07 ms │        88.46 / 89.84 ±1.98 / 93.74 ms │     no change │
   │ QQuery 40 │        23.29 / 23.72 ±0.23 / 23.91 ms │        22.97 / 23.54 ±0.76 / 25.03 ms │     no change │
   │ QQuery 41 │        11.59 / 12.21 ±0.99 / 14.17 ms │        10.88 / 11.13 ±0.25 / 11.61 ms │ +1.10x faster │
   │ QQuery 42 │        23.50 / 23.97 ±0.75 / 25.45 ms │        23.31 / 23.64 ±0.43 / 24.46 ms │     no change │
   │ QQuery 43 │           5.05 / 5.25 ±0.17 / 5.53 ms │           5.10 / 5.22 ±0.14 / 5.48 ms │     no change │
   │ QQuery 44 │           9.42 / 9.51 ±0.12 / 9.74 ms │           9.13 / 9.30 ±0.16 / 9.55 ms │     no change │
   │ QQuery 45 │        38.19 / 39.60 ±1.09 / 41.10 ms │        37.36 / 37.73 ±0.25 / 38.08 ms │     no change │
   │ QQuery 46 │        12.08 / 12.39 ±0.25 / 12.76 ms │        11.61 / 11.87 ±0.23 / 12.20 ms │     no change │
   │ QQuery 47 │     231.84 / 237.58 ±4.81 / 244.97 ms │     219.46 / 224.85 ±3.48 / 229.56 ms │ +1.06x faster │
   │ QQuery 48 │        94.79 / 96.77 ±1.69 / 99.64 ms │        94.07 / 95.43 ±1.16 / 97.33 ms │     no change │
   │ QQuery 49 │        75.34 / 76.17 ±1.00 / 77.99 ms │        74.70 / 75.65 ±0.65 / 76.64 ms │     no change │
   │ QQuery 50 │        58.18 / 59.78 ±2.09 / 63.83 ms │        58.16 / 59.06 ±0.84 / 60.64 ms │     no change │
   │ QQuery 51 │        91.99 / 93.82 ±1.35 / 96.11 ms │        91.62 / 93.66 ±1.71 / 96.49 ms │     no change │
   │ QQuery 52 │        23.86 / 24.09 ±0.13 / 24.22 ms │        23.47 / 23.67 ±0.11 / 23.80 ms │     no change │
   │ QQuery 53 │        29.17 / 29.34 ±0.12 / 29.52 ms │        28.41 / 28.68 ±0.21 / 29.03 ms │     no change │
   │ QQuery 54 │        54.37 / 56.40 ±2.90 / 62.12 ms │        54.02 / 54.61 ±0.76 / 56.02 ms │     no change │
   │ QQuery 55 │        23.66 / 23.92 ±0.25 / 24.32 ms │        22.98 / 23.35 ±0.59 / 24.52 ms │     no change │
   │ QQuery 56 │        38.59 / 39.01 ±0.27 / 39.33 ms │        37.79 / 39.49 ±1.56 / 42.37 ms │     no change │
   │ QQuery 57 │     173.33 / 174.73 ±1.52 / 177.66 ms │     173.80 / 175.13 ±1.34 / 177.44 ms │     no change │
   │ QQuery 58 │     111.69 / 112.99 ±0.84 / 114.25 ms │     110.28 / 112.37 ±2.58 / 117.33 ms │     no change │
   │ QQuery 59 │     117.92 / 120.37 ±1.82 / 122.80 ms │     117.62 / 118.48 ±1.31 / 121.06 ms │     no change │
   │ QQuery 60 │        38.66 / 39.66 ±0.53 / 40.21 ms │        38.99 / 39.82 ±0.94 / 41.61 ms │     no change │
   │ QQuery 61 │        12.25 / 12.49 ±0.15 / 12.65 ms │        12.15 / 12.28 ±0.14 / 12.52 ms │     no change │
   │ QQuery 62 │        46.36 / 46.68 ±0.18 / 46.87 ms │        45.58 / 46.40 ±0.58 / 47.35 ms │     no change │
   │ QQuery 63 │        29.47 / 31.25 ±3.12 / 37.46 ms │        29.04 / 30.20 ±1.71 / 33.59 ms │     no change │
   │ QQuery 64 │     401.64 / 405.95 ±4.48 / 413.91 ms │     404.29 / 406.72 ±3.15 / 412.83 ms │     no change │
   │ QQuery 65 │     121.31 / 124.89 ±2.90 / 129.98 ms │     122.20 / 123.29 ±0.63 / 123.90 ms │     no change │
   │ QQuery 66 │        80.58 / 82.92 ±3.84 / 90.54 ms │        79.57 / 82.16 ±3.65 / 89.39 ms │     no change │
   │ QQuery 67 │     241.97 / 247.21 ±4.46 / 252.96 ms │     238.00 / 242.71 ±4.78 / 251.22 ms │     no change │
   │ QQuery 68 │        12.01 / 12.14 ±0.12 / 12.32 ms │        11.88 / 12.10 ±0.26 / 12.60 ms │     no change │
   │ QQuery 69 │        56.60 / 57.00 ±0.34 / 57.45 ms │        56.32 / 56.61 ±0.27 / 57.13 ms │     no change │
   │ QQuery 70 │     105.47 / 113.31 ±8.41 / 129.65 ms │     104.66 / 109.72 ±4.39 / 115.79 ms │     no change │
   │ QQuery 71 │        34.46 / 34.66 ±0.26 / 35.16 ms │        34.29 / 34.94 ±0.83 / 36.57 ms │     no change │
   │ QQuery 72 │ 2033.09 / 2067.92 ±20.33 / 2089.90 ms │ 1999.15 / 2059.49 ±41.44 / 2127.87 ms │     no change │
   │ QQuery 73 │        10.11 / 10.21 ±0.12 / 10.43 ms │         9.91 / 10.08 ±0.22 / 10.52 ms │     no change │
   │ QQuery 74 │     177.93 / 184.41 ±5.29 / 193.86 ms │     184.55 / 187.29 ±2.76 / 190.79 ms │     no change │
   │ QQuery 75 │     147.09 / 151.99 ±5.76 / 163.17 ms │     149.15 / 150.02 ±0.61 / 150.63 ms │     no change │
   │ QQuery 76 │        34.91 / 35.29 ±0.29 / 35.79 ms │        35.40 / 35.55 ±0.15 / 35.81 ms │     no change │
   │ QQuery 77 │        60.35 / 60.68 ±0.30 / 61.04 ms │        61.80 / 62.61 ±0.91 / 64.24 ms │     no change │
   │ QQuery 78 │     193.99 / 196.87 ±3.27 / 201.58 ms │     197.97 / 202.26 ±2.73 / 205.64 ms │     no change │
   │ QQuery 79 │        66.98 / 68.70 ±2.01 / 71.80 ms │        66.87 / 68.61 ±2.35 / 73.17 ms │     no change │
   │ QQuery 80 │       97.60 / 98.75 ±1.05 / 100.03 ms │       98.42 / 99.91 ±0.88 / 100.87 ms │     no change │
   │ QQuery 81 │        25.61 / 25.86 ±0.27 / 26.21 ms │        26.41 / 26.66 ±0.29 / 27.23 ms │     no change │
   │ QQuery 82 │        15.93 / 19.10 ±5.87 / 30.83 ms │        16.54 / 17.60 ±1.90 / 21.40 ms │ +1.09x faster │
   │ QQuery 83 │        40.24 / 40.61 ±0.25 / 40.99 ms │        40.51 / 42.09 ±2.20 / 46.42 ms │     no change │
   │ QQuery 84 │        30.20 / 30.89 ±0.79 / 32.45 ms │        30.04 / 30.63 ±0.76 / 32.11 ms │     no change │
   │ QQuery 85 │     106.08 / 107.10 ±0.77 / 108.12 ms │     105.69 / 106.43 ±0.43 / 106.91 ms │     no change │
   │ QQuery 86 │        25.38 / 27.69 ±4.12 / 35.91 ms │        25.65 / 27.25 ±2.83 / 32.91 ms │     no change │
   │ QQuery 87 │        63.24 / 64.14 ±0.98 / 65.82 ms │        61.92 / 63.74 ±1.34 / 66.06 ms │     no change │
   │ QQuery 88 │        62.53 / 63.13 ±0.36 / 63.64 ms │        63.40 / 63.78 ±0.22 / 64.07 ms │     no change │
   │ QQuery 89 │        35.03 / 35.53 ±0.47 / 36.28 ms │        35.28 / 35.51 ±0.16 / 35.78 ms │     no change │
   │ QQuery 90 │        17.09 / 17.35 ±0.26 / 17.78 ms │        17.09 / 17.28 ±0.23 / 17.71 ms │     no change │
   │ QQuery 91 │        45.80 / 48.12 ±3.83 / 55.76 ms │        46.24 / 47.47 ±1.31 / 49.33 ms │     no change │
   │ QQuery 92 │        29.62 / 30.56 ±0.94 / 32.33 ms │        29.44 / 30.02 ±0.42 / 30.53 ms │     no change │
   │ QQuery 93 │        49.33 / 50.34 ±0.80 / 51.18 ms │        49.56 / 50.69 ±1.16 / 52.93 ms │     no change │
   │ QQuery 94 │        37.44 / 37.96 ±0.30 / 38.33 ms │        38.00 / 38.39 ±0.25 / 38.69 ms │     no change │
   │ QQuery 95 │        79.35 / 80.89 ±1.53 / 83.76 ms │        80.35 / 81.56 ±1.27 / 83.93 ms │     no change │
   │ QQuery 96 │        23.99 / 25.04 ±1.71 / 28.45 ms │        23.99 / 24.19 ±0.28 / 24.72 ms │     no change │
   │ QQuery 97 │        46.41 / 47.01 ±0.35 / 47.44 ms │        46.79 / 47.50 ±0.75 / 48.91 ms │     no change │
   │ QQuery 98 │        41.75 / 42.28 ±0.52 / 43.11 ms │        42.32 / 43.11 ±0.58 / 43.90 ms │     no change │
   │ QQuery 99 │        69.72 / 71.28 ±2.60 / 76.46 ms │        69.72 / 70.47 ±1.02 / 72.49 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 9803.24ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 9781.20ms │
   │ Average Time (HEAD)                                  │   99.02ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   98.80ms │
   │ Queries Faster                                       │         3 │
   │ Queries Slower                                       │         1 │
   │ Queries with No Change                               │        95 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 1.9 GiB |
   | Avg memory | 1.4 GiB |
   | CPU user | 222.2s |
   | CPU sys | 5.6s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 2.2 GiB |
   | Avg memory | 1.6 GiB |
   | CPU user | 216.9s |
   | CPU sys | 5.7s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5374321355

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5374297430)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5374297430-1851-d4grz 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5373794237

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5373757117)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5373757117-1848-5sx5t 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (4865205f97c7ad3ef19d44e2e2275e14304c0d48) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..4865205f97c7ad3ef19d44e2e2275e14304c0d48)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5400052719

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5400020690)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5400020690-1908-t5p4g 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8c1ada76c5bf9da164594ac9ff6609c885) to 26b40dd (merge-base) [diff](https://github.com/apache/datafusion/compare/26b40dd781d2eeab1acc38370a7069a8b9ebe450..35dd8d8c1ada76c5bf9da164594ac9ff6609c885)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5400270245

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5400020690)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8c1ada76c5bf9da164594ac9ff6609c885) to 26b40dd (merge-base) [diff](https://github.com/apache/datafusion/compare/26b40dd781d2eeab1acc38370a7069a8b9ebe450..35dd8d8c1ada76c5bf9da164594ac9ff6609c885)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │    1.21 ms │                               1.20 ms │     no change │
   │ QQuery 1  │   11.54 ms │                              11.95 ms │     no change │
   │ QQuery 2  │   35.69 ms │                              36.38 ms │     no change │
   │ QQuery 3  │   30.76 ms │                              31.11 ms │     no change │
   │ QQuery 4  │  222.36 ms │                             223.84 ms │     no change │
   │ QQuery 5  │  270.71 ms │                             272.10 ms │     no change │
   │ QQuery 6  │    1.24 ms │                               1.25 ms │     no change │
   │ QQuery 7  │   13.03 ms │                              13.13 ms │     no change │
   │ QQuery 8  │  326.37 ms │                             325.71 ms │     no change │
   │ QQuery 9  │  456.11 ms │                             456.91 ms │     no change │
   │ QQuery 10 │   70.85 ms │                              70.92 ms │     no change │
   │ QQuery 11 │   82.07 ms │                              81.60 ms │     no change │
   │ QQuery 12 │  265.78 ms │                             267.45 ms │     no change │
   │ QQuery 13 │  364.36 ms │                             373.98 ms │     no change │
   │ QQuery 14 │  284.46 ms │                             283.18 ms │     no change │
   │ QQuery 15 │  270.99 ms │                             274.76 ms │     no change │
   │ QQuery 16 │  611.87 ms │                             613.28 ms │     no change │
   │ QQuery 17 │  616.74 ms │                             613.62 ms │     no change │
   │ QQuery 18 │ 1250.11 ms │                            1297.89 ms │     no change │
   │ QQuery 19 │   27.49 ms │                              27.27 ms │     no change │
   │ QQuery 20 │  517.16 ms │                             516.10 ms │     no change │
   │ QQuery 21 │  519.43 ms │                             514.84 ms │     no change │
   │ QQuery 22 │  989.84 ms │                             991.20 ms │     no change │
   │ QQuery 23 │ 3055.04 ms │                            3100.77 ms │     no change │
   │ QQuery 24 │   41.21 ms │                              41.27 ms │     no change │
   │ QQuery 25 │  110.81 ms │                             111.01 ms │     no change │
   │ QQuery 26 │   42.42 ms │                              42.43 ms │     no change │
   │ QQuery 27 │  513.08 ms │                             514.18 ms │     no change │
   │ QQuery 28 │ 2874.53 ms │                            2898.77 ms │     no change │
   │ QQuery 29 │   40.56 ms │                              40.49 ms │     no change │
   │ QQuery 30 │  302.15 ms │                             298.37 ms │     no change │
   │ QQuery 31 │  275.55 ms │                             279.35 ms │     no change │
   │ QQuery 32 │  945.40 ms │                             957.83 ms │     no change │
   │ QQuery 33 │ 1495.10 ms │                            1443.54 ms │     no change │
   │ QQuery 34 │ 1499.45 ms │                            1474.46 ms │     no change │
   │ QQuery 35 │  281.06 ms │                             283.12 ms │     no change │
   │ QQuery 36 │   68.73 ms │                              66.47 ms │     no change │
   │ QQuery 37 │   35.25 ms │                              36.21 ms │     no change │
   │ QQuery 38 │   41.79 ms │                              41.89 ms │     no change │
   │ QQuery 39 │  155.44 ms │                             145.66 ms │ +1.07x faster │
   │ QQuery 40 │   14.83 ms │                              14.10 ms │     no change │
   │ QQuery 41 │   13.95 ms │                              13.72 ms │     no change │
   │ QQuery 42 │   13.50 ms │                              13.01 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 19060.03ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 19116.34ms │
   │ Average Time (HEAD)                                  │   443.26ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   444.57ms │
   │ Queries Faster                                       │          1 │
   │ Queries Slower                                       │          0 │
   │ Queries with No Change                               │         42 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.21 / 3.90 ±5.32 / 14.54 ms │          1.20 / 4.02 ±5.50 / 15.02 ms │     no change │
   │ QQuery 1  │        11.54 / 11.94 ±0.32 / 12.52 ms │        11.95 / 12.31 ±0.31 / 12.84 ms │     no change │
   │ QQuery 2  │        35.69 / 35.98 ±0.29 / 36.45 ms │        36.38 / 36.62 ±0.20 / 36.96 ms │     no change │
   │ QQuery 3  │        30.76 / 31.64 ±0.74 / 32.56 ms │        31.11 / 31.69 ±0.94 / 33.57 ms │     no change │
   │ QQuery 4  │     222.36 / 224.18 ±1.25 / 225.62 ms │     223.84 / 227.42 ±3.66 / 234.02 ms │     no change │
   │ QQuery 5  │     270.71 / 273.92 ±1.85 / 275.55 ms │     272.10 / 274.67 ±2.33 / 278.65 ms │     no change │
   │ QQuery 6  │           1.24 / 1.39 ±0.21 / 1.81 ms │           1.25 / 1.39 ±0.22 / 1.82 ms │     no change │
   │ QQuery 7  │        13.03 / 13.10 ±0.07 / 13.22 ms │        13.13 / 13.27 ±0.11 / 13.47 ms │     no change │
   │ QQuery 8  │     326.37 / 330.10 ±3.84 / 335.59 ms │     325.71 / 331.21 ±2.96 / 334.15 ms │     no change │
   │ QQuery 9  │     456.11 / 461.00 ±4.11 / 466.79 ms │     456.91 / 462.54 ±4.14 / 468.72 ms │     no change │
   │ QQuery 10 │        70.85 / 75.35 ±8.02 / 91.35 ms │        70.92 / 71.60 ±0.49 / 72.32 ms │     no change │
   │ QQuery 11 │        82.07 / 85.08 ±5.27 / 95.59 ms │        81.60 / 82.08 ±0.63 / 83.32 ms │     no change │
   │ QQuery 12 │     265.78 / 271.47 ±3.72 / 276.46 ms │     267.45 / 272.57 ±3.51 / 276.54 ms │     no change │
   │ QQuery 13 │    364.36 / 378.92 ±11.00 / 396.80 ms │    373.98 / 390.62 ±20.56 / 429.80 ms │     no change │
   │ QQuery 14 │     284.46 / 289.15 ±3.83 / 294.95 ms │     283.18 / 287.79 ±3.12 / 293.00 ms │     no change │
   │ QQuery 15 │     270.99 / 277.02 ±5.93 / 287.39 ms │     274.76 / 281.82 ±5.36 / 291.08 ms │     no change │
   │ QQuery 16 │     611.87 / 622.33 ±8.03 / 631.91 ms │    613.28 / 630.84 ±11.76 / 644.76 ms │     no change │
   │ QQuery 17 │    616.74 / 632.88 ±25.21 / 682.85 ms │    613.62 / 629.88 ±11.51 / 646.66 ms │     no change │
   │ QQuery 18 │ 1250.11 / 1291.84 ±45.45 / 1380.14 ms │ 1297.89 / 1317.19 ±11.30 / 1329.61 ms │     no change │
   │ QQuery 19 │        27.49 / 31.18 ±6.64 / 44.44 ms │       27.27 / 40.90 ±17.48 / 70.38 ms │  1.31x slower │
   │ QQuery 20 │    517.16 / 526.88 ±11.16 / 547.43 ms │    516.10 / 530.44 ±12.17 / 548.28 ms │     no change │
   │ QQuery 21 │     519.43 / 522.37 ±1.97 / 525.53 ms │     514.84 / 529.62 ±9.34 / 539.51 ms │     no change │
   │ QQuery 22 │    989.84 / 997.35 ±8.44 / 1013.22 ms │    991.20 / 999.53 ±7.23 / 1009.38 ms │     no change │
   │ QQuery 23 │ 3055.04 / 3100.92 ±36.47 / 3153.10 ms │ 3100.77 / 3144.67 ±39.44 / 3191.26 ms │     no change │
   │ QQuery 24 │        41.21 / 43.77 ±4.35 / 52.44 ms │        41.27 / 43.14 ±2.07 / 46.25 ms │     no change │
   │ QQuery 25 │     110.81 / 114.95 ±4.68 / 121.06 ms │     111.01 / 113.84 ±5.47 / 124.78 ms │     no change │
   │ QQuery 26 │        42.42 / 43.91 ±1.69 / 46.96 ms │        42.43 / 45.06 ±2.31 / 48.03 ms │     no change │
   │ QQuery 27 │     513.08 / 516.87 ±3.66 / 523.87 ms │     514.18 / 516.58 ±1.66 / 518.41 ms │     no change │
   │ QQuery 28 │ 2874.53 / 2902.31 ±19.95 / 2926.27 ms │  2898.77 / 2908.61 ±7.73 / 2920.44 ms │     no change │
   │ QQuery 29 │       40.56 / 59.69 ±22.96 / 91.94 ms │        40.49 / 41.71 ±1.81 / 45.29 ms │ +1.43x faster │
   │ QQuery 30 │     302.15 / 307.54 ±3.66 / 313.34 ms │     298.37 / 311.93 ±9.79 / 327.33 ms │     no change │
   │ QQuery 31 │     275.55 / 288.07 ±7.12 / 296.75 ms │    279.35 / 297.22 ±13.65 / 321.61 ms │     no change │
   │ QQuery 32 │    945.40 / 965.74 ±15.55 / 988.43 ms │     957.83 / 963.68 ±7.74 / 977.84 ms │     no change │
   │ QQuery 33 │ 1495.10 / 1516.01 ±17.58 / 1542.07 ms │ 1443.54 / 1468.45 ±15.68 / 1487.63 ms │     no change │
   │ QQuery 34 │ 1499.45 / 1529.09 ±25.93 / 1574.14 ms │  1474.46 / 1487.91 ±8.75 / 1500.56 ms │     no change │
   │ QQuery 35 │    281.06 / 316.76 ±48.81 / 411.89 ms │    283.12 / 322.01 ±54.60 / 428.37 ms │     no change │
   │ QQuery 36 │        68.73 / 72.95 ±3.55 / 77.90 ms │        66.47 / 73.43 ±6.41 / 85.40 ms │     no change │
   │ QQuery 37 │       35.25 / 44.95 ±14.02 / 71.87 ms │        36.21 / 38.27 ±1.05 / 39.03 ms │ +1.17x faster │
   │ QQuery 38 │        41.79 / 43.89 ±1.49 / 46.46 ms │        41.89 / 45.62 ±4.00 / 53.33 ms │     no change │
   │ QQuery 39 │     155.44 / 163.33 ±7.69 / 175.33 ms │     145.66 / 150.82 ±5.83 / 161.97 ms │ +1.08x faster │
   │ QQuery 40 │        14.83 / 15.26 ±0.41 / 16.03 ms │        14.10 / 15.16 ±1.97 / 19.11 ms │     no change │
   │ QQuery 41 │        13.95 / 14.26 ±0.30 / 14.80 ms │        13.72 / 13.93 ±0.17 / 14.18 ms │     no change │
   │ QQuery 42 │        13.50 / 13.65 ±0.10 / 13.78 ms │        13.01 / 15.63 ±4.76 / 25.14 ms │  1.15x slower │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 19462.87ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 19477.69ms │
   │ Average Time (HEAD)                                  │   452.62ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   452.97ms │
   │ Queries Faster                                       │          3 │
   │ Queries Slower                                       │          2 │
   │ Queries with No Change                               │         38 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 11.0 GiB |
   | Avg memory | 4.2 GiB |
   | CPU user | 993.5s |
   | CPU sys | 71.4s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 12.9 GiB |
   | Avg memory | 4.8 GiB |
   | CPU user | 991.7s |
   | CPU sys | 72.3s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5674937369

   Hi @Rich-T-kid This looks good to me at the moment. Let's resolve the conflicts and re-run the benchmarks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675329963

   Hi @kumarUjjawal, you asked to view the benchmark queue (https://github.com/apache/datafusion/pull/24227#issuecomment-5675329490).
   
   | Comment | Repo | PR | User | Benchmarks | Status |
   | --- | --- | --- | --- | --- | --- |
   | [#5675290885](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885) | apache/datafusion | #24227 | kumarUjjawal | ["tpch"] | running |
   | [#5675290885](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885) | apache/datafusion | #24227 | kumarUjjawal | ["tpcds"] | running |
   | [#5675290885](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885) | apache/datafusion | #24227 | kumarUjjawal | ["clickbench_partitioned"] | running |
   | [#5675290885](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885) | apache/datafusion | #24227 | kumarUjjawal | ["wide_schema"] | running |
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675682727

   Benchmark for [this request](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
   
   Benchmarks requested: `wide_schema`
   
   <details><summary>Runner log (last 40 lines)</summary>
   
   ```
   2026-09-15T05:38:30.475334Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/24227, benchmarks=wide_schema
   2026-09-15T05:38:30.534029Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
   2026-09-15T05:38:30.534127Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
   2026-09-15T05:38:35.542760Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/24227/head:rich-T-kid/introduce-rle-parquet-flag", "main"], cwd="/workspace/datafusion-branch"
   2026-09-15T05:38:40.544667Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "rich-T-kid/introduce-rle-parquet-flag"], cwd="/workspace/datafusion-branch"
   2026-09-15T05:38:45.546276Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
   2026-09-15T05:38:50.548946Z  INFO benchmark_controller::runner::bench_datafusion === Checking out custom changed ref === changed_ref=cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f
   2026-09-15T05:38:50.548962Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/24227/head"], cwd="/workspace/datafusion-branch"
   2026-09-15T05:38:55.551151Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin"], cwd="/workspace/datafusion-branch"
   2026-09-15T05:39:00.553444Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["-c", "advice.detachedHead=false", "checkout", "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"], cwd="/workspace/datafusion-branch"
   2026-09-15T05:39:05.555440Z  INFO benchmark_controller::runner::bench_datafusion === Cloning merge-base ===
   2026-09-15T05:39:05.555458Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-base"], cwd="/"
   2026-09-15T05:39:10.557640Z  INFO benchmark_controller::runner::bench_datafusion === Checking out custom baseline ref === baseline_ref=cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f
   2026-09-15T05:39:10.557655Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/24227/head"], cwd="/workspace/datafusion-base"
   2026-09-15T05:39:15.559224Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin"], cwd="/workspace/datafusion-base"
   2026-09-15T05:39:20.561573Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["-c", "advice.detachedHead=false", "checkout", "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"], cwd="/workspace/datafusion-base"
   2026-09-15T05:39:25.563640Z  INFO benchmark_controller::runner::shell running command cmd=rustc, args=["--version"], cwd="/"
   2026-09-15T05:39:30.569501Z  INFO benchmark_controller::runner::shell running command cmd=cargo, args=["metadata", "--no-deps", "--format-version", "1"], cwd="/workspace/datafusion-branch/benchmarks"
   2026-09-15T05:39:35.574517Z  INFO benchmark_controller::runner::bench_datafusion === Compiling dfbench for PR branch and merge-base in parallel ===
   2026-09-15T05:39:35.601275Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["rev-parse", "HEAD"], cwd="/workspace/datafusion-branch"
   2026-09-15T05:39:40.609555Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["rev-parse", "HEAD"], cwd="/workspace/datafusion-base"
   2026-09-15T05:39:45.612219Z  INFO benchmark_controller::runner::controller_client post_comment _repo=apache/datafusion, _pr_number=24227, job_id=2367
   2026-09-15T05:39:46.649712Z  INFO benchmark_controller::runner::bench_datafusion === Waiting for builds ===
   2026-09-15T05:53:32.776134Z  INFO benchmark_controller::runner::bench_datafusion === Builds complete ===
   2026-09-15T05:53:32.776146Z  INFO benchmark_controller::runner::bench_datafusion === Setting up bench runner ===
   2026-09-15T05:53:32.776156Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-bench"], cwd="/"
   2026-09-15T05:53:37.778745Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["-c", "advice.detachedHead=false", "checkout", "origin/main"], cwd="/workspace/datafusion-bench"
   2026-09-15T05:53:42.781261Z  INFO benchmark_controller::runner::shell running command cmd=cp, args=["-r", "/data/tpch-answers/.", "/workspace/datafusion-bench/benchmarks/data/tpch_sf1/answers"], cwd="/"
   2026-09-15T05:53:47.783876Z  INFO benchmark_controller::runner::shell running command cmd=cp, args=["-r", "/data/tpch-answers/.", "/workspace/datafusion-bench/benchmarks/data/tpch_sf10/answers"], cwd="/"
   2026-09-15T05:53:52.785214Z  INFO benchmark_controller::runner::bench_datafusion ** Creating data if needed for wide_schema **
   2026-09-15T05:53:52.785901Z  INFO benchmark_controller::runner::shell running command cmd=/scripts/cache_data.sh, args=["wide_schema", "/workspace/datafusion-bench/benchmarks"], cwd="/workspace/datafusion-bench/benchmarks"
   2026-09-15T06:06:12.956609Z  INFO benchmark_controller::runner::bench_datafusion ** Running wide_schema baseline **
   2026-09-15T06:06:12.956752Z  INFO benchmark_controller::runner::shell running command (monitored) cmd=env, args=["DATAFUSION_DIR=/workspace/datafusion-base", "RESULTS_NAME=HEAD", "DATAFUSION_RUNTIME_TEMP_DIRECTORY=/workspace/spill-base-wide_schema", "SQL_CARGO_COMMAND=cargo bench --bench sql -- --save-baseline HEAD", "DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY=false", "./bench.sh", "run", "wide_schema"], cwd="/workspace/datafusion-bench/benchmarks"
   ```
   
   </details>
   
   <details><summary>Kubernetes message</summary>
   
   ```
   Job has reached the specified backoff limit
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3911164931


##########
datafusion/datasource-parquet/src/metadata.rs:
##########
@@ -439,6 +448,68 @@ impl<'a> DFParquetMetadata<'a> {
                     .coerce()
             })
             .unwrap_or(schema);
+
+        let schema = if self.enable_rle_to_dictionary {
+            let schema_descr = file_metadata.schema_descr();
+            // Top-level columns that have a dictionary page in at least one row group.
+            let dict_cols: HashSet<String> = metadata
+                .row_groups()
+                .iter()
+                .flat_map(|rg| {
+                    rg.columns()
+                        .iter()
+                        .enumerate()
+                        .filter_map(|(col_idx, col)| {
+                            col.dictionary_page_offset()?;

Review Comment:
   These hosted benchmarks do not appear to exercise this feature: enable_rle_to_dictionary defaults to false, and the reported run configuration does not enable it. They also target 35dd8d8, not the current head. Since a dictionary-page offset does not guarantee that later pages remain dictionary encoded, could we add current-head enabled/disabled comparisons for both low-cardinality data and a high-cardinality dictionary-fallback column, including peak memory?



##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -135,6 +157,114 @@ pub fn apply_file_schema_type_coercions(
     ))
 }
 
+fn dictionary_value_type(data_type: &DataType) -> Option<&DataType> {
+    match data_type {
+        DataType::Dictionary(_, value_type) => Some(value_type.as_ref()),
+        _ => None,
+    }
+}
+
+// Find the value type that can represent both sides without narrowing offsets
+// or crossing string/binary families.
+fn common_dictionary_value_type(
+    field_type: &DataType,
+    dictionary_value_type: &DataType,
+) -> Option<DataType> {
+    let field_type = match field_type {
+        DataType::Dictionary(_, field_value_type) => field_value_type.as_ref(),
+        _ => field_type,
+    };
+
+    match (field_type, dictionary_value_type) {
+        (DataType::Utf8, DataType::Utf8) => Some(DataType::Utf8),
+        (DataType::Utf8 | DataType::LargeUtf8, DataType::Utf8 | DataType::LargeUtf8) => {
+            Some(DataType::LargeUtf8)
+        }
+        (DataType::Binary, DataType::Binary) => Some(DataType::Binary),
+        (
+            DataType::Binary | DataType::LargeBinary,
+            DataType::Binary | DataType::LargeBinary,
+        ) => Some(DataType::LargeBinary),
+        _ => None,
+    }
+}
+
+/// Allows safe widening into the table dictionary type.
+/// - `Utf8` to `Dictionary(Int32, LargeUtf8)`: allowed
+/// - `LargeBinary` to `Dictionary(Int32, Binary)`: rejected
+fn can_promote_to_dictionary_type(
+    file_field_type: &DataType,
+    table_dictionary_type: &DataType,
+) -> bool {
+    dictionary_value_type(table_dictionary_type).is_some_and(|dictionary_value_type| {
+        common_dictionary_value_type(file_field_type, dictionary_value_type)
+            .is_some_and(|common_type| &common_type == dictionary_value_type)
+    })
+}
+
+/// Normalize per-file schemas so that a column promoted to `Dictionary` in
+/// *any* file is promoted to the same `Dictionary` type in *all* files.
+///
+/// This lets [`Schema::try_merge`] accept directories that mix dictionary and
+/// plain encodings for the same column.
+pub(crate) fn uniform_dict_schemas(schemas: Vec<Schema>) -> Vec<Schema> {
+    // First pass: record the dictionary type for every column that is Dictionary in
+    // at least one schema.
+    let mut dict_types: HashMap<String, DataType> = HashMap::new();
+    for schema in &schemas {
+        for field in schema.fields() {
+            if matches!(field.data_type(), DataType::Dictionary(_, _)) {
+                dict_types
+                    .entry(field.name().clone())
+                    .or_insert_with(|| field.data_type().clone());
+            }
+        }
+    }
+    if dict_types.is_empty() {
+        return schemas;
+    }
+
+    // Widen the recorded dictionary value type before promoting plain fields.
+    for schema in &schemas {
+        for field in schema.fields() {
+            let Some(dict_type) = dict_types.get_mut(field.name()) else {
+                continue;
+            };
+            let DataType::Dictionary(key_type, value_type) = dict_type else {

Review Comment:
   dict_types preserves the key type from the first dictionary schema, but can_promote_to_dictionary_type compares only value types. A mixed Dictionary(Int8, Utf8) / Dictionary(Int32, Utf8) directory can therefore be normalized to Int8, and Arrow will fail once a later dictionary has more than 127 values. Could this choose a safe common key type at least Int32 for feature-promoted columns and add a mixed-key/high-cardinality regression test?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3916700865


##########
datafusion/datasource-parquet/src/metadata.rs:
##########
@@ -439,6 +448,68 @@ impl<'a> DFParquetMetadata<'a> {
                     .coerce()
             })
             .unwrap_or(schema);
+
+        let schema = if self.enable_rle_to_dictionary {
+            let schema_descr = file_metadata.schema_descr();
+            // Top-level columns that have a dictionary page in at least one row group.
+            let dict_cols: HashSet<String> = metadata
+                .row_groups()
+                .iter()
+                .flat_map(|rg| {
+                    rg.columns()
+                        .iter()
+                        .enumerate()
+                        .filter_map(|(col_idx, col)| {
+                            col.dictionary_page_offset()?;

Review Comment:
   these exist in the PR thread. I didn't do a good job distinguishing when the flag was on or off.
   
   I can enable the flag again and run the benchmarks. from the previous times that I did run this it caused regressions in query performance as expected.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5628877339

   @kumarUjjawal could you trigger the CI for this pr? its also ready for review when you get a chance 🙇 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364887275

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364728282)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │    1.25 ms │                               1.20 ms │     no change │
   │ QQuery 1  │   11.85 ms │                              11.84 ms │     no change │
   │ QQuery 2  │   36.91 ms │                              36.42 ms │     no change │
   │ QQuery 3  │   31.16 ms │                              31.14 ms │     no change │
   │ QQuery 4  │  225.04 ms │                             225.90 ms │     no change │
   │ QQuery 5  │  274.21 ms │                            1470.40 ms │  5.36x slower │
   │ QQuery 6  │    1.31 ms │                               1.22 ms │ +1.07x faster │
   │ QQuery 7  │   13.33 ms │                              13.14 ms │     no change │
   │ QQuery 8  │  323.36 ms │                             339.24 ms │     no change │
   │ QQuery 9  │  449.97 ms │                             463.93 ms │     no change │
   │ QQuery 10 │   70.26 ms │                             204.82 ms │  2.92x slower │
   │ QQuery 11 │   81.55 ms │                             215.10 ms │  2.64x slower │
   │ QQuery 12 │  268.08 ms │                            1525.75 ms │  5.69x slower │
   │ QQuery 13 │  361.64 ms │                            2055.28 ms │  5.68x slower │
   │ QQuery 14 │  281.31 ms │                            1618.77 ms │  5.75x slower │
   │ QQuery 15 │  270.02 ms │                             280.14 ms │     no change │
   │ QQuery 16 │  613.25 ms │                            3559.52 ms │  5.80x slower │
   │ QQuery 17 │  611.98 ms │                            3503.14 ms │  5.72x slower │
   │ QQuery 18 │ 1240.88 ms │                            6460.11 ms │  5.21x slower │
   │ QQuery 19 │   27.53 ms │                              27.40 ms │     no change │
   │ QQuery 20 │  516.97 ms │                            1045.42 ms │  2.02x slower │
   │ QQuery 21 │  512.78 ms │                            1369.99 ms │  2.67x slower │
   │ QQuery 22 │  987.85 ms │                            3261.80 ms │  3.30x slower │
   │ QQuery 23 │ 3066.60 ms │                            9286.14 ms │  3.03x slower │
   │ QQuery 24 │   41.10 ms │                              96.14 ms │  2.34x slower │
   │ QQuery 25 │  111.03 ms │                             359.92 ms │  3.24x slower │
   │ QQuery 26 │   41.88 ms │                              97.35 ms │  2.32x slower │
   │ QQuery 27 │  514.38 ms │                            1131.24 ms │  2.20x slower │
   │ QQuery 28 │ 2878.29 ms │                            3521.51 ms │  1.22x slower │
   │ QQuery 29 │   41.13 ms │                              40.90 ms │     no change │
   │ QQuery 30 │  303.08 ms │                             545.64 ms │  1.80x slower │
   │ QQuery 31 │  291.70 ms │                             543.66 ms │  1.86x slower │
   │ QQuery 32 │  931.39 ms │                            1005.71 ms │  1.08x slower │
   │ QQuery 33 │ 1455.91 ms │                           13765.01 ms │  9.45x slower │
   │ QQuery 34 │ 1472.45 ms │                           13636.56 ms │  9.26x slower │
   │ QQuery 35 │  284.66 ms │                             280.93 ms │     no change │
   │ QQuery 36 │   66.44 ms │                             167.22 ms │  2.52x slower │
   │ QQuery 37 │   36.03 ms │                              96.69 ms │  2.68x slower │
   │ QQuery 38 │   41.86 ms │                              80.22 ms │  1.92x slower │
   │ QQuery 39 │  146.05 ms │                             370.57 ms │  2.54x slower │
   │ QQuery 40 │   14.10 ms │                              14.30 ms │     no change │
   │ QQuery 41 │   14.20 ms │                              13.73 ms │     no change │
   │ QQuery 42 │   13.55 ms │                              13.17 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 18978.33ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 72788.30ms │
   │ Average Time (HEAD)                                  │   441.36ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  1692.75ms │
   │ Queries Faster                                       │          1 │
   │ Queries Slower                                       │         27 │
   │ Queries with No Change                               │         15 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃     rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.25 / 3.95 ±5.35 / 14.64 ms │              1.20 / 3.89 ±5.33 / 14.55 ms │     no change │
   │ QQuery 1  │        11.85 / 12.15 ±0.16 / 12.28 ms │            11.84 / 11.98 ±0.13 / 12.20 ms │     no change │
   │ QQuery 2  │        36.91 / 37.07 ±0.23 / 37.51 ms │            36.42 / 36.49 ±0.09 / 36.67 ms │     no change │
   │ QQuery 3  │        31.16 / 31.86 ±0.67 / 33.01 ms │            31.14 / 31.36 ±0.22 / 31.77 ms │     no change │
   │ QQuery 4  │     225.04 / 228.60 ±3.66 / 235.54 ms │         225.90 / 231.15 ±3.44 / 235.01 ms │     no change │
   │ QQuery 5  │     274.21 / 277.70 ±2.66 / 281.01 ms │     1470.40 / 1519.51 ±46.43 / 1605.62 ms │  5.47x slower │
   │ QQuery 6  │           1.31 / 1.46 ±0.24 / 1.93 ms │               1.22 / 1.36 ±0.22 / 1.78 ms │ +1.08x faster │
   │ QQuery 7  │        13.33 / 13.52 ±0.19 / 13.78 ms │           13.14 / 19.74 ±12.80 / 45.34 ms │  1.46x slower │
   │ QQuery 8  │     323.36 / 329.33 ±4.42 / 334.61 ms │         339.24 / 347.66 ±9.26 / 365.50 ms │  1.06x slower │
   │ QQuery 9  │     449.97 / 459.96 ±5.19 / 464.77 ms │         463.93 / 477.61 ±7.47 / 486.09 ms │     no change │
   │ QQuery 10 │        70.26 / 71.70 ±0.78 / 72.42 ms │         204.82 / 210.27 ±9.28 / 228.72 ms │  2.93x slower │
   │ QQuery 11 │        81.55 / 83.02 ±1.20 / 84.82 ms │         215.10 / 215.91 ±0.47 / 216.49 ms │  2.60x slower │
   │ QQuery 12 │     268.08 / 271.28 ±4.01 / 279.12 ms │     1525.75 / 1565.25 ±29.61 / 1597.44 ms │  5.77x slower │
   │ QQuery 13 │    361.64 / 374.53 ±10.98 / 388.45 ms │     2055.28 / 2129.46 ±56.53 / 2198.21 ms │  5.69x slower │
   │ QQuery 14 │     281.31 / 291.13 ±6.54 / 299.92 ms │     1618.77 / 1658.70 ±51.69 / 1757.86 ms │  5.70x slower │
   │ QQuery 15 │     270.02 / 281.64 ±9.79 / 294.07 ms │         280.14 / 290.99 ±6.89 / 298.58 ms │     no change │
   │ QQuery 16 │     613.25 / 618.69 ±4.97 / 625.86 ms │     3559.52 / 3656.96 ±74.44 / 3779.61 ms │  5.91x slower │
   │ QQuery 17 │     611.98 / 623.76 ±6.96 / 632.17 ms │    3503.14 / 3679.27 ±119.17 / 3824.98 ms │  5.90x slower │
   │ QQuery 18 │ 1240.88 / 1264.74 ±14.09 / 1281.91 ms │    6460.11 / 6750.28 ±196.49 / 7048.69 ms │  5.34x slower │
   │ QQuery 19 │       27.53 / 36.79 ±16.61 / 69.99 ms │            27.40 / 27.82 ±0.30 / 28.28 ms │ +1.32x faster │
   │ QQuery 20 │     516.97 / 528.11 ±9.24 / 544.32 ms │     1045.42 / 1064.94 ±16.54 / 1085.98 ms │  2.02x slower │
   │ QQuery 21 │     512.78 / 519.24 ±4.20 / 523.91 ms │      1369.99 / 1377.47 ±7.79 / 1390.63 ms │  2.65x slower │
   │ QQuery 22 │    987.85 / 995.69 ±6.16 / 1005.12 ms │     3261.80 / 3309.13 ±31.18 / 3358.91 ms │  3.32x slower │
   │ QQuery 23 │ 3066.60 / 3097.52 ±26.79 / 3128.45 ms │    9286.14 / 9421.53 ±141.86 / 9630.21 ms │  3.04x slower │
   │ QQuery 24 │        41.10 / 45.18 ±5.27 / 55.32 ms │          96.14 / 103.09 ±7.91 / 115.49 ms │  2.28x slower │
   │ QQuery 25 │     111.03 / 112.44 ±1.39 / 114.96 ms │         359.92 / 364.31 ±3.40 / 370.36 ms │  3.24x slower │
   │ QQuery 26 │        41.88 / 42.35 ±0.40 / 42.88 ms │           97.35 / 99.64 ±2.07 / 103.47 ms │  2.35x slower │
   │ QQuery 27 │     514.38 / 522.63 ±5.29 / 528.35 ms │      1131.24 / 1143.75 ±7.06 / 1153.06 ms │  2.19x slower │
   │ QQuery 28 │ 2878.29 / 2920.57 ±26.11 / 2955.02 ms │     3521.51 / 3558.89 ±22.95 / 3587.36 ms │  1.22x slower │
   │ QQuery 29 │       41.13 / 53.42 ±16.77 / 83.95 ms │            40.90 / 44.22 ±5.88 / 55.97 ms │ +1.21x faster │
   │ QQuery 30 │     303.08 / 311.22 ±7.03 / 321.37 ms │         545.64 / 552.82 ±6.20 / 563.80 ms │  1.78x slower │
   │ QQuery 31 │     291.70 / 297.20 ±4.95 / 305.93 ms │        543.66 / 556.28 ±10.93 / 574.33 ms │  1.87x slower │
   │ QQuery 32 │    931.39 / 954.42 ±20.10 / 984.42 ms │     1005.71 / 1090.31 ±73.97 / 1221.22 ms │  1.14x slower │
   │ QQuery 33 │ 1455.91 / 1471.48 ±11.68 / 1492.18 ms │ 13765.01 / 14072.13 ±224.61 / 14404.25 ms │  9.56x slower │
   │ QQuery 34 │ 1472.45 / 1495.42 ±12.09 / 1505.41 ms │ 13636.56 / 14064.81 ±485.68 / 14871.80 ms │  9.41x slower │
   │ QQuery 35 │    284.66 / 303.06 ±30.90 / 364.48 ms │       280.93 / 351.31 ±123.91 / 598.95 ms │  1.16x slower │
   │ QQuery 36 │        66.44 / 72.18 ±4.52 / 78.97 ms │         167.22 / 176.81 ±8.42 / 192.02 ms │  2.45x slower │
   │ QQuery 37 │        36.03 / 38.52 ±4.44 / 47.38 ms │          96.69 / 101.35 ±5.01 / 108.66 ms │  2.63x slower │
   │ QQuery 38 │        41.86 / 46.22 ±5.93 / 57.99 ms │            80.22 / 85.43 ±3.29 / 90.26 ms │  1.85x slower │
   │ QQuery 39 │     146.05 / 155.08 ±9.04 / 168.15 ms │        370.57 / 386.15 ±12.28 / 405.09 ms │  2.49x slower │
   │ QQuery 40 │        14.10 / 14.91 ±0.81 / 16.44 ms │            14.30 / 15.75 ±2.02 / 19.71 ms │  1.06x slower │
   │ QQuery 41 │        14.20 / 17.64 ±4.59 / 26.29 ms │            13.73 / 14.02 ±0.29 / 14.57 ms │ +1.26x faster │
   │ QQuery 42 │        13.55 / 13.70 ±0.09 / 13.84 ms │            13.17 / 14.25 ±1.74 / 17.71 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 19341.07ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 74834.05ms │
   │ Average Time (HEAD)                                  │   449.79ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  1740.33ms │
   │ Queries Faster                                       │          4 │
   │ Queries Slower                                       │         31 │
   │ Queries with No Change                               │          8 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 11.3 GiB |
   | Avg memory | 4.6 GiB |
   | CPU user | 993.6s |
   | CPU sys | 67.5s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 375.1s |
   | Peak memory | 12.8 GiB |
   | Avg memory | 5.8 GiB |
   | CPU user | 3281.3s |
   | CPU sys | 200.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364746749

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364728282)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5364728282-1828-n9hqq 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3962343162


##########
datafusion/sqllogictest/test_files/parquet_rle_to_dictionary.slt:
##########
@@ -0,0 +1,334 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Tests for datafusion.execution.parquet.enable_rle_to_dictionary. The flag
+# applies only when DataFusion infers the table schema; explicit schemas are
+# not promoted.
+
+# Write with dictionary_enabled=true to guarantee RLE_DICTIONARY encoding.
+query I
+COPY (
+  SELECT column1 AS product_category, column2 AS status
+  FROM (VALUES
+    ('electronics', 'active'),
+    ('clothing',    'active'),
+    ('electronics', 'inactive'),
+    ('books',       'inactive'),
+    ('furniture',   'pending'),
+    ('books',       'active')
+  )
+)
+TO 'test_files/scratch/parquet_rle_to_dictionary/products.parquet'
+STORED AS PARQUET
+OPTIONS ('format.dictionary_enabled' true);
+----
+6
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = false;
+
+statement ok
+CREATE EXTERNAL TABLE products_utf8
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/products.parquet';
+
+query TT
+SELECT DISTINCT arrow_typeof(product_category), arrow_typeof(status) FROM products_utf8;
+----
+Utf8View Utf8View
+
+query TI rowsort
+SELECT product_category, COUNT(*) FROM products_utf8 GROUP BY product_category;
+----
+books 2
+clothing 1
+electronics 2
+furniture 1
+
+statement ok
+DROP TABLE products_utf8;
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = true;
+
+statement ok
+CREATE EXTERNAL TABLE products_dict
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/products.parquet';
+
+query TT
+SELECT DISTINCT arrow_typeof(product_category), arrow_typeof(status) FROM products_dict;
+----
+Dictionary(Int32, Utf8) Dictionary(Int32, Utf8)
+
+query TI rowsort
+SELECT product_category, COUNT(*) FROM products_dict GROUP BY product_category;
+----
+books 2
+clothing 1
+electronics 2
+furniture 1
+
+# Predicates and aggregates must work against dictionary scan output.
+query TT rowsort
+SELECT product_category, status FROM products_dict WHERE status = 'active';
+----
+books active
+clothing active
+electronics active
+
+query TI rowsort
+SELECT product_category, COUNT(*) FROM products_dict WHERE status != 'inactive' GROUP BY product_category;
+----
+books 1
+clothing 1
+electronics 1
+furniture 1
+
+query TT
+SELECT DISTINCT arrow_typeof(product_category), arrow_typeof(product_count)
+FROM (
+  SELECT product_category, COUNT(*) AS product_count
+  FROM products_dict
+  WHERE status != 'inactive'
+  GROUP BY product_category
+);
+----
+Dictionary(Int32, Utf8) Int64
+
+statement ok
+DROP TABLE products_dict;
+
+# Per-column writer options: only columns with dictionary pages are promoted.
+
+statement ok
+COPY (
+  SELECT column1 AS env, column2 AS region, column3 AS build
+  FROM (VALUES
+    ('prod',    'us-east', 'debug'),
+    ('staging', 'us-west', 'release'),
+    ('prod',    'us-east', 'debug')
+  )
+)
+TO 'test_files/scratch/parquet_rle_to_dictionary/selective.parquet'
+STORED AS PARQUET
+OPTIONS (
+  'format.dictionary_enabled' false,
+  'format.dictionary_enabled::env' true,
+  'format.dictionary_enabled::region' true
+);
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = false;
+
+statement ok
+CREATE EXTERNAL TABLE selective_utf8
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/selective.parquet';
+
+query TTT
+SELECT DISTINCT arrow_typeof(env), arrow_typeof(region), arrow_typeof(build)
+FROM selective_utf8;
+----
+Utf8View Utf8View Utf8View
+
+statement ok
+DROP TABLE selective_utf8;
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = true;
+
+statement ok
+CREATE EXTERNAL TABLE selective_dict
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/selective.parquet';
+
+query TTT
+SELECT DISTINCT arrow_typeof(env), arrow_typeof(region), arrow_typeof(build)
+FROM selective_dict;
+----
+Dictionary(Int32, Utf8) Dictionary(Int32, Utf8) Utf8View
+
+statement ok
+DROP TABLE selective_dict;
+
+# Cross-file mixed encoding: one file has dictionary pages and one is plain.
+# Schema inference must merge both as Dictionary(Int32, Utf8).
+
+statement ok
+COPY (SELECT column1 AS category FROM (VALUES ('electronics'), ('electronics')))
+TO 'test_files/scratch/parquet_rle_to_dictionary/mixed/rle.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' true);
+
+statement ok
+COPY (SELECT column1 AS category FROM (VALUES ('books'), ('books')))
+TO 'test_files/scratch/parquet_rle_to_dictionary/mixed/plain.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' false);
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = true;
+
+statement ok
+CREATE EXTERNAL TABLE mixed_encoding
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/mixed/';
+
+query TI
+SELECT arrow_typeof(category), COUNT(*) FROM mixed_encoding GROUP BY arrow_typeof(category);
+----
+Dictionary(Int32, Utf8) 4
+
+statement ok
+DROP TABLE mixed_encoding;
+
+# Mixed encoding with high cardinality: the plain file contains more than 128
+# distinct values so a narrow (Int8) dictionary key would overflow. After the
+# fix, uniform_dict_schemas widens the shared key to at least Int32.
+# Both files must be scannable and return the correct row count.
+
+statement ok
+COPY (
+  SELECT v AS category
+  FROM (
+    VALUES
+      ('v000'),('v001'),('v002'),('v003'),('v004'),('v005'),('v006'),('v007'),
+      ('v008'),('v009'),('v010'),('v011'),('v012'),('v013'),('v014'),('v015'),
+      ('v016'),('v017'),('v018'),('v019'),('v020'),('v021'),('v022'),('v023'),
+      ('v024'),('v025'),('v026'),('v027'),('v028'),('v029'),('v030'),('v031'),
+      ('v032'),('v033'),('v034'),('v035'),('v036'),('v037'),('v038'),('v039'),
+      ('v040'),('v041'),('v042'),('v043'),('v044'),('v045'),('v046'),('v047'),
+      ('v048'),('v049'),('v050'),('v051'),('v052'),('v053'),('v054'),('v055'),
+      ('v056'),('v057'),('v058'),('v059'),('v060'),('v061'),('v062'),('v063'),
+      ('v064'),('v065'),('v066'),('v067'),('v068'),('v069'),('v070'),('v071'),
+      ('v072'),('v073'),('v074'),('v075'),('v076'),('v077'),('v078'),('v079'),
+      ('v080'),('v081'),('v082'),('v083'),('v084'),('v085'),('v086'),('v087'),
+      ('v088'),('v089'),('v090'),('v091'),('v092'),('v093'),('v094'),('v095'),
+      ('v096'),('v097'),('v098'),('v099'),('v100'),('v101'),('v102'),('v103'),
+      ('v104'),('v105'),('v106'),('v107'),('v108'),('v109'),('v110'),('v111'),
+      ('v112'),('v113'),('v114'),('v115'),('v116'),('v117'),('v118'),('v119'),
+      ('v120'),('v121'),('v122'),('v123'),('v124'),('v125'),('v126'),('v127'),
+      ('v128'),('v129')
+  ) AS t(v)
+)
+TO 'test_files/scratch/parquet_rle_to_dictionary/high_card/plain.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' false);
+
+statement ok
+COPY (SELECT column1 AS category FROM (VALUES ('rle_a'), ('rle_b'), ('rle_a')))
+TO 'test_files/scratch/parquet_rle_to_dictionary/high_card/rle.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' true);

Review Comment:
   updated the test to cast to a int8 first before overflowing https://github.com/apache/datafusion/commit/4d607b74af918f514fb4f521b9b307a3458a5546



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364966534

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364868497)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f09f1555c30e2f0b88db32a80d7bcd87062bcc5d) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f09f1555c30e2f0b88db32a80d7bcd87062bcc5d)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    5.82 ms │                               5.71 ms │     no change │
   │ QQuery 2  │   81.08 ms │                              81.97 ms │     no change │
   │ QQuery 3  │   29.28 ms │                              28.81 ms │     no change │
   │ QQuery 4  │  477.86 ms │                             648.14 ms │  1.36x slower │
   │ QQuery 5  │   52.42 ms │                              52.10 ms │     no change │
   │ QQuery 6  │   36.20 ms │                              34.86 ms │     no change │
   │ QQuery 7  │   93.76 ms │                              95.60 ms │     no change │
   │ QQuery 8  │   36.99 ms │                              37.03 ms │     no change │
   │ QQuery 9  │   51.00 ms │                              52.62 ms │     no change │
   │ QQuery 10 │   63.31 ms │                              63.56 ms │     no change │
   │ QQuery 11 │  318.79 ms │                             433.00 ms │  1.36x slower │
   │ QQuery 12 │   29.05 ms │                              25.98 ms │ +1.12x faster │
   │ QQuery 13 │  116.81 ms │                             118.22 ms │     no change │
   │ QQuery 14 │  414.22 ms │                             418.70 ms │     no change │
   │ QQuery 15 │   57.00 ms │                              67.17 ms │  1.18x slower │
   │ QQuery 16 │    6.70 ms │                               7.42 ms │  1.11x slower │
   │ QQuery 17 │   79.63 ms │                              81.83 ms │     no change │
   │ QQuery 18 │  122.77 ms │                             125.06 ms │     no change │
   │ QQuery 19 │   41.16 ms │                              41.77 ms │     no change │
   │ QQuery 20 │   35.29 ms │                              30.95 ms │ +1.14x faster │
   │ QQuery 21 │   17.16 ms │                              17.04 ms │     no change │
   │ QQuery 22 │   62.70 ms │                              72.31 ms │  1.15x slower │
   │ QQuery 23 │  341.24 ms │                             339.19 ms │     no change │
   │ QQuery 24 │  222.15 ms │                             222.82 ms │     no change │
   │ QQuery 25 │  109.81 ms │                             109.28 ms │     no change │
   │ QQuery 26 │   57.83 ms │                              58.42 ms │     no change │
   │ QQuery 27 │    6.32 ms │                               6.79 ms │  1.07x slower │
   │ QQuery 28 │   55.76 ms │                              60.38 ms │  1.08x slower │
   │ QQuery 29 │   96.44 ms │                              97.22 ms │     no change │
   │ QQuery 30 │   32.78 ms │                              38.92 ms │  1.19x slower │
   │ QQuery 31 │  110.02 ms │                             116.00 ms │  1.05x slower │
   │ QQuery 32 │   20.34 ms │                              20.00 ms │     no change │
   │ QQuery 33 │   37.75 ms │                              37.61 ms │     no change │
   │ QQuery 34 │    9.96 ms │                               9.62 ms │     no change │
   │ QQuery 35 │   72.55 ms │                              73.00 ms │     no change │
   │ QQuery 36 │    6.07 ms │                               6.31 ms │     no change │
   │ QQuery 37 │    6.88 ms │                               7.77 ms │  1.13x slower │
   │ QQuery 38 │   61.81 ms │                              77.80 ms │  1.26x slower │
   │ QQuery 39 │   90.34 ms │                              89.73 ms │     no change │
   │ QQuery 40 │   23.17 ms │                              23.70 ms │     no change │
   │ QQuery 41 │   11.35 ms │                              12.59 ms │  1.11x slower │
   │ QQuery 42 │   23.54 ms │                              23.76 ms │     no change │
   │ QQuery 43 │    5.44 ms │                               5.85 ms │  1.07x slower │
   │ QQuery 44 │    9.57 ms │                               9.92 ms │     no change │
   │ QQuery 45 │   37.72 ms │                              37.16 ms │     no change │
   │ QQuery 46 │   11.96 ms │                              11.95 ms │     no change │
   │ QQuery 47 │  224.47 ms │                             212.51 ms │ +1.06x faster │
   │ QQuery 48 │   95.41 ms │                              96.17 ms │     no change │
   │ QQuery 49 │   78.70 ms │                              76.06 ms │     no change │
   │ QQuery 50 │   62.03 ms │                              59.35 ms │     no change │
   │ QQuery 51 │   96.67 ms │                              92.03 ms │     no change │
   │ QQuery 52 │   25.37 ms │                              23.85 ms │ +1.06x faster │
   │ QQuery 53 │   30.78 ms │                              28.83 ms │ +1.07x faster │
   │ QQuery 54 │   57.35 ms │                              53.90 ms │ +1.06x faster │
   │ QQuery 55 │   24.68 ms │                              23.36 ms │ +1.06x faster │
   │ QQuery 56 │   41.37 ms │                              38.76 ms │ +1.07x faster │
   │ QQuery 57 │  188.44 ms │                             163.02 ms │ +1.16x faster │
   │ QQuery 58 │  119.06 ms │                             109.04 ms │ +1.09x faster │
   │ QQuery 59 │  122.99 ms │                             115.50 ms │ +1.06x faster │
   │ QQuery 60 │   41.85 ms │                              39.32 ms │ +1.06x faster │
   │ QQuery 61 │   13.51 ms │                              14.07 ms │     no change │
   │ QQuery 62 │   49.04 ms │                              48.57 ms │     no change │
   │ QQuery 63 │   31.18 ms │                              31.12 ms │     no change │
   │ QQuery 64 │  407.80 ms │                             444.77 ms │  1.09x slower │
   │ QQuery 65 │  124.63 ms │                             133.45 ms │  1.07x slower │
   │ QQuery 66 │   81.53 ms │                              87.86 ms │  1.08x slower │
   │ QQuery 67 │  234.88 ms │                             288.71 ms │  1.23x slower │
   │ QQuery 68 │   12.26 ms │                              13.47 ms │  1.10x slower │
   │ QQuery 69 │   57.06 ms │                              61.15 ms │  1.07x slower │
   │ QQuery 70 │  105.64 ms │                             109.70 ms │     no change │
   │ QQuery 71 │   34.99 ms │                              37.55 ms │  1.07x slower │
   │ QQuery 72 │ 1934.24 ms │                            1829.61 ms │ +1.06x faster │
   │ QQuery 73 │    9.88 ms │                              10.24 ms │     no change │
   │ QQuery 74 │  167.27 ms │                             234.36 ms │  1.40x slower │
   │ QQuery 75 │  146.53 ms │                             163.35 ms │  1.11x slower │
   │ QQuery 76 │   34.87 ms │                              36.09 ms │     no change │
   │ QQuery 77 │   60.60 ms │                              61.97 ms │     no change │
   │ QQuery 78 │  195.66 ms │                             197.02 ms │     no change │
   │ QQuery 79 │   66.58 ms │                              67.57 ms │     no change │
   │ QQuery 80 │   99.18 ms │                             102.62 ms │     no change │
   │ QQuery 81 │   25.89 ms │                              30.63 ms │  1.18x slower │
   │ QQuery 82 │   16.21 ms │                              17.34 ms │  1.07x slower │
   │ QQuery 83 │   39.43 ms │                              40.41 ms │     no change │
   │ QQuery 84 │   29.73 ms │                              33.06 ms │  1.11x slower │
   │ QQuery 85 │  105.92 ms │                             110.50 ms │     no change │
   │ QQuery 86 │   25.04 ms │                              25.81 ms │     no change │
   │ QQuery 87 │   62.00 ms │                              78.24 ms │  1.26x slower │
   │ QQuery 88 │   62.64 ms │                              63.43 ms │     no change │
   │ QQuery 89 │   35.13 ms │                              36.27 ms │     no change │
   │ QQuery 90 │   17.03 ms │                              17.11 ms │     no change │
   │ QQuery 91 │   45.48 ms │                              46.55 ms │     no change │
   │ QQuery 92 │   28.80 ms │                              29.12 ms │     no change │
   │ QQuery 93 │   48.78 ms │                              49.66 ms │     no change │
   │ QQuery 94 │   37.74 ms │                              37.36 ms │     no change │
   │ QQuery 95 │   79.77 ms │                              80.01 ms │     no change │
   │ QQuery 96 │   23.61 ms │                              23.64 ms │     no change │
   │ QQuery 97 │   46.98 ms │                              46.93 ms │     no change │
   │ QQuery 98 │   42.90 ms │                              38.17 ms │ +1.12x faster │
   │ QQuery 99 │   69.76 ms │                              69.46 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │  9609.17ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 10014.31ms │
   │ Average Time (HEAD)                                  │    97.06ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   101.15ms │
   │ Queries Faster                                       │         14 │
   │ Queries Slower                                       │         26 │
   │ Queries with No Change                               │         59 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                   HEAD ┃  rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │            5.82 / 6.35 ±0.91 / 8.17 ms │            5.71 / 6.23 ±0.86 / 7.94 ms │     no change │
   │ QQuery 2  │         81.08 / 81.58 ±0.36 / 82.17 ms │         81.97 / 82.55 ±0.34 / 82.97 ms │     no change │
   │ QQuery 3  │         29.28 / 29.60 ±0.25 / 30.00 ms │         28.81 / 28.98 ±0.15 / 29.26 ms │     no change │
   │ QQuery 4  │      477.86 / 485.71 ±6.40 / 497.17 ms │      648.14 / 657.53 ±7.49 / 666.07 ms │  1.35x slower │
   │ QQuery 5  │         52.42 / 53.07 ±0.50 / 53.68 ms │         52.10 / 55.49 ±4.58 / 64.57 ms │     no change │
   │ QQuery 6  │         36.20 / 36.53 ±0.28 / 37.05 ms │         34.86 / 35.37 ±0.27 / 35.59 ms │     no change │
   │ QQuery 7  │         93.76 / 94.93 ±0.80 / 96.13 ms │         95.60 / 96.60 ±1.10 / 98.65 ms │     no change │
   │ QQuery 8  │         36.99 / 38.55 ±1.87 / 42.08 ms │         37.03 / 37.48 ±0.40 / 38.19 ms │     no change │
   │ QQuery 9  │         51.00 / 53.55 ±2.55 / 58.15 ms │         52.62 / 55.59 ±4.66 / 64.87 ms │     no change │
   │ QQuery 10 │         63.31 / 63.85 ±0.32 / 64.24 ms │         63.56 / 63.89 ±0.31 / 64.31 ms │     no change │
   │ QQuery 11 │     318.79 / 351.56 ±19.79 / 371.60 ms │      433.00 / 437.92 ±4.50 / 443.33 ms │  1.25x slower │
   │ QQuery 12 │         29.05 / 30.19 ±0.71 / 31.20 ms │         25.98 / 26.33 ±0.34 / 26.83 ms │ +1.15x faster │
   │ QQuery 13 │      116.81 / 118.31 ±1.08 / 120.09 ms │      118.22 / 119.98 ±1.58 / 122.48 ms │     no change │
   │ QQuery 14 │      414.22 / 419.94 ±4.15 / 425.41 ms │     418.70 / 434.53 ±11.29 / 445.71 ms │     no change │
   │ QQuery 15 │         57.00 / 57.56 ±0.65 / 58.72 ms │         67.17 / 72.44 ±2.95 / 75.46 ms │  1.26x slower │
   │ QQuery 16 │            6.70 / 6.84 ±0.18 / 7.19 ms │            7.42 / 7.65 ±0.18 / 7.97 ms │  1.12x slower │
   │ QQuery 17 │         79.63 / 80.67 ±1.28 / 83.14 ms │         81.83 / 84.17 ±2.54 / 88.78 ms │     no change │
   │ QQuery 18 │      122.77 / 124.03 ±1.17 / 125.99 ms │      125.06 / 125.67 ±0.52 / 126.62 ms │     no change │
   │ QQuery 19 │         41.16 / 41.56 ±0.25 / 41.81 ms │         41.77 / 42.13 ±0.58 / 43.28 ms │     no change │
   │ QQuery 20 │         35.29 / 36.09 ±0.77 / 37.40 ms │         30.95 / 31.44 ±0.38 / 32.00 ms │ +1.15x faster │
   │ QQuery 21 │         17.16 / 17.38 ±0.19 / 17.63 ms │         17.04 / 17.32 ±0.28 / 17.82 ms │     no change │
   │ QQuery 22 │         62.70 / 63.89 ±0.60 / 64.32 ms │         72.31 / 73.17 ±0.55 / 73.82 ms │  1.15x slower │
   │ QQuery 23 │      341.24 / 343.32 ±2.41 / 347.51 ms │     339.19 / 352.16 ±17.80 / 387.32 ms │     no change │
   │ QQuery 24 │      222.15 / 225.85 ±3.60 / 232.54 ms │      222.82 / 226.56 ±2.78 / 229.67 ms │     no change │
   │ QQuery 25 │      109.81 / 110.06 ±0.26 / 110.52 ms │      109.28 / 111.04 ±1.09 / 112.34 ms │     no change │
   │ QQuery 26 │         57.83 / 58.89 ±1.31 / 61.42 ms │         58.42 / 59.63 ±1.41 / 62.34 ms │     no change │
   │ QQuery 27 │            6.32 / 6.42 ±0.12 / 6.66 ms │            6.79 / 6.94 ±0.19 / 7.30 ms │  1.08x slower │
   │ QQuery 28 │         55.76 / 59.34 ±2.03 / 61.97 ms │         60.38 / 60.99 ±0.52 / 61.89 ms │     no change │
   │ QQuery 29 │         96.44 / 97.17 ±0.84 / 98.63 ms │        97.22 / 99.17 ±1.79 / 101.84 ms │     no change │
   │ QQuery 30 │         32.78 / 34.64 ±2.15 / 38.09 ms │         38.92 / 40.23 ±1.81 / 43.81 ms │  1.16x slower │
   │ QQuery 31 │      110.02 / 111.14 ±0.88 / 112.72 ms │      116.00 / 116.43 ±0.30 / 116.91 ms │     no change │
   │ QQuery 32 │         20.34 / 20.72 ±0.23 / 21.03 ms │         20.00 / 20.27 ±0.27 / 20.76 ms │     no change │
   │ QQuery 33 │         37.75 / 38.22 ±0.42 / 38.97 ms │         37.61 / 37.94 ±0.24 / 38.28 ms │     no change │
   │ QQuery 34 │          9.96 / 11.74 ±1.95 / 14.58 ms │           9.62 / 9.87 ±0.20 / 10.19 ms │ +1.19x faster │
   │ QQuery 35 │         72.55 / 73.12 ±0.47 / 73.54 ms │         73.00 / 73.85 ±1.43 / 76.71 ms │     no change │
   │ QQuery 36 │            6.07 / 6.18 ±0.15 / 6.47 ms │            6.31 / 6.46 ±0.25 / 6.95 ms │     no change │
   │ QQuery 37 │            6.88 / 6.98 ±0.10 / 7.10 ms │           7.77 / 8.67 ±1.43 / 11.52 ms │  1.24x slower │
   │ QQuery 38 │         61.81 / 62.82 ±0.60 / 63.59 ms │         77.80 / 78.58 ±0.74 / 79.88 ms │  1.25x slower │
   │ QQuery 39 │         90.34 / 91.84 ±2.47 / 96.76 ms │         89.73 / 90.44 ±0.58 / 91.43 ms │     no change │
   │ QQuery 40 │         23.17 / 23.27 ±0.06 / 23.34 ms │         23.70 / 24.87 ±1.16 / 27.00 ms │  1.07x slower │
   │ QQuery 41 │         11.35 / 11.47 ±0.10 / 11.60 ms │         12.59 / 12.86 ±0.22 / 13.13 ms │  1.12x slower │
   │ QQuery 42 │         23.54 / 23.82 ±0.17 / 24.01 ms │         23.76 / 24.33 ±0.43 / 24.95 ms │     no change │
   │ QQuery 43 │            5.44 / 5.54 ±0.12 / 5.76 ms │            5.85 / 5.97 ±0.14 / 6.23 ms │  1.08x slower │
   │ QQuery 44 │            9.57 / 9.70 ±0.11 / 9.87 ms │           9.92 / 9.99 ±0.04 / 10.03 ms │     no change │
   │ QQuery 45 │         37.72 / 38.14 ±0.63 / 39.40 ms │         37.16 / 37.49 ±0.36 / 38.10 ms │     no change │
   │ QQuery 46 │         11.96 / 12.12 ±0.14 / 12.36 ms │         11.95 / 12.11 ±0.14 / 12.28 ms │     no change │
   │ QQuery 47 │      224.47 / 229.12 ±5.05 / 238.52 ms │      212.51 / 214.56 ±2.46 / 219.38 ms │ +1.07x faster │
   │ QQuery 48 │        95.41 / 98.68 ±5.62 / 109.91 ms │         96.17 / 97.37 ±1.07 / 99.30 ms │     no change │
   │ QQuery 49 │         78.70 / 80.72 ±1.64 / 82.82 ms │         76.06 / 76.73 ±0.59 / 77.75 ms │     no change │
   │ QQuery 50 │         62.03 / 64.26 ±3.37 / 70.97 ms │         59.35 / 60.86 ±2.26 / 65.33 ms │ +1.06x faster │
   │ QQuery 51 │        96.67 / 98.59 ±2.04 / 102.33 ms │         92.03 / 94.80 ±1.82 / 96.50 ms │     no change │
   │ QQuery 52 │         25.37 / 25.88 ±0.29 / 26.16 ms │         23.85 / 24.03 ±0.22 / 24.46 ms │ +1.08x faster │
   │ QQuery 53 │         30.78 / 30.98 ±0.21 / 31.37 ms │         28.83 / 29.22 ±0.31 / 29.64 ms │ +1.06x faster │
   │ QQuery 54 │         57.35 / 60.07 ±3.93 / 67.82 ms │         53.90 / 55.56 ±1.70 / 58.83 ms │ +1.08x faster │
   │ QQuery 55 │         24.68 / 24.92 ±0.22 / 25.28 ms │         23.36 / 23.88 ±0.43 / 24.56 ms │     no change │
   │ QQuery 56 │         41.37 / 42.04 ±0.46 / 42.69 ms │         38.76 / 39.40 ±0.52 / 40.25 ms │ +1.07x faster │
   │ QQuery 57 │      188.44 / 189.80 ±1.59 / 192.87 ms │      163.02 / 165.14 ±3.13 / 171.31 ms │ +1.15x faster │
   │ QQuery 58 │      119.06 / 121.78 ±3.42 / 128.38 ms │      109.04 / 110.61 ±1.04 / 112.30 ms │ +1.10x faster │
   │ QQuery 59 │      122.99 / 123.48 ±0.37 / 124.01 ms │      115.50 / 117.54 ±1.72 / 120.16 ms │     no change │
   │ QQuery 60 │         41.85 / 42.50 ±0.49 / 43.14 ms │         39.32 / 40.31 ±0.55 / 40.83 ms │ +1.05x faster │
   │ QQuery 61 │         13.51 / 15.49 ±3.65 / 22.78 ms │         14.07 / 14.31 ±0.24 / 14.71 ms │ +1.08x faster │
   │ QQuery 62 │         49.04 / 49.42 ±0.25 / 49.68 ms │         48.57 / 48.94 ±0.26 / 49.35 ms │     no change │
   │ QQuery 63 │         31.18 / 31.70 ±0.34 / 32.18 ms │         31.12 / 32.61 ±2.22 / 36.93 ms │     no change │
   │ QQuery 64 │     407.80 / 435.78 ±14.56 / 447.87 ms │      444.77 / 450.82 ±4.62 / 457.69 ms │     no change │
   │ QQuery 65 │      124.63 / 129.45 ±4.29 / 137.36 ms │      133.45 / 136.73 ±3.47 / 143.39 ms │  1.06x slower │
   │ QQuery 66 │         81.53 / 81.99 ±0.33 / 82.42 ms │         87.86 / 90.30 ±3.57 / 97.32 ms │  1.10x slower │
   │ QQuery 67 │      234.88 / 242.28 ±4.24 / 247.89 ms │      288.71 / 291.36 ±3.53 / 297.89 ms │  1.20x slower │
   │ QQuery 68 │         12.26 / 12.53 ±0.32 / 13.16 ms │         13.47 / 13.67 ±0.15 / 13.92 ms │  1.09x slower │
   │ QQuery 69 │         57.06 / 59.26 ±3.12 / 65.44 ms │         61.15 / 64.02 ±4.85 / 73.71 ms │  1.08x slower │
   │ QQuery 70 │      105.64 / 109.07 ±3.22 / 114.87 ms │      109.70 / 112.87 ±3.60 / 119.79 ms │     no change │
   │ QQuery 71 │         34.99 / 35.22 ±0.16 / 35.46 ms │         37.55 / 37.77 ±0.22 / 38.10 ms │  1.07x slower │
   │ QQuery 72 │ 1934.24 / 2128.22 ±129.25 / 2325.00 ms │ 1829.61 / 1965.85 ±157.56 / 2272.48 ms │ +1.08x faster │
   │ QQuery 73 │          9.88 / 10.12 ±0.20 / 10.45 ms │         10.24 / 10.40 ±0.12 / 10.54 ms │     no change │
   │ QQuery 74 │      167.27 / 171.10 ±4.44 / 179.69 ms │     234.36 / 256.99 ±14.56 / 273.28 ms │  1.50x slower │
   │ QQuery 75 │      146.53 / 148.52 ±1.63 / 151.39 ms │      163.35 / 164.44 ±0.99 / 165.81 ms │  1.11x slower │
   │ QQuery 76 │         34.87 / 35.43 ±0.40 / 35.96 ms │         36.09 / 38.53 ±3.55 / 45.55 ms │  1.09x slower │
   │ QQuery 77 │         60.60 / 61.40 ±0.79 / 62.87 ms │         61.97 / 63.04 ±1.26 / 65.46 ms │     no change │
   │ QQuery 78 │      195.66 / 198.04 ±1.55 / 200.52 ms │      197.02 / 199.29 ±2.01 / 202.24 ms │     no change │
   │ QQuery 79 │         66.58 / 66.83 ±0.16 / 67.03 ms │         67.57 / 68.25 ±0.40 / 68.78 ms │     no change │
   │ QQuery 80 │       99.18 / 104.11 ±8.91 / 121.90 ms │      102.62 / 107.29 ±5.78 / 117.55 ms │     no change │
   │ QQuery 81 │         25.89 / 26.29 ±0.25 / 26.64 ms │         30.63 / 30.90 ±0.28 / 31.34 ms │  1.18x slower │
   │ QQuery 82 │         16.21 / 16.66 ±0.51 / 17.59 ms │         17.34 / 17.87 ±0.36 / 18.32 ms │  1.07x slower │
   │ QQuery 83 │         39.43 / 39.81 ±0.26 / 40.04 ms │         40.41 / 40.73 ±0.29 / 41.23 ms │     no change │
   │ QQuery 84 │         29.73 / 32.79 ±5.66 / 44.09 ms │         33.06 / 33.18 ±0.19 / 33.55 ms │     no change │
   │ QQuery 85 │      105.92 / 107.41 ±0.99 / 108.95 ms │      110.50 / 111.79 ±1.19 / 113.82 ms │     no change │
   │ QQuery 86 │         25.04 / 25.36 ±0.27 / 25.86 ms │         25.81 / 26.55 ±0.44 / 27.11 ms │     no change │
   │ QQuery 87 │         62.00 / 64.72 ±4.28 / 73.25 ms │         78.24 / 80.07 ±2.04 / 83.95 ms │  1.24x slower │
   │ QQuery 88 │         62.64 / 63.30 ±0.35 / 63.69 ms │         63.43 / 64.77 ±1.45 / 67.59 ms │     no change │
   │ QQuery 89 │         35.13 / 35.71 ±0.37 / 36.17 ms │         36.27 / 36.80 ±0.41 / 37.52 ms │     no change │
   │ QQuery 90 │         17.03 / 17.31 ±0.22 / 17.60 ms │         17.11 / 17.47 ±0.23 / 17.75 ms │     no change │
   │ QQuery 91 │         45.48 / 46.12 ±0.46 / 46.77 ms │         46.55 / 46.79 ±0.23 / 47.11 ms │     no change │
   │ QQuery 92 │         28.80 / 30.28 ±1.61 / 33.37 ms │         29.12 / 29.74 ±0.44 / 30.17 ms │     no change │
   │ QQuery 93 │         48.78 / 52.22 ±4.67 / 61.35 ms │         49.66 / 51.48 ±1.30 / 53.19 ms │     no change │
   │ QQuery 94 │         37.74 / 38.35 ±0.81 / 39.94 ms │         37.36 / 38.05 ±0.51 / 38.65 ms │     no change │
   │ QQuery 95 │         79.77 / 80.74 ±1.01 / 82.31 ms │         80.01 / 80.54 ±0.50 / 81.42 ms │     no change │
   │ QQuery 96 │         23.61 / 23.94 ±0.31 / 24.49 ms │         23.64 / 24.00 ±0.21 / 24.30 ms │     no change │
   │ QQuery 97 │         46.98 / 49.04 ±2.58 / 54.10 ms │         46.93 / 48.31 ±2.27 / 52.83 ms │     no change │
   │ QQuery 98 │         42.90 / 43.35 ±0.40 / 44.06 ms │         38.17 / 38.97 ±0.62 / 39.99 ms │ +1.11x faster │
   │ QQuery 99 │         69.76 / 70.51 ±0.46 / 71.05 ms │         69.46 / 69.78 ±0.40 / 70.38 ms │     no change │
   └───────────┴────────────────────────────────────────┴────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │  9996.92ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 10324.58ms │
   │ Average Time (HEAD)                                  │   100.98ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   104.29ms │
   │ Queries Faster                                       │         15 │
   │ Queries Slower                                       │         24 │
   │ Queries with No Change                               │         60 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 1.9 GiB |
   | Avg memory | 1.3 GiB |
   | CPU user | 223.8s |
   | CPU sys | 6.3s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 1.8 GiB |
   | Avg memory | 1.3 GiB |
   | CPU user | 220.9s |
   | CPU sys | 6.1s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5373993176

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5373757117)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (4865205f97c7ad3ef19d44e2e2275e14304c0d48) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..4865205f97c7ad3ef19d44e2e2275e14304c0d48)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 0  │    1.21 ms │                               1.21 ms │    no change │
   │ QQuery 1  │   11.59 ms │                              12.19 ms │ 1.05x slower │
   │ QQuery 2  │   36.71 ms │                              37.35 ms │    no change │
   │ QQuery 3  │   30.73 ms │                              31.05 ms │    no change │
   │ QQuery 4  │  218.40 ms │                             224.18 ms │    no change │
   │ QQuery 5  │  271.25 ms │                             275.27 ms │    no change │
   │ QQuery 6  │    1.25 ms │                               1.26 ms │    no change │
   │ QQuery 7  │   12.96 ms │                              13.06 ms │    no change │
   │ QQuery 8  │  322.60 ms │                             332.74 ms │    no change │
   │ QQuery 9  │  448.11 ms │                             464.14 ms │    no change │
   │ QQuery 10 │   69.51 ms │                              69.42 ms │    no change │
   │ QQuery 11 │   80.93 ms │                              80.48 ms │    no change │
   │ QQuery 12 │  267.86 ms │                             264.51 ms │    no change │
   │ QQuery 13 │  362.48 ms │                             363.13 ms │    no change │
   │ QQuery 14 │  281.95 ms │                             280.91 ms │    no change │
   │ QQuery 15 │  270.76 ms │                             267.05 ms │    no change │
   │ QQuery 16 │  602.20 ms │                             610.85 ms │    no change │
   │ QQuery 17 │  614.82 ms │                             622.54 ms │    no change │
   │ QQuery 18 │ 1232.66 ms │                            1270.95 ms │    no change │
   │ QQuery 19 │   27.63 ms │                              27.10 ms │    no change │
   │ QQuery 20 │  518.10 ms │                             516.85 ms │    no change │
   │ QQuery 21 │  522.65 ms │                             519.45 ms │    no change │
   │ QQuery 22 │  981.44 ms │                             990.69 ms │    no change │
   │ QQuery 23 │ 3007.91 ms │                            3028.72 ms │    no change │
   │ QQuery 24 │   40.53 ms │                              41.27 ms │    no change │
   │ QQuery 25 │  109.71 ms │                             109.80 ms │    no change │
   │ QQuery 26 │   40.82 ms │                              41.30 ms │    no change │
   │ QQuery 27 │  516.20 ms │                             511.17 ms │    no change │
   │ QQuery 28 │ 2934.25 ms │                            2902.32 ms │    no change │
   │ QQuery 29 │   40.70 ms │                              40.20 ms │    no change │
   │ QQuery 30 │  305.04 ms │                             296.66 ms │    no change │
   │ QQuery 31 │  286.84 ms │                             281.61 ms │    no change │
   │ QQuery 32 │  942.52 ms │                             897.72 ms │    no change │
   │ QQuery 33 │ 1457.21 ms │                            1421.75 ms │    no change │
   │ QQuery 34 │ 1480.59 ms │                            1447.89 ms │    no change │
   │ QQuery 35 │  278.53 ms │                             277.85 ms │    no change │
   │ QQuery 36 │   65.82 ms │                              67.65 ms │    no change │
   │ QQuery 37 │   35.53 ms │                              35.57 ms │    no change │
   │ QQuery 38 │   40.81 ms │                              40.96 ms │    no change │
   │ QQuery 39 │  133.63 ms │                             138.92 ms │    no change │
   │ QQuery 40 │   14.27 ms │                              14.17 ms │    no change │
   │ QQuery 41 │   14.15 ms │                              13.57 ms │    no change │
   │ QQuery 42 │   13.04 ms │                              13.14 ms │    no change │
   └───────────┴────────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 18945.93ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 18898.61ms │
   │ Average Time (HEAD)                                  │   440.60ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   439.50ms │
   │ Queries Faster                                       │          0 │
   │ Queries Slower                                       │          1 │
   │ Queries with No Change                               │         42 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.21 / 3.95 ±5.39 / 14.73 ms │          1.21 / 3.98 ±5.41 / 14.80 ms │     no change │
   │ QQuery 1  │        11.59 / 11.95 ±0.31 / 12.52 ms │        12.19 / 12.30 ±0.14 / 12.58 ms │     no change │
   │ QQuery 2  │        36.71 / 37.25 ±0.29 / 37.51 ms │        37.35 / 37.40 ±0.04 / 37.44 ms │     no change │
   │ QQuery 3  │        30.73 / 32.51 ±1.06 / 33.89 ms │        31.05 / 31.30 ±0.14 / 31.44 ms │     no change │
   │ QQuery 4  │     218.40 / 225.15 ±3.95 / 228.94 ms │     224.18 / 227.90 ±2.58 / 231.01 ms │     no change │
   │ QQuery 5  │     271.25 / 273.69 ±2.24 / 277.40 ms │     275.27 / 280.59 ±5.05 / 289.58 ms │     no change │
   │ QQuery 6  │           1.25 / 1.40 ±0.22 / 1.82 ms │           1.26 / 1.40 ±0.22 / 1.82 ms │     no change │
   │ QQuery 7  │        12.96 / 13.16 ±0.14 / 13.39 ms │        13.06 / 13.26 ±0.12 / 13.45 ms │     no change │
   │ QQuery 8  │     322.60 / 326.32 ±3.17 / 330.04 ms │     332.74 / 335.95 ±2.37 / 339.47 ms │     no change │
   │ QQuery 9  │     448.11 / 463.77 ±9.83 / 475.01 ms │     464.14 / 472.20 ±6.73 / 479.63 ms │     no change │
   │ QQuery 10 │        69.51 / 70.28 ±0.39 / 70.50 ms │        69.42 / 70.10 ±0.47 / 70.81 ms │     no change │
   │ QQuery 11 │        80.93 / 84.32 ±4.57 / 93.34 ms │        80.48 / 81.80 ±1.27 / 83.51 ms │     no change │
   │ QQuery 12 │     267.86 / 274.23 ±5.00 / 281.96 ms │     264.51 / 269.61 ±4.76 / 276.08 ms │     no change │
   │ QQuery 13 │    362.48 / 373.43 ±11.92 / 396.65 ms │    363.13 / 375.69 ±11.11 / 390.94 ms │     no change │
   │ QQuery 14 │     281.95 / 291.38 ±5.27 / 297.14 ms │     280.91 / 287.75 ±5.09 / 294.67 ms │     no change │
   │ QQuery 15 │     270.76 / 275.15 ±4.40 / 281.49 ms │     267.05 / 273.86 ±4.85 / 282.00 ms │     no change │
   │ QQuery 16 │     602.20 / 613.05 ±7.49 / 622.48 ms │     610.85 / 619.45 ±5.89 / 625.35 ms │     no change │
   │ QQuery 17 │     614.82 / 620.73 ±3.96 / 626.32 ms │    622.54 / 639.41 ±16.49 / 667.97 ms │     no change │
   │ QQuery 18 │ 1232.66 / 1273.42 ±23.77 / 1303.02 ms │ 1270.95 / 1286.87 ±22.01 / 1327.56 ms │     no change │
   │ QQuery 19 │        27.63 / 30.84 ±6.20 / 43.24 ms │        27.10 / 31.20 ±7.61 / 46.42 ms │     no change │
   │ QQuery 20 │     518.10 / 524.48 ±6.98 / 538.09 ms │     516.85 / 526.55 ±6.10 / 534.28 ms │     no change │
   │ QQuery 21 │     522.65 / 525.72 ±2.48 / 529.38 ms │     519.45 / 521.28 ±1.32 / 523.49 ms │     no change │
   │ QQuery 22 │     981.44 / 987.97 ±3.82 / 992.19 ms │   990.69 / 1004.64 ±9.13 / 1014.91 ms │     no change │
   │ QQuery 23 │ 3007.91 / 3056.78 ±39.80 / 3124.40 ms │ 3028.72 / 3078.07 ±31.51 / 3121.78 ms │     no change │
   │ QQuery 24 │        40.53 / 44.56 ±7.04 / 58.63 ms │        41.27 / 45.49 ±5.29 / 55.07 ms │     no change │
   │ QQuery 25 │     109.71 / 111.29 ±1.00 / 112.70 ms │     109.80 / 110.77 ±0.65 / 111.67 ms │     no change │
   │ QQuery 26 │        40.82 / 41.41 ±0.69 / 42.70 ms │        41.30 / 43.21 ±3.41 / 50.03 ms │     no change │
   │ QQuery 27 │     516.20 / 523.87 ±7.41 / 533.95 ms │     511.17 / 522.62 ±9.76 / 540.64 ms │     no change │
   │ QQuery 28 │ 2934.25 / 2950.01 ±15.34 / 2975.91 ms │ 2902.32 / 2940.03 ±32.99 / 2998.13 ms │     no change │
   │ QQuery 29 │        40.70 / 47.22 ±7.55 / 60.21 ms │        40.20 / 42.22 ±3.54 / 49.29 ms │ +1.12x faster │
   │ QQuery 30 │     305.04 / 309.91 ±4.56 / 317.81 ms │    296.66 / 311.49 ±15.05 / 337.82 ms │     no change │
   │ QQuery 31 │     286.84 / 291.40 ±5.82 / 301.97 ms │    281.61 / 297.11 ±14.03 / 320.80 ms │     no change │
   │ QQuery 32 │   942.52 / 965.00 ±22.21 / 1006.36 ms │    897.72 / 926.16 ±20.21 / 955.80 ms │     no change │
   │ QQuery 33 │ 1457.21 / 1477.90 ±21.95 / 1516.69 ms │ 1421.75 / 1471.06 ±36.46 / 1518.88 ms │     no change │
   │ QQuery 34 │ 1480.59 / 1502.14 ±20.15 / 1525.84 ms │ 1447.89 / 1505.39 ±43.17 / 1581.31 ms │     no change │
   │ QQuery 35 │    278.53 / 314.30 ±37.26 / 367.98 ms │    277.85 / 300.48 ±22.97 / 342.28 ms │     no change │
   │ QQuery 36 │        65.82 / 68.30 ±1.66 / 70.85 ms │        67.65 / 75.23 ±9.33 / 89.73 ms │  1.10x slower │
   │ QQuery 37 │        35.53 / 41.82 ±4.88 / 48.57 ms │        35.57 / 37.64 ±2.60 / 42.71 ms │ +1.11x faster │
   │ QQuery 38 │        40.81 / 47.56 ±6.23 / 58.18 ms │        40.96 / 44.47 ±2.77 / 47.99 ms │ +1.07x faster │
   │ QQuery 39 │    133.63 / 152.56 ±10.24 / 162.67 ms │     138.92 / 147.66 ±6.25 / 155.51 ms │     no change │
   │ QQuery 40 │        14.27 / 16.89 ±4.54 / 25.93 ms │        14.17 / 15.00 ±0.58 / 15.89 ms │ +1.13x faster │
   │ QQuery 41 │        14.15 / 16.89 ±3.58 / 23.33 ms │        13.57 / 14.11 ±0.41 / 14.65 ms │ +1.20x faster │
   │ QQuery 42 │        13.04 / 13.31 ±0.20 / 13.55 ms │        13.14 / 14.58 ±2.45 / 19.47 ms │  1.10x slower │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 19327.25ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 19347.26ms │
   │ Average Time (HEAD)                                  │   449.47ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   449.94ms │
   │ Queries Faster                                       │          5 │
   │ Queries Slower                                       │          2 │
   │ Queries with No Change                               │         36 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 12.2 GiB |
   | Avg memory | 4.3 GiB |
   | CPU user | 990.8s |
   | CPU sys | 68.0s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 11.1 GiB |
   | Avg memory | 4.4 GiB |
   | CPU user | 988.9s |
   | CPU sys | 68.9s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5373971684

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5373757117)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (4865205f97c7ad3ef19d44e2e2275e14304c0d48) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..4865205f97c7ad3ef19d44e2e2275e14304c0d48)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    5.81 ms │                               6.26 ms │  1.08x slower │
   │ QQuery 2  │   81.89 ms │                              83.84 ms │     no change │
   │ QQuery 3  │   29.32 ms │                              30.05 ms │     no change │
   │ QQuery 4  │  489.00 ms │                             482.87 ms │     no change │
   │ QQuery 5  │   52.88 ms │                              52.04 ms │     no change │
   │ QQuery 6  │   36.05 ms │                              36.12 ms │     no change │
   │ QQuery 7  │   94.35 ms │                              93.56 ms │     no change │
   │ QQuery 8  │   36.61 ms │                              36.42 ms │     no change │
   │ QQuery 9  │   52.34 ms │                              51.12 ms │     no change │
   │ QQuery 10 │   63.02 ms │                              62.31 ms │     no change │
   │ QQuery 11 │  308.24 ms │                             309.16 ms │     no change │
   │ QQuery 12 │   28.65 ms │                              29.02 ms │     no change │
   │ QQuery 13 │  118.69 ms │                             117.52 ms │     no change │
   │ QQuery 14 │  440.17 ms │                             414.34 ms │ +1.06x faster │
   │ QQuery 15 │   73.67 ms │                              57.62 ms │ +1.28x faster │
   │ QQuery 16 │    7.67 ms │                               6.82 ms │ +1.12x faster │
   │ QQuery 17 │   81.22 ms │                              78.80 ms │     no change │
   │ QQuery 18 │  125.29 ms │                             125.53 ms │     no change │
   │ QQuery 19 │   42.45 ms │                              43.12 ms │     no change │
   │ QQuery 20 │   37.06 ms │                              39.90 ms │  1.08x slower │
   │ QQuery 21 │   17.19 ms │                              18.42 ms │  1.07x slower │
   │ QQuery 22 │   63.06 ms │                              74.19 ms │  1.18x slower │
   │ QQuery 23 │  342.29 ms │                             345.70 ms │     no change │
   │ QQuery 24 │  222.15 ms │                             221.06 ms │     no change │
   │ QQuery 25 │  109.26 ms │                             110.64 ms │     no change │
   │ QQuery 26 │   57.90 ms │                              56.95 ms │     no change │
   │ QQuery 27 │    6.43 ms │                               6.25 ms │     no change │
   │ QQuery 28 │   60.20 ms │                              56.18 ms │ +1.07x faster │
   │ QQuery 29 │   98.22 ms │                              96.71 ms │     no change │
   │ QQuery 30 │   32.78 ms │                              32.01 ms │     no change │
   │ QQuery 31 │  110.43 ms │                             109.60 ms │     no change │
   │ QQuery 32 │   20.49 ms │                              19.91 ms │     no change │
   │ QQuery 33 │   37.75 ms │                              37.55 ms │     no change │
   │ QQuery 34 │   10.02 ms │                               9.91 ms │     no change │
   │ QQuery 35 │   72.81 ms │                              72.06 ms │     no change │
   │ QQuery 36 │    5.77 ms │                               5.73 ms │     no change │
   │ QQuery 37 │    6.75 ms │                               6.78 ms │     no change │
   │ QQuery 38 │   61.31 ms │                              61.75 ms │     no change │
   │ QQuery 39 │   89.61 ms │                              89.91 ms │     no change │
   │ QQuery 40 │   23.07 ms │                              22.65 ms │     no change │
   │ QQuery 41 │   11.26 ms │                              10.98 ms │     no change │
   │ QQuery 42 │   23.44 ms │                              23.32 ms │     no change │
   │ QQuery 43 │    5.28 ms │                               5.19 ms │     no change │
   │ QQuery 44 │    9.34 ms │                               9.28 ms │     no change │
   │ QQuery 45 │   37.70 ms │                              38.92 ms │     no change │
   │ QQuery 46 │   11.99 ms │                              12.23 ms │     no change │
   │ QQuery 47 │  224.14 ms │                             230.09 ms │     no change │
   │ QQuery 48 │   96.92 ms │                              95.32 ms │     no change │
   │ QQuery 49 │   77.88 ms │                              76.24 ms │     no change │
   │ QQuery 50 │   58.98 ms │                              59.44 ms │     no change │
   │ QQuery 51 │   91.26 ms │                              90.96 ms │     no change │
   │ QQuery 52 │   24.10 ms │                              23.81 ms │     no change │
   │ QQuery 53 │   29.68 ms │                              29.16 ms │     no change │
   │ QQuery 54 │   54.35 ms │                              54.32 ms │     no change │
   │ QQuery 55 │   23.52 ms │                              23.03 ms │     no change │
   │ QQuery 56 │   38.54 ms │                              37.93 ms │     no change │
   │ QQuery 57 │  177.13 ms │                             175.29 ms │     no change │
   │ QQuery 58 │  112.74 ms │                             111.91 ms │     no change │
   │ QQuery 59 │  118.67 ms │                             117.24 ms │     no change │
   │ QQuery 60 │   41.64 ms │                              40.15 ms │     no change │
   │ QQuery 61 │   13.75 ms │                              12.98 ms │ +1.06x faster │
   │ QQuery 62 │   49.51 ms │                              47.87 ms │     no change │
   │ QQuery 63 │   30.95 ms │                              29.91 ms │     no change │
   │ QQuery 64 │  428.41 ms │                             404.58 ms │ +1.06x faster │
   │ QQuery 65 │  128.27 ms │                             125.80 ms │     no change │
   │ QQuery 66 │   83.29 ms │                              80.41 ms │     no change │
   │ QQuery 67 │  239.18 ms │                             248.48 ms │     no change │
   │ QQuery 68 │   12.01 ms │                              13.28 ms │  1.11x slower │
   │ QQuery 69 │   57.19 ms │                              60.35 ms │  1.06x slower │
   │ QQuery 70 │  106.96 ms │                             112.58 ms │  1.05x slower │
   │ QQuery 71 │   34.77 ms │                              36.87 ms │  1.06x slower │
   │ QQuery 72 │ 1979.86 ms │                            2044.15 ms │     no change │
   │ QQuery 73 │    9.59 ms │                              10.72 ms │  1.12x slower │
   │ QQuery 74 │  170.14 ms │                             196.31 ms │  1.15x slower │
   │ QQuery 75 │  149.05 ms │                             146.09 ms │     no change │
   │ QQuery 76 │   35.10 ms │                              35.94 ms │     no change │
   │ QQuery 77 │   60.97 ms │                              63.33 ms │     no change │
   │ QQuery 78 │  195.19 ms │                             196.02 ms │     no change │
   │ QQuery 79 │   66.53 ms │                              66.51 ms │     no change │
   │ QQuery 80 │   98.93 ms │                              98.57 ms │     no change │
   │ QQuery 81 │   25.60 ms │                              25.86 ms │     no change │
   │ QQuery 82 │   16.17 ms │                              16.03 ms │     no change │
   │ QQuery 83 │   39.97 ms │                              39.02 ms │     no change │
   │ QQuery 84 │   29.49 ms │                              29.50 ms │     no change │
   │ QQuery 85 │  104.90 ms │                             107.33 ms │     no change │
   │ QQuery 86 │   24.71 ms │                              25.42 ms │     no change │
   │ QQuery 87 │   61.61 ms │                              62.02 ms │     no change │
   │ QQuery 88 │   63.12 ms │                              63.83 ms │     no change │
   │ QQuery 89 │   35.27 ms │                              35.57 ms │     no change │
   │ QQuery 90 │   16.69 ms │                              16.98 ms │     no change │
   │ QQuery 91 │   45.06 ms │                              45.35 ms │     no change │
   │ QQuery 92 │   29.59 ms │                              28.92 ms │     no change │
   │ QQuery 93 │   49.20 ms │                              49.81 ms │     no change │
   │ QQuery 94 │   37.27 ms │                              38.05 ms │     no change │
   │ QQuery 95 │   79.04 ms │                              79.87 ms │     no change │
   │ QQuery 96 │   24.07 ms │                              24.25 ms │     no change │
   │ QQuery 97 │   47.37 ms │                              46.90 ms │     no change │
   │ QQuery 98 │   42.81 ms │                              42.68 ms │     no change │
   │ QQuery 99 │   69.90 ms │                              69.61 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 9711.93ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 9752.93ms │
   │ Average Time (HEAD)                                  │   98.10ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   98.51ms │
   │ Queries Faster                                       │         6 │
   │ Queries Slower                                       │        10 │
   │ Queries with No Change                               │        83 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                   HEAD ┃  rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │            5.81 / 6.35 ±0.99 / 8.31 ms │            6.26 / 6.87 ±0.86 / 8.57 ms │  1.08x slower │
   │ QQuery 2  │         81.89 / 82.19 ±0.28 / 82.62 ms │         83.84 / 84.58 ±0.51 / 85.25 ms │     no change │
   │ QQuery 3  │         29.32 / 29.55 ±0.22 / 29.87 ms │         30.05 / 30.61 ±0.35 / 31.13 ms │     no change │
   │ QQuery 4  │      489.00 / 494.87 ±5.06 / 503.70 ms │     482.87 / 515.64 ±25.27 / 544.71 ms │     no change │
   │ QQuery 5  │         52.88 / 53.49 ±0.41 / 53.99 ms │         52.04 / 52.41 ±0.34 / 53.06 ms │     no change │
   │ QQuery 6  │         36.05 / 36.50 ±0.47 / 37.32 ms │         36.12 / 36.29 ±0.18 / 36.63 ms │     no change │
   │ QQuery 7  │         94.35 / 94.65 ±0.27 / 95.12 ms │        93.56 / 96.29 ±4.29 / 104.83 ms │     no change │
   │ QQuery 8  │         36.61 / 37.15 ±0.42 / 37.67 ms │         36.42 / 36.87 ±0.52 / 37.79 ms │     no change │
   │ QQuery 9  │         52.34 / 54.96 ±1.64 / 57.07 ms │         51.12 / 52.86 ±1.76 / 56.22 ms │     no change │
   │ QQuery 10 │         63.02 / 63.43 ±0.33 / 63.86 ms │         62.31 / 62.75 ±0.33 / 63.25 ms │     no change │
   │ QQuery 11 │      308.24 / 314.21 ±4.38 / 318.72 ms │      309.16 / 312.77 ±1.95 / 314.69 ms │     no change │
   │ QQuery 12 │         28.65 / 29.24 ±0.39 / 29.67 ms │         29.02 / 29.16 ±0.14 / 29.42 ms │     no change │
   │ QQuery 13 │      118.69 / 119.44 ±0.90 / 121.13 ms │      117.52 / 118.52 ±0.69 / 119.61 ms │     no change │
   │ QQuery 14 │      440.17 / 443.87 ±2.26 / 445.70 ms │      414.34 / 419.45 ±5.70 / 430.35 ms │ +1.06x faster │
   │ QQuery 15 │         73.67 / 74.31 ±0.64 / 75.19 ms │         57.62 / 58.54 ±0.50 / 59.16 ms │ +1.27x faster │
   │ QQuery 16 │            7.67 / 7.77 ±0.11 / 7.97 ms │            6.82 / 6.99 ±0.19 / 7.34 ms │ +1.11x faster │
   │ QQuery 17 │         81.22 / 84.73 ±3.84 / 91.50 ms │         78.80 / 80.77 ±2.68 / 86.08 ms │     no change │
   │ QQuery 18 │      125.29 / 128.30 ±1.57 / 129.74 ms │      125.53 / 128.94 ±2.25 / 131.82 ms │     no change │
   │ QQuery 19 │         42.45 / 43.70 ±1.86 / 47.41 ms │         43.12 / 43.44 ±0.26 / 43.74 ms │     no change │
   │ QQuery 20 │         37.06 / 37.98 ±1.16 / 40.25 ms │         39.90 / 42.32 ±2.00 / 46.00 ms │  1.11x slower │
   │ QQuery 21 │         17.19 / 17.56 ±0.24 / 17.91 ms │         18.42 / 18.81 ±0.34 / 19.31 ms │  1.07x slower │
   │ QQuery 22 │         63.06 / 64.03 ±0.89 / 65.71 ms │         74.19 / 75.00 ±0.67 / 76.08 ms │  1.17x slower │
   │ QQuery 23 │      342.29 / 349.04 ±5.73 / 358.43 ms │     345.70 / 393.15 ±23.79 / 407.93 ms │  1.13x slower │
   │ QQuery 24 │      222.15 / 223.90 ±1.33 / 225.33 ms │      221.06 / 227.29 ±4.66 / 231.94 ms │     no change │
   │ QQuery 25 │      109.26 / 110.94 ±1.39 / 112.86 ms │      110.64 / 113.32 ±1.77 / 115.64 ms │     no change │
   │ QQuery 26 │         57.90 / 58.38 ±0.31 / 58.78 ms │         56.95 / 57.31 ±0.30 / 57.84 ms │     no change │
   │ QQuery 27 │            6.43 / 6.58 ±0.20 / 6.96 ms │            6.25 / 6.35 ±0.15 / 6.65 ms │     no change │
   │ QQuery 28 │         60.20 / 61.08 ±0.70 / 62.31 ms │         56.18 / 59.63 ±1.81 / 61.15 ms │     no change │
   │ QQuery 29 │       98.22 / 100.55 ±3.91 / 108.31 ms │       96.71 / 101.19 ±4.50 / 109.07 ms │     no change │
   │ QQuery 30 │         32.78 / 33.03 ±0.22 / 33.36 ms │         32.01 / 32.56 ±0.32 / 32.87 ms │     no change │
   │ QQuery 31 │      110.43 / 111.14 ±0.77 / 112.58 ms │      109.60 / 110.90 ±2.14 / 115.18 ms │     no change │
   │ QQuery 32 │         20.49 / 21.96 ±2.27 / 26.47 ms │         19.91 / 20.32 ±0.34 / 20.73 ms │ +1.08x faster │
   │ QQuery 33 │         37.75 / 38.78 ±0.53 / 39.17 ms │         37.55 / 37.79 ±0.21 / 38.11 ms │     no change │
   │ QQuery 34 │         10.02 / 10.27 ±0.20 / 10.61 ms │          9.91 / 10.21 ±0.27 / 10.70 ms │     no change │
   │ QQuery 35 │         72.81 / 74.10 ±1.03 / 75.70 ms │         72.06 / 72.61 ±0.46 / 73.33 ms │     no change │
   │ QQuery 36 │            5.77 / 6.03 ±0.21 / 6.38 ms │            5.73 / 5.88 ±0.17 / 6.20 ms │     no change │
   │ QQuery 37 │            6.75 / 6.83 ±0.07 / 6.91 ms │            6.78 / 6.88 ±0.10 / 7.04 ms │     no change │
   │ QQuery 38 │         61.31 / 61.94 ±0.81 / 63.54 ms │         61.75 / 63.22 ±0.97 / 64.05 ms │     no change │
   │ QQuery 39 │         89.61 / 90.11 ±0.51 / 90.94 ms │         89.91 / 91.22 ±1.23 / 93.55 ms │     no change │
   │ QQuery 40 │         23.07 / 23.42 ±0.39 / 24.07 ms │         22.65 / 23.07 ±0.26 / 23.48 ms │     no change │
   │ QQuery 41 │         11.26 / 11.47 ±0.16 / 11.74 ms │         10.98 / 11.15 ±0.19 / 11.51 ms │     no change │
   │ QQuery 42 │         23.44 / 23.59 ±0.11 / 23.75 ms │         23.32 / 23.78 ±0.31 / 24.12 ms │     no change │
   │ QQuery 43 │            5.28 / 5.38 ±0.15 / 5.67 ms │            5.19 / 5.33 ±0.17 / 5.67 ms │     no change │
   │ QQuery 44 │            9.34 / 9.57 ±0.14 / 9.75 ms │          9.28 / 10.87 ±2.99 / 16.85 ms │  1.14x slower │
   │ QQuery 45 │         37.70 / 39.82 ±2.30 / 44.26 ms │         38.92 / 40.10 ±1.09 / 41.61 ms │     no change │
   │ QQuery 46 │         11.99 / 12.82 ±0.72 / 13.85 ms │         12.23 / 12.82 ±0.41 / 13.32 ms │     no change │
   │ QQuery 47 │      224.14 / 228.63 ±2.66 / 232.23 ms │      230.09 / 234.12 ±3.43 / 239.71 ms │     no change │
   │ QQuery 48 │         96.92 / 97.67 ±0.70 / 98.63 ms │         95.32 / 95.86 ±0.36 / 96.24 ms │     no change │
   │ QQuery 49 │         77.88 / 81.33 ±4.15 / 89.45 ms │         76.24 / 79.83 ±4.37 / 88.13 ms │     no change │
   │ QQuery 50 │         58.98 / 60.45 ±1.14 / 62.27 ms │         59.44 / 59.92 ±0.39 / 60.50 ms │     no change │
   │ QQuery 51 │         91.26 / 92.19 ±1.05 / 94.17 ms │        90.96 / 95.54 ±6.36 / 108.16 ms │     no change │
   │ QQuery 52 │         24.10 / 26.85 ±3.94 / 34.68 ms │         23.81 / 23.99 ±0.15 / 24.17 ms │ +1.12x faster │
   │ QQuery 53 │         29.68 / 30.37 ±0.72 / 31.73 ms │         29.16 / 29.27 ±0.07 / 29.40 ms │     no change │
   │ QQuery 54 │         54.35 / 54.84 ±0.42 / 55.55 ms │         54.32 / 54.90 ±0.55 / 55.63 ms │     no change │
   │ QQuery 55 │         23.52 / 23.68 ±0.22 / 24.11 ms │         23.03 / 23.38 ±0.20 / 23.63 ms │     no change │
   │ QQuery 56 │         38.54 / 39.51 ±0.79 / 40.91 ms │         37.93 / 38.33 ±0.24 / 38.60 ms │     no change │
   │ QQuery 57 │      177.13 / 179.72 ±3.88 / 187.39 ms │      175.29 / 179.13 ±4.43 / 187.78 ms │     no change │
   │ QQuery 58 │      112.74 / 114.44 ±1.30 / 116.22 ms │      111.91 / 113.92 ±2.22 / 117.93 ms │     no change │
   │ QQuery 59 │      118.67 / 120.69 ±2.13 / 124.50 ms │      117.24 / 117.78 ±0.45 / 118.37 ms │     no change │
   │ QQuery 60 │         41.64 / 42.31 ±0.60 / 43.23 ms │         40.15 / 41.87 ±1.27 / 43.88 ms │     no change │
   │ QQuery 61 │         13.75 / 14.36 ±0.82 / 15.96 ms │         12.98 / 13.04 ±0.10 / 13.25 ms │ +1.10x faster │
   │ QQuery 62 │         49.51 / 49.82 ±0.17 / 49.98 ms │         47.87 / 48.12 ±0.24 / 48.57 ms │     no change │
   │ QQuery 63 │         30.95 / 31.37 ±0.36 / 31.88 ms │         29.91 / 30.18 ±0.15 / 30.32 ms │     no change │
   │ QQuery 64 │      428.41 / 445.19 ±9.77 / 458.59 ms │      404.58 / 412.98 ±9.09 / 430.36 ms │ +1.08x faster │
   │ QQuery 65 │      128.27 / 132.16 ±3.57 / 138.77 ms │      125.80 / 126.43 ±0.74 / 127.84 ms │     no change │
   │ QQuery 66 │         83.29 / 84.26 ±0.65 / 85.19 ms │         80.41 / 80.85 ±0.61 / 82.05 ms │     no change │
   │ QQuery 67 │      239.18 / 245.34 ±5.60 / 254.87 ms │     248.48 / 264.37 ±16.49 / 290.41 ms │  1.08x slower │
   │ QQuery 68 │         12.01 / 12.63 ±0.39 / 13.06 ms │         13.28 / 13.80 ±0.70 / 15.18 ms │  1.09x slower │
   │ QQuery 69 │         57.19 / 57.39 ±0.13 / 57.57 ms │         60.35 / 60.91 ±0.42 / 61.61 ms │  1.06x slower │
   │ QQuery 70 │      106.96 / 108.61 ±1.39 / 110.89 ms │      112.58 / 114.37 ±1.95 / 117.93 ms │  1.05x slower │
   │ QQuery 71 │         34.77 / 35.83 ±1.03 / 37.70 ms │         36.87 / 39.19 ±2.20 / 43.04 ms │  1.09x slower │
   │ QQuery 72 │ 1979.86 / 2123.61 ±147.32 / 2378.82 ms │ 2044.15 / 2196.34 ±111.83 / 2310.68 ms │     no change │
   │ QQuery 73 │           9.59 / 9.82 ±0.18 / 10.14 ms │         10.72 / 10.96 ±0.26 / 11.43 ms │  1.12x slower │
   │ QQuery 74 │      170.14 / 175.21 ±3.25 / 179.25 ms │      196.31 / 212.07 ±8.13 / 219.52 ms │  1.21x slower │
   │ QQuery 75 │      149.05 / 152.90 ±3.94 / 159.53 ms │      146.09 / 147.49 ±0.88 / 148.85 ms │     no change │
   │ QQuery 76 │         35.10 / 35.70 ±0.58 / 36.74 ms │         35.94 / 39.25 ±4.58 / 48.31 ms │  1.10x slower │
   │ QQuery 77 │         60.97 / 61.36 ±0.27 / 61.70 ms │         63.33 / 63.70 ±0.37 / 64.39 ms │     no change │
   │ QQuery 78 │      195.19 / 197.90 ±3.13 / 203.78 ms │      196.02 / 205.45 ±8.25 / 218.02 ms │     no change │
   │ QQuery 79 │         66.53 / 67.93 ±1.08 / 69.65 ms │         66.51 / 66.91 ±0.31 / 67.42 ms │     no change │
   │ QQuery 80 │       98.93 / 100.69 ±1.69 / 103.84 ms │       98.57 / 104.66 ±5.62 / 111.80 ms │     no change │
   │ QQuery 81 │         25.60 / 25.76 ±0.30 / 26.35 ms │         25.86 / 26.10 ±0.28 / 26.65 ms │     no change │
   │ QQuery 82 │         16.17 / 16.69 ±0.85 / 18.38 ms │         16.03 / 16.21 ±0.19 / 16.47 ms │     no change │
   │ QQuery 83 │         39.97 / 42.81 ±2.78 / 46.80 ms │         39.02 / 39.37 ±0.22 / 39.72 ms │ +1.09x faster │
   │ QQuery 84 │         29.49 / 29.99 ±0.32 / 30.47 ms │         29.50 / 29.61 ±0.10 / 29.74 ms │     no change │
   │ QQuery 85 │      104.90 / 106.12 ±1.23 / 108.48 ms │      107.33 / 109.06 ±1.91 / 111.81 ms │     no change │
   │ QQuery 86 │         24.71 / 24.95 ±0.26 / 25.32 ms │         25.42 / 25.52 ±0.09 / 25.70 ms │     no change │
   │ QQuery 87 │         61.61 / 62.83 ±0.96 / 64.10 ms │         62.02 / 62.99 ±0.77 / 64.11 ms │     no change │
   │ QQuery 88 │         63.12 / 63.44 ±0.21 / 63.67 ms │         63.83 / 64.93 ±1.11 / 66.67 ms │     no change │
   │ QQuery 89 │         35.27 / 35.70 ±0.44 / 36.54 ms │         35.57 / 36.32 ±0.55 / 37.27 ms │     no change │
   │ QQuery 90 │         16.69 / 16.87 ±0.21 / 17.28 ms │         16.98 / 17.10 ±0.11 / 17.30 ms │     no change │
   │ QQuery 91 │         45.06 / 45.56 ±0.48 / 46.39 ms │         45.35 / 45.60 ±0.44 / 46.48 ms │     no change │
   │ QQuery 92 │         29.59 / 30.54 ±1.21 / 32.83 ms │         28.92 / 29.42 ±0.64 / 30.68 ms │     no change │
   │ QQuery 93 │         49.20 / 50.31 ±0.93 / 51.94 ms │         49.81 / 50.55 ±0.59 / 51.37 ms │     no change │
   │ QQuery 94 │         37.27 / 37.89 ±0.53 / 38.60 ms │         38.05 / 38.60 ±0.46 / 39.32 ms │     no change │
   │ QQuery 95 │         79.04 / 80.14 ±0.91 / 81.45 ms │         79.87 / 80.34 ±0.30 / 80.81 ms │     no change │
   │ QQuery 96 │         24.07 / 25.10 ±1.79 / 28.67 ms │         24.25 / 24.35 ±0.09 / 24.50 ms │     no change │
   │ QQuery 97 │         47.37 / 47.92 ±0.47 / 48.64 ms │         46.90 / 48.64 ±2.36 / 53.19 ms │     no change │
   │ QQuery 98 │         42.81 / 43.29 ±0.41 / 43.88 ms │         42.68 / 43.25 ±0.38 / 43.84 ms │     no change │
   │ QQuery 99 │         69.90 / 70.14 ±0.16 / 70.34 ms │         69.61 / 70.14 ±0.40 / 70.66 ms │     no change │
   └───────────┴────────────────────────────────────────┴────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │  9999.82ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 10143.86ms │
   │ Average Time (HEAD)                                  │   101.01ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   102.46ms │
   │ Queries Faster                                       │          8 │
   │ Queries Slower                                       │         14 │
   │ Queries with No Change                               │         77 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 2.1 GiB |
   | Avg memory | 1.5 GiB |
   | CPU user | 225.4s |
   | CPU sys | 6.4s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 2.2 GiB |
   | Avg memory | 1.4 GiB |
   | CPU user | 227.9s |
   | CPU sys | 6.2s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5374316367

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5374297430)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5374297430-1850-t29qh 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5690822111

   @kumarUjjawal any status update on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675446751

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f (cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f) to cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f [diff](https://github.com/apache/datafusion/compare/cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f..cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   env:
     DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
   baseline:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
   changed:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 37.99 ms │                              38.06 ms │    no change │
   │ QQuery 2  │ 18.53 ms │                              19.17 ms │    no change │
   │ QQuery 3  │ 28.03 ms │                              28.40 ms │    no change │
   │ QQuery 4  │ 17.14 ms │                              17.32 ms │    no change │
   │ QQuery 5  │ 35.12 ms │                              35.14 ms │    no change │
   │ QQuery 6  │ 16.05 ms │                              15.97 ms │    no change │
   │ QQuery 7  │ 40.40 ms │                              40.74 ms │    no change │
   │ QQuery 8  │ 40.58 ms │                              40.68 ms │    no change │
   │ QQuery 9  │ 47.98 ms │                              50.30 ms │    no change │
   │ QQuery 10 │ 41.88 ms │                              60.36 ms │ 1.44x slower │
   │ QQuery 11 │ 13.10 ms │                              13.63 ms │    no change │
   │ QQuery 12 │ 23.80 ms │                              23.74 ms │    no change │
   │ QQuery 13 │ 38.69 ms │                              51.41 ms │ 1.33x slower │
   │ QQuery 14 │ 24.19 ms │                              24.14 ms │    no change │
   │ QQuery 15 │ 30.67 ms │                              30.81 ms │    no change │
   │ QQuery 16 │ 13.58 ms │                              14.23 ms │    no change │
   │ QQuery 17 │ 70.69 ms │                              71.49 ms │    no change │
   │ QQuery 18 │ 61.01 ms │                              99.29 ms │ 1.63x slower │
   │ QQuery 19 │ 32.64 ms │                              31.65 ms │    no change │
   │ QQuery 20 │ 31.61 ms │                              35.90 ms │ 1.14x slower │
   │ QQuery 21 │ 55.16 ms │                              52.74 ms │    no change │
   │ QQuery 22 │ 13.57 ms │                              15.63 ms │ 1.15x slower │
   └───────────┴──────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 732.41ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 810.78ms │
   │ Average Time (HEAD)                                  │  33.29ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  36.85ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        5 │
   │ Queries with No Change                               │       17 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 37.99 / 39.01 ±1.26 / 41.04 ms │        38.06 / 38.85 ±1.03 / 40.81 ms │     no change │
   │ QQuery 2  │ 18.53 / 19.13 ±0.68 / 20.45 ms │        19.17 / 19.73 ±0.65 / 20.98 ms │     no change │
   │ QQuery 3  │ 28.03 / 29.52 ±1.27 / 31.46 ms │        28.40 / 28.65 ±0.17 / 28.90 ms │     no change │
   │ QQuery 4  │ 17.14 / 17.50 ±0.29 / 17.92 ms │        17.32 / 17.51 ±0.11 / 17.62 ms │     no change │
   │ QQuery 5  │ 35.12 / 36.41 ±1.24 / 38.72 ms │        35.14 / 35.48 ±0.19 / 35.71 ms │     no change │
   │ QQuery 6  │ 16.05 / 18.18 ±3.82 / 25.81 ms │        15.97 / 16.05 ±0.06 / 16.16 ms │ +1.13x faster │
   │ QQuery 7  │ 40.40 / 41.95 ±0.87 / 43.10 ms │        40.74 / 42.62 ±1.36 / 44.40 ms │     no change │
   │ QQuery 8  │ 40.58 / 41.57 ±1.46 / 44.47 ms │        40.68 / 42.22 ±2.54 / 47.28 ms │     no change │
   │ QQuery 9  │ 47.98 / 49.97 ±1.40 / 52.04 ms │        50.30 / 50.94 ±0.49 / 51.75 ms │     no change │
   │ QQuery 10 │ 41.88 / 42.59 ±1.04 / 44.65 ms │        60.36 / 61.66 ±0.98 / 62.85 ms │  1.45x slower │
   │ QQuery 11 │ 13.10 / 13.30 ±0.22 / 13.71 ms │        13.63 / 13.72 ±0.06 / 13.80 ms │     no change │
   │ QQuery 12 │ 23.80 / 24.82 ±0.91 / 26.42 ms │        23.74 / 23.99 ±0.17 / 24.21 ms │     no change │
   │ QQuery 13 │ 38.69 / 38.88 ±0.14 / 39.08 ms │        51.41 / 52.41 ±1.59 / 55.57 ms │  1.35x slower │
   │ QQuery 14 │ 24.19 / 24.45 ±0.22 / 24.81 ms │        24.14 / 24.35 ±0.15 / 24.52 ms │     no change │
   │ QQuery 15 │ 30.67 / 30.79 ±0.07 / 30.86 ms │        30.81 / 31.89 ±1.64 / 35.15 ms │     no change │
   │ QQuery 16 │ 13.58 / 13.86 ±0.26 / 14.31 ms │        14.23 / 14.61 ±0.22 / 14.85 ms │  1.05x slower │
   │ QQuery 17 │ 70.69 / 71.53 ±0.48 / 72.15 ms │        71.49 / 73.43 ±1.53 / 76.09 ms │     no change │
   │ QQuery 18 │ 61.01 / 62.06 ±0.72 / 63.23 ms │      99.29 / 103.32 ±4.09 / 110.66 ms │  1.66x slower │
   │ QQuery 19 │ 32.64 / 32.96 ±0.25 / 33.34 ms │        31.65 / 31.98 ±0.36 / 32.66 ms │     no change │
   │ QQuery 20 │ 31.61 / 31.84 ±0.23 / 32.18 ms │        35.90 / 36.32 ±0.27 / 36.76 ms │  1.14x slower │
   │ QQuery 21 │ 55.16 / 55.56 ±0.40 / 56.27 ms │        52.74 / 54.75 ±2.39 / 59.20 ms │     no change │
   │ QQuery 22 │ 13.57 / 13.93 ±0.24 / 14.23 ms │        15.63 / 15.80 ±0.14 / 15.97 ms │  1.13x slower │
   └───────────┴────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 749.79ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 830.28ms │
   │ Average Time (HEAD)                                  │  34.08ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  37.74ms │
   │ Queries Faster                                       │        1 │
   │ Queries Slower                                       │        6 │
   │ Queries with No Change                               │       15 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Memory Pool Peaks</summary>
   
   Peak `MemoryPool` reservation per query — what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
   
   Base: `cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f` | Changed: `cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f`
   
   **`tpch` — `tpch_sf1`**
   
   | Query | Base | Changed | Change |
   | --- | --- | --- | --- |
   | Query 1 | 30.0 MiB | 30.0 MiB | +0.1% |
   | Query 2 | 25.2 MiB | 16.0 MiB | -36.3% |
   | Query 3 | 11.4 MiB | 11.2 MiB | -2.2% |
   | Query 4 | 33.5 MiB | 22.2 MiB | -33.8% |
   | Query 5 | 42.4 MiB | 32.2 MiB | -24.2% |
   | Query 6 | 832 B | 832 B | +0.0% |
   | Query 7 | 97.2 MiB | 100.7 MiB | +3.5% |
   | Query 8 | 30.1 MiB | 30.1 MiB | -0.0% |
   | Query 9 | 134.9 MiB | 133.8 MiB | -0.8% |
   | Query 10 | 37.9 MiB | 68.2 MiB | +79.8% |
   | Query 11 | 41.5 MiB | 31.5 MiB | -24.1% |
   | Query 12 | 20.0 MiB | 10.1 MiB | -49.4% |
   | Query 13 | 62.4 MiB | 52.5 MiB | -16.0% |
   | Query 14 | 4.8 MiB | 4.8 MiB | +0.0% |
   | Query 15 | 12.3 MiB | 13.1 MiB | +6.4% |
   | Query 16 | 44.5 MiB | 32.7 MiB | -26.6% |
   | Query 17 | 141.3 MiB | 141.3 MiB | +0.0% |
   | Query 18 | 308.1 MiB | 152.9 MiB | -50.4% |
   | Query 19 | 721.9 KiB | 215.8 KiB | -70.1% |
   | Query 20 | 58.8 MiB | 64.9 MiB | +10.3% |
   | Query 21 | 39.5 MiB | 32.9 MiB | -16.7% |
   | Query 22 | 32.3 MiB | 35.0 MiB | +8.6% |
   
   **Pool accounting vs. process RSS**
   
   Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
   
   | Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
   | --- | --- | --- | --- | --- | --- |
   | tpch | base (`cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f`) | 308.1 MiB | 1.2 GiB | 944.0 MiB | 4.1× |
   | tpch | changed (`cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f`) | 152.9 MiB | 1.1 GiB | 985.6 MiB | 7.4× |
   
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.2 GiB |
   | Avg memory | 506.4 MiB |
   | CPU user | 21.0s |
   | CPU sys | 1.7s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.1 GiB |
   | Avg memory | 658.8 MiB |
   | CPU user | 21.3s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5719142962

   Do we have any query where this option makes the query faster? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal merged PR #24227:
URL: https://github.com/apache/datafusion/pull/24227


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5995789307

   > > I can rebase this before we merge
   > 
   > Please resolve the conflicts.
   
   @kumarUjjawal fixed. should be good to go now 🚀 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513684577

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513429687)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (896eebe1dee3bd2845c6f6ffec1ecda7be808125) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..896eebe1dee3bd2845c6f6ffec1ecda7be808125)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.75 ms │                              37.91 ms │    no change │
   │ QQuery 2  │ 18.60 ms │                              18.98 ms │    no change │
   │ QQuery 3  │ 28.81 ms │                              28.17 ms │    no change │
   │ QQuery 4  │ 17.55 ms │                              17.57 ms │    no change │
   │ QQuery 5  │ 35.73 ms │                              35.41 ms │    no change │
   │ QQuery 6  │ 16.03 ms │                              15.83 ms │    no change │
   │ QQuery 7  │ 41.49 ms │                              41.31 ms │    no change │
   │ QQuery 8  │ 40.74 ms │                              40.89 ms │    no change │
   │ QQuery 9  │ 49.57 ms │                              50.52 ms │    no change │
   │ QQuery 10 │ 42.14 ms │                              59.99 ms │ 1.42x slower │
   │ QQuery 11 │ 13.47 ms │                              13.51 ms │    no change │
   │ QQuery 12 │ 23.83 ms │                              23.62 ms │    no change │
   │ QQuery 13 │ 39.26 ms │                              50.95 ms │ 1.30x slower │
   │ QQuery 14 │ 24.21 ms │                              24.02 ms │    no change │
   │ QQuery 15 │ 30.93 ms │                              30.98 ms │    no change │
   │ QQuery 16 │ 13.86 ms │                              14.03 ms │    no change │
   │ QQuery 17 │ 70.46 ms │                              71.18 ms │    no change │
   │ QQuery 18 │ 60.87 ms │                              97.92 ms │ 1.61x slower │
   │ QQuery 19 │ 32.85 ms │                              31.91 ms │    no change │
   │ QQuery 20 │ 31.42 ms │                              35.55 ms │ 1.13x slower │
   │ QQuery 21 │ 55.90 ms │                              54.74 ms │    no change │
   │ QQuery 22 │ 13.97 ms │                              16.79 ms │ 1.20x slower │
   └───────────┴──────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 740.46ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 811.77ms │
   │ Average Time (HEAD)                                  │  33.66ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  36.90ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        5 │
   │ Queries with No Change                               │       17 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.75 / 40.65 ±1.51 / 42.26 ms │        37.91 / 38.69 ±1.01 / 40.63 ms │    no change │
   │ QQuery 2  │ 18.60 / 18.84 ±0.25 / 19.30 ms │        18.98 / 19.24 ±0.21 / 19.54 ms │    no change │
   │ QQuery 3  │ 28.81 / 29.65 ±1.07 / 31.69 ms │        28.17 / 28.88 ±0.74 / 30.27 ms │    no change │
   │ QQuery 4  │ 17.55 / 17.85 ±0.21 / 18.20 ms │        17.57 / 18.13 ±0.44 / 18.90 ms │    no change │
   │ QQuery 5  │ 35.73 / 35.95 ±0.14 / 36.09 ms │        35.41 / 36.20 ±0.93 / 38.02 ms │    no change │
   │ QQuery 6  │ 16.03 / 16.59 ±0.80 / 18.17 ms │        15.83 / 15.96 ±0.11 / 16.17 ms │    no change │
   │ QQuery 7  │ 41.49 / 42.68 ±1.00 / 44.26 ms │        41.31 / 43.10 ±1.87 / 46.70 ms │    no change │
   │ QQuery 8  │ 40.74 / 41.78 ±1.19 / 44.00 ms │        40.89 / 41.50 ±0.69 / 42.84 ms │    no change │
   │ QQuery 9  │ 49.57 / 50.23 ±0.68 / 51.05 ms │        50.52 / 51.23 ±0.54 / 52.18 ms │    no change │
   │ QQuery 10 │ 42.14 / 43.47 ±1.20 / 44.98 ms │        59.99 / 60.75 ±0.45 / 61.34 ms │ 1.40x slower │
   │ QQuery 11 │ 13.47 / 13.70 ±0.15 / 13.93 ms │        13.51 / 13.81 ±0.24 / 14.16 ms │    no change │
   │ QQuery 12 │ 23.83 / 24.33 ±0.43 / 25.01 ms │        23.62 / 24.02 ±0.26 / 24.40 ms │    no change │
   │ QQuery 13 │ 39.26 / 40.49 ±1.59 / 43.57 ms │        50.95 / 52.09 ±1.25 / 54.45 ms │ 1.29x slower │
   │ QQuery 14 │ 24.21 / 25.14 ±1.22 / 27.53 ms │        24.02 / 24.26 ±0.13 / 24.37 ms │    no change │
   │ QQuery 15 │ 30.93 / 31.58 ±1.06 / 33.68 ms │        30.98 / 31.15 ±0.18 / 31.49 ms │    no change │
   │ QQuery 16 │ 13.86 / 14.10 ±0.18 / 14.40 ms │        14.03 / 14.29 ±0.17 / 14.54 ms │    no change │
   │ QQuery 17 │ 70.46 / 71.99 ±1.20 / 74.12 ms │        71.18 / 72.19 ±0.95 / 73.99 ms │    no change │
   │ QQuery 18 │ 60.87 / 62.05 ±1.36 / 64.03 ms │      97.92 / 100.17 ±1.46 / 101.83 ms │ 1.61x slower │
   │ QQuery 19 │ 32.85 / 33.41 ±0.76 / 34.90 ms │        31.91 / 32.17 ±0.25 / 32.56 ms │    no change │
   │ QQuery 20 │ 31.42 / 31.92 ±0.61 / 33.03 ms │        35.55 / 36.22 ±0.79 / 37.75 ms │ 1.13x slower │
   │ QQuery 21 │ 55.90 / 56.10 ±0.14 / 56.33 ms │        54.74 / 59.10 ±4.56 / 67.87 ms │ 1.05x slower │
   │ QQuery 22 │ 13.97 / 14.16 ±0.13 / 14.36 ms │        16.79 / 17.18 ±0.55 / 18.25 ms │ 1.21x slower │
   └───────────┴────────────────────────────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 756.65ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 830.33ms │
   │ Average Time (HEAD)                                  │  34.39ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  37.74ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        6 │
   │ Queries with No Change                               │       16 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.2 GiB |
   | Avg memory | 489.9 MiB |
   | CPU user | 21.1s |
   | CPU sys | 1.7s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.1 GiB |
   | Avg memory | 649.9 MiB |
   | CPU user | 21.4s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5514091549

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513845153)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (6166c31c21bdd7965f348ae7a126a2d1e85a6216) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..6166c31c21bdd7965f348ae7a126a2d1e85a6216)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    5.49 ms │                               5.53 ms │     no change │
   │ QQuery 2  │   80.66 ms │                              80.12 ms │     no change │
   │ QQuery 3  │   28.39 ms │                              28.55 ms │     no change │
   │ QQuery 4  │  471.91 ms │                             483.52 ms │     no change │
   │ QQuery 5  │   51.62 ms │                              51.41 ms │     no change │
   │ QQuery 6  │   35.98 ms │                              35.82 ms │     no change │
   │ QQuery 7  │   74.62 ms │                              74.56 ms │     no change │
   │ QQuery 8  │   35.79 ms │                              36.14 ms │     no change │
   │ QQuery 9  │   52.19 ms │                              49.99 ms │     no change │
   │ QQuery 10 │   62.00 ms │                              62.53 ms │     no change │
   │ QQuery 11 │  297.69 ms │                             297.54 ms │     no change │
   │ QQuery 12 │   28.31 ms │                              28.22 ms │     no change │
   │ QQuery 13 │  117.09 ms │                             118.14 ms │     no change │
   │ QQuery 14 │  412.64 ms │                             415.84 ms │     no change │
   │ QQuery 15 │   56.88 ms │                              57.57 ms │     no change │
   │ QQuery 16 │    6.67 ms │                               6.82 ms │     no change │
   │ QQuery 17 │   79.23 ms │                              79.38 ms │     no change │
   │ QQuery 18 │  103.37 ms │                             104.12 ms │     no change │
   │ QQuery 19 │   41.18 ms │                              41.37 ms │     no change │
   │ QQuery 20 │   34.72 ms │                              35.12 ms │     no change │
   │ QQuery 21 │   17.05 ms │                              17.34 ms │     no change │
   │ QQuery 22 │   62.22 ms │                              63.26 ms │     no change │
   │ QQuery 23 │  308.00 ms │                             311.16 ms │     no change │
   │ QQuery 24 │  190.89 ms │                             197.19 ms │     no change │
   │ QQuery 25 │  108.86 ms │                             109.75 ms │     no change │
   │ QQuery 26 │   48.20 ms │                              48.30 ms │     no change │
   │ QQuery 27 │    5.97 ms │                               5.99 ms │     no change │
   │ QQuery 28 │   55.86 ms │                              55.73 ms │     no change │
   │ QQuery 29 │   96.71 ms │                              96.00 ms │     no change │
   │ QQuery 30 │   32.01 ms │                              31.70 ms │     no change │
   │ QQuery 31 │  110.41 ms │                             110.38 ms │     no change │
   │ QQuery 32 │   19.92 ms │                              19.95 ms │     no change │
   │ QQuery 33 │   37.05 ms │                              37.47 ms │     no change │
   │ QQuery 34 │   10.01 ms │                               9.97 ms │     no change │
   │ QQuery 35 │   72.41 ms │                              71.85 ms │     no change │
   │ QQuery 36 │    5.58 ms │                               5.57 ms │     no change │
   │ QQuery 37 │    6.74 ms │                               6.73 ms │     no change │
   │ QQuery 38 │   61.14 ms │                              60.94 ms │     no change │
   │ QQuery 39 │   88.88 ms │                              89.31 ms │     no change │
   │ QQuery 40 │   23.20 ms │                              23.82 ms │     no change │
   │ QQuery 41 │   10.91 ms │                              10.98 ms │     no change │
   │ QQuery 42 │   23.36 ms │                              24.12 ms │     no change │
   │ QQuery 43 │    5.13 ms │                               5.27 ms │     no change │
   │ QQuery 44 │    9.04 ms │                               9.26 ms │     no change │
   │ QQuery 45 │   38.22 ms │                              38.76 ms │     no change │
   │ QQuery 46 │   12.11 ms │                              12.12 ms │     no change │
   │ QQuery 47 │  222.65 ms │                             224.11 ms │     no change │
   │ QQuery 48 │   94.82 ms │                              95.48 ms │     no change │
   │ QQuery 49 │   70.06 ms │                              70.76 ms │     no change │
   │ QQuery 50 │   58.92 ms │                              58.53 ms │     no change │
   │ QQuery 51 │   92.30 ms │                              93.24 ms │     no change │
   │ QQuery 52 │   23.95 ms │                              23.85 ms │     no change │
   │ QQuery 53 │   29.30 ms │                              28.72 ms │     no change │
   │ QQuery 54 │   53.91 ms │                              53.64 ms │     no change │
   │ QQuery 55 │   23.30 ms │                              23.18 ms │     no change │
   │ QQuery 56 │   38.56 ms │                              38.42 ms │     no change │
   │ QQuery 57 │  174.65 ms │                             175.68 ms │     no change │
   │ QQuery 58 │  110.37 ms │                             111.88 ms │     no change │
   │ QQuery 59 │  117.96 ms │                             117.25 ms │     no change │
   │ QQuery 60 │   39.38 ms │                              38.94 ms │     no change │
   │ QQuery 61 │   11.95 ms │                              11.92 ms │     no change │
   │ QQuery 62 │   45.08 ms │                              45.81 ms │     no change │
   │ QQuery 63 │   28.88 ms │                              29.41 ms │     no change │
   │ QQuery 64 │  364.70 ms │                             368.87 ms │     no change │
   │ QQuery 65 │  124.70 ms │                             122.50 ms │     no change │
   │ QQuery 66 │   80.56 ms │                              81.28 ms │     no change │
   │ QQuery 67 │  239.94 ms │                             239.40 ms │     no change │
   │ QQuery 68 │   11.78 ms │                              11.74 ms │     no change │
   │ QQuery 69 │   56.25 ms │                              57.38 ms │     no change │
   │ QQuery 70 │  104.45 ms │                             109.94 ms │  1.05x slower │
   │ QQuery 71 │   35.12 ms │                              35.52 ms │     no change │
   │ QQuery 72 │ 1786.55 ms │                            1690.14 ms │ +1.06x faster │
   │ QQuery 73 │    9.61 ms │                               9.45 ms │     no change │
   │ QQuery 74 │  168.79 ms │                             166.18 ms │     no change │
   │ QQuery 75 │  145.30 ms │                             146.14 ms │     no change │
   │ QQuery 76 │   34.56 ms │                              35.04 ms │     no change │
   │ QQuery 77 │   59.92 ms │                              60.47 ms │     no change │
   │ QQuery 78 │  216.58 ms │                             218.22 ms │     no change │
   │ QQuery 79 │   66.50 ms │                              66.16 ms │     no change │
   │ QQuery 80 │   98.14 ms │                              96.67 ms │     no change │
   │ QQuery 81 │   25.57 ms │                              24.94 ms │     no change │
   │ QQuery 82 │   15.84 ms │                              15.72 ms │     no change │
   │ QQuery 83 │   33.35 ms │                              33.40 ms │     no change │
   │ QQuery 84 │   29.44 ms │                              29.15 ms │     no change │
   │ QQuery 85 │  101.06 ms │                             101.72 ms │     no change │
   │ QQuery 86 │   24.62 ms │                              24.95 ms │     no change │
   │ QQuery 87 │   62.13 ms │                              61.24 ms │     no change │
   │ QQuery 88 │   62.43 ms │                              62.49 ms │     no change │
   │ QQuery 89 │   34.85 ms │                              34.85 ms │     no change │
   │ QQuery 90 │   16.66 ms │                              16.62 ms │     no change │
   │ QQuery 91 │   45.31 ms │                              44.64 ms │     no change │
   │ QQuery 92 │   28.62 ms │                              28.58 ms │     no change │
   │ QQuery 93 │   50.16 ms │                              49.63 ms │     no change │
   │ QQuery 94 │   37.47 ms │                              37.13 ms │     no change │
   │ QQuery 95 │   80.40 ms │                              80.00 ms │     no change │
   │ QQuery 96 │   23.96 ms │                              23.80 ms │     no change │
   │ QQuery 97 │   52.67 ms │                              53.18 ms │     no change │
   │ QQuery 98 │   42.00 ms │                              41.99 ms │     no change │
   │ QQuery 99 │   69.40 ms │                              70.09 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 9211.75ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 9155.26ms │
   │ Average Time (HEAD)                                  │   93.05ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   92.48ms │
   │ Queries Faster                                       │         1 │
   │ Queries Slower                                       │         1 │
   │ Queries with No Change                               │        97 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │           5.49 / 5.99 ±0.93 / 7.85 ms │           5.53 / 6.01 ±0.85 / 7.72 ms │     no change │
   │ QQuery 2  │        80.66 / 80.91 ±0.14 / 81.02 ms │        80.12 / 80.53 ±0.30 / 80.87 ms │     no change │
   │ QQuery 3  │        28.39 / 28.72 ±0.26 / 29.11 ms │        28.55 / 28.82 ±0.24 / 29.26 ms │     no change │
   │ QQuery 4  │     471.91 / 478.84 ±3.99 / 482.68 ms │     483.52 / 484.81 ±0.97 / 486.41 ms │     no change │
   │ QQuery 5  │        51.62 / 52.19 ±0.53 / 53.17 ms │        51.41 / 54.73 ±5.34 / 65.38 ms │     no change │
   │ QQuery 6  │        35.98 / 36.26 ±0.23 / 36.62 ms │        35.82 / 36.43 ±0.33 / 36.81 ms │     no change │
   │ QQuery 7  │        74.62 / 75.01 ±0.23 / 75.27 ms │        74.56 / 75.36 ±0.64 / 76.26 ms │     no change │
   │ QQuery 8  │        35.79 / 36.32 ±0.35 / 36.84 ms │        36.14 / 36.56 ±0.38 / 37.11 ms │     no change │
   │ QQuery 9  │        52.19 / 53.44 ±0.93 / 54.76 ms │        49.99 / 54.89 ±4.88 / 64.23 ms │     no change │
   │ QQuery 10 │        62.00 / 62.62 ±0.36 / 63.14 ms │        62.53 / 63.19 ±0.37 / 63.58 ms │     no change │
   │ QQuery 11 │     297.69 / 299.75 ±2.11 / 303.28 ms │     297.54 / 301.38 ±2.13 / 303.75 ms │     no change │
   │ QQuery 12 │        28.31 / 28.80 ±0.28 / 29.17 ms │        28.22 / 28.50 ±0.34 / 29.10 ms │     no change │
   │ QQuery 13 │     117.09 / 117.76 ±0.35 / 118.09 ms │     118.14 / 119.12 ±1.62 / 122.33 ms │     no change │
   │ QQuery 14 │     412.64 / 417.57 ±3.15 / 420.90 ms │     415.84 / 418.15 ±1.69 / 420.10 ms │     no change │
   │ QQuery 15 │        56.88 / 58.43 ±2.57 / 63.52 ms │        57.57 / 58.49 ±0.67 / 59.61 ms │     no change │
   │ QQuery 16 │           6.67 / 6.85 ±0.23 / 7.29 ms │           6.82 / 7.00 ±0.18 / 7.35 ms │     no change │
   │ QQuery 17 │        79.23 / 80.65 ±1.70 / 83.86 ms │        79.38 / 81.43 ±1.68 / 84.07 ms │     no change │
   │ QQuery 18 │     103.37 / 104.71 ±1.30 / 106.43 ms │     104.12 / 104.78 ±0.63 / 105.98 ms │     no change │
   │ QQuery 19 │        41.18 / 41.50 ±0.25 / 41.90 ms │        41.37 / 41.90 ±0.39 / 42.48 ms │     no change │
   │ QQuery 20 │        34.72 / 35.21 ±0.30 / 35.53 ms │        35.12 / 35.37 ±0.21 / 35.63 ms │     no change │
   │ QQuery 21 │        17.05 / 17.26 ±0.26 / 17.68 ms │        17.34 / 17.97 ±0.97 / 19.86 ms │     no change │
   │ QQuery 22 │        62.22 / 63.01 ±0.43 / 63.43 ms │        63.26 / 64.20 ±1.08 / 65.74 ms │     no change │
   │ QQuery 23 │     308.00 / 309.23 ±1.51 / 312.12 ms │     311.16 / 313.97 ±3.77 / 321.43 ms │     no change │
   │ QQuery 24 │     190.89 / 192.50 ±0.96 / 193.74 ms │     197.19 / 201.37 ±5.32 / 211.44 ms │     no change │
   │ QQuery 25 │     108.86 / 110.73 ±1.23 / 112.29 ms │     109.75 / 113.69 ±3.17 / 118.22 ms │     no change │
   │ QQuery 26 │        48.20 / 48.78 ±0.69 / 50.09 ms │        48.30 / 48.79 ±0.43 / 49.49 ms │     no change │
   │ QQuery 27 │           5.97 / 6.14 ±0.21 / 6.53 ms │           5.99 / 6.14 ±0.21 / 6.54 ms │     no change │
   │ QQuery 28 │        55.86 / 58.67 ±1.44 / 59.76 ms │        55.73 / 61.50 ±3.96 / 68.11 ms │     no change │
   │ QQuery 29 │       96.71 / 98.55 ±1.27 / 100.48 ms │       96.00 / 98.48 ±2.04 / 101.95 ms │     no change │
   │ QQuery 30 │        32.01 / 32.65 ±0.46 / 33.25 ms │        31.70 / 32.11 ±0.24 / 32.43 ms │     no change │
   │ QQuery 31 │     110.41 / 112.23 ±1.67 / 114.82 ms │     110.38 / 111.09 ±0.38 / 111.43 ms │     no change │
   │ QQuery 32 │        19.92 / 20.30 ±0.34 / 20.90 ms │        19.95 / 20.51 ±0.40 / 21.14 ms │     no change │
   │ QQuery 33 │        37.05 / 37.74 ±0.37 / 38.08 ms │        37.47 / 38.64 ±1.65 / 41.83 ms │     no change │
   │ QQuery 34 │        10.01 / 10.23 ±0.18 / 10.40 ms │         9.97 / 10.09 ±0.14 / 10.37 ms │     no change │
   │ QQuery 35 │        72.41 / 72.70 ±0.24 / 73.14 ms │        71.85 / 72.55 ±0.44 / 73.14 ms │     no change │
   │ QQuery 36 │           5.58 / 5.76 ±0.23 / 6.21 ms │           5.57 / 5.71 ±0.25 / 6.21 ms │     no change │
   │ QQuery 37 │           6.74 / 6.83 ±0.05 / 6.90 ms │          6.73 / 7.94 ±2.29 / 12.51 ms │  1.16x slower │
   │ QQuery 38 │        61.14 / 61.79 ±0.46 / 62.25 ms │        60.94 / 62.35 ±0.96 / 63.88 ms │     no change │
   │ QQuery 39 │        88.88 / 89.49 ±0.40 / 90.03 ms │        89.31 / 89.66 ±0.29 / 90.17 ms │     no change │
   │ QQuery 40 │        23.20 / 23.77 ±0.33 / 24.09 ms │        23.82 / 24.53 ±0.62 / 25.59 ms │     no change │
   │ QQuery 41 │        10.91 / 11.02 ±0.11 / 11.18 ms │        10.98 / 11.21 ±0.17 / 11.37 ms │     no change │
   │ QQuery 42 │        23.36 / 23.58 ±0.21 / 23.97 ms │        24.12 / 25.11 ±0.53 / 25.60 ms │  1.06x slower │
   │ QQuery 43 │           5.13 / 5.25 ±0.17 / 5.58 ms │           5.27 / 5.37 ±0.13 / 5.62 ms │     no change │
   │ QQuery 44 │         9.04 / 10.32 ±2.36 / 15.04 ms │           9.26 / 9.39 ±0.12 / 9.61 ms │ +1.10x faster │
   │ QQuery 45 │        38.22 / 39.42 ±0.88 / 40.85 ms │        38.76 / 38.92 ±0.20 / 39.30 ms │     no change │
   │ QQuery 46 │        12.11 / 12.66 ±0.62 / 13.83 ms │        12.12 / 12.42 ±0.30 / 12.94 ms │     no change │
   │ QQuery 47 │     222.65 / 227.94 ±3.95 / 233.58 ms │     224.11 / 231.18 ±5.57 / 238.18 ms │     no change │
   │ QQuery 48 │        94.82 / 95.57 ±0.60 / 96.57 ms │        95.48 / 96.20 ±0.46 / 96.83 ms │     no change │
   │ QQuery 49 │        70.06 / 70.60 ±0.59 / 71.72 ms │        70.76 / 71.31 ±0.62 / 72.52 ms │     no change │
   │ QQuery 50 │        58.92 / 59.57 ±0.41 / 60.08 ms │        58.53 / 60.10 ±1.36 / 62.55 ms │     no change │
   │ QQuery 51 │       92.30 / 94.84 ±2.95 / 100.44 ms │        93.24 / 94.95 ±1.24 / 97.11 ms │     no change │
   │ QQuery 52 │        23.95 / 24.56 ±0.53 / 25.42 ms │        23.85 / 24.09 ±0.18 / 24.31 ms │     no change │
   │ QQuery 53 │        29.30 / 29.72 ±0.29 / 30.02 ms │        28.72 / 29.35 ±0.70 / 30.71 ms │     no change │
   │ QQuery 54 │        53.91 / 54.41 ±0.28 / 54.65 ms │        53.64 / 54.46 ±0.55 / 55.31 ms │     no change │
   │ QQuery 55 │        23.30 / 23.42 ±0.11 / 23.62 ms │        23.18 / 23.37 ±0.15 / 23.53 ms │     no change │
   │ QQuery 56 │        38.56 / 39.05 ±0.34 / 39.53 ms │        38.42 / 40.55 ±2.89 / 46.23 ms │     no change │
   │ QQuery 57 │     174.65 / 175.35 ±0.57 / 176.25 ms │     175.68 / 179.01 ±2.33 / 182.98 ms │     no change │
   │ QQuery 58 │     110.37 / 112.20 ±1.69 / 115.17 ms │     111.88 / 113.78 ±1.06 / 115.05 ms │     no change │
   │ QQuery 59 │     117.96 / 118.98 ±0.74 / 120.25 ms │     117.25 / 119.16 ±2.35 / 123.79 ms │     no change │
   │ QQuery 60 │        39.38 / 39.63 ±0.17 / 39.89 ms │        38.94 / 39.77 ±0.82 / 40.80 ms │     no change │
   │ QQuery 61 │        11.95 / 12.15 ±0.22 / 12.56 ms │        11.92 / 12.05 ±0.16 / 12.35 ms │     no change │
   │ QQuery 62 │        45.08 / 45.54 ±0.39 / 46.22 ms │        45.81 / 46.94 ±1.42 / 49.72 ms │     no change │
   │ QQuery 63 │        28.88 / 29.49 ±0.56 / 30.54 ms │        29.41 / 29.63 ±0.18 / 29.94 ms │     no change │
   │ QQuery 64 │     364.70 / 370.26 ±5.23 / 380.19 ms │     368.87 / 373.16 ±3.56 / 378.02 ms │     no change │
   │ QQuery 65 │     124.70 / 125.57 ±0.60 / 126.17 ms │     122.50 / 125.76 ±2.77 / 130.37 ms │     no change │
   │ QQuery 66 │        80.56 / 82.70 ±2.86 / 88.35 ms │        81.28 / 82.28 ±1.11 / 84.42 ms │     no change │
   │ QQuery 67 │     239.94 / 244.37 ±4.14 / 250.22 ms │     239.40 / 243.56 ±2.21 / 245.82 ms │     no change │
   │ QQuery 68 │        11.78 / 11.93 ±0.16 / 12.23 ms │        11.74 / 11.98 ±0.21 / 12.36 ms │     no change │
   │ QQuery 69 │        56.25 / 56.65 ±0.27 / 56.97 ms │        57.38 / 59.64 ±2.40 / 62.79 ms │  1.05x slower │
   │ QQuery 70 │     104.45 / 108.97 ±5.27 / 118.84 ms │     109.94 / 110.62 ±0.42 / 111.06 ms │     no change │
   │ QQuery 71 │        35.12 / 35.62 ±0.36 / 36.11 ms │        35.52 / 37.43 ±2.93 / 43.22 ms │  1.05x slower │
   │ QQuery 72 │ 1786.55 / 1849.50 ±40.52 / 1914.12 ms │ 1690.14 / 1760.22 ±71.70 / 1873.30 ms │     no change │
   │ QQuery 73 │         9.61 / 10.08 ±0.28 / 10.41 ms │          9.45 / 9.98 ±0.34 / 10.41 ms │     no change │
   │ QQuery 74 │     168.79 / 170.96 ±1.90 / 173.80 ms │     166.18 / 169.50 ±2.13 / 172.58 ms │     no change │
   │ QQuery 75 │     145.30 / 147.05 ±1.33 / 149.22 ms │     146.14 / 147.78 ±1.81 / 151.25 ms │     no change │
   │ QQuery 76 │        34.56 / 34.89 ±0.17 / 35.04 ms │        35.04 / 35.47 ±0.43 / 36.30 ms │     no change │
   │ QQuery 77 │        59.92 / 62.43 ±3.99 / 70.40 ms │        60.47 / 61.39 ±1.20 / 63.68 ms │     no change │
   │ QQuery 78 │     216.58 / 226.80 ±8.29 / 236.23 ms │     218.22 / 225.25 ±4.77 / 233.09 ms │     no change │
   │ QQuery 79 │        66.50 / 67.26 ±1.07 / 69.35 ms │        66.16 / 66.93 ±0.46 / 67.41 ms │     no change │
   │ QQuery 80 │       98.14 / 99.08 ±1.02 / 100.91 ms │       96.67 / 99.17 ±1.33 / 100.35 ms │     no change │
   │ QQuery 81 │        25.57 / 27.35 ±2.95 / 33.23 ms │        24.94 / 26.58 ±2.08 / 30.69 ms │     no change │
   │ QQuery 82 │        15.84 / 16.42 ±0.34 / 16.76 ms │        15.72 / 16.54 ±1.07 / 18.62 ms │     no change │
   │ QQuery 83 │        33.35 / 33.76 ±0.29 / 34.20 ms │        33.40 / 35.48 ±3.45 / 42.35 ms │  1.05x slower │
   │ QQuery 84 │        29.44 / 30.39 ±1.59 / 33.55 ms │        29.15 / 29.49 ±0.32 / 30.06 ms │     no change │
   │ QQuery 85 │     101.06 / 101.88 ±1.03 / 103.85 ms │     101.72 / 103.68 ±1.52 / 106.15 ms │     no change │
   │ QQuery 86 │        24.62 / 27.24 ±4.84 / 36.91 ms │        24.95 / 25.79 ±0.96 / 27.59 ms │ +1.06x faster │
   │ QQuery 87 │        62.13 / 63.28 ±1.24 / 65.63 ms │        61.24 / 62.26 ±0.68 / 63.28 ms │     no change │
   │ QQuery 88 │        62.43 / 63.05 ±0.45 / 63.60 ms │        62.49 / 62.99 ±0.27 / 63.30 ms │     no change │
   │ QQuery 89 │        34.85 / 35.28 ±0.31 / 35.64 ms │        34.85 / 35.62 ±0.49 / 36.07 ms │     no change │
   │ QQuery 90 │        16.66 / 18.97 ±4.33 / 27.62 ms │        16.62 / 16.83 ±0.22 / 17.18 ms │ +1.13x faster │
   │ QQuery 91 │        45.31 / 45.73 ±0.32 / 46.00 ms │        44.64 / 45.33 ±0.72 / 46.60 ms │     no change │
   │ QQuery 92 │        28.62 / 29.63 ±0.78 / 30.86 ms │        28.58 / 29.27 ±0.57 / 30.02 ms │     no change │
   │ QQuery 93 │        50.16 / 50.82 ±0.59 / 51.85 ms │        49.63 / 50.61 ±0.65 / 51.54 ms │     no change │
   │ QQuery 94 │        37.47 / 37.80 ±0.27 / 38.23 ms │        37.13 / 38.27 ±1.00 / 40.11 ms │     no change │
   │ QQuery 95 │        80.40 / 81.90 ±1.81 / 85.37 ms │        80.00 / 80.89 ±0.83 / 82.40 ms │     no change │
   │ QQuery 96 │        23.96 / 24.53 ±0.71 / 25.90 ms │        23.80 / 24.83 ±1.75 / 28.33 ms │     no change │
   │ QQuery 97 │        52.67 / 54.17 ±1.45 / 56.65 ms │        53.18 / 54.38 ±1.14 / 56.51 ms │     no change │
   │ QQuery 98 │        42.00 / 42.36 ±0.47 / 43.29 ms │        41.99 / 43.12 ±0.87 / 44.41 ms │     no change │
   │ QQuery 99 │        69.40 / 69.96 ±0.35 / 70.31 ms │        70.09 / 70.46 ±0.28 / 70.89 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 9393.75ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 9359.60ms │
   │ Average Time (HEAD)                                  │   94.89ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   94.54ms │
   │ Queries Faster                                       │         3 │
   │ Queries Slower                                       │         5 │
   │ Queries with No Change                               │        91 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 2.1 GiB |
   | Avg memory | 1.4 GiB |
   | CPU user | 205.3s |
   | CPU sys | 5.5s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 2.1 GiB |
   | Avg memory | 1.5 GiB |
   | CPU user | 200.9s |
   | CPU sys | 5.8s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5514079377

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513845153)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (6166c31c21bdd7965f348ae7a126a2d1e85a6216) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..6166c31c21bdd7965f348ae7a126a2d1e85a6216)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.75 ms │                              38.57 ms │    no change │
   │ QQuery 2  │ 18.69 ms │                              18.66 ms │    no change │
   │ QQuery 3  │ 28.91 ms │                              29.15 ms │    no change │
   │ QQuery 4  │ 17.75 ms │                              18.48 ms │    no change │
   │ QQuery 5  │ 35.39 ms │                              38.57 ms │ 1.09x slower │
   │ QQuery 6  │ 15.98 ms │                              17.32 ms │ 1.08x slower │
   │ QQuery 7  │ 42.10 ms │                              45.96 ms │ 1.09x slower │
   │ QQuery 8  │ 42.00 ms │                              43.88 ms │    no change │
   │ QQuery 9  │ 51.40 ms │                              54.71 ms │ 1.06x slower │
   │ QQuery 10 │ 43.78 ms │                              46.10 ms │ 1.05x slower │
   │ QQuery 11 │ 14.14 ms │                              14.78 ms │    no change │
   │ QQuery 12 │ 25.12 ms │                              26.07 ms │    no change │
   │ QQuery 13 │ 42.99 ms │                              47.00 ms │ 1.09x slower │
   │ QQuery 14 │ 25.20 ms │                              26.52 ms │ 1.05x slower │
   │ QQuery 15 │ 31.02 ms │                              33.86 ms │ 1.09x slower │
   │ QQuery 16 │ 14.03 ms │                              15.53 ms │ 1.11x slower │
   │ QQuery 17 │ 71.91 ms │                              86.72 ms │ 1.21x slower │
   │ QQuery 18 │ 59.53 ms │                              60.75 ms │    no change │
   │ QQuery 19 │ 32.74 ms │                              32.96 ms │    no change │
   │ QQuery 20 │ 31.47 ms │                              31.86 ms │    no change │
   │ QQuery 21 │ 55.67 ms │                              55.27 ms │    no change │
   │ QQuery 22 │ 14.03 ms │                              14.16 ms │    no change │
   └───────────┴──────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 752.62ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 796.90ms │
   │ Average Time (HEAD)                                  │  34.21ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  36.22ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │       10 │
   │ Queries with No Change                               │       12 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.75 / 40.41 ±1.35 / 41.81 ms │        38.57 / 39.34 ±1.11 / 41.56 ms │    no change │
   │ QQuery 2  │ 18.69 / 19.07 ±0.24 / 19.34 ms │        18.66 / 18.98 ±0.31 / 19.53 ms │    no change │
   │ QQuery 3  │ 28.91 / 29.46 ±0.52 / 30.36 ms │        29.15 / 30.08 ±1.02 / 32.07 ms │    no change │
   │ QQuery 4  │ 17.75 / 18.20 ±0.42 / 18.74 ms │        18.48 / 18.63 ±0.10 / 18.74 ms │    no change │
   │ QQuery 5  │ 35.39 / 35.87 ±0.38 / 36.48 ms │        38.57 / 40.20 ±1.12 / 41.96 ms │ 1.12x slower │
   │ QQuery 6  │ 15.98 / 16.15 ±0.13 / 16.33 ms │        17.32 / 17.42 ±0.08 / 17.56 ms │ 1.08x slower │
   │ QQuery 7  │ 42.10 / 42.94 ±1.08 / 45.06 ms │        45.96 / 47.39 ±1.48 / 50.15 ms │ 1.10x slower │
   │ QQuery 8  │ 42.00 / 43.24 ±1.77 / 46.72 ms │        43.88 / 45.19 ±1.30 / 47.48 ms │    no change │
   │ QQuery 9  │ 51.40 / 52.20 ±0.85 / 53.41 ms │        54.71 / 55.61 ±0.87 / 57.09 ms │ 1.07x slower │
   │ QQuery 10 │ 43.78 / 45.42 ±1.97 / 48.57 ms │        46.10 / 46.48 ±0.41 / 47.27 ms │    no change │
   │ QQuery 11 │ 14.14 / 14.38 ±0.19 / 14.62 ms │        14.78 / 15.33 ±0.88 / 17.09 ms │ 1.07x slower │
   │ QQuery 12 │ 25.12 / 25.57 ±0.39 / 26.24 ms │        26.07 / 27.08 ±1.05 / 28.82 ms │ 1.06x slower │
   │ QQuery 13 │ 42.99 / 44.13 ±1.56 / 47.19 ms │        47.00 / 47.65 ±1.01 / 49.65 ms │ 1.08x slower │
   │ QQuery 14 │ 25.20 / 25.38 ±0.11 / 25.52 ms │        26.52 / 26.84 ±0.49 / 27.80 ms │ 1.06x slower │
   │ QQuery 15 │ 31.02 / 31.42 ±0.50 / 32.40 ms │        33.86 / 34.63 ±0.70 / 35.66 ms │ 1.10x slower │
   │ QQuery 16 │ 14.03 / 14.32 ±0.22 / 14.57 ms │        15.53 / 15.68 ±0.24 / 16.15 ms │ 1.09x slower │
   │ QQuery 17 │ 71.91 / 72.62 ±0.48 / 73.18 ms │        86.72 / 87.89 ±1.06 / 89.59 ms │ 1.21x slower │
   │ QQuery 18 │ 59.53 / 62.94 ±3.97 / 70.44 ms │        60.75 / 65.55 ±3.93 / 70.49 ms │    no change │
   │ QQuery 19 │ 32.74 / 33.24 ±0.35 / 33.73 ms │        32.96 / 33.93 ±1.06 / 35.78 ms │    no change │
   │ QQuery 20 │ 31.47 / 31.89 ±0.55 / 32.97 ms │        31.86 / 32.13 ±0.31 / 32.70 ms │    no change │
   │ QQuery 21 │ 55.67 / 56.84 ±0.93 / 58.23 ms │        55.27 / 57.13 ±1.53 / 59.88 ms │    no change │
   │ QQuery 22 │ 14.03 / 14.15 ±0.14 / 14.40 ms │        14.16 / 14.39 ±0.13 / 14.50 ms │    no change │
   └───────────┴────────────────────────────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 769.83ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 817.54ms │
   │ Average Time (HEAD)                                  │  34.99ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  37.16ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │       11 │
   │ Queries with No Change                               │       11 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.1 GiB |
   | Avg memory | 486.4 MiB |
   | CPU user | 21.5s |
   | CPU sys | 1.9s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.0 GiB |
   | Avg memory | 630.7 MiB |
   | CPU user | 23.1s |
   | CPU sys | 1.9s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364740537

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364728282)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5364728282-1827-xzmbg 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5309972905

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion v54.1.0 (current)
          Built [  60.821s] (current)
        Parsing datafusion v54.1.0 (current)
         Parsed [   0.035s] (current)
       Building datafusion v54.1.0 (baseline)
          Built [  58.577s] (baseline)
        Parsing datafusion v54.1.0 (baseline)
         Parsed [   0.034s] (baseline)
       Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.601s] 223 checks: 223 pass, 31 skip
        Summary no semver update required
       Finished [ 121.795s] datafusion
       Building datafusion-common v54.1.0 (current)
          Built [  34.685s] (current)
        Parsing datafusion-common v54.1.0 (current)
         Parsed [   0.060s] (current)
       Building datafusion-common v54.1.0 (baseline)
          Built [  34.284s] (baseline)
        Parsing datafusion-common v54.1.0 (baseline)
         Parsed [   0.063s] (baseline)
       Checking datafusion-common v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.645s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
   
   --- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
   
   Description:
   A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1196
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [  70.837s] datafusion-common
       Building datafusion-datasource-parquet v54.1.0 (current)
          Built [  48.961s] (current)
        Parsing datafusion-datasource-parquet v54.1.0 (current)
         Parsed [   0.032s] (current)
       Building datafusion-datasource-parquet v54.1.0 (baseline)
          Built [  48.795s] (baseline)
        Parsing datafusion-datasource-parquet v54.1.0 (baseline)
         Parsed [   0.033s] (baseline)
       Checking datafusion-datasource-parquet v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.150s] 223 checks: 223 pass, 31 skip
        Summary no semver update required
       Finished [  99.352s] datafusion-datasource-parquet
       Building datafusion-proto v54.1.0 (current)
          Built [  54.025s] (current)
        Parsing datafusion-proto v54.1.0 (current)
         Parsed [   0.017s] (current)
       Building datafusion-proto v54.1.0 (baseline)
          Built [  54.414s] (baseline)
        Parsing datafusion-proto v54.1.0 (baseline)
         Parsed [   0.019s] (baseline)
       Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.125s] 223 checks: 223 pass, 31 skip
        Summary no semver update required
       Finished [ 109.886s] datafusion-proto
       Building datafusion-proto-common v54.1.0 (current)
          Built [  22.424s] (current)
        Parsing datafusion-proto-common v54.1.0 (current)
         Parsed [   0.046s] (current)
       Building datafusion-proto-common v54.1.0 (baseline)
          Built [  22.563s] (baseline)
        Parsing datafusion-proto-common v54.1.0 (baseline)
         Parsed [   0.048s] (baseline)
       Checking datafusion-proto-common v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   1.058s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
   
   --- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
   
   Description:
   A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:868
     field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:868
     field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:868
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [  47.265s] datafusion-proto-common
       Building datafusion-proto-models v54.1.0 (current)
          Built [  25.553s] (current)
        Parsing datafusion-proto-models v54.1.0 (current)
         Parsed [   0.129s] (current)
       Building datafusion-proto-models v54.1.0 (baseline)
          Built [  25.327s] (baseline)
        Parsing datafusion-proto-models v54.1.0 (baseline)
         Parsed [   0.128s] (baseline)
       Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   1.765s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
   
   --- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
   
   Description:
   A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:868
     field ParquetOptions.enable_rle_to_dictionary in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:868
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [  54.495s] datafusion-proto-models
       Building datafusion-sqllogictest v54.1.0 (current)
          Built [ 101.048s] (current)
        Parsing datafusion-sqllogictest v54.1.0 (current)
         Parsed [   0.021s] (current)
       Building datafusion-sqllogictest v54.1.0 (baseline)
          Built [ 101.122s] (baseline)
        Parsing datafusion-sqllogictest v54.1.0 (baseline)
         Parsed [   0.022s] (baseline)
       Checking datafusion-sqllogictest v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.103s] 223 checks: 223 pass, 31 skip
        Summary no semver update required
       Finished [ 205.091s] datafusion-sqllogictest
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5404645612

   @kumarUjjawal 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5860790032

   @kumarUjjawal seems we didn't get much of a response. @alamb & @adriangb have some context on what were doing, they may have input
   
   cc @asolimando for visibility 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513845153

   run benchmarks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364728282

   run benchmarks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364868497

   run benchmarks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5345862338

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5345832494)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5345832494-1769-zjwzs 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (449355e7c5098104ce4868fa3ce347ca95a24ce3) to bbf739a (merge-base) [diff](https://github.com/apache/datafusion/compare/bbf739a3cfe1d5cc7646e37250c947c6a56b5b0e..449355e7c5098104ce4868fa3ce347ca95a24ce3)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch10
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5374297430

   run benchmarks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5873009385

   > Is the goal of this PR to improve performance for some usecases? If so, do we have any benchmark results that show the performance improves?
   
   @alamb yes & no. The end goal is faster queries & providing users with more control over how their data is read. This initial PR solves the second problem. As for the query speedup, I would have expected a larger speedup for certain queries. As we suspected, and as @yinli-systems pointed out [here](https://github.com/apache/datafusion/issues/24117#issuecomment-5671192732), there is a cutoff for the usefulness of dictionaries. I made a follow-up issue to address this and bridge the performance gaps.
   
   On a slightly unrelated note, it will be much easier to track performance changes and gains with this PR, as we can just run the regular benchmark suite with this flag enabled. This means a faster iteration cycle.
   
   I plan on going through the list of optimizations pointed out in the issue and then running the benchmarks with the flag enabled vs disabled. for any other discrepancies with performance ill do in in depth profile to see where we may be missing optimizations


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5629505825

   > its also ready for review when you get a chance 🙇
   
   It's on my list for this weekend. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5692251700

   > @kumarUjjawal any status update on this?
   
   Based on the reported benchmark number: 
   
   - TPC-H is approximately 10.7% slower.
   
   - TPC-DS is approximately 4–5% slower, although it uses less memory.
   
   - ClickBench is approximately 3.75–3.83× slower overall. Q33 and Q34 are approximately 9× slower, and both process memory and MemoryPool reservations increase.
   
   - wide_schema failed while running the disabled baseline, so it does not provide a comparison.
   
   These results reproduce the earlier performance direction. I think the implementation needs to avoid automatically promoting high-cardinality or dictionary-fallback columns.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["yinli-systems (via GitHub)" <gi...@apache.org>]

yinli-systems commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5672875242

   I checked the remaining `cargo test datafusion-cli (amd64)` failure because #24227 is the scan dependency for the #24117 optimizer prototype.
   
   All four failed tests stop during MinIO setup before exercising this PR: the runner cannot pull `minio/minio:RELEASE.2025-02-28T09-55-16Z` and receives Docker Hub 404 / access denied. Current `main` has already moved the integration test image to `quay.io/minio/minio`, so this looks like a stale merge-run infrastructure failure rather than a code failure in #24227. A fresh CI run against current `main` should exercise the corrected image.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675320003

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5675290885-2365-msp2d 6.12.94+ #1 SMP Tue Aug  4 08:44:15 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f (cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f) to cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f [diff](https://github.com/apache/datafusion/compare/cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f..cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   env:
     DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
   baseline:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
   changed:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675329490

   show benchmark queue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5698308429

   Thank you @Rich-T-kid I will give another pass.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5808731011

   Thank you @Rich-T-kid for moving this forward. Since this is a larger change I will need second pair of eyes on this before this can be approved. Let's see if anyone has time to take a look. Can you share the work in discord that might bring some people over.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3962339698


##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -135,6 +157,169 @@ pub fn apply_file_schema_type_coercions(
     ))
 }
 
+// Find the value type that can represent both sides without narrowing offsets
+// or crossing string/binary families.
+fn common_dictionary_value_type(
+    field_type: &DataType,
+    dictionary_value_type: &DataType,
+) -> Option<DataType> {
+    let field_type = match field_type {
+        DataType::Dictionary(_, field_value_type) => field_value_type.as_ref(),
+        _ => field_type,
+    };
+
+    match (field_type, dictionary_value_type) {
+        (DataType::Utf8, DataType::Utf8) => Some(DataType::Utf8),
+        (DataType::Utf8 | DataType::LargeUtf8, DataType::Utf8 | DataType::LargeUtf8) => {
+            Some(DataType::LargeUtf8)
+        }
+        (DataType::Binary, DataType::Binary) => Some(DataType::Binary),
+        (
+            DataType::Binary | DataType::LargeBinary,
+            DataType::Binary | DataType::LargeBinary,
+        ) => Some(DataType::LargeBinary),
+        _ => None,
+    }
+}
+
+// Same family (both signed or both unsigned): return the wider member.

Review Comment:
   > Could this select UInt64 for that case, or reject the normalization instead of silently narrowing it?
   
   with https://github.com/apache/datafusion/pull/24227/commits/6398041e1fcb0536c5a43a2708af58ea8cddd8a8 the type now widens instead of narrowing silently 



##########
datafusion/sqllogictest/test_files/parquet_rle_to_dictionary.slt:
##########
@@ -0,0 +1,334 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Tests for datafusion.execution.parquet.enable_rle_to_dictionary. The flag
+# applies only when DataFusion infers the table schema; explicit schemas are
+# not promoted.
+
+# Write with dictionary_enabled=true to guarantee RLE_DICTIONARY encoding.
+query I
+COPY (
+  SELECT column1 AS product_category, column2 AS status
+  FROM (VALUES
+    ('electronics', 'active'),
+    ('clothing',    'active'),
+    ('electronics', 'inactive'),
+    ('books',       'inactive'),
+    ('furniture',   'pending'),
+    ('books',       'active')
+  )
+)
+TO 'test_files/scratch/parquet_rle_to_dictionary/products.parquet'
+STORED AS PARQUET
+OPTIONS ('format.dictionary_enabled' true);
+----
+6
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = false;
+
+statement ok
+CREATE EXTERNAL TABLE products_utf8
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/products.parquet';
+
+query TT
+SELECT DISTINCT arrow_typeof(product_category), arrow_typeof(status) FROM products_utf8;
+----
+Utf8View Utf8View
+
+query TI rowsort
+SELECT product_category, COUNT(*) FROM products_utf8 GROUP BY product_category;
+----
+books 2
+clothing 1
+electronics 2
+furniture 1
+
+statement ok
+DROP TABLE products_utf8;
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = true;
+
+statement ok
+CREATE EXTERNAL TABLE products_dict
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/products.parquet';
+
+query TT
+SELECT DISTINCT arrow_typeof(product_category), arrow_typeof(status) FROM products_dict;
+----
+Dictionary(Int32, Utf8) Dictionary(Int32, Utf8)
+
+query TI rowsort
+SELECT product_category, COUNT(*) FROM products_dict GROUP BY product_category;
+----
+books 2
+clothing 1
+electronics 2
+furniture 1
+
+# Predicates and aggregates must work against dictionary scan output.
+query TT rowsort
+SELECT product_category, status FROM products_dict WHERE status = 'active';
+----
+books active
+clothing active
+electronics active
+
+query TI rowsort
+SELECT product_category, COUNT(*) FROM products_dict WHERE status != 'inactive' GROUP BY product_category;
+----
+books 1
+clothing 1
+electronics 1
+furniture 1
+
+query TT
+SELECT DISTINCT arrow_typeof(product_category), arrow_typeof(product_count)
+FROM (
+  SELECT product_category, COUNT(*) AS product_count
+  FROM products_dict
+  WHERE status != 'inactive'
+  GROUP BY product_category
+);
+----
+Dictionary(Int32, Utf8) Int64
+
+statement ok
+DROP TABLE products_dict;
+
+# Per-column writer options: only columns with dictionary pages are promoted.
+
+statement ok
+COPY (
+  SELECT column1 AS env, column2 AS region, column3 AS build
+  FROM (VALUES
+    ('prod',    'us-east', 'debug'),
+    ('staging', 'us-west', 'release'),
+    ('prod',    'us-east', 'debug')
+  )
+)
+TO 'test_files/scratch/parquet_rle_to_dictionary/selective.parquet'
+STORED AS PARQUET
+OPTIONS (
+  'format.dictionary_enabled' false,
+  'format.dictionary_enabled::env' true,
+  'format.dictionary_enabled::region' true
+);
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = false;
+
+statement ok
+CREATE EXTERNAL TABLE selective_utf8
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/selective.parquet';
+
+query TTT
+SELECT DISTINCT arrow_typeof(env), arrow_typeof(region), arrow_typeof(build)
+FROM selective_utf8;
+----
+Utf8View Utf8View Utf8View
+
+statement ok
+DROP TABLE selective_utf8;
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = true;
+
+statement ok
+CREATE EXTERNAL TABLE selective_dict
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/selective.parquet';
+
+query TTT
+SELECT DISTINCT arrow_typeof(env), arrow_typeof(region), arrow_typeof(build)
+FROM selective_dict;
+----
+Dictionary(Int32, Utf8) Dictionary(Int32, Utf8) Utf8View
+
+statement ok
+DROP TABLE selective_dict;
+
+# Cross-file mixed encoding: one file has dictionary pages and one is plain.
+# Schema inference must merge both as Dictionary(Int32, Utf8).
+
+statement ok
+COPY (SELECT column1 AS category FROM (VALUES ('electronics'), ('electronics')))
+TO 'test_files/scratch/parquet_rle_to_dictionary/mixed/rle.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' true);
+
+statement ok
+COPY (SELECT column1 AS category FROM (VALUES ('books'), ('books')))
+TO 'test_files/scratch/parquet_rle_to_dictionary/mixed/plain.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' false);
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = true;
+
+statement ok
+CREATE EXTERNAL TABLE mixed_encoding
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/mixed/';
+
+query TI
+SELECT arrow_typeof(category), COUNT(*) FROM mixed_encoding GROUP BY arrow_typeof(category);
+----
+Dictionary(Int32, Utf8) 4
+
+statement ok
+DROP TABLE mixed_encoding;
+
+# Mixed encoding with high cardinality: the plain file contains more than 128
+# distinct values so a narrow (Int8) dictionary key would overflow. After the
+# fix, uniform_dict_schemas widens the shared key to at least Int32.
+# Both files must be scannable and return the correct row count.
+
+statement ok
+COPY (
+  SELECT v AS category
+  FROM (
+    VALUES
+      ('v000'),('v001'),('v002'),('v003'),('v004'),('v005'),('v006'),('v007'),
+      ('v008'),('v009'),('v010'),('v011'),('v012'),('v013'),('v014'),('v015'),
+      ('v016'),('v017'),('v018'),('v019'),('v020'),('v021'),('v022'),('v023'),
+      ('v024'),('v025'),('v026'),('v027'),('v028'),('v029'),('v030'),('v031'),
+      ('v032'),('v033'),('v034'),('v035'),('v036'),('v037'),('v038'),('v039'),
+      ('v040'),('v041'),('v042'),('v043'),('v044'),('v045'),('v046'),('v047'),
+      ('v048'),('v049'),('v050'),('v051'),('v052'),('v053'),('v054'),('v055'),
+      ('v056'),('v057'),('v058'),('v059'),('v060'),('v061'),('v062'),('v063'),
+      ('v064'),('v065'),('v066'),('v067'),('v068'),('v069'),('v070'),('v071'),
+      ('v072'),('v073'),('v074'),('v075'),('v076'),('v077'),('v078'),('v079'),
+      ('v080'),('v081'),('v082'),('v083'),('v084'),('v085'),('v086'),('v087'),
+      ('v088'),('v089'),('v090'),('v091'),('v092'),('v093'),('v094'),('v095'),
+      ('v096'),('v097'),('v098'),('v099'),('v100'),('v101'),('v102'),('v103'),
+      ('v104'),('v105'),('v106'),('v107'),('v108'),('v109'),('v110'),('v111'),
+      ('v112'),('v113'),('v114'),('v115'),('v116'),('v117'),('v118'),('v119'),
+      ('v120'),('v121'),('v122'),('v123'),('v124'),('v125'),('v126'),('v127'),
+      ('v128'),('v129')
+  ) AS t(v)
+)
+TO 'test_files/scratch/parquet_rle_to_dictionary/high_card/plain.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' false);
+
+statement ok
+COPY (SELECT column1 AS category FROM (VALUES ('rle_a'), ('rle_b'), ('rle_a')))
+TO 'test_files/scratch/parquet_rle_to_dictionary/high_card/rle.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' true);

Review Comment:
   updated the test to cast to a int8 first before overflowing https://github.com/apache/datafusion/pull/24227/commits/6398041e1fcb0536c5a43a2708af58ea8cddd8a8



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5719360116

   > Do we have any query where this option makes the query faster?
   
   @alamb theres 6 queries that tpcds show sped up but these are more than likely within noise. I think theres still a decent chunk of work left to do in aggregations/filter operators to make dictionary arrays faster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5281381819

   > If I understand correctly the goal is to evaluate filters during filter pushdown against dictionary / RLE encoded columns? 
   
   no not exactly. The goal of this PR is to keep RLE parquet columns in their compacted form by materializing them as dictionary arrays instead of regular strings.
   
   > We can't propagate these dynamic type changes to the rest of the query plan / scan. Is that right?
   
   exactly! This is why it needs to be done as early in the plan as possible. we inspect `DFParquetMetadata::fetch_schema` to see if any RLE columns exist, if so change the plan type from `utf8/binary` to `dict<_,utf8/binary>`. Thanks to some plumbing in arrow-rs it will handle the conversions correctly returning a dictionary array.
   
   [this can make reads 60x faster](https://github.com/apache/arrow-rs/pull/1180)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3878078785


##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -438,6 +568,25 @@ pub fn transform_schema_to_view(schema: &Schema) -> Schema {
             DataType::Binary | DataType::LargeBinary => {
                 field_with_new_type(field, DataType::BinaryView)
             }
+            // Also rewrite the value type inside dictionary columns so that
+            // dict<_, Utf8> / dict<_, LargeUtf8> become dict<_, Utf8View> and
+            // dict<_, Binary> / dict<_, LargeBinary> become dict<_, BinaryView>.
+            DataType::Dictionary(key_type, value_type) => {

Review Comment:
   This arm runs whenever `schema_force_view_types` is on, which is the default, and it does not look at `enable_rle_to_dictionary`. With the flag off, `DESCRIBE dict_hash_10` in
     `dictionary.slt` returns `Dictionary(Int32, Utf8View)` instead of `Dictionary(Int32, Utf8)`. The same change makes `test_statistics_from_parquet_metadata_dictionary` return `Absent` for
     `null_count`, so the column loses its statistics and its row group pruning. `transform_binary_to_string` gained the same unconditional arm. Would it work to drop both arms and keep the
     view rewrite out of this PR?



##########
datafusion/datasource-parquet/src/file_format.rs:
##########
@@ -402,7 +426,12 @@ impl FileFormat for ParquetFormat {
         }
         drop(seen);
 
-        let schemas = schemas.into_iter().map(|(_, schema)| schema);
+        // Normalize dict-promoted schemas before merging so mixed dict/plain files merge cleanly.
+        let mut schemas: Vec<Schema> =
+            schemas.into_iter().map(|(_, schema)| schema).collect();

Review Comment:
     This gate reads only the global flag, but `fetch_schema` promotes when either the flag or the allowlist is set. So an allowlist with the flag off promotes per file and then skips this  normalization. A directory that mixes dictionary and plain files then fails `Schema::try_merge` at registration. Would it work to use the same condition here that `fetch_schema` uses?



##########
datafusion/datasource-parquet/src/file_format.rs:
##########
@@ -210,6 +213,22 @@ impl ParquetFormat {
         &self.options
     }
 
+    /// Restrict RLE->Dictionary promotion to a named subset of columns.
+    ///
+    /// Only columns in `columns` that also have dictionary pages in the file
+    /// are promoted to `Dictionary(Int32, …)` in the inferred schema.
+    /// Overrides `enable_rle_to_dictionary` when set; an empty set disables
+    /// promotion entirely. See <https://github.com/apache/datafusion/issues/24113>.
+    pub fn with_rle_column_allowlist(mut self, columns: HashSet<String>) -> Self {

Review Comment:
   Nothing in the tree calls `with_rle_column_allowlist` or `rle_column_allowlist`, and no test covers the allowlist path. New public API on `ParquetFormat` is hard to remove later, and the shape of #24113 can still change. I'd leave the allowlist out of this PR and add it with the per-column work that needs it.



##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -135,6 +157,114 @@ pub fn apply_file_schema_type_coercions(
     ))
 }
 
+fn dictionary_value_type(data_type: &DataType) -> Option<&DataType> {
+    match data_type {
+        DataType::Dictionary(_, value_type) => Some(value_type.as_ref()),
+        _ => None,
+    }
+}
+
+// Find the value type that can represent both sides without narrowing offsets
+// or crossing string/binary families.
+fn common_dictionary_value_type(

Review Comment:
   This helper has no arm for `Utf8View` or `BinaryView`, so `can_promote_to_dictionary_type` returns false for every view value type. With `schema_force_view_types` on, inference produces
     `Dictionary(Int32, Utf8View)`, and the opener then declines to coerce the file column. Am I right that the dictionary array then comes from a later cast, not from arrow-rs?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5345832494

   run benchmark tpch10
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5285122363

   ideally we surface columns that are physically RLE_DICTIONARY-encoded in the parquet file as Arrow `Dictionary(Int32, Utf8)` arrays rather than decoding them back to plain `Utf8`.
   
   To know whether a specific column is RLE_DICTIONARY-encoded you need to read the parquet file footer. For the `infer_schema` path this happens at table registration, but for explicit schemas (`CREATE EXTERNAL TABLE (col VARCHAR)`) and direct `ParquetSource` construction no footer is ever read during planning, so per-column encoding information isn't available for all paths.
   
   Downstream physical operators (`FilterExec`, `AggregateExec`) are compiled against the scan's declared output schema during physical planning, before any files are opened. If the scan declares `Utf8` but produces `Dictionary(Int32, Utf8)` at execution time that's a type mismatch.
   
   So when the flag is enabled we promote all string/binary columns to dict at planning time, not just the ones that are actually RLE-encoded, because that's the only way to guarantee schema consistency across all parquet scan paths without introducing file I/O into the planning stage.
   
   I feel like i'm missing something here. if we could take a peak at the parquets metadata before physical planning and change the schema for all operators from the point forward that would be perfect. Im not sure this is currently possible


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513885045

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513845153)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5513845153-2095-9nv4b 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (6166c31c21bdd7965f348ae7a126a2d1e85a6216) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..6166c31c21bdd7965f348ae7a126a2d1e85a6216)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885

   run benchmark tpch tpcds clickbench_partitioned wide_schema
   ```yaml
   env:
     DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
   baseline:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
   changed:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5551465444

   > im a little confused by what you mean by this
   
   ParquetOptions is a public struct and is not #[non_exhaustive], so downstream crates can construct it with an exhaustive ParquetOptions { ... } literal. After this PR adds enable_rle_to_dictionary, those existing literals fail to compile with E0063: missing field enable_rle_to_dictionary.
   
   The docs/source/library-user-guide/upgrading/56.0.0.md should mention that affected callers must add enable_rle_to_dictionary: false or migrate to constructing from ParquetOptions::default().


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5345661768

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5345449022)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (449355e7c5098104ce4868fa3ce347ca95a24ce3) to bbf739a (merge-base) [diff](https://github.com/apache/datafusion/compare/bbf739a3cfe1d5cc7646e37250c947c6a56b5b0e..449355e7c5098104ce4868fa3ce347ca95a24ce3)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    5.69 ms │                               5.80 ms │     no change │
   │ QQuery 2  │   81.33 ms │                              83.54 ms │     no change │
   │ QQuery 3  │   28.99 ms │                              29.19 ms │     no change │
   │ QQuery 4  │  493.09 ms │                             668.91 ms │  1.36x slower │
   │ QQuery 5  │   52.50 ms │                              53.41 ms │     no change │
   │ QQuery 6  │   36.20 ms │                              35.04 ms │     no change │
   │ QQuery 7  │   94.90 ms │                              96.05 ms │     no change │
   │ QQuery 8  │   37.61 ms │                              37.76 ms │     no change │
   │ QQuery 9  │   52.90 ms │                              50.03 ms │ +1.06x faster │
   │ QQuery 10 │   63.70 ms │                              64.42 ms │     no change │
   │ QQuery 11 │  302.15 ms │                             435.47 ms │  1.44x slower │
   │ QQuery 12 │   29.59 ms │                              28.68 ms │     no change │
   │ QQuery 13 │  118.35 ms │                             124.50 ms │  1.05x slower │
   │ QQuery 14 │  421.16 ms │                             419.58 ms │     no change │
   │ QQuery 15 │   57.55 ms │                              66.72 ms │  1.16x slower │
   │ QQuery 16 │    6.82 ms │                               6.96 ms │     no change │
   │ QQuery 17 │   79.33 ms │                              81.52 ms │     no change │
   │ QQuery 18 │  124.06 ms │                             125.91 ms │     no change │
   │ QQuery 19 │   41.71 ms │                              42.01 ms │     no change │
   │ QQuery 20 │   36.41 ms │                              32.02 ms │ +1.14x faster │
   │ QQuery 21 │   17.68 ms │                              17.91 ms │     no change │
   │ QQuery 22 │   63.00 ms │                              73.22 ms │  1.16x slower │
   │ QQuery 23 │  346.85 ms │                             345.19 ms │     no change │
   │ QQuery 24 │  224.09 ms │                             225.42 ms │     no change │
   │ QQuery 25 │  109.10 ms │                             109.59 ms │     no change │
   │ QQuery 26 │   58.42 ms │                              59.36 ms │     no change │
   │ QQuery 27 │    6.36 ms │                               6.72 ms │  1.06x slower │
   │ QQuery 28 │   61.05 ms │                              60.58 ms │     no change │
   │ QQuery 29 │   96.99 ms │                              98.17 ms │     no change │
   │ QQuery 30 │   32.79 ms │                              39.48 ms │  1.20x slower │
   │ QQuery 31 │  111.26 ms │                             117.45 ms │  1.06x slower │
   │ QQuery 32 │   20.12 ms │                              20.43 ms │     no change │
   │ QQuery 33 │   38.14 ms │                              38.13 ms │     no change │
   │ QQuery 34 │    9.99 ms │                               9.92 ms │     no change │
   │ QQuery 35 │   73.26 ms │                              73.45 ms │     no change │
   │ QQuery 36 │    5.85 ms │                               6.18 ms │  1.06x slower │
   │ QQuery 37 │    6.85 ms │                               7.81 ms │  1.14x slower │
   │ QQuery 38 │   62.73 ms │                              78.64 ms │  1.25x slower │
   │ QQuery 39 │   90.65 ms │                              90.70 ms │     no change │
   │ QQuery 40 │   23.17 ms │                              23.82 ms │     no change │
   │ QQuery 41 │   11.39 ms │                              12.54 ms │  1.10x slower │
   │ QQuery 42 │   24.10 ms │                              23.93 ms │     no change │
   │ QQuery 43 │    5.19 ms │                               5.40 ms │     no change │
   │ QQuery 44 │    9.50 ms │                               9.54 ms │     no change │
   │ QQuery 45 │   38.50 ms │                              37.87 ms │     no change │
   │ QQuery 46 │   12.35 ms │                              11.77 ms │     no change │
   │ QQuery 47 │  229.24 ms │                             211.22 ms │ +1.09x faster │
   │ QQuery 48 │   95.82 ms │                              98.01 ms │     no change │
   │ QQuery 49 │   77.94 ms │                              76.91 ms │     no change │
   │ QQuery 50 │   59.67 ms │                              59.74 ms │     no change │
   │ QQuery 51 │   91.55 ms │                              93.15 ms │     no change │
   │ QQuery 52 │   23.89 ms │                              24.48 ms │     no change │
   │ QQuery 53 │   29.23 ms │                              29.68 ms │     no change │
   │ QQuery 54 │   55.02 ms │                              55.24 ms │     no change │
   │ QQuery 55 │   23.74 ms │                              23.79 ms │     no change │
   │ QQuery 56 │   39.41 ms │                              39.50 ms │     no change │
   │ QQuery 57 │  177.14 ms │                             164.35 ms │ +1.08x faster │
   │ QQuery 58 │  112.76 ms │                             110.45 ms │     no change │
   │ QQuery 59 │  117.71 ms │                             115.81 ms │     no change │
   │ QQuery 60 │   39.29 ms │                              40.37 ms │     no change │
   │ QQuery 61 │   12.60 ms │                              12.49 ms │     no change │
   │ QQuery 62 │   47.28 ms │                              46.76 ms │     no change │
   │ QQuery 63 │   29.45 ms │                              29.52 ms │     no change │
   │ QQuery 64 │  410.28 ms │                             419.10 ms │     no change │
   │ QQuery 65 │  122.89 ms │                             124.23 ms │     no change │
   │ QQuery 66 │   81.92 ms │                              81.38 ms │     no change │
   │ QQuery 67 │  247.05 ms │                             248.38 ms │     no change │
   │ QQuery 68 │   12.05 ms │                              11.98 ms │     no change │
   │ QQuery 69 │   57.87 ms │                              58.78 ms │     no change │
   │ QQuery 70 │  109.29 ms │                             104.49 ms │     no change │
   │ QQuery 71 │   35.70 ms │                              35.77 ms │     no change │
   │ QQuery 72 │ 1961.07 ms │                            1766.17 ms │ +1.11x faster │
   │ QQuery 73 │   10.04 ms │                               9.58 ms │     no change │
   │ QQuery 74 │  173.17 ms │                             234.33 ms │  1.35x slower │
   │ QQuery 75 │  148.29 ms │                             150.06 ms │     no change │
   │ QQuery 76 │   35.78 ms │                              36.33 ms │     no change │
   │ QQuery 77 │   61.96 ms │                              62.19 ms │     no change │
   │ QQuery 78 │  197.46 ms │                             198.86 ms │     no change │
   │ QQuery 79 │   66.84 ms │                              67.65 ms │     no change │
   │ QQuery 80 │   99.04 ms │                             101.52 ms │     no change │
   │ QQuery 81 │   25.95 ms │                              29.98 ms │  1.16x slower │
   │ QQuery 82 │   16.25 ms │                              17.53 ms │  1.08x slower │
   │ QQuery 83 │   40.36 ms │                              39.81 ms │     no change │
   │ QQuery 84 │   30.22 ms │                              32.61 ms │  1.08x slower │
   │ QQuery 85 │  106.34 ms │                             108.90 ms │     no change │
   │ QQuery 86 │   25.34 ms │                              24.95 ms │     no change │
   │ QQuery 87 │   63.02 ms │                              79.17 ms │  1.26x slower │
   │ QQuery 88 │   64.40 ms │                              64.56 ms │     no change │
   │ QQuery 89 │   35.37 ms │                              36.35 ms │     no change │
   │ QQuery 90 │   17.33 ms │                              17.16 ms │     no change │
   │ QQuery 91 │   46.60 ms │                              47.00 ms │     no change │
   │ QQuery 92 │   29.78 ms │                              29.11 ms │     no change │
   │ QQuery 93 │   49.90 ms │                              49.73 ms │     no change │
   │ QQuery 94 │   37.83 ms │                              38.01 ms │     no change │
   │ QQuery 95 │   80.89 ms │                              79.99 ms │     no change │
   │ QQuery 96 │   24.23 ms │                              24.42 ms │     no change │
   │ QQuery 97 │   47.40 ms │                              46.92 ms │     no change │
   │ QQuery 98 │   42.98 ms │                              38.67 ms │ +1.11x faster │
   │ QQuery 99 │   70.93 ms │                              70.12 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 9670.92ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 9908.01ms │
   │ Average Time (HEAD)                                  │   97.69ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  100.08ms │
   │ Queries Faster                                       │         6 │
   │ Queries Slower                                       │        17 │
   │ Queries with No Change                               │        76 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │           5.69 / 6.21 ±0.91 / 8.03 ms │           5.80 / 6.28 ±0.91 / 8.10 ms │     no change │
   │ QQuery 2  │        81.33 / 81.72 ±0.34 / 82.19 ms │        83.54 / 83.83 ±0.16 / 83.99 ms │     no change │
   │ QQuery 3  │        28.99 / 29.42 ±0.24 / 29.63 ms │        29.19 / 29.40 ±0.12 / 29.57 ms │     no change │
   │ QQuery 4  │     493.09 / 494.80 ±1.60 / 497.80 ms │     668.91 / 676.80 ±5.30 / 683.07 ms │  1.37x slower │
   │ QQuery 5  │        52.50 / 53.09 ±0.47 / 53.95 ms │        53.41 / 55.20 ±2.71 / 60.56 ms │     no change │
   │ QQuery 6  │        36.20 / 36.53 ±0.23 / 36.85 ms │        35.04 / 35.44 ±0.22 / 35.69 ms │     no change │
   │ QQuery 7  │        94.90 / 95.17 ±0.24 / 95.59 ms │        96.05 / 96.68 ±0.63 / 97.73 ms │     no change │
   │ QQuery 8  │        37.61 / 38.99 ±1.93 / 42.79 ms │        37.76 / 39.14 ±2.14 / 43.40 ms │     no change │
   │ QQuery 9  │        52.90 / 54.95 ±1.15 / 56.20 ms │        50.03 / 52.74 ±1.52 / 54.38 ms │     no change │
   │ QQuery 10 │        63.70 / 63.97 ±0.23 / 64.36 ms │        64.42 / 64.85 ±0.33 / 65.18 ms │     no change │
   │ QQuery 11 │     302.15 / 309.40 ±5.52 / 318.07 ms │    435.47 / 460.68 ±27.36 / 511.32 ms │  1.49x slower │
   │ QQuery 12 │        29.59 / 29.70 ±0.09 / 29.83 ms │        28.68 / 30.64 ±3.54 / 37.72 ms │     no change │
   │ QQuery 13 │     118.35 / 120.64 ±2.02 / 123.71 ms │     124.50 / 126.61 ±1.52 / 129.09 ms │     no change │
   │ QQuery 14 │     421.16 / 423.97 ±2.35 / 427.77 ms │     419.58 / 425.92 ±9.27 / 444.21 ms │     no change │
   │ QQuery 15 │        57.55 / 61.16 ±2.16 / 63.05 ms │        66.72 / 68.29 ±2.44 / 73.15 ms │  1.12x slower │
   │ QQuery 16 │           6.82 / 7.04 ±0.18 / 7.28 ms │           6.96 / 7.15 ±0.23 / 7.58 ms │     no change │
   │ QQuery 17 │        79.33 / 81.07 ±1.42 / 83.56 ms │        81.52 / 83.08 ±2.07 / 87.16 ms │     no change │
   │ QQuery 18 │     124.06 / 125.27 ±1.09 / 127.14 ms │     125.91 / 126.53 ±0.72 / 127.87 ms │     no change │
   │ QQuery 19 │        41.71 / 42.38 ±0.78 / 43.85 ms │        42.01 / 42.54 ±0.45 / 43.34 ms │     no change │
   │ QQuery 20 │        36.41 / 37.14 ±0.82 / 38.30 ms │        32.02 / 32.20 ±0.16 / 32.49 ms │ +1.15x faster │
   │ QQuery 21 │        17.68 / 17.76 ±0.07 / 17.86 ms │        17.91 / 18.60 ±0.83 / 20.18 ms │     no change │
   │ QQuery 22 │        63.00 / 63.79 ±0.69 / 65.05 ms │        73.22 / 74.22 ±1.12 / 76.37 ms │  1.16x slower │
   │ QQuery 23 │     346.85 / 349.07 ±2.16 / 352.86 ms │     345.19 / 347.29 ±2.00 / 350.43 ms │     no change │
   │ QQuery 24 │     224.09 / 226.34 ±1.77 / 229.48 ms │     225.42 / 227.89 ±2.85 / 233.20 ms │     no change │
   │ QQuery 25 │     109.10 / 110.96 ±1.49 / 112.92 ms │     109.59 / 111.84 ±2.84 / 117.14 ms │     no change │
   │ QQuery 26 │        58.42 / 59.72 ±1.44 / 62.42 ms │        59.36 / 60.84 ±1.34 / 63.37 ms │     no change │
   │ QQuery 27 │           6.36 / 6.49 ±0.17 / 6.83 ms │           6.72 / 6.88 ±0.16 / 7.18 ms │  1.06x slower │
   │ QQuery 28 │        61.05 / 61.53 ±0.30 / 61.93 ms │        60.58 / 62.53 ±1.96 / 66.26 ms │     no change │
   │ QQuery 29 │        96.99 / 97.94 ±0.91 / 99.32 ms │       98.17 / 99.65 ±1.72 / 102.90 ms │     no change │
   │ QQuery 30 │        32.79 / 33.51 ±1.18 / 35.87 ms │        39.48 / 40.46 ±1.27 / 42.96 ms │  1.21x slower │
   │ QQuery 31 │     111.26 / 112.25 ±0.52 / 112.73 ms │     117.45 / 118.65 ±1.21 / 120.73 ms │  1.06x slower │
   │ QQuery 32 │        20.12 / 20.81 ±0.41 / 21.40 ms │        20.43 / 20.82 ±0.29 / 21.16 ms │     no change │
   │ QQuery 33 │        38.14 / 38.59 ±0.46 / 39.33 ms │        38.13 / 38.99 ±1.22 / 41.39 ms │     no change │
   │ QQuery 34 │         9.99 / 10.40 ±0.35 / 10.95 ms │         9.92 / 10.08 ±0.19 / 10.43 ms │     no change │
   │ QQuery 35 │        73.26 / 74.56 ±1.27 / 76.58 ms │        73.45 / 74.08 ±0.36 / 74.57 ms │     no change │
   │ QQuery 36 │           5.85 / 6.01 ±0.26 / 6.53 ms │           6.18 / 6.37 ±0.20 / 6.76 ms │  1.06x slower │
   │ QQuery 37 │           6.85 / 6.93 ±0.06 / 7.01 ms │           7.81 / 7.88 ±0.06 / 7.97 ms │  1.14x slower │
   │ QQuery 38 │        62.73 / 63.72 ±0.74 / 64.63 ms │        78.64 / 79.10 ±0.47 / 79.96 ms │  1.24x slower │
   │ QQuery 39 │        90.65 / 93.69 ±2.97 / 98.76 ms │        90.70 / 92.23 ±1.13 / 94.08 ms │     no change │
   │ QQuery 40 │        23.17 / 23.70 ±0.37 / 24.20 ms │        23.82 / 24.01 ±0.18 / 24.35 ms │     no change │
   │ QQuery 41 │        11.39 / 11.65 ±0.20 / 11.94 ms │        12.54 / 12.77 ±0.17 / 13.05 ms │  1.10x slower │
   │ QQuery 42 │        24.10 / 24.37 ±0.32 / 24.97 ms │        23.93 / 24.42 ±0.36 / 25.02 ms │     no change │
   │ QQuery 43 │           5.19 / 5.30 ±0.16 / 5.62 ms │           5.40 / 5.51 ±0.14 / 5.79 ms │     no change │
   │ QQuery 44 │           9.50 / 9.61 ±0.08 / 9.74 ms │           9.54 / 9.67 ±0.09 / 9.79 ms │     no change │
   │ QQuery 45 │        38.50 / 40.26 ±1.92 / 43.96 ms │        37.87 / 38.09 ±0.15 / 38.26 ms │ +1.06x faster │
   │ QQuery 46 │        12.35 / 12.72 ±0.32 / 13.15 ms │        11.77 / 12.10 ±0.21 / 12.29 ms │     no change │
   │ QQuery 47 │     229.24 / 232.30 ±3.31 / 238.54 ms │     211.22 / 216.22 ±3.13 / 219.37 ms │ +1.07x faster │
   │ QQuery 48 │        95.82 / 97.22 ±1.41 / 99.90 ms │       98.01 / 98.92 ±0.71 / 100.06 ms │     no change │
   │ QQuery 49 │        77.94 / 79.48 ±1.84 / 82.99 ms │        76.91 / 77.31 ±0.27 / 77.75 ms │     no change │
   │ QQuery 50 │        59.67 / 60.83 ±1.62 / 64.02 ms │        59.74 / 61.30 ±1.68 / 63.90 ms │     no change │
   │ QQuery 51 │        91.55 / 93.06 ±1.27 / 95.39 ms │        93.15 / 94.29 ±1.01 / 96.00 ms │     no change │
   │ QQuery 52 │        23.89 / 24.36 ±0.24 / 24.54 ms │        24.48 / 24.72 ±0.19 / 25.04 ms │     no change │
   │ QQuery 53 │        29.23 / 29.61 ±0.35 / 30.15 ms │        29.68 / 29.84 ±0.10 / 29.99 ms │     no change │
   │ QQuery 54 │        55.02 / 56.28 ±1.27 / 58.73 ms │        55.24 / 56.34 ±1.18 / 58.13 ms │     no change │
   │ QQuery 55 │        23.74 / 24.07 ±0.35 / 24.68 ms │        23.79 / 23.99 ±0.18 / 24.32 ms │     no change │
   │ QQuery 56 │        39.41 / 40.01 ±0.36 / 40.48 ms │        39.50 / 39.79 ±0.20 / 40.02 ms │     no change │
   │ QQuery 57 │     177.14 / 178.94 ±1.38 / 180.95 ms │     164.35 / 166.37 ±1.30 / 167.62 ms │ +1.08x faster │
   │ QQuery 58 │     112.76 / 114.38 ±1.49 / 116.89 ms │     110.45 / 111.92 ±1.53 / 114.85 ms │     no change │
   │ QQuery 59 │     117.71 / 118.72 ±1.41 / 121.52 ms │     115.81 / 116.37 ±0.68 / 117.66 ms │     no change │
   │ QQuery 60 │        39.29 / 40.14 ±0.47 / 40.57 ms │        40.37 / 40.85 ±0.46 / 41.60 ms │     no change │
   │ QQuery 61 │        12.60 / 13.31 ±1.17 / 15.63 ms │        12.49 / 12.71 ±0.16 / 12.98 ms │     no change │
   │ QQuery 62 │        47.28 / 48.17 ±1.00 / 50.06 ms │        46.76 / 47.86 ±1.67 / 51.19 ms │     no change │
   │ QQuery 63 │        29.45 / 29.93 ±0.46 / 30.70 ms │        29.52 / 29.76 ±0.14 / 29.91 ms │     no change │
   │ QQuery 64 │     410.28 / 413.79 ±3.33 / 418.88 ms │     419.10 / 424.11 ±4.25 / 431.41 ms │     no change │
   │ QQuery 65 │     122.89 / 126.72 ±5.05 / 136.64 ms │     124.23 / 126.70 ±1.73 / 129.61 ms │     no change │
   │ QQuery 66 │        81.92 / 83.56 ±1.88 / 87.20 ms │        81.38 / 82.60 ±2.29 / 87.18 ms │     no change │
   │ QQuery 67 │     247.05 / 249.82 ±2.55 / 253.22 ms │     248.38 / 250.02 ±0.88 / 250.67 ms │     no change │
   │ QQuery 68 │        12.05 / 12.52 ±0.25 / 12.83 ms │        11.98 / 12.15 ±0.24 / 12.62 ms │     no change │
   │ QQuery 69 │        57.87 / 61.25 ±5.43 / 72.07 ms │        58.78 / 59.00 ±0.29 / 59.58 ms │     no change │
   │ QQuery 70 │     109.29 / 111.50 ±2.36 / 115.17 ms │     104.49 / 106.93 ±2.13 / 110.57 ms │     no change │
   │ QQuery 71 │        35.70 / 35.96 ±0.16 / 36.14 ms │        35.77 / 36.11 ±0.33 / 36.69 ms │     no change │
   │ QQuery 72 │ 1961.07 / 2039.07 ±62.78 / 2150.82 ms │ 1766.17 / 1888.08 ±85.53 / 2006.14 ms │ +1.08x faster │
   │ QQuery 73 │        10.04 / 10.18 ±0.13 / 10.43 ms │          9.58 / 9.78 ±0.19 / 10.09 ms │     no change │
   │ QQuery 74 │     173.17 / 175.77 ±1.86 / 178.65 ms │     234.33 / 240.53 ±3.85 / 245.15 ms │  1.37x slower │
   │ QQuery 75 │     148.29 / 152.39 ±3.40 / 157.85 ms │     150.06 / 152.77 ±3.83 / 160.25 ms │     no change │
   │ QQuery 76 │        35.78 / 35.91 ±0.10 / 36.03 ms │        36.33 / 36.66 ±0.19 / 36.84 ms │     no change │
   │ QQuery 77 │        61.96 / 63.53 ±1.78 / 66.86 ms │        62.19 / 62.61 ±0.25 / 62.89 ms │     no change │
   │ QQuery 78 │     197.46 / 200.28 ±3.03 / 205.74 ms │     198.86 / 202.06 ±2.40 / 205.98 ms │     no change │
   │ QQuery 79 │        66.84 / 68.19 ±1.58 / 71.21 ms │        67.65 / 68.31 ±0.44 / 68.83 ms │     no change │
   │ QQuery 80 │      99.04 / 100.90 ±1.13 / 102.25 ms │     101.52 / 104.03 ±2.46 / 108.47 ms │     no change │
   │ QQuery 81 │        25.95 / 26.37 ±0.40 / 27.08 ms │        29.98 / 30.65 ±0.38 / 31.05 ms │  1.16x slower │
   │ QQuery 82 │        16.25 / 18.70 ±2.27 / 22.16 ms │        17.53 / 17.70 ±0.18 / 17.95 ms │ +1.06x faster │
   │ QQuery 83 │        40.36 / 40.91 ±0.42 / 41.63 ms │        39.81 / 40.46 ±0.44 / 41.02 ms │     no change │
   │ QQuery 84 │        30.22 / 30.33 ±0.11 / 30.49 ms │        32.61 / 33.78 ±1.89 / 37.54 ms │  1.11x slower │
   │ QQuery 85 │     106.34 / 108.61 ±2.09 / 112.26 ms │     108.90 / 110.51 ±1.32 / 112.92 ms │     no change │
   │ QQuery 86 │        25.34 / 27.29 ±1.95 / 30.77 ms │        24.95 / 25.38 ±0.38 / 25.85 ms │ +1.08x faster │
   │ QQuery 87 │        63.02 / 63.53 ±0.35 / 63.95 ms │        79.17 / 80.23 ±1.35 / 82.88 ms │  1.26x slower │
   │ QQuery 88 │        64.40 / 64.60 ±0.23 / 64.98 ms │        64.56 / 65.06 ±0.48 / 65.95 ms │     no change │
   │ QQuery 89 │        35.37 / 36.08 ±0.48 / 36.65 ms │        36.35 / 36.98 ±0.46 / 37.57 ms │     no change │
   │ QQuery 90 │        17.33 / 17.82 ±0.64 / 19.06 ms │        17.16 / 17.58 ±0.35 / 18.08 ms │     no change │
   │ QQuery 91 │        46.60 / 47.27 ±0.59 / 48.31 ms │        47.00 / 47.57 ±0.59 / 48.70 ms │     no change │
   │ QQuery 92 │        29.78 / 30.00 ±0.17 / 30.15 ms │        29.11 / 31.12 ±1.91 / 34.59 ms │     no change │
   │ QQuery 93 │        49.90 / 50.46 ±0.66 / 51.59 ms │        49.73 / 51.32 ±1.14 / 52.85 ms │     no change │
   │ QQuery 94 │        37.83 / 38.15 ±0.33 / 38.57 ms │        38.01 / 38.48 ±0.40 / 38.96 ms │     no change │
   │ QQuery 95 │        80.89 / 82.67 ±1.12 / 84.31 ms │        79.99 / 80.58 ±0.59 / 81.40 ms │     no change │
   │ QQuery 96 │        24.23 / 24.52 ±0.22 / 24.91 ms │        24.42 / 25.53 ±1.83 / 29.17 ms │     no change │
   │ QQuery 97 │        47.40 / 47.72 ±0.34 / 48.37 ms │        46.92 / 47.82 ±0.82 / 49.12 ms │     no change │
   │ QQuery 98 │        42.98 / 44.04 ±0.72 / 45.22 ms │        38.67 / 39.20 ±0.40 / 39.90 ms │ +1.12x faster │
   │ QQuery 99 │        70.93 / 71.94 ±1.87 / 75.67 ms │        70.12 / 70.29 ±0.12 / 70.43 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │  9867.48ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 10170.24ms │
   │ Average Time (HEAD)                                  │    99.67ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   102.73ms │
   │ Queries Faster                                       │          8 │
   │ Queries Slower                                       │         15 │
   │ Queries with No Change                               │         76 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 2.3 GiB |
   | Avg memory | 1.6 GiB |
   | CPU user | 215.7s |
   | CPU sys | 6.2s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 1.8 GiB |
   | Avg memory | 1.3 GiB |
   | CPU user | 214.4s |
   | CPU sys | 5.9s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5345642658

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5345449022)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (449355e7c5098104ce4868fa3ce347ca95a24ce3) to bbf739a (merge-base) [diff](https://github.com/apache/datafusion/compare/bbf739a3cfe1d5cc7646e37250c947c6a56b5b0e..449355e7c5098104ce4868fa3ce347ca95a24ce3)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.35 ms │                              38.02 ms │    no change │
   │ QQuery 2  │ 19.15 ms │                              21.53 ms │ 1.12x slower │
   │ QQuery 3  │ 30.47 ms │                              30.69 ms │    no change │
   │ QQuery 4  │ 17.27 ms │                              17.45 ms │    no change │
   │ QQuery 5  │ 39.38 ms │                              38.15 ms │    no change │
   │ QQuery 6  │ 15.98 ms │                              16.05 ms │    no change │
   │ QQuery 7  │ 43.84 ms │                              43.78 ms │    no change │
   │ QQuery 8  │ 41.76 ms │                              42.53 ms │    no change │
   │ QQuery 9  │ 48.78 ms │                              49.74 ms │    no change │
   │ QQuery 10 │ 41.91 ms │                              59.94 ms │ 1.43x slower │
   │ QQuery 11 │ 13.39 ms │                              13.41 ms │    no change │
   │ QQuery 12 │ 24.21 ms │                              23.60 ms │    no change │
   │ QQuery 13 │ 33.44 ms │                              42.99 ms │ 1.29x slower │
   │ QQuery 14 │ 23.00 ms │                              23.21 ms │    no change │
   │ QQuery 15 │ 30.57 ms │                              30.64 ms │    no change │
   │ QQuery 16 │ 13.69 ms │                              14.08 ms │    no change │
   │ QQuery 17 │ 69.97 ms │                              69.08 ms │    no change │
   │ QQuery 18 │ 58.82 ms │                             104.11 ms │ 1.77x slower │
   │ QQuery 19 │ 32.89 ms │                              34.39 ms │    no change │
   │ QQuery 20 │ 31.58 ms │                              39.05 ms │ 1.24x slower │
   │ QQuery 21 │ 55.51 ms │                              62.11 ms │ 1.12x slower │
   │ QQuery 22 │ 13.73 ms │                              17.10 ms │ 1.25x slower │
   └───────────┴──────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 737.70ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 831.66ms │
   │ Average Time (HEAD)                                  │  33.53ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  37.80ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        7 │
   │ Queries with No Change                               │       15 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 38.35 / 39.61 ±2.07 / 43.73 ms │        38.02 / 39.09 ±1.20 / 40.87 ms │    no change │
   │ QQuery 2  │ 19.15 / 19.42 ±0.21 / 19.73 ms │        21.53 / 21.97 ±0.28 / 22.35 ms │ 1.13x slower │
   │ QQuery 3  │ 30.47 / 32.14 ±1.23 / 33.54 ms │        30.69 / 32.99 ±1.22 / 34.25 ms │    no change │
   │ QQuery 4  │ 17.27 / 17.66 ±0.36 / 18.35 ms │        17.45 / 17.95 ±0.69 / 19.29 ms │    no change │
   │ QQuery 5  │ 39.38 / 39.59 ±0.20 / 39.96 ms │        38.15 / 40.04 ±0.97 / 40.88 ms │    no change │
   │ QQuery 6  │ 15.98 / 16.47 ±0.56 / 17.56 ms │        16.05 / 16.43 ±0.44 / 17.21 ms │    no change │
   │ QQuery 7  │ 43.84 / 45.22 ±1.04 / 46.99 ms │        43.78 / 45.75 ±1.32 / 47.31 ms │    no change │
   │ QQuery 8  │ 41.76 / 42.33 ±0.36 / 42.74 ms │        42.53 / 42.75 ±0.30 / 43.34 ms │    no change │
   │ QQuery 9  │ 48.78 / 49.71 ±0.70 / 50.57 ms │        49.74 / 50.43 ±0.68 / 51.66 ms │    no change │
   │ QQuery 10 │ 41.91 / 42.27 ±0.22 / 42.52 ms │        59.94 / 60.22 ±0.26 / 60.67 ms │ 1.42x slower │
   │ QQuery 11 │ 13.39 / 13.59 ±0.22 / 13.88 ms │        13.41 / 13.97 ±0.63 / 15.16 ms │    no change │
   │ QQuery 12 │ 24.21 / 24.50 ±0.21 / 24.79 ms │        23.60 / 23.81 ±0.18 / 24.04 ms │    no change │
   │ QQuery 13 │ 33.44 / 34.11 ±0.46 / 34.76 ms │        42.99 / 44.38 ±1.47 / 47.24 ms │ 1.30x slower │
   │ QQuery 14 │ 23.00 / 23.21 ±0.18 / 23.47 ms │        23.21 / 24.05 ±1.31 / 26.66 ms │    no change │
   │ QQuery 15 │ 30.57 / 30.84 ±0.22 / 31.24 ms │        30.64 / 30.81 ±0.13 / 31.03 ms │    no change │
   │ QQuery 16 │ 13.69 / 13.88 ±0.17 / 14.10 ms │        14.08 / 14.32 ±0.37 / 15.05 ms │    no change │
   │ QQuery 17 │ 69.97 / 70.42 ±0.44 / 71.12 ms │        69.08 / 71.41 ±2.73 / 76.73 ms │    no change │
   │ QQuery 18 │ 58.82 / 59.32 ±0.42 / 59.95 ms │     104.11 / 109.70 ±3.27 / 113.71 ms │ 1.85x slower │
   │ QQuery 19 │ 32.89 / 34.08 ±1.39 / 36.30 ms │        34.39 / 35.53 ±0.96 / 37.25 ms │    no change │
   │ QQuery 20 │ 31.58 / 32.38 ±1.03 / 34.28 ms │        39.05 / 39.51 ±0.33 / 40.01 ms │ 1.22x slower │
   │ QQuery 21 │ 55.51 / 56.69 ±1.09 / 58.73 ms │        62.11 / 64.51 ±2.23 / 68.00 ms │ 1.14x slower │
   │ QQuery 22 │ 13.73 / 13.92 ±0.10 / 14.00 ms │        17.10 / 17.38 ±0.19 / 17.66 ms │ 1.25x slower │
   └───────────┴────────────────────────────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 751.36ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 857.01ms │
   │ Average Time (HEAD)                                  │  34.15ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  38.96ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        7 │
   │ Queries with No Change                               │       15 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.2 GiB |
   | Avg memory | 525.5 MiB |
   | CPU user | 21.5s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.0 GiB |
   | Avg memory | 662.3 MiB |
   | CPU user | 21.6s |
   | CPU sys | 1.7s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364843512

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364728282)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f7fe23ed045d3fa5ac3c01cbc8ffc9247d7ec624)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    5.65 ms │                               5.75 ms │     no change │
   │ QQuery 2  │   80.25 ms │                              82.92 ms │     no change │
   │ QQuery 3  │   28.80 ms │                              28.64 ms │     no change │
   │ QQuery 4  │  471.59 ms │                             644.85 ms │  1.37x slower │
   │ QQuery 5  │   51.96 ms │                              52.44 ms │     no change │
   │ QQuery 6  │   35.76 ms │                              34.34 ms │     no change │
   │ QQuery 7  │   93.85 ms │                              94.29 ms │     no change │
   │ QQuery 8  │   36.62 ms │                              37.06 ms │     no change │
   │ QQuery 9  │   52.52 ms │                              50.03 ms │     no change │
   │ QQuery 10 │   62.68 ms │                              63.86 ms │     no change │
   │ QQuery 11 │  289.40 ms │                             425.71 ms │  1.47x slower │
   │ QQuery 12 │   28.24 ms │                              25.83 ms │ +1.09x faster │
   │ QQuery 13 │  116.49 ms │                             118.40 ms │     no change │
   │ QQuery 14 │  415.60 ms │                             412.98 ms │     no change │
   │ QQuery 15 │   66.87 ms │                              64.55 ms │     no change │
   │ QQuery 16 │    7.50 ms │                               6.85 ms │ +1.09x faster │
   │ QQuery 17 │   83.24 ms │                              81.63 ms │     no change │
   │ QQuery 18 │  123.64 ms │                             123.55 ms │     no change │
   │ QQuery 19 │   40.86 ms │                              41.68 ms │     no change │
   │ QQuery 20 │   35.33 ms │                              31.51 ms │ +1.12x faster │
   │ QQuery 21 │   17.14 ms │                              17.01 ms │     no change │
   │ QQuery 22 │   62.86 ms │                              72.20 ms │  1.15x slower │
   │ QQuery 23 │  338.22 ms │                             337.11 ms │     no change │
   │ QQuery 24 │  221.53 ms │                             225.34 ms │     no change │
   │ QQuery 25 │  108.65 ms │                             109.68 ms │     no change │
   │ QQuery 26 │   57.82 ms │                              58.59 ms │     no change │
   │ QQuery 27 │    6.45 ms │                               6.69 ms │     no change │
   │ QQuery 28 │   60.23 ms │                              60.02 ms │     no change │
   │ QQuery 29 │   96.12 ms │                              98.45 ms │     no change │
   │ QQuery 30 │   33.29 ms │                              38.96 ms │  1.17x slower │
   │ QQuery 31 │  109.95 ms │                             114.74 ms │     no change │
   │ QQuery 32 │   20.09 ms │                              19.68 ms │     no change │
   │ QQuery 33 │   37.60 ms │                              37.56 ms │     no change │
   │ QQuery 34 │   10.24 ms │                               9.63 ms │ +1.06x faster │
   │ QQuery 35 │   72.77 ms │                              71.80 ms │     no change │
   │ QQuery 36 │    5.82 ms │                               6.14 ms │  1.05x slower │
   │ QQuery 37 │    6.84 ms │                               7.62 ms │  1.11x slower │
   │ QQuery 38 │   61.39 ms │                              77.39 ms │  1.26x slower │
   │ QQuery 39 │   89.26 ms │                              90.14 ms │     no change │
   │ QQuery 40 │   23.29 ms │                              23.35 ms │     no change │
   │ QQuery 41 │   11.31 ms │                              12.56 ms │  1.11x slower │
   │ QQuery 42 │   23.54 ms │                              23.67 ms │     no change │
   │ QQuery 43 │    5.47 ms │                               5.71 ms │     no change │
   │ QQuery 44 │    9.64 ms │                               9.50 ms │     no change │
   │ QQuery 45 │   37.38 ms │                              36.77 ms │     no change │
   │ QQuery 46 │   11.83 ms │                              11.77 ms │     no change │
   │ QQuery 47 │  223.04 ms │                             210.18 ms │ +1.06x faster │
   │ QQuery 48 │   95.23 ms │                              96.48 ms │     no change │
   │ QQuery 49 │   75.46 ms │                              76.32 ms │     no change │
   │ QQuery 50 │   58.60 ms │                              58.77 ms │     no change │
   │ QQuery 51 │   91.66 ms │                              90.84 ms │     no change │
   │ QQuery 52 │   23.82 ms │                              23.73 ms │     no change │
   │ QQuery 53 │   28.65 ms │                              29.12 ms │     no change │
   │ QQuery 54 │   53.43 ms │                              54.72 ms │     no change │
   │ QQuery 55 │   23.30 ms │                              23.11 ms │     no change │
   │ QQuery 56 │   38.32 ms │                              38.96 ms │     no change │
   │ QQuery 57 │  174.56 ms │                             163.12 ms │ +1.07x faster │
   │ QQuery 58 │  111.04 ms │                             108.07 ms │     no change │
   │ QQuery 59 │  117.19 ms │                             114.85 ms │     no change │
   │ QQuery 60 │   38.96 ms │                              39.84 ms │     no change │
   │ QQuery 61 │   12.33 ms │                              12.26 ms │     no change │
   │ QQuery 62 │   47.94 ms │                              45.74 ms │     no change │
   │ QQuery 63 │   31.27 ms │                              28.96 ms │ +1.08x faster │
   │ QQuery 64 │  437.60 ms │                             414.79 ms │ +1.05x faster │
   │ QQuery 65 │  128.87 ms │                             124.25 ms │     no change │
   │ QQuery 66 │   87.70 ms │                              81.19 ms │ +1.08x faster │
   │ QQuery 67 │  279.87 ms │                             289.35 ms │     no change │
   │ QQuery 68 │   13.18 ms │                              13.26 ms │     no change │
   │ QQuery 69 │   60.90 ms │                              61.25 ms │     no change │
   │ QQuery 70 │  113.41 ms │                             111.28 ms │     no change │
   │ QQuery 71 │   37.48 ms │                              38.29 ms │     no change │
   │ QQuery 72 │ 2075.99 ms │                            1932.52 ms │ +1.07x faster │
   │ QQuery 73 │    9.61 ms │                               9.48 ms │     no change │
   │ QQuery 74 │  166.49 ms │                             233.01 ms │  1.40x slower │
   │ QQuery 75 │  147.18 ms │                             148.66 ms │     no change │
   │ QQuery 76 │   36.42 ms │                              37.16 ms │     no change │
   │ QQuery 77 │   61.02 ms │                              64.39 ms │  1.06x slower │
   │ QQuery 78 │  196.17 ms │                             195.62 ms │     no change │
   │ QQuery 79 │   66.17 ms │                              69.52 ms │  1.05x slower │
   │ QQuery 80 │   98.10 ms │                              98.77 ms │     no change │
   │ QQuery 81 │   25.59 ms │                              29.70 ms │  1.16x slower │
   │ QQuery 82 │   16.05 ms │                              17.06 ms │  1.06x slower │
   │ QQuery 83 │   39.33 ms │                              38.88 ms │     no change │
   │ QQuery 84 │   29.98 ms │                              32.46 ms │  1.08x slower │
   │ QQuery 85 │  105.73 ms │                             108.25 ms │     no change │
   │ QQuery 86 │   25.01 ms │                              25.87 ms │     no change │
   │ QQuery 87 │   61.91 ms │                              78.19 ms │  1.26x slower │
   │ QQuery 88 │   62.23 ms │                              63.81 ms │     no change │
   │ QQuery 89 │   34.96 ms │                              36.45 ms │     no change │
   │ QQuery 90 │   16.84 ms │                              17.33 ms │     no change │
   │ QQuery 91 │   45.27 ms │                              46.08 ms │     no change │
   │ QQuery 92 │   28.88 ms │                              28.68 ms │     no change │
   │ QQuery 93 │   50.45 ms │                              49.14 ms │     no change │
   │ QQuery 94 │   38.54 ms │                              37.50 ms │     no change │
   │ QQuery 95 │   79.00 ms │                              79.62 ms │     no change │
   │ QQuery 96 │   23.88 ms │                              23.96 ms │     no change │
   │ QQuery 97 │   46.61 ms │                              46.70 ms │     no change │
   │ QQuery 98 │   42.42 ms │                              38.66 ms │ +1.10x faster │
   │ QQuery 99 │   69.41 ms │                              68.90 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │  9768.62ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 10014.00ms │
   │ Average Time (HEAD)                                  │    98.67ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   101.15ms │
   │ Queries Faster                                       │         11 │
   │ Queries Slower                                       │         15 │
   │ Queries with No Change                               │         73 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃  rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │           5.65 / 6.14 ±0.89 / 7.92 ms │            5.75 / 6.29 ±0.89 / 8.06 ms │     no change │
   │ QQuery 2  │        80.25 / 80.74 ±0.32 / 81.18 ms │         82.92 / 83.22 ±0.30 / 83.71 ms │     no change │
   │ QQuery 3  │        28.80 / 29.00 ±0.17 / 29.31 ms │         28.64 / 28.92 ±0.23 / 29.30 ms │     no change │
   │ QQuery 4  │     471.59 / 478.96 ±5.27 / 485.67 ms │      644.85 / 653.69 ±7.73 / 664.57 ms │  1.36x slower │
   │ QQuery 5  │        51.96 / 52.46 ±0.40 / 53.13 ms │         52.44 / 52.68 ±0.15 / 52.86 ms │     no change │
   │ QQuery 6  │        35.76 / 36.09 ±0.34 / 36.73 ms │         34.34 / 34.64 ±0.24 / 34.96 ms │     no change │
   │ QQuery 7  │        93.85 / 94.52 ±0.61 / 95.52 ms │         94.29 / 95.08 ±0.49 / 95.67 ms │     no change │
   │ QQuery 8  │        36.62 / 38.34 ±2.46 / 43.10 ms │         37.06 / 37.31 ±0.18 / 37.54 ms │     no change │
   │ QQuery 9  │        52.52 / 53.21 ±0.60 / 54.10 ms │         50.03 / 52.55 ±1.36 / 53.85 ms │     no change │
   │ QQuery 10 │        62.68 / 63.17 ±0.32 / 63.64 ms │         63.86 / 64.06 ±0.16 / 64.28 ms │     no change │
   │ QQuery 11 │     289.40 / 293.64 ±4.94 / 302.95 ms │      425.71 / 431.01 ±3.15 / 435.37 ms │  1.47x slower │
   │ QQuery 12 │        28.24 / 28.66 ±0.40 / 29.42 ms │         25.83 / 26.17 ±0.32 / 26.73 ms │ +1.10x faster │
   │ QQuery 13 │     116.49 / 116.95 ±0.34 / 117.54 ms │      118.40 / 118.98 ±0.44 / 119.47 ms │     no change │
   │ QQuery 14 │     415.60 / 420.92 ±3.90 / 427.00 ms │      412.98 / 416.53 ±2.40 / 420.22 ms │     no change │
   │ QQuery 15 │        66.87 / 67.65 ±0.97 / 69.56 ms │         64.55 / 66.13 ±1.78 / 69.57 ms │     no change │
   │ QQuery 16 │           7.50 / 7.65 ±0.20 / 8.04 ms │            6.85 / 7.00 ±0.23 / 7.45 ms │ +1.09x faster │
   │ QQuery 17 │        83.24 / 84.96 ±1.63 / 87.48 ms │         81.63 / 82.69 ±1.16 / 84.61 ms │     no change │
   │ QQuery 18 │     123.64 / 129.44 ±4.62 / 134.28 ms │      123.55 / 125.35 ±1.89 / 128.72 ms │     no change │
   │ QQuery 19 │        40.86 / 41.30 ±0.27 / 41.61 ms │         41.68 / 42.73 ±1.84 / 46.38 ms │     no change │
   │ QQuery 20 │        35.33 / 36.21 ±0.99 / 37.99 ms │         31.51 / 31.97 ±0.34 / 32.50 ms │ +1.13x faster │
   │ QQuery 21 │        17.14 / 17.43 ±0.20 / 17.66 ms │         17.01 / 17.25 ±0.14 / 17.39 ms │     no change │
   │ QQuery 22 │        62.86 / 64.12 ±1.45 / 66.70 ms │         72.20 / 72.79 ±0.42 / 73.36 ms │  1.14x slower │
   │ QQuery 23 │     338.22 / 348.69 ±7.61 / 359.77 ms │     337.11 / 355.57 ±18.59 / 384.77 ms │     no change │
   │ QQuery 24 │     221.53 / 224.46 ±3.95 / 232.30 ms │      225.34 / 231.79 ±5.16 / 240.98 ms │     no change │
   │ QQuery 25 │     108.65 / 109.53 ±1.16 / 111.83 ms │      109.68 / 111.79 ±1.73 / 114.52 ms │     no change │
   │ QQuery 26 │        57.82 / 59.84 ±2.80 / 65.36 ms │         58.59 / 59.07 ±0.30 / 59.41 ms │     no change │
   │ QQuery 27 │           6.45 / 6.61 ±0.16 / 6.89 ms │           6.69 / 8.30 ±2.86 / 14.01 ms │  1.26x slower │
   │ QQuery 28 │        60.23 / 60.74 ±0.37 / 61.39 ms │         60.02 / 60.38 ±0.35 / 61.03 ms │     no change │
   │ QQuery 29 │        96.12 / 96.68 ±0.76 / 98.14 ms │       98.45 / 100.15 ±1.98 / 103.90 ms │     no change │
   │ QQuery 30 │        33.29 / 34.77 ±2.44 / 39.64 ms │         38.96 / 40.96 ±2.90 / 46.72 ms │  1.18x slower │
   │ QQuery 31 │     109.95 / 110.74 ±0.88 / 112.07 ms │      114.74 / 115.76 ±0.74 / 116.98 ms │     no change │
   │ QQuery 32 │        20.09 / 20.40 ±0.36 / 21.11 ms │         19.68 / 20.07 ±0.41 / 20.86 ms │     no change │
   │ QQuery 33 │        37.60 / 38.04 ±0.56 / 39.14 ms │         37.56 / 38.25 ±0.61 / 39.13 ms │     no change │
   │ QQuery 34 │        10.24 / 11.76 ±1.93 / 15.39 ms │           9.63 / 9.81 ±0.21 / 10.23 ms │ +1.20x faster │
   │ QQuery 35 │        72.77 / 73.81 ±0.99 / 75.67 ms │         71.80 / 72.50 ±0.92 / 74.28 ms │     no change │
   │ QQuery 36 │           5.82 / 5.98 ±0.25 / 6.47 ms │            6.14 / 6.26 ±0.17 / 6.60 ms │     no change │
   │ QQuery 37 │           6.84 / 6.92 ±0.08 / 7.04 ms │            7.62 / 8.04 ±0.29 / 8.52 ms │  1.16x slower │
   │ QQuery 38 │        61.39 / 62.02 ±0.46 / 62.67 ms │         77.39 / 78.48 ±1.28 / 80.97 ms │  1.27x slower │
   │ QQuery 39 │        89.26 / 90.04 ±0.51 / 90.73 ms │         90.14 / 92.56 ±2.80 / 97.74 ms │     no change │
   │ QQuery 40 │        23.29 / 23.71 ±0.26 / 23.97 ms │         23.35 / 23.96 ±0.45 / 24.74 ms │     no change │
   │ QQuery 41 │        11.31 / 11.47 ±0.15 / 11.76 ms │         12.56 / 12.69 ±0.19 / 13.05 ms │  1.11x slower │
   │ QQuery 42 │        23.54 / 24.08 ±0.60 / 25.17 ms │         23.67 / 25.03 ±1.76 / 28.17 ms │     no change │
   │ QQuery 43 │           5.47 / 5.56 ±0.12 / 5.80 ms │            5.71 / 5.87 ±0.21 / 6.27 ms │  1.06x slower │
   │ QQuery 44 │           9.64 / 9.73 ±0.11 / 9.92 ms │            9.50 / 9.69 ±0.12 / 9.85 ms │     no change │
   │ QQuery 45 │        37.38 / 38.40 ±0.73 / 39.12 ms │         36.77 / 37.55 ±0.62 / 38.50 ms │     no change │
   │ QQuery 46 │        11.83 / 12.06 ±0.17 / 12.28 ms │         11.77 / 11.92 ±0.13 / 12.15 ms │     no change │
   │ QQuery 47 │     223.04 / 225.37 ±2.66 / 230.32 ms │      210.18 / 212.26 ±2.11 / 216.32 ms │ +1.06x faster │
   │ QQuery 48 │       95.23 / 99.29 ±5.76 / 110.71 ms │         96.48 / 97.18 ±1.02 / 99.19 ms │     no change │
   │ QQuery 49 │        75.46 / 77.26 ±1.01 / 78.35 ms │         76.32 / 77.64 ±1.09 / 78.99 ms │     no change │
   │ QQuery 50 │        58.60 / 61.06 ±3.96 / 68.95 ms │         58.77 / 60.39 ±2.13 / 64.59 ms │     no change │
   │ QQuery 51 │        91.66 / 93.87 ±1.49 / 95.90 ms │         90.84 / 93.39 ±1.41 / 94.96 ms │     no change │
   │ QQuery 52 │        23.82 / 24.13 ±0.29 / 24.55 ms │         23.73 / 23.95 ±0.23 / 24.40 ms │     no change │
   │ QQuery 53 │        28.65 / 29.04 ±0.26 / 29.42 ms │         29.12 / 29.24 ±0.11 / 29.43 ms │     no change │
   │ QQuery 54 │        53.43 / 54.65 ±0.65 / 55.34 ms │         54.72 / 56.01 ±1.91 / 59.73 ms │     no change │
   │ QQuery 55 │        23.30 / 23.96 ±0.48 / 24.61 ms │         23.11 / 23.91 ±0.71 / 25.08 ms │     no change │
   │ QQuery 56 │        38.32 / 38.92 ±0.64 / 40.13 ms │         38.96 / 39.31 ±0.34 / 39.94 ms │     no change │
   │ QQuery 57 │     174.56 / 176.55 ±1.63 / 179.39 ms │      163.12 / 165.36 ±2.36 / 168.35 ms │ +1.07x faster │
   │ QQuery 58 │     111.04 / 112.22 ±0.96 / 113.78 ms │      108.07 / 109.76 ±1.41 / 111.95 ms │     no change │
   │ QQuery 59 │     117.19 / 118.57 ±1.98 / 122.44 ms │      114.85 / 116.33 ±1.38 / 118.46 ms │     no change │
   │ QQuery 60 │        38.96 / 39.72 ±0.44 / 40.35 ms │         39.84 / 40.54 ±0.54 / 41.38 ms │     no change │
   │ QQuery 61 │        12.33 / 12.65 ±0.27 / 13.10 ms │         12.26 / 12.43 ±0.24 / 12.89 ms │     no change │
   │ QQuery 62 │        47.94 / 50.37 ±3.66 / 57.59 ms │         45.74 / 46.05 ±0.19 / 46.24 ms │ +1.09x faster │
   │ QQuery 63 │        31.27 / 31.62 ±0.30 / 32.12 ms │         28.96 / 29.31 ±0.22 / 29.58 ms │ +1.08x faster │
   │ QQuery 64 │     437.60 / 442.56 ±2.74 / 445.91 ms │      414.79 / 420.29 ±5.31 / 429.46 ms │ +1.05x faster │
   │ QQuery 65 │     128.87 / 131.93 ±2.54 / 136.63 ms │      124.25 / 127.17 ±2.86 / 132.65 ms │     no change │
   │ QQuery 66 │        87.70 / 89.19 ±1.91 / 92.93 ms │         81.19 / 85.59 ±4.10 / 92.29 ms │     no change │
   │ QQuery 67 │     279.87 / 284.26 ±3.20 / 288.13 ms │      289.35 / 293.48 ±2.13 / 295.24 ms │     no change │
   │ QQuery 68 │        13.18 / 13.25 ±0.07 / 13.37 ms │         13.26 / 13.45 ±0.16 / 13.72 ms │     no change │
   │ QQuery 69 │        60.90 / 63.95 ±4.11 / 72.03 ms │         61.25 / 61.80 ±0.51 / 62.74 ms │     no change │
   │ QQuery 70 │     113.41 / 115.86 ±1.50 / 118.14 ms │      111.28 / 113.56 ±2.47 / 118.17 ms │     no change │
   │ QQuery 71 │        37.48 / 39.97 ±4.64 / 49.25 ms │         38.29 / 38.70 ±0.32 / 39.07 ms │     no change │
   │ QQuery 72 │ 2075.99 / 2130.90 ±47.35 / 2196.54 ms │ 1932.52 / 2087.13 ±112.13 / 2266.59 ms │     no change │
   │ QQuery 73 │          9.61 / 9.87 ±0.24 / 10.26 ms │           9.48 / 9.66 ±0.20 / 10.00 ms │     no change │
   │ QQuery 74 │     166.49 / 168.98 ±1.93 / 171.85 ms │      233.01 / 235.78 ±1.58 / 237.41 ms │  1.40x slower │
   │ QQuery 75 │     147.18 / 150.24 ±2.18 / 153.39 ms │      148.66 / 151.56 ±3.79 / 158.90 ms │     no change │
   │ QQuery 76 │        36.42 / 37.04 ±0.47 / 37.83 ms │         37.16 / 38.27 ±0.81 / 39.23 ms │     no change │
   │ QQuery 77 │        61.02 / 61.95 ±0.67 / 62.90 ms │         64.39 / 66.18 ±2.73 / 71.61 ms │  1.07x slower │
   │ QQuery 78 │     196.17 / 198.16 ±1.54 / 200.86 ms │     195.62 / 209.57 ±12.37 / 228.50 ms │  1.06x slower │
   │ QQuery 79 │        66.17 / 68.48 ±3.41 / 75.17 ms │         69.52 / 69.89 ±0.35 / 70.36 ms │     no change │
   │ QQuery 80 │      98.10 / 100.09 ±1.71 / 103.26 ms │       98.77 / 101.67 ±2.27 / 104.31 ms │     no change │
   │ QQuery 81 │        25.59 / 25.87 ±0.24 / 26.20 ms │         29.70 / 29.87 ±0.09 / 29.98 ms │  1.15x slower │
   │ QQuery 82 │        16.05 / 16.27 ±0.14 / 16.45 ms │         17.06 / 17.74 ±1.16 / 20.05 ms │  1.09x slower │
   │ QQuery 83 │        39.33 / 42.46 ±5.26 / 52.93 ms │         38.88 / 39.37 ±0.55 / 40.43 ms │ +1.08x faster │
   │ QQuery 84 │        29.98 / 30.57 ±0.37 / 30.95 ms │         32.46 / 33.21 ±0.86 / 34.85 ms │  1.09x slower │
   │ QQuery 85 │     105.73 / 107.38 ±1.60 / 110.43 ms │      108.25 / 109.00 ±0.91 / 110.62 ms │     no change │
   │ QQuery 86 │        25.01 / 25.32 ±0.35 / 25.99 ms │         25.87 / 26.08 ±0.35 / 26.78 ms │     no change │
   │ QQuery 87 │        61.91 / 63.83 ±2.66 / 69.10 ms │         78.19 / 80.27 ±3.02 / 86.23 ms │  1.26x slower │
   │ QQuery 88 │        62.23 / 63.56 ±0.85 / 64.84 ms │         63.81 / 64.80 ±1.42 / 67.60 ms │     no change │
   │ QQuery 89 │        34.96 / 35.35 ±0.50 / 36.33 ms │         36.45 / 37.01 ±0.34 / 37.43 ms │     no change │
   │ QQuery 90 │        16.84 / 17.03 ±0.17 / 17.31 ms │         17.33 / 17.53 ±0.19 / 17.86 ms │     no change │
   │ QQuery 91 │        45.27 / 45.55 ±0.20 / 45.83 ms │         46.08 / 46.32 ±0.19 / 46.59 ms │     no change │
   │ QQuery 92 │        28.88 / 32.18 ±4.46 / 40.97 ms │         28.68 / 30.29 ±1.83 / 33.86 ms │ +1.06x faster │
   │ QQuery 93 │        50.45 / 51.56 ±1.10 / 53.07 ms │         49.14 / 50.03 ±0.72 / 51.00 ms │     no change │
   │ QQuery 94 │        38.54 / 39.02 ±0.59 / 40.14 ms │         37.50 / 38.05 ±0.45 / 38.86 ms │     no change │
   │ QQuery 95 │        79.00 / 80.35 ±0.92 / 81.57 ms │         79.62 / 79.98 ±0.19 / 80.13 ms │     no change │
   │ QQuery 96 │        23.88 / 25.51 ±3.12 / 31.75 ms │         23.96 / 25.23 ±2.35 / 29.92 ms │     no change │
   │ QQuery 97 │        46.61 / 47.23 ±0.54 / 48.00 ms │         46.70 / 47.07 ±0.22 / 47.25 ms │     no change │
   │ QQuery 98 │        42.42 / 43.42 ±0.53 / 43.96 ms │         38.66 / 39.68 ±1.25 / 42.06 ms │ +1.09x faster │
   │ QQuery 99 │        69.41 / 69.81 ±0.28 / 70.23 ms │         68.90 / 69.41 ±0.43 / 70.10 ms │     no change │
   └───────────┴───────────────────────────────────────┴────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │  9965.79ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 10321.23ms │
   │ Average Time (HEAD)                                  │   100.66ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   104.25ms │
   │ Queries Faster                                       │         12 │
   │ Queries Slower                                       │         16 │
   │ Queries with No Change                               │         71 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 2.1 GiB |
   | Avg memory | 1.4 GiB |
   | CPU user | 224.1s |
   | CPU sys | 6.2s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 1.8 GiB |
   | Avg memory | 1.3 GiB |
   | CPU user | 222.4s |
   | CPU sys | 5.8s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5374322806

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5374297430)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5374297430-1849-gmv77 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5374538200

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5374297430)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 0  │    1.20 ms │                               1.20 ms │    no change │
   │ QQuery 1  │   11.60 ms │                              11.88 ms │    no change │
   │ QQuery 2  │   35.99 ms │                              37.25 ms │    no change │
   │ QQuery 3  │   30.51 ms │                              30.78 ms │    no change │
   │ QQuery 4  │  222.33 ms │                             222.86 ms │    no change │
   │ QQuery 5  │  268.62 ms │                             270.86 ms │    no change │
   │ QQuery 6  │    1.29 ms │                               1.25 ms │    no change │
   │ QQuery 7  │   12.89 ms │                              13.01 ms │    no change │
   │ QQuery 8  │  322.88 ms │                             320.54 ms │    no change │
   │ QQuery 9  │  447.23 ms │                             454.78 ms │    no change │
   │ QQuery 10 │   68.68 ms │                              69.36 ms │    no change │
   │ QQuery 11 │   79.99 ms │                              79.93 ms │    no change │
   │ QQuery 12 │  265.04 ms │                             268.67 ms │    no change │
   │ QQuery 13 │  362.00 ms │                             363.97 ms │    no change │
   │ QQuery 14 │  276.42 ms │                             279.99 ms │    no change │
   │ QQuery 15 │  264.23 ms │                             267.49 ms │    no change │
   │ QQuery 16 │  606.22 ms │                             606.52 ms │    no change │
   │ QQuery 17 │  610.64 ms │                             608.89 ms │    no change │
   │ QQuery 18 │ 1232.34 ms │                            1226.80 ms │    no change │
   │ QQuery 19 │   27.12 ms │                              27.14 ms │    no change │
   │ QQuery 20 │  514.11 ms │                             512.72 ms │    no change │
   │ QQuery 21 │  513.50 ms │                             517.66 ms │    no change │
   │ QQuery 22 │  985.01 ms │                             990.41 ms │    no change │
   │ QQuery 23 │ 2998.24 ms │                            3069.01 ms │    no change │
   │ QQuery 24 │   40.66 ms │                              41.06 ms │    no change │
   │ QQuery 25 │  110.25 ms │                             109.66 ms │    no change │
   │ QQuery 26 │   40.68 ms │                              41.06 ms │    no change │
   │ QQuery 27 │  510.34 ms │                             511.57 ms │    no change │
   │ QQuery 28 │ 2880.18 ms │                            2898.95 ms │    no change │
   │ QQuery 29 │   40.63 ms │                              40.40 ms │    no change │
   │ QQuery 30 │  296.79 ms │                             301.31 ms │    no change │
   │ QQuery 31 │  272.79 ms │                             284.74 ms │    no change │
   │ QQuery 32 │  922.27 ms │                             908.35 ms │    no change │
   │ QQuery 33 │ 1431.77 ms │                            1436.29 ms │    no change │
   │ QQuery 34 │ 1464.67 ms │                            1452.04 ms │    no change │
   │ QQuery 35 │  269.53 ms │                             274.85 ms │    no change │
   │ QQuery 36 │   66.16 ms │                              67.56 ms │    no change │
   │ QQuery 37 │   36.34 ms │                              35.26 ms │    no change │
   │ QQuery 38 │   40.69 ms │                              40.81 ms │    no change │
   │ QQuery 39 │  134.90 ms │                             150.44 ms │ 1.12x slower │
   │ QQuery 40 │   13.92 ms │                              13.84 ms │    no change │
   │ QQuery 41 │   13.44 ms │                              13.51 ms │    no change │
   │ QQuery 42 │   12.97 ms │                              13.42 ms │    no change │
   └───────────┴────────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 18757.06ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 18888.08ms │
   │ Average Time (HEAD)                                  │   436.21ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   439.26ms │
   │ Queries Faster                                       │          0 │
   │ Queries Slower                                       │          1 │
   │ Queries with No Change                               │         42 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.20 / 3.89 ±5.30 / 14.48 ms │          1.20 / 3.92 ±5.38 / 14.69 ms │     no change │
   │ QQuery 1  │        11.60 / 11.85 ±0.17 / 12.08 ms │        11.88 / 12.16 ±0.25 / 12.55 ms │     no change │
   │ QQuery 2  │        35.99 / 36.54 ±0.55 / 37.54 ms │        37.25 / 37.40 ±0.15 / 37.65 ms │     no change │
   │ QQuery 3  │        30.51 / 34.82 ±3.53 / 40.09 ms │        30.78 / 30.96 ±0.18 / 31.25 ms │ +1.12x faster │
   │ QQuery 4  │     222.33 / 225.35 ±4.18 / 233.63 ms │     222.86 / 225.62 ±2.14 / 228.60 ms │     no change │
   │ QQuery 5  │     268.62 / 273.24 ±2.34 / 274.98 ms │     270.86 / 272.97 ±1.43 / 274.92 ms │     no change │
   │ QQuery 6  │           1.29 / 1.42 ±0.21 / 1.84 ms │           1.25 / 1.40 ±0.22 / 1.83 ms │     no change │
   │ QQuery 7  │        12.89 / 13.06 ±0.14 / 13.23 ms │        13.01 / 13.22 ±0.13 / 13.38 ms │     no change │
   │ QQuery 8  │     322.88 / 326.09 ±1.71 / 327.99 ms │     320.54 / 326.22 ±4.46 / 333.62 ms │     no change │
   │ QQuery 9  │     447.23 / 454.34 ±4.25 / 460.20 ms │     454.78 / 460.22 ±2.99 / 463.23 ms │     no change │
   │ QQuery 10 │        68.68 / 69.46 ±0.62 / 70.41 ms │        69.36 / 71.09 ±2.09 / 75.11 ms │     no change │
   │ QQuery 11 │        79.99 / 84.07 ±7.57 / 99.19 ms │        79.93 / 82.03 ±1.91 / 85.30 ms │     no change │
   │ QQuery 12 │     265.04 / 272.52 ±4.85 / 279.48 ms │     268.67 / 274.44 ±2.95 / 276.77 ms │     no change │
   │ QQuery 13 │     362.00 / 369.65 ±6.76 / 381.53 ms │    363.97 / 380.22 ±16.16 / 410.70 ms │     no change │
   │ QQuery 14 │     276.42 / 282.63 ±4.45 / 288.34 ms │     279.99 / 285.86 ±6.39 / 295.57 ms │     no change │
   │ QQuery 15 │     264.23 / 270.63 ±6.57 / 282.49 ms │    267.49 / 287.00 ±30.72 / 348.20 ms │  1.06x slower │
   │ QQuery 16 │     606.22 / 610.80 ±5.39 / 620.99 ms │     606.52 / 616.95 ±7.25 / 627.09 ms │     no change │
   │ QQuery 17 │     610.64 / 620.59 ±8.48 / 632.98 ms │    608.89 / 630.05 ±21.91 / 668.21 ms │     no change │
   │ QQuery 18 │ 1232.34 / 1248.76 ±15.71 / 1275.07 ms │ 1226.80 / 1270.37 ±25.20 / 1302.96 ms │     no change │
   │ QQuery 19 │        27.12 / 27.38 ±0.31 / 27.95 ms │       27.14 / 38.62 ±14.62 / 63.57 ms │  1.41x slower │
   │ QQuery 20 │     514.11 / 523.08 ±7.95 / 532.91 ms │     512.72 / 519.40 ±4.88 / 526.09 ms │     no change │
   │ QQuery 21 │     513.50 / 518.00 ±3.62 / 523.55 ms │     517.66 / 521.72 ±4.92 / 531.01 ms │     no change │
   │ QQuery 22 │   985.01 / 994.28 ±10.89 / 1014.53 ms │    990.41 / 999.31 ±5.61 / 1005.30 ms │     no change │
   │ QQuery 23 │ 2998.24 / 3037.67 ±27.20 / 3078.27 ms │ 3069.01 / 3090.61 ±15.06 / 3113.76 ms │     no change │
   │ QQuery 24 │       40.66 / 52.83 ±22.94 / 98.69 ms │        41.06 / 46.68 ±6.92 / 58.37 ms │ +1.13x faster │
   │ QQuery 25 │     110.25 / 111.27 ±0.91 / 112.68 ms │     109.66 / 115.54 ±9.90 / 135.30 ms │     no change │
   │ QQuery 26 │        40.68 / 41.37 ±0.61 / 42.50 ms │        41.06 / 43.67 ±3.82 / 51.23 ms │  1.06x slower │
   │ QQuery 27 │     510.34 / 523.08 ±9.69 / 539.98 ms │     511.57 / 516.23 ±4.87 / 524.72 ms │     no change │
   │ QQuery 28 │ 2880.18 / 2900.59 ±13.56 / 2918.85 ms │ 2898.95 / 2916.87 ±12.90 / 2937.00 ms │     no change │
   │ QQuery 29 │      40.63 / 60.91 ±34.47 / 129.43 ms │        40.40 / 41.56 ±1.76 / 45.05 ms │ +1.47x faster │
   │ QQuery 30 │     296.79 / 305.32 ±9.60 / 324.10 ms │     301.31 / 307.16 ±3.61 / 312.02 ms │     no change │
   │ QQuery 31 │     272.79 / 284.81 ±9.51 / 299.35 ms │     284.74 / 289.89 ±6.24 / 302.00 ms │     no change │
   │ QQuery 32 │    922.27 / 939.56 ±10.84 / 952.84 ms │    908.35 / 937.06 ±21.76 / 975.12 ms │     no change │
   │ QQuery 33 │ 1431.77 / 1465.02 ±21.66 / 1491.35 ms │ 1436.29 / 1459.17 ±18.70 / 1482.21 ms │     no change │
   │ QQuery 34 │ 1464.67 / 1511.86 ±63.60 / 1636.17 ms │ 1452.04 / 1484.97 ±34.91 / 1548.10 ms │     no change │
   │ QQuery 35 │    269.53 / 287.95 ±22.55 / 332.06 ms │    274.85 / 327.66 ±62.02 / 431.42 ms │  1.14x slower │
   │ QQuery 36 │        66.16 / 73.59 ±7.34 / 86.77 ms │        67.56 / 73.49 ±6.56 / 85.44 ms │     no change │
   │ QQuery 37 │        36.34 / 40.96 ±4.92 / 47.04 ms │        35.26 / 39.88 ±4.32 / 45.20 ms │     no change │
   │ QQuery 38 │        40.69 / 42.80 ±1.19 / 44.09 ms │        40.81 / 42.21 ±0.94 / 43.69 ms │     no change │
   │ QQuery 39 │    134.90 / 150.75 ±13.32 / 166.71 ms │     150.44 / 155.42 ±4.38 / 160.98 ms │     no change │
   │ QQuery 40 │        13.92 / 14.63 ±0.63 / 15.64 ms │        13.84 / 16.07 ±2.69 / 20.53 ms │  1.10x slower │
   │ QQuery 41 │        13.44 / 13.78 ±0.33 / 14.28 ms │        13.51 / 13.74 ±0.26 / 14.21 ms │     no change │
   │ QQuery 42 │        12.97 / 13.16 ±0.19 / 13.46 ms │        13.42 / 13.65 ±0.27 / 14.19 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 19144.36ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 19292.67ms │
   │ Average Time (HEAD)                                  │   445.22ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   448.67ms │
   │ Queries Faster                                       │          3 │
   │ Queries Slower                                       │          5 │
   │ Queries with No Change                               │         35 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 12.9 GiB |
   | Avg memory | 4.8 GiB |
   | CPU user | 986.2s |
   | CPU sys | 65.9s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 11.7 GiB |
   | Avg memory | 4.5 GiB |
   | CPU user | 986.5s |
   | CPU sys | 67.8s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3959087988


##########
datafusion/sqllogictest/test_files/parquet_rle_to_dictionary.slt:
##########
@@ -0,0 +1,334 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Tests for datafusion.execution.parquet.enable_rle_to_dictionary. The flag
+# applies only when DataFusion infers the table schema; explicit schemas are
+# not promoted.
+
+# Write with dictionary_enabled=true to guarantee RLE_DICTIONARY encoding.
+query I
+COPY (
+  SELECT column1 AS product_category, column2 AS status
+  FROM (VALUES
+    ('electronics', 'active'),
+    ('clothing',    'active'),
+    ('electronics', 'inactive'),
+    ('books',       'inactive'),
+    ('furniture',   'pending'),
+    ('books',       'active')
+  )
+)
+TO 'test_files/scratch/parquet_rle_to_dictionary/products.parquet'
+STORED AS PARQUET
+OPTIONS ('format.dictionary_enabled' true);
+----
+6
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = false;
+
+statement ok
+CREATE EXTERNAL TABLE products_utf8
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/products.parquet';
+
+query TT
+SELECT DISTINCT arrow_typeof(product_category), arrow_typeof(status) FROM products_utf8;
+----
+Utf8View Utf8View
+
+query TI rowsort
+SELECT product_category, COUNT(*) FROM products_utf8 GROUP BY product_category;
+----
+books 2
+clothing 1
+electronics 2
+furniture 1
+
+statement ok
+DROP TABLE products_utf8;
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = true;
+
+statement ok
+CREATE EXTERNAL TABLE products_dict
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/products.parquet';
+
+query TT
+SELECT DISTINCT arrow_typeof(product_category), arrow_typeof(status) FROM products_dict;
+----
+Dictionary(Int32, Utf8) Dictionary(Int32, Utf8)
+
+query TI rowsort
+SELECT product_category, COUNT(*) FROM products_dict GROUP BY product_category;
+----
+books 2
+clothing 1
+electronics 2
+furniture 1
+
+# Predicates and aggregates must work against dictionary scan output.
+query TT rowsort
+SELECT product_category, status FROM products_dict WHERE status = 'active';
+----
+books active
+clothing active
+electronics active
+
+query TI rowsort
+SELECT product_category, COUNT(*) FROM products_dict WHERE status != 'inactive' GROUP BY product_category;
+----
+books 1
+clothing 1
+electronics 1
+furniture 1
+
+query TT
+SELECT DISTINCT arrow_typeof(product_category), arrow_typeof(product_count)
+FROM (
+  SELECT product_category, COUNT(*) AS product_count
+  FROM products_dict
+  WHERE status != 'inactive'
+  GROUP BY product_category
+);
+----
+Dictionary(Int32, Utf8) Int64
+
+statement ok
+DROP TABLE products_dict;
+
+# Per-column writer options: only columns with dictionary pages are promoted.
+
+statement ok
+COPY (
+  SELECT column1 AS env, column2 AS region, column3 AS build
+  FROM (VALUES
+    ('prod',    'us-east', 'debug'),
+    ('staging', 'us-west', 'release'),
+    ('prod',    'us-east', 'debug')
+  )
+)
+TO 'test_files/scratch/parquet_rle_to_dictionary/selective.parquet'
+STORED AS PARQUET
+OPTIONS (
+  'format.dictionary_enabled' false,
+  'format.dictionary_enabled::env' true,
+  'format.dictionary_enabled::region' true
+);
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = false;
+
+statement ok
+CREATE EXTERNAL TABLE selective_utf8
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/selective.parquet';
+
+query TTT
+SELECT DISTINCT arrow_typeof(env), arrow_typeof(region), arrow_typeof(build)
+FROM selective_utf8;
+----
+Utf8View Utf8View Utf8View
+
+statement ok
+DROP TABLE selective_utf8;
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = true;
+
+statement ok
+CREATE EXTERNAL TABLE selective_dict
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/selective.parquet';
+
+query TTT
+SELECT DISTINCT arrow_typeof(env), arrow_typeof(region), arrow_typeof(build)
+FROM selective_dict;
+----
+Dictionary(Int32, Utf8) Dictionary(Int32, Utf8) Utf8View
+
+statement ok
+DROP TABLE selective_dict;
+
+# Cross-file mixed encoding: one file has dictionary pages and one is plain.
+# Schema inference must merge both as Dictionary(Int32, Utf8).
+
+statement ok
+COPY (SELECT column1 AS category FROM (VALUES ('electronics'), ('electronics')))
+TO 'test_files/scratch/parquet_rle_to_dictionary/mixed/rle.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' true);
+
+statement ok
+COPY (SELECT column1 AS category FROM (VALUES ('books'), ('books')))
+TO 'test_files/scratch/parquet_rle_to_dictionary/mixed/plain.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' false);
+
+statement ok
+set datafusion.execution.parquet.enable_rle_to_dictionary = true;
+
+statement ok
+CREATE EXTERNAL TABLE mixed_encoding
+STORED AS PARQUET
+LOCATION 'test_files/scratch/parquet_rle_to_dictionary/mixed/';
+
+query TI
+SELECT arrow_typeof(category), COUNT(*) FROM mixed_encoding GROUP BY arrow_typeof(category);
+----
+Dictionary(Int32, Utf8) 4
+
+statement ok
+DROP TABLE mixed_encoding;
+
+# Mixed encoding with high cardinality: the plain file contains more than 128
+# distinct values so a narrow (Int8) dictionary key would overflow. After the
+# fix, uniform_dict_schemas widens the shared key to at least Int32.
+# Both files must be scannable and return the correct row count.
+
+statement ok
+COPY (
+  SELECT v AS category
+  FROM (
+    VALUES
+      ('v000'),('v001'),('v002'),('v003'),('v004'),('v005'),('v006'),('v007'),
+      ('v008'),('v009'),('v010'),('v011'),('v012'),('v013'),('v014'),('v015'),
+      ('v016'),('v017'),('v018'),('v019'),('v020'),('v021'),('v022'),('v023'),
+      ('v024'),('v025'),('v026'),('v027'),('v028'),('v029'),('v030'),('v031'),
+      ('v032'),('v033'),('v034'),('v035'),('v036'),('v037'),('v038'),('v039'),
+      ('v040'),('v041'),('v042'),('v043'),('v044'),('v045'),('v046'),('v047'),
+      ('v048'),('v049'),('v050'),('v051'),('v052'),('v053'),('v054'),('v055'),
+      ('v056'),('v057'),('v058'),('v059'),('v060'),('v061'),('v062'),('v063'),
+      ('v064'),('v065'),('v066'),('v067'),('v068'),('v069'),('v070'),('v071'),
+      ('v072'),('v073'),('v074'),('v075'),('v076'),('v077'),('v078'),('v079'),
+      ('v080'),('v081'),('v082'),('v083'),('v084'),('v085'),('v086'),('v087'),
+      ('v088'),('v089'),('v090'),('v091'),('v092'),('v093'),('v094'),('v095'),
+      ('v096'),('v097'),('v098'),('v099'),('v100'),('v101'),('v102'),('v103'),
+      ('v104'),('v105'),('v106'),('v107'),('v108'),('v109'),('v110'),('v111'),
+      ('v112'),('v113'),('v114'),('v115'),('v116'),('v117'),('v118'),('v119'),
+      ('v120'),('v121'),('v122'),('v123'),('v124'),('v125'),('v126'),('v127'),
+      ('v128'),('v129')
+  ) AS t(v)
+)
+TO 'test_files/scratch/parquet_rle_to_dictionary/high_card/plain.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' false);
+
+statement ok
+COPY (SELECT column1 AS category FROM (VALUES ('rle_a'), ('rle_b'), ('rle_a')))
+TO 'test_files/scratch/parquet_rle_to_dictionary/high_card/rle.parquet'
+STORED AS PARQUET OPTIONS ('format.dictionary_enabled' true);

Review Comment:
   The new unit test correctly covers `Dictionary(Int8, Utf8) + Utf8`, but I do not think this SQLLogicTest exercises that regression. `rle.parquet` is written from a plain `Utf8` expression, so RLE inference promotes it directly to `Dictionary(Int32, Utf8)` before `uniform_dict_schemas` runs. The previous implementation would therefore also scan these 130 values successfully.
   
   Could the dictionary file be written from an actual `Dictionary(Int8, Utf8)` array—using `arrow_cast` if supported here, or a Rust Parquet-writer test then combined with the 130-value plain file? That would verify the full inference and scan path that previously overflowed.



##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -135,6 +157,169 @@ pub fn apply_file_schema_type_coercions(
     ))
 }
 
+// Find the value type that can represent both sides without narrowing offsets
+// or crossing string/binary families.
+fn common_dictionary_value_type(
+    field_type: &DataType,
+    dictionary_value_type: &DataType,
+) -> Option<DataType> {
+    let field_type = match field_type {
+        DataType::Dictionary(_, field_value_type) => field_value_type.as_ref(),
+        _ => field_type,
+    };
+
+    match (field_type, dictionary_value_type) {
+        (DataType::Utf8, DataType::Utf8) => Some(DataType::Utf8),
+        (DataType::Utf8 | DataType::LargeUtf8, DataType::Utf8 | DataType::LargeUtf8) => {
+            Some(DataType::LargeUtf8)
+        }
+        (DataType::Binary, DataType::Binary) => Some(DataType::Binary),
+        (
+            DataType::Binary | DataType::LargeBinary,
+            DataType::Binary | DataType::LargeBinary,
+        ) => Some(DataType::LargeBinary),
+        _ => None,
+    }
+}
+
+// Same family (both signed or both unsigned): return the wider member.

Review Comment:
   The mixed signed/unsigned case still appears to narrow the key type: the test expects `UInt64 + Int64 → Int64`, although `Int64` cannot represent the full `UInt64` key domain. The comment says Parquet supports only signed dictionary keys, but arrow-rs's byte-array dictionary reader supports `UInt8`, `UInt16`, `UInt32`, and `UInt64` keys too.
   
   Could this select `UInt64` for that case, or reject the normalization instead of silently narrowing it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675462154

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f (cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f) to cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f [diff](https://github.com/apache/datafusion/compare/cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f..cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   env:
     DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
   baseline:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
   changed:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    5.52 ms │                               5.87 ms │  1.06x slower │
   │ QQuery 2  │   80.27 ms │                              84.30 ms │  1.05x slower │
   │ QQuery 3  │   28.74 ms │                              29.52 ms │     no change │
   │ QQuery 4  │  488.18 ms │                             664.61 ms │  1.36x slower │
   │ QQuery 5  │   52.24 ms │                              52.35 ms │     no change │
   │ QQuery 6  │   36.32 ms │                              34.83 ms │     no change │
   │ QQuery 7  │   74.32 ms │                              76.49 ms │     no change │
   │ QQuery 8  │   36.50 ms │                              37.36 ms │     no change │
   │ QQuery 9  │   52.80 ms │                              51.99 ms │     no change │
   │ QQuery 10 │   61.24 ms │                              63.35 ms │     no change │
   │ QQuery 11 │  306.53 ms │                             437.80 ms │  1.43x slower │
   │ QQuery 12 │   28.98 ms │                              25.73 ms │ +1.13x faster │
   │ QQuery 13 │  117.36 ms │                             121.36 ms │     no change │
   │ QQuery 14 │  415.32 ms │                             419.35 ms │     no change │
   │ QQuery 15 │   57.00 ms │                              65.28 ms │  1.15x slower │
   │ QQuery 16 │    6.53 ms │                               6.67 ms │     no change │
   │ QQuery 17 │   80.10 ms │                              82.14 ms │     no change │
   │ QQuery 18 │  103.91 ms │                             106.01 ms │     no change │
   │ QQuery 19 │   41.37 ms │                              42.25 ms │     no change │
   │ QQuery 20 │   35.46 ms │                              31.79 ms │ +1.12x faster │
   │ QQuery 21 │   17.16 ms │                              17.27 ms │     no change │
   │ QQuery 22 │   62.97 ms │                              72.40 ms │  1.15x slower │
   │ QQuery 23 │  313.60 ms │                             314.76 ms │     no change │
   │ QQuery 24 │  195.62 ms │                             196.51 ms │     no change │
   │ QQuery 25 │  110.00 ms │                             109.55 ms │     no change │
   │ QQuery 26 │   48.70 ms │                              50.47 ms │     no change │
   │ QQuery 27 │    6.14 ms │                               6.66 ms │  1.08x slower │
   │ QQuery 28 │   56.91 ms │                              56.84 ms │     no change │
   │ QQuery 29 │   97.68 ms │                              99.98 ms │     no change │
   │ QQuery 30 │   31.75 ms │                              39.76 ms │  1.25x slower │
   │ QQuery 31 │  110.59 ms │                             115.76 ms │     no change │
   │ QQuery 32 │   20.18 ms │                              20.25 ms │     no change │
   │ QQuery 33 │   38.64 ms │                              37.90 ms │     no change │
   │ QQuery 34 │    9.97 ms │                               9.76 ms │     no change │
   │ QQuery 35 │   71.85 ms │                              72.93 ms │     no change │
   │ QQuery 36 │    5.76 ms │                               6.03 ms │     no change │
   │ QQuery 37 │    6.80 ms │                               7.87 ms │  1.16x slower │
   │ QQuery 38 │   61.59 ms │                              77.88 ms │  1.26x slower │
   │ QQuery 39 │   88.93 ms │                              91.27 ms │     no change │
   │ QQuery 40 │   23.87 ms │                              24.62 ms │     no change │
   │ QQuery 41 │   11.18 ms │                              12.49 ms │  1.12x slower │
   │ QQuery 42 │   23.52 ms │                              23.91 ms │     no change │
   │ QQuery 43 │    5.06 ms │                               5.29 ms │     no change │
   │ QQuery 44 │    9.31 ms │                               9.15 ms │     no change │
   │ QQuery 45 │   39.34 ms │                              38.25 ms │     no change │
   │ QQuery 46 │   12.14 ms │                              11.87 ms │     no change │
   │ QQuery 47 │  227.46 ms │                             211.65 ms │ +1.07x faster │
   │ QQuery 48 │   95.07 ms │                              97.39 ms │     no change │
   │ QQuery 49 │   70.91 ms │                              70.92 ms │     no change │
   │ QQuery 50 │   59.05 ms │                              60.04 ms │     no change │
   │ QQuery 51 │   91.14 ms │                              91.66 ms │     no change │
   │ QQuery 52 │   23.91 ms │                              24.04 ms │     no change │
   │ QQuery 53 │   29.16 ms │                              29.39 ms │     no change │
   │ QQuery 54 │   54.06 ms │                              55.36 ms │     no change │
   │ QQuery 55 │   23.38 ms │                              23.65 ms │     no change │
   │ QQuery 56 │   38.95 ms │                              39.45 ms │     no change │
   │ QQuery 57 │  175.66 ms │                             163.08 ms │ +1.08x faster │
   │ QQuery 58 │  113.67 ms │                             108.32 ms │     no change │
   │ QQuery 59 │  117.84 ms │                             115.97 ms │     no change │
   │ QQuery 60 │   39.44 ms │                              39.60 ms │     no change │
   │ QQuery 61 │   12.10 ms │                              12.26 ms │     no change │
   │ QQuery 62 │   46.29 ms │                              46.77 ms │     no change │
   │ QQuery 63 │   29.62 ms │                              29.43 ms │     no change │
   │ QQuery 64 │  364.39 ms │                             376.51 ms │     no change │
   │ QQuery 65 │  122.72 ms │                             125.13 ms │     no change │
   │ QQuery 66 │   80.64 ms │                              78.98 ms │     no change │
   │ QQuery 67 │  243.75 ms │                             248.87 ms │     no change │
   │ QQuery 68 │   11.97 ms │                              12.04 ms │     no change │
   │ QQuery 69 │   56.25 ms │                              57.95 ms │     no change │
   │ QQuery 70 │  104.35 ms │                             101.56 ms │     no change │
   │ QQuery 71 │   35.17 ms │                              35.90 ms │     no change │
   │ QQuery 72 │ 1840.58 ms │                            1761.34 ms │     no change │
   │ QQuery 73 │   10.30 ms │                               9.63 ms │ +1.07x faster │
   │ QQuery 74 │  170.78 ms │                             230.11 ms │  1.35x slower │
   │ QQuery 75 │  145.69 ms │                             146.43 ms │     no change │
   │ QQuery 76 │   34.46 ms │                              36.07 ms │     no change │
   │ QQuery 77 │   60.89 ms │                              62.01 ms │     no change │
   │ QQuery 78 │  221.60 ms │                             223.40 ms │     no change │
   │ QQuery 79 │   66.06 ms │                              67.51 ms │     no change │
   │ QQuery 80 │   98.48 ms │                              99.60 ms │     no change │
   │ QQuery 81 │   25.93 ms │                              30.45 ms │  1.17x slower │
   │ QQuery 82 │   16.15 ms │                              17.33 ms │  1.07x slower │
   │ QQuery 83 │   34.23 ms │                              33.75 ms │     no change │
   │ QQuery 84 │   30.00 ms │                              32.01 ms │  1.07x slower │
   │ QQuery 85 │  103.72 ms │                             105.75 ms │     no change │
   │ QQuery 86 │   25.07 ms │                              25.09 ms │     no change │
   │ QQuery 87 │   62.62 ms │                              78.51 ms │  1.25x slower │
   │ QQuery 88 │   64.08 ms │                              64.73 ms │     no change │
   │ QQuery 89 │   36.18 ms │                              36.55 ms │     no change │
   │ QQuery 90 │   17.26 ms │                              17.43 ms │     no change │
   │ QQuery 91 │   44.93 ms │                              46.22 ms │     no change │
   │ QQuery 92 │   29.59 ms │                              29.65 ms │     no change │
   │ QQuery 93 │   50.41 ms │                              51.03 ms │     no change │
   │ QQuery 94 │   38.79 ms │                              38.55 ms │     no change │
   │ QQuery 95 │   80.53 ms │                              80.99 ms │     no change │
   │ QQuery 96 │   24.27 ms │                              23.75 ms │     no change │
   │ QQuery 97 │   52.15 ms │                              51.65 ms │     no change │
   │ QQuery 98 │   44.16 ms │                              38.51 ms │ +1.15x faster │
   │ QQuery 99 │   71.10 ms │                              69.69 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 9354.95ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 9732.14ms │
   │ Average Time (HEAD)                                  │   94.49ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   98.30ms │
   │ Queries Faster                                       │         6 │
   │ Queries Slower                                       │        16 │
   │ Queries with No Change                               │        77 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │           5.52 / 6.04 ±0.90 / 7.84 ms │           5.87 / 6.38 ±0.95 / 8.29 ms │  1.06x slower │
   │ QQuery 2  │        80.27 / 80.53 ±0.22 / 80.86 ms │        84.30 / 84.59 ±0.18 / 84.84 ms │  1.05x slower │
   │ QQuery 3  │        28.74 / 29.17 ±0.33 / 29.61 ms │        29.52 / 29.75 ±0.19 / 30.10 ms │     no change │
   │ QQuery 4  │     488.18 / 492.46 ±3.26 / 497.11 ms │    664.61 / 674.27 ±10.66 / 689.43 ms │  1.37x slower │
   │ QQuery 5  │        52.24 / 52.55 ±0.27 / 52.98 ms │        52.35 / 54.06 ±3.02 / 60.08 ms │     no change │
   │ QQuery 6  │        36.32 / 36.47 ±0.17 / 36.81 ms │        34.83 / 35.32 ±0.46 / 36.19 ms │     no change │
   │ QQuery 7  │        74.32 / 75.01 ±0.47 / 75.72 ms │        76.49 / 76.84 ±0.32 / 77.35 ms │     no change │
   │ QQuery 8  │        36.50 / 38.24 ±3.26 / 44.76 ms │        37.36 / 37.51 ±0.12 / 37.67 ms │     no change │
   │ QQuery 9  │        52.80 / 54.33 ±1.49 / 57.15 ms │        51.99 / 55.64 ±3.35 / 61.97 ms │     no change │
   │ QQuery 10 │        61.24 / 61.56 ±0.35 / 62.23 ms │        63.35 / 63.91 ±0.37 / 64.49 ms │     no change │
   │ QQuery 11 │     306.53 / 309.10 ±1.96 / 311.89 ms │     437.80 / 442.65 ±3.48 / 447.05 ms │  1.43x slower │
   │ QQuery 12 │        28.98 / 29.43 ±0.37 / 29.90 ms │        25.73 / 26.55 ±0.52 / 27.33 ms │ +1.11x faster │
   │ QQuery 13 │     117.36 / 119.12 ±2.06 / 123.09 ms │     121.36 / 122.92 ±1.38 / 124.93 ms │     no change │
   │ QQuery 14 │     415.32 / 419.44 ±2.52 / 422.88 ms │     419.35 / 423.38 ±4.02 / 430.22 ms │     no change │
   │ QQuery 15 │        57.00 / 58.37 ±1.15 / 60.16 ms │        65.28 / 66.36 ±0.61 / 67.09 ms │  1.14x slower │
   │ QQuery 16 │           6.53 / 6.72 ±0.25 / 7.22 ms │           6.67 / 6.91 ±0.25 / 7.37 ms │     no change │
   │ QQuery 17 │        80.10 / 82.54 ±3.39 / 89.20 ms │        82.14 / 86.19 ±3.84 / 93.01 ms │     no change │
   │ QQuery 18 │     103.91 / 104.40 ±0.56 / 105.44 ms │     106.01 / 107.20 ±0.82 / 108.23 ms │     no change │
   │ QQuery 19 │        41.37 / 41.64 ±0.27 / 42.13 ms │        42.25 / 43.93 ±1.75 / 46.52 ms │  1.05x slower │
   │ QQuery 20 │        35.46 / 36.26 ±1.00 / 38.17 ms │        31.79 / 32.24 ±0.26 / 32.50 ms │ +1.12x faster │
   │ QQuery 21 │        17.16 / 17.32 ±0.10 / 17.49 ms │        17.27 / 17.64 ±0.21 / 17.83 ms │     no change │
   │ QQuery 22 │        62.97 / 63.75 ±0.58 / 64.59 ms │        72.40 / 73.53 ±0.72 / 74.35 ms │  1.15x slower │
   │ QQuery 23 │     313.60 / 317.20 ±2.55 / 320.22 ms │     314.76 / 318.38 ±2.74 / 321.14 ms │     no change │
   │ QQuery 24 │     195.62 / 196.68 ±0.92 / 198.12 ms │     196.51 / 200.46 ±3.45 / 204.72 ms │     no change │
   │ QQuery 25 │     110.00 / 112.55 ±2.14 / 116.24 ms │     109.55 / 112.11 ±1.76 / 114.27 ms │     no change │
   │ QQuery 26 │        48.70 / 48.91 ±0.17 / 49.11 ms │        50.47 / 53.08 ±2.97 / 57.95 ms │  1.09x slower │
   │ QQuery 27 │           6.14 / 6.32 ±0.15 / 6.53 ms │           6.66 / 6.83 ±0.13 / 6.98 ms │  1.08x slower │
   │ QQuery 28 │        56.91 / 60.06 ±1.59 / 61.07 ms │        56.84 / 60.36 ±1.86 / 62.27 ms │     no change │
   │ QQuery 29 │      97.68 / 100.05 ±2.40 / 103.23 ms │      99.98 / 101.52 ±1.80 / 104.52 ms │     no change │
   │ QQuery 30 │        31.75 / 32.39 ±0.37 / 32.83 ms │        39.76 / 41.81 ±3.34 / 48.46 ms │  1.29x slower │
   │ QQuery 31 │     110.59 / 112.54 ±2.23 / 116.88 ms │     115.76 / 116.91 ±0.67 / 117.84 ms │     no change │
   │ QQuery 32 │        20.18 / 21.49 ±1.57 / 24.55 ms │        20.25 / 20.71 ±0.34 / 21.13 ms │     no change │
   │ QQuery 33 │        38.64 / 39.14 ±0.65 / 40.40 ms │        37.90 / 39.73 ±2.71 / 45.13 ms │     no change │
   │ QQuery 34 │         9.97 / 10.19 ±0.23 / 10.52 ms │         9.76 / 10.41 ±0.62 / 11.51 ms │     no change │
   │ QQuery 35 │        71.85 / 73.05 ±1.29 / 75.33 ms │        72.93 / 73.53 ±0.36 / 73.89 ms │     no change │
   │ QQuery 36 │           5.76 / 5.91 ±0.22 / 6.33 ms │           6.03 / 6.20 ±0.24 / 6.68 ms │     no change │
   │ QQuery 37 │           6.80 / 6.92 ±0.10 / 7.05 ms │           7.87 / 8.32 ±0.29 / 8.67 ms │  1.20x slower │
   │ QQuery 38 │        61.59 / 62.45 ±0.74 / 63.56 ms │        77.88 / 79.00 ±1.32 / 81.59 ms │  1.27x slower │
   │ QQuery 39 │        88.93 / 89.80 ±0.80 / 91.16 ms │        91.27 / 93.32 ±3.23 / 99.71 ms │     no change │
   │ QQuery 40 │        23.87 / 26.09 ±3.25 / 32.41 ms │        24.62 / 25.03 ±0.24 / 25.36 ms │     no change │
   │ QQuery 41 │        11.18 / 11.43 ±0.18 / 11.69 ms │        12.49 / 12.62 ±0.11 / 12.77 ms │  1.10x slower │
   │ QQuery 42 │        23.52 / 24.02 ±0.53 / 24.85 ms │        23.91 / 24.29 ±0.39 / 25.05 ms │     no change │
   │ QQuery 43 │           5.06 / 5.18 ±0.15 / 5.47 ms │           5.29 / 5.44 ±0.20 / 5.83 ms │  1.05x slower │
   │ QQuery 44 │           9.31 / 9.42 ±0.09 / 9.56 ms │           9.15 / 9.41 ±0.21 / 9.72 ms │     no change │
   │ QQuery 45 │        39.34 / 39.85 ±0.71 / 41.24 ms │        38.25 / 39.93 ±2.20 / 44.23 ms │     no change │
   │ QQuery 46 │        12.14 / 12.37 ±0.30 / 12.94 ms │        11.87 / 12.08 ±0.23 / 12.48 ms │     no change │
   │ QQuery 47 │     227.46 / 229.40 ±1.05 / 230.30 ms │     211.65 / 216.12 ±4.46 / 222.15 ms │ +1.06x faster │
   │ QQuery 48 │       95.07 / 96.78 ±2.86 / 102.48 ms │        97.39 / 98.55 ±0.81 / 99.91 ms │     no change │
   │ QQuery 49 │        70.91 / 71.85 ±1.01 / 73.49 ms │        70.92 / 72.68 ±2.59 / 77.77 ms │     no change │
   │ QQuery 50 │        59.05 / 59.59 ±0.56 / 60.55 ms │        60.04 / 61.69 ±2.42 / 66.50 ms │     no change │
   │ QQuery 51 │       91.14 / 95.05 ±3.36 / 100.91 ms │        91.66 / 93.35 ±0.92 / 94.22 ms │     no change │
   │ QQuery 52 │        23.91 / 24.40 ±0.35 / 24.87 ms │        24.04 / 24.36 ±0.22 / 24.60 ms │     no change │
   │ QQuery 53 │        29.16 / 29.48 ±0.21 / 29.81 ms │        29.39 / 30.33 ±1.23 / 32.68 ms │     no change │
   │ QQuery 54 │        54.06 / 54.67 ±0.32 / 54.91 ms │        55.36 / 56.69 ±1.45 / 59.41 ms │     no change │
   │ QQuery 55 │        23.38 / 23.64 ±0.21 / 23.97 ms │        23.65 / 24.00 ±0.38 / 24.65 ms │     no change │
   │ QQuery 56 │        38.95 / 40.66 ±2.56 / 45.72 ms │        39.45 / 40.03 ±0.60 / 41.12 ms │     no change │
   │ QQuery 57 │     175.66 / 177.66 ±1.90 / 180.96 ms │     163.08 / 165.25 ±2.72 / 170.58 ms │ +1.08x faster │
   │ QQuery 58 │     113.67 / 114.69 ±1.20 / 117.02 ms │     108.32 / 111.43 ±2.86 / 116.26 ms │     no change │
   │ QQuery 59 │     117.84 / 119.83 ±2.19 / 123.76 ms │     115.97 / 117.07 ±1.52 / 120.06 ms │     no change │
   │ QQuery 60 │        39.44 / 39.79 ±0.37 / 40.29 ms │        39.60 / 40.31 ±0.41 / 40.80 ms │     no change │
   │ QQuery 61 │        12.10 / 12.75 ±1.13 / 15.01 ms │        12.26 / 12.53 ±0.26 / 12.99 ms │     no change │
   │ QQuery 62 │        46.29 / 46.72 ±0.43 / 47.53 ms │        46.77 / 47.74 ±1.47 / 50.65 ms │     no change │
   │ QQuery 63 │        29.62 / 30.03 ±0.31 / 30.54 ms │        29.43 / 30.85 ±1.67 / 34.08 ms │     no change │
   │ QQuery 64 │     364.39 / 367.41 ±1.85 / 369.72 ms │     376.51 / 382.24 ±4.49 / 389.13 ms │     no change │
   │ QQuery 65 │     122.72 / 125.57 ±2.28 / 128.93 ms │     125.13 / 128.69 ±2.46 / 132.77 ms │     no change │
   │ QQuery 66 │        80.64 / 82.29 ±2.38 / 86.90 ms │        78.98 / 83.01 ±3.74 / 88.64 ms │     no change │
   │ QQuery 67 │     243.75 / 249.20 ±3.94 / 255.10 ms │     248.87 / 255.11 ±4.33 / 260.85 ms │     no change │
   │ QQuery 68 │        11.97 / 13.62 ±3.03 / 19.68 ms │        12.04 / 12.33 ±0.24 / 12.60 ms │ +1.10x faster │
   │ QQuery 69 │        56.25 / 56.76 ±0.55 / 57.83 ms │        57.95 / 58.19 ±0.31 / 58.80 ms │     no change │
   │ QQuery 70 │     104.35 / 106.26 ±1.60 / 108.20 ms │     101.56 / 105.27 ±3.64 / 111.49 ms │     no change │
   │ QQuery 71 │        35.17 / 35.52 ±0.26 / 35.90 ms │        35.90 / 36.30 ±0.59 / 37.46 ms │     no change │
   │ QQuery 72 │ 1840.58 / 1882.18 ±42.35 / 1935.08 ms │ 1761.34 / 1854.04 ±58.70 / 1919.91 ms │     no change │
   │ QQuery 73 │        10.30 / 10.75 ±0.32 / 11.13 ms │          9.63 / 9.85 ±0.22 / 10.12 ms │ +1.09x faster │
   │ QQuery 74 │     170.78 / 173.20 ±2.15 / 176.12 ms │     230.11 / 234.68 ±3.95 / 241.20 ms │  1.35x slower │
   │ QQuery 75 │     145.69 / 149.38 ±2.49 / 152.71 ms │     146.43 / 149.23 ±3.44 / 155.72 ms │     no change │
   │ QQuery 76 │        34.46 / 35.52 ±0.58 / 36.19 ms │        36.07 / 36.35 ±0.33 / 36.96 ms │     no change │
   │ QQuery 77 │        60.89 / 61.71 ±0.62 / 62.71 ms │        62.01 / 62.42 ±0.40 / 63.09 ms │     no change │
   │ QQuery 78 │     221.60 / 226.31 ±3.15 / 230.36 ms │     223.40 / 228.02 ±3.22 / 231.60 ms │     no change │
   │ QQuery 79 │        66.06 / 67.95 ±1.89 / 71.53 ms │        67.51 / 68.81 ±1.50 / 71.70 ms │     no change │
   │ QQuery 80 │      98.48 / 100.48 ±1.71 / 102.61 ms │      99.60 / 104.28 ±5.79 / 115.65 ms │     no change │
   │ QQuery 81 │        25.93 / 26.58 ±0.89 / 28.34 ms │        30.45 / 30.88 ±0.34 / 31.29 ms │  1.16x slower │
   │ QQuery 82 │        16.15 / 16.49 ±0.25 / 16.89 ms │        17.33 / 17.80 ±0.49 / 18.74 ms │  1.08x slower │
   │ QQuery 83 │        34.23 / 35.05 ±0.75 / 36.07 ms │        33.75 / 34.00 ±0.13 / 34.11 ms │     no change │
   │ QQuery 84 │        30.00 / 30.95 ±1.54 / 34.03 ms │        32.01 / 32.24 ±0.13 / 32.39 ms │     no change │
   │ QQuery 85 │     103.72 / 105.15 ±1.32 / 107.47 ms │     105.75 / 109.52 ±3.57 / 115.59 ms │     no change │
   │ QQuery 86 │        25.07 / 26.47 ±1.74 / 29.85 ms │        25.09 / 25.53 ±0.28 / 25.88 ms │     no change │
   │ QQuery 87 │        62.62 / 63.37 ±0.60 / 64.39 ms │        78.51 / 79.21 ±0.53 / 79.86 ms │  1.25x slower │
   │ QQuery 88 │        64.08 / 64.50 ±0.23 / 64.72 ms │        64.73 / 66.43 ±2.05 / 69.90 ms │     no change │
   │ QQuery 89 │        36.18 / 36.94 ±0.61 / 37.97 ms │        36.55 / 36.95 ±0.39 / 37.67 ms │     no change │
   │ QQuery 90 │        17.26 / 17.45 ±0.25 / 17.92 ms │        17.43 / 17.64 ±0.30 / 18.22 ms │     no change │
   │ QQuery 91 │        44.93 / 45.83 ±0.78 / 47.09 ms │        46.22 / 46.73 ±0.27 / 46.99 ms │     no change │
   │ QQuery 92 │        29.59 / 29.80 ±0.16 / 30.07 ms │        29.65 / 29.99 ±0.36 / 30.50 ms │     no change │
   │ QQuery 93 │        50.41 / 52.12 ±1.67 / 54.87 ms │        51.03 / 51.98 ±0.75 / 52.78 ms │     no change │
   │ QQuery 94 │        38.79 / 39.87 ±1.51 / 42.82 ms │        38.55 / 39.90 ±1.22 / 42.15 ms │     no change │
   │ QQuery 95 │        80.53 / 82.41 ±1.09 / 83.64 ms │        80.99 / 81.58 ±0.81 / 83.18 ms │     no change │
   │ QQuery 96 │        24.27 / 24.38 ±0.06 / 24.45 ms │        23.75 / 23.97 ±0.17 / 24.25 ms │     no change │
   │ QQuery 97 │        52.15 / 53.59 ±1.21 / 55.28 ms │        51.65 / 53.16 ±1.28 / 55.11 ms │     no change │
   │ QQuery 98 │        44.16 / 44.99 ±0.82 / 46.48 ms │        38.51 / 39.24 ±0.56 / 39.84 ms │ +1.15x faster │
   │ QQuery 99 │        71.10 / 71.49 ±0.22 / 71.74 ms │        69.69 / 70.23 ±0.41 / 70.74 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 9517.03ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 9978.00ms │
   │ Average Time (HEAD)                                  │   96.13ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  100.79ms │
   │ Queries Faster                                       │         7 │
   │ Queries Slower                                       │        18 │
   │ Queries with No Change                               │        74 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Memory Pool Peaks</summary>
   
   Peak `MemoryPool` reservation per query — what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
   
   Base: `cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f` | Changed: `cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f`
   
   **`tpcds` — `tpcds_sf1`**
   
   | Query | Base | Changed | Change |
   | --- | --- | --- | --- |
   | Query 1 | 128.4 KiB | 64.3 KiB | -50.0% |
   | Query 2 | 83.6 MiB | 80.1 MiB | -4.2% |
   | Query 3 | 4.5 MiB | 2.5 MiB | -44.0% |
   | Query 4 | 154.2 MiB | 83.3 MiB | -46.0% |
   | Query 5 | 12.7 MiB | 11.7 MiB | -8.2% |
   | Query 6 | 7.8 MiB | 4.5 MiB | -42.1% |
   | Query 7 | 9.1 MiB | 3.4 MiB | -62.5% |
   | Query 8 | 2.4 MiB | 1.2 MiB | -50.8% |
   | Query 9 | 32.1 KiB | 32.1 KiB | +0.0% |
   | Query 10 | 4.4 MiB | 2.0 MiB | -54.0% |
   | Query 11 | 101.5 MiB | 55.8 MiB | -45.0% |
   | Query 12 | 131.9 MiB | 116.6 MiB | -11.6% |
   | Query 13 | 17.4 MiB | 11.8 MiB | -31.8% |
   | Query 14 | 160.5 MiB | 160.3 MiB | -0.1% |
   | Query 15 | 6.8 MiB | 4.2 MiB | -37.6% |
   | Query 16 | 161.6 KiB | 66.0 KiB | -59.2% |
   | Query 17 | 37.7 MiB | 37.7 MiB | -0.1% |
   | Query 18 | 21.3 MiB | 18.4 MiB | -13.4% |
   | Query 19 | 7.8 MiB | 2.8 MiB | -64.6% |
   | Query 20 | 132.8 MiB | 126.5 MiB | -4.7% |
   | Query 21 | 5.6 MiB | 2.7 MiB | -52.5% |
   | Query 22 | 91.5 MiB | 59.1 MiB | -35.4% |
   | Query 23 | 216.9 MiB | 210.3 MiB | -3.0% |
   | Query 24 | 46.8 MiB | 32.6 MiB | -30.5% |
   | Query 25 | 79.6 MiB | 79.2 MiB | -0.6% |
   | Query 26 | 8.9 MiB | 3.6 MiB | -59.3% |
   | Query 27 | 1.9 MiB | 1.3 MiB | -28.2% |
   | Query 28 | 1.5 MiB | 1.3 MiB | -9.0% |
   | Query 29 | 91.2 MiB | 91.6 MiB | +0.4% |
   | Query 30 | 6.9 MiB | 3.6 MiB | -48.1% |
   | Query 31 | 32.0 MiB | 21.5 MiB | -33.0% |
   | Query 32 | 2.7 MiB | 2.7 MiB | +0.0% |
   | Query 33 | 6.3 MiB | 4.8 MiB | -24.8% |
   | Query 34 | 7.9 MiB | 2.9 MiB | -62.9% |
   | Query 35 | 25.8 MiB | 16.0 MiB | -37.9% |
   | Query 36 | 10.6 MiB | 10.3 MiB | -2.4% |
   | Query 37 | 3.8 MiB | 3.1 MiB | -17.7% |
   | Query 38 | 20.3 MiB | 9.5 MiB | -53.1% |
   | Query 39 | 44.3 MiB | 54.7 MiB | +23.6% |
   | Query 40 | 13.8 MiB | 10.8 MiB | -21.3% |
   | Query 41 | 2.2 MiB | 1.1 MiB | -48.5% |
   | Query 42 | 2.2 MiB | 1.2 MiB | -45.3% |
   | Query 43 | 192.4 KiB | 96.6 KiB | -49.8% |
   | Query 44 | 192.4 KiB | 96.6 KiB | -49.8% |
   | Query 45 | 10.7 MiB | 5.7 MiB | -46.8% |
   | Query 46 | 8.0 MiB | 4.2 MiB | -47.8% |
   | Query 47 | 163.0 MiB | 129.8 MiB | -20.4% |
   | Query 48 | 15.5 MiB | 9.9 MiB | -35.8% |
   | Query 49 | 51.7 MiB | 43.0 MiB | -17.0% |
   | Query 50 | 13.6 MiB | 13.9 MiB | +2.2% |
   | Query 51 | 177.8 MiB | 178.1 MiB | +0.2% |
   | Query 52 | 3.9 MiB | 1.9 MiB | -51.2% |
   | Query 53 | 30.8 MiB | 30.8 MiB | +0.0% |
   | Query 54 | 3.3 MiB | 1.9 MiB | -41.8% |
   | Query 55 | 3.8 MiB | 1.6 MiB | -56.7% |
   | Query 56 | 5.7 MiB | 3.9 MiB | -30.2% |
   | Query 57 | 156.9 MiB | 128.5 MiB | -18.1% |
   | Query 58 | 6.4 MiB | 4.3 MiB | -33.0% |
   | Query 59 | 24.0 MiB | 20.4 MiB | -14.9% |
   | Query 60 | 7.8 MiB | 6.3 MiB | -19.0% |
   | Query 61 | 1.3 MiB | 1.3 MiB | +0.0% |
   | Query 62 | 7.0 MiB | 4.0 MiB | -42.5% |
   | Query 63 | 20.8 MiB | 20.8 MiB | -0.0% |
   | Query 64 | 86.3 MiB | 93.9 MiB | +8.8% |
   | Query 65 | 30.7 MiB | 21.0 MiB | -31.6% |
   | Query 66 | 12.2 MiB | 13.3 MiB | +9.4% |
   | Query 67 | 517.5 MiB | 141.9 MiB | -72.6% |
   | Query 68 | 8.0 MiB | 4.2 MiB | -47.8% |
   | Query 69 | 47.9 MiB | 3.6 MiB | -92.4% |
   | Query 70 | 51.9 MiB | 39.5 MiB | -23.9% |
   | Query 71 | 52.9 MiB | 41.8 MiB | -21.0% |
   | Query 72 | 72.8 MiB | 75.5 MiB | +3.7% |
   | Query 73 | 7.9 MiB | 2.9 MiB | -62.9% |
   | Query 74 | 57.6 MiB | 33.7 MiB | -41.4% |
   | Query 75 | 24.3 MiB | 23.2 MiB | -4.5% |
   | Query 76 | 8.0 MiB | 6.7 MiB | -16.4% |
   | Query 77 | 3.5 MiB | 3.5 MiB | -0.1% |
   | Query 78 | 137.4 MiB | 137.4 MiB | +0.0% |
   | Query 79 | 10.3 MiB | 8.8 MiB | -14.4% |
   | Query 80 | 42.3 MiB | 41.4 MiB | -2.3% |
   | Query 81 | 17.5 MiB | 5.0 MiB | -71.4% |
   | Query 82 | 3.8 MiB | 3.8 MiB | -0.0% |
   | Query 83 | 11.4 MiB | 8.3 MiB | -27.7% |
   | Query 84 | 10.9 MiB | 10.5 MiB | -3.3% |
   | Query 85 | 20.2 MiB | 15.2 MiB | -24.5% |
   | Query 86 | 105.5 MiB | 82.1 MiB | -22.1% |
   | Query 87 | 20.8 MiB | 12.3 MiB | -40.8% |
   | Query 88 | 2.1 MiB | 1.8 MiB | -12.2% |
   | Query 89 | 130.4 MiB | 112.4 MiB | -13.8% |
   | Query 90 | 619.0 KiB | 619.0 KiB | +0.0% |
   | Query 91 | 22.7 MiB | 30.9 MiB | +36.1% |
   | Query 92 | 2.7 MiB | 2.7 MiB | +0.0% |
   | Query 93 | 13.8 MiB | 14.3 MiB | +3.7% |
   | Query 94 | 3.8 MiB | 3.6 MiB | -7.3% |
   | Query 95 | 14.3 MiB | 14.3 MiB | +0.0% |
   | Query 96 | 224.7 KiB | 224.7 KiB | +0.0% |
   | Query 97 | 32.3 MiB | 32.0 MiB | -0.9% |
   | Query 98 | 133.1 MiB | 116.9 MiB | -12.1% |
   | Query 99 | 7.1 MiB | 1.7 MiB | -76.5% |
   
   **Pool accounting vs. process RSS**
   
   Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
   
   | Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
   | --- | --- | --- | --- | --- | --- |
   | tpcds | base (`cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f`) | 517.5 MiB | 2.2 GiB | 1.7 GiB | 4.3× |
   | tpcds | changed (`cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f`) | 210.3 MiB | 1.8 GiB | 1.6 GiB | 8.8× |
   
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 2.2 GiB |
   | Avg memory | 1.5 GiB |
   | CPU user | 201.6s |
   | CPU sys | 5.7s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 1.8 GiB |
   | Avg memory | 1.3 GiB |
   | CPU user | 209.1s |
   | CPU sys | 5.9s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r4119052171


##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -84,6 +84,183 @@ pub fn apply_file_schema_type_coercions(
     ))
 }
 
+/// Like [`apply_file_schema_type_coercions`], but also coerces compatible
+/// string/binary file fields to dictionary types already present in the table
+/// schema.
+pub(crate) fn apply_file_schema_type_coercions_with_rle(
+    table_schema: &Schema,
+    file_schema: &Schema,
+    enable_rle_to_dictionary: bool,
+) -> Option<Schema> {
+    let mut needs_view_transform = false;
+    let mut needs_string_transform = false;
+    let mut needs_nested_transform = false;
+    let mut needs_dict_transform = false;
+
+    // Create a mapping of table field names to their data types for fast lookup
+    // and simultaneously check if we need any transformations
+    let table_fields: HashMap<_, _> = table_schema
+        .fields()
+        .iter()
+        .map(|field| {
+            let data_type = field.data_type();
+            // Check if we need view type transformation
+            if matches!(data_type, &DataType::Utf8View | &DataType::BinaryView) {
+                needs_view_transform = true;
+            }
+            // Check if we need string type transformation
+            if matches!(
+                data_type,
+                &DataType::Utf8 | &DataType::LargeUtf8 | &DataType::Utf8View
+            ) {
+                needs_string_transform = true;
+            }
+            // Nested fields can need transformations even when their parent does not.
+            if matches!(
+                data_type,
+                DataType::Struct(_)
+                    | DataType::List(_)
+                    | DataType::LargeList(_)
+                    | DataType::ListView(_)
+                    | DataType::LargeListView(_)
+                    | DataType::FixedSizeList(_, _)
+                    | DataType::Map(_, _)
+            ) {
+                needs_nested_transform = true;
+            }
+            if enable_rle_to_dictionary
+                && matches!(data_type, &DataType::Dictionary(_, _))
+            {
+                needs_dict_transform = true;
+            }
+
+            (field.name(), data_type)
+        })
+        .collect();
+
+    // Early return if no transformation needed
+    if !needs_view_transform
+        && !needs_string_transform
+        && !needs_nested_transform
+        && !needs_dict_transform
+    {
+        return None;
+    }
+
+    let fields: Vec<Arc<Field>> = file_schema

Review Comment:
   This allocates a new vector and clones every field for common string or nested schemas. It also does this when `enable_rle_to_dictionary` is false and no field changes. The existing `coerce_fields` helper allocates only after the first change. Please add dictionary promotion to the existing coercion path instead of copying the full implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5285046082

   @adriangb your comments are addressed in https://github.com/apache/datafusion/pull/24227/commits/8bc183beaf0a59fcd5566598ee436910adb54c7f 
   Previously the promotion only applied when the schema was inferred via `CREATE EXTERNAL TABLE` (no explicit columns). Explicit schemas (`CREATE EXTERNAL TABLE (col VARCHAR)`) and custom `TableProvider` implementations never touched `infer_schema` so the flag had no effect on those paths.
   
   `ParquetSource::with_table_parquet_options` now promotes all string/binary fields to `Dictionary(Int32, …)` at planning time regardless of how the schema was established, since every `ParquetSource` passes through this method.
   
   To keep planning and execution in sync, the opener also rewrites the physical read schema so arrow-rs produces dict arrays. Without the planning-time promotion, downstream operators (FilterExec, AggregateExec) would be compiled against `Utf8` while the scan produces `Dictionary(Int32, Utf8)` — a type mismatch at execution time.
   
   
   ## test
   Added tests covering:
   - Unit tests in `datasource-parquet`: flag propagation, RLE-encoded and plain-encoded files with the flag on/off
   - Integration tests in `core/tests/parquet/schema_coercion.rs`: scan → filter → project using `ParquetSource` directly (no `infer_schema`) to validate the explicit-schema path; end-to-end scan → filter → aggregate via `SessionContext` and SQL to validate the full planning pipeline
   - SLT (`parquet_rle_to_dictionary.slt`): `arrow_typeof` assertions with the flag on/off, GROUP BY over dict columns to confirm aggregation works correctly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5533719016

   
   thank you for the review @kumarUjjawal, the latest commit should address your comments.
   
   > Adding enable_rle_to_dictionary to public ParquetOptions breaks downstream exhaustive struct literals. I think we should mention it in the upgrade guide.
   
   im a little confused by what you mean by this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3878263400


##########
datafusion/datasource-parquet/src/file_format.rs:
##########
@@ -210,6 +213,22 @@ impl ParquetFormat {
         &self.options
     }
 
+    /// Restrict RLE->Dictionary promotion to a named subset of columns.
+    ///
+    /// Only columns in `columns` that also have dictionary pages in the file
+    /// are promoted to `Dictionary(Int32, …)` in the inferred schema.
+    /// Overrides `enable_rle_to_dictionary` when set; an empty set disables
+    /// promotion entirely. See <https://github.com/apache/datafusion/issues/24113>.
+    pub fn with_rle_column_allowlist(mut self, columns: HashSet<String>) -> Self {

Review Comment:
   this makes sense to me, less things to introduce in 1 PR the easier it will be to catch bugs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513429687

   run benchmarks 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513470137

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513429687)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5513429687-2084-mzjxp 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (896eebe1dee3bd2845c6f6ffec1ecda7be808125) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..896eebe1dee3bd2845c6f6ffec1ecda7be808125)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513887813

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513845153)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5513845153-2096-hchcs 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (6166c31c21bdd7965f348ae7a126a2d1e85a6216) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..6166c31c21bdd7965f348ae7a126a2d1e85a6216)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513891168

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513845153)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5513845153-2097-kz4ls 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (6166c31c21bdd7965f348ae7a126a2d1e85a6216) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..6166c31c21bdd7965f348ae7a126a2d1e85a6216)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513422947

   ## following results are `enable_rle_to_dictionary` = `true`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364898372

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364868497)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5364868497-1831-jgmcr 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f09f1555c30e2f0b88db32a80d7bcd87062bcc5d) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f09f1555c30e2f0b88db32a80d7bcd87062bcc5d)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5345983251

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5345832494)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (449355e7c5098104ce4868fa3ce347ca95a24ce3) to bbf739a (merge-base) [diff](https://github.com/apache/datafusion/compare/bbf739a3cfe1d5cc7646e37250c947c6a56b5b0e..449355e7c5098104ce4868fa3ce347ca95a24ce3)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch10
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf10.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃      HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │ 310.35 ms │                             304.48 ms │     no change │
   │ QQuery 2  │ 102.58 ms │                             134.80 ms │  1.31x slower │
   │ QQuery 3  │ 224.12 ms │                             229.31 ms │     no change │
   │ QQuery 4  │ 112.19 ms │                             113.78 ms │     no change │
   │ QQuery 5  │ 345.56 ms │                             348.76 ms │     no change │
   │ QQuery 6  │ 123.62 ms │                             121.53 ms │     no change │
   │ QQuery 7  │ 466.80 ms │                             461.40 ms │     no change │
   │ QQuery 8  │ 379.99 ms │                             376.72 ms │     no change │
   │ QQuery 9  │ 550.32 ms │                             557.28 ms │     no change │
   │ QQuery 10 │ 302.01 ms │                             619.30 ms │  2.05x slower │
   │ QQuery 11 │  84.78 ms │                              87.48 ms │     no change │
   │ QQuery 12 │ 177.53 ms │                             159.91 ms │ +1.11x faster │
   │ QQuery 13 │ 280.78 ms │                             370.17 ms │  1.32x slower │
   │ QQuery 14 │ 167.03 ms │                             170.45 ms │     no change │
   │ QQuery 15 │ 300.22 ms │                             308.60 ms │     no change │
   │ QQuery 16 │  64.56 ms │                              64.19 ms │     no change │
   │ QQuery 17 │ 613.74 ms │                             612.26 ms │     no change │
   │ QQuery 18 │ 687.08 ms │                            1546.60 ms │  2.25x slower │
   │ QQuery 19 │ 241.04 ms │                             231.04 ms │     no change │
   │ QQuery 20 │ 278.84 ms │                             286.57 ms │     no change │
   │ QQuery 21 │ 649.45 ms │                             625.01 ms │     no change │
   │ QQuery 22 │  58.28 ms │                              66.21 ms │  1.14x slower │
   └───────────┴───────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 6520.87ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 7795.86ms │
   │ Average Time (HEAD)                                  │  296.40ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  354.36ms │
   │ Queries Faster                                       │         1 │
   │ Queries Slower                                       │         5 │
   │ Queries with No Change                               │        16 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf10.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                               HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │  310.35 / 312.60 ±1.37 / 314.64 ms │     304.48 / 305.24 ±0.75 / 306.66 ms │     no change │
   │ QQuery 2  │  102.58 / 106.60 ±3.19 / 111.81 ms │     134.80 / 137.54 ±1.55 / 139.10 ms │  1.29x slower │
   │ QQuery 3  │  224.12 / 230.67 ±4.67 / 236.13 ms │     229.31 / 237.00 ±5.92 / 242.84 ms │     no change │
   │ QQuery 4  │  112.19 / 113.47 ±1.00 / 115.20 ms │     113.78 / 114.44 ±0.52 / 115.29 ms │     no change │
   │ QQuery 5  │  345.56 / 353.20 ±8.00 / 366.85 ms │     348.76 / 357.82 ±7.86 / 371.87 ms │     no change │
   │ QQuery 6  │  123.62 / 125.56 ±2.52 / 130.47 ms │     121.53 / 123.08 ±1.17 / 124.69 ms │     no change │
   │ QQuery 7  │  466.80 / 474.46 ±8.06 / 486.49 ms │     461.40 / 475.14 ±9.45 / 488.11 ms │     no change │
   │ QQuery 8  │  379.99 / 386.54 ±5.38 / 394.10 ms │     376.72 / 379.53 ±2.39 / 382.20 ms │     no change │
   │ QQuery 9  │  550.32 / 555.98 ±4.85 / 562.69 ms │     557.28 / 566.44 ±7.55 / 576.93 ms │     no change │
   │ QQuery 10 │  302.01 / 310.20 ±4.85 / 317.16 ms │     619.30 / 635.77 ±9.40 / 645.96 ms │  2.05x slower │
   │ QQuery 11 │     84.78 / 90.35 ±5.03 / 96.46 ms │        87.48 / 91.33 ±4.31 / 99.44 ms │     no change │
   │ QQuery 12 │  177.53 / 178.61 ±0.61 / 179.21 ms │     159.91 / 167.14 ±7.38 / 180.83 ms │ +1.07x faster │
   │ QQuery 13 │ 280.78 / 296.72 ±13.40 / 320.50 ms │    370.17 / 391.30 ±19.47 / 423.12 ms │  1.32x slower │
   │ QQuery 14 │  167.03 / 173.45 ±5.44 / 181.34 ms │     170.45 / 171.77 ±1.04 / 173.57 ms │     no change │
   │ QQuery 15 │  300.22 / 303.68 ±2.97 / 307.28 ms │     308.60 / 310.75 ±2.12 / 314.38 ms │     no change │
   │ QQuery 16 │     64.56 / 68.06 ±5.21 / 78.41 ms │        64.19 / 67.63 ±4.82 / 77.10 ms │     no change │
   │ QQuery 17 │  613.74 / 622.09 ±9.00 / 637.06 ms │     612.26 / 622.66 ±7.72 / 631.41 ms │     no change │
   │ QQuery 18 │  687.08 / 698.88 ±8.74 / 708.99 ms │ 1546.60 / 1599.66 ±40.22 / 1667.96 ms │  2.29x slower │
   │ QQuery 19 │ 241.04 / 257.11 ±22.23 / 300.12 ms │     231.04 / 243.81 ±9.85 / 257.80 ms │ +1.05x faster │
   │ QQuery 20 │  278.84 / 283.75 ±3.46 / 288.15 ms │     286.57 / 290.91 ±2.60 / 294.72 ms │     no change │
   │ QQuery 21 │ 649.45 / 660.99 ±11.13 / 677.41 ms │     625.01 / 634.20 ±8.65 / 649.15 ms │     no change │
   │ QQuery 22 │     58.28 / 62.70 ±4.01 / 68.06 ms │        66.21 / 68.08 ±1.64 / 70.47 ms │  1.09x slower │
   └───────────┴────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 6665.69ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 7991.22ms │
   │ Average Time (HEAD)                                  │  302.99ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  363.24ms │
   │ Queries Faster                                       │         2 │
   │ Queries Slower                                       │         5 │
   │ Queries with No Change                               │        15 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch10 — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 35.0s |
   | Peak memory | 4.8 GiB |
   | Avg memory | 1.5 GiB |
   | CPU user | 339.1s |
   | CPU sys | 18.5s |
   | Peak spill | 0 B |
   
   **tpch10 — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 45.0s |
   | Peak memory | 2.8 GiB |
   | Avg memory | 1.2 GiB |
   | CPU user | 354.1s |
   | CPU sys | 19.9s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5400020690

   run benchmarks 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5370432706

   TLDR: the current state of this PR is that it enables direct reads of RLE_DICTIONARY-encoded Parquet columns into dictionary arrays. The main issue right now is a performance regression on some queries, since the flag currently promotes every variable-length column to a dictionary type regardless of cardinality, causing high cardinality columns (5-15 million) to degrade query performance. I still think it's worth keeping the flag: it lets us benchmark dictionary-encoding-related changes, as @alamb pointed out [here](https://github.com/apache/datafusion/pull/24418#pullrequestreview-4965416771), and this PR lays the groundwork for the follow-up PRs in #24111 that will add fine-grained, per-column tuning. The flag will also be turned off by default
   
   
   @alamb @adriangb 
   the PR isn't ready for review I'm more so checking that this is the correct direction to be moving in


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5675321414

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5675290885)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5675290885-2366-xftgp 6.12.94+ #1 SMP Tue Aug  4 08:44:15 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f (cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f) to cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f [diff](https://github.com/apache/datafusion/compare/cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f..cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   env:
     DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
   baseline:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
   changed:
     ref: "cb3a57806e64fa8d95a5ec2e226d34ba5a97af3f"
     env:
       DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-6000992664

   thanks @kumarUjjawal 🙇 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5999996004

   Thank you @Rich-T-kid 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513805728

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513429687)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (896eebe1dee3bd2845c6f6ffec1ecda7be808125) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..896eebe1dee3bd2845c6f6ffec1ecda7be808125)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃       Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
   │ QQuery 0  │    1.29 ms │                               1.26 ms │    no change │
   │ QQuery 1  │   12.23 ms │                              12.01 ms │    no change │
   │ QQuery 2  │   37.64 ms │                              36.62 ms │    no change │
   │ QQuery 3  │   32.67 ms │                              31.80 ms │    no change │
   │ QQuery 4  │  249.52 ms │                             240.65 ms │    no change │
   │ QQuery 5  │  292.67 ms │                            1470.39 ms │ 5.02x slower │
   │ QQuery 6  │    1.34 ms │                               1.30 ms │    no change │
   │ QQuery 7  │   14.49 ms │                              14.17 ms │    no change │
   │ QQuery 8  │  363.25 ms │                             372.16 ms │    no change │
   │ QQuery 9  │  499.50 ms │                             524.22 ms │    no change │
   │ QQuery 10 │   74.92 ms │                              76.21 ms │    no change │
   │ QQuery 11 │   86.20 ms │                              87.83 ms │    no change │
   │ QQuery 12 │  294.39 ms │                            1383.87 ms │ 4.70x slower │
   │ QQuery 13 │  402.40 ms │                            2450.72 ms │ 6.09x slower │
   │ QQuery 14 │  308.90 ms │                            1557.05 ms │ 5.04x slower │
   │ QQuery 15 │  307.47 ms │                             309.82 ms │    no change │
   │ QQuery 16 │  662.11 ms │                            3766.12 ms │ 5.69x slower │
   │ QQuery 17 │  669.33 ms │                            3807.81 ms │ 5.69x slower │
   │ QQuery 18 │ 1346.29 ms │                            6703.73 ms │ 4.98x slower │
   │ QQuery 19 │   28.97 ms │                              28.35 ms │    no change │
   │ QQuery 20 │  524.49 ms │                            1069.84 ms │ 2.04x slower │
   │ QQuery 21 │  520.61 ms │                            1090.47 ms │ 2.09x slower │
   │ QQuery 22 │ 1011.48 ms │                            2444.99 ms │ 2.42x slower │
   │ QQuery 23 │ 3187.07 ms │                            4788.00 ms │ 1.50x slower │
   │ QQuery 24 │   41.89 ms │                              64.05 ms │ 1.53x slower │
   │ QQuery 25 │  114.66 ms │                             193.50 ms │ 1.69x slower │
   │ QQuery 26 │   42.23 ms │                              65.17 ms │ 1.54x slower │
   │ QQuery 27 │  524.04 ms │                                  FAIL │ incomparable │
   │ QQuery 28 │ 3045.73 ms │                                  FAIL │ incomparable │
   │ QQuery 29 │   42.09 ms │                              41.79 ms │    no change │
   │ QQuery 30 │  322.94 ms │                             383.73 ms │ 1.19x slower │
   │ QQuery 31 │  305.36 ms │                             365.33 ms │ 1.20x slower │
   │ QQuery 32 │ 1023.43 ms │                            1094.86 ms │ 1.07x slower │
   │ QQuery 33 │ 1578.98 ms │                           14363.04 ms │ 9.10x slower │
   │ QQuery 34 │ 1616.70 ms │                           14191.49 ms │ 8.78x slower │
   │ QQuery 35 │  313.78 ms │                             312.46 ms │    no change │
   │ QQuery 36 │   68.67 ms │                             179.55 ms │ 2.61x slower │
   │ QQuery 37 │   37.11 ms │                              53.30 ms │ 1.44x slower │
   │ QQuery 38 │   43.95 ms │                              72.40 ms │ 1.65x slower │
   │ QQuery 39 │  153.58 ms │                             301.06 ms │ 1.96x slower │
   │ QQuery 40 │   14.81 ms │                              14.93 ms │    no change │
   │ QQuery 41 │   14.46 ms │                              14.75 ms │    no change │
   │ QQuery 42 │   14.30 ms │                              14.15 ms │    no change │
   └───────────┴────────────┴───────────────────────────────────────┴──────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 16678.17ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 63994.95ms │
   │ Average Time (HEAD)                                  │   406.78ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  1560.85ms │
   │ Queries Faster                                       │          0 │
   │ Queries Slower                                       │         23 │
   │ Queries with No Change                               │         18 │
   │ Queries with Failure                                 │          2 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃     rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │          1.29 / 4.20 ±5.71 / 15.63 ms │              1.26 / 4.17 ±5.65 / 15.48 ms │     no change │
   │ QQuery 1  │        12.23 / 13.01 ±0.44 / 13.52 ms │            12.01 / 12.58 ±0.32 / 12.86 ms │     no change │
   │ QQuery 2  │        37.64 / 37.93 ±0.23 / 38.29 ms │            36.62 / 36.91 ±0.21 / 37.20 ms │     no change │
   │ QQuery 3  │        32.67 / 33.32 ±0.89 / 35.07 ms │            31.80 / 32.13 ±0.33 / 32.59 ms │     no change │
   │ QQuery 4  │     249.52 / 254.26 ±3.21 / 258.96 ms │         240.65 / 245.69 ±3.21 / 248.63 ms │     no change │
   │ QQuery 5  │     292.67 / 297.71 ±4.37 / 303.98 ms │     1470.39 / 1561.32 ±55.34 / 1625.33 ms │  5.24x slower │
   │ QQuery 6  │           1.34 / 1.49 ±0.22 / 1.93 ms │               1.30 / 1.47 ±0.25 / 1.95 ms │     no change │
   │ QQuery 7  │        14.49 / 14.56 ±0.08 / 14.71 ms │            14.17 / 18.91 ±9.28 / 37.47 ms │  1.30x slower │
   │ QQuery 8  │     363.25 / 368.18 ±3.69 / 374.47 ms │         372.16 / 382.21 ±5.43 / 387.75 ms │     no change │
   │ QQuery 9  │    499.50 / 522.82 ±18.70 / 551.72 ms │         524.22 / 531.73 ±9.53 / 550.25 ms │     no change │
   │ QQuery 10 │        74.92 / 75.76 ±0.48 / 76.40 ms │           76.21 / 81.87 ±9.18 / 100.10 ms │  1.08x slower │
   │ QQuery 11 │        86.20 / 87.13 ±0.90 / 88.42 ms │            87.83 / 90.15 ±3.80 / 97.73 ms │     no change │
   │ QQuery 12 │     294.39 / 302.09 ±4.91 / 307.80 ms │     1383.87 / 1466.98 ±66.20 / 1562.23 ms │  4.86x slower │
   │ QQuery 13 │     402.40 / 409.26 ±5.30 / 416.46 ms │    2450.72 / 2896.53 ±441.53 / 3539.04 ms │  7.08x slower │
   │ QQuery 14 │     308.90 / 316.62 ±5.04 / 323.77 ms │     1557.05 / 1611.24 ±47.42 / 1670.77 ms │  5.09x slower │
   │ QQuery 15 │     307.47 / 318.41 ±9.90 / 332.52 ms │         309.82 / 317.08 ±5.86 / 326.30 ms │     no change │
   │ QQuery 16 │     662.11 / 673.16 ±8.52 / 686.44 ms │     3766.12 / 3903.76 ±92.88 / 4013.20 ms │  5.80x slower │
   │ QQuery 17 │     669.33 / 679.36 ±9.82 / 695.16 ms │     3807.81 / 3893.64 ±49.29 / 3958.90 ms │  5.73x slower │
   │ QQuery 18 │ 1346.29 / 1392.27 ±33.84 / 1425.54 ms │    6703.73 / 7007.51 ±194.14 / 7289.01 ms │  5.03x slower │
   │ QQuery 19 │        28.97 / 33.68 ±6.92 / 47.46 ms │            28.35 / 28.81 ±0.44 / 29.62 ms │ +1.17x faster │
   │ QQuery 20 │    524.49 / 537.88 ±10.24 / 549.91 ms │     1069.84 / 1099.25 ±40.82 / 1180.08 ms │  2.04x slower │
   │ QQuery 21 │     520.61 / 532.87 ±9.43 / 544.86 ms │     1090.47 / 1108.95 ±18.68 / 1136.11 ms │  2.08x slower │
   │ QQuery 22 │  1011.48 / 1026.45 ±9.97 / 1039.98 ms │     2444.99 / 2501.73 ±36.20 / 2539.50 ms │  2.44x slower │
   │ QQuery 23 │  3187.07 / 3202.86 ±8.09 / 3208.90 ms │    4788.00 / 4931.46 ±177.10 / 5272.95 ms │  1.54x slower │
   │ QQuery 24 │       41.89 / 54.08 ±19.00 / 91.42 ms │           64.05 / 74.65 ±12.31 / 90.55 ms │  1.38x slower │
   │ QQuery 25 │     114.66 / 117.50 ±4.41 / 126.05 ms │         193.50 / 196.26 ±2.43 / 199.52 ms │  1.67x slower │
   │ QQuery 26 │        42.23 / 42.63 ±0.33 / 43.07 ms │            65.17 / 65.97 ±0.86 / 67.30 ms │  1.55x slower │
   │ QQuery 27 │     524.04 / 531.44 ±5.38 / 539.62 ms │                                      FAIL │  incomparable │
   │ QQuery 28 │ 3045.73 / 3070.33 ±25.82 / 3108.16 ms │                                      FAIL │  incomparable │
   │ QQuery 29 │       42.09 / 56.75 ±14.27 / 80.80 ms │            41.79 / 42.09 ±0.20 / 42.40 ms │ +1.35x faster │
   │ QQuery 30 │     322.94 / 332.23 ±6.69 / 343.01 ms │         383.73 / 390.73 ±6.51 / 398.92 ms │  1.18x slower │
   │ QQuery 31 │    305.36 / 321.05 ±18.87 / 358.13 ms │         365.33 / 376.71 ±6.83 / 385.20 ms │  1.17x slower │
   │ QQuery 32 │ 1023.43 / 1057.61 ±26.39 / 1093.94 ms │     1094.86 / 1169.75 ±63.00 / 1257.53 ms │  1.11x slower │
   │ QQuery 33 │ 1578.98 / 1598.55 ±16.42 / 1625.92 ms │ 14363.04 / 14720.05 ±368.88 / 15399.07 ms │  9.21x slower │
   │ QQuery 34 │ 1616.70 / 1649.83 ±40.29 / 1728.72 ms │ 14191.49 / 14720.67 ±269.70 / 14913.30 ms │  8.92x slower │
   │ QQuery 35 │     313.78 / 322.87 ±5.63 / 330.28 ms │        312.46 / 328.80 ±21.86 / 371.75 ms │     no change │
   │ QQuery 36 │        68.67 / 76.28 ±5.27 / 82.76 ms │        179.55 / 192.06 ±11.88 / 212.79 ms │  2.52x slower │
   │ QQuery 37 │        37.11 / 39.34 ±2.57 / 43.43 ms │            53.30 / 58.79 ±5.16 / 67.46 ms │  1.49x slower │
   │ QQuery 38 │        43.95 / 46.49 ±2.43 / 49.58 ms │            72.40 / 80.77 ±5.90 / 90.61 ms │  1.74x slower │
   │ QQuery 39 │     153.58 / 165.96 ±6.99 / 172.47 ms │        301.06 / 326.19 ±15.67 / 342.21 ms │  1.97x slower │
   │ QQuery 40 │        14.81 / 15.68 ±0.87 / 17.35 ms │            14.93 / 15.75 ±0.71 / 17.00 ms │     no change │
   │ QQuery 41 │        14.46 / 15.81 ±1.95 / 19.66 ms │            14.75 / 16.54 ±3.13 / 22.78 ms │     no change │
   │ QQuery 42 │        14.30 / 16.50 ±3.72 / 23.89 ms │            14.15 / 16.45 ±4.05 / 24.53 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 17064.43ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 66558.29ms │
   │ Average Time (HEAD)                                  │   416.21ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  1623.37ms │
   │ Queries Faster                                       │          2 │
   │ Queries Slower                                       │         25 │
   │ Queries with No Change                               │         14 │
   │ Queries with Failure                                 │          2 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 105.0s |
   | Peak memory | 10.3 GiB |
   | Avg memory | 4.4 GiB |
   | CPU user | 1054.5s |
   | CPU sys | 80.1s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 335.1s |
   | Peak memory | 13.4 GiB |
   | Avg memory | 6.2 GiB |
   | CPU user | 2736.9s |
   | CPU sys | 206.4s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513829469

   ## following results are `enable_rle_to_dictionary` = `false`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513455669

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513429687)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5513429687-2083-kpqnf 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (896eebe1dee3bd2845c6f6ffec1ecda7be808125) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..896eebe1dee3bd2845c6f6ffec1ecda7be808125)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5513462391

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513429687)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5513429687-2085-x7mb7 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (896eebe1dee3bd2845c6f6ffec1ecda7be808125) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..896eebe1dee3bd2845c6f6ffec1ecda7be808125)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5514148671

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5513845153)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (6166c31c21bdd7965f348ae7a126a2d1e85a6216) to d2b626c (merge-base) [diff](https://github.com/apache/datafusion/compare/d2b626cc93616b5bf80b7ca2a079e9859d992e32..6166c31c21bdd7965f348ae7a126a2d1e85a6216)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │    1.24 ms │                               1.25 ms │     no change │
   │ QQuery 1  │   12.07 ms │                              11.84 ms │     no change │
   │ QQuery 2  │   36.59 ms │                              36.20 ms │     no change │
   │ QQuery 3  │   31.66 ms │                              30.61 ms │     no change │
   │ QQuery 4  │  229.16 ms │                             222.83 ms │     no change │
   │ QQuery 5  │  275.77 ms │                             272.01 ms │     no change │
   │ QQuery 6  │    1.25 ms │                               1.26 ms │     no change │
   │ QQuery 7  │   13.08 ms │                              13.15 ms │     no change │
   │ QQuery 8  │  330.80 ms │                             328.46 ms │     no change │
   │ QQuery 9  │  453.76 ms │                             465.10 ms │     no change │
   │ QQuery 10 │   69.02 ms │                              75.01 ms │  1.09x slower │
   │ QQuery 11 │   79.85 ms │                              88.90 ms │  1.11x slower │
   │ QQuery 12 │  263.29 ms │                             307.71 ms │  1.17x slower │
   │ QQuery 13 │  360.09 ms │                             420.03 ms │  1.17x slower │
   │ QQuery 14 │  283.81 ms │                             322.20 ms │  1.14x slower │
   │ QQuery 15 │  270.45 ms │                             328.74 ms │  1.22x slower │
   │ QQuery 16 │  614.57 ms │                             628.11 ms │     no change │
   │ QQuery 17 │  623.68 ms │                             634.21 ms │     no change │
   │ QQuery 18 │ 1281.84 ms │                            1275.38 ms │     no change │
   │ QQuery 19 │   28.83 ms │                              27.34 ms │ +1.05x faster │
   │ QQuery 20 │  526.74 ms │                             517.35 ms │     no change │
   │ QQuery 21 │  513.68 ms │                             516.75 ms │     no change │
   │ QQuery 22 │  983.28 ms │                             989.84 ms │     no change │
   │ QQuery 23 │ 3017.23 ms │                            3062.20 ms │     no change │
   │ QQuery 24 │   43.30 ms │                              41.06 ms │ +1.05x faster │
   │ QQuery 25 │  115.39 ms │                             111.91 ms │     no change │
   │ QQuery 26 │   43.42 ms │                              41.74 ms │     no change │
   │ QQuery 27 │  544.97 ms │                             514.74 ms │ +1.06x faster │
   │ QQuery 28 │ 2926.06 ms │                            2894.56 ms │     no change │
   │ QQuery 29 │   42.05 ms │                              42.74 ms │     no change │
   │ QQuery 30 │  305.27 ms │                             300.98 ms │     no change │
   │ QQuery 31 │  279.93 ms │                             283.12 ms │     no change │
   │ QQuery 32 │  936.19 ms │                             927.75 ms │     no change │
   │ QQuery 33 │ 1450.38 ms │                            1456.72 ms │     no change │
   │ QQuery 34 │ 1526.27 ms │                            1509.53 ms │     no change │
   │ QQuery 35 │  283.89 ms │                             273.63 ms │     no change │
   │ QQuery 36 │   67.96 ms │                              70.78 ms │     no change │
   │ QQuery 37 │   35.41 ms │                              35.10 ms │     no change │
   │ QQuery 38 │   41.77 ms │                              40.46 ms │     no change │
   │ QQuery 39 │  149.12 ms │                             138.35 ms │ +1.08x faster │
   │ QQuery 40 │   13.68 ms │                              13.95 ms │     no change │
   │ QQuery 41 │   13.43 ms │                              13.62 ms │     no change │
   │ QQuery 42 │   12.81 ms │                              13.00 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 19133.04ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 19300.22ms │
   │ Average Time (HEAD)                                  │   444.95ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   448.84ms │
   │ Queries Faster                                       │          4 │
   │ Queries Slower                                       │          6 │
   │ Queries with No Change                               │         33 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                   HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │           1.24 / 4.20 ±5.76 / 15.71 ms │          1.25 / 3.99 ±5.41 / 14.81 ms │     no change │
   │ QQuery 1  │         12.07 / 12.35 ±0.18 / 12.60 ms │        11.84 / 12.04 ±0.16 / 12.24 ms │     no change │
   │ QQuery 2  │         36.59 / 36.99 ±0.33 / 37.44 ms │        36.20 / 36.88 ±0.89 / 38.64 ms │     no change │
   │ QQuery 3  │         31.66 / 32.40 ±0.88 / 34.01 ms │        30.61 / 31.21 ±0.59 / 32.25 ms │     no change │
   │ QQuery 4  │      229.16 / 237.41 ±9.53 / 250.93 ms │     222.83 / 228.44 ±2.96 / 231.02 ms │     no change │
   │ QQuery 5  │      275.77 / 277.44 ±1.32 / 279.48 ms │     272.01 / 276.16 ±2.16 / 278.40 ms │     no change │
   │ QQuery 6  │            1.25 / 1.40 ±0.23 / 1.86 ms │           1.26 / 1.41 ±0.21 / 1.83 ms │     no change │
   │ QQuery 7  │         13.08 / 13.39 ±0.16 / 13.52 ms │        13.15 / 13.27 ±0.08 / 13.40 ms │     no change │
   │ QQuery 8  │      330.80 / 334.32 ±1.88 / 336.20 ms │     328.46 / 332.14 ±2.56 / 334.73 ms │     no change │
   │ QQuery 9  │      453.76 / 464.42 ±9.82 / 482.63 ms │    465.10 / 486.20 ±19.02 / 512.16 ms │     no change │
   │ QQuery 10 │         69.02 / 69.78 ±0.77 / 71.24 ms │        75.01 / 77.09 ±3.19 / 83.42 ms │  1.10x slower │
   │ QQuery 11 │         79.85 / 81.15 ±0.94 / 82.63 ms │        88.90 / 91.55 ±3.75 / 98.95 ms │  1.13x slower │
   │ QQuery 12 │      263.29 / 269.66 ±4.10 / 274.16 ms │     307.71 / 319.80 ±9.95 / 337.49 ms │  1.19x slower │
   │ QQuery 13 │     360.09 / 382.27 ±15.77 / 406.68 ms │     420.03 / 431.00 ±9.91 / 446.23 ms │  1.13x slower │
   │ QQuery 14 │     283.81 / 309.79 ±13.73 / 321.61 ms │     322.20 / 326.87 ±3.10 / 331.62 ms │  1.06x slower │
   │ QQuery 15 │     270.45 / 286.21 ±15.90 / 313.78 ms │    328.74 / 342.45 ±14.13 / 359.72 ms │  1.20x slower │
   │ QQuery 16 │      614.57 / 626.82 ±8.88 / 641.74 ms │    628.11 / 690.67 ±35.82 / 730.64 ms │  1.10x slower │
   │ QQuery 17 │      623.68 / 638.28 ±9.64 / 648.70 ms │    634.21 / 664.43 ±24.24 / 705.34 ms │     no change │
   │ QQuery 18 │  1281.84 / 1306.44 ±14.78 / 1323.41 ms │ 1275.38 / 1296.97 ±17.02 / 1323.49 ms │     no change │
   │ QQuery 19 │         28.83 / 31.90 ±3.09 / 37.53 ms │        27.34 / 31.15 ±6.62 / 44.33 ms │     no change │
   │ QQuery 20 │      526.74 / 537.31 ±6.16 / 544.27 ms │     517.35 / 524.68 ±3.88 / 528.64 ms │     no change │
   │ QQuery 21 │      513.68 / 521.23 ±7.69 / 532.51 ms │     516.75 / 534.36 ±9.59 / 545.49 ms │     no change │
   │ QQuery 22 │      983.28 / 987.96 ±2.85 / 991.29 ms │    989.84 / 998.23 ±5.14 / 1005.93 ms │     no change │
   │ QQuery 23 │ 3017.23 / 3180.58 ±116.30 / 3314.16 ms │ 3062.20 / 3144.65 ±55.46 / 3228.96 ms │     no change │
   │ QQuery 24 │         43.30 / 49.19 ±9.93 / 69.03 ms │        41.06 / 44.20 ±4.52 / 53.16 ms │ +1.11x faster │
   │ QQuery 25 │      115.39 / 117.74 ±1.43 / 119.30 ms │     111.91 / 115.52 ±5.52 / 126.53 ms │     no change │
   │ QQuery 26 │         43.42 / 46.78 ±4.25 / 55.15 ms │        41.74 / 42.90 ±1.47 / 45.73 ms │ +1.09x faster │
   │ QQuery 27 │      544.97 / 557.64 ±9.53 / 573.37 ms │     514.74 / 519.94 ±4.06 / 527.14 ms │ +1.07x faster │
   │ QQuery 28 │  2926.06 / 2949.52 ±21.11 / 2983.44 ms │ 2894.56 / 3003.22 ±87.81 / 3096.49 ms │     no change │
   │ QQuery 29 │         42.05 / 42.56 ±0.51 / 43.53 ms │       42.74 / 61.47 ±23.12 / 95.77 ms │  1.44x slower │
   │ QQuery 30 │     305.27 / 338.84 ±27.28 / 380.81 ms │    300.98 / 325.51 ±31.12 / 385.56 ms │     no change │
   │ QQuery 31 │     279.93 / 295.84 ±10.65 / 309.32 ms │    283.12 / 303.20 ±29.24 / 360.92 ms │     no change │
   │ QQuery 32 │     936.19 / 953.06 ±20.55 / 991.94 ms │ 927.75 / 1043.47 ±100.97 / 1201.40 ms │  1.09x slower │
   │ QQuery 33 │  1450.38 / 1517.91 ±70.77 / 1648.88 ms │ 1456.72 / 1498.65 ±35.86 / 1564.48 ms │     no change │
   │ QQuery 34 │   1526.27 / 1535.51 ±7.10 / 1544.81 ms │ 1509.53 / 1555.65 ±38.88 / 1622.57 ms │     no change │
   │ QQuery 35 │     283.89 / 303.93 ±23.85 / 340.85 ms │    273.63 / 310.68 ±63.26 / 436.84 ms │     no change │
   │ QQuery 36 │         67.96 / 73.47 ±3.87 / 78.28 ms │        70.78 / 73.70 ±2.42 / 76.50 ms │     no change │
   │ QQuery 37 │         35.41 / 36.82 ±1.60 / 39.76 ms │        35.10 / 36.92 ±1.95 / 40.26 ms │     no change │
   │ QQuery 38 │         41.77 / 45.35 ±2.56 / 49.04 ms │        40.46 / 42.24 ±1.22 / 43.77 ms │ +1.07x faster │
   │ QQuery 39 │      149.12 / 157.71 ±6.61 / 167.88 ms │     138.35 / 147.12 ±8.22 / 162.58 ms │ +1.07x faster │
   │ QQuery 40 │         13.68 / 17.10 ±3.45 / 21.51 ms │        13.95 / 15.49 ±2.23 / 19.89 ms │ +1.10x faster │
   │ QQuery 41 │         13.43 / 13.67 ±0.19 / 13.96 ms │        13.62 / 14.07 ±0.37 / 14.61 ms │     no change │
   │ QQuery 42 │         12.81 / 12.96 ±0.17 / 13.27 ms │        13.00 / 13.19 ±0.18 / 13.49 ms │     no change │
   └───────────┴────────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 19709.30ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 20062.80ms │
   │ Average Time (HEAD)                                  │   458.36ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   466.58ms │
   │ Queries Faster                                       │          6 │
   │ Queries Slower                                       │          9 │
   │ Queries with No Change                               │         28 │
   │ Queries with Failure                                 │          0 │
   └──────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 12.2 GiB |
   | Avg memory | 4.1 GiB |
   | CPU user | 1008.8s |
   | CPU sys | 72.2s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 105.0s |
   | Peak memory | 11.5 GiB |
   | Avg memory | 4.2 GiB |
   | CPU user | 1022.8s |
   | CPU sys | 74.7s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3920726384


##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -135,6 +157,114 @@ pub fn apply_file_schema_type_coercions(
     ))
 }
 
+fn dictionary_value_type(data_type: &DataType) -> Option<&DataType> {
+    match data_type {
+        DataType::Dictionary(_, value_type) => Some(value_type.as_ref()),
+        _ => None,
+    }
+}
+
+// Find the value type that can represent both sides without narrowing offsets
+// or crossing string/binary families.
+fn common_dictionary_value_type(
+    field_type: &DataType,
+    dictionary_value_type: &DataType,
+) -> Option<DataType> {
+    let field_type = match field_type {
+        DataType::Dictionary(_, field_value_type) => field_value_type.as_ref(),
+        _ => field_type,
+    };
+
+    match (field_type, dictionary_value_type) {
+        (DataType::Utf8, DataType::Utf8) => Some(DataType::Utf8),
+        (DataType::Utf8 | DataType::LargeUtf8, DataType::Utf8 | DataType::LargeUtf8) => {
+            Some(DataType::LargeUtf8)
+        }
+        (DataType::Binary, DataType::Binary) => Some(DataType::Binary),
+        (
+            DataType::Binary | DataType::LargeBinary,
+            DataType::Binary | DataType::LargeBinary,
+        ) => Some(DataType::LargeBinary),
+        _ => None,
+    }
+}
+
+/// Allows safe widening into the table dictionary type.
+/// - `Utf8` to `Dictionary(Int32, LargeUtf8)`: allowed
+/// - `LargeBinary` to `Dictionary(Int32, Binary)`: rejected
+fn can_promote_to_dictionary_type(
+    file_field_type: &DataType,
+    table_dictionary_type: &DataType,
+) -> bool {
+    dictionary_value_type(table_dictionary_type).is_some_and(|dictionary_value_type| {
+        common_dictionary_value_type(file_field_type, dictionary_value_type)
+            .is_some_and(|common_type| &common_type == dictionary_value_type)
+    })
+}
+
+/// Normalize per-file schemas so that a column promoted to `Dictionary` in
+/// *any* file is promoted to the same `Dictionary` type in *all* files.
+///
+/// This lets [`Schema::try_merge`] accept directories that mix dictionary and
+/// plain encodings for the same column.
+pub(crate) fn uniform_dict_schemas(schemas: Vec<Schema>) -> Vec<Schema> {
+    // First pass: record the dictionary type for every column that is Dictionary in
+    // at least one schema.
+    let mut dict_types: HashMap<String, DataType> = HashMap::new();
+    for schema in &schemas {
+        for field in schema.fields() {
+            if matches!(field.data_type(), DataType::Dictionary(_, _)) {
+                dict_types
+                    .entry(field.name().clone())
+                    .or_insert_with(|| field.data_type().clone());
+            }
+        }
+    }
+    if dict_types.is_empty() {
+        return schemas;
+    }
+
+    // Widen the recorded dictionary value type before promoting plain fields.
+    for schema in &schemas {
+        for field in schema.fields() {
+            let Some(dict_type) = dict_types.get_mut(field.name()) else {
+                continue;
+            };
+            let DataType::Dictionary(key_type, value_type) = dict_type else {

Review Comment:
   nice catch, the latest commit addresses this https://github.com/apache/datafusion/pull/24227/commits/46ccfb87f3e34350616bd26459ee653e89ab08d2. the output dictionary type is the largest type that can fit both/all types. in follow up PR for the epic it will be important that we take the cardinality sum into account to use the smallest key that fits all the distinct values



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5374461529

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5374297430)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2) to db8796d (merge-base) [diff](https://github.com/apache/datafusion/compare/db8796d24fc2b34f3cb6a489267723c01551db78..b0b0c0ef0fae922ab31b7cd392b78e5fe654afd2)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃    Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ QQuery 1  │ 38.33 ms │                              38.28 ms │ no change │
   │ QQuery 2  │ 18.87 ms │                              19.31 ms │ no change │
   │ QQuery 3  │ 30.27 ms │                              30.28 ms │ no change │
   │ QQuery 4  │ 17.40 ms │                              17.12 ms │ no change │
   │ QQuery 5  │ 37.76 ms │                              37.86 ms │ no change │
   │ QQuery 6  │ 15.84 ms │                              16.04 ms │ no change │
   │ QQuery 7  │ 43.54 ms │                              44.31 ms │ no change │
   │ QQuery 8  │ 41.96 ms │                              41.94 ms │ no change │
   │ QQuery 9  │ 48.06 ms │                              48.88 ms │ no change │
   │ QQuery 10 │ 41.59 ms │                              41.62 ms │ no change │
   │ QQuery 11 │ 13.29 ms │                              13.32 ms │ no change │
   │ QQuery 12 │ 23.63 ms │                              23.64 ms │ no change │
   │ QQuery 13 │ 32.48 ms │                              31.71 ms │ no change │
   │ QQuery 14 │ 23.08 ms │                              22.81 ms │ no change │
   │ QQuery 15 │ 30.29 ms │                              30.41 ms │ no change │
   │ QQuery 16 │ 13.68 ms │                              13.69 ms │ no change │
   │ QQuery 17 │ 69.79 ms │                              69.36 ms │ no change │
   │ QQuery 18 │ 59.15 ms │                              57.57 ms │ no change │
   │ QQuery 19 │ 32.17 ms │                              32.22 ms │ no change │
   │ QQuery 20 │ 31.13 ms │                              31.37 ms │ no change │
   │ QQuery 21 │ 54.49 ms │                              54.39 ms │ no change │
   │ QQuery 22 │ 13.58 ms │                              14.16 ms │ no change │
   └───────────┴──────────┴───────────────────────────────────────┴───────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 730.37ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 730.29ms │
   │ Average Time (HEAD)                                  │  33.20ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  33.19ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        0 │
   │ Queries with No Change                               │       22 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃    Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ QQuery 1  │ 38.33 / 39.58 ±1.31 / 41.35 ms │        38.28 / 39.26 ±0.98 / 41.11 ms │ no change │
   │ QQuery 2  │ 18.87 / 20.13 ±1.39 / 22.17 ms │        19.31 / 19.50 ±0.17 / 19.83 ms │ no change │
   │ QQuery 3  │ 30.27 / 32.34 ±1.41 / 33.77 ms │        30.28 / 32.81 ±1.34 / 34.11 ms │ no change │
   │ QQuery 4  │ 17.40 / 17.95 ±0.54 / 18.97 ms │        17.12 / 17.40 ±0.18 / 17.66 ms │ no change │
   │ QQuery 5  │ 37.76 / 39.36 ±1.35 / 41.19 ms │        37.86 / 39.84 ±1.14 / 41.40 ms │ no change │
   │ QQuery 6  │ 15.84 / 15.97 ±0.09 / 16.08 ms │        16.04 / 16.13 ±0.11 / 16.32 ms │ no change │
   │ QQuery 7  │ 43.54 / 44.52 ±0.74 / 45.82 ms │        44.31 / 45.33 ±0.81 / 46.30 ms │ no change │
   │ QQuery 8  │ 41.96 / 42.90 ±0.87 / 44.53 ms │        41.94 / 42.39 ±0.23 / 42.58 ms │ no change │
   │ QQuery 9  │ 48.06 / 49.55 ±0.82 / 50.43 ms │        48.88 / 49.60 ±0.57 / 50.23 ms │ no change │
   │ QQuery 10 │ 41.59 / 42.51 ±1.30 / 44.99 ms │        41.62 / 41.92 ±0.18 / 42.14 ms │ no change │
   │ QQuery 11 │ 13.29 / 14.21 ±0.56 / 14.82 ms │        13.32 / 13.61 ±0.25 / 13.96 ms │ no change │
   │ QQuery 12 │ 23.63 / 24.18 ±0.32 / 24.60 ms │        23.64 / 24.82 ±1.16 / 26.79 ms │ no change │
   │ QQuery 13 │ 32.48 / 33.81 ±1.02 / 35.44 ms │        31.71 / 32.88 ±0.90 / 33.85 ms │ no change │
   │ QQuery 14 │ 23.08 / 23.24 ±0.19 / 23.61 ms │        22.81 / 23.04 ±0.16 / 23.22 ms │ no change │
   │ QQuery 15 │ 30.29 / 31.15 ±1.15 / 33.42 ms │        30.41 / 30.86 ±0.75 / 32.35 ms │ no change │
   │ QQuery 16 │ 13.68 / 14.05 ±0.64 / 15.32 ms │        13.69 / 13.88 ±0.16 / 14.10 ms │ no change │
   │ QQuery 17 │ 69.79 / 70.48 ±0.46 / 71.21 ms │        69.36 / 70.77 ±1.03 / 72.06 ms │ no change │
   │ QQuery 18 │ 59.15 / 60.49 ±1.37 / 63.11 ms │        57.57 / 59.45 ±1.02 / 60.52 ms │ no change │
   │ QQuery 19 │ 32.17 / 32.53 ±0.34 / 33.11 ms │        32.22 / 32.59 ±0.32 / 33.14 ms │ no change │
   │ QQuery 20 │ 31.13 / 32.01 ±1.18 / 34.34 ms │        31.37 / 31.80 ±0.63 / 33.04 ms │ no change │
   │ QQuery 21 │ 54.49 / 55.67 ±0.89 / 56.79 ms │        54.39 / 56.76 ±1.82 / 59.36 ms │ no change │
   │ QQuery 22 │ 13.58 / 13.92 ±0.30 / 14.28 ms │        14.16 / 14.29 ±0.11 / 14.48 ms │ no change │
   └───────────┴────────────────────────────────┴───────────────────────────────────────┴───────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃          ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 750.53ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 748.93ms │
   │ Average Time (HEAD)                                  │  34.11ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │  34.04ms │
   │ Queries Faster                                       │        0 │
   │ Queries Slower                                       │        0 │
   │ Queries with No Change                               │       22 │
   │ Queries with Failure                                 │        0 │
   └──────────────────────────────────────────────────────┴──────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.2 GiB |
   | Avg memory | 518.7 MiB |
   | CPU user | 21.3s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.2 GiB |
   | Avg memory | 527.2 MiB |
   | CPU user | 21.4s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5400240601

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5400020690)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8c1ada76c5bf9da164594ac9ff6609c885) to 26b40dd (merge-base) [diff](https://github.com/apache/datafusion/compare/26b40dd781d2eeab1acc38370a7069a8b9ebe450..35dd8d8c1ada76c5bf9da164594ac9ff6609c885)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │    6.01 ms │                               6.18 ms │     no change │
   │ QQuery 2  │   82.12 ms │                              84.20 ms │     no change │
   │ QQuery 3  │   29.24 ms │                              30.19 ms │     no change │
   │ QQuery 4  │  480.96 ms │                             567.74 ms │  1.18x slower │
   │ QQuery 5  │   51.47 ms │                              54.42 ms │  1.06x slower │
   │ QQuery 6  │   35.98 ms │                              39.46 ms │  1.10x slower │
   │ QQuery 7  │   75.16 ms │                              78.81 ms │     no change │
   │ QQuery 8  │   36.94 ms │                              38.07 ms │     no change │
   │ QQuery 9  │   53.73 ms │                              53.71 ms │     no change │
   │ QQuery 10 │   65.01 ms │                              65.42 ms │     no change │
   │ QQuery 11 │  297.85 ms │                             301.34 ms │     no change │
   │ QQuery 12 │   29.31 ms │                              28.55 ms │     no change │
   │ QQuery 13 │  118.87 ms │                             117.44 ms │     no change │
   │ QQuery 14 │  411.59 ms │                             411.19 ms │     no change │
   │ QQuery 15 │   56.46 ms │                              56.98 ms │     no change │
   │ QQuery 16 │    6.67 ms │                               6.76 ms │     no change │
   │ QQuery 17 │   78.77 ms │                              79.03 ms │     no change │
   │ QQuery 18 │  103.62 ms │                             105.71 ms │     no change │
   │ QQuery 19 │   40.49 ms │                              41.02 ms │     no change │
   │ QQuery 20 │   35.07 ms │                              35.36 ms │     no change │
   │ QQuery 21 │   17.23 ms │                              17.24 ms │     no change │
   │ QQuery 22 │   62.34 ms │                              62.45 ms │     no change │
   │ QQuery 23 │  304.79 ms │                             308.78 ms │     no change │
   │ QQuery 24 │  195.33 ms │                             191.17 ms │     no change │
   │ QQuery 25 │  107.82 ms │                             114.09 ms │  1.06x slower │
   │ QQuery 26 │   48.37 ms │                              51.49 ms │  1.06x slower │
   │ QQuery 27 │    6.07 ms │                               6.69 ms │  1.10x slower │
   │ QQuery 28 │   55.47 ms │                              58.49 ms │  1.05x slower │
   │ QQuery 29 │   95.13 ms │                              98.01 ms │     no change │
   │ QQuery 30 │   31.29 ms │                              33.84 ms │  1.08x slower │
   │ QQuery 31 │  108.73 ms │                             109.82 ms │     no change │
   │ QQuery 32 │   21.68 ms │                              19.86 ms │ +1.09x faster │
   │ QQuery 33 │   39.63 ms │                              37.16 ms │ +1.07x faster │
   │ QQuery 34 │   10.87 ms │                               9.84 ms │ +1.10x faster │
   │ QQuery 35 │   80.37 ms │                              72.53 ms │ +1.11x faster │
   │ QQuery 36 │    6.39 ms │                               5.69 ms │ +1.12x faster │
   │ QQuery 37 │    7.38 ms │                               6.67 ms │ +1.11x faster │
   │ QQuery 38 │   60.69 ms │                              61.19 ms │     no change │
   │ QQuery 39 │   87.87 ms │                              88.56 ms │     no change │
   │ QQuery 40 │   23.73 ms │                              23.56 ms │     no change │
   │ QQuery 41 │   10.99 ms │                              11.05 ms │     no change │
   │ QQuery 42 │   23.18 ms │                              23.31 ms │     no change │
   │ QQuery 43 │    5.08 ms │                               5.11 ms │     no change │
   │ QQuery 44 │    9.07 ms │                               9.18 ms │     no change │
   │ QQuery 45 │   36.35 ms │                              37.09 ms │     no change │
   │ QQuery 46 │   12.04 ms │                              11.80 ms │     no change │
   │ QQuery 47 │  224.09 ms │                             227.39 ms │     no change │
   │ QQuery 48 │   95.35 ms │                              95.04 ms │     no change │
   │ QQuery 49 │   71.03 ms │                              70.96 ms │     no change │
   │ QQuery 50 │   58.30 ms │                              58.42 ms │     no change │
   │ QQuery 51 │   92.00 ms │                              92.11 ms │     no change │
   │ QQuery 52 │   23.96 ms │                              23.85 ms │     no change │
   │ QQuery 53 │   29.11 ms │                              29.15 ms │     no change │
   │ QQuery 54 │   53.75 ms │                              53.78 ms │     no change │
   │ QQuery 55 │   22.89 ms │                              23.05 ms │     no change │
   │ QQuery 56 │   38.52 ms │                              38.27 ms │     no change │
   │ QQuery 57 │  173.60 ms │                             173.34 ms │     no change │
   │ QQuery 58 │  110.53 ms │                             110.36 ms │     no change │
   │ QQuery 59 │  116.67 ms │                             117.70 ms │     no change │
   │ QQuery 60 │   39.36 ms │                              38.76 ms │     no change │
   │ QQuery 61 │   11.78 ms │                              12.17 ms │     no change │
   │ QQuery 62 │   45.11 ms │                              45.75 ms │     no change │
   │ QQuery 63 │   28.92 ms │                              28.83 ms │     no change │
   │ QQuery 64 │  364.65 ms │                             360.34 ms │     no change │
   │ QQuery 65 │  121.73 ms │                             131.01 ms │  1.08x slower │
   │ QQuery 66 │   79.71 ms │                              80.49 ms │     no change │
   │ QQuery 67 │  237.29 ms │                             242.12 ms │     no change │
   │ QQuery 68 │   11.86 ms │                              11.79 ms │     no change │
   │ QQuery 69 │   56.65 ms │                              56.87 ms │     no change │
   │ QQuery 70 │  105.44 ms │                             106.74 ms │     no change │
   │ QQuery 71 │   34.66 ms │                              34.99 ms │     no change │
   │ QQuery 72 │ 1814.65 ms │                            1729.95 ms │     no change │
   │ QQuery 73 │    9.81 ms │                               9.54 ms │     no change │
   │ QQuery 74 │  167.39 ms │                             168.33 ms │     no change │
   │ QQuery 75 │  145.35 ms │                             144.70 ms │     no change │
   │ QQuery 76 │   34.85 ms │                              34.69 ms │     no change │
   │ QQuery 77 │   60.75 ms │                              61.26 ms │     no change │
   │ QQuery 78 │  214.76 ms │                             219.74 ms │     no change │
   │ QQuery 79 │   65.99 ms │                              71.02 ms │  1.08x slower │
   │ QQuery 80 │   97.42 ms │                             106.60 ms │  1.09x slower │
   │ QQuery 81 │   25.16 ms │                              25.77 ms │     no change │
   │ QQuery 82 │   16.00 ms │                              15.88 ms │     no change │
   │ QQuery 83 │   33.30 ms │                              33.32 ms │     no change │
   │ QQuery 84 │   29.05 ms │                              29.25 ms │     no change │
   │ QQuery 85 │  102.02 ms │                             101.90 ms │     no change │
   │ QQuery 86 │   25.11 ms │                              25.00 ms │     no change │
   │ QQuery 87 │   61.97 ms │                              61.00 ms │     no change │
   │ QQuery 88 │   63.40 ms │                              63.67 ms │     no change │
   │ QQuery 89 │   35.72 ms │                              35.36 ms │     no change │
   │ QQuery 90 │   17.14 ms │                              16.86 ms │     no change │
   │ QQuery 91 │   44.36 ms │                              44.96 ms │     no change │
   │ QQuery 92 │   29.17 ms │                              28.14 ms │     no change │
   │ QQuery 93 │   49.49 ms │                              48.69 ms │     no change │
   │ QQuery 94 │   38.12 ms │                              37.79 ms │     no change │
   │ QQuery 95 │   78.69 ms │                              78.35 ms │     no change │
   │ QQuery 96 │   23.85 ms │                              23.43 ms │     no change │
   │ QQuery 97 │   51.73 ms │                              51.89 ms │     no change │
   │ QQuery 98 │   42.53 ms │                              42.65 ms │     no change │
   │ QQuery 99 │   69.79 ms │                              70.02 ms │     no change │
   └───────────┴────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 9257.77ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 9319.27ms │
   │ Average Time (HEAD)                                  │   93.51ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   94.13ms │
   │ Queries Faster                                       │         6 │
   │ Queries Slower                                       │        11 │
   │ Queries with No Change                               │        82 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and rich-T-kid_introduce-rle-parquet-flag
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ rich-T-kid_introduce-rle-parquet-flag ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │           6.01 / 6.52 ±0.86 / 8.24 ms │           6.18 / 6.80 ±0.88 / 8.54 ms │     no change │
   │ QQuery 2  │        82.12 / 82.72 ±0.35 / 83.15 ms │        84.20 / 84.39 ±0.21 / 84.76 ms │     no change │
   │ QQuery 3  │        29.24 / 29.78 ±0.32 / 30.06 ms │        30.19 / 30.62 ±0.25 / 30.93 ms │     no change │
   │ QQuery 4  │     480.96 / 487.60 ±3.97 / 493.20 ms │     567.74 / 571.26 ±3.24 / 576.35 ms │  1.17x slower │
   │ QQuery 5  │        51.47 / 52.18 ±0.60 / 53.26 ms │        54.42 / 55.07 ±0.45 / 55.65 ms │  1.06x slower │
   │ QQuery 6  │        35.98 / 36.46 ±0.27 / 36.78 ms │        39.46 / 40.19 ±0.44 / 40.72 ms │  1.10x slower │
   │ QQuery 7  │        75.16 / 75.48 ±0.26 / 75.78 ms │        78.81 / 80.34 ±1.70 / 83.52 ms │  1.06x slower │
   │ QQuery 8  │        36.94 / 39.57 ±3.75 / 46.97 ms │        38.07 / 38.62 ±0.32 / 38.97 ms │     no change │
   │ QQuery 9  │        53.73 / 55.58 ±1.55 / 58.06 ms │        53.71 / 56.26 ±2.89 / 61.63 ms │     no change │
   │ QQuery 10 │        65.01 / 65.66 ±0.61 / 66.58 ms │        65.42 / 67.08 ±2.15 / 71.31 ms │     no change │
   │ QQuery 11 │    297.85 / 327.25 ±28.12 / 370.14 ms │    301.34 / 312.03 ±14.59 / 340.92 ms │     no change │
   │ QQuery 12 │        29.31 / 29.76 ±0.29 / 30.22 ms │        28.55 / 29.27 ±0.49 / 29.91 ms │     no change │
   │ QQuery 13 │     118.87 / 120.83 ±1.79 / 123.73 ms │     117.44 / 119.63 ±2.89 / 125.24 ms │     no change │
   │ QQuery 14 │     411.59 / 415.34 ±3.57 / 421.98 ms │     411.19 / 415.94 ±2.72 / 418.77 ms │     no change │
   │ QQuery 15 │        56.46 / 57.65 ±0.87 / 59.14 ms │        56.98 / 57.68 ±0.59 / 58.61 ms │     no change │
   │ QQuery 16 │           6.67 / 6.82 ±0.20 / 7.21 ms │           6.76 / 6.95 ±0.27 / 7.49 ms │     no change │
   │ QQuery 17 │        78.77 / 79.77 ±0.78 / 81.07 ms │        79.03 / 85.15 ±4.89 / 93.39 ms │  1.07x slower │
   │ QQuery 18 │     103.62 / 105.10 ±1.82 / 108.60 ms │     105.71 / 110.69 ±2.50 / 112.17 ms │  1.05x slower │
   │ QQuery 19 │        40.49 / 41.05 ±0.46 / 41.70 ms │        41.02 / 41.43 ±0.39 / 42.15 ms │     no change │
   │ QQuery 20 │        35.07 / 35.27 ±0.19 / 35.60 ms │        35.36 / 36.36 ±0.78 / 37.68 ms │     no change │
   │ QQuery 21 │        17.23 / 17.56 ±0.23 / 17.80 ms │        17.24 / 17.43 ±0.23 / 17.84 ms │     no change │
   │ QQuery 22 │        62.34 / 64.16 ±1.26 / 66.28 ms │        62.45 / 63.08 ±0.79 / 64.64 ms │     no change │
   │ QQuery 23 │     304.79 / 307.07 ±1.56 / 309.39 ms │     308.78 / 310.47 ±1.34 / 311.88 ms │     no change │
   │ QQuery 24 │     195.33 / 198.27 ±3.72 / 205.45 ms │     191.17 / 198.22 ±7.11 / 211.37 ms │     no change │
   │ QQuery 25 │     107.82 / 110.19 ±1.63 / 112.50 ms │     114.09 / 115.62 ±1.65 / 118.75 ms │     no change │
   │ QQuery 26 │        48.37 / 48.93 ±0.46 / 49.48 ms │        51.49 / 52.22 ±0.47 / 52.84 ms │  1.07x slower │
   │ QQuery 27 │           6.07 / 6.24 ±0.20 / 6.61 ms │           6.69 / 6.81 ±0.14 / 7.06 ms │  1.09x slower │
   │ QQuery 28 │        55.47 / 58.36 ±2.18 / 60.48 ms │        58.49 / 63.82 ±4.48 / 72.12 ms │  1.09x slower │
   │ QQuery 29 │        95.13 / 95.92 ±0.44 / 96.36 ms │       98.01 / 99.58 ±0.89 / 100.64 ms │     no change │
   │ QQuery 30 │        31.29 / 31.73 ±0.27 / 32.07 ms │        33.84 / 34.14 ±0.18 / 34.30 ms │  1.08x slower │
   │ QQuery 31 │     108.73 / 113.82 ±3.78 / 118.31 ms │     109.82 / 112.28 ±2.00 / 115.17 ms │     no change │
   │ QQuery 32 │        21.68 / 22.62 ±1.39 / 25.37 ms │        19.86 / 20.36 ±0.31 / 20.71 ms │ +1.11x faster │
   │ QQuery 33 │        39.63 / 40.06 ±0.33 / 40.54 ms │        37.16 / 37.94 ±0.77 / 39.42 ms │ +1.06x faster │
   │ QQuery 34 │        10.87 / 11.06 ±0.29 / 11.62 ms │         9.84 / 10.17 ±0.25 / 10.47 ms │ +1.09x faster │
   │ QQuery 35 │        80.37 / 81.81 ±1.26 / 83.97 ms │        72.53 / 75.03 ±2.65 / 80.07 ms │ +1.09x faster │
   │ QQuery 36 │           6.39 / 6.59 ±0.14 / 6.75 ms │           5.69 / 5.80 ±0.16 / 6.12 ms │ +1.14x faster │
   │ QQuery 37 │           7.38 / 7.52 ±0.09 / 7.65 ms │           6.67 / 6.92 ±0.24 / 7.23 ms │ +1.09x faster │
   │ QQuery 38 │        60.69 / 64.36 ±2.84 / 67.96 ms │        61.19 / 61.43 ±0.21 / 61.70 ms │     no change │
   │ QQuery 39 │        87.87 / 92.46 ±3.68 / 95.87 ms │        88.56 / 89.71 ±0.67 / 90.55 ms │     no change │
   │ QQuery 40 │        23.73 / 23.99 ±0.18 / 24.30 ms │        23.56 / 24.93 ±1.84 / 28.56 ms │     no change │
   │ QQuery 41 │        10.99 / 12.04 ±1.91 / 15.86 ms │        11.05 / 11.18 ±0.18 / 11.52 ms │ +1.08x faster │
   │ QQuery 42 │        23.18 / 23.77 ±0.44 / 24.30 ms │        23.31 / 24.23 ±0.74 / 25.24 ms │     no change │
   │ QQuery 43 │           5.08 / 5.20 ±0.19 / 5.58 ms │           5.11 / 5.24 ±0.17 / 5.59 ms │     no change │
   │ QQuery 44 │           9.07 / 9.17 ±0.07 / 9.27 ms │           9.18 / 9.31 ±0.14 / 9.56 ms │     no change │
   │ QQuery 45 │        36.35 / 38.05 ±1.53 / 40.83 ms │        37.09 / 37.93 ±0.44 / 38.27 ms │     no change │
   │ QQuery 46 │        12.04 / 12.37 ±0.30 / 12.92 ms │        11.80 / 12.02 ±0.22 / 12.41 ms │     no change │
   │ QQuery 47 │     224.09 / 226.03 ±1.64 / 228.44 ms │     227.39 / 228.65 ±1.41 / 231.23 ms │     no change │
   │ QQuery 48 │       95.35 / 96.78 ±2.28 / 101.33 ms │        95.04 / 96.78 ±1.39 / 98.65 ms │     no change │
   │ QQuery 49 │        71.03 / 71.63 ±0.40 / 72.25 ms │        70.96 / 71.62 ±0.79 / 73.11 ms │     no change │
   │ QQuery 50 │        58.30 / 59.20 ±0.77 / 60.49 ms │        58.42 / 58.96 ±0.35 / 59.48 ms │     no change │
   │ QQuery 51 │        92.00 / 93.72 ±1.43 / 95.76 ms │        92.11 / 93.99 ±1.21 / 95.53 ms │     no change │
   │ QQuery 52 │        23.96 / 24.35 ±0.47 / 24.94 ms │        23.85 / 24.25 ±0.41 / 25.02 ms │     no change │
   │ QQuery 53 │        29.11 / 29.54 ±0.39 / 30.23 ms │        29.15 / 29.23 ±0.05 / 29.31 ms │     no change │
   │ QQuery 54 │        53.75 / 54.46 ±1.00 / 56.44 ms │        53.78 / 54.16 ±0.29 / 54.66 ms │     no change │
   │ QQuery 55 │        22.89 / 23.28 ±0.28 / 23.61 ms │        23.05 / 23.27 ±0.20 / 23.60 ms │     no change │
   │ QQuery 56 │        38.52 / 39.56 ±1.64 / 42.81 ms │        38.27 / 38.95 ±0.76 / 40.10 ms │     no change │
   │ QQuery 57 │     173.60 / 174.95 ±1.08 / 176.47 ms │     173.34 / 174.41 ±1.45 / 177.27 ms │     no change │
   │ QQuery 58 │     110.53 / 112.56 ±1.24 / 113.93 ms │     110.36 / 111.71 ±1.99 / 115.66 ms │     no change │
   │ QQuery 59 │     116.67 / 118.24 ±1.80 / 121.75 ms │     117.70 / 118.75 ±0.70 / 119.77 ms │     no change │
   │ QQuery 60 │        39.36 / 39.78 ±0.42 / 40.53 ms │        38.76 / 39.54 ±0.45 / 40.04 ms │     no change │
   │ QQuery 61 │        11.78 / 11.95 ±0.15 / 12.24 ms │        12.17 / 12.30 ±0.16 / 12.62 ms │     no change │
   │ QQuery 62 │        45.11 / 45.51 ±0.35 / 45.96 ms │        45.75 / 46.19 ±0.22 / 46.39 ms │     no change │
   │ QQuery 63 │        28.92 / 29.14 ±0.16 / 29.33 ms │        28.83 / 29.32 ±0.40 / 30.00 ms │     no change │
   │ QQuery 64 │     364.65 / 367.59 ±2.37 / 371.51 ms │    360.34 / 372.93 ±13.23 / 391.07 ms │     no change │
   │ QQuery 65 │     121.73 / 125.36 ±2.15 / 127.22 ms │     131.01 / 134.79 ±3.55 / 140.63 ms │  1.08x slower │
   │ QQuery 66 │        79.71 / 81.59 ±2.30 / 86.01 ms │        80.49 / 83.16 ±1.95 / 85.19 ms │     no change │
   │ QQuery 67 │     237.29 / 241.67 ±2.37 / 244.27 ms │     242.12 / 243.19 ±1.19 / 244.92 ms │     no change │
   │ QQuery 68 │        11.86 / 12.01 ±0.12 / 12.21 ms │        11.79 / 11.93 ±0.18 / 12.29 ms │     no change │
   │ QQuery 69 │        56.65 / 59.31 ±4.19 / 67.66 ms │        56.87 / 57.42 ±0.43 / 58.17 ms │     no change │
   │ QQuery 70 │     105.44 / 109.38 ±3.24 / 114.24 ms │     106.74 / 110.48 ±3.84 / 117.18 ms │     no change │
   │ QQuery 71 │        34.66 / 35.28 ±0.53 / 35.88 ms │        34.99 / 36.13 ±1.41 / 38.75 ms │     no change │
   │ QQuery 72 │ 1814.65 / 1923.76 ±93.95 / 2076.49 ms │ 1729.95 / 1819.57 ±71.16 / 1891.11 ms │ +1.06x faster │
   │ QQuery 73 │         9.81 / 10.17 ±0.47 / 11.00 ms │         9.54 / 10.14 ±0.37 / 10.61 ms │     no change │
   │ QQuery 74 │     167.39 / 170.45 ±3.46 / 177.19 ms │     168.33 / 170.03 ±1.45 / 172.31 ms │     no change │
   │ QQuery 75 │     145.35 / 146.15 ±1.13 / 148.35 ms │     144.70 / 147.54 ±3.46 / 154.31 ms │     no change │
   │ QQuery 76 │        34.85 / 36.12 ±2.29 / 40.69 ms │        34.69 / 36.05 ±1.85 / 39.69 ms │     no change │
   │ QQuery 77 │        60.75 / 61.49 ±0.60 / 62.56 ms │        61.26 / 63.43 ±2.75 / 68.76 ms │     no change │
   │ QQuery 78 │     214.76 / 219.32 ±2.84 / 222.56 ms │    219.74 / 238.99 ±14.20 / 258.88 ms │  1.09x slower │
   │ QQuery 79 │        65.99 / 66.17 ±0.16 / 66.43 ms │        71.02 / 71.72 ±0.39 / 72.21 ms │  1.08x slower │
   │ QQuery 80 │      97.42 / 101.23 ±5.38 / 111.89 ms │     106.60 / 107.76 ±1.31 / 110.24 ms │  1.06x slower │
   │ QQuery 81 │        25.16 / 25.55 ±0.30 / 26.01 ms │        25.77 / 27.28 ±1.25 / 29.05 ms │  1.07x slower │
   │ QQuery 82 │        16.00 / 16.29 ±0.19 / 16.52 ms │        15.88 / 16.61 ±0.98 / 18.53 ms │     no change │
   │ QQuery 83 │        33.30 / 33.53 ±0.23 / 33.90 ms │        33.32 / 33.76 ±0.45 / 34.57 ms │     no change │
   │ QQuery 84 │        29.05 / 31.13 ±3.81 / 38.74 ms │        29.25 / 29.37 ±0.12 / 29.58 ms │ +1.06x faster │
   │ QQuery 85 │     102.02 / 104.93 ±3.58 / 111.97 ms │     101.90 / 104.16 ±2.38 / 108.66 ms │     no change │
   │ QQuery 86 │        25.11 / 25.38 ±0.31 / 25.97 ms │        25.00 / 25.41 ±0.23 / 25.65 ms │     no change │
   │ QQuery 87 │        61.97 / 62.62 ±0.44 / 63.28 ms │        61.00 / 61.62 ±0.49 / 62.39 ms │     no change │
   │ QQuery 88 │        63.40 / 65.15 ±2.25 / 69.59 ms │        63.67 / 65.01 ±1.61 / 68.16 ms │     no change │
   │ QQuery 89 │        35.72 / 36.15 ±0.32 / 36.56 ms │        35.36 / 36.10 ±0.73 / 37.25 ms │     no change │
   │ QQuery 90 │        17.14 / 17.31 ±0.13 / 17.55 ms │        16.86 / 17.02 ±0.18 / 17.36 ms │     no change │
   │ QQuery 91 │        44.36 / 44.91 ±0.42 / 45.43 ms │        44.96 / 45.24 ±0.19 / 45.53 ms │     no change │
   │ QQuery 92 │        29.17 / 29.55 ±0.31 / 30.04 ms │        28.14 / 28.42 ±0.28 / 28.93 ms │     no change │
   │ QQuery 93 │        49.49 / 51.25 ±2.44 / 55.97 ms │        48.69 / 49.94 ±1.00 / 51.14 ms │     no change │
   │ QQuery 94 │        38.12 / 38.95 ±0.96 / 40.73 ms │        37.79 / 39.12 ±1.61 / 42.27 ms │     no change │
   │ QQuery 95 │        78.69 / 79.91 ±0.64 / 80.42 ms │        78.35 / 80.00 ±1.04 / 81.33 ms │     no change │
   │ QQuery 96 │        23.85 / 24.03 ±0.16 / 24.31 ms │        23.43 / 23.67 ±0.26 / 24.16 ms │     no change │
   │ QQuery 97 │        51.73 / 53.41 ±1.50 / 55.61 ms │        51.89 / 52.90 ±1.50 / 55.88 ms │     no change │
   │ QQuery 98 │        42.53 / 43.49 ±0.76 / 44.87 ms │        42.65 / 43.39 ±0.88 / 45.03 ms │     no change │
   │ QQuery 99 │        69.79 / 71.15 ±1.37 / 72.96 ms │        70.02 / 70.30 ±0.29 / 70.76 ms │     no change │
   └───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                    ┃           ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   │ Total Time (HEAD)                                    │ 9526.07ms │
   │ Total Time (rich-T-kid_introduce-rle-parquet-flag)   │ 9568.26ms │
   │ Average Time (HEAD)                                  │   96.22ms │
   │ Average Time (rich-T-kid_introduce-rle-parquet-flag) │   96.65ms │
   │ Queries Faster                                       │         9 │
   │ Queries Slower                                       │        15 │
   │ Queries with No Change                               │        75 │
   │ Queries with Failure                                 │         0 │
   └──────────────────────────────────────────────────────┴───────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 2.0 GiB |
   | Avg memory | 1.4 GiB |
   | CPU user | 205.9s |
   | CPU sys | 5.2s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 2.2 GiB |
   | Avg memory | 1.5 GiB |
   | CPU user | 203.4s |
   | CPU sys | 5.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5400053441

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5400020690)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5400020690-1907-njl57 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (35dd8d8c1ada76c5bf9da164594ac9ff6609c885) to 26b40dd (merge-base) [diff](https://github.com/apache/datafusion/compare/26b40dd781d2eeab1acc38370a7069a8b9ebe450..35dd8d8c1ada76c5bf9da164594ac9ff6609c885)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3793151424


##########
datafusion/datasource-parquet/src/metadata.rs:
##########
@@ -384,6 +395,74 @@ impl<'a> DFParquetMetadata<'a> {
                     .coerce()
             })
             .unwrap_or(schema);
+
+        // Promote before force_view_types so dictionary columns survive the
+        // Utf8 -> Utf8View pass.
+        let schema = if self.enable_rle_to_dictionary {
+            let rle_cols: HashSet<String> = {
+                let schema_descr = file_metadata.schema_descr();
+                metadata
+                    .row_groups()
+                    .iter()
+                    .flat_map(|rg| {
+                        rg.columns()
+                            .iter()
+                            .enumerate()
+                            .filter_map(|(col_idx, col)| {
+                                col.dictionary_page_offset()?;
+                                let desc = schema_descr.column(col_idx);
+                                // Only promote top-level columns. For nested columns
+                                // the Parquet leaf name (e.g. "element" inside a List)
+                                // does not match the Arrow top-level field name (e.g.
+                                // "tags"), so using the leaf name here would miss the
+                                // target field or promote an unrelated one.
+                                let parts = desc.path().parts();
+                                (parts.len() == 1).then(|| parts[0].clone())
+                            })
+                    })
+                    .collect()
+            };
+            if rle_cols.is_empty() {
+                schema
+            } else {
+                let promoted: Vec<_> = schema
+                    .fields()
+                    .iter()
+                    .map(|field| {
+                        if !rle_cols.contains(field.name()) {
+                            return Arc::clone(field);
+                        }
+                        let dict_value_type = match field.data_type() {
+                            DataType::Utf8 => Some(DataType::Utf8),
+                            DataType::LargeUtf8 => Some(DataType::LargeUtf8),
+                            DataType::Binary => Some(DataType::Binary),
+                            DataType::LargeBinary => Some(DataType::LargeBinary),
+                            _ => None,
+                        };
+                        dict_value_type.map_or_else(
+                            || Arc::clone(field),
+                            |value_type| {
+                                Arc::new(
+                                    arrow::datatypes::Field::new(

Review Comment:
   nit remove this path prefix



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["Rich-T-kid (via GitHub)" <gi...@apache.org>]

Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3750955824


##########
datafusion/common/src/config.rs:
##########
@@ -1200,6 +1200,13 @@ config_namespace! {
         /// Defaults to 20.
         pub max_in_list_size: usize, default = 20
 
+        /// (reading) If true, string and binary columns that are dictionary-encoded in
+        /// the parquet file are read as `Dictionary<Int32, Utf8>` / `Dictionary<Int32, Binary>`

Review Comment:
   I choose to keep the key type hard coded to int32 instead of u64, would not mind changing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["codecov-commenter (via GitHub)" <gi...@apache.org>]

codecov-commenter commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5245122461

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24227?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `90.17544%` with `28 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 80.99%. Comparing base ([`e08aed1`](https://app.codecov.io/gh/apache/datafusion/commit/e08aed1e5de41dcf81d529140dae07723b942a5e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d90e9df`](https://app.codecov.io/gh/apache/datafusion/commit/d90e9df87a8fe1defeb4959c06d525b09c8ba0cc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 38 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24227?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [datafusion/proto-common/src/generated/pbjson.rs](https://app.codecov.io/gh/apache/datafusion/pull/24227?src=pr&el=tree&filepath=datafusion%2Fproto-common%2Fsrc%2Fgenerated%2Fpbjson.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by1jb21tb24vc3JjL2dlbmVyYXRlZC9wYmpzb24ucnM=) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24227?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/datasource-parquet/src/metadata.rs](https://app.codecov.io/gh/apache/datafusion/pull/24227?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fmetadata.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL21ldGFkYXRhLnJz) | 87.17% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24227?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/datasource-parquet/src/opener/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/24227?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fopener%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL29wZW5lci9tb2QucnM=) | 96.73% | [4 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24227?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/datasource-parquet/src/source.rs](https://app.codecov.io/gh/apache/datafusion/pull/24227?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fsource.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL3NvdXJjZS5ycw==) | 94.36% | [2 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24227?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #24227      +/-   ##
   ==========================================
   - Coverage   81.04%   80.99%   -0.05%     
   ==========================================
     Files        1106     1106              
     Lines      380984   383652    +2668     
     Branches   380984   383652    +2668     
   ==========================================
   + Hits       308769   310754    +1985     
   - Misses      53968    54574     +606     
   - Partials    18247    18324      +77     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24227?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5345487759

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5345449022)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5345449022-1764-vzmzr 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (449355e7c5098104ce4868fa3ce347ca95a24ce3) to bbf739a (merge-base) [diff](https://github.com/apache/datafusion/compare/bbf739a3cfe1d5cc7646e37250c947c6a56b5b0e..449355e7c5098104ce4868fa3ce347ca95a24ce3)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364898249

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364868497)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5364868497-1832-fcspb 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f09f1555c30e2f0b88db32a80d7bcd87062bcc5d) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f09f1555c30e2f0b88db32a80d7bcd87062bcc5d)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["adriangbot (via GitHub)" <gi...@apache.org>]

adriangbot commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5364885593

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24227#issuecomment-5364868497)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5364868497-1833-9xcfg 6.12.85+ #1 SMP Sat Jun 27 09:31:30 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing rich-T-kid/introduce-rle-parquet-flag (f09f1555c30e2f0b88db32a80d7bcd87062bcc5d) to 8332cfa (merge-base) [diff](https://github.com/apache/datafusion/compare/8332cfafb37aa9209eacdbb098afc3e4bcd7f59a..f09f1555c30e2f0b88db32a80d7bcd87062bcc5d)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5629513612

   > could you trigger the CI for this pr?
   
   There seems to be conflicts, if you resolve it the ci should automatically start.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["kumarUjjawal (via GitHub)" <gi...@apache.org>]

kumarUjjawal commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r4119051027


##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -84,6 +84,183 @@ pub fn apply_file_schema_type_coercions(
     ))
 }
 
+/// Like [`apply_file_schema_type_coercions`], but also coerces compatible
+/// string/binary file fields to dictionary types already present in the table
+/// schema.
+pub(crate) fn apply_file_schema_type_coercions_with_rle(
+    table_schema: &Schema,
+    file_schema: &Schema,
+    enable_rle_to_dictionary: bool,
+) -> Option<Schema> {
+    let mut needs_view_transform = false;
+    let mut needs_string_transform = false;
+    let mut needs_nested_transform = false;
+    let mut needs_dict_transform = false;
+
+    // Create a mapping of table field names to their data types for fast lookup
+    // and simultaneously check if we need any transformations
+    let table_fields: HashMap<_, _> = table_schema
+        .fields()
+        .iter()
+        .map(|field| {
+            let data_type = field.data_type();
+            // Check if we need view type transformation
+            if matches!(data_type, &DataType::Utf8View | &DataType::BinaryView) {
+                needs_view_transform = true;
+            }
+            // Check if we need string type transformation
+            if matches!(
+                data_type,
+                &DataType::Utf8 | &DataType::LargeUtf8 | &DataType::Utf8View
+            ) {
+                needs_string_transform = true;
+            }
+            // Nested fields can need transformations even when their parent does not.
+            if matches!(
+                data_type,
+                DataType::Struct(_)
+                    | DataType::List(_)
+                    | DataType::LargeList(_)
+                    | DataType::ListView(_)
+                    | DataType::LargeListView(_)
+                    | DataType::FixedSizeList(_, _)
+                    | DataType::Map(_, _)
+            ) {
+                needs_nested_transform = true;
+            }
+            if enable_rle_to_dictionary
+                && matches!(data_type, &DataType::Dictionary(_, _))
+            {
+                needs_dict_transform = true;
+            }
+
+            (field.name(), data_type)
+        })
+        .collect();
+
+    // Early return if no transformation needed
+    if !needs_view_transform
+        && !needs_string_transform
+        && !needs_nested_transform
+        && !needs_dict_transform
+    {
+        return None;
+    }
+
+    let fields: Vec<Arc<Field>> = file_schema
+        .fields()
+        .iter()
+        .map(|field| {
+            let field_name = field.name();
+            let field_type = field.data_type();
+
+            // Look up the corresponding field type in the table schema
+            if let Some(table_type) = table_fields.get(field_name) {
+                match (table_type, field_type) {
+                    // table schema uses string type, coerce the file schema to use string type
+                    (
+                        &DataType::Utf8,
+                        DataType::Binary | DataType::LargeBinary | DataType::BinaryView,
+                    ) => {
+                        return field_with_new_type(field, DataType::Utf8);
+                    }
+                    // table schema uses large string type, coerce the file schema to use large string type
+                    (
+                        &DataType::LargeUtf8,
+                        DataType::Binary | DataType::LargeBinary | DataType::BinaryView,
+                    ) => {
+                        return field_with_new_type(field, DataType::LargeUtf8);
+                    }
+                    // table schema uses string view type, coerce the file schema to use view type
+                    (
+                        &DataType::Utf8View,
+                        DataType::Binary | DataType::LargeBinary | DataType::BinaryView,
+                    ) => {
+                        return field_with_new_type(field, DataType::Utf8View);
+                    }
+                    // Handle view type conversions
+                    (&DataType::Utf8View, DataType::Utf8 | DataType::LargeUtf8) => {
+                        return field_with_new_type(field, DataType::Utf8View);
+                    }
+                    (&DataType::BinaryView, DataType::Binary | DataType::LargeBinary) => {
+                        return field_with_new_type(field, DataType::BinaryView);
+                    }
+                    // Apply the same coercions to matching fields inside structs.
+                    (DataType::Struct(table_fields), DataType::Struct(file_fields)) => {
+                        if let Some(schema) = apply_file_schema_type_coercions(
+                            &Schema::new(table_fields.clone()),
+                            &Schema::new(file_fields.clone()),
+                        ) {
+                            return field_with_new_type(
+                                field,
+                                DataType::Struct(schema.fields),
+                            );
+                        }
+                    }
+                    // Container children match by position, regardless of their names.
+                    (DataType::List(table_child), DataType::List(file_child))
+                    | (
+                        DataType::LargeList(table_child),
+                        DataType::LargeList(file_child),
+                    )
+                    | (DataType::ListView(table_child), DataType::ListView(file_child))
+                    | (
+                        DataType::LargeListView(table_child),
+                        DataType::LargeListView(file_child),
+                    )
+                    | (
+                        DataType::FixedSizeList(table_child, _),
+                        DataType::FixedSizeList(file_child, _),
+                    )
+                    | (DataType::Map(table_child, _), DataType::Map(file_child, _)) => {

Review Comment:
   Please keep the existing map-specific coercion here. This call treats map entries as structs and matches their children by name. Parquet often uses `key` and `value`, while the table schema can use `keys` and `values`. The current `coerce_map_entries` helper matches them by position. It also keeps binary map values on the validating cast path. This function runs when `enable_rle_to_dictionary` is false, so the rebase changes existing behavior. Please reuse the existing helper pipeline and add dictionary promotion to it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] introduce optional rle reads from parquet [datafusion]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5871396646

   Is the goal of this PR to improve performance for some usecases? If so, do we have any benchmark results that show the performance improves?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org