You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/08/04 23:16:55 UTC

[PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

adriangb opened a new pull request, #24093:
URL: https://github.com/apache/datafusion/pull/24093

   ## Which issue does this PR close?
   
   - Closes #.
   
   ## Rationale for this change
   
   `--simulate-latency` wraps the local filesystem in `LatencyObjectStore` so benchmarks can be run against S3-like IO without a network. It is not a faithful model of one, in three ways that matter.
   
   **It overrides `get_ranges`.** Neither `AmazonS3` nor `GoogleCloudStorage` implements that method; they inherit the `ObjectStore` default, which merges ranges less than `OBJECT_STORE_COALESCE_DEFAULT` (1MiB) apart and issues the merged chunks as up to 10 *concurrent* GETs. `LatencyObjectStore` overrides it, sleeps once, and delegates to `LocalFileSystem::get_ranges` — so it charges one round trip where S3 pays one per coalesced chunk, and it inherits local positional reads with no coalescing at all. How many chunks a vectored read produces is data dependent (column layout, projection, gap distances), which is exactly what an IO optimisation changes, so the simulation is blind to the thing being measured.
   
   **It has no per-byte cost.** A 4 byte footer read and a 100MB column chunk read cost the same. Coalescing across a gap trades bytes for round trips; without a bytes term the simulation says that trade is always free.
   
   **It charges one latency for an entire LIST stream.** Both S3 and GCS cap a listing response at 1000 keys, and each page needs the continuation token from the one before it, so pages are serial round trips. Listing 5000 files is five round trips, not one.
   
   ## What changes are included in this PR?
   
   Replaces `LatencyObjectStore` with `SimulatedObjectStore`, which presents a local filesystem the way a remote store presents itself.
   
   The organising principle is that it implements **only** the HTTP-shaped primitives a real remote store implements, and leaves `get_ranges`, `list_with_offset` and `rename_opts` to the `ObjectStore` defaults — exactly as `AmazonS3` and `GoogleCloudStorage` leave them. Everything arrow-rs does above the wire (coalescing, fan-out, pagination) then runs against the simulator unmodified.
   
   This is deliberately the opposite of the advice in `ObjectStore`'s "Wrappers" doc section, which tells wrappers to implement every method so they do not lose the wrapped store's overrides. That advice is for observability wrappers; here the wrapped store's overrides are the problem. The module documents this, and there is a comment against adding `#[deny(clippy::missing_trait_methods)]`.
   
   Each simulated request pays for:
   
   1. a connection from a bounded pool (`max_concurrent_requests`, default 128),
   2. time to first byte, drawn from the same S3-shaped distribution as before,
   3. transfer time at `connection_bytes_per_second` (default 100MB/s), charged as the body is consumed.
   
   A `HEAD` is charged (1) only — its `GetResult` reports a range spanning the whole object even though no bytes move, and charging that as a transfer made every metadata probe cost as much as downloading the file.
   
   Bounded reads take a single blocking read; reads above `max_eager_read_bytes` stream, so a whole-object GET of a multi-GB CSV is not buffered in memory.
   
   The latency sampler now hashes its draw counter rather than round-robining. A 20-entry table cycled against `coalesce_ranges`' fixed waves of 10 meant every vectored read saw the same two fixed sets of latencies, and a read issuing exactly 10 or 20 requests always landed on the table mean.
   
   ### What is deliberately not modelled
   
   arrow-rs never splits one large range into several smaller concurrent requests — `merge_ranges` has no upper bound on merged size and no store chunks a large GET — so neither does this. A 200MB coalesced range is one GET on one connection, and is bandwidth bound. Retries, multipart part uploads and per-prefix throttling are also not modelled.
   
   ## Are these changes tested?
   
   Yes — 10 new tests.
   
   The ones carrying the argument above:
   
   - `distant_ranges_become_separate_requests` / `nearby_ranges_are_coalesced_into_one_request` — ranges more than 1MiB apart reach the wire as separate requests; nearer ones are merged into one. The old store reported one in both cases.
   - `projecting_more_columns_costs_more_requests` — end to end through a real DataFusion Parquet scan: two distant column chunks requested in a single `get_byte_ranges` call cost more requests than one. The old store could not express this difference.
   - `head_requests_do_not_pay_for_a_body`, `transfer_time_scales_with_bytes`, `list_is_charged_per_page`.
   
   `cargo test -p datafusion-benchmarks` passes (133 tests). `cargo clippy -p datafusion-benchmarks --all-targets --no-deps -- -D warnings` is clean.
   
   ### Benchmark results
   
   TPC-H SF1 parquet, 3 iterations, all 22 queries, run back to back on the same machine. "old sim" is `LatencyObjectStore` at the parent commit; "new sim" is this PR. Both with `--simulate-latency`.
   
   | Metric | Value |
   |---|---|
   | Total, no simulation | 5.33 s |
   | Total, old sim | 22.55 s |
   | Total, new sim | 26.75 s |
   | new / old | **1.19x** |
   
   The total is not the interesting number. The spread is: per query, new/old ranges from **0.90x to 2.04x**, because the new model differentiates queries that the old one charged identically.
   
   | Query | no sim | old sim | new sim | new/old |
   |---|---|---|---|---|
   | q22 | 23 ms | 529 ms | 1076 ms | 2.04x |
   | q21 | 153 ms | 2344 ms | 3850 ms | 1.64x |
   | q12 | 94 ms | 831 ms | 1160 ms | 1.40x |
   | q20 | 133 ms | 1016 ms | 1392 ms | 1.37x |
   | q4 | 46 ms | 812 ms | 1101 ms | 1.36x |
   | … | | | | |
   | q15 | 99 ms | 1267 ms | 1146 ms | 0.90x |
   
   Queries whose vectored reads fan out into many distant chunks (q22, q21) get materially more expensive; the small decreases at the bottom of the table are within run-to-run variance and I would not read anything into them. Relative query ranking changes between the two models, which is the point: an IO change evaluated under the old model was being scored against a cost function that could not see request count.
   
   Numbers from previous `--simulate-latency` runs are **not** comparable across this change.
   
   ## Are there any user-facing changes?
   
   No public API changes — this is benchmark-only. `--simulate-latency` and `SIMULATE_LATENCY` keep their names and meaning; their help text and `bench.sh` documentation are updated to describe what is now modelled.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-6150155534

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24093#issuecomment-6150107152)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6150107152-3358-2z456 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing claude/storage-latency-simulation-c76ce0 (228f0257239757cd24445b3b1f3b7be19c8d20a7) to 45de64c (merge-base) [diff](https://github.com/apache/datafusion/compare/45de64cfc199345c8f218acd6d18634105403776..228f0257239757cd24445b3b1f3b7be19c8d20a7)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   env:
     SIMULATE_LATENCY: "true"
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#discussion_r4256769865


##########
benchmarks/src/util/simulated_object_store.rs:
##########
@@ -0,0 +1,506 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! An [`ObjectStore`] that makes a local filesystem behave like S3 or GCS.
+//!
+//! Each request costs:
+//!
+//! 1. a connection from a pool of [`MAX_CONNECTIONS`],
+//! 2. a time to first byte, drawn at random from a latency distribution,
+//! 3. a transfer time at [`BYTES_PER_SECOND`] for each response body.
+//!
+//! The store does not override [`ObjectStore::get_ranges`]. `AmazonS3` and
+//! `GoogleCloudStorage` do not override it either. The trait default merges
+//! ranges that are less than 1MiB apart and sends each merged range as a
+//! separate GET. Thus the simulated request count is the same as on S3. A
+//! wrapper that overrides `get_ranges` charges one request where S3 charges
+//! several.
+//!
+//! The store does not model request failures, retries or a shared network
+//! limit.
+
+use std::fmt;
+use std::sync::Arc;
+use std::time::Duration;
+
+use async_trait::async_trait;
+use futures::StreamExt;
+use futures::stream::BoxStream;
+use object_store::path::Path;
+use object_store::{
+    CopyOptions, GetOptions, GetResult, GetResultPayload, ListResult, MultipartUpload,
+    ObjectMeta, ObjectStore, PutMultipartOptions, PutOptions, PutPayload, PutResult,
+    Result,
+};
+use rand::Rng;
+use tokio::sync::{OwnedSemaphorePermit, Semaphore};
+use tokio::time::Instant;
+
+/// GET time to first byte, inspired by S3 latencies.
+/// Sorted: 25,25,28,28,30,30,30,30,32,32,35, 70,85,100,100,110, 130,150, 180,200
+/// P50≈32ms, P90≈110ms, P99≈200ms
+const GET_TTFB_MS: &[u64] = &[

Review Comment:
   or P99 is even below 100ms in most cases



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#discussion_r4261414113


##########
benchmarks/src/util/simulated_object_store.rs:
##########
@@ -0,0 +1,506 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! An [`ObjectStore`] that makes a local filesystem behave like S3 or GCS.
+//!
+//! Each request costs:
+//!
+//! 1. a connection from a pool of [`MAX_CONNECTIONS`],
+//! 2. a time to first byte, drawn at random from a latency distribution,
+//! 3. a transfer time at [`BYTES_PER_SECOND`] for each response body.
+//!
+//! The store does not override [`ObjectStore::get_ranges`]. `AmazonS3` and
+//! `GoogleCloudStorage` do not override it either. The trait default merges
+//! ranges that are less than 1MiB apart and sends each merged range as a
+//! separate GET. Thus the simulated request count is the same as on S3. A
+//! wrapper that overrides `get_ranges` charges one request where S3 charges
+//! several.
+//!
+//! The store does not model request failures, retries or a shared network
+//! limit.
+
+use std::fmt;
+use std::sync::Arc;
+use std::time::Duration;
+
+use async_trait::async_trait;
+use futures::StreamExt;
+use futures::stream::BoxStream;
+use object_store::path::Path;
+use object_store::{
+    CopyOptions, GetOptions, GetResult, GetResultPayload, ListResult, MultipartUpload,
+    ObjectMeta, ObjectStore, PutMultipartOptions, PutOptions, PutPayload, PutResult,
+    Result,
+};
+use rand::Rng;
+use tokio::sync::{OwnedSemaphorePermit, Semaphore};
+use tokio::time::Instant;
+
+/// GET time to first byte, inspired by S3 latencies.
+/// Sorted: 25,25,28,28,30,30,30,30,32,32,35, 70,85,100,100,110, 130,150, 180,200
+/// P50≈32ms, P90≈110ms, P99≈200ms
+const GET_TTFB_MS: &[u64] = &[

Review Comment:
   I re-measured against S3 and GCS and added both. So it will be a mix. I think that's reasonable? I don't think this has to be perfect, there are *many* object store implemetnations out there. The point is to differentiate from a local SSD.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-5188344474

   Good idea @adriangb let me know when it's reviewable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-6136427061

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-5185824434

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24093?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `76.19048%` with `90 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 80.90%. Comparing base ([`39d5064`](https://app.codecov.io/gh/apache/datafusion/commit/39d50641424dfbc9fadad9cad17619c64d06076b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`af1b50b`](https://app.codecov.io/gh/apache/datafusion/commit/af1b50b17a6abe4376963f7a7cf24429b97bbb4c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 48 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24093?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [benchmarks/src/util/simulated\_object\_store/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/24093?src=pr&el=tree&filepath=benchmarks%2Fsrc%2Futil%2Fsimulated_object_store%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YmVuY2htYXJrcy9zcmMvdXRpbC9zaW11bGF0ZWRfb2JqZWN0X3N0b3JlL21vZC5ycw==) | 73.92% | [84 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24093?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [benchmarks/src/util/options.rs](https://app.codecov.io/gh/apache/datafusion/pull/24093?src=pr&el=tree&filepath=benchmarks%2Fsrc%2Futil%2Foptions.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YmVuY2htYXJrcy9zcmMvdXRpbC9vcHRpb25zLnJz) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24093?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...chmarks/src/util/simulated\_object\_store/latency.rs](https://app.codecov.io/gh/apache/datafusion/pull/24093?src=pr&el=tree&filepath=benchmarks%2Fsrc%2Futil%2Fsimulated_object_store%2Flatency.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YmVuY2htYXJrcy9zcmMvdXRpbC9zaW11bGF0ZWRfb2JqZWN0X3N0b3JlL2xhdGVuY3kucnM=) | 97.91% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24093?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #24093      +/-   ##
   ==========================================
   + Coverage   80.85%   80.90%   +0.04%     
   ==========================================
     Files        1099     1103       +4     
     Lines      374304   376672    +2368     
     Branches   374304   376672    +2368     
   ==========================================
   + Hits       302642   304739    +2097     
   - Misses      53607    53808     +201     
   - Partials    18055    18125      +70     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24093?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-6150107152

   run benchmarks
   
   ```yaml
   env:
     SIMULATE_LATENCY: "true"
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#discussion_r4256756807


##########
benchmarks/src/util/simulated_object_store.rs:
##########
@@ -0,0 +1,506 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! An [`ObjectStore`] that makes a local filesystem behave like S3 or GCS.
+//!
+//! Each request costs:
+//!
+//! 1. a connection from a pool of [`MAX_CONNECTIONS`],
+//! 2. a time to first byte, drawn at random from a latency distribution,
+//! 3. a transfer time at [`BYTES_PER_SECOND`] for each response body.
+//!
+//! The store does not override [`ObjectStore::get_ranges`]. `AmazonS3` and
+//! `GoogleCloudStorage` do not override it either. The trait default merges
+//! ranges that are less than 1MiB apart and sends each merged range as a
+//! separate GET. Thus the simulated request count is the same as on S3. A
+//! wrapper that overrides `get_ranges` charges one request where S3 charges
+//! several.
+//!
+//! The store does not model request failures, retries or a shared network
+//! limit.
+
+use std::fmt;
+use std::sync::Arc;
+use std::time::Duration;
+
+use async_trait::async_trait;
+use futures::StreamExt;
+use futures::stream::BoxStream;
+use object_store::path::Path;
+use object_store::{
+    CopyOptions, GetOptions, GetResult, GetResultPayload, ListResult, MultipartUpload,
+    ObjectMeta, ObjectStore, PutMultipartOptions, PutOptions, PutPayload, PutResult,
+    Result,
+};
+use rand::Rng;
+use tokio::sync::{OwnedSemaphorePermit, Semaphore};
+use tokio::time::Instant;
+
+/// GET time to first byte, inspired by S3 latencies.
+/// Sorted: 25,25,28,28,30,30,30,30,32,32,35, 70,85,100,100,110, 130,150, 180,200
+/// P50≈32ms, P90≈110ms, P99≈200ms
+const GET_TTFB_MS: &[u64] = &[

Review Comment:
   Interesting numbers!
   
   I feel like S3 has a bit better P50/P90 numbers in my experience (I think around 30ms and around 100ms on some rough graphs).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-6139325355

   @Dandandan I've revived this, rescoped it to be smaller and given it a review pass. Could you ptal?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-6150167875

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24093#issuecomment-6150107152)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6150107152-3357-npw5z 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing claude/storage-latency-simulation-c76ce0 (228f0257239757cd24445b3b1f3b7be19c8d20a7) to 45de64c (merge-base) [diff](https://github.com/apache/datafusion/compare/45de64cfc199345c8f218acd6d18634105403776..228f0257239757cd24445b3b1f3b7be19c8d20a7)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   env:
     SIMULATE_LATENCY: "true"
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-6150686261

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24093#issuecomment-6150107152)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing claude/storage-latency-simulation-c76ce0 (228f0257239757cd24445b3b1f3b7be19c8d20a7) to 45de64c (merge-base) [diff](https://github.com/apache/datafusion/compare/45de64cfc199345c8f218acd6d18634105403776..228f0257239757cd24445b3b1f3b7be19c8d20a7)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   env:
     SIMULATE_LATENCY: "true"
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and claude_storage-latency-simulation-c76ce0
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ claude_storage-latency-simulation-c76ce0 ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │    1.22 ms │                                  1.20 ms │     no change │
   │ QQuery 1  │ 1842.89 ms │                               1641.83 ms │ +1.12x faster │
   │ QQuery 2  │ 2002.55 ms │                               2479.97 ms │  1.24x slower │
   │ QQuery 3  │ 2095.88 ms │                               2110.89 ms │     no change │
   │ QQuery 4  │ 2254.62 ms │                               2189.31 ms │     no change │
   │ QQuery 5  │ 2394.02 ms │                               2435.43 ms │     no change │
   │ QQuery 6  │    1.29 ms │                                  1.25 ms │     no change │
   │ QQuery 7  │ 1794.81 ms │                               1711.44 ms │     no change │
   │ QQuery 8  │ 2330.70 ms │                               2404.94 ms │     no change │
   │ QQuery 9  │ 2689.55 ms │                               3637.71 ms │  1.35x slower │
   │ QQuery 10 │ 2171.79 ms │                               2724.26 ms │  1.25x slower │
   │ QQuery 11 │ 2324.96 ms │                               2621.21 ms │  1.13x slower │
   │ QQuery 12 │ 2254.05 ms │                               2413.06 ms │  1.07x slower │
   │ QQuery 13 │ 2446.62 ms │                               3218.78 ms │  1.32x slower │
   │ QQuery 14 │ 2343.99 ms │                               2433.90 ms │     no change │
   │ QQuery 15 │ 2409.99 ms │                               2299.29 ms │     no change │
   │ QQuery 16 │ 2578.48 ms │                               3501.32 ms │  1.36x slower │
   │ QQuery 17 │ 2747.57 ms │                               3403.66 ms │  1.24x slower │
   │ QQuery 18 │ 3208.30 ms │                               4177.87 ms │  1.30x slower │
   │ QQuery 19 │ 1697.91 ms │                               1665.21 ms │     no change │
   │ QQuery 20 │ 2535.77 ms │                               4799.86 ms │  1.89x slower │
   │ QQuery 21 │ 2497.15 ms │                               5106.32 ms │  2.04x slower │
   │ QQuery 22 │ 3141.64 ms │                               6611.66 ms │  2.10x slower │
   │ QQuery 23 │ 5397.43 ms │                              18054.26 ms │  3.34x slower │
   │ QQuery 24 │  408.24 ms │                                568.31 ms │  1.39x slower │
   │ QQuery 25 │ 2232.07 ms │                               2318.57 ms │     no change │
   │ QQuery 26 │  525.89 ms │                                620.19 ms │  1.18x slower │
   │ QQuery 27 │ 2602.04 ms │                               4963.52 ms │  1.91x slower │
   │ QQuery 28 │ 4826.36 ms │                               6713.97 ms │  1.39x slower │
   │ QQuery 29 │ 2122.66 ms │                               1836.95 ms │ +1.16x faster │
   │ QQuery 30 │ 2280.92 ms │                               3404.77 ms │  1.49x slower │
   │ QQuery 31 │ 2284.80 ms │                               4073.63 ms │  1.78x slower │
   │ QQuery 32 │ 2643.10 ms │                               4094.32 ms │  1.55x slower │
   │ QQuery 33 │ 3540.48 ms │                               5650.44 ms │  1.60x slower │
   │ QQuery 34 │ 3504.56 ms │                               5747.44 ms │  1.64x slower │
   │ QQuery 35 │ 2310.77 ms │                               2317.10 ms │     no change │
   │ QQuery 36 │  139.45 ms │                                321.53 ms │  2.31x slower │
   │ QQuery 37 │  113.93 ms │                                177.54 ms │  1.56x slower │
   │ QQuery 38 │  112.60 ms │                                334.45 ms │  2.97x slower │
   │ QQuery 39 │  219.75 ms │                                572.34 ms │  2.60x slower │
   │ QQuery 40 │  115.64 ms │                                184.70 ms │  1.60x slower │
   │ QQuery 41 │  118.17 ms │                                254.13 ms │  2.15x slower │
   │ QQuery 42 │   71.05 ms │                                232.51 ms │  3.27x slower │
   └───────────┴────────────┴──────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                       ┃             ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                       │  85335.63ms │
   │ Total Time (claude_storage-latency-simulation-c76ce0)   │ 126031.04ms │
   │ Average Time (HEAD)                                     │   1984.55ms │
   │ Average Time (claude_storage-latency-simulation-c76ce0) │   2930.95ms │
   │ Queries Faster                                          │           2 │
   │ Queries Slower                                          │          29 │
   │ Queries with No Change                                  │          12 │
   │ Queries with Failure                                    │           0 │
   └─────────────────────────────────────────────────────────┴─────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and claude_storage-latency-simulation-c76ce0
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                   HEAD ┃  claude_storage-latency-simulation-c76ce0 ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 0  │           1.22 / 4.51 ±6.42 / 17.36 ms │              1.20 / 4.44 ±6.35 / 17.13 ms │     no change │
   │ QQuery 1  │ 1842.89 / 2012.25 ±149.50 / 2207.47 ms │     1641.83 / 1688.34 ±35.86 / 1751.66 ms │ +1.19x faster │
   │ QQuery 2  │ 2002.55 / 2260.29 ±137.62 / 2399.00 ms │     2479.97 / 2588.73 ±97.64 / 2754.61 ms │  1.15x slower │
   │ QQuery 3  │ 2095.88 / 2198.55 ±135.51 / 2461.03 ms │     2110.89 / 2201.30 ±78.69 / 2325.83 ms │     no change │
   │ QQuery 4  │ 2254.62 / 2404.22 ±114.24 / 2541.60 ms │    2189.31 / 2369.01 ±107.40 / 2493.66 ms │     no change │
   │ QQuery 5  │  2394.02 / 2461.62 ±51.55 / 2538.69 ms │     2435.43 / 2545.60 ±85.15 / 2675.06 ms │     no change │
   │ QQuery 6  │            1.29 / 1.44 ±0.25 / 1.93 ms │               1.25 / 1.41 ±0.25 / 1.90 ms │     no change │
   │ QQuery 7  │  1794.81 / 1917.25 ±98.41 / 2019.67 ms │    1711.44 / 1867.12 ±134.04 / 2098.38 ms │     no change │
   │ QQuery 8  │  2330.70 / 2433.09 ±57.40 / 2488.90 ms │    2404.94 / 2678.09 ±156.86 / 2829.77 ms │  1.10x slower │
   │ QQuery 9  │  2689.55 / 2805.35 ±80.19 / 2905.91 ms │     3637.71 / 3709.80 ±60.32 / 3781.74 ms │  1.32x slower │
   │ QQuery 10 │  2171.79 / 2223.93 ±40.74 / 2296.38 ms │     2724.26 / 2835.08 ±79.11 / 2917.45 ms │  1.27x slower │
   │ QQuery 11 │  2324.96 / 2384.18 ±43.88 / 2434.60 ms │    2621.21 / 2782.19 ±152.30 / 3071.27 ms │  1.17x slower │
   │ QQuery 12 │ 2254.05 / 2426.86 ±121.27 / 2586.47 ms │    2413.06 / 2578.06 ±106.09 / 2703.66 ms │  1.06x slower │
   │ QQuery 13 │  2446.62 / 2580.79 ±79.54 / 2669.78 ms │    3218.78 / 3353.85 ±120.74 / 3544.65 ms │  1.30x slower │
   │ QQuery 14 │  2343.99 / 2492.45 ±85.79 / 2608.66 ms │    2433.90 / 2622.87 ±118.87 / 2771.06 ms │  1.05x slower │
   │ QQuery 15 │  2409.99 / 2517.23 ±80.37 / 2651.66 ms │    2299.29 / 2464.40 ±133.18 / 2677.95 ms │     no change │
   │ QQuery 16 │ 2578.48 / 2738.69 ±104.93 / 2859.60 ms │     3501.32 / 3593.52 ±92.31 / 3725.99 ms │  1.31x slower │
   │ QQuery 17 │  2747.57 / 2825.11 ±58.92 / 2895.21 ms │    3403.66 / 3581.11 ±134.76 / 3751.82 ms │  1.27x slower │
   │ QQuery 18 │  3208.30 / 3248.69 ±38.06 / 3317.78 ms │     4177.87 / 4306.07 ±81.42 / 4408.65 ms │  1.33x slower │
   │ QQuery 19 │  1697.91 / 1817.63 ±75.74 / 1928.92 ms │    1665.21 / 1834.06 ±102.37 / 1984.35 ms │     no change │
   │ QQuery 20 │  2535.77 / 2653.51 ±66.80 / 2706.12 ms │     4799.86 / 4931.90 ±82.96 / 5047.63 ms │  1.86x slower │
   │ QQuery 21 │  2497.15 / 2656.71 ±89.37 / 2771.86 ms │     5106.32 / 5223.91 ±86.57 / 5375.49 ms │  1.97x slower │
   │ QQuery 22 │  3141.64 / 3197.71 ±72.94 / 3339.58 ms │     6611.66 / 6726.16 ±93.03 / 6876.96 ms │  2.10x slower │
   │ QQuery 23 │ 5397.43 / 5605.51 ±129.79 / 5776.44 ms │ 18054.26 / 18274.91 ±208.65 / 18602.17 ms │  3.26x slower │
   │ QQuery 24 │     408.24 / 473.35 ±61.74 / 581.34 ms │        568.31 / 643.84 ±66.68 / 765.87 ms │  1.36x slower │
   │ QQuery 25 │  2232.07 / 2303.42 ±87.87 / 2467.35 ms │     2318.57 / 2445.98 ±91.13 / 2602.96 ms │  1.06x slower │
   │ QQuery 26 │     525.89 / 561.66 ±40.59 / 640.01 ms │        620.19 / 701.22 ±69.20 / 822.38 ms │  1.25x slower │
   │ QQuery 27 │  2602.04 / 2738.72 ±83.52 / 2829.16 ms │    4963.52 / 5158.08 ±180.38 / 5489.33 ms │  1.88x slower │
   │ QQuery 28 │ 4826.36 / 5038.39 ±118.73 / 5171.44 ms │    6713.97 / 6851.37 ±139.23 / 7104.97 ms │  1.36x slower │
   │ QQuery 29 │  2122.66 / 2248.18 ±96.24 / 2356.25 ms │     1836.95 / 1912.29 ±75.21 / 2035.22 ms │ +1.18x faster │
   │ QQuery 30 │ 2280.92 / 2486.52 ±151.67 / 2653.05 ms │    3404.77 / 3583.89 ±152.42 / 3780.34 ms │  1.44x slower │
   │ QQuery 31 │ 2284.80 / 2422.64 ±103.03 / 2584.74 ms │    4073.63 / 4223.79 ±167.11 / 4542.19 ms │  1.74x slower │
   │ QQuery 32 │  2643.10 / 2774.21 ±92.24 / 2856.07 ms │    4094.32 / 4232.03 ±138.93 / 4443.27 ms │  1.53x slower │
   │ QQuery 33 │ 3540.48 / 3702.85 ±129.91 / 3835.10 ms │     5650.44 / 5713.45 ±60.83 / 5819.52 ms │  1.54x slower │
   │ QQuery 34 │ 3504.56 / 3670.63 ±132.67 / 3912.28 ms │     5747.44 / 5877.43 ±98.72 / 6004.70 ms │  1.60x slower │
   │ QQuery 35 │ 2310.77 / 2480.91 ±135.01 / 2708.05 ms │     2317.10 / 2378.12 ±37.73 / 2433.01 ms │     no change │
   │ QQuery 36 │     139.45 / 234.87 ±93.42 / 405.77 ms │        321.53 / 481.59 ±85.30 / 560.35 ms │  2.05x slower │
   │ QQuery 37 │     113.93 / 151.40 ±32.36 / 200.30 ms │        177.54 / 267.42 ±58.86 / 317.84 ms │  1.77x slower │
   │ QQuery 38 │     112.60 / 214.52 ±57.50 / 263.25 ms │        334.45 / 444.45 ±79.27 / 534.10 ms │  2.07x slower │
   │ QQuery 39 │     219.75 / 269.30 ±50.83 / 360.80 ms │        572.34 / 642.22 ±95.76 / 825.29 ms │  2.38x slower │
   │ QQuery 40 │     115.64 / 166.27 ±39.62 / 221.44 ms │        184.70 / 289.42 ±92.04 / 458.04 ms │  1.74x slower │
   │ QQuery 41 │     118.17 / 148.31 ±20.84 / 183.39 ms │        254.13 / 307.88 ±48.22 / 393.25 ms │  2.08x slower │
   │ QQuery 42 │       71.05 / 84.88 ±22.29 / 129.36 ms │        232.51 / 289.68 ±35.74 / 328.39 ms │  3.41x slower │
   └───────────┴────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                       ┃             ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                       │  90038.62ms │
   │ Total Time (claude_storage-latency-simulation-c76ce0)   │ 131206.06ms │
   │ Average Time (HEAD)                                     │   2093.92ms │
   │ Average Time (claude_storage-latency-simulation-c76ce0) │   3051.30ms │
   │ Queries Faster                                          │           2 │
   │ Queries Slower                                          │          32 │
   │ Queries with No Change                                  │           9 │
   │ Queries with Failure                                    │           0 │
   └─────────────────────────────────────────────────────────┴─────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 455.1s |
   | Peak memory | 16.8 GiB |
   | Avg memory | 2.5 GiB |
   | CPU user | 1020.5s |
   | CPU sys | 152.7s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 660.1s |
   | Peak memory | 16.4 GiB |
   | Avg memory | 2.8 GiB |
   | CPU user | 1313.1s |
   | CPU sys | 543.2s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-6150165867

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24093#issuecomment-6150107152)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6150107152-3356-s4hqs 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing claude/storage-latency-simulation-c76ce0 (228f0257239757cd24445b3b1f3b7be19c8d20a7) to 45de64c (merge-base) [diff](https://github.com/apache/datafusion/compare/45de64cfc199345c8f218acd6d18634105403776..228f0257239757cd24445b3b1f3b7be19c8d20a7)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark clickbench_partitioned
   env:
     SIMULATE_LATENCY: "true"
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-6150518640

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24093#issuecomment-6150107152)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing claude/storage-latency-simulation-c76ce0 (228f0257239757cd24445b3b1f3b7be19c8d20a7) to 45de64c (merge-base) [diff](https://github.com/apache/datafusion/compare/45de64cfc199345c8f218acd6d18634105403776..228f0257239757cd24445b3b1f3b7be19c8d20a7)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   env:
     SIMULATE_LATENCY: "true"
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and claude_storage-latency-simulation-c76ce0
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ claude_storage-latency-simulation-c76ce0 ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │  137.09 ms │                                 24.80 ms │ +5.53x faster │
   │ QQuery 2  │  650.84 ms │                                404.46 ms │ +1.61x faster │
   │ QQuery 3  │  262.45 ms │                                183.09 ms │ +1.43x faster │
   │ QQuery 4  │ 1133.27 ms │                               1364.65 ms │  1.20x slower │
   │ QQuery 5  │  528.91 ms │                                369.15 ms │ +1.43x faster │
   │ QQuery 6  │  393.58 ms │                                334.98 ms │ +1.17x faster │
   │ QQuery 7  │  289.39 ms │                                395.67 ms │  1.37x slower │
   │ QQuery 8  │  553.58 ms │                                258.62 ms │ +2.14x faster │
   │ QQuery 9  │ 1271.16 ms │                                480.56 ms │ +2.65x faster │
   │ QQuery 10 │  611.70 ms │                                453.49 ms │ +1.35x faster │
   │ QQuery 11 │  828.56 ms │                                999.30 ms │  1.21x slower │
   │ QQuery 12 │  297.85 ms │                                143.39 ms │ +2.08x faster │
   │ QQuery 13 │  366.04 ms │                                559.47 ms │  1.53x slower │
   │ QQuery 14 │ 3606.33 ms │                               3240.39 ms │ +1.11x faster │
   │ QQuery 15 │  290.78 ms │                                254.56 ms │ +1.14x faster │
   │ QQuery 16 │  118.56 ms │                                 23.83 ms │ +4.97x faster │
   │ QQuery 17 │  571.82 ms │                                600.64 ms │  1.05x slower │
   │ QQuery 18 │  446.90 ms │                                456.99 ms │     no change │
   │ QQuery 19 │  432.64 ms │                                407.89 ms │ +1.06x faster │
   │ QQuery 20 │  359.63 ms │                                146.12 ms │ +2.46x faster │
   │ QQuery 21 │  483.71 ms │                                215.12 ms │ +2.25x faster │
   │ QQuery 22 │  375.39 ms │                                283.00 ms │ +1.33x faster │
   │ QQuery 23 │ 2127.77 ms │                               1869.42 ms │ +1.14x faster │
   │ QQuery 24 │ 1593.05 ms │                               1556.40 ms │     no change │
   │ QQuery 25 │  732.91 ms │                                485.78 ms │ +1.51x faster │
   │ QQuery 26 │  211.04 ms │                                299.22 ms │  1.42x slower │
   │ QQuery 27 │  216.84 ms │                                 56.06 ms │ +3.87x faster │
   │ QQuery 28 │  268.68 ms │                                288.17 ms │  1.07x slower │
   │ QQuery 29 │  602.59 ms │                                625.93 ms │     no change │
   │ QQuery 30 │  553.13 ms │                                356.30 ms │ +1.55x faster │
   │ QQuery 31 │ 1055.22 ms │                               1134.08 ms │  1.07x slower │
   │ QQuery 32 │  325.00 ms │                                268.45 ms │ +1.21x faster │
   │ QQuery 33 │  454.89 ms │                                301.10 ms │ +1.51x faster │
   │ QQuery 34 │  228.90 ms │                                 81.65 ms │ +2.80x faster │
   │ QQuery 35 │  521.45 ms │                                522.00 ms │     no change │
   │ QQuery 36 │   36.29 ms │                                 24.43 ms │ +1.49x faster │
   │ QQuery 37 │  111.57 ms │                                 76.94 ms │ +1.45x faster │
   │ QQuery 38 │  480.77 ms │                                420.92 ms │ +1.14x faster │
   │ QQuery 39 │ 1674.19 ms │                               1410.01 ms │ +1.19x faster │
   │ QQuery 40 │  412.95 ms │                                182.94 ms │ +2.26x faster │
   │ QQuery 41 │  239.20 ms │                                 80.05 ms │ +2.99x faster │
   │ QQuery 42 │  262.58 ms │                                254.82 ms │     no change │
   │ QQuery 43 │   38.66 ms │                                 23.58 ms │ +1.64x faster │
   │ QQuery 44 │  143.28 ms │                                 39.86 ms │ +3.59x faster │
   │ QQuery 45 │  435.12 ms │                                205.21 ms │ +2.12x faster │
   │ QQuery 46 │  222.96 ms │                                168.94 ms │ +1.32x faster │
   │ QQuery 47 │  909.90 ms │                               1085.77 ms │  1.19x slower │
   │ QQuery 48 │  519.83 ms │                                412.63 ms │ +1.26x faster │
   │ QQuery 49 │  386.87 ms │                                317.29 ms │ +1.22x faster │
   │ QQuery 50 │  571.85 ms │                                347.77 ms │ +1.64x faster │
   │ QQuery 51 │  367.97 ms │                                337.27 ms │ +1.09x faster │
   │ QQuery 52 │  171.97 ms │                                217.64 ms │  1.27x slower │
   │ QQuery 53 │  299.12 ms │                                323.98 ms │  1.08x slower │
   │ QQuery 54 │  823.52 ms │                                689.27 ms │ +1.19x faster │
   │ QQuery 55 │  265.97 ms │                                208.35 ms │ +1.28x faster │
   │ QQuery 56 │  468.31 ms │                                420.29 ms │ +1.11x faster │
   │ QQuery 57 │  990.57 ms │                                975.16 ms │     no change │
   │ QQuery 58 │  981.14 ms │                                903.35 ms │ +1.09x faster │
   │ QQuery 59 │  855.83 ms │                                665.62 ms │ +1.29x faster │
   │ QQuery 60 │  548.20 ms │                                448.57 ms │ +1.22x faster │
   │ QQuery 61 │  304.09 ms │                                 93.07 ms │ +3.27x faster │
   │ QQuery 62 │  336.74 ms │                                231.07 ms │ +1.46x faster │
   │ QQuery 63 │  303.79 ms │                                217.23 ms │ +1.40x faster │
   │ QQuery 64 │ 2691.18 ms │                               2271.09 ms │ +1.18x faster │
   │ QQuery 65 │  306.01 ms │                                470.82 ms │  1.54x slower │
   │ QQuery 66 │  449.39 ms │                                351.07 ms │ +1.28x faster │
   │ QQuery 67 │  614.90 ms │                                583.62 ms │ +1.05x faster │
   │ QQuery 68 │  360.16 ms │                                121.55 ms │ +2.96x faster │
   │ QQuery 69 │  708.44 ms │                                525.09 ms │ +1.35x faster │
   │ QQuery 70 │  604.31 ms │                                535.99 ms │ +1.13x faster │
   │ QQuery 71 │  474.47 ms │                                357.98 ms │ +1.33x faster │
   │ QQuery 72 │ 2337.03 ms │                               2179.13 ms │ +1.07x faster │
   │ QQuery 73 │  160.37 ms │                                 71.72 ms │ +2.24x faster │
   │ QQuery 74 │  683.92 ms │                                616.65 ms │ +1.11x faster │
   │ QQuery 75 │  864.21 ms │                                785.56 ms │ +1.10x faster │
   │ QQuery 76 │  414.87 ms │                                267.25 ms │ +1.55x faster │
   │ QQuery 77 │  653.58 ms │                                500.90 ms │ +1.30x faster │
   │ QQuery 78 │  524.12 ms │                                604.38 ms │  1.15x slower │
   │ QQuery 79 │  354.84 ms │                                340.70 ms │     no change │
   │ QQuery 80 │  594.20 ms │                                608.65 ms │     no change │
   │ QQuery 81 │  582.62 ms │                                321.69 ms │ +1.81x faster │
   │ QQuery 82 │  422.29 ms │                                287.91 ms │ +1.47x faster │
   │ QQuery 83 │  636.38 ms │                                602.75 ms │ +1.06x faster │
   │ QQuery 84 │  254.23 ms │                                176.44 ms │ +1.44x faster │
   │ QQuery 85 │  563.17 ms │                                489.65 ms │ +1.15x faster │
   │ QQuery 86 │  176.65 ms │                                199.22 ms │  1.13x slower │
   │ QQuery 87 │  483.66 ms │                                433.96 ms │ +1.11x faster │
   │ QQuery 88 │  690.13 ms │                                481.86 ms │ +1.43x faster │
   │ QQuery 89 │  325.46 ms │                                264.60 ms │ +1.23x faster │
   │ QQuery 90 │  282.56 ms │                                197.65 ms │ +1.43x faster │
   │ QQuery 91 │  658.43 ms │                                438.32 ms │ +1.50x faster │
   │ QQuery 92 │  507.15 ms │                                239.12 ms │ +2.12x faster │
   │ QQuery 93 │  268.55 ms │                                433.43 ms │  1.61x slower │
   │ QQuery 94 │  413.93 ms │                                335.73 ms │ +1.23x faster │
   │ QQuery 95 │  641.22 ms │                                507.30 ms │ +1.26x faster │
   │ QQuery 96 │  336.18 ms │                                200.69 ms │ +1.68x faster │
   │ QQuery 97 │  293.31 ms │                                250.46 ms │ +1.17x faster │
   │ QQuery 98 │  192.67 ms │                                248.28 ms │  1.29x slower │
   │ QQuery 99 │  519.37 ms │                                383.81 ms │ +1.35x faster │
   └───────────┴────────────┴──────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                       ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                       │ 57108.86ms │
   │ Total Time (claude_storage-latency-simulation-c76ce0)   │ 47911.85ms │
   │ Average Time (HEAD)                                     │   576.86ms │
   │ Average Time (claude_storage-latency-simulation-c76ce0) │   483.96ms │
   │ Queries Faster                                          │         75 │
   │ Queries Slower                                          │         16 │
   │ Queries with No Change                                  │          8 │
   │ Queries with Failure                                    │          0 │
   └─────────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and claude_storage-latency-simulation-c76ce0
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                   HEAD ┃ claude_storage-latency-simulation-c76ce0 ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │     137.09 / 178.72 ±27.41 / 206.40 ms │           24.80 / 29.03 ±3.61 / 35.70 ms │ +6.16x faster │
   │ QQuery 2  │     650.84 / 718.15 ±48.58 / 779.73 ms │      404.46 / 581.48 ±114.72 / 749.22 ms │ +1.24x faster │
   │ QQuery 3  │     262.45 / 342.17 ±70.45 / 437.27 ms │       183.09 / 258.27 ±51.52 / 325.44 ms │ +1.32x faster │
   │ QQuery 4  │  1133.27 / 1236.59 ±92.24 / 1404.90 ms │    1364.65 / 1448.60 ±60.24 / 1513.21 ms │  1.17x slower │
   │ QQuery 5  │     528.91 / 559.54 ±20.91 / 582.74 ms │       369.15 / 444.26 ±62.99 / 539.15 ms │ +1.26x faster │
   │ QQuery 6  │    393.58 / 580.37 ±105.50 / 700.74 ms │      334.98 / 523.69 ±215.71 / 929.30 ms │ +1.11x faster │
   │ QQuery 7  │    289.39 / 557.94 ±145.45 / 682.01 ms │       395.67 / 496.84 ±57.44 / 559.63 ms │ +1.12x faster │
   │ QQuery 8  │     553.58 / 612.89 ±42.17 / 672.40 ms │      258.62 / 431.81 ±115.17 / 621.45 ms │ +1.42x faster │
   │ QQuery 9  │ 1271.16 / 1523.93 ±204.91 / 1769.31 ms │       480.56 / 548.99 ±34.80 / 572.65 ms │ +2.78x faster │
   │ QQuery 10 │     611.70 / 749.69 ±96.38 / 898.31 ms │       453.49 / 604.81 ±80.34 / 678.34 ms │ +1.24x faster │
   │ QQuery 11 │    828.56 / 901.70 ±75.97 / 1046.77 ms │     999.30 / 1074.02 ±69.92 / 1164.15 ms │  1.19x slower │
   │ QQuery 12 │     297.85 / 402.57 ±78.28 / 528.77 ms │       143.39 / 236.91 ±69.07 / 355.08 ms │ +1.70x faster │
   │ QQuery 13 │    366.04 / 684.22 ±160.98 / 798.82 ms │      559.47 / 696.33 ±153.42 / 973.77 ms │     no change │
   │ QQuery 14 │ 3606.33 / 3889.93 ±151.90 / 4011.31 ms │   3240.39 / 3513.67 ±246.03 / 3885.04 ms │ +1.11x faster │
   │ QQuery 15 │     290.78 / 434.16 ±98.49 / 558.87 ms │       254.56 / 369.37 ±86.08 / 514.11 ms │ +1.18x faster │
   │ QQuery 16 │     118.56 / 172.14 ±31.17 / 208.28 ms │           23.83 / 26.90 ±2.56 / 30.91 ms │ +6.40x faster │
   │ QQuery 17 │     571.82 / 696.21 ±81.14 / 801.46 ms │      600.64 / 816.59 ±116.58 / 937.52 ms │  1.17x slower │
   │ QQuery 18 │    446.90 / 588.51 ±110.49 / 718.38 ms │       456.99 / 526.53 ±97.21 / 713.39 ms │ +1.12x faster │
   │ QQuery 19 │    432.64 / 617.41 ±145.53 / 853.20 ms │      407.89 / 587.15 ±167.72 / 857.95 ms │     no change │
   │ QQuery 20 │     359.63 / 444.75 ±59.39 / 534.62 ms │      146.12 / 268.75 ±112.35 / 445.20 ms │ +1.65x faster │
   │ QQuery 21 │     483.71 / 521.10 ±31.14 / 564.39 ms │      215.12 / 416.31 ±132.20 / 607.22 ms │ +1.25x faster │
   │ QQuery 22 │    375.39 / 483.49 ±107.04 / 642.63 ms │       283.00 / 366.62 ±69.04 / 485.35 ms │ +1.32x faster │
   │ QQuery 23 │ 2127.77 / 2345.08 ±153.97 / 2584.66 ms │   1869.42 / 2062.21 ±161.41 / 2322.02 ms │ +1.14x faster │
   │ QQuery 24 │ 1593.05 / 1850.80 ±197.00 / 2108.05 ms │   1556.40 / 1777.26 ±132.85 / 1916.67 ms │     no change │
   │ QQuery 25 │      732.91 / 743.28 ±9.25 / 759.33 ms │      485.78 / 668.33 ±126.84 / 822.16 ms │ +1.11x faster │
   │ QQuery 26 │    211.04 / 421.23 ±126.24 / 533.41 ms │       299.22 / 398.38 ±53.12 / 445.37 ms │ +1.06x faster │
   │ QQuery 27 │     216.84 / 301.61 ±70.68 / 387.84 ms │        56.06 / 118.31 ±72.79 / 260.97 ms │ +2.55x faster │
   │ QQuery 28 │     268.68 / 380.82 ±57.76 / 428.71 ms │       288.17 / 311.98 ±19.64 / 338.45 ms │ +1.22x faster │
   │ QQuery 29 │    602.59 / 768.42 ±143.47 / 999.58 ms │       625.93 / 706.82 ±72.98 / 838.65 ms │ +1.09x faster │
   │ QQuery 30 │    553.13 / 672.96 ±115.45 / 872.17 ms │       356.30 / 442.91 ±72.36 / 532.29 ms │ +1.52x faster │
   │ QQuery 31 │ 1055.22 / 1300.72 ±163.56 / 1558.29 ms │   1134.08 / 1449.68 ±312.21 / 2021.30 ms │  1.11x slower │
   │ QQuery 32 │     325.00 / 455.29 ±95.45 / 599.38 ms │      268.45 / 464.48 ±117.58 / 587.14 ms │     no change │
   │ QQuery 33 │     454.89 / 552.11 ±73.11 / 640.04 ms │       301.10 / 409.87 ±62.47 / 478.11 ms │ +1.35x faster │
   │ QQuery 34 │     228.90 / 312.31 ±47.09 / 364.15 ms │        81.65 / 147.34 ±66.15 / 273.13 ms │ +2.12x faster │
   │ QQuery 35 │     521.45 / 652.59 ±93.43 / 806.65 ms │       522.00 / 635.24 ±85.31 / 749.59 ms │     no change │
   │ QQuery 36 │       36.29 / 72.48 ±31.15 / 117.28 ms │           24.43 / 27.74 ±4.13 / 35.61 ms │ +2.61x faster │
   │ QQuery 37 │     111.57 / 241.02 ±75.35 / 318.18 ms │        76.94 / 122.73 ±30.34 / 168.26 ms │ +1.96x faster │
   │ QQuery 38 │     480.77 / 605.07 ±86.27 / 692.68 ms │       420.92 / 555.73 ±95.72 / 708.68 ms │ +1.09x faster │
   │ QQuery 39 │ 1674.19 / 1890.37 ±145.29 / 2051.60 ms │    1410.01 / 1517.11 ±80.25 / 1628.56 ms │ +1.25x faster │
   │ QQuery 40 │     412.95 / 475.00 ±41.20 / 540.55 ms │       182.94 / 309.52 ±93.07 / 470.37 ms │ +1.53x faster │
   │ QQuery 41 │     239.20 / 259.52 ±31.21 / 320.93 ms │         80.05 / 89.18 ±10.04 / 107.82 ms │ +2.91x faster │
   │ QQuery 42 │    262.58 / 373.83 ±109.83 / 508.03 ms │       254.82 / 311.88 ±45.66 / 379.41 ms │ +1.20x faster │
   │ QQuery 43 │      38.66 / 111.64 ±61.47 / 186.78 ms │           23.58 / 27.89 ±4.22 / 34.94 ms │ +4.00x faster │
   │ QQuery 44 │     143.28 / 216.85 ±56.39 / 292.07 ms │          39.86 / 51.79 ±10.32 / 68.09 ms │ +4.19x faster │
   │ QQuery 45 │     435.12 / 554.15 ±64.67 / 631.60 ms │      205.21 / 436.04 ±213.39 / 709.32 ms │ +1.27x faster │
   │ QQuery 46 │     222.96 / 326.56 ±54.50 / 379.65 ms │       168.94 / 263.49 ±87.38 / 404.71 ms │ +1.24x faster │
   │ QQuery 47 │  909.90 / 1239.07 ±165.97 / 1362.30 ms │   1085.77 / 1302.08 ±170.91 / 1611.47 ms │  1.05x slower │
   │ QQuery 48 │    519.83 / 647.92 ±117.39 / 817.74 ms │       412.63 / 512.28 ±79.01 / 611.26 ms │ +1.26x faster │
   │ QQuery 49 │     386.87 / 522.52 ±69.94 / 583.69 ms │      317.29 / 465.61 ±102.95 / 589.48 ms │ +1.12x faster │
   │ QQuery 50 │     571.85 / 690.25 ±73.00 / 794.31 ms │      347.77 / 594.83 ±144.55 / 782.31 ms │ +1.16x faster │
   │ QQuery 51 │     367.97 / 511.85 ±86.88 / 614.63 ms │       337.27 / 447.17 ±85.28 / 551.05 ms │ +1.14x faster │
   │ QQuery 52 │    171.97 / 305.32 ±100.44 / 424.04 ms │       217.64 / 253.02 ±24.32 / 291.43 ms │ +1.21x faster │
   │ QQuery 53 │     299.12 / 397.43 ±80.58 / 544.28 ms │       323.98 / 375.31 ±43.60 / 448.50 ms │ +1.06x faster │
   │ QQuery 54 │   823.52 / 968.40 ±126.52 / 1203.48 ms │       689.27 / 767.44 ±65.00 / 848.65 ms │ +1.26x faster │
   │ QQuery 55 │     265.97 / 392.76 ±97.61 / 558.77 ms │      208.35 / 338.44 ±116.62 / 532.77 ms │ +1.16x faster │
   │ QQuery 56 │     468.31 / 576.12 ±73.44 / 693.50 ms │       420.29 / 480.67 ±49.00 / 542.17 ms │ +1.20x faster │
   │ QQuery 57 │  990.57 / 1119.66 ±117.23 / 1319.75 ms │     975.16 / 1154.82 ±98.55 / 1272.87 ms │     no change │
   │ QQuery 58 │  981.14 / 1275.76 ±193.50 / 1539.08 ms │    903.35 / 1161.84 ±146.59 / 1334.89 ms │ +1.10x faster │
   │ QQuery 59 │  855.83 / 1023.03 ±158.10 / 1292.78 ms │      665.62 / 864.79 ±137.25 / 995.65 ms │ +1.18x faster │
   │ QQuery 60 │     548.20 / 622.58 ±39.43 / 657.65 ms │      448.57 / 536.73 ±120.71 / 765.79 ms │ +1.16x faster │
   │ QQuery 61 │     304.09 / 373.78 ±43.69 / 415.45 ms │        93.07 / 163.91 ±40.39 / 214.52 ms │ +2.28x faster │
   │ QQuery 62 │     336.74 / 486.84 ±99.62 / 650.08 ms │       231.07 / 407.25 ±95.96 / 513.46 ms │ +1.20x faster │
   │ QQuery 63 │     303.79 / 405.29 ±75.84 / 537.23 ms │       217.23 / 372.01 ±88.06 / 471.44 ms │ +1.09x faster │
   │ QQuery 64 │ 2691.18 / 3094.60 ±279.96 / 3408.63 ms │   2271.09 / 2650.41 ±281.33 / 3075.62 ms │ +1.17x faster │
   │ QQuery 65 │    306.01 / 557.13 ±141.49 / 720.60 ms │      470.82 / 674.67 ±143.54 / 840.30 ms │  1.21x slower │
   │ QQuery 66 │     449.39 / 488.69 ±36.48 / 543.13 ms │      351.07 / 473.95 ±109.58 / 667.99 ms │     no change │
   │ QQuery 67 │     614.90 / 669.05 ±42.22 / 712.19 ms │       583.62 / 603.16 ±19.82 / 640.57 ms │ +1.11x faster │
   │ QQuery 68 │     360.16 / 409.35 ±47.61 / 477.14 ms │      121.55 / 232.31 ±100.73 / 396.53 ms │ +1.76x faster │
   │ QQuery 69 │   708.44 / 850.00 ±106.67 / 1022.28 ms │       525.09 / 568.84 ±51.29 / 641.20 ms │ +1.49x faster │
   │ QQuery 70 │   604.31 / 767.85 ±147.71 / 1045.60 ms │       535.99 / 643.41 ±89.14 / 763.17 ms │ +1.19x faster │
   │ QQuery 71 │     474.47 / 490.22 ±12.64 / 506.33 ms │       357.98 / 442.48 ±53.91 / 516.42 ms │ +1.11x faster │
   │ QQuery 72 │ 2337.03 / 2484.39 ±205.04 / 2888.55 ms │   2179.13 / 2349.45 ±123.84 / 2565.20 ms │ +1.06x faster │
   │ QQuery 73 │     160.37 / 227.92 ±45.91 / 302.71 ms │        71.72 / 107.98 ±48.01 / 203.04 ms │ +2.11x faster │
   │ QQuery 74 │   683.92 / 804.45 ±146.79 / 1079.39 ms │     616.65 / 870.19 ±192.80 / 1148.33 ms │  1.08x slower │
   │ QQuery 75 │    864.21 / 926.33 ±70.38 / 1054.91 ms │       785.56 / 832.70 ±47.26 / 922.55 ms │ +1.11x faster │
   │ QQuery 76 │     414.87 / 514.10 ±59.74 / 590.05 ms │       267.25 / 437.75 ±99.33 / 543.20 ms │ +1.17x faster │
   │ QQuery 77 │     653.58 / 696.28 ±41.50 / 756.61 ms │       500.90 / 601.73 ±85.26 / 712.61 ms │ +1.16x faster │
   │ QQuery 78 │    524.12 / 716.66 ±163.71 / 929.28 ms │       604.38 / 700.26 ±77.58 / 834.66 ms │     no change │
   │ QQuery 79 │    354.84 / 489.62 ±106.06 / 652.02 ms │       340.70 / 403.78 ±54.38 / 496.41 ms │ +1.21x faster │
   │ QQuery 80 │     594.20 / 697.40 ±63.09 / 784.30 ms │      608.65 / 774.71 ±107.83 / 905.64 ms │  1.11x slower │
   │ QQuery 81 │     582.62 / 654.17 ±42.93 / 713.40 ms │      321.69 / 593.30 ±161.86 / 783.22 ms │ +1.10x faster │
   │ QQuery 82 │    422.29 / 546.56 ±114.04 / 737.43 ms │      287.91 / 441.60 ±117.42 / 630.20 ms │ +1.24x faster │
   │ QQuery 83 │   636.38 / 829.60 ±127.68 / 1025.86 ms │      602.75 / 792.91 ±149.23 / 986.40 ms │     no change │
   │ QQuery 84 │    254.23 / 475.91 ±118.82 / 581.33 ms │       176.44 / 295.90 ±85.87 / 374.32 ms │ +1.61x faster │
   │ QQuery 85 │     563.17 / 748.44 ±94.27 / 824.17 ms │       489.65 / 592.13 ±76.50 / 724.26 ms │ +1.26x faster │
   │ QQuery 86 │     176.65 / 272.28 ±74.84 / 380.62 ms │       199.22 / 251.45 ±57.57 / 362.41 ms │ +1.08x faster │
   │ QQuery 87 │     483.66 / 556.18 ±77.23 / 698.93 ms │       433.96 / 560.07 ±97.67 / 673.48 ms │     no change │
   │ QQuery 88 │     690.13 / 765.67 ±42.03 / 803.56 ms │       481.86 / 528.68 ±39.55 / 576.20 ms │ +1.45x faster │
   │ QQuery 89 │     325.46 / 464.98 ±72.05 / 534.04 ms │       264.60 / 372.85 ±71.86 / 481.61 ms │ +1.25x faster │
   │ QQuery 90 │     282.56 / 353.05 ±48.60 / 434.36 ms │       197.65 / 265.25 ±39.25 / 308.70 ms │ +1.33x faster │
   │ QQuery 91 │    658.43 / 766.45 ±109.29 / 922.84 ms │       438.32 / 507.51 ±97.51 / 699.46 ms │ +1.51x faster │
   │ QQuery 92 │     507.15 / 522.71 ±20.32 / 562.43 ms │      239.12 / 416.21 ±106.46 / 558.73 ms │ +1.26x faster │
   │ QQuery 93 │     268.55 / 362.16 ±73.92 / 476.86 ms │       433.43 / 500.13 ±48.42 / 569.51 ms │  1.38x slower │
   │ QQuery 94 │    413.93 / 553.90 ±181.40 / 906.73 ms │      335.73 / 515.02 ±133.83 / 709.52 ms │ +1.08x faster │
   │ QQuery 95 │     641.22 / 804.52 ±99.98 / 913.33 ms │      507.30 / 761.95 ±164.08 / 913.74 ms │ +1.06x faster │
   │ QQuery 96 │     336.18 / 422.40 ±84.36 / 571.32 ms │       200.69 / 323.15 ±64.90 / 390.80 ms │ +1.31x faster │
   │ QQuery 97 │    293.31 / 495.60 ±107.43 / 589.78 ms │       250.46 / 338.70 ±60.58 / 438.36 ms │ +1.46x faster │
   │ QQuery 98 │     192.67 / 353.23 ±95.72 / 468.03 ms │       248.28 / 306.13 ±51.02 / 371.16 ms │ +1.15x faster │
   │ QQuery 99 │     519.37 / 608.39 ±84.63 / 746.58 ms │      383.81 / 491.41 ±110.91 / 696.21 ms │ +1.24x faster │
   └───────────┴────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                       ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                       │ 69778.64ms │
   │ Total Time (claude_storage-latency-simulation-c76ce0)   │ 59993.15ms │
   │ Average Time (HEAD)                                     │   704.83ms │
   │ Average Time (claude_storage-latency-simulation-c76ce0) │   605.99ms │
   │ Queries Faster                                          │         80 │
   │ Queries Slower                                          │          9 │
   │ Queries with No Change                                  │         10 │
   │ Queries with Failure                                    │          0 │
   └─────────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 355.1s |
   | Peak memory | 990.6 MiB |
   | Avg memory | 543.5 MiB |
   | CPU user | 211.8s |
   | CPU sys | 11.2s |
   | Peak spill | 0 B |
   
   **tpcds — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 305.1s |
   | Peak memory | 902.3 MiB |
   | Avg memory | 368.4 MiB |
   | CPU user | 228.7s |
   | CPU sys | 31.8s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #24093:
URL: https://github.com/apache/datafusion/pull/24093


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#discussion_r4255435344


##########
benchmarks/src/util/simulated_object_store.rs:
##########
@@ -0,0 +1,506 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! An [`ObjectStore`] that makes a local filesystem behave like S3 or GCS.
+//!
+//! Each request costs:
+//!
+//! 1. a connection from a pool of [`MAX_CONNECTIONS`],
+//! 2. a time to first byte, drawn at random from a latency distribution,
+//! 3. a transfer time at [`BYTES_PER_SECOND`] for each response body.
+//!
+//! The store does not override [`ObjectStore::get_ranges`]. `AmazonS3` and
+//! `GoogleCloudStorage` do not override it either. The trait default merges
+//! ranges that are less than 1MiB apart and sends each merged range as a
+//! separate GET. Thus the simulated request count is the same as on S3. A
+//! wrapper that overrides `get_ranges` charges one request where S3 charges
+//! several.
+//!
+//! The store does not model request failures, retries or a shared network
+//! limit.
+
+use std::fmt;
+use std::sync::Arc;
+use std::time::Duration;
+
+use async_trait::async_trait;
+use futures::StreamExt;
+use futures::stream::BoxStream;
+use object_store::path::Path;
+use object_store::{
+    CopyOptions, GetOptions, GetResult, GetResultPayload, ListResult, MultipartUpload,
+    ObjectMeta, ObjectStore, PutMultipartOptions, PutOptions, PutPayload, PutResult,
+    Result,
+};
+use rand::Rng;
+use tokio::sync::{OwnedSemaphorePermit, Semaphore};
+use tokio::time::Instant;
+
+/// GET time to first byte, inspired by S3 latencies.
+/// Sorted: 25,25,28,28,30,30,30,30,32,32,35, 70,85,100,100,110, 130,150, 180,200
+/// P50≈32ms, P90≈110ms, P99≈200ms
+const GET_TTFB_MS: &[u64] = &[

Review Comment:
   Would be good to measure/callibrat3 it against s3 / gcs to confirm this is realistic. Did you try that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#discussion_r4255625414


##########
benchmarks/src/util/simulated_object_store.rs:
##########
@@ -0,0 +1,506 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! An [`ObjectStore`] that makes a local filesystem behave like S3 or GCS.
+//!
+//! Each request costs:
+//!
+//! 1. a connection from a pool of [`MAX_CONNECTIONS`],
+//! 2. a time to first byte, drawn at random from a latency distribution,
+//! 3. a transfer time at [`BYTES_PER_SECOND`] for each response body.
+//!
+//! The store does not override [`ObjectStore::get_ranges`]. `AmazonS3` and
+//! `GoogleCloudStorage` do not override it either. The trait default merges
+//! ranges that are less than 1MiB apart and sends each merged range as a
+//! separate GET. Thus the simulated request count is the same as on S3. A
+//! wrapper that overrides `get_ranges` charges one request where S3 charges
+//! several.
+//!
+//! The store does not model request failures, retries or a shared network
+//! limit.
+
+use std::fmt;
+use std::sync::Arc;
+use std::time::Duration;
+
+use async_trait::async_trait;
+use futures::StreamExt;
+use futures::stream::BoxStream;
+use object_store::path::Path;
+use object_store::{
+    CopyOptions, GetOptions, GetResult, GetResultPayload, ListResult, MultipartUpload,
+    ObjectMeta, ObjectStore, PutMultipartOptions, PutOptions, PutPayload, PutResult,
+    Result,
+};
+use rand::Rng;
+use tokio::sync::{OwnedSemaphorePermit, Semaphore};
+use tokio::time::Instant;
+
+/// GET time to first byte, inspired by S3 latencies.
+/// Sorted: 25,25,28,28,30,30,30,30,32,32,35, 70,85,100,100,110, 130,150, 180,200
+/// P50≈32ms, P90≈110ms, P99≈200ms
+const GET_TTFB_MS: &[u64] = &[

Review Comment:
   No, but I agree we should. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] bench: model remote object storage faithfully for --simulate-latency [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24093:
URL: https://github.com/apache/datafusion/pull/24093#issuecomment-6150376696

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24093#issuecomment-6150107152)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing claude/storage-latency-simulation-c76ce0 (228f0257239757cd24445b3b1f3b7be19c8d20a7) to 45de64c (merge-base) [diff](https://github.com/apache/datafusion/compare/45de64cfc199345c8f218acd6d18634105403776..228f0257239757cd24445b3b1f3b7be19c8d20a7)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   env:
     SIMULATE_LATENCY: "true"
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and claude_storage-latency-simulation-c76ce0
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ claude_storage-latency-simulation-c76ce0 ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │  471.88 ms │                                531.58 ms │  1.13x slower │
   │ QQuery 2  │  825.73 ms │                                660.11 ms │ +1.25x faster │
   │ QQuery 3  │  821.64 ms │                                915.55 ms │  1.11x slower │
   │ QQuery 4  │  587.37 ms │                                776.08 ms │  1.32x slower │
   │ QQuery 5  │  849.05 ms │                                921.58 ms │  1.09x slower │
   │ QQuery 6  │  451.42 ms │                                470.62 ms │     no change │
   │ QQuery 7  │ 1005.17 ms │                                870.31 ms │ +1.15x faster │
   │ QQuery 8  │  810.49 ms │                                916.69 ms │  1.13x slower │
   │ QQuery 9  │ 1173.43 ms │                               1015.74 ms │ +1.16x faster │
   │ QQuery 10 │  666.73 ms │                                818.39 ms │  1.23x slower │
   │ QQuery 11 │  691.86 ms │                                590.25 ms │ +1.17x faster │
   │ QQuery 12 │  635.24 ms │                                929.79 ms │  1.46x slower │
   │ QQuery 13 │  243.58 ms │                                489.73 ms │  2.01x slower │
   │ QQuery 14 │  670.75 ms │                                609.85 ms │ +1.10x faster │
   │ QQuery 15 │ 1056.03 ms │                               1140.06 ms │  1.08x slower │
   │ QQuery 16 │  195.71 ms │                                291.39 ms │  1.49x slower │
   │ QQuery 17 │ 1021.79 ms │                               1150.97 ms │  1.13x slower │
   │ QQuery 18 │ 1097.98 ms │                               1289.77 ms │  1.17x slower │
   │ QQuery 19 │  576.76 ms │                                682.59 ms │  1.18x slower │
   │ QQuery 20 │  822.73 ms │                                837.01 ms │     no change │
   │ QQuery 21 │ 1746.65 ms │                               1924.38 ms │  1.10x slower │
   │ QQuery 22 │  478.62 ms │                                381.61 ms │ +1.25x faster │
   └───────────┴────────────┴──────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                       ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                       │ 16900.61ms │
   │ Total Time (claude_storage-latency-simulation-c76ce0)   │ 18214.08ms │
   │ Average Time (HEAD)                                     │   768.21ms │
   │ Average Time (claude_storage-latency-simulation-c76ce0) │   827.91ms │
   │ Queries Faster                                          │          6 │
   │ Queries Slower                                          │         14 │
   │ Queries with No Change                                  │          2 │
   │ Queries with Failure                                    │          0 │
   └─────────────────────────────────────────────────────────┴────────────┘
   
   Distribution per query (min / mean ±stddev / max):
   
   Comparing HEAD and claude_storage-latency-simulation-c76ce0
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                   HEAD ┃ claude_storage-latency-simulation-c76ce0 ┃        Change ┃
   ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   │ QQuery 1  │     471.88 / 563.90 ±64.44 / 665.76 ms │       531.58 / 621.88 ±68.88 / 735.58 ms │  1.10x slower │
   │ QQuery 2  │    825.73 / 928.38 ±72.68 / 1019.54 ms │       660.11 / 802.01 ±93.19 / 907.02 ms │ +1.16x faster │
   │ QQuery 3  │     821.64 / 866.19 ±37.27 / 934.85 ms │    915.55 / 1102.41 ±146.53 / 1298.77 ms │  1.27x slower │
   │ QQuery 4  │    587.37 / 775.91 ±100.43 / 864.42 ms │      776.08 / 920.61 ±84.50 / 1001.85 ms │  1.19x slower │
   │ QQuery 5  │    849.05 / 909.36 ±89.74 / 1086.99 ms │      921.58 / 976.63 ±59.17 / 1085.20 ms │  1.07x slower │
   │ QQuery 6  │     451.42 / 539.33 ±88.98 / 700.44 ms │       470.62 / 597.40 ±87.61 / 721.60 ms │  1.11x slower │
   │ QQuery 7  │  1005.17 / 1035.02 ±21.41 / 1062.23 ms │      870.31 / 980.48 ±86.13 / 1076.56 ms │ +1.06x faster │
   │ QQuery 8  │  810.49 / 1017.64 ±129.19 / 1174.51 ms │    916.69 / 1173.80 ±211.12 / 1527.74 ms │  1.15x slower │
   │ QQuery 9  │ 1173.43 / 1456.66 ±191.76 / 1766.41 ms │   1015.74 / 1396.59 ±203.89 / 1625.01 ms │     no change │
   │ QQuery 10 │   666.73 / 882.20 ±155.11 / 1132.84 ms │     818.39 / 985.44 ±136.80 / 1209.61 ms │  1.12x slower │
   │ QQuery 11 │     691.86 / 812.45 ±77.37 / 929.00 ms │     590.25 / 819.44 ±199.03 / 1109.89 ms │     no change │
   │ QQuery 12 │     635.24 / 756.19 ±86.15 / 878.32 ms │    929.79 / 1063.17 ±101.57 / 1204.10 ms │  1.41x slower │
   │ QQuery 13 │    243.58 / 423.13 ±103.11 / 543.11 ms │       489.73 / 517.07 ±23.18 / 546.69 ms │  1.22x slower │
   │ QQuery 14 │     670.75 / 718.55 ±38.55 / 787.00 ms │      609.85 / 751.95 ±128.91 / 961.10 ms │     no change │
   │ QQuery 15 │  1056.03 / 1168.94 ±68.90 / 1248.39 ms │   1140.06 / 1223.98 ±101.22 / 1385.42 ms │     no change │
   │ QQuery 16 │     195.71 / 382.49 ±93.85 / 445.21 ms │       291.39 / 363.72 ±68.72 / 478.21 ms │     no change │
   │ QQuery 17 │ 1021.79 / 1264.48 ±161.57 / 1441.25 ms │   1150.97 / 1384.46 ±140.17 / 1544.60 ms │  1.09x slower │
   │ QQuery 18 │ 1097.98 / 1303.92 ±111.88 / 1432.92 ms │   1289.77 / 1418.77 ±142.70 / 1685.60 ms │  1.09x slower │
   │ QQuery 19 │    576.76 / 736.69 ±142.12 / 940.23 ms │     682.59 / 805.94 ±154.94 / 1095.09 ms │  1.09x slower │
   │ QQuery 20 │   822.73 / 932.30 ±114.53 / 1072.12 ms │    837.01 / 1004.42 ±148.14 / 1223.30 ms │  1.08x slower │
   │ QQuery 21 │ 1746.65 / 1989.05 ±159.28 / 2182.45 ms │   1924.38 / 2126.91 ±157.92 / 2352.52 ms │  1.07x slower │
   │ QQuery 22 │     478.62 / 502.15 ±28.20 / 537.01 ms │      381.61 / 506.16 ±106.65 / 641.40 ms │     no change │
   └───────────┴────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                       ┃            ┃
   ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   │ Total Time (HEAD)                                       │ 19964.91ms │
   │ Total Time (claude_storage-latency-simulation-c76ce0)   │ 21543.25ms │
   │ Average Time (HEAD)                                     │   907.50ms │
   │ Average Time (claude_storage-latency-simulation-c76ce0) │   979.24ms │
   │ Queries Faster                                          │          2 │
   │ Queries Slower                                          │         14 │
   │ Queries with No Change                                  │          6 │
   │ Queries with Failure                                    │          0 │
   └─────────────────────────────────────────────────────────┴────────────┘
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 105.0s |
   | Peak memory | 745.7 MiB |
   | Avg memory | 456.2 MiB |
   | CPU user | 28.1s |
   | CPU sys | 3.4s |
   | Peak spill | 0 B |
   
   **tpch — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 110.0s |
   | Peak memory | 597.4 MiB |
   | Avg memory | 359.0 MiB |
   | CPU user | 40.3s |
   | CPU sys | 18.2s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org