You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "james-willis (via GitHub)" <gi...@apache.org> on 2026/10/01 22:29:39 UTC

[PR] perf(rust/sedona-raster-gdal): read OutDb files concurrently under a shared I/O budget [sedona-db]

james-willis opened a new pull request, #1385:
URL: https://github.com/apache/sedona-db/pull/1385

   `GdalLoader::load` read all of a call's requests one after another in a single `spawn_blocking` task. `EnsureLoadedExec` hands it one byte-budget slice per call, so every OutDb file in a slice was read in series. For COGs on object storage that is one chain of round trips per file, end to end. The trait docs already expect backends to fan out, and `ZarrLoader` does.
   
   ## What changes
   
   - Requests are grouped by file. Each file is read by its own `spawn_blocking` task, and results come back in request order. When several requests name one file (one per band), that file is opened once and its bands are read in turn on one thread. GDAL handles never cross threads, and the per-thread dataset cache, the dtype check and `#band=N` handling are unchanged.
   - An I/O budget bounds the files read at once. It is a `tokio::sync::Semaphore` shared by every `load` call on the loader and its clones, so it caps all partitions together rather than each one. It is configured with `GdalLoader::with_concurrency` and `GdalLoader::concurrency`, mirroring `ZarrLoader`. The default is `DEFAULT_LOAD_CONCURRENCY = 32`.
   - Cancellation: each call keeps its `CancelOnDrop` flag, now shared by all of the call's tasks.
     - Dropping the call stops running reads at their next strip, and files still waiting for a permit never start.
     - If one file fails, `try_collect` returns its error, and the same guard stops the other reads.
     - The permit moves into the blocking closure, so an abandoned read keeps its slot in the budget until it has actually returned.
   - `benches/gdal_io_concurrency.rs` is modelled on `zarr_io_concurrency.rs`. It sweeps the budget over 64 COGs, with one caller and with eight callers sharing the budget. It runs against the local filesystem and against an HTTP server that adds a fixed latency to every request, read through `/vsicurl/`. Each iteration reads freshly named hard links, so every read is a cold open.
   
   ## Choosing the default
   
   Bench results (Apple M-series, 12 cores; 64 COGs of 1024 × 1024 uint8, DEFLATE):
   
   | budget | HTTP +10 ms | HTTP +50 ms | local |
   |---|---|---|---|
   | 1 | 3.69 s | 14.4 s | 80.7 ms |
   | 8 | 437 ms | 1.80 s | 12.0 ms |
   | 16 | 215 ms | 911 ms | 10.4 ms |
   | 32 | 112 ms | 459 ms | 9.7 ms |
   | 64 | 63 ms | 237 ms | 8.7 ms |
   
   These rows are for one caller. Eight callers sharing the budget match them within 2%. Over HTTP the time halves every time the budget doubles, until every file is in flight at once, so the bench alone has no knee. On local files a larger budget never made things worse.
   
   The knee comes from real object storage. The workload was 640 public SpatialBench SF1 COGs (1113 × 1113 uint8 ZSTD, read unsigned from `s3://wherobots-benchmark-prod`), read over a home connection, all from one batch:
   
   ```sql
   SELECT count(*), count(m) FROM (
     SELECT RS_SummaryStats(RS_FromPath(assets['tasmax']), 'mean', 1) AS m
     FROM temporal WHERE footprint_id < 80)
   ```
   
   | build | query time | COGs/s |
   |---|---|---|
   | main (nightly 0.5.0a260) | 206 s, 254 s | 2.5–3.1 |
   | this PR, budget 1 | 210 s | 3.0 |
   | this PR, budget 8 | 63.7 s | 10.0 |
   | this PR, budget 16 | 54.9 s | 11.7 |
   | this PR, budget 32 (default) | 51.1 s, 51.3 s, 52.4 s | 12.5 |
   | this PR, budget 64 | 50.1 s | 12.8 |
   | this PR, budget 128 | 50.3 s | 12.7 |
   
   On both builds, about 41 s of every run goes to `RS_FromPath`'s own opens, which happen before any load: a query that only calls `RS_Width(RS_FromPath(...))` takes 40–41 s. Subtracting that, loading went from 169 s to about 10 s at budget 32, roughly 17×. Past 32 the store is the limit, not the budget. 64 was not taken, for two reasons:
   - Each blocking thread keeps its own dataset cache, so a bigger budget means more threads, each holding open datasets.
   - In the bench, peak open file descriptors grew with the budget (about 650 at 8, 1,300 at 16 and 2,200 at 32 or more, where the 64 distinct files cap it).
   
   Memory in flight does not grow with the budget, because `EnsureLoadedExec` already caps the bytes a single call returns.
   
   ## Notes for review
   
   - The fan-out spreads local-file datasets over more blocking threads. Reading 400 local files in one batch peaked at about 410 open descriptors on this branch (at budgets 8 to 64) against 45 on main. The per-thread cache is kept as it was. Not keeping datasets open in the loader path is a possible follow-up if that matters; the session chunk cache already serves repeat loads.
   - `RS_FromPath`'s serial opens are now most of what is left on remote data. That is a separate change.
   
   ## Tests
   
   - Rust (`sedona-raster-gdal`):
     - request order across files and bands, at budget 1 and at the default;
     - a 48-file batch at budget 8;
     - eight concurrent calls on clones with a budget of 2 never having more than 2 files in flight;
     - one missing file failing the whole call at several budgets, with all permits returned;
     - dropping a call that is waiting for budget starts no reads and leaks no permits;
     - `with_concurrency` clamping to at least 1.
   - Python: `test_rs_ensureloaded_many_files_in_one_batch` loads 40 interleaved rows over five fixture files in one batch and checks each row against that file's mean computed on its own.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(rust/sedona-raster-gdal): open and read OutDb files concurrently under one shared I/O budget [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #1385:
URL: https://github.com/apache/sedona-db/pull/1385#discussion_r4341948572


##########
rust/sedona-raster-gdal/benches/gdal_io_concurrency.rs:
##########
@@ -0,0 +1,296 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The I/O budget of `GdalLoader` (`with_concurrency`): how long a fixed
+//! set of COGs takes to load as the budget grows, with one caller and with
+//! several callers sharing the budget the way DataFusion partitions do. Two
+//! stores: the page-cached local filesystem, where reads are cheap and
+//! decoding dominates, and an HTTP server (read through `/vsicurl/`) that
+//! sleeps for a fixed time on every request, a stand-in for object storage
+//! where the round trips dominate. The knee of the HTTP curve is what the
+//! default budget is chosen from; the local curve shows what that budget
+//! costs when there is no latency to hide.
+//!
+//! Every iteration reads the files under fresh names (hard links in a new
+//! directory, made outside the timed section), so neither GDAL's caches nor
+//! the loader's per-thread dataset cache serves a later iteration from an
+//! earlier one: each read is a cold open, as it is for a query that reads
+//! each COG once.
+//!
+//! Run with `cargo bench -p sedona-raster-gdal --bench gdal_io_concurrency`.
+//! `GDAL_BENCH_LATENCY_MS` sets the HTTP store's per-request latency
+//! (default 10). The HTTP store needs `python3` on the path; without it that
+//! half is skipped.
+
+use std::net::{TcpListener, TcpStream};
+use std::path::Path;
+use std::process::{Child, Command, Stdio};
+use std::sync::atomic::{AtomicUsize, Ordering};
+use std::time::{Duration, Instant};
+
+use criterion::{BatchSize, BenchmarkId, Criterion, criterion_group, criterion_main};
+use sedona_gdal::global::with_global_gdal;
+use sedona_gdal::raster::types::{Buffer as GdalBuffer, GdalDataType};
+use sedona_raster::raster_loader::{AsyncRasterLoader, RasterLoadRequest};
+use sedona_raster::view_entries::ViewEntries;
+use sedona_raster_gdal::GdalLoader;
+use sedona_schema::raster::BandDataType;
+use tempfile::TempDir;
+
+const FILES: usize = 64;
+const SIDE: usize = 1024;
+const CALLERS: [usize; 2] = [1, 8];
+const BUDGETS: [usize; 8] = [1, 2, 4, 8, 16, 32, 64, 128];
+
+/// One single-band uint8 DEFLATE COG of `SIDE × SIDE` (512-pixel tiles),
+/// written to `dir/cog.tif`.
+fn build_file(dir: &TempDir) {
+    let first = dir.path().join("cog.tif");
+    with_global_gdal(|gdal| {
+        let mem = gdal
+            .create_mem_dataset(SIDE, SIDE, 1, GdalDataType::UInt8)
+            .unwrap();
+        mem.set_geo_transform(&[0.0, 1.0, 0.0, SIDE as f64, 0.0, -1.0])
+            .unwrap();
+        // A gradient with some noise, so tiles compress like real data
+        // rather than to nothing.
+        let pixels: Vec<u8> = (0..SIDE * SIDE)
+            .map(|i| ((i % SIDE + i / SIDE) as u8) ^ ((i * 2654435761) >> 28) as u8)
+            .collect();
+        let mut buffer = GdalBuffer::new((SIDE, SIDE), pixels);
+        mem.rasterband(1)
+            .unwrap()
+            .write((0, 0), (SIDE, SIDE), &mut buffer)
+            .unwrap();
+        let cog = gdal.get_driver_by_name("COG").unwrap();
+        mem.create_copy(&cog, &first.to_string_lossy(), &["COMPRESS=DEFLATE"])
+            .unwrap();
+    })
+    .unwrap();
+}
+
+/// A fresh directory `g{generation}` under `root` holding `FILES` hard links
+/// to `root/cog.tif`, the previous generation's directory removed.
+fn fresh_generation(root: &Path, generation: usize) -> String {
+    if generation > 0 {
+        let _ = std::fs::remove_dir_all(root.join(format!("g{}", generation - 1)));
+    }
+    let name = format!("g{generation}");
+    std::fs::create_dir(root.join(&name)).unwrap();
+    for i in 0..FILES {
+        std::fs::hard_link(
+            root.join("cog.tif"),
+            root.join(&name).join(format!("cog{i}.tif")),
+        )
+        .unwrap();
+    }
+    name
+}
+
+/// `callers` tasks, each loading its share of the files through clones of
+/// one loader, so they share its budget like partitions share the
+/// registered loader.
+async fn load_all(prefix: &str, generation: &str, loader: &GdalLoader, callers: usize) {
+    let per_caller = FILES / callers;
+    let tasks: Vec<_> = (0..callers)
+        .map(|c| {
+            let loader = loader.clone();
+            let uris: Vec<String> = (c * per_caller..(c + 1) * per_caller)
+                .map(|i| format!("{prefix}/{generation}/cog{i}.tif#band=1"))
+                .collect();
+            tokio::spawn(async move {
+                let shape = [SIDE as i64, SIDE as i64];
+                let view = ViewEntries::identity_for_shape(&shape);
+                let reqs: Vec<RasterLoadRequest> = uris
+                    .iter()
+                    .map(|uri| RasterLoadRequest {
+                        uri,
+                        dim_names: &["y", "x"],
+                        source_shape: &shape,
+                        view: &view,
+                        data_type: BandDataType::UInt8,
+                    })
+                    .collect();
+                let refs: Vec<&RasterLoadRequest> = reqs.iter().collect();
+                loader.load(&refs).await.unwrap();
+            })
+        })
+        .collect();
+    for task in tasks {
+        task.await.unwrap();
+    }
+}
+
+fn sweep(
+    c: &mut Criterion,
+    group_name: &str,
+    root: &Path,
+    prefix: &str,
+    rt: &tokio::runtime::Runtime,
+    generation: &AtomicUsize,
+) {
+    let mut group = c.benchmark_group(group_name);
+    group.sample_size(10);
+    for callers in CALLERS {
+        for budget in BUDGETS {
+            let loader = GdalLoader::new().with_concurrency(budget);
+            group.bench_with_input(
+                BenchmarkId::new(format!("callers{callers}"), budget),
+                &budget,
+                |b, _| {
+                    b.iter_batched(
+                        || fresh_generation(root, generation.fetch_add(1, Ordering::Relaxed)),
+                        |g| rt.block_on(load_all(prefix, &g, &loader, callers)),
+                        BatchSize::PerIteration,
+                    )
+                },
+            );
+        }
+    }
+    group.finish();
+}
+
+/// A Python HTTP server over `root` that sleeps `latency` before answering
+/// every request and serves byte ranges (which `/vsicurl/` relies on), with
+/// keep-alive and a deep listen backlog so that only the latency is
+/// measured; killed on drop.
+struct LatencyServer {
+    child: Child,
+    port: u16,
+}

Review Comment:
   I think we have one of these based on axum in the zarr crate that doesn't involve calling into Python



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(rust/sedona-raster-gdal): open and read OutDb files concurrently under one shared I/O budget [sedona-db]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1385:
URL: https://github.com/apache/sedona-db/pull/1385#issuecomment-6351220696

   I handled the code comments from dewey as so:
   
   1. this is true but I'm going to punt on it for now I think. #1483 
   2 and 3. [claude assisted writing] both came from building the budget on a tokio semaphore it did not need. It is now a counter behind a mutex and a condition variable (f1741203f), with no runtime underneath, so `acquire_blocking` works inside `futures::executor::block_on` and any `usize` is a valid limit. Waiters hold tickets and are served in arrival order, and `try_acquire` never takes a slot ahead of a queued waiter. Shrinking just lowers the limit and admits no one until the operations in flight drop below it, so the permit-debt bookkeeping went away too. `sedona-raster` no longer depends on tokio or futures outside its tests. Both cases are regression tests.
   
   On the reuse points: loader reads are `SpawnedTask`s now, and each file's buffers go straight into their requests' slots the way the Zarr loader does, so the final sort is gone (5de0f165e). 
   
   Agreed on not copying the async acquire; there is no semaphore left to copy from.
   
   On pyarrow and lance: Lance's reader is a read-ahead kind of deal which has a lot of implications on how its designed. I would generally describe it as more complex than our and those benefits mostly coming in the circumstance where you are performing that read ahead.
   
   pyarrow: on pixel reads our implementation looks a lot like theirs, except we retire threads after an idle timeout. In `RS_FromPath` we use batch-local threads in order to handle some lifetime issues. I'm investigating if we can align frompath to look more like the ensure loaded case, but might defer that work to a subsequent PR
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(rust/sedona-raster-gdal): open and read OutDb files concurrently under one shared I/O budget [sedona-db]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1385:
URL: https://github.com/apache/sedona-db/pull/1385#issuecomment-6332740295

   agree! lets not block this because of this. lets merge it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(rust/sedona-raster-gdal): open and read OutDb files concurrently under one shared I/O budget [sedona-db]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1385:
URL: https://github.com/apache/sedona-db/pull/1385#issuecomment-6332508588

   I’d strongly prefer a dedicated raster data source as the main bulk-loading path here. We went through a similar progression in wherobots-compute: we started with RS_FromPath, then moved bulk loading into a dedicated raster data source. That gave us much more room to optimize reading and execution at the scan level.
   
   Can we take the same approach in SedonaDB? The shared I/O budget and concurrent pixel loader in this PR seem useful underneath either interface. I’d keep RS_FromPath for expression-level use, but put bulk-read scheduling in the data source.
   
   I’d like us to settle that direction before merging more scheduling logic into RS_FromPath. Could we structure this work as the shared foundation for a dedicated raster source and make that source the next concrete step?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(rust/sedona-raster-gdal): open and read OutDb files concurrently under one shared I/O budget [sedona-db]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1385:
URL: https://github.com/apache/sedona-db/pull/1385#discussion_r4352401078


##########
rust/sedona-raster-gdal/benches/gdal_io_concurrency.rs:
##########
@@ -0,0 +1,296 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The I/O budget of `GdalLoader` (`with_concurrency`): how long a fixed
+//! set of COGs takes to load as the budget grows, with one caller and with
+//! several callers sharing the budget the way DataFusion partitions do. Two
+//! stores: the page-cached local filesystem, where reads are cheap and
+//! decoding dominates, and an HTTP server (read through `/vsicurl/`) that
+//! sleeps for a fixed time on every request, a stand-in for object storage
+//! where the round trips dominate. The knee of the HTTP curve is what the
+//! default budget is chosen from; the local curve shows what that budget
+//! costs when there is no latency to hide.
+//!
+//! Every iteration reads the files under fresh names (hard links in a new
+//! directory, made outside the timed section), so neither GDAL's caches nor
+//! the loader's per-thread dataset cache serves a later iteration from an
+//! earlier one: each read is a cold open, as it is for a query that reads
+//! each COG once.
+//!
+//! Run with `cargo bench -p sedona-raster-gdal --bench gdal_io_concurrency`.
+//! `GDAL_BENCH_LATENCY_MS` sets the HTTP store's per-request latency
+//! (default 10). The HTTP store needs `python3` on the path; without it that
+//! half is skipped.
+
+use std::net::{TcpListener, TcpStream};
+use std::path::Path;
+use std::process::{Child, Command, Stdio};
+use std::sync::atomic::{AtomicUsize, Ordering};
+use std::time::{Duration, Instant};
+
+use criterion::{BatchSize, BenchmarkId, Criterion, criterion_group, criterion_main};
+use sedona_gdal::global::with_global_gdal;
+use sedona_gdal::raster::types::{Buffer as GdalBuffer, GdalDataType};
+use sedona_raster::raster_loader::{AsyncRasterLoader, RasterLoadRequest};
+use sedona_raster::view_entries::ViewEntries;
+use sedona_raster_gdal::GdalLoader;
+use sedona_schema::raster::BandDataType;
+use tempfile::TempDir;
+
+const FILES: usize = 64;
+const SIDE: usize = 1024;
+const CALLERS: [usize; 2] = [1, 8];
+const BUDGETS: [usize; 8] = [1, 2, 4, 8, 16, 32, 64, 128];
+
+/// One single-band uint8 DEFLATE COG of `SIDE × SIDE` (512-pixel tiles),
+/// written to `dir/cog.tif`.
+fn build_file(dir: &TempDir) {
+    let first = dir.path().join("cog.tif");
+    with_global_gdal(|gdal| {
+        let mem = gdal
+            .create_mem_dataset(SIDE, SIDE, 1, GdalDataType::UInt8)
+            .unwrap();
+        mem.set_geo_transform(&[0.0, 1.0, 0.0, SIDE as f64, 0.0, -1.0])
+            .unwrap();
+        // A gradient with some noise, so tiles compress like real data
+        // rather than to nothing.
+        let pixels: Vec<u8> = (0..SIDE * SIDE)
+            .map(|i| ((i % SIDE + i / SIDE) as u8) ^ ((i * 2654435761) >> 28) as u8)
+            .collect();
+        let mut buffer = GdalBuffer::new((SIDE, SIDE), pixels);
+        mem.rasterband(1)
+            .unwrap()
+            .write((0, 0), (SIDE, SIDE), &mut buffer)
+            .unwrap();
+        let cog = gdal.get_driver_by_name("COG").unwrap();
+        mem.create_copy(&cog, &first.to_string_lossy(), &["COMPRESS=DEFLATE"])
+            .unwrap();
+    })
+    .unwrap();
+}
+
+/// A fresh directory `g{generation}` under `root` holding `FILES` hard links
+/// to `root/cog.tif`, the previous generation's directory removed.
+fn fresh_generation(root: &Path, generation: usize) -> String {
+    if generation > 0 {
+        let _ = std::fs::remove_dir_all(root.join(format!("g{}", generation - 1)));
+    }
+    let name = format!("g{generation}");
+    std::fs::create_dir(root.join(&name)).unwrap();
+    for i in 0..FILES {
+        std::fs::hard_link(
+            root.join("cog.tif"),
+            root.join(&name).join(format!("cog{i}.tif")),
+        )
+        .unwrap();
+    }
+    name
+}
+
+/// `callers` tasks, each loading its share of the files through clones of
+/// one loader, so they share its budget like partitions share the
+/// registered loader.
+async fn load_all(prefix: &str, generation: &str, loader: &GdalLoader, callers: usize) {
+    let per_caller = FILES / callers;
+    let tasks: Vec<_> = (0..callers)
+        .map(|c| {
+            let loader = loader.clone();
+            let uris: Vec<String> = (c * per_caller..(c + 1) * per_caller)
+                .map(|i| format!("{prefix}/{generation}/cog{i}.tif#band=1"))
+                .collect();
+            tokio::spawn(async move {
+                let shape = [SIDE as i64, SIDE as i64];
+                let view = ViewEntries::identity_for_shape(&shape);
+                let reqs: Vec<RasterLoadRequest> = uris
+                    .iter()
+                    .map(|uri| RasterLoadRequest {
+                        uri,
+                        dim_names: &["y", "x"],
+                        source_shape: &shape,
+                        view: &view,
+                        data_type: BandDataType::UInt8,
+                    })
+                    .collect();
+                let refs: Vec<&RasterLoadRequest> = reqs.iter().collect();
+                loader.load(&refs).await.unwrap();
+            })
+        })
+        .collect();
+    for task in tasks {
+        task.await.unwrap();
+    }
+}
+
+fn sweep(
+    c: &mut Criterion,
+    group_name: &str,
+    root: &Path,
+    prefix: &str,
+    rt: &tokio::runtime::Runtime,
+    generation: &AtomicUsize,
+) {
+    let mut group = c.benchmark_group(group_name);
+    group.sample_size(10);
+    for callers in CALLERS {
+        for budget in BUDGETS {
+            let loader = GdalLoader::new().with_concurrency(budget);
+            group.bench_with_input(
+                BenchmarkId::new(format!("callers{callers}"), budget),
+                &budget,
+                |b, _| {
+                    b.iter_batched(
+                        || fresh_generation(root, generation.fetch_add(1, Ordering::Relaxed)),
+                        |g| rt.block_on(load_all(prefix, &g, &loader, callers)),
+                        BatchSize::PerIteration,
+                    )
+                },
+            );
+        }
+    }
+    group.finish();
+}
+
+/// A Python HTTP server over `root` that sleeps `latency` before answering
+/// every request and serves byte ranges (which `/vsicurl/` relies on), with
+/// keep-alive and a deep listen backlog so that only the latency is
+/// measured; killed on drop.
+struct LatencyServer {
+    child: Child,
+    port: u16,
+}

Review Comment:
   We did not have the latency but I added it there and use that now. good call!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(rust/sedona-raster-gdal): open and read OutDb files concurrently under one shared I/O budget [sedona-db]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis merged PR #1385:
URL: https://github.com/apache/sedona-db/pull/1385


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(rust/sedona-raster-gdal): open and read OutDb files concurrently under one shared I/O budget [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #1385:
URL: https://github.com/apache/sedona-db/pull/1385#issuecomment-6346625031

   > I’d strongly prefer a dedicated raster data source as the main bulk-loading path here
   
   I also think this is a good idea (and agree it's not blocking here), although it's also very useful to have the scalar function for syntax. We can optimize calls to `RS_FromPath()` into a scan when appropriate but in the meantime we do want queries like `SELECT RS_Value(geom, RS_FromPath('foofy.tif')) FROM big_geometry_table)` to work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(rust/sedona-raster-gdal): open and read OutDb files concurrently under one shared I/O budget [sedona-db]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1385:
URL: https://github.com/apache/sedona-db/pull/1385#issuecomment-6332727369

   @jiayuasu this logic backs anything that loads from raster files using GDAL. So since RS_FromPath exists we want it to use this.
   
   The logic in this PR also back RS_EnsureLoad. So I don't see any reason to hold off on merging this. Any data source we build will also benefit from these changes.
   
   If building that loader starts to duplicate code from RS_FromPath we can refactor logic out to be shared between the two codepaths.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(rust/sedona-raster-gdal): open and read OutDb files concurrently under one shared I/O budget [sedona-db]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1385:
URL: https://github.com/apache/sedona-db/pull/1385#issuecomment-6332754382

   yes just need to finalize the revisions I made from dewey's review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org