[I] new feature: caller-owned buffer reads (fill a destination buffer directly) [opendal]

["chitralverma (via GitHub)" <gi...@apache.org>]

chitralverma opened a new issue, #7909:
URL: https://github.com/apache/opendal/issues/7909

   # Feature Description
   
   Add an optional read path that fills a **caller-owned destination buffer** directly, so that capable services and transports can write bytes into memory the caller already owns without OpenDAL first materializing an intermediate
   owned `Buffer`/`Bytes`.
   
   Today OpenDAL's reader pipeline always yields owned `Buffer`/`Bytes` chunks. The closest public API, `Reader::read_into(&mut impl BufMut, range)`, still consumes those owned chunks from the reader stream and copies them into the
   caller's `BufMut`:
   
   ```rust
   pub async fn read_into(
       &self,
       buf: &mut impl BufMut,
       range: impl Into<BytesRange>,
   ) -> Result<usize> {
       let mut stream = self.clone().into_stream(range).await?;
       let mut read = 0;
       loop {
           let Some(bs) = stream.try_next().await? else {
               return Ok(read);
           };
           read += bs.len();
           buf.put(bs); // copy into caller memory
       }
   }
   ```
   
   With `&mut [u8]` as the `BufMut`, `bytes::BufMut::put_slice` performs a `copy_from_slice` into the caller's allocation. So the effective path is:
   
   ```text
   backend/transport -> OpenDAL Buffer/Bytes -> caller-owned buffer
                                            one memcpy
   ```
   
   The request is for an API where the caller supplies the destination and, on
   capable backends, the backend writes into it directly:
   
   ```text
   backend/transport -> caller-owned destination      (no intermediate data buffer)
   ```
   
   # Problem and Solution
   
   ## Problem
   
   Many callers do not own the shape of their destination memory — it is dictated by a foreign API, a hardware constraint, or a preallocated pool — and they require OpenDAL to *fill that specific memory*. In these cases OpenDAL's `Buffer`-first model forces an extra copy that has no semantic purpose.
   
   Concrete cases where the caller owns the destination and cannot adopt an OpenDAL `Buffer`:
   
   - **FFI / foreign filesystem bridges.** C/C++ `pread`-style contracts (`read(void *buffer, size_t nr_bytes, off_t offset)`) hand OpenDAL a raw destination pointer by value and require it to be filled. The buffer is owned and freed by the foreign runtime; OpenDAL cannot return or substitute its own `Buffer`, so today every such bridge pays a memcpy per read.
   - **Columnar / Parquet readers.** Arrow-style readers assemble column chunks into preallocated, correctly-aligned buffers (often part of a larger arena or a `MutableBuffer`). They want positioned range reads landed straight into those buffers rather than into a transient `Buffer` that is immediately copied and dropped.
   - **Buffer pools and arenas.** Query engines and network servers read into slabs from a fixed-capacity pool to bound allocation and improve cache locality. A `Buffer`-returning read defeats the pool because the data lands in OpenDAL-owned memory first.
   - **Special-purpose memory.** Destinations such as `mmap` regions, GPU pinned host memory, `O_DIRECT`-aligned buffers, or `io_uring`-registered buffers must be written in place. They cannot be replaced by an OpenDAL allocation.
   
   The extra copy is most visible exactly where network latency does *not* hide memory bandwidth: local `fs`, fast attached storage, the in-memory `memory` service, and cache hits (e.g. a warm cache layer). For large scans, many concurrent positioned reads, or high-throughput links, the redundant copy shows up as CPU cycles, memory bandwidth, and allocator pressure.
   
   ## Proposed solution
   
   Introduce a caller-owned buffer read at two levels, gated by capability with a safe fallback:
   
   1. **Public high-level method** on `Reader`, e.g.
   
      ```rust
      /// Read exactly `dst.len()` bytes for `range` into caller-owned memory.
      pub async fn read_exact_into(
          &self,
          dst: &mut [u8],
          range: Range<u64>,
      ) -> Result<()>;
      ```
   
      with a fixed-capacity, `pread`-like contract that matches positioned reads.
   
   2. **Optional raw capability** on the positioned-read path so capable services fill the destination directly, for example an additive method alongside the existing `PositionRead::read_at`:
   
      ```rust
      async fn read_at_into(
          handle: &Self::Handle,
          offset: u64,
          dst: &mut [u8],
      ) -> Result<usize>;
      ```
   
      Services that can write into a foreign destination (starting with `fs` via platform positioned reads, and `memory` via `copy_from_slice`) implement it. Everything else falls back to the existing `read_at` + one copy, so no service
      is required to change and behavior is preserved.
   
   The existing `read`/`read_into`/`fetch`/stream APIs and their zero-copy-within-OpenDAL semantics stay exactly as they are; this is purely additive.
   
   A pragmatic first increment: land the raw capability + fs implementation + generic fallback + the `Reader` method, benchmark it against `read_into`, then consider extending to transports and other services. A full design (capability
   advertisement, layer composition — retry/timeout/cancellation with borrowed memory, short-read/EOF semantics, `MaybeUninit` vs initialized slices) is worth an RFC; i'll be happy to draft that if this issue makes sense.
   
   # Additional Context
   
   - [ ] Yes, I am willing to contribute to the development of this feature.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] new feature: caller-owned buffer reads (fill a destination buffer directly) [opendal]

["dentiny (via GitHub)" <gi...@apache.org>]

dentiny commented on issue #7909:
URL: https://github.com/apache/opendal/issues/7909#issuecomment-4960595250

   Hi @chitralverma thanks for raising this!
   
   I actually like buffer-based solution a lot, one big reason is for caller-owned memory, it's hard to implement [hedged request](https://github.com/apache/opendal/issues/7189).
   
   In terms of DuckDB integration, which both of us care about, I raise a separate issue [here](https://github.com/duckdb/duckdb/discussions/21546).
   In the latest v2.0 coming late this year, I introduced a buffer-wise data structure [here](https://github.com/duckdb/duckdb/blob/main/src/include/duckdb/storage/external_file_cache/file_buffer_handle_group.hpp), which should play with opendal better, but it's only used for DuckDB external file cache for now, and it needs some work to change the interface and prove it's more performant.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] new feature: caller-owned buffer reads (fill a destination buffer directly) [opendal]

["chitralverma (via GitHub)" <gi...@apache.org>]

chitralverma commented on issue #7909:
URL: https://github.com/apache/opendal/issues/7909#issuecomment-4960467473

   @Xuanwo @dentiny  what're your thoughts on this ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org