You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2026/07/24 12:35:10 UTC
[PR] RFC: Add limited read [opendal]
Xuanwo opened a new pull request, #7945:
URL: https://github.com/apache/opendal/pull/7945
# Which issue does this PR close?
Refs #7938.
# Rationale for this change
A bounded read range is exact, so callers that only need up to a small number of bytes must stat the object first or handle a short object as an error. OpenDAL needs an at-most read for probes such as file header and format detection without an extra metadata request.
# What changes are included in this PR?
This PR proposes `ReadOptions::limit` and `Operator::read_with(...).limit(n)`. The proposal keeps exact bounded range behavior, uses the existing raw `open` and `read` split, and carries only a private `exact` boolean in core instead of adding a new raw operation or planning abstraction.
# Are there any user-facing changes?
Yes. The proposed API returns between zero and the configured limit on clean EOF. Existing reads that do not set `limit` keep their current behavior. This PR contains the RFC only; it does not implement the API.
# AI Usage Statement
This proposal was drafted with assistance from OpenAI Codex powered by GPT-5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "akx (via GitHub)" <gi...@apache.org>.
akx commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3657335118
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
+beyond the end of the object, the result is empty.
+
+`range` without `limit` keeps its current behavior:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(0..16 * 1024)
+ .await?;
+```
+
+This succeeds only when the complete 16 KiB range is available. A clean end of
+file before the requested end remains an error.
+
+For a non-empty limit, `limit` changes only clean end-of-file handling. A
+missing object, a failed condition, a permission failure, or a transport error
+still fails the read.
+
+# Reference-level explanation
+
+## Public API
+
+`ReadOptions` gains one field:
+
+```rust,ignore
+pub struct ReadOptions {
+ pub range: BytesRange,
+ pub limit: Option<u64>,
+ // Existing fields.
+}
+```
+
+`FutureRead` gains the matching builder:
+
+```rust,ignore
+pub fn limit(mut self, limit: u64) -> Self {
+ self.args.limit = Some(limit);
+ self
+}
+```
+
+`limit` applies after `range`. Core converts the two options into one physical
+`BytesRange`:
+
+| Options | Physical range | Completion |
+| --- | --- | --- |
+| no `range`, `limit(n)` | offset 0, size `n` | at most `n` |
+| `range(offset..)`, `limit(n)` | offset `offset`, size `n` | at most `n` |
+| `range(start..end)`, `limit(n)` | offset `start`, size `min(end - start, n)` | at most that size |
+| bounded `range` without `limit` | unchanged | exact |
+
+The initial API rejects combining a suffix range with `limit`. A suffix is
+relative to the object end, so resolving it without metadata would defeat the
+no-`stat` property.
+
+`limit(0)` returns an empty buffer without checking the object, consistent with
+an empty range.
+
+Combining `limit` with a suffix range, an explicit `chunk` size, or a
+`concurrent` value greater than one returns `ErrorKind::ConfigInvalid` before
+storage I/O.
+
+## Exact completion
+
+Core carries a private boolean named `exact` with the normalized range. It does
+not introduce a public type or a new raw abstraction.
+
+- A regular bounded, non-suffix range sets `exact` to `true`.
+- A read with `limit` sets `exact` to `false`.
+- An open-ended or suffix range sets `exact` to `false`.
+
+The buffer stream tracks the number of bytes it yields. At clean end of file, it
+compares that count with the bounded range size only when `exact` is `true`.
+This preserves current exact range behavior while allowing limited reads to
+finish early.
+
+The `exact` flag belongs to the public read execution path. It is not passed to
+services because services should not decide whether a caller accepts a short
+result.
Review Comment:
I'm not sure about this. Could a service do a better job if it knew about the request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #7945:
URL: https://github.com/apache/opendal/pull/7945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3656018145
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
Review Comment:
Sorry I didn't get this. We are going to add `limit`, and this examples shows how `limit` works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3851503854
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
+beyond the end of the object, the result is empty.
+
+`range` without `limit` keeps its current behavior:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(0..16 * 1024)
+ .await?;
+```
+
+This succeeds only when the complete 16 KiB range is available. A clean end of
+file before the requested end remains an error.
+
+For a non-empty limit, `limit` changes only clean end-of-file handling. A
+missing object, a failed condition, a permission failure, or a transport error
+still fails the read.
+
+# Reference-level explanation
+
+## Public API
+
+`ReadOptions` gains one field:
+
+```rust,ignore
+pub struct ReadOptions {
+ pub range: BytesRange,
+ pub limit: Option<u64>,
+ // Existing fields.
+}
+```
+
+`FutureRead` gains the matching builder:
+
+```rust,ignore
+pub fn limit(mut self, limit: u64) -> Self {
+ self.args.limit = Some(limit);
+ self
+}
+```
+
+`limit` applies after `range`. Core converts the two options into one physical
+`BytesRange`:
+
+| Options | Physical range | Completion |
+| --- | --- | --- |
+| no `range`, `limit(n)` | offset 0, size `n` | at most `n` |
+| `range(offset..)`, `limit(n)` | offset `offset`, size `n` | at most `n` |
+| `range(start..end)`, `limit(n)` | offset `start`, size `min(end - start, n)` | at most that size |
+| bounded `range` without `limit` | unchanged | exact |
+
+The initial API rejects combining a suffix range with `limit`. A suffix is
+relative to the object end, so resolving it without metadata would defeat the
+no-`stat` property.
Review Comment:
updated。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3656018145
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
Review Comment:
Sorry I didn't get this. We are going to add a new option called `limit`, and this examples shows how `limit` works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3655869640
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
Review Comment:
I'm not sure this aligns with the current behavior? I checked with quick unit test seems we return `RangeNotSatisfied` for out-of-bound access.
Example-1 for memory backend
```rs
// under core/core/src/services/memory/backend.rs
#[cfg(test)]
mod tests {
use super::MEMORY_SCHEME;
use crate::Buffer;
use crate::BytesRange;
use crate::ErrorKind;
use crate::Operator;
use crate::Result;
#[tokio::test]
async fn test_bounded_read_beyond_eof_returns_range_not_satisfied() -> Result<()> {
let op = Operator::via_iter(MEMORY_SCHEME, [])?;
op.write("test", Buffer::from("abc")).await?;
// RFC-7945 proposes normalizing range(4096..).limit(1024) into this
// bounded physical range. Memory rejects it before core can apply
// at-most completion semantics.
let physical_range = BytesRange::new(4096, Some(1024));
let err = op
.reader("test")
.await?
.read(physical_range)
.await
.unwrap_err();
assert_eq!(err.kind(), ErrorKind::RangeNotSatisfied);
Ok(())
}
}
```
Example-2 for local filesystem backend
```rs
// under core/services/fs/src/backend.rs
#[cfg(test)]
mod tests {
use opendal_core::Buffer;
use opendal_core::BytesRange;
use opendal_core::ErrorKind;
use opendal_core::Operator;
use super::*;
#[tokio::test]
async fn test_bounded_read_beyond_eof_returns_range_not_satisfied() -> Result<()> {
let root = tempfile::TempDir::new().expect("tempdir must be created");
let op = Operator::new(FsBuilder::default().root(&root.path().to_string_lossy()))?;
op.write("test", Buffer::from("abc")).await?;
let physical_range = BytesRange::new(4096, Some(1024));
let err = op
.reader("test")
.await?
.read(physical_range)
.await
.unwrap_err();
assert_eq!(err.kind(), ErrorKind::RangeNotSatisfied);
Ok(())
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3656201087
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
Review Comment:
Oh, yes. Makes sense 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3656126055
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
Review Comment:
The RFC here mentioned
```
If the offset is at or beyond the end of the object, the result is empty.
```
It reads to me that, with `limit` option specified, users will get an empty buffer instead of an error.
- Our current behavior seems to reject and explicitly return `RangeNotSatisfied` (see my rust code above) -- it's a behavior change
- HTTP spec indicates 416 error code is intended for out-of-range request: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/416#malformed_range_request
- S3 documents that Range follows the HTTP Range header semantics: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html?utm_source=chatgpt.com#API_GetObject_RequestSyntax
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "akx (via GitHub)" <gi...@apache.org>.
akx commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3657328097
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
+beyond the end of the object, the result is empty.
+
+`range` without `limit` keeps its current behavior:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(0..16 * 1024)
+ .await?;
+```
+
+This succeeds only when the complete 16 KiB range is available. A clean end of
+file before the requested end remains an error.
+
+For a non-empty limit, `limit` changes only clean end-of-file handling. A
+missing object, a failed condition, a permission failure, or a transport error
+still fails the read.
+
+# Reference-level explanation
+
+## Public API
+
+`ReadOptions` gains one field:
+
+```rust,ignore
+pub struct ReadOptions {
+ pub range: BytesRange,
+ pub limit: Option<u64>,
+ // Existing fields.
+}
+```
+
+`FutureRead` gains the matching builder:
+
+```rust,ignore
+pub fn limit(mut self, limit: u64) -> Self {
+ self.args.limit = Some(limit);
+ self
+}
+```
+
+`limit` applies after `range`. Core converts the two options into one physical
+`BytesRange`:
+
+| Options | Physical range | Completion |
+| --- | --- | --- |
+| no `range`, `limit(n)` | offset 0, size `n` | at most `n` |
+| `range(offset..)`, `limit(n)` | offset `offset`, size `n` | at most `n` |
+| `range(start..end)`, `limit(n)` | offset `start`, size `min(end - start, n)` | at most that size |
+| bounded `range` without `limit` | unchanged | exact |
+
+The initial API rejects combining a suffix range with `limit`. A suffix is
+relative to the object end, so resolving it without metadata would defeat the
+no-`stat` property.
Review Comment:
I'm not sure this rejection is needed. If suffix ranges are currently supported anyway, a suffix range (END-1024..END) + limit 512, say, is equivalent to (END-1024..END-512) and resolvable just the same?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "akx (via GitHub)" <gi...@apache.org>.
akx commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3657302965
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
Review Comment:
Yeah, starting a read past end of file sounds like it should be RangeNotSatisfied, even if short reads would be allowed by the caller. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3656018145
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
Review Comment:
Sorry I didn't get this. We are going to add a new option called `limit`, and this examples shows how `limit` works. It's true and of course that current tests will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RFC: Add limited read [opendal]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on code in PR #7945:
URL: https://github.com/apache/opendal/pull/7945#discussion_r3656126055
##########
core/core/src/docs/rfcs/7945_limited_read.md:
##########
@@ -0,0 +1,270 @@
+- Proposal Name: `limited_read`
+- Start Date: 2026-07-24
+- RFC PR: [apache/opendal#7945](https://github.com/apache/opendal/pull/7945)
+- Tracking Issue: [apache/opendal#7938](https://github.com/apache/opendal/issues/7938)
+
+# Summary
+
+Add `limit` to `Operator::read_with` and `ReadOptions`.
+
+`range` continues to request an exact bounded range. `limit` caps the amount of
+data returned but accepts a clean end of file before the cap. This supports
+small probes, such as reading up to 16 KiB from an object whose size is unknown,
+without a preceding `stat`.
+
+The implementation reuses the existing read path. It adds no raw operation or
+new planning type. Core only carries an `exact` boolean so the component that
+collects a stream knows whether a clean short read is an error.
+
+# Motivation
+
+Callers often need only the beginning of an object:
+
+- inspect a file signature or header;
+- detect a format;
+- parse metadata stored near the beginning;
+- sample a small object without first knowing its size.
+
+Today a caller can use `range(0..N)`, but a bounded range is exact. Reading an
+object shorter than `N` fails even though the bytes that do exist are sufficient
+for these use cases. The caller can avoid the error by calling `stat` first and
+then choosing a smaller range, but that adds a request and introduces a race
+between metadata lookup and data access.
+
+OpenDAL needs an at-most read alongside its existing exact range read.
+
+# Guide-level explanation
+
+Use `limit` when any number of bytes from zero through the limit is a successful
+result:
+
+```rust
+let header = op
+ .read_with("path/to/file")
+ .limit(16 * 1024)
+ .await?;
+```
+
+If the object contains at least 16 KiB, this returns 16 KiB. If it contains less,
+this returns the whole object. OpenDAL does not perform a `stat` before reading.
+
+`limit` can start at an offset:
+
+```rust
+let data = op
+ .read_with("path/to/file")
+ .range(4096..)
+ .limit(1024)
+ .await?;
+```
+
+This returns at most 1024 bytes starting at offset 4096. If the offset is at or
Review Comment:
The RFC here mentioned
```
If the offset is at or beyond the end of the object, the result is empty.
```
It reads to me that, with `limit` option specified, users will get an empty buffer instead of an error.
- Our current behavior seems to reject and explicitly return `RangeNotSatisfied` (see my rust code above) -- it's a behavior change
- HTTP spec indicates 416 error code is intended for out-of-range request: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/416#malformed_range_request
- S3 documents that Range follows the HTTP Range header semantics: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html?utm_source=chatgpt.com#API_GetObject_RequestSyntax
So I'm wondering if it's better to keep the current behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org