You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "chitralverma (via GitHub)" <gi...@apache.org> on 2025/03/26 08:12:41 UTC
[I] new feature: Support Vectored I/O via `read*` and `write*` [opendal]
chitralverma opened a new issue, #5881:
URL: https://github.com/apache/opendal/issues/5881
### Feature Description
Currently, OpenDAL does not provide built-in support for vectored I/O (scatter/ gather I/O) in its `read*` and `write*` methods. This feature would allow optimized batch reads and writes by enabling multiple non-contiguous byte ranges to be processed in a single operation when supported by the underlying storage backend.
Initially this can be exposed via `read_vectored` and `write_vectored` and later be the default way by merging into the existing `read*` and `write*` methods.
**Use Cases**
- **Remote Object Stores (S3, GCS, Azure):** Parallel range requests to reduce network round-trips.
- **Distributed File Systems:** Efficient scatter/gather operations for improved performance.
- **Big Data Workloads:** Optimized partial reads for high-throughput analytics.
Would love to hear thoughts from the maintainers on feasibility and implementation approach!
### Problem and Solution
**Feature Request**
- expose vectored_read and vectored_write as capabilities on Operator
- Utilize backend capabilities where available (e.g., HTTP Range requests for S3/GCS/Azure, multipart uploads for writing).
- Provide a fallback mechanism for backends that do not natively support vectored I/O.
- Ensure compatibility with existing OpenDAL abstractions.
### Additional Context
_No response_
### Are you willing to contribute to the development of this feature?
- [ ] Yes, I am willing to contribute to the development of this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support Vectored I/O for capable services [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #5881: new feature: Support Vectored I/O for capable services
URL: https://github.com/apache/opendal/issues/5881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support Vectored I/O for capable services [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5881:
URL: https://github.com/apache/opendal/issues/5881#issuecomment-2765399491
Hi, the Vectored I/O you described in this issue refers to two different things from the storage perspective.
- For local fs, vectored I/O refers to a set of syscall, like [read_vectored][https://doc.rust-lang.org/stable/std/io/trait.Read.html#method.read_vectored].
- But for cloud storage services s3, gcs, the `vectored I/O` here more like concurrent reading/writing.
APIs like `read_vectored` are useful but have some limitations: the slices are vectored but contiguous, which means users can't read ranges like `1MiB-2MiB` and `3MiB-4MiB` by this API. In our current context, it's more of an optimization we can apply to the `fs` service to improve contiguously read and write performance.
From what I understand so far, the most useful feature is allowing users to perform concurrent reading and writing. OpenDAL is implemented in this way:
```rust
// Read a large file with 4MiB as chunk at 8 concurrent reading requests.
op.read_with(path).chunk(4*1024*1024).concurrent(8).await?;
// Write a large file with 8MiB as chunk at 8 concurrent writing requests.
op.write_with(path, a_super_large_bs).chunk(8*1024*1024).concurrent(8).await?;
```
There are also `Reader` and `Writer` versions of this API that enable users to implement more dynamic logic.
---
Please let me know if I understand correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support Vectored I/O for capable services [opendal]
Posted by "chitralverma (via GitHub)" <gi...@apache.org>.
chitralverma commented on issue #5881:
URL: https://github.com/apache/opendal/issues/5881#issuecomment-2786446821
@Xuanwo this ticket most came from [this](https://github.com/apache/opendal/blob/7ee3babcc70d30721e0acca2cb8008fc38a28795/core/src/types/read/futures_async_reader.rs#L107) comment and the comparison with [object_store crate (point 3)](https://docs.rs/object_store/latest/object_store/#highlights).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support Vectored I/O for capable services [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5881:
URL: https://github.com/apache/opendal/issues/5881#issuecomment-2804807432
Closing as no actions to take. I will improve https://github.com/apache/opendal/issues/4121 instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support Vectored I/O for capable services [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5881:
URL: https://github.com/apache/opendal/issues/5881#issuecomment-2788274328
> [@Xuanwo](https://github.com/Xuanwo) this ticket most came from [this](https://github.com/apache/opendal/blob/7ee3babcc70d30721e0acca2cb8008fc38a28795/core/src/types/read/futures_async_reader.rs#L107) comment and the comparison with [object_store crate (point 3)](https://docs.rs/object_store/latest/object_store/#highlights).
Understood. OpenDAL provides similiar vectored IO like `object_store` do, which is the `concurrent` feature in opendal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] new feature: Support Vectored I/O for capable services [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5881:
URL: https://github.com/apache/opendal/issues/5881#issuecomment-2765431449
I created the following issues to allow our users can benefits from our Buffer in-memory layout.
- https://github.com/apache/opendal/issues/5913
- https://github.com/apache/opendal/issues/5914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org