You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by ho...@apache.org on 2024/04/28 03:23:12 UTC
(opendal) branch feat-concurrent-blocking-read created (now d5ad3cffe9)
This is an automated email from the ASF dual-hosted git repository.
hoslo pushed a change to branch feat-concurrent-blocking-read
in repository https://gitbox.apache.org/repos/asf/opendal.git
at d5ad3cffe9 feat(core/types): Implement concurrent read for blocking read
This branch includes the following new commits:
new d5ad3cffe9 feat(core/types): Implement concurrent read for blocking read
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(opendal) 01/01: feat(core/types): Implement concurrent read for blocking read
Posted by ho...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hoslo pushed a commit to branch feat-concurrent-blocking-read
in repository https://gitbox.apache.org/repos/asf/opendal.git
commit d5ad3cffe9cb19dacb0f5f0f0932bb64084f7e5a
Author: Xuanwo <gi...@xuanwo.io>
AuthorDate: Sat Apr 27 16:18:10 2024 +0800
feat(core/types): Implement concurrent read for blocking read
---
core/Cargo.lock | 1 +
core/Cargo.toml | 1 +
core/src/docs/upgrade.md | 86 +++++++++++++++++++++++++
core/src/types/blocking_read/blocking_reader.rs | 82 +++++++++++++++++++----
core/src/types/operator/blocking_operator.rs | 12 ++--
core/src/types/operator/operator_functions.rs | 41 ++++++++----
6 files changed, 193 insertions(+), 30 deletions(-)
diff --git a/core/Cargo.lock b/core/Cargo.lock
index 7faf93611c..f3113dc15e 100644
--- a/core/Cargo.lock
+++ b/core/Cargo.lock
@@ -4802,6 +4802,7 @@ dependencies = [
"quick-xml",
"r2d2",
"rand 0.8.5",
+ "rayon",
"redb",
"redis",
"reqsign",
diff --git a/core/Cargo.toml b/core/Cargo.toml
index 75f26150e5..0e27e249b2 100644
--- a/core/Cargo.toml
+++ b/core/Cargo.toml
@@ -351,6 +351,7 @@ tracing = { version = "0.1", optional = true }
# for layers-dtrace
probe = { version = "0.5.1", optional = true }
crc32c = "0.6.5"
+rayon = "1.10.0"
[target.'cfg(target_arch = "wasm32")'.dependencies]
getrandom = { version = "0.2", features = ["js"] }
diff --git a/core/src/docs/upgrade.md b/core/src/docs/upgrade.md
index a9bed6cf0b..7e9d2e5d57 100644
--- a/core/src/docs/upgrade.md
+++ b/core/src/docs/upgrade.md
@@ -2,6 +2,10 @@
## Public API
+### MSRV Changed to 1.75
+
+Since 0.46, OpenDAL requires Rust 1.75.0 or later to use features like [`RPITIT`](https://rust-lang.github.io/rfcs/3425-return-position-impl-trait-in-traits.html) and [`AFIT`](https://rust-lang.github.io/rfcs/3185-static-async-fn-in-trait.html).
+
### Services Feature Flag
Starting with version 0.46, OpenDAL only includes the memory service by default to prevent compiling unnecessary service code. To use other services, please activate their respective feature flags.
@@ -12,6 +16,88 @@ Additionally, we have removed all `reqwest`-related feature flags:
- The `rustls` feature is no longer enabled by default; it must be activated manually.
- OpenDAL no longer offers the `trust-dns` option; users should configure the client builder directly.
+### Range Based Read
+
+Since v0.46, OpenDAL transformed it's Read IO trait to range based instead of stateful poll based IO. This change will make the IO more efficient, easier for concurrency and ready for completion based IO.
+
+`opendal::Reader` now have APIs like:
+
+```rust
+let r = op.reader("test.txt").await?;
+let buf = r.read(1024..2048).await?;
+```
+
+### Buffer Based IO
+
+Since version 0.46, OpenDAL features a native `Buffer` struct that supports both contiguous and non-contiguous buffers. This update enhances IO efficiency by minimizing unnecessary byte copying and enabling vectored IO.
+
+OpenDAL's `Reader` will return `Buffer` and `Writer` will accept `Buffer` as input. Users who have implemented their own IO traits should update their code to use the new `Buffer` struct.
+
+```rust
+let r = op.reader("test.txt").await?;
+// read returns `Buffer`
+let buf: Buffer = r.read(1024..2048).await?;
+
+let w = op.writer("test.txt").await?;
+
+// Buffer can be created from continues bytes.
+w.write("hello, world").await?;
+// Buffer can also be created from non-continues bytes.
+w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]).await?;
+
+// Make sure file has been written completely.
+w.close().await?;
+```
+
+To enhance usability, we've integrated bridges into `bytes::Buf` and `bytes::BufMut`, allowing users to directly interact with the bytes API.
+
+```rust
+let r = op.reader("test.txt").await?;
+let mut bs = vec![];
+// read_into accepts bytes::BufMut
+let buf: Buffer = r.read_into(&mut bs, 1024..2048).await?;
+
+let w = op.writer("test.txt").await?;
+
+// write_from accepts bytes::Buf
+w.write_from("hello, world".as_bytes()).await?;
+
+// Make sure file has been written completely.
+w.close().await?;
+```
+
+### Bridge API
+
+OpenDAL's `Reader` and `Writer` previously implemented APIs such as `AsyncRead` and `AsyncWrite` directly. This design was not user-friendly, as it could lead to unexpected costs that users were unaware of in advance.
+
+Since v0.46, OpenDAL provides bridge APIs for `Reader` and `Writer` instead.
+
+```rust
+let r = op.reader("test.txt").await?;
+
+// Convert into futures AsyncRead + AsyncSeek.
+let reader = r.into_futures_async_read(1024..2048);
+// Convert into futures bytes stream.
+let stream = r.into_bytes_stream(1024..2048);
+
+let w = op.writer("test.txt").await?;
+
+// Convert into futures AsyncWrite
+let writer = w.into_futures_async_write();
+// Convert into futures bytes sink;
+let sink = w.into_bytes_sink();
+```
+
+## Raw API
+
+### Async in IO trait
+
+Since version 0.46, OpenDAL has adopted Rust's native [`async_in_trait`](https://blog.rust-lang.org/2023/12/21/async-fn-rpit-in-traits.html) for our core IO traits, including `oio::Read`, `oio::Write`, and `oio::List`.
+
+This update eliminates the need for manually written, poll-based state machines and simplifies the codebase. Consequently, OpenDAL now requires Rust version 1.75.0 or later.
+
+Users who have implemented their own IO traits should update their code to use the new async trait syntax.
+
# Upgrade to v0.45
## Public API
diff --git a/core/src/types/blocking_read/blocking_reader.rs b/core/src/types/blocking_read/blocking_reader.rs
index 59b3e3bc1f..a6403f87ab 100644
--- a/core/src/types/blocking_read/blocking_reader.rs
+++ b/core/src/types/blocking_read/blocking_reader.rs
@@ -21,6 +21,8 @@ use std::ops::RangeBounds;
use bytes::Buf;
use bytes::BufMut;
+use log::debug;
+use rayon::prelude::*;
use crate::raw::oio::BlockingRead;
use crate::raw::*;
@@ -30,6 +32,7 @@ use crate::*;
/// manner.
pub struct BlockingReader {
pub(crate) inner: oio::BlockingReader,
+ options: OpReader,
}
impl BlockingReader {
@@ -40,10 +43,15 @@ impl BlockingReader {
///
/// We don't want to expose those details to users so keep this function
/// in crate only.
- pub(crate) fn create(acc: FusedAccessor, path: &str, op: OpRead) -> crate::Result<Self> {
+ pub(crate) fn create(
+ acc: FusedAccessor,
+ path: &str,
+ op: OpRead,
+ options: OpReader,
+ ) -> crate::Result<Self> {
let (_, r) = acc.blocking_read(path, op)?;
- Ok(BlockingReader { inner: r })
+ Ok(BlockingReader { inner: r, options })
}
/// Read give range from reader into [`Buffer`].
@@ -76,21 +84,69 @@ impl BlockingReader {
let mut bufs = Vec::new();
let mut offset = start;
+ let concurrent = self.options.concurrent() as u64;
+
+
+ let (interval_size, mut intervals) = end
+ .map(|end| {
+ // let interval_size = (end - start + concurrent - 1) / concurrent;
+ let interval_size = (end - start) / concurrent;
+ let remainder = (end - start) % concurrent;
+ let intervals: Vec<(u64, u64)> = (0..concurrent)
+ .map(|i| {
+ let interval_start = start + i * interval_size + remainder.min(i);
+ let interval_end =
+ interval_start + interval_size + if i < remainder { 1 } else { 0 };
+ (interval_start, interval_end)
+ })
+ .filter(|(interval_start, interval_end)| interval_start != interval_end)
+ .collect();
+ (interval_size, intervals)
+ })
+ .unwrap_or({
+ // TODO: use service preferred io size instead.
+ let interval_size = 4 * 1024 * 1024;
+ let intervals: Vec<(u64, u64)> = (0..concurrent)
+ .map(|i| {
+ let current = start + i * interval_size;
+ (current, current + interval_size)
+ })
+ .collect();
+ (interval_size, intervals)
+ });
loop {
- // TODO: use service preferred io size instead.
- let limit = end.map(|end| end - offset).unwrap_or(4 * 1024 * 1024) as usize;
- let bs = self.inner.read_at(offset, limit)?;
- let n = bs.remaining();
- bufs.push(bs);
- if n < limit {
- return Ok(bufs.into_iter().flatten().collect());
+
+ let results: Vec<Result<(usize, Buffer)>> = intervals
+ .into_par_iter()
+ .map(|(start, end)| -> Result<(usize, Buffer)> {
+ let limit = (end - start) as usize;
+
+ let bs = self.inner.read_at(start as u64, limit)?;
+ let n = bs.remaining();
+
+ Ok((n, bs))
+ })
+ .collect();
+ for result in results {
+ let result = result?;
+ bufs.push(result.1);
+ if result.0 < interval_size as usize {
+ return Ok(bufs.into_iter().flatten().collect());
+ }
+
+ offset += result.0 as u64;
+ if Some(offset) == end {
+ return Ok(bufs.into_iter().flatten().collect());
+ }
}
- offset += n as u64;
- if Some(offset) == end {
- return Ok(bufs.into_iter().flatten().collect());
- }
+ intervals = (0..concurrent)
+ .map(|i| {
+ let current = offset + i * interval_size;
+ (current, current + interval_size)
+ })
+ .collect();
}
}
diff --git a/core/src/types/operator/blocking_operator.rs b/core/src/types/operator/blocking_operator.rs
index 458aa2342b..68361fac54 100644
--- a/core/src/types/operator/blocking_operator.rs
+++ b/core/src/types/operator/blocking_operator.rs
@@ -385,8 +385,8 @@ impl BlockingOperator {
FunctionRead(OperatorFunction::new(
self.inner().clone(),
path,
- (OpRead::default(), BytesRange::default()),
- |inner, path, (args, range)| {
+ (OpRead::default(), BytesRange::default(), OpReader::default()),
+ |inner, path, (args, range, options)| {
if !validate_path(&path, EntryMode::FILE) {
return Err(
Error::new(ErrorKind::IsADirectory, "read path is a directory")
@@ -396,7 +396,7 @@ impl BlockingOperator {
);
}
- let r = BlockingReader::create(inner, &path, args)?;
+ let r = BlockingReader::create(inner, &path, args, options)?;
let buf = r.read(range.to_range())?;
Ok(buf)
},
@@ -443,8 +443,8 @@ impl BlockingOperator {
FunctionReader(OperatorFunction::new(
self.inner().clone(),
path,
- OpRead::default(),
- |inner, path, args| {
+ (OpRead::default(), OpReader::default()),
+ |inner, path, (args, options)| {
if !validate_path(&path, EntryMode::FILE) {
return Err(
Error::new(ErrorKind::IsADirectory, "reader path is a directory")
@@ -454,7 +454,7 @@ impl BlockingOperator {
);
}
- BlockingReader::create(inner.clone(), &path, args)
+ BlockingReader::create(inner.clone(), &path, args, options)
},
))
}
diff --git a/core/src/types/operator/operator_functions.rs b/core/src/types/operator/operator_functions.rs
index 64e4fcf038..ef584f884c 100644
--- a/core/src/types/operator/operator_functions.rs
+++ b/core/src/types/operator/operator_functions.rs
@@ -320,12 +320,14 @@ impl FunctionLister {
/// Function that generated by [`BlockingOperator::read_with`].
///
/// Users can add more options by public functions provided by this struct.
-pub struct FunctionRead(pub(crate) OperatorFunction<(OpRead, BytesRange), Buffer>);
+pub struct FunctionRead(pub(crate) OperatorFunction<(OpRead, BytesRange, OpReader), Buffer>);
impl FunctionRead {
/// Set the range for this operation.
pub fn range(mut self, range: impl RangeBounds<u64>) -> Self {
- self.0 = self.0.map_args(|(args, _)| (args, range.into()));
+ self.0 = self
+ .0
+ .map_args(|(args, _, options)| (args, range.into(), options));
self
}
@@ -334,19 +336,30 @@ impl FunctionRead {
pub fn call(self) -> Result<Buffer> {
self.0.call()
}
+
+ /// Set the concurrent read task amount.
+ pub fn concurrent(mut self, concurrent: usize) -> Self {
+ self.0 = self
+ .0
+ .map_args(|(args, range, options)| (args, range, options.with_concurrent(concurrent)));
+ self
+ }
}
/// Function that generated by [`BlockingOperator::reader_with`].
///
/// Users can add more options by public functions provided by this struct.
-pub struct FunctionReader(pub(crate) OperatorFunction<OpRead, BlockingReader>);
+pub struct FunctionReader(pub(crate) OperatorFunction<(OpRead, OpReader), BlockingReader>);
impl FunctionReader {
/// Sets the content-disposition header that should be send back by the remote read operation.
pub fn override_content_disposition(mut self, content_disposition: &str) -> Self {
- self.0 = self
- .0
- .map_args(|args| args.with_override_content_disposition(content_disposition));
+ self.0 = self.0.map_args(|(args, options)| {
+ (
+ args.with_override_content_disposition(content_disposition),
+ options,
+ )
+ });
self
}
@@ -354,7 +367,7 @@ impl FunctionReader {
pub fn override_cache_control(mut self, cache_control: &str) -> Self {
self.0 = self
.0
- .map_args(|args| args.with_override_cache_control(cache_control));
+ .map_args(|(args, options)| (args.with_override_cache_control(cache_control), options));
self
}
@@ -362,25 +375,31 @@ impl FunctionReader {
pub fn override_content_type(mut self, content_type: &str) -> Self {
self.0 = self
.0
- .map_args(|args| args.with_override_content_type(content_type));
+ .map_args(|(args, options)| (args.with_override_content_type(content_type), options));
self
}
/// Set the If-Match for this operation.
pub fn if_match(mut self, v: &str) -> Self {
- self.0 = self.0.map_args(|args| args.with_if_match(v));
+ self.0 = self
+ .0
+ .map_args(|(args, options)| (args.with_if_match(v), options));
self
}
/// Set the If-None-Match for this operation.
pub fn if_none_match(mut self, v: &str) -> Self {
- self.0 = self.0.map_args(|args| args.with_if_none_match(v));
+ self.0 = self
+ .0
+ .map_args(|(args, options)| (args.with_if_none_match(v), options));
self
}
/// Set the version for this operation.
pub fn version(mut self, v: &str) -> Self {
- self.0 = self.0.map_args(|args| args.with_version(v));
+ self.0 = self
+ .0
+ .map_args(|(args, options)| (args.with_version(v), options));
self
}