[PR] docs(core): Add performance guide for concurrent write [opendal]

["Xuanwo (via GitHub)" <gi...@apache.org>]

Xuanwo opened a new pull request, #6011:
URL: https://github.com/apache/opendal/pull/6011

   # Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Part of https://github.com/apache/opendal/issues/4121
   
   # Rationale for this change
   
   Add performance guide for concurrently write.
   
   # What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   # Are there any user-facing changes?
   
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!---
   If there are any breaking changes to public APIs, please add the `breaking-changes` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(core): Add performance guide for concurrent write [opendal]

["Xuanwo (via GitHub)" <gi...@apache.org>]

Xuanwo merged PR #6011:
URL: https://github.com/apache/opendal/pull/6011


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(core): Add performance guide for concurrent write [opendal]

["chitralverma (via GitHub)" <gi...@apache.org>]

chitralverma commented on code in PR #6011:
URL: https://github.com/apache/opendal/pull/6011#discussion_r2041600688


##########
core/src/docs/performance/concurrent_write.md:
##########
@@ -0,0 +1,99 @@
+# Concurrent Write
+
+OpenDAL writes data sequentially by default.
+
+```rust
+# use opendal::Operator;
+# async fn test() {
+let w = op.writer("test.txt").await?;
+w.write(data1).await?;
+w.write(data2).await?;
+w.close().await?;
+# }
+```
+
+Most of the time, this can't maximize write performance due to limitations on a single connection. We can perform concurrent writes to improve performance.
+
+```rust
+# use opendal::Operator;
+# async fn test() {
+let w = op
+    .writer_with("test.txt")
+    .concurrent(8)
+    .await?;
+
+w.write(data1).await?;
+w.write(data2).await?;
+w.close().await?;
+# }
+```
+
+After setting `concurrent`, OpenDAL will attempt to write the specified file concurrently. The maximum level of concurrency is determined by the `concurrent` parameter. By default, it is set to 1, indicating a sequential write operation.
+
+Under the hood, OpenDAL maintains a task queue to manage concurrent writes. It spawns asynchronous tasks in the background using the [`Executor`][crate::Executor] and tracks the status of each task. The task queue is flushed when the writer is closed, allowing data to be written concurrently without blocking the main thread.
+
+Take our example here, the write of `data1` will not block the write of `data2`. The two writes will be executed concurrently.
+
+The underlying implementation of concurrent writes may vary depending on the backend. For instance, the `s3` backend leverages the S3 Multipart Uploads API to handle concurrent writes, while the `azblob` backend utilizes the Block API for the same purpose.
+
+## Tuning
+
+There are two parameters that can be tuned to optimize concurrent writes:
+
+- `concurrent`: This parameter controls the maximum number of concurrent writes. Increasing this value can improve performance, but it may also increase resource usage. The default value is 1.

Review Comment:
   @Xuanwo instead of "increase resource usage" can we be a little more explicit with which resource utilization is increase (cpu/ mem, disk buffering etc.), same for other sections.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org