You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2026/09/14 07:40:41 UTC

[PR] perf(bindings/python): avoid staging copies in AsyncFile writes [opendal]

Xuanwo opened a new pull request, #8298:
URL: https://github.com/apache/opendal/pull/8298

   # Which issue does this PR close?
   
   Related to #8246.
   
   # Rationale for this change
   
   `AsyncFile.write` retains immutable Python bytes, but passing them through `FuturesAsyncWriter.write_all` copies them into the adapter's staging buffer before the core writer performs storage chunking. Avoiding that remaining copy reduces upload CPU and memory use.
   
   # What changes are included in this PR?
   
   Use the existing owner-preserving Python-to-`Buffer` conversion and the core `BufferSink` for async writes. The sink retains pending write state across cancellation. Keep small writes coalesced with a 256 KiB fallback when neither the caller nor the service provides a chunk size.
   
   Regression coverage checks Python owner lifetime, cancellation with the original buffer pointer preserved, mixed write sizes, request coalescing, close errors and resuming a cancelled close.
   
   # Are there any user-facing changes?
   
   No API changes. `AsyncFile.write` continues to accept `bytes` and return the full accepted byte count. Context-manager completion and close error behavior are preserved. Buffered and in-flight writes can retain the original Python bytes until the core releases them. Synchronous `File` is unchanged.
   
   ## Benchmark
   
   Real S3 upload of the same 59,158,238-byte Parquet file with 8 MiB reads/parts and CRC32C, including read, write and close. Before: `8a4a4ed976b49afa582518df7e84c2e7b253e5c7`; after: `174e04ea3a9e32dc74e165af58ea91f2afd61210`. Both release builds used the same dependency lockfile. Measurements used a `c7i.8xlarge` in `us-east-2`, limited to four CPUs and 1 GiB RAM, with 30 balanced pairs per concurrency.
   
   | Configured concurrency | Median elapsed, before → after | Median process CPU, before → after | Median peak RSS, before → after |
   | --- | ---: | ---: | ---: |
   | 1 | 978 → 982 ms | 71.4 → 60.1 ms | 73.9 → 63.9 MiB |
   | 4 | 416 → 391 ms | 78.3 → 61.7 ms | 97.9 → 89.1 MiB |
   | 8 | 326 → 319 ms | 80.1 → 62.7 ms | 106.6 → 90.5 MiB |
   
   Resource savings are consistent; elapsed-time gains are modest and workload-dependent. C1 has no clear latency improvement. Network bandwidth-allowance counters increased in some C4 samples and every C8 sample. Separate request traces also found that the shared scheduler can transiently exceed configured C4, so the C4 gain cannot be attributed entirely to copying. This PR does not change that scheduler or claim a universal speedup.
   
   Every measured upload passed independent full SHA-256 readback and multipart checksum validation. Additional real-S3 checks covered empty, tiny, exact-part, non-multiple and mixed writes. Local cancellation/close regression tests passed; real-S3 tests did not inject retries or cancellation. Direct allocation counting remains outside the collected evidence, so this PR leaves #8246 open for the remaining validation scope.
   
   # AI Usage Statement
   
   AI assisted implementation, regression tests and benchmark analysis under human direction and review. Measurement limitations and remaining validation gaps are stated above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(bindings/python): avoid staging copies in AsyncFile writes [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #8298:
URL: https://github.com/apache/opendal/pull/8298


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org