[I] bug(core): failed tasks can retain payloads and delay caller errors [opendal]

["v0y4g3r (via GitHub)" <gi...@apache.org>]

v0y4g3r opened a new issue, #8325:
URL: https://github.com/apache/opendal/issues/8325

   ### OpenDAL version or commit
   
   `2698e90a1131a477fc3382465f1c28bd2202395e`
   
   ### Affected service or component
   
   `core::raw::ConcurrentTasks`, used by `PositionWriter`, `MultipartWriter`, and `BlockWriter`
   
   ### Describe the bug
   
   After #8224, successful task inputs are released before result collection, but failed tasks still retain their retry inputs. Those failed tasks also provide prefetch capacity, and the scheduler normally collects results only when the queue is full or the caller explicitly drains it.
   
   With the default writer settings (`concurrent=8`, `prefetch=8192`), a backend that repeatedly fails quickly can leave 8,200 completed failed tasks queued before a caller observes an error. In a deterministic 8 KiB chunk probe, those tasks retained 67,182,592 bytes of input. With an 8 MiB chunk, the same task count corresponds to approximately 64.07 GiB of retained input.
   
   The caller also does not see the error promptly: the first backend failure can complete during write call 2, while `Writer::write` does not return an error until write call 8,202.
   
   ### Steps to Reproduce
   
   The deterministic reproduction uses the real `PositionWriter`, `MultipartWriter`, and `BlockWriter` implementations from the affected commit, with `bytes=1.11.1` and the default writer settings (`concurrent=8`, `prefetch=8192`).
   
   It:
   
   - injects temporary or permanent errors at the raw `oio` storage boundary of a locally controlled backend;
   - tracks payload ownership through `Bytes::from_owner`, so the measurements observe payload lifetime rather than RSS;
   - waits after each accepted write until the earlier background task has fully completed, so later submissions cannot be explained by a task that has not had an opportunity to run;
   - checks payload bytes, offsets, multipart part metadata, block ordering, temporary-error recovery, permanent-error abort behavior, and close behavior.
   
   ### Actual Behavior
   
   For each of `PositionWriter`, `MultipartWriter`, and `BlockWriter` with `concurrent=8`, `prefetch=8192`, and fast injected failures:
   
   | Backend failure scenario | Accepted writes before caller error | Failed completed tasks | Live payloads | Payload bytes |
   |---|---:|---:|---:|---:|
   | All writes fail temporarily | 8,201 | 8,200 | 8,201 | 67,182,592 |
   | All writes fail permanently | 8,201 | 8,200 | 8,201 | 67,182,592 |
   | Only the first submitted write fails permanently | 8,201 | 1 | 2 | 16,384 |
   
   Progression while every write fails quickly:
   
   ```text
   accepted=2    finished=1    backend_failures=1    live=2
   accepted=1024 finished=1023 backend_failures=1023 live=1024
   accepted=4096 finished=4095 backend_failures=4095 live=4096
   accepted=8201 finished=8200 backend_failures=8200 live=8201
   FIRST_ERROR call=8202 backend_first_failure_call=2
   ```
   
   `concurrent=1` returns the backend error on the second write call. Direct `ConcurrentTasks` checks show that the retained failed-task count scales with the configured prefetch value.
   
   This measurement tracks payload ownership lifetime in the component under test; it is not a production RSS measurement.
   
   ### Expected Behavior
   
   A completed failed task should not provide capacity for thousands of new submissions while its retry input remains queued. After the first backend failure is known to the scheduler, the next write call should apply backpressure and process that failure instead of continuing to accept large amounts of new input.
   
   Temporary errors should still preserve the retry input and ordered results. Retaining that input is required for an in-place retry: the backend operation may already have consumed or dropped its cloned buffer, and retrying needs the same payload to avoid lost or mismatched data. The issue is not the retained input itself; it is that completed failed tasks also provide additional prefetch capacity and can remain uncollected until the queue is full or the caller closes the writer.
   
   ### Additional Context
   
   - The issue affects the shared scheduler rather than a single storage service. It can affect filesystem and object-store writes that use position, multipart, or block writers.
   - The reproduction intentionally uses the affected core component with a controlled local backend; it does not claim a new service-specific protocol bug.
   - A proposed scheduler-level fix was validated locally with 240 core unit tests, deterministic scheduler regressions, and a 32-scenario writer probe. Those local checks do not replace real-service performance validation.
   
   ### Reproduction confirmation
   
   - [x] I reproduced this issue against the affected component. For a service-specific report, I used the actual service rather than only a mock, emulator, or source-code analysis.
   
   ### Are you willing to submit a PR to fix this bug?
   
   - [x] Yes, I would like to submit a PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] bug(core): failed tasks can retain payloads and delay caller errors [opendal]

["Xuanwo (via GitHub)" <gi...@apache.org>]

Xuanwo closed issue #8325: bug(core): failed tasks can retain payloads and delay caller errors
URL: https://github.com/apache/opendal/issues/8325


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org