[I] new feature (or bug?): concurrent limit layer and concurrent read could lead to deadlock [opendal]

["dentiny (via GitHub)" <gi...@apache.org>]

dentiny opened a new issue, #7338:
URL: https://github.com/apache/opendal/issues/7338

   ### Feature Description
   
   Correct me if wrong, this is my understand on how concurrent read works:
   - When users do `read_with().chunk().concurrent(C)`, the concurrent tasks will create C slots and try to issue as many HTTP requests as possible.
   - If concurrent limit layer applied, HTTP requests need to acquire a permit first, and the acquired permit keeps valid until the stream gets dropped.
   - But the issue is, concurrent tasks hold the ownership of the opened streams, and only try to sync when the operation-level concurrent limit reaches (the `C` above)
   
   In a detailed example,
   - Suppose concurrent limit is set to 2, and we issue 4 read operations, which will be turned into 16 HTTP requests
   - Assume the HTTP request is sent in the order of
     + read operation 1, HTTP request 1 -> acquires permit and holds
     + read operation 2, HTTP request 1 -> acquires permit and holds
     + Later all requests get blocked at permit acquisition
   
   I found this issue annoying because it doesn't always reproduce in production.
   
   ### Problem and Solution
   
   To workaround, I implement my own concurrent limit layer. Two main goals:
   - No non-deterministic deadlock
   - Have a process-wise concurrent limit for HTTP requests, which serves as a workaround for [missing TCP connection pool](https://github.com/apache/opendal/pull/7244)
   
   Before every read operation, I attempted as CAS operation to acquire a batch of permits, which get released when the read operation finishes.
   
   I'm wondering if it's possible to connect (1) concurrent tasks and  (2) concurrent limit layer (or self-implemented layers) better, so it's aware of the current actual limit?
   For example, pass a functor to chunk reader?
   
   ### Additional Context
   
   _No response_
   
   ### Are you willing to contribute to the development of this feature?
   
   - [ ] Yes, I am willing to contribute to the development of this feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] new feature (or bug?): concurrent limit layer and concurrent read could lead to deadlock [opendal]

["Xuanwo (via GitHub)" <gi...@apache.org>]

Xuanwo closed issue #7338: new feature (or bug?): concurrent limit layer and concurrent read could lead to deadlock
URL: https://github.com/apache/opendal/issues/7338


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org