[I] [client] Isolate historical lookup in-flight requests [fluss]

["luoyuxia (via GitHub)" <gi...@apache.org>]

luoyuxia opened a new issue, #3766:
URL: https://github.com/apache/fluss/issues/3766

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   ### Description
   
   Normal and historical lookup requests currently share the `maxInFlightReuqestsSemaphore` in `LookupSender`. This keeps the initial historical lookup implementation simple, but lake lookups can be much slower than normal lookups. A burst of historical requests can therefore consume all in-flight permits and prevent normal lookup requests from being sent until lake requests complete.
   
   FIP-28 proposes separating the capacity available to historical lookups through `client.lookup.historical-inflight-ratio`, with historical requests capped at a small portion of the overall lookup concurrency. The initial implementation intentionally omitted this configuration after discussion because historical lookups were expected to be infrequent.
   
   We should revisit this assumption and isolate or reserve in-flight capacity for historical lookups if shared permits can cause normal lookup starvation. The implementation should:
   
   - prevent historical lookup traffic from consuming all normal lookup permits;
   - preserve the existing overall lookup in-flight limit;
   - define validation and rounding behavior for small limits and ratio values;
   - add coverage showing that normal lookups can proceed while historical lookups are slow.
   
   ### Willingness to contribute
   
   - [ ] I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [client] Isolate historical lookup in-flight requests [fluss]

["vvvdwbvvv (via GitHub)" <gi...@apache.org>]

vvvdwbvvv commented on issue #3766:
URL: https://github.com/apache/fluss/issues/3766#issuecomment-5086033112

    I’m considering keeping the existing global limit N and adding a historical cap H. Normal lookups may use all N permits, while historical lookups must acquire
     both permits non-blockingly and are re-enqueued when H is full.
   
     This preserves normal-only throughput and guarantees total in-flight requests <= N and historical requests <= H. The trade-off is that historical lookups may
     wait under sustained normal traffic.
   
     Would this priority trade-off be acceptable? If so, I propose ratio in (0, 1), N >= 2, and H = max(1, floor(N * ratio)), treating the ratio as a cap rather
     than reserved capacity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [client] Isolate historical lookup in-flight requests [fluss]

["vvvdwbvvv (via GitHub)" <gi...@apache.org>]

vvvdwbvvv commented on issue #3766:
URL: https://github.com/apache/fluss/issues/3766#issuecomment-5078019213

   @luoyuxia I would like to contribute on this issue, can you assign me?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org