[GitHub] [airflow] BeatlesMD opened a new issue #15082: Impose rate limits for task starts as pool feature

[GitBox <gi...@apache.org>]

BeatlesMD opened a new issue #15082:
URL: https://github.com/apache/airflow/issues/15082


   **Description**
   
   As a pool feature, queue task starts such that task initiation is distributed over time according to a sliding window rate limit (may be more easily implemented as task initiation cool down within a pool).
   
   **Use case / motivation**
   
   APIs commonly will impose certain techniques to limit rate of requests (sliding window, fixed window, token bucket, leaky bucket, etc). While task retries may resolve the issue, all failures could potentially be avoided if there was a feature to match the endpoint's programatic behavior (I suggest sliding window as it has other benefits).
   
   A sliding window rate limiter could also be used to stagger task/request initiation to a legacy system. There are a number of reasons why you may want to stagger requests to a legacy system, such as if the beginning portion of a request is the most resource intensive within the foreign system, or if the legacy system itself does not itself provide its own rate-limiting signals.
   
   It may be more easily implemented as a cool down between a pooled task initiation and the next queued task start, but figured I'd frame the feature request to match other rate-limiting strategies and techniques commonly seen.
   
   **Related Issues**
   
   Potentially https://github.com/apache/airflow/issues/8789 ?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #15082: Impose rate limits for task starts as pool feature

[GitBox <gi...@apache.org>]

boring-cyborg[bot] commented on issue #15082:
URL: https://github.com/apache/airflow/issues/15082#issuecomment-809820912


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org