[D] Dynamic allocation of pool slots for expanded tasks [airflow]

["curious-bug (via GitHub)" <gi...@apache.org>]

GitHub user curious-bug created a discussion: Dynamic allocation of pool slots for expanded tasks

I use airflow to orchestrate execution of queries on our SQL DB.  There are mapped tasks where some expanded parameters lead to heavy DB usage that can cause failures if to many queries are running at once.
As such, I want to allow a only small number of these heavy queries to run at once while still allowing the lighter queries to be more parallelized. The current solution is two sets of mapped tasks with different numbers of used pool slots but I'd like to make it just one task with pool sizing being automatically assigned during the task expansion since the current setup is pretty messy.

Basically, I'd like to do the following:
```python
# Xcom form is like [{'pool_slots': a, 'params': b}, ...]
Operator.partial(
    task_id="dynamic_pool_sizing",
    pool="my_pool",
    ...
).expand_kwargs(xcom_with_pool_sizes)
```

I'm aware of Issue https://github.com/apache/airflow/issues/35803 was wondering if this situation would be handled differently by Airflow.

GitHub link: https://github.com/apache/airflow/discussions/53774

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Dynamic allocation of pool slots for expanded tasks [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

GitHub user potiuk added a comment to the discussion: Dynamic allocation of pool slots for expanded tasks

Possibly you are looking for `task_concurrency` - basic feature of the operator.

GitHub link: https://github.com/apache/airflow/discussions/53774#discussioncomment-13894792

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Dynamic allocation of pool slots for expanded tasks [airflow]

["curious-bug (via GitHub)" <gi...@apache.org>]

GitHub user curious-bug added a comment to the discussion: Dynamic allocation of pool slots for expanded tasks

No, concurrency is handled by the pool for me.
I'm asking if its possible to dynamically set the pool sizing of a single task during map/expansion

GitHub link: https://github.com/apache/airflow/discussions/53774#discussioncomment-13912785

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org