You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jens Scheffler <js...@apache.org> on 2026/03/15 22:29:09 UTC
[LAZY CONSENSUS] Counting "Deferred" task status as running
Hi Airflow Devs,
as discussed in
https://lists.apache.org/thread/vvjpo6q6h0j4f80dq36g4oo0mrp5ldbq about
how deferred status of tasks is counted into running, after discussion
has "calmed down" I'd like to call for a LAZY consensus with the
following outcome:
* Short term, as some (like Bosch) is affected by deferred tasks not
counting into Dag / Task / TI concurrency we continue with Fix PR
https://github.com/apache/airflow/pull/61769
* We make the counting of deferred status a config flag so that (on a
global level) depending on needs deferred can be counted-in or not.
o Means technically: All 3 concurrency maps
"dag_run_active_tasks_map (dag_id+run_id)",
"task_concurrency_map (dag_id+task_id)" and
"task_dagrun_concurrency_map (dag_id+run_id+task_id)" will
either include deferred or not
* Defaulting to current behavior which does NOT count deferred in.
As the discussion revealed there is a general demand to configure
deferred counting on a per Dag level (like it is today with pools) we
agree on developing a mid-term concept that concurrency can be defined
with more parameters. But this would take a bit more time.
If somebody objects to this please respond to this email in the next 72h
as calling for LAZY CONSENSUS, meaning until Wednesday March 18th, 23:30
CET.
Target would be that the config flag can get into 3.2.0 as a short term
remidiation.
Jens
Re: [LAZY CONSENSUS] Counting "Deferred" task status as running
Posted by Jens Scheffler <js...@apache.org>.
Lazy consensus passed.
On 16.03.26 20:47, Jens Scheffler wrote:
> Hi Eugene,
>
> I'd say for now the `parallelism` parameter falls into the category:
>
>> As the discussion revealed there is a general demand to configure
>> deferred counting on a per Dag level (like it is today with pools) we
>> agree on developing a mid-term concept that concurrency can be
>> defined with more parameters. But this would take a bit more time.
>
> Jens
>
> On 16.03.26 09:34, Eugen Kosteev wrote:
>> Hi Jens.
>>
>> What about parallelism parameter?
>> https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parallelism
>>
>>
>> Will deferred tasks count also as running there?
>>
>> - Eugene
>>
>> On Sun, Mar 15, 2026 at 11:29 PM Jens Scheffler<js...@apache.org>
>> wrote:
>>
>>> Hi Airflow Devs,
>>>
>>> as discussed in
>>> https://lists.apache.org/thread/vvjpo6q6h0j4f80dq36g4oo0mrp5ldbq about
>>> how deferred status of tasks is counted into running, after discussion
>>> has "calmed down" I'd like to call for a LAZY consensus with the
>>> following outcome:
>>>
>>> * Short term, as some (like Bosch) is affected by deferred tasks not
>>> counting into Dag / Task / TI concurrency we continue with Fix PR
>>> https://github.com/apache/airflow/pull/61769
>>> * We make the counting of deferred status a config flag so that
>>> (on a
>>> global level) depending on needs deferred can be counted-in or
>>> not.
>>> o Means technically: All 3 concurrency maps
>>> "dag_run_active_tasks_map (dag_id+run_id)",
>>> "task_concurrency_map (dag_id+task_id)" and
>>> "task_dagrun_concurrency_map (dag_id+run_id+task_id)" will
>>> either include deferred or not
>>> * Defaulting to current behavior which does NOT count deferred in.
>>>
>>> As the discussion revealed there is a general demand to configure
>>> deferred counting on a per Dag level (like it is today with pools) we
>>> agree on developing a mid-term concept that concurrency can be defined
>>> with more parameters. But this would take a bit more time.
>>>
>>> If somebody objects to this please respond to this email in the next
>>> 72h
>>> as calling for LAZY CONSENSUS, meaning until Wednesday March 18th,
>>> 23:30
>>> CET.
>>>
>>> Target would be that the config flag can get into 3.2.0 as a short term
>>> remidiation.
>>>
>>> Jens
>>>
>>
Re: [LAZY CONSENSUS] Counting "Deferred" task status as running
Posted by Jens Scheffler <js...@apache.org>.
Hi Eugene,
I'd say for now the `parallelism` parameter falls into the category:
> As the discussion revealed there is a general demand to configure deferred counting on a per Dag level (like it is today with pools) we agree on developing a mid-term concept that concurrency can be defined with more parameters. But this would take a bit more time.
Jens
On 16.03.26 09:34, Eugen Kosteev wrote:
> Hi Jens.
>
> What about parallelism parameter?
> https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parallelism
>
> Will deferred tasks count also as running there?
>
> - Eugene
>
> On Sun, Mar 15, 2026 at 11:29 PM Jens Scheffler<js...@apache.org> wrote:
>
>> Hi Airflow Devs,
>>
>> as discussed in
>> https://lists.apache.org/thread/vvjpo6q6h0j4f80dq36g4oo0mrp5ldbq about
>> how deferred status of tasks is counted into running, after discussion
>> has "calmed down" I'd like to call for a LAZY consensus with the
>> following outcome:
>>
>> * Short term, as some (like Bosch) is affected by deferred tasks not
>> counting into Dag / Task / TI concurrency we continue with Fix PR
>> https://github.com/apache/airflow/pull/61769
>> * We make the counting of deferred status a config flag so that (on a
>> global level) depending on needs deferred can be counted-in or not.
>> o Means technically: All 3 concurrency maps
>> "dag_run_active_tasks_map (dag_id+run_id)",
>> "task_concurrency_map (dag_id+task_id)" and
>> "task_dagrun_concurrency_map (dag_id+run_id+task_id)" will
>> either include deferred or not
>> * Defaulting to current behavior which does NOT count deferred in.
>>
>> As the discussion revealed there is a general demand to configure
>> deferred counting on a per Dag level (like it is today with pools) we
>> agree on developing a mid-term concept that concurrency can be defined
>> with more parameters. But this would take a bit more time.
>>
>> If somebody objects to this please respond to this email in the next 72h
>> as calling for LAZY CONSENSUS, meaning until Wednesday March 18th, 23:30
>> CET.
>>
>> Target would be that the config flag can get into 3.2.0 as a short term
>> remidiation.
>>
>> Jens
>>
>
Re: [LAZY CONSENSUS] Counting "Deferred" task status as running
Posted by Eugen Kosteev <eu...@kosteev.com>.
Hi Jens.
What about parallelism parameter?
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parallelism
Will deferred tasks count also as running there?
- Eugene
On Sun, Mar 15, 2026 at 11:29 PM Jens Scheffler <js...@apache.org> wrote:
> Hi Airflow Devs,
>
> as discussed in
> https://lists.apache.org/thread/vvjpo6q6h0j4f80dq36g4oo0mrp5ldbq about
> how deferred status of tasks is counted into running, after discussion
> has "calmed down" I'd like to call for a LAZY consensus with the
> following outcome:
>
> * Short term, as some (like Bosch) is affected by deferred tasks not
> counting into Dag / Task / TI concurrency we continue with Fix PR
> https://github.com/apache/airflow/pull/61769
> * We make the counting of deferred status a config flag so that (on a
> global level) depending on needs deferred can be counted-in or not.
> o Means technically: All 3 concurrency maps
> "dag_run_active_tasks_map (dag_id+run_id)",
> "task_concurrency_map (dag_id+task_id)" and
> "task_dagrun_concurrency_map (dag_id+run_id+task_id)" will
> either include deferred or not
> * Defaulting to current behavior which does NOT count deferred in.
>
> As the discussion revealed there is a general demand to configure
> deferred counting on a per Dag level (like it is today with pools) we
> agree on developing a mid-term concept that concurrency can be defined
> with more parameters. But this would take a bit more time.
>
> If somebody objects to this please respond to this email in the next 72h
> as calling for LAZY CONSENSUS, meaning until Wednesday March 18th, 23:30
> CET.
>
> Target would be that the config flag can get into 3.2.0 as a short term
> remidiation.
>
> Jens
>
--
Eugene