You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "MaksymSofer (via GitHub)" <gi...@apache.org> on 2025/03/15 07:25:01 UTC

[D] How parallelism works in KubernetesExecutor? [airflow]

GitHub user MaksymSofer created a discussion: How parallelism works in KubernetesExecutor?

Hello,
I use Airflow in GKE, parallelism is set to 32
i have three dags that can bring (even on theoretical level only 12 tasks). 
but i see constant issue with Scheduler going into 
`Executor parallelism limit reached. 0 open slots.`

Checking executor code, i see that open_slots defined as:

[open_slots = self.parallelism - len(self.running)](https://github.com/apache/airflow/blob/945083250d2a4c58344c4d2db64f83b6caa06a6a/airflow/executors/base_executor.py#L244C13-L244C62)

I spotted that KubernetesExecutor leave "Succeeded" pods - is it possible that they calculated into "running" amount?

Please help me.




GitHub link: https://github.com/apache/airflow/discussions/47805

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How parallelism works in KubernetesExecutor? [airflow]

Posted by "MaksymSofer (via GitHub)" <gi...@apache.org>.

GitHub user MaksymSofer edited a discussion: How parallelism works in KubernetesExecutor?

Hello,
I use Airflow 2.10.1 in GKE, parallelism is set to 32
i have three dags that can bring (even on theoretical level only 12 simultaneous tasks). 
but i see constant issue with Scheduler going into 
`Executor parallelism limit reached. 0 open slots.`

Checking executor code, i see that open_slots defined as:

[open_slots = self.parallelism - len(self.running)](https://github.com/apache/airflow/blob/945083250d2a4c58344c4d2db64f83b6caa06a6a/airflow/executors/base_executor.py#L244C13-L244C62)

I spotted that KubernetesExecutor leave "Succeeded" pods - is it possible that they calculated into "running" amount?

Please help me.




GitHub link: https://github.com/apache/airflow/discussions/47805

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How parallelism works in KubernetesExecutor? [airflow]

Posted by "MaksymSofer (via GitHub)" <gi...@apache.org>.

GitHub user MaksymSofer added a comment to the discussion: How parallelism works in KubernetesExecutor?

Any one?

GitHub link: https://github.com/apache/airflow/discussions/47805#discussioncomment-12713412

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How parallelism works in KubernetesExecutor? [airflow]

Posted by "MaksymSofer (via GitHub)" <gi...@apache.org>.

GitHub user MaksymSofer added a comment to the discussion: How parallelism works in KubernetesExecutor?

Did that but not found answer on why airflow thinks I have running pods, while I dont

GitHub link: https://github.com/apache/airflow/discussions/47805#discussioncomment-12756906

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How parallelism works in KubernetesExecutor? [airflow]

Posted by "MaksymSofer (via GitHub)" <gi...@apache.org>.

GitHub user MaksymSofer added a comment to the discussion: How parallelism works in KubernetesExecutor?

I enabled sending of traces to tempo and now i see that numbers actually count succeded pods

![image](https://github.com/user-attachments/assets/20f0f74d-7458-42e2-a535-19ebbb99abb1)
this correspond to tempo saved trace:

![image](https://github.com/user-attachments/assets/56dbe148-171d-4899-9811-1662a36bdf54)


GitHub link: https://github.com/apache/airflow/discussions/47805#discussioncomment-12509490

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How parallelism works in KubernetesExecutor? [airflow]

Posted by "MaksymSofer (via GitHub)" <gi...@apache.org>.

GitHub user MaksymSofer added a comment to the discussion: How parallelism works in KubernetesExecutor?

Scheduler again went into "no open slots" state and cluster overview (from airflow ui) show this:
![image](https://github.com/user-attachments/assets/cfabd69f-0113-46f3-9766-e6e9e93eed62)
while only two running tasks are:
![image](https://github.com/user-attachments/assets/ffaba7d7-3ae7-4020-9ed3-4f4f830a9bb1)

which 18 tasks scheduler see with "no state" i dont know,,,


GitHub link: https://github.com/apache/airflow/discussions/47805#discussioncomment-12512641

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How parallelism works in KubernetesExecutor? [airflow]

Posted by "MaksymSofer (via GitHub)" <gi...@apache.org>.

GitHub user MaksymSofer edited a discussion: How parallelism works in KubernetesExecutor?

Hello,
I use Airflow 2.10.1 in GKE, parallelism is set to 32
i have three dags that can bring (even on theoretical level only 12 tasks). 
but i see constant issue with Scheduler going into 
`Executor parallelism limit reached. 0 open slots.`

Checking executor code, i see that open_slots defined as:

[open_slots = self.parallelism - len(self.running)](https://github.com/apache/airflow/blob/945083250d2a4c58344c4d2db64f83b6caa06a6a/airflow/executors/base_executor.py#L244C13-L244C62)

I spotted that KubernetesExecutor leave "Succeeded" pods - is it possible that they calculated into "running" amount?

Please help me.




GitHub link: https://github.com/apache/airflow/discussions/47805

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How parallelism works in KubernetesExecutor? [airflow]

Posted by "jsjasonseba (via GitHub)" <gi...@apache.org>.

GitHub user jsjasonseba edited a comment on the discussion: How parallelism works in KubernetesExecutor?

Hi @MaksymSofer, I use KubernetesExecutor and succeeded pods are not counted as running tasks.

You should try browsing task instances from Airflow UI and filter by status to check if the task counts match. There you can identify running tasks and also figure out the no status tasks you mentioned.

GitHub link: https://github.com/apache/airflow/discussions/47805#discussioncomment-12755334

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How parallelism works in KubernetesExecutor? [airflow]

Posted by "jsjasonseba (via GitHub)" <gi...@apache.org>.

GitHub user jsjasonseba added a comment to the discussion: How parallelism works in KubernetesExecutor?

Hi @MaksymSofer, I use KubernetesExecutor and succeeded pods are not counted as running tasks.

You should try browsing task instances from Airflow UI and filter by status to check if the task counts match. There you can identify running tasks and also firugre out the no status tasks you mentioned.

GitHub link: https://github.com/apache/airflow/discussions/47805#discussioncomment-12755334

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org