You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "wer860612 (via GitHub)" <gi...@apache.org> on 2025/05/03 17:18:03 UTC

[D] Task pod fails to update MySQL due to "Too many connections", Istio sidecar exits, pod stuck in error state (Airflow 2.6.2, KubernetesExecutor) [airflow]

GitHub user wer860612 created a discussion: Task pod fails to update MySQL due to "Too many connections", Istio sidecar exits, pod stuck in error state (Airflow 2.6.2, KubernetesExecutor)

Hi everyone,

I'm running Apache Airflow 2.6.2 with the KubernetesExecutor in a Kubernetes environment. My task pods are injected with an Istio sidecar, so each pod has two containers: the task container and the Istio proxy container.

Recently, I've encountered an issue where a task completes its logic successfully, but fails when attempting to update the MySQL metadata database with session.commit() due to this error:

```
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1040, 'Too many connections')
```
After this error, I noticed the Istio sidecar container closed, leaving the pod in a 1/2 error status (one container completed, one still running). The task container doesn't crash, but the pod gets stuck in this state and is never marked as "zombie" by the scheduler 
```
NAME           READY   STATUS
my-task-pod    1/2     Error
```
So far, these pods just stay in this half-broken state until I manually delete them.

Questions:
Why does the Istio container terminate in this situation? Is it due to network or healthcheck issues when the DB is unreachable?

Why doesn't the scheduler mark this task as zombie? Is it because the task never registered as "RUNNING" in the metadata DB?

Are there best practices to avoid this issue entirely — e.g., connection pool tuning, Istio retry configs, or Airflow-level safeguards?

I have already adjusted the MySQL max_connections setting so that this issue no longer occurs, but I still want to understand the root cause.

Any insights or suggestions are greatly appreciated!

Thanks!

GitHub link: https://github.com/apache/airflow/discussions/50151

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org