You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "davidjfrickert (via GitHub)" <gi...@apache.org> on 2026/04/16 03:38:41 UTC
[I] Upgrade from 3.1.7 to 3.2.0 stuck on Database migrations (MySQL) [airflow]
davidjfrickert opened a new issue, #65350:
URL: https://github.com/apache/airflow/issues/65350
### Under which category would you file this issue?
Helm chart
### Apache Airflow version
3.1.7 -> 3.2.0
### What happened and how to reproduce it?
Upgraded Airflow from 3.1.7 and 3.2.0. The db migrations pod was stuck for 20+ minutes.
Some logs:
```
2026-04-16T02:51:13.804504Z [info ] Context impl MySQLImpl. [alembic.runtime.migration] loc=migration.py:210
2026-04-16T02:51:13.804607Z [info ] Will assume non-transactional DDL. [alembic.runtime.migration] loc=migration.py:213
2026-04-16T02:51:13.806593Z [info ] Migrating the Airflow database (MySQL) [airflow.utils.db] loc=db.py:1179
2026-04-16T02:51:13.806680Z [info ] MySQL: Committing session to release metadata locks [airflow.utils.db] loc=db.py:794
2026-04-16T02:58:44.621134Z [info ] Context impl MySQLImpl. [alembic.runtime.migration] loc=migration.py:210
2026-04-16T02:58:44.621234Z [info ] Will assume non-transactional DDL. [alembic.runtime.migration] loc=migration.py:213
2026-04-16T02:58:44.687475Z [info ] Running upgrade cc92b33c6709 -> 82dbd68e6171, Add composite index (ti_id, id DESC) to task_reschedule. [alembic.runtime.migration] loc=migration.py:621
```
First, it was stuck for 7.5 minutes even before it started doing any migrations, then the actual migration (Add composite index) was stuck for 20+ minutes.
The only way to get this to finish was to scale down all pods, which then let this migration finish. Is this expected?
We are using MySQL.
Graph of processes waiting for lock. Started migrations around start of this graph and ended when scaled down pods and let migration finish.
<img width="1340" height="281" alt="Image" src="https://github.com/user-attachments/assets/337bba0b-ea1a-4c11-84e8-f2c393bc4c2e" />
### What you think should happen instead?
Helm chart should handle migrations cleanly without needing user to scale down workloads.
### Operating System
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
1.20.0 (latest released)
### Kubernetes Version
v1.30.14
### Helm Chart configuration
```
defaultAirflowTag: "3.2.0"
airflowVersion: "3.2.0"
executor: "KubernetesExecutor"
allowPodLaunching: true
# Avoid having the helm chart managing these secret which causes it to rotate on upgrades and causes issues for DAGs or web UI access
jwtSecretName: airflow-jwt-secret
fernetKeySecretName: airflow-fernet-key
apiSecretKeySecretName: airflow-api-secret-key
env:
- name: AIRFLOW__API__EXPOSE_CONFIG
value: "False"
- name: AIRFLOW__API__BASE_URL
value: "https://(redacted)"
- name: AIRFLOW__EMAIL__EMAIL_BACKEND
value: "airflow.utils.email.send_email_smtp"
- name: AIRFLOW__SMTP__SMTP_HOST
value: "redacted"
- name: AIRFLOW__SMTP__SMTP_MAIL_FROM
value: "redacted"
- name: AIRFLOW__SMTP__SMTP_STARTTLS
value: "False"
- name: AIRFLOW__WEBSERVER__SHOW_TRIGGER_FORM_IF_NO_PARAMS
value: "True"
- name: AIRFLOW__WEBSERVER__WARN_DEPLOYMENT_EXPOSURE
value: "False"
## in Airflow V2 parallelism=0 meant infinite.
## in V3, a value must be supplied
- name: AIRFLOW__CORE__PARALLELISM
value: "256"
# dags are mounted via localPath
dags:
persistence:
enabled: false
logs:
persistence:
enabled: true
#size: 5Gi
#annotations: {}
#storageClassName: "rook-cephfs-fs00"
existingClaim: airflow-logs
redis:
enabled: false
postgresql:
enabled: false
statsd:
enabled: false
extraEnv: |
- name: AIRFLOW_KUBE_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: AIRFLOW_ENV
valueFrom:
configMapKeyRef:
name: airflow.deployment
key: environment
extraEnvFrom: |
- secretRef:
name: airflow-okta-creds
- secretRef:
name: airflow-admin-user-pwd
data:
metadataSecretName: airflow-db-conn
resultBackendSecretName: airflow-db-conn
# Root-level tolerations and affinity apply to all components (scheduler, webserver, triggerer, etc.)
# including the KubernetesExecutor pod template.
# If desired to have different rules for the pod template, set workers.affinity/workers.tolerations instead.
# See: https://github.com/apache/airflow/blob/main/chart/files/pod-template-file.kubernetes-helm-yaml
tolerations:
- key: airflow
operator: Equal
value: "true"
effect: NoSchedule
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node-role.kubernetes.io/worker-airflow
operator: In
values:
- "true"
webserver:
defaultUser:
enabled: true
role: Admin
username: redacted
firstName: redacted
lastName: redacted
email: redacted
# Note: this is only applied on account creation, if account already exists details won't be updated
createUserJob:
ttlSecondsAfterFinished: 300
command: ~
args:
- "bash"
- "-c"
- |-
exec \
airflow users create \
-r "{{ .Values.webserver.defaultUser.role }}" \
-u "{{ .Values.webserver.defaultUser.username }}" \
-e "{{ .Values.webserver.defaultUser.email }}" \
-f "{{ .Values.webserver.defaultUser.firstName }}" \
-l "{{ .Values.webserver.defaultUser.lastName }}" \
-p "${AIRFLOW_ADMIN_PASSWORD}"
applyCustomEnv: true
securityContexts:
pod:
runAsUser: 50000
runAsGroup: 0
fsGroup: 50000
# changing permissions operation is slow, especially on big volumes like the logs volume
# so, only change if the root permissions do not match the expected permissions
fsGroupChangePolicy: "OnRootMismatch"
runAsNonRoot: true
triggerer:
replicas: 2
resources:
requests:
memory: "500Mi"
cpu: "1000m"
limits:
memory: "1Gi"
cpu: "2000m"
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node-role.kubernetes.io/worker-airflow
operator: In
values:
- "true"
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: component
operator: In
values:
- triggerer
topologyKey: "kubernetes.io/hostname"
extraVolumes:
- name: dags
hostPath:
path: redacted
type: Directory
extraVolumeMounts:
- name: dags
mountPath: /opt/airflow/dags
readOnly: true
subPath: dags-v3
scheduler:
replicas: 2
resources:
requests:
memory: "4Gi"
cpu: "2"
limits:
memory: "8Gi"
cpu: "4"
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node-role.kubernetes.io/worker-airflow
operator: In
values:
- "true"
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: component
operator: In
values:
- scheduler
topologyKey: "kubernetes.io/hostname"
dagProcessor:
replicas: 2
resources:
# dagProcessor has memory leak increasing around 300MB/day
requests:
memory: "4Gi"
cpu: "1"
limits:
memory: "8Gi"
cpu: "2"
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node-role.kubernetes.io/worker-airflow
operator: In
values:
- "true"
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: component
operator: In
values:
- dag-processor
topologyKey: "kubernetes.io/hostname"
extraVolumes:
- name: dags
hostPath:
path: redacted
type: Directory
extraVolumeMounts:
- name: dags
mountPath: /opt/airflow/dags
readOnly: true
subPath: dags-v3
apiServer:
replicas: 2
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "3Gi"
cpu: "2000m"
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node-role.kubernetes.io/worker-airflow
operator: In
values:
- "true"
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: component
operator: In
values:
- api-server
topologyKey: "kubernetes.io/hostname"
env:
## this should achieve the same as the community chart's `airflow.extraPipPackages`. Note: underscore is necessary, not a typo!
- name: _PIP_ADDITIONAL_REQUIREMENTS
value: "airflow-exporter==1.6.0"
service:
type: NodePort
ports:
- name: api-server
port: "{{ .Values.ports.apiServer }}"
nodePort: 30010
## Uncomment below to enable Okta
## note: does not work when port-forwarding
apiServerConfig: |
import os
from flask_appbuilder.security.manager import AUTH_OAUTH
AUTH_TYPE = AUTH_OAUTH
# registration configs
AUTH_USER_REGISTRATION = True # allow users who are not already in the FAB DB
AUTH_USER_REGISTRATION_ROLE = "Viewer" # this role will be given in addition to any AUTH_ROLES_MAPPING
# the list of providers which the user can choose from
OAUTH_PROVIDERS = [
{
"name": "okta",
"icon": "fa-circle-o",
"token_key": "access_token",
"remote_app": {
"client_id": os.environ["OKTA_CLIENT_ID"],
"client_secret": os.environ["OKTA_CLIENT_SECRET"],
"api_base_url": "redacted",
"client_kwargs": {"scope": "openid profile email groups"},
"server_metadata_url": "redacted",
"access_token_url": "redacted",
"authorize_url": "redacted",
},
},
]
# a mapping from the values of `userinfo["role_keys"]` to a list of FAB roles
AUTH_ROLES_MAPPING = {
"Viewer": ["Viewer"],
"redacted": ["Admin"],
}
# if we should replace ALL the user's roles each login, or only on registration
AUTH_ROLES_SYNC_AT_LOGIN = True
# force users to re-auth after 30min of inactivity (to keep roles in sync)
PERMANENT_SESSION_LIFETIME = 1800
## this is a bit of a confusing section. workers section is technically for deploying CeleryExecutors which we don't use
## but extraVolumes/extraVolumeMounts of this section is used to append to pod manifest of launched pods by airflow
## As of chart 1.20.0, workers.resources is deprecated in favor of workers.kubernetes.resources
workers:
extraVolumes:
- name: dags
hostPath:
path: redacted
type: Directory
extraVolumeMounts:
- name: dags
mountPath: /opt/airflow/dags
readOnly: true
subPath: dags-v3
kubernetes:
resources:
requests:
memory: "500Mi"
cpu: "200m"
limits:
memory: "1Gi"
cpu: "2000m"
```
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade from 3.1.7 to 3.2.0 stuck on Database migrations (MySQL) [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #65350: Upgrade from 3.1.7 to 3.2.0 stuck on Database migrations (MySQL)
URL: https://github.com/apache/airflow/issues/65350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade from 3.1.7 to 3.2.0 stuck on Database migrations (MySQL) [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #65350:
URL: https://github.com/apache/airflow/issues/65350#issuecomment-4986689661
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade from 3.1.7 to 3.2.0 stuck on Database migrations (MySQL) [airflow]
Posted by "florian-meyrueis-al (via GitHub)" <gi...@apache.org>.
florian-meyrueis-al commented on issue #65350:
URL: https://github.com/apache/airflow/issues/65350#issuecomment-4313184960
I don't know if this usefull, but i face a similar issue upgrading from 3.1.7 to 3.2.1, with upgrade of fab provider 3.6.1
We have a postgres db instead of mysql, but i was able to see, the migrate db job stuck on this message : Fix fab db inconsistencies (which looks to be a revision in fab provider 3.5.0).
I could see in the database instance that a session was locking all the transaction of the migrate db job.
this session was doing some query on fab tables like ab_roles (i suspect it comes from the createUserJob)
I've killed this session and the migrate job was instantly running and completing well.
So it doesn't really look like excatly this issue, but there is one common point: somewhere in the database transaction, locks seems to be handled improperly.
I could try to provide locking query if you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade from 3.1.7 to 3.2.0 stuck on Database migrations (MySQL) [airflow]
Posted by "paolofacchinetti (via GitHub)" <gi...@apache.org>.
paolofacchinetti commented on issue #65350:
URL: https://github.com/apache/airflow/issues/65350#issuecomment-4313904469
as @florian-meyrueis-al said, can also attest that my migration was stuck at line:
```
Running upgrade 6709f7a774b9 -> 02ca36b0235b, Fix fab db inconsistencies. [alembic.runtime.migration] loc=migration.py:621
```
Querying pg_stat_activity + pg_blocking_pids() showed the migration blocked by an unrelated session that had been idle in transaction for ~28 minutes, holding locks from a FAB permission lookup (SELECT joining ab_user, ab_role, ab_permission, ab_view_menu). pg_terminate_backend() on the blocking PID unblocked the migration and it completed immediately.
```
BLOCKED SESSION (migration)
─────────────────────────────────────────────────────────────────────────
pid : 3089
user : postgres
state : active
wait_event_type : Lock
wait_event : relation
query : ALTER TABLE ab_permission_view_role
DROP CONSTRAINT ab_permission_view_role_role_id_fkey
BLOCKING SESSION
─────────────────────────────────────────────────────────────────────────
pid : 32659
user : postgres
state : idle in transaction
xact_age : 00:28:43
query : SELECT ab_user_1.id AS ab_user_1_id,
ab_role.id AS ab_role_id,
ab_role.name AS ab_role_name,
ab_permission_1.id AS ab_permission_1_id,
ab_permission_1.name AS ab_permission_1_name,
ab_view_menu_1.id AS ...
[truncated — FAB permission lookup joining
ab_user / ab_role / ab_permission / ab_view_menu]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade from 3.1.7 to 3.2.0 stuck on Database migrations (MySQL) [airflow]
Posted by "bastienmenis (via GitHub)" <gi...@apache.org>.
bastienmenis commented on issue #65350:
URL: https://github.com/apache/airflow/issues/65350#issuecomment-4444154196
@paolofacchinetti Thanks, we had exactly the same issue and using `pg_stat_activity` + `pg_blocking_pids` + `pg_terminate_backend` unblocked the migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade from 3.1.7 to 3.2.0 stuck on Database migrations (MySQL) [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #65350:
URL: https://github.com/apache/airflow/issues/65350#issuecomment-4266362153
Looks like stuck on 82dbd68e6171 which is 3.1.8 migration, I tried and I was not able to replicate this. How much data you have in task_reschedule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade from 3.1.7 to 3.2.0 stuck on Database migrations (MySQL) [airflow]
Posted by "davidjfrickert (via GitHub)" <gi...@apache.org>.
davidjfrickert commented on issue #65350:
URL: https://github.com/apache/airflow/issues/65350#issuecomment-4277152196
> Looks like stuck on 82dbd68e6171 which is 3.1.8 migration, I tried and I was not able to replicate this. How much data you have in task_reschedule?
It's a fresh instance with no DAGs yet. I think the issue is not the amount of data but the migration can't even start due to table lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Upgrade from 3.1.7 to 3.2.0 stuck on Database migrations (MySQL) [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #65350:
URL: https://github.com/apache/airflow/issues/65350#issuecomment-4898933259
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org