You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "IvanovOleg (via GitHub)" <gi...@apache.org> on 2024/04/11 15:35:52 UTC
[I] Components stuck waiting on migrations [airflow]
IvanovOleg opened a new issue, #38937:
URL: https://github.com/apache/airflow/issues/38937
### Official Helm Chart version
1.13.1 (latest released)
### Apache Airflow version
2.8.3
### Kubernetes Version
Openshift 4.14.12
### Helm Chart configuration
```yaml
scheduler:
securityContext:
fsGroup: 1000720000
runAsGroup: 1000720000
runAsUser: 1000720000
waitForMigrations:
securityContexts:
container:
runAsGroup: 0
workers:
securityContext:
fsGroup: 1000720000
runAsGroup: 1000720000
runAsUser: 1000720000
waitForMigrations:
securityContexts:
container:
runAsGroup: 0
triggerer:
securityContext:
fsGroup: 1000720000
runAsGroup: 1000720000
runAsUser: 1000720000
waitForMigrations:
securityContexts:
container:
runAsGroup: 0
webserver:
securityContext:
fsGroup: 1000720000
runAsUser: 1000720000
statsd:
securityContext:
runAsUser: 1000720000
redis:
securityContext:
runAsUser: 1000720000
runAsGroup: 1000720000
postgresql:
primary:
podSecurityContext:
enabled: false
containerSecurityContext:
enabled: false
readReplicas:
podSecurityContext:
enabled: false
containerSecurityContext:
enabled: false
```
### Docker Image customizations
_No response_
### What happened
Webserver, worker, triggerer, shceduler stuck with following init container issue:
```
[2024-04-11T15:10:52.326+0000] {db.py:799} INFO - Waiting for migrations... 0 second(s)
[2024-04-11T15:10:53.329+0000] {db.py:799} INFO - Waiting for migrations... 1 second(s)
[2024-04-11T15:10:54.330+0000] {db.py:799} INFO - Waiting for migrations... 2 second(s)
[2024-04-11T15:10:55.334+0000] {db.py:799} INFO - Waiting for migrations... 3 second(s)
[2024-04-11T15:10:56.335+0000] {db.py:799} INFO - Waiting for migrations... 4 second(s)
[2024-04-11T15:10:57.337+0000] {db.py:799} INFO - Waiting for migrations... 5 second(s)
[2024-04-11T15:10:58.339+0000] {db.py:799} INFO - Waiting for migrations... 6 second(s)
[2024-04-11T15:10:59.341+0000] {db.py:799} INFO - Waiting for migrations... 7 second(s)
[2024-04-11T15:11:00.342+0000] {db.py:799} INFO - Waiting for migrations... 8 second(s)
[2024-04-11T15:11:01.343+0000] {db.py:799} INFO - Waiting for migrations... 9 second(s)
[2024-04-11T15:11:02.345+0000] {db.py:799} INFO - Waiting for migrations... 10 second(s)
[2024-04-11T15:11:03.347+0000] {db.py:799} INFO - Waiting for migrations... 11 second(s)
[2024-04-11T15:11:04.349+0000] {db.py:799} INFO - Waiting for migrations... 12 second(s)
[2024-04-11T15:11:05.350+0000] {db.py:799} INFO - Waiting for migrations... 13 second(s)
[2024-04-11T15:11:06.353+0000] {db.py:799} INFO - Waiting for migrations... 14 second(s)
[2024-04-11T15:11:07.355+0000] {db.py:799} INFO - Waiting for migrations... 15 second(s)
[2024-04-11T15:11:08.356+0000] {db.py:799} INFO - Waiting for migrations... 16 second(s)
[2024-04-11T15:11:09.358+0000] {db.py:799} INFO - Waiting for migrations... 17 second(s)
[2024-04-11T15:11:10.359+0000] {db.py:799} INFO - Waiting for migrations... 18 second(s)
[2024-04-11T15:11:11.361+0000] {db.py:799} INFO - Waiting for migrations... 19 second(s)
[2024-04-11T15:11:12.363+0000] {db.py:799} INFO - Waiting for migrations... 20 second(s)
[2024-04-11T15:11:13.365+0000] {db.py:799} INFO - Waiting for migrations... 21 second(s)
[2024-04-11T15:11:14.366+0000] {db.py:799} INFO - Waiting for migrations... 22 second(s)
[2024-04-11T15:11:15.368+0000] {db.py:799} INFO - Waiting for migrations... 23 second(s)
[2024-04-11T15:11:16.369+0000] {db.py:799} INFO - Waiting for migrations... 24 second(s)
[2024-04-11T15:11:17.370+0000] {db.py:799} INFO - Waiting for migrations... 25 second(s)
[2024-04-11T15:11:18.372+0000] {db.py:799} INFO - Waiting for migrations... 26 second(s)
[2024-04-11T15:11:19.373+0000] {db.py:799} INFO - Waiting for migrations... 27 second(s)
[2024-04-11T15:11:20.374+0000] {db.py:799} INFO - Waiting for migrations... 28 second(s)
[2024-04-11T15:11:21.375+0000] {db.py:799} INFO - Waiting for migrations... 29 second(s)
[2024-04-11T15:11:22.377+0000] {db.py:799} INFO - Waiting for migrations... 30 second(s)
[2024-04-11T15:11:23.378+0000] {db.py:799} INFO - Waiting for migrations... 31 second(s)
[2024-04-11T15:11:24.380+0000] {db.py:799} INFO - Waiting for migrations... 32 second(s)
[2024-04-11T15:11:25.382+0000] {db.py:799} INFO - Waiting for migrations... 33 second(s)
[2024-04-11T15:11:26.383+0000] {db.py:799} INFO - Waiting for migrations... 34 second(s)
[2024-04-11T15:11:27.384+0000] {db.py:799} INFO - Waiting for migrations... 35 second(s)
[2024-04-11T15:11:28.385+0000] {db.py:799} INFO - Waiting for migrations... 36 second(s)
[2024-04-11T15:11:29.386+0000] {db.py:799} INFO - Waiting for migrations... 37 second(s)
[2024-04-11T15:11:30.400+0000] {db.py:799} INFO - Waiting for migrations... 38 second(s)
[2024-04-11T15:11:31.422+0000] {db.py:799} INFO - Waiting for migrations... 39 second(s)
[2024-04-11T15:11:32.424+0000] {db.py:799} INFO - Waiting for migrations... 40 second(s)
[2024-04-11T15:11:33.426+0000] {db.py:799} INFO - Waiting for migrations... 41 second(s)
[2024-04-11T15:11:34.427+0000] {db.py:799} INFO - Waiting for migrations... 42 second(s)
[2024-04-11T15:11:35.429+0000] {db.py:799} INFO - Waiting for migrations... 43 second(s)
[2024-04-11T15:11:36.430+0000] {db.py:799} INFO - Waiting for migrations... 44 second(s)
[2024-04-11T15:11:37.431+0000] {db.py:799} INFO - Waiting for migrations... 45 second(s)
[2024-04-11T15:11:38.434+0000] {db.py:799} INFO - Waiting for migrations... 46 second(s)
[2024-04-11T15:11:39.436+0000] {db.py:799} INFO - Waiting for migrations... 47 second(s)
[2024-04-11T15:11:40.438+0000] {db.py:799} INFO - Waiting for migrations... 48 second(s)
[2024-04-11T15:11:41.439+0000] {db.py:799} INFO - Waiting for migrations... 49 second(s)
[2024-04-11T15:11:42.441+0000] {db.py:799} INFO - Waiting for migrations... 50 second(s)
[2024-04-11T15:11:43.443+0000] {db.py:799} INFO - Waiting for migrations... 51 second(s)
[2024-04-11T15:11:44.444+0000] {db.py:799} INFO - Waiting for migrations... 52 second(s)
[2024-04-11T15:11:45.446+0000] {db.py:799} INFO - Waiting for migrations... 53 second(s)
[2024-04-11T15:11:46.448+0000] {db.py:799} INFO - Waiting for migrations... 54 second(s)
[2024-04-11T15:11:47.449+0000] {db.py:799} INFO - Waiting for migrations... 55 second(s)
[2024-04-11T15:11:48.471+0000] {db.py:799} INFO - Waiting for migrations... 56 second(s)
[2024-04-11T15:11:49.473+0000] {db.py:799} INFO - Waiting for migrations... 57 second(s)
[2024-04-11T15:11:50.475+0000] {db.py:799} INFO - Waiting for migrations... 58 second(s)
[2024-04-11T15:11:51.476+0000] {db.py:799} INFO - Waiting for migrations... 59 second(s)
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 57, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_config.py", line 49, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 190, in check_migrations
db.check_migrations(timeout=args.migration_wait_timeout)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 800, in check_migrations
raise TimeoutError(
TimeoutError: There are still unapplied migrations after 60 seconds. MigrationHead(s) in DB: set() | Migration Head(s) in Source Code: {'88344c1d9134'}
```
Waiting doesn't help. In postgres I see following errors:
```
postgresql 15:08:12.02 INFO ==>
postgresql 15:08:12.03 INFO ==> Welcome to the Bitnami postgresql container
postgresql 15:08:12.03 INFO ==> Subscribe to project updates by watching https://github.com/bitnami/containers
postgresql 15:08:12.04 INFO ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgresql 15:08:12.04 INFO ==>
postgresql 15:08:12.05 INFO ==> ** Starting PostgreSQL setup **
postgresql 15:08:12.07 INFO ==> Validating settings in POSTGRESQL_* env vars..
postgresql 15:08:12.08 INFO ==> Loading custom pre-init scripts...
postgresql 15:08:12.08 INFO ==> Initializing PostgreSQL database...
postgresql 15:08:12.10 INFO ==> pg_hba.conf file not detected. Generating it...
postgresql 15:08:12.10 INFO ==> Generating local authentication configuration
postgresql 15:08:13.32 INFO ==> Starting PostgreSQL in background...
postgresql 15:08:13.44 INFO ==> Changing password of postgres
postgresql 15:08:13.47 INFO ==> Configuring replication parameters
postgresql 15:08:13.49 INFO ==> Configuring synchronous_replication
postgresql 15:08:13.49 INFO ==> Configuring fsync
postgresql 15:08:13.53 INFO ==> Stopping PostgreSQL...
waiting for server to shut down.... done
server stopped
postgresql 15:08:13.63 INFO ==> Loading custom scripts...
postgresql 15:08:13.64 INFO ==> Enabling remote connections
postgresql 15:08:13.64 INFO ==> ** PostgreSQL setup finished! **
postgresql 15:08:13.65 INFO ==> ** Starting PostgreSQL **
2024-04-11 15:08:13.675 GMT [1] LOG: pgaudit extension initialized
2024-04-11 15:08:13.686 GMT [1] LOG: starting PostgreSQL 16.1 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2024-04-11 15:08:13.686 GMT [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2024-04-11 15:08:13.686 GMT [1] LOG: listening on IPv6 address "::", port 5432
2024-04-11 15:08:13.694 GMT [1] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2024-04-11 15:08:13.707 GMT [123] LOG: database system was shut down at 2024-04-11 15:08:13 GMT
2024-04-11 15:08:13.722 GMT [1] LOG: database system is ready to accept connections
2024-04-11 15:08:21.948 GMT [134] ERROR: relation "log" does not exist at character 13
2024-04-11 15:08:21.948 GMT [134] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:08:14.855577+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-webserver-76c8cb49f-ms2gr", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:08:22.713 GMT [135] ERROR: relation "log" does not exist at character 13
2024-04-11 15:08:22.713 GMT [135] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:08:15.605933+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-scheduler-58c4b49fc-r98hh", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:08:22.905 GMT [136] ERROR: relation "log" does not exist at character 13
2024-04-11 15:08:22.905 GMT [136] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:08:15.815946+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-worker-0", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:08:23.033 GMT [137] ERROR: relation "log" does not exist at character 13
2024-04-11 15:08:23.033 GMT [137] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:08:15.928602+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-triggerer-0", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:09:30.672 GMT [229] ERROR: relation "log" does not exist at character 13
2024-04-11 15:09:30.672 GMT [229] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:09:30.657757+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-scheduler-58c4b49fc-r98hh", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:09:30.684 GMT [230] ERROR: relation "log" does not exist at character 13
2024-04-11 15:09:30.684 GMT [230] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:09:30.657674+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-webserver-76c8cb49f-ms2gr", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:09:31.501 GMT [231] ERROR: relation "log" does not exist at character 13
2024-04-11 15:09:31.501 GMT [231] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:09:31.492209+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-worker-0", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:09:31.543 GMT [232] ERROR: relation "log" does not exist at character 13
2024-04-11 15:09:31.543 GMT [232] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:09:31.533532+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-triggerer-0", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:10:49.517 GMT [338] ERROR: relation "log" does not exist at character 13
2024-04-11 15:10:49.517 GMT [338] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:10:49.507573+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-scheduler-58c4b49fc-r98hh", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:10:50.612 GMT [353] ERROR: relation "log" does not exist at character 13
2024-04-11 15:10:50.612 GMT [353] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:10:50.602994+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-worker-0", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:10:51.768 GMT [355] ERROR: relation "log" does not exist at character 13
2024-04-11 15:10:51.768 GMT [355] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:10:51.760706+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-webserver-76c8cb49f-ms2gr", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:10:52.789 GMT [357] ERROR: relation "log" does not exist at character 13
2024-04-11 15:10:52.789 GMT [357] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:10:52.779490+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-triggerer-0", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:12:22.476 GMT [489] ERROR: relation "log" does not exist at character 13
2024-04-11 15:12:22.476 GMT [489] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:12:22.462657+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-scheduler-58c4b49fc-r98hh", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:12:24.925 GMT [491] ERROR: relation "log" does not exist at character 13
2024-04-11 15:12:24.925 GMT [491] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:12:24.915888+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-webserver-76c8cb49f-ms2gr", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:12:24.978 GMT [492] ERROR: relation "log" does not exist at character 13
2024-04-11 15:12:24.978 GMT [492] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:12:24.970969+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-triggerer-0", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:12:25.957 GMT [493] ERROR: relation "log" does not exist at character 13
2024-04-11 15:12:25.957 GMT [493] STATEMENT: INSERT INTO log (dttm, event, owner, extra) VALUES ('2024-04-11T15:12:25.947215+00:00'::timestamptz, 'cli_check', 'default', '{"host_name": "airflow-worker-0", "full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:13:13.806 GMT [121] LOG: checkpoint starting: time
2024-04-11 15:13:18.052 GMT [121] LOG: checkpoint complete: wrote 45 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.210 s, sync=0.012 s, total=4.246 s; sync files=12, longest=0.003 s, average=0.001 s; distance=261 kB, estimate=261 kB; lsn=0/1528A00, redo lsn=0/15289C8
```
### What you think should happen instead
Migration completed successfully
### How to reproduce
deploy helm chart
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Components stuck waiting on migrations [airflow]
Posted by "yriveiro (via GitHub)" <gi...@apache.org>.
yriveiro commented on issue #38937:
URL: https://github.com/apache/airflow/issues/38937#issuecomment-2435685294
I think was this one https://airflow.apache.org/docs/helm-chart/stable/index.html#installing-the-chart-with-argo-cd-flux-rancher-or-terraform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Components stuck waiting on migrations [airflow]
Posted by "simonherbert (via GitHub)" <gi...@apache.org>.
simonherbert commented on issue #38937:
URL: https://github.com/apache/airflow/issues/38937#issuecomment-2254450725
> Missed a documentation part related to the ArgoCD deployment
What documentation part did you miss?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Components stuck waiting on migrations [airflow]
Posted by "IvanovOleg (via GitHub)" <gi...@apache.org>.
IvanovOleg closed issue #38937: Components stuck waiting on migrations
URL: https://github.com/apache/airflow/issues/38937
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Components stuck waiting on migrations [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #38937:
URL: https://github.com/apache/airflow/issues/38937#issuecomment-2049980172
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Components stuck waiting on migrations [airflow]
Posted by "IvanovOleg (via GitHub)" <gi...@apache.org>.
IvanovOleg commented on issue #38937:
URL: https://github.com/apache/airflow/issues/38937#issuecomment-2057526719
Missed a documentation part related to the ArgoCD deployment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org