You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/06 15:58:22 UTC
[GitHub] [airflow] gardnerdev opened a new issue #10201: Worker unexpected CrashLoopBack
gardnerdev opened a new issue #10201:
URL: https://github.com/apache/airflow/issues/10201
During usage of airflow from helm package I encountered this error
> │ wait-for-airflow-migrations DB_BACKEND=postgresql │
│ wait-for-airflow-migrations DB_HOST=fi-airflow-postgresql.fi.svc.cluster.local │
│ wait-for-airflow-migrations DB_PORT=5432 │
│ wait-for-airflow-migrations [2020-08-06 15:25:17,173] {migration.py:155} INFO - Context impl PostgresqlImpl. │
│ wait-for-airflow-migrations [2020-08-06 15:25:17,173] {migration.py:162} INFO - Will assume transactional DDL. │
│ wait-for-airflow-migrations stream closed │
│ worker DB_BACKEND=postgresql │
│ worker DB_HOST=fi-airflow-postgresql.fi.svc.cluster.local │
│ worker DB_PORT=5432 │
│ worker DB_BACKEND=postgresql │
│ worker DB_HOST=fi-airflow-postgresql.fi.svc.cluster.local │
│ worker DB_PORT=5432 │
│ worker [2020-08-06 15:26:51,326: INFO/MainProcess] Connected to redis://:**@fi-airflow-redis:6379/0 │
│ worker [2020-08-06 15:26:51,341: INFO/MainProcess] mingle: searching for neighbors │
│ worker [2020-08-06 15:26:52,434: INFO/MainProcess] mingle: all alone │
│ worker [2020-08-06 15:26:52,454: INFO/MainProcess] celery@fi-airflow-worker-596bd66d58-zqwlj ready. │
│ worker [2020-08-06 15:26:52,454: ERROR/MainProcess] Process 'ForkPoolWorker-14' pid:51 exited with 'signal 9 (SIGKILL)' │
│ worker [2020-08-06 15:26:52,465: ERROR/MainProcess] Process 'ForkPoolWorker-13' pid:50 exited with 'signal 9 (SIGKILL)' │
│ worker [2020-08-06 15:26:52,475: ERROR/MainProcess] Process 'ForkPoolWorker-12' pid:49 exited with 'signal 9 (SIGKILL)' │
│ worker [2020-08-06 15:26:52,486: ERROR/MainProcess] Process 'ForkPoolWorker-11' pid:48 exited with 'signal 9 (SIGKILL)' │
│ worker [2020-08-06 15:26:52,496: ERROR/MainProcess] Process 'ForkPoolWorker-10' pid:45 exited with 'signal 9 (SIGKILL)' │
│ worker [2020-08-06 15:26:52,506: ERROR/MainProcess] Process 'ForkPoolWorker-9' pid:44 exited with 'signal 9 (SIGKILL)' │
│ worker [2020-08-06 15:26:52,517: ERROR/MainProcess] Process 'ForkPoolWorker-8' pid:32 exited with 'signal 9 (SIGKILL)' │
│ worker [2020-08-06 15:26:52,527: ERROR/MainProcess] Process 'ForkPoolWorker-6' pid:30 exited with 'signal 9 (SIGKILL)' │
│ worker [2020-08-06 15:26:52,538: ERROR/MainProcess] Process 'ForkPoolWorker-4' pid:28 exited with 'signal 9 (SIGKILL)' │
│ worker [2020-08-06 15:26:52,641: CRITICAL/MainProcess] Unrecoverable error: WorkerLostError('Could not start worker processes') │
│ worker Traceback (most recent call last): │
│ worker File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start │
│ worker self.blueprint.start(self) │
│ worker File "/home/airflow/.local/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start │
│ worker step.start(parent) │
│ worker File "/home/airflow/.local/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start │
│ worker return self.obj.start() │
│ worker File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start │
│ worker blueprint.start(self) │
│ worker File "/home/airflow/.local/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start │
│ worker step.start(parent) │
│ worker File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 599, in start │
│ worker c.loop(*c.loop_args()) │
│ worker File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/loops.py", line 59, in asynloop │
│ worker raise WorkerLostError('Could not start worker processes') │
│ worker billiard.exceptions.WorkerLostError: Could not start worker processes
**Apache Airflow version**:
1.10.10
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
1.18.3
**Environment**:
Kubernetes Cluster
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**Anything else we need to know**:
I changed redis from chart to secured one from bitnami repository:
```
repository: bitnami/redis
tag: 6.0.6
```
<!--
How often does this problem occur? Once? Every time etc?
Yes, every time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gardnerdev commented on issue #10201: Worker unexpected CrashLoopBack
Posted by GitBox <gi...@apache.org>.
gardnerdev commented on issue #10201:
URL: https://github.com/apache/airflow/issues/10201#issuecomment-670028221
Resolved by provisioning more resources for workers
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gardnerdev closed issue #10201: Worker unexpected CrashLoopBack
Posted by GitBox <gi...@apache.org>.
gardnerdev closed issue #10201:
URL: https://github.com/apache/airflow/issues/10201
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gardnerdev closed issue #10201: Worker unexpected CrashLoopBack
Posted by GitBox <gi...@apache.org>.
gardnerdev closed issue #10201:
URL: https://github.com/apache/airflow/issues/10201
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org