You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Yuri Bendana (JIRA)" <ji...@apache.org> on 2016/06/08 01:49:20 UTC
[jira] [Commented] (AIRFLOW-166) Webserver times out using systemd
script
[ https://issues.apache.org/jira/browse/AIRFLOW-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319846#comment-15319846 ]
Yuri Bendana commented on AIRFLOW-166:
--------------------------------------
The airflow-webserver.service does pass in the --pid flag
{{ExecStart=/bin/airflow webserver --pid /run/airflow/webserver.pid}}
and I verified that the location is writable by the airflow user. When I run {{airflow webserver}} from the command line it does create the pid file. I tried passing in the --log, --stdout, and --stderr but it didn't have any effect. I then took a look at cli.py and noticed that they aren't used in the call to gunicorn. gunicorn doesn't have --stdout and --stderr flags but it does have an --error-logfile flag. I tried adding it to the run_args in cli.py but I then get an 'ImportError: Import by filename is not supported.' However, I'm able to successfully run it from the command line although I had to remove the parenthesis after cached_app:
{{gunicorn -w 4 -k sync -t 120 -b 0.0.0.0:8080 -n airflow-webserver -p /opt/airflow/airflow-webserver.pid --error-logfile /opt/airflow/airflow-webserver.log airflow.www.app:cached_app}}
Any reason why the gunicorn error logfile is not set in cli.py?
> Webserver times out using systemd script
> ----------------------------------------
>
> Key: AIRFLOW-166
> URL: https://issues.apache.org/jira/browse/AIRFLOW-166
> Project: Apache Airflow
> Issue Type: Bug
> Affects Versions: Airflow 1.7.1
> Environment: CentOS 7
> Reporter: Yuri Bendana
>
> I just upgraded to 1.7.1.2 from 1.6 and I'm having a problem starting the webserver using the systemd script. This used to work fine. The issue is that it starts and then just hangs, no error is reported and it finally times out after about a minute. I tried starting it from the command line and it works fine without timing out. I also ran it in daemon mode with -D and again it seems to be fine. Any thoughts on how to debug this?
> Here's the log output:
> {code}
> May 23 16:27:50 ybendana-linux systemd: Starting Airflow webserver daemon...
> May 23 16:27:51 ybendana-linux airflow: [2016-05-23 16:27:51,444] {__init__.py:36} INFO - Using executor LocalExecutor
> May 23 16:27:53 ybendana-linux airflow: ____________ _____________
> May 23 16:27:53 ybendana-linux airflow: ____ |__( )_________ __/__ /________ __
> May 23 16:27:53 ybendana-linux airflow: ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
> May 23 16:27:53 ybendana-linux airflow: ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
> May 23 16:27:53 ybendana-linux airflow: _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
> May 23 16:27:53 ybendana-linux airflow: [2016-05-23 16:27:53,446] {models.py:154} INFO - Filling up the DagBag from /opt/airflow/dags
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [15960] [INFO] Starting gunicorn 19.3.0
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [15960] [INFO] Listening at: http://0.0.0.0:8080 (15960)
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [15960] [INFO] Using worker: sync
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [16067] [INFO] Booting worker with pid: 16067
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [16069] [INFO] Booting worker with pid: 16069
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [16070] [INFO] Booting worker with pid: 16070
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55 +0000] [16071] [INFO] Booting worker with pid: 16071
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55,876] {__init__.py:36} INFO - Using executor LocalExecutor
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55,950] {__init__.py:36} INFO - Using executor LocalExecutor
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55,972] {__init__.py:36} INFO - Using executor LocalExecutor
> May 23 16:27:55 ybendana-linux airflow: [2016-05-23 16:27:55,997] {__init__.py:36} INFO - Using executor LocalExecutor
> May 23 16:27:57 ybendana-linux airflow: [2016-05-23 16:27:57,885] {models.py:154} INFO - Filling up the DagBag from /opt/airflow/dags
> May 23 16:27:57 ybendana-linux airflow: [2016-05-23 16:27:57,951] {models.py:154} INFO - Filling up the DagBag from /opt/airflow/dags
> May 23 16:27:57 ybendana-linux airflow: [2016-05-23 16:27:57,983] {models.py:154} INFO - Filling up the DagBag from /opt/airflow/dags
> May 23 16:27:58 ybendana-linux airflow: [2016-05-23 16:27:58,014] {models.py:154} INFO - Filling up the DagBag from /opt/airflow/dags
> May 23 16:29:20 ybendana-linux systemd: airflow-webserver.service start operation timed out. Terminating.
> May 23 16:29:20 ybendana-linux airflow: [2016-05-23 16:29:20 +0000] [16070] [INFO] Worker exiting (pid: 16070)
> May 23 16:29:20 ybendana-linux airflow: [2016-05-23 16:29:20 +0000] [15960] [INFO] Handling signal: term
> May 23 16:29:20 ybendana-linux airflow: [2016-05-23 16:29:20 +0000] [16071] [INFO] Worker exiting (pid: 16071)
> May 23 16:29:20 ybendana-linux airflow: [2016-05-23 16:29:20 +0000] [16069] [INFO] Worker exiting (pid: 16069)
> May 23 16:29:20 ybendana-linux airflow: [2016-05-23 16:29:20 +0000] [16067] [INFO] Worker exiting (pid: 16067)
> May 23 16:29:21 ybendana-linux airflow: [2016-05-23 16:29:21 +0000] [15960] [INFO] Shutting down: Master
> May 23 16:29:21 ybendana-linux systemd: Failed to start Airflow webserver daemon.
> May 23 16:29:21 ybendana-linux systemd: Unit airflow-webserver.service entered failed state.
> May 23 16:29:21 ybendana-linux systemd: airflow-webserver.service failed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)