You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/23 06:18:56 UTC
[GitHub] [airflow] nevinbaiju opened a new issue #17185: Unable to start airflow webserver in the background in docker container.
nevinbaiju opened a new issue #17185:
URL: https://github.com/apache/airflow/issues/17185
Apache Airflow version: 2.1.2
Environment: continuumio/anaconda3 docker image
- OS (e.g. from /etc/os-release): Debian 10 Buster
- Kernel: Linux 242e786962c8 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64 GNU/Linux
- Install tools: Docker
- Others: Host OS is windows 10
What happened:
I have installed airflow inside the docker container. And to run the webserver, I used the command
`airflow webserver -D`
to run in the background, however, the application is still running in the foreground and I have to open another terminal to run the airflow scheduler. The same issue is faced when trying to run the airflow scheduler in the background.
What you expected to happen: To start the airflow web server in the background.
How to reproduce it:
1. Use the Dockerfile attached to run the image.
2. Start a terminal in the container.
3. Run `airflow webserver -D`
The Dockerfile:
`
FROM continuumio/anaconda3
WORKDIR /airflow
ENV AIRFLOW_HOME /airflow
ENV AIRFLOW_VERSION 2.1.2
ENV PYTHON_VERSION 3.8
ENV CONSTRAINT_URL "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
RUN apt-get update -y
RUN apt-get install -y --no-install-recommends \
freetds-bin \
ldap-utils \
libffi-dev \
libsasl2-2 \
libsasl2-modules \
libssl1.1 \
locales \
lsb-release \
sasl2-bin \
sqlite3 \
unixodbc
RUN pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
RUN airflow db init
RUN airflow users create \
--username admin \
--firstname admin \
--lastname admin \
--password admin \
--role Admin \
--email admin@admin.org
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17185:
URL: https://github.com/apache/airflow/issues/17185#issuecomment-885491972
There is something wrong with your conda base installation. I could not reproduced in using the official image so I used yours. And airflow is simply not working in your image (likely the master process locked itself out when exiting) - you have errors in your ~/airlfow/webserver-err.log :
```
File "/opt/conda/lib/python3.8/site-packages/psutil/_common.py", line 447, in wrapper
ret = self._cache[fun]
AttributeError: _cache
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/airflow", line 8, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 480, in webserver
monitor_gunicorn(gunicorn_master_proc.pid)
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 445, in monitor_gunicorn
GunicornMonitor(
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 212, in start
self._check_workers()
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 227, in _check_workers
num_ready_workers_running = self._get_num_ready_workers_running()
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 131, in _get_num_ready_workers_running
workers = psutil.Process(self.gunicorn_master_proc.pid).children()
File "/opt/conda/lib/python3.8/site-packages/psutil/__init__.py", line 272, in wrapper
return fun(self, *args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/psutil/__init__.py", line 905, in children
child = Process(pid)
File "/opt/conda/lib/python3.8/site-packages/psutil/__init__.py", line 326, in __init__
self._init(pid)
File "/opt/conda/lib/python3.8/site-packages/psutil/__init__.py", line 354, in _init
self.create_time()
File "/opt/conda/lib/python3.8/site-packages/psutil/__init__.py", line 710, in create_time
self._create_time = self._proc.create_time()
File "/opt/conda/lib/python3.8/site-packages/psutil/_pslinux.py", line 1576, in wrapper
return fun(self, *args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/psutil/_pslinux.py", line 1788, in create_time
ctime = float(self._parse_stat_file()['create_time'])
File "/opt/conda/lib/python3.8/site-packages/psutil/_pslinux.py", line 1576, in wrapper
return fun(self, *args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/psutil/_common.py", line 450, in wrapper
return fun(self)
File "/opt/conda/lib/python3.8/site-packages/psutil/_pslinux.py", line 1619, in _parse_stat_file
data = f.read()
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 435, in kill_proc
if gunicorn_master_proc.poll() is not None:
AttributeError: 'Process' object has no attribute 'poll'
```
You can try to diagnose your image (likely it is based on some old/unsupported base images. However it's definitly somethin you have to investigate on your own.
Instead I heartily recommend to you the official image of Airflow image - it is based on LTS buster slim image (And we always use most recent images with all security patches. There are various ways you can customize and extend image and we have comprehensive documentation: https://airflow.apache.org/docs/docker-stack/index.html and we will be able to help you if you have problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17185:
URL: https://github.com/apache/airflow/issues/17185#issuecomment-885491972
There is something wrong with your conda base installation. I could not reproduced in using the official image so I used yours. And airflow is simply not working in your image (likely the master process locked itself out when exiting) - you have errors in your ~/airlfow/airflow-webserver.err:
```
File "/opt/conda/lib/python3.8/site-packages/psutil/_common.py", line 447, in wrapper
ret = self._cache[fun]
AttributeError: _cache
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/airflow", line 8, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 480, in webserver
monitor_gunicorn(gunicorn_master_proc.pid)
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 445, in monitor_gunicorn
GunicornMonitor(
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 212, in start
self._check_workers()
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 227, in _check_workers
num_ready_workers_running = self._get_num_ready_workers_running()
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 131, in _get_num_ready_workers_running
workers = psutil.Process(self.gunicorn_master_proc.pid).children()
File "/opt/conda/lib/python3.8/site-packages/psutil/__init__.py", line 272, in wrapper
return fun(self, *args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/psutil/__init__.py", line 905, in children
child = Process(pid)
File "/opt/conda/lib/python3.8/site-packages/psutil/__init__.py", line 326, in __init__
self._init(pid)
File "/opt/conda/lib/python3.8/site-packages/psutil/__init__.py", line 354, in _init
self.create_time()
File "/opt/conda/lib/python3.8/site-packages/psutil/__init__.py", line 710, in create_time
self._create_time = self._proc.create_time()
File "/opt/conda/lib/python3.8/site-packages/psutil/_pslinux.py", line 1576, in wrapper
return fun(self, *args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/psutil/_pslinux.py", line 1788, in create_time
ctime = float(self._parse_stat_file()['create_time'])
File "/opt/conda/lib/python3.8/site-packages/psutil/_pslinux.py", line 1576, in wrapper
return fun(self, *args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/psutil/_common.py", line 450, in wrapper
return fun(self)
File "/opt/conda/lib/python3.8/site-packages/psutil/_pslinux.py", line 1619, in _parse_stat_file
data = f.read()
File "/opt/conda/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 435, in kill_proc
if gunicorn_master_proc.poll() is not None:
AttributeError: 'Process' object has no attribute 'poll'
```
You can try to diagnose your image (likely it is based on some old/unsupported base images. However it's definitly somethin you have to investigate on your own.
Instead I heartily recommend to you the official image of Airflow image - it is based on LTS buster slim image (And we always use most recent images with all security patches. There are various ways you can customize and extend image and we have comprehensive documentation: https://airflow.apache.org/docs/docker-stack/index.html and we will be able to help you if you have problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17185:
URL: https://github.com/apache/airflow/issues/17185#issuecomment-885538743
All works perfectly for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17185:
URL: https://github.com/apache/airflow/issues/17185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17185:
URL: https://github.com/apache/airflow/issues/17185#issuecomment-885426310
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nevinbaiju commented on issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
nevinbaiju commented on issue #17185:
URL: https://github.com/apache/airflow/issues/17185#issuecomment-885525959
@potiuk please pardon if it is not the right way to use this.
I tried with an official image of airflow and tried running `airflow webserver -D` from the terminal and still, the issue persists.
This is the Dockerfile I used:
`
FROM apache/airflow:latest
ENTRYPOINT [ "/bin/bash" ]
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17185:
URL: https://github.com/apache/airflow/issues/17185#issuecomment-885535780
Wrong entrypoint. You need to run the image with 'bash' command and see if it works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nevinbaiju edited a comment on issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
nevinbaiju edited a comment on issue #17185:
URL: https://github.com/apache/airflow/issues/17185#issuecomment-885525959
@potiuk please pardon if it is not the right way to use this.
I tried with an official image of airflow and tried running `airflow webserver -D` from the terminal and still, the issue persists.
This is the Dockerfile I used:
```
FROM apache/airflow:latest
ENTRYPOINT [ "/bin/bash" ]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nevinbaiju commented on issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
nevinbaiju commented on issue #17185:
URL: https://github.com/apache/airflow/issues/17185#issuecomment-885541046
Yes, I just checked it and it is working fine. Thank you so much for the help, I really appreciate it. It was a mistake on my part, I was using the entrypoint wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17185:
URL: https://github.com/apache/airflow/issues/17185#issuecomment-885538627
```
docker run -it apache/airflow:latest bash
airflow@1abe97747ca7:/opt/airflow$ airflow webserver -D
[2021-07-23 10:11:16,569] {cli_action_loggers.py:105} WARNING - Failed to log action with (sqlite3.OperationalError) no such table: log
[SQL: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('2021-07-23 10:11:16.565715', None, None, 'cli_webserver', None, 'airflow', '{"host_name": "1abe97747ca7", "full_command": "[\'/home/airflow/.local/bin/airflow\', \'webserver\', \'-D\']"}')]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-07-23 10:11:16,590] {dagbag.py:496} INFO - Filling up the DagBag from /dev/null
[2021-07-23 10:11:16,690] {manager.py:784} WARNING - No user yet created, use flask fab command to do it.
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat:
=================================================================
airflow@1abe97747ca7:/opt/airflow$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nevinbaiju commented on issue #17185: Unable to start airflow webserver in the background in docker container.
Posted by GitBox <gi...@apache.org>.
nevinbaiju commented on issue #17185:
URL: https://github.com/apache/airflow/issues/17185#issuecomment-885538624
There is still no luck there. However, is this the way one is supposed to start the webserver inside the airflow container?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org