You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/01 11:04:57 UTC
[GitHub] [airflow] christophreimer opened a new issue #16752: DockerOperator fails with HTTPError: 400 Client Error
christophreimer opened a new issue #16752:
URL: https://github.com/apache/airflow/issues/16752
**Apache Airflow version**: 2.1.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): -
**Environment**:
- **Cloud provider or hardware configuration**: OpenStack Victoria
- **OS** (e.g. from /etc/os-release): Ubuntu 20.04.2 LTS
- **Kernel** (e.g. `uname -a`): Linux 5.4.0-1041-kvm
- **Install tools**: pip
- **Others**:
- Docker version 20.10.7, build f0df350
- docker-py 5.0.0
**What happened**:
Created a DAG with basic tasks in it as defined hereafter.
DummyOperator --> DockerOperator --> DummyOperator
*DockerOperator Instance*:
````python
DockerOperator(
task_id="test",
image="debian:sid-slim",
command="hostname",
mounts=[Mount("/eodc", "/eodc", type="bind")]
)
````
**What you expected to happen**:
Pull image, run container and execute hostname in it.
**How to reproduce it**:
Setup Airflow 2.1.0 and latest Docker provider
**Anything else we need to know**:
Task Log Output:
````
[2021-07-01 10:36:39,537] {base_task_runner.py:115} INFO - Job 11: Subtask test /opt/airflow/lib/python3.8/site-packages/airflow/configuration.py:346 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
[2021-07-01 10:36:39,856] {base_task_runner.py:115} INFO - Job 11: Subtask test /opt/airflow/lib/python3.8/site-packages/airflow/configuration.py:346 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2021-07-01 10:36:40,032] {base_task_runner.py:115} INFO - Job 11: Subtask test /opt/airflow/lib/python3.8/site-packages/airflow/configuration.py:346 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
[2021-07-01 10:36:40,243] {base_task_runner.py:115} INFO - Job 11: Subtask test [[34m2021-07-01 10:36:40,243[0m] {[34mdagbag.py:[0m487} INFO[0m - Filling up the DagBag from /home/gfm_ops/gfm_dags/data_preparation.py[0m
[2021-07-01 10:36:40,645] {taskinstance.py:1280} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=airflow@example.com
AIRFLOW_CTX_DAG_OWNER=gfm_ops
AIRFLOW_CTX_DAG_ID=GFM-data-preparation
AIRFLOW_CTX_TASK_ID=test
AIRFLOW_CTX_EXECUTION_DATE=2021-07-01T10:36:37.229404+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-07-01T10:36:37.229404+00:00
[2021-07-01 10:36:40,666] {docker.py:231} INFO - Starting docker container from image debian:sid-slim
[2021-07-01 10:36:40,678] {taskinstance.py:1481} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/opt/airflow/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/containers/create
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1137, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/opt/airflow/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 319, in execute
return self._run_image()
File "/opt/airflow/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 237, in _run_image
self.container = self.cli.create_container(
File "/opt/airflow/lib/python3.8/site-packages/docker/api/container.py", line 430, in create_container
return self.create_container_from_config(config, name)
File "/opt/airflow/lib/python3.8/site-packages/docker/api/container.py", line 441, in create_container_from_config
return self._result(res, True)
File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
self._raise_for_status(response)
File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/opt/airflow/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.41/containers/create: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmp6h76o3kr")
[2021-07-01 10:36:40,680] {base_task_runner.py:115} INFO - Job 11: Subtask test Running <TaskInstance: GFM-data-preparation.test 2021-07-01T10:36:37.229404+00:00 [running]> on host 10.10.103.238
[2021-07-01 10:36:40,684] {base_task_runner.py:115} INFO - Job 11: Subtask test Traceback (most recent call last):
[2021-07-01 10:36:40,684] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
[2021-07-01 10:36:40,684] {base_task_runner.py:115} INFO - Job 11: Subtask test response.raise_for_status()
[2021-07-01 10:36:40,684] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
[2021-07-01 10:36:40,684] {base_task_runner.py:115} INFO - Job 11: Subtask test raise HTTPError(http_error_msg, response=self)
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/containers/create
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test During handling of the above exception, another exception occurred:
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test Traceback (most recent call last):
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1137, in _run_raw_task
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test self._prepare_and_execute_task_with_callbacks(context, task)
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test result = self._execute_task(context, task_copy)
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
[2021-07-01 10:36:40,685] {base_task_runner.py:115} INFO - Job 11: Subtask test result = task_copy.execute(context=context)
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 319, in execute
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test return self._run_image()
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 237, in _run_image
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test self.container = self.cli.create_container(
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/docker/api/container.py", line 430, in create_container
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test return self.create_container_from_config(config, name)
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/docker/api/container.py", line 441, in create_container_from_config
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test return self._result(res, True)
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 274, in _result
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test self._raise_for_status(response)
[2021-07-01 10:36:40,686] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test raise create_api_error_from_http_exception(e)
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test raise cls(e, response=response, explanation=explanation)
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.41/containers/create: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmp6h76o3kr")
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test During handling of the above exception, another exception occurred:
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test Traceback (most recent call last):
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/usr/bin/airflow", line 8, in <module>
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test sys.exit(main())
[2021-07-01 10:36:40,687] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test args.func(args)
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test return func(*args, **kwargs)
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test return f(*args, **kwargs)
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 237, in task_run
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test _run_task_by_selected_method(args, dag, ti)
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 66, in _run_task_by_selected_method
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test _run_raw_task(args, ti)
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 148, in _run_raw_task
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test ti._run_raw_task( # pylint: disable=protected-access
[2021-07-01 10:36:40,688] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test return func(*args, session=session, **kwargs)
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1177, in _run_raw_task
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test self.handle_failure(e, test_mode, error_file=error_file)
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test return func(*args, session=session, **kwargs)
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1487, in handle_failure
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test set_error_file(error_file, error)
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test File "/opt/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 124, in set_error_file
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test with open(error_file, "wb") as fd:
[2021-07-01 10:36:40,689] {base_task_runner.py:115} INFO - Job 11: Subtask test PermissionError: [Errno 13] Permission denied: '/tmp/tmpnzzqxiqz'
[2021-07-01 10:36:40,932] {local_task_job.py:151} INFO - Task exited with return code 1
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16752: DockerOperator fails with HTTPError: 400 Client Error
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16752:
URL: https://github.com/apache/airflow/issues/16752#issuecomment-872151682
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16752: DockerOperator fails with HTTPError: 400 Client Error
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16752:
URL: https://github.com/apache/airflow/issues/16752#issuecomment-872137347
Seems this is a problem with permissions of the user that your arflow workers runs on (or maybe the way how your /tmp folder is mounted), it does not seem Airflow problem but your deployment. Can you please login to the worker and run the docker command that you are expecting to run (with the --bind command) manually to see if you have the problem and solve it before you try airflow again?
```
PermissionError: [Errno 13] Permission denied: '/tmp/tmpnzzqxiqz'
[2021-07-01 10:36:40,932] {local_task_job.py:151} INFO - Task exited with return code 1
```
Closing after we can see that the deployment works but you still have the problem. Let us know if you have more checking done and you still see the problem (with more logs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16752: DockerOperator fails with HTTPError: 400 Client Error
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16752:
URL: https://github.com/apache/airflow/issues/16752#issuecomment-872138949
There is something rather wrong with "/tmp" folder permissions (look at the last error where temporary log file could not be created in "/tmp" folder either).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16752: DockerOperator fails with HTTPError: 400 Client Error
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16752:
URL: https://github.com/apache/airflow/issues/16752#issuecomment-872294885
You need to check your docker engine logs. My guess is that your docker engine has the same permission problem with /tmp folder (but this is a wild guess).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] christophreimer commented on issue #16752: DockerOperator fails with HTTPError: 400 Client Error
Posted by GitBox <gi...@apache.org>.
christophreimer commented on issue #16752:
URL: https://github.com/apache/airflow/issues/16752#issuecomment-872290162
Thx @potiuk, resolved the issue with the /tmp folder permissions. In the systemd service the option PrivateTmp=True was set which was the root cause of it. But still the docker operator doesn't execute successfully.
````bash
2021-07-01 14:12:41,220] {taskinstance.py:1481} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/airflow/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/opt/airflow/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=lastest&fromImage=registry.eodc.eu%2Fjrc-gfm%2Feodc%2Fdata-interfaces
````
I guess the operator tries to pull the image and the corresponding connection details are missing.
However, the image is already on the host, so a docker pull shouldn't be executed per default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #16752: DockerOperator fails with HTTPError: 400 Client Error
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #16752:
URL: https://github.com/apache/airflow/issues/16752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org