You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/22 00:15:39 UTC
[GitHub] [airflow] zengbotang opened a new issue, #23159: docker container still running while dag run failed
zengbotang opened a new issue, #23159:
URL: https://github.com/apache/airflow/issues/23159
### Apache Airflow version
2.1.4
### What happened
I have operator run with docker .
When dag run failed , docker.py try to remove container but remove failed and got the following error:
`2022-04-20 00:03:50,381] {taskinstance.py:1463} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 301, in _run_image_with_mounts
for line in lines:
File "/home/airflow/.local/lib/python3.8/site-packages/docker/types/daemon.py", line 32, in __next__
return next(self._stream)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 412, in <genexpr>
gen = (data for (_, data) in gen)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/socket.py", line 92, in frames_iter_no_tty
(stream, n) = next_frame_header(socket)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/socket.py", line 64, in next_frame_header
data = read_exactly(socket, 8)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/socket.py", line 49, in read_exactly
next_data = read(socket, n - len(data))
File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/socket.py", line 29, in read
select.select([socket], [], [])
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1238, in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/home/airflow/.local/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.35/containers/de4cd812f8b0dcc448d591d1bd28fa736b1712237c8c8848919be512938bd515?v=False&link=False&force=False
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1165, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1283, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1313, in _execute_task
result = task_copy.execute(context=context)
File "/usr/local/airflow/dags/operators/byx_base_operator.py", line 611, in execute
raise e
File "/usr/local/airflow/dags/operators/byx_base_operator.py", line 591, in execute
self.execute_job(context)
File "/usr/local/airflow/dags/operators/byx_datax_operator.py", line 93, in execute_job
result = call_datax.execute(context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 343, in execute
return self._run_image()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 265, in _run_image
return self._run_image_with_mounts(self.mounts, add_tmp_variable=False)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py", line 317, in _run_image_with_mounts
self.cli.remove_container(self.container['Id'])
File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/container.py", line 1010, in remove_container
self._raise_for_status(res)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/home/airflow/.local/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.35/containers/de4cd812f8b0dcc448d591d1bd28fa736b1712237c8c8848919be512938bd515?v=False&link=False&force=False: Conflict ("You cannot remove a running container de4cd812f8b0dcc448d591d1bd28fa736b1712237c8c8848919be512938bd515. Stop the container before attempting removal or force remove")
`
### What you think should happen instead
the container should removed successful when dag run failed
### How to reproduce
step 1: create a dag with execute DockerOperator operation
step 2: trigger dag
step 3: mark dag run to failed simulate dag run failed, and the remove container failed error will appear and the docker container still running.
### Operating System
NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2"
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #23159: docker container still running while dag run failed
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #23159: docker container still running while dag run failed
URL: https://github.com/apache/airflow/issues/23159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #23159: docker container still running while dag run failed
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #23159:
URL: https://github.com/apache/airflow/issues/23159#issuecomment-1105875457
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org