You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/13 10:34:15 UTC
[GitHub] [airflow] maxim317 opened a new issue, #28332: DockerSwarmOperator placement dont work
maxim317 opened a new issue, #28332:
URL: https://github.com/apache/airflow/issues/28332
### Apache Airflow Provider(s)
docker
### Versions of Apache Airflow Providers
3.3.0
### Apache Airflow version
2.5.0
### Operating System
Ubuntu 20.04
### Deployment
Docker-Compose
### Deployment details
t1 = DockerSwarmOperator(
api_version='auto',
docker_url='unix://var/run/docker.sock', # Set your docker URL
mounts=[
Mount(source="/mnt/gfd/airflow/data", target="/home", type="bind"),
],
command=[
"/bin/bash",
"-c",
"/bin/pwd > pwd.file",
],
image='helloworld',
auto_remove=True,
task_id='dockerSwarmTaskSolver',
enable_logging=False,
pool_slots=2,
pool="solvers",
placement="constraints:node.hostname==SUPERserver1",
dag=dag,
)
### What happened
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/docker/operators/docker_swarm.py", line 122, in execute
return self._run_service()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/docker/operators/docker_swarm.py", line 128, in _run_service
self.service = self.cli.create_service(
File "/home/airflow/.local/lib/python3.10/site-packages/docker/utils/decorators.py", line 34, in wrapper
return f(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/docker/api/service.py", line 189, in create_service
return self._result(
File "/home/airflow/.local/lib/python3.10/site-packages/docker/api/client.py", line 274, in _result
self._raise_for_status(response)
File "/home/airflow/.local/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e) from e
File "/home/airflow/.local/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.41/services/create: Bad Request ("**json: cannot unmarshal string into Go struct field TaskSpec.TaskTemplate.Placement of type swarm.Placement**")
[2022-12-13, 13:21:50 MSK] {taskinstance.py:1322} INFO - Marking task as FAILED. dag_id=opt_final_DOCKER_7_BIGSOLVER_1, task_id=dockerSwarmTaskSolver, execution_date=20221213T102144, start_date=20221213T102150, end_date=20221213T102150
[2022-12-13, 13:21:51 MSK] {standard_task_runner.py:100} ERROR - Failed to execute job 49633 for task dockerSwarmTaskSolver (400 Client Error for http+docker://localhost/v1.41/services/create: Bad Request ("json: cannot unmarshal string into Go struct field TaskSpec.TaskTemplate.Placement of type swarm.Placement"); 1069977)
[2022-12-13, 13:21:51 MSK] {local_task_job.py:159} INFO - Task exited with return code 1
### What you think should happen instead
How to make my container run on the node specified in the placement?
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] maxim317 commented on issue #28332: DockerSwarmOperator placement dont work
Posted by GitBox <gi...@apache.org>.
maxim317 commented on issue #28332:
URL: https://github.com/apache/airflow/issues/28332#issuecomment-1359328451
solution
in begin Dag file add:
`from docker.types import Placement`
in task add:
`placement=Placement(constraints=['node.hostname==SUPERserver1']),`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28332: DockerSwarmOperator placement dont work
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28332:
URL: https://github.com/apache/airflow/issues/28332#issuecomment-1348154635
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28332: DockerSwarmOperator placement dont work
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28332:
URL: https://github.com/apache/airflow/issues/28332#issuecomment-1354043409
@maxim317 you should provide [docker.types.Placement](https://docker-py.readthedocs.io/en/stable/api.html?highlight=placement#docker.types.Placement) or list of them, instead of just string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis closed issue #28332: DockerSwarmOperator placement dont work
Posted by GitBox <gi...@apache.org>.
Taragolis closed issue #28332: DockerSwarmOperator placement dont work
URL: https://github.com/apache/airflow/issues/28332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org