You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/07 14:23:44 UTC
[GitHub] [airflow] AlessioM opened a new pull request #13541: added support for device_requests in DockerOperator
AlessioM opened a new pull request #13541:
URL: https://github.com/apache/airflow/pull/13541
closes: #9492
* this uses the device_requests parameter when creating the docker container to enable container to access host devices
* feature was introduced in docker-py [here](https://github.com/docker/docker-py/pull/2471) and released in [4.3.0](https://docker-py.readthedocs.io/en/stable/change-log.html)
* had to increase the docker-py version to >=4.3, breaking changes mentioned in Changelog of docker-py (above) should have no effect
## intended use
```
with_gpu = DockerOperator(
task_id='with_gpu',
image='nvidia/cuda:11.0-base',
device_requests=[
docker.types.DeviceRequest(count=-1, capabilities=[['gpu']])
]
)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tadej-redstone commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
tadej-redstone commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-764622950
@AlessioM perfect, thanks for adding it. How's it going with the PR otherwise? Do you need some help? I could try and contribute, as I'd love to see this feature added soon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-935722042
If you want to take over and submit it and lead to completion - feel free :). Create a new PR please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-792387970
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] PKramek commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
PKramek commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-943371055
Are there any updates on the matter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tadej-redstone commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
tadej-redstone commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-764622950
@AlessioM perfect, thanks for adding it. How's it going with the PR otherwise? Do you need some help? I could try and contribute, as I'd love to see this feature added soon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AlessioM commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
AlessioM commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-764549730
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AlessioM commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
AlessioM commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-764671190
I created another repo with some docker-compose files to test the PR:
https://github.com/AlessioM/airflow-docker-gpu
it should work in theory, problem is I didn't find time to test it properly, especially since I needed to upgrade docker-py to >4.3 (required for the device_requests param) and I don't know if there will be some side effect somewhere
also I'm nut 100% certain how the device_request behaves when the scheduler is executed in a docker container (probably without GPU support), that's something I'd like to test beforehand
so any input there is more than welcome, but I'll try to cnt testing this weekend
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #13541:
URL: https://github.com/apache/airflow/pull/13541
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tadej-redstone removed a comment on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
tadej-redstone removed a comment on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-762849033
Hey @AlessioM , would you consider also adding the `runtime` parameter? It's also a way to use GPUs with docker, by specifying `runtime='nvidia'`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AlessioM commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
AlessioM commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-764549730
added the runtime parameter: see https://docker-py.readthedocs.io/en/stable/api.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tadej-redstone commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
tadej-redstone commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-762848799
Hey @AlessioM , would you consider also adding the `runtime` parameter? It's also a way to use GPUs with docker, by specifying `runtime='nvidia'`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alberthier commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
alberthier commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-935688549
The proposed PR has been closed as there was no activity on it but the code works properly and fixes this issue.
Airflow now depends on docker 5.0.2 so the `device_requests` parameter is now available in `APIClient`.
Is there any chance to see this PR merged ? Or should a new one be submitted ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-756271722
[The Workflow run](https://github.com/apache/airflow/actions/runs/469458321) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AlessioM commented on pull request #13541: added support for device_requests in DockerOperator
Posted by GitBox <gi...@apache.org>.
AlessioM commented on pull request #13541:
URL: https://github.com/apache/airflow/pull/13541#issuecomment-756386424
this will propably only work if the scheduler is running directly (i.e. not in a docker container itself) on the machine which will execute the tasks (because the device request call is executed from the scheduler and hence "sees" only the local devices)
it might be possible to run the scheduler in a docker container using `--gpus=all` and making it thus "see" the same devices as the host, but I haven't tested that yet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org