You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/07/26 01:15:38 UTC
[GitHub] [incubator-mxnet] yajiedesign opened a new issue #18793: Can we install docker on Windows CI machine?
yajiedesign opened a new issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793
Prepare to deploy CD in docker like Linux.
Can docker be installed on the machine and start Windows container mode?
leonard
@leezu
@szha
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-669995116
@szha
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign edited a comment on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign edited a comment on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-668335017
However, these dockerfiles cannot be used directly because they contain a large number of external files.
I am upload all file to onedriver https://1drv.ms/u/s!AgSFk5eCdAIXkVy8o_B54S1_n8ML?e=qkuJUu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
josephevans commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-713662769
I have a new Windows AMI created with Cuda 11.0 and docker, but having issues with testing. Builds work, but unit tests are failing with: OSError: exception: access violation writing 0x0000000000000000
I believe this may be due to OpenCV or OpenBLAS dependencies.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-670074764
> these dockerfiles cannot be used directly because they contain a large number of external files.
Our CI actually needs to be able to build the container (at least that's how we do it for Linux containers). Should we request NVidia to provide containers a la https://hub.docker.com/r/nvidia/cuda/ for windows?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-668124238
Thank you @yajiedesign. Can you also share the Dockerfile for https://hub.docker.com/r/yajiedesign/mxnet_cd_demo
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
josephevans commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-727294275
OK guys, I finally got new AMIs launched for Windows builds that include docker. They are currently in-place with Cuda 11.0 installed. You should be able to start migrating to containerized builds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-670281672
@leezu if NVidia can provide windows cuda containers is the best.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign edited a comment on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign edited a comment on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-667505476
@leezu @szha I have successfully made cuda100 images, and completed the compilation and packaging in the docker container.
next we need to install docker in aws machine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
josephevans commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-680990433
So just to recap, we need the following installed on a Windows AMI:
- docker
- python3
- jenkins slave run script (in python)
- java runtime environment (to run jenkins slave client)
- NVidia driver
- CUDA libraries/toolkit
Anything else?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sandeep-krishnamurthy commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
sandeep-krishnamurthy commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-679600988
> I can spend some time creating Windows AMIs with docker installed. Can I use the existing AMIs we currently use for Windows CI? What version of docker and what installation steps are you using?
@josephevans - Can you please help on how to help @yajiedesign and get this work merged to CI?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-668350981
I am upload all file to onedriver https://1drv.ms/u/s!AgSFk5eCdAIXkVy8o_B54S1_n8ML?e=qkuJUu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-667855574
I am push a domo image to dockerhub yajiedesign/mxnet_cd_demo:1.0
run c:\t\run_demo.bat start build.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-680892287
Then the NV driver and CUDA are needed to run the test, and the tool chain is still placed in the docker container.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign edited a comment on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign edited a comment on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-668335017
https://github.com/yajiedesign/mxnet_cd_demo
However, these dockerfiles cannot be used directly because they contain a large number of external files.
I am upload all file to onedriver https://1drv.ms/u/s!AgSFk5eCdAIXkVy8o_B54S1_n8ML?e=qkuJUu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-667505476
@leezu @szha I have successfully made cuda10 images, and completed the compilation and packaging in the docker container.
next we need to install docker in aws machine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
josephevans commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-681011230
@marcoabreu Yeah, I'm familiar with the scripts to create the AMIs. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
szha commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-706795516
what's the status for this? It's been almost three months.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] yajiedesign removed a comment on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign removed a comment on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-668334177
https://github.com/yajiedesign/mxnet_cd_demo
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
josephevans commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-670086423
I can spend some time creating Windows AMIs with docker installed. Can I use the existing AMIs we currently use for Windows CI? What version of docker and what installation steps are you using?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-667506665
Can you please show a demo before the team takes on the effort to update the ami?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign edited a comment on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign edited a comment on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-664712383
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign edited a comment on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign edited a comment on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-670240559
@josephevans I use ltsc2019 base image,You can choose the Windows version based on this page:https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility?tabs=windows-server-2004%2Cwindows-10-2004
the docker use the last version.I am use win10,so I donwload and run win10 docker setup and next next next.
win server look this :https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-docker/configure-docker-daemon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-668335017
However, these dockerfiles cannot be used directly because they contain a large number of external files. The total is about 4GB.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-668334177
https://github.com/yajiedesign/mxnet_cd_demo
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign removed a comment on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign removed a comment on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-668350981
I am upload all file to onedriver https://1drv.ms/u/s!AgSFk5eCdAIXkVy8o_B54S1_n8ML?e=qkuJUu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-680670552
@josephevans For security reasons, I'm not going to run tests on CD machines. So maybe I just need to docker.jenkins slave,python.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-680679301
From an ami perspective, we are not differentiating between CD and CI machines. They all use the same image
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign edited a comment on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign edited a comment on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-680670552
@josephevans For security reasons, I'm not going to run tests on CD machines. So maybe I just need to docker.jenkins slave,python.All toolchains will be installed in the docker container. and Windows docker cannot run cuda now .The nv drive is also unnecessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
szha commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-670282320
> @leezu if NVidia can provide windows cuda containers is the best.
cc @TristonC who tends to have magical ways of making such things happen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
josephevans commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-680132801
I don't think we want to re-use an existing Windows CI AMI for this, because it has a lot of stuff installed that may not be needed. Are you planning on installing all the build tools (MSVS, nvcc, etc) in the docker container instead of on the host system? Besides docker, what other software needs to be installed on the base AMI? Does NVidia drivers need to be installed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-680992712
Just to make sure: you are aware of the scripts to automatically create a windows slave ami, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-667514804
I try to gen a demo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-670240559
@josephevans I use ltsc2019 base image,You can choose the Windows version based on this page:https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility?tabs=windows-server-2004%2Cwindows-10-2004
the docker use the last version.I am use win10,so I donwload and run win10 docker setup and next next next etc.
win server look this :https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-docker/configure-docker-daemon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] yajiedesign commented on issue #18793: [CD]Can we install docker on Windows CI/CD machine?
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #18793:
URL: https://github.com/apache/incubator-mxnet/issues/18793#issuecomment-664712383
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org