You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/04/27 19:49:11 UTC

[GitHub] [incubator-mxnet] leezu opened a new pull request #18179: Enable docker cache build for images listed in docker-compose.yml

leezu opened a new pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179


   Enable docker cache build for images listed in docker-compose.yml
   Ignore `docker-compose.yml` related files in `ci/build_cache.py`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] leezu commented on pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#issuecomment-620375272


   The run you point out fails due to flakyness of dockerhub.
   Dockerhub suddenly rejects our credentials and we get `[2020-04-28T01:55:45.399Z] denied: requested access to the resource is denied`
   
   After retrigger, that error is gone. However, there's another error to be fixed by
   https://github.com/apache/incubator-mxnet/pull/18183


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#discussion_r416159358



##########
File path: ci/Jenkinsfile_docker_cache
##########
@@ -38,6 +38,7 @@ core_logic: {
         timeout(time: total_timeout, unit: 'MINUTES') {
           utils.init_git()
           sh "python3 ./ci/docker_cache.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}"
+          sh "cd ci && docker-compose -f docker/docker-compose.yml build --parallel && docker-compose -f docker/docker-compose.yml push "

Review comment:
       Can you pull that into docker_cache.py to leverage parallelisation wrt non updated images and also providing one single interface?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#discussion_r416182368



##########
File path: ci/Jenkinsfile_docker_cache
##########
@@ -38,6 +38,7 @@ core_logic: {
         timeout(time: total_timeout, unit: 'MINUTES') {
           utils.init_git()
           sh "python3 ./ci/docker_cache.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}"
+          sh "cd ci && docker-compose -f docker/docker-compose.yml build --parallel && docker-compose -f docker/docker-compose.yml push "

Review comment:
       I'm not sure what parallelism you refer to. Is `docker-compose build --parallel` not sufficient?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#discussion_r416182047



##########
File path: ci/Jenkinsfile_docker_cache
##########
@@ -38,6 +38,7 @@ core_logic: {
         timeout(time: total_timeout, unit: 'MINUTES') {
           utils.init_git()
           sh "python3 ./ci/docker_cache.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}"
+          sh "cd ci && docker-compose -f docker/docker-compose.yml build --parallel && docker-compose -f docker/docker-compose.yml push "

Review comment:
       The idea is to delete `docker_cache.py` later, so the `docker-compose -f docker/docker-compose.yml build --parallel && docker-compose -f docker/docker-compose.yml push` would be the unified interface.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] ChaiBapchya commented on a change in pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on a change in pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#discussion_r416231669



##########
File path: tests/README.md
##########
@@ -62,13 +62,21 @@ Ninja is a build tool (like make) that prioritizes building speed. If you will b
     
 ## Runing Python Tests Within Docker
 
-1. To run tests inside docker run the following comamdn
-    ```
-    ci/build.py --platform {PLATFORM} /work/runtime_functions.sh {RUNTIME_FUNCTION}
-    ```
+To run tests inside docker, you first need to install `docker` and `docker-compose` on your machine.
+
+On Ubuntu you may install them via `sudo apt-get install docker.io docker-compose`
+and set them up via `sudo usermod $(whoami) -G docker -a`.
+
+Then, to run tests inside docker run the following command

Review comment:
       Can you also add `pip3 install -r ci/requirements.txt`
   Thanks




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#issuecomment-620196707


   Hey @leezu , Thanks for submitting the PR 
   All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands: 
   - To trigger all jobs: @mxnet-bot run ci [all] 
   - To trigger specific jobs: @mxnet-bot run ci [job1, job2] 
   *** 
   **CI supported jobs**: [unix-gpu, sanity, website, centos-cpu, miscellaneous, windows-cpu, unix-cpu, clang, edge, centos-gpu, windows-gpu]
   *** 
   _Note_: 
    Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin. 
   All CI tests must pass before the PR can be merged. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#discussion_r416222468



##########
File path: ci/Jenkinsfile_docker_cache
##########
@@ -38,6 +38,7 @@ core_logic: {
         timeout(time: total_timeout, unit: 'MINUTES') {
           utils.init_git()
           sh "python3 ./ci/docker_cache.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}"
+          sh "cd ci && docker-compose -f docker/docker-compose.yml build --parallel && docker-compose -f docker/docker-compose.yml push "

Review comment:
       I understand. 
   
   It's not sufficient in the sense of that docker-cache and docker compose are run in sequence. Alternatively, create a second job so it runs on another node?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#discussion_r416230937



##########
File path: ci/Jenkinsfile_docker_cache
##########
@@ -38,6 +38,7 @@ core_logic: {
         timeout(time: total_timeout, unit: 'MINUTES') {
           utils.init_git()
           sh "python3 ./ci/docker_cache.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}"
+          sh "cd ci && docker-compose -f docker/docker-compose.yml build --parallel && docker-compose -f docker/docker-compose.yml push "

Review comment:
       `docker-cache` will only be used temporarily. I think the current solution is fine given that the first step (`sh "python3 ./ci/docker_cache.py`) will be deleted later.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] ChaiBapchya commented on pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#issuecomment-620332320


   @leezu it's supposed to fail on docker cache refresh?
   http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/restricted-docker-cache-refresh/detail/master/2830/pipeline


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] leezu commented on pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#issuecomment-620318773


   @mxnet-bot run ci [sanity]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#discussion_r416222468



##########
File path: ci/Jenkinsfile_docker_cache
##########
@@ -38,6 +38,7 @@ core_logic: {
         timeout(time: total_timeout, unit: 'MINUTES') {
           utils.init_git()
           sh "python3 ./ci/docker_cache.py --docker-registry ${env.DOCKER_CACHE_REGISTRY}"
+          sh "cd ci && docker-compose -f docker/docker-compose.yml build --parallel && docker-compose -f docker/docker-compose.yml push "

Review comment:
       I understand. 
   
   It's not sufficient in the sense of that docker-cache and docker compose are run in sequence. Alternatively, create a second parallel step so it runs on another node?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#discussion_r416256492



##########
File path: tests/README.md
##########
@@ -62,13 +62,21 @@ Ninja is a build tool (like make) that prioritizes building speed. If you will b
     
 ## Runing Python Tests Within Docker
 
-1. To run tests inside docker run the following comamdn
-    ```
-    ci/build.py --platform {PLATFORM} /work/runtime_functions.sh {RUNTIME_FUNCTION}
-    ```
+To run tests inside docker, you first need to install `docker` and `docker-compose` on your machine.
+
+On Ubuntu you may install them via `sudo apt-get install docker.io docker-compose`
+and set them up via `sudo usermod $(whoami) -G docker -a`.
+
+Then, to run tests inside docker run the following command

Review comment:
       Let's add it in a separate PR to unblock merging this fix. We can also consider getting rid of that dependency by moving relying on standard tools instead of using our custom scripts.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] ChaiBapchya commented on a change in pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on a change in pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#discussion_r416299329



##########
File path: tests/README.md
##########
@@ -62,13 +62,21 @@ Ninja is a build tool (like make) that prioritizes building speed. If you will b
     
 ## Runing Python Tests Within Docker
 
-1. To run tests inside docker run the following comamdn
-    ```
-    ci/build.py --platform {PLATFORM} /work/runtime_functions.sh {RUNTIME_FUNCTION}
-    ```
+To run tests inside docker, you first need to install `docker` and `docker-compose` on your machine.
+
+On Ubuntu you may install them via `sudo apt-get install docker.io docker-compose`
+and set them up via `sudo usermod $(whoami) -G docker -a`.
+
+Then, to run tests inside docker run the following command

Review comment:
       Makes sense. But custom scripts sort of make job easier for lazy people like me :p having to google steps for getting those standard tools. What do you reckon? Apart from taking the onus of ensuring testing is correct rather than leaving it to users to find their way out of the problems that may come up with setting up those standard tools on their own.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18179: Enable docker cache build for images listed in docker-compose.yml

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18179:
URL: https://github.com/apache/incubator-mxnet/pull/18179#issuecomment-620318808


   Jenkins CI successfully triggered : [sanity]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org