You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/05/02 17:49:16 UTC
[GitHub] [tvm] areusch opened a new issue, #11195: [CI Problem] docker build can fail due to concurrent docker system prune
areusch opened a new issue, #11195:
URL: https://github.com/apache/tvm/issues/11195
Seeing strange build failures due to missing docker images:
[1.](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-11170/1/pipeline/61/)
```
[2022-04-28T21:00:02.003Z] Step 22/76 : RUN bash /install/ubuntu_install_java.sh
[2022-04-28T21:00:02.569Z] No such image: sha256:784a469a44d1bddcaf168d978bee1c0ee32de4a157142e77598083a72cf44501
```
[2.](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-11148/8/pipeline/)
```
[2022-05-02T17:00:23.338Z] unknown parent image ID sha256:1e211c29ce68f4552a3e76cc8151456ea41f58740ee3c0fcb5a3cd35175fcd6d
[2022-05-02T17:00:23.338Z] ERROR: docker build failed.
```
Hypothesis is that periodically, the CI workers run `docker system prune` to remove stale CI images and ensure PRs don't run out of disk space on the executor nodes. This prune nukes the in-progress docker build containers if run at just the right time and prevents the builds from succeeding.
Proposed workaround is to only prune images older than e.g. 20m, which can be done with a flag to docker system prune.
cc @konturn @driazati
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on issue #11195: [CI Problem] docker build can fail due to concurrent docker system prune
Posted by GitBox <gi...@apache.org>.
areusch commented on issue #11195:
URL: https://github.com/apache/tvm/issues/11195#issuecomment-1115465433
in practice our docker containers are built with --no-cache, so only the FROM image should be older than e.g. `<longest_step_time> + <fudge_factor>` min. i think we could do either thing, but the prune does help to ensure we don't have a rogue CI run that allocates tons of disk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] driazati closed issue #11195: [CI Problem] docker build can fail due to concurrent docker system prune
Posted by GitBox <gi...@apache.org>.
driazati closed issue #11195: [CI Problem] docker build can fail due to concurrent docker system prune
URL: https://github.com/apache/tvm/issues/11195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] konturn commented on issue #11195: [CI Problem] docker build can fail due to concurrent docker system prune
Posted by GitBox <gi...@apache.org>.
konturn commented on issue #11195:
URL: https://github.com/apache/tvm/issues/11195#issuecomment-1116293196
Ah I see--well in any case I just opened https://github.com/tlc-pack/ci-packer/pull/4, which adds a prune filter as proposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] konturn commented on issue #11195: [CI Problem] docker build can fail due to concurrent docker system prune
Posted by GitBox <gi...@apache.org>.
konturn commented on issue #11195:
URL: https://github.com/apache/tvm/issues/11195#issuecomment-1115222281
Setting an age filter on docker prune won't completely resolve the issue, as it's possible for docker builds to use older cached layers, and then to subsequently fail on the `docker push` operation if a prune occurs in between the cache and the push. A more sustainable solution would be to limit the lifetime of CI nodes (e.g., https://aws.amazon.com/about-aws/whats-new/2019/11/amazon-ec2-auto-scaling-supports-max-instance-lifetime/) and remove the system prune operation altogether. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] driazati commented on issue #11195: [CI Problem] docker build can fail due to concurrent docker system prune
Posted by GitBox <gi...@apache.org>.
driazati commented on issue #11195:
URL: https://github.com/apache/tvm/issues/11195#issuecomment-1209990834
This should be fixed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org