You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/12/10 13:43:13 UTC
[GitHub] [tvm] manupa-arm opened a new issue #9705: [CI Problem] CI node out of memory
manupa-arm opened a new issue #9705:
URL: https://github.com/apache/tvm/issues/9705
Examples :
* https://ci.tlcpack.ai/blue/organizations/jenkins/docker-images-ci%2Fdaily-docker-image-rebuild/detail/daily-docker-image-rebuild/143/pipeline
* https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/175/pipeline/
This started with a timeout on https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/173/pipeline.
It seems like the CI nodes starts with already past images. Should we clean them ?
Possible resolutions for both TVM's Jenkins file and daily-docker-image-rebuild Jenkins file:
* Doing the cleanup_docker_image("ci_cpu") before the build is started
* Do a full prune and pull the docker images from the hub
Any other ideas to proceed are also welcome!
cc : @leandron @tqchen @areusch @u99127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] manupa-arm commented on issue #9705: [CI Problem] CI node out of memory
Posted by GitBox <gi...@apache.org>.
manupa-arm commented on issue #9705:
URL: https://github.com/apache/tvm/issues/9705#issuecomment-992306803
Thanks! @leandron
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] leandron commented on issue #9705: [CI Problem] CI node out of memory
Posted by GitBox <gi...@apache.org>.
leandron commented on issue #9705:
URL: https://github.com/apache/tvm/issues/9705#issuecomment-992303400
I temporarily disabled this machine on Jenkins, due to storage issues, FYI.
https://ci.tlcpack.ai/computer/octo.aws.c4.44.242.35.246/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on issue #9705: [CI Problem] CI node out of memory
Posted by GitBox <gi...@apache.org>.
areusch commented on issue #9705:
URL: https://github.com/apache/tvm/issues/9705#issuecomment-993806967
from Noah:
- docker prune script on static nodes (which I have running as a cron job) doesn't pass the 'volumes' flag
- manually fixed 5 nodes which were full (think 246 was one of them)
- permanent fix requires a rolling update of the fleet to bring in a new AMI. no time for this right now.
let's leave this bug open til we do the rolling update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org