You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "Liam-Sturge (via GitHub)" <gi...@apache.org> on 2024/02/09 15:55:30 UTC
[I] [CI Problem] CI-arm build queue blocked [tvm]
Liam-Sturge opened a new issue, #16549:
URL: https://github.com/apache/tvm/issues/16549
Jobs that require an arm agent are struggling to find an Graviton-3 executor in Jenkins. There are hundreds of builds queued for ci-arm. Some of the builds queued are stuck. This appears to have started on 8th February.
### Branch/PR Failing
https://github.com/apache/tvm/pull/16183 may have caused the backlog of jobs. Recently a large amount of commits were pushed simultaneously for this PR and builds have been scheduled for each commit. I have attached a graph of the agent queue and allocation, which shows a big spike in queued jobs.
### Jenkins Link
Build logs on these jobs indicate that some of the builds are stuck. They have ended with an unfinished status, but remain in the queue. See this console log as one example of many:
https://ci.tlcpack.ai/job/tvm-arm/job/PR-16183/495/console
### Triage
Please refer to the list of label tags [here](https://github.com/apache/tvm/wiki/Issue-Triage-Labels) to find the relevant tags and add them below in a bullet format (example below).
* needs-triage
CC @lhutton1 @konturn @tqchen @yongwww
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CI Problem] CI-arm build queue blocked [tvm]
Posted by "Liam-Sturge (via GitHub)" <gi...@apache.org>.
Liam-Sturge closed issue #16549: [CI Problem] CI-arm build queue blocked
URL: https://github.com/apache/tvm/issues/16549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CI Problem] CI-arm build queue blocked [tvm]
Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on issue #16549:
URL: https://github.com/apache/tvm/issues/16549#issuecomment-1936484947
Huh, that definitely wasn't intended with that PR. The large number of commits was due to rebasing onto main, where I had expected that only the last commit on the rebased branch would be built, as the intermediate commits don't require testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CI Problem] CI-arm build queue blocked [tvm]
Posted by "Liam-Sturge (via GitHub)" <gi...@apache.org>.
Liam-Sturge commented on issue #16549:
URL: https://github.com/apache/tvm/issues/16549#issuecomment-1938804215
Hi @Lunderberg, I too would have expected that only the last commit on the rebased branch would have been built. Really seems odd that this isn't the default behaviour here.
Thanks for looking in to it and getting the queue moving again. I agree that setting the Jenkins option `disableConcurrentBuilds(abortPrevious: true)` sounds like a sensible idea. For now, I am happy to close this issue as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CI Problem] CI-arm build queue blocked [tvm]
Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on issue #16549:
URL: https://github.com/apache/tvm/issues/16549#issuecomment-1936489408
This PR is also one that impacts `packed_func.h`, which is included by practically everything in TVM. So, not only is it re-building TVM once for each rebased commit, but they're also full rebuilds that can't benefit from `ccache`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CI Problem] CI-arm build queue blocked [tvm]
Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on issue #16549:
URL: https://github.com/apache/tvm/issues/16549#issuecomment-1936553425
Short-term, I've stopped all tasks related to #16183, and the ARM queue is recovering. All remaining tasks are related to other PRs, as it works through the backlog.
Long-term, it looks like there's a Jenkins option `disableConcurrentBuilds(abortPrevious: true)` ([stack overflow link](https://stackoverflow.com/questions/46343348/build-only-latest-commit-on-branch-with-jenkins), [GH link](https://github.com/jenkinsci/workflow-job-plugin/pull/200))) that we should enable. If there's two concurrent builds for the same PR, it would cancel the previous one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CI Problem] CI-arm build queue blocked [tvm]
Posted by "lhutton1 (via GitHub)" <gi...@apache.org>.
lhutton1 commented on issue #16549:
URL: https://github.com/apache/tvm/issues/16549#issuecomment-2009288622
This issue also seems to have occurred on #16425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org