[GitHub] marcoabreu opened a new pull request #13383: Add utility slave

[GitBox <gi...@apache.org>]

marcoabreu opened a new pull request #13383: Add utility slave
URL: https://github.com/apache/incubator-mxnet/pull/13383
 
 
   This PR adds utility slaves.
   
   The idea of utility slaves is to offload miscellaneous lifecycle-tasks that are not actually part of the task but rather involved to keep the pipeline alive. We should not process these tasks on the same slaves as actual workload since this has multiple implications:
   1. mxnetlinux-cpu slaves are very powerful and only have 3 executors. We don't need 72 CPU cores to load a groovy script
   2. Lifecycle tasks like publishing the GitHub status to PRs share the same queue as regular workload. This might result in a PR not being notified that a job has been queued (or finished) until a mxnetlinux-cpu worker is free. This might confuse the user.
   3. Since a lifecycle task might be only executed when it reaches the top of the queue, the follow-up jobs will only be added to the queue after the lifecycle job has been processed. Only at this point, the real jobs will be enqueued. This means that it has two wait for it's part twice. Especially considering our auto scaling gets triggered every 5 minutes, it could mean that the real workload misses multiple scaling cycles since the lifecycle job hasn't passed yet and thus the "real" jobs haven't been scheduled yet.
   
   Basically what this replaces is the concept of a high-priority queue. Since we don't have that system in Jenkins (and neither do we want to use expensive slaves for that), we can instead make these slaves with a lot of executors and a special label.
   
   This job is NOT intended to run ANY custom workload but only to execute Jenkins operations that require a node context. Usually, these tasks would run on the Jenkins master, but I disabled that for security reasons.
   
   The slave would be a c5.large with 2vCPUs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services