You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by shane knapp <sk...@berkeley.edu> on 2018/07/30 23:31:38 UTC

[build system] two workers will be reimaged w/ubuntu tomorrow

my testing is going really well, and i think we're --->this<--- close to
porting all of the spark builds to ubuntu!

TL;DR:   i am NOT planning on moving all builds to centos until after
august 8th.  i WOULD like to move the PRB to ubuntu before then.

anyways:

once these two smoke test builds pass, i'll be somewhere @ ~99% certainty
that we can just move all of the spark builds (except docs) en masse!

https://amplab.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/59/
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6-ubuntu-test/887/

we're starting to run in to build node availability bottlenecks for some of
our lab builds, so we will be reinstalling the following two workers
w/ubuntu tomorrow:
amp-jenkins-worker-07
amp-jenkins-worker-08

this will give us some breathing room, and we shouldn't have any noticeable
impact on the spark builds.

the biggest worry is the pull request builder...  i'll update that job and
let a couple of PRBs run through before moving it back to the centos hosts.

if the regular PRB builds pass happily, this will let us move it from
centos to ubuntu *before* 2.4 is cut, and gives us two things:

1) unblock https://github.com/apache/spark/pull/21584
2) stop needing 2 builds for pull requests (one for regular tests on
centos, one to test against minikube on ubuntu).

questions/comments/concerns?

shane
-- 
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu