[build system] massive jenkins infrastructure changes forthcoming

[shane knapp <sk...@berkeley.edu>]

TL;DR:  amplab is becomine riselab, and is much more C++ oriented.
centos 6 is so far behind, and i'm already having to roll C++
compilers and various libraries by hand.  centos 7 is an absolute
no-go, so we'll be moving the jenkins workers over to a recent (TBD)
version of ubuntu server.  also, we'll finally get jenkins upgraded to
the latest LTS version, as well as our insanely out of date plugins.
riselab (me) will still run the build system, btw.

oh, we'll also have a macOS worker!

well, that was still pretty long.  :)

anyways, you have the gist of it.  this is something we're going to do
slowly, so as to not interrupt any spark, alluxio or lab builds.

i'll be spinning up a master and two worker ubuntu nodes, and then
port a couple of builds over and get the major kinks worked out.
then, early next year, we can point the new master at the old workers,
and one-by-one reinstall and deploy them w/ubuntu.

i'll be reaching out to some individuals (you know who you are) as
things progress.

if we do this right, we'll have minimal service interruptions and end
up w/a clean and fresh jenkins.  this is the opposite of our current
jenkins, which is at least 4 years old and is super-glued and
duct-taped together.

the ubuntu staging servers should be ready early next week, but i
don't foresee much work happening until after thanksgiving.

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [build system] massive jenkins infrastructure changes forthcoming

[shane knapp <sk...@berkeley.edu>]

On Thu, Nov 17, 2016 at 4:52 PM, Reynold Xin <rx...@databricks.com> wrote:
> Thanks for the headsup, Shane.
>
no problem!

i'm really looking forward to starting w/a much cleaner slate that
what we have now.

not only are we locked in a jenkins/plugin version dependency hell
that keeps us from updating easily and regularly (esp WRT security
issues), but our jenkins logs completely useless for debugging due to
stack trace spam.

no joke, about 80%+ of the logs are stack traces.  :(

anyways, i will post more updates as they come in.

shane

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: [build system] massive jenkins infrastructure changes forthcoming

[Reynold Xin <rx...@databricks.com>]

Thanks for the headsup, Shane.


On Thu, Nov 17, 2016 at 2:33 PM, shane knapp <sk...@berkeley.edu> wrote:

> TL;DR:  amplab is becomine riselab, and is much more C++ oriented.
> centos 6 is so far behind, and i'm already having to roll C++
> compilers and various libraries by hand.  centos 7 is an absolute
> no-go, so we'll be moving the jenkins workers over to a recent (TBD)
> version of ubuntu server.  also, we'll finally get jenkins upgraded to
> the latest LTS version, as well as our insanely out of date plugins.
> riselab (me) will still run the build system, btw.
>
> oh, we'll also have a macOS worker!
>
> well, that was still pretty long.  :)
>
> anyways, you have the gist of it.  this is something we're going to do
> slowly, so as to not interrupt any spark, alluxio or lab builds.
>
> i'll be spinning up a master and two worker ubuntu nodes, and then
> port a couple of builds over and get the major kinks worked out.
> then, early next year, we can point the new master at the old workers,
> and one-by-one reinstall and deploy them w/ubuntu.
>
> i'll be reaching out to some individuals (you know who you are) as
> things progress.
>
> if we do this right, we'll have minimal service interruptions and end
> up w/a clean and fresh jenkins.  this is the opposite of our current
> jenkins, which is at least 4 years old and is super-glued and
> duct-taped together.
>
> the ubuntu staging servers should be ready early next week, but i
> don't foresee much work happening until after thanksgiving.
>