You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by shane knapp <sk...@berkeley.edu> on 2015/04/03 18:59:54 UTC
extended jenkins downtime, thursday april 9th 7am-noon PDT (moving to
anaconda python & more)
welcome to python2.7+, java 8 and more! :)
i'll be doing a major upgrade to our build system next thursday morning.
here's a quick list of what's going on:
* installation of anaconda python on all worker nodes
* installation of pypy 2.5.1 (python 2.7) on all nodes
* matching installation of python modules for the current system python
(2.6), and anaconda python (2.6, 2.7 and 3.4)
- anaconda python 2.7 will be the default for all workers (this has
stealthily been the case on amp-jenkins-worker-01 for the past two weeks,
and i've noticed no test failures)
- you can now use anaconda environments to specify which version of
python to use in your tests: http://www.continuum.io/blog/conda
* installation of new python 2.7 modules: pymongo requests six pymongo
requests six python-crontab
* bare-bones mongodb installation on all workers
* installation of java 1.6 and 1.8 internal to jenkins
- jobs will default to the system java, which is 1.7.0_75
- if you want to run your tests w/java 6 or 8, you can select the JDK
version of your choice in the job configuration page (it'll be towards the
top)
these changes have actually all been tested against a variety of builds
(yay staging!) and while i'm certain that i have all of the kinks worked
out, i'm going to schedule a longer downtime so that i have a chance to
identify and squash any problems that surface.
thanks to josh rosen, k. shankari and davies liu for helping me test all of
this and get it working.
shane
Re: extended jenkins downtime, thursday april 9th 7am-noon PDT
(moving to anaconda python & more)
Posted by shane knapp <sk...@berkeley.edu>.
yep, everything is installed (and i just checked again). the path for
python 3.4 is /home/anaconda/bin/envs/py3k/bin, which you can find by
either manually prepending it to the PATH variable or running 'source
activate py3k' in the test.
On Tue, Apr 14, 2015 at 11:41 AM, Davies Liu <da...@databricks.com> wrote:
> Hey Shane,
>
> Have you updated all the jenkins slaves?
>
> There is a run with old configurations (no Python 3, with 130 minutes
> timeout), see
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/666/consoleFull
>
> Davies
>
> On Thu, Apr 9, 2015 at 10:18 AM, shane knapp <sk...@berkeley.edu> wrote:
> > ok, we're looking good. i'll keep an eye on this for the rest of the
> day,
> > and if you happen to notice any infrastructure failures before i do (i
> > updated a LOT), please let me know immediately! :)
> >
> > On Thu, Apr 9, 2015 at 8:38 AM, shane knapp <sk...@berkeley.edu> wrote:
> >
> >> things are looking pretty good and i expect to be done within an hour.
> >> i've got some test builds running right now, and will give the green
> light
> >> when they successfully complete.
> >>
> >> On Thu, Apr 9, 2015 at 7:29 AM, shane knapp <sk...@berkeley.edu>
> wrote:
> >>
> >>> and this is now happening.
> >>>
> >>> On Tue, Apr 7, 2015 at 4:38 PM, shane knapp <sk...@berkeley.edu>
> wrote:
> >>>
> >>>> reminder! this is happening thurday morning.
> >>>>
> >>>> On Fri, Apr 3, 2015 at 9:59 AM, shane knapp <sk...@berkeley.edu>
> wrote:
> >>>>
> >>>>> welcome to python2.7+, java 8 and more! :)
> >>>>>
> >>>>> i'll be doing a major upgrade to our build system next thursday
> >>>>> morning. here's a quick list of what's going on:
> >>>>>
> >>>>> * installation of anaconda python on all worker nodes
> >>>>>
> >>>>> * installation of pypy 2.5.1 (python 2.7) on all nodes
> >>>>>
> >>>>> * matching installation of python modules for the current system
> python
> >>>>> (2.6), and anaconda python (2.6, 2.7 and 3.4)
> >>>>> - anaconda python 2.7 will be the default for all workers (this has
> >>>>> stealthily been the case on amp-jenkins-worker-01 for the past two
> weeks,
> >>>>> and i've noticed no test failures)
> >>>>> - you can now use anaconda environments to specify which version of
> >>>>> python to use in your tests: http://www.continuum.io/blog/conda
> >>>>>
> >>>>> * installation of new python 2.7 modules: pymongo requests six
> pymongo
> >>>>> requests six python-crontab
> >>>>>
> >>>>> * bare-bones mongodb installation on all workers
> >>>>>
> >>>>> * installation of java 1.6 and 1.8 internal to jenkins
> >>>>> - jobs will default to the system java, which is 1.7.0_75
> >>>>> - if you want to run your tests w/java 6 or 8, you can select the
> JDK
> >>>>> version of your choice in the job configuration page (it'll be
> towards the
> >>>>> top)
> >>>>>
> >>>>> these changes have actually all been tested against a variety of
> builds
> >>>>> (yay staging!) and while i'm certain that i have all of the kinks
> worked
> >>>>> out, i'm going to schedule a longer downtime so that i have a chance
> to
> >>>>> identify and squash any problems that surface.
> >>>>>
> >>>>> thanks to josh rosen, k. shankari and davies liu for helping me test
> >>>>> all of this and get it working.
> >>>>>
> >>>>> shane
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
Re: extended jenkins downtime, thursday april 9th 7am-noon PDT
(moving to anaconda python & more)
Posted by Davies Liu <da...@databricks.com>.
Hey Shane,
Have you updated all the jenkins slaves?
There is a run with old configurations (no Python 3, with 130 minutes
timeout), see https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/666/consoleFull
Davies
On Thu, Apr 9, 2015 at 10:18 AM, shane knapp <sk...@berkeley.edu> wrote:
> ok, we're looking good. i'll keep an eye on this for the rest of the day,
> and if you happen to notice any infrastructure failures before i do (i
> updated a LOT), please let me know immediately! :)
>
> On Thu, Apr 9, 2015 at 8:38 AM, shane knapp <sk...@berkeley.edu> wrote:
>
>> things are looking pretty good and i expect to be done within an hour.
>> i've got some test builds running right now, and will give the green light
>> when they successfully complete.
>>
>> On Thu, Apr 9, 2015 at 7:29 AM, shane knapp <sk...@berkeley.edu> wrote:
>>
>>> and this is now happening.
>>>
>>> On Tue, Apr 7, 2015 at 4:38 PM, shane knapp <sk...@berkeley.edu> wrote:
>>>
>>>> reminder! this is happening thurday morning.
>>>>
>>>> On Fri, Apr 3, 2015 at 9:59 AM, shane knapp <sk...@berkeley.edu> wrote:
>>>>
>>>>> welcome to python2.7+, java 8 and more! :)
>>>>>
>>>>> i'll be doing a major upgrade to our build system next thursday
>>>>> morning. here's a quick list of what's going on:
>>>>>
>>>>> * installation of anaconda python on all worker nodes
>>>>>
>>>>> * installation of pypy 2.5.1 (python 2.7) on all nodes
>>>>>
>>>>> * matching installation of python modules for the current system python
>>>>> (2.6), and anaconda python (2.6, 2.7 and 3.4)
>>>>> - anaconda python 2.7 will be the default for all workers (this has
>>>>> stealthily been the case on amp-jenkins-worker-01 for the past two weeks,
>>>>> and i've noticed no test failures)
>>>>> - you can now use anaconda environments to specify which version of
>>>>> python to use in your tests: http://www.continuum.io/blog/conda
>>>>>
>>>>> * installation of new python 2.7 modules: pymongo requests six pymongo
>>>>> requests six python-crontab
>>>>>
>>>>> * bare-bones mongodb installation on all workers
>>>>>
>>>>> * installation of java 1.6 and 1.8 internal to jenkins
>>>>> - jobs will default to the system java, which is 1.7.0_75
>>>>> - if you want to run your tests w/java 6 or 8, you can select the JDK
>>>>> version of your choice in the job configuration page (it'll be towards the
>>>>> top)
>>>>>
>>>>> these changes have actually all been tested against a variety of builds
>>>>> (yay staging!) and while i'm certain that i have all of the kinks worked
>>>>> out, i'm going to schedule a longer downtime so that i have a chance to
>>>>> identify and squash any problems that surface.
>>>>>
>>>>> thanks to josh rosen, k. shankari and davies liu for helping me test
>>>>> all of this and get it working.
>>>>>
>>>>> shane
>>>>>
>>>>
>>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: extended jenkins downtime, thursday april 9th 7am-noon PDT
(moving to anaconda python & more)
Posted by shane knapp <sk...@berkeley.edu>.
ok, we're looking good. i'll keep an eye on this for the rest of the day,
and if you happen to notice any infrastructure failures before i do (i
updated a LOT), please let me know immediately! :)
On Thu, Apr 9, 2015 at 8:38 AM, shane knapp <sk...@berkeley.edu> wrote:
> things are looking pretty good and i expect to be done within an hour.
> i've got some test builds running right now, and will give the green light
> when they successfully complete.
>
> On Thu, Apr 9, 2015 at 7:29 AM, shane knapp <sk...@berkeley.edu> wrote:
>
>> and this is now happening.
>>
>> On Tue, Apr 7, 2015 at 4:38 PM, shane knapp <sk...@berkeley.edu> wrote:
>>
>>> reminder! this is happening thurday morning.
>>>
>>> On Fri, Apr 3, 2015 at 9:59 AM, shane knapp <sk...@berkeley.edu> wrote:
>>>
>>>> welcome to python2.7+, java 8 and more! :)
>>>>
>>>> i'll be doing a major upgrade to our build system next thursday
>>>> morning. here's a quick list of what's going on:
>>>>
>>>> * installation of anaconda python on all worker nodes
>>>>
>>>> * installation of pypy 2.5.1 (python 2.7) on all nodes
>>>>
>>>> * matching installation of python modules for the current system python
>>>> (2.6), and anaconda python (2.6, 2.7 and 3.4)
>>>> - anaconda python 2.7 will be the default for all workers (this has
>>>> stealthily been the case on amp-jenkins-worker-01 for the past two weeks,
>>>> and i've noticed no test failures)
>>>> - you can now use anaconda environments to specify which version of
>>>> python to use in your tests: http://www.continuum.io/blog/conda
>>>>
>>>> * installation of new python 2.7 modules: pymongo requests six pymongo
>>>> requests six python-crontab
>>>>
>>>> * bare-bones mongodb installation on all workers
>>>>
>>>> * installation of java 1.6 and 1.8 internal to jenkins
>>>> - jobs will default to the system java, which is 1.7.0_75
>>>> - if you want to run your tests w/java 6 or 8, you can select the JDK
>>>> version of your choice in the job configuration page (it'll be towards the
>>>> top)
>>>>
>>>> these changes have actually all been tested against a variety of builds
>>>> (yay staging!) and while i'm certain that i have all of the kinks worked
>>>> out, i'm going to schedule a longer downtime so that i have a chance to
>>>> identify and squash any problems that surface.
>>>>
>>>> thanks to josh rosen, k. shankari and davies liu for helping me test
>>>> all of this and get it working.
>>>>
>>>> shane
>>>>
>>>
>>>
>>
>
Re: extended jenkins downtime, thursday april 9th 7am-noon PDT
(moving to anaconda python & more)
Posted by shane knapp <sk...@berkeley.edu>.
things are looking pretty good and i expect to be done within an hour.
i've got some test builds running right now, and will give the green light
when they successfully complete.
On Thu, Apr 9, 2015 at 7:29 AM, shane knapp <sk...@berkeley.edu> wrote:
> and this is now happening.
>
> On Tue, Apr 7, 2015 at 4:38 PM, shane knapp <sk...@berkeley.edu> wrote:
>
>> reminder! this is happening thurday morning.
>>
>> On Fri, Apr 3, 2015 at 9:59 AM, shane knapp <sk...@berkeley.edu> wrote:
>>
>>> welcome to python2.7+, java 8 and more! :)
>>>
>>> i'll be doing a major upgrade to our build system next thursday morning.
>>> here's a quick list of what's going on:
>>>
>>> * installation of anaconda python on all worker nodes
>>>
>>> * installation of pypy 2.5.1 (python 2.7) on all nodes
>>>
>>> * matching installation of python modules for the current system python
>>> (2.6), and anaconda python (2.6, 2.7 and 3.4)
>>> - anaconda python 2.7 will be the default for all workers (this has
>>> stealthily been the case on amp-jenkins-worker-01 for the past two weeks,
>>> and i've noticed no test failures)
>>> - you can now use anaconda environments to specify which version of
>>> python to use in your tests: http://www.continuum.io/blog/conda
>>>
>>> * installation of new python 2.7 modules: pymongo requests six pymongo
>>> requests six python-crontab
>>>
>>> * bare-bones mongodb installation on all workers
>>>
>>> * installation of java 1.6 and 1.8 internal to jenkins
>>> - jobs will default to the system java, which is 1.7.0_75
>>> - if you want to run your tests w/java 6 or 8, you can select the JDK
>>> version of your choice in the job configuration page (it'll be towards the
>>> top)
>>>
>>> these changes have actually all been tested against a variety of builds
>>> (yay staging!) and while i'm certain that i have all of the kinks worked
>>> out, i'm going to schedule a longer downtime so that i have a chance to
>>> identify and squash any problems that surface.
>>>
>>> thanks to josh rosen, k. shankari and davies liu for helping me test all
>>> of this and get it working.
>>>
>>> shane
>>>
>>
>>
>
Re: extended jenkins downtime, thursday april 9th 7am-noon PDT
(moving to anaconda python & more)
Posted by shane knapp <sk...@berkeley.edu>.
and this is now happening.
On Tue, Apr 7, 2015 at 4:38 PM, shane knapp <sk...@berkeley.edu> wrote:
> reminder! this is happening thurday morning.
>
> On Fri, Apr 3, 2015 at 9:59 AM, shane knapp <sk...@berkeley.edu> wrote:
>
>> welcome to python2.7+, java 8 and more! :)
>>
>> i'll be doing a major upgrade to our build system next thursday morning.
>> here's a quick list of what's going on:
>>
>> * installation of anaconda python on all worker nodes
>>
>> * installation of pypy 2.5.1 (python 2.7) on all nodes
>>
>> * matching installation of python modules for the current system python
>> (2.6), and anaconda python (2.6, 2.7 and 3.4)
>> - anaconda python 2.7 will be the default for all workers (this has
>> stealthily been the case on amp-jenkins-worker-01 for the past two weeks,
>> and i've noticed no test failures)
>> - you can now use anaconda environments to specify which version of
>> python to use in your tests: http://www.continuum.io/blog/conda
>>
>> * installation of new python 2.7 modules: pymongo requests six pymongo
>> requests six python-crontab
>>
>> * bare-bones mongodb installation on all workers
>>
>> * installation of java 1.6 and 1.8 internal to jenkins
>> - jobs will default to the system java, which is 1.7.0_75
>> - if you want to run your tests w/java 6 or 8, you can select the JDK
>> version of your choice in the job configuration page (it'll be towards the
>> top)
>>
>> these changes have actually all been tested against a variety of builds
>> (yay staging!) and while i'm certain that i have all of the kinks worked
>> out, i'm going to schedule a longer downtime so that i have a chance to
>> identify and squash any problems that surface.
>>
>> thanks to josh rosen, k. shankari and davies liu for helping me test all
>> of this and get it working.
>>
>> shane
>>
>
>
Re: extended jenkins downtime, thursday april 9th 7am-noon PDT
(moving to anaconda python & more)
Posted by shane knapp <sk...@berkeley.edu>.
reminder! this is happening thurday morning.
On Fri, Apr 3, 2015 at 9:59 AM, shane knapp <sk...@berkeley.edu> wrote:
> welcome to python2.7+, java 8 and more! :)
>
> i'll be doing a major upgrade to our build system next thursday morning.
> here's a quick list of what's going on:
>
> * installation of anaconda python on all worker nodes
>
> * installation of pypy 2.5.1 (python 2.7) on all nodes
>
> * matching installation of python modules for the current system python
> (2.6), and anaconda python (2.6, 2.7 and 3.4)
> - anaconda python 2.7 will be the default for all workers (this has
> stealthily been the case on amp-jenkins-worker-01 for the past two weeks,
> and i've noticed no test failures)
> - you can now use anaconda environments to specify which version of
> python to use in your tests: http://www.continuum.io/blog/conda
>
> * installation of new python 2.7 modules: pymongo requests six pymongo
> requests six python-crontab
>
> * bare-bones mongodb installation on all workers
>
> * installation of java 1.6 and 1.8 internal to jenkins
> - jobs will default to the system java, which is 1.7.0_75
> - if you want to run your tests w/java 6 or 8, you can select the JDK
> version of your choice in the job configuration page (it'll be towards the
> top)
>
> these changes have actually all been tested against a variety of builds
> (yay staging!) and while i'm certain that i have all of the kinks worked
> out, i'm going to schedule a longer downtime so that i have a chance to
> identify and squash any problems that surface.
>
> thanks to josh rosen, k. shankari and davies liu for helping me test all
> of this and get it working.
>
> shane
>