You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Erik Erlandson <ee...@redhat.com> on 2018/08/15 19:33:46 UTC
[DISCUSS] SparkR support on k8s back-end for Spark 2.4
The SparkR support PR is finished, along with integration testing, however
Shane has requested that the integration testing not be enabled until after
the 2.4 release because it requires the OS updates he wants to test *after*
the release.
The integration testing can be run locally, and so the question at hand is:
would the PMC be willing to consider inclusion of the SparkR for 2.4, based
on local verification of the testing? The PySpark PR was merged under
similar circumstances: the testing was verified locally and the PR was
merged before the testing was enabled for jenkins.
Cheers,
Erik
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by Ilan Filonenko <if...@cornell.edu>.
The SparkR support PR includes integration testing that can be tested on a
local Minikube instance by merely running the distribution with appropriate
flags (--r) and running the integration-tests similarly to as you would on
any k8s test. Maybe some others could locally test this, if there is any
hesitation. I, for one, don't see a problem with merging as is and having
the Jenkins build be configured for R distribution support via the Ubuntu
update when Shane gets to it, similar to as we did for PySpark. The PR can
be found here <https://github.com/apache/spark/pull/21584>.
Furthermore, I am wondering if there is any desire to ensure SparkR support
(for k8s) is merged by 2.4 by the community, and if so, then maybe merging
this without the Jenkins build seems even more appropriate.
Best,
Ilan Filonenko
On Wed, Aug 15, 2018 at 3:33 PM, Erik Erlandson <ee...@redhat.com> wrote:
> The SparkR support PR is finished, along with integration testing, however
> Shane has requested that the integration testing not be enabled until after
> the 2.4 release because it requires the OS updates he wants to test *after*
> the release.
>
> The integration testing can be run locally, and so the question at hand
> is: would the PMC be willing to consider inclusion of the SparkR for 2.4,
> based on local verification of the testing? The PySpark PR was merged under
> similar circumstances: the testing was verified locally and the PR was
> merged before the testing was enabled for jenkins.
>
> Cheers,
> Erik
>
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by Ilan Filonenko <if...@cornell.edu>.
Correct, the OS change and updates would require more testing, from what
Shane has told me, and could potentially surface some issue that could
delay a major release.
So yes, the release manager would need to run the tests manually and after
the release we would switch to a fully integrated Jenkins that would run
those same tests on the newly updated workers.
On Wed, Aug 15, 2018 at 3:47 PM, Reynold Xin <rx...@databricks.com> wrote:
> Personally I'd love for R support to be in 2.4, but I don't consider
> something "Done" unless tests are running ... Is the proposal: the release
> manager manually run the R tests when preparing the release, and switch
> over to fully integrated Jenkins after 2.4.0 is released?
>
> On Wed, Aug 15, 2018 at 2:45 PM Reynold Xin <rx...@databricks.com> wrote:
>
>> What's the reason we don't want to do the OS updates right now? Is it due
>> to the unpredictability of potential issues that might happen and end up
>> delaying 2.4 release?
>>
>>
>> On Wed, Aug 15, 2018 at 2:33 PM Erik Erlandson <ee...@redhat.com>
>> wrote:
>>
>>> The SparkR support PR is finished, along with integration testing,
>>> however Shane has requested that the integration testing not be enabled
>>> until after the 2.4 release because it requires the OS updates he wants to
>>> test *after* the release.
>>>
>>> The integration testing can be run locally, and so the question at hand
>>> is: would the PMC be willing to consider inclusion of the SparkR for 2.4,
>>> based on local verification of the testing? The PySpark PR was merged under
>>> similar circumstances: the testing was verified locally and the PR was
>>> merged before the testing was enabled for jenkins.
>>>
>>> Cheers,
>>> Erik
>>>
>>
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by Reynold Xin <rx...@databricks.com>.
Personally I'd love for R support to be in 2.4, but I don't consider
something "Done" unless tests are running ... Is the proposal: the release
manager manually run the R tests when preparing the release, and switch
over to fully integrated Jenkins after 2.4.0 is released?
On Wed, Aug 15, 2018 at 2:45 PM Reynold Xin <rx...@databricks.com> wrote:
> What's the reason we don't want to do the OS updates right now? Is it due
> to the unpredictability of potential issues that might happen and end up
> delaying 2.4 release?
>
>
> On Wed, Aug 15, 2018 at 2:33 PM Erik Erlandson <ee...@redhat.com>
> wrote:
>
>> The SparkR support PR is finished, along with integration testing,
>> however Shane has requested that the integration testing not be enabled
>> until after the 2.4 release because it requires the OS updates he wants to
>> test *after* the release.
>>
>> The integration testing can be run locally, and so the question at hand
>> is: would the PMC be willing to consider inclusion of the SparkR for 2.4,
>> based on local verification of the testing? The PySpark PR was merged under
>> similar circumstances: the testing was verified locally and the PR was
>> merged before the testing was enabled for jenkins.
>>
>> Cheers,
>> Erik
>>
>
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by Felix Cheung <fe...@hotmail.com>.
I fully support merging that SparkR support on k8s.
If Ilan and other are willing to manually validate the RC I’m happy to voucher for it (I’m not 100% sure i have capacity to test it that way but certainly will try)
Also +1 on revisiting Jenkins builds. tbh I’m not sure we depend on them too much during the release process due to reason Marcelo listed, for the last 4+ releases...
________________________________
From: Ilan Filonenko <if...@cornell.edu>
Sent: Thursday, August 16, 2018 10:27 AM
To: shane knapp
Cc: Erik Erlandson; Wenchen Fan; Marcelo Vanzin; Reynold?Xin; Spark dev list
Subject: Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Okay, if there is a consensus in merging the SparkR feature and its respective tests separately, I will split up the current PR to allow for the feature to be merged while the e2e tests to be used for locally testing until we are ready to merge (similar to PySpark). Will give it another day to hear other opinions, but otherwise working under the impression that the OS upgrade will not happen before the 2.4 cut.
On Thu, Aug 16, 2018 at 12:53 PM, shane knapp <sk...@berkeley.edu>> wrote:
On Thu, Aug 16, 2018 at 9:49 AM, Erik Erlandson <ee...@redhat.com>> wrote:
IMO sparkR support makes sense to merge for 2.4, as long as the release wranglers agree that local integration testing is sufficiently convincing.
i agree w/this. testing for this stuff specifically will happen within a couple of weeks after the 2.4 cut.
Part of the intent here is to allow this to happen without Shane having to reorganize his complex upgrade schedule and make it even more complicated.
this. exactly. :)
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by Ilan Filonenko <if...@cornell.edu>.
Okay, if there is a consensus in merging the SparkR feature and its
respective tests separately, I will split up the current PR to allow for
the feature to be merged while the e2e tests to be used for locally testing
until we are ready to merge (similar to PySpark). Will give it another day
to hear other opinions, but otherwise working under the impression that the
OS upgrade will not happen before the 2.4 cut.
On Thu, Aug 16, 2018 at 12:53 PM, shane knapp <sk...@berkeley.edu> wrote:
> On Thu, Aug 16, 2018 at 9:49 AM, Erik Erlandson <ee...@redhat.com>
> wrote:
>
>> IMO sparkR support makes sense to merge for 2.4, as long as the release
>> wranglers agree that local integration testing is sufficiently convincing.
>>
>
> i agree w/this. testing for this stuff specifically will happen within a
> couple of weeks after the 2.4 cut.
>
>
>> Part of the intent here is to allow this to happen without Shane having
>> to reorganize his complex upgrade schedule and make it even more
>> complicated.
>>
>> this. exactly. :)
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by shane knapp <sk...@berkeley.edu>.
On Thu, Aug 16, 2018 at 9:49 AM, Erik Erlandson <ee...@redhat.com> wrote:
> IMO sparkR support makes sense to merge for 2.4, as long as the release
> wranglers agree that local integration testing is sufficiently convincing.
>
i agree w/this. testing for this stuff specifically will happen within a
couple of weeks after the 2.4 cut.
> Part of the intent here is to allow this to happen without Shane having to
> reorganize his complex upgrade schedule and make it even more complicated.
>
> this. exactly. :)
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by Erik Erlandson <ee...@redhat.com>.
IMO sparkR support makes sense to merge for 2.4, as long as the release
wranglers agree that local integration testing is sufficiently convincing.
Part of the intent here is to allow this to happen without Shane having to
reorganize his complex upgrade schedule and make it even more complicated.
On Wed, Aug 15, 2018 at 7:08 PM, Wenchen Fan <cl...@gmail.com> wrote:
> I'm also happy to see we have R support on k8s for Spark 2.4. I'll do the
> manual testing for it if we don't want to upgrade the OS now. If the Python
> support is also merged in this way, I think we can merge the R support PR
> too?
>
> On Thu, Aug 16, 2018 at 7:23 AM shane knapp <sk...@berkeley.edu> wrote:
>
>>
>>> What is the current purpose of these builds?
>>>
>>> to be honest, i have absolutely no idea. :)
>>
>> these were set up a long time ago, in a galaxy far far away, by someone
>> who is not me.
>>
>>
>>> - spark-docs seems to be building the docs, is that the only place
>>> where the docs build is tested?
>>>
>>> i think so...
>>
>>
>>> In the last many releases we've moved away from using jenkins jobs for
>>> preparing the packages, and the scripts have changed a lot to be
>>> friendlier to people running them locally (they even support docker
>>> now, and have flags to run "test" builds that don't require
>>> credentials such as GPG keys).
>>>
>>> awesome++
>>
>>
>>> Perhaps we should think about revamping these jobs instead of keeping
>>> them as is.
>>>
>>
>> i fully support this. which is exactly why i punted on even trying to
>> get them ported over to the ubuntu nodes.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by Wenchen Fan <cl...@gmail.com>.
I'm also happy to see we have R support on k8s for Spark 2.4. I'll do the
manual testing for it if we don't want to upgrade the OS now. If the Python
support is also merged in this way, I think we can merge the R support PR
too?
On Thu, Aug 16, 2018 at 7:23 AM shane knapp <sk...@berkeley.edu> wrote:
>
>> What is the current purpose of these builds?
>>
>> to be honest, i have absolutely no idea. :)
>
> these were set up a long time ago, in a galaxy far far away, by someone
> who is not me.
>
>
>> - spark-docs seems to be building the docs, is that the only place
>> where the docs build is tested?
>>
>> i think so...
>
>
>> In the last many releases we've moved away from using jenkins jobs for
>> preparing the packages, and the scripts have changed a lot to be
>> friendlier to people running them locally (they even support docker
>> now, and have flags to run "test" builds that don't require
>> credentials such as GPG keys).
>>
>> awesome++
>
>
>> Perhaps we should think about revamping these jobs instead of keeping
>> them as is.
>>
>
> i fully support this. which is exactly why i punted on even trying to get
> them ported over to the ubuntu nodes.
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by shane knapp <sk...@berkeley.edu>.
>
>
> What is the current purpose of these builds?
>
> to be honest, i have absolutely no idea. :)
these were set up a long time ago, in a galaxy far far away, by someone who
is not me.
> - spark-docs seems to be building the docs, is that the only place
> where the docs build is tested?
>
> i think so...
> In the last many releases we've moved away from using jenkins jobs for
> preparing the packages, and the scripts have changed a lot to be
> friendlier to people running them locally (they even support docker
> now, and have flags to run "test" builds that don't require
> credentials such as GPG keys).
>
> awesome++
> Perhaps we should think about revamping these jobs instead of keeping
> them as is.
>
i fully support this. which is exactly why i punted on even trying to get
them ported over to the ubuntu nodes.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by Marcelo Vanzin <va...@cloudera.com.INVALID>.
On Wed, Aug 15, 2018 at 1:35 PM, shane knapp <sk...@berkeley.edu> wrote:
> in fact, i don't see us getting rid of all of the centos machines until EOY
> (see my above comment, re docs, release etc). these are the builds that
> will remain on centos for the near future:
> https://rise.cs.berkeley.edu/jenkins/label/spark-release/
> https://rise.cs.berkeley.edu/jenkins/label/spark-packaging/
> https://rise.cs.berkeley.edu/jenkins/label/spark-docs/
What is the current purpose of these builds?
- spark-release hasn't been triggered in a long time.
- spark-packaging seems to have been failing miserably for the last 10 months.
- spark-docs seems to be building the docs, is that the only place
where the docs build is tested?
In the last many releases we've moved away from using jenkins jobs for
preparing the packages, and the scripts have changed a lot to be
friendlier to people running them locally (they even support docker
now, and have flags to run "test" builds that don't require
credentials such as GPG keys).
Perhaps we should think about revamping these jobs instead of keeping
them as is.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by shane knapp <sk...@berkeley.edu>.
On Wed, Aug 15, 2018 at 12:45 PM, Reynold Xin <rx...@databricks.com> wrote:
> What's the reason we don't want to do the OS updates right now? Is it due
> to the unpredictability of potential issues that might happen and end up
> delaying 2.4 release?
>
> that is exactly it... i haven't had a chance to test everything (esp
docs, release builds, etc), and since there have been many changes pushed
to various branches, some builds are failing in different ways on centos vs
ubuntu. things are very, very fluid right now.
we are SUPER close to being able to move the vast majority of builds to
ubuntu, but even after than happens there will almost definitely be a
non-zero amount of babysitting and debugging that needs to happen. all of
that babysitting and debugging would potentially delay the 2.4 cut even
more.
in fact, i don't see us getting rid of all of the centos machines until EOY
(see my above comment, re docs, release etc). these are the builds that
will remain on centos for the near future:
https://rise.cs.berkeley.edu/jenkins/label/spark-release/
https://rise.cs.berkeley.edu/jenkins/label/spark-packaging/
https://rise.cs.berkeley.edu/jenkins/label/spark-docs/
these all run on amp-jenkins-worker-01, which will remain untouched.
HOWEVER:
i'm 99%+ certain that i can port the PRB builder over to ubuntu w/a pared
down python 3.5 installation. i have a build running now (
https://rise.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/74/)
and if it behaves appropriately (meaning testing all the right things
w/py35), i should be able to get two more non-critical centos build nodes
reinstalled w/ubuntu and deployed by EOW. this will give us enough jenkins
executors and allow PRB throughput to not decrease.
if the dev community is ok w/taking this plunge, i will make it happen.
shane (who wants everyone to remember that it's just little old me running
this... not a team of people) ;)
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
Posted by Reynold Xin <rx...@databricks.com>.
What's the reason we don't want to do the OS updates right now? Is it due
to the unpredictability of potential issues that might happen and end up
delaying 2.4 release?
On Wed, Aug 15, 2018 at 2:33 PM Erik Erlandson <ee...@redhat.com> wrote:
> The SparkR support PR is finished, along with integration testing, however
> Shane has requested that the integration testing not be enabled until after
> the 2.4 release because it requires the OS updates he wants to test *after*
> the release.
>
> The integration testing can be run locally, and so the question at hand
> is: would the PMC be willing to consider inclusion of the SparkR for 2.4,
> based on local verification of the testing? The PySpark PR was merged under
> similar circumstances: the testing was verified locally and the PR was
> merged before the testing was enabled for jenkins.
>
> Cheers,
> Erik
>