You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by shane knapp <sk...@berkeley.edu> on 2019/04/10 18:37:06 UTC
[SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all
active branches
details here (see most recent comments for current state of things):
https://issues.apache.org/jira/browse/SPARK-25079
my PR for these changes:
https://github.com/apache/spark/pull/24266
we're doing this because we need to support arrow 0.12.1:
https://issues.apache.org/jira/browse/SPARK-27276
a couple of things:
* i won't be switching things over until april 11th or 12th at the earliest.
* this change will require a short (~15-20min) downtime to switch over
from 3.4 to 3.6.
* updating python will impact all active branches, so i will need to
backport this pr: https://github.com/apache/spark/pull/24266
question: which other branches should i be applying this change to?
definitely master and 2.4... also 2.3? 2.2?
thanks in advance,
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all
active branches
Posted by Bryan Cutler <cu...@gmail.com>.
Great work, thanks Shane!
On Thu, Apr 18, 2019 at 2:46 PM shane knapp <sk...@berkeley.edu> wrote:
> alrighty folks, the future is here and we'll be moving to python 3.6
> monday!
>
> all three PRs are green!
> master PR: https://github.com/apache/spark/pull/24266
> 2.4 PR: https://github.com/apache/spark/pull/24379
> 2.3 PR: https://github.com/apache/spark/pull/24380
>
> more detailed email coming out this afternoon about the upgrade.
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all
active branches
Posted by shane knapp <sk...@berkeley.edu>.
alrighty folks, the future is here and we'll be moving to python 3.6 monday!
all three PRs are green!
master PR: https://github.com/apache/spark/pull/24266
2.4 PR: https://github.com/apache/spark/pull/24379
2.3 PR: https://github.com/apache/spark/pull/24380
more detailed email coming out this afternoon about the upgrade.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all
active branches
Posted by shane knapp <sk...@berkeley.edu>.
2.4 PR: https://github.com/apache/spark/pull/24379
2.3 PR: https://github.com/apache/spark/pull/24380
both of these branches failed pretty spectacularly during my non-PR testing
in the pyspark sql tests, but let's see how they fare when things are run
automagically by jenkins.
shane
On Mon, Apr 15, 2019 at 11:49 AM shane knapp <sk...@berkeley.edu> wrote:
> 1) i absolutely do not want to test against more than two python
> versions. consider my foot to have been put down on that. :)
>
> 2) i'll start testing against 2.3 and 2.4 now (last week was a bit crazy,
> so i didn't get around to it).
>
> once i'm happy w/the 2.3 and 2.4 results, i'll follow up here and get the
> upgrade schedule posted.
>
> On Wed, Apr 10, 2019 at 3:57 PM Sean Owen <sr...@gmail.com> wrote:
>
>> My guess is that branch 2.3, 2.4 still work with Python 3.6 even if
>> we're not testing that. If we're only testing against 3.6, I guess we
>> wouldn't detect if something broke for 3.4 which is still 'supported'
>> in 2.3.x, 2.4.x? For 3.x, no problem, let's raise the minimum anyway.
>> This could be fine if we accept that small risk and it keeps the
>> testing env from getting crazy; I don't actually want to test vs N
>> Python versions anyway.
>>
>> On Wed, Apr 10, 2019 at 4:49 PM shane knapp <sk...@berkeley.edu> wrote:
>> >
>> > ok, i'll create a PR for each branch so we can test things that way,
>> rather than backporting.
>> >
>> > the bigger question i have is how far back, branch-wise, are we willing
>> to support w/regards to tests?
>> >
>> > On Wed, Apr 10, 2019 at 12:16 PM Sean Owen <sr...@gmail.com> wrote:
>> >>
>> >> In theory Spark 2.4 supports Python 3.4; would this mean it's now just
>> >> tested vs 3.6? that's not out of the question, but can the older
>> >> branches continue to test on older versions or is that super complex?
>> >>
>> >> On Wed, Apr 10, 2019 at 1:37 PM shane knapp <sk...@berkeley.edu>
>> wrote:
>> >> >
>> >> > details here (see most recent comments for current state of things):
>> >> > https://issues.apache.org/jira/browse/SPARK-25079
>> >> >
>> >> > my PR for these changes:
>> >> > https://github.com/apache/spark/pull/24266
>> >> >
>> >> > we're doing this because we need to support arrow 0.12.1:
>> >> > https://issues.apache.org/jira/browse/SPARK-27276
>> >> >
>> >> > a couple of things:
>> >> >
>> >> > * i won't be switching things over until april 11th or 12th at the
>> earliest.
>> >> > * this change will require a short (~15-20min) downtime to switch
>> over from 3.4 to 3.6.
>> >> > * updating python will impact all active branches, so i will need
>> to backport this pr: https://github.com/apache/spark/pull/24266
>> >> >
>> >> > question: which other branches should i be applying this change
>> to? definitely master and 2.4... also 2.3? 2.2?
>> >> >
>> >> > thanks in advance,
>> >> >
>> >> > shane
>> >> > --
>> >> > Shane Knapp
>> >> > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> >> > https://rise.cs.berkeley.edu
>> >
>> >
>> >
>> > --
>> > Shane Knapp
>> > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > https://rise.cs.berkeley.edu
>>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all
active branches
Posted by shane knapp <sk...@berkeley.edu>.
1) i absolutely do not want to test against more than two python
versions. consider my foot to have been put down on that. :)
2) i'll start testing against 2.3 and 2.4 now (last week was a bit crazy,
so i didn't get around to it).
once i'm happy w/the 2.3 and 2.4 results, i'll follow up here and get the
upgrade schedule posted.
On Wed, Apr 10, 2019 at 3:57 PM Sean Owen <sr...@gmail.com> wrote:
> My guess is that branch 2.3, 2.4 still work with Python 3.6 even if
> we're not testing that. If we're only testing against 3.6, I guess we
> wouldn't detect if something broke for 3.4 which is still 'supported'
> in 2.3.x, 2.4.x? For 3.x, no problem, let's raise the minimum anyway.
> This could be fine if we accept that small risk and it keeps the
> testing env from getting crazy; I don't actually want to test vs N
> Python versions anyway.
>
> On Wed, Apr 10, 2019 at 4:49 PM shane knapp <sk...@berkeley.edu> wrote:
> >
> > ok, i'll create a PR for each branch so we can test things that way,
> rather than backporting.
> >
> > the bigger question i have is how far back, branch-wise, are we willing
> to support w/regards to tests?
> >
> > On Wed, Apr 10, 2019 at 12:16 PM Sean Owen <sr...@gmail.com> wrote:
> >>
> >> In theory Spark 2.4 supports Python 3.4; would this mean it's now just
> >> tested vs 3.6? that's not out of the question, but can the older
> >> branches continue to test on older versions or is that super complex?
> >>
> >> On Wed, Apr 10, 2019 at 1:37 PM shane knapp <sk...@berkeley.edu>
> wrote:
> >> >
> >> > details here (see most recent comments for current state of things):
> >> > https://issues.apache.org/jira/browse/SPARK-25079
> >> >
> >> > my PR for these changes:
> >> > https://github.com/apache/spark/pull/24266
> >> >
> >> > we're doing this because we need to support arrow 0.12.1:
> >> > https://issues.apache.org/jira/browse/SPARK-27276
> >> >
> >> > a couple of things:
> >> >
> >> > * i won't be switching things over until april 11th or 12th at the
> earliest.
> >> > * this change will require a short (~15-20min) downtime to switch
> over from 3.4 to 3.6.
> >> > * updating python will impact all active branches, so i will need to
> backport this pr: https://github.com/apache/spark/pull/24266
> >> >
> >> > question: which other branches should i be applying this change to?
> definitely master and 2.4... also 2.3? 2.2?
> >> >
> >> > thanks in advance,
> >> >
> >> > shane
> >> > --
> >> > Shane Knapp
> >> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> >> > https://rise.cs.berkeley.edu
> >
> >
> >
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all
active branches
Posted by Sean Owen <sr...@gmail.com>.
My guess is that branch 2.3, 2.4 still work with Python 3.6 even if
we're not testing that. If we're only testing against 3.6, I guess we
wouldn't detect if something broke for 3.4 which is still 'supported'
in 2.3.x, 2.4.x? For 3.x, no problem, let's raise the minimum anyway.
This could be fine if we accept that small risk and it keeps the
testing env from getting crazy; I don't actually want to test vs N
Python versions anyway.
On Wed, Apr 10, 2019 at 4:49 PM shane knapp <sk...@berkeley.edu> wrote:
>
> ok, i'll create a PR for each branch so we can test things that way, rather than backporting.
>
> the bigger question i have is how far back, branch-wise, are we willing to support w/regards to tests?
>
> On Wed, Apr 10, 2019 at 12:16 PM Sean Owen <sr...@gmail.com> wrote:
>>
>> In theory Spark 2.4 supports Python 3.4; would this mean it's now just
>> tested vs 3.6? that's not out of the question, but can the older
>> branches continue to test on older versions or is that super complex?
>>
>> On Wed, Apr 10, 2019 at 1:37 PM shane knapp <sk...@berkeley.edu> wrote:
>> >
>> > details here (see most recent comments for current state of things):
>> > https://issues.apache.org/jira/browse/SPARK-25079
>> >
>> > my PR for these changes:
>> > https://github.com/apache/spark/pull/24266
>> >
>> > we're doing this because we need to support arrow 0.12.1:
>> > https://issues.apache.org/jira/browse/SPARK-27276
>> >
>> > a couple of things:
>> >
>> > * i won't be switching things over until april 11th or 12th at the earliest.
>> > * this change will require a short (~15-20min) downtime to switch over from 3.4 to 3.6.
>> > * updating python will impact all active branches, so i will need to backport this pr: https://github.com/apache/spark/pull/24266
>> >
>> > question: which other branches should i be applying this change to? definitely master and 2.4... also 2.3? 2.2?
>> >
>> > thanks in advance,
>> >
>> > shane
>> > --
>> > Shane Knapp
>> > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all
active branches
Posted by shane knapp <sk...@berkeley.edu>.
ok, i'll create a PR for each branch so we can test things that way, rather
than backporting.
the bigger question i have is how far back, branch-wise, are we willing to
support w/regards to tests?
On Wed, Apr 10, 2019 at 12:16 PM Sean Owen <sr...@gmail.com> wrote:
> In theory Spark 2.4 supports Python 3.4; would this mean it's now just
> tested vs 3.6? that's not out of the question, but can the older
> branches continue to test on older versions or is that super complex?
>
> On Wed, Apr 10, 2019 at 1:37 PM shane knapp <sk...@berkeley.edu> wrote:
> >
> > details here (see most recent comments for current state of things):
> > https://issues.apache.org/jira/browse/SPARK-25079
> >
> > my PR for these changes:
> > https://github.com/apache/spark/pull/24266
> >
> > we're doing this because we need to support arrow 0.12.1:
> > https://issues.apache.org/jira/browse/SPARK-27276
> >
> > a couple of things:
> >
> > * i won't be switching things over until april 11th or 12th at the
> earliest.
> > * this change will require a short (~15-20min) downtime to switch over
> from 3.4 to 3.6.
> > * updating python will impact all active branches, so i will need to
> backport this pr: https://github.com/apache/spark/pull/24266
> >
> > question: which other branches should i be applying this change to?
> definitely master and 2.4... also 2.3? 2.2?
> >
> > thanks in advance,
> >
> > shane
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all
active branches
Posted by Sean Owen <sr...@gmail.com>.
In theory Spark 2.4 supports Python 3.4; would this mean it's now just
tested vs 3.6? that's not out of the question, but can the older
branches continue to test on older versions or is that super complex?
On Wed, Apr 10, 2019 at 1:37 PM shane knapp <sk...@berkeley.edu> wrote:
>
> details here (see most recent comments for current state of things):
> https://issues.apache.org/jira/browse/SPARK-25079
>
> my PR for these changes:
> https://github.com/apache/spark/pull/24266
>
> we're doing this because we need to support arrow 0.12.1:
> https://issues.apache.org/jira/browse/SPARK-27276
>
> a couple of things:
>
> * i won't be switching things over until april 11th or 12th at the earliest.
> * this change will require a short (~15-20min) downtime to switch over from 3.4 to 3.6.
> * updating python will impact all active branches, so i will need to backport this pr: https://github.com/apache/spark/pull/24266
>
> question: which other branches should i be applying this change to? definitely master and 2.4... also 2.3? 2.2?
>
> thanks in advance,
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org