You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by shane knapp <sk...@berkeley.edu> on 2018/08/13 20:14:59 UTC
[discuss][minor] impending python 3.x jenkins upgrade... 3.5.x? 3.6.x?
hey everyone!
i was checking out the EOL/release cycle for python 3.5 and it looks like
we'll have 3.5.6 released in early 2019.
this got me to thinking: instead of 3.5, what about 3.6?
i looked around, and according to the 'docs' and 'collective wisdom of the
internets', 3.5 and 3.6 should be fully backwards-compatible w/3.4.
of course, this needs to be taken w/a grain of salt, as we're mostly
focused on actual python package requirements, rather than worrying about
core python functionality.
thoughts? comments?
thanks in advance,
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [discuss][minor] impending python 3.x jenkins upgrade... 3.5.x? 3.6.x?
Posted by shane knapp <sk...@berkeley.edu>.
i'll start my testing w/3.6 tomorrow.
On Mon, Aug 20, 2018 at 1:30 PM, Li Jin <ic...@gmail.com> wrote:
> Thanks for looking into this Shane. If we can only have a single python 3
> version, I agree 3.6 would be better than 3.5. Otherwise, ideally I think
> it would be nice to test all supported 3.x versions (latest micros should
> be fine).
>
> On Mon, Aug 20, 2018 at 7:07 PM shane knapp <sk...@berkeley.edu> wrote:
>
>> initially, i'd like to just choose one version to have the primary tests
>> against, but i'm also not opposed to supporting more of a matrix. the
>> biggest problem i see w/this approach, however, is that of build monitoring
>> and long-term ownership. this is why we have a relatively restrictive
>> current deployment.
>>
>> another thing i've been noticing during this project is that we have a
>> lot of flaky tests... for instance, i'm literally having every other build
>> fail on my (relatively) up-to-date PRB fork:
>> https://amplab.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/
>>
>> (i'm testing more than python here, otherwise i could just build a spark
>> distro and run the python tests against that)
>>
>> i'll also set up a 3.6 env tomorrow and start testing against that. i'm
>> pretty confident about 3.5, tho.
>>
>> shane
>>
>> On Mon, Aug 20, 2018 at 11:33 AM, Bryan Cutler <cu...@gmail.com> wrote:
>>
>>> Thanks for looking into this Shane! If we are choosing a single python
>>> 3.x, I think 3.6 would be good. It might still be nice to test against
>>> other versions too, so we can catch any issues. Is it possible to have more
>>> exhaustive testing as part of a nightly or scheduled build? As a point of
>>> reference for Python 3.6, Arrow is using this version for CI.
>>>
>>> Bryan
>>>
>>> On Sun, Aug 19, 2018 at 9:49 PM Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>>
>>>> Actually Python 3.7 is released (https://www.python.org/
>>>> downloads/release/python-370/) too and I fixed the compatibility
>>>> issues accordingly - https://github.com/apache/spark/pull/21714
>>>> There has been an issue for 3.6 (comparing to lower versions of Python
>>>> including 3.5) - https://github.com/apache/spark/pull/16429
>>>>
>>>> I am not yet sure what's the best matrix for it actually. In case of R,
>>>> we test lowest version in Jenkins and highest version via AppVeyor FWIW.
>>>> I don't have a strong preference opinion on this since we have been
>>>> having compatibility issues for each Python version.
>>>>
>>>>
>>>> 2018년 8월 14일 (화) 오전 4:15, shane knapp <sk...@berkeley.edu>님이 작성:
>>>>
>>>>> hey everyone!
>>>>>
>>>>> i was checking out the EOL/release cycle for python 3.5 and it looks
>>>>> like we'll have 3.5.6 released in early 2019.
>>>>>
>>>>> this got me to thinking: instead of 3.5, what about 3.6?
>>>>>
>>>>> i looked around, and according to the 'docs' and 'collective wisdom of
>>>>> the internets', 3.5 and 3.6 should be fully backwards-compatible w/3.4.
>>>>>
>>>>> of course, this needs to be taken w/a grain of salt, as we're mostly
>>>>> focused on actual python package requirements, rather than worrying about
>>>>> core python functionality.
>>>>>
>>>>> thoughts? comments?
>>>>>
>>>>> thanks in advance,
>>>>>
>>>>> shane
>>>>> --
>>>>> Shane Knapp
>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>> https://rise.cs.berkeley.edu
>>>>>
>>>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [discuss][minor] impending python 3.x jenkins upgrade... 3.5.x? 3.6.x?
Posted by Li Jin <ic...@gmail.com>.
Thanks for looking into this Shane. If we can only have a single python 3
version, I agree 3.6 would be better than 3.5. Otherwise, ideally I think
it would be nice to test all supported 3.x versions (latest micros should
be fine).
On Mon, Aug 20, 2018 at 7:07 PM shane knapp <sk...@berkeley.edu> wrote:
> initially, i'd like to just choose one version to have the primary tests
> against, but i'm also not opposed to supporting more of a matrix. the
> biggest problem i see w/this approach, however, is that of build monitoring
> and long-term ownership. this is why we have a relatively restrictive
> current deployment.
>
> another thing i've been noticing during this project is that we have a lot
> of flaky tests... for instance, i'm literally having every other build
> fail on my (relatively) up-to-date PRB fork:
> https://amplab.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/
>
> (i'm testing more than python here, otherwise i could just build a spark
> distro and run the python tests against that)
>
> i'll also set up a 3.6 env tomorrow and start testing against that. i'm
> pretty confident about 3.5, tho.
>
> shane
>
> On Mon, Aug 20, 2018 at 11:33 AM, Bryan Cutler <cu...@gmail.com> wrote:
>
>> Thanks for looking into this Shane! If we are choosing a single python
>> 3.x, I think 3.6 would be good. It might still be nice to test against
>> other versions too, so we can catch any issues. Is it possible to have more
>> exhaustive testing as part of a nightly or scheduled build? As a point of
>> reference for Python 3.6, Arrow is using this version for CI.
>>
>> Bryan
>>
>> On Sun, Aug 19, 2018 at 9:49 PM Hyukjin Kwon <gu...@gmail.com> wrote:
>>
>>> Actually Python 3.7 is released (
>>> https://www.python.org/downloads/release/python-370/) too and I fixed
>>> the compatibility issues accordingly -
>>> https://github.com/apache/spark/pull/21714
>>> There has been an issue for 3.6 (comparing to lower versions of Python
>>> including 3.5) - https://github.com/apache/spark/pull/16429
>>>
>>> I am not yet sure what's the best matrix for it actually. In case of R,
>>> we test lowest version in Jenkins and highest version via AppVeyor FWIW.
>>> I don't have a strong preference opinion on this since we have been
>>> having compatibility issues for each Python version.
>>>
>>>
>>> 2018년 8월 14일 (화) 오전 4:15, shane knapp <sk...@berkeley.edu>님이 작성:
>>>
>>>> hey everyone!
>>>>
>>>> i was checking out the EOL/release cycle for python 3.5 and it looks
>>>> like we'll have 3.5.6 released in early 2019.
>>>>
>>>> this got me to thinking: instead of 3.5, what about 3.6?
>>>>
>>>> i looked around, and according to the 'docs' and 'collective wisdom of
>>>> the internets', 3.5 and 3.6 should be fully backwards-compatible w/3.4.
>>>>
>>>> of course, this needs to be taken w/a grain of salt, as we're mostly
>>>> focused on actual python package requirements, rather than worrying about
>>>> core python functionality.
>>>>
>>>> thoughts? comments?
>>>>
>>>> thanks in advance,
>>>>
>>>> shane
>>>> --
>>>> Shane Knapp
>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>> https://rise.cs.berkeley.edu
>>>>
>>>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
Re: [discuss][minor] impending python 3.x jenkins upgrade... 3.5.x? 3.6.x?
Posted by shane knapp <sk...@berkeley.edu>.
initially, i'd like to just choose one version to have the primary tests
against, but i'm also not opposed to supporting more of a matrix. the
biggest problem i see w/this approach, however, is that of build monitoring
and long-term ownership. this is why we have a relatively restrictive
current deployment.
another thing i've been noticing during this project is that we have a lot
of flaky tests... for instance, i'm literally having every other build
fail on my (relatively) up-to-date PRB fork:
https://amplab.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/
(i'm testing more than python here, otherwise i could just build a spark
distro and run the python tests against that)
i'll also set up a 3.6 env tomorrow and start testing against that. i'm
pretty confident about 3.5, tho.
shane
On Mon, Aug 20, 2018 at 11:33 AM, Bryan Cutler <cu...@gmail.com> wrote:
> Thanks for looking into this Shane! If we are choosing a single python
> 3.x, I think 3.6 would be good. It might still be nice to test against
> other versions too, so we can catch any issues. Is it possible to have more
> exhaustive testing as part of a nightly or scheduled build? As a point of
> reference for Python 3.6, Arrow is using this version for CI.
>
> Bryan
>
> On Sun, Aug 19, 2018 at 9:49 PM Hyukjin Kwon <gu...@gmail.com> wrote:
>
>> Actually Python 3.7 is released (https://www.python.org/
>> downloads/release/python-370/) too and I fixed the compatibility issues
>> accordingly - https://github.com/apache/spark/pull/21714
>> There has been an issue for 3.6 (comparing to lower versions of Python
>> including 3.5) - https://github.com/apache/spark/pull/16429
>>
>> I am not yet sure what's the best matrix for it actually. In case of R,
>> we test lowest version in Jenkins and highest version via AppVeyor FWIW.
>> I don't have a strong preference opinion on this since we have been
>> having compatibility issues for each Python version.
>>
>>
>> 2018년 8월 14일 (화) 오전 4:15, shane knapp <sk...@berkeley.edu>님이 작성:
>>
>>> hey everyone!
>>>
>>> i was checking out the EOL/release cycle for python 3.5 and it looks
>>> like we'll have 3.5.6 released in early 2019.
>>>
>>> this got me to thinking: instead of 3.5, what about 3.6?
>>>
>>> i looked around, and according to the 'docs' and 'collective wisdom of
>>> the internets', 3.5 and 3.6 should be fully backwards-compatible w/3.4.
>>>
>>> of course, this needs to be taken w/a grain of salt, as we're mostly
>>> focused on actual python package requirements, rather than worrying about
>>> core python functionality.
>>>
>>> thoughts? comments?
>>>
>>> thanks in advance,
>>>
>>> shane
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: [discuss][minor] impending python 3.x jenkins upgrade... 3.5.x? 3.6.x?
Posted by Bryan Cutler <cu...@gmail.com>.
Thanks for looking into this Shane! If we are choosing a single python
3.x, I think 3.6 would be good. It might still be nice to test against
other versions too, so we can catch any issues. Is it possible to have more
exhaustive testing as part of a nightly or scheduled build? As a point of
reference for Python 3.6, Arrow is using this version for CI.
Bryan
On Sun, Aug 19, 2018 at 9:49 PM Hyukjin Kwon <gu...@gmail.com> wrote:
> Actually Python 3.7 is released (
> https://www.python.org/downloads/release/python-370/) too and I fixed the
> compatibility issues accordingly -
> https://github.com/apache/spark/pull/21714
> There has been an issue for 3.6 (comparing to lower versions of Python
> including 3.5) - https://github.com/apache/spark/pull/16429
>
> I am not yet sure what's the best matrix for it actually. In case of R, we
> test lowest version in Jenkins and highest version via AppVeyor FWIW.
> I don't have a strong preference opinion on this since we have been having
> compatibility issues for each Python version.
>
>
> 2018년 8월 14일 (화) 오전 4:15, shane knapp <sk...@berkeley.edu>님이 작성:
>
>> hey everyone!
>>
>> i was checking out the EOL/release cycle for python 3.5 and it looks like
>> we'll have 3.5.6 released in early 2019.
>>
>> this got me to thinking: instead of 3.5, what about 3.6?
>>
>> i looked around, and according to the 'docs' and 'collective wisdom of
>> the internets', 3.5 and 3.6 should be fully backwards-compatible w/3.4.
>>
>> of course, this needs to be taken w/a grain of salt, as we're mostly
>> focused on actual python package requirements, rather than worrying about
>> core python functionality.
>>
>> thoughts? comments?
>>
>> thanks in advance,
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
Re: [discuss][minor] impending python 3.x jenkins upgrade... 3.5.x? 3.6.x?
Posted by Hyukjin Kwon <gu...@gmail.com>.
Actually Python 3.7 is released (
https://www.python.org/downloads/release/python-370/) too and I fixed the
compatibility issues accordingly -
https://github.com/apache/spark/pull/21714
There has been an issue for 3.6 (comparing to lower versions of Python
including 3.5) - https://github.com/apache/spark/pull/16429
I am not yet sure what's the best matrix for it actually. In case of R, we
test lowest version in Jenkins and highest version via AppVeyor FWIW.
I don't have a strong preference opinion on this since we have been having
compatibility issues for each Python version.
2018년 8월 14일 (화) 오전 4:15, shane knapp <sk...@berkeley.edu>님이 작성:
> hey everyone!
>
> i was checking out the EOL/release cycle for python 3.5 and it looks like
> we'll have 3.5.6 released in early 2019.
>
> this got me to thinking: instead of 3.5, what about 3.6?
>
> i looked around, and according to the 'docs' and 'collective wisdom of the
> internets', 3.5 and 3.6 should be fully backwards-compatible w/3.4.
>
> of course, this needs to be taken w/a grain of salt, as we're mostly
> focused on actual python package requirements, rather than worrying about
> core python functionality.
>
> thoughts? comments?
>
> thanks in advance,
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>