You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by shane knapp <sk...@berkeley.edu> on 2019/04/01 15:58:51 UTC
Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]
i'd much prefer that we minimize the number of python versions that we test
against... would 2.7 and 3.6 be sufficient?
On Fri, Mar 29, 2019 at 10:23 PM Felix Cheung <fe...@hotmail.com>
wrote:
> I don’t take it as Sept 2019 is end of life for python 3.5 tho. It’s just
> saying the next release.
>
> In any case I think in the next release it will be great to get more
> Python 3.x release test coverage.
>
>
>
> ------------------------------
> *From:* shane knapp <sk...@berkeley.edu>
> *Sent:* Friday, March 29, 2019 4:46 PM
> *To:* Bryan Cutler
> *Cc:* Felix Cheung; Hyukjin Kwon; dev
> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]
>
> i'm not opposed to 3.6 at all.
>
> On Fri, Mar 29, 2019 at 4:16 PM Bryan Cutler <cu...@gmail.com> wrote:
>
>> PyArrow dropping Python 3.4 was mainly due to support going away at
>> Conda-Forge and other dependencies also dropping it. I think we better
>> upgrade Jenkins Python while we are at it. Are you all against jumping to
>> Python 3.6 so we are not in the same boat in September?
>>
>> On Thu, Mar 28, 2019 at 7:58 PM Felix Cheung <fe...@hotmail.com>
>> wrote:
>>
>>> 3.4 is end of life but 3.5 is not. From your link
>>>
>>> we expect to release Python 3.5.8 around September 2019.
>>>
>>>
>>>
>>> ------------------------------
>>> *From:* shane knapp <sk...@berkeley.edu>
>>> *Sent:* Thursday, March 28, 2019 7:54 PM
>>> *To:* Hyukjin Kwon
>>> *Cc:* Bryan Cutler; dev; Felix Cheung
>>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x
>>> [SPARK-27276]
>>>
>>> looks like the same for 3.5...
>>> https://www.python.org/dev/peps/pep-0478/
>>>
>>> let's pick a python version and start testing.
>>>
>>> On Thu, Mar 28, 2019 at 7:52 PM shane knapp <sk...@berkeley.edu> wrote:
>>>
>>>>
>>>>> If there was, it looks inevitable to upgrade Jenkins\s Python from 3.4
>>>>> to 3.5.
>>>>>
>>>>> this is inevitable. 3.4s final release was 10 days ago (
>>>> https://www.python.org/dev/peps/pep-0429/) so we're basically EOL.
>>>>
>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]
Posted by shane knapp <sk...@berkeley.edu>.
i am totally fine w/waiting a few days for the latest arrow release... not
at all a problem.
On Tue, Apr 2, 2019 at 9:14 AM Bryan Cutler <cu...@gmail.com> wrote:
> Nice work Shane! That all sounds good to me. We might want to use pyarrow
> 0.12.1 though, there is a major bug that was fixed, but we can discuss in
> the PR. I will put up the code changes in the next few days.
> Felix, I think you're right about Python 3.5, they just list one upcoming
> release and that's not necessarily the last. Comparing the histories, it
> might still be soon though. I think using 3.6 will be fine, as a point of
> reference, pyarrow CI uses 2.7 and 3.6.
>
> On Mon, Apr 1, 2019 at 3:09 PM shane knapp <sk...@berkeley.edu> wrote:
>
>> well now! color me completely surprised... i decided to whip up a fresh
>> python3.6.8 conda environment this morning to "see if things just worked".
>>
>> well, apparently they do! :)
>>
>> regardless, this is pretty awesome news as i will be able to easily
>> update the 'py3k' python3.4 environment to a fresh, less bloated, but still
>> package-complete python3.6.8 environment (including pyarrow 0.12.0, pandas
>> 0.24.2, scipy 1.2.1).
>>
>> i tested this pretty extensively today on both the ubuntu and centos
>> workers, and i think i'm ready to pull the trigger for a build-system-wide
>> upgrade... however, i'll be out wednesday through friday this week and
>> don't want to make a massive change before disappearing for a few days.
>>
>> so: how does early next week sound for the python upgrade? :)
>>
>> shane
>>
>> On Mon, Apr 1, 2019 at 8:58 AM shane knapp <sk...@berkeley.edu> wrote:
>>
>>> i'd much prefer that we minimize the number of python versions that we
>>> test against... would 2.7 and 3.6 be sufficient?
>>>
>>> On Fri, Mar 29, 2019 at 10:23 PM Felix Cheung <fe...@hotmail.com>
>>> wrote:
>>>
>>>> I don’t take it as Sept 2019 is end of life for python 3.5 tho. It’s
>>>> just saying the next release.
>>>>
>>>> In any case I think in the next release it will be great to get more
>>>> Python 3.x release test coverage.
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* shane knapp <sk...@berkeley.edu>
>>>> *Sent:* Friday, March 29, 2019 4:46 PM
>>>> *To:* Bryan Cutler
>>>> *Cc:* Felix Cheung; Hyukjin Kwon; dev
>>>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x
>>>> [SPARK-27276]
>>>>
>>>> i'm not opposed to 3.6 at all.
>>>>
>>>> On Fri, Mar 29, 2019 at 4:16 PM Bryan Cutler <cu...@gmail.com> wrote:
>>>>
>>>>> PyArrow dropping Python 3.4 was mainly due to support going away at
>>>>> Conda-Forge and other dependencies also dropping it. I think we better
>>>>> upgrade Jenkins Python while we are at it. Are you all against jumping to
>>>>> Python 3.6 so we are not in the same boat in September?
>>>>>
>>>>> On Thu, Mar 28, 2019 at 7:58 PM Felix Cheung <
>>>>> felixcheung_m@hotmail.com> wrote:
>>>>>
>>>>>> 3.4 is end of life but 3.5 is not. From your link
>>>>>>
>>>>>> we expect to release Python 3.5.8 around September 2019.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> *From:* shane knapp <sk...@berkeley.edu>
>>>>>> *Sent:* Thursday, March 28, 2019 7:54 PM
>>>>>> *To:* Hyukjin Kwon
>>>>>> *Cc:* Bryan Cutler; dev; Felix Cheung
>>>>>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x
>>>>>> [SPARK-27276]
>>>>>>
>>>>>> looks like the same for 3.5...
>>>>>> https://www.python.org/dev/peps/pep-0478/
>>>>>>
>>>>>> let's pick a python version and start testing.
>>>>>>
>>>>>> On Thu, Mar 28, 2019 at 7:52 PM shane knapp <sk...@berkeley.edu>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>> If there was, it looks inevitable to upgrade Jenkins\s Python from
>>>>>>>> 3.4 to 3.5.
>>>>>>>>
>>>>>>>> this is inevitable. 3.4s final release was 10 days ago (
>>>>>>> https://www.python.org/dev/peps/pep-0429/) so we're basically EOL.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Shane Knapp
>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>>> https://rise.cs.berkeley.edu
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Shane Knapp
>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>> https://rise.cs.berkeley.edu
>>>>
>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]
Posted by Bryan Cutler <cu...@gmail.com>.
Nice work Shane! That all sounds good to me. We might want to use pyarrow
0.12.1 though, there is a major bug that was fixed, but we can discuss in
the PR. I will put up the code changes in the next few days.
Felix, I think you're right about Python 3.5, they just list one upcoming
release and that's not necessarily the last. Comparing the histories, it
might still be soon though. I think using 3.6 will be fine, as a point of
reference, pyarrow CI uses 2.7 and 3.6.
On Mon, Apr 1, 2019 at 3:09 PM shane knapp <sk...@berkeley.edu> wrote:
> well now! color me completely surprised... i decided to whip up a fresh
> python3.6.8 conda environment this morning to "see if things just worked".
>
> well, apparently they do! :)
>
> regardless, this is pretty awesome news as i will be able to easily update
> the 'py3k' python3.4 environment to a fresh, less bloated, but still
> package-complete python3.6.8 environment (including pyarrow 0.12.0, pandas
> 0.24.2, scipy 1.2.1).
>
> i tested this pretty extensively today on both the ubuntu and centos
> workers, and i think i'm ready to pull the trigger for a build-system-wide
> upgrade... however, i'll be out wednesday through friday this week and
> don't want to make a massive change before disappearing for a few days.
>
> so: how does early next week sound for the python upgrade? :)
>
> shane
>
> On Mon, Apr 1, 2019 at 8:58 AM shane knapp <sk...@berkeley.edu> wrote:
>
>> i'd much prefer that we minimize the number of python versions that we
>> test against... would 2.7 and 3.6 be sufficient?
>>
>> On Fri, Mar 29, 2019 at 10:23 PM Felix Cheung <fe...@hotmail.com>
>> wrote:
>>
>>> I don’t take it as Sept 2019 is end of life for python 3.5 tho. It’s
>>> just saying the next release.
>>>
>>> In any case I think in the next release it will be great to get more
>>> Python 3.x release test coverage.
>>>
>>>
>>>
>>> ------------------------------
>>> *From:* shane knapp <sk...@berkeley.edu>
>>> *Sent:* Friday, March 29, 2019 4:46 PM
>>> *To:* Bryan Cutler
>>> *Cc:* Felix Cheung; Hyukjin Kwon; dev
>>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x
>>> [SPARK-27276]
>>>
>>> i'm not opposed to 3.6 at all.
>>>
>>> On Fri, Mar 29, 2019 at 4:16 PM Bryan Cutler <cu...@gmail.com> wrote:
>>>
>>>> PyArrow dropping Python 3.4 was mainly due to support going away at
>>>> Conda-Forge and other dependencies also dropping it. I think we better
>>>> upgrade Jenkins Python while we are at it. Are you all against jumping to
>>>> Python 3.6 so we are not in the same boat in September?
>>>>
>>>> On Thu, Mar 28, 2019 at 7:58 PM Felix Cheung <fe...@hotmail.com>
>>>> wrote:
>>>>
>>>>> 3.4 is end of life but 3.5 is not. From your link
>>>>>
>>>>> we expect to release Python 3.5.8 around September 2019.
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> *From:* shane knapp <sk...@berkeley.edu>
>>>>> *Sent:* Thursday, March 28, 2019 7:54 PM
>>>>> *To:* Hyukjin Kwon
>>>>> *Cc:* Bryan Cutler; dev; Felix Cheung
>>>>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x
>>>>> [SPARK-27276]
>>>>>
>>>>> looks like the same for 3.5...
>>>>> https://www.python.org/dev/peps/pep-0478/
>>>>>
>>>>> let's pick a python version and start testing.
>>>>>
>>>>> On Thu, Mar 28, 2019 at 7:52 PM shane knapp <sk...@berkeley.edu>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>> If there was, it looks inevitable to upgrade Jenkins\s Python from
>>>>>>> 3.4 to 3.5.
>>>>>>>
>>>>>>> this is inevitable. 3.4s final release was 10 days ago (
>>>>>> https://www.python.org/dev/peps/pep-0429/) so we're basically EOL.
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Shane Knapp
>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>> https://rise.cs.berkeley.edu
>>>>>
>>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]
Posted by shane knapp <sk...@berkeley.edu>.
well now! color me completely surprised... i decided to whip up a fresh
python3.6.8 conda environment this morning to "see if things just worked".
well, apparently they do! :)
regardless, this is pretty awesome news as i will be able to easily update
the 'py3k' python3.4 environment to a fresh, less bloated, but still
package-complete python3.6.8 environment (including pyarrow 0.12.0, pandas
0.24.2, scipy 1.2.1).
i tested this pretty extensively today on both the ubuntu and centos
workers, and i think i'm ready to pull the trigger for a build-system-wide
upgrade... however, i'll be out wednesday through friday this week and
don't want to make a massive change before disappearing for a few days.
so: how does early next week sound for the python upgrade? :)
shane
On Mon, Apr 1, 2019 at 8:58 AM shane knapp <sk...@berkeley.edu> wrote:
> i'd much prefer that we minimize the number of python versions that we
> test against... would 2.7 and 3.6 be sufficient?
>
> On Fri, Mar 29, 2019 at 10:23 PM Felix Cheung <fe...@hotmail.com>
> wrote:
>
>> I don’t take it as Sept 2019 is end of life for python 3.5 tho. It’s just
>> saying the next release.
>>
>> In any case I think in the next release it will be great to get more
>> Python 3.x release test coverage.
>>
>>
>>
>> ------------------------------
>> *From:* shane knapp <sk...@berkeley.edu>
>> *Sent:* Friday, March 29, 2019 4:46 PM
>> *To:* Bryan Cutler
>> *Cc:* Felix Cheung; Hyukjin Kwon; dev
>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]
>>
>> i'm not opposed to 3.6 at all.
>>
>> On Fri, Mar 29, 2019 at 4:16 PM Bryan Cutler <cu...@gmail.com> wrote:
>>
>>> PyArrow dropping Python 3.4 was mainly due to support going away at
>>> Conda-Forge and other dependencies also dropping it. I think we better
>>> upgrade Jenkins Python while we are at it. Are you all against jumping to
>>> Python 3.6 so we are not in the same boat in September?
>>>
>>> On Thu, Mar 28, 2019 at 7:58 PM Felix Cheung <fe...@hotmail.com>
>>> wrote:
>>>
>>>> 3.4 is end of life but 3.5 is not. From your link
>>>>
>>>> we expect to release Python 3.5.8 around September 2019.
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* shane knapp <sk...@berkeley.edu>
>>>> *Sent:* Thursday, March 28, 2019 7:54 PM
>>>> *To:* Hyukjin Kwon
>>>> *Cc:* Bryan Cutler; dev; Felix Cheung
>>>> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x
>>>> [SPARK-27276]
>>>>
>>>> looks like the same for 3.5...
>>>> https://www.python.org/dev/peps/pep-0478/
>>>>
>>>> let's pick a python version and start testing.
>>>>
>>>> On Thu, Mar 28, 2019 at 7:52 PM shane knapp <sk...@berkeley.edu>
>>>> wrote:
>>>>
>>>>>
>>>>>> If there was, it looks inevitable to upgrade Jenkins\s Python from
>>>>>> 3.4 to 3.5.
>>>>>>
>>>>>> this is inevitable. 3.4s final release was 10 days ago (
>>>>> https://www.python.org/dev/peps/pep-0429/) so we're basically EOL.
>>>>>
>>>>
>>>>
>>>> --
>>>> Shane Knapp
>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>> https://rise.cs.berkeley.edu
>>>>
>>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu