Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

[Holden Karau <ho...@pigscanfly.ca>]

I may have watched too many buzzfeed videos this week but the steps are:
1) git checkout the PR in question
2) Run tox -e lint_py2,lint_py3

This is important since Python 3 isn't installed on the Jenkins workers
just yet and we have some tests to catch basic invalid Python 3 which we
can slowly grow as we fix the issues and you can help us keep moving
forward!

If step 1 is too much work I like using the hub program I find it helps me
with this part of my workflow in other projects. That being said you don't
have to do this, we'll fix whatever errors come up, so if this is going to
slow your workflow down or you otherwise don't like it feel free to pass
along.

-- 
Twitter: https://twitter.com/holdenkarau

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

[Ahmet Altay <al...@google.com>]

Sent https://github.com/apache/beam/pull/4801 to enable py3 lint for
precommits.

On Fri, Mar 2, 2018 at 11:23 AM, Robert Bradshaw <ro...@google.com>
wrote:

> To address the first point, 3.4 is almost certainly sufficient for our
> needs (running lint_py3 to prevent regressions). Also, +1 that automating
> this is going to be much more effective than asking users to manually do
> extra steps.
>
> Long-term, we should definitely support 3.5+, definitely not support 3.1
> or below, and probably look at usage numbers/customers/what's shipped with
> different OSes for 3.2, 3.3, 3.4 (my guess is that 3.4 is the lowest we'll
> want to go).
>

Thank you for this explanation. Filed (
https://issues.apache.org/jira/browse/BEAM-3781) as a reminder to figure
this out once have some reasonable python 3 support.


>
>
> On Fri, Mar 2, 2018 at 11:06 AM Ahmet Altay <al...@google.com> wrote:
>
>> This is great. Let's enable py3 lint tests in Jenkins.
>>
>> Side question, what python 3 version we should target as the minimum
>> supported version in Beam?
>>
>> On Fri, Mar 2, 2018 at 10:31 AM, Holden Karau <ho...@pigscanfly.ca>
>> wrote:
>>
>>> 3.4.3 is from Feb 2015, and for what it’s worth the minimum version of
>>> Python in Spark is  3.4. We could enable lint tests in Jenkins and see how
>>> they go?
>>>
>>> On Fri, Mar 2, 2018 at 10:06 AM Alan Myrvold <am...@google.com>
>>> wrote:
>>>
>>>> I ran "python3 --version" on each worker and all showed python 3.4.3.
>>>> Is that too old?
>>>>
>>>>
>>>> On Fri, Mar 2, 2018 at 10:04 AM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> That is my understanding as well, it is requires attention from infra.
>>>>> Could anyone help with this? I know we worked with infra before, what is
>>>>> the best way to approach this?
>>>>>
>>>>> On Fri, Mar 2, 2018 at 9:50 AM, Holden Karau <ho...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I agree, however I'm of the impression it's blocked on infra? (e.g.
>>>>>> it's important but out of my hands).
>>>>>>
>>>>>> On Mar 1, 2018 11:05 PM, "Ahmet Altay" <al...@google.com> wrote:
>>>>>>
>>>>>>> I think we should prioritize the issue of installing Python 3 on the
>>>>>>> workers (https://issues.apache.org/jira/browse/BEAM-3671). I would
>>>>>>> appreciate if folks pay attention to these 2 steps but I am worried that it
>>>>>>> will be easily forgotten.
>>>>>>>
>>>>>>> On Thu, Mar 1, 2018 at 6:51 PM, Holden Karau <ho...@pigscanfly.ca>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I may have watched too many buzzfeed videos this week but the steps
>>>>>>>> are:
>>>>>>>> 1) git checkout the PR in question
>>>>>>>> 2) Run tox -e lint_py2,lint_py3
>>>>>>>>
>>>>>>>> This is important since Python 3 isn't installed on the Jenkins
>>>>>>>> workers just yet and we have some tests to catch basic invalid Python 3
>>>>>>>> which we can slowly grow as we fix the issues and you can help us keep
>>>>>>>> moving forward!
>>>>>>>>
>>>>>>>> If step 1 is too much work I like using the hub program I find it
>>>>>>>> helps me with this part of my workflow in other projects. That being said
>>>>>>>> you don't have to do this, we'll fix whatever errors come up, so if this is
>>>>>>>> going to slow your workflow down or you otherwise don't like it feel free
>>>>>>>> to pass along.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>
>>

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

[Robert Bradshaw <ro...@google.com>]

To address the first point, 3.4 is almost certainly sufficient for our
needs (running lint_py3 to prevent regressions). Also, +1 that automating
this is going to be much more effective than asking users to manually do
extra steps.

Long-term, we should definitely support 3.5+, definitely not support 3.1 or
below, and probably look at usage numbers/customers/what's shipped with
different OSes for 3.2, 3.3, 3.4 (my guess is that 3.4 is the lowest we'll
want to go).


On Fri, Mar 2, 2018 at 11:06 AM Ahmet Altay <al...@google.com> wrote:

> This is great. Let's enable py3 lint tests in Jenkins.
>
> Side question, what python 3 version we should target as the minimum
> supported version in Beam?
>
> On Fri, Mar 2, 2018 at 10:31 AM, Holden Karau <ho...@pigscanfly.ca>
> wrote:
>
>> 3.4.3 is from Feb 2015, and for what it’s worth the minimum version of
>> Python in Spark is  3.4. We could enable lint tests in Jenkins and see how
>> they go?
>>
>> On Fri, Mar 2, 2018 at 10:06 AM Alan Myrvold <am...@google.com> wrote:
>>
>>> I ran "python3 --version" on each worker and all showed python 3.4.3. Is
>>> that too old?
>>>
>>>
>>> On Fri, Mar 2, 2018 at 10:04 AM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> That is my understanding as well, it is requires attention from infra.
>>>> Could anyone help with this? I know we worked with infra before, what is
>>>> the best way to approach this?
>>>>
>>>> On Fri, Mar 2, 2018 at 9:50 AM, Holden Karau <ho...@gmail.com>
>>>> wrote:
>>>>
>>>>> I agree, however I'm of the impression it's blocked on infra? (e.g.
>>>>> it's important but out of my hands).
>>>>>
>>>>> On Mar 1, 2018 11:05 PM, "Ahmet Altay" <al...@google.com> wrote:
>>>>>
>>>>>> I think we should prioritize the issue of installing Python 3 on the
>>>>>> workers (https://issues.apache.org/jira/browse/BEAM-3671). I would
>>>>>> appreciate if folks pay attention to these 2 steps but I am worried that it
>>>>>> will be easily forgotten.
>>>>>>
>>>>>> On Thu, Mar 1, 2018 at 6:51 PM, Holden Karau <ho...@pigscanfly.ca>
>>>>>> wrote:
>>>>>>
>>>>>>> I may have watched too many buzzfeed videos this week but the steps
>>>>>>> are:
>>>>>>> 1) git checkout the PR in question
>>>>>>> 2) Run tox -e lint_py2,lint_py3
>>>>>>>
>>>>>>> This is important since Python 3 isn't installed on the Jenkins
>>>>>>> workers just yet and we have some tests to catch basic invalid Python 3
>>>>>>> which we can slowly grow as we fix the issues and you can help us keep
>>>>>>> moving forward!
>>>>>>>
>>>>>>> If step 1 is too much work I like using the hub program I find it
>>>>>>> helps me with this part of my workflow in other projects. That being said
>>>>>>> you don't have to do this, we'll fix whatever errors come up, so if this is
>>>>>>> going to slow your workflow down or you otherwise don't like it feel free
>>>>>>> to pass along.
>>>>>>>
>>>>>>> --
>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>>
>>>>>>
>>>>>>
>>>> --
>> Twitter: https://twitter.com/holdenkarau
>>
>
>

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

[Ahmet Altay <al...@google.com>]

This is great. Let's enable py3 lint tests in Jenkins.

Side question, what python 3 version we should target as the minimum
supported version in Beam?

On Fri, Mar 2, 2018 at 10:31 AM, Holden Karau <ho...@pigscanfly.ca> wrote:

> 3.4.3 is from Feb 2015, and for what it’s worth the minimum version of
> Python in Spark is  3.4. We could enable lint tests in Jenkins and see how
> they go?
>
> On Fri, Mar 2, 2018 at 10:06 AM Alan Myrvold <am...@google.com> wrote:
>
>> I ran "python3 --version" on each worker and all showed python 3.4.3. Is
>> that too old?
>>
>>
>> On Fri, Mar 2, 2018 at 10:04 AM Ahmet Altay <al...@google.com> wrote:
>>
>>> That is my understanding as well, it is requires attention from infra.
>>> Could anyone help with this? I know we worked with infra before, what is
>>> the best way to approach this?
>>>
>>> On Fri, Mar 2, 2018 at 9:50 AM, Holden Karau <ho...@gmail.com>
>>> wrote:
>>>
>>>> I agree, however I'm of the impression it's blocked on infra? (e.g.
>>>> it's important but out of my hands).
>>>>
>>>> On Mar 1, 2018 11:05 PM, "Ahmet Altay" <al...@google.com> wrote:
>>>>
>>>>> I think we should prioritize the issue of installing Python 3 on the
>>>>> workers (https://issues.apache.org/jira/browse/BEAM-3671). I would
>>>>> appreciate if folks pay attention to these 2 steps but I am worried that it
>>>>> will be easily forgotten.
>>>>>
>>>>> On Thu, Mar 1, 2018 at 6:51 PM, Holden Karau <ho...@pigscanfly.ca>
>>>>> wrote:
>>>>>
>>>>>> I may have watched too many buzzfeed videos this week but the steps
>>>>>> are:
>>>>>> 1) git checkout the PR in question
>>>>>> 2) Run tox -e lint_py2,lint_py3
>>>>>>
>>>>>> This is important since Python 3 isn't installed on the Jenkins
>>>>>> workers just yet and we have some tests to catch basic invalid Python 3
>>>>>> which we can slowly grow as we fix the issues and you can help us keep
>>>>>> moving forward!
>>>>>>
>>>>>> If step 1 is too much work I like using the hub program I find it
>>>>>> helps me with this part of my workflow in other projects. That being said
>>>>>> you don't have to do this, we'll fix whatever errors come up, so if this is
>>>>>> going to slow your workflow down or you otherwise don't like it feel free
>>>>>> to pass along.
>>>>>>
>>>>>> --
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>
>>>>>
>>>>>
>>> --
> Twitter: https://twitter.com/holdenkarau
>

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

[Holden Karau <ho...@pigscanfly.ca>]

3.4.3 is from Feb 2015, and for what it’s worth the minimum version of
Python in Spark is  3.4. We could enable lint tests in Jenkins and see how
they go?

On Fri, Mar 2, 2018 at 10:06 AM Alan Myrvold <am...@google.com> wrote:

> I ran "python3 --version" on each worker and all showed python 3.4.3. Is
> that too old?
>
>
> On Fri, Mar 2, 2018 at 10:04 AM Ahmet Altay <al...@google.com> wrote:
>
>> That is my understanding as well, it is requires attention from infra.
>> Could anyone help with this? I know we worked with infra before, what is
>> the best way to approach this?
>>
>> On Fri, Mar 2, 2018 at 9:50 AM, Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> I agree, however I'm of the impression it's blocked on infra? (e.g. it's
>>> important but out of my hands).
>>>
>>> On Mar 1, 2018 11:05 PM, "Ahmet Altay" <al...@google.com> wrote:
>>>
>>>> I think we should prioritize the issue of installing Python 3 on the
>>>> workers (https://issues.apache.org/jira/browse/BEAM-3671). I would
>>>> appreciate if folks pay attention to these 2 steps but I am worried that it
>>>> will be easily forgotten.
>>>>
>>>> On Thu, Mar 1, 2018 at 6:51 PM, Holden Karau <ho...@pigscanfly.ca>
>>>> wrote:
>>>>
>>>>> I may have watched too many buzzfeed videos this week but the steps
>>>>> are:
>>>>> 1) git checkout the PR in question
>>>>> 2) Run tox -e lint_py2,lint_py3
>>>>>
>>>>> This is important since Python 3 isn't installed on the Jenkins
>>>>> workers just yet and we have some tests to catch basic invalid Python 3
>>>>> which we can slowly grow as we fix the issues and you can help us keep
>>>>> moving forward!
>>>>>
>>>>> If step 1 is too much work I like using the hub program I find it
>>>>> helps me with this part of my workflow in other projects. That being said
>>>>> you don't have to do this, we'll fix whatever errors come up, so if this is
>>>>> going to slow your workflow down or you otherwise don't like it feel free
>>>>> to pass along.
>>>>>
>>>>> --
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>
>>>>
>>>>
>> --
Twitter: https://twitter.com/holdenkarau

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

[Alan Myrvold <am...@google.com>]

I ran "python3 --version" on each worker and all showed python 3.4.3. Is
that too old?


On Fri, Mar 2, 2018 at 10:04 AM Ahmet Altay <al...@google.com> wrote:

> That is my understanding as well, it is requires attention from infra.
> Could anyone help with this? I know we worked with infra before, what is
> the best way to approach this?
>
> On Fri, Mar 2, 2018 at 9:50 AM, Holden Karau <ho...@gmail.com>
> wrote:
>
>> I agree, however I'm of the impression it's blocked on infra? (e.g. it's
>> important but out of my hands).
>>
>> On Mar 1, 2018 11:05 PM, "Ahmet Altay" <al...@google.com> wrote:
>>
>>> I think we should prioritize the issue of installing Python 3 on the
>>> workers (https://issues.apache.org/jira/browse/BEAM-3671). I would
>>> appreciate if folks pay attention to these 2 steps but I am worried that it
>>> will be easily forgotten.
>>>
>>> On Thu, Mar 1, 2018 at 6:51 PM, Holden Karau <ho...@pigscanfly.ca>
>>> wrote:
>>>
>>>> I may have watched too many buzzfeed videos this week but the steps are:
>>>> 1) git checkout the PR in question
>>>> 2) Run tox -e lint_py2,lint_py3
>>>>
>>>> This is important since Python 3 isn't installed on the Jenkins workers
>>>> just yet and we have some tests to catch basic invalid Python 3 which we
>>>> can slowly grow as we fix the issues and you can help us keep moving
>>>> forward!
>>>>
>>>> If step 1 is too much work I like using the hub program I find it helps
>>>> me with this part of my workflow in other projects. That being said you
>>>> don't have to do this, we'll fix whatever errors come up, so if this is
>>>> going to slow your workflow down or you otherwise don't like it feel free
>>>> to pass along.
>>>>
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>>
>>>
>>>
>

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

[Ahmet Altay <al...@google.com>]

That is my understanding as well, it is requires attention from infra.
Could anyone help with this? I know we worked with infra before, what is
the best way to approach this?

On Fri, Mar 2, 2018 at 9:50 AM, Holden Karau <ho...@gmail.com> wrote:

> I agree, however I'm of the impression it's blocked on infra? (e.g. it's
> important but out of my hands).
>
> On Mar 1, 2018 11:05 PM, "Ahmet Altay" <al...@google.com> wrote:
>
>> I think we should prioritize the issue of installing Python 3 on the
>> workers (https://issues.apache.org/jira/browse/BEAM-3671). I would
>> appreciate if folks pay attention to these 2 steps but I am worried that it
>> will be easily forgotten.
>>
>> On Thu, Mar 1, 2018 at 6:51 PM, Holden Karau <ho...@pigscanfly.ca>
>> wrote:
>>
>>> I may have watched too many buzzfeed videos this week but the steps are:
>>> 1) git checkout the PR in question
>>> 2) Run tox -e lint_py2,lint_py3
>>>
>>> This is important since Python 3 isn't installed on the Jenkins workers
>>> just yet and we have some tests to catch basic invalid Python 3 which we
>>> can slowly grow as we fix the issues and you can help us keep moving
>>> forward!
>>>
>>> If step 1 is too much work I like using the hub program I find it helps
>>> me with this part of my workflow in other projects. That being said you
>>> don't have to do this, we'll fix whatever errors come up, so if this is
>>> going to slow your workflow down or you otherwise don't like it feel free
>>> to pass along.
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>
>>

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

[Holden Karau <ho...@gmail.com>]

I agree, however I'm of the impression it's blocked on infra? (e.g. it's
important but out of my hands).

On Mar 1, 2018 11:05 PM, "Ahmet Altay" <al...@google.com> wrote:

> I think we should prioritize the issue of installing Python 3 on the
> workers (https://issues.apache.org/jira/browse/BEAM-3671). I would
> appreciate if folks pay attention to these 2 steps but I am worried that it
> will be easily forgotten.
>
> On Thu, Mar 1, 2018 at 6:51 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> I may have watched too many buzzfeed videos this week but the steps are:
>> 1) git checkout the PR in question
>> 2) Run tox -e lint_py2,lint_py3
>>
>> This is important since Python 3 isn't installed on the Jenkins workers
>> just yet and we have some tests to catch basic invalid Python 3 which we
>> can slowly grow as we fix the issues and you can help us keep moving
>> forward!
>>
>> If step 1 is too much work I like using the hub program I find it helps
>> me with this part of my workflow in other projects. That being said you
>> don't have to do this, we'll fix whatever errors come up, so if this is
>> going to slow your workflow down or you otherwise don't like it feel free
>> to pass along.
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>>
>
>

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

[Ahmet Altay <al...@google.com>]

I think we should prioritize the issue of installing Python 3 on the
workers (https://issues.apache.org/jira/browse/BEAM-3671). I would
appreciate if folks pay attention to these 2 steps but I am worried that it
will be easily forgotten.

On Thu, Mar 1, 2018 at 6:51 PM, Holden Karau <ho...@pigscanfly.ca> wrote:

> I may have watched too many buzzfeed videos this week but the steps are:
> 1) git checkout the PR in question
> 2) Run tox -e lint_py2,lint_py3
>
> This is important since Python 3 isn't installed on the Jenkins workers
> just yet and we have some tests to catch basic invalid Python 3 which we
> can slowly grow as we fix the issues and you can help us keep moving
> forward!
>
> If step 1 is too much work I like using the hub program I find it helps me
> with this part of my workflow in other projects. That being said you don't
> have to do this, we'll fix whatever errors come up, so if this is going to
> slow your workflow down or you otherwise don't like it feel free to pass
> along.
>
> --
> Twitter: https://twitter.com/holdenkarau
>