You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yetus.apache.org by Andrew Wang <an...@cloudera.com> on 2016/08/24 19:21:10 UTC
[DISCUSS] Requiring Python 2.7
Hi dev@,
This topic came up on YETUS-445. I'd like to make the case for requiring
Python 2.7.
As a bit of background, the main offender here is RHEL6. RHEL7 has been out
for two years and has some penetration, but RHEL6 is still out there and
supported.
My take on it:
* This tool is aimed at release managers, and no one runs RHEL6 on their
personal dev machine.
* Even for build machines, Docker is not supported on RHEL6 (
https://access.redhat.com/solutions/1378023). FWIW, Docker is a requirement
for building a Hadoop release and used by Hadoop precommit.
* If you really do need to use RHEL6, you can use conda (
http://conda.pydata.org/miniconda.html) to install Python 2.7 quite easily.
I've used it, and it's great.
Thoughts?
Andrew
Re: [DISCUSS] Requiring Python 2.7
Posted by Chris Nauroth <cn...@hortonworks.com>.
In my own experience, I’ve had to work under all of the following policies:
1. Write Python code however you want, and any necessary upgrades are fine.
2. You must stick to Python version X, but installing additional third-party libraries is acceptable.
3. You must stick to “vanilla” Python version X, and additional third-party libraries cannot be installed.
My suggestion to consider bundling argparse and front-loading it on PYTHONPATH is a workaround that I have used in scenario 3 in some cases (when the library’s licensing allowed it). It’s a trade-off between the convenience of whatever the library in question provides and the additional build and deployment complexity. This was considered acceptable under policy, because use of the library was constrained to my application and not installed globally.
However, as you pointed out, we’re not currently doing this in Yetus. We currently prompt to install the additional libraries. I think this is acceptable. Policy 3 was an unusual circumstance for me. If we get a user asking for vanilla Python 2.6 support, then it’s something we could explore again.
--Chris Nauroth
On 9/6/16, 10:35 AM, "Andrew Wang" <an...@cloudera.com> wrote:
I already updated YETUS-445 for Python 2.6 support, so it's not blocked on
this conversation.
Chris, in the situation you describe where "IT policies prevented me from
performing non-essential/non-security related upgrades or installing new
software", does this policy not also apply to third party libraries
installed via pip? I honestly don't see the difference, since both conda
and pip can be used without root, and install things to your local homedir.
This is why I keep bringing up third-party libraries. If the restriction is
not just Python 2.6 but vanilla Python 2.6, that's also something that
should be documented, and we should retrofit RDM and YETUS-445 to remove
those dependencies.
On Tue, Sep 6, 2016 at 10:23 AM, Chris Nauroth <cn...@hortonworks.com>
wrote:
> I’m just catching up on this thread now.
>
> I fall in the camp that would prefer not to dictate a requirement of
> Python 2.7 for anything in Yetus. My reasoning agrees with what Allen has
> already described. I have been in the situation that he describes, needing
> to maintain build infrastructure on old OSes and old Python versions.
> Upgrading the OS or installing Python 2.7 via conda or a custom rpm was not
> an option. IT policies prevented me from performing
> non-essential/non-security related upgrades or installing new software.
> Instead, the solution was to stick to Python code compatible with the old
> versions.
>
> I haven’t reviewed the code for YETUS-445, but the motivation for this
> topic coming up now seems to be argparse. I think Ajay’s proposal for
> handling that sounds fine. Alternatively, we could explore if it’s
> feasible to check in a copy of argparse to be front-loaded onto PYTHONPATH.
>
> --Chris Nauroth
>
> On 8/24/16, 9:45 PM, "Ajay Yadava" <aj...@gmail.com> wrote:
>
> If the only motivation for Python 2.7 for this script is to use
> argparse
> library then it can be installed on Python 2.6 and hence shouldn't be
> an
> issue IMO. We have external dependencies for RDM as well. A simple
> try-catch will ensure that 2.6 users know about the external
> dependency.
> Does this work?
>
> On Thu, Aug 25, 2016, 9:22 AM Dima Spivak <ds...@cloudera.com>
> wrote:
>
> > What if we compromise by creating some sort of designation for tools
> > that require runs to use Docker mode? This way, people could still
> use
> > Andrew's awesome script either on their own or in Apache Infra (even
> > precommit!) without unfortunate surprises when someone tries to run
> it on
> > an older OS.
> >
> > On Wednesday, August 24, 2016, Andrew Wang <andrew.wang@cloudera.com
> >
> > wrote:
> >
> > > On Wed, Aug 24, 2016 at 4:46 PM, Allen Wittenauer <
> > > aw@effectivemachines.com <javascript:;>>
> > > wrote:
> > >
> > > >
> > > > > On Aug 24, 2016, at 3:49 PM, Andrew Wang <
> andrew.wang@cloudera.com
> > > <javascript:;>>
> > > > wrote:
> > > > > My reasoning here has been that this tool will primarily be
> run by
> > > > release
> > > > > managers, and RMs are very likely to have Python 2.7 on their
> > machine.
> > > >
> > > > Whereas I believe the opposite: there's a good chance if
> one is
> > > > the RM for enterprise software, the machine you are building on
> > probably
> > > > doesn't have python 2.7 on it because you aren't doing this work
> on
> > your
> > > > desktop! It's extremely common to have build servers that are
> several
> > OS
> > > > revs behind because software tends to be more upwardly
> compatible....
> > > >
> > > > > ASF infra and servers in general are not really a target.
> > > >
> > > > Umm, yeah, they kind of are. While there are certainly
> issues
> > > with
> > > > them, they do serve as sort of a model of what is out in the real
> > world.
> > > > It's old and crufty and that's how a lot of software is built.
> Where
> > > > people build software is *exactly* our target.
> > > >
> > > > Here's a fun anecdote. Firefox v48 is the first version
> to
> > drop
> > > > support for Mac OS X 10.6, 10.7, and 10.8. This means they were
> almost
> > > > certainly building on 10.6. So up until earlier this year, they
> were
> > > using
> > > > a 7 year old OS that most definitely didn't have python 2.7 on
> it by
> > > > default. Mozilla, by many accounts, would be considered quite
> > aggressive
> > > > but they can get away with it because they are targeting the
> desktop.
> > > >
> > > >
> > > I think we're talking past each other here, since this is not a
> build
> > tool.
> > > It's an API checking tool. It only needs to be run on a developer's
> > > computer, not all boxes at ASF. And, like I've been saying, you
> can just
> > > install Python 2.7 like you would any other dependency. RDM
> requires `pip
> > > install python-dateutils` for instance.
> > >
> > >
> > > > > Maybe one day if
> > > > > we want to run this as part of precommit, but then it can be
> optional
> > > > like
> > > > > gradle and docker.
> > > >
> > > >
> > > > It sends really bizarre mixed messages if one tool has
> > different
> > > > python requirements than another just because someone wanted to
> use a
> > > > different option parsing library.
> > >
> > >
> > > I don't know about "really bizzare"; this is the same problem as
> any
> > > external dependency (e.g. python-dateutils).
> > >
> > > The real issue here is the lack of a standardized runtime
> environment. Is
> > > there build/packaging for Yetus? Some of this could be handled by
> > wrappers
> > > that created virtualenvs or downloaded conda as appropriate.
> > >
> >
> >
> > --
> > -Dima
> >
> --
> Regards
> Ajay Yadava
>
>
>
Re: [DISCUSS] Requiring Python 2.7
Posted by Andrew Wang <an...@cloudera.com>.
I already updated YETUS-445 for Python 2.6 support, so it's not blocked on
this conversation.
Chris, in the situation you describe where "IT policies prevented me from
performing non-essential/non-security related upgrades or installing new
software", does this policy not also apply to third party libraries
installed via pip? I honestly don't see the difference, since both conda
and pip can be used without root, and install things to your local homedir.
This is why I keep bringing up third-party libraries. If the restriction is
not just Python 2.6 but vanilla Python 2.6, that's also something that
should be documented, and we should retrofit RDM and YETUS-445 to remove
those dependencies.
On Tue, Sep 6, 2016 at 10:23 AM, Chris Nauroth <cn...@hortonworks.com>
wrote:
> I’m just catching up on this thread now.
>
> I fall in the camp that would prefer not to dictate a requirement of
> Python 2.7 for anything in Yetus. My reasoning agrees with what Allen has
> already described. I have been in the situation that he describes, needing
> to maintain build infrastructure on old OSes and old Python versions.
> Upgrading the OS or installing Python 2.7 via conda or a custom rpm was not
> an option. IT policies prevented me from performing
> non-essential/non-security related upgrades or installing new software.
> Instead, the solution was to stick to Python code compatible with the old
> versions.
>
> I haven’t reviewed the code for YETUS-445, but the motivation for this
> topic coming up now seems to be argparse. I think Ajay’s proposal for
> handling that sounds fine. Alternatively, we could explore if it’s
> feasible to check in a copy of argparse to be front-loaded onto PYTHONPATH.
>
> --Chris Nauroth
>
> On 8/24/16, 9:45 PM, "Ajay Yadava" <aj...@gmail.com> wrote:
>
> If the only motivation for Python 2.7 for this script is to use
> argparse
> library then it can be installed on Python 2.6 and hence shouldn't be
> an
> issue IMO. We have external dependencies for RDM as well. A simple
> try-catch will ensure that 2.6 users know about the external
> dependency.
> Does this work?
>
> On Thu, Aug 25, 2016, 9:22 AM Dima Spivak <ds...@cloudera.com>
> wrote:
>
> > What if we compromise by creating some sort of designation for tools
> > that require runs to use Docker mode? This way, people could still
> use
> > Andrew's awesome script either on their own or in Apache Infra (even
> > precommit!) without unfortunate surprises when someone tries to run
> it on
> > an older OS.
> >
> > On Wednesday, August 24, 2016, Andrew Wang <andrew.wang@cloudera.com
> >
> > wrote:
> >
> > > On Wed, Aug 24, 2016 at 4:46 PM, Allen Wittenauer <
> > > aw@effectivemachines.com <javascript:;>>
> > > wrote:
> > >
> > > >
> > > > > On Aug 24, 2016, at 3:49 PM, Andrew Wang <
> andrew.wang@cloudera.com
> > > <javascript:;>>
> > > > wrote:
> > > > > My reasoning here has been that this tool will primarily be
> run by
> > > > release
> > > > > managers, and RMs are very likely to have Python 2.7 on their
> > machine.
> > > >
> > > > Whereas I believe the opposite: there's a good chance if
> one is
> > > > the RM for enterprise software, the machine you are building on
> > probably
> > > > doesn't have python 2.7 on it because you aren't doing this work
> on
> > your
> > > > desktop! It's extremely common to have build servers that are
> several
> > OS
> > > > revs behind because software tends to be more upwardly
> compatible....
> > > >
> > > > > ASF infra and servers in general are not really a target.
> > > >
> > > > Umm, yeah, they kind of are. While there are certainly
> issues
> > > with
> > > > them, they do serve as sort of a model of what is out in the real
> > world.
> > > > It's old and crufty and that's how a lot of software is built.
> Where
> > > > people build software is *exactly* our target.
> > > >
> > > > Here's a fun anecdote. Firefox v48 is the first version
> to
> > drop
> > > > support for Mac OS X 10.6, 10.7, and 10.8. This means they were
> almost
> > > > certainly building on 10.6. So up until earlier this year, they
> were
> > > using
> > > > a 7 year old OS that most definitely didn't have python 2.7 on
> it by
> > > > default. Mozilla, by many accounts, would be considered quite
> > aggressive
> > > > but they can get away with it because they are targeting the
> desktop.
> > > >
> > > >
> > > I think we're talking past each other here, since this is not a
> build
> > tool.
> > > It's an API checking tool. It only needs to be run on a developer's
> > > computer, not all boxes at ASF. And, like I've been saying, you
> can just
> > > install Python 2.7 like you would any other dependency. RDM
> requires `pip
> > > install python-dateutils` for instance.
> > >
> > >
> > > > > Maybe one day if
> > > > > we want to run this as part of precommit, but then it can be
> optional
> > > > like
> > > > > gradle and docker.
> > > >
> > > >
> > > > It sends really bizarre mixed messages if one tool has
> > different
> > > > python requirements than another just because someone wanted to
> use a
> > > > different option parsing library.
> > >
> > >
> > > I don't know about "really bizzare"; this is the same problem as
> any
> > > external dependency (e.g. python-dateutils).
> > >
> > > The real issue here is the lack of a standardized runtime
> environment. Is
> > > there build/packaging for Yetus? Some of this could be handled by
> > wrappers
> > > that created virtualenvs or downloaded conda as appropriate.
> > >
> >
> >
> > --
> > -Dima
> >
> --
> Regards
> Ajay Yadava
>
>
>
Re: [DISCUSS] Requiring Python 2.7
Posted by Chris Nauroth <cn...@hortonworks.com>.
I’m just catching up on this thread now.
I fall in the camp that would prefer not to dictate a requirement of Python 2.7 for anything in Yetus. My reasoning agrees with what Allen has already described. I have been in the situation that he describes, needing to maintain build infrastructure on old OSes and old Python versions. Upgrading the OS or installing Python 2.7 via conda or a custom rpm was not an option. IT policies prevented me from performing non-essential/non-security related upgrades or installing new software. Instead, the solution was to stick to Python code compatible with the old versions.
I haven’t reviewed the code for YETUS-445, but the motivation for this topic coming up now seems to be argparse. I think Ajay’s proposal for handling that sounds fine. Alternatively, we could explore if it’s feasible to check in a copy of argparse to be front-loaded onto PYTHONPATH.
--Chris Nauroth
On 8/24/16, 9:45 PM, "Ajay Yadava" <aj...@gmail.com> wrote:
If the only motivation for Python 2.7 for this script is to use argparse
library then it can be installed on Python 2.6 and hence shouldn't be an
issue IMO. We have external dependencies for RDM as well. A simple
try-catch will ensure that 2.6 users know about the external dependency.
Does this work?
On Thu, Aug 25, 2016, 9:22 AM Dima Spivak <ds...@cloudera.com> wrote:
> What if we compromise by creating some sort of designation for tools
> that require runs to use Docker mode? This way, people could still use
> Andrew's awesome script either on their own or in Apache Infra (even
> precommit!) without unfortunate surprises when someone tries to run it on
> an older OS.
>
> On Wednesday, August 24, 2016, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > On Wed, Aug 24, 2016 at 4:46 PM, Allen Wittenauer <
> > aw@effectivemachines.com <javascript:;>>
> > wrote:
> >
> > >
> > > > On Aug 24, 2016, at 3:49 PM, Andrew Wang <andrew.wang@cloudera.com
> > <javascript:;>>
> > > wrote:
> > > > My reasoning here has been that this tool will primarily be run by
> > > release
> > > > managers, and RMs are very likely to have Python 2.7 on their
> machine.
> > >
> > > Whereas I believe the opposite: there's a good chance if one is
> > > the RM for enterprise software, the machine you are building on
> probably
> > > doesn't have python 2.7 on it because you aren't doing this work on
> your
> > > desktop! It's extremely common to have build servers that are several
> OS
> > > revs behind because software tends to be more upwardly compatible....
> > >
> > > > ASF infra and servers in general are not really a target.
> > >
> > > Umm, yeah, they kind of are. While there are certainly issues
> > with
> > > them, they do serve as sort of a model of what is out in the real
> world.
> > > It's old and crufty and that's how a lot of software is built. Where
> > > people build software is *exactly* our target.
> > >
> > > Here's a fun anecdote. Firefox v48 is the first version to
> drop
> > > support for Mac OS X 10.6, 10.7, and 10.8. This means they were almost
> > > certainly building on 10.6. So up until earlier this year, they were
> > using
> > > a 7 year old OS that most definitely didn't have python 2.7 on it by
> > > default. Mozilla, by many accounts, would be considered quite
> aggressive
> > > but they can get away with it because they are targeting the desktop.
> > >
> > >
> > I think we're talking past each other here, since this is not a build
> tool.
> > It's an API checking tool. It only needs to be run on a developer's
> > computer, not all boxes at ASF. And, like I've been saying, you can just
> > install Python 2.7 like you would any other dependency. RDM requires `pip
> > install python-dateutils` for instance.
> >
> >
> > > > Maybe one day if
> > > > we want to run this as part of precommit, but then it can be optional
> > > like
> > > > gradle and docker.
> > >
> > >
> > > It sends really bizarre mixed messages if one tool has
> different
> > > python requirements than another just because someone wanted to use a
> > > different option parsing library.
> >
> >
> > I don't know about "really bizzare"; this is the same problem as any
> > external dependency (e.g. python-dateutils).
> >
> > The real issue here is the lack of a standardized runtime environment. Is
> > there build/packaging for Yetus? Some of this could be handled by
> wrappers
> > that created virtualenvs or downloaded conda as appropriate.
> >
>
>
> --
> -Dima
>
--
Regards
Ajay Yadava
Re: [DISCUSS] Requiring Python 2.7
Posted by Ajay Yadava <aj...@gmail.com>.
If the only motivation for Python 2.7 for this script is to use argparse
library then it can be installed on Python 2.6 and hence shouldn't be an
issue IMO. We have external dependencies for RDM as well. A simple
try-catch will ensure that 2.6 users know about the external dependency.
Does this work?
On Thu, Aug 25, 2016, 9:22 AM Dima Spivak <ds...@cloudera.com> wrote:
> What if we compromise by creating some sort of designation for tools
> that require runs to use Docker mode? This way, people could still use
> Andrew's awesome script either on their own or in Apache Infra (even
> precommit!) without unfortunate surprises when someone tries to run it on
> an older OS.
>
> On Wednesday, August 24, 2016, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > On Wed, Aug 24, 2016 at 4:46 PM, Allen Wittenauer <
> > aw@effectivemachines.com <javascript:;>>
> > wrote:
> >
> > >
> > > > On Aug 24, 2016, at 3:49 PM, Andrew Wang <andrew.wang@cloudera.com
> > <javascript:;>>
> > > wrote:
> > > > My reasoning here has been that this tool will primarily be run by
> > > release
> > > > managers, and RMs are very likely to have Python 2.7 on their
> machine.
> > >
> > > Whereas I believe the opposite: there's a good chance if one is
> > > the RM for enterprise software, the machine you are building on
> probably
> > > doesn't have python 2.7 on it because you aren't doing this work on
> your
> > > desktop! It's extremely common to have build servers that are several
> OS
> > > revs behind because software tends to be more upwardly compatible....
> > >
> > > > ASF infra and servers in general are not really a target.
> > >
> > > Umm, yeah, they kind of are. While there are certainly issues
> > with
> > > them, they do serve as sort of a model of what is out in the real
> world.
> > > It's old and crufty and that's how a lot of software is built. Where
> > > people build software is *exactly* our target.
> > >
> > > Here's a fun anecdote. Firefox v48 is the first version to
> drop
> > > support for Mac OS X 10.6, 10.7, and 10.8. This means they were almost
> > > certainly building on 10.6. So up until earlier this year, they were
> > using
> > > a 7 year old OS that most definitely didn't have python 2.7 on it by
> > > default. Mozilla, by many accounts, would be considered quite
> aggressive
> > > but they can get away with it because they are targeting the desktop.
> > >
> > >
> > I think we're talking past each other here, since this is not a build
> tool.
> > It's an API checking tool. It only needs to be run on a developer's
> > computer, not all boxes at ASF. And, like I've been saying, you can just
> > install Python 2.7 like you would any other dependency. RDM requires `pip
> > install python-dateutils` for instance.
> >
> >
> > > > Maybe one day if
> > > > we want to run this as part of precommit, but then it can be optional
> > > like
> > > > gradle and docker.
> > >
> > >
> > > It sends really bizarre mixed messages if one tool has
> different
> > > python requirements than another just because someone wanted to use a
> > > different option parsing library.
> >
> >
> > I don't know about "really bizzare"; this is the same problem as any
> > external dependency (e.g. python-dateutils).
> >
> > The real issue here is the lack of a standardized runtime environment. Is
> > there build/packaging for Yetus? Some of this could be handled by
> wrappers
> > that created virtualenvs or downloaded conda as appropriate.
> >
>
>
> --
> -Dima
>
--
Regards
Ajay Yadava
Re: [DISCUSS] Requiring Python 2.7
Posted by Dima Spivak <ds...@cloudera.com>.
What if we compromise by creating some sort of designation for tools
that require runs to use Docker mode? This way, people could still use
Andrew's awesome script either on their own or in Apache Infra (even
precommit!) without unfortunate surprises when someone tries to run it on
an older OS.
On Wednesday, August 24, 2016, Andrew Wang <an...@cloudera.com> wrote:
> On Wed, Aug 24, 2016 at 4:46 PM, Allen Wittenauer <
> aw@effectivemachines.com <javascript:;>>
> wrote:
>
> >
> > > On Aug 24, 2016, at 3:49 PM, Andrew Wang <andrew.wang@cloudera.com
> <javascript:;>>
> > wrote:
> > > My reasoning here has been that this tool will primarily be run by
> > release
> > > managers, and RMs are very likely to have Python 2.7 on their machine.
> >
> > Whereas I believe the opposite: there's a good chance if one is
> > the RM for enterprise software, the machine you are building on probably
> > doesn't have python 2.7 on it because you aren't doing this work on your
> > desktop! It's extremely common to have build servers that are several OS
> > revs behind because software tends to be more upwardly compatible....
> >
> > > ASF infra and servers in general are not really a target.
> >
> > Umm, yeah, they kind of are. While there are certainly issues
> with
> > them, they do serve as sort of a model of what is out in the real world.
> > It's old and crufty and that's how a lot of software is built. Where
> > people build software is *exactly* our target.
> >
> > Here's a fun anecdote. Firefox v48 is the first version to drop
> > support for Mac OS X 10.6, 10.7, and 10.8. This means they were almost
> > certainly building on 10.6. So up until earlier this year, they were
> using
> > a 7 year old OS that most definitely didn't have python 2.7 on it by
> > default. Mozilla, by many accounts, would be considered quite aggressive
> > but they can get away with it because they are targeting the desktop.
> >
> >
> I think we're talking past each other here, since this is not a build tool.
> It's an API checking tool. It only needs to be run on a developer's
> computer, not all boxes at ASF. And, like I've been saying, you can just
> install Python 2.7 like you would any other dependency. RDM requires `pip
> install python-dateutils` for instance.
>
>
> > > Maybe one day if
> > > we want to run this as part of precommit, but then it can be optional
> > like
> > > gradle and docker.
> >
> >
> > It sends really bizarre mixed messages if one tool has different
> > python requirements than another just because someone wanted to use a
> > different option parsing library.
>
>
> I don't know about "really bizzare"; this is the same problem as any
> external dependency (e.g. python-dateutils).
>
> The real issue here is the lack of a standardized runtime environment. Is
> there build/packaging for Yetus? Some of this could be handled by wrappers
> that created virtualenvs or downloaded conda as appropriate.
>
--
-Dima
Re: [DISCUSS] Requiring Python 2.7
Posted by Andrew Wang <an...@cloudera.com>.
On Wed, Aug 24, 2016 at 4:46 PM, Allen Wittenauer <aw...@effectivemachines.com>
wrote:
>
> > On Aug 24, 2016, at 3:49 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> > My reasoning here has been that this tool will primarily be run by
> release
> > managers, and RMs are very likely to have Python 2.7 on their machine.
>
> Whereas I believe the opposite: there's a good chance if one is
> the RM for enterprise software, the machine you are building on probably
> doesn't have python 2.7 on it because you aren't doing this work on your
> desktop! It's extremely common to have build servers that are several OS
> revs behind because software tends to be more upwardly compatible....
>
> > ASF infra and servers in general are not really a target.
>
> Umm, yeah, they kind of are. While there are certainly issues with
> them, they do serve as sort of a model of what is out in the real world.
> It's old and crufty and that's how a lot of software is built. Where
> people build software is *exactly* our target.
>
> Here's a fun anecdote. Firefox v48 is the first version to drop
> support for Mac OS X 10.6, 10.7, and 10.8. This means they were almost
> certainly building on 10.6. So up until earlier this year, they were using
> a 7 year old OS that most definitely didn't have python 2.7 on it by
> default. Mozilla, by many accounts, would be considered quite aggressive
> but they can get away with it because they are targeting the desktop.
>
>
I think we're talking past each other here, since this is not a build tool.
It's an API checking tool. It only needs to be run on a developer's
computer, not all boxes at ASF. And, like I've been saying, you can just
install Python 2.7 like you would any other dependency. RDM requires `pip
install python-dateutils` for instance.
> > Maybe one day if
> > we want to run this as part of precommit, but then it can be optional
> like
> > gradle and docker.
>
>
> It sends really bizarre mixed messages if one tool has different
> python requirements than another just because someone wanted to use a
> different option parsing library.
I don't know about "really bizzare"; this is the same problem as any
external dependency (e.g. python-dateutils).
The real issue here is the lack of a standardized runtime environment. Is
there build/packaging for Yetus? Some of this could be handled by wrappers
that created virtualenvs or downloaded conda as appropriate.
Re: [DISCUSS] Requiring Python 2.7
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Aug 24, 2016, at 3:49 PM, Andrew Wang <an...@cloudera.com> wrote:
> My reasoning here has been that this tool will primarily be run by release
> managers, and RMs are very likely to have Python 2.7 on their machine.
Whereas I believe the opposite: there's a good chance if one is the RM for enterprise software, the machine you are building on probably doesn't have python 2.7 on it because you aren't doing this work on your desktop! It's extremely common to have build servers that are several OS revs behind because software tends to be more upwardly compatible....
> ASF infra and servers in general are not really a target.
Umm, yeah, they kind of are. While there are certainly issues with them, they do serve as sort of a model of what is out in the real world. It's old and crufty and that's how a lot of software is built. Where people build software is *exactly* our target.
Here's a fun anecdote. Firefox v48 is the first version to drop support for Mac OS X 10.6, 10.7, and 10.8. This means they were almost certainly building on 10.6. So up until earlier this year, they were using a 7 year old OS that most definitely didn't have python 2.7 on it by default. Mozilla, by many accounts, would be considered quite aggressive but they can get away with it because they are targeting the desktop.
> Maybe one day if
> we want to run this as part of precommit, but then it can be optional like
> gradle and docker.
It sends really bizarre mixed messages if one tool has different python requirements than another just because someone wanted to use a different option parsing library.
Re: [DISCUSS] Requiring Python 2.7
Posted by Andrew Wang <an...@cloudera.com>.
On Wed, Aug 24, 2016 at 3:01 PM, Allen Wittenauer <aw...@effectivemachines.com>
wrote:
>
> > On Aug 24, 2016, at 2:41 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> >
> > Given the presence of conda, I don't see Python 2.7 as that different as
> > any other third-party dependency. For instance, test-patch has all kinds
> of
> > integrations that aren't available out-of-the-box on RHEL6 like Docker
> and
> > Gradle.
>
> As soon as we can make python 2.7 as optional as gradle and
> docker, I'll be all for it.
Well, the check compatibility tool at YETUS-445 is standalone and new code,
so in that sense it is optional. I'm only talking about bumping the
required version for this one tool.
My reasoning here has been that this tool will primarily be run by release
managers, and RMs are very likely to have Python 2.7 on their machine.
ASF infra and servers in general are not really a target. Maybe one day if
we want to run this as part of precommit, but then it can be optional like
gradle and docker.
Re: [DISCUSS] Requiring Python 2.7
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Aug 24, 2016, at 2:41 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> Given the presence of conda, I don't see Python 2.7 as that different as
> any other third-party dependency. For instance, test-patch has all kinds of
> integrations that aren't available out-of-the-box on RHEL6 like Docker and
> Gradle.
As soon as we can make python 2.7 as optional as gradle and docker, I'll be all for it.
Re: [DISCUSS] Requiring Python 2.7
Posted by Andrew Wang <an...@cloudera.com>.
On Wed, Aug 24, 2016 at 2:09 PM, Allen Wittenauer <aw...@effectivemachines.com>
wrote:
>
> > On Aug 24, 2016, at 1:04 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> >
> >> I'm pretty sure you have access to a machine with Python 2.7 though, or
> > the means to install it. That's what I meant by "no one runs RHEL6 on
> their
> > personal dev machine".
> >
> > Same applies to the alternative OSes.
>
> ... and with that, you've pretty much convinced me that setting
> the minimum to python 2.7 is not worth pursuing.
>
> "Oh you should go build half of your software over here and the
> other half on this other box and then manually merge the results" is not
> very user-friendly at all.
>
> I didn't say you should build half here and half there, I said that you
could either:
a) run your entire workflow on a more modern machine, that has Python 2.7
via a package manager
b) install Python 2.7 via conda
Given the presence of conda, I don't see Python 2.7 as that different as
any other third-party dependency. For instance, test-patch has all kinds of
integrations that aren't available out-of-the-box on RHEL6 like Docker and
Gradle. If people can install those, they can definitely also install
Python 2.7 via conda.
Re: [DISCUSS] Requiring Python 2.7
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Aug 24, 2016, at 1:04 PM, Andrew Wang <an...@cloudera.com> wrote:
>
>> I'm pretty sure you have access to a machine with Python 2.7 though, or
> the means to install it. That's what I meant by "no one runs RHEL6 on their
> personal dev machine".
>
> Same applies to the alternative OSes.
... and with that, you've pretty much convinced me that setting the minimum to python 2.7 is not worth pursuing.
"Oh you should go build half of your software over here and the other half on this other box and then manually merge the results" is not very user-friendly at all.
Re: [DISCUSS] Requiring Python 2.7
Posted by Andrew Wang <an...@cloudera.com>.
>
> That also doesn't cover other operating systems... I think
> Solaris 10 shipped with python 2.6 and it's still very much supported by
> Oracle. It also happens to be what the ASF Solaris boxes are running... I
> think AIX is in a similar situation.
>
> > My take on it:
> >
> > * This tool is aimed at release managers, and no one runs RHEL6 on their
> > personal dev machine.
>
> *looks at VM running CentOS 6 where he tested the release of Yetus
> as the RM and other bits before*
>
> I'm pretty sure you have access to a machine with Python 2.7 though, or
the means to install it. That's what I meant by "no one runs RHEL6 on their
personal dev machine".
Same applies to the alternative OSes.
> > * Even for build machines, Docker is not supported on RHEL6 (
> > https://access.redhat.com/solutions/1378023). FWIW, Docker is a
> requirement
> > for building a Hadoop release and used by Hadoop precommit.
>
> That doesn't really matter much. ASF and Hadoop != rest of
> universe. We're trying to build something usable by lots of folks here...
>
> This was just a comment on RHEL6 being old and not developer friendly.
> * If you really do need to use RHEL6, you can use conda (
> > http://conda.pydata.org/miniconda.html) to install Python 2.7 quite
> easily.
> > I've used it, and it's great.
> >
> > Thoughts?
>
> The only thing I'm convinced of is that python is still horribly
> designed. :) We should rewrite it all in perl5. :)
>
Could you clarify your position on bumping to 2.7, particular if conda
suffices?
Thanks,
Andrew
Re: [DISCUSS] Requiring Python 2.7
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Aug 24, 2016, at 12:59 PM, Dima Spivak <ds...@cloudera.com> wrote:
>
> While I definitely appreciate your position, Allen, I agree with Andrew
> here. As much as I love Bash (just ask my frustrated colleagues...),
It's funny that you brought up bash in this context.
There are lots and lots and lots of times I wish test-patch would have been built with bash 4. It would make life a lot easier in lots of places. So why don't we use it? Because it would cut out a chunk of the user base.
I view the python 2.7 argument (at least today) in the same vein. I'm not convinced that we're in a place where saying that 2.7 is the minimum is viable, especially when parts of the ASF's own infrastructure can't run it. That's extremely hypocritical.
> I worry that we'll close the door on a lot of useful
> contributions if we maintain this as a requirement.
I think it should be understood by contributors that one of the challenges of working with QA tools is that there is an expectation that they should work in particularly rugged environments. They generally don't get the privilege of dictating requirements. These tools are being used to help make some other project better. Apache Yetus is not the core of the end product, just one of many cogs. The more requirements we put in place, the fewer places the project will be invited to help out.
Re: [DISCUSS] Requiring Python 2.7
Posted by Dima Spivak <ds...@cloudera.com>.
While I definitely appreciate your position, Allen, I agree with Andrew
here. As much as I love Bash (just ask my frustrated colleagues...), Python
really makes life significantly easier when it comes to maintainable
scripts, 2.7 in particular. The example over in YETUS-445 specifically
pointed to argparse; it's not fun to write command line parsers with
inferior functionality and readability to maintain support for a 10 year
old interpreter. I worry that we'll close the door on a lot of useful
contributions if we maintain this as a requirement.
On Wednesday, August 24, 2016, Allen Wittenauer <aw...@effectivemachines.com>
wrote:
>
> > On Aug 24, 2016, at 12:21 PM, Andrew Wang <andrew.wang@cloudera.com
> <javascript:;>> wrote:
> >
> > Hi dev@,
> >
> > This topic came up on YETUS-445. I'd like to make the case for requiring
> > Python 2.7.
> >
> > As a bit of background, the main offender here is RHEL6. RHEL7 has been
> out
> > for two years and has some penetration, but RHEL6 is still out there and
> > supported.
>
> I think I know of maybe two sizable shops that are using RHEL7.
> Everyone else is either still on RHEL6 or derivatives. (FWIW: mainly due
> to systemd. One company I know of has gone so far to avoid systemd that
> they forked from CentOS 6 and are now building their own distributions.)
>
> That also doesn't cover other operating systems... I think
> Solaris 10 shipped with python 2.6 and it's still very much supported by
> Oracle. It also happens to be what the ASF Solaris boxes are running... I
> think AIX is in a similar situation.
>
> > My take on it:
> >
> > * This tool is aimed at release managers, and no one runs RHEL6 on their
> > personal dev machine.
>
> *looks at VM running CentOS 6 where he tested the release of Yetus
> as the RM and other bits before*
>
> > * Even for build machines, Docker is not supported on RHEL6 (
> > https://access.redhat.com/solutions/1378023). FWIW, Docker is a
> requirement
> > for building a Hadoop release and used by Hadoop precommit.
>
> That doesn't really matter much. ASF and Hadoop != rest of
> universe. We're trying to build something usable by lots of folks here...
>
> > * If you really do need to use RHEL6, you can use conda (
> > http://conda.pydata.org/miniconda.html) to install Python 2.7 quite
> easily.
> > I've used it, and it's great.
> >
> > Thoughts?
>
> The only thing I'm convinced of is that python is still horribly
> designed. :) We should rewrite it all in perl5. :)
>
>
>
--
-Dima
Re: [DISCUSS] Requiring Python 2.7
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Aug 24, 2016, at 12:21 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> Hi dev@,
>
> This topic came up on YETUS-445. I'd like to make the case for requiring
> Python 2.7.
>
> As a bit of background, the main offender here is RHEL6. RHEL7 has been out
> for two years and has some penetration, but RHEL6 is still out there and
> supported.
I think I know of maybe two sizable shops that are using RHEL7. Everyone else is either still on RHEL6 or derivatives. (FWIW: mainly due to systemd. One company I know of has gone so far to avoid systemd that they forked from CentOS 6 and are now building their own distributions.)
That also doesn't cover other operating systems... I think Solaris 10 shipped with python 2.6 and it's still very much supported by Oracle. It also happens to be what the ASF Solaris boxes are running... I think AIX is in a similar situation.
> My take on it:
>
> * This tool is aimed at release managers, and no one runs RHEL6 on their
> personal dev machine.
*looks at VM running CentOS 6 where he tested the release of Yetus as the RM and other bits before*
> * Even for build machines, Docker is not supported on RHEL6 (
> https://access.redhat.com/solutions/1378023). FWIW, Docker is a requirement
> for building a Hadoop release and used by Hadoop precommit.
That doesn't really matter much. ASF and Hadoop != rest of universe. We're trying to build something usable by lots of folks here...
> * If you really do need to use RHEL6, you can use conda (
> http://conda.pydata.org/miniconda.html) to install Python 2.7 quite easily.
> I've used it, and it's great.
>
> Thoughts?
The only thing I'm convinced of is that python is still horribly designed. :) We should rewrite it all in perl5. :)