You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yetus.apache.org by Chris Nauroth <cn...@hortonworks.com> on 2016/09/06 17:23:00 UTC
Re: [DISCUSS] Requiring Python 2.7
I’m just catching up on this thread now.
I fall in the camp that would prefer not to dictate a requirement of Python 2.7 for anything in Yetus. My reasoning agrees with what Allen has already described. I have been in the situation that he describes, needing to maintain build infrastructure on old OSes and old Python versions. Upgrading the OS or installing Python 2.7 via conda or a custom rpm was not an option. IT policies prevented me from performing non-essential/non-security related upgrades or installing new software. Instead, the solution was to stick to Python code compatible with the old versions.
I haven’t reviewed the code for YETUS-445, but the motivation for this topic coming up now seems to be argparse. I think Ajay’s proposal for handling that sounds fine. Alternatively, we could explore if it’s feasible to check in a copy of argparse to be front-loaded onto PYTHONPATH.
--Chris Nauroth
On 8/24/16, 9:45 PM, "Ajay Yadava" <aj...@gmail.com> wrote:
If the only motivation for Python 2.7 for this script is to use argparse
library then it can be installed on Python 2.6 and hence shouldn't be an
issue IMO. We have external dependencies for RDM as well. A simple
try-catch will ensure that 2.6 users know about the external dependency.
Does this work?
On Thu, Aug 25, 2016, 9:22 AM Dima Spivak <ds...@cloudera.com> wrote:
> What if we compromise by creating some sort of designation for tools
> that require runs to use Docker mode? This way, people could still use
> Andrew's awesome script either on their own or in Apache Infra (even
> precommit!) without unfortunate surprises when someone tries to run it on
> an older OS.
>
> On Wednesday, August 24, 2016, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > On Wed, Aug 24, 2016 at 4:46 PM, Allen Wittenauer <
> > aw@effectivemachines.com <javascript:;>>
> > wrote:
> >
> > >
> > > > On Aug 24, 2016, at 3:49 PM, Andrew Wang <andrew.wang@cloudera.com
> > <javascript:;>>
> > > wrote:
> > > > My reasoning here has been that this tool will primarily be run by
> > > release
> > > > managers, and RMs are very likely to have Python 2.7 on their
> machine.
> > >
> > > Whereas I believe the opposite: there's a good chance if one is
> > > the RM for enterprise software, the machine you are building on
> probably
> > > doesn't have python 2.7 on it because you aren't doing this work on
> your
> > > desktop! It's extremely common to have build servers that are several
> OS
> > > revs behind because software tends to be more upwardly compatible....
> > >
> > > > ASF infra and servers in general are not really a target.
> > >
> > > Umm, yeah, they kind of are. While there are certainly issues
> > with
> > > them, they do serve as sort of a model of what is out in the real
> world.
> > > It's old and crufty and that's how a lot of software is built. Where
> > > people build software is *exactly* our target.
> > >
> > > Here's a fun anecdote. Firefox v48 is the first version to
> drop
> > > support for Mac OS X 10.6, 10.7, and 10.8. This means they were almost
> > > certainly building on 10.6. So up until earlier this year, they were
> > using
> > > a 7 year old OS that most definitely didn't have python 2.7 on it by
> > > default. Mozilla, by many accounts, would be considered quite
> aggressive
> > > but they can get away with it because they are targeting the desktop.
> > >
> > >
> > I think we're talking past each other here, since this is not a build
> tool.
> > It's an API checking tool. It only needs to be run on a developer's
> > computer, not all boxes at ASF. And, like I've been saying, you can just
> > install Python 2.7 like you would any other dependency. RDM requires `pip
> > install python-dateutils` for instance.
> >
> >
> > > > Maybe one day if
> > > > we want to run this as part of precommit, but then it can be optional
> > > like
> > > > gradle and docker.
> > >
> > >
> > > It sends really bizarre mixed messages if one tool has
> different
> > > python requirements than another just because someone wanted to use a
> > > different option parsing library.
> >
> >
> > I don't know about "really bizzare"; this is the same problem as any
> > external dependency (e.g. python-dateutils).
> >
> > The real issue here is the lack of a standardized runtime environment. Is
> > there build/packaging for Yetus? Some of this could be handled by
> wrappers
> > that created virtualenvs or downloaded conda as appropriate.
> >
>
>
> --
> -Dima
>
--
Regards
Ajay Yadava
Re: [DISCUSS] Requiring Python 2.7
Posted by Chris Nauroth <cn...@hortonworks.com>.
In my own experience, I’ve had to work under all of the following policies:
1. Write Python code however you want, and any necessary upgrades are fine.
2. You must stick to Python version X, but installing additional third-party libraries is acceptable.
3. You must stick to “vanilla” Python version X, and additional third-party libraries cannot be installed.
My suggestion to consider bundling argparse and front-loading it on PYTHONPATH is a workaround that I have used in scenario 3 in some cases (when the library’s licensing allowed it). It’s a trade-off between the convenience of whatever the library in question provides and the additional build and deployment complexity. This was considered acceptable under policy, because use of the library was constrained to my application and not installed globally.
However, as you pointed out, we’re not currently doing this in Yetus. We currently prompt to install the additional libraries. I think this is acceptable. Policy 3 was an unusual circumstance for me. If we get a user asking for vanilla Python 2.6 support, then it’s something we could explore again.
--Chris Nauroth
On 9/6/16, 10:35 AM, "Andrew Wang" <an...@cloudera.com> wrote:
I already updated YETUS-445 for Python 2.6 support, so it's not blocked on
this conversation.
Chris, in the situation you describe where "IT policies prevented me from
performing non-essential/non-security related upgrades or installing new
software", does this policy not also apply to third party libraries
installed via pip? I honestly don't see the difference, since both conda
and pip can be used without root, and install things to your local homedir.
This is why I keep bringing up third-party libraries. If the restriction is
not just Python 2.6 but vanilla Python 2.6, that's also something that
should be documented, and we should retrofit RDM and YETUS-445 to remove
those dependencies.
On Tue, Sep 6, 2016 at 10:23 AM, Chris Nauroth <cn...@hortonworks.com>
wrote:
> I’m just catching up on this thread now.
>
> I fall in the camp that would prefer not to dictate a requirement of
> Python 2.7 for anything in Yetus. My reasoning agrees with what Allen has
> already described. I have been in the situation that he describes, needing
> to maintain build infrastructure on old OSes and old Python versions.
> Upgrading the OS or installing Python 2.7 via conda or a custom rpm was not
> an option. IT policies prevented me from performing
> non-essential/non-security related upgrades or installing new software.
> Instead, the solution was to stick to Python code compatible with the old
> versions.
>
> I haven’t reviewed the code for YETUS-445, but the motivation for this
> topic coming up now seems to be argparse. I think Ajay’s proposal for
> handling that sounds fine. Alternatively, we could explore if it’s
> feasible to check in a copy of argparse to be front-loaded onto PYTHONPATH.
>
> --Chris Nauroth
>
> On 8/24/16, 9:45 PM, "Ajay Yadava" <aj...@gmail.com> wrote:
>
> If the only motivation for Python 2.7 for this script is to use
> argparse
> library then it can be installed on Python 2.6 and hence shouldn't be
> an
> issue IMO. We have external dependencies for RDM as well. A simple
> try-catch will ensure that 2.6 users know about the external
> dependency.
> Does this work?
>
> On Thu, Aug 25, 2016, 9:22 AM Dima Spivak <ds...@cloudera.com>
> wrote:
>
> > What if we compromise by creating some sort of designation for tools
> > that require runs to use Docker mode? This way, people could still
> use
> > Andrew's awesome script either on their own or in Apache Infra (even
> > precommit!) without unfortunate surprises when someone tries to run
> it on
> > an older OS.
> >
> > On Wednesday, August 24, 2016, Andrew Wang <andrew.wang@cloudera.com
> >
> > wrote:
> >
> > > On Wed, Aug 24, 2016 at 4:46 PM, Allen Wittenauer <
> > > aw@effectivemachines.com <javascript:;>>
> > > wrote:
> > >
> > > >
> > > > > On Aug 24, 2016, at 3:49 PM, Andrew Wang <
> andrew.wang@cloudera.com
> > > <javascript:;>>
> > > > wrote:
> > > > > My reasoning here has been that this tool will primarily be
> run by
> > > > release
> > > > > managers, and RMs are very likely to have Python 2.7 on their
> > machine.
> > > >
> > > > Whereas I believe the opposite: there's a good chance if
> one is
> > > > the RM for enterprise software, the machine you are building on
> > probably
> > > > doesn't have python 2.7 on it because you aren't doing this work
> on
> > your
> > > > desktop! It's extremely common to have build servers that are
> several
> > OS
> > > > revs behind because software tends to be more upwardly
> compatible....
> > > >
> > > > > ASF infra and servers in general are not really a target.
> > > >
> > > > Umm, yeah, they kind of are. While there are certainly
> issues
> > > with
> > > > them, they do serve as sort of a model of what is out in the real
> > world.
> > > > It's old and crufty and that's how a lot of software is built.
> Where
> > > > people build software is *exactly* our target.
> > > >
> > > > Here's a fun anecdote. Firefox v48 is the first version
> to
> > drop
> > > > support for Mac OS X 10.6, 10.7, and 10.8. This means they were
> almost
> > > > certainly building on 10.6. So up until earlier this year, they
> were
> > > using
> > > > a 7 year old OS that most definitely didn't have python 2.7 on
> it by
> > > > default. Mozilla, by many accounts, would be considered quite
> > aggressive
> > > > but they can get away with it because they are targeting the
> desktop.
> > > >
> > > >
> > > I think we're talking past each other here, since this is not a
> build
> > tool.
> > > It's an API checking tool. It only needs to be run on a developer's
> > > computer, not all boxes at ASF. And, like I've been saying, you
> can just
> > > install Python 2.7 like you would any other dependency. RDM
> requires `pip
> > > install python-dateutils` for instance.
> > >
> > >
> > > > > Maybe one day if
> > > > > we want to run this as part of precommit, but then it can be
> optional
> > > > like
> > > > > gradle and docker.
> > > >
> > > >
> > > > It sends really bizarre mixed messages if one tool has
> > different
> > > > python requirements than another just because someone wanted to
> use a
> > > > different option parsing library.
> > >
> > >
> > > I don't know about "really bizzare"; this is the same problem as
> any
> > > external dependency (e.g. python-dateutils).
> > >
> > > The real issue here is the lack of a standardized runtime
> environment. Is
> > > there build/packaging for Yetus? Some of this could be handled by
> > wrappers
> > > that created virtualenvs or downloaded conda as appropriate.
> > >
> >
> >
> > --
> > -Dima
> >
> --
> Regards
> Ajay Yadava
>
>
>
Re: [DISCUSS] Requiring Python 2.7
Posted by Andrew Wang <an...@cloudera.com>.
I already updated YETUS-445 for Python 2.6 support, so it's not blocked on
this conversation.
Chris, in the situation you describe where "IT policies prevented me from
performing non-essential/non-security related upgrades or installing new
software", does this policy not also apply to third party libraries
installed via pip? I honestly don't see the difference, since both conda
and pip can be used without root, and install things to your local homedir.
This is why I keep bringing up third-party libraries. If the restriction is
not just Python 2.6 but vanilla Python 2.6, that's also something that
should be documented, and we should retrofit RDM and YETUS-445 to remove
those dependencies.
On Tue, Sep 6, 2016 at 10:23 AM, Chris Nauroth <cn...@hortonworks.com>
wrote:
> I’m just catching up on this thread now.
>
> I fall in the camp that would prefer not to dictate a requirement of
> Python 2.7 for anything in Yetus. My reasoning agrees with what Allen has
> already described. I have been in the situation that he describes, needing
> to maintain build infrastructure on old OSes and old Python versions.
> Upgrading the OS or installing Python 2.7 via conda or a custom rpm was not
> an option. IT policies prevented me from performing
> non-essential/non-security related upgrades or installing new software.
> Instead, the solution was to stick to Python code compatible with the old
> versions.
>
> I haven’t reviewed the code for YETUS-445, but the motivation for this
> topic coming up now seems to be argparse. I think Ajay’s proposal for
> handling that sounds fine. Alternatively, we could explore if it’s
> feasible to check in a copy of argparse to be front-loaded onto PYTHONPATH.
>
> --Chris Nauroth
>
> On 8/24/16, 9:45 PM, "Ajay Yadava" <aj...@gmail.com> wrote:
>
> If the only motivation for Python 2.7 for this script is to use
> argparse
> library then it can be installed on Python 2.6 and hence shouldn't be
> an
> issue IMO. We have external dependencies for RDM as well. A simple
> try-catch will ensure that 2.6 users know about the external
> dependency.
> Does this work?
>
> On Thu, Aug 25, 2016, 9:22 AM Dima Spivak <ds...@cloudera.com>
> wrote:
>
> > What if we compromise by creating some sort of designation for tools
> > that require runs to use Docker mode? This way, people could still
> use
> > Andrew's awesome script either on their own or in Apache Infra (even
> > precommit!) without unfortunate surprises when someone tries to run
> it on
> > an older OS.
> >
> > On Wednesday, August 24, 2016, Andrew Wang <andrew.wang@cloudera.com
> >
> > wrote:
> >
> > > On Wed, Aug 24, 2016 at 4:46 PM, Allen Wittenauer <
> > > aw@effectivemachines.com <javascript:;>>
> > > wrote:
> > >
> > > >
> > > > > On Aug 24, 2016, at 3:49 PM, Andrew Wang <
> andrew.wang@cloudera.com
> > > <javascript:;>>
> > > > wrote:
> > > > > My reasoning here has been that this tool will primarily be
> run by
> > > > release
> > > > > managers, and RMs are very likely to have Python 2.7 on their
> > machine.
> > > >
> > > > Whereas I believe the opposite: there's a good chance if
> one is
> > > > the RM for enterprise software, the machine you are building on
> > probably
> > > > doesn't have python 2.7 on it because you aren't doing this work
> on
> > your
> > > > desktop! It's extremely common to have build servers that are
> several
> > OS
> > > > revs behind because software tends to be more upwardly
> compatible....
> > > >
> > > > > ASF infra and servers in general are not really a target.
> > > >
> > > > Umm, yeah, they kind of are. While there are certainly
> issues
> > > with
> > > > them, they do serve as sort of a model of what is out in the real
> > world.
> > > > It's old and crufty and that's how a lot of software is built.
> Where
> > > > people build software is *exactly* our target.
> > > >
> > > > Here's a fun anecdote. Firefox v48 is the first version
> to
> > drop
> > > > support for Mac OS X 10.6, 10.7, and 10.8. This means they were
> almost
> > > > certainly building on 10.6. So up until earlier this year, they
> were
> > > using
> > > > a 7 year old OS that most definitely didn't have python 2.7 on
> it by
> > > > default. Mozilla, by many accounts, would be considered quite
> > aggressive
> > > > but they can get away with it because they are targeting the
> desktop.
> > > >
> > > >
> > > I think we're talking past each other here, since this is not a
> build
> > tool.
> > > It's an API checking tool. It only needs to be run on a developer's
> > > computer, not all boxes at ASF. And, like I've been saying, you
> can just
> > > install Python 2.7 like you would any other dependency. RDM
> requires `pip
> > > install python-dateutils` for instance.
> > >
> > >
> > > > > Maybe one day if
> > > > > we want to run this as part of precommit, but then it can be
> optional
> > > > like
> > > > > gradle and docker.
> > > >
> > > >
> > > > It sends really bizarre mixed messages if one tool has
> > different
> > > > python requirements than another just because someone wanted to
> use a
> > > > different option parsing library.
> > >
> > >
> > > I don't know about "really bizzare"; this is the same problem as
> any
> > > external dependency (e.g. python-dateutils).
> > >
> > > The real issue here is the lack of a standardized runtime
> environment. Is
> > > there build/packaging for Yetus? Some of this could be handled by
> > wrappers
> > > that created virtualenvs or downloaded conda as appropriate.
> > >
> >
> >
> > --
> > -Dima
> >
> --
> Regards
> Ajay Yadava
>
>
>