You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "EKC (Erik Cederstrand)" <EK...@novozymes.com.INVALID> on 2018/10/03 10:01:00 UTC
"setup.py test" is being naughty
Hi all,
I wanted to contribute a simple patch, and as a good open source citizen I wanted to also contribute a test. So I git clone from GitHub, create a virtualenv and run "setup.py test". First experience is that my /etc/krb5.conf is overwritten, which means my account is locked out of all systems here at work. I recovered from that, only to find out that ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub were also overwritten. Now I'm not very amused.
Did I miss something in CONTRIBUTING.md?
Erik
Re: "setup.py test" is being naughty
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
We're working hard to get rid of the tight Travis integration and moving to
a Docker based setup. I think it should be very easy to get a Docker up and
running which is packed with the required dependencies. Unfortunately we're
not there yet. Also the tox layer feels a bit redundant to me, since we're
using Docker now.
Cheers, Fokko
Op wo 3 okt. 2018 om 15:08 schreef Jarek Potiuk <Ja...@polidea.com>:
> Local testing works well for a number of unit tests when run from the IDE.
> We of course run full suite of tests via docker environment but our own
> test classess/modules are run using local python environment. It's the
> easiest way to configure local python virtualenv with IntelliJ/Pycharm for
> one. You can - in recent version of PyCharm/IntelliJ - have docker python
> environment setup, but there are certain downsides of using it
> (speed/mounting local volumes with sources etc.).
>
> So I think we should not really discourage running at least some tests
> locally. Maybe (if there are not many of those) we could identify the tests
> which require the full-blown docker environment and mark them with
> skipUnless and only have them executed when we are inside dockerized
> environment for unit tests ?
>
> J.
>
>
> On Wed, Oct 3, 2018 at 1:48 PM Holden Karau <ho...@pigscanfly.ca> wrote:
>
> > I think (in the short term) discontinuing local testing and telling folks
> > to use the docker based approach makes more sense (many of the tests
> have a
> > complex set of dependencies that don't make sense to try and test
> locally).
> > What do other folks think?
> >
> > On Wed, Oct 3, 2018 at 4:45 AM EKC (Erik Cederstrand)
> > <EK...@novozymes.com.invalid> wrote:
> >
> > > The test suite is also trying to create /usr/local/bin/airflow, which
> > > means I can't run the test suite on a machine that actually uses
> > > /usr/local/bin/airflow. And the default config file doesn't find the
> > MySQL
> > > server I set up locally. I'm trying the Docker-based test environment
> > now.
> > >
> > >
> > > It seems the local test setup either needs polishing or should be
> > > discontinued.
> > >
> > >
> > > Erik
> > >
> > > ________________________________
> > > From: EKC (Erik Cederstrand)
> > > Sent: Wednesday, October 3, 2018 12:01:00 PM
> > > To: dev@airflow.incubator.apache.org
> > > Subject: "setup.py test" is being naughty
> > >
> > >
> > > Hi all,
> > >
> > >
> > > I wanted to contribute a simple patch, and as a good open source
> citizen
> > I
> > > wanted to also contribute a test. So I git clone from GitHub, create a
> > > virtualenv and run "setup.py test". First experience is that my
> > > /etc/krb5.conf is overwritten, which means my account is locked out of
> > all
> > > systems here at work. I recovered from that, only to find out that
> > > ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub were also overwritten. Now I'm not
> > very
> > > amused.
> > >
> > >
> > > Did I miss something in CONTRIBUTING.md?
> > >
> > >
> > > Erik
> > >
> >
> >
> > --
> > Twitter: https://twitter.com/holdenkarau
> > Books (Learning Spark, High Performance Spark, etc.):
> > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> >
>
>
> --
>
> *Jarek Potiuk, Principal Software Engineer*
> Mobile: +48 660 796 129
>
Re: "setup.py test" is being naughty
Posted by Jarek Potiuk <Ja...@polidea.com>.
Local testing works well for a number of unit tests when run from the IDE.
We of course run full suite of tests via docker environment but our own
test classess/modules are run using local python environment. It's the
easiest way to configure local python virtualenv with IntelliJ/Pycharm for
one. You can - in recent version of PyCharm/IntelliJ - have docker python
environment setup, but there are certain downsides of using it
(speed/mounting local volumes with sources etc.).
So I think we should not really discourage running at least some tests
locally. Maybe (if there are not many of those) we could identify the tests
which require the full-blown docker environment and mark them with
skipUnless and only have them executed when we are inside dockerized
environment for unit tests ?
J.
On Wed, Oct 3, 2018 at 1:48 PM Holden Karau <ho...@pigscanfly.ca> wrote:
> I think (in the short term) discontinuing local testing and telling folks
> to use the docker based approach makes more sense (many of the tests have a
> complex set of dependencies that don't make sense to try and test locally).
> What do other folks think?
>
> On Wed, Oct 3, 2018 at 4:45 AM EKC (Erik Cederstrand)
> <EK...@novozymes.com.invalid> wrote:
>
> > The test suite is also trying to create /usr/local/bin/airflow, which
> > means I can't run the test suite on a machine that actually uses
> > /usr/local/bin/airflow. And the default config file doesn't find the
> MySQL
> > server I set up locally. I'm trying the Docker-based test environment
> now.
> >
> >
> > It seems the local test setup either needs polishing or should be
> > discontinued.
> >
> >
> > Erik
> >
> > ________________________________
> > From: EKC (Erik Cederstrand)
> > Sent: Wednesday, October 3, 2018 12:01:00 PM
> > To: dev@airflow.incubator.apache.org
> > Subject: "setup.py test" is being naughty
> >
> >
> > Hi all,
> >
> >
> > I wanted to contribute a simple patch, and as a good open source citizen
> I
> > wanted to also contribute a test. So I git clone from GitHub, create a
> > virtualenv and run "setup.py test". First experience is that my
> > /etc/krb5.conf is overwritten, which means my account is locked out of
> all
> > systems here at work. I recovered from that, only to find out that
> > ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub were also overwritten. Now I'm not
> very
> > amused.
> >
> >
> > Did I miss something in CONTRIBUTING.md?
> >
> >
> > Erik
> >
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
--
*Jarek Potiuk, Principal Software Engineer*
Mobile: +48 660 796 129
Re: "setup.py test" is being naughty
Posted by Holden Karau <ho...@pigscanfly.ca>.
I think (in the short term) discontinuing local testing and telling folks
to use the docker based approach makes more sense (many of the tests have a
complex set of dependencies that don't make sense to try and test locally).
What do other folks think?
On Wed, Oct 3, 2018 at 4:45 AM EKC (Erik Cederstrand)
<EK...@novozymes.com.invalid> wrote:
> The test suite is also trying to create /usr/local/bin/airflow, which
> means I can't run the test suite on a machine that actually uses
> /usr/local/bin/airflow. And the default config file doesn't find the MySQL
> server I set up locally. I'm trying the Docker-based test environment now.
>
>
> It seems the local test setup either needs polishing or should be
> discontinued.
>
>
> Erik
>
> ________________________________
> From: EKC (Erik Cederstrand)
> Sent: Wednesday, October 3, 2018 12:01:00 PM
> To: dev@airflow.incubator.apache.org
> Subject: "setup.py test" is being naughty
>
>
> Hi all,
>
>
> I wanted to contribute a simple patch, and as a good open source citizen I
> wanted to also contribute a test. So I git clone from GitHub, create a
> virtualenv and run "setup.py test". First experience is that my
> /etc/krb5.conf is overwritten, which means my account is locked out of all
> systems here at work. I recovered from that, only to find out that
> ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub were also overwritten. Now I'm not very
> amused.
>
>
> Did I miss something in CONTRIBUTING.md?
>
>
> Erik
>
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: "setup.py test" is being naughty
Posted by "EKC (Erik Cederstrand)" <EK...@novozymes.com.INVALID>.
The test suite is also trying to create /usr/local/bin/airflow, which means I can't run the test suite on a machine that actually uses /usr/local/bin/airflow. And the default config file doesn't find the MySQL server I set up locally. I'm trying the Docker-based test environment now.
It seems the local test setup either needs polishing or should be discontinued.
Erik
________________________________
From: EKC (Erik Cederstrand)
Sent: Wednesday, October 3, 2018 12:01:00 PM
To: dev@airflow.incubator.apache.org
Subject: "setup.py test" is being naughty
Hi all,
I wanted to contribute a simple patch, and as a good open source citizen I wanted to also contribute a test. So I git clone from GitHub, create a virtualenv and run "setup.py test". First experience is that my /etc/krb5.conf is overwritten, which means my account is locked out of all systems here at work. I recovered from that, only to find out that ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub were also overwritten. Now I'm not very amused.
Did I miss something in CONTRIBUTING.md?
Erik
Re: "setup.py test" is being naughty
Posted by Holden Karau <ho...@pigscanfly.ca>.
That's a really good point. The docs suggest running in docker but we don't
warn people about the downsides of running the tests outside of docker and
we should probably at least warn folks if it looks like they are running on
bare metal.
On Wed, Oct 3, 2018 at 3:01 AM EKC (Erik Cederstrand)
<EK...@novozymes.com.invalid> wrote:
> Hi all,
>
>
> I wanted to contribute a simple patch, and as a good open source citizen I
> wanted to also contribute a test. So I git clone from GitHub, create a
> virtualenv and run "setup.py test". First experience is that my
> /etc/krb5.conf is overwritten, which means my account is locked out of all
> systems here at work. I recovered from that, only to find out that
> ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub were also overwritten. Now I'm not very
> amused.
>
>
> Did I miss something in CONTRIBUTING.md?
>
>
> Erik
>
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau