You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2019/09/02 16:06:03 UTC
Re: "Breeze" dev environment
Hey Kevin,
Thanks for those - really helpful explanations! Sorry for spamming everyone
but I post answers here as maybe some people might find it useful for their
own usage of Breeze. I am also adding Elena - who is the technical writer
who started to work on onboarding documentation so that she can see the
kind of issues and questions and problems developers might have with
Airflow environment.
On Fri, Aug 30, 2019 at 5:34 AM Kevin Yang <yr...@gmail.com> wrote:
> Oh sry somehow I missed a detail there, it was the virtualenv use case
> where I need to do the pip install. The docker approach went smoothily. I
> recall two test cases where I need to install packages, one is from the gcp
> operator tests and the other is a celery executor test( sry I can't
> remember the exact test names).
>
>
Regarding the virtualenv - the `*--initialize-virtualenv*` flag on purpose
installs only `*devel*` extras (not `*devel_all*` nor `*devel_ci*` nor even
`*devel_minreq*`) - you are not able to run fully-fledged airflow with the
virtualenv only. This way you have much bigger chance that it will actually
work out-of-the-box on your host machine and allow you to run most of the '
*unit*' tests with sqlite database only. This virtualenv is not really
intended to run more complex system tests that require GCP or celery
executor tests. There are a number of prerequisites that are outside of pip
that your host would have to fulfill (for example installed mysql
client/server, hadoop clients, kubernetes etc.). When you have both Linux
and MacOS environments for those it's not a trivial thing to setup and
support. That's why we have the integration test environment in the first
place - to make it common across all the OS/environments and configuration
each developer has on their hosts. It Dockerises all such dependencies and
adds a number of scripts that makes it rather independent from host
configuration. The idea is that this Breeze environment and not the Local
Virtualenv is used to run most of the non-trivial (integration/system
tests) and to run Airflow itself.
You can still add other extras via `pip install .[EXTRA]` and you can
install other extras there (mysql/hadoop/gcp etc.). No problem with it :).
Some "integration" (GCP and others) tests will even work in this
environment as they just need the pip packages and nothing else to run so
you can use it this way. Maybe we can make it explicit in the new
documentation. We are also starting some discussions on AIP-4
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>
that
might result in clear separation of Unit tests/Integration tests/System
tests so that we know which tests run in which environment.
> (One more little thing: `./breeze --build-docs` seems to fail, I've being
> using `./scripts/ci/ci_docs.sh` instead. And wanna use this opportunity to
> complain that debugging doc building error is really a headache :) )
>
The doc building via breeze should be fixed now (both master and
v1-10-tests). Also in the new approach (as of few weeks or days - can't
remember :) ) debugging doc errors should be much easier than it used to.
When you build docs (either via breeze or via script directly) we do not
remove now the temporary auto-generated API documentation. The errors
communicated by sphinx should then refer to the actual files available in
'_api' directory after you run the script. Those auto-generated files are
generated in files that are mounted from your source's "docs" directory and
you should see the generated files there.
I think that should help a lot when dealing with documentation errors :).
It helped me a lot at least.
>
> On Thu, Aug 29, 2019 at 4:29 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > Fantastic. Thanks. Do you recall which tests/packages they were? I'd love
> > to see whats's the root cause. We are using the same image in CI and the
> > tests pass, so I wonder what could be the reason for that.
> >
> > On Thu, Aug 29, 2019 at 4:13 AM Kevin Yang <yr...@gmail.com> wrote:
> >
> > > Was looking forward to this since last meet up in bay area. Tried both
> > the
> > > docker solution and the virtualenv IDE( PyCharm) solution today and it
> > was
> > > great. Great job Jarek and thank you so much.
> > >
> > > (One minor thing that may just be problem on my side: some tests
> complain
> > > about missing package so I did a `pip install -e ".[all]"` and it went
> > > smoothily afterwards)
> > >
> > >
> > > Cheers,
> > > Kevin Y
> > >
> > > On Wed, Aug 28, 2019 at 12:04 PM Felix Uellendall
> <feluelle@pm.me.invalid
> > >
> > > wrote:
> > >
> > > > Great job Jarek! Thanks. I am looking forward to try it out as soon
> as
> > I
> > > > am back from holiday :)
> > > >
> > > > Felix
> > > >
> > > > Sent from ProtonMail mobile
> > > >
> > > > -------- Original Message --------
> > > > On Aug 28, 2019, 10:34, Jarek Potiuk wrote:
> > > >
> > > > > Just FYI. We had some "teething" problems (of course) at the
> workshop
> > > but
> > > > > fixes to it are already merged to master (thanks Kaxil for the
> > review)!
> > > > >
> > > > > J.
> > > > >
> > > > > On Wed, Aug 28, [2019](tel:2019) at 2:17 AM Chao-Han Tsai <
> > > > milton0825@gmail.com> wrote:
> > > > >
> > > > >> Thanks Jarek for the hard work. It really improved the airflow
> > > > development
> > > > >> experience a lot.
> > > > >> I remember the struggling I experienced to reproduce CI bug
> locally
> > > and
> > > > I
> > > > >> am
> > > > >> glad that those days are now gone. This really lowered the barrier
> > to
> > > > >> contribute to Airflow and
> > > > >> I look forward to more people joining the airflow community!
> > > > >>
> > > > >> Great work.
> > > > >>
> > > > >> On Tue, Aug 27, [2019](tel:2019) at 3:08 PM Sergio Kef <
> > > > sergiokef@gmail.com> wrote:
> > > > >>
> > > > >> > Amazing! congrats and thank you.
> > > > >> >
> > > > >> >
> > > > >> > On Tue, 27 Aug [2019](tel:2019) at 22:25, Jarek Potiuk <
> > > > Jarek.Potiuk@polidea.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hello everyone,
> > > > >> > >
> > > > >> > > After a long time in the making I merged "Breeze" environment
> > > which
> > > > is
> > > > >> a
> > > > >> > > swiss-knife of everything testing in Airflow. It should make
> it
> > > even
> > > > >> > easier
> > > > >> > > to contribute to Airflow (easier to test your changes locally
> > > while
> > > > >> > > development in general).
> > > > >> > >
> > > > >> > > There is a fairly comprehensive documentation in
> > > > >> > > https://github.com/apache/airflow/blob/master/BREEZE.rst that
> > you
> > > > can
> > > > >> > > follow to get started and start using it.
> > > > >> > >
> > > > >> > > Accidentally (or not :)) we have a workshop today in NYC (
> > > > >> > >
> > > https://www.meetup.com/NYC-Apache-Airflow-Meetup/events/264141360/)
> > > > >> > where
> > > > >> > > I
> > > > >> > > will show breeze and have a number of new users using it -
> this
> > > > might
> > > > >> > > result in some incremental fixes/improvements.
> > > > >> > >
> > > > >> > > I expect some small teething problems over the next few days
> > and I
> > > > will
> > > > >> > be
> > > > >> > > watching all channels to help to solve them. Feel free to ping
> > me
> > > in
> > > > >> case
> > > > >> > > you try to use Breeze and have problems - slack is probably
> best
> > > in
> > > > >> such
> > > > >> > > case.
> > > > >> > >
> > > > >> > > J
> > > > >> > >
> > > > >> > > --
> > > > >> > >
> > > > >> > > Jarek Potiuk
> > > > >> > > Polidea <https://www.polidea.com/> | Principal Software
> > Engineer
> > > > >> > >
> > > > >> > > M: [+48 660 796 129](tel:+48660796129)
> > > > <[+48660796129](tel:+48660796129)>
> > > > >> > > [image: Polidea] <https://www.polidea.com/>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> Chao-Han Tsai
> > > > >>
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > > M: [+48 660 796 129](tel:+48660796129)
> > > <[+48660796129](tel:+48660796129)>
> > > > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: "Breeze" dev environment
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hey Elena, All,
Since we have solved some teething problems with Breeze I agreed with Elena
that we clearly separate out the two environments we have for Airflow -
Local Virtualenv and Breeze. I prepared a doc-only PR
https://github.com/apache/airflow/pull/6001 where advantages/disadvantages
of using both are explained and instruction how to use either of them are
separate out to separate .rst files.
Elena will further work on it to make it better structured and easier to
understand. If you have some ideas how to improve the documentation - feel
free to comment here.
J.
On Mon, Sep 2, 2019 at 6:06 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Hey Kevin,
>
> Thanks for those - really helpful explanations! Sorry for spamming
> everyone but I post answers here as maybe some people might find it useful
> for their own usage of Breeze. I am also adding Elena - who is the
> technical writer who started to work on onboarding documentation so that
> she can see the kind of issues and questions and problems developers might
> have with Airflow environment.
>
> On Fri, Aug 30, 2019 at 5:34 AM Kevin Yang <yr...@gmail.com> wrote:
>
>> Oh sry somehow I missed a detail there, it was the virtualenv use case
>> where I need to do the pip install. The docker approach went smoothily. I
>> recall two test cases where I need to install packages, one is from the
>> gcp
>> operator tests and the other is a celery executor test( sry I can't
>> remember the exact test names).
>>
>>
> Regarding the virtualenv - the `*--initialize-virtualenv*` flag on
> purpose installs only `*devel*` extras (not `*devel_all*` nor `*devel_ci*`
> nor even `*devel_minreq*`) - you are not able to run fully-fledged
> airflow with the virtualenv only. This way you have much bigger chance that
> it will actually work out-of-the-box on your host machine and allow you to
> run most of the '*unit*' tests with sqlite database only. This virtualenv
> is not really intended to run more complex system tests that require GCP or
> celery executor tests. There are a number of prerequisites that are outside
> of pip that your host would have to fulfill (for example installed mysql
> client/server, hadoop clients, kubernetes etc.). When you have both Linux
> and MacOS environments for those it's not a trivial thing to setup and
> support. That's why we have the integration test environment in the first
> place - to make it common across all the OS/environments and configuration
> each developer has on their hosts. It Dockerises all such dependencies and
> adds a number of scripts that makes it rather independent from host
> configuration. The idea is that this Breeze environment and not the Local
> Virtualenv is used to run most of the non-trivial (integration/system
> tests) and to run Airflow itself.
>
> You can still add other extras via `pip install .[EXTRA]` and you can
> install other extras there (mysql/hadoop/gcp etc.). No problem with it :).
> Some "integration" (GCP and others) tests will even work in this
> environment as they just need the pip packages and nothing else to run so
> you can use it this way. Maybe we can make it explicit in the new
> documentation. We are also starting some discussions on AIP-4
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems> that
> might result in clear separation of Unit tests/Integration tests/System
> tests so that we know which tests run in which environment.
>
>
>> (One more little thing: `./breeze --build-docs` seems to fail, I've being
>> using `./scripts/ci/ci_docs.sh` instead. And wanna use this opportunity to
>> complain that debugging doc building error is really a headache :) )
>>
>
> The doc building via breeze should be fixed now (both master and
> v1-10-tests). Also in the new approach (as of few weeks or days - can't
> remember :) ) debugging doc errors should be much easier than it used to.
> When you build docs (either via breeze or via script directly) we do not
> remove now the temporary auto-generated API documentation. The errors
> communicated by sphinx should then refer to the actual files available in
> '_api' directory after you run the script. Those auto-generated files are
> generated in files that are mounted from your source's "docs" directory and
> you should see the generated files there.
>
> I think that should help a lot when dealing with documentation errors :).
> It helped me a lot at least.
>
>
>>
>> On Thu, Aug 29, 2019 at 4:29 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> > Fantastic. Thanks. Do you recall which tests/packages they were? I'd
>> love
>> > to see whats's the root cause. We are using the same image in CI and the
>> > tests pass, so I wonder what could be the reason for that.
>> >
>> > On Thu, Aug 29, 2019 at 4:13 AM Kevin Yang <yr...@gmail.com> wrote:
>> >
>> > > Was looking forward to this since last meet up in bay area. Tried both
>> > the
>> > > docker solution and the virtualenv IDE( PyCharm) solution today and it
>> > was
>> > > great. Great job Jarek and thank you so much.
>> > >
>> > > (One minor thing that may just be problem on my side: some tests
>> complain
>> > > about missing package so I did a `pip install -e ".[all]"` and it went
>> > > smoothily afterwards)
>> > >
>> > >
>> > > Cheers,
>> > > Kevin Y
>> > >
>> > > On Wed, Aug 28, 2019 at 12:04 PM Felix Uellendall
>> <feluelle@pm.me.invalid
>> > >
>> > > wrote:
>> > >
>> > > > Great job Jarek! Thanks. I am looking forward to try it out as soon
>> as
>> > I
>> > > > am back from holiday :)
>> > > >
>> > > > Felix
>> > > >
>> > > > Sent from ProtonMail mobile
>> > > >
>> > > > -------- Original Message --------
>> > > > On Aug 28, 2019, 10:34, Jarek Potiuk wrote:
>> > > >
>> > > > > Just FYI. We had some "teething" problems (of course) at the
>> workshop
>> > > but
>> > > > > fixes to it are already merged to master (thanks Kaxil for the
>> > review)!
>> > > > >
>> > > > > J.
>> > > > >
>> > > > > On Wed, Aug 28, [2019](tel:2019) at 2:17 AM Chao-Han Tsai <
>> > > > milton0825@gmail.com> wrote:
>> > > > >
>> > > > >> Thanks Jarek for the hard work. It really improved the airflow
>> > > > development
>> > > > >> experience a lot.
>> > > > >> I remember the struggling I experienced to reproduce CI bug
>> locally
>> > > and
>> > > > I
>> > > > >> am
>> > > > >> glad that those days are now gone. This really lowered the
>> barrier
>> > to
>> > > > >> contribute to Airflow and
>> > > > >> I look forward to more people joining the airflow community!
>> > > > >>
>> > > > >> Great work.
>> > > > >>
>> > > > >> On Tue, Aug 27, [2019](tel:2019) at 3:08 PM Sergio Kef <
>> > > > sergiokef@gmail.com> wrote:
>> > > > >>
>> > > > >> > Amazing! congrats and thank you.
>> > > > >> >
>> > > > >> >
>> > > > >> > On Tue, 27 Aug [2019](tel:2019) at 22:25, Jarek Potiuk <
>> > > > Jarek.Potiuk@polidea.com>
>> > > > >> > wrote:
>> > > > >> >
>> > > > >> > > Hello everyone,
>> > > > >> > >
>> > > > >> > > After a long time in the making I merged "Breeze" environment
>> > > which
>> > > > is
>> > > > >> a
>> > > > >> > > swiss-knife of everything testing in Airflow. It should make
>> it
>> > > even
>> > > > >> > easier
>> > > > >> > > to contribute to Airflow (easier to test your changes locally
>> > > while
>> > > > >> > > development in general).
>> > > > >> > >
>> > > > >> > > There is a fairly comprehensive documentation in
>> > > > >> > > https://github.com/apache/airflow/blob/master/BREEZE.rst
>> that
>> > you
>> > > > can
>> > > > >> > > follow to get started and start using it.
>> > > > >> > >
>> > > > >> > > Accidentally (or not :)) we have a workshop today in NYC (
>> > > > >> > >
>> > > https://www.meetup.com/NYC-Apache-Airflow-Meetup/events/264141360/)
>> > > > >> > where
>> > > > >> > > I
>> > > > >> > > will show breeze and have a number of new users using it -
>> this
>> > > > might
>> > > > >> > > result in some incremental fixes/improvements.
>> > > > >> > >
>> > > > >> > > I expect some small teething problems over the next few days
>> > and I
>> > > > will
>> > > > >> > be
>> > > > >> > > watching all channels to help to solve them. Feel free to
>> ping
>> > me
>> > > in
>> > > > >> case
>> > > > >> > > you try to use Breeze and have problems - slack is probably
>> best
>> > > in
>> > > > >> such
>> > > > >> > > case.
>> > > > >> > >
>> > > > >> > > J
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > >
>> > > > >> > > Jarek Potiuk
>> > > > >> > > Polidea <https://www.polidea.com/> | Principal Software
>> > Engineer
>> > > > >> > >
>> > > > >> > > M: [+48 660 796 129](tel:+48660796129)
>> > > > <[+48660796129](tel:+48660796129)>
>> > > > >> > > [image: Polidea] <https://www.polidea.com/>
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >>
>> > > > >> Chao-Han Tsai
>> > > > >>
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Jarek Potiuk
>> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > > > >
>> > > > > M: [+48 660 796 129](tel:+48660796129)
>> > > <[+48660796129](tel:+48660796129)>
>> > > > > [image: Polidea] <https://www.polidea.com/>
>> > >
>> >
>> >
>> > --
>> >
>> > Jarek Potiuk
>> > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >
>> > M: +48 660 796 129 <+48660796129>
>> > [image: Polidea] <https://www.polidea.com/>
>> >
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>