You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2019/10/18 13:30:25 UTC
Re: [DISCUSS] Smarter test execution for CI (trivial changes without
full tests)
Seems that our tests got way smarter now :).
I just implemented the "smartness" it with
https://github.com/apache/airflow/pull/6321 and trying to workaround
Kubernetes problem we have :).
The doc changes are short enough that there is no need to optimise those.
Doc-test only should execute much much faster now :).
For details see https://issues.apache.org/jira/browse/AIRFLOW-5649
J.
On Fri, Aug 23, 2019 at 4:09 PM James Meickle
<jm...@quantopian.com.invalid> wrote:
> GitHub recently introduced the idea of "Draft" PRs:
> https://github.blog/2019-02-14-introducing-draft-pull-requests/
>
> Could we do something similar either with that system or something else?
> Run a minimal set until it's marked as "ready for testing", and then run a
> larger suite.
>
> On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Maybe your 4th point covers this but there are frequent Doc only changes.
> > In this case we should not run "real-test" but only static checks - mypy,
> > pylint. flake8, doc generation.
> >
> > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > On top of moving out from Travis (I will resume working on it next
> week)
> > I
> > > thought about some ways to improve the feedback cycle time we have with
> > CI
> > > (super long now).
> > >
> > > Maybe we should consider being smarter with running tests only when
> > really
> > > needed.
> > >
> > > After introducing pre-commit framework, I thought that it is rather
> smart
> > > in selecting which tests to run locally based on what files changed.
> It's
> > > not perfect of course and there are edge cases where we change .xml
> files
> > > and python tests stop running (for example), but maybe we can introduce
> > > some smartness in our test execution scripts based on similar
> principles.
> > >
> > > First of all we should always run all tests after master is merged
> > (catches
> > > edge cases missed by partial running in PR). Then in PRs we could run
> > > partial tests:
> > >
> > > 1) Do not run Python tests if none of .py files change
> > >
> > > 2) Do not run Doc generation if neither .py nor .rst files change
> > >
> > > 3) [This might be controversial / not catching a lot of problems] -
> only
> > > run related tests - for corresponding packages - when .py files change.
> > >
> > > 4) Only run "real unit tests" if none of the operators/hooks/sensors
> > change
> > > (we would have to introduce a way to distinguish unit tests - requiring
> > > only basic Airflow + database - from integration tests where
> > > hooks/sensors/operators change). This could be done using the
> > > slimmer/smaller/future production CI image - without the overhead of
> > > starting up all the services.
> > >
> > > 5) Run only static checks + real/related tests in PR for
> > > "drafts/in-progress PRs" but only trigger full tests when someone marks
> > it
> > > as "Ready to merge" (via label or comment in PR).
> > >
> > > Of course if we can make it faster with no-Travis setup running full
> > tests
> > > in all PRs makes more sense, but maybe some of those options above are
> > also
> > > acceptable.
> > >
> > > Comments?
> > >
> > > J.
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Smarter test execution for CI (trivial changes without
full tests)
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Awesome Jarek, that was my fear as well. Good to hear that we still run the
full test suite.\
Cheers, Fokko
Op vr 18 okt. 2019 om 15:47 schreef Daniel Imberman <
daniel.imberman@gmail.com>:
> Oh then that sounds fine in that case :)
>
> via Newton Mail [
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2
> ]
> On Fri, Oct 18, 2019 at 10:46 AM, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> It's even more than daily. In the latest version every time after we merge
> to master full suite of tests will be run - which means that we can
> immediately see which commit caused the problem. It's only at the PR level
> the tests will be "smart".
>
> On Fri, Oct 18, 2019 at 3:43 PM Daniel Imberman <daniel.imberman@gmail.com
> >
> wrote:
>
> > I’m still somewhat hesitant on this as it could allow regressions to peak
> > through, though as long as we’re still doing the daily build, and with
> our
> > soon-to-be-created prerelease load testing I think we should be okay.
> >
> > On Fri, Oct 18, 2019 at 10:30 AM, Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > wrote:
> > Seems that our tests got way smarter now :).
> >
> > I just implemented the "smartness" it with
> > https://github.com/apache/airflow/pull/6321 and trying to workaround
> > Kubernetes problem we have :).
> > The doc changes are short enough that there is no need to optimise those.
> > Doc-test only should execute much much faster now :).
> > For details see https://issues.apache.org/jira/browse/AIRFLOW-5649
> >
> > J.
> >
> > On Fri, Aug 23, 2019 at 4:09 PM James Meickle
> > <jm...@quantopian.com.invalid> wrote:
> >
> > > GitHub recently introduced the idea of "Draft" PRs:
> > > https://github.blog/2019-02-14-introducing-draft-pull-requests/
> > >
> > > Could we do something similar either with that system or something
> else?
> > > Run a minimal set until it's marked as "ready for testing", and then
> run
> > a
> > > larger suite.
> > >
> > > On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >
> > > > Maybe your 4th point covers this but there are frequent Doc only
> > changes.
> > > > In this case we should not run "real-test" but only static checks -
> > mypy,
> > > > pylint. flake8, doc generation.
> > > >
> > > > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > On top of moving out from Travis (I will resume working on it next
> > > week)
> > > > I
> > > > > thought about some ways to improve the feedback cycle time we have
> > with
> > > > CI
> > > > > (super long now).
> > > > >
> > > > > Maybe we should consider being smarter with running tests only when
> > > > really
> > > > > needed.
> > > > >
> > > > > After introducing pre-commit framework, I thought that it is rather
> > > smart
> > > > > in selecting which tests to run locally based on what files
> changed.
> > > It's
> > > > > not perfect of course and there are edge cases where we change .xml
> > > files
> > > > > and python tests stop running (for example), but maybe we can
> > introduce
> > > > > some smartness in our test execution scripts based on similar
> > > principles.
> > > > >
> > > > > First of all we should always run all tests after master is merged
> > > > (catches
> > > > > edge cases missed by partial running in PR). Then in PRs we could
> run
> > > > > partial tests:
> > > > >
> > > > > 1) Do not run Python tests if none of .py files change
> > > > >
> > > > > 2) Do not run Doc generation if neither .py nor .rst files change
> > > > >
> > > > > 3) [This might be controversial / not catching a lot of problems] -
> > > only
> > > > > run related tests - for corresponding packages - when .py files
> > change.
> > > > >
> > > > > 4) Only run "real unit tests" if none of the
> operators/hooks/sensors
> > > > change
> > > > > (we would have to introduce a way to distinguish unit tests -
> > requiring
> > > > > only basic Airflow + database - from integration tests where
> > > > > hooks/sensors/operators change). This could be done using the
> > > > > slimmer/smaller/future production CI image - without the overhead
> of
> > > > > starting up all the services.
> > > > >
> > > > > 5) Run only static checks + real/related tests in PR for
> > > > > "drafts/in-progress PRs" but only trigger full tests when someone
> > marks
> > > > it
> > > > > as "Ready to merge" (via label or comment in PR).
> > > > >
> > > > > Of course if we can make it faster with no-Travis setup running
> full
> > > > tests
> > > > > in all PRs makes more sense, but maybe some of those options above
> > are
> > > > also
> > > > > acceptable.
> > > > >
> > > > > Comments?
> > > > >
> > > > > J.
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129 <+48660796129>
> > > > > [image: Polidea] <https://www.polidea.com/>
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Smarter test execution for CI (trivial changes
without full tests)
Posted by Daniel Imberman <da...@gmail.com>.
Oh then that sounds fine in that case :)
via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2]
On Fri, Oct 18, 2019 at 10:46 AM, Jarek Potiuk <Ja...@polidea.com> wrote:
It's even more than daily. In the latest version every time after we merge
to master full suite of tests will be run - which means that we can
immediately see which commit caused the problem. It's only at the PR level
the tests will be "smart".
On Fri, Oct 18, 2019 at 3:43 PM Daniel Imberman <da...@gmail.com>
wrote:
> I’m still somewhat hesitant on this as it could allow regressions to peak
> through, though as long as we’re still doing the daily build, and with our
> soon-to-be-created prerelease load testing I think we should be okay.
>
> On Fri, Oct 18, 2019 at 10:30 AM, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> Seems that our tests got way smarter now :).
>
> I just implemented the "smartness" it with
> https://github.com/apache/airflow/pull/6321 and trying to workaround
> Kubernetes problem we have :).
> The doc changes are short enough that there is no need to optimise those.
> Doc-test only should execute much much faster now :).
> For details see https://issues.apache.org/jira/browse/AIRFLOW-5649
>
> J.
>
> On Fri, Aug 23, 2019 at 4:09 PM James Meickle
> <jm...@quantopian.com.invalid> wrote:
>
> > GitHub recently introduced the idea of "Draft" PRs:
> > https://github.blog/2019-02-14-introducing-draft-pull-requests/
> >
> > Could we do something similar either with that system or something else?
> > Run a minimal set until it's marked as "ready for testing", and then run
> a
> > larger suite.
> >
> > On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Maybe your 4th point covers this but there are frequent Doc only
> changes.
> > > In this case we should not run "real-test" but only static checks -
> mypy,
> > > pylint. flake8, doc generation.
> > >
> > > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > On top of moving out from Travis (I will resume working on it next
> > week)
> > > I
> > > > thought about some ways to improve the feedback cycle time we have
> with
> > > CI
> > > > (super long now).
> > > >
> > > > Maybe we should consider being smarter with running tests only when
> > > really
> > > > needed.
> > > >
> > > > After introducing pre-commit framework, I thought that it is rather
> > smart
> > > > in selecting which tests to run locally based on what files changed.
> > It's
> > > > not perfect of course and there are edge cases where we change .xml
> > files
> > > > and python tests stop running (for example), but maybe we can
> introduce
> > > > some smartness in our test execution scripts based on similar
> > principles.
> > > >
> > > > First of all we should always run all tests after master is merged
> > > (catches
> > > > edge cases missed by partial running in PR). Then in PRs we could run
> > > > partial tests:
> > > >
> > > > 1) Do not run Python tests if none of .py files change
> > > >
> > > > 2) Do not run Doc generation if neither .py nor .rst files change
> > > >
> > > > 3) [This might be controversial / not catching a lot of problems] -
> > only
> > > > run related tests - for corresponding packages - when .py files
> change.
> > > >
> > > > 4) Only run "real unit tests" if none of the operators/hooks/sensors
> > > change
> > > > (we would have to introduce a way to distinguish unit tests -
> requiring
> > > > only basic Airflow + database - from integration tests where
> > > > hooks/sensors/operators change). This could be done using the
> > > > slimmer/smaller/future production CI image - without the overhead of
> > > > starting up all the services.
> > > >
> > > > 5) Run only static checks + real/related tests in PR for
> > > > "drafts/in-progress PRs" but only trigger full tests when someone
> marks
> > > it
> > > > as "Ready to merge" (via label or comment in PR).
> > > >
> > > > Of course if we can make it faster with no-Travis setup running full
> > > tests
> > > > in all PRs makes more sense, but maybe some of those options above
> are
> > > also
> > > > acceptable.
> > > >
> > > > Comments?
> > > >
> > > > J.
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Smarter test execution for CI (trivial changes without
full tests)
Posted by Jarek Potiuk <Ja...@polidea.com>.
It's even more than daily. In the latest version every time after we merge
to master full suite of tests will be run - which means that we can
immediately see which commit caused the problem. It's only at the PR level
the tests will be "smart".
On Fri, Oct 18, 2019 at 3:43 PM Daniel Imberman <da...@gmail.com>
wrote:
> I’m still somewhat hesitant on this as it could allow regressions to peak
> through, though as long as we’re still doing the daily build, and with our
> soon-to-be-created prerelease load testing I think we should be okay.
>
> On Fri, Oct 18, 2019 at 10:30 AM, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> Seems that our tests got way smarter now :).
>
> I just implemented the "smartness" it with
> https://github.com/apache/airflow/pull/6321 and trying to workaround
> Kubernetes problem we have :).
> The doc changes are short enough that there is no need to optimise those.
> Doc-test only should execute much much faster now :).
> For details see https://issues.apache.org/jira/browse/AIRFLOW-5649
>
> J.
>
> On Fri, Aug 23, 2019 at 4:09 PM James Meickle
> <jm...@quantopian.com.invalid> wrote:
>
> > GitHub recently introduced the idea of "Draft" PRs:
> > https://github.blog/2019-02-14-introducing-draft-pull-requests/
> >
> > Could we do something similar either with that system or something else?
> > Run a minimal set until it's marked as "ready for testing", and then run
> a
> > larger suite.
> >
> > On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Maybe your 4th point covers this but there are frequent Doc only
> changes.
> > > In this case we should not run "real-test" but only static checks -
> mypy,
> > > pylint. flake8, doc generation.
> > >
> > > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > On top of moving out from Travis (I will resume working on it next
> > week)
> > > I
> > > > thought about some ways to improve the feedback cycle time we have
> with
> > > CI
> > > > (super long now).
> > > >
> > > > Maybe we should consider being smarter with running tests only when
> > > really
> > > > needed.
> > > >
> > > > After introducing pre-commit framework, I thought that it is rather
> > smart
> > > > in selecting which tests to run locally based on what files changed.
> > It's
> > > > not perfect of course and there are edge cases where we change .xml
> > files
> > > > and python tests stop running (for example), but maybe we can
> introduce
> > > > some smartness in our test execution scripts based on similar
> > principles.
> > > >
> > > > First of all we should always run all tests after master is merged
> > > (catches
> > > > edge cases missed by partial running in PR). Then in PRs we could run
> > > > partial tests:
> > > >
> > > > 1) Do not run Python tests if none of .py files change
> > > >
> > > > 2) Do not run Doc generation if neither .py nor .rst files change
> > > >
> > > > 3) [This might be controversial / not catching a lot of problems] -
> > only
> > > > run related tests - for corresponding packages - when .py files
> change.
> > > >
> > > > 4) Only run "real unit tests" if none of the operators/hooks/sensors
> > > change
> > > > (we would have to introduce a way to distinguish unit tests -
> requiring
> > > > only basic Airflow + database - from integration tests where
> > > > hooks/sensors/operators change). This could be done using the
> > > > slimmer/smaller/future production CI image - without the overhead of
> > > > starting up all the services.
> > > >
> > > > 5) Run only static checks + real/related tests in PR for
> > > > "drafts/in-progress PRs" but only trigger full tests when someone
> marks
> > > it
> > > > as "Ready to merge" (via label or comment in PR).
> > > >
> > > > Of course if we can make it faster with no-Travis setup running full
> > > tests
> > > > in all PRs makes more sense, but maybe some of those options above
> are
> > > also
> > > > acceptable.
> > > >
> > > > Comments?
> > > >
> > > > J.
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Smarter test execution for CI (trivial changes
without full tests)
Posted by Daniel Imberman <da...@gmail.com>.
I’m still somewhat hesitant on this as it could allow regressions to peak through, though as long as we’re still doing the daily build, and with our soon-to-be-created prerelease load testing I think we should be okay.
On Fri, Oct 18, 2019 at 10:30 AM, Jarek Potiuk <Ja...@polidea.com> wrote:
Seems that our tests got way smarter now :).
I just implemented the "smartness" it with
https://github.com/apache/airflow/pull/6321 and trying to workaround
Kubernetes problem we have :).
The doc changes are short enough that there is no need to optimise those.
Doc-test only should execute much much faster now :).
For details see https://issues.apache.org/jira/browse/AIRFLOW-5649
J.
On Fri, Aug 23, 2019 at 4:09 PM James Meickle
<jm...@quantopian.com.invalid> wrote:
> GitHub recently introduced the idea of "Draft" PRs:
> https://github.blog/2019-02-14-introducing-draft-pull-requests/
>
> Could we do something similar either with that system or something else?
> Run a minimal set until it's marked as "ready for testing", and then run a
> larger suite.
>
> On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Maybe your 4th point covers this but there are frequent Doc only changes.
> > In this case we should not run "real-test" but only static checks - mypy,
> > pylint. flake8, doc generation.
> >
> > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > On top of moving out from Travis (I will resume working on it next
> week)
> > I
> > > thought about some ways to improve the feedback cycle time we have with
> > CI
> > > (super long now).
> > >
> > > Maybe we should consider being smarter with running tests only when
> > really
> > > needed.
> > >
> > > After introducing pre-commit framework, I thought that it is rather
> smart
> > > in selecting which tests to run locally based on what files changed.
> It's
> > > not perfect of course and there are edge cases where we change .xml
> files
> > > and python tests stop running (for example), but maybe we can introduce
> > > some smartness in our test execution scripts based on similar
> principles.
> > >
> > > First of all we should always run all tests after master is merged
> > (catches
> > > edge cases missed by partial running in PR). Then in PRs we could run
> > > partial tests:
> > >
> > > 1) Do not run Python tests if none of .py files change
> > >
> > > 2) Do not run Doc generation if neither .py nor .rst files change
> > >
> > > 3) [This might be controversial / not catching a lot of problems] -
> only
> > > run related tests - for corresponding packages - when .py files change.
> > >
> > > 4) Only run "real unit tests" if none of the operators/hooks/sensors
> > change
> > > (we would have to introduce a way to distinguish unit tests - requiring
> > > only basic Airflow + database - from integration tests where
> > > hooks/sensors/operators change). This could be done using the
> > > slimmer/smaller/future production CI image - without the overhead of
> > > starting up all the services.
> > >
> > > 5) Run only static checks + real/related tests in PR for
> > > "drafts/in-progress PRs" but only trigger full tests when someone marks
> > it
> > > as "Ready to merge" (via label or comment in PR).
> > >
> > > Of course if we can make it faster with no-Travis setup running full
> > tests
> > > in all PRs makes more sense, but maybe some of those options above are
> > also
> > > acceptable.
> > >
> > > Comments?
> > >
> > > J.
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>