You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2019/08/23 13:39:39 UTC
[DISCUSS] Smarter test execution for CI (trivial changes without full tests)
Hello everyone,
On top of moving out from Travis (I will resume working on it next week) I
thought about some ways to improve the feedback cycle time we have with CI
(super long now).
Maybe we should consider being smarter with running tests only when really
needed.
After introducing pre-commit framework, I thought that it is rather smart
in selecting which tests to run locally based on what files changed. It's
not perfect of course and there are edge cases where we change .xml files
and python tests stop running (for example), but maybe we can introduce
some smartness in our test execution scripts based on similar principles.
First of all we should always run all tests after master is merged (catches
edge cases missed by partial running in PR). Then in PRs we could run
partial tests:
1) Do not run Python tests if none of .py files change
2) Do not run Doc generation if neither .py nor .rst files change
3) [This might be controversial / not catching a lot of problems] - only
run related tests - for corresponding packages - when .py files change.
4) Only run "real unit tests" if none of the operators/hooks/sensors change
(we would have to introduce a way to distinguish unit tests - requiring
only basic Airflow + database - from integration tests where
hooks/sensors/operators change). This could be done using the
slimmer/smaller/future production CI image - without the overhead of
starting up all the services.
5) Run only static checks + real/related tests in PR for
"drafts/in-progress PRs" but only trigger full tests when someone marks it
as "Ready to merge" (via label or comment in PR).
Of course if we can make it faster with no-Travis setup running full tests
in all PRs makes more sense, but maybe some of those options above are also
acceptable.
Comments?
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Smarter test execution for CI (trivial changes without
full tests)
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Awesome Jarek, that was my fear as well. Good to hear that we still run the
full test suite.\
Cheers, Fokko
Op vr 18 okt. 2019 om 15:47 schreef Daniel Imberman <
daniel.imberman@gmail.com>:
> Oh then that sounds fine in that case :)
>
> via Newton Mail [
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2
> ]
> On Fri, Oct 18, 2019 at 10:46 AM, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> It's even more than daily. In the latest version every time after we merge
> to master full suite of tests will be run - which means that we can
> immediately see which commit caused the problem. It's only at the PR level
> the tests will be "smart".
>
> On Fri, Oct 18, 2019 at 3:43 PM Daniel Imberman <daniel.imberman@gmail.com
> >
> wrote:
>
> > I’m still somewhat hesitant on this as it could allow regressions to peak
> > through, though as long as we’re still doing the daily build, and with
> our
> > soon-to-be-created prerelease load testing I think we should be okay.
> >
> > On Fri, Oct 18, 2019 at 10:30 AM, Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > wrote:
> > Seems that our tests got way smarter now :).
> >
> > I just implemented the "smartness" it with
> > https://github.com/apache/airflow/pull/6321 and trying to workaround
> > Kubernetes problem we have :).
> > The doc changes are short enough that there is no need to optimise those.
> > Doc-test only should execute much much faster now :).
> > For details see https://issues.apache.org/jira/browse/AIRFLOW-5649
> >
> > J.
> >
> > On Fri, Aug 23, 2019 at 4:09 PM James Meickle
> > <jm...@quantopian.com.invalid> wrote:
> >
> > > GitHub recently introduced the idea of "Draft" PRs:
> > > https://github.blog/2019-02-14-introducing-draft-pull-requests/
> > >
> > > Could we do something similar either with that system or something
> else?
> > > Run a minimal set until it's marked as "ready for testing", and then
> run
> > a
> > > larger suite.
> > >
> > > On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >
> > > > Maybe your 4th point covers this but there are frequent Doc only
> > changes.
> > > > In this case we should not run "real-test" but only static checks -
> > mypy,
> > > > pylint. flake8, doc generation.
> > > >
> > > > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > On top of moving out from Travis (I will resume working on it next
> > > week)
> > > > I
> > > > > thought about some ways to improve the feedback cycle time we have
> > with
> > > > CI
> > > > > (super long now).
> > > > >
> > > > > Maybe we should consider being smarter with running tests only when
> > > > really
> > > > > needed.
> > > > >
> > > > > After introducing pre-commit framework, I thought that it is rather
> > > smart
> > > > > in selecting which tests to run locally based on what files
> changed.
> > > It's
> > > > > not perfect of course and there are edge cases where we change .xml
> > > files
> > > > > and python tests stop running (for example), but maybe we can
> > introduce
> > > > > some smartness in our test execution scripts based on similar
> > > principles.
> > > > >
> > > > > First of all we should always run all tests after master is merged
> > > > (catches
> > > > > edge cases missed by partial running in PR). Then in PRs we could
> run
> > > > > partial tests:
> > > > >
> > > > > 1) Do not run Python tests if none of .py files change
> > > > >
> > > > > 2) Do not run Doc generation if neither .py nor .rst files change
> > > > >
> > > > > 3) [This might be controversial / not catching a lot of problems] -
> > > only
> > > > > run related tests - for corresponding packages - when .py files
> > change.
> > > > >
> > > > > 4) Only run "real unit tests" if none of the
> operators/hooks/sensors
> > > > change
> > > > > (we would have to introduce a way to distinguish unit tests -
> > requiring
> > > > > only basic Airflow + database - from integration tests where
> > > > > hooks/sensors/operators change). This could be done using the
> > > > > slimmer/smaller/future production CI image - without the overhead
> of
> > > > > starting up all the services.
> > > > >
> > > > > 5) Run only static checks + real/related tests in PR for
> > > > > "drafts/in-progress PRs" but only trigger full tests when someone
> > marks
> > > > it
> > > > > as "Ready to merge" (via label or comment in PR).
> > > > >
> > > > > Of course if we can make it faster with no-Travis setup running
> full
> > > > tests
> > > > > in all PRs makes more sense, but maybe some of those options above
> > are
> > > > also
> > > > > acceptable.
> > > > >
> > > > > Comments?
> > > > >
> > > > > J.
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129 <+48660796129>
> > > > > [image: Polidea] <https://www.polidea.com/>
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Smarter test execution for CI (trivial changes
without full tests)
Posted by Daniel Imberman <da...@gmail.com>.
Oh then that sounds fine in that case :)
via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2]
On Fri, Oct 18, 2019 at 10:46 AM, Jarek Potiuk <Ja...@polidea.com> wrote:
It's even more than daily. In the latest version every time after we merge
to master full suite of tests will be run - which means that we can
immediately see which commit caused the problem. It's only at the PR level
the tests will be "smart".
On Fri, Oct 18, 2019 at 3:43 PM Daniel Imberman <da...@gmail.com>
wrote:
> I’m still somewhat hesitant on this as it could allow regressions to peak
> through, though as long as we’re still doing the daily build, and with our
> soon-to-be-created prerelease load testing I think we should be okay.
>
> On Fri, Oct 18, 2019 at 10:30 AM, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> Seems that our tests got way smarter now :).
>
> I just implemented the "smartness" it with
> https://github.com/apache/airflow/pull/6321 and trying to workaround
> Kubernetes problem we have :).
> The doc changes are short enough that there is no need to optimise those.
> Doc-test only should execute much much faster now :).
> For details see https://issues.apache.org/jira/browse/AIRFLOW-5649
>
> J.
>
> On Fri, Aug 23, 2019 at 4:09 PM James Meickle
> <jm...@quantopian.com.invalid> wrote:
>
> > GitHub recently introduced the idea of "Draft" PRs:
> > https://github.blog/2019-02-14-introducing-draft-pull-requests/
> >
> > Could we do something similar either with that system or something else?
> > Run a minimal set until it's marked as "ready for testing", and then run
> a
> > larger suite.
> >
> > On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Maybe your 4th point covers this but there are frequent Doc only
> changes.
> > > In this case we should not run "real-test" but only static checks -
> mypy,
> > > pylint. flake8, doc generation.
> > >
> > > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > On top of moving out from Travis (I will resume working on it next
> > week)
> > > I
> > > > thought about some ways to improve the feedback cycle time we have
> with
> > > CI
> > > > (super long now).
> > > >
> > > > Maybe we should consider being smarter with running tests only when
> > > really
> > > > needed.
> > > >
> > > > After introducing pre-commit framework, I thought that it is rather
> > smart
> > > > in selecting which tests to run locally based on what files changed.
> > It's
> > > > not perfect of course and there are edge cases where we change .xml
> > files
> > > > and python tests stop running (for example), but maybe we can
> introduce
> > > > some smartness in our test execution scripts based on similar
> > principles.
> > > >
> > > > First of all we should always run all tests after master is merged
> > > (catches
> > > > edge cases missed by partial running in PR). Then in PRs we could run
> > > > partial tests:
> > > >
> > > > 1) Do not run Python tests if none of .py files change
> > > >
> > > > 2) Do not run Doc generation if neither .py nor .rst files change
> > > >
> > > > 3) [This might be controversial / not catching a lot of problems] -
> > only
> > > > run related tests - for corresponding packages - when .py files
> change.
> > > >
> > > > 4) Only run "real unit tests" if none of the operators/hooks/sensors
> > > change
> > > > (we would have to introduce a way to distinguish unit tests -
> requiring
> > > > only basic Airflow + database - from integration tests where
> > > > hooks/sensors/operators change). This could be done using the
> > > > slimmer/smaller/future production CI image - without the overhead of
> > > > starting up all the services.
> > > >
> > > > 5) Run only static checks + real/related tests in PR for
> > > > "drafts/in-progress PRs" but only trigger full tests when someone
> marks
> > > it
> > > > as "Ready to merge" (via label or comment in PR).
> > > >
> > > > Of course if we can make it faster with no-Travis setup running full
> > > tests
> > > > in all PRs makes more sense, but maybe some of those options above
> are
> > > also
> > > > acceptable.
> > > >
> > > > Comments?
> > > >
> > > > J.
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Smarter test execution for CI (trivial changes without
full tests)
Posted by Jarek Potiuk <Ja...@polidea.com>.
It's even more than daily. In the latest version every time after we merge
to master full suite of tests will be run - which means that we can
immediately see which commit caused the problem. It's only at the PR level
the tests will be "smart".
On Fri, Oct 18, 2019 at 3:43 PM Daniel Imberman <da...@gmail.com>
wrote:
> I’m still somewhat hesitant on this as it could allow regressions to peak
> through, though as long as we’re still doing the daily build, and with our
> soon-to-be-created prerelease load testing I think we should be okay.
>
> On Fri, Oct 18, 2019 at 10:30 AM, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> Seems that our tests got way smarter now :).
>
> I just implemented the "smartness" it with
> https://github.com/apache/airflow/pull/6321 and trying to workaround
> Kubernetes problem we have :).
> The doc changes are short enough that there is no need to optimise those.
> Doc-test only should execute much much faster now :).
> For details see https://issues.apache.org/jira/browse/AIRFLOW-5649
>
> J.
>
> On Fri, Aug 23, 2019 at 4:09 PM James Meickle
> <jm...@quantopian.com.invalid> wrote:
>
> > GitHub recently introduced the idea of "Draft" PRs:
> > https://github.blog/2019-02-14-introducing-draft-pull-requests/
> >
> > Could we do something similar either with that system or something else?
> > Run a minimal set until it's marked as "ready for testing", and then run
> a
> > larger suite.
> >
> > On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Maybe your 4th point covers this but there are frequent Doc only
> changes.
> > > In this case we should not run "real-test" but only static checks -
> mypy,
> > > pylint. flake8, doc generation.
> > >
> > > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > On top of moving out from Travis (I will resume working on it next
> > week)
> > > I
> > > > thought about some ways to improve the feedback cycle time we have
> with
> > > CI
> > > > (super long now).
> > > >
> > > > Maybe we should consider being smarter with running tests only when
> > > really
> > > > needed.
> > > >
> > > > After introducing pre-commit framework, I thought that it is rather
> > smart
> > > > in selecting which tests to run locally based on what files changed.
> > It's
> > > > not perfect of course and there are edge cases where we change .xml
> > files
> > > > and python tests stop running (for example), but maybe we can
> introduce
> > > > some smartness in our test execution scripts based on similar
> > principles.
> > > >
> > > > First of all we should always run all tests after master is merged
> > > (catches
> > > > edge cases missed by partial running in PR). Then in PRs we could run
> > > > partial tests:
> > > >
> > > > 1) Do not run Python tests if none of .py files change
> > > >
> > > > 2) Do not run Doc generation if neither .py nor .rst files change
> > > >
> > > > 3) [This might be controversial / not catching a lot of problems] -
> > only
> > > > run related tests - for corresponding packages - when .py files
> change.
> > > >
> > > > 4) Only run "real unit tests" if none of the operators/hooks/sensors
> > > change
> > > > (we would have to introduce a way to distinguish unit tests -
> requiring
> > > > only basic Airflow + database - from integration tests where
> > > > hooks/sensors/operators change). This could be done using the
> > > > slimmer/smaller/future production CI image - without the overhead of
> > > > starting up all the services.
> > > >
> > > > 5) Run only static checks + real/related tests in PR for
> > > > "drafts/in-progress PRs" but only trigger full tests when someone
> marks
> > > it
> > > > as "Ready to merge" (via label or comment in PR).
> > > >
> > > > Of course if we can make it faster with no-Travis setup running full
> > > tests
> > > > in all PRs makes more sense, but maybe some of those options above
> are
> > > also
> > > > acceptable.
> > > >
> > > > Comments?
> > > >
> > > > J.
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Smarter test execution for CI (trivial changes
without full tests)
Posted by Daniel Imberman <da...@gmail.com>.
I’m still somewhat hesitant on this as it could allow regressions to peak through, though as long as we’re still doing the daily build, and with our soon-to-be-created prerelease load testing I think we should be okay.
On Fri, Oct 18, 2019 at 10:30 AM, Jarek Potiuk <Ja...@polidea.com> wrote:
Seems that our tests got way smarter now :).
I just implemented the "smartness" it with
https://github.com/apache/airflow/pull/6321 and trying to workaround
Kubernetes problem we have :).
The doc changes are short enough that there is no need to optimise those.
Doc-test only should execute much much faster now :).
For details see https://issues.apache.org/jira/browse/AIRFLOW-5649
J.
On Fri, Aug 23, 2019 at 4:09 PM James Meickle
<jm...@quantopian.com.invalid> wrote:
> GitHub recently introduced the idea of "Draft" PRs:
> https://github.blog/2019-02-14-introducing-draft-pull-requests/
>
> Could we do something similar either with that system or something else?
> Run a minimal set until it's marked as "ready for testing", and then run a
> larger suite.
>
> On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Maybe your 4th point covers this but there are frequent Doc only changes.
> > In this case we should not run "real-test" but only static checks - mypy,
> > pylint. flake8, doc generation.
> >
> > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > On top of moving out from Travis (I will resume working on it next
> week)
> > I
> > > thought about some ways to improve the feedback cycle time we have with
> > CI
> > > (super long now).
> > >
> > > Maybe we should consider being smarter with running tests only when
> > really
> > > needed.
> > >
> > > After introducing pre-commit framework, I thought that it is rather
> smart
> > > in selecting which tests to run locally based on what files changed.
> It's
> > > not perfect of course and there are edge cases where we change .xml
> files
> > > and python tests stop running (for example), but maybe we can introduce
> > > some smartness in our test execution scripts based on similar
> principles.
> > >
> > > First of all we should always run all tests after master is merged
> > (catches
> > > edge cases missed by partial running in PR). Then in PRs we could run
> > > partial tests:
> > >
> > > 1) Do not run Python tests if none of .py files change
> > >
> > > 2) Do not run Doc generation if neither .py nor .rst files change
> > >
> > > 3) [This might be controversial / not catching a lot of problems] -
> only
> > > run related tests - for corresponding packages - when .py files change.
> > >
> > > 4) Only run "real unit tests" if none of the operators/hooks/sensors
> > change
> > > (we would have to introduce a way to distinguish unit tests - requiring
> > > only basic Airflow + database - from integration tests where
> > > hooks/sensors/operators change). This could be done using the
> > > slimmer/smaller/future production CI image - without the overhead of
> > > starting up all the services.
> > >
> > > 5) Run only static checks + real/related tests in PR for
> > > "drafts/in-progress PRs" but only trigger full tests when someone marks
> > it
> > > as "Ready to merge" (via label or comment in PR).
> > >
> > > Of course if we can make it faster with no-Travis setup running full
> > tests
> > > in all PRs makes more sense, but maybe some of those options above are
> > also
> > > acceptable.
> > >
> > > Comments?
> > >
> > > J.
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Smarter test execution for CI (trivial changes without
full tests)
Posted by Jarek Potiuk <Ja...@polidea.com>.
Seems that our tests got way smarter now :).
I just implemented the "smartness" it with
https://github.com/apache/airflow/pull/6321 and trying to workaround
Kubernetes problem we have :).
The doc changes are short enough that there is no need to optimise those.
Doc-test only should execute much much faster now :).
For details see https://issues.apache.org/jira/browse/AIRFLOW-5649
J.
On Fri, Aug 23, 2019 at 4:09 PM James Meickle
<jm...@quantopian.com.invalid> wrote:
> GitHub recently introduced the idea of "Draft" PRs:
> https://github.blog/2019-02-14-introducing-draft-pull-requests/
>
> Could we do something similar either with that system or something else?
> Run a minimal set until it's marked as "ready for testing", and then run a
> larger suite.
>
> On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Maybe your 4th point covers this but there are frequent Doc only changes.
> > In this case we should not run "real-test" but only static checks - mypy,
> > pylint. flake8, doc generation.
> >
> > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > On top of moving out from Travis (I will resume working on it next
> week)
> > I
> > > thought about some ways to improve the feedback cycle time we have with
> > CI
> > > (super long now).
> > >
> > > Maybe we should consider being smarter with running tests only when
> > really
> > > needed.
> > >
> > > After introducing pre-commit framework, I thought that it is rather
> smart
> > > in selecting which tests to run locally based on what files changed.
> It's
> > > not perfect of course and there are edge cases where we change .xml
> files
> > > and python tests stop running (for example), but maybe we can introduce
> > > some smartness in our test execution scripts based on similar
> principles.
> > >
> > > First of all we should always run all tests after master is merged
> > (catches
> > > edge cases missed by partial running in PR). Then in PRs we could run
> > > partial tests:
> > >
> > > 1) Do not run Python tests if none of .py files change
> > >
> > > 2) Do not run Doc generation if neither .py nor .rst files change
> > >
> > > 3) [This might be controversial / not catching a lot of problems] -
> only
> > > run related tests - for corresponding packages - when .py files change.
> > >
> > > 4) Only run "real unit tests" if none of the operators/hooks/sensors
> > change
> > > (we would have to introduce a way to distinguish unit tests - requiring
> > > only basic Airflow + database - from integration tests where
> > > hooks/sensors/operators change). This could be done using the
> > > slimmer/smaller/future production CI image - without the overhead of
> > > starting up all the services.
> > >
> > > 5) Run only static checks + real/related tests in PR for
> > > "drafts/in-progress PRs" but only trigger full tests when someone marks
> > it
> > > as "Ready to merge" (via label or comment in PR).
> > >
> > > Of course if we can make it faster with no-Travis setup running full
> > tests
> > > in all PRs makes more sense, but maybe some of those options above are
> > also
> > > acceptable.
> > >
> > > Comments?
> > >
> > > J.
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Smarter test execution for CI (trivial changes without
full tests)
Posted by James Meickle <jm...@quantopian.com.INVALID>.
GitHub recently introduced the idea of "Draft" PRs:
https://github.blog/2019-02-14-introducing-draft-pull-requests/
Could we do something similar either with that system or something else?
Run a minimal set until it's marked as "ready for testing", and then run a
larger suite.
On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <ka...@gmail.com> wrote:
> Maybe your 4th point covers this but there are frequent Doc only changes.
> In this case we should not run "real-test" but only static checks - mypy,
> pylint. flake8, doc generation.
>
> On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > Hello everyone,
> >
> > On top of moving out from Travis (I will resume working on it next week)
> I
> > thought about some ways to improve the feedback cycle time we have with
> CI
> > (super long now).
> >
> > Maybe we should consider being smarter with running tests only when
> really
> > needed.
> >
> > After introducing pre-commit framework, I thought that it is rather smart
> > in selecting which tests to run locally based on what files changed. It's
> > not perfect of course and there are edge cases where we change .xml files
> > and python tests stop running (for example), but maybe we can introduce
> > some smartness in our test execution scripts based on similar principles.
> >
> > First of all we should always run all tests after master is merged
> (catches
> > edge cases missed by partial running in PR). Then in PRs we could run
> > partial tests:
> >
> > 1) Do not run Python tests if none of .py files change
> >
> > 2) Do not run Doc generation if neither .py nor .rst files change
> >
> > 3) [This might be controversial / not catching a lot of problems] - only
> > run related tests - for corresponding packages - when .py files change.
> >
> > 4) Only run "real unit tests" if none of the operators/hooks/sensors
> change
> > (we would have to introduce a way to distinguish unit tests - requiring
> > only basic Airflow + database - from integration tests where
> > hooks/sensors/operators change). This could be done using the
> > slimmer/smaller/future production CI image - without the overhead of
> > starting up all the services.
> >
> > 5) Run only static checks + real/related tests in PR for
> > "drafts/in-progress PRs" but only trigger full tests when someone marks
> it
> > as "Ready to merge" (via label or comment in PR).
> >
> > Of course if we can make it faster with no-Travis setup running full
> tests
> > in all PRs makes more sense, but maybe some of those options above are
> also
> > acceptable.
> >
> > Comments?
> >
> > J.
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
Re: [DISCUSS] Smarter test execution for CI (trivial changes without
full tests)
Posted by Kaxil Naik <ka...@gmail.com>.
Maybe your 4th point covers this but there are frequent Doc only changes.
In this case we should not run "real-test" but only static checks - mypy,
pylint. flake8, doc generation.
On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Hello everyone,
>
> On top of moving out from Travis (I will resume working on it next week) I
> thought about some ways to improve the feedback cycle time we have with CI
> (super long now).
>
> Maybe we should consider being smarter with running tests only when really
> needed.
>
> After introducing pre-commit framework, I thought that it is rather smart
> in selecting which tests to run locally based on what files changed. It's
> not perfect of course and there are edge cases where we change .xml files
> and python tests stop running (for example), but maybe we can introduce
> some smartness in our test execution scripts based on similar principles.
>
> First of all we should always run all tests after master is merged (catches
> edge cases missed by partial running in PR). Then in PRs we could run
> partial tests:
>
> 1) Do not run Python tests if none of .py files change
>
> 2) Do not run Doc generation if neither .py nor .rst files change
>
> 3) [This might be controversial / not catching a lot of problems] - only
> run related tests - for corresponding packages - when .py files change.
>
> 4) Only run "real unit tests" if none of the operators/hooks/sensors change
> (we would have to introduce a way to distinguish unit tests - requiring
> only basic Airflow + database - from integration tests where
> hooks/sensors/operators change). This could be done using the
> slimmer/smaller/future production CI image - without the overhead of
> starting up all the services.
>
> 5) Run only static checks + real/related tests in PR for
> "drafts/in-progress PRs" but only trigger full tests when someone marks it
> as "Ready to merge" (via label or comment in PR).
>
> Of course if we can make it faster with no-Travis setup running full tests
> in all PRs makes more sense, but maybe some of those options above are also
> acceptable.
>
> Comments?
>
> J.
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>