You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/24 12:56:50 UTC
[GitHub] [airflow] potiuk opened a new issue #10507: Selective builds for different types tests
potiuk opened a new issue #10507:
URL: https://github.com/apache/airflow/issues/10507
**Description**
After we've implemented AIP-21 with backport packages and quite recently #10422 "Optimisation of Image building on CI" - we have now an interesting opportunity to speed up the tests even more. Previously each job had 4-10 minutes of overhead which is now down to just 1 minute of image pulling (so it cannot be optimized further). This is very little overhead and now we can have more jobs running in parallel for one job. This might speed up significantly most of our builds.
While the "merges" on master should build the full suite of tests, we can limit the tests run on PR:
1) If core python files change - run all the tests for all backends and versions.
2) If only files from certain provider fail - run only tests for that provider and all providers that depend on it (we already have all the information to know which providers depend on each other). This can be done only for one backend (and all python versions) - for example, sqlite. Maybe some edge-cases in providers that require mysql/postgres.
With this change, if someone changes only one provider, the tests should run in few minutes rather than hours.
The best is to add another test type "Providers" with different matrix of runs and add custom filtering depending on the files modified by the PR (we already do similar filtering in kubernetes tests). This means we need more jobs (that's why reusing image built only once was so important).
This can be implemented in two steps:
- [ ] extracting common composite test run step (which can be used to run all types of tests including the MySQL/Postgres ones) : https://docs.github.com/en/actions/creating-actions/creating-a-composite-run-steps-action
- [ ] implement the "Providers" type of test job with filtering only the tests for impacted providers (both direct and dependent providers)
**Use case / motivation**
Most contributors - only contributing code to providers should wait less than 10 minutes for the result of their PR build (in terms of elapsed time since the builds starts - there might be some queuing time that we might have no influence on).
**Related Issues**
#10422
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10507: Selective builds for different types tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10507:
URL: https://github.com/apache/airflow/issues/10507#issuecomment-679110300
cc: @kaxil @dimberman @feluelle @turbaszek @mik-laj -> I think we want that :). Any comments are welcome. I am happy to collaborate with others rather than develop it myself so that you can also learn how our CI works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #10507: [OPTIMISATION] Selective builds for different types of tests
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #10507:
URL: https://github.com/apache/airflow/issues/10507
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10507: [OPTIMISATION] Selective builds for different types of tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10507:
URL: https://github.com/apache/airflow/issues/10507#issuecomment-694189094
> I will be happy to collaborate on this one :) -- let me know how should we proceed.
Cool! When we merge the #10959 (which introduces "programmability"), I will open a PR where I will propose a mechanism and some implementation for some of the "programmable" criteria and I will be happy to iterate on it and discuss all the different "variations" we might want and edge cases. We need to be rather careful with that - to not miss some important dependencies and cases. I have my proposal where I introduce not only several variations but I also add unit tests covering it, to make it robust and will be happy to collaborate on that :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #10507: [OPTIMISATION] Selective builds for different types of tests
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #10507:
URL: https://github.com/apache/airflow/issues/10507#issuecomment-694155078
I will be happy to collaborate on this one :) -- let me know how should we proceed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org