You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/20 13:38:43 UTC

[GitHub] [airflow] potiuk opened a new pull request #14332: Attempts to stabilize and improve speed of static checks

potiuk opened a new pull request #14332:
URL: https://github.com/apache/airflow/pull/14332

This change attempts to stabilize pylint checks. Since we have
recently added self-hosted runners with multiple CPUS, seems that
re-enabling parallel mode for pylint makes perfect sense as we will
finally be able to use the parallelism and speed up static checks
significantly.

Previously the tests were run in single-processor mode in attempt
to avoid random mistakes where different files were processed in
different processes. This led to random pylint or mypy problems
and aither false-positives or false negatives before especially
when it came to circular dependencies. but since we are now past
heavy refactoring, this should be no problem for future changes
and occasional false positive/negative is less disruptive than
long checks.

The attempt is made to apply sorting order in the files passed
to pylint. This should provide more stability in the results
of running the tests in PR and in master.

We had some custom pylint plugins that prevented using of pylint
parallelism. For now we are giving up on one of the plugins
(no asserts use) and we rely on committer's review on that (we
have a rule in place to only use asserts in tests). The other
plugin was replaced by coming back to separation of "main code"
and "test code" and applying different rules to those - we have
now two different configuration files|: pylintrc and
pylintrc-tests to control settings for those two different cases.

Mypy and flake8 have been parallelized at the level of pre-commits.

By implementing those changes we are likely to speed up the
tests on self-hosted runners 6x-8x times.

<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.

Feel free to ping committers for the review!

In case of existing issue, reference it using one of the following:

closes: #ISSUE
related: #ISSUE

How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->

---
**^ Add meaningful description above**

Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org