You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2021/02/21 09:22:24 UTC
[airflow] branch master updated: Attempts to stabilize and improve
speed of static checks (#14332)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new 82cb041 Attempts to stabilize and improve speed of static checks (#14332)
82cb041 is described below
commit 82cb041a8f7d9856ef8db9426d82595e8e3a5c42
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Feb 21 10:22:17 2021 +0100
Attempts to stabilize and improve speed of static checks (#14332)
This change attempts to stabilize pylint checks. Since we have
recently added self-hosted runners with multiple CPUS, seems that
re-enabling parallel mode for pylint makes perfect sense as we will
finally be able to use the parallelism and speed up static checks
significantly.
Previously the tests were run in single-processor mode in attempt
to avoid random mistakes where different files were processed in
different processes. This led to random pylint or mypy problems
and aither false-positives or false negatives before especially
when it came to circular dependencies. but since we are now past
heavy refactoring, this should be no problem for future changes
and occasional false positive/negative is less disruptive than
long checks.
The attempt is made to apply sorting order in the files passed
to pylint. This should provide more stability in the results
of running the tests in PR and in master.
We had some custom pylint plugins that prevented using of pylint
parallelism. For now we are giving up on one of the plugins
(no asserts use) and we rely on committer's review on that (we
have a rule in place to only use asserts in tests). The other
plugin was replaced by coming back to separation of "main code"
and "test code" and applying different rules to those - we have
now two different configuration files|: pylintrc and
pylintrc-tests to control settings for those two different cases.
Mypy and flake8 have been parallelized at the level of pre-commits.
By implementing those changes we are likely to speed up the
tests on self-hosted runners 6x-8x times.
Also caching of pre-commit environments (including the
pre-commit installation) is improved. Previously we only cached the
environment created by the pre-commit, but with this change we
also cache the pre-commit installation in .local directory - this
should save 1-2 minutes of the job.
---
.dockerignore | 1 +
.github/workflows/ci.yml | 57 ++++++++++++-------
.pre-commit-config.yaml | 33 ++++++-----
pylintrc | 4 +-
pylintrc => pylintrc-tests | 11 +++-
scripts/ci/docker-compose/local.yml | 1 +
scripts/ci/libraries/_build_images.sh | 26 ++++-----
scripts/ci/libraries/_local_mounts.sh | 1 +
scripts/ci/libraries/_start_end.sh | 4 +-
.../ci/static_checks/run_basic_static_checks.sh | 2 +-
scripts/ci/static_checks/run_static_checks.sh | 4 +-
scripts/in_container/run_pylint.sh | 4 +-
setup.py | 2 +-
tests/airflow_pylint/__init__.py | 16 ------
tests/airflow_pylint/disable_checks_for_tests.py | 65 ----------------------
tests/airflow_pylint/do_not_use_asserts.py | 44 ---------------
tests/test_utils/perf/dags/elastic_dag.py | 2 -
17 files changed, 92 insertions(+), 185 deletions(-)
diff --git a/.dockerignore b/.dockerignore
index 70494e6..b5a61d3 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -52,6 +52,7 @@
!.flake8
!.dockerignore
!pylintrc
+!pylintrc-tests
!pytest.ini
!CHANGELOG.txt
!LICENSE
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index bdcf82e..d1f2a20 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -275,17 +275,23 @@ jobs:
uses: actions/setup-python@v2
with:
python-version: ${{needs.build-info.outputs.defaultPythonVersion}}
- - name: Cache pre-commit env
- uses: actions/cache@v2
- with:
- path: ~/.cache/pre-commit
- key: pre-commit-no-pylint-${{ hashFiles('.pre-commit-config.yaml') }}
- restore-keys: pre-commit-no-pylint-
- name: "Free space"
run: ./scripts/ci/tools/ci_free_space_on_ci.sh
if: "!contains(needs.build-info.outputs.runsOn, 'self-hosted')"
- name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
+ - name: "Cache pre-commit local-installation"
+ uses: actions/cache@v2
+ with:
+ path: ~/.local
+ key: pre-commit-local-installation-${{ hashFiles('setup.py', 'setup.cfg') }}
+ restore-keys: pre-commit-local-installation-
+ - name: "Cache pre-commit envs: no-pylint"
+ uses: actions/cache@v2
+ with:
+ path: ~/.cache/pre-commit
+ key: pre-commit-no-pylint-${{ hashFiles('.pre-commit-config.yaml') }}
+ restore-keys: pre-commit-no-pylint-
- name: "Static checks: except pylint"
run: ./scripts/ci/static_checks/run_static_checks.sh
env:
@@ -313,12 +319,6 @@ jobs:
uses: actions/setup-python@v2
with:
python-version: ${{needs.build-info.outputs.defaultPythonVersion}}
- - name: Cache pre-commit env
- uses: actions/cache@v2
- with:
- path: ~/.cache/pre-commit
- key: pre-commit-basic-${{ hashFiles('.pre-commit-config.yaml') }}
- restore-keys: pre-commit-no-basic-
- name: >
Fetch incoming commit ${{ github.sha }} with its parent
uses: actions/checkout@v2
@@ -326,9 +326,20 @@ jobs:
ref: ${{ github.sha }}
fetch-depth: 2
persist-credentials: false
+ - name: "Cache pre-commit local installation"
+ uses: actions/cache@v2
+ with:
+ path: ~/.local
+ key: pre-commit-local-installation-${{ hashFiles('setup.py', 'setup.cfg') }}
+ restore-keys: pre-commit-local-installation-
+ - name: "Cache pre-commit envs - basic"
+ uses: actions/cache@v2
+ with:
+ path: ~/.cache/pre-commit
+ key: pre-commit-basic-${{ hashFiles('.pre-commit-config.yaml') }}
+ restore-keys: pre-commit-basic-
- name: "Static checks: basic checks only"
- run: |
- ./scripts/ci/static_checks/run_basic_static_checks.sh "${{ github.sha }}"
+ run: ./scripts/ci/static_checks/run_basic_static_checks.sh "${{ github.sha }}"
env:
VERBOSE: false
@@ -353,17 +364,23 @@ jobs:
uses: actions/setup-python@v2
with:
python-version: ${{needs.build-info.outputs.defaultPythonVersion}}
- - name: "Cache pre-commit env"
- uses: actions/cache@v2
- with:
- path: ~/.cache/pre-commit
- key: pre-commit-pylint-${{ hashFiles('.pre-commit-config.yaml') }}
- restore-keys: pre-commit-pylint-
- name: "Free space"
run: ./scripts/ci/tools/ci_free_space_on_ci.sh
if: "!contains(needs.build-info.outputs.runsOn, 'self-hosted')"
- name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
+ - name: "Cache pre-commit local-installation"
+ uses: actions/cache@v2
+ with:
+ path: ~/.local
+ key: pre-commit-local-installation-${{ hashFiles('setup.py', 'setup.cfg') }}
+ restore-keys: pre-commit-local-installation-
+ - name: "Cache pre-commit envs - pylint"
+ uses: actions/cache@v2
+ with:
+ path: ~/.cache/pre-commit
+ key: pre-commit-pylint-${{ hashFiles('.pre-commit-config.yaml') }}
+ restore-keys: pre-commit-pylint-
- name: "Static checks: pylint"
run: ./scripts/ci/static_checks/run_static_checks.sh pylint
env:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index d0c94e4..458fa88 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -548,6 +548,13 @@ repos:
files: airflow/config_templates/config.yml$
require_serial: true
additional_dependencies: ['jsonschema==3.2.0', 'PyYAML==5.3.1', 'requests==2.25.0']
+ - id: flynt
+ name: Convert to f-strings with flynt
+ entry: flynt
+ language: python
+ language_version: python3
+ additional_dependencies: ['flynt']
+ files: \.py$
## ADD MOST PRE-COMMITS ABOVE THAT LINE
# The below pre-commits are those requiring CI image to be built
- id: build
@@ -567,26 +574,33 @@ repos:
language: system
entry: "./scripts/ci/pre_commit/pre_commit_mypy.sh"
files: ^chart/.*\.py$
- require_serial: true
+ require_serial: false
- id: mypy
name: Run mypy for /docs/ folder
language: system
entry: "./scripts/ci/pre_commit/pre_commit_mypy.sh"
files: ^docs/.*\.py$
exclude: rtd-deprecation
- require_serial: true
+ require_serial: false
- id: pylint
- name: Run pylint
+ name: Run pylint for main code
language: system
- entry: "./scripts/ci/pre_commit/pre_commit_pylint.sh"
+ entry: "./scripts/ci/pre_commit/pre_commit_pylint.sh --rcfile pylintrc"
files: \.py$
- exclude: ^scripts/.*\.py$|^dev|^provider_packages|^chart
+ exclude: ^scripts/.*\.py$|^dev|^provider_packages|^chart|^tests|^kubernetes_tests
+ pass_filenames: true
+ require_serial: true
+ - id: pylint
+ name: Run pylint for tests
+ language: system
+ entry: "./scripts/ci/pre_commit/pre_commit_pylint.sh --rcfile pylintrc-tests"
+ files: ^tests/.*\.py$
pass_filenames: true
require_serial: true
- id: pylint
name: Run pylint for helm chart tests
language: system
- entry: "./scripts/ci/pre_commit/pre_commit_pylint.sh"
+ entry: "./scripts/ci/pre_commit/pre_commit_pylint.sh --rcfile pylintrc-tests"
files: ^chart/.*\.py$
pass_filenames: true
require_serial: true
@@ -602,11 +616,4 @@ repos:
entry: "./scripts/ci/pre_commit/pre_commit_in_container_bats_test.sh"
files: ^tests/bats/in_container/.*.bats$|^scripts/in_container/.*sh
pass_filenames: false
- - id: flynt
- name: Convert to f-strings with flynt
- entry: flynt
- language: python
- language_version: python3
- additional_dependencies: ['flynt']
- files: \.py$
## ONLY ADD PRE-COMMITS HERE THAT REQUIRE CI IMAGE
diff --git a/pylintrc b/pylintrc
index cd9a060..c2253d7 100644
--- a/pylintrc
+++ b/pylintrc
@@ -29,7 +29,7 @@ ignore-patterns=
# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the
# number of processors available to use.
-jobs=1
+jobs=0
# Control the amount of potential inferred values when inferring a single
# object. This can help the performance when dealing with large functions or
@@ -38,7 +38,7 @@ limit-inference-results=100
# List of plugins (as comma separated values of python modules names) to load,
# usually to register additional checkers.
-load-plugins=tests.airflow_pylint.do_not_use_asserts,tests.airflow_pylint.disable_checks_for_tests
+#load-plugins=tests.airflow_pylint.do_not_use_asserts,tests.airflow_pylint.disable_checks_for_tests
# Pickle collected data for later comparisons.
persistent=yes
diff --git a/pylintrc b/pylintrc-tests
similarity index 98%
copy from pylintrc
copy to pylintrc-tests
index cd9a060..5b360a9 100644
--- a/pylintrc
+++ b/pylintrc-tests
@@ -29,7 +29,7 @@ ignore-patterns=
# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the
# number of processors available to use.
-jobs=1
+jobs=0
# Control the amount of potential inferred values when inferring a single
# object. This can help the performance when dealing with large functions or
@@ -38,7 +38,7 @@ limit-inference-results=100
# List of plugins (as comma separated values of python modules names) to load,
# usually to register additional checkers.
-load-plugins=tests.airflow_pylint.do_not_use_asserts,tests.airflow_pylint.disable_checks_for_tests
+#load-plugins=tests.airflow_pylint.do_not_use_asserts,tests.airflow_pylint.disable_checks_for_tests
# Pickle collected data for later comparisons.
persistent=yes
@@ -155,7 +155,12 @@ disable=print-statement,
missing-module-docstring,
import-outside-toplevel, # We import outside toplevel to avoid cyclic imports
raise-missing-from, # We don't use raise...from
- misplaced-comparison-constant
+ misplaced-comparison-constant,
+ # Those checks are disabled for tests only
+ missing-docstring,
+ no-self-use,
+ too-many-public-methods,
+ protected-access
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
diff --git a/scripts/ci/docker-compose/local.yml b/scripts/ci/docker-compose/local.yml
index 641f93f..56a06f1 100644
--- a/scripts/ci/docker-compose/local.yml
+++ b/scripts/ci/docker-compose/local.yml
@@ -47,6 +47,7 @@ services:
- ../../../hooks:/opt/airflow/hooks:cached
- ../../../logs:/root/airflow/logs:cached
- ../../../pylintrc:/opt/airflow/pylintrc:cached
+ - ../../../pylintrc-tests:/opt/airflow/pylintrc-tests:cached
- ../../../pyproject.toml:/opt/airflow/pyproject.toml:cached
- ../../../pytest.ini:/opt/airflow/pytest.ini:cached
- ../../../scripts:/opt/airflow/scripts:cached
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index ae2b076..ccb913b 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -468,14 +468,14 @@ function build_image::configure_github_docker_registry() {
# For now ghcr.io can only authenticate using Personal Access Token with package access scope.
# There are plans to implement GITHUB_TOKEN authentication but this is not implemented yet
token="${CONTAINER_REGISTRY_TOKEN=}"
- echo
- echo "Using CONTAINER_REGISTRY_TOKEN!"
- echo
+ verbosity::print_info
+ verbosity::print_info "Using CONTAINER_REGISTRY_TOKEN!"
+ verbosity::print_info
elif [[ "${GITHUB_REGISTRY}" == "docker.pkg.github.com" ]]; then
token="${GITHUB_TOKEN}"
- echo
- echo "Using GITHUB_TOKEN!"
- echo
+ verbosity::print_info
+ verbosity::print_info "Using GITHUB_TOKEN!"
+ verbosity::print_info
else
echo
echo "${COLOR_RED}ERROR: Bad value of '${GITHUB_REGISTRY}'. Should be either 'ghcr.io' or 'docker.pkg.github.com'!${COLOR_RESET}"
@@ -483,9 +483,9 @@ function build_image::configure_github_docker_registry() {
exit 1
fi
if [[ -z "${token}" ]] ; then
- echo
- echo "Skip logging in to Github Registry. No Token available!"
- echo
+ verbosity::print_info
+ verbosity::print_info "Skip logging in to Github Registry. No Token available!"
+ verbosity::print_info
fi
if [[ -n "${token}" ]]; then
echo "${token}" | docker login \
@@ -493,14 +493,14 @@ function build_image::configure_github_docker_registry() {
--password-stdin \
"${GITHUB_REGISTRY}"
else
- echo "Skip Login to GitHub Registry ${GITHUB_REGISTRY} as token is missing"
+ verbosity::print_info "Skip Login to GitHub Registry ${GITHUB_REGISTRY} as token is missing"
fi
- echo "Make sure experimental docker features are enabled"
+ verbosity::print_info "Make sure experimental docker features are enabled"
local new_config
new_config=$(jq '.experimental = "enabled"' "${HOME}/.docker/config.json")
echo "${new_config}" > "${HOME}/.docker/config.json"
- echo "Docker config after change:"
- echo "${new_config}"
+ verbosity::print_info "Docker config after change:"
+ verbosity::print_info "${new_config}"
start_end::group_end
fi
}
diff --git a/scripts/ci/libraries/_local_mounts.sh b/scripts/ci/libraries/_local_mounts.sh
index 4925140..9f06752 100644
--- a/scripts/ci/libraries/_local_mounts.sh
+++ b/scripts/ci/libraries/_local_mounts.sh
@@ -43,6 +43,7 @@ function local_mounts::generate_local_mounts_list {
"$prefix"hooks:/opt/airflow/hooks:cached
"$prefix"logs:/root/airflow/logs:cached
"$prefix"pylintrc:/opt/airflow/pylintrc:cached
+ "$prefix"pylintrc-tests:/opt/airflow/pylintrc-tests:cached
"$prefix"pyproject.toml:/opt/airflow/pyproject.toml:cached
"$prefix"pytest.ini:/opt/airflow/pytest.ini:cached
"$prefix"scripts:/opt/airflow/scripts:cached
diff --git a/scripts/ci/libraries/_start_end.sh b/scripts/ci/libraries/_start_end.sh
index 7db7aaf..d4490cb 100644
--- a/scripts/ci/libraries/_start_end.sh
+++ b/scripts/ci/libraries/_start_end.sh
@@ -18,7 +18,7 @@
# Starts group for Github Actions - makes logs much more readable
function start_end::group_start {
- if [[ ${GITHUB_ACTIONS=} == "true" ]]; then
+ if [[ ${GITHUB_ACTIONS=} == "true" && ${PRINT_INFO_FROM_SCRIPTS} != "false" ]]; then
echo "::group::${1}"
else
echo
@@ -29,7 +29,7 @@ function start_end::group_start {
# Ends group for Github Actions
function start_end::group_end {
- if [[ ${GITHUB_ACTIONS=} == "true" ]]; then
+ if [[ ${GITHUB_ACTIONS=} == "true" && ${PRINT_INFO_FROM_SCRIPTS} != "false" ]]; then
echo -e "\033[0m" # Disable any colors set in the group
echo "::endgroup::"
fi
diff --git a/scripts/ci/static_checks/run_basic_static_checks.sh b/scripts/ci/static_checks/run_basic_static_checks.sh
index e017bec..847b373 100755
--- a/scripts/ci/static_checks/run_basic_static_checks.sh
+++ b/scripts/ci/static_checks/run_basic_static_checks.sh
@@ -35,7 +35,7 @@ fi
COMMIT_SHA="${1}"
shift
-python -m pip install pre-commit \
+python -m pip install --user pre-commit \
--constraint "https://raw.githubusercontent.com/apache/airflow/${DEFAULT_CONSTRAINTS_BRANCH}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
if [[ $# == "0" ]]; then
diff --git a/scripts/ci/static_checks/run_static_checks.sh b/scripts/ci/static_checks/run_static_checks.sh
index 4ceb668..7777119 100755
--- a/scripts/ci/static_checks/run_static_checks.sh
+++ b/scripts/ci/static_checks/run_static_checks.sh
@@ -29,9 +29,11 @@ build_images::prepare_ci_build
build_images::rebuild_ci_image_if_needed
-python -m pip install pre-commit \
+python -m pip install --user pre-commit \
--constraint "https://raw.githubusercontent.com/apache/airflow/${DEFAULT_CONSTRAINTS_BRANCH}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
+export PATH=~/.local/bin:${PATH}
+
if [[ $# == "0" ]]; then
pre-commit run --all-files --show-diff-on-failure --color always
else
diff --git a/scripts/in_container/run_pylint.sh b/scripts/in_container/run_pylint.sh
index d62a1b5..f1b8306 100755
--- a/scripts/in_container/run_pylint.sh
+++ b/scripts/in_container/run_pylint.sh
@@ -19,7 +19,7 @@
. "$( dirname "${BASH_SOURCE[0]}" )/_in_container_script_init.sh"
# Pylint is _very_ unhappy with implicit namespaces, so for this test only, we need to make it not
-trap "rm airflow/providers/__init__.py" EXIT
+trap "rm -f airflow/providers/__init__.py" EXIT
touch airflow/providers/__init__.py
@@ -43,7 +43,7 @@ if [[ ${#@} == "0" ]]; then
-name "*.py" \
-not -name 'webserver_config.py' | \
grep ".*.py$" | \
- grep -vFf scripts/ci/pylint_todo.txt | xargs pylint --output-format=colorized
+ grep -vFf scripts/ci/pylint_todo.txt | sort | xargs pylint -j 0 --output-format=colorized
else
/usr/local/bin/pylint --output-format=colorized "$@"
fi
diff --git a/setup.py b/setup.py
index a5e8869..26d2b71 100644
--- a/setup.py
+++ b/setup.py
@@ -493,7 +493,7 @@ devel = [
'paramiko',
'pipdeptree',
'pre-commit',
- 'pylint==2.6.0',
+ 'pylint',
'pysftp',
'pytest~=6.0',
'pytest-cov',
diff --git a/tests/airflow_pylint/__init__.py b/tests/airflow_pylint/__init__.py
deleted file mode 100644
index 13a8339..0000000
--- a/tests/airflow_pylint/__init__.py
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
diff --git a/tests/airflow_pylint/disable_checks_for_tests.py b/tests/airflow_pylint/disable_checks_for_tests.py
deleted file mode 100644
index cbdf9b2..0000000
--- a/tests/airflow_pylint/disable_checks_for_tests.py
+++ /dev/null
@@ -1,65 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-
-from astroid import MANAGER, scoped_nodes
-from pylint.lint import PyLinter
-
-DISABLED_CHECKS_FOR_TESTS = (
- "missing-docstring, no-self-use, too-many-public-methods, protected-access, do-not-use-asserts"
-)
-
-
-def register(_: PyLinter):
- """
- Skip registering any plugin. This is not a real plugin - we only need it to register transform before
- running pylint.
-
- :param _:
- :return:
- """
-
-
-def transform(mod):
- """
- It's a small hack but one that gives us a lot of speedup in pylint tests. We are replacing the first
- line of the file with pylint-disable (or update existing one) when file name start with `test_` or
- (for providers) when it is the full path of the package (both cases occur in pylint)
-
- :param mod: astroid module
- :return: None
- """
- if (
- mod.name.startswith("test_")
- or mod.name.startswith("tests.")
- or mod.name.startswith("kubernetes_tests.")
- or mod.name.startswith("chart.")
- ):
- decoded_lines = mod.stream().read().decode("utf-8").split("\n")
- if decoded_lines[0].startswith("# pylint: disable="):
- decoded_lines[0] = decoded_lines[0] + " " + DISABLED_CHECKS_FOR_TESTS
- elif decoded_lines[0].startswith("#") or decoded_lines[0].strip() == "":
- decoded_lines[0] = "# pylint: disable=" + DISABLED_CHECKS_FOR_TESTS
- else:
- raise Exception(
- f"The first line of module {mod.name} is not a comment or empty. " f"Please make sure it is!"
- )
- # pylint will read from `.file_bytes` attribute later when tokenization
- mod.file_bytes = "\n".join(decoded_lines).encode("utf-8")
-
-
-MANAGER.register_transform(scoped_nodes.Module, transform)
diff --git a/tests/airflow_pylint/do_not_use_asserts.py b/tests/airflow_pylint/do_not_use_asserts.py
deleted file mode 100644
index 47a0e20..0000000
--- a/tests/airflow_pylint/do_not_use_asserts.py
+++ /dev/null
@@ -1,44 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-from pylint.checkers import BaseChecker
-from pylint.interfaces import IAstroidChecker
-from pylint.lint import PyLinter
-
-
-class DoNotUseAssertsChecker(BaseChecker):
- __implements__ = IAstroidChecker
-
- name = 'do-not-use-asserts'
- priority = -1
- msgs = {
- 'E7401': (
- 'Do not use asserts.',
- 'do-not-use-asserts',
- 'Asserts should not be used in the main Airflow code.',
- ),
- }
-
- def visit_assert(self, node):
- self.add_message(
- self.name,
- node=node,
- )
-
-
-def register(linter: PyLinter):
- linter.register_checker(DoNotUseAssertsChecker(linter))
diff --git a/tests/test_utils/perf/dags/elastic_dag.py b/tests/test_utils/perf/dags/elastic_dag.py
index 9aa0a4d..a57a328 100644
--- a/tests/test_utils/perf/dags/elastic_dag.py
+++ b/tests/test_utils/perf/dags/elastic_dag.py
@@ -40,12 +40,10 @@ def parse_time_delta(time_str: str):
"""
parts = RE_TIME_DELTA.match(time_str)
- # pylint: disable=do-not-use-asserts
assert parts is not None, (
f"Could not parse any time information from '{time_str}'. "
f"Examples of valid strings: '8h', '2d8h5m20s', '2m4s'"
)
- # pylint: enable=do-not-use-asserts
time_params = {name: float(param) for name, param in parts.groupdict().items() if param}
return timedelta(**time_params) # type: ignore