You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2021/06/04 20:03:01 UTC
[airflow] branch main updated: Add pre-commit hook for Boring
Cyborg (#16260)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 014edbc Add pre-commit hook for Boring Cyborg (#16260)
014edbc is described below
commit 014edbc71fa8752f48047c52bb850dc1e0cd7c0f
Author: Kamil BreguĊa <mi...@users.noreply.github.com>
AuthorDate: Fri Jun 4 22:02:36 2021 +0200
Add pre-commit hook for Boring Cyborg (#16260)
I noticed that the cyborg configuration contains frequently unused patterns that result from moving files. To spot these problems before merging, I add a pre-commit hook that checks to see if each pattern has at least one match.
---
.github/boring-cyborg.yml | 3 +-
.pre-commit-config.yaml | 7 +++
BREEZE.rst | 29 ++++++------
STATIC_CODE_CHECKS.rst | 2 +
breeze-complete | 1 +
scripts/ci/pre_commit/pre_commit_boring_cyborg.py | 57 +++++++++++++++++++++++
6 files changed, 83 insertions(+), 16 deletions(-)
diff --git a/.github/boring-cyborg.yml b/.github/boring-cyborg.yml
index 8afc30e..834265e 100644
--- a/.github/boring-cyborg.yml
+++ b/.github/boring-cyborg.yml
@@ -129,7 +129,6 @@ labelPRBasedOnFilePath:
- airflow/ui/**/*
area:CLI:
- - airflow/bin/cli.py
- airflow/cli/**/*.py
- tests/cli/**/*.py
- docs/apache-airflow/cli-and-env-variables-ref.rst
@@ -160,7 +159,7 @@ labelPRBasedOnFilePath:
- airflow/task/task_runner/**/*
- airflow/utils/dag_processing.py
- docs/apache-airflow/executor/**/*
- - docs/apache-airflow/scheduler.rst
+ - docs/apache-airflow/concepts/scheduler.rst
- tests/executors/**/*
- tests/jobs/**/*
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index c5546ed..f0f4906 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -447,6 +447,13 @@ repos:
files: ^airflow/migrations/versions/.*\.(py)$
pass_filenames: false
require_serial: true
+ - id: boring-cyborg
+ name: Checks for Boring Cyborg configuration consistency
+ language: python
+ entry: ./scripts/ci/pre_commit/pre_commit_boring_cyborg.py
+ pass_filenames: false
+ require_serial: true
+ additional_dependencies: ['pyyaml', 'termcolor==1.1.0', 'wcmatch==8.2']
- id: sort-in-the-wild
name: Sort INTHEWILD.md alphabetically
entry: ./scripts/ci/pre_commit/pre_commit_sort_in_the_wild.sh
diff --git a/BREEZE.rst b/BREEZE.rst
index 98b3df3..8af8ec2 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -2233,20 +2233,21 @@ This is the current syntax for `./breeze <./breeze>`_:
all all-but-pylint airflow-config-yaml airflow-providers-available
airflow-provider-yaml-files-ok base-operator bats-tests bats-in-container-tests
- black blacken-docs build build-providers-dependencies check-apache-license
- check-builtin-literals check-executables-have-shebangs check-hooks-apply
- check-integrations check-merge-conflict check-xml consistent-pylint
- daysago-import-check debug-statements detect-private-key doctoc dont-use-safe-filter
- end-of-file-fixer fix-encoding-pragma flake8 flynt forbid-tabs helm-lint identity
- incorrect-use-of-LoggingMixin insert-license isort json-schema language-matters
- lint-dockerfile lint-openapi markdownlint mermaid mixed-line-ending mypy mypy-helm
- no-providers-in-core-examples no-relative-imports pre-commit-descriptions
- pre-commit-hook-names pretty-format-json provide-create-sessions providers-init-file
- provider-yamls pydevd pydocstyle pylint pylint-tests python-no-log-warn pyupgrade
- restrict-start_date rst-backticks setup-order setup-extra-packages shellcheck
- sort-in-the-wild sort-spelling-wordlist stylelint trailing-whitespace ui-lint
- update-breeze-file update-extras update-local-yml-file update-setup-cfg-file
- verify-db-migrations-documented version-sync www-lint yamllint
+ black blacken-docs boring-cyborg build build-providers-dependencies
+ check-apache-license check-builtin-literals check-executables-have-shebangs
+ check-hooks-apply check-integrations check-merge-conflict check-xml
+ consistent-pylint daysago-import-check debug-statements detect-private-key doctoc
+ dont-use-safe-filter end-of-file-fixer fix-encoding-pragma flake8 flynt forbid-tabs
+ helm-lint identity incorrect-use-of-LoggingMixin insert-license isort json-schema
+ language-matters lint-dockerfile lint-openapi markdownlint mermaid mixed-line-ending
+ mypy mypy-helm no-providers-in-core-examples no-relative-imports
+ pre-commit-descriptions pre-commit-hook-names pretty-format-json
+ provide-create-sessions providers-init-file provider-yamls pydevd pydocstyle pylint
+ pylint-tests python-no-log-warn pyupgrade restrict-start_date rst-backticks
+ setup-order setup-extra-packages shellcheck sort-in-the-wild sort-spelling-wordlist
+ stylelint trailing-whitespace ui-lint update-breeze-file update-extras
+ update-local-yml-file update-setup-cfg-file verify-db-migrations-documented
+ version-sync www-lint yamllint
You can pass extra arguments including options to the pre-commit framework as
<EXTRA_ARGS> passed after --. For example:
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index 7c95e98..c11f5ed 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -62,6 +62,8 @@ require Breeze Docker images to be installed locally:
----------------------------------- ---------------------------------------------------------------- ------------
``blacken-docs`` Run black on python code blocks in documentation files
----------------------------------- ---------------------------------------------------------------- ------------
+``boring-cyborg`` Checks for Boring Cyborg configuration consistency
+----------------------------------- ---------------------------------------------------------------- ------------
``build`` Builds image for mypy, pylint, flake8 *
----------------------------------- ---------------------------------------------------------------- ------------
``build-providers-dependencies`` Regenerates the JSON file with cross-provider dependencies
diff --git a/breeze-complete b/breeze-complete
index 4c182a2..9ba1bd7 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -86,6 +86,7 @@ bats-tests
bats-in-container-tests
black
blacken-docs
+boring-cyborg
build
build-providers-dependencies
check-apache-license
diff --git a/scripts/ci/pre_commit/pre_commit_boring_cyborg.py b/scripts/ci/pre_commit/pre_commit_boring_cyborg.py
new file mode 100755
index 0000000..48a410a
--- /dev/null
+++ b/scripts/ci/pre_commit/pre_commit_boring_cyborg.py
@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import subprocess
+import sys
+from pathlib import Path
+
+import yaml
+from termcolor import colored
+from wcmatch import glob
+
+if __name__ not in ("__main__", "__mp_main__"):
+ raise SystemExit(
+ "This file is intended to be executed as an executable program. You cannot use it as a module."
+ f"To run this script, run the ./{__file__} command"
+ )
+
+CONFIG_KEY = 'labelPRBasedOnFilePath'
+
+current_files = subprocess.check_output(["git", "ls-files"]).decode().splitlines()
+git_root = Path(subprocess.check_output(['git', 'rev-parse', '--show-toplevel']).decode().strip())
+cyborg_config_path = git_root / ".github" / "boring-cyborg.yml"
+cyborg_config = yaml.safe_load(cyborg_config_path.read_text())
+if CONFIG_KEY not in cyborg_config:
+ raise SystemExit(f"Missing section {CONFIG_KEY}")
+
+errors = []
+for label, patterns in cyborg_config[CONFIG_KEY].items():
+ for pattern in patterns:
+ if glob.globfilter(current_files, pattern, flags=glob.G | glob.E):
+ continue
+ yaml_path = f'{CONFIG_KEY}.{label}'
+ errors.append(
+ f"Unused pattern [{colored(pattern, 'cyan')}] in [{colored(yaml_path, 'cyan')}] section."
+ )
+
+if errors:
+ print(f"Found {colored(str(len(errors)), 'red')} problems:")
+ print("\n".join(errors))
+ sys.exit(1)
+else:
+ print("No found problems. Have a good day!")