You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2023/03/08 14:44:15 UTC
[airflow] branch v2-5-test updated (4e2af12f99 -> b87990dc9d)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a change to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 4e2af12f99 Make static checks generated file more stable accross the board (#29080)
new 81ca55c1e2 Fix bugs introduced when moving the code of issue generation to breeze (#28683)
new 6180b2d01c Fix pyzmq warning stopping provider package tests (#28901)
new a0045495a4 Removal of flake8 file has broken breeze bind mounts (#28909)
new 2b377377e4 Fix a few remaining references to flake8 (#28915)
new 7945514559 Update boring-cyborg.yml with more PR labels (#28955)
new 626b1514d4 Exclude dependedbot PRs from status testing report (#28954)
new 04d1c17efc Fix hot loop with a burst of log messages (#28778) (#28780)
new fbada04257 Fix `project-name` generation for Providers `test-type` in Breeze (#29030)
new d56606ea97 Fix asset compilation in start-airflow (#29185)
new d2840e97cf Fix commit message for constraints (#29215)
new 4451aba667 Fix `breeze static_checks` type arg (#29292)
new 3a3932440b Upgrade PIP to 23.0 version (#29421)
new 6a240c3b26 Bump stale bot version (#29491)
new cf89b09a62 Use "all" integrations to run tests when "all" is used (#29517)
new 458c41fda6 Migrate breeze unit tests to pytest. (#29639)
new 99b1cc6707 Do not crash when a version fails to parse (#29685)
new ffdb121523 Propose to upgrade to newer dependencies in case build CI image fail (#29705)
new b37ec74e02 Upgrade pip to version 23.0.1 (#29767)
new d0966cfc2c Limit importlib-metadata backwport to < 5.0.0 (#29924)
new b87990dc9d Make sure DOCKER_BUILDKIT=1 variable is set for all builds (#29928)
The 20 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.dockerignore | 1 -
.gitattributes | 1 -
.github/boring-cyborg.yml | 60 +++++++++++++++++++++-
.github/workflows/ci.yml | 4 ++
.github/workflows/stale.yml | 2 +-
CONTRIBUTING.rst | 2 +-
CONTRIBUTORS_QUICK_START.rst | 4 +-
Dockerfile | 7 +--
Dockerfile.ci | 13 ++++-
IMAGES.rst | 2 +-
STATIC_CODE_CHECKS.rst | 18 +++----
.../airflow_breeze/commands/ci_image_commands.py | 47 +++++++++--------
.../airflow_breeze/commands/developer_commands.py | 10 ++--
.../commands/production_image_commands.py | 6 ++-
.../commands/release_management_commands.py | 8 +--
.../airflow_breeze/commands/testing_commands.py | 6 ++-
dev/breeze/src/airflow_breeze/global_constants.py | 2 +-
.../src/airflow_breeze/params/shell_params.py | 4 +-
.../airflow_breeze/utils/docker_command_utils.py | 1 -
.../utils/find_newer_dependencies.py | 5 +-
dev/breeze/src/airflow_breeze/utils/run_utils.py | 12 +++--
dev/breeze/tests/test_run_utils.py | 3 +-
dev/prepare_release_issue.py | 3 ++
docs/docker-stack/build-arg-ref.rst | 2 +-
images/breeze/output-commands-hash.txt | 6 +--
...t_release-management_generate-issue-content.svg | 60 +++++++++++-----------
images/breeze/output_static-checks.svg | 2 +-
scripts/ci/constraints/ci_commit_constraints.sh | 8 ++-
scripts/ci/docker-compose/local.yml | 3 --
.../ci/pre_commit/pre_commit_compile_www_assets.py | 1 -
scripts/docker/clean-logs.sh | 3 +-
scripts/docker/common.sh | 2 +-
scripts/docker/entrypoint_ci.sh | 9 ++++
scripts/in_container/verify_providers.py | 5 ++
setup.cfg | 5 +-
35 files changed, 219 insertions(+), 108 deletions(-)
[airflow] 14/20: Use "all" integrations to run tests when "all" is used (#29517)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit cf89b09a62e173f0fbdee13d4f6bea3a65ab7401
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Feb 14 00:10:12 2023 +0100
Use "all" integrations to run tests when "all" is used (#29517)
The change #29449 made a distinction between "all" and "available"
tests and it turned out that "All" test type uses "available" tests.
The name "all" is actually wrong now. It should be renamed to be
"all-with-tests" or similar, but this should be done in a separate PR
(cherry picked from commit 72c3817a44eea5005761ae3b621e8c39fde136ad)
---
dev/breeze/src/airflow_breeze/params/shell_params.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/params/shell_params.py b/dev/breeze/src/airflow_breeze/params/shell_params.py
index fad96e81f5..fb7ebe0c30 100644
--- a/dev/breeze/src/airflow_breeze/params/shell_params.py
+++ b/dev/breeze/src/airflow_breeze/params/shell_params.py
@@ -23,6 +23,7 @@ from pathlib import Path
from airflow_breeze.branch_defaults import AIRFLOW_BRANCH, DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
from airflow_breeze.global_constants import (
+ ALL_INTEGRATIONS,
ALLOWED_BACKENDS,
ALLOWED_CONSTRAINTS_MODES_CI,
ALLOWED_INSTALLATION_PACKAGE_FORMATS,
@@ -31,7 +32,6 @@ from airflow_breeze.global_constants import (
ALLOWED_POSTGRES_VERSIONS,
ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS,
APACHE_AIRFLOW_GITHUB_REPOSITORY,
- AVAILABLE_INTEGRATIONS,
DOCKER_DEFAULT_PLATFORM,
MOUNT_ALL,
MOUNT_REMOVE,
@@ -238,7 +238,7 @@ class ShellParams:
if self.include_mypy_volume:
compose_file_list.append(DOCKER_COMPOSE_DIR / "mypy.yml")
if "all" in self.integration:
- integrations = AVAILABLE_INTEGRATIONS
+ integrations = ALL_INTEGRATIONS
else:
integrations = self.integration
if len(integrations) > 0:
[airflow] 05/20: Update boring-cyborg.yml with more PR labels (#28955)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7945514559bbb481d47cbc2fc0d93a5aff5bb76a
Author: eladkal <45...@users.noreply.github.com>
AuthorDate: Mon Jan 16 00:02:05 2023 +0200
Update boring-cyborg.yml with more PR labels (#28955)
(cherry picked from commit 3d5e66b52f8b066b7297fd6c437d3461208d48f8)
---
.github/boring-cyborg.yml | 58 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)
diff --git a/.github/boring-cyborg.yml b/.github/boring-cyborg.yml
index a10d8a5bf0..2e6644551b 100644
--- a/.github/boring-cyborg.yml
+++ b/.github/boring-cyborg.yml
@@ -54,6 +54,64 @@ labelPRBasedOnFilePath:
- docs/apache-airflow-providers-snowflake/**/*
- tests/providers/snowflake/**/*
+ provider:DBs:
+ - airflow/providers/mysql/**/*
+ - docs/apache-airflow-providers-mysql/**/*
+ - tests/providers/mysql/**/*
+ - airflow/providers/postgres/**/*
+ - docs/apache-airflow-providers-postgres/**/*
+ - tests/providers/postgres/**/*
+ - airflow/providers/sqlite/**/*
+ - docs/apache-airflow-providers-sqlite/**/*
+ - tests/providers/sqlite/**/*
+ - airflow/providers/oracle/**/*
+ - docs/apache-airflow-providers-oracle/**/*
+ - tests/providers/oracle/**/*
+ - airflow/providers/influxdb/**/*
+ - docs/apache-airflow-providers-influxdb/**/*
+ - tests/providers/influxdb/**/*
+ - airflow/providers/microsoft/mssql/**/*
+ - docs/apache-airflow-providers-microsoft-mssql/**/*
+ - tests/providers/microsoft/mssql/**/*
+
+ provider:Protocols:
+ - airflow/providers/http/**/*
+ - docs/apache-airflow-providers-http/**/*
+ - tests/providers/http/**/*
+ - airflow/providers/ftp/**/*
+ - docs/apache-airflow-providers-ftp/**/*
+ - tests/providers/ftp/**/*
+ - airflow/providers/ssh/**/*
+ - docs/apache-airflow-providers-ssh/**/*
+ - tests/providers/ssh/**/*
+ - airflow/providers/jdbc/**/*
+ - docs/apache-airflow-providers-jdbc/**/*
+ - tests/providers/jdbc/**/*
+
+ provider:Docker:
+ - airflow/providers/docker/**/*
+ - docs/apache-airflow-providers-docker/**/*
+ - tests/providers/docker/**/*
+
+ provider:message-apps:
+ - airflow/providers/slack/**/*
+ - docs/apache-airflow-providers-slack/**/*
+ - tests/providers/slack/**/*
+ - airflow/providers/opsgenie/**/*
+ - docs/apache-airflow-providers-opsgenie/**/*
+ - tests/providers/opsgenie/**/*
+ - airflow/providers/telegram/**/*
+ - docs/apache-airflow-providers-telegram/**/*
+ - tests/providers/telegram/**/*
+
+ provider:trino/presto:
+ - airflow/providers/trino/**/*
+ - docs/apache-airflow-providers-trino/**/*
+ - tests/providers/trino/**/*
+ - airflow/providers/presto/**/*
+ - docs/apache-airflow-providers-presto/**/*
+ - tests/providers/presto/**/*
+
area:providers:
- airflow/providers/**/*
- docs/apache-airflow-providers-*/**/*
[airflow] 11/20: Fix `breeze static_checks` type arg (#29292)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4451aba667e9a6abf9e0790fec6bc0b4106ebcaf
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Wed Feb 1 18:54:38 2023 -0600
Fix `breeze static_checks` type arg (#29292)
pre-commit does not support specifying multiple hooks to run, so I've
removed support for multiple args in our static_checks command.
(cherry picked from commit d80b583db07197c8c3d0549a805e83ceaaf10d52)
---
dev/breeze/src/airflow_breeze/commands/developer_commands.py | 10 +++++-----
images/breeze/output-commands-hash.txt | 2 +-
images/breeze/output_static-checks.svg | 2 +-
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands.py b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
index 9db96edff7..3e2a54a700 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
@@ -373,9 +373,9 @@ def build_docs(
@click.option(
"-t",
"--type",
- help="Type(s) of the static checks to run (multiple can be added).",
+ "type_",
+ help="Type(s) of the static checks to run.",
type=BetterChoice(PRE_COMMIT_LIST),
- multiple=True,
)
@click.option("-a", "--all-files", help="Run checks on all files.", is_flag=True)
@click.option("-f", "--file", help="List of files to run the checks on.", type=click.Path(), multiple=True)
@@ -404,7 +404,7 @@ def static_checks(
show_diff_on_failure: bool,
last_commit: bool,
commit_ref: str,
- type: tuple[str],
+ type_: str,
file: Iterable[str],
precommit_args: tuple,
github_repository: str,
@@ -415,8 +415,8 @@ def static_checks(
if last_commit and commit_ref:
get_console().print("\n[error]You cannot specify both --last-commit and --commit-ref[/]\n")
sys.exit(1)
- for single_check in type:
- command_to_execute.append(single_check)
+ if type_:
+ command_to_execute.append(type_)
if all_files:
command_to_execute.append("--all-files")
if show_diff_on_failure:
diff --git a/images/breeze/output-commands-hash.txt b/images/breeze/output-commands-hash.txt
index 46163a381d..4d5ffeb5fd 100644
--- a/images/breeze/output-commands-hash.txt
+++ b/images/breeze/output-commands-hash.txt
@@ -55,7 +55,7 @@ setup:version:123b462a421884dc2320ffc5e54b2478
setup:f383b9236f6141f95276136ccd9217f5
shell:affbf6f7f469408d0af47f75c6a38f6c
start-airflow:109728919a0dd5c5ff5640ae86ba9e90
-static-checks:06708a5e0c50a6fc6cd18c2413431168
+static-checks:16ff0bbf7b8b2ca9ac7341b23b69b6f0
stop:e5aa686b4e53707ced4039d8414d5cd6
testing:docker-compose-tests:b86c044b24138af0659a05ed6331576c
testing:helm-tests:94a442e7f3f63b34c4831a84d165690a
diff --git a/images/breeze/output_static-checks.svg b/images/breeze/output_static-checks.svg
index 7934d746c9..968779aab9 100644
--- a/images/breeze/output_static-checks.svg
+++ b/images/breeze/output_static-checks.svg
@@ -219,7 +219,7 @@
</text><text class="breeze-static-checks-r2" x="12.2" y="93.2" textLength="219.6" clip-path="url(#breeze-static-checks-line-3)">Run static checks.</text><text class="breeze-static-checks-r2" x="1464" y="93.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-3)">
</text><text class="breeze-static-checks-r2" x="1464" y="117.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-4)">
</text><text class="breeze-static-checks-r5" x="0" y="142" textLength="24.4" clip-path="url(#breeze-static-checks-line-5)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="142" textLength="219.6" clip-path="url(#breeze-static-checks-line-5)"> Pre-commit flags </text><text class="breeze-static-checks-r5" x="244" y="142" textLength="1195.6" clip-path="url(#breeze-static-checks-line-5)">──────────────────────────────────────────────────────────────────────────────── [...]
-</text><text class="breeze-static-checks-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-6)">│</text><text class="breeze-static-checks-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-6)">-</text><text class="breeze-static-checks-r4" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-static-checks-line-6)">-type</text><text class="breeze-static-checks-r6" x="317.2" y="166.4" textLength="24.4" clip-path="url(#breeze- [...]
+</text><text class="breeze-static-checks-r5" x="0" y="166.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-6)">│</text><text class="breeze-static-checks-r4" x="24.4" y="166.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-6)">-</text><text class="breeze-static-checks-r4" x="36.6" y="166.4" textLength="61" clip-path="url(#breeze-static-checks-line-6)">-type</text><text class="breeze-static-checks-r6" x="317.2" y="166.4" textLength="24.4" clip-path="url(#breeze- [...]
</text><text class="breeze-static-checks-r5" x="0" y="190.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-7)">│</text><text class="breeze-static-checks-r7" x="366" y="190.8" textLength="1073.6" clip-path="url(#breeze-static-checks-line-7)">(all | black | blacken-docs | check-airflow-config-yaml-consistent |                    </text><text c [...]
</text><text class="breeze-static-checks-r5" x="0" y="215.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-8)">│</text><text class="breeze-static-checks-r7" x="366" y="215.2" textLength="1073.6" clip-path="url(#breeze-static-checks-line-8)">check-airflow-provider-compatibility | check-apache-license-rat |                       </text><text class= [...]
</text><text class="breeze-static-checks-r5" x="0" y="239.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-9)">│</text><text class="breeze-static-checks-r7" x="366" y="239.6" textLength="1073.6" clip-path="url(#breeze-static-checks-line-9)">check-base-operator-partial-arguments | check-base-operator-usage |                     </text><text class="breeze-st [...]
[airflow] 12/20: Upgrade PIP to 23.0 version (#29421)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3a3932440b2e7d86f2cf968b9eefeb410ea9d560
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Feb 8 10:51:26 2023 +0100
Upgrade PIP to 23.0 version (#29421)
(cherry picked from commit c310fb9255ba458b2842315f65f59758b76df9d5)
---
Dockerfile | 4 ++--
Dockerfile.ci | 4 ++--
IMAGES.rst | 2 +-
dev/breeze/src/airflow_breeze/global_constants.py | 2 +-
docs/docker-stack/build-arg-ref.rst | 2 +-
scripts/docker/common.sh | 2 +-
6 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index 6eb79e57f6..6c55e18664 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -48,7 +48,7 @@ ARG AIRFLOW_VERSION="2.4.3"
ARG PYTHON_BASE_IMAGE="python:3.7-slim-bullseye"
-ARG AIRFLOW_PIP_VERSION=22.3.1
+ARG AIRFLOW_PIP_VERSION=23.0
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
ARG AIRFLOW_IMAGE_README_URL="https://raw.githubusercontent.com/apache/airflow/main/docs/docker-stack/README.md"
@@ -408,7 +408,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="22.3.1"
+ export AIRFLOW_PIP_VERSION="23.0"
fi
fi
}
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 76afec0321..0855b8a6ca 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -368,7 +368,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="22.3.1"
+ export AIRFLOW_PIP_VERSION="23.0"
fi
fi
}
@@ -1139,7 +1139,7 @@ ARG AIRFLOW_CI_BUILD_EPOCH="4"
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
# By default in the image, we are installing all providers when installing from sources
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
-ARG AIRFLOW_PIP_VERSION=22.3.1
+ARG AIRFLOW_PIP_VERSION=23.0
# Setup PIP
# By default PIP install run without cache to make image smaller
ARG PIP_NO_CACHE_DIR="true"
diff --git a/IMAGES.rst b/IMAGES.rst
index f6b78aee0f..5df50eeb56 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -457,7 +457,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| ``ADDITIONAL_DEV_APT_ENV`` | | Additional env variables defined |
| | | when installing dev deps |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``AIRFLOW_PIP_VERSION`` | ``22.3.1`` | PIP version used. |
+| ``AIRFLOW_PIP_VERSION`` | ``23.0`` | PIP version used. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``PIP_PROGRESS_BAR`` | ``on`` | Progress bar for PIP installation |
+------------------------------------------+------------------------------------------+------------------------------------------+
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index 323efd1c19..895dbeca4d 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -70,7 +70,7 @@ ALLOWED_POSTGRES_VERSIONS = ["11", "12", "13", "14", "15"]
ALLOWED_MYSQL_VERSIONS = ["5.7", "8"]
ALLOWED_MSSQL_VERSIONS = ["2017-latest", "2019-latest"]
-PIP_VERSION = "22.3.1"
+PIP_VERSION = "23.0"
@lru_cache(maxsize=None)
diff --git a/docs/docker-stack/build-arg-ref.rst b/docs/docker-stack/build-arg-ref.rst
index e37e6f29b9..4453cf7bd7 100644
--- a/docs/docker-stack/build-arg-ref.rst
+++ b/docs/docker-stack/build-arg-ref.rst
@@ -45,7 +45,7 @@ Those are the most common arguments that you use when you want to build a custom
+------------------------------------------+------------------------------------------+---------------------------------------------+
| ``AIRFLOW_USER_HOME_DIR`` | ``/home/airflow`` | Home directory of the Airflow user. |
+------------------------------------------+------------------------------------------+---------------------------------------------+
-| ``AIRFLOW_PIP_VERSION`` | ``22.3.1`` | PIP version used. |
+| ``AIRFLOW_PIP_VERSION`` | ``23.0`` | PIP version used. |
+------------------------------------------+------------------------------------------+---------------------------------------------+
| ``ADDITIONAL_PIP_INSTALL_FLAGS`` | | additional ``pip`` flags passed to the |
| | | installation commands (except when |
diff --git a/scripts/docker/common.sh b/scripts/docker/common.sh
index fab02a3b4f..c6a9f593c8 100644
--- a/scripts/docker/common.sh
+++ b/scripts/docker/common.sh
@@ -42,7 +42,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="22.3.1"
+ export AIRFLOW_PIP_VERSION="23.0"
fi
fi
}
[airflow] 19/20: Limit importlib-metadata backwport to < 5.0.0 (#29924)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d0966cfc2c27c7953c297cc3254844eaebcfd92b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Mar 5 06:11:04 2023 +0100
Limit importlib-metadata backwport to < 5.0.0 (#29924)
The Importlib-metadata 5 breaks importing Celery < 5.3.0. Until
Celery 5.3.0 gets released (it is pre-release/beta now) we should
workaround the problem by limiting importlib-metadata to < 5.0.0.
(cherry picked from commit 4ea457a5ca694eab150e02a26c0b3ef62858fbca)
---
setup.cfg | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/setup.cfg b/setup.cfg
index e55108fb92..39a73305a8 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -101,7 +101,10 @@ install_requires =
graphviz>=0.12
gunicorn>=20.1.0
httpx
- importlib_metadata>=1.7;python_version<"3.9"
+ # Importlib-metadata 5 is breaking Celery import due to regression it introduced
+ # This was tracked and fixed in https://github.com/celery/celery/pull/7785 but it is not released yet
+ # We can remove the < 5.0.0 limitation hwne Celery 5.3.0 gets released and we bump celeryt o >= 5.3.0
+ importlib_metadata>=1.7,<5.0.0;python_version<"3.9"
importlib_resources>=5.2;python_version<"3.9"
itsdangerous>=2.0
jinja2>=3.0.0
[airflow] 18/20: Upgrade pip to version 23.0.1 (#29767)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b37ec74e02800cd91db7239a263479568a4cdf81
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Feb 25 19:14:57 2023 +0100
Upgrade pip to version 23.0.1 (#29767)
(cherry picked from commit 851cedf088a95347038c8a59e21361d276f7b9b6)
---
Dockerfile | 4 ++--
Dockerfile.ci | 4 ++--
IMAGES.rst | 2 +-
dev/breeze/src/airflow_breeze/global_constants.py | 2 +-
docs/docker-stack/build-arg-ref.rst | 2 +-
scripts/docker/common.sh | 2 +-
6 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index 6c55e18664..5906ee85ba 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -48,7 +48,7 @@ ARG AIRFLOW_VERSION="2.4.3"
ARG PYTHON_BASE_IMAGE="python:3.7-slim-bullseye"
-ARG AIRFLOW_PIP_VERSION=23.0
+ARG AIRFLOW_PIP_VERSION=23.0.1
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
ARG AIRFLOW_IMAGE_README_URL="https://raw.githubusercontent.com/apache/airflow/main/docs/docker-stack/README.md"
@@ -408,7 +408,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="23.0"
+ export AIRFLOW_PIP_VERSION="23.0.1"
fi
fi
}
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 0855b8a6ca..eea7f0e9b6 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -368,7 +368,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="23.0"
+ export AIRFLOW_PIP_VERSION="23.0.1"
fi
fi
}
@@ -1139,7 +1139,7 @@ ARG AIRFLOW_CI_BUILD_EPOCH="4"
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
# By default in the image, we are installing all providers when installing from sources
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
-ARG AIRFLOW_PIP_VERSION=23.0
+ARG AIRFLOW_PIP_VERSION=23.0.1
# Setup PIP
# By default PIP install run without cache to make image smaller
ARG PIP_NO_CACHE_DIR="true"
diff --git a/IMAGES.rst b/IMAGES.rst
index 5df50eeb56..88f8c9ca9a 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -457,7 +457,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| ``ADDITIONAL_DEV_APT_ENV`` | | Additional env variables defined |
| | | when installing dev deps |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``AIRFLOW_PIP_VERSION`` | ``23.0`` | PIP version used. |
+| ``AIRFLOW_PIP_VERSION`` | ``23.0.1`` | PIP version used. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``PIP_PROGRESS_BAR`` | ``on`` | Progress bar for PIP installation |
+------------------------------------------+------------------------------------------+------------------------------------------+
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index 895dbeca4d..cd83967d86 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -70,7 +70,7 @@ ALLOWED_POSTGRES_VERSIONS = ["11", "12", "13", "14", "15"]
ALLOWED_MYSQL_VERSIONS = ["5.7", "8"]
ALLOWED_MSSQL_VERSIONS = ["2017-latest", "2019-latest"]
-PIP_VERSION = "23.0"
+PIP_VERSION = "23.0.1"
@lru_cache(maxsize=None)
diff --git a/docs/docker-stack/build-arg-ref.rst b/docs/docker-stack/build-arg-ref.rst
index 4453cf7bd7..0c0bcec6cd 100644
--- a/docs/docker-stack/build-arg-ref.rst
+++ b/docs/docker-stack/build-arg-ref.rst
@@ -45,7 +45,7 @@ Those are the most common arguments that you use when you want to build a custom
+------------------------------------------+------------------------------------------+---------------------------------------------+
| ``AIRFLOW_USER_HOME_DIR`` | ``/home/airflow`` | Home directory of the Airflow user. |
+------------------------------------------+------------------------------------------+---------------------------------------------+
-| ``AIRFLOW_PIP_VERSION`` | ``23.0`` | PIP version used. |
+| ``AIRFLOW_PIP_VERSION`` | ``23.0.1`` | PIP version used. |
+------------------------------------------+------------------------------------------+---------------------------------------------+
| ``ADDITIONAL_PIP_INSTALL_FLAGS`` | | additional ``pip`` flags passed to the |
| | | installation commands (except when |
diff --git a/scripts/docker/common.sh b/scripts/docker/common.sh
index c6a9f593c8..a3b1fe6f35 100644
--- a/scripts/docker/common.sh
+++ b/scripts/docker/common.sh
@@ -42,7 +42,7 @@ function common::get_airflow_version_specification() {
function common::override_pip_version_if_needed() {
if [[ -n ${AIRFLOW_VERSION} ]]; then
if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
- export AIRFLOW_PIP_VERSION="23.0"
+ export AIRFLOW_PIP_VERSION="23.0.1"
fi
fi
}
[airflow] 06/20: Exclude dependedbot PRs from status testing report (#28954)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 626b1514d4b022b7e368df6479aa7b3641338248
Author: eladkal <45...@users.noreply.github.com>
AuthorDate: Mon Jan 16 01:13:09 2023 +0200
Exclude dependedbot PRs from status testing report (#28954)
(cherry picked from commit d5ac1b057f96d94062a92d968fe40f4371eb5da0)
---
dev/prepare_release_issue.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/dev/prepare_release_issue.py b/dev/prepare_release_issue.py
index 9811d6051e..3009a151a8 100755
--- a/dev/prepare_release_issue.py
+++ b/dev/prepare_release_issue.py
@@ -289,6 +289,9 @@ def generate_issue_content(
console.print(f"[red]The PR #{pr_number} could not be found[/]")
continue
+ if pr.user.login == "dependabot":
+ console.print(f"[yellow]Skipping PR #{pr_number} as it was created by dependabot[/]")
+ continue
# Ignore doc-only and skipped PRs
label_names = [label.name for label in pr.labels]
if "type:doc-only" in label_names or "changelog:skip" in label_names:
[airflow] 09/20: Fix asset compilation in start-airflow (#29185)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d56606ea97975794d9d7ef47d79ed4aada66c3bb
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Jan 26 23:40:56 2023 +0100
Fix asset compilation in start-airflow (#29185)
The recent change #29080 introduced missing black import in the
pre-commit that run the compilation. The compilation happened in
the background thread and it's ouptut was only visible in the
asset_compilation output file
This PR fixes the black import problem by removing unnecessary import,
but it will also stop start-airflow and exit with error as well as
surface the output of asset compilation to the console.
(cherry picked from commit c231f1174d67d05384832b75f6ca7c02f5fda73c)
---
Dockerfile.ci | 9 +++++++++
dev/breeze/src/airflow_breeze/utils/run_utils.py | 12 +++++++++---
scripts/ci/pre_commit/pre_commit_compile_www_assets.py | 1 -
scripts/docker/entrypoint_ci.sh | 9 +++++++++
4 files changed, 27 insertions(+), 4 deletions(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 369faab2ab..76afec0321 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -623,6 +623,15 @@ If it does not complete soon, you might want to stop it and remove file lock:
fi
done
fi
+ if [ -f "${AIRFLOW_SOURCES}/.build/www/asset_compile.out" ]; then
+ echo
+ echo "${COLOR_RED}The asset compilation failed. Exiting.${COLOR_RESET}"
+ echo
+ cat "${AIRFLOW_SOURCES}/.build/www/asset_compile.out"
+ rm "${AIRFLOW_SOURCES}/.build/www/asset_compile.out"
+ echo
+ exit 1
+ fi
}
if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
diff --git a/dev/breeze/src/airflow_breeze/utils/run_utils.py b/dev/breeze/src/airflow_breeze/utils/run_utils.py
index ac0f689814..0209caf19a 100644
--- a/dev/breeze/src/airflow_breeze/utils/run_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/run_utils.py
@@ -418,16 +418,22 @@ def _run_compile_internally(command_to_execute: list[str], dev: bool) -> RunComm
pass
try:
with SoftFileLock(WWW_ASSET_COMPILE_LOCK, timeout=5):
- with open(WWW_ASSET_OUT_FILE, "w") as f:
- return run_command(
+ with open(WWW_ASSET_OUT_FILE, "w") as output_file:
+ result = run_command(
command_to_execute,
check=False,
no_output_dump_on_exception=True,
text=True,
env=env,
stderr=subprocess.STDOUT,
- stdout=f,
+ stdout=output_file,
)
+ if result.returncode == 0:
+ try:
+ WWW_ASSET_OUT_FILE.unlink()
+ except FileNotFoundError:
+ pass
+ return result
except Timeout:
get_console().print("[error]Another asset compilation is running. Exiting[/]\n")
get_console().print("[warning]If you are sure there is no other compilation,[/]")
diff --git a/scripts/ci/pre_commit/pre_commit_compile_www_assets.py b/scripts/ci/pre_commit/pre_commit_compile_www_assets.py
index 4733a1460b..8c43f72fd2 100755
--- a/scripts/ci/pre_commit/pre_commit_compile_www_assets.py
+++ b/scripts/ci/pre_commit/pre_commit_compile_www_assets.py
@@ -24,7 +24,6 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.resolve())) # make sure common_precommit_utils is imported
from common_precommit_utils import get_directory_hash # isort: skip # noqa E402
-from common_precommit_black_utils import black_format # isort: skip # noqa E402
AIRFLOW_SOURCES_PATH = Path(__file__).parents[3].resolve()
WWW_HASH_FILE = AIRFLOW_SOURCES_PATH / ".build" / "www" / "hash.txt"
diff --git a/scripts/docker/entrypoint_ci.sh b/scripts/docker/entrypoint_ci.sh
index 9454626d6c..85bf24e009 100755
--- a/scripts/docker/entrypoint_ci.sh
+++ b/scripts/docker/entrypoint_ci.sh
@@ -72,6 +72,15 @@ If it does not complete soon, you might want to stop it and remove file lock:
fi
done
fi
+ if [ -f "${AIRFLOW_SOURCES}/.build/www/asset_compile.out" ]; then
+ echo
+ echo "${COLOR_RED}The asset compilation failed. Exiting.${COLOR_RESET}"
+ echo
+ cat "${AIRFLOW_SOURCES}/.build/www/asset_compile.out"
+ rm "${AIRFLOW_SOURCES}/.build/www/asset_compile.out"
+ echo
+ exit 1
+ fi
}
if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
[airflow] 10/20: Fix commit message for constraints (#29215)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d2840e97cf0e2839856329152369f8368b628a1d
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Jan 29 14:18:35 2023 +0100
Fix commit message for constraints (#29215)
A long time ago in a galaxy far away ...
We had a code that make our CI variables independent from the
CI Actions used and we used CI_* prefixes to distinguish those
values. This reflected the time when we moved from Travis to
GitHub Actions and we wanted to keep the possibility to support
both. Those variables have been since removed, but there was
that one single planet (err. script) where the remnant of the old
Jedi order (err. variables) remained and they caused rather
meaningless commit messages for constraints.
This PR updates the commit messages to be more meaningful,
including direct links to the action that updated the constraints
and helpful description on how you can easily run breeze with the
exact set of dependencies that were used for it.
(cherry picked from commit c7aa0bf91ce2624db3f3075d7e2a3bf4927dca15)
---
scripts/ci/constraints/ci_commit_constraints.sh | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/constraints/ci_commit_constraints.sh b/scripts/ci/constraints/ci_commit_constraints.sh
index 3d4485db58..1170b0fdcf 100755
--- a/scripts/ci/constraints/ci_commit_constraints.sh
+++ b/scripts/ci/constraints/ci_commit_constraints.sh
@@ -23,8 +23,12 @@ git diff --color --exit-code --ignore-matching-lines="^#.*" || \
git commit --all --message "Updating constraints. Github run id:${GITHUB_RUN_ID}
This update in constraints is automatically committed by the CI 'constraints-push' step based on
-HEAD of '${CI_REF}' in '${CI_TARGET_REPO}'
-with commit sha ${COMMIT_SHA}.
+'${GITHUB_REF}' in the '${GITHUB_REPOSITORY}' repository with commit sha ${GITHUB_SHA}.
+
+The action that build those constraints can be found at https://github.com/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/
+
+The image tag used for that build was: ${IMAGE_TAG}. You can enter Breeze environment
+with this image by running 'breeze shell --image-tag ${IMAGE_TAG}'
All tests passed in this build so we determined we can push the updated constraints.
[airflow] 16/20: Do not crash when a version fails to parse (#29685)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 99b1cc6707005f9639a22ade23ad4312954a8be2
Author: Tzu-ping Chung <ur...@gmail.com>
AuthorDate: Wed Feb 22 20:13:08 2023 +0800
Do not crash when a version fails to parse (#29685)
Old files on PyPI may contain version numbers that are non-standard and
can't be properly parsed. Those are no longer allowed for new versions,
so we can safely ignore those versions since they must be ancient.
(cherry picked from commit 8430d6014aef801ddea3c7f2eb56b76a8d291b8c)
---
dev/breeze/src/airflow_breeze/utils/find_newer_dependencies.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/dev/breeze/src/airflow_breeze/utils/find_newer_dependencies.py b/dev/breeze/src/airflow_breeze/utils/find_newer_dependencies.py
index ecedb8b35f..fb4079548f 100644
--- a/dev/breeze/src/airflow_breeze/utils/find_newer_dependencies.py
+++ b/dev/breeze/src/airflow_breeze/utils/find_newer_dependencies.py
@@ -109,7 +109,10 @@ def get_releases_and_upload_times(package, min_date, current_version, tz) -> lis
releases: list[tuple[Any, Any]] = []
for release_version, release_info in package_info["releases"].items():
if release_info and not release_info[0]["yanked"]:
- parsed_version = version.parse(release_version)
+ try:
+ parsed_version = version.parse(release_version)
+ except version.InvalidVersion:
+ continue
if (
parsed_version.is_prerelease
or parsed_version.is_devrelease
[airflow] 17/20: Propose to upgrade to newer dependencies in case build CI image fail (#29705)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ffdb121523ac24ea108dde9c00645cc75dc22768
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Feb 23 03:00:05 2023 +0100
Propose to upgrade to newer dependencies in case build CI image fail (#29705)
* Propose to upgrade to newer dependencies in case build CI image fail
When build image fails it might be because there might be conflicts
with current constraints. In case this happens we propose to add
`--upgrade-to-newer-dependencies` flag.
* Update dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
Co-authored-by: Raphaël Vandon <11...@users.noreply.github.com>
* Update dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
Co-authored-by: Raphaël Vandon <11...@users.noreply.github.com>
---------
Co-authored-by: Raphaël Vandon <11...@users.noreply.github.com>
(cherry picked from commit f03885421e1ec39280d9dd94cbed73151f6f19ba)
---
.../airflow_breeze/commands/ci_image_commands.py | 41 ++++++++++++----------
1 file changed, 23 insertions(+), 18 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
index b8f996dde6..6b72895952 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
@@ -504,24 +504,29 @@ def run_build_ci_image(
check=False,
output=output,
)
- if (
- build_command_result.returncode != 0
- and ci_image_params.upgrade_on_failure
- and not ci_image_params.upgrade_to_newer_dependencies
- ):
- ci_image_params.upgrade_to_newer_dependencies = True
- get_console().print(
- "[warning]Attempting to build with upgrade_to_newer_dependencies on failure"
- )
- build_command_result = run_command(
- prepare_docker_build_command(
- image_params=ci_image_params,
- ),
- cwd=AIRFLOW_SOURCES_ROOT,
- text=True,
- check=False,
- output=output,
- )
+ if build_command_result.returncode != 0 and not ci_image_params.upgrade_to_newer_dependencies:
+ if ci_image_params.upgrade_on_failure:
+ ci_image_params.upgrade_to_newer_dependencies = True
+ get_console().print(
+ "[warning]Attempting to build with upgrade_to_newer_dependencies on failure"
+ )
+ build_command_result = run_command(
+ prepare_docker_build_command(
+ image_params=ci_image_params,
+ ),
+ cwd=AIRFLOW_SOURCES_ROOT,
+ text=True,
+ check=False,
+ output=output,
+ )
+ else:
+ get_console().print(
+ "[warning]Your image build failed. It could be caused by conflicting dependencies."
+ )
+ get_console().print(
+ "[info]Run "
+ "`breeze ci-image build --upgrade-to-newer-dependencies` to upgrade them.\n"
+ )
if build_command_result.returncode == 0:
if ci_image_params.tag_as_latest:
build_command_result = tag_image_as_latest(image_params=ci_image_params, output=output)
[airflow] 03/20: Removal of flake8 file has broken breeze bind mounts (#28909)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a0045495a4d5b0e41df7434429513993b53e37d6
Author: Niko Oliveira <on...@amazon.com>
AuthorDate: Thu Jan 12 23:53:11 2023 -0800
Removal of flake8 file has broken breeze bind mounts (#28909)
Don't try mount .flake8 anymore, it has been removed in #28893
(cherry picked from commit 118d6505a523ef8370e193ad5053d61cdf3cb6b0)
---
dev/breeze/src/airflow_breeze/utils/docker_command_utils.py | 1 -
scripts/ci/docker-compose/local.yml | 3 ---
2 files changed, 4 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
index b990e44169..fe2b6238fd 100644
--- a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
@@ -77,7 +77,6 @@ VOLUMES_FOR_SELECTED_MOUNTS = [
(".build", "/opt/airflow/.build"),
(".coveragerc", "/opt/airflow/.coveragerc"),
(".dockerignore", "/opt/airflow/.dockerignore"),
- (".flake8", "/opt/airflow/.flake8"),
(".github", "/opt/airflow/.github"),
(".inputrc", "/root/.inputrc"),
(".rat-excludes", "/opt/airflow/.rat-excludes"),
diff --git a/scripts/ci/docker-compose/local.yml b/scripts/ci/docker-compose/local.yml
index 45a18acceb..0d80695e75 100644
--- a/scripts/ci/docker-compose/local.yml
+++ b/scripts/ci/docker-compose/local.yml
@@ -42,9 +42,6 @@ services:
- type: bind
source: ../../../.dockerignore
target: /opt/airflow/.dockerignore
- - type: bind
- source: ../../../.flake8
- target: /opt/airflow/.flake8
- type: bind
source: ../../../.github
target: /opt/airflow/.github
[airflow] 07/20: Fix hot loop with a burst of log messages (#28778) (#28780)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 04d1c17efc64497e69499910b2808bd1a452ab4e
Author: Ivan Reche <iv...@gmail.com>
AuthorDate: Mon Jan 16 14:05:34 2023 -0300
Fix hot loop with a burst of log messages (#28778) (#28780)
* Fix hot loop with a burst of log messages (#28778)
* Add missing inline in Dockerfile
(cherry picked from commit 4b1a36f833b77d3f0bec78958d1fb9f360b7b11b)
---
Dockerfile | 3 ++-
scripts/docker/clean-logs.sh | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index 3da82f2d2b..6eb79e57f6 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1025,7 +1025,8 @@ while true; do
xargs -0 rm -f
seconds=$(( $(date -u +%s) % EVERY))
- (( seconds < 1 )) || sleep $((EVERY - seconds))
+ (( seconds < 1 )) || sleep $((EVERY - seconds - 1))
+ sleep 1
done
EOF
diff --git a/scripts/docker/clean-logs.sh b/scripts/docker/clean-logs.sh
index 57b6e8b605..0c775b14e6 100644
--- a/scripts/docker/clean-logs.sh
+++ b/scripts/docker/clean-logs.sh
@@ -36,5 +36,6 @@ while true; do
xargs -0 rm -f
seconds=$(( $(date -u +%s) % EVERY))
- (( seconds < 1 )) || sleep $((EVERY - seconds))
+ (( seconds < 1 )) || sleep $((EVERY - seconds - 1))
+ sleep 1
done
[airflow] 15/20: Migrate breeze unit tests to pytest. (#29639)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 458c41fda6c065496dbc096353dc32c2ef9ee98b
Author: Vedant Lodha <ve...@hotmail.com>
AuthorDate: Mon Feb 20 21:50:46 2023 +0530
Migrate breeze unit tests to pytest. (#29639)
This change removes unittest.Testcase dependency for breeze tests which was just being used for an assert statement.
fixes #29305
(cherry picked from commit d721701e14debe7e8a411680aba821ed800ccb36)
---
dev/breeze/tests/test_run_utils.py | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/dev/breeze/tests/test_run_utils.py b/dev/breeze/tests/test_run_utils.py
index 78187f974a..2ff2fb8810 100644
--- a/dev/breeze/tests/test_run_utils.py
+++ b/dev/breeze/tests/test_run_utils.py
@@ -19,7 +19,6 @@ from __future__ import annotations
import os
import stat
from pathlib import Path
-from unittest import TestCase
from airflow_breeze.utils.run_utils import (
change_directory_permission,
@@ -51,4 +50,4 @@ def test_filter_out_none():
dict_input_with_none = {"sample": None, "sample1": "One", "sample2": "Two", "samplen": None}
expected_dict_output = {"sample1": "One", "sample2": "Two"}
output_dict = filter_out_none(**dict_input_with_none)
- TestCase().assertDictEqual(output_dict, expected_dict_output)
+ assert output_dict == expected_dict_output
[airflow] 02/20: Fix pyzmq warning stopping provider package tests (#28901)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6180b2d01c2a20c2ae38fbe424862b86d1ce7dd0
Author: Ash Berlin-Taylor <as...@apache.org>
AuthorDate: Thu Jan 12 21:48:14 2023 +0000
Fix pyzmq warning stopping provider package tests (#28901)
(cherry picked from commit a1f23fea12b5e49c5fc86a1402758da5111e9383)
---
scripts/in_container/verify_providers.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/scripts/in_container/verify_providers.py b/scripts/in_container/verify_providers.py
index d46e32caa3..2af7f63e0b 100755
--- a/scripts/in_container/verify_providers.py
+++ b/scripts/in_container/verify_providers.py
@@ -221,6 +221,11 @@ KNOWN_DEPRECATED_MESSAGES: set[tuple[str, str]] = {
"urllib3 2.x. Read more in this issue: https://github.com/urllib3/urllib3/issues/2680",
"requests_toolbelt",
),
+ (
+ "zmq.eventloop.ioloop is deprecated in pyzmq 17. pyzmq now works with default tornado and asyncio "
+ "eventloops.",
+ "jupyter_client",
+ ),
}
KNOWN_COMMON_DEPRECATED_MESSAGES: set[str] = {
[airflow] 13/20: Bump stale bot version (#29491)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6a240c3b261cb3250d869ec920080c432bf4ea54
Author: eladkal <45...@users.noreply.github.com>
AuthorDate: Sun Feb 12 22:25:51 2023 +0200
Bump stale bot version (#29491)
(cherry picked from commit ca7161b2ecab8e7851a823ab22354934caf89bfd)
---
.github/workflows/stale.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml
index e9f8943513..08c40b978d 100644
--- a/.github/workflows/stale.yml
+++ b/.github/workflows/stale.yml
@@ -29,7 +29,7 @@ jobs:
stale:
runs-on: ubuntu-20.04
steps:
- - uses: actions/stale@v5
+ - uses: actions/stale@v7
with:
stale-pr-message: >
This pull request has been automatically marked as stale because it has not had
[airflow] 01/20: Fix bugs introduced when moving the code of issue generation to breeze (#28683)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 81ca55c1e24970d9108c9a0c3d835a8e92b3549f
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jan 2 19:59:23 2023 +0100
Fix bugs introduced when moving the code of issue generation to breeze (#28683)
The #28352 introduced subtle bugs which failed when runnint it during
provider generation.
Also added running the command automatically in CI.
(cherry picked from commit 69ab7d8252f830d8c1a013d34f8305a16da26bcf)
---
.github/workflows/ci.yml | 4 ++
.../commands/release_management_commands.py | 8 +--
images/breeze/output-commands-hash.txt | 4 +-
...t_release-management_generate-issue-content.svg | 60 +++++++++++-----------
4 files changed, 41 insertions(+), 35 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 589582ae8a..6e2056ff7f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -736,6 +736,10 @@ jobs:
- name: "Verify sdist packages with twine"
run: pipx install twine && twine check dist/*.tar.gz
if: matrix.package-format == 'sdist'
+ - name: "Test issue generation automatically"
+ run: >
+ breeze release-management generate-issue-content --only-available-in-dist --disable-progress
+ if: matrix.package-format == 'wheel'
- name: "Install and test provider packages and airflow via ${{matrix.package-format}} files"
run: breeze release-management verify-provider-packages --use-packages-from-dist
env:
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index 287ab93a43..612d07a32e 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -781,6 +781,7 @@ def get_prs_for_package(package_id: str) -> list[int]:
help="Only consider package ids with packages prepared in the dist folder",
)
@click.option("--excluded-pr-list", type=str, help="Coma-separated list of PRs to exclude from the issue.")
+@click.option("--disable-progress", is_flag=True, help="Disable progress bar")
@argument_packages
def generate_issue_content(
packages: list[str],
@@ -788,6 +789,7 @@ def generate_issue_content(
suffix: str,
only_available_in_dist: bool,
excluded_pr_list: str,
+ disable_progress: bool,
):
import jinja2
import yaml
@@ -812,14 +814,14 @@ def generate_issue_content(
all_prs: set[int] = set()
provider_prs: dict[str, list[int]] = {}
if only_available_in_dist:
- files_in_dist = os.listdir(str(APACHE_AIRFLOW_GITHUB_REPOSITORY / "dist"))
+ files_in_dist = os.listdir(str(AIRFLOW_SOURCES_ROOT / "dist"))
prepared_package_ids = []
for package_id in packages:
if not only_available_in_dist or is_package_in_dist(files_in_dist, package_id):
get_console().print(f"Extracting PRs for provider {package_id}")
prepared_package_ids.append(package_id)
else:
- get_console.print(
+ get_console().print(
f"Skipping extracting PRs for provider {package_id} as it is missing in dist"
)
continue
@@ -829,7 +831,7 @@ def generate_issue_content(
g = Github(github_token)
repo = g.get_repo("apache/airflow")
pull_requests: dict[int, PullRequest.PullRequest | Issue.Issue] = {}
- with Progress(console=get_console()) as progress:
+ with Progress(console=get_console(), disable=disable_progress) as progress:
task = progress.add_task(f"Retrieving {len(all_prs)} PRs ", total=len(all_prs))
pr_list = list(all_prs)
for i in range(len(pr_list)):
diff --git a/images/breeze/output-commands-hash.txt b/images/breeze/output-commands-hash.txt
index d159616b63..46163a381d 100644
--- a/images/breeze/output-commands-hash.txt
+++ b/images/breeze/output-commands-hash.txt
@@ -37,7 +37,7 @@ prod-image:verify:31bc5efada1d70a0a31990025db1a093
prod-image:a4013428dc7f71a1defc3778d2efe3dc
release-management:create-minor-branch:6a01066dce15e09fb269a8385626657c
release-management:generate-constraints:ae30d6ad49a1b2c15b61cb29080fd957
-release-management:generate-issue-content:24218438f9e85e7c92258aadebbb19de
+release-management:generate-issue-content:afea5d82089751fabfa8bd1bfa36ea21
release-management:prepare-airflow-package:3ac14ea6d2b09614959c0ec4fd564789
release-management:prepare-provider-documentation:3fe5ead9887c518d1b397d1103dc0025
release-management:prepare-provider-packages:40144cb01afc56f6a4f92d9e117e546e
@@ -45,7 +45,7 @@ release-management:release-prod-images:c9bc40938e0efad49e51ef66e83f9527
release-management:start-rc-process:6aafbaceabd7b67b9a1af4c2f59abc4c
release-management:start-release:acb384d86e02ff5fde1bf971897be17c
release-management:verify-provider-packages:8d3c6362657d76bb3cd064fed5596e84
-release-management:5505ed89019e969cf1e7efb309dee46e
+release-management:eecf81579915b400e13193d1a14ec455
setup:autocomplete:03343478bf1d0cf9c101d454cdb63b68
setup:check-all-params-in-groups:5b3dc1f6e630510c8f52c3d0687b3b2a
setup:config:3ffcd35dd24b486ddf1d08b797e3d017
diff --git a/images/breeze/output_release-management_generate-issue-content.svg b/images/breeze/output_release-management_generate-issue-content.svg
index 0c38887451..c83998c8eb 100644
--- a/images/breeze/output_release-management_generate-issue-content.svg
+++ b/images/breeze/output_release-management_generate-issue-content.svg
@@ -35,8 +35,8 @@
.breeze-release-management-generate-issue-content-r1 { fill: #c5c8c6;font-weight: bold }
.breeze-release-management-generate-issue-content-r2 { fill: #c5c8c6 }
.breeze-release-management-generate-issue-content-r3 { fill: #d0b344;font-weight: bold }
-.breeze-release-management-generate-issue-content-r4 { fill: #868887 }
-.breeze-release-management-generate-issue-content-r5 { fill: #68a0b3;font-weight: bold }
+.breeze-release-management-generate-issue-content-r4 { fill: #68a0b3;font-weight: bold }
+.breeze-release-management-generate-issue-content-r5 { fill: #868887 }
.breeze-release-management-generate-issue-content-r6 { fill: #8d7b39 }
.breeze-release-management-generate-issue-content-r7 { fill: #98a84b;font-weight: bold }
</style>
@@ -151,37 +151,37 @@
<g class="breeze-release-management-generate-issue-content-matrix">
<text class="breeze-release-management-generate-issue-content-r2" x="1464" y="20" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-0)">
-</text><text class="breeze-release-management-generate-issue-content-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-release-management-generate-issue-content-line-1)">Usage: </text><text class="breeze-release-management-generate-issue-content-r1" x="97.6" y="44.4" textLength="1244.4" clip-path="url(#breeze-release-management-generate-issue-content-line-1)">breeze release-management generate-issue-content [OPTIONS] [airbyte | alibaba  [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="68.8" textLength="1427.4" clip-path="url(#breeze-release-management-generate-issue-content-line-2)">                                                   & [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="93.2" textLength="1415.2" clip-path="url(#breeze-release-management-generate-issue-content-line-3)">                                                   & [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="117.6" textLength="1366.4" clip-path="url(#breeze-release-management-generate-issue-content-line-4)">                                                    [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="142" textLength="1403" clip-path="url(#breeze-release-management-generate-issue-content-line-5)">                                                    [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="166.4" textLength="1439.6" clip-path="url(#breeze-release-management-generate-issue-content-line-6)">                                                    [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="190.8" textLength="1403" clip-path="url(#breeze-release-management-generate-issue-content-line-7)">                                                   &# [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="215.2" textLength="1439.6" clip-path="url(#breeze-release-management-generate-issue-content-line-8)">                                                    [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="239.6" textLength="1403" clip-path="url(#breeze-release-management-generate-issue-content-line-9)">                                                   &# [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="264" textLength="1415.2" clip-path="url(#breeze-release-management-generate-issue-content-line-10)">                                                   & [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="288.4" textLength="1329.8" clip-path="url(#breeze-release-management-generate-issue-content-line-11)">                                                    [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="312.8" textLength="1427.4" clip-path="url(#breeze-release-management-generate-issue-content-line-12)">                                                    [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="337.2" textLength="1366.4" clip-path="url(#breeze-release-management-generate-issue-content-line-13)">                                                    [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="361.6" textLength="1390.8" clip-path="url(#breeze-release-management-generate-issue-content-line-14)">                                                    [...]
-</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="386" textLength="1403" clip-path="url(#breeze-release-management-generate-issue-content-line-15)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r3" x="12.2" y="44.4" textLength="85.4" clip-path="url(#breeze-release-management-generate-issue-content-line-1)">Usage: </text><text class="breeze-release-management-generate-issue-content-r1" x="97.6" y="44.4" textLength="610" clip-path="url(#breeze-release-management-generate-issue-content-line-1)">breeze release-management generate-issue-content [</text><text class="breeze-release-management-gene [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="68.8" textLength="1439.6" clip-path="url(#breeze-release-management-generate-issue-content-line-2)">                                                   & [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="93.2" textLength="1354.2" clip-path="url(#breeze-release-management-generate-issue-content-line-3)">                                                   & [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="117.6" textLength="1427.4" clip-path="url(#breeze-release-management-generate-issue-content-line-4)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="142" textLength="1366.4" clip-path="url(#breeze-release-management-generate-issue-content-line-5)">                                                   &# [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="166.4" textLength="1390.8" clip-path="url(#breeze-release-management-generate-issue-content-line-6)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="190.8" textLength="1415.2" clip-path="url(#breeze-release-management-generate-issue-content-line-7)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="215.2" textLength="1427.4" clip-path="url(#breeze-release-management-generate-issue-content-line-8)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="239.6" textLength="1317.6" clip-path="url(#breeze-release-management-generate-issue-content-line-9)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="264" textLength="1390.8" clip-path="url(#breeze-release-management-generate-issue-content-line-10)">                                                   & [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="288.4" textLength="1427.4" clip-path="url(#breeze-release-management-generate-issue-content-line-11)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="312.8" textLength="1390.8" clip-path="url(#breeze-release-management-generate-issue-content-line-12)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="337.2" textLength="1378.6" clip-path="url(#breeze-release-management-generate-issue-content-line-13)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="361.6" textLength="1378.6" clip-path="url(#breeze-release-management-generate-issue-content-line-14)">                                                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r1" x="12.2" y="386" textLength="1146.8" clip-path="url(#breeze-release-management-generate-issue-content-line-15)">                                                   & [...]
</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="410.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-16)">
</text><text class="breeze-release-management-generate-issue-content-r2" x="12.2" y="434.8" textLength="585.6" clip-path="url(#breeze-release-management-generate-issue-content-line-17)">Generates content for issue to test the release.</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="434.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-17)">
</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="459.2" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-18)">
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="483.6" textLength="24.4" clip-path="url(#breeze-release-management-generate-issue-content-line-19)">╭─</text><text class="breeze-release-management-generate-issue-content-r4" x="24.4" y="483.6" textLength="366" clip-path="url(#breeze-release-management-generate-issue-content-line-19)"> Generate issue content flags </text><text class="breeze-release-management-generate-issue-content- [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="508" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-20)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-20)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="508" textLength="85.4" clip-path="url(#bre [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-21)">│</text><text class="breeze-release-management-generate-issue-content-r2" x="366" y="532.4" textLength="1073.6" clip-path="url(#breeze-release-management-generate-issue-content-line-21)">variable set. Can be generated with:          &# [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-22)">│</text><text class="breeze-release-management-generate-issue-content-r2" x="366" y="556.8" textLength="1073.6" clip-path="url(#breeze-release-management-generate-issue-content-line-22)">https://github.com/settings/tokens/new?description=Read%20sssues&scopes=repo:status     </text [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-23)">│</text><text class="breeze-release-management-generate-issue-content-r6" x="366" y="581.2" textLength="1073.6" clip-path="url(#breeze-release-management-generate-issue-content-line-23)">(TEXT)                    [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-24)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="605.6" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-24)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="605.6" textLength="85.4" clip-path="ur [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="630" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-25)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-25)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="630" textLength="61" clip-path="url(#breez [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-26)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-26)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="654.4" textLength="109.8" clip-path="u [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-27)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-27)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="678.8" textLength="97.6" clip-path="ur [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="703.2" textLength="1464" clip-path="url(#breeze-release-management-generate-issue-content-line-28)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="703.2" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-28)">
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="727.6" textLength="24.4" clip-path="url(#breeze-release-management-generate-issue-content-line-29)">╭─</text><text class="breeze-release-management-generate-issue-content-r4" x="24.4" y="727.6" textLength="195.2" clip-path="url(#breeze-release-management-generate-issue-content-line-29)"> Common options </text><text class="breeze-release-management-generate-issue-content-r4" x="219.6" y="727.6 [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="752" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-30)">│</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-30)">-</text><text class="breeze-release-management-generate-issue-content-r5" x="36.6" y="752" textLength="61" clip-path="url(#breez [...]
-</text><text class="breeze-release-management-generate-issue-content-r4" x="0" y="776.4" textLength="1464" clip-path="url(#breeze-release-management-generate-issue-content-line-31)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="776.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-31)">
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="483.6" textLength="24.4" clip-path="url(#breeze-release-management-generate-issue-content-line-19)">╭─</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="483.6" textLength="366" clip-path="url(#breeze-release-management-generate-issue-content-line-19)"> Generate issue content flags </text><text class="breeze-release-management-generate-issue-content- [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-20)">│</text><text class="breeze-release-management-generate-issue-content-r4" x="24.4" y="508" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-20)">-</text><text class="breeze-release-management-generate-issue-content-r4" x="36.6" y="508" textLength="85.4" clip-path="url(#bre [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-21)">│</text><text class="breeze-release-management-generate-issue-content-r2" x="366" y="532.4" textLength="1073.6" clip-path="url(#breeze-release-management-generate-issue-content-line-21)">variable set. Can be generated with:          &# [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-22)">│</text><text class="breeze-release-management-generate-issue-content-r2" x="366" y="556.8" textLength="1073.6" clip-path="url(#breeze-release-management-generate-issue-content-line-22)">https://github.com/settings/tokens/new?description=Read%20sssues&scopes=repo:status     </text [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-23)">│</text><text class="breeze-release-management-generate-issue-content-r6" x="366" y="581.2" textLength="1073.6" clip-path="url(#breeze-release-management-generate-issue-content-line-23)">(TEXT)                    [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-24)">│</text><text class="breeze-release-management-generate-issue-content-r4" x="24.4" y="605.6" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-24)">-</text><text class="breeze-release-management-generate-issue-content-r4" x="36.6" y="605.6" textLength="85.4" clip-path="ur [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-25)">│</text><text class="breeze-release-management-generate-issue-content-r4" x="24.4" y="630" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-25)">-</text><text class="breeze-release-management-generate-issue-content-r4" x="36.6" y="630" textLength="61" clip-path="url(#breez [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-26)">│</text><text class="breeze-release-management-generate-issue-content-r4" x="24.4" y="654.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-26)">-</text><text class="breeze-release-management-generate-issue-content-r4" x="36.6" y="654.4" textLength="109.8" clip-path="u [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-27)">│</text><text class="breeze-release-management-generate-issue-content-r4" x="24.4" y="678.8" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-27)">-</text><text class="breeze-release-management-generate-issue-content-r4" x="36.6" y="678.8" textLength="97.6" clip-path="ur [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="703.2" textLength="1464" clip-path="url(#breeze-release-management-generate-issue-content-line-28)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="703.2" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-28)">
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="727.6" textLength="24.4" clip-path="url(#breeze-release-management-generate-issue-content-line-29)">╭─</text><text class="breeze-release-management-generate-issue-content-r5" x="24.4" y="727.6" textLength="195.2" clip-path="url(#breeze-release-management-generate-issue-content-line-29)"> Common options </text><text class="breeze-release-management-generate-issue-content-r5" x="219.6" y="727.6 [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-30)">│</text><text class="breeze-release-management-generate-issue-content-r4" x="24.4" y="752" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-30)">-</text><text class="breeze-release-management-generate-issue-content-r4" x="36.6" y="752" textLength="61" clip-path="url(#breez [...]
+</text><text class="breeze-release-management-generate-issue-content-r5" x="0" y="776.4" textLength="1464" clip-path="url(#breeze-release-management-generate-issue-content-line-31)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-release-management-generate-issue-content-r2" x="1464" y="776.4" textLength="12.2" clip-path="url(#breeze-release-management-generate-issue-content-line-31)">
</text>
</g>
</g>
[airflow] 04/20: Fix a few remaining references to flake8 (#28915)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2b377377e4e2ae08885d206f1b9f4aa0b3958f4a
Author: Ash Berlin-Taylor <as...@apache.org>
AuthorDate: Fri Jan 13 10:09:19 2023 +0000
Fix a few remaining references to flake8 (#28915)
Some are just docs, others stopped some `breeze` commands from working
(cherry picked from commit 71340fcc4d393484bc47ac4ca73e37b06062f978)
---
.dockerignore | 1 -
.gitattributes | 1 -
.github/boring-cyborg.yml | 2 +-
CONTRIBUTING.rst | 2 +-
CONTRIBUTORS_QUICK_START.rst | 4 ++--
STATIC_CODE_CHECKS.rst | 18 +++++++++---------
6 files changed, 13 insertions(+), 15 deletions(-)
diff --git a/.dockerignore b/.dockerignore
index 045b730630..bf1cd0f3b5 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -45,7 +45,6 @@
!.coveragerc
!.rat-excludes
-!.flake8
!.dockerignore
!pytest.ini
!RELEASE_NOTES.rst
diff --git a/.gitattributes b/.gitattributes
index 4ee74a9d8a..583924fead 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -30,7 +30,6 @@ CONTRIBUTORS_QUICK_START.rst export-ignore
.pre-commit-config.yaml export-ignore
.mailmap export-ignore
.editorconfig export-ignore
-.flake8 export-ignore
.inputrc export-ignore
.codespellignorelines export-ignore
.gitmodules export-ignore
diff --git a/.github/boring-cyborg.yml b/.github/boring-cyborg.yml
index de8a944d70..a10d8a5bf0 100644
--- a/.github/boring-cyborg.yml
+++ b/.github/boring-cyborg.yml
@@ -213,7 +213,7 @@ firstPRWelcomeComment: >
Here are some useful points:
- - Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits](
+ - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits](
https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
will help you with that.
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index a5e00305c1..3a6b7eb29a 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -892,7 +892,7 @@ as described in the static code checks documentation.
Coding style and best practices
===============================
-Most of our coding style rules are enforced programmatically by flake8 and mypy (which are run automatically
+Most of our coding style rules are enforced programmatically by ruff and mypy (which are run automatically
on every pull request), but there are some rules that are not yet automated and are more Airflow specific or
semantic than style
diff --git a/CONTRIBUTORS_QUICK_START.rst b/CONTRIBUTORS_QUICK_START.rst
index bffd4967c8..da37b03da8 100644
--- a/CONTRIBUTORS_QUICK_START.rst
+++ b/CONTRIBUTORS_QUICK_START.rst
@@ -532,8 +532,8 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r
$ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
black...............................................................Passed
- $ pre-commit run run-flake8 --files airflow/decorators.py tests/utils/test_task_group.py
- Run flake8..........................................................Passed
+ $ pre-commit run ruff --files airflow/decorators.py tests/utils/test_task_group.py
+ Run ruff............................................................Passed
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index a31a29171e..6d1684acd9 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -119,8 +119,8 @@ require Breeze Docker image to be build locally.
In case you have a problem with running particular ``pre-commit`` check you can still continue using the
benefits of having ``pre-commit`` installed, with some of the checks disabled. In order to disable
checks you might need to set ``SKIP`` environment variable to coma-separated list of checks to skip. For example
- when you want to skip some checks (flake/mypy for example), you should be able to do it by setting
- ``export SKIP=run-flake8,run-mypy``. You can also add this to your ``.bashrc`` or ``.zshrc`` if you
+ when you want to skip some checks (ruff/mypy for example), you should be able to do it by setting
+ ``export SKIP=ruff,run-mypy``. You can also add this to your ``.bashrc`` or ``.zshrc`` if you
do not want to set it manually every time you enter the terminal.
In case you do not have breeze image configured locally, you can also disable all checks that require
@@ -386,7 +386,7 @@ code. But you can run pre-commit hooks manually as needed.
.. code-block:: bash
- SKIP=run-mypy,run-flake8 pre-commit run --all-files
+ SKIP=run-mypy,ruff pre-commit run --all-files
You can always skip running the tests by providing ``--no-verify`` flag to the
@@ -416,17 +416,17 @@ Run the ``mypy`` check for all files:
breeze static-checks --type run-mypy --all-files
-Run the ``flake8`` check for the ``tests.core.py`` file with verbose output:
+Run the ``ruff`` check for the ``tests.core.py`` file with verbose output:
.. code-block:: bash
- breeze static-checks --type run-flake8 --file tests/core.py --verbose
+ breeze static-checks --type ruff --file tests/core.py --verbose
-Run the ``flake8`` check for the ``tests.core`` package with verbose output:
+Run the ``ruff for the ``tests.core`` package with verbose output:
.. code-block:: bash
- breeze static-checks --type run-flake8 --file tests/core/* --verbose
+ breeze static-checks --type ruff --file tests/core/* --verbose
Run all checks for the currently staged files:
@@ -461,10 +461,10 @@ Printing the commands while executing:
.. code-block:: bash
- VERBOSE="true" pre-commit run --verbose run-flake8
+ VERBOSE="true" pre-commit run --verbose ruff
Just performing dry run:
.. code-block:: bash
- DRY_RUN="true" pre-commit run --verbose run-flake8
+ DRY_RUN="true" pre-commit run --verbose ruff
[airflow] 20/20: Make sure DOCKER_BUILDKIT=1 variable is set for all builds (#29928)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b87990dc9d68553e05eafc75201045b0e1447445
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Mar 5 11:24:34 2023 +0100
Make sure DOCKER_BUILDKIT=1 variable is set for all builds (#29928)
The DOCKER_BUILDKIT=1 was only set in some specific cases, but since
our images are using buildkit features, we should always use
buildkit when building the images.
(cherry picked from commit 0f6e0ee611a81046838ecc852bdc84155e335ecf)
---
dev/breeze/src/airflow_breeze/commands/ci_image_commands.py | 6 ++++--
dev/breeze/src/airflow_breeze/commands/production_image_commands.py | 6 ++++--
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
index 6b72895952..d37c3e313b 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_image_commands.py
@@ -464,9 +464,9 @@ def run_build_ci_image(
output=output,
)
else:
+ env = os.environ.copy()
+ env["DOCKER_BUILDKIT"] = "1"
if ci_image_params.empty_image:
- env = os.environ.copy()
- env["DOCKER_BUILDKIT"] = "1"
get_console(output=output).print(
f"\n[info]Building empty CI Image for Python {ci_image_params.python}\n"
)
@@ -502,6 +502,7 @@ def run_build_ci_image(
cwd=AIRFLOW_SOURCES_ROOT,
text=True,
check=False,
+ env=env,
output=output,
)
if build_command_result.returncode != 0 and not ci_image_params.upgrade_to_newer_dependencies:
@@ -515,6 +516,7 @@ def run_build_ci_image(
image_params=ci_image_params,
),
cwd=AIRFLOW_SOURCES_ROOT,
+ env=env,
text=True,
check=False,
output=output,
diff --git a/dev/breeze/src/airflow_breeze/commands/production_image_commands.py b/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
index 8cc058741b..66256b7e9e 100644
--- a/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/production_image_commands.py
@@ -482,9 +482,9 @@ def run_build_production_image(
if prod_image_params.prepare_buildx_cache:
build_command_result = build_cache(image_params=prod_image_params, output=output)
else:
+ env = os.environ.copy()
+ env["DOCKER_BUILDKIT"] = "1"
if prod_image_params.empty_image:
- env = os.environ.copy()
- env["DOCKER_BUILDKIT"] = "1"
get_console(output=output).print(
f"\n[info]Building empty PROD Image for Python {prod_image_params.python}\n"
)
@@ -504,6 +504,7 @@ def run_build_production_image(
),
cwd=AIRFLOW_SOURCES_ROOT,
check=False,
+ env=env,
text=True,
output=output,
)
@@ -523,6 +524,7 @@ def run_build_production_image(
cwd=AIRFLOW_SOURCES_ROOT,
check=False,
text=True,
+ env=env,
output=output,
)
if build_command_result.returncode == 0:
[airflow] 08/20: Fix `project-name` generation for Providers `test-type` in Breeze (#29030)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit fbada0425747035b43d34210945ca092c8093452
Author: Josh Fell <48...@users.noreply.github.com>
AuthorDate: Thu Jan 19 06:37:21 2023 -0500
Fix `project-name` generation for Providers `test-type` in Breeze (#29030)
The `breeze testing tests --test-type "Providers[...]"` command would fail for an invalid `project-name` for `docker compose` if the provider(s) specified required a dotted path (e.g. `microsoft.azure`, `dbt.cloud`, etc.).
This fixes the `project-name` generation in Breeze when using the `Providers` test type for those dotted-path providers.
(cherry picked from commit be31214dcf14db39b7a5f422ca272cdc13e08268)
---
dev/breeze/src/airflow_breeze/commands/testing_commands.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands.py b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
index 74c3069417..86561cac58 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
@@ -17,6 +17,7 @@
from __future__ import annotations
import os
+import re
import sys
from datetime import datetime
@@ -199,8 +200,9 @@ def _run_test(
return result.returncode, f"Test: {exec_shell_params.test_type}"
-def _file_name_from_test_type(test_type):
- return test_type.lower().replace("[", "_").replace("]", "").replace(",", "_")[:30]
+def _file_name_from_test_type(test_type: str):
+ test_type_no_brackets = test_type.lower().replace("[", "_").replace("]", "")
+ return re.sub("[,\.]", "_", test_type_no_brackets)[:30]
def _run_tests_in_pool(