You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by je...@apache.org on 2024/04/26 23:00:16 UTC
(airflow) branch v2-9-test updated (e61cb8fa41 -> c9cc72608d)
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a change to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from e61cb8fa41 Apply PROVIDE_PROJECT_ID mypy workaround across Google provider (#39129)
new 1d40343286 Fix typo with DatasetOrTimeSchedule as it was renamed (#38576)
new 70d31c0977 Raise deprecated warning in `airflow dags backfill` only if `-I`/`--ignore-first-depends-on-past` provided (#38676)
new c501d3a1f6 Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods (#37757)
new 83e7dec335 Improve audit log events filter (#38719)
new 16256a2514 Bump undici from 5.28.3 to 5.28.4 in /airflow/www (#38751)
new c6c589f5a4 Use async db calls in WorkflowTrigger (#38689)
new 6d36c80d0b Fix 'use case' spelling (#38817)
new a0249d4a9e Remove decorator from rendering fields example (#38827)
new 3b84410342 Update version added field in config.yml (#38840)
new 672f8ad02a Remove a couple items from the 2.9.0 changelog (#38866)
new 84b76e02fc Fix dag run link params (#38873)
new 72b2ec0ea4 Fix typo in templates-ref.rst (#38865)
new 1725139a42 Fix problem that links in legacy log view can not be clicked (#38882)
new 2659704629 Fix implicit cartessian product in AirflowSecurityManagerV2 (#38913)
new e9986043a1 Limit redis image to 7.2 in sample docker-compose.yaml (#38938)
new df37d60d49 Fix `SAWarning` 'Coercing Subquery object into a select() for use in IN()' (#38926)
new 39dd593027 Fix dag run selection (#38941)
new 2e25153a52 (external_task.py): changed the background color of external task in graph (#38969)
new b35f67dc4c Fix calendar view with no DAG Run. (#38964)
new 5c6736daeb Don't wait for DagRun lock in mini scheduler (#38914)
new 1f70db4794 Use debug level for minischeduler skip (#38976)
new adfabe897b Add in Trove classifiers Python 3.12 support (#39004)
new 82a1b6abde Add `inherit_cache` attribute for `CreateTableAs` custom SA Clause (#38985)
new d6e397fe11 Correcting an example taskflow example (#39015)
new 083a781d1f Undeprecate `BaseXCom.get_one` method for now (#38991)
new 3487fccfe2 Bugfix: Move rendering of `map_index_template` so it renders for failed tasks as long as it was defined before the point of failure (#38902)
new 983ab58769 Check whether `AUTH_ROLE_PUBLIC` is set in `check_authentication` (#39012)
new 394821f47a Hostname was "redacted", not "redact"; remove it when there is no context (#39037)
new 2970e35e1d Fix doc redirects (#39039)
new 672d47ebf0 [docs] update `DagBag` class docstring to include all params (#38814)
new 0602de2222 Data aware scheduling docs edits (#38687)
new ed4cd93eca update document (#39068)
new 5fcf36cc65 Check that the dataset<>task exists before trying to render graph (#39069)
new 7cd476ed50 Graph view improvements (#38940)
new 6cee9b0d56 Improve rendered_template ux in react dag page (#39122)
new 2091072167 Improve task filtering UX (#39119)
new fd0226e068 Add k8s 1.29 to supported version in docs (#39168)
new 092ceaed55 Update `is_authorized_custom_view` from auth manager to handle custom actions (#39167)
new 8e458d979e Add warning about run/import Airflow under the Windows (#39196)
new c5836745e8 Add link to `user` and `role` commands (#39224)
new 7cd311dc78 Use grid view for Task Instance's `log_url` (#39183)
new 899d4e8587 Fixed side effect of menu filtering causing disappearing menus (#39229)
new f299149da4 doc: dynamictaskmapping pythonoperator op_kwargs (#39242)
new 505ecd9538 Get served logs when remote or executor logs not available for non-running task try (#39177)
new d2ae8c9755 Force to use Airflow Deprecation warnings categories on `@deprecated` decorator (#39205)
new df9db0cb15 Add Grid button to Task Instance view (#39223)
new d5d8b588f7 Add workaround for datetime-local input in firefox (#39261)
new c9cc72608d Fix trigger kwarg encryption migration (#39246)
The 48 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.pre-commit-config.yaml | 14 ++
RELEASE_NOTES.rst | 2 -
airflow/__init__.py | 16 +-
airflow/api/common/experimental/get_code.py | 6 +-
.../api/common/experimental/get_dag_run_state.py | 3 +-
airflow/api/common/experimental/get_task.py | 3 +-
.../api/common/experimental/get_task_instance.py | 4 +-
airflow/api/common/experimental/pool.py | 10 +-
.../endpoints/import_error_endpoint.py | 8 +-
airflow/api_connexion/security.py | 6 +
airflow/auth/managers/base_auth_manager.py | 21 ++-
airflow/auth/managers/fab/fab_auth_manager.py | 5 +-
.../auth/managers/fab/security_manager/override.py | 4 +-
airflow/cli/commands/dag_command.py | 13 +-
airflow/compat/functools.pyi | 27 ---
airflow/config_templates/config.yml | 2 +-
.../0140_2_9_0_update_trigger_kwargs_type.py | 46 ++++-
airflow/models/abstractoperator.py | 4 +-
airflow/models/dagbag.py | 5 +
airflow/models/mappedoperator.py | 8 +-
airflow/models/taskinstance.py | 38 ++--
airflow/models/trigger.py | 10 +-
airflow/models/xcom.py | 2 -
.../amazon/aws/auth_manager/avp/entities.py | 2 +-
.../amazon/aws/auth_manager/avp/facade.py | 7 +-
.../amazon/aws/auth_manager/aws_auth_manager.py | 2 +-
.../providers/fab/auth_manager/fab_auth_manager.py | 7 +-
airflow/reproducible_build.yaml | 4 +-
airflow/sensors/external_task.py | 4 +-
airflow/triggers/external_task.py | 46 ++---
airflow/utils/db.py | 39 -----
airflow/utils/db_cleanup.py | 6 +-
airflow/utils/log/file_task_handler.py | 12 +-
airflow/utils/task_group.py | 9 +-
airflow/utils/weight_rule.py | 4 +-
airflow/www/auth.py | 6 +-
airflow/www/security.py | 4 +-
airflow/www/security_manager.py | 8 +-
airflow/www/static/js/api/useGridData.ts | 32 +++-
.../static/js/cluster-activity/nav/FilterBar.tsx | 9 +-
airflow/www/static/js/components/DateTimeInput.tsx | 84 +++++++++
airflow/www/static/js/dag/Main.tsx | 14 +-
airflow/www/static/js/dag/TaskName.tsx | 1 -
airflow/www/static/js/dag/details/AuditLog.tsx | 73 ++++----
airflow/www/static/js/dag/details/FilterTasks.tsx | 76 +++++---
airflow/www/static/js/dag/details/Header.tsx | 13 +-
airflow/www/static/js/dag/details/dag/Calendar.tsx | 6 +-
.../www/static/js/dag/details/graph/DagNode.tsx | 34 ++--
airflow/www/static/js/dag/details/graph/index.tsx | 15 +-
airflow/www/static/js/dag/details/index.tsx | 2 +-
.../static/js/dag/details/taskInstance/Details.tsx | 59 ++++---
.../js/dag/details/taskInstance/Logs/utils.ts | 1 +
.../www/static/js/dag/details/taskInstance/Nav.tsx | 10 +-
airflow/www/static/js/dag/grid/renderTaskRows.tsx | 5 -
airflow/www/static/js/dag/nav/FilterBar.tsx | 6 +-
airflow/www/static/js/dag/useFilters.tsx | 1 +
airflow/www/static/js/dag/useSelection.ts | 7 -
airflow/www/static/js/ti_log.js | 8 +-
airflow/www/static/js/types/index.ts | 1 +
airflow/www/static/js/utils/graph.ts | 11 +-
airflow/www/templates/airflow/dag.html | 2 +-
airflow/www/templates/airflow/task_instance.html | 3 +
airflow/www/utils.py | 4 +-
airflow/www/views.py | 48 ++---
airflow/www/yarn.lock | 6 +-
clients/python/pyproject.toml | 1 +
contributing-docs/08_static_code_checks.rst | 4 +
dev/breeze/doc/images/output_static-checks.svg | 152 ++++++++--------
dev/breeze/doc/images/output_static-checks.txt | 2 +-
dev/breeze/src/airflow_breeze/pre_commit_ids.py | 2 +
.../authoring-and-scheduling/datasets.rst | 65 ++++---
.../dynamic-task-mapping.rst | 18 ++
docs/apache-airflow/cli-and-env-variables-ref.rst | 1 +
docs/apache-airflow/core-concepts/operators.rst | 2 +-
docs/apache-airflow/extra-packages-ref.rst | 4 +-
.../howto/docker-compose/docker-compose.yaml | 4 +-
docs/apache-airflow/img/airflow_erd.sha256 | 2 +-
docs/apache-airflow/img/airflow_erd.svg | 4 +-
docs/apache-airflow/installation/prerequisites.rst | 2 +-
docs/apache-airflow/migrations-ref.rst | 2 +-
docs/apache-airflow/redirects.txt | 15 +-
docs/apache-airflow/templates-ref.rst | 2 +-
docs/apache-airflow/tutorial/taskflow.rst | 1 +
docs/conf.py | 2 +-
hatch_build.py | 1 +
pyproject.toml | 8 +-
scripts/ci/pre_commit/check_deprecations.py | 194 +++++++++++++++++++++
scripts/ci/pre_commit/compat_cache_on_methods.py | 69 ++++++++
tests/api_connexion/conftest.py | 15 +-
.../endpoints/test_config_endpoint.py | 22 +++
.../endpoints/test_connection_endpoint.py | 89 ++++++++++
tests/api_connexion/endpoints/test_dag_endpoint.py | 99 +++++++++++
.../endpoints/test_dag_run_endpoint.py | 185 ++++++++++++++++++++
.../endpoints/test_dag_source_endpoint.py | 16 ++
.../endpoints/test_dag_warning_endpoint.py | 12 ++
.../endpoints/test_dataset_endpoint.py | 186 +++++++++++++++++++-
.../endpoints/test_event_log_endpoint.py | 44 +++++
.../auth/backend/test_basic_auth.py | 6 +-
tests/auth/managers/test_base_auth_manager.py | 42 ++++-
tests/cli/commands/test_dag_command.py | 35 ++--
tests/models/test_mappedoperator.py | 29 +++
tests/models/test_taskinstance.py | 9 +-
tests/models/test_trigger.py | 17 ++
.../test_role_and_permission_endpoint.py | 63 +++++++
tests/providers/fab/auth_manager/conftest.py | 15 +-
.../fab/auth_manager/test_fab_auth_manager.py | 13 +-
tests/providers/smtp/notifications/test_smtp.py | 2 +-
tests/triggers/test_external_task.py | 153 ++++++++++------
tests/utils/test_log_handlers.py | 14 +-
tests/www/views/test_views_tasks.py | 20 +++
110 files changed, 2002 insertions(+), 589 deletions(-)
delete mode 100644 airflow/compat/functools.pyi
create mode 100644 airflow/www/static/js/components/DateTimeInput.tsx
create mode 100755 scripts/ci/pre_commit/check_deprecations.py
create mode 100755 scripts/ci/pre_commit/compat_cache_on_methods.py
(airflow) 37/48: Add k8s 1.29 to supported version in docs (#39168)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit fd0226e068d6ade76f558d37b8e75f218fe9ea44
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Mon Apr 22 11:56:28 2024 -0400
Add k8s 1.29 to supported version in docs (#39168)
(cherry picked from commit 897f00d3cafb8a7ef9c1444997784d693c09611f)
---
docs/apache-airflow/installation/prerequisites.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow/installation/prerequisites.rst b/docs/apache-airflow/installation/prerequisites.rst
index 07ccc903c7..c253217a29 100644
--- a/docs/apache-airflow/installation/prerequisites.rst
+++ b/docs/apache-airflow/installation/prerequisites.rst
@@ -28,7 +28,7 @@ Airflow™ is tested with:
* MySQL: 8.0, `Innovation <https://dev.mysql.com/blog-archive/introducing-mysql-innovation-and-long-term-support-lts-versions>`_
* SQLite: 3.15.0+
-* Kubernetes: 1.25, 1.26, 1.27, 1.28
+* Kubernetes: 1.25, 1.26, 1.27, 1.28, 1.29
The minimum memory required we recommend Airflow to run with is 4GB, but the actual requirements depend
wildly on the deployment options you have
(airflow) 35/48: Improve rendered_template ux in react dag page (#39122)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6cee9b0d565b3e9d0dbefc6c614f2654073222eb
Author: Brent Bovenzi <br...@astronomer.io>
AuthorDate: Fri Apr 19 00:22:19 2024 -0400
Improve rendered_template ux in react dag page (#39122)
(cherry picked from commit 50334230b03bf3c1f6ff11ea7808dde1a79df98a)
---
.../static/js/dag/details/taskInstance/Details.tsx | 59 +++++++++++++---------
.../www/static/js/dag/details/taskInstance/Nav.tsx | 10 +---
airflow/www/templates/airflow/dag.html | 1 -
3 files changed, 38 insertions(+), 32 deletions(-)
diff --git a/airflow/www/static/js/dag/details/taskInstance/Details.tsx b/airflow/www/static/js/dag/details/taskInstance/Details.tsx
index 72d8273ce6..80e26d79ca 100644
--- a/airflow/www/static/js/dag/details/taskInstance/Details.tsx
+++ b/airflow/www/static/js/dag/details/taskInstance/Details.tsx
@@ -231,29 +231,6 @@ const Details = ({ gridInstance, taskInstance, group }: Props) => {
</Td>
</Tr>
)}
- {taskInstance?.renderedFields && (
- <>
- {Object.keys(taskInstance.renderedFields).map((key) => {
- const renderedFields = taskInstance.renderedFields as Record<
- string,
- unknown
- >;
- if (renderedFields[key]) {
- return (
- <Tr key={key}>
- <Td>{key}</Td>
- <Td>
- <Code fontSize="md">
- {JSON.stringify(renderedFields[key])}
- </Code>
- </Td>
- </Tr>
- );
- }
- return null;
- })}
- </>
- )}
{!!taskInstance?.pool && (
<Tr>
<Td>Pool</Td>
@@ -300,6 +277,42 @@ const Details = ({ gridInstance, taskInstance, group }: Props) => {
)}
</Tbody>
</Table>
+ {taskInstance?.renderedFields && (
+ <Box mt={3}>
+ <Text as="strong" mb={3}>
+ Rendered Templates
+ </Text>
+ <Table>
+ <Tbody>
+ {Object.keys(taskInstance.renderedFields).map((key) => {
+ const renderedFields = taskInstance.renderedFields as Record<
+ string,
+ unknown
+ >;
+ let field = renderedFields[key];
+ if (field) {
+ if (typeof field !== "string") {
+ try {
+ field = JSON.stringify(field);
+ } catch (e) {
+ // skip
+ }
+ }
+ return (
+ <Tr key={key}>
+ <Td>{key}</Td>
+ <Td>
+ <Code fontSize="md">{field as string}</Code>
+ </Td>
+ </Tr>
+ );
+ }
+ return null;
+ })}
+ </Tbody>
+ </Table>
+ </Box>
+ )}
</Box>
);
};
diff --git a/airflow/www/static/js/dag/details/taskInstance/Nav.tsx b/airflow/www/static/js/dag/details/taskInstance/Nav.tsx
index 1e5d84d8ba..88560d5dac 100644
--- a/airflow/www/static/js/dag/details/taskInstance/Nav.tsx
+++ b/airflow/www/static/js/dag/details/taskInstance/Nav.tsx
@@ -29,7 +29,6 @@ const dagId = getMetaValue("dag_id");
const isK8sExecutor = getMetaValue("k8s_or_k8scelery_executor") === "True";
const taskInstancesUrl = getMetaValue("task_instances_list_url");
const renderedK8sUrl = getMetaValue("rendered_k8s_url");
-const renderedTemplatesUrl = getMetaValue("rendered_templates_url");
const taskUrl = getMetaValue("task_url");
const gridUrl = getMetaValue("grid_url");
@@ -50,7 +49,6 @@ const Nav = forwardRef<HTMLDivElement, Props>(
map_index: mapIndex ?? -1,
});
const detailsLink = `${taskUrl}&${params}`;
- const renderedLink = `${renderedTemplatesUrl}&${params}`;
const k8sLink = `${renderedK8sUrl}&${params}`;
const listParams = new URLSearchParamsWrapper({
_flt_3_dag_id: dagId,
@@ -79,7 +77,6 @@ const Nav = forwardRef<HTMLDivElement, Props>(
{(!isMapped || mapIndex !== undefined) && (
<>
<LinkButton href={detailsLink}>More Details</LinkButton>
- <LinkButton href={renderedLink}>Rendered Template</LinkButton>
{isK8sExecutor && (
<LinkButton href={k8sLink}>K8s Pod Spec</LinkButton>
)}
@@ -88,11 +85,8 @@ const Nav = forwardRef<HTMLDivElement, Props>(
)}
</>
)}
- <LinkButton
- href={allInstancesLink}
- title="View all instances across all DAG runs"
- >
- List Instances, all runs
+ <LinkButton href={allInstancesLink} title="View all">
+ List All Instances
</LinkButton>
</Flex>
);
diff --git a/airflow/www/templates/airflow/dag.html b/airflow/www/templates/airflow/dag.html
index d9ef3c139a..29f535b4b6 100644
--- a/airflow/www/templates/airflow/dag.html
+++ b/airflow/www/templates/airflow/dag.html
@@ -65,7 +65,6 @@
<meta name="graph_url" content="{{ url_for('Airflow.graph', dag_id=dag.dag_id, root=root) }}">
<meta name="task_url" content="{{ url_for('Airflow.task', dag_id=dag.dag_id) }}">
<meta name="log_url" content="{{ url_for('Airflow.log', dag_id=dag.dag_id) }}">
- <meta name="rendered_templates_url" content="{{ url_for('Airflow.rendered_templates', dag_id=dag.dag_id) }}">
<meta name="rendered_k8s_url" content="{{ url_for('Airflow.rendered_k8s', dag_id=dag.dag_id) }}">
<meta name="task_instances_list_url" content="{{ url_for('TaskInstanceModelView.list') }}">
<meta name="tag_index_url" content="{{ url_for('Airflow.index', tags='_TAG_NAME_') }}">
(airflow) 13/48: Fix problem that links in legacy log view can not be clicked (#38882)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1725139a42925ca369d36101e0af694af5486c3f
Author: Jens Scheffler <95...@users.noreply.github.com>
AuthorDate: Wed Apr 10 13:40:45 2024 +0200
Fix problem that links in legacy log view can not be clicked (#38882)
* Remove problems by double-linkification from coloring
* Fix problem that links in legacy log view can not be clicked
(cherry picked from commit 87acf61f574daf47ce9e03a986e352a2c727f4ce)
---
airflow/www/static/js/dag/details/taskInstance/Logs/utils.ts | 1 +
airflow/www/static/js/ti_log.js | 8 ++++++--
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/airflow/www/static/js/dag/details/taskInstance/Logs/utils.ts b/airflow/www/static/js/dag/details/taskInstance/Logs/utils.ts
index a7043cb32f..1d41b3dccd 100644
--- a/airflow/www/static/js/dag/details/taskInstance/Logs/utils.ts
+++ b/airflow/www/static/js/dag/details/taskInstance/Logs/utils.ts
@@ -62,6 +62,7 @@ export const parseLogs = (
const parsedLines: Array<string> = [];
const fileSources: Set<string> = new Set();
const ansiUp = new AnsiUp();
+ ansiUp.url_allowlist = {};
const urlRegex = /((https?:\/\/|http:\/\/)[^\s]+)/g;
// Detect log groups which can be collapsed
diff --git a/airflow/www/static/js/ti_log.js b/airflow/www/static/js/ti_log.js
index 465af4be6d..1fd66acee6 100644
--- a/airflow/www/static/js/ti_log.js
+++ b/airflow/www/static/js/ti_log.js
@@ -111,6 +111,7 @@ function autoTailingLog(tryNumber, metadata = null, autoTailing = false) {
// Text coloring, detect urls and log timestamps
const ansiUp = new AnsiUp();
+ ansiUp.url_allowlist = {};
// Detect urls and log timestamps
const urlRegex =
/http(s)?:\/\/[\w.-]+(\.?:[\w.-]+)*([/?#][\w\-._~:/?#[\]@!$&'()*+,;=.%]+)?/g;
@@ -194,11 +195,14 @@ function handleLogGroupClick(e) {
if (e.target.id?.endsWith(unfoldIdSuffix)) {
e.target.style.display = "none";
e.target.nextSibling.style.display = "inline";
- } else if (e.target.id?.endsWith(foldIdSuffix)) {
+ return false;
+ }
+ if (e.target.id?.endsWith(foldIdSuffix)) {
e.target.parentNode.style.display = "none";
e.target.parentNode.previousSibling.style.display = "inline";
+ return false;
}
- return false;
+ return true;
}
function setDownloadUrl(tryNumber) {
(airflow) 20/48: Don't wait for DagRun lock in mini scheduler (#38914)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5c6736daeb9910f3b7c5f7c5b166d07841f86261
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Wed Apr 10 20:06:37 2024 -0700
Don't wait for DagRun lock in mini scheduler (#38914)
We should just bail if something else is already "minischeduling". We already catch OperationalError so there's nothing else we need to change for this.
Sometimes (e.g. with certain task mapping scenarios) many tasks from same dag run are trying to "minischedule" at the same time. If something else is already locking, it's probably better to just move on and not wait.
(cherry picked from commit afe8e82531cece4f3aeaea465b985ebd7257e71d)
---
airflow/models/taskinstance.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
index 2f0d6e27bf..6dedf4bfd2 100644
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -3439,6 +3439,7 @@ class TaskInstance(Base, LoggingMixin):
run_id=ti.run_id,
),
session=session,
+ nowait=True,
).one()
task = ti.task
(airflow) 27/48: Check whether `AUTH_ROLE_PUBLIC` is set in `check_authentication` (#39012)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 983ab5876945bc399806efa7d426afac29eb6025
Author: Wei Lee <we...@gmail.com>
AuthorDate: Mon Apr 15 22:37:33 2024 +0800
Check whether `AUTH_ROLE_PUBLIC` is set in `check_authentication` (#39012)
(cherry picked from commit 21f08ddd93f7ff8ee18f1fec1778f3d016e3db00)
---
airflow/api_connexion/security.py | 6 +
tests/api_connexion/conftest.py | 15 +-
.../endpoints/test_config_endpoint.py | 22 +++
.../endpoints/test_connection_endpoint.py | 89 ++++++++++
tests/api_connexion/endpoints/test_dag_endpoint.py | 99 +++++++++++
.../endpoints/test_dag_run_endpoint.py | 185 ++++++++++++++++++++
.../endpoints/test_dag_source_endpoint.py | 16 ++
.../endpoints/test_dag_warning_endpoint.py | 12 ++
.../endpoints/test_dataset_endpoint.py | 186 ++++++++++++++++++++-
.../endpoints/test_event_log_endpoint.py | 44 +++++
.../test_role_and_permission_endpoint.py | 63 +++++++
tests/providers/fab/auth_manager/conftest.py | 15 +-
12 files changed, 749 insertions(+), 3 deletions(-)
diff --git a/airflow/api_connexion/security.py b/airflow/api_connexion/security.py
index 1cc044d9dd..660bc6cce2 100644
--- a/airflow/api_connexion/security.py
+++ b/airflow/api_connexion/security.py
@@ -49,6 +49,12 @@ def check_authentication() -> None:
response = auth.requires_authentication(Response)()
if response.status_code == 200:
return
+
+ # Even if the current_user is anonymous, the AUTH_ROLE_PUBLIC might still have permission.
+ appbuilder = get_airflow_app().appbuilder
+ if appbuilder.get_app.config.get("AUTH_ROLE_PUBLIC", None):
+ return
+
# since this handler only checks authentication, not authorization,
# we should always return 401
raise Unauthenticated(headers=response.headers)
diff --git a/tests/api_connexion/conftest.py b/tests/api_connexion/conftest.py
index c860a78f27..481f07fe73 100644
--- a/tests/api_connexion/conftest.py
+++ b/tests/api_connexion/conftest.py
@@ -40,7 +40,9 @@ def minimal_app_for_api():
)
def factory():
with conf_vars({("api", "auth_backends"): "tests.test_utils.remote_user_api_auth_backend"}):
- return app.create_app(testing=True, config={"WTF_CSRF_ENABLED": False}) # type:ignore
+ _app = app.create_app(testing=True, config={"WTF_CSRF_ENABLED": False}) # type:ignore
+ _app.config["AUTH_ROLE_PUBLIC"] = None
+ return _app
return factory()
@@ -67,3 +69,14 @@ def dagbag():
)
DagBag(include_examples=True, read_dags_from_db=False).sync_to_db()
return DagBag(include_examples=True, read_dags_from_db=True)
+
+
+@pytest.fixture
+def set_auto_role_public(request):
+ app = request.getfixturevalue("minimal_app_for_api")
+ auto_role_public = app.config["AUTH_ROLE_PUBLIC"]
+ app.config["AUTH_ROLE_PUBLIC"] = request.param
+
+ yield
+
+ app.config["AUTH_ROLE_PUBLIC"] = auto_role_public
diff --git a/tests/api_connexion/endpoints/test_config_endpoint.py b/tests/api_connexion/endpoints/test_config_endpoint.py
index c091c4ef1c..3dd5814e5d 100644
--- a/tests/api_connexion/endpoints/test_config_endpoint.py
+++ b/tests/api_connexion/endpoints/test_config_endpoint.py
@@ -222,6 +222,16 @@ class TestGetConfig:
assert response.status_code == 403
assert "chose not to expose" in response.json["detail"]
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ response = self.client.get("/api/v1/config", headers={"Accept": "application/json"})
+
+ assert response.status_code == expected_status_code
+
class TestGetValue:
@pytest.fixture(autouse=True)
@@ -339,3 +349,15 @@ class TestGetValue:
)
assert response.status_code == 403
assert "chose not to expose" in response.json["detail"]
+
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ response = self.client.get(
+ "/api/v1/config/section/smtp/option/smtp_mail_from", headers={"Accept": "application/json"}
+ )
+
+ assert response.status_code == expected_status_code
diff --git a/tests/api_connexion/endpoints/test_connection_endpoint.py b/tests/api_connexion/endpoints/test_connection_endpoint.py
index 0d9debeb98..4f85f53e1e 100644
--- a/tests/api_connexion/endpoints/test_connection_endpoint.py
+++ b/tests/api_connexion/endpoints/test_connection_endpoint.py
@@ -112,6 +112,22 @@ class TestDeleteConnection(TestConnectionEndpoint):
)
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 204)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ connection_model = Connection(conn_id="test-connection", conn_type="test_type")
+ session.add(connection_model)
+ session.commit()
+ conn = session.query(Connection).all()
+ assert len(conn) == 1
+
+ response = self.client.delete("/api/v1/connections/test-connection")
+
+ assert response.status_code == expected_status_code
+
class TestGetConnection(TestConnectionEndpoint):
def test_should_respond_200(self, session):
@@ -177,6 +193,31 @@ class TestGetConnection(TestConnectionEndpoint):
assert_401(response)
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ connection_model = Connection(
+ conn_id="test-connection-id",
+ conn_type="mysql",
+ description="test description",
+ host="mysql",
+ login="login",
+ schema="testschema",
+ port=80,
+ extra='{"param": "value"}',
+ )
+ session.add(connection_model)
+ session.commit()
+ result = session.query(Connection).all()
+ assert len(result) == 1
+
+ response = self.client.get("/api/v1/connections/test-connection-id")
+
+ assert response.status_code == expected_status_code
+
class TestGetConnections(TestConnectionEndpoint):
def test_should_respond_200(self, session):
@@ -255,6 +296,16 @@ class TestGetConnections(TestConnectionEndpoint):
assert_401(response)
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ response = self.client.get("/api/v1/connections")
+
+ assert response.status_code == expected_status_code
+
class TestGetConnectionsPagination(TestConnectionEndpoint):
@pytest.mark.parametrize(
@@ -528,6 +579,21 @@ class TestPatchConnection(TestConnectionEndpoint):
assert_401(response)
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ self._create_connection(session)
+
+ response = self.client.patch(
+ "/api/v1/connections/test-connection-id",
+ json={"connection_id": "test-connection-id", "conn_type": "test_type", "extra": '{"key": "var"}'},
+ )
+
+ assert response.status_code == expected_status_code
+
class TestPostConnection(TestConnectionEndpoint):
def test_post_should_respond_200(self, session):
@@ -609,6 +675,18 @@ class TestPostConnection(TestConnectionEndpoint):
assert_401(response)
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ response = self.client.post(
+ "/api/v1/connections", json={"connection_id": "test-connection-id", "conn_type": "test_type"}
+ )
+
+ assert response.status_code == expected_status_code
+
class TestConnection(TestConnectionEndpoint):
@mock.patch.dict(os.environ, {"AIRFLOW__CORE__TEST_CONNECTION": "Enabled"})
@@ -662,3 +740,14 @@ class TestConnection(TestConnectionEndpoint):
"Testing connections is disabled in Airflow configuration. "
"Contact your deployment admin to enable it."
)
+
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ @mock.patch.dict(os.environ, {"AIRFLOW__CORE__TEST_CONNECTION": "Enabled"})
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ payload = {"connection_id": "test-connection-id", "conn_type": "sqlite"}
+ response = self.client.post("/api/v1/connections/test", json=payload)
+ assert response.status_code == expected_status_code
diff --git a/tests/api_connexion/endpoints/test_dag_endpoint.py b/tests/api_connexion/endpoints/test_dag_endpoint.py
index 8578f633cf..b514faba27 100644
--- a/tests/api_connexion/endpoints/test_dag_endpoint.py
+++ b/tests/api_connexion/endpoints/test_dag_endpoint.py
@@ -317,6 +317,24 @@ class TestGetDag(TestDagEndpoint):
)
assert response.status_code == 400, f"Current code: {response.status_code}"
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ dag_model = DagModel(
+ dag_id="TEST_DAG_1",
+ fileloc="/tmp/dag_1.py",
+ schedule_interval=None,
+ is_paused=False,
+ )
+ session.add(dag_model)
+ session.commit()
+
+ response = self.client.get("/api/v1/dags/TEST_DAG_1")
+ assert response.status_code == expected_status_code
+
class TestGetDagDetails(TestDagEndpoint):
def test_should_respond_200(self, url_safe_serializer):
@@ -728,6 +746,18 @@ class TestGetDagDetails(TestDagEndpoint):
)
assert response.status_code == 400, f"Current code: {response.status_code}"
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, url_safe_serializer):
+ self._create_dag_model_for_details_endpoint(self.dag_id)
+ url_safe_serializer.dumps("/tmp/dag.py")
+ response = self.client.get(f"/api/v1/dags/{self.dag_id}/details")
+
+ assert response.status_code == expected_status_code
+
class TestGetDags(TestDagEndpoint):
@provide_session
@@ -1259,6 +1289,22 @@ class TestGetDags(TestDagEndpoint):
assert response.status_code == 400, f"Current code: {response.status_code}"
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ self._create_dag_models(2)
+ self._create_deactivated_dag()
+
+ dags_query = session.query(DagModel).filter(~DagModel.is_subdag)
+ assert len(dags_query.all()) == 3
+
+ response = self.client.get("api/v1/dags")
+
+ assert response.status_code == expected_status_code
+
class TestPatchDag(TestDagEndpoint):
def test_should_respond_200_on_patch_is_paused(self, url_safe_serializer, session):
@@ -1485,6 +1531,24 @@ class TestPatchDag(TestDagEndpoint):
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(
+ self, set_auto_role_public, expected_status_code, url_safe_serializer, session
+ ):
+ url_safe_serializer.dumps("/tmp/dag_1.py")
+ dag_model = self._create_dag_model()
+ payload = {"is_paused": False}
+ response = self.client.patch(
+ f"/api/v1/dags/{dag_model.dag_id}",
+ json=payload,
+ )
+
+ assert response.status_code == expected_status_code
+
class TestPatchDags(TestDagEndpoint):
@provide_session
@@ -2291,6 +2355,29 @@ class TestPatchDags(TestDagEndpoint):
)
assert response.status_code == 400
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(
+ self, set_auto_role_public, expected_status_code, session, url_safe_serializer
+ ):
+ url_safe_serializer.dumps("/tmp/dag_1.py")
+ url_safe_serializer.dumps("/tmp/dag_2.py")
+ self._create_dag_models(2)
+ self._create_deactivated_dag()
+
+ dags_query = session.query(DagModel).filter(~DagModel.is_subdag)
+ assert len(dags_query.all()) == 3
+
+ response = self.client.patch(
+ "/api/v1/dags?dag_id_pattern=~",
+ json={"is_paused": False},
+ )
+
+ assert response.status_code == expected_status_code
+
class TestDeleteDagEndpoint(TestDagEndpoint):
def test_that_dag_can_be_deleted(self, session):
@@ -2342,3 +2429,15 @@ class TestDeleteDagEndpoint(TestDagEndpoint):
environ_overrides={"REMOTE_USER": "test_no_permissions"},
)
assert response.status_code == 403
+
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 204)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ self._create_dag_models(1)
+
+ response = self.client.delete("/api/v1/dags/TEST_DAG_1")
+
+ assert response.status_code == expected_status_code
diff --git a/tests/api_connexion/endpoints/test_dag_run_endpoint.py b/tests/api_connexion/endpoints/test_dag_run_endpoint.py
index f6ace16099..5182ef427e 100644
--- a/tests/api_connexion/endpoints/test_dag_run_endpoint.py
+++ b/tests/api_connexion/endpoints/test_dag_run_endpoint.py
@@ -214,6 +214,18 @@ class TestDeleteDagRun(TestDagRunEndpoint):
)
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 204)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ session.add_all(self._create_test_dag_run())
+ session.commit()
+ response = self.client.delete("api/v1/dags/TEST_DAG_ID/dagRuns/TEST_DAG_RUN_ID_1")
+
+ assert response.status_code == expected_status_code
+
class TestGetDagRun(TestDagRunEndpoint):
def test_should_respond_200(self, session):
@@ -333,6 +345,29 @@ class TestGetDagRun(TestDagRunEndpoint):
)
assert response.status_code == 400, f"Current code: {response.status_code}"
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ dagrun_model = DagRun(
+ dag_id="TEST_DAG_ID",
+ run_id="TEST_DAG_RUN_ID",
+ run_type=DagRunType.MANUAL,
+ execution_date=timezone.parse(self.default_time),
+ start_date=timezone.parse(self.default_time),
+ external_trigger=True,
+ state="running",
+ )
+ session.add(dagrun_model)
+ session.commit()
+ result = session.query(DagRun).all()
+ assert len(result) == 1
+
+ response = self.client.get("api/v1/dags/TEST_DAG_ID/dagRuns/TEST_DAG_RUN_ID")
+ assert response.status_code == expected_status_code
+
class TestGetDagRuns(TestDagRunEndpoint):
def test_should_respond_200(self, session):
@@ -508,6 +543,18 @@ class TestGetDagRuns(TestDagRunEndpoint):
)
assert response.status_code == 400, f"Current code: {response.status_code}"
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ self._create_test_dag_run()
+ result = session.query(DagRun).all()
+ assert len(result) == 2
+ response = self.client.get("api/v1/dags/TEST_DAG_ID/dagRuns")
+ assert response.status_code == expected_status_code
+
class TestGetDagRunsPagination(TestDagRunEndpoint):
@pytest.mark.parametrize(
@@ -931,6 +978,18 @@ class TestGetDagRunBatch(TestDagRunEndpoint):
assert_401(response)
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ self._create_test_dag_run()
+
+ response = self.client.post("api/v1/dags/~/dagRuns/list", json={"dag_ids": ["TEST_DAG_ID"]})
+
+ assert response.status_code == expected_status_code
+
class TestGetDagRunBatchPagination(TestDagRunEndpoint):
@pytest.mark.parametrize(
@@ -1564,6 +1623,26 @@ class TestPostDagRun(TestDagRunEndpoint):
)
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ execution_date = "2020-11-10T08:25:56.939143+00:00"
+ logical_date = "2020-11-10T08:25:56.939143+00:00"
+ self._create_dag("TEST_DAG_ID")
+
+ response = self.client.post(
+ "api/v1/dags/TEST_DAG_ID/dagRuns",
+ json={
+ "execution_date": execution_date,
+ "logical_date": logical_date,
+ },
+ )
+
+ assert response.status_code == expected_status_code
+
class TestPatchDagRunState(TestDagRunEndpoint):
@pytest.mark.parametrize("state", ["failed", "success", "queued"])
@@ -1687,6 +1766,31 @@ class TestPatchDagRunState(TestDagRunEndpoint):
)
assert response.status_code == 404
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, dag_maker, session):
+ dag_id = "TEST_DAG_ID"
+ dag_run_id = "TEST_DAG_RUN_ID"
+ with dag_maker(dag_id) as dag:
+ task = EmptyOperator(task_id="task_id", dag=dag)
+ self.app.dag_bag.bag_dag(dag, root_dag=dag)
+ dr = dag_maker.create_dagrun(run_id=dag_run_id, run_type=DagRunType.SCHEDULED)
+ ti = dr.get_task_instance(task_id="task_id")
+ ti.task = task
+ ti.state = State.RUNNING
+ session.merge(ti)
+ session.commit()
+
+ response = self.client.patch(
+ f"api/v1/dags/{dag_id}/dagRuns/{dag_run_id}",
+ json={"state": "failed"},
+ )
+
+ assert response.status_code == expected_status_code
+
class TestClearDagRun(TestDagRunEndpoint):
def test_should_respond_200(self, dag_maker, session):
@@ -1822,6 +1926,31 @@ class TestClearDagRun(TestDagRunEndpoint):
)
assert response.status_code == 404
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, dag_maker, session):
+ dag_id = "TEST_DAG_ID"
+ dag_run_id = "TEST_DAG_RUN_ID"
+ with dag_maker(dag_id) as dag:
+ task = EmptyOperator(task_id="task_id", dag=dag)
+ self.app.dag_bag.bag_dag(dag, root_dag=dag)
+ dr = dag_maker.create_dagrun(run_id=dag_run_id, run_type=DagRunType.SCHEDULED)
+ ti = dr.get_task_instance(task_id="task_id")
+ ti.task = task
+ ti.state = State.RUNNING
+ session.merge(ti)
+ session.commit()
+
+ response = self.client.patch(
+ f"api/v1/dags/{dag_id}/dagRuns/{dag_run_id}",
+ json={"state": "failed"},
+ )
+
+ assert response.status_code == expected_status_code
+
@pytest.mark.need_serialized_dag
class TestGetDagRunDatasetTriggerEvents(TestDagRunEndpoint):
@@ -1916,6 +2045,42 @@ class TestGetDagRunDatasetTriggerEvents(TestDagRunEndpoint):
assert_401(response)
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, dag_maker, session):
+ dataset1 = Dataset(uri="ds1")
+
+ with dag_maker(dag_id="source_dag", start_date=timezone.utcnow(), session=session):
+ EmptyOperator(task_id="task", outlets=[dataset1])
+ dr = dag_maker.create_dagrun()
+ ti = dr.task_instances[0]
+
+ ds1_id = session.query(DatasetModel.id).filter_by(uri=dataset1.uri).scalar()
+ event = DatasetEvent(
+ dataset_id=ds1_id,
+ source_task_id=ti.task_id,
+ source_dag_id=ti.dag_id,
+ source_run_id=ti.run_id,
+ source_map_index=ti.map_index,
+ )
+ session.add(event)
+
+ with dag_maker(dag_id="TEST_DAG_ID", start_date=timezone.utcnow(), session=session):
+ pass
+ dr = dag_maker.create_dagrun(run_id="TEST_DAG_RUN_ID", run_type=DagRunType.DATASET_TRIGGERED)
+ dr.consumed_dataset_events.append(event)
+
+ session.commit()
+ assert event.timestamp
+
+ response = self.client.get(
+ "api/v1/dags/TEST_DAG_ID/dagRuns/TEST_DAG_RUN_ID/upstreamDatasetEvents",
+ )
+ assert response.status_code == expected_status_code
+
class TestSetDagRunNote(TestDagRunEndpoint):
def test_should_respond_200(self, dag_maker, session):
@@ -2046,3 +2211,23 @@ class TestSetDagRunNote(TestDagRunEndpoint):
json={"note": "I am setting a note with anonymous user"},
)
assert response.status_code == 200
+
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ dag_runs: list[DagRun] = self._create_test_dag_run(DagRunState.SUCCESS)
+ session.add_all(dag_runs)
+ session.commit()
+ created_dr: DagRun = dag_runs[0]
+ new_note_value = "My super cool DagRun notes"
+ response = self.client.patch(
+ f"api/v1/dags/{created_dr.dag_id}/dagRuns/{created_dr.run_id}/setNote",
+ json={"note": new_note_value},
+ )
+
+ session.query(DagRun).filter(DagRun.run_id == created_dr.run_id).first()
+
+ assert response.status_code == expected_status_code
diff --git a/tests/api_connexion/endpoints/test_dag_source_endpoint.py b/tests/api_connexion/endpoints/test_dag_source_endpoint.py
index d48d7e1c02..14c7d1534d 100644
--- a/tests/api_connexion/endpoints/test_dag_source_endpoint.py
+++ b/tests/api_connexion/endpoints/test_dag_source_endpoint.py
@@ -202,3 +202,19 @@ class TestGetSource:
)
assert response.status_code == 403
assert read_dag.status_code == 200
+
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, url_safe_serializer):
+ dagbag = DagBag(dag_folder=EXAMPLE_DAG_FILE)
+ dagbag.sync_to_db()
+ test_dag: DAG = dagbag.dags[TEST_DAG_ID]
+ self._get_dag_file_docstring(test_dag.fileloc)
+
+ url = f"/api/v1/dagSources/{url_safe_serializer.dumps(test_dag.fileloc)}"
+ response = self.client.get(url, headers={"Accept": "text/plain"})
+
+ assert response.status_code == expected_status_code
diff --git a/tests/api_connexion/endpoints/test_dag_warning_endpoint.py b/tests/api_connexion/endpoints/test_dag_warning_endpoint.py
index 9310956d24..cc398329b9 100644
--- a/tests/api_connexion/endpoints/test_dag_warning_endpoint.py
+++ b/tests/api_connexion/endpoints/test_dag_warning_endpoint.py
@@ -170,3 +170,15 @@ class TestGetDagWarningEndpoint(TestBaseDagWarning):
query_string={"dag_id": "dag1"},
)
assert response.status_code == 403
+
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ response = self.client.get(
+ "/api/v1/dagWarnings",
+ query_string={"dag_id": "dag1", "warning_type": "non-existent pool"},
+ )
+ assert response.status_code == expected_status_code
diff --git a/tests/api_connexion/endpoints/test_dataset_endpoint.py b/tests/api_connexion/endpoints/test_dataset_endpoint.py
index 29192a3c65..4c3ead353c 100644
--- a/tests/api_connexion/endpoints/test_dataset_endpoint.py
+++ b/tests/api_connexion/endpoints/test_dataset_endpoint.py
@@ -143,6 +143,22 @@ class TestGetDatasetEndpoint(TestDatasetEndpoint):
response = self.client.get(f"/api/v1/datasets/{urllib.parse.quote('s3://bucket/key', safe='')}")
assert_401(response)
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ self._create_dataset(session)
+ assert session.query(DatasetModel).count() == 1
+
+ with assert_queries_count(5):
+ response = self.client.get(
+ f"/api/v1/datasets/{urllib.parse.quote('s3://bucket/key', safe='')}",
+ )
+
+ assert response.status_code == expected_status_code
+
class TestGetDatasets(TestDatasetEndpoint):
def test_should_respond_200(self, session):
@@ -313,6 +329,31 @@ class TestGetDatasets(TestDatasetEndpoint):
response_data = response.json
assert len(response_data["datasets"]) == expected_num
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ datasets = [
+ DatasetModel(
+ id=i,
+ uri=f"s3://bucket/key/{i}",
+ extra={"foo": "bar"},
+ created_at=timezone.parse(self.default_time),
+ updated_at=timezone.parse(self.default_time),
+ )
+ for i in [1, 2]
+ ]
+ session.add_all(datasets)
+ session.commit()
+ assert session.query(DatasetModel).count() == 2
+
+ with assert_queries_count(8):
+ response = self.client.get("/api/v1/datasets")
+
+ assert response.status_code == expected_status_code
+
class TestGetDatasetsEndpointPagination(TestDatasetEndpoint):
@pytest.mark.parametrize(
@@ -579,6 +620,32 @@ class TestGetDatasetEvents(TestDatasetEndpoint):
"total_entries": 1,
}
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ self._create_dataset(session)
+ common = {
+ "dataset_id": 1,
+ "extra": {"foo": "bar"},
+ "source_dag_id": "foo",
+ "source_task_id": "bar",
+ "source_run_id": "custom",
+ "source_map_index": -1,
+ "created_dagruns": [],
+ }
+
+ events = [DatasetEvent(id=i, timestamp=timezone.parse(self.default_time), **common) for i in [1, 2]]
+ session.add_all(events)
+ session.commit()
+ assert session.query(DatasetEvent).count() == 2
+
+ response = self.client.get("/api/v1/datasets/events")
+
+ assert response.status_code == expected_status_code
+
class TestPostDatasetEvents(TestDatasetEndpoint):
@pytest.fixture
@@ -650,6 +717,19 @@ class TestPostDatasetEvents(TestDatasetEndpoint):
response = self.client.post("/api/v1/datasets/events", json={"dataset_uri": "TEST_DATASET_URI"})
assert_401(response)
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ @pytest.mark.usefixtures("time_freezer")
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, session):
+ self._create_dataset(session)
+ event_payload = {"dataset_uri": "s3://bucket/key", "extra": {"foo": "bar"}}
+ response = self.client.post("/api/v1/datasets/events", json=event_payload)
+
+ assert response.status_code == expected_status_code
+
class TestGetDatasetEventsEndpointPagination(TestDatasetEndpoint):
@pytest.mark.parametrize(
@@ -820,6 +900,27 @@ class TestGetDagDatasetQueuedEvent(TestQueuedEventEndpoint):
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ @pytest.mark.usefixtures("time_freezer")
+ def test_with_auth_role_public_set(
+ self, set_auto_role_public, expected_status_code, create_dummy_dag, session
+ ):
+ dag, _ = create_dummy_dag()
+ dag_id = dag.dag_id
+ dataset_id = self._create_dataset(session).id
+ self._create_dataset_dag_run_queues(dag_id, dataset_id, session)
+ dataset_uri = "s3://bucket/key"
+
+ response = self.client.get(
+ f"/api/v1/dags/{dag_id}/datasets/queuedEvent/{dataset_uri}",
+ )
+
+ assert response.status_code == expected_status_code
+
class TestDeleteDagDatasetQueuedEvent(TestDatasetEndpoint):
def test_delete_should_respond_204(self, session, create_dummy_dag):
@@ -881,7 +982,7 @@ class TestDeleteDagDatasetQueuedEvent(TestDatasetEndpoint):
class TestGetDagDatasetQueuedEvents(TestQueuedEventEndpoint):
@pytest.mark.usefixtures("time_freezer")
- def test_should_respond_200(self, session, create_dummy_dag, time_freezer):
+ def test_should_respond_200(self, session, create_dummy_dag):
dag, _ = create_dummy_dag()
dag_id = dag.dag_id
dataset_id = self._create_dataset(session).id
@@ -937,6 +1038,24 @@ class TestGetDagDatasetQueuedEvents(TestQueuedEventEndpoint):
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(
+ self, set_auto_role_public, expected_status_code, session, create_dummy_dag
+ ):
+ dag, _ = create_dummy_dag()
+ dag_id = dag.dag_id
+ dataset_id = self._create_dataset(session).id
+ self._create_dataset_dag_run_queues(dag_id, dataset_id, session)
+
+ response = self.client.get(
+ f"/api/v1/dags/{dag_id}/datasets/queuedEvent",
+ )
+ assert response.status_code == expected_status_code
+
class TestDeleteDagDatasetQueuedEvents(TestDatasetEndpoint):
def test_should_respond_404(self):
@@ -972,6 +1091,31 @@ class TestDeleteDagDatasetQueuedEvents(TestDatasetEndpoint):
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 204)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(
+ self, set_auto_role_public, expected_status_code, session, create_dummy_dag
+ ):
+ dag, _ = create_dummy_dag()
+ dag_id = dag.dag_id
+ dataset_uri = "s3://bucket/key"
+ dataset_id = self._create_dataset(session).id
+
+ ddrq = DatasetDagRunQueue(target_dag_id=dag_id, dataset_id=dataset_id)
+ session.add(ddrq)
+ session.commit()
+ conn = session.query(DatasetDagRunQueue).all()
+ assert len(conn) == 1
+
+ response = self.client.delete(
+ f"/api/v1/dags/{dag_id}/datasets/queuedEvent/{dataset_uri}",
+ )
+
+ assert response.status_code == expected_status_code
+
class TestGetDatasetQueuedEvents(TestQueuedEventEndpoint):
@pytest.mark.usefixtures("time_freezer")
@@ -1032,6 +1176,26 @@ class TestGetDatasetQueuedEvents(TestQueuedEventEndpoint):
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ @pytest.mark.usefixtures("time_freezer")
+ def test_with_auth_role_public_set(
+ self, set_auto_role_public, expected_status_code, session, create_dummy_dag
+ ):
+ dag, _ = create_dummy_dag()
+ dag_id = dag.dag_id
+ dataset_id = self._create_dataset(session).id
+ self._create_dataset_dag_run_queues(dag_id, dataset_id, session)
+
+ response = self.client.get(
+ f"/api/v1/dags/{dag_id}/datasets/queuedEvent",
+ )
+
+ assert response.status_code == expected_status_code
+
class TestDeleteDatasetQueuedEvents(TestQueuedEventEndpoint):
def test_delete_should_respond_204(self, session, create_dummy_dag):
@@ -1083,3 +1247,23 @@ class TestDeleteDatasetQueuedEvents(TestQueuedEventEndpoint):
)
assert response.status_code == 403
+
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 204)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(
+ self, set_auto_role_public, expected_status_code, session, create_dummy_dag
+ ):
+ dag, _ = create_dummy_dag()
+ dag_id = dag.dag_id
+ dataset_id = self._create_dataset(session).id
+ self._create_dataset_dag_run_queues(dag_id, dataset_id, session)
+ dataset_uri = "s3://bucket/key"
+
+ response = self.client.delete(
+ f"/api/v1/datasets/queuedEvent/{dataset_uri}",
+ )
+
+ assert response.status_code == expected_status_code
diff --git a/tests/api_connexion/endpoints/test_event_log_endpoint.py b/tests/api_connexion/endpoints/test_event_log_endpoint.py
index 6e71a86b94..6738858ddd 100644
--- a/tests/api_connexion/endpoints/test_event_log_endpoint.py
+++ b/tests/api_connexion/endpoints/test_event_log_endpoint.py
@@ -109,6 +109,21 @@ class TestEventLogEndpoint:
def teardown_method(self) -> None:
clear_db_logs()
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, log_model):
+ event_log_id = log_model.id
+ response = self.client.get(
+ f"/api/v1/eventLogs/{event_log_id}", environ_overrides={"REMOTE_USER": "test"}
+ )
+
+ response = self.client.get("/api/v1/eventLogs")
+
+ assert response.status_code == expected_status_code
+
class TestGetEventLog(TestEventLogEndpoint):
def test_should_respond_200(self, log_model):
@@ -152,6 +167,18 @@ class TestGetEventLog(TestEventLogEndpoint):
)
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code, log_model):
+ event_log_id = log_model.id
+
+ response = self.client.get(f"/api/v1/eventLogs/{event_log_id}")
+
+ assert response.status_code == expected_status_code
+
class TestGetEventLogs(TestEventLogEndpoint):
def test_should_respond_200(self, session, create_log_model):
@@ -349,6 +376,23 @@ class TestGetEventLogs(TestEventLogEndpoint):
assert response_data["total_entries"] == 1
assert {"cli_scheduler"} == {x["event"] for x in response_data["event_logs"]}
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(
+ self, set_auto_role_public, expected_status_code, create_log_model, session
+ ):
+ log_model_3 = Log(event="cli_scheduler", owner="root", extra='{"host_name": "e24b454f002a"}')
+ log_model_3.dttm = self.default_time_2
+
+ session.add(log_model_3)
+ session.flush()
+ response = self.client.get("/api/v1/eventLogs")
+
+ assert response.status_code == expected_status_code
+
class TestGetEventLogPagination(TestEventLogEndpoint):
@pytest.mark.parametrize(
diff --git a/tests/providers/fab/auth_manager/api_endpoints/test_role_and_permission_endpoint.py b/tests/providers/fab/auth_manager/api_endpoints/test_role_and_permission_endpoint.py
index 33b85d8c18..b8970d797e 100644
--- a/tests/providers/fab/auth_manager/api_endpoints/test_role_and_permission_endpoint.py
+++ b/tests/providers/fab/auth_manager/api_endpoints/test_role_and_permission_endpoint.py
@@ -102,6 +102,15 @@ class TestGetRoleEndpoint(TestRoleEndpoint):
)
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ response = self.client.get("/auth/fab/v1/roles/Admin")
+ assert response.status_code == expected_status_code
+
class TestGetRolesEndpoint(TestRoleEndpoint):
def test_should_response_200(self):
@@ -131,6 +140,15 @@ class TestGetRolesEndpoint(TestRoleEndpoint):
)
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ response = self.client.get("/auth/fab/v1/roles")
+ assert response.status_code == expected_status_code
+
class TestGetRolesEndpointPaginationandFilter(TestRoleEndpoint):
@pytest.mark.parametrize(
@@ -184,6 +202,15 @@ class TestGetPermissionsEndpoint(TestRoleEndpoint):
)
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ response = self.client.get("/auth/fab/v1/permissions")
+ assert response.status_code == expected_status_code
+
class TestPostRole(TestRoleEndpoint):
def test_post_should_respond_200(self):
@@ -313,6 +340,19 @@ class TestPostRole(TestRoleEndpoint):
)
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ payload = {
+ "name": "Test2",
+ "actions": [{"resource": {"name": "Connections"}, "action": {"name": "can_create"}}],
+ }
+ response = self.client.post("/auth/fab/v1/roles", json=payload)
+ assert response.status_code == expected_status_code
+
class TestDeleteRole(TestRoleEndpoint):
def test_delete_should_respond_204(self, session):
@@ -347,6 +387,16 @@ class TestDeleteRole(TestRoleEndpoint):
)
assert response.status_code == 403
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 204)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ role = create_role(self.app, "mytestrole")
+ response = self.client.delete(f"/auth/fab/v1/roles/{role.name}")
+ assert response.status_code == expected_status_code
+
class TestPatchRole(TestRoleEndpoint):
@pytest.mark.parametrize(
@@ -522,3 +572,16 @@ class TestPatchRole(TestRoleEndpoint):
environ_overrides={"REMOTE_USER": "test_no_permissions"},
)
assert response.status_code == 403
+
+ @pytest.mark.parametrize(
+ "set_auto_role_public, expected_status_code",
+ (("Public", 403), ("Admin", 200)),
+ indirect=["set_auto_role_public"],
+ )
+ def test_with_auth_role_public_set(self, set_auto_role_public, expected_status_code):
+ role = create_role(self.app, "mytestrole")
+ response = self.client.patch(
+ f"/auth/fab/v1/roles/{role.name}",
+ json={"name": "mytest"},
+ )
+ assert response.status_code == expected_status_code
diff --git a/tests/providers/fab/auth_manager/conftest.py b/tests/providers/fab/auth_manager/conftest.py
index 66707ef53d..6b4feb143f 100644
--- a/tests/providers/fab/auth_manager/conftest.py
+++ b/tests/providers/fab/auth_manager/conftest.py
@@ -35,6 +35,19 @@ def minimal_app_for_auth_api():
)
def factory():
with conf_vars({("api", "auth_backends"): "tests.test_utils.remote_user_api_auth_backend"}):
- return app.create_app(testing=True, config={"WTF_CSRF_ENABLED": False}) # type:ignore
+ _app = app.create_app(testing=True, config={"WTF_CSRF_ENABLED": False}) # type:ignore
+ _app.config["AUTH_ROLE_PUBLIC"] = None
+ return _app
return factory()
+
+
+@pytest.fixture
+def set_auto_role_public(request):
+ app = request.getfixturevalue("minimal_app_for_auth_api")
+ auto_role_public = app.config["AUTH_ROLE_PUBLIC"]
+ app.config["AUTH_ROLE_PUBLIC"] = request.param
+
+ yield
+
+ app.config["AUTH_ROLE_PUBLIC"] = auto_role_public
(airflow) 01/48: Fix typo with DatasetOrTimeSchedule as it was renamed (#38576)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1d40343286864942f32f4be6a7e3789790ea2fc2
Author: Ankit Chaurasia <86...@users.noreply.github.com>
AuthorDate: Thu Mar 28 11:31:32 2024 +0545
Fix typo with DatasetOrTimeSchedule as it was renamed (#38576)
(cherry picked from commit 9a5974af48a8253d0e71bc2ab3179ee62371348b)
---
docs/apache-airflow/authoring-and-scheduling/datasets.rst | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/docs/apache-airflow/authoring-and-scheduling/datasets.rst b/docs/apache-airflow/authoring-and-scheduling/datasets.rst
index 354a69ff7e..1102420dd4 100644
--- a/docs/apache-airflow/authoring-and-scheduling/datasets.rst
+++ b/docs/apache-airflow/authoring-and-scheduling/datasets.rst
@@ -330,8 +330,6 @@ Combining Dataset and Time-Based Schedules
DatasetTimetable Integration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-With the introduction of ``DatasetTimetable``, it is now possible to schedule DAGs based on both dataset events and time-based schedules. This feature offers flexibility for scenarios where a DAG needs to be triggered by data updates as well as run periodically according to a fixed timetable.
+With the introduction of ``DatasetOrTimeSchedule``, it is now possible to schedule DAGs based on both dataset events and time-based schedules. This feature offers flexibility for scenarios where a DAG needs to be triggered by data updates as well as run periodically according to a fixed timetable.
-For more detailed information on ``DatasetTimetable`` and its usage, refer to the corresponding section in :ref:`DatasetTimetable <dataset-timetable-section>`.
-
-These examples illustrate how Airflow's conditional dataset expressions can be used to create complex data-dependent scheduling scenarios, providing precise control over when DAGs are triggered in response to data updates.
+For more detailed information on ``DatasetOrTimeSchedule`` and its usage, refer to the corresponding section in :ref:`DatasetOrTimeSchedule <dataset-timetable-section>`.
(airflow) 32/48: update document (#39068)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ed4cd93eca7f95b8723d2a21e50e52dd7f95d871
Author: humit <jh...@naver.com>
AuthorDate: Wed Apr 17 05:46:14 2024 +0900
update document (#39068)
(cherry picked from commit b56c7cd3352bf8f493e081bda895058320122b51)
---
docs/apache-airflow/extra-packages-ref.rst | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/docs/apache-airflow/extra-packages-ref.rst b/docs/apache-airflow/extra-packages-ref.rst
index 91922dd5fc..0a9a73600d 100644
--- a/docs/apache-airflow/extra-packages-ref.rst
+++ b/docs/apache-airflow/extra-packages-ref.rst
@@ -57,8 +57,6 @@ python dependencies for the provided package.
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| google-auth | ``pip install 'apache-airflow[google-auth]'`` | Google auth backend |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
-| graphviz | ``pip install 'apache-airflow[graphvis]'`` | Enables exporting DAGs to .dot graphical output |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| graphviz | ``pip install 'apache-airflow[graphviz]'`` | Graphviz renderer for converting DAG to graphical output |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| kerberos | ``pip install 'apache-airflow[kerberos]'`` | Kerberos integration for Kerberized services (Hadoop, Presto, Trino) |
@@ -333,7 +331,7 @@ pre-installed when Airflow is installed.
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| openlineage | ``pip install 'apache-airflow[openlineage]'`` | Sending OpenLineage events | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
-| opensearch | ``pip install 'apache-airflow[opensearch]'`` | Opensearch hooks and operators | |
+| opensearch | ``pip install 'apache-airflow[opensearch]'`` | Opensearch hooks and operators | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| papermill | ``pip install 'apache-airflow[papermill]'`` | Papermill hooks and operators | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
(airflow) 18/48: (external_task.py): changed the background color of external task in graph (#38969)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2e25153a522a3567e69ae602ea7bceecae921a72
Author: Idan Tepper <65...@users.noreply.github.com>
AuthorDate: Fri Apr 12 21:50:47 2024 +0300
(external_task.py): changed the background color of external task in graph (#38969)
Co-authored-by: idantepper@gmail.com <id...@github.com>
(cherry picked from commit 8fad98d34d6493483e934cd76eb82ea97bb58fd1)
---
airflow/sensors/external_task.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/airflow/sensors/external_task.py b/airflow/sensors/external_task.py
index 32f5992001..08dab8ee7b 100644
--- a/airflow/sensors/external_task.py
+++ b/airflow/sensors/external_task.py
@@ -148,7 +148,7 @@ class ExternalTaskSensor(BaseSensorOperator):
"""
template_fields = ["external_dag_id", "external_task_id", "external_task_ids", "external_task_group_id"]
- ui_color = "#19647e"
+ ui_color = "#4db7db"
operator_extra_links = [ExternalDagLink()]
def __init__(
@@ -473,7 +473,7 @@ class ExternalTaskMarker(EmptyOperator):
"""
template_fields = ["external_dag_id", "external_task_id", "execution_date"]
- ui_color = "#19647e"
+ ui_color = "#4db7db"
operator_extra_links = [ExternalDagLink()]
# The _serialized_fields are lazily loaded when get_serialized_fields() method is called
(airflow) 43/48: doc: dynamictaskmapping pythonoperator op_kwargs (#39242)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f299149da4681b680050fc5e5cf68ce667a31102
Author: raphaelauv <ra...@gmail.com>
AuthorDate: Thu Apr 25 06:40:06 2024 +0200
doc: dynamictaskmapping pythonoperator op_kwargs (#39242)
Co-authored-by: raphaelauv <ra...@users.noreply.github.com>
(cherry picked from commit 6781c632e3ba7fd4404f21c8b932e1374a4a9322)
---
.../authoring-and-scheduling/dynamic-task-mapping.rst | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/docs/apache-airflow/authoring-and-scheduling/dynamic-task-mapping.rst b/docs/apache-airflow/authoring-and-scheduling/dynamic-task-mapping.rst
index acf9cfba3d..739bc6fee9 100644
--- a/docs/apache-airflow/authoring-and-scheduling/dynamic-task-mapping.rst
+++ b/docs/apache-airflow/authoring-and-scheduling/dynamic-task-mapping.rst
@@ -291,6 +291,24 @@ Sometimes an upstream needs to specify multiple arguments to a downstream operat
This produces two task instances at run-time printing ``1`` and ``2`` respectively.
+Also it's possible to mix ``expand_kwargs`` with most of the operators arguments like the ``op_kwargs`` of the PythonOperator
+
+.. code-block:: python
+
+ def print_args(x, y):
+ print(x)
+ print(y)
+ return x + y
+
+
+ PythonOperator.partial(task_id="task-1", python_callable=print_args).expand_kwargs(
+ [
+ {"op_kwargs": {"x": 1, "y": 2}, "show_return_value_in_logs": True},
+ {"op_kwargs": {"x": 3, "y": 4}, "show_return_value_in_logs": False},
+ ]
+ )
+
+
Similar to ``expand``, you can also map against a XCom that returns a list of dicts, or a list of XComs each returning a dict. Re-using the S3 example above, you can use a mapped task to perform "branching" and copy files to different buckets:
.. code-block:: python
(airflow) 23/48: Add `inherit_cache` attribute for `CreateTableAs` custom SA Clause (#38985)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 82a1b6abdeeb225b75f53fcfa9d74ba258a4aee9
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Sun Apr 14 17:19:17 2024 +0400
Add `inherit_cache` attribute for `CreateTableAs` custom SA Clause (#38985)
(cherry picked from commit bdb8f749ee462d508bc9da43b16d18a599ff0e7b)
---
airflow/utils/db_cleanup.py | 2 ++
pyproject.toml | 1 +
2 files changed, 3 insertions(+)
diff --git a/airflow/utils/db_cleanup.py b/airflow/utils/db_cleanup.py
index 4475209b77..7fe158e605 100644
--- a/airflow/utils/db_cleanup.py
+++ b/airflow/utils/db_cleanup.py
@@ -218,6 +218,8 @@ def _subquery_keep_last(*, recency_column, keep_last_filters, group_by_columns,
class CreateTableAs(Executable, ClauseElement):
"""Custom sqlalchemy clause element for CTAS operations."""
+ inherit_cache = False
+
def __init__(self, name, query):
self.name = name
self.query = query
diff --git a/pyproject.toml b/pyproject.toml
index fbc7b7a9d7..4107396af3 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -482,6 +482,7 @@ filterwarnings = [
# Avoid building cartesian product which might impact performance
"error:SELECT statement has a cartesian product between FROM:sqlalchemy.exc.SAWarning:airflow",
'error:Coercing Subquery object into a select\(\) for use in IN\(\):sqlalchemy.exc.SAWarning:airflow',
+ 'error:Class.*will not make use of SQL compilation caching',
"ignore::DeprecationWarning:flask_appbuilder.filemanager",
"ignore::DeprecationWarning:flask_appbuilder.widgets",
# https://github.com/dpgaspar/Flask-AppBuilder/pull/1940
(airflow) 05/48: Bump undici from 5.28.3 to 5.28.4 in /airflow/www (#38751)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 16256a2514d4fff6d4ccca106edb4bc280846d6e
Author: Dependabot [bot] <49...@users.noreply.github.com>
AuthorDate: Thu Apr 4 21:15:48 2024 +0300
Bump undici from 5.28.3 to 5.28.4 in /airflow/www (#38751)
Bumps [undici](https://github.com/nodejs/undici) from 5.28.3 to 5.28.4.
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v5.28.3...v5.28.4)
---
updated-dependencies:
- dependency-name: undici
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
(cherry picked from commit ff33a05335530055adebb815e3313d43eac71b25)
---
airflow/www/yarn.lock | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/airflow/www/yarn.lock b/airflow/www/yarn.lock
index 600a4b7a3f..c5b0d6e438 100644
--- a/airflow/www/yarn.lock
+++ b/airflow/www/yarn.lock
@@ -11330,9 +11330,9 @@ unbox-primitive@^1.0.2:
which-boxed-primitive "^1.0.2"
undici@^5.4.0:
- version "5.28.3"
- resolved "https://registry.yarnpkg.com/undici/-/undici-5.28.3.tgz#a731e0eff2c3fcfd41c1169a869062be222d1e5b"
- integrity sha512-3ItfzbrhDlINjaP0duwnNsKpDQk3acHI3gVJ1z4fmwMK31k5G9OVIAMLSIaP6w4FaGkaAkN6zaQO9LUvZ1t7VA==
+ version "5.28.4"
+ resolved "https://registry.yarnpkg.com/undici/-/undici-5.28.4.tgz#6b280408edb6a1a604a9b20340f45b422e373068"
+ integrity sha512-72RFADWFqKmUb2hmmvNODKL3p9hcB6Gt2DOQMis1SEBaV6a4MH8soBvzg+95CYhCKPFedut2JY9bMfrDl9D23g==
dependencies:
"@fastify/busboy" "^2.0.0"
(airflow) 06/48: Use async db calls in WorkflowTrigger (#38689)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c6c589f5a495a19769e3274e7bfe04dde50f62a7
Author: Steven Schaerer <53...@users.noreply.github.com>
AuthorDate: Thu Apr 4 21:55:28 2024 +0200
Use async db calls in WorkflowTrigger (#38689)
* Use async db calls in WorkflowTrigger
* address PR comments
* deprecate TaskStateTrigger with proper category
(cherry picked from commit e6eec0cfad424e402fe2a03b42818e706f0685ba)
---
airflow/triggers/external_task.py | 46 ++++++-----
tests/triggers/test_external_task.py | 153 ++++++++++++++++++++++-------------
2 files changed, 122 insertions(+), 77 deletions(-)
diff --git a/airflow/triggers/external_task.py b/airflow/triggers/external_task.py
index 5c7361a15b..a5de817c35 100644
--- a/airflow/triggers/external_task.py
+++ b/airflow/triggers/external_task.py
@@ -21,8 +21,10 @@ import typing
from typing import Any
from asgiref.sync import sync_to_async
+from deprecated import deprecated
from sqlalchemy import func
+from airflow.exceptions import RemovedInAirflow3Warning
from airflow.models import DagRun, TaskInstance
from airflow.triggers.base import BaseTrigger, TriggerEvent
from airflow.utils.sensor_helper import _get_count
@@ -98,13 +100,7 @@ class WorkflowTrigger(BaseTrigger):
"""Check periodically tasks, task group or dag status."""
while True:
if self.failed_states:
- failed_count = _get_count(
- self.execution_dates,
- self.external_task_ids,
- self.external_task_group_id,
- self.external_dag_id,
- self.failed_states,
- )
+ failed_count = await self._get_count(self.failed_states)
if failed_count > 0:
yield TriggerEvent({"status": "failed"})
return
@@ -112,30 +108,38 @@ class WorkflowTrigger(BaseTrigger):
yield TriggerEvent({"status": "success"})
return
if self.skipped_states:
- skipped_count = _get_count(
- self.execution_dates,
- self.external_task_ids,
- self.external_task_group_id,
- self.external_dag_id,
- self.skipped_states,
- )
+ skipped_count = await self._get_count(self.skipped_states)
if skipped_count > 0:
yield TriggerEvent({"status": "skipped"})
return
- allowed_count = _get_count(
- self.execution_dates,
- self.external_task_ids,
- self.external_task_group_id,
- self.external_dag_id,
- self.allowed_states,
- )
+ allowed_count = await self._get_count(self.allowed_states)
if allowed_count == len(self.execution_dates):
yield TriggerEvent({"status": "success"})
return
self.log.info("Sleeping for %s seconds", self.poke_interval)
await asyncio.sleep(self.poke_interval)
+ @sync_to_async
+ def _get_count(self, states: typing.Iterable[str] | None) -> int:
+ """
+ Get the count of records against dttm filter and states. Async wrapper for _get_count.
+
+ :param states: task or dag states
+ :return The count of records.
+ """
+ return _get_count(
+ dttm_filter=self.execution_dates,
+ external_task_ids=self.external_task_ids,
+ external_task_group_id=self.external_task_group_id,
+ external_dag_id=self.external_dag_id,
+ states=states,
+ )
+
+@deprecated(
+ reason="TaskStateTrigger has been deprecated and will be removed in future.",
+ category=RemovedInAirflow3Warning,
+)
class TaskStateTrigger(BaseTrigger):
"""
Waits asynchronously for a task in a different DAG to complete for a specific logical date.
diff --git a/tests/triggers/test_external_task.py b/tests/triggers/test_external_task.py
index fe773049fa..8ce6d89a3a 100644
--- a/tests/triggers/test_external_task.py
+++ b/tests/triggers/test_external_task.py
@@ -18,11 +18,13 @@ from __future__ import annotations
import asyncio
import datetime
+import time
from unittest import mock
import pytest
from sqlalchemy.exc import SQLAlchemyError
+from airflow.exceptions import RemovedInAirflow3Warning
from airflow.models.dag import DAG
from airflow.models.dagrun import DagRun
from airflow.models.taskinstance import TaskInstance
@@ -41,11 +43,10 @@ class TestWorkflowTrigger:
STATES = ["success", "fail"]
@mock.patch("airflow.triggers.external_task._get_count")
- @mock.patch("asyncio.sleep")
@pytest.mark.asyncio
- async def test_task_workflow_trigger_success(self, mock_sleep, mock_get_count):
+ async def test_task_workflow_trigger_success(self, mock_get_count):
"""check the db count get called correctly."""
- mock_get_count.return_value = 1
+ mock_get_count.side_effect = mocked_get_count
trigger = WorkflowTrigger(
external_dag_id=self.DAG_ID,
execution_dates=[timezone.datetime(2022, 1, 1)],
@@ -54,19 +55,29 @@ class TestWorkflowTrigger:
poke_interval=0.2,
)
- generator = trigger.run()
- await generator.asend(None)
+ gen = trigger.run()
+ trigger_task = asyncio.create_task(gen.__anext__())
+ fake_task = asyncio.create_task(fake_async_fun())
+ await trigger_task
+ assert fake_task.done() # confirm that get_count is done in an async fashion
+ assert trigger_task.done()
+ result = trigger_task.result()
+ assert result.payload == {"status": "success"}
mock_get_count.assert_called_once_with(
- [timezone.datetime(2022, 1, 1)], ["external_task_op"], None, "external_task", ["success", "fail"]
+ dttm_filter=[timezone.datetime(2022, 1, 1)],
+ external_task_ids=["external_task_op"],
+ external_task_group_id=None,
+ external_dag_id="external_task",
+ states=["success", "fail"],
)
# test that it returns after yielding
with pytest.raises(StopAsyncIteration):
- await generator.__anext__()
+ await gen.__anext__()
@mock.patch("airflow.triggers.external_task._get_count")
@pytest.mark.asyncio
async def test_task_workflow_trigger_failed(self, mock_get_count):
- mock_get_count.return_value = 1
+ mock_get_count.side_effect = mocked_get_count
trigger = WorkflowTrigger(
external_dag_id=self.DAG_ID,
execution_dates=[timezone.datetime(2022, 1, 1)],
@@ -77,13 +88,19 @@ class TestWorkflowTrigger:
gen = trigger.run()
trigger_task = asyncio.create_task(gen.__anext__())
+ fake_task = asyncio.create_task(fake_async_fun())
await trigger_task
- assert trigger_task.done() is True
+ assert fake_task.done() # confirm that get_count is done in an async fashion
+ assert trigger_task.done()
result = trigger_task.result()
assert isinstance(result, TriggerEvent)
assert result.payload == {"status": "failed"}
mock_get_count.assert_called_once_with(
- [timezone.datetime(2022, 1, 1)], ["external_task_op"], None, "external_task", ["success", "fail"]
+ dttm_filter=[timezone.datetime(2022, 1, 1)],
+ external_task_ids=["external_task_op"],
+ external_task_group_id=None,
+ external_dag_id="external_task",
+ states=["success", "fail"],
)
# test that it returns after yielding
with pytest.raises(StopAsyncIteration):
@@ -104,12 +121,16 @@ class TestWorkflowTrigger:
gen = trigger.run()
trigger_task = asyncio.create_task(gen.__anext__())
await trigger_task
- assert trigger_task.done() is True
+ assert trigger_task.done()
result = trigger_task.result()
assert isinstance(result, TriggerEvent)
assert result.payload == {"status": "success"}
mock_get_count.assert_called_once_with(
- [timezone.datetime(2022, 1, 1)], ["external_task_op"], None, "external_task", ["success", "fail"]
+ dttm_filter=[timezone.datetime(2022, 1, 1)],
+ external_task_ids=["external_task_op"],
+ external_task_group_id=None,
+ external_dag_id="external_task",
+ states=["success", "fail"],
)
# test that it returns after yielding
with pytest.raises(StopAsyncIteration):
@@ -118,7 +139,7 @@ class TestWorkflowTrigger:
@mock.patch("airflow.triggers.external_task._get_count")
@pytest.mark.asyncio
async def test_task_workflow_trigger_skipped(self, mock_get_count):
- mock_get_count.return_value = 1
+ mock_get_count.side_effect = mocked_get_count
trigger = WorkflowTrigger(
external_dag_id=self.DAG_ID,
execution_dates=[timezone.datetime(2022, 1, 1)],
@@ -129,13 +150,19 @@ class TestWorkflowTrigger:
gen = trigger.run()
trigger_task = asyncio.create_task(gen.__anext__())
+ fake_task = asyncio.create_task(fake_async_fun())
await trigger_task
- assert trigger_task.done() is True
+ assert fake_task.done() # confirm that get_count is done in an async fashion
+ assert trigger_task.done()
result = trigger_task.result()
assert isinstance(result, TriggerEvent)
assert result.payload == {"status": "skipped"}
mock_get_count.assert_called_once_with(
- [timezone.datetime(2022, 1, 1)], ["external_task_op"], None, "external_task", ["success", "fail"]
+ dttm_filter=[timezone.datetime(2022, 1, 1)],
+ external_task_ids=["external_task_op"],
+ external_task_group_id=None,
+ external_dag_id="external_task",
+ states=["success", "fail"],
)
@mock.patch("airflow.triggers.external_task._get_count")
@@ -153,7 +180,7 @@ class TestWorkflowTrigger:
gen = trigger.run()
trigger_task = asyncio.create_task(gen.__anext__())
await trigger_task
- assert trigger_task.done() is True
+ assert trigger_task.done()
result = trigger_task.result()
assert isinstance(result, TriggerEvent)
assert result.payload == {"status": "success"}
@@ -222,14 +249,15 @@ class TestTaskStateTrigger:
session.add(instance)
session.commit()
- trigger = TaskStateTrigger(
- dag_id=dag.dag_id,
- task_id=instance.task_id,
- states=self.STATES,
- execution_dates=[timezone.datetime(2022, 1, 1)],
- poll_interval=0.2,
- trigger_start_time=trigger_start_time,
- )
+ with pytest.warns(RemovedInAirflow3Warning, match="TaskStateTrigger has been deprecated"):
+ trigger = TaskStateTrigger(
+ dag_id=dag.dag_id,
+ task_id=instance.task_id,
+ states=self.STATES,
+ execution_dates=[timezone.datetime(2022, 1, 1)],
+ poll_interval=0.2,
+ trigger_start_time=trigger_start_time,
+ )
task = asyncio.create_task(trigger.run().__anext__())
await asyncio.sleep(0.5)
@@ -252,14 +280,15 @@ class TestTaskStateTrigger:
trigger_start_time = utcnow()
mock_utcnow.return_value = trigger_start_time + datetime.timedelta(seconds=61)
- trigger = TaskStateTrigger(
- dag_id="dag1",
- task_id="task1",
- states=self.STATES,
- execution_dates=[timezone.datetime(2022, 1, 1)],
- poll_interval=0.2,
- trigger_start_time=trigger_start_time,
- )
+ with pytest.warns(RemovedInAirflow3Warning, match="TaskStateTrigger has been deprecated"):
+ trigger = TaskStateTrigger(
+ dag_id="dag1",
+ task_id="task1",
+ states=self.STATES,
+ execution_dates=[timezone.datetime(2022, 1, 1)],
+ poll_interval=0.2,
+ trigger_start_time=trigger_start_time,
+ )
trigger.count_running_dags = mock.AsyncMock()
trigger.count_running_dags.return_value = 0
@@ -284,14 +313,15 @@ class TestTaskStateTrigger:
trigger_start_time = utcnow()
mock_utcnow.return_value = trigger_start_time + datetime.timedelta(seconds=20)
- trigger = TaskStateTrigger(
- dag_id="dag1",
- task_id="task1",
- states=self.STATES,
- execution_dates=[timezone.datetime(2022, 1, 1)],
- poll_interval=0.2,
- trigger_start_time=trigger_start_time,
- )
+ with pytest.warns(RemovedInAirflow3Warning, match="TaskStateTrigger has been deprecated"):
+ trigger = TaskStateTrigger(
+ dag_id="dag1",
+ task_id="task1",
+ states=self.STATES,
+ execution_dates=[timezone.datetime(2022, 1, 1)],
+ poll_interval=0.2,
+ trigger_start_time=trigger_start_time,
+ )
trigger.count_running_dags = mock.AsyncMock()
trigger.count_running_dags.return_value = 0
@@ -331,14 +361,15 @@ class TestTaskStateTrigger:
trigger_start_time + datetime.timedelta(seconds=20),
]
- trigger = TaskStateTrigger(
- dag_id="dag1",
- task_id="task1",
- states=self.STATES,
- execution_dates=[timezone.datetime(2022, 1, 1)],
- poll_interval=0.2,
- trigger_start_time=trigger_start_time,
- )
+ with pytest.warns(RemovedInAirflow3Warning, match="TaskStateTrigger has been deprecated"):
+ trigger = TaskStateTrigger(
+ dag_id="dag1",
+ task_id="task1",
+ states=self.STATES,
+ execution_dates=[timezone.datetime(2022, 1, 1)],
+ poll_interval=0.2,
+ trigger_start_time=trigger_start_time,
+ )
trigger.count_running_dags = mock.AsyncMock()
trigger.count_running_dags.side_effect = [SQLAlchemyError]
@@ -358,14 +389,15 @@ class TestTaskStateTrigger:
and classpath.
"""
trigger_start_time = utcnow()
- trigger = TaskStateTrigger(
- dag_id=self.DAG_ID,
- task_id=self.TASK_ID,
- states=self.STATES,
- execution_dates=[timezone.datetime(2022, 1, 1)],
- poll_interval=5,
- trigger_start_time=trigger_start_time,
- )
+ with pytest.warns(RemovedInAirflow3Warning, match="TaskStateTrigger has been deprecated"):
+ trigger = TaskStateTrigger(
+ dag_id=self.DAG_ID,
+ task_id=self.TASK_ID,
+ states=self.STATES,
+ execution_dates=[timezone.datetime(2022, 1, 1)],
+ poll_interval=5,
+ trigger_start_time=trigger_start_time,
+ )
classpath, kwargs = trigger.serialize()
assert classpath == "airflow.triggers.external_task.TaskStateTrigger"
assert kwargs == {
@@ -438,3 +470,12 @@ class TestDagStateTrigger:
"execution_dates": [timezone.datetime(2022, 1, 1)],
"poll_interval": 5,
}
+
+
+def mocked_get_count(*args, **kwargs):
+ time.sleep(0.0001)
+ return 1
+
+
+async def fake_async_fun():
+ await asyncio.sleep(0.00005)
(airflow) 42/48: Fixed side effect of menu filtering causing disappearing menus (#39229)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 899d4e858735bda153305b268c8c06937adf841d
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Apr 24 19:09:09 2024 +0200
Fixed side effect of menu filtering causing disappearing menus (#39229)
The default implementation of filter_permitted_menu_items had a
subtle side-effect. The filtering on child items was done in-place
and modified the menu itself, so it was enough to get the same
worker serve requests for multiple users for the same menu to get
the items removed for subsequent user - even if they had permission
to see it.
Deepcopying the menu items before filtering them should fix the problem
Fixes: #39204
Fixes: #39135
(cherry picked from commit 0d2c0c5cf04ef886a8211820d0dc2f4cd8c47251)
---
airflow/auth/managers/base_auth_manager.py | 14 ++++++++--
tests/auth/managers/test_base_auth_manager.py | 40 +++++++++++++++++++++++++++
2 files changed, 51 insertions(+), 3 deletions(-)
diff --git a/airflow/auth/managers/base_auth_manager.py b/airflow/auth/managers/base_auth_manager.py
index 7bb4e92889..44fc53a66e 100644
--- a/airflow/auth/managers/base_auth_manager.py
+++ b/airflow/auth/managers/base_auth_manager.py
@@ -21,6 +21,7 @@ from abc import abstractmethod
from functools import cached_property
from typing import TYPE_CHECKING, Container, Literal, Sequence
+from flask_appbuilder.menu import MenuItem
from sqlalchemy import select
from airflow.auth.managers.models.resource_details import (
@@ -34,7 +35,6 @@ from airflow.utils.session import NEW_SESSION, provide_session
if TYPE_CHECKING:
from flask import Blueprint
- from flask_appbuilder.menu import MenuItem
from sqlalchemy.orm import Session
from airflow.auth.managers.models.base_user import BaseUser
@@ -397,13 +397,21 @@ class BaseAuthManager(LoggingMixin):
)
accessible_items = []
for menu_item in items:
+ menu_item_copy = MenuItem(
+ name=menu_item.name,
+ icon=menu_item.icon,
+ label=menu_item.label,
+ childs=[],
+ baseview=menu_item.baseview,
+ cond=menu_item.cond,
+ )
if menu_item.childs:
accessible_children = []
for child in menu_item.childs:
if self.security_manager.has_access(ACTION_CAN_ACCESS_MENU, child.name):
accessible_children.append(child)
- menu_item.childs = accessible_children
- accessible_items.append(menu_item)
+ menu_item_copy.childs = accessible_children
+ accessible_items.append(menu_item_copy)
return accessible_items
@abstractmethod
diff --git a/tests/auth/managers/test_base_auth_manager.py b/tests/auth/managers/test_base_auth_manager.py
index 64d33f6065..a39b60787c 100644
--- a/tests/auth/managers/test_base_auth_manager.py
+++ b/tests/auth/managers/test_base_auth_manager.py
@@ -313,3 +313,43 @@ class TestBaseAuthManager:
assert result[1].name == "item3"
assert len(result[1].childs) == 1
assert result[1].childs[0].name == "item3.1"
+
+ @patch.object(EmptyAuthManager, "security_manager")
+ def test_filter_permitted_menu_items_twice(self, mock_security_manager, auth_manager):
+ mock_security_manager.has_access.side_effect = [
+ # 1st call
+ True, # menu 1
+ False, # menu 2
+ True, # menu 3
+ True, # Item 3.1
+ False, # Item 3.2
+ # 2nd call
+ False, # menu 1
+ True, # menu 2
+ True, # menu 3
+ False, # Item 3.1
+ True, # Item 3.2
+ ]
+
+ menu = Menu()
+ menu.add_link("item1")
+ menu.add_link("item2")
+ menu.add_link("item3")
+ menu.add_link("item3.1", category="item3")
+ menu.add_link("item3.2", category="item3")
+
+ result = auth_manager.filter_permitted_menu_items(menu.get_list())
+
+ assert len(result) == 2
+ assert result[0].name == "item1"
+ assert result[1].name == "item3"
+ assert len(result[1].childs) == 1
+ assert result[1].childs[0].name == "item3.1"
+
+ result = auth_manager.filter_permitted_menu_items(menu.get_list())
+
+ assert len(result) == 2
+ assert result[0].name == "item2"
+ assert result[1].name == "item3"
+ assert len(result[1].childs) == 1
+ assert result[1].childs[0].name == "item3.2"
(airflow) 31/48: Data aware scheduling docs edits (#38687)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0602de222202eef86e7c7c49da77d4937f7c9956
Author: Laura Zdanski <25...@users.noreply.github.com>
AuthorDate: Tue Apr 16 08:43:35 2024 -0400
Data aware scheduling docs edits (#38687)
(cherry picked from commit d82246a0a78f451184f2d8cce1fb8f9ae122c31f)
---
.../authoring-and-scheduling/datasets.rst | 63 +++++++++++-----------
1 file changed, 31 insertions(+), 32 deletions(-)
diff --git a/docs/apache-airflow/authoring-and-scheduling/datasets.rst b/docs/apache-airflow/authoring-and-scheduling/datasets.rst
index 1102420dd4..641f1ca1d6 100644
--- a/docs/apache-airflow/authoring-and-scheduling/datasets.rst
+++ b/docs/apache-airflow/authoring-and-scheduling/datasets.rst
@@ -23,7 +23,7 @@ Data-aware scheduling
Quickstart
----------
-In addition to scheduling DAGs based upon time, they can also be scheduled based upon a task updating a dataset.
+In addition to scheduling DAGs based on time, you can also schedule DAGs to run based on when a task updates a dataset.
.. code-block:: python
@@ -51,9 +51,9 @@ In addition to scheduling DAGs based upon time, they can also be scheduled based
What is a "dataset"?
--------------------
-An Airflow dataset is a stand-in for a logical grouping of data. Datasets may be updated by upstream "producer" tasks, and dataset updates contribute to scheduling downstream "consumer" DAGs.
+An Airflow dataset is a logical grouping of data. Upstream producer tasks can update datasets, and dataset updates contribute to scheduling downstream consumer DAGs.
-A dataset is defined by a Uniform Resource Identifier (URI):
+`Uniform Resource Identifier (URI) <https://en.wikipedia.org/wiki/Uniform_Resource_Identifier>`_ define datasets:
.. code-block:: python
@@ -61,20 +61,20 @@ A dataset is defined by a Uniform Resource Identifier (URI):
example_dataset = Dataset("s3://dataset-bucket/example.csv")
-Airflow makes no assumptions about the content or location of the data represented by the URI. It is treated as a string, so any use of regular expressions (eg ``input_\d+.csv``) or file glob patterns (eg ``input_2022*.csv``) as an attempt to create multiple datasets from one declaration will not work.
+Airflow makes no assumptions about the content or location of the data represented by the URI, and treats the URI like a string. This means that Airflow treats any regular expressions, like ``input_\d+.csv``, or file glob patterns, such as ``input_2022*.csv``, as an attempt to create multiple datasets from one declaration, and they will not work.
-A dataset should be created with a valid URI. Airflow core and providers define various URI schemes that you can use, such as ``file`` (core), ``postgres`` (by the Postgres provider), and ``s3`` (by the Amazon provider). Third-party providers and plugins may also provide their own schemes. These pre-defined schemes have individual semantics that are expected to be followed.
+You must create datasets with a valid URI. Airflow core and providers define various URI schemes that you can use, such as ``file`` (core), ``postgres`` (by the Postgres provider), and ``s3`` (by the Amazon provider). Third-party providers and plugins might also provide their own schemes. These pre-defined schemes have individual semantics that are expected to be followed.
What is valid URI?
------------------
-Technically, the URI must conform to the valid character set in RFC 3986. If you don't know what this means, that's basically ASCII alphanumeric characters, plus ``%``, ``-``, ``_``, ``.``, and ``~``. To identify a resource that cannot be represented by URI-safe characters, encode the resource name with `percent-encoding <https://en.wikipedia.org/wiki/Percent-encoding>`_.
+Technically, the URI must conform to the valid character set in RFC 3986, which is basically ASCII alphanumeric characters, plus ``%``, ``-``, ``_``, ``.``, and ``~``. To identify a resource that cannot be represented by URI-safe characters, encode the resource name with `percent-encoding <https://en.wikipedia.org/wiki/Percent-encoding>`_.
The URI is also case sensitive, so ``s3://example/dataset`` and ``s3://Example/Dataset`` are considered different. Note that the *host* part of the URI is also case sensitive, which differs from RFC 3986.
Do not use the ``airflow`` scheme, which is is reserved for Airflow's internals.
-Airflow always prefers using lower cases in schemes, and case sensitivity is needed in the host part to correctly distinguish between resources.
+Airflow always prefers using lower cases in schemes, and case sensitivity is needed in the host part of the URI to correctly distinguish between resources.
.. code-block:: python
@@ -82,7 +82,7 @@ Airflow always prefers using lower cases in schemes, and case sensitivity is nee
reserved = Dataset("airflow://example_dataset")
not_ascii = Dataset("èxample_datašet")
-If you wish to define datasets with a scheme without additional semantic constraints, use a scheme with the prefix ``x-``. Airflow will skip any semantic validation on URIs with such schemes.
+If you want to define datasets with a scheme that doesn't include additional semantic constraints, use a scheme with the prefix ``x-``. Airflow skips any semantic validation on URIs with these schemes.
.. code-block:: python
@@ -99,10 +99,10 @@ The identifier does not have to be absolute; it can be a scheme-less, relative U
Non-absolute identifiers are considered plain strings that do not carry any semantic meanings to Airflow.
-Extra information
------------------
+Extra information on dataset
+----------------------------
-If needed, an extra dictionary can be included in a Dataset:
+If needed, you can include an extra dictionary in a dataset:
.. code-block:: python
@@ -128,12 +128,12 @@ This extra information does not affect a dataset's identity. This means a DAG wi
...,
)
-.. note:: **Security Note:** Dataset URI and extra fields are not encrypted, they are stored in cleartext, in Airflow's metadata database. Do NOT store any sensitive values, especially credentials, in dataset URIs or extra key values!
+.. note:: **Security Note:** Dataset URI and extra fields are not encrypted, they are stored in cleartext in Airflow's metadata database. Do NOT store any sensitive values, especially credentials, in either dataset URIs or extra key values!
How to use datasets in your DAGs
--------------------------------
-You can use datasets to specify data dependencies in your DAGs. Take the following example:
+You can use datasets to specify data dependencies in your DAGs. The following example shows how after the ``producer`` task in the ``producer`` DAG successfully completes, Airflow schedules the ``consumer`` DAG. Airflow marks a dataset as ``updated`` only if the task completes successfully. If the task fails or if it is skipped, no update occurs, and Airflow doesn't schedule the ``consumer`` DAG.
.. code-block:: python
@@ -145,15 +145,14 @@ You can use datasets to specify data dependencies in your DAGs. Take the followi
with DAG(dag_id="consumer", schedule=[example_dataset], ...):
...
-Once the ``producer`` task in the ``producer`` DAG has completed successfully, Airflow schedules the ``consumer`` DAG. A dataset will be marked as updated only if the task completes successfully — if the task fails or if it is skipped, no update occurs, and the ``consumer`` DAG will not be scheduled.
-A listing of the relationships between datasets and DAGs can be found in the
+You can find a listing of the relationships between datasets and DAGs in the
:ref:`Datasets View<ui:datasets-view>`
Multiple Datasets
-----------------
-As the ``schedule`` parameter is a list, DAGs can require multiple datasets, and the DAG will be scheduled once **all** datasets it consumes have been updated at least once since the last time it was run:
+Because the ``schedule`` parameter is a list, DAGs can require multiple datasets. Airflow schedules a DAG after **all** datasets the DAG consumes have been updated at least once since the last time the DAG ran:
.. code-block:: python
@@ -169,7 +168,7 @@ As the ``schedule`` parameter is a list, DAGs can require multiple datasets, and
...
-If one dataset is updated multiple times before all consumed datasets have been updated, the downstream DAG will still only be run once, as shown in this illustration:
+If one dataset is updated multiple times before all consumed datasets update, the downstream DAG still only runs once, as shown in this illustration:
.. ::
ASCII art representation of this diagram
@@ -224,10 +223,10 @@ If one dataset is updated multiple times before all consumed datasets have been
}
-Fetching information from a Triggering Dataset Event
+Fetching information from a triggering dataset event
----------------------------------------------------
-A triggered DAG can fetch information from the Dataset that triggered it using the ``triggering_dataset_events`` template or parameter.
+A triggered DAG can fetch information from the dataset that triggered it using the ``triggering_dataset_events`` template or parameter.
See more at :ref:`templates-ref`.
Example:
@@ -261,27 +260,27 @@ Example:
print_triggering_dataset_events()
-Note that this example is using `(.values() | first | first) <https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.first>`_ to fetch the first of one Dataset given to the DAG, and the first of one DatasetEvent for that Dataset. An implementation may be quite complex if you have multiple Datasets, potentially with multiple DatasetEvents.
+Note that this example is using `(.values() | first | first) <https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.first>`_ to fetch the first of one dataset given to the DAG, and the first of one DatasetEvent for that dataset. An implementation can be quite complex if you have multiple datasets, potentially with multiple DatasetEvents.
-Advanced Dataset Scheduling with Conditional Expressions
+Advanced dataset scheduling with conditional expressions
--------------------------------------------------------
-Apache Airflow introduces advanced scheduling capabilities that leverage conditional expressions with datasets. This feature allows Airflow users to define complex dependencies for DAG executions based on dataset updates, using logical operators for more granular control over workflow triggers.
+Apache Airflow includes advanced scheduling capabilities that use conditional expressions with datasets. This feature allows you to define complex dependencies for DAG executions based on dataset updates, using logical operators for more control on workflow triggers.
-Logical Operators for Datasets
+Logical operators for datasets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Airflow supports two logical operators for combining dataset conditions:
- **AND (``&``)**: Specifies that the DAG should be triggered only after all of the specified datasets have been updated.
-- **OR (``|``)**: Specifies that the DAG should be triggered when any one of the specified datasets is updated.
+- **OR (``|``)**: Specifies that the DAG should be triggered when any of the specified datasets is updated.
-These operators enable the expression of complex dataset update conditions, enhancing the dynamism and flexibility of Airflow workflows.
+These operators enable you to configure your Airflow workflows to use more complex dataset update conditions, making them more dynamic and flexible.
-Example Usage
+Example Use
-------------
-**Scheduling Based on Multiple Dataset Updates**
+**Scheduling based on multiple dataset updates**
To schedule a DAG to run only when two specific datasets have both been updated, use the AND operator (``&``):
@@ -297,9 +296,9 @@ To schedule a DAG to run only when two specific datasets have both been updated,
):
...
-**Scheduling Based on Any Dataset Update**
+**Scheduling based on any dataset update**
-To trigger a DAG execution when either of two datasets is updated, apply the OR operator (``|``):
+To trigger a DAG execution when either one of two datasets is updated, apply the OR operator (``|``):
.. code-block:: python
@@ -325,11 +324,11 @@ For scenarios requiring more intricate conditions, such as triggering a DAG when
):
...
-Combining Dataset and Time-Based Schedules
+Combining dataset and time-based schedules
------------------------------------------
DatasetTimetable Integration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-With the introduction of ``DatasetOrTimeSchedule``, it is now possible to schedule DAGs based on both dataset events and time-based schedules. This feature offers flexibility for scenarios where a DAG needs to be triggered by data updates as well as run periodically according to a fixed timetable.
+You can schedule DAGs based on both dataset events and time-based schedules using ``DatasetOrTimeSchedule``. This allows you to create workflows when a DAG needs both to be triggered by data updates and run periodically according to a fixed timetable.
-For more detailed information on ``DatasetOrTimeSchedule`` and its usage, refer to the corresponding section in :ref:`DatasetOrTimeSchedule <dataset-timetable-section>`.
+For more detailed information on ``DatasetOrTimeSchedule``, refer to the corresponding section in :ref:`DatasetOrTimeSchedule <dataset-timetable-section>`.
(airflow) 10/48: Remove a couple items from the 2.9.0 changelog (#38866)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 672f8ad02aef0d74332d877abc0bb36d490ccab7
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Tue Apr 9 13:56:09 2024 -0400
Remove a couple items from the 2.9.0 changelog (#38866)
One isn't a core change and one was reverted.
(cherry picked from commit 6027aa58f5ef23ad2b6bd20e649447e237a0c0d7)
---
RELEASE_NOTES.rst | 2 --
airflow/reproducible_build.yaml | 4 ++--
2 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst
index 5589ae34b6..5533fad32c 100644
--- a/RELEASE_NOTES.rst
+++ b/RELEASE_NOTES.rst
@@ -127,7 +127,6 @@ New Features
- Introduce mechanism to support multiple executor configuration (#37635)
- Add color formatting for ANSI chars in logs from task executions (#37985)
- Add the dataset_expression as part of DagModel and DAGDetailSchema (#37826)
-- Add TaskFail entries to Gantt chart (#37918)
- Allow longer rendered_map_index (#37798)
- Inherit the run_ordering from DatasetTriggeredTimetable for DatasetOrTimeSchedule (#37775)
- Implement AIP-60 Dataset URI formats (#37005)
@@ -150,7 +149,6 @@ New Features
- Create new Metrics with Tagging (#36528)
- Add support for openlineage to AFS and common.io (#36410)
- Introduce ``@task.bash`` TaskFlow decorator (#30176, #37875)
-- Added functionality to automatically ingest custom airflow.cfg file upon startup (#36289)
Improvements
""""""""""""
diff --git a/airflow/reproducible_build.yaml b/airflow/reproducible_build.yaml
index 918f75fb1d..fc6a3933e8 100644
--- a/airflow/reproducible_build.yaml
+++ b/airflow/reproducible_build.yaml
@@ -1,2 +1,2 @@
-release-notes-hash: 54dac17350ee872bcd60731a6392d9f7
-source-date-epoch: 1712436084
+release-notes-hash: 416d01241f2b6ed259e8d991fb7ac1f8
+source-date-epoch: 1712672348
(airflow) 03/48: Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods (#37757)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c501d3a1f6e3743566a34c20a4e6514710e48df6
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Wed Apr 3 12:42:16 2024 +0400
Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods (#37757)
* Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods
* Rename pre-commit script
(cherry picked from commit c65b08399d11410795b470640eb4fe6f42748c36)
---
.pre-commit-config.yaml | 7 +++
airflow/compat/functools.pyi | 27 ----------
airflow/models/abstractoperator.py | 4 +-
airflow/models/mappedoperator.py | 8 +--
airflow/utils/task_group.py | 4 +-
airflow/utils/weight_rule.py | 4 +-
contributing-docs/08_static_code_checks.rst | 2 +
dev/breeze/doc/images/output_static-checks.svg | 26 ++++-----
dev/breeze/doc/images/output_static-checks.txt | 2 +-
dev/breeze/src/airflow_breeze/pre_commit_ids.py | 1 +
hatch_build.py | 1 +
pyproject.toml | 3 +-
scripts/ci/pre_commit/compat_cache_on_methods.py | 69 ++++++++++++++++++++++++
13 files changed, 106 insertions(+), 52 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index db7325ef13..b2f998afa9 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -980,6 +980,13 @@ repos:
files: \.py$
exclude: ^.*/.*_vendor/
additional_dependencies: ['rich>=12.4.4']
+ - id: check-compat-cache-on-methods
+ name: Check that compat cache do not use on class methods
+ entry: ./scripts/ci/pre_commit/compat_cache_on_methods.py
+ language: python
+ pass_filenames: true
+ files: ^airflow/.*\.py$
+ exclude: ^.*/.*_vendor/
- id: lint-chart-schema
name: Lint chart/values.schema.json file
entry: ./scripts/ci/pre_commit/chart_schema.py
diff --git a/airflow/compat/functools.pyi b/airflow/compat/functools.pyi
deleted file mode 100644
index 7d1bef5939..0000000000
--- a/airflow/compat/functools.pyi
+++ /dev/null
@@ -1,27 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# This stub exists to work around false linter errors due to python/mypy#10408.
-# TODO: Remove this file after the upstream fix is available in our toolchain.
-from __future__ import annotations
-
-from typing import TypeVar
-
-T = TypeVar("T")
-
-def cache(f: T) -> T: ...
diff --git a/airflow/models/abstractoperator.py b/airflow/models/abstractoperator.py
index 380c6fcf00..5d9469749c 100644
--- a/airflow/models/abstractoperator.py
+++ b/airflow/models/abstractoperator.py
@@ -23,9 +23,9 @@ from abc import abstractproperty
from functools import cached_property
from typing import TYPE_CHECKING, Any, Callable, ClassVar, Collection, Iterable, Iterator, Sequence
+import methodtools
from sqlalchemy import select
-from airflow.compat.functools import cache
from airflow.configuration import conf
from airflow.exceptions import AirflowException
from airflow.models.expandinput import NotFullyPopulated
@@ -492,7 +492,7 @@ class AbstractOperator(Templater, DAGNode):
return link.get_link(self.unmap(None), ti.dag_run.logical_date) # type: ignore[misc]
return link.get_link(self.unmap(None), ti_key=ti.key)
- @cache
+ @methodtools.lru_cache(maxsize=None)
def get_parse_time_mapped_ti_count(self) -> int:
"""
Return the number of mapped task instances that can be created on DAG run creation.
diff --git a/airflow/models/mappedoperator.py b/airflow/models/mappedoperator.py
index 994e041d9f..ba5e8c174d 100644
--- a/airflow/models/mappedoperator.py
+++ b/airflow/models/mappedoperator.py
@@ -24,8 +24,8 @@ import warnings
from typing import TYPE_CHECKING, Any, ClassVar, Collection, Iterable, Iterator, Mapping, Sequence, Union
import attr
+import methodtools
-from airflow.compat.functools import cache
from airflow.exceptions import AirflowException, UnmappableOperator
from airflow.models.abstractoperator import (
DEFAULT_IGNORE_FIRST_DEPENDS_ON_PAST,
@@ -334,8 +334,8 @@ class MappedOperator(AbstractOperator):
f"{self.task_id!r}."
)
+ @methodtools.lru_cache(maxsize=None)
@classmethod
- @cache
def get_serialized_fields(cls):
# Not using 'cls' here since we only want to serialize base fields.
return frozenset(attr.fields_dict(MappedOperator)) - {
@@ -351,8 +351,8 @@ class MappedOperator(AbstractOperator):
"_on_failure_fail_dagrun",
}
+ @methodtools.lru_cache(maxsize=None)
@staticmethod
- @cache
def deps_for(operator_class: type[BaseOperator]) -> frozenset[BaseTIDep]:
operator_deps = operator_class.deps
if not isinstance(operator_deps, collections.abc.Set):
@@ -784,7 +784,7 @@ class MappedOperator(AbstractOperator):
for operator, _ in XComArg.iter_xcom_references(self._get_specified_expand_input()):
yield operator
- @cache
+ @methodtools.lru_cache(maxsize=None)
def get_parse_time_mapped_ti_count(self) -> int:
current_count = self._get_specified_expand_input().get_parse_time_mapped_ti_count()
try:
diff --git a/airflow/utils/task_group.py b/airflow/utils/task_group.py
index 5f79600c47..50ce7b4089 100644
--- a/airflow/utils/task_group.py
+++ b/airflow/utils/task_group.py
@@ -25,9 +25,9 @@ import operator
import weakref
from typing import TYPE_CHECKING, Any, Generator, Iterator, Sequence
+import methodtools
import re2
-from airflow.compat.functools import cache
from airflow.exceptions import (
AirflowDagCycleException,
AirflowException,
@@ -586,7 +586,7 @@ class MappedTaskGroup(TaskGroup):
for op, _ in XComArg.iter_xcom_references(self._expand_input):
yield op
- @cache
+ @methodtools.lru_cache(maxsize=None)
def get_parse_time_mapped_ti_count(self) -> int:
"""
Return the Number of instances a task in this group should be mapped to, when a DAG run is created.
diff --git a/airflow/utils/weight_rule.py b/airflow/utils/weight_rule.py
index f65f2fa77e..a63358b032 100644
--- a/airflow/utils/weight_rule.py
+++ b/airflow/utils/weight_rule.py
@@ -19,7 +19,7 @@ from __future__ import annotations
from enum import Enum
-from airflow.compat.functools import cache
+import methodtools
class WeightRule(str, Enum):
@@ -34,8 +34,8 @@ class WeightRule(str, Enum):
"""Check if weight rule is valid."""
return weight_rule in cls.all_weight_rules()
+ @methodtools.lru_cache(maxsize=None)
@classmethod
- @cache
def all_weight_rules(cls) -> set[str]:
"""Return all weight rules."""
return set(cls.__members__.values())
diff --git a/contributing-docs/08_static_code_checks.rst b/contributing-docs/08_static_code_checks.rst
index 0b331bf3e9..b02c8749a7 100644
--- a/contributing-docs/08_static_code_checks.rst
+++ b/contributing-docs/08_static_code_checks.rst
@@ -146,6 +146,8 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-cncf-k8s-only-for-executors | Check cncf.kubernetes imports used for executors only | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
+| check-compat-cache-on-methods | Check that compat cache do not use on class methods | |
++-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-core-deprecation-classes | Verify usage of Airflow deprecation classes in core | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-daysago-import-from-utils | Make sure days_ago is imported from airflow.utils.dates | |
diff --git a/dev/breeze/doc/images/output_static-checks.svg b/dev/breeze/doc/images/output_static-checks.svg
index 679db3dfee..a4c1734c30 100644
--- a/dev/breeze/doc/images/output_static-checks.svg
+++ b/dev/breeze/doc/images/output_static-checks.svg
@@ -319,19 +319,19 @@
</text><text class="breeze-static-checks-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-static-checks-line-10)">│</text><text class="breeze-static-checks-r7" x="451.4" y="264" textLength="988.2" clip-path="url(#breeze-static-checks-line-10)">check-base-operator-partial-arguments | check-base-operator-usage |              </text><text class="breeze-static-checks-r5" x="1451.8" y="264" textLeng [...]
</text><text class="breeze-static-checks-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-11)">│</text><text class="breeze-static-checks-r7" x="451.4" y="288.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-11)">check-boring-cyborg-configuration | check-breeze-top-dependencies-limited |      </text><text class="breeze-static-checks-r5" x="1451.8" y="288.4" textLength="12.2" clip-path="url(#breeze-s [...]
</text><text class="breeze-static-checks-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-12)">│</text><text class="breeze-static-checks-r7" x="451.4" y="312.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-12)">check-builtin-literals | check-changelog-has-no-duplicates |                     </text><text class="breeze-static [...]
-</text><text class="breeze-static-checks-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-13)">│</text><text class="breeze-static-checks-r7" x="451.4" y="337.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-13)">check-cncf-k8s-only-for-executors | check-core-deprecation-classes |             </text><text class="breeze-static-checks-r5" x="1451.8" y="337.2" textLen [...]
-</text><text class="breeze-static-checks-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-14)">│</text><text class="breeze-static-checks-r7" x="451.4" y="361.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-14)">check-daysago-import-from-utils | check-decorated-operator-implements-custom-name</text><text class="breeze-static-checks-r5" x="1451.8" y="361.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-14)">│</text><tex [...]
-</text><text class="breeze-static-checks-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-static-checks-line-15)">│</text><text class="breeze-static-checks-r7" x="451.4" y="386" textLength="988.2" clip-path="url(#breeze-static-checks-line-15)">| check-deferrable-default-value | check-docstring-param-types |                 </text><text class="breeze-static-checks-r5" x="1451 [...]
-</text><text class="breeze-static-checks-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-16)">│</text><text class="breeze-static-checks-r7" x="451.4" y="410.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-16)">check-example-dags-urls | check-executables-have-shebangs |                      </text><text class="breeze-s [...]
-</text><text class="breeze-static-checks-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-17)">│</text><text class="breeze-static-checks-r7" x="451.4" y="434.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-17)">check-extra-packages-references | check-extras-order | check-fab-migrations |    </text><text class="breeze-static-checks-r5" x="1451.8" y="434.8" textLength="12.2" clip-path="url(#breeze-s [...]
-</text><text class="breeze-static-checks-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-18)">│</text><text class="breeze-static-checks-r7" x="451.4" y="459.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-18)">check-for-inclusive-language | check-google-re2-as-dependency |                  </text><text class="breeze-static-checks-r5" x=" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-19)">│</text><text class="breeze-static-checks-r7" x="451.4" y="483.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-19)">check-hatch-build-order | check-hooks-apply | check-incorrect-use-of-LoggingMixin</text><text class="breeze-static-checks-r5" x="1451.8" y="483.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-19)">│< [...]
-</text><text class="breeze-static-checks-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-static-checks-line-20)">│</text><text class="breeze-static-checks-r7" x="451.4" y="508" textLength="988.2" clip-path="url(#breeze-static-checks-line-20)">| check-init-decorator-arguments | check-integrations-list-consistent |          </text><text class="breeze-static-checks-r5" x="1451.8" y="508" textLength="12.2" clip- [...]
-</text><text class="breeze-static-checks-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-21)">│</text><text class="breeze-static-checks-r7" x="451.4" y="532.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-21)">check-lazy-logging | check-links-to-example-dags-do-not-use-hardcoded-versions | </text><text class="breeze-static-checks-r5" x="1451.8" y="532.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-21)">│< [...]
-</text><text class="breeze-static-checks-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-22)">│</text><text class="breeze-static-checks-r7" x="451.4" y="556.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-22)">check-merge-conflict | check-newsfragments-are-valid |                           </t [...]
-</text><text class="breeze-static-checks-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-23)">│</text><text class="breeze-static-checks-r7" x="451.4" y="581.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-23)">check-no-airflow-deprecation-in-providers | check-no-providers-in-core-examples |</text><text class="breeze-static-checks-r5" x="1451.8" y="581.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-23)">│</text [...]
-</text><text class="breeze-static-checks-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-24)">│</text><text class="breeze-static-checks-r7" x="451.4" y="605.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-24)">check-only-new-session-with-provide-session |                               [...]
-</text><text class="breeze-static-checks-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-static-checks-line-25)">│</text><text class="breeze-static-checks-r7" x="451.4" y="630" textLength="988.2" clip-path="url(#breeze-static-checks-line-25)">check-persist-credentials-disabled-in-github-workflows |                         </text><text class="breeze- [...]
+</text><text class="breeze-static-checks-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-13)">│</text><text class="breeze-static-checks-r7" x="451.4" y="337.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-13)">check-cncf-k8s-only-for-executors | check-compat-cache-on-methods |              </text><text class="breeze-static-checks-r5" x="1451.8" y="337.2" te [...]
+</text><text class="breeze-static-checks-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-14)">│</text><text class="breeze-static-checks-r7" x="451.4" y="361.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-14)">check-core-deprecation-classes | check-daysago-import-from-utils |               </text><text class="breeze-static-checks-r5" x="1451.8" y="361. [...]
+</text><text class="breeze-static-checks-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-static-checks-line-15)">│</text><text class="breeze-static-checks-r7" x="451.4" y="386" textLength="988.2" clip-path="url(#breeze-static-checks-line-15)">check-decorated-operator-implements-custom-name | check-deferrable-default-value </text><text class="breeze-static-checks-r5" x="1451.8" y="386" textLength="12.2" clip-path="url(#breeze-static-checks-line-15)">│</text><text [...]
+</text><text class="breeze-static-checks-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-16)">│</text><text class="breeze-static-checks-r7" x="451.4" y="410.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-16)">| check-docstring-param-types | check-example-dags-urls |                        </text><text [...]
+</text><text class="breeze-static-checks-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-17)">│</text><text class="breeze-static-checks-r7" x="451.4" y="434.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-17)">check-executables-have-shebangs | check-extra-packages-references |              </text><text class="breeze-static-checks-r5" x="1451.8" y="434.8" te [...]
+</text><text class="breeze-static-checks-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-18)">│</text><text class="breeze-static-checks-r7" x="451.4" y="459.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-18)">check-extras-order | check-fab-migrations | check-for-inclusive-language |       </text><text class="breeze-static-checks-r5" x="1451.8" y="459.2" textLength="12.2" clip-path [...]
+</text><text class="breeze-static-checks-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-19)">│</text><text class="breeze-static-checks-r7" x="451.4" y="483.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-19)">check-google-re2-as-dependency | check-hatch-build-order | check-hooks-apply |   </text><text class="breeze-static-checks-r5" x="1451.8" y="483.6" textLength="12.2" clip-path="url(#breeze-static [...]
+</text><text class="breeze-static-checks-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-static-checks-line-20)">│</text><text class="breeze-static-checks-r7" x="451.4" y="508" textLength="988.2" clip-path="url(#breeze-static-checks-line-20)">check-incorrect-use-of-LoggingMixin | check-init-decorator-arguments |           </text><text class="breeze-static-checks-r5" x="1451.8" y="508" textLength="12.2" clip- [...]
+</text><text class="breeze-static-checks-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-21)">│</text><text class="breeze-static-checks-r7" x="451.4" y="532.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-21)">check-integrations-list-consistent | check-lazy-logging |                        </text><text class [...]
+</text><text class="breeze-static-checks-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-22)">│</text><text class="breeze-static-checks-r7" x="451.4" y="556.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-22)">check-links-to-example-dags-do-not-use-hardcoded-versions | check-merge-conflict </text><text class="breeze-static-checks-r5" x="1451.8" y="556.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-22)">│</text [...]
+</text><text class="breeze-static-checks-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-23)">│</text><text class="breeze-static-checks-r7" x="451.4" y="581.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-23)">| check-newsfragments-are-valid | check-no-airflow-deprecation-in-providers |    </text><text class="breeze-static-checks-r5" x="1451.8" y="581.2" textLength="12.2" clip-path="url(#breeze-static [...]
+</text><text class="breeze-static-checks-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-24)">│</text><text class="breeze-static-checks-r7" x="451.4" y="605.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-24)">check-no-providers-in-core-examples | check-only-new-session-with-provide-session</text><text class="breeze-static-checks-r5" x="1451.8" y="605.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-24)">│</text><tex [...]
+</text><text class="breeze-static-checks-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-static-checks-line-25)">│</text><text class="breeze-static-checks-r7" x="451.4" y="630" textLength="988.2" clip-path="url(#breeze-static-checks-line-25)">| check-persist-credentials-disabled-in-github-workflows |                       </text><text class="breeze-stati [...]
</text><text class="breeze-static-checks-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-26)">│</text><text class="breeze-static-checks-r7" x="451.4" y="654.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-26)">check-pre-commit-information-consistent | check-provide-create-sessions-imports |</text><text class="breeze-static-checks-r5" x="1451.8" y="654.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-26)">│</text [...]
</text><text class="breeze-static-checks-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-27)">│</text><text class="breeze-static-checks-r7" x="451.4" y="678.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-27)">check-provider-docs-valid | check-provider-yaml-valid |                          </text>< [...]
</text><text class="breeze-static-checks-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-28)">│</text><text class="breeze-static-checks-r7" x="451.4" y="703.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-28)">check-providers-init-file-missing | check-providers-subpackages-init-file-exist |</text><text class="breeze-static-checks-r5" x="1451.8" y="703.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-28)">│</text [...]
diff --git a/dev/breeze/doc/images/output_static-checks.txt b/dev/breeze/doc/images/output_static-checks.txt
index 66ca569ffb..ebec23568e 100644
--- a/dev/breeze/doc/images/output_static-checks.txt
+++ b/dev/breeze/doc/images/output_static-checks.txt
@@ -1 +1 @@
-e8fa3d7a6215d2565dc536cbc50e0465
+e83bbc2b97062b42c940992d0c2d80f6
diff --git a/dev/breeze/src/airflow_breeze/pre_commit_ids.py b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
index 516fc85c51..eb5c46e1ab 100644
--- a/dev/breeze/src/airflow_breeze/pre_commit_ids.py
+++ b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
@@ -38,6 +38,7 @@ PRE_COMMIT_LIST = [
"check-builtin-literals",
"check-changelog-has-no-duplicates",
"check-cncf-k8s-only-for-executors",
+ "check-compat-cache-on-methods",
"check-core-deprecation-classes",
"check-daysago-import-from-utils",
"check-decorated-operator-implements-custom-name",
diff --git a/hatch_build.py b/hatch_build.py
index ca85510780..439e486ec5 100644
--- a/hatch_build.py
+++ b/hatch_build.py
@@ -463,6 +463,7 @@ DEPENDENCIES = [
"markupsafe>=1.1.1",
"marshmallow-oneofschema>=2.0.1",
"mdit-py-plugins>=0.3.0",
+ "methodtools>=0.4.7",
"opentelemetry-api>=1.15.0",
"opentelemetry-exporter-otlp",
"packaging>=14.0",
diff --git a/pyproject.toml b/pyproject.toml
index 2fc9981328..7ab695097c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -284,9 +284,10 @@ extend-select = [
"D419",
"PGH004", # Use specific rule codes when using noqa
"PGH005", # Invalid unittest.mock.Mock methods/attributes/properties
- "B006", # Checks for uses of mutable objects as function argument defaults.
"S101", # Checks use `assert` outside the test cases, test cases should be added into the exclusions
"B004", # Checks for use of hasattr(x, "__call__") and replaces it with callable(x)
+ "B006", # Checks for uses of mutable objects as function argument defaults.
+ "B019", # Use of functools.lru_cache or functools.cache on methods can lead to memory leaks
]
ignore = [
"D203",
diff --git a/scripts/ci/pre_commit/compat_cache_on_methods.py b/scripts/ci/pre_commit/compat_cache_on_methods.py
new file mode 100755
index 0000000000..5fee74ff2a
--- /dev/null
+++ b/scripts/ci/pre_commit/compat_cache_on_methods.py
@@ -0,0 +1,69 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import ast
+import pathlib
+import sys
+
+COMPAT_MODULE = "airflow.compat.functools"
+
+
+def check_test_file(file: str) -> int:
+ node = ast.parse(pathlib.Path(file).read_text("utf-8"), file)
+ if not (classes := [c for c in node.body if isinstance(c, ast.ClassDef)]):
+ # Exit early if module doesn't contain any classes
+ return 0
+
+ compat_cache_aliases = []
+ for stmt in node.body:
+ if not isinstance(stmt, ast.ImportFrom) or stmt.module != COMPAT_MODULE:
+ continue
+ for alias in stmt.names:
+ if "cache" in alias.name:
+ compat_cache_aliases.append(alias.asname or alias.name)
+ if not compat_cache_aliases:
+ # Exit early in case if there are no imports from `airflow.compat.functools.cache`
+ return 0
+
+ found = 0
+ for klass in classes:
+ for cls_stmt in klass.body:
+ if not isinstance(cls_stmt, ast.FunctionDef) or not cls_stmt.decorator_list:
+ continue
+ for decorator in cls_stmt.decorator_list:
+ if (isinstance(decorator, ast.Name) and decorator.id in compat_cache_aliases) or (
+ isinstance(decorator, ast.Attribute) and decorator.attr in compat_cache_aliases
+ ):
+ found += 1
+ prefix = f"{file}:{decorator.lineno}:"
+ print(f"{prefix} Use of `{COMPAT_MODULE}.cache` on methods can lead to memory leaks")
+
+ return found
+
+
+def main(*args: str) -> int:
+ errors = sum(check_test_file(file) for file in args[1:])
+ if not errors:
+ return 0
+ print(f"Found {errors} error{'s' if errors > 1 else ''}.")
+ return 1
+
+
+if __name__ == "__main__":
+ sys.exit(main(*sys.argv))
(airflow) 17/48: Fix dag run selection (#38941)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 39dd5930279af2dd90bcb7c4106b9a931901b8c9
Author: Brent Bovenzi <br...@astronomer.io>
AuthorDate: Thu Apr 11 16:36:22 2024 -0400
Fix dag run selection (#38941)
* Fix dag run link params
* Do run_id checks inside of the grid_data hook
* remove firstRunId context
(cherry picked from commit f87772f52aeaad2c99eb4c0b1f985b9aa4810ea9)
---
airflow/www/static/js/api/useGridData.ts | 32 ++++++++++++++++++++++------
airflow/www/static/js/dag/Main.tsx | 14 +-----------
airflow/www/static/js/dag/details/Header.tsx | 11 ++++------
airflow/www/static/js/dag/useSelection.ts | 7 ------
airflow/www/templates/airflow/dag.html | 1 +
airflow/www/views.py | 27 ++++-------------------
6 files changed, 36 insertions(+), 56 deletions(-)
diff --git a/airflow/www/static/js/api/useGridData.ts b/airflow/www/static/js/api/useGridData.ts
index ee9325e852..1a633acea3 100644
--- a/airflow/www/static/js/api/useGridData.ts
+++ b/airflow/www/static/js/api/useGridData.ts
@@ -33,9 +33,9 @@ import useFilters, {
FILTER_UPSTREAM_PARAM,
ROOT_PARAM,
} from "src/dag/useFilters";
-import type { Task, DagRun, RunOrdering } from "src/types";
+import type { Task, DagRun, RunOrdering, API } from "src/types";
import { camelCase } from "lodash";
-import useSelection, { RUN_ID } from "src/dag/useSelection";
+import useSelection from "src/dag/useSelection";
const DAG_ID_PARAM = "dag_id";
@@ -80,9 +80,12 @@ const useGridData = () => {
filterDownstream,
filterUpstream,
},
+ onBaseDateChange,
} = useFilters();
- const { firstRunIdSetByUrl } = useSelection();
-
+ const {
+ onSelect,
+ selected: { taskId, runId },
+ } = useSelection();
const query = useQuery(
[
"gridData",
@@ -93,7 +96,7 @@ const useGridData = () => {
root,
filterUpstream,
filterDownstream,
- firstRunIdSetByUrl,
+ runId,
],
async () => {
const params = {
@@ -105,11 +108,28 @@ const useGridData = () => {
[NUM_RUNS_PARAM]: numRuns,
[RUN_TYPE_PARAM]: runType,
[RUN_STATE_PARAM]: runState,
- [RUN_ID]: firstRunIdSetByUrl || "",
};
const response = await axios.get<AxiosResponse, GridData>(gridDataUrl, {
params,
});
+ if (runId && !response.dagRuns.find((dr) => dr.runId === runId)) {
+ const dagRunUrl = getMetaValue("dag_run_url")
+ .replace("__DAG_ID__", dagId)
+ .replace("__DAG_RUN_ID__", runId);
+
+ // If the run id cannot be found in the response, try fetching it to see if its real and then adjust the base date filter
+ try {
+ const selectedRun = await axios.get<AxiosResponse, API.DAGRun>(
+ dagRunUrl
+ );
+ if (selectedRun?.executionDate) {
+ onBaseDateChange(selectedRun.executionDate);
+ }
+ // otherwise the run_id isn't valid and we should unselect it
+ } catch (e) {
+ onSelect({ taskId });
+ }
+ }
// turn off auto refresh if there are no active runs
if (!areActiveRuns(response.dagRuns)) stopRefresh();
return response;
diff --git a/airflow/www/static/js/dag/Main.tsx b/airflow/www/static/js/dag/Main.tsx
index 2d70b452d5..144aba94a9 100644
--- a/airflow/www/static/js/dag/Main.tsx
+++ b/airflow/www/static/js/dag/Main.tsx
@@ -47,7 +47,6 @@ import FilterBar from "./nav/FilterBar";
import LegendRow from "./nav/LegendRow";
import useToggleGroups from "./useToggleGroups";
import keyboardShortcutIdentifier from "./keyboardShortcutIdentifier";
-import { DagRunSelectionContext, RUN_ID } from "./useSelection";
const detailsPanelKey = "hideDetailsPanel";
const minPanelWidth = 300;
@@ -74,7 +73,7 @@ const headerHeight =
10
) || 0;
-const MainInContext = () => {
+const Main = () => {
const {
data: { groups },
isLoading,
@@ -319,15 +318,4 @@ const MainInContext = () => {
);
};
-const Main = () => {
- const [searchParams] = useSearchParams();
- const [firstRunIdSetByUrl] = useState(searchParams.get(RUN_ID));
-
- return (
- <DagRunSelectionContext.Provider value={firstRunIdSetByUrl}>
- <MainInContext />
- </DagRunSelectionContext.Provider>
- );
-};
-
export default Main;
diff --git a/airflow/www/static/js/dag/details/Header.tsx b/airflow/www/static/js/dag/details/Header.tsx
index e1222270d4..d13dfa6c81 100644
--- a/airflow/www/static/js/dag/details/Header.tsx
+++ b/airflow/www/static/js/dag/details/Header.tsx
@@ -51,17 +51,14 @@ const Header = ({ mapIndex }: Props) => {
} = useSelection();
const dagRun = dagRuns.find((r) => r.runId === runId);
-
const group = getTask({ taskId, task: groups });
- // If runId and/or taskId can't be found remove the selection
+ // If taskId can't be found remove the selection
useEffect(() => {
- if (runId && !dagRun && taskId && !group) {
- clearSelection();
- } else if (runId && !dagRun) {
- onSelect({ taskId });
+ if (taskId && !group) {
+ onSelect({ runId });
}
- }, [dagRun, taskId, group, runId, onSelect, clearSelection]);
+ }, [taskId, group, onSelect, runId]);
let runLabel;
if (dagRun && runId) {
diff --git a/airflow/www/static/js/dag/useSelection.ts b/airflow/www/static/js/dag/useSelection.ts
index 00ead40c52..04113a870c 100644
--- a/airflow/www/static/js/dag/useSelection.ts
+++ b/airflow/www/static/js/dag/useSelection.ts
@@ -17,7 +17,6 @@
* under the License.
*/
-import { createContext, useContext } from "react";
import { useSearchParams } from "react-router-dom";
import {
LIMIT_PARAM,
@@ -35,13 +34,8 @@ export interface SelectionProps {
mapIndex?: number;
}
-// The first run_id need to be treated differently from the selection, because it is used in backend to
-// calculate the base_date, which we don't want jumping around when user is clicking in the grid.
-export const DagRunSelectionContext = createContext<string | null>(null);
-
const useSelection = () => {
const [searchParams, setSearchParams] = useSearchParams();
- const firstRunIdSetByUrl = useContext(DagRunSelectionContext);
// Clear selection, but keep other search params
const clearSelection = () => {
@@ -99,7 +93,6 @@ const useSelection = () => {
},
clearSelection,
onSelect,
- firstRunIdSetByUrl,
};
};
diff --git a/airflow/www/templates/airflow/dag.html b/airflow/www/templates/airflow/dag.html
index 56bfc27070..d9ef3c139a 100644
--- a/airflow/www/templates/airflow/dag.html
+++ b/airflow/www/templates/airflow/dag.html
@@ -84,6 +84,7 @@
<meta name="datasets_api" content="{{ url_for('/api/v1.airflow_api_connexion_endpoints_dataset_endpoint_get_datasets') }}">
<meta name="event_logs_api" content="{{ url_for('/api/v1.airflow_api_connexion_endpoints_event_log_endpoint_get_event_logs') }}">
<meta name="audit_log_url" content="{{ url_for('LogModelView.list') }}">
+ <meta name="dag_run_url" content="{{ url_for('/api/v1.airflow_api_connexion_endpoints_dag_run_endpoint_get_dag_run', dag_id='__DAG_ID__', dag_run_id='__DAG_RUN_ID__') }}">
<!-- End Urls -->
<meta name="is_paused" content="{{ dag_is_paused }}">
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 55aa5a5e50..ce0727ab4d 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -3167,7 +3167,6 @@ class Airflow(AirflowBaseView):
def grid_data(self):
"""Return grid data."""
dag_id = request.args.get("dag_id")
- run_id = request.args.get("dag_run_id")
dag = get_airflow_app().dag_bag.get_dag(dag_id)
if not dag:
@@ -3185,43 +3184,25 @@ class Airflow(AirflowBaseView):
if num_runs is None:
num_runs = conf.getint("webserver", "default_dag_run_display_number")
- dagrun = None
- if run_id:
- with create_session() as session:
- dagrun = dag.get_dagrun(run_id=run_id, session=session)
- if not dagrun:
- return {"error": f"can't find dag_run_id={run_id}"}, 404
- base_date = dagrun.execution_date
- else:
- try:
- base_date = timezone.parse(request.args["base_date"], strict=True)
- except (KeyError, ValueError):
- base_date = dag.get_latest_execution_date() or timezone.utcnow()
+ try:
+ base_date = timezone.parse(request.args["base_date"], strict=True)
+ except (KeyError, ValueError):
+ base_date = dag.get_latest_execution_date() or timezone.utcnow()
with create_session() as session:
query = select(DagRun).where(DagRun.dag_id == dag.dag_id, DagRun.execution_date <= base_date)
run_types = request.args.getlist("run_type")
if run_types:
- if run_id:
- return {"error": "Can not provide filters when dag_run_id filter is selected."}, 400
query = query.where(DagRun.run_type.in_(run_types))
run_states = request.args.getlist("run_state")
if run_states:
- if run_id:
- return {"error": "Can not provide filters when dag_run_id filter is selected."}, 400
query = query.where(DagRun.state.in_(run_states))
dag_runs = wwwutils.sorted_dag_runs(
query, ordering=dag.timetable.run_ordering, limit=num_runs, session=session
)
- if dagrun:
- found_requested_run_id = any(True for d in dag_runs if d.run_id == run_id)
- if not found_requested_run_id:
- return {
- "error": f"Dag with dag_run_id={run_id} found, but not in selected time range or filters."
- }, 404
encoded_runs = [wwwutils.encode_dag_run(dr, json_encoder=utils_json.WebEncoder) for dr in dag_runs]
data = {
(airflow) 04/48: Improve audit log events filter (#38719)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 83e7dec335d40e3435607172c3b348c245dc39a6
Author: Brent Bovenzi <br...@astronomer.io>
AuthorDate: Thu Apr 4 12:56:27 2024 -0400
Improve audit log events filter (#38719)
(cherry picked from commit b094da448f0adbf24d6492b62748625c49da2e5b)
---
airflow/www/static/js/dag/details/AuditLog.tsx | 73 +++++++++++++++-----------
1 file changed, 41 insertions(+), 32 deletions(-)
diff --git a/airflow/www/static/js/dag/details/AuditLog.tsx b/airflow/www/static/js/dag/details/AuditLog.tsx
index e30fbbe68e..d991812781 100644
--- a/airflow/www/static/js/dag/details/AuditLog.tsx
+++ b/airflow/www/static/js/dag/details/AuditLog.tsx
@@ -29,6 +29,9 @@ import {
Input,
SimpleGrid,
Button,
+ RadioGroup,
+ Stack,
+ Radio,
} from "@chakra-ui/react";
import { createColumnHelper } from "@tanstack/react-table";
import { snakeCase } from "lodash";
@@ -74,12 +77,21 @@ const AuditLog = ({ taskId, run }: Props) => {
const { tableURLState, setTableURLState } = useTableURLState({
sorting: [{ id: "when", desc: true }],
});
- const [includedEvents, setIncludedEvents] = useState(
- configIncludedEvents.length ? configIncludedEvents.split(",") : []
- );
- const [excludedEvents, setExcludedEvents] = useState(
- configExcludedEvents.length ? configExcludedEvents.split(",") : []
- );
+ const includedEvents = configIncludedEvents.length
+ ? configIncludedEvents.split(",")
+ : [];
+ const excludedEvents = configExcludedEvents.length
+ ? configExcludedEvents.split(",")
+ : [];
+
+ let defaultEventFilter = "include";
+ let defaultEvents = includedEvents;
+ if (!includedEvents.length && excludedEvents.length) {
+ defaultEventFilter = "exclude";
+ defaultEvents = excludedEvents;
+ }
+ const [eventFilter, setEventFilter] = useState(defaultEventFilter);
+ const [events, setEvents] = useState(defaultEvents);
const sort = tableURLState.sorting[0];
const orderBy = sort ? `${sort.desc ? "-" : ""}${snakeCase(sort.id)}` : "";
@@ -94,8 +106,10 @@ const AuditLog = ({ taskId, run }: Props) => {
limit: tableURLState.pagination.pageSize,
offset:
tableURLState.pagination.pageIndex * tableURLState.pagination.pageSize,
- includedEvents: includedEvents ? includedEvents.join(",") : undefined,
- excludedEvents: excludedEvents ? excludedEvents.join(",") : undefined,
+ includedEvents:
+ eventFilter === "include" && events ? events.join(",") : undefined,
+ excludedEvents:
+ eventFilter === "exclude" && events ? events.join(",") : undefined,
});
const columns = useMemo(() => {
@@ -155,31 +169,17 @@ const AuditLog = ({ taskId, run }: Props) => {
}),
};
- const excludedEventsSelectProps = useChakraSelectProps<Option, true>({
+ const eventsSelectProps = useChakraSelectProps<Option, true>({
isMulti: true,
+ noOptionsMessage: () => "Type to add new event",
tagVariant: "solid",
- value: excludedEvents.map((e) => ({
+ value: events.map((e) => ({
label: e,
value: e,
})),
onChange: (options) => {
- setExcludedEvents((options || []).map(({ value }) => value));
+ setEvents((options || []).map(({ value }) => value));
},
- placeholder: "Type to filter an event",
- chakraStyles,
- });
-
- const includedEventsSelectProps = useChakraSelectProps<Option, true>({
- isMulti: true,
- tagVariant: "solid",
- value: includedEvents.map((e) => ({
- label: e,
- value: e,
- })),
- onChange: (options) => {
- setIncludedEvents((options || []).map(({ value }) => value));
- },
- placeholder: "Type to filter an event",
chakraStyles,
});
@@ -246,12 +246,21 @@ const AuditLog = ({ taskId, run }: Props) => {
<FormHelperText />
</FormControl>
<FormControl>
- <FormLabel>Events to exclude</FormLabel>
- <CreatableSelect {...excludedEventsSelectProps} />
- </FormControl>
- <FormControl>
- <FormLabel>Events to include</FormLabel>
- <CreatableSelect {...includedEventsSelectProps} />
+ <FormLabel display="flex" alignItems="center">
+ Events to
+ <RadioGroup
+ onChange={setEventFilter}
+ value={eventFilter}
+ display="inline-flex"
+ ml={2}
+ >
+ <Stack direction="row">
+ <Radio value="include">Include</Radio>
+ <Radio value="exclude">Exclude</Radio>
+ </Stack>
+ </RadioGroup>
+ </FormLabel>
+ <CreatableSelect {...eventsSelectProps} />
</FormControl>
</SimpleGrid>
<NewTable
(airflow) 16/48: Fix `SAWarning` 'Coercing Subquery object into a select() for use in IN()' (#38926)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit df37d60d498d9bfac073bb1aa59ca6734a7fe6db
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Thu Apr 11 23:18:26 2024 +0400
Fix `SAWarning` 'Coercing Subquery object into a select() for use in IN()' (#38926)
(cherry picked from commit 05ba268d05787bfbec6b1f5eefde2c90c64bf6b5)
---
airflow/api_connexion/endpoints/import_error_endpoint.py | 8 +++-----
airflow/utils/db_cleanup.py | 4 +---
airflow/www/views.py | 5 +----
pyproject.toml | 1 +
4 files changed, 6 insertions(+), 12 deletions(-)
diff --git a/airflow/api_connexion/endpoints/import_error_endpoint.py b/airflow/api_connexion/endpoints/import_error_endpoint.py
index a56cb97c41..274d842d18 100644
--- a/airflow/api_connexion/endpoints/import_error_endpoint.py
+++ b/airflow/api_connexion/endpoints/import_error_endpoint.py
@@ -94,11 +94,9 @@ def get_import_errors(
if not can_read_all_dags:
# if the user doesn't have access to all DAGs, only display errors from visible DAGs
readable_dag_ids = security.get_readable_dags()
- dagfiles_subq = (
- select(DagModel.fileloc).distinct().where(DagModel.dag_id.in_(readable_dag_ids)).subquery()
- )
- query = query.where(ImportErrorModel.filename.in_(dagfiles_subq))
- count_query = count_query.where(ImportErrorModel.filename.in_(dagfiles_subq))
+ dagfiles_stmt = select(DagModel.fileloc).distinct().where(DagModel.dag_id.in_(readable_dag_ids))
+ query = query.where(ImportErrorModel.filename.in_(dagfiles_stmt))
+ count_query = count_query.where(ImportErrorModel.filename.in_(dagfiles_stmt))
total_entries = session.scalars(count_query).one()
import_errors = session.scalars(query.offset(offset).limit(limit)).all()
diff --git a/airflow/utils/db_cleanup.py b/airflow/utils/db_cleanup.py
index e119c4228b..4475209b77 100644
--- a/airflow/utils/db_cleanup.py
+++ b/airflow/utils/db_cleanup.py
@@ -186,9 +186,7 @@ def _do_delete(*, query, orm_model, skip_archive, session):
if dialect_name == "sqlite":
pk_cols = source_table.primary_key.columns
delete = source_table.delete().where(
- tuple_(*pk_cols).in_(
- select(*[target_table.c[x.name] for x in source_table.primary_key.columns]).subquery()
- )
+ tuple_(*pk_cols).in_(select(*[target_table.c[x.name] for x in source_table.primary_key.columns]))
)
else:
delete = source_table.delete().where(
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 8a5ea38270..55aa5a5e50 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -952,10 +952,7 @@ class Airflow(AirflowBaseView):
# if the user doesn't have access to all DAGs, only display errors from visible DAGs
import_errors = import_errors.where(
errors.ImportError.filename.in_(
- select(DagModel.fileloc)
- .distinct()
- .where(DagModel.dag_id.in_(filter_dag_ids))
- .subquery()
+ select(DagModel.fileloc).distinct().where(DagModel.dag_id.in_(filter_dag_ids))
)
)
diff --git a/pyproject.toml b/pyproject.toml
index 82e3cbc528..1448a01dac 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -480,6 +480,7 @@ filterwarnings = [
"error::pytest.PytestCollectionWarning",
# Avoid building cartesian product which might impact performance
"error:SELECT statement has a cartesian product between FROM:sqlalchemy.exc.SAWarning:airflow",
+ 'error:Coercing Subquery object into a select\(\) for use in IN\(\):sqlalchemy.exc.SAWarning:airflow',
"ignore::DeprecationWarning:flask_appbuilder.filemanager",
"ignore::DeprecationWarning:flask_appbuilder.widgets",
# https://github.com/dpgaspar/Flask-AppBuilder/pull/1940
(airflow) 29/48: Fix doc redirects (#39039)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2970e35e1d2483d815be6efc2ddeffa634b2e445
Author: Vincent <97...@users.noreply.github.com>
AuthorDate: Tue Apr 16 04:34:50 2024 -0300
Fix doc redirects (#39039)
* Fix doc redirects
* Fix other redirects + add explanations about "../"
(cherry picked from commit 40bddb3686ea4411027b4d1358173ff253736976)
---
docs/apache-airflow/redirects.txt | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/docs/apache-airflow/redirects.txt b/docs/apache-airflow/redirects.txt
index 5b2408c950..058e95df1f 100644
--- a/docs/apache-airflow/redirects.txt
+++ b/docs/apache-airflow/redirects.txt
@@ -19,8 +19,15 @@
administration-and-deployment/security/index.rst security/index.rst
administration-and-deployment/security/kerberos.rst security/kerberos.rst
-administration-and-deployment/security/access-control.rst ../../apache-airflow-providers-fab/stable/auth-manager/access-control.rst
-administration-and-deployment/security/access-control/index.rst ../../apache-airflow-providers-fab/stable/auth-manager/access-control.rst
+
+# Moving FAB auth manager related docs to providers
+## The ``../`` indicates that it will move to the root of the docs directory, unlike the rest of
+## the entries here which move to docs/apache-airflow.
+## It's okay to include ``/stable/``, because there's no relationship between a version of
+## Airflow and the version of any provider package.
+
+administration-and-deployment/security/access-control.rst ../apache-airflow-providers-fab/stable/auth-manager/access-control.rst
+administration-and-deployment/security/access-control/index.rst ../apache-airflow-providers-fab/stable/auth-manager/access-control.rst
administration-and-deployment/security/api.rst security/api.rst
administration-and-deployment/security/audit_logs.rst security/audit_logs.rst
administration-and-deployment/security/flower.rst security/flower.rst
@@ -37,7 +44,7 @@ howto/use-alternative-secrets-backend.rst security/secrets/secrets-backend/index
security.rst security/index.rst
# Move the documentation from core to FAB provider
-security/access-control.rst ../../apache-airflow-providers-fab/stable/auth-manager/access-control.rst
+security/access-control.rst ../apache-airflow-providers-fab/stable/auth-manager/access-control.rst
# Operators guides
howto/operator/external.rst howto/operator/external_task_sensor.rst
@@ -47,7 +54,7 @@ howto/customize-dag-ui-page-instance-name.rst howto/customize-ui.rst#customizing
howto/customize-state-colors-ui.rst howto/customize-ui.rst#customizing-state-colours
# Web UI
-howto/add-new-role.rst ../../apache-airflow-providers-fab/stable/auth-manager/access-control.rst
+howto/add-new-role.rst ../apache-airflow-providers-fab/stable/auth-manager/access-control.rst
# Set up a database
howto/initialize-database.rst howto/set-up-database.rst
(airflow) 22/48: Add in Trove classifiers Python 3.12 support (#39004)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit adfabe897b156dfee44206de35a7a24b1e231a28
Author: Wei Lee <we...@gmail.com>
AuthorDate: Sun Apr 14 18:34:31 2024 +0800
Add in Trove classifiers Python 3.12 support (#39004)
(cherry picked from commit c2f96ff38d500e3c62e09992fdba179dd5b56045)
---
clients/python/pyproject.toml | 1 +
pyproject.toml | 1 +
2 files changed, 2 insertions(+)
diff --git a/clients/python/pyproject.toml b/clients/python/pyproject.toml
index d197812d36..88a5291d3d 100644
--- a/clients/python/pyproject.toml
+++ b/clients/python/pyproject.toml
@@ -46,6 +46,7 @@ classifiers = [
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
"Topic :: System :: Monitoring",
]
diff --git a/pyproject.toml b/pyproject.toml
index 1448a01dac..fbc7b7a9d7 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -59,6 +59,7 @@ classifiers = [
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
"Topic :: System :: Monitoring",
]
(airflow) 39/48: Add warning about run/import Airflow under the Windows (#39196)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8e458d979ee269bdf6e51dd22e1c083808c429db
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Tue Apr 23 14:21:46 2024 +0400
Add warning about run/import Airflow under the Windows (#39196)
(cherry picked from commit b82c428697b5210020c3251f1796cb6101386039)
---
airflow/__init__.py | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/airflow/__init__.py b/airflow/__init__.py
index f3aa6c3277..197f52c84a 100644
--- a/airflow/__init__.py
+++ b/airflow/__init__.py
@@ -21,6 +21,7 @@ __version__ = "2.9.0"
import os
import sys
+import warnings
if os.environ.get("_AIRFLOW_PATCH_GEVENT"):
# If you are using gevents and start airflow webserver, you might want to run gevent monkeypatching
@@ -30,6 +31,17 @@ if os.environ.get("_AIRFLOW_PATCH_GEVENT"):
patch_all()
+if sys.platform == "win32":
+ warnings.warn(
+ "Airflow currently can be run on POSIX-compliant Operating Systems. For development, "
+ "it is regularly tested on fairly modern Linux Distros and recent versions of macOS. "
+ "On Windows you can run it via WSL2 (Windows Subsystem for Linux 2) or via Linux Containers. "
+ "The work to add Windows support is tracked via https://github.com/apache/airflow/issues/10388, "
+ "but it is not a high priority.",
+ category=RuntimeWarning,
+ stacklevel=1,
+ )
+
# The configuration module initializes and validates the conf object as a side effect the first
# time it is imported. If it is not imported before importing the settings module, the conf
# object will then be initted/validated as a side effect of it being imported in settings,
@@ -76,8 +88,6 @@ def __getattr__(name: str):
module_path, attr_name, deprecated = __lazy_imports.get(name, ("", "", False))
if not module_path:
if name.startswith("PY3") and (py_minor := name[3:]) in ("6", "7", "8", "9", "10", "11", "12"):
- import warnings
-
warnings.warn(
f"Python version constraint {name!r} is deprecated and will be removed in the future. "
f"Please get version info from the 'sys.version_info'.",
@@ -87,8 +97,6 @@ def __getattr__(name: str):
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
elif deprecated:
- import warnings
-
warnings.warn(
f"Import {name!r} directly from the airflow module is deprecated and "
f"will be removed in the future. Please import it from 'airflow{module_path}.{attr_name}'.",
(airflow) 30/48: [docs] update `DagBag` class docstring to include all params (#38814)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 672d47ebf062b9b152ff407289db9959c9579db5
Author: Kalyan <ka...@live.com>
AuthorDate: Tue Apr 16 17:52:57 2024 +0530
[docs] update `DagBag` class docstring to include all params (#38814)
* update docstring for DagBag class
* break long line
* fix space
Signed-off-by: kalyanr <ka...@live.com>
---------
Signed-off-by: kalyanr <ka...@live.com>
(cherry picked from commit b00274156b28cce6a68dac20e667d21d00310016)
---
airflow/models/dagbag.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/airflow/models/dagbag.py b/airflow/models/dagbag.py
index ce9bf5587b..1902c985be 100644
--- a/airflow/models/dagbag.py
+++ b/airflow/models/dagbag.py
@@ -89,11 +89,16 @@ class DagBag(LoggingMixin):
:param dag_folder: the folder to scan to find DAGs
:param include_examples: whether to include the examples that ship
with airflow or not
+ :param safe_mode: when ``False``, scans all python modules for dags.
+ When ``True`` uses heuristics (files containing ``DAG`` and ``airflow`` strings)
+ to filter python modules to scan for dags.
:param read_dags_from_db: Read DAGs from DB if ``True`` is passed.
If ``False`` DAGs are read from python files.
+ :param store_serialized_dags: deprecated parameter, same effect as `read_dags_from_db`
:param load_op_links: Should the extra operator link be loaded via plugins when
de-serializing the DAG? This flag is set to False in Scheduler so that Extra Operator links
are not loaded to not run User code in Scheduler.
+ :param collect_dags: when True, collects dags during class initialization.
"""
def __init__(
(airflow) 21/48: Use debug level for minischeduler skip (#38976)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1f70db4794656a9cc86c9c862ee2658c060cf9f3
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Fri Apr 12 16:04:16 2024 -0700
Use debug level for minischeduler skip (#38976)
Now that we are using nowait, it will be more common that the minicheduler skips. Leaving it at info level will cause unnecessary alarm to users.
(cherry picked from commit d03ba594b3158c127c1c1b3f1d0c31fb93104367)
---
airflow/models/taskinstance.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
index 6dedf4bfd2..42849f88f1 100644
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -3486,7 +3486,7 @@ class TaskInstance(Base, LoggingMixin):
except OperationalError as e:
# Any kind of DB error here is _non fatal_ as this block is just an optimisation.
- cls.logger().info(
+ cls.logger().debug(
"Skipping mini scheduling run due to exception: %s",
e.statement,
exc_info=True,
(airflow) 33/48: Check that the dataset<>task exists before trying to render graph (#39069)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5fcf36cc652a98f5405626d263d93ad55251a959
Author: Brent Bovenzi <br...@astronomer.io>
AuthorDate: Tue Apr 16 21:54:30 2024 -0400
Check that the dataset<>task exists before trying to render graph (#39069)
(cherry picked from commit 57c6a4bfdcee03ac11d7e1aff167a91453acceb8)
---
airflow/www/static/js/dag/details/graph/index.tsx | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/airflow/www/static/js/dag/details/graph/index.tsx b/airflow/www/static/js/dag/details/graph/index.tsx
index d7445f189a..e86801f28f 100644
--- a/airflow/www/static/js/dag/details/graph/index.tsx
+++ b/airflow/www/static/js/dag/details/graph/index.tsx
@@ -34,7 +34,7 @@ import { BiCollapse, BiExpand } from "react-icons/bi";
import { useDatasets, useGraphData, useGridData } from "src/api";
import useSelection from "src/dag/useSelection";
-import { getMetaValue, useOffsetTop } from "src/utils";
+import { getMetaValue, getTask, useOffsetTop } from "src/utils";
import { useGraphLayout } from "src/utils/graph";
import Edge from "src/components/Graph/Edge";
import type { DepNode, WebserverEdge } from "src/types";
@@ -68,6 +68,10 @@ const Graph = ({
const [hasRendered, setHasRendered] = useState(false);
const [isZoomedOut, setIsZoomedOut] = useState(false);
+ const {
+ data: { dagRuns, groups },
+ } = useGridData();
+
useEffect(() => {
setArrange(data?.arrange || "LR");
}, [data?.arrange]);
@@ -104,7 +108,11 @@ const Graph = ({
);
const consumingDag = dataset?.consumingDags?.find((d) => d.dagId === dagId);
if (dataset.id) {
- if (producingTask?.taskId) {
+ // check that the task is in the graph
+ if (
+ producingTask?.taskId &&
+ getTask({ taskId: producingTask?.taskId, task: groups })
+ ) {
datasetEdges.push({
sourceId: producingTask.taskId,
targetId: dataset.id.toString(),
@@ -130,9 +138,6 @@ const Graph = ({
const { selected } = useSelection();
- const {
- data: { dagRuns, groups },
- } = useGridData();
const { colors } = useTheme();
const { getZoom, fitView } = useReactFlow();
const latestDagRunId = dagRuns[dagRuns.length - 1]?.runId;
(airflow) 47/48: Add workaround for datetime-local input in firefox (#39261)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d5d8b588f72fc8b91a7a5a7a16bcbe4cf82b1fe8
Author: Brent Bovenzi <br...@astronomer.io>
AuthorDate: Thu Apr 25 13:21:17 2024 -0400
Add workaround for datetime-local input in firefox (#39261)
(cherry picked from commit 5638bc3f502e656b1b3c8b765ad1c84434017b41)
---
.../static/js/cluster-activity/nav/FilterBar.tsx | 9 ++-
airflow/www/static/js/components/DateTimeInput.tsx | 84 ++++++++++++++++++++++
airflow/www/static/js/dag/nav/FilterBar.tsx | 6 +-
3 files changed, 91 insertions(+), 8 deletions(-)
diff --git a/airflow/www/static/js/cluster-activity/nav/FilterBar.tsx b/airflow/www/static/js/cluster-activity/nav/FilterBar.tsx
index b4f840cb5f..7e458ab5ab 100644
--- a/airflow/www/static/js/cluster-activity/nav/FilterBar.tsx
+++ b/airflow/www/static/js/cluster-activity/nav/FilterBar.tsx
@@ -19,7 +19,7 @@
/* global moment */
-import { Box, Button, Flex, Input, Text } from "@chakra-ui/react";
+import { Box, Button, Flex, Text } from "@chakra-ui/react";
import React from "react";
import { useTimezone } from "src/context/timezone";
@@ -29,6 +29,7 @@ import {
getDuration,
} from "src/datetime_utils";
import useFilters from "src/cluster-activity/useFilters";
+import DateTimeInput from "src/components/DateTimeInput";
const FilterBar = () => {
const { filters, onStartDateChange, onEndDateChange, clearFilters } =
@@ -61,9 +62,8 @@ const FilterBar = () => {
<Text fontSize="sm" as="b" position="absolute" mt="-14px" ml={1}>
Start Date
</Text>
- <Input
+ <DateTimeInput
{...inputStyles}
- type="datetime-local"
value={formattedStartDate || ""}
onChange={(e) => onStartDateChange(e.target.value)}
/>
@@ -72,9 +72,8 @@ const FilterBar = () => {
<Text fontSize="sm" as="b" position="absolute" mt="-14px" ml={1}>
End Date
</Text>
- <Input
+ <DateTimeInput
{...inputStyles}
- type="datetime-local"
value={formattedEndDate || ""}
onChange={(e) => onEndDateChange(e.target.value)}
/>
diff --git a/airflow/www/static/js/components/DateTimeInput.tsx b/airflow/www/static/js/components/DateTimeInput.tsx
new file mode 100644
index 0000000000..1d6b32ef1c
--- /dev/null
+++ b/airflow/www/static/js/components/DateTimeInput.tsx
@@ -0,0 +1,84 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/* global moment */
+
+import React from "react";
+import { Flex, Input, InputProps } from "@chakra-ui/react";
+
+const DateTimeInput = (props: InputProps) => {
+ // Firefox and safari do not support datetime-local so we need to put a date and time input next to each other
+ const userAgent = navigator.userAgent.toLowerCase();
+ const unsupportedBrowser =
+ userAgent.includes("firefox") || userAgent.includes("safari");
+
+ if (!unsupportedBrowser) return <Input type="datetime-local" {...props} />;
+
+ const { value, onChange, ...rest } = props;
+ // @ts-ignore
+ let datetime = moment(value as string);
+ // @ts-ignore
+ datetime = datetime.isValid() ? datetime : moment();
+ // @ts-ignore
+ const date = moment(datetime).format("YYYY-MM-DD");
+ // @ts-ignore
+ const time = moment(datetime).format("HH:mm:ss");
+ return (
+ <Flex>
+ <Input
+ type="date"
+ value={date}
+ onChange={(e) => {
+ if (onChange)
+ onChange({
+ ...e,
+ target: {
+ ...e.target,
+ value: `${e.target.value}T${time}`,
+ },
+ });
+ }}
+ {...rest}
+ borderRightWidth={0}
+ borderRightRadius={0}
+ paddingInlineEnd={0}
+ />
+ <Input
+ type="time"
+ value={time}
+ onChange={(e) => {
+ if (onChange)
+ onChange({
+ ...e,
+ target: {
+ ...e.target,
+ value: `${date}T${e.target.value}`,
+ },
+ });
+ }}
+ {...rest}
+ borderLeftWidth={0}
+ borderLeftRadius={0}
+ paddingInlineStart={0}
+ />
+ </Flex>
+ );
+};
+
+export default DateTimeInput;
diff --git a/airflow/www/static/js/dag/nav/FilterBar.tsx b/airflow/www/static/js/dag/nav/FilterBar.tsx
index f03ea734b8..2a0b356893 100644
--- a/airflow/www/static/js/dag/nav/FilterBar.tsx
+++ b/airflow/www/static/js/dag/nav/FilterBar.tsx
@@ -19,7 +19,7 @@
/* global moment */
-import { Box, Button, Flex, Input, Select } from "@chakra-ui/react";
+import { Box, Button, Flex, Select } from "@chakra-ui/react";
import MultiSelect from "src/components/MultiSelect";
import React from "react";
import type { DagRun, RunState, TaskState } from "src/types";
@@ -30,6 +30,7 @@ import { useChakraSelectProps } from "chakra-react-select";
import { useTimezone } from "src/context/timezone";
import { isoFormatWithoutTZ } from "src/datetime_utils";
import useFilters from "src/dag/useFilters";
+import DateTimeInput from "src/components/DateTimeInput";
declare const filtersOptions: {
dagStates: RunState[];
@@ -124,9 +125,8 @@ const FilterBar = () => {
<Flex backgroundColor="blackAlpha.200" p={4} justifyContent="space-between">
<Flex ml={10}>
<Box px={2}>
- <Input
+ <DateTimeInput
{...inputStyles}
- type="datetime-local"
value={formattedTime || ""}
onChange={(e) => onBaseDateChange(e.target.value)}
{...(isBaseDateDefault ? {} : filteredStyles)}
(airflow) 02/48: Raise deprecated warning in `airflow dags backfill` only if `-I`/`--ignore-first-depends-on-past` provided (#38676)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 70d31c09776849554195b939af1f707a24d58037
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Tue Apr 2 23:30:39 2024 +0400
Raise deprecated warning in `airflow dags backfill` only if `-I`/`--ignore-first-depends-on-past` provided (#38676)
(cherry picked from commit e58cd4b519a867a06cf4898d8481bececddb9b88)
---
airflow/cli/commands/dag_command.py | 13 ++++++-------
tests/cli/commands/test_dag_command.py | 35 ++++++++++++++++++++++++----------
2 files changed, 31 insertions(+), 17 deletions(-)
diff --git a/airflow/cli/commands/dag_command.py b/airflow/cli/commands/dag_command.py
index 2a8031cf90..7f64f9006d 100644
--- a/airflow/cli/commands/dag_command.py
+++ b/airflow/cli/commands/dag_command.py
@@ -133,10 +133,12 @@ def dag_backfill(args, dag: list[DAG] | DAG | None = None) -> None:
"""Create backfill job or dry run for a DAG or list of DAGs using regex."""
logging.basicConfig(level=settings.LOGGING_LEVEL, format=settings.SIMPLE_LOG_FORMAT)
signal.signal(signal.SIGTERM, sigint_handler)
- warnings.warn(
- "--ignore-first-depends-on-past is deprecated as the value is always set to True",
- category=RemovedInAirflow3Warning,
- )
+ if args.ignore_first_depends_on_past:
+ warnings.warn(
+ "--ignore-first-depends-on-past is deprecated as the value is always set to True",
+ category=RemovedInAirflow3Warning,
+ )
+ args.ignore_first_depends_on_past = True
if not args.treat_dag_id_as_regex and args.treat_dag_as_regex:
warnings.warn(
@@ -145,9 +147,6 @@ def dag_backfill(args, dag: list[DAG] | DAG | None = None) -> None:
)
args.treat_dag_id_as_regex = args.treat_dag_as_regex
- if args.ignore_first_depends_on_past is False:
- args.ignore_first_depends_on_past = True
-
if not args.start_date and not args.end_date:
raise AirflowException("Provide a start_date and/or end_date")
diff --git a/tests/cli/commands/test_dag_command.py b/tests/cli/commands/test_dag_command.py
index cac538139b..fb472fbdb0 100644
--- a/tests/cli/commands/test_dag_command.py
+++ b/tests/cli/commands/test_dag_command.py
@@ -17,6 +17,7 @@
# under the License.
from __future__ import annotations
+import argparse
import contextlib
import json
import os
@@ -34,7 +35,7 @@ from airflow.api_connexion.schemas.dag_schema import DAGSchema, dag_schema
from airflow.cli import cli_parser
from airflow.cli.commands import dag_command
from airflow.decorators import task
-from airflow.exceptions import AirflowException
+from airflow.exceptions import AirflowException, RemovedInAirflow3Warning
from airflow.models import DagBag, DagModel, DagRun
from airflow.models.baseoperator import BaseOperator
from airflow.models.dag import _run_inline_trigger
@@ -61,6 +62,8 @@ pytestmark = pytest.mark.db_test
class TestCliDags:
+ parser: argparse.ArgumentParser
+
@classmethod
def setup_class(cls):
cls.dagbag = DagBag(include_examples=True)
@@ -328,8 +331,15 @@ class TestCliDags:
assert "OUT" in out
assert "ERR" in out
+ @pytest.mark.parametrize(
+ "cli_arg",
+ [
+ pytest.param("-I", id="short"),
+ pytest.param("--ignore-first-depends-on-past", id="full"),
+ ],
+ )
@mock.patch("airflow.cli.commands.dag_command.DAG.run")
- def test_cli_backfill_depends_on_past(self, mock_run):
+ def test_cli_backfill_deprecated_ignore_first_depends_on_past(self, mock_run, cli_arg: str):
"""
Test that CLI respects -I argument
@@ -345,11 +355,12 @@ class TestCliDags:
"--local",
"--start-date",
run_date.isoformat(),
- "--ignore-first-depends-on-past",
+ cli_arg,
]
dag = self.dagbag.get_dag(dag_id)
- dag_command.dag_backfill(self.parser.parse_args(args), dag=dag)
+ with pytest.warns(RemovedInAirflow3Warning, match="ignore-first-depends-on-past is deprecated"):
+ dag_command.dag_backfill(self.parser.parse_args(args), dag=dag)
mock_run.assert_called_once_with(
start_date=run_date,
@@ -369,11 +380,16 @@ class TestCliDags:
disable_retry=False,
)
+ @pytest.mark.parametrize(
+ "cli_arg",
+ [
+ pytest.param("-B", id="short"),
+ pytest.param("--run-backwards", id="full"),
+ ],
+ )
@mock.patch("airflow.cli.commands.dag_command.DAG.run")
- def test_cli_backfill_depends_on_past_backwards(self, mock_run):
- """
- Test that CLI respects -B argument and raises on interaction with depends_on_past
- """
+ def test_cli_backfill_depends_on_past_run_backwards(self, mock_run, cli_arg: str):
+ """Test that CLI respects -B argument."""
dag_id = "test_depends_on_past"
start_date = DEFAULT_DATE + timedelta(days=1)
end_date = start_date + timedelta(days=1)
@@ -386,8 +402,7 @@ class TestCliDags:
start_date.isoformat(),
"--end-date",
end_date.isoformat(),
- "--ignore-first-depends-on-past",
- "--run-backwards",
+ cli_arg,
]
dag = self.dagbag.get_dag(dag_id)
(airflow) 41/48: Use grid view for Task Instance's `log_url` (#39183)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7cd311dc78a82c057b971912fdb4d70457e06638
Author: Madison Swain-Bowden <bo...@spu.edu>
AuthorDate: Wed Apr 24 09:41:43 2024 -0700
Use grid view for Task Instance's `log_url` (#39183)
* Use grid view for Task Instance's `log_url`
* Update other tests
(cherry picked from commit 6db6fef357f2adadf94956238fd14eb34aebfe34)
---
airflow/models/taskinstance.py | 10 ++++++----
tests/models/test_taskinstance.py | 9 +++++----
tests/providers/smtp/notifications/test_smtp.py | 2 +-
3 files changed, 12 insertions(+), 9 deletions(-)
diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
index 88a495e9eb..42824fa586 100644
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -1719,15 +1719,17 @@ class TaskInstance(Base, LoggingMixin):
@property
def log_url(self) -> str:
"""Log URL for TaskInstance."""
- iso = quote(self.execution_date.isoformat())
+ run_id = quote(self.run_id)
base_url = conf.get_mandatory_value("webserver", "BASE_URL")
return (
f"{base_url}"
- "/log"
- f"?execution_date={iso}"
+ f"/dags"
+ f"/{self.dag_id}"
+ f"/grid"
+ f"?dag_run_id={run_id}"
f"&task_id={self.task_id}"
- f"&dag_id={self.dag_id}"
f"&map_index={self.map_index}"
+ "&tab=logs"
)
@property
diff --git a/tests/models/test_taskinstance.py b/tests/models/test_taskinstance.py
index 120856dbbf..d1cb500dec 100644
--- a/tests/models/test_taskinstance.py
+++ b/tests/models/test_taskinstance.py
@@ -1981,14 +1981,15 @@ class TestTaskInstance:
assert 1 == tis2[("task_3", 0)].get_num_running_task_instances(session=session, same_dagrun=True)
def test_log_url(self, create_task_instance):
- ti = create_task_instance(dag_id="dag", task_id="op", execution_date=timezone.datetime(2018, 1, 1))
+ ti = create_task_instance(dag_id="my_dag", task_id="op", execution_date=timezone.datetime(2018, 1, 1))
expected_url = (
- "http://localhost:8080/log?"
- "execution_date=2018-01-01T00%3A00%3A00%2B00%3A00"
+ "http://localhost:8080"
+ "/dags/my_dag/grid"
+ "?dag_run_id=test"
"&task_id=op"
- "&dag_id=dag"
"&map_index=-1"
+ "&tab=logs"
)
assert ti.log_url == expected_url
diff --git a/tests/providers/smtp/notifications/test_smtp.py b/tests/providers/smtp/notifications/test_smtp.py
index 9878d99786..f1a71e2b53 100644
--- a/tests/providers/smtp/notifications/test_smtp.py
+++ b/tests/providers/smtp/notifications/test_smtp.py
@@ -129,7 +129,7 @@ class TestSmtpNotifier:
from_email=conf.get("smtp", "smtp_mail_from"),
to="test_reciver@test.com",
subject="DAG dag - Task op - Run ID test in State None",
- html_content="""<!DOCTYPE html>\n<html>\n <head>\n <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />\n <meta name="viewport" content="width=device-width">\n </head>\n<body>\n <table role="presentation">\n \n <tr>\n <td>Run ID:</td>\n <td>test</td>\n </tr>\n <tr>\n <td>Try:</td>\n <td>1 of 1</td>\n </tr>\n <tr>\n <td>Task State:</td>\n [...]
+ html_content="""<!DOCTYPE html>\n<html>\n <head>\n <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />\n <meta name="viewport" content="width=device-width">\n </head>\n<body>\n <table role="presentation">\n \n <tr>\n <td>Run ID:</td>\n <td>test</td>\n </tr>\n <tr>\n <td>Try:</td>\n <td>1 of 1</td>\n </tr>\n <tr>\n <td>Task State:</td>\n [...]
smtp_conn_id="smtp_default",
files=None,
cc=None,
(airflow) 12/48: Fix typo in templates-ref.rst (#38865)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 72b2ec0ea49027666a2e1976b56503904556ed10
Author: Kolmakov Alexey <al...@gmail.com>
AuthorDate: Tue Apr 9 23:00:38 2024 +0300
Fix typo in templates-ref.rst (#38865)
(cherry picked from commit 90aa4bb1da72f88b4927ca33e72feb55282e3c02)
---
docs/apache-airflow/templates-ref.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow/templates-ref.rst b/docs/apache-airflow/templates-ref.rst
index dd05fcc831..8f5e3ff070 100644
--- a/docs/apache-airflow/templates-ref.rst
+++ b/docs/apache-airflow/templates-ref.rst
@@ -39,7 +39,7 @@ Variable Type Description
``{{ data_interval_start }}`` `pendulum.DateTime`_ Start of the data interval. Added in version 2.2.
``{{ data_interval_end }}`` `pendulum.DateTime`_ End of the data interval. Added in version 2.2.
``{{ logical_date }}`` `pendulum.DateTime`_ | A date-time that logically identifies the current DAG run. This value does not contain any semantics, but is simply a value for identification.
- | Use ``data_interval_start`` and ``date_interval_end`` instead if you want a value that has real-world semantics,
+ | Use ``data_interval_start`` and ``data_interval_end`` instead if you want a value that has real-world semantics,
| such as to get a slice of rows from the database based on timestamps.
``{{ ds }}`` str | The DAG run's logical date as ``YYYY-MM-DD``.
| Same as ``{{ logical_date | ds }}``.
(airflow) 11/48: Fix dag run link params (#38873)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 84b76e02fc7b8352fa132dd3926f69db08682b53
Author: Brent Bovenzi <br...@astronomer.io>
AuthorDate: Tue Apr 9 14:23:13 2024 -0400
Fix dag run link params (#38873)
(cherry picked from commit 43522a7b8460753cb0e588ed79d76e34adf529ba)
---
airflow/www/static/js/dag/useFilters.tsx | 1 +
airflow/www/utils.py | 4 ++--
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/airflow/www/static/js/dag/useFilters.tsx b/airflow/www/static/js/dag/useFilters.tsx
index a83598beb8..5a1a49b686 100644
--- a/airflow/www/static/js/dag/useFilters.tsx
+++ b/airflow/www/static/js/dag/useFilters.tsx
@@ -182,6 +182,7 @@ const useFilters = (): FilterHookReturn => {
searchParams.delete(BASE_DATE_PARAM);
searchParams.delete(RUN_TYPE_PARAM);
searchParams.delete(RUN_STATE_PARAM);
+ searchParams.delete(EXECUTION_DATE_PARAM);
setSearchParams(searchParams);
};
diff --git a/airflow/www/utils.py b/airflow/www/utils.py
index d73bff9a64..995833d6d6 100644
--- a/airflow/www/utils.py
+++ b/airflow/www/utils.py
@@ -532,8 +532,8 @@ def dag_run_link(attr):
"""Generate a URL to the Graph view for a DagRun."""
dag_id = attr.get("dag_id")
run_id = attr.get("run_id")
- execution_date = attr.get("dag_run.execution_date") or attr.get("execution_date")
- url = url_for("Airflow.graph", dag_id=dag_id, run_id=run_id, execution_date=execution_date)
+
+ url = url_for("Airflow.graph", dag_id=dag_id, dag_run_id=run_id)
return Markup('<a href="{url}">{run_id}</a>').format(url=url, run_id=run_id)
(airflow) 19/48: Fix calendar view with no DAG Run. (#38964)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b35f67dc4cf695aa18885eb1fba445e8699ba847
Author: Karthikeyan Singaravelan <ti...@gmail.com>
AuthorDate: Sat Apr 13 00:30:39 2024 +0530
Fix calendar view with no DAG Run. (#38964)
(cherry picked from commit 5f6f4a5f4654d64efb4c0cabb6e0e054d13a2e0f)
---
airflow/www/static/js/dag/details/dag/Calendar.tsx | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/airflow/www/static/js/dag/details/dag/Calendar.tsx b/airflow/www/static/js/dag/details/dag/Calendar.tsx
index 0c9e5afb93..6826ab5d67 100644
--- a/airflow/www/static/js/dag/details/dag/Calendar.tsx
+++ b/airflow/www/static/js/dag/details/dag/Calendar.tsx
@@ -21,7 +21,7 @@
import React from "react";
import type { EChartsOption } from "echarts";
-import { Box, Spinner, Flex } from "@chakra-ui/react";
+import { Box, Spinner, Flex, Text } from "@chakra-ui/react";
import ReactECharts from "src/components/ReactECharts";
import { useCalendarData } from "src/api";
@@ -37,6 +37,10 @@ const Calendar = () => {
const { dagStates } = calendarData;
+ if (dagStates.length < 1) {
+ return <Text>Calendar view requires at least one DAG Run.</Text>;
+ }
+
const startDate = dagStates[0].date;
const endDate = dagStates[dagStates.length - 1].date;
// @ts-ignore
(airflow) 08/48: Remove decorator from rendering fields example (#38827)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a0249d4a9ecda1c0dc270c52eb5b19ab4cc0f460
Author: Elad Kalif <45...@users.noreply.github.com>
AuthorDate: Mon Apr 8 19:09:17 2024 +0300
Remove decorator from rendering fields example (#38827)
* Remove decorator from rendering fields example
* fix
(cherry picked from commit 285f037dbcc0f5e23c2d6ac99bfd6cab86c96ac3)
---
docs/apache-airflow/core-concepts/operators.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow/core-concepts/operators.rst b/docs/apache-airflow/core-concepts/operators.rst
index c7193789bc..0330ef43fb 100644
--- a/docs/apache-airflow/core-concepts/operators.rst
+++ b/docs/apache-airflow/core-concepts/operators.rst
@@ -246,9 +246,9 @@ you can pass ``render_template_as_native_obj=True`` to the DAG as follows:
return json.loads(data_string)
- @task(task_id="transform")
def transform(order_data):
print(type(order_data))
+ total_order_value = 0
for value in order_data.values():
total_order_value += value
return {"total_order_value": total_order_value}
(airflow) 34/48: Graph view improvements (#38940)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7cd476ed506b804d3900c2bd5ffa9dfef0592fb1
Author: Brent Bovenzi <br...@astronomer.io>
AuthorDate: Wed Apr 17 08:59:59 2024 -0400
Graph view improvements (#38940)
* Initial pass at graph updates again
* Initial pass
* fixes
* Fix task_group tests
(cherry picked from commit 0fbe711dfa73d416a4ddd60caa29f1406cf0774c)
---
airflow/utils/task_group.py | 5 ++++
airflow/www/static/js/dag/TaskName.tsx | 1 -
.../www/static/js/dag/details/graph/DagNode.tsx | 34 ++++++++++++++--------
airflow/www/static/js/dag/grid/renderTaskRows.tsx | 5 ----
airflow/www/static/js/types/index.ts | 1 +
airflow/www/static/js/utils/graph.ts | 11 +++----
6 files changed, 34 insertions(+), 23 deletions(-)
diff --git a/airflow/utils/task_group.py b/airflow/utils/task_group.py
index 50ce7b4089..37b03e1907 100644
--- a/airflow/utils/task_group.py
+++ b/airflow/utils/task_group.py
@@ -679,13 +679,17 @@ class TaskGroupContext:
def task_group_to_dict(task_item_or_group):
"""Create a nested dict representation of this TaskGroup and its children used to construct the Graph."""
from airflow.models.abstractoperator import AbstractOperator
+ from airflow.models.mappedoperator import MappedOperator
if isinstance(task := task_item_or_group, AbstractOperator):
setup_teardown_type = {}
+ is_mapped = {}
if task.is_setup is True:
setup_teardown_type["setupTeardownType"] = "setup"
elif task.is_teardown is True:
setup_teardown_type["setupTeardownType"] = "teardown"
+ if isinstance(task, MappedOperator):
+ is_mapped["isMapped"] = True
return {
"id": task.task_id,
"value": {
@@ -694,6 +698,7 @@ def task_group_to_dict(task_item_or_group):
"style": f"fill:{task.ui_color};",
"rx": 5,
"ry": 5,
+ **is_mapped,
**setup_teardown_type,
},
}
diff --git a/airflow/www/static/js/dag/TaskName.tsx b/airflow/www/static/js/dag/TaskName.tsx
index 05f851c18d..6b5f0a90bd 100644
--- a/airflow/www/static/js/dag/TaskName.tsx
+++ b/airflow/www/static/js/dag/TaskName.tsx
@@ -54,7 +54,6 @@ const TaskName = ({
data-testid={id}
color={colors.gray[800]}
fontSize={isZoomedOut ? 24 : undefined}
- textAlign="justify"
{...rest}
>
<chakra.span onClick={onClick}>
diff --git a/airflow/www/static/js/dag/details/graph/DagNode.tsx b/airflow/www/static/js/dag/details/graph/DagNode.tsx
index ec04f73c17..7a448da880 100644
--- a/airflow/www/static/js/dag/details/graph/DagNode.tsx
+++ b/airflow/www/static/js/dag/details/graph/DagNode.tsx
@@ -18,7 +18,7 @@
*/
import React from "react";
-import { Box, Flex, Text } from "@chakra-ui/react";
+import { Flex, Text, useTheme } from "@chakra-ui/react";
import type { NodeProps } from "reactflow";
import { SimpleStatus } from "src/dag/StatusBox";
@@ -53,10 +53,11 @@ const DagNode = ({
}: NodeProps<CustomNodeProps>) => {
const { onSelect } = useSelection();
const containerRef = useContainerRef();
+ const { colors } = useTheme();
if (!task) return null;
- const bg = isOpen ? "blackAlpha.50" : "white";
+ const groupBg = isOpen ? `${colors.blue[500]}15` : "blue.50";
const { isMapped } = task;
const mappedStates = instance?.mappedStates;
@@ -67,9 +68,9 @@ const DagNode = ({
: label;
let operatorTextColor = "";
- let operatorBG = "";
+ let operatorBg = "";
if (style) {
- [, operatorBG] = style.split(":");
+ [, operatorBg] = style.split(":");
}
if (labelStyle) {
@@ -83,6 +84,12 @@ const DagNode = ({
? stateColors[instance.state]
: "gray.400";
+ let borderWidth = 2;
+ if (isZoomedOut) {
+ if (isSelected) borderWidth = 10;
+ else borderWidth = 6;
+ } else if (isSelected) borderWidth = 4;
+
return (
<Tooltip
label={
@@ -95,15 +102,16 @@ const DagNode = ({
placement="top"
openDelay={hoverDelay}
>
- <Box
+ <Flex
borderRadius={isZoomedOut ? 10 : 5}
- borderWidth={(isSelected ? 4 : 2) * (isZoomedOut ? 3 : 1)}
+ borderWidth={borderWidth}
borderColor={nodeBorderColor}
+ wordBreak="break-word"
bg={
- !task.children?.length && operatorBG
+ !task.children?.length && operatorBg
? // Fade the operator color to clash less with the task instance status
- `color-mix(in srgb, ${operatorBG.replace(";", "")} 80%, white)`
- : bg
+ `color-mix(in srgb, ${operatorBg.replace(";", "")} 80%, white)`
+ : groupBg
}
height={`${height}px`}
width={`${width}px`}
@@ -121,6 +129,10 @@ const DagNode = ({
}}
px={isZoomedOut ? 1 : 2}
mt={isZoomedOut ? -2 : 0}
+ alignItems={isZoomedOut && !isOpen ? "center" : undefined}
+ justifyContent={isZoomedOut && !isOpen ? "center" : undefined}
+ flexDirection="column"
+ overflow="wrap"
>
<TaskName
label={taskName}
@@ -131,9 +143,7 @@ const DagNode = ({
onToggleCollapse();
}}
setupTeardownType={setupTeardownType}
- fontWeight="bold"
isZoomedOut={isZoomedOut}
- mt={isZoomedOut ? -2 : 0}
noOfLines={2}
/>
{!isZoomedOut && (
@@ -159,7 +169,7 @@ const DagNode = ({
)}
</>
)}
- </Box>
+ </Flex>
</Tooltip>
);
};
diff --git a/airflow/www/static/js/dag/grid/renderTaskRows.tsx b/airflow/www/static/js/dag/grid/renderTaskRows.tsx
index dd77284fe6..9d688345bb 100644
--- a/airflow/www/static/js/dag/grid/renderTaskRows.tsx
+++ b/airflow/www/static/js/dag/grid/renderTaskRows.tsx
@@ -182,11 +182,6 @@ const Row = (props: RowProps) => {
pl={level * 4 + 4}
setupTeardownType={task.setupTeardownType}
pr={4}
- fontWeight={
- isGroup || (task.isMapped && !isParentMapped)
- ? "bold"
- : "normal"
- }
noOfLines={1}
/>
</Td>
diff --git a/airflow/www/static/js/types/index.ts b/airflow/www/static/js/types/index.ts
index cf97aa8e04..79ed13c9bb 100644
--- a/airflow/www/static/js/types/index.ts
+++ b/airflow/www/static/js/types/index.ts
@@ -143,6 +143,7 @@ interface DepNode {
labelStyle?: string;
style?: string;
setupTeardownType?: "setup" | "teardown";
+ isMapped?: boolean;
};
children?: DepNode[];
edges?: MidEdge[];
diff --git a/airflow/www/static/js/utils/graph.ts b/airflow/www/static/js/utils/graph.ts
index 7db3767994..dde4fe64c3 100644
--- a/airflow/www/static/js/utils/graph.ts
+++ b/airflow/www/static/js/utils/graph.ts
@@ -173,8 +173,10 @@ const generateGraph = ({
}));
closedGroupIds.push(id);
}
- const extraLabelLength =
- value.label.length > 20 ? value.label.length - 19 : 0;
+
+ const label = value.isMapped ? `${value.label} [100]` : value.label;
+ const labelLength = getTextWidth(label, font);
+ const width = labelLength > 200 ? labelLength : 200;
return {
id,
@@ -184,9 +186,8 @@ const generateGraph = ({
isJoinNode,
childCount,
},
- // Make tasks with long names wider
- width: isJoinNode ? 10 : 200 + extraLabelLength * 5,
- height: isJoinNode ? 10 : 70,
+ width: isJoinNode ? 10 : width,
+ height: isJoinNode ? 10 : 80,
};
};
(airflow) 44/48: Get served logs when remote or executor logs not available for non-running task try (#39177)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 505ecd95381610ed073a3bb5cfd9d773335b8a2b
Author: Kalle Ahlström <71...@users.noreply.github.com>
AuthorDate: Thu Apr 25 16:19:37 2024 +0300
Get served logs when remote or executor logs not available for non-running task try (#39177)
(cherry picked from commit eca077b3b994813a09942497704ec61e35efd7d5)
---
airflow/utils/log/file_task_handler.py | 12 ++++++------
tests/utils/test_log_handlers.py | 14 ++++++++++++--
2 files changed, 18 insertions(+), 8 deletions(-)
diff --git a/airflow/utils/log/file_task_handler.py b/airflow/utils/log/file_task_handler.py
index a5a2da6062..6d35b230b4 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -343,10 +343,7 @@ class FileTaskHandler(logging.Handler):
executor_messages: list[str] = []
executor_logs: list[str] = []
served_logs: list[str] = []
- is_running = ti.try_number == try_number and ti.state in (
- TaskInstanceState.RUNNING,
- TaskInstanceState.DEFERRED,
- )
+ is_in_running_or_deferred = ti.state in (TaskInstanceState.RUNNING, TaskInstanceState.DEFERRED)
with suppress(NotImplementedError):
remote_messages, remote_logs = self._read_remote_logs(ti, try_number, metadata)
messages_list.extend(remote_messages)
@@ -361,7 +358,9 @@ class FileTaskHandler(logging.Handler):
worker_log_full_path = Path(self.local_base, worker_log_rel_path)
local_messages, local_logs = self._read_from_local(worker_log_full_path)
messages_list.extend(local_messages)
- if is_running and not executor_messages:
+ if is_in_running_or_deferred and not executor_messages and not remote_logs:
+ # While task instance is still running and we don't have either executor nor remote logs, look for served logs
+ # This is for cases when users have not setup remote logging nor shared drive for logs
served_messages, served_logs = self._read_from_logs_server(ti, worker_log_rel_path)
messages_list.extend(served_messages)
elif ti.state not in State.unfinished and not (local_logs or remote_logs):
@@ -381,11 +380,12 @@ class FileTaskHandler(logging.Handler):
)
log_pos = len(logs)
messages = "".join([f"*** {x}\n" for x in messages_list])
+ end_of_log = ti.try_number != try_number or not is_in_running_or_deferred
if metadata and "log_pos" in metadata:
previous_chars = metadata["log_pos"]
logs = logs[previous_chars:] # Cut off previously passed log test as new tail
out_message = logs if "log_pos" in (metadata or {}) else messages + logs
- return out_message, {"end_of_log": not is_running, "log_pos": log_pos}
+ return out_message, {"end_of_log": end_of_log, "log_pos": log_pos}
@staticmethod
def _get_pod_namespace(ti: TaskInstance):
diff --git a/tests/utils/test_log_handlers.py b/tests/utils/test_log_handlers.py
index 7f84b6d9f1..13f1e4de0e 100644
--- a/tests/utils/test_log_handlers.py
+++ b/tests/utils/test_log_handlers.py
@@ -304,7 +304,7 @@ class TestFileTaskLogHandler:
def test__read_for_celery_executor_fallbacks_to_worker(self, create_task_instance):
"""Test for executors which do not have `get_task_log` method, it fallbacks to reading
- log from worker. But it happens only for the latest try_number."""
+ log from worker if and only if remote logs aren't found"""
executor_name = "CeleryExecutor"
ti = create_task_instance(
@@ -325,7 +325,17 @@ class TestFileTaskLogHandler:
fth._read_from_logs_server.assert_called_once()
assert actual == ("*** this message\nthis\nlog\ncontent", {"end_of_log": False, "log_pos": 16})
- # Previous try_number is from remote logs without reaching worker server
+ # Previous try_number should return served logs when remote logs aren't implemented
+ fth._read_from_logs_server = mock.Mock()
+ fth._read_from_logs_server.return_value = ["served logs try_number=1"], ["this\nlog\ncontent"]
+ actual = fth._read(ti=ti, try_number=1)
+ fth._read_from_logs_server.assert_called_once()
+ assert actual == (
+ "*** served logs try_number=1\nthis\nlog\ncontent",
+ {"end_of_log": True, "log_pos": 16},
+ )
+
+ # When remote_logs is implemented, previous try_number is from remote logs without reaching worker server
fth._read_from_logs_server.reset_mock()
fth._read_remote_logs = mock.Mock()
fth._read_remote_logs.return_value = ["remote logs"], ["remote\nlog\ncontent"]
(airflow) 46/48: Add Grid button to Task Instance view (#39223)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit df9db0cb15e2acfe768218f5401550c17b302cde
Author: Carlos Sánchez Páez <ca...@feverup.com>
AuthorDate: Thu Apr 25 17:17:47 2024 +0200
Add Grid button to Task Instance view (#39223)
(cherry picked from commit 8c556dab0db01324f40685f879b3410a488c834b)
---
airflow/www/templates/airflow/task_instance.html | 3 +++
airflow/www/views.py | 6 ++++++
tests/www/views/test_views_tasks.py | 20 ++++++++++++++++++++
3 files changed, 29 insertions(+)
diff --git a/airflow/www/templates/airflow/task_instance.html b/airflow/www/templates/airflow/task_instance.html
index f7faff3a53..9c5953ae28 100644
--- a/airflow/www/templates/airflow/task_instance.html
+++ b/airflow/www/templates/airflow/task_instance.html
@@ -39,6 +39,9 @@
{{ url_for(endpoint, dag_id=dag.dag_id, task_id=task_id, execution_date=execution_date) }}
{%- endif -%}
{% endmacro -%}
+ <li><a href="{{ url_for('Airflow.grid', dag_id=dag.dag_id, dag_run_id=dag_run_id ,task_id=task_id, map_index=map_index) }}">
+ <span class="material-icons" aria-hidden="true">grid_on</span>
+ Grid</a></li>
<li><a href="{{ ti_url('Airflow.task') }}">
<span class="material-icons" aria-hidden="true">details</span>
Task Instance Details</a></li>
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 5bdebe5d91..b82f844f71 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -1393,6 +1393,7 @@ class Airflow(AirflowBaseView):
show_trigger_form_if_no_params=conf.getboolean(
"webserver", "show_trigger_form_if_no_params"
),
+ dag_run_id=dag_run.run_id if dag_run else "",
html_dict=html_dict,
dag=dag,
task_id=task_id,
@@ -1458,6 +1459,7 @@ class Airflow(AirflowBaseView):
"airflow/ti_code.html",
show_trigger_form_if_no_params=conf.getboolean("webserver", "show_trigger_form_if_no_params"),
html_dict=html_dict,
+ dag_run_id=dag_run.run_id if dag_run else "",
dag=dag,
task_id=task_id,
task_display_name=task.task_display_name,
@@ -1522,6 +1524,7 @@ class Airflow(AirflowBaseView):
return self.render_template(
"airflow/ti_code.html",
show_trigger_form_if_no_params=conf.getboolean("webserver", "show_trigger_form_if_no_params"),
+ dag_run_id=dag_run.run_id if dag_run else "",
html_dict={"k8s": content},
dag=dag,
task_id=task_id,
@@ -1648,6 +1651,7 @@ class Airflow(AirflowBaseView):
show_trigger_form_if_no_params=conf.getboolean("webserver", "show_trigger_form_if_no_params"),
logs=logs,
dag=dag_model,
+ dag_run_id=ti.run_id if ti else "",
title="Log by attempts",
dag_id=dag_id,
task_id=task_id,
@@ -1804,6 +1808,7 @@ class Airflow(AirflowBaseView):
show_trigger_form_if_no_params=conf.getboolean("webserver", "show_trigger_form_if_no_params"),
task_attrs=task_attrs,
ti_attrs=ti_attrs,
+ dag_run_id=ti.run_id if ti else "",
failed_dep_reasons=failed_dep_reasons or no_failed_deps_result,
task_id=task_id,
execution_date=execution_date,
@@ -1856,6 +1861,7 @@ class Airflow(AirflowBaseView):
show_trigger_form_if_no_params=conf.getboolean("webserver", "show_trigger_form_if_no_params"),
attributes=attributes,
task_id=task_id,
+ dag_run_id=ti.run_id if ti else "",
task_display_name=ti.task_display_name,
execution_date=execution_date,
map_index=map_index,
diff --git a/tests/www/views/test_views_tasks.py b/tests/www/views/test_views_tasks.py
index 90cc5bbc1c..83f57d0751 100644
--- a/tests/www/views/test_views_tasks.py
+++ b/tests/www/views/test_views_tasks.py
@@ -159,22 +159,42 @@ def client_ti_without_dag_edit(app):
],
id="delete-dag-button-normal",
),
+ pytest.param(
+ f"task?task_id=runme_0&dag_id=example_bash_operator&execution_date={DEFAULT_VAL}",
+ ["Grid"],
+ id="task-grid-button",
+ ),
pytest.param(
f"task?task_id=runme_0&dag_id=example_bash_operator&execution_date={DEFAULT_VAL}",
["Task Instance Details"],
id="task",
),
+ pytest.param(
+ f"log?task_id=runme_0&dag_id=example_bash_operator&execution_date={DEFAULT_VAL}",
+ ["Grid"],
+ id="log-grid-button",
+ ),
pytest.param(
f"xcom?task_id=runme_0&dag_id=example_bash_operator&execution_date={DEFAULT_VAL}",
["XCom"],
id="xcom",
),
+ pytest.param(
+ f"xcom?task_id=runme_0&dag_id=example_bash_operator&execution_date={DEFAULT_VAL}",
+ ["Grid"],
+ id="xcom-grid-button",
+ ),
pytest.param("xcom/list", ["List XComs"], id="xcom-list"),
pytest.param(
f"rendered-templates?task_id=runme_0&dag_id=example_bash_operator&execution_date={DEFAULT_VAL}",
["Rendered Template"],
id="rendered-templates",
),
+ pytest.param(
+ f"rendered-templates?task_id=runme_0&dag_id=example_bash_operator&execution_date={DEFAULT_VAL}",
+ ["Grid"],
+ id="rendered-templates-grid-button",
+ ),
pytest.param(
"object/graph_data?dag_id=example_bash_operator",
["runme_1"],
(airflow) 40/48: Add link to `user` and `role` commands (#39224)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c5836745e875a612adca3558b8c79862c66dc82e
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Apr 24 13:05:32 2024 +0200
Add link to `user` and `role` commands (#39224)
Fixes: #39197
(cherry picked from commit fd1b5f5819098b04bec4e651a875ca46235e25a6)
---
docs/apache-airflow/cli-and-env-variables-ref.rst | 1 +
1 file changed, 1 insertion(+)
diff --git a/docs/apache-airflow/cli-and-env-variables-ref.rst b/docs/apache-airflow/cli-and-env-variables-ref.rst
index b71379f4da..29db4bb6a5 100644
--- a/docs/apache-airflow/cli-and-env-variables-ref.rst
+++ b/docs/apache-airflow/cli-and-env-variables-ref.rst
@@ -40,6 +40,7 @@ contributed by the community providers:
* Celery Executor and related CLI commands: :doc:`apache-airflow-providers-celery:cli-ref`
* Kubernetes Executor and related CLI commands: :doc:`apache-airflow-providers-cncf-kubernetes:cli-ref`
* AWS and related CLI commands: :doc:`apache-airflow-providers-amazon:cli-ref`
+* The ``users`` and ``roles`` CLI commands are described in FAB provider documentation :doc:`apache-airflow-providers-fab:cli-ref`
.. argparse::
:module: airflow.cli.cli_parser
(airflow) 09/48: Update version added field in config.yml (#38840)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3b84410342738d0b639e1fa47c8ce74b2d199658
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Mon Apr 8 17:15:43 2024 +0100
Update version added field in config.yml (#38840)
(cherry picked from commit 76477b0977c464a8548043ac0651704a3aed9b2c)
---
airflow/config_templates/config.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow/config_templates/config.yml b/airflow/config_templates/config.yml
index 4b086dda5e..c07dcb42cc 100644
--- a/airflow/config_templates/config.yml
+++ b/airflow/config_templates/config.yml
@@ -1956,7 +1956,7 @@ webserver:
Require confirmation when changing a DAG in the web UI. This is to prevent accidental changes
to a DAG that may be running on sensitive environments like production.
When set to True, confirmation dialog will be shown when a user tries to Pause/Unpause, Trigger a DAG
- version_added: ~
+ version_added: 2.9.0
type: boolean
example: ~
default: "False"
(airflow) 15/48: Limit redis image to 7.2 in sample docker-compose.yaml (#38938)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e9986043a10d93a2252fbb2ee36d355128505578
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Thu Apr 11 22:58:47 2024 +0400
Limit redis image to 7.2 in sample docker-compose.yaml (#38938)
(cherry picked from commit cd88ddb4f1617e3d9855e8927d3d7aa4b6daea9a)
---
docs/apache-airflow/howto/docker-compose/docker-compose.yaml | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/docs/apache-airflow/howto/docker-compose/docker-compose.yaml b/docs/apache-airflow/howto/docker-compose/docker-compose.yaml
index 9effc2d504..92b0eab937 100644
--- a/docs/apache-airflow/howto/docker-compose/docker-compose.yaml
+++ b/docs/apache-airflow/howto/docker-compose/docker-compose.yaml
@@ -102,7 +102,9 @@ services:
restart: always
redis:
- image: redis:latest
+ # Redis is limited to 7.2-bookworm due to licencing change
+ # https://redis.io/blog/redis-adopts-dual-source-available-licensing/
+ image: redis:7.2-bookworm
expose:
- 6379
healthcheck:
(airflow) 07/48: Fix 'use case' spelling (#38817)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6d36c80d0b3f497bb3baa0864e497ee69ac65505
Author: Tzu-ping Chung <ur...@gmail.com>
AuthorDate: Mon Apr 8 15:27:04 2024 +0800
Fix 'use case' spelling (#38817)
The top-level site uses 'use case' but the documentation uses
'use-case'. The former seems to be the more popular spelling.
(cherry picked from commit 4cd15f8926b6aa2b25dbef3c961d545d6cbf72be)
---
docs/conf.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/conf.py b/docs/conf.py
index 90c9b68e6f..e959544a09 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -382,7 +382,7 @@ html_theme_options["navbar_links"] = [
{"href": "/community/", "text": "Community"},
{"href": "/meetups/", "text": "Meetups"},
{"href": "/docs/", "text": "Documentation"},
- {"href": "/use-cases/", "text": "Use-cases"},
+ {"href": "/use-cases/", "text": "Use Cases"},
{"href": "/announcements/", "text": "Announcements"},
{"href": "/blog/", "text": "Blog"},
{"href": "/ecosystem/", "text": "Ecosystem"},
(airflow) 45/48: Force to use Airflow Deprecation warnings categories on `@deprecated` decorator (#39205)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d2ae8c97550eaeaefca309d4b5878546d66895c9
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Thu Apr 25 17:44:22 2024 +0400
Force to use Airflow Deprecation warnings categories on `@deprecated` decorator (#39205)
* Force to use Airflow Deprecation warnings categories on @deprecated decorator
* Catch warning in Experimental API test
(cherry picked from commit c2ef1daea65885db04610eaa7e02555500aec0e0)
---
.pre-commit-config.yaml | 7 +
airflow/api/common/experimental/get_code.py | 6 +-
.../api/common/experimental/get_dag_run_state.py | 3 +-
airflow/api/common/experimental/get_task.py | 3 +-
.../api/common/experimental/get_task_instance.py | 4 +-
airflow/api/common/experimental/pool.py | 10 +-
airflow/auth/managers/fab/fab_auth_manager.py | 5 +-
.../auth/managers/fab/security_manager/override.py | 4 +-
airflow/www/security.py | 4 +-
contributing-docs/08_static_code_checks.rst | 2 +
dev/breeze/doc/images/output_static-checks.svg | 152 ++++++++--------
dev/breeze/doc/images/output_static-checks.txt | 2 +-
dev/breeze/src/airflow_breeze/pre_commit_ids.py | 1 +
scripts/ci/pre_commit/check_deprecations.py | 194 +++++++++++++++++++++
.../auth/backend/test_basic_auth.py | 6 +-
15 files changed, 313 insertions(+), 90 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index b2f998afa9..308d63a615 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -987,6 +987,13 @@ repos:
pass_filenames: true
files: ^airflow/.*\.py$
exclude: ^.*/.*_vendor/
+ - id: check-code-deprecations
+ name: Check deprecations categories in decorators
+ entry: ./scripts/ci/pre_commit/check_deprecations.py
+ language: python
+ pass_filenames: true
+ files: ^airflow/.*\.py$
+ exclude: ^.*/.*_vendor/
- id: lint-chart-schema
name: Lint chart/values.schema.json file
entry: ./scripts/ci/pre_commit/chart_schema.py
diff --git a/airflow/api/common/experimental/get_code.py b/airflow/api/common/experimental/get_code.py
index f618690a92..96aadc441b 100644
--- a/airflow/api/common/experimental/get_code.py
+++ b/airflow/api/common/experimental/get_code.py
@@ -22,11 +22,13 @@ from __future__ import annotations
from deprecated import deprecated
from airflow.api.common.experimental import check_and_get_dag
-from airflow.exceptions import AirflowException, DagCodeNotFound
+from airflow.exceptions import AirflowException, DagCodeNotFound, RemovedInAirflow3Warning
from airflow.models.dagcode import DagCode
-@deprecated(reason="Use DagCode().get_code_by_fileloc() instead", version="2.2.4")
+@deprecated(
+ reason="Use DagCode().get_code_by_fileloc() instead", version="2.2.4", category=RemovedInAirflow3Warning
+)
def get_code(dag_id: str) -> str:
"""Return python code of a given dag_id.
diff --git a/airflow/api/common/experimental/get_dag_run_state.py b/airflow/api/common/experimental/get_dag_run_state.py
index 8905aa4f04..6c7cd96c2b 100644
--- a/airflow/api/common/experimental/get_dag_run_state.py
+++ b/airflow/api/common/experimental/get_dag_run_state.py
@@ -24,12 +24,13 @@ from typing import TYPE_CHECKING
from deprecated import deprecated
from airflow.api.common.experimental import check_and_get_dag, check_and_get_dagrun
+from airflow.exceptions import RemovedInAirflow3Warning
if TYPE_CHECKING:
from datetime import datetime
-@deprecated(reason="Use DagRun().get_state() instead", version="2.2.4")
+@deprecated(reason="Use DagRun().get_state() instead", version="2.2.4", category=RemovedInAirflow3Warning)
def get_dag_run_state(dag_id: str, execution_date: datetime) -> dict[str, str]:
"""Return the Dag Run state identified by the given dag_id and execution_date.
diff --git a/airflow/api/common/experimental/get_task.py b/airflow/api/common/experimental/get_task.py
index 6bc58d0328..7e25ff4ffd 100644
--- a/airflow/api/common/experimental/get_task.py
+++ b/airflow/api/common/experimental/get_task.py
@@ -24,12 +24,13 @@ from typing import TYPE_CHECKING
from deprecated import deprecated
from airflow.api.common.experimental import check_and_get_dag
+from airflow.exceptions import RemovedInAirflow3Warning
if TYPE_CHECKING:
from airflow.models import TaskInstance
-@deprecated(reason="Use DAG().get_task", version="2.2.4")
+@deprecated(reason="Use DAG().get_task", version="2.2.4", category=RemovedInAirflow3Warning)
def get_task(dag_id: str, task_id: str) -> TaskInstance:
"""Return the task object identified by the given dag_id and task_id."""
dag = check_and_get_dag(dag_id, task_id)
diff --git a/airflow/api/common/experimental/get_task_instance.py b/airflow/api/common/experimental/get_task_instance.py
index 8a22e2c24e..722120534d 100644
--- a/airflow/api/common/experimental/get_task_instance.py
+++ b/airflow/api/common/experimental/get_task_instance.py
@@ -24,14 +24,14 @@ from typing import TYPE_CHECKING
from deprecated import deprecated
from airflow.api.common.experimental import check_and_get_dag, check_and_get_dagrun
-from airflow.exceptions import TaskInstanceNotFound
+from airflow.exceptions import RemovedInAirflow3Warning, TaskInstanceNotFound
from airflow.models import TaskInstance
if TYPE_CHECKING:
from datetime import datetime
-@deprecated(version="2.2.4", reason="Use DagRun.get_task_instance instead")
+@deprecated(version="2.2.4", reason="Use DagRun.get_task_instance instead", category=RemovedInAirflow3Warning)
def get_task_instance(dag_id: str, task_id: str, execution_date: datetime) -> TaskInstance:
"""Return the task instance identified by the given dag_id, task_id and execution_date."""
dag = check_and_get_dag(dag_id, task_id)
diff --git a/airflow/api/common/experimental/pool.py b/airflow/api/common/experimental/pool.py
index cfa1ffcc71..e3d99be8c1 100644
--- a/airflow/api/common/experimental/pool.py
+++ b/airflow/api/common/experimental/pool.py
@@ -24,7 +24,7 @@ from typing import TYPE_CHECKING
from deprecated import deprecated
from sqlalchemy import select
-from airflow.exceptions import AirflowBadRequest, PoolNotFound
+from airflow.exceptions import AirflowBadRequest, PoolNotFound, RemovedInAirflow3Warning
from airflow.models import Pool
from airflow.utils.session import NEW_SESSION, provide_session
@@ -32,7 +32,7 @@ if TYPE_CHECKING:
from sqlalchemy.orm import Session
-@deprecated(reason="Use Pool.get_pool() instead", version="2.2.4")
+@deprecated(reason="Use Pool.get_pool() instead", version="2.2.4", category=RemovedInAirflow3Warning)
@provide_session
def get_pool(name, session: Session = NEW_SESSION):
"""Get pool by a given name."""
@@ -46,14 +46,14 @@ def get_pool(name, session: Session = NEW_SESSION):
return pool
-@deprecated(reason="Use Pool.get_pools() instead", version="2.2.4")
+@deprecated(reason="Use Pool.get_pools() instead", version="2.2.4", category=RemovedInAirflow3Warning)
@provide_session
def get_pools(session: Session = NEW_SESSION):
"""Get all pools."""
return session.scalars(select(Pool)).all()
-@deprecated(reason="Use Pool.create_pool() instead", version="2.2.4")
+@deprecated(reason="Use Pool.create_pool() instead", version="2.2.4", category=RemovedInAirflow3Warning)
@provide_session
def create_pool(name, slots, description, session: Session = NEW_SESSION):
"""Create a pool with given parameters."""
@@ -84,7 +84,7 @@ def create_pool(name, slots, description, session: Session = NEW_SESSION):
return pool
-@deprecated(reason="Use Pool.delete_pool() instead", version="2.2.4")
+@deprecated(reason="Use Pool.delete_pool() instead", version="2.2.4", category=RemovedInAirflow3Warning)
@provide_session
def delete_pool(name, session: Session = NEW_SESSION):
"""Delete pool by a given name."""
diff --git a/airflow/auth/managers/fab/fab_auth_manager.py b/airflow/auth/managers/fab/fab_auth_manager.py
index ffbda9c808..aa77656639 100644
--- a/airflow/auth/managers/fab/fab_auth_manager.py
+++ b/airflow/auth/managers/fab/fab_auth_manager.py
@@ -19,11 +19,14 @@ from __future__ import annotations
from deprecated import deprecated
+from airflow.exceptions import RemovedInAirflow3Warning
from airflow.providers.fab.auth_manager.fab_auth_manager import FabAuthManager as FabAuthManagerProvider
@deprecated(
- reason="Use airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager instead", version="2.9.0"
+ reason="Use airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager instead",
+ version="2.9.0",
+ category=RemovedInAirflow3Warning,
)
class FabAuthManager(FabAuthManagerProvider):
"""
diff --git a/airflow/auth/managers/fab/security_manager/override.py b/airflow/auth/managers/fab/security_manager/override.py
index 96b98a268b..cae7da7e92 100644
--- a/airflow/auth/managers/fab/security_manager/override.py
+++ b/airflow/auth/managers/fab/security_manager/override.py
@@ -19,6 +19,7 @@ from __future__ import annotations
from deprecated import deprecated
+from airflow.exceptions import RemovedInAirflow3Warning
from airflow.providers.fab.auth_manager.security_manager.override import (
FabAirflowSecurityManagerOverride as FabProviderAirflowSecurityManagerOverride,
)
@@ -27,7 +28,8 @@ from airflow.providers.fab.auth_manager.security_manager.override import (
@deprecated(
reason="If you want to override the security manager, you should inherit from "
"`airflow.providers.fab.auth_manager.security_manager.override.FabAirflowSecurityManagerOverride` "
- "instead"
+ "instead",
+ category=RemovedInAirflow3Warning,
)
class FabAirflowSecurityManagerOverride(FabProviderAirflowSecurityManagerOverride):
"""
diff --git a/airflow/www/security.py b/airflow/www/security.py
index 9cb21cf45a..69c90d65eb 100644
--- a/airflow/www/security.py
+++ b/airflow/www/security.py
@@ -18,6 +18,7 @@ from __future__ import annotations
from deprecated import deprecated
+from airflow.exceptions import RemovedInAirflow3Warning
from airflow.providers.fab.auth_manager.security_manager.override import FabAirflowSecurityManagerOverride
EXISTING_ROLES = {
@@ -32,7 +33,8 @@ EXISTING_ROLES = {
@deprecated(
reason="If you want to override the security manager, you should inherit from "
"`airflow.providers.fab.auth_manager.security_manager.override.FabAirflowSecurityManagerOverride` "
- "instead"
+ "instead",
+ category=RemovedInAirflow3Warning,
)
class AirflowSecurityManager(FabAirflowSecurityManagerOverride):
"""Placeholder, just here to avoid breaking the code of users who inherit from this.
diff --git a/contributing-docs/08_static_code_checks.rst b/contributing-docs/08_static_code_checks.rst
index b02c8749a7..e3fcf28ed4 100644
--- a/contributing-docs/08_static_code_checks.rst
+++ b/contributing-docs/08_static_code_checks.rst
@@ -146,6 +146,8 @@ require Breeze Docker image to be built locally.
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-cncf-k8s-only-for-executors | Check cncf.kubernetes imports used for executors only | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
+| check-code-deprecations | Check deprecations categories in decorators | |
++-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-compat-cache-on-methods | Check that compat cache do not use on class methods | |
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
| check-core-deprecation-classes | Verify usage of Airflow deprecation classes in core | |
diff --git a/dev/breeze/doc/images/output_static-checks.svg b/dev/breeze/doc/images/output_static-checks.svg
index a4c1734c30..f1d369044a 100644
--- a/dev/breeze/doc/images/output_static-checks.svg
+++ b/dev/breeze/doc/images/output_static-checks.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 2099.6" xmlns="http://www.w3.org/2000/svg">
+<svg class="rich-terminal" viewBox="0 0 1482 2124.0" xmlns="http://www.w3.org/2000/svg">
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-static-checks-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="2048.6" />
+ <rect x="0" y="0" width="1463.0" height="2073.0" />
</clipPath>
<clipPath id="breeze-static-checks-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -294,9 +294,12 @@
<clipPath id="breeze-static-checks-line-82">
<rect x="0" y="2002.3" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-static-checks-line-83">
+ <rect x="0" y="2026.7" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2097.6" rx="8"/><text class="breeze-static-checks-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: static-checks</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1" x="1" y="1" width="1480" height="2122" rx="8"/><text class="breeze-static-checks-title" fill="#c5c8c6" text-anchor="middle" x="740" y="27">Command: static-checks</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -319,77 +322,78 @@
</text><text class="breeze-static-checks-r5" x="0" y="264" textLength="12.2" clip-path="url(#breeze-static-checks-line-10)">│</text><text class="breeze-static-checks-r7" x="451.4" y="264" textLength="988.2" clip-path="url(#breeze-static-checks-line-10)">check-base-operator-partial-arguments | check-base-operator-usage |              </text><text class="breeze-static-checks-r5" x="1451.8" y="264" textLeng [...]
</text><text class="breeze-static-checks-r5" x="0" y="288.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-11)">│</text><text class="breeze-static-checks-r7" x="451.4" y="288.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-11)">check-boring-cyborg-configuration | check-breeze-top-dependencies-limited |      </text><text class="breeze-static-checks-r5" x="1451.8" y="288.4" textLength="12.2" clip-path="url(#breeze-s [...]
</text><text class="breeze-static-checks-r5" x="0" y="312.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-12)">│</text><text class="breeze-static-checks-r7" x="451.4" y="312.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-12)">check-builtin-literals | check-changelog-has-no-duplicates |                     </text><text class="breeze-static [...]
-</text><text class="breeze-static-checks-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-13)">│</text><text class="breeze-static-checks-r7" x="451.4" y="337.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-13)">check-cncf-k8s-only-for-executors | check-compat-cache-on-methods |              </text><text class="breeze-static-checks-r5" x="1451.8" y="337.2" te [...]
-</text><text class="breeze-static-checks-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-14)">│</text><text class="breeze-static-checks-r7" x="451.4" y="361.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-14)">check-core-deprecation-classes | check-daysago-import-from-utils |               </text><text class="breeze-static-checks-r5" x="1451.8" y="361. [...]
-</text><text class="breeze-static-checks-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-static-checks-line-15)">│</text><text class="breeze-static-checks-r7" x="451.4" y="386" textLength="988.2" clip-path="url(#breeze-static-checks-line-15)">check-decorated-operator-implements-custom-name | check-deferrable-default-value </text><text class="breeze-static-checks-r5" x="1451.8" y="386" textLength="12.2" clip-path="url(#breeze-static-checks-line-15)">│</text><text [...]
-</text><text class="breeze-static-checks-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-16)">│</text><text class="breeze-static-checks-r7" x="451.4" y="410.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-16)">| check-docstring-param-types | check-example-dags-urls |                        </text><text [...]
-</text><text class="breeze-static-checks-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-17)">│</text><text class="breeze-static-checks-r7" x="451.4" y="434.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-17)">check-executables-have-shebangs | check-extra-packages-references |              </text><text class="breeze-static-checks-r5" x="1451.8" y="434.8" te [...]
-</text><text class="breeze-static-checks-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-18)">│</text><text class="breeze-static-checks-r7" x="451.4" y="459.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-18)">check-extras-order | check-fab-migrations | check-for-inclusive-language |       </text><text class="breeze-static-checks-r5" x="1451.8" y="459.2" textLength="12.2" clip-path [...]
-</text><text class="breeze-static-checks-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-19)">│</text><text class="breeze-static-checks-r7" x="451.4" y="483.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-19)">check-google-re2-as-dependency | check-hatch-build-order | check-hooks-apply |   </text><text class="breeze-static-checks-r5" x="1451.8" y="483.6" textLength="12.2" clip-path="url(#breeze-static [...]
-</text><text class="breeze-static-checks-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-static-checks-line-20)">│</text><text class="breeze-static-checks-r7" x="451.4" y="508" textLength="988.2" clip-path="url(#breeze-static-checks-line-20)">check-incorrect-use-of-LoggingMixin | check-init-decorator-arguments |           </text><text class="breeze-static-checks-r5" x="1451.8" y="508" textLength="12.2" clip- [...]
-</text><text class="breeze-static-checks-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-21)">│</text><text class="breeze-static-checks-r7" x="451.4" y="532.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-21)">check-integrations-list-consistent | check-lazy-logging |                        </text><text class [...]
-</text><text class="breeze-static-checks-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-22)">│</text><text class="breeze-static-checks-r7" x="451.4" y="556.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-22)">check-links-to-example-dags-do-not-use-hardcoded-versions | check-merge-conflict </text><text class="breeze-static-checks-r5" x="1451.8" y="556.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-22)">│</text [...]
-</text><text class="breeze-static-checks-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-23)">│</text><text class="breeze-static-checks-r7" x="451.4" y="581.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-23)">| check-newsfragments-are-valid | check-no-airflow-deprecation-in-providers |    </text><text class="breeze-static-checks-r5" x="1451.8" y="581.2" textLength="12.2" clip-path="url(#breeze-static [...]
-</text><text class="breeze-static-checks-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-24)">│</text><text class="breeze-static-checks-r7" x="451.4" y="605.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-24)">check-no-providers-in-core-examples | check-only-new-session-with-provide-session</text><text class="breeze-static-checks-r5" x="1451.8" y="605.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-24)">│</text><tex [...]
-</text><text class="breeze-static-checks-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-static-checks-line-25)">│</text><text class="breeze-static-checks-r7" x="451.4" y="630" textLength="988.2" clip-path="url(#breeze-static-checks-line-25)">| check-persist-credentials-disabled-in-github-workflows |                       </text><text class="breeze-stati [...]
-</text><text class="breeze-static-checks-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-26)">│</text><text class="breeze-static-checks-r7" x="451.4" y="654.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-26)">check-pre-commit-information-consistent | check-provide-create-sessions-imports |</text><text class="breeze-static-checks-r5" x="1451.8" y="654.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-26)">│</text [...]
-</text><text class="breeze-static-checks-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-27)">│</text><text class="breeze-static-checks-r7" x="451.4" y="678.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-27)">check-provider-docs-valid | check-provider-yaml-valid |                          </text>< [...]
-</text><text class="breeze-static-checks-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-28)">│</text><text class="breeze-static-checks-r7" x="451.4" y="703.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-28)">check-providers-init-file-missing | check-providers-subpackages-init-file-exist |</text><text class="breeze-static-checks-r5" x="1451.8" y="703.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-28)">│</text [...]
-</text><text class="breeze-static-checks-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-29)">│</text><text class="breeze-static-checks-r7" x="451.4" y="727.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-29)">check-pydevd-left-in-code | check-revision-heads-map |                           </t [...]
-</text><text class="breeze-static-checks-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-static-checks-line-30)">│</text><text class="breeze-static-checks-r7" x="451.4" y="752" textLength="988.2" clip-path="url(#breeze-static-checks-line-30)">check-safe-filter-usage-in-html | check-sql-dependency-common-data-structure |   </text><text class="breeze-static-checks-r5" x="1451.8" y="752" textLength="12.2" clip-path="url(#breeze-static-checks-line-30) [...]
-</text><text class="breeze-static-checks-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-31)">│</text><text class="breeze-static-checks-r7" x="451.4" y="776.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-31)">check-start-date-not-used-in-defaults | check-system-tests-present |             </text><text class="breeze-static-checks-r5" x="1451.8" y="776.4" textLen [...]
-</text><text class="breeze-static-checks-r5" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-32)">│</text><text class="breeze-static-checks-r7" x="451.4" y="800.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-32)">check-system-tests-tocs | check-tests-in-the-right-folders |                     </text><text class="breeze-static [...]
-</text><text class="breeze-static-checks-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-33)">│</text><text class="breeze-static-checks-r7" x="451.4" y="825.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-33)">check-tests-unittest-testcase | check-urlparse-usage-in-code |                   </text><text class="breeze-static-checks-r5 [...]
-</text><text class="breeze-static-checks-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-34)">│</text><text class="breeze-static-checks-r7" x="451.4" y="849.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-34)">check-usage-of-re2-over-re | check-xml | codespell | compile-www-assets |        </text><text class="breeze-static-checks-r5" x="1451.8" y="849.6" textLength=" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-static-checks-line-35)">│</text><text class="breeze-static-checks-r7" x="451.4" y="874" textLength="988.2" clip-path="url(#breeze-static-checks-line-35)">compile-www-assets-dev | create-missing-init-py-files-tests | debug-statements | </text><text class="breeze-static-checks-r5" x="1451.8" y="874" textLength="12.2" clip-path="url(#breeze-static-checks-line-35) [...]
-</text><text class="breeze-static-checks-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-36)">│</text><text class="breeze-static-checks-r7" x="451.4" y="898.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-36)">detect-private-key | doctoc | end-of-file-fixer | fix-encoding-pragma | flynt |  </text><text class="breeze-static-checks-r5" x="1451.8" y="898.4" textLength="12.2" clip-path="url [...]
-</text><text class="breeze-static-checks-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-37)">│</text><text class="breeze-static-checks-r7" x="451.4" y="922.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-37)">generate-airflow-diagrams | generate-pypi-readme | identity | insert-license |   </text><text class="breeze-static-checks-r5" x="1451.8" y="922.8" textLength="12.2" clip-path="url(#bre [...]
-</text><text class="breeze-static-checks-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-38)">│</text><text class="breeze-static-checks-r7" x="451.4" y="947.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-38)">kubeconform | lint-chart-schema | lint-css | lint-dockerfile | lint-helm-chart | </text><text class="breeze-static-checks-r5" x="1451.8" y="947.2" textLength="12.2" clip-path="url(#bre [...]
-</text><text class="breeze-static-checks-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-39)">│</text><text class="breeze-static-checks-r7" x="451.4" y="971.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-39)">lint-json-schema | lint-markdown | lint-openapi | mixed-line-ending |            </text><text class="breeze-static-checks-r5" x="1451.8" y= [...]
-</text><text class="breeze-static-checks-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-static-checks-line-40)">│</text><text class="breeze-static-checks-r7" x="451.4" y="996" textLength="988.2" clip-path="url(#breeze-static-checks-line-40)">mypy-airflow | mypy-dev | mypy-docs | mypy-providers | pretty-format-json |      </text><text class="breeze-static-checks-r5" x="1451.8" y="996" textLength="12.2" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-41)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1020.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-41)">pylint | python-no-log-warn | replace-bad-characters | rst-backticks | ruff |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1020.4" textLength="12.2" c [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-42)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1044.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-42)">ruff-format | shellcheck | trailing-whitespace | ts-compile-format-lint-www |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1044.8" textLength="12.2" clip-path=" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-43)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1069.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-43)">update-black-version | update-breeze-cmd-output |                             [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-44)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1093.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-44)">update-breeze-readme-config-hash | update-build-dependencies |                   </text><text class="breeze-static-checks- [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-static-checks-line-45)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1118" textLength="988.2" clip-path="url(#breeze-static-checks-line-45)">update-chart-dependencies | update-common-sql-api-stubs | update-er-diagram |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1118" textLength="12.2" clip-path="url(#breeze-stat [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-46)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1142.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-46)">update-extras | update-in-the-wild-to-be-sorted |                             [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-47)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1166.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-47)">update-inlined-dockerfile-scripts | update-installed-providers-to-be-sorted |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1166.8" textLength="12.2" clip-path="url(#breeze-static-c [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-48)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1191.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-48)">update-installers | update-local-yml-file | update-migration-references |        </text><text class="breeze-static-checks-r5" x="1451.8" y="1191.2" textLength="12.2" c [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-49)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1215.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-49)">update-providers-dependencies | update-reproducible-source-date-epoch |          </text><text class="breeze-static-checks-r5" x="1451.8" y="1215.6" textLength="12.2" c [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-static-checks-line-50)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1240" textLength="988.2" clip-path="url(#breeze-static-checks-line-50)">update-spelling-wordlist-to-be-sorted | update-supported-versions |              </text><text class="breeze-static-checks-r5" x="1451.8" y="1240" textL [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-51)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1264.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-51)">update-vendored-in-k8s-json-schema | update-version | validate-operators-init |  </text><text class="breeze-static-checks-r5" x="1451.8" y="1264.4" textLength="12.2" clip-path="url(#breeze-static-c [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-52)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1288.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-52)">yamllint)                                     & [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1313.2" textLength="61" clip-path="url(#breeze-static-checks-line-53)">-show</text><text class="breeze-static-checks-r4" x="97.6" y="1313.2" textLength="195.2" clip-path="url(# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-54)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1337.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-54)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1337.6" textLength="134.2" clip-path="url(#breeze-static-checks-line-54)">-initialize</text><text class="breeze-static-checks-r4" x="170.8" y="1337.6" textLength="146.4" clip-p [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1362" textLength="12.2" clip-path="url(#breeze-static-checks-line-55)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1362" textLength="12.2" clip-path="url(#breeze-static-checks-line-55)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1362" textLength="48.8" clip-path="url(#breeze-static-checks-line-55)">-max</text><text class="breeze-static-checks-r4" x="85.4" y="1362" textLength="292.8" clip-path="url(#breeze- [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1386.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-56)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1386.4" textLength="854" clip-path="url(#breeze-static-checks-line-56)">(INTEGER RANGE)                                    [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-57)">│</text><text class="breeze-static-checks-r5" x="451.4" y="1410.8" textLength="854" clip-path="url(#breeze-static-checks-line-57)">[default: 3; 1<=x<=10]                                 [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1435.2" textLength="1464" clip-path="url(#breeze-static-checks-line-58)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1435.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-58)">
-</text><text class="breeze-static-checks-r5" x="0" y="1459.6" textLength="24.4" clip-path="url(#breeze-static-checks-line-59)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1459.6" textLength="463.6" clip-path="url(#breeze-static-checks-line-59)"> Selecting files to run the checks on </text><text class="breeze-static-checks-r5" x="488" y="1459.6" textLength="951.6" clip-path="url(#breeze-static-checks-line-59)">──────────────────────── [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1484" textLength="12.2" clip-path="url(#breeze-static-checks-line-60)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1484" textLength="12.2" clip-path="url(#breeze-static-checks-line-60)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1484" textLength="61" clip-path="url(#breeze-static-checks-line-60)">-file</text><text class="breeze-static-checks-r6" x="256.2" y="1484" textLength="24.4" clip-path="url(#breeze-s [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-61)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-61)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1508.4" textLength="48.8" clip-path="url(#breeze-static-checks-line-61)">-all</text><text class="breeze-static-checks-r4" x="85.4" y="1508.4" textLength="73.2" clip-path="url(# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-62)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-62)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1532.8" textLength="85.4" clip-path="url(#breeze-static-checks-line-62)">-commit</text><text class="breeze-static-checks-r4" x="122" y="1532.8" textLength="48.8" clip-path="url [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-63)">│</text><text class="breeze-static-checks-r1" x="305" y="1557.2" textLength="183" clip-path="url(#breeze-static-checks-line-63)">exclusive with </text><text class="breeze-static-checks-r4" x="488" y="1557.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-63)">-</text><text class="breeze-static-checks-r4" x="500.2" y="1557.2" textLength="61" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1581.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-64)">│</text><text class="breeze-static-checks-r7" x="305" y="1581.6" textLength="1134.6" clip-path="url(#breeze-static-checks-line-64)">(TEXT)                                       [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-static-checks-line-65)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1606" textLength="12.2" clip-path="url(#breeze-static-checks-line-65)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1606" textLength="61" clip-path="url(#breeze-static-checks-line-65)">-last</text><text class="breeze-static-checks-r4" x="97.6" y="1606" textLength="85.4" clip-path="url(#breeze-st [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-66)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1630.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-66)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1630.4" textLength="61" clip-path="url(#breeze-static-checks-line-66)">-only</text><text class="breeze-static-checks-r4" x="97.6" y="1630.4" textLength="134.2" clip-path="url(# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-67)">│</text><text class="breeze-static-checks-r1" x="305" y="1654.8" textLength="1134.6" clip-path="url(#breeze-static-checks-line-67)">branch and HEAD of your branch.                              [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1679.2" textLength="1464" clip-path="url(#breeze-static-checks-line-68)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1679.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-68)">
-</text><text class="breeze-static-checks-r5" x="0" y="1703.6" textLength="24.4" clip-path="url(#breeze-static-checks-line-69)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1703.6" textLength="463.6" clip-path="url(#breeze-static-checks-line-69)"> Building image before running checks </text><text class="breeze-static-checks-r5" x="488" y="1703.6" textLength="951.6" clip-path="url(#breeze-static-checks-line-69)">────────────────────────────────── [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1728" textLength="12.2" clip-path="url(#breeze-static-checks-line-70)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1728" textLength="12.2" clip-path="url(#breeze-static-checks-line-70)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1728" textLength="61" clip-path="url(#breeze-static-checks-line-70)">-skip</text><text class="breeze-static-checks-r4" x="97.6" y="1728" textLength="244" clip-path="url(#breeze-sta [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-71)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1752.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-71)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1752.4" textLength="73.2" clip-path="url(#breeze-static-checks-line-71)">-force</text><text class="breeze-static-checks-r4" x="109.8" y="1752.4" textLength="73.2" clip-path="ur [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1776.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-72)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1776.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-72)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1776.8" textLength="73.2" clip-path="url(#breeze-static-checks-line-72)">-image</text><text class="breeze-static-checks-r4" x="109.8" y="1776.8" textLength="48.8" clip-path="ur [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1801.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-73)">│</text><text class="breeze-static-checks-r7" x="414.8" y="1801.2" textLength="963.8" clip-path="url(#breeze-static-checks-line-73)">(TEXT)                                      [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-74)">│</text><text class="breeze-static-checks-r5" x="414.8" y="1825.6" textLength="963.8" clip-path="url(#breeze-static-checks-line-74)">[default: latest]                                    [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-static-checks-line-75)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1850" textLength="12.2" clip-path="url(#breeze-static-checks-line-75)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1850" textLength="85.4" clip-path="url(#breeze-static-checks-line-75)">-github</text><text class="breeze-static-checks-r4" x="122" y="1850" textLength="134.2" clip-path="url(#breez [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1874.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-76)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1874.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-76)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1874.4" textLength="97.6" clip-path="url(#breeze-static-checks-line-76)">-builder</text><text class="breeze-static-checks-r1" x="414.8" y="1874.4" textLength="756.4" clip-path= [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1898.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-77)">│</text><text class="breeze-static-checks-r5" x="414.8" y="1898.8" textLength="756.4" clip-path="url(#breeze-static-checks-line-77)">[default: autodetect]                                  &# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1923.2" textLength="1464" clip-path="url(#breeze-static-checks-line-78)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1923.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-78)">
-</text><text class="breeze-static-checks-r5" x="0" y="1947.6" textLength="24.4" clip-path="url(#breeze-static-checks-line-79)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1947.6" textLength="195.2" clip-path="url(#breeze-static-checks-line-79)"> Common options </text><text class="breeze-static-checks-r5" x="219.6" y="1947.6" textLength="1220" clip-path="url(#breeze-static-checks-line-79)">────────────────────────────────────────────────────────────────────── [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1972" textLength="12.2" clip-path="url(#breeze-static-checks-line-80)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1972" textLength="12.2" clip-path="url(#breeze-static-checks-line-80)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1972" textLength="48.8" clip-path="url(#breeze-static-checks-line-80)">-dry</text><text class="breeze-static-checks-r4" x="85.4" y="1972" textLength="48.8" clip-path="url(#breeze-s [...]
-</text><text class="breeze-static-checks-r5" x="0" y="1996.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-81)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1996.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-81)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1996.4" textLength="97.6" clip-path="url(#breeze-static-checks-line-81)">-verbose</text><text class="breeze-static-checks-r6" x="158.6" y="1996.4" textLength="24.4" clip-path=" [...]
-</text><text class="breeze-static-checks-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-82)">│</text><text class="breeze-static-checks-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-82)">-</text><text class="breeze-static-checks-r4" x="36.6" y="2020.8" textLength="61" clip-path="url(#breeze-static-checks-line-82)">-help</text><text class="breeze-static-checks-r6" x="158.6" y="2020.8" textLength="24.4" clip-path="url(# [...]
-</text><text class="breeze-static-checks-r5" x="0" y="2045.2" textLength="1464" clip-path="url(#breeze-static-checks-line-83)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="2045.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-83)">
+</text><text class="breeze-static-checks-r5" x="0" y="337.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-13)">│</text><text class="breeze-static-checks-r7" x="451.4" y="337.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-13)">check-cncf-k8s-only-for-executors | check-code-deprecations |                    </text><text class="breeze-static-chec [...]
+</text><text class="breeze-static-checks-r5" x="0" y="361.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-14)">│</text><text class="breeze-static-checks-r7" x="451.4" y="361.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-14)">check-compat-cache-on-methods | check-core-deprecation-classes |                 </text><text class="breeze-static-checks-r5" x="1451. [...]
+</text><text class="breeze-static-checks-r5" x="0" y="386" textLength="12.2" clip-path="url(#breeze-static-checks-line-15)">│</text><text class="breeze-static-checks-r7" x="451.4" y="386" textLength="988.2" clip-path="url(#breeze-static-checks-line-15)">check-daysago-import-from-utils | check-decorated-operator-implements-custom-name</text><text class="breeze-static-checks-r5" x="1451.8" y="386" textLength="12.2" clip-path="url(#breeze-static-checks-line-15)">│</text><text clas [...]
+</text><text class="breeze-static-checks-r5" x="0" y="410.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-16)">│</text><text class="breeze-static-checks-r7" x="451.4" y="410.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-16)">| check-deferrable-default-value | check-docstring-param-types |                 </text><text class="breeze-static-checks-r5" x=" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="434.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-17)">│</text><text class="breeze-static-checks-r7" x="451.4" y="434.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-17)">check-example-dags-urls | check-executables-have-shebangs |                      </text><text class="breeze-s [...]
+</text><text class="breeze-static-checks-r5" x="0" y="459.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-18)">│</text><text class="breeze-static-checks-r7" x="451.4" y="459.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-18)">check-extra-packages-references | check-extras-order | check-fab-migrations |    </text><text class="breeze-static-checks-r5" x="1451.8" y="459.2" textLength="12.2" clip-path="url(#breeze-s [...]
+</text><text class="breeze-static-checks-r5" x="0" y="483.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-19)">│</text><text class="breeze-static-checks-r7" x="451.4" y="483.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-19)">check-for-inclusive-language | check-google-re2-as-dependency |                  </text><text class="breeze-static-checks-r5" x=" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="508" textLength="12.2" clip-path="url(#breeze-static-checks-line-20)">│</text><text class="breeze-static-checks-r7" x="451.4" y="508" textLength="988.2" clip-path="url(#breeze-static-checks-line-20)">check-hatch-build-order | check-hooks-apply | check-incorrect-use-of-LoggingMixin</text><text class="breeze-static-checks-r5" x="1451.8" y="508" textLength="12.2" clip-path="url(#breeze-static-checks-line-20)">│</text> [...]
+</text><text class="breeze-static-checks-r5" x="0" y="532.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-21)">│</text><text class="breeze-static-checks-r7" x="451.4" y="532.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-21)">| check-init-decorator-arguments | check-integrations-list-consistent |          </text><text class="breeze-static-checks-r5" x="1451.8" y="532.4" textLength="12.2" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="556.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-22)">│</text><text class="breeze-static-checks-r7" x="451.4" y="556.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-22)">check-lazy-logging | check-links-to-example-dags-do-not-use-hardcoded-versions | </text><text class="breeze-static-checks-r5" x="1451.8" y="556.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-22)">│< [...]
+</text><text class="breeze-static-checks-r5" x="0" y="581.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-23)">│</text><text class="breeze-static-checks-r7" x="451.4" y="581.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-23)">check-merge-conflict | check-newsfragments-are-valid |                           </t [...]
+</text><text class="breeze-static-checks-r5" x="0" y="605.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-24)">│</text><text class="breeze-static-checks-r7" x="451.4" y="605.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-24)">check-no-airflow-deprecation-in-providers | check-no-providers-in-core-examples |</text><text class="breeze-static-checks-r5" x="1451.8" y="605.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-24)">│</text [...]
+</text><text class="breeze-static-checks-r5" x="0" y="630" textLength="12.2" clip-path="url(#breeze-static-checks-line-25)">│</text><text class="breeze-static-checks-r7" x="451.4" y="630" textLength="988.2" clip-path="url(#breeze-static-checks-line-25)">check-only-new-session-with-provide-session |                               &# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="654.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-26)">│</text><text class="breeze-static-checks-r7" x="451.4" y="654.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-26)">check-persist-credentials-disabled-in-github-workflows |                         </text><text class="bre [...]
+</text><text class="breeze-static-checks-r5" x="0" y="678.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-27)">│</text><text class="breeze-static-checks-r7" x="451.4" y="678.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-27)">check-pre-commit-information-consistent | check-provide-create-sessions-imports |</text><text class="breeze-static-checks-r5" x="1451.8" y="678.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-27)">│</text [...]
+</text><text class="breeze-static-checks-r5" x="0" y="703.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-28)">│</text><text class="breeze-static-checks-r7" x="451.4" y="703.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-28)">check-provider-docs-valid | check-provider-yaml-valid |                          </text>< [...]
+</text><text class="breeze-static-checks-r5" x="0" y="727.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-29)">│</text><text class="breeze-static-checks-r7" x="451.4" y="727.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-29)">check-providers-init-file-missing | check-providers-subpackages-init-file-exist |</text><text class="breeze-static-checks-r5" x="1451.8" y="727.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-29)">│</text [...]
+</text><text class="breeze-static-checks-r5" x="0" y="752" textLength="12.2" clip-path="url(#breeze-static-checks-line-30)">│</text><text class="breeze-static-checks-r7" x="451.4" y="752" textLength="988.2" clip-path="url(#breeze-static-checks-line-30)">check-pydevd-left-in-code | check-revision-heads-map |                           </text> [...]
+</text><text class="breeze-static-checks-r5" x="0" y="776.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-31)">│</text><text class="breeze-static-checks-r7" x="451.4" y="776.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-31)">check-safe-filter-usage-in-html | check-sql-dependency-common-data-structure |   </text><text class="breeze-static-checks-r5" x="1451.8" y="776.4" textLength="12.2" clip-path="url(#breeze-static-checks-li [...]
+</text><text class="breeze-static-checks-r5" x="0" y="800.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-32)">│</text><text class="breeze-static-checks-r7" x="451.4" y="800.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-32)">check-start-date-not-used-in-defaults | check-system-tests-present |             </text><text class="breeze-static-checks-r5" x="1451.8" y="800.8" textLen [...]
+</text><text class="breeze-static-checks-r5" x="0" y="825.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-33)">│</text><text class="breeze-static-checks-r7" x="451.4" y="825.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-33)">check-system-tests-tocs | check-tests-in-the-right-folders |                     </text><text class="breeze-static [...]
+</text><text class="breeze-static-checks-r5" x="0" y="849.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-34)">│</text><text class="breeze-static-checks-r7" x="451.4" y="849.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-34)">check-tests-unittest-testcase | check-urlparse-usage-in-code |                   </text><text class="breeze-static-checks-r5 [...]
+</text><text class="breeze-static-checks-r5" x="0" y="874" textLength="12.2" clip-path="url(#breeze-static-checks-line-35)">│</text><text class="breeze-static-checks-r7" x="451.4" y="874" textLength="988.2" clip-path="url(#breeze-static-checks-line-35)">check-usage-of-re2-over-re | check-xml | codespell | compile-www-assets |        </text><text class="breeze-static-checks-r5" x="1451.8" y="874" textLength="12.2" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="898.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-36)">│</text><text class="breeze-static-checks-r7" x="451.4" y="898.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-36)">compile-www-assets-dev | create-missing-init-py-files-tests | debug-statements | </text><text class="breeze-static-checks-r5" x="1451.8" y="898.4" textLength="12.2" clip-path="url(#breeze-static-checks-li [...]
+</text><text class="breeze-static-checks-r5" x="0" y="922.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-37)">│</text><text class="breeze-static-checks-r7" x="451.4" y="922.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-37)">detect-private-key | doctoc | end-of-file-fixer | fix-encoding-pragma | flynt |  </text><text class="breeze-static-checks-r5" x="1451.8" y="922.8" textLength="12.2" clip-path="url [...]
+</text><text class="breeze-static-checks-r5" x="0" y="947.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-38)">│</text><text class="breeze-static-checks-r7" x="451.4" y="947.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-38)">generate-airflow-diagrams | generate-pypi-readme | identity | insert-license |   </text><text class="breeze-static-checks-r5" x="1451.8" y="947.2" textLength="12.2" clip-path="url(#bre [...]
+</text><text class="breeze-static-checks-r5" x="0" y="971.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-39)">│</text><text class="breeze-static-checks-r7" x="451.4" y="971.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-39)">kubeconform | lint-chart-schema | lint-css | lint-dockerfile | lint-helm-chart | </text><text class="breeze-static-checks-r5" x="1451.8" y="971.6" textLength="12.2" clip-path="url(#bre [...]
+</text><text class="breeze-static-checks-r5" x="0" y="996" textLength="12.2" clip-path="url(#breeze-static-checks-line-40)">│</text><text class="breeze-static-checks-r7" x="451.4" y="996" textLength="988.2" clip-path="url(#breeze-static-checks-line-40)">lint-json-schema | lint-markdown | lint-openapi | mixed-line-ending |            </text><text class="breeze-static-checks-r5" x="1451.8" y="996 [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1020.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-41)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1020.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-41)">mypy-airflow | mypy-dev | mypy-docs | mypy-providers | pretty-format-json |      </text><text class="breeze-static-checks-r5" x="1451.8" y="1020.4" textLengt [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1044.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-42)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1044.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-42)">pylint | python-no-log-warn | replace-bad-characters | rst-backticks | ruff |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1044.8" textLength="12.2" c [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1069.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-43)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1069.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-43)">ruff-format | shellcheck | trailing-whitespace | ts-compile-format-lint-www |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1069.2" textLength="12.2" clip-path=" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1093.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-44)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1093.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-44)">update-black-version | update-breeze-cmd-output |                             [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1118" textLength="12.2" clip-path="url(#breeze-static-checks-line-45)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1118" textLength="988.2" clip-path="url(#breeze-static-checks-line-45)">update-breeze-readme-config-hash | update-build-dependencies |                   </text><text class="breeze-static-checks-r5" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1142.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-46)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1142.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-46)">update-chart-dependencies | update-common-sql-api-stubs | update-er-diagram |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1142.4" textLength="12.2" clip-path="url(#breez [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1166.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-47)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1166.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-47)">update-extras | update-in-the-wild-to-be-sorted |                             [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1191.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-48)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1191.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-48)">update-inlined-dockerfile-scripts | update-installed-providers-to-be-sorted |    </text><text class="breeze-static-checks-r5" x="1451.8" y="1191.2" textLength="12.2" clip-path="url(#breeze-static-c [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1215.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-49)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1215.6" textLength="988.2" clip-path="url(#breeze-static-checks-line-49)">update-installers | update-local-yml-file | update-migration-references |        </text><text class="breeze-static-checks-r5" x="1451.8" y="1215.6" textLength="12.2" c [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1240" textLength="12.2" clip-path="url(#breeze-static-checks-line-50)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1240" textLength="988.2" clip-path="url(#breeze-static-checks-line-50)">update-providers-dependencies | update-reproducible-source-date-epoch |          </text><text class="breeze-static-checks-r5" x="1451.8" y="1240" textLength="12.2" clip-pa [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1264.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-51)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1264.4" textLength="988.2" clip-path="url(#breeze-static-checks-line-51)">update-spelling-wordlist-to-be-sorted | update-supported-versions |              </text><text class="breeze-static-checks-r5" x="1451.8" y="1264.4" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-52)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1288.8" textLength="988.2" clip-path="url(#breeze-static-checks-line-52)">update-vendored-in-k8s-json-schema | update-version | validate-operators-init |  </text><text class="breeze-static-checks-r5" x="1451.8" y="1288.8" textLength="12.2" clip-path="url(#breeze-static-c [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1313.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-53)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1313.2" textLength="988.2" clip-path="url(#breeze-static-checks-line-53)">yamllint)                                     & [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1337.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-54)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1337.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-54)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1337.6" textLength="61" clip-path="url(#breeze-static-checks-line-54)">-show</text><text class="breeze-static-checks-r4" x="97.6" y="1337.6" textLength="195.2" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1362" textLength="12.2" clip-path="url(#breeze-static-checks-line-55)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1362" textLength="12.2" clip-path="url(#breeze-static-checks-line-55)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1362" textLength="134.2" clip-path="url(#breeze-static-checks-line-55)">-initialize</text><text class="breeze-static-checks-r4" x="170.8" y="1362" textLength="146.4" clip-path="url [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1386.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-56)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1386.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-56)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1386.4" textLength="48.8" clip-path="url(#breeze-static-checks-line-56)">-max</text><text class="breeze-static-checks-r4" x="85.4" y="1386.4" textLength="292.8" clip-path="url( [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1410.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-57)">│</text><text class="breeze-static-checks-r7" x="451.4" y="1410.8" textLength="854" clip-path="url(#breeze-static-checks-line-57)">(INTEGER RANGE)                                    [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1435.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-58)">│</text><text class="breeze-static-checks-r5" x="451.4" y="1435.2" textLength="854" clip-path="url(#breeze-static-checks-line-58)">[default: 3; 1<=x<=10]                                 [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1459.6" textLength="1464" clip-path="url(#breeze-static-checks-line-59)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1459.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-59)">
+</text><text class="breeze-static-checks-r5" x="0" y="1484" textLength="24.4" clip-path="url(#breeze-static-checks-line-60)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1484" textLength="463.6" clip-path="url(#breeze-static-checks-line-60)"> Selecting files to run the checks on </text><text class="breeze-static-checks-r5" x="488" y="1484" textLength="951.6" clip-path="url(#breeze-static-checks-line-60)">────────────────────────────── [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1508.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-61)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1508.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-61)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1508.4" textLength="61" clip-path="url(#breeze-static-checks-line-61)">-file</text><text class="breeze-static-checks-r6" x="256.2" y="1508.4" textLength="24.4" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1532.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-62)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1532.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-62)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1532.8" textLength="48.8" clip-path="url(#breeze-static-checks-line-62)">-all</text><text class="breeze-static-checks-r4" x="85.4" y="1532.8" textLength="73.2" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1557.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-63)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1557.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-63)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1557.2" textLength="85.4" clip-path="url(#breeze-static-checks-line-63)">-commit</text><text class="breeze-static-checks-r4" x="122" y="1557.2" textLength="48.8" clip-path="url [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1581.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-64)">│</text><text class="breeze-static-checks-r1" x="305" y="1581.6" textLength="183" clip-path="url(#breeze-static-checks-line-64)">exclusive with </text><text class="breeze-static-checks-r4" x="488" y="1581.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-64)">-</text><text class="breeze-static-checks-r4" x="500.2" y="1581.6" textLength="61" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1606" textLength="12.2" clip-path="url(#breeze-static-checks-line-65)">│</text><text class="breeze-static-checks-r7" x="305" y="1606" textLength="1134.6" clip-path="url(#breeze-static-checks-line-65)">(TEXT)                                       [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1630.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-66)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1630.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-66)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1630.4" textLength="61" clip-path="url(#breeze-static-checks-line-66)">-last</text><text class="breeze-static-checks-r4" x="97.6" y="1630.4" textLength="85.4" clip-path="url(#b [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1654.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-67)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1654.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-67)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1654.8" textLength="61" clip-path="url(#breeze-static-checks-line-67)">-only</text><text class="breeze-static-checks-r4" x="97.6" y="1654.8" textLength="134.2" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1679.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-68)">│</text><text class="breeze-static-checks-r1" x="305" y="1679.2" textLength="1134.6" clip-path="url(#breeze-static-checks-line-68)">branch and HEAD of your branch.                              [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1703.6" textLength="1464" clip-path="url(#breeze-static-checks-line-69)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1703.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-69)">
+</text><text class="breeze-static-checks-r5" x="0" y="1728" textLength="24.4" clip-path="url(#breeze-static-checks-line-70)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1728" textLength="463.6" clip-path="url(#breeze-static-checks-line-70)"> Building image before running checks </text><text class="breeze-static-checks-r5" x="488" y="1728" textLength="951.6" clip-path="url(#breeze-static-checks-line-70)">──────────────────────────────────────── [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1752.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-71)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1752.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-71)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1752.4" textLength="61" clip-path="url(#breeze-static-checks-line-71)">-skip</text><text class="breeze-static-checks-r4" x="97.6" y="1752.4" textLength="244" clip-path="url(#br [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1776.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-72)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1776.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-72)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1776.8" textLength="73.2" clip-path="url(#breeze-static-checks-line-72)">-force</text><text class="breeze-static-checks-r4" x="109.8" y="1776.8" textLength="73.2" clip-path="ur [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1801.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-73)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1801.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-73)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1801.2" textLength="73.2" clip-path="url(#breeze-static-checks-line-73)">-image</text><text class="breeze-static-checks-r4" x="109.8" y="1801.2" textLength="48.8" clip-path="ur [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1825.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-74)">│</text><text class="breeze-static-checks-r7" x="414.8" y="1825.6" textLength="963.8" clip-path="url(#breeze-static-checks-line-74)">(TEXT)                                      [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1850" textLength="12.2" clip-path="url(#breeze-static-checks-line-75)">│</text><text class="breeze-static-checks-r5" x="414.8" y="1850" textLength="963.8" clip-path="url(#breeze-static-checks-line-75)">[default: latest]                                    [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1874.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-76)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1874.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-76)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1874.4" textLength="85.4" clip-path="url(#breeze-static-checks-line-76)">-github</text><text class="breeze-static-checks-r4" x="122" y="1874.4" textLength="134.2" clip-path="ur [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1898.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-77)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1898.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-77)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1898.8" textLength="97.6" clip-path="url(#breeze-static-checks-line-77)">-builder</text><text class="breeze-static-checks-r1" x="414.8" y="1898.8" textLength="756.4" clip-path= [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1923.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-78)">│</text><text class="breeze-static-checks-r5" x="414.8" y="1923.2" textLength="756.4" clip-path="url(#breeze-static-checks-line-78)">[default: autodetect]                                  &# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1947.6" textLength="1464" clip-path="url(#breeze-static-checks-line-79)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="1947.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-79)">
+</text><text class="breeze-static-checks-r5" x="0" y="1972" textLength="24.4" clip-path="url(#breeze-static-checks-line-80)">╭─</text><text class="breeze-static-checks-r5" x="24.4" y="1972" textLength="195.2" clip-path="url(#breeze-static-checks-line-80)"> Common options </text><text class="breeze-static-checks-r5" x="219.6" y="1972" textLength="1220" clip-path="url(#breeze-static-checks-line-80)">──────────────────────────────────────────────────────────────────────────── [...]
+</text><text class="breeze-static-checks-r5" x="0" y="1996.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-81)">│</text><text class="breeze-static-checks-r4" x="24.4" y="1996.4" textLength="12.2" clip-path="url(#breeze-static-checks-line-81)">-</text><text class="breeze-static-checks-r4" x="36.6" y="1996.4" textLength="48.8" clip-path="url(#breeze-static-checks-line-81)">-dry</text><text class="breeze-static-checks-r4" x="85.4" y="1996.4" textLength="48.8" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="2020.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-82)">│</text><text class="breeze-static-checks-r4" x="24.4" y="2020.8" textLength="12.2" clip-path="url(#breeze-static-checks-line-82)">-</text><text class="breeze-static-checks-r4" x="36.6" y="2020.8" textLength="97.6" clip-path="url(#breeze-static-checks-line-82)">-verbose</text><text class="breeze-static-checks-r6" x="158.6" y="2020.8" textLength="24.4" clip-path=" [...]
+</text><text class="breeze-static-checks-r5" x="0" y="2045.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-83)">│</text><text class="breeze-static-checks-r4" x="24.4" y="2045.2" textLength="12.2" clip-path="url(#breeze-static-checks-line-83)">-</text><text class="breeze-static-checks-r4" x="36.6" y="2045.2" textLength="61" clip-path="url(#breeze-static-checks-line-83)">-help</text><text class="breeze-static-checks-r6" x="158.6" y="2045.2" textLength="24.4" clip-path="url(# [...]
+</text><text class="breeze-static-checks-r5" x="0" y="2069.6" textLength="1464" clip-path="url(#breeze-static-checks-line-84)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text class="breeze-static-checks-r1" x="1464" y="2069.6" textLength="12.2" clip-path="url(#breeze-static-checks-line-84)">
</text>
</g>
</g>
diff --git a/dev/breeze/doc/images/output_static-checks.txt b/dev/breeze/doc/images/output_static-checks.txt
index ebec23568e..78b9775ad8 100644
--- a/dev/breeze/doc/images/output_static-checks.txt
+++ b/dev/breeze/doc/images/output_static-checks.txt
@@ -1 +1 @@
-e83bbc2b97062b42c940992d0c2d80f6
+bff57d0c14fb8cd14a682610c408b8fd
diff --git a/dev/breeze/src/airflow_breeze/pre_commit_ids.py b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
index eb5c46e1ab..2ca41946e1 100644
--- a/dev/breeze/src/airflow_breeze/pre_commit_ids.py
+++ b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
@@ -38,6 +38,7 @@ PRE_COMMIT_LIST = [
"check-builtin-literals",
"check-changelog-has-no-duplicates",
"check-cncf-k8s-only-for-executors",
+ "check-code-deprecations",
"check-compat-cache-on-methods",
"check-core-deprecation-classes",
"check-daysago-import-from-utils",
diff --git a/scripts/ci/pre_commit/check_deprecations.py b/scripts/ci/pre_commit/check_deprecations.py
new file mode 100755
index 0000000000..aa01e56065
--- /dev/null
+++ b/scripts/ci/pre_commit/check_deprecations.py
@@ -0,0 +1,194 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import ast
+import sys
+from functools import lru_cache
+from pathlib import Path
+
+allowed_warnings: dict[str, tuple[str, ...]] = {
+ "airflow": (
+ "airflow.exceptions.RemovedInAirflow3Warning",
+ "airflow.utils.context.AirflowContextDeprecationWarning",
+ ),
+ "providers": ("airflow.exceptions.AirflowProviderDeprecationWarning",),
+}
+compatible_decorators: frozenset[tuple[str, ...]] = frozenset(
+ [
+ # PEP 702 decorators
+ ("warnings", "deprecated"),
+ ("typing_extensions", "deprecated"),
+ # `Deprecated` package decorators
+ ("deprecated", "deprecated"),
+ ("deprecated", "classic", "deprecated"),
+ ]
+)
+
+
+@lru_cache(maxsize=None)
+def allowed_group_warnings(group: str) -> tuple[str, tuple[str, ...]]:
+ group_warnings = allowed_warnings[group]
+ if len(group_warnings) == 1:
+ return f"expected {group_warnings[0]} type", group_warnings
+ else:
+ return f"expected one of {', '.join(group_warnings)} types", group_warnings
+
+
+def built_import_from(import_from: ast.ImportFrom) -> list[str]:
+ result: list[str] = []
+ module_name = import_from.module
+ if not module_name:
+ return result
+
+ imports_levels = module_name.count(".") + 1
+ for import_path in compatible_decorators:
+ if imports_levels >= len(import_path):
+ continue
+ if module_name != ".".join(import_path[:imports_levels]):
+ continue
+ for name in import_from.names:
+ if name.name == import_path[imports_levels]:
+ alias: str = name.asname or name.name
+ remaining_part = len(import_path) - imports_levels - 1
+ if remaining_part > 0:
+ alias = ".".join([alias, *import_path[-remaining_part:]])
+ result.append(alias)
+ return result
+
+
+def built_import(import_clause: ast.Import) -> list[str]:
+ result = []
+ for name in import_clause.names:
+ module_name = name.name
+ imports_levels = module_name.count(".") + 1
+ for import_path in compatible_decorators:
+ if imports_levels > len(import_path):
+ continue
+ if module_name != ".".join(import_path[:imports_levels]):
+ continue
+
+ alias: str = name.asname or module_name
+ remaining_part = len(import_path) - imports_levels
+ if remaining_part > 0:
+ alias = ".".join([alias, *import_path[-remaining_part:]])
+ result.append(alias)
+ return result
+
+
+def found_compatible_decorators(mod: ast.Module) -> tuple[str, ...]:
+ result = []
+ for node in mod.body:
+ if not isinstance(node, (ast.ImportFrom, ast.Import)):
+ continue
+ result.extend(built_import_from(node) if isinstance(node, ast.ImportFrom) else built_import(node))
+ return tuple(sorted(set(result)))
+
+
+def resolve_name(obj: ast.Attribute | ast.Name) -> str:
+ name = ""
+ while True:
+ if isinstance(obj, ast.Name):
+ name = f"{obj.id}.{name}" if name else obj.id
+ break
+ elif isinstance(obj, ast.Attribute):
+ name = f"{obj.attr}.{name}" if name else obj.attr
+ obj = obj.value # type: ignore[assignment]
+ else:
+ msg = f"Expected to got ast.Name or ast.Attribute but got {type(obj).__name__!r}."
+ raise SystemExit(msg)
+
+ return name
+
+
+def resolve_decorator_name(obj: ast.Call | ast.Attribute | ast.Name) -> str:
+ return resolve_name(obj.func if isinstance(obj, ast.Call) else obj) # type: ignore[arg-type]
+
+
+def check_decorators(mod: ast.Module, file: str, file_group: str) -> int:
+ if not (decorators_names := found_compatible_decorators(mod)):
+ # There are no expected decorators into module, exit early
+ return 0
+
+ errors = 0
+ for node in ast.walk(mod):
+ if not hasattr(node, "decorator_list"):
+ continue
+
+ for decorator in node.decorator_list:
+ decorator_name = resolve_decorator_name(decorator)
+ if decorator_name not in decorators_names:
+ continue
+
+ expected_types, warns_types = allowed_group_warnings(file_group)
+ category_keyword: ast.keyword | None = next(
+ filter(lambda k: k and k.arg == "category", decorator.keywords), None
+ )
+ if category_keyword is None:
+ errors += 1
+ print(
+ f"{file}:{decorator.lineno}: Missing `category` keyword on decorator @{decorator_name}, "
+ f"{expected_types}"
+ )
+ continue
+ elif not hasattr(category_keyword, "value"):
+ continue
+ category_value_ast = category_keyword.value
+
+ warns_types = allowed_warnings[file_group]
+ if isinstance(category_value_ast, (ast.Name, ast.Attribute)):
+ category_value = resolve_name(category_value_ast)
+ if not any(cv.endswith(category_value) for cv in warns_types):
+ errors += 1
+ print(
+ f"{file}:{category_keyword.lineno}: "
+ f"category={category_value}, but {expected_types}"
+ )
+ elif isinstance(category_value_ast, ast.Constant):
+ errors += 1
+ print(
+ f"{file}:{category_keyword.lineno}: "
+ f"category=Literal[{category_value_ast.value!r}], but {expected_types}"
+ )
+
+ return errors
+
+
+def check_file(file: str) -> int:
+ file_path = Path(file)
+ if not file_path.as_posix().startswith("airflow"):
+ # Not expected file, exit early
+ return 0
+ file_group = "providers" if file_path.as_posix().startswith("airflow/providers") else "airflow"
+ ast_module = ast.parse(file_path.read_text("utf-8"), file)
+ errors = 0
+ errors += check_decorators(ast_module, file, file_group)
+ return errors
+
+
+def main(*args: str) -> int:
+ errors = sum(check_file(file) for file in args[1:])
+ if not errors:
+ return 0
+ print(f"Found {errors} error{'s' if errors > 1 else ''}.")
+ return 1
+
+
+if __name__ == "__main__":
+ sys.exit(main(*sys.argv))
diff --git a/tests/api_experimental/auth/backend/test_basic_auth.py b/tests/api_experimental/auth/backend/test_basic_auth.py
index 0d84465dd0..a7f7a2a1cd 100644
--- a/tests/api_experimental/auth/backend/test_basic_auth.py
+++ b/tests/api_experimental/auth/backend/test_basic_auth.py
@@ -21,6 +21,7 @@ from base64 import b64encode
import pytest
from flask_login import current_user
+from airflow.exceptions import RemovedInAirflow3Warning
from tests.test_utils.db import clear_db_pools
pytestmark = pytest.mark.db_test
@@ -49,7 +50,10 @@ class TestBasicAuth:
clear_db_pools()
with self.app.test_client() as test_client:
- response = test_client.get("/api/experimental/pools", headers={"Authorization": token})
+ with pytest.warns(RemovedInAirflow3Warning, match=r"Use Pool.get_pools\(\) instead"):
+ # Experimental client itself deprecated, no reason to change to actual methods
+ # It should be removed in the same time: Airflow 3.0
+ response = test_client.get("/api/experimental/pools", headers={"Authorization": token})
assert current_user.email == "test@fab.org"
assert response.status_code == 200
(airflow) 48/48: Fix trigger kwarg encryption migration (#39246)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c9cc72608d66b1481ef1029499ce54e30b178b12
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Thu Apr 25 15:20:40 2024 -0400
Fix trigger kwarg encryption migration (#39246)
Do the encryption in the migration itself, and fix support for offline
migrations as well.
The offline up migration won't actually encrypt the trigger kwargs as there
isn't a safe way to accomplish that, so the decryption processes checks
and short circuits if it isn't encrypted.
The offline down migration will now print out a warning that the offline
migration will fail if there are any running triggers. I think this is
the best we can do for that scenario (and folks willing to do offline
migrations will hopefully be able to understand the situation).
This also solves the "encrypting the already encrypted kwargs" bug in
2.9.0.
(cherry picked from commit adeb7f7cba2ab2b16be2e006c17e140fe91fdf77)
---
.../0140_2_9_0_update_trigger_kwargs_type.py | 46 +++++++++++++++++++---
airflow/models/trigger.py | 10 ++++-
airflow/utils/db.py | 39 ------------------
docs/apache-airflow/img/airflow_erd.sha256 | 2 +-
docs/apache-airflow/img/airflow_erd.svg | 4 +-
docs/apache-airflow/migrations-ref.rst | 2 +-
tests/models/test_trigger.py | 17 ++++++++
7 files changed, 70 insertions(+), 50 deletions(-)
diff --git a/airflow/migrations/versions/0140_2_9_0_update_trigger_kwargs_type.py b/airflow/migrations/versions/0140_2_9_0_update_trigger_kwargs_type.py
index dbde1201e4..2d57686e43 100644
--- a/airflow/migrations/versions/0140_2_9_0_update_trigger_kwargs_type.py
+++ b/airflow/migrations/versions/0140_2_9_0_update_trigger_kwargs_type.py
@@ -16,18 +16,22 @@
# specific language governing permissions and limitations
# under the License.
-"""update trigger kwargs type
+"""update trigger kwargs type and encrypt
Revision ID: 1949afb29106
Revises: ee1467d4aa35
Create Date: 2024-03-17 22:09:09.406395
"""
+import json
+from textwrap import dedent
+
+from alembic import context, op
import sqlalchemy as sa
+from sqlalchemy.orm import lazyload
+from airflow.serialization.serialized_objects import BaseSerialization
from airflow.models.trigger import Trigger
-from alembic import op
-
from airflow.utils.sqlalchemy import ExtendedJSON
# revision identifiers, used by Alembic.
@@ -38,13 +42,43 @@ depends_on = None
airflow_version = "2.9.0"
+def get_session() -> sa.orm.Session:
+ conn = op.get_bind()
+ sessionmaker = sa.orm.sessionmaker()
+ return sessionmaker(bind=conn)
+
def upgrade():
- """Update trigger kwargs type to string"""
+ """Update trigger kwargs type to string and encrypt"""
with op.batch_alter_table("trigger") as batch_op:
batch_op.alter_column("kwargs", type_=sa.Text(), )
+ if not context.is_offline_mode():
+ session = get_session()
+ try:
+ for trigger in session.query(Trigger).options(lazyload(Trigger.task_instance)):
+ trigger.kwargs = trigger.kwargs
+ session.commit()
+ finally:
+ session.close()
+
def downgrade():
- """Unapply update trigger kwargs type to string"""
+ """Unapply update trigger kwargs type to string and encrypt"""
+ if context.is_offline_mode():
+ print(dedent("""
+ ------------
+ -- WARNING: Unable to decrypt trigger kwargs automatically in offline mode!
+ -- If any trigger rows exist when you do an offline downgrade, the migration will fail.
+ ------------
+ """))
+ else:
+ session = get_session()
+ try:
+ for trigger in session.query(Trigger).options(lazyload(Trigger.task_instance)):
+ trigger.encrypted_kwargs = json.dumps(BaseSerialization.serialize(trigger.kwargs))
+ session.commit()
+ finally:
+ session.close()
+
with op.batch_alter_table("trigger") as batch_op:
- batch_op.alter_column("kwargs", type_=ExtendedJSON(), postgresql_using="kwargs::json")
+ batch_op.alter_column("kwargs", type_=ExtendedJSON(), postgresql_using='kwargs::json')
diff --git a/airflow/models/trigger.py b/airflow/models/trigger.py
index 32fec3b343..cde76ca241 100644
--- a/airflow/models/trigger.py
+++ b/airflow/models/trigger.py
@@ -116,7 +116,15 @@ class Trigger(Base):
from airflow.models.crypto import get_fernet
from airflow.serialization.serialized_objects import BaseSerialization
- decrypted_kwargs = json.loads(get_fernet().decrypt(encrypted_kwargs.encode("utf-8")).decode("utf-8"))
+ # We weren't able to encrypt the kwargs in all migration paths,
+ # so we need to handle the case where they are not encrypted.
+ # Triggers aren't long lasting, so we can skip encrypting them now.
+ if encrypted_kwargs.startswith("{"):
+ decrypted_kwargs = json.loads(encrypted_kwargs)
+ else:
+ decrypted_kwargs = json.loads(
+ get_fernet().decrypt(encrypted_kwargs.encode("utf-8")).decode("utf-8")
+ )
return BaseSerialization.deserialize(decrypted_kwargs)
diff --git a/airflow/utils/db.py b/airflow/utils/db.py
index c0d282a587..b7997498bb 100644
--- a/airflow/utils/db.py
+++ b/airflow/utils/db.py
@@ -972,33 +972,6 @@ def synchronize_log_template(*, session: Session = NEW_SESSION) -> None:
session.add(LogTemplate(filename=filename, elasticsearch_id=elasticsearch_id))
-def encrypt_trigger_kwargs(*, session: Session) -> None:
- """Encrypt trigger kwargs."""
- from airflow.models.trigger import Trigger
- from airflow.serialization.serialized_objects import BaseSerialization
-
- for trigger in session.query(Trigger):
- # convert serialized dict to string and encrypt it
- trigger.kwargs = BaseSerialization.deserialize(json.loads(trigger.encrypted_kwargs))
- session.commit()
-
-
-def decrypt_trigger_kwargs(*, session: Session) -> None:
- """Decrypt trigger kwargs."""
- from airflow.models.trigger import Trigger
- from airflow.serialization.serialized_objects import BaseSerialization
-
- if not inspect(session.bind).has_table(Trigger.__tablename__):
- # table does not exist, nothing to do
- # this can happen when we downgrade to an old version before the Trigger table was added
- return
-
- for trigger in session.scalars(select(Trigger.encrypted_kwargs)):
- # decrypt the string and convert it to serialized dict
- trigger.encrypted_kwargs = json.dumps(BaseSerialization.serialize(trigger.kwargs))
- session.commit()
-
-
def check_conn_id_duplicates(session: Session) -> Iterable[str]:
"""
Check unique conn_id in connection table.
@@ -1666,12 +1639,6 @@ def upgradedb(
_reserialize_dags(session=session)
add_default_pool_if_not_exists(session=session)
synchronize_log_template(session=session)
- if _revision_greater(
- config,
- _REVISION_HEADS_MAP["2.9.0"],
- _get_current_revision(session=session),
- ):
- encrypt_trigger_kwargs(session=session)
@provide_session
@@ -1744,12 +1711,6 @@ def downgrade(*, to_revision, from_revision=None, show_sql_only=False, session:
else:
log.info("Applying downgrade migrations.")
command.downgrade(config, revision=to_revision, sql=show_sql_only)
- if _revision_greater(
- config,
- _REVISION_HEADS_MAP["2.9.0"],
- to_revision,
- ):
- decrypt_trigger_kwargs(session=session)
def drop_airflow_models(connection):
diff --git a/docs/apache-airflow/img/airflow_erd.sha256 b/docs/apache-airflow/img/airflow_erd.sha256
index 09f84daea2..6f623a5a64 100644
--- a/docs/apache-airflow/img/airflow_erd.sha256
+++ b/docs/apache-airflow/img/airflow_erd.sha256
@@ -1 +1 @@
-2a24225537326f38be5df14e0b7a8dca867122093e0fa932f1a11ac12d1fb11c
\ No newline at end of file
+3eb263f117248f914f64bf7cf44757526ecc00f222677629a11602f3bae7cdf0
\ No newline at end of file
diff --git a/docs/apache-airflow/img/airflow_erd.svg b/docs/apache-airflow/img/airflow_erd.svg
index dc32fe0566..3dbb55dd4b 100644
--- a/docs/apache-airflow/img/airflow_erd.svg
+++ b/docs/apache-airflow/img/airflow_erd.svg
@@ -1375,14 +1375,14 @@
<g id="edge43" class="edge">
<title>task_instance--xcom</title>
<path fill="none" stroke="#7f7f7f" stroke-dasharray="5,2" d="M1198.1,-831.83C1228.72,-828.75 1260.55,-825.32 1290.36,-821.86"/>
-<text text-anchor="start" x="1280.36" y="-825.66" font-family="Times,serif" font-size="14.00">1</text>
+<text text-anchor="start" x="1259.36" y="-825.66" font-family="Times,serif" font-size="14.00">0..N</text>
<text text-anchor="start" x="1198.1" y="-835.63" font-family="Times,serif" font-size="14.00">1</text>
</g>
<!-- task_instance--xcom -->
<g id="edge44" class="edge">
<title>task_instance--xcom</title>
<path fill="none" stroke="#7f7f7f" stroke-dasharray="5,2" d="M1198.1,-845.13C1228.72,-842.41 1260.55,-838.85 1290.36,-834.82"/>
-<text text-anchor="start" x="1259.36" y="-838.62" font-family="Times,serif" font-size="14.00">0..N</text>
+<text text-anchor="start" x="1280.36" y="-838.62" font-family="Times,serif" font-size="14.00">1</text>
<text text-anchor="start" x="1198.1" y="-848.93" font-family="Times,serif" font-size="14.00">1</text>
</g>
<!-- rendered_task_instance_fields -->
diff --git a/docs/apache-airflow/migrations-ref.rst b/docs/apache-airflow/migrations-ref.rst
index 76ad56631f..e8faea6d0d 100644
--- a/docs/apache-airflow/migrations-ref.rst
+++ b/docs/apache-airflow/migrations-ref.rst
@@ -39,7 +39,7 @@ Here's the list of all the Database Migrations that are executed via when you ru
+---------------------------------+-------------------+-------------------+--------------------------------------------------------------+
| Revision ID | Revises ID | Airflow Version | Description |
+=================================+===================+===================+==============================================================+
-| ``1949afb29106`` (head) | ``ee1467d4aa35`` | ``2.9.0`` | update trigger kwargs type |
+| ``1949afb29106`` (head) | ``ee1467d4aa35`` | ``2.9.0`` | update trigger kwargs type and encrypt |
+---------------------------------+-------------------+-------------------+--------------------------------------------------------------+
| ``ee1467d4aa35`` | ``b4078ac230a1`` | ``2.9.0`` | add display name for dag and task instance |
+---------------------------------+-------------------+-------------------+--------------------------------------------------------------+
diff --git a/tests/models/test_trigger.py b/tests/models/test_trigger.py
index a3dd6ce35a..6be2086f34 100644
--- a/tests/models/test_trigger.py
+++ b/tests/models/test_trigger.py
@@ -17,6 +17,7 @@
from __future__ import annotations
import datetime
+import json
from typing import Any, AsyncIterator
import pytest
@@ -27,6 +28,7 @@ from airflow.jobs.job import Job
from airflow.jobs.triggerer_job_runner import TriggererJobRunner
from airflow.models import TaskInstance, Trigger
from airflow.operators.empty import EmptyOperator
+from airflow.serialization.serialized_objects import BaseSerialization
from airflow.triggers.base import BaseTrigger, TriggerEvent
from airflow.utils import timezone
from airflow.utils.session import create_session
@@ -378,3 +380,18 @@ def test_serialize_sensitive_kwargs():
assert isinstance(trigger_row.encrypted_kwargs, str)
assert "value1" not in trigger_row.encrypted_kwargs
assert "value2" not in trigger_row.encrypted_kwargs
+
+
+def test_kwargs_not_encrypted():
+ """
+ Tests that we don't decrypt kwargs if they aren't encrypted.
+ We weren't able to encrypt the kwargs in all migration paths.
+ """
+ trigger = Trigger(classpath="airflow.triggers.testing.SuccessTrigger", kwargs={})
+ # force the `encrypted_kwargs` to be unencrypted, like they would be after an offline upgrade
+ trigger.encrypted_kwargs = json.dumps(
+ BaseSerialization.serialize({"param1": "value1", "param2": "value2"})
+ )
+
+ assert trigger.kwargs["param1"] == "value1"
+ assert trigger.kwargs["param2"] == "value2"
(airflow) 25/48: Undeprecate `BaseXCom.get_one` method for now (#38991)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 083a781d1f01818cc1ecba05b1651880d5addae8
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Mon Apr 15 12:51:28 2024 +0400
Undeprecate `BaseXCom.get_one` method for now (#38991)
(cherry picked from commit 260d3fd6e2c7437e69f9a2a701923ee8333b9535)
---
airflow/models/xcom.py | 2 --
1 file changed, 2 deletions(-)
diff --git a/airflow/models/xcom.py b/airflow/models/xcom.py
index be55837770..f9afda123c 100644
--- a/airflow/models/xcom.py
+++ b/airflow/models/xcom.py
@@ -29,7 +29,6 @@ from functools import cached_property, wraps
from typing import TYPE_CHECKING, Any, Generator, Iterable, cast, overload
import attr
-from deprecated import deprecated
from sqlalchemy import (
Column,
ForeignKeyConstraint,
@@ -370,7 +369,6 @@ class BaseXCom(TaskInstanceDependencies, LoggingMixin):
@staticmethod
@provide_session
@internal_api_call
- @deprecated
def get_one(
execution_date: datetime.datetime | None = None,
key: str | None = None,
(airflow) 26/48: Bugfix: Move rendering of `map_index_template` so it renders for failed tasks as long as it was defined before the point of failure (#38902)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3487fccfe27d959f6ed1b407cfb613635f88a839
Author: Tamara Janina Fingerlin <90...@users.noreply.github.com>
AuthorDate: Mon Apr 15 13:05:31 2024 +0200
Bugfix: Move rendering of `map_index_template` so it renders for failed tasks as long as it was defined before the point of failure (#38902)
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
(cherry picked from commit 456ec48d12be02ca7266f021a16e01abb5d4c5a3)
---
airflow/models/taskinstance.py | 25 ++++++++++++++++++-------
tests/models/test_mappedoperator.py | 29 +++++++++++++++++++++++++++++
2 files changed, 47 insertions(+), 7 deletions(-)
diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
index 42849f88f1..88a495e9eb 100644
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -2658,19 +2658,30 @@ class TaskInstance(Base, LoggingMixin):
previous_state=TaskInstanceState.QUEUED, task_instance=self, session=session
)
- # Execute the task
+ def _render_map_index(context: Context, *, jinja_env: jinja2.Environment | None) -> str | None:
+ """Render named map index if the DAG author defined map_index_template at the task level."""
+ if jinja_env is None or (template := context.get("map_index_template")) is None:
+ return None
+ rendered_map_index = jinja_env.from_string(template).render(context)
+ log.debug("Map index rendered as %s", rendered_map_index)
+ return rendered_map_index
+
+ # Execute the task.
with set_current_context(context):
- result = self._execute_task(context, task_orig)
+ try:
+ result = self._execute_task(context, task_orig)
+ except Exception:
+ # If the task failed, swallow rendering error so it doesn't mask the main error.
+ with contextlib.suppress(jinja2.TemplateSyntaxError, jinja2.UndefinedError):
+ self.rendered_map_index = _render_map_index(context, jinja_env=jinja_env)
+ raise
+ else: # If the task succeeded, render normally to let rendering error bubble up.
+ self.rendered_map_index = _render_map_index(context, jinja_env=jinja_env)
# Run post_execute callback
# Is never MappedOperator at this point
self.task.post_execute(context=context, result=result) # type: ignore[union-attr]
- # DAG authors define map_index_template at the task level
- if jinja_env is not None and (template := context.get("map_index_template")) is not None:
- rendered_map_index = self.rendered_map_index = jinja_env.from_string(template).render(context)
- self.log.info("Map index rendered as %s", rendered_map_index)
-
Stats.incr(f"operator_successes_{self.task.task_type}", tags=self.stats_tags)
# Same metric with tagging
Stats.incr("operator_successes", tags={**self.stats_tags, "task_type": self.task.task_type})
diff --git a/tests/models/test_mappedoperator.py b/tests/models/test_mappedoperator.py
index 64304cf306..0351fab05d 100644
--- a/tests/models/test_mappedoperator.py
+++ b/tests/models/test_mappedoperator.py
@@ -635,6 +635,33 @@ def _create_mapped_with_name_template_taskflow(*, task_id, map_names, template):
return task1.expand(map_name=map_names)
+def _create_named_map_index_renders_on_failure_classic(*, task_id, map_names, template):
+ class HasMapName(BaseOperator):
+ def __init__(self, *, map_name: str, **kwargs):
+ super().__init__(**kwargs)
+ self.map_name = map_name
+
+ def execute(self, context):
+ context["map_name"] = self.map_name
+ raise AirflowSkipException("Imagine this task failed!")
+
+ return HasMapName.partial(task_id=task_id, map_index_template=template).expand(
+ map_name=map_names,
+ )
+
+
+def _create_named_map_index_renders_on_failure_taskflow(*, task_id, map_names, template):
+ from airflow.operators.python import get_current_context
+
+ @task(task_id=task_id, map_index_template=template)
+ def task1(map_name):
+ context = get_current_context()
+ context["map_name"] = map_name
+ raise AirflowSkipException("Imagine this task failed!")
+
+ return task1.expand(map_name=map_names)
+
+
@pytest.mark.parametrize(
"template, expected_rendered_names",
[
@@ -649,6 +676,8 @@ def _create_mapped_with_name_template_taskflow(*, task_id, map_names, template):
[
pytest.param(_create_mapped_with_name_template_classic, id="classic"),
pytest.param(_create_mapped_with_name_template_taskflow, id="taskflow"),
+ pytest.param(_create_named_map_index_renders_on_failure_classic, id="classic-failure"),
+ pytest.param(_create_named_map_index_renders_on_failure_taskflow, id="taskflow-failure"),
],
)
def test_expand_mapped_task_instance_with_named_index(
(airflow) 36/48: Improve task filtering UX (#39119)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 20910721674cf3c7fdef159a48a0661a5f91b708
Author: Brent Bovenzi <br...@astronomer.io>
AuthorDate: Mon Apr 22 10:15:07 2024 -0400
Improve task filtering UX (#39119)
* Improve filtering grid/graph/gantt filtering by task UX
* only show map index if greater than -1
(cherry picked from commit 596e9ea10c1e7359d63532b74f3b7f12b42599e5)
---
airflow/www/static/js/dag/details/FilterTasks.tsx | 76 +++++++++++++++--------
airflow/www/static/js/dag/details/Header.tsx | 2 +-
airflow/www/static/js/dag/details/index.tsx | 2 +-
3 files changed, 52 insertions(+), 28 deletions(-)
diff --git a/airflow/www/static/js/dag/details/FilterTasks.tsx b/airflow/www/static/js/dag/details/FilterTasks.tsx
index 33d830b74d..7bc9b976b9 100644
--- a/airflow/www/static/js/dag/details/FilterTasks.tsx
+++ b/airflow/www/static/js/dag/details/FilterTasks.tsx
@@ -28,9 +28,11 @@ import {
} from "@chakra-ui/react";
import useFilters from "src/dag/useFilters";
import { MdArrowDropDown } from "react-icons/md";
+import { useGridData } from "src/api";
+import { getTask } from "src/utils";
interface Props {
- taskId: string;
+ taskId: string | null;
}
const FilterTasks = ({ taskId }: Props) => {
@@ -40,34 +42,48 @@ const FilterTasks = ({ taskId }: Props) => {
resetRoot,
} = useFilters();
- const onFilterUpstream = () =>
- onFilterTasksChange({
- root: taskId,
- filterUpstream: true,
- filterDownstream: false,
- });
+ const {
+ data: { groups },
+ } = useGridData();
- const onFilterDownstream = () =>
- onFilterTasksChange({
- root: taskId,
- filterUpstream: false,
- filterDownstream: true,
- });
+ const label = "Filter upstream and/or downstream of a task";
- const onFilterAll = () =>
- onFilterTasksChange({
- root: taskId,
- filterUpstream: true,
- filterDownstream: true,
- });
+ if (!root && !taskId) return null;
- const label = "Filter upstream and/or downstream of a task";
+ const onFilterUpstream = () => {
+ if (taskId)
+ onFilterTasksChange({
+ root: taskId,
+ filterUpstream: true,
+ filterDownstream: false,
+ });
+ };
+
+ const onFilterDownstream = () => {
+ if (taskId)
+ onFilterTasksChange({
+ root: taskId,
+ filterUpstream: false,
+ filterDownstream: true,
+ });
+ };
+
+ const onFilterAll = () => {
+ if (taskId)
+ onFilterTasksChange({
+ root: taskId,
+ filterUpstream: true,
+ filterDownstream: true,
+ });
+ };
+
+ const task = root ? getTask({ taskId: root, task: groups }) : undefined;
return (
<Menu>
<MenuButton
as={Button}
- variant="outline"
+ variant={root ? "solid" : "outline"}
colorScheme="blue"
transition="all 0.2s"
title={label}
@@ -75,15 +91,23 @@ const FilterTasks = ({ taskId }: Props) => {
mt={2}
>
<Flex>
- {!root ? "Filter Tasks " : "Clear Task Filter "}
+ {root
+ ? `Filtered on ${task?.label || "unknown"}`
+ : "Filter DAG by task"}
<MdArrowDropDown size="16px" />
</Flex>
</MenuButton>
<MenuList>
- <MenuItem onClick={onFilterUpstream}>Filter Upstream</MenuItem>
- <MenuItem onClick={onFilterDownstream}>Filter Downstream</MenuItem>
- <MenuItem onClick={onFilterAll}>Filter Upstream & Downstream</MenuItem>
- {!!root && <MenuItem onClick={resetRoot}>Reset Root</MenuItem>}
+ {!!root && <MenuItem onClick={resetRoot}>Reset</MenuItem>}
+ {!!taskId && (
+ <>
+ <MenuItem onClick={onFilterUpstream}>Only upstream</MenuItem>
+ <MenuItem onClick={onFilterDownstream}>Only downstream</MenuItem>
+ <MenuItem onClick={onFilterAll}>
+ Both upstream & downstream
+ </MenuItem>
+ </>
+ )}
</MenuList>
</Menu>
);
diff --git a/airflow/www/static/js/dag/details/Header.tsx b/airflow/www/static/js/dag/details/Header.tsx
index d13dfa6c81..78f30f3620 100644
--- a/airflow/www/static/js/dag/details/Header.tsx
+++ b/airflow/www/static/js/dag/details/Header.tsx
@@ -124,7 +124,7 @@ const Header = ({ mapIndex }: Props) => {
</BreadcrumbLink>
</BreadcrumbItem>
)}
- {mapIndex !== undefined && (
+ {mapIndex !== undefined && mapIndex !== -1 && (
<BreadcrumbItem isCurrentPage mt={4}>
<BreadcrumbLink
_hover={isMappedTaskDetails ? { cursor: "default" } : undefined}
diff --git a/airflow/www/static/js/dag/details/index.tsx b/airflow/www/static/js/dag/details/index.tsx
index 96c9a7c100..b459a82266 100644
--- a/airflow/www/static/js/dag/details/index.tsx
+++ b/airflow/www/static/js/dag/details/index.tsx
@@ -287,7 +287,7 @@ const Details = ({
/>
</>
)}
- {taskId && runId && <FilterTasks taskId={taskId} />}
+ <FilterTasks taskId={taskId} />
</Flex>
</Flex>
<Divider my={2} />
(airflow) 28/48: Hostname was "redacted", not "redact"; remove it when there is no context (#39037)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 394821f47a1575daa201a4f1d2542817d5f653da
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Mon Apr 15 14:09:16 2024 -0400
Hostname was "redacted", not "redact"; remove it when there is no context (#39037)
(cherry picked from commit 82e51760a12a95c6b79e0ca14c575a920a62b2ca)
---
airflow/www/auth.py | 6 ++----
airflow/www/views.py | 10 +++++-----
2 files changed, 7 insertions(+), 9 deletions(-)
diff --git a/airflow/www/auth.py b/airflow/www/auth.py
index 1b3753f87d..f3f36e05e3 100644
--- a/airflow/www/auth.py
+++ b/airflow/www/auth.py
@@ -167,7 +167,7 @@ def _has_access(*, is_authorized: bool, func: Callable, args, kwargs):
return (
render_template(
"airflow/no_roles_permissions.html",
- hostname=get_hostname() if conf.getboolean("webserver", "EXPOSE_HOSTNAME") else "redact",
+ hostname=get_hostname() if conf.getboolean("webserver", "EXPOSE_HOSTNAME") else "",
logout_url=get_auth_manager().get_url_logout(),
),
403,
@@ -234,9 +234,7 @@ def has_access_dag(method: ResourceMethod, access_entity: DagAccessEntity | None
return (
render_template(
"airflow/no_roles_permissions.html",
- hostname=get_hostname()
- if conf.getboolean("webserver", "EXPOSE_HOSTNAME")
- else "redact",
+ hostname=get_hostname() if conf.getboolean("webserver", "EXPOSE_HOSTNAME") else "",
logout_url=get_auth_manager().get_url_logout(),
),
403,
diff --git a/airflow/www/views.py b/airflow/www/views.py
index ce0727ab4d..5bdebe5d91 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -632,7 +632,7 @@ def not_found(error):
return (
render_template(
"airflow/error.html",
- hostname=get_hostname() if conf.getboolean("webserver", "EXPOSE_HOSTNAME") else "redact",
+ hostname=get_hostname() if conf.getboolean("webserver", "EXPOSE_HOSTNAME") else "",
status_code=404,
error_message="Page cannot be found.",
),
@@ -645,7 +645,7 @@ def method_not_allowed(error):
return (
render_template(
"airflow/error.html",
- hostname=get_hostname() if conf.getboolean("webserver", "EXPOSE_HOSTNAME") else "redact",
+ hostname=get_hostname() if conf.getboolean("webserver", "EXPOSE_HOSTNAME") else "",
status_code=405,
error_message="Received an invalid request.",
),
@@ -659,11 +659,11 @@ def show_traceback(error):
return (
render_template(
"airflow/traceback.html",
- python_version=sys.version.split(" ")[0] if is_logged_in else "redact",
- airflow_version=version if is_logged_in else "redact",
+ python_version=sys.version.split(" ")[0] if is_logged_in else "redacted",
+ airflow_version=version if is_logged_in else "redacted",
hostname=get_hostname()
if conf.getboolean("webserver", "EXPOSE_HOSTNAME") and is_logged_in
- else "redact",
+ else "redacted",
info=traceback.format_exc()
if conf.getboolean("webserver", "EXPOSE_STACKTRACE") and is_logged_in
else "Error! Please contact server admin.",
(airflow) 38/48: Update `is_authorized_custom_view` from auth manager to handle custom actions (#39167)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 092ceaed55fff2d09c5a975b255d33d3d29106b9
Author: Vincent <97...@users.noreply.github.com>
AuthorDate: Mon Apr 22 12:36:26 2024 -0400
Update `is_authorized_custom_view` from auth manager to handle custom actions (#39167)
(cherry picked from commit f7a2f6032544defa8a00d1f7fa90e91d27eb3a8e)
---
airflow/auth/managers/base_auth_manager.py | 7 +++++--
airflow/providers/amazon/aws/auth_manager/avp/entities.py | 2 +-
airflow/providers/amazon/aws/auth_manager/avp/facade.py | 7 +++++--
.../providers/amazon/aws/auth_manager/aws_auth_manager.py | 2 +-
airflow/providers/fab/auth_manager/fab_auth_manager.py | 7 +++++--
tests/auth/managers/test_base_auth_manager.py | 2 +-
tests/providers/fab/auth_manager/test_fab_auth_manager.py | 13 ++++++++++++-
7 files changed, 30 insertions(+), 10 deletions(-)
diff --git a/airflow/auth/managers/base_auth_manager.py b/airflow/auth/managers/base_auth_manager.py
index 4d5c249235..7bb4e92889 100644
--- a/airflow/auth/managers/base_auth_manager.py
+++ b/airflow/auth/managers/base_auth_manager.py
@@ -237,7 +237,7 @@ class BaseAuthManager(LoggingMixin):
@abstractmethod
def is_authorized_custom_view(
- self, *, method: ResourceMethod, resource_name: str, user: BaseUser | None = None
+ self, *, method: ResourceMethod | str, resource_name: str, user: BaseUser | None = None
):
"""
Return whether the user is authorized to perform a given action on a custom view.
@@ -246,7 +246,10 @@ class BaseAuthManager(LoggingMixin):
the auth manager is used as part of the environment. It can also be a view defined as part of a
plugin defined by a user.
- :param method: the method to perform
+ :param method: the method to perform.
+ The method can also be a string if the action has been defined in a plugin.
+ In that case, the action can be anything (e.g. can_do).
+ See https://github.com/apache/airflow/issues/39144
:param resource_name: the name of the resource
:param user: the user to perform the action on. If not provided (or None), it uses the current user
"""
diff --git a/airflow/providers/amazon/aws/auth_manager/avp/entities.py b/airflow/providers/amazon/aws/auth_manager/avp/entities.py
index f2c6376729..8c2e8855b8 100644
--- a/airflow/providers/amazon/aws/auth_manager/avp/entities.py
+++ b/airflow/providers/amazon/aws/auth_manager/avp/entities.py
@@ -55,7 +55,7 @@ def get_entity_type(resource_type: AvpEntities) -> str:
return AVP_PREFIX_ENTITIES + resource_type.value
-def get_action_id(resource_type: AvpEntities, method: ResourceMethod):
+def get_action_id(resource_type: AvpEntities, method: ResourceMethod | str):
"""
Return action id.
diff --git a/airflow/providers/amazon/aws/auth_manager/avp/facade.py b/airflow/providers/amazon/aws/auth_manager/avp/facade.py
index 010531155e..4bb9515004 100644
--- a/airflow/providers/amazon/aws/auth_manager/avp/facade.py
+++ b/airflow/providers/amazon/aws/auth_manager/avp/facade.py
@@ -75,7 +75,7 @@ class AwsAuthManagerAmazonVerifiedPermissionsFacade(LoggingMixin):
def is_authorized(
self,
*,
- method: ResourceMethod,
+ method: ResourceMethod | str,
entity_type: AvpEntities,
user: AwsAuthManagerUser | None,
entity_id: str | None = None,
@@ -86,7 +86,10 @@ class AwsAuthManagerAmazonVerifiedPermissionsFacade(LoggingMixin):
Check whether the user has permissions to access given resource.
- :param method: the method to perform
+ :param method: the method to perform.
+ The method can also be a string if the action has been defined in a plugin.
+ In that case, the action can be anything (e.g. can_do).
+ See https://github.com/apache/airflow/issues/39144
:param entity_type: the entity type the user accesses
:param user: the user
:param entity_id: the entity ID the user accesses. If not provided, all entities of the type will be
diff --git a/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py b/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py
index eb202acfca..1d2bcf1b19 100644
--- a/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py
+++ b/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py
@@ -207,7 +207,7 @@ class AwsAuthManager(BaseAuthManager):
)
def is_authorized_custom_view(
- self, *, method: ResourceMethod, resource_name: str, user: BaseUser | None = None
+ self, *, method: ResourceMethod | str, resource_name: str, user: BaseUser | None = None
):
return self.avp_facade.is_authorized(
method=method,
diff --git a/airflow/providers/fab/auth_manager/fab_auth_manager.py b/airflow/providers/fab/auth_manager/fab_auth_manager.py
index 7533a214aa..ea5ec403ac 100644
--- a/airflow/providers/fab/auth_manager/fab_auth_manager.py
+++ b/airflow/providers/fab/auth_manager/fab_auth_manager.py
@@ -268,11 +268,14 @@ class FabAuthManager(BaseAuthManager):
)
def is_authorized_custom_view(
- self, *, method: ResourceMethod, resource_name: str, user: BaseUser | None = None
+ self, *, method: ResourceMethod | str, resource_name: str, user: BaseUser | None = None
):
if not user:
user = self.get_user()
- fab_action_name = get_fab_action_from_method_map()[method]
+ if method in get_fab_action_from_method_map():
+ fab_action_name = get_fab_action_from_method_map()[method]
+ else:
+ fab_action_name = method
return (fab_action_name, resource_name) in self._get_user_permissions(user)
@provide_session
diff --git a/tests/auth/managers/test_base_auth_manager.py b/tests/auth/managers/test_base_auth_manager.py
index 04191c4838..64d33f6065 100644
--- a/tests/auth/managers/test_base_auth_manager.py
+++ b/tests/auth/managers/test_base_auth_manager.py
@@ -95,7 +95,7 @@ class EmptyAuthManager(BaseAuthManager):
raise NotImplementedError()
def is_authorized_custom_view(
- self, *, method: ResourceMethod, resource_name: str, user: BaseUser | None = None
+ self, *, method: ResourceMethod | str, resource_name: str, user: BaseUser | None = None
):
raise NotImplementedError()
diff --git a/tests/providers/fab/auth_manager/test_fab_auth_manager.py b/tests/providers/fab/auth_manager/test_fab_auth_manager.py
index 773651fafe..b5f654e382 100644
--- a/tests/providers/fab/auth_manager/test_fab_auth_manager.py
+++ b/tests/providers/fab/auth_manager/test_fab_auth_manager.py
@@ -392,10 +392,21 @@ class TestFabAuthManager:
[(ACTION_CAN_READ, "custom_resource2")],
False,
),
+ (
+ "DUMMY",
+ "custom_resource",
+ [("DUMMY", "custom_resource")],
+ True,
+ ),
],
)
def test_is_authorized_custom_view(
- self, method: ResourceMethod, resource_name: str, user_permissions, expected_result, auth_manager
+ self,
+ method: ResourceMethod | str,
+ resource_name: str,
+ user_permissions,
+ expected_result,
+ auth_manager,
):
user = Mock()
user.perms = user_permissions
(airflow) 14/48: Fix implicit cartessian product in AirflowSecurityManagerV2 (#38913)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 26597046291a7f53b8a73165c19fd946d5887778
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Thu Apr 11 14:24:03 2024 +0400
Fix implicit cartessian product in AirflowSecurityManagerV2 (#38913)
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
(cherry picked from commit e16dc0fc2862bfa5e90e7f080f12b36c992972b7)
---
airflow/www/security_manager.py | 8 ++++----
pyproject.toml | 2 ++
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/airflow/www/security_manager.py b/airflow/www/security_manager.py
index c63de34068..88c2caa32a 100644
--- a/airflow/www/security_manager.py
+++ b/airflow/www/security_manager.py
@@ -199,8 +199,8 @@ class AirflowSecurityManagerV2(LoggingMixin):
if not resource_pk:
return None
composite_pk = json.loads(resource_pk)
- ti = session.scalar(
- select(DagRun)
+ dag_id = session.scalar(
+ select(TaskInstance.dag_id)
.where(
TaskInstance.dag_id == composite_pk[0],
TaskInstance.task_id == composite_pk[1],
@@ -209,9 +209,9 @@ class AirflowSecurityManagerV2(LoggingMixin):
)
.limit(1)
)
- if not ti:
+ if not dag_id:
raise AirflowException("Task instance not found")
- return ti.dag_id
+ return dag_id
def get_pool_name(resource_pk):
if not resource_pk:
diff --git a/pyproject.toml b/pyproject.toml
index 7ab695097c..82e3cbc528 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -478,6 +478,8 @@ norecursedirs = [
log_level = "INFO"
filterwarnings = [
"error::pytest.PytestCollectionWarning",
+ # Avoid building cartesian product which might impact performance
+ "error:SELECT statement has a cartesian product between FROM:sqlalchemy.exc.SAWarning:airflow",
"ignore::DeprecationWarning:flask_appbuilder.filemanager",
"ignore::DeprecationWarning:flask_appbuilder.widgets",
# https://github.com/dpgaspar/Flask-AppBuilder/pull/1940
(airflow) 24/48: Correcting an example taskflow example (#39015)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-9-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d6e397fe11c5b8840f06f1588707da2f442a86eb
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Mon Apr 15 13:53:50 2024 +0530
Correcting an example taskflow example (#39015)
(cherry picked from commit bfd347759d5e86ec3d10c1d78306c44d47aadfca)
---
docs/apache-airflow/tutorial/taskflow.rst | 1 +
1 file changed, 1 insertion(+)
diff --git a/docs/apache-airflow/tutorial/taskflow.rst b/docs/apache-airflow/tutorial/taskflow.rst
index 6d5b3bed25..fbfe7aaf6b 100644
--- a/docs/apache-airflow/tutorial/taskflow.rst
+++ b/docs/apache-airflow/tutorial/taskflow.rst
@@ -453,6 +453,7 @@ Building this dependency is shown in the code below:
"""
order_data_file = "/tmp/order_data.csv"
order_data_df = pd.read_csv(order_data_file)
+ return order_data_df
file_task = FileSensor(task_id="check_file", filepath="/tmp/order_data.csv")