You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2023/03/07 16:16:07 UTC
[airflow] branch v2-5-test updated (c2625fb75c -> c0cd3397f7)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a change to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from c2625fb75c Update issue triage policy (#29554)
new 60d3bdd73f Bump undici from 5.9.1 to 5.19.1 (#29583)
new 46f9cb90d6 Specific use-case: adding packages via requirements.txt in compose (#29598)
new 195d47aa58 fix clear dag run openapi spec responses by adding additional return type (#29600)
new e716b7509c convert moment with timezone to UTC instead of raising an exception (#29606)
new 3aa0c008fc Fix Scheduler crash when clear a previous run of a normal task that is now a mapped task (#29645)
new 18bda7a70f FIx formatting of Dataset inlet/outlet note in TaskFlow concepts (#29678)
new 3297c943bd Move extra links position in grid view (#29703)
new 940464b991 Correct version label for data interval variables in documentation (#29704)
new 47fe890435 Update docs re: template_fields typing and subclasses (#29725)
new 4bbc298703 some fixes in the issue triage process doc (#29730)
new 3cba77fd15 Description of dag_processing.last_duration (#29740)
new 521fb2c090 DAG list sorting lost when switching page (#29756)
new 1d997fd381 Fix compile www assets dev (#29769)
new 9f8895f2d4 better description for limit in api (#29773)
new cc334d563b Removed continue for not in (#29791)
new 9eac84439b Update ref anchor for env var link in Connection how-to doc (#29816)
new 5add348e59 Add a check for not templateable fields (#29821)
new 29b7bf204e POST /dagRuns API should 404 if dag not active (#29860)
new 1ed564fbee Correct link to best_practice(s) section (#29876)
new 995dffc584 replace tree view with grid view in docs (#29879)
new c962688473 Correct argument name of Workday timetable in timetable.rst (#29896)
new 09e90630f0 Normalising the usage of generate_pages (#29898)
new c0cd3397f7 Update min version of python-deamon to fix containerd file limits (#29916)
The 23 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
ISSUE_TRIAGE_PROCESS.rst | 4 +-
.../api_connexion/endpoints/dag_run_endpoint.py | 2 +-
airflow/api_connexion/openapi/v1.yaml | 4 +-
airflow/config_templates/config.yml | 11 ++--
airflow/config_templates/default_airflow.cfg | 11 ++--
airflow/models/dagrun.py | 8 +++
airflow/models/taskinstance.py | 19 ++++++
airflow/serialization/serialized_objects.py | 6 ++
airflow/triggers/temporal.py | 4 +-
.../static/js/dag/details/taskInstance/index.tsx | 2 +-
airflow/www/static/js/types/api-generated.ts | 3 +-
airflow/www/utils.py | 70 ++++++++++++++++-----
airflow/www/views.py | 12 ++--
airflow/www/yarn.lock | 20 +++++-
.../connections.rst | 2 +-
.../logging-monitoring/metrics.rst | 2 +-
docs/apache-airflow/best-practices.rst | 2 +-
docs/apache-airflow/cli-and-env-variables-ref.rst | 2 +-
docs/apache-airflow/core-concepts/operators.rst | 3 +-
docs/apache-airflow/core-concepts/taskflow.rst | 5 +-
docs/apache-airflow/howto/connection.rst | 7 ++-
docs/apache-airflow/howto/custom-operator.rst | 72 +++++++++++++++++++---
docs/apache-airflow/howto/docker-compose/index.rst | 34 ++++++++++
.../howto/dynamic-dag-generation.rst | 2 +-
docs/apache-airflow/howto/timetable.rst | 2 +-
docs/apache-airflow/templates-ref.rst | 8 +--
.../pre_commit_compile_www_assets_dev.py | 1 +
setup.cfg | 2 +-
.../endpoints/test_dag_run_endpoint.py | 15 ++++-
tests/models/test_dagrun.py | 67 +++++++++++++++-----
tests/models/test_taskinstance.py | 16 +++++
tests/serialization/test_dag_serialization.py | 18 +++++-
tests/triggers/test_temporal.py | 14 ++++-
tests/www/test_utils.py | 50 ++++++++++++++-
34 files changed, 409 insertions(+), 91 deletions(-)
[airflow] 12/23: DAG list sorting lost when switching page (#29756)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 521fb2c090fc0d26509d66959dbac385b284d3f6
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Thu Mar 2 20:29:43 2023 +0530
DAG list sorting lost when switching page (#29756)
* Adding sorting keys to generate_pages
* Handling review comments and using the new keys
* Fixing Brent review comments
* Adding docstring
---------
Co-authored-by: Amogh <ad...@cloudera.com>
(cherry picked from commit c8cd90fa92c1597300dbbad4366c2bef49ef6390)
---
airflow/www/utils.py | 70 ++++++++++++++++++++++++++++++++++++++-----------
airflow/www/views.py | 5 ++--
tests/www/test_utils.py | 50 +++++++++++++++++++++++++++++++++--
3 files changed, 106 insertions(+), 19 deletions(-)
diff --git a/airflow/www/utils.py b/airflow/www/utils.py
index 5381709e35..bd5b84cb5e 100644
--- a/airflow/www/utils.py
+++ b/airflow/www/utils.py
@@ -211,7 +211,16 @@ def get_params(**kwargs):
return urlencode({d: v for d, v in kwargs.items() if v is not None}, True)
-def generate_pages(current_page, num_of_pages, search=None, status=None, tags=None, window=7):
+def generate_pages(
+ current_page,
+ num_of_pages,
+ search=None,
+ status=None,
+ tags=None,
+ window=7,
+ sorting_key=None,
+ sorting_direction=None,
+):
"""
Generates the HTML for a paging component using a similar logic to the paging
auto-generated by Flask managed views. The paging component defines a number of
@@ -230,6 +239,9 @@ def generate_pages(current_page, num_of_pages, search=None, status=None, tags=No
:param status: 'all', 'active', or 'paused'
:param tags: array of strings of the current filtered tags
:param window: the number of pages to be shown in the paging component (7 default)
+ :param sorting_key: the sorting key selected for dags, None indicates that sorting is not needed/provided
+ :param sorting_direction: direction of sorting, 'asc' or 'desc',
+ None indicates that sorting is not needed/provided
:return: the HTML string of the paging component
"""
void_link = "javascript:void(0)"
@@ -267,9 +279,15 @@ def generate_pages(current_page, num_of_pages, search=None, status=None, tags=No
is_disabled = "disabled" if current_page <= 0 else ""
- first_node_link = (
- void_link if is_disabled else f"?{get_params(page=0, search=search, status=status, tags=tags)}"
+ qs = get_params(
+ page=0,
+ search=search,
+ status=status,
+ tags=tags,
+ sorting_key=sorting_key,
+ sorting_direction=sorting_direction,
)
+ first_node_link = void_link if is_disabled else f"?{qs}"
output.append(
first_node.format(
href_link=first_node_link,
@@ -279,7 +297,15 @@ def generate_pages(current_page, num_of_pages, search=None, status=None, tags=No
page_link = void_link
if current_page > 0:
- page_link = f"?{get_params(page=current_page - 1, search=search, status=status, tags=tags)}"
+ qs = get_params(
+ page=current_page - 1,
+ search=search,
+ status=status,
+ tags=tags,
+ sorting_key=sorting_key,
+ sorting_direction=sorting_direction,
+ )
+ page_link = f"?{qs}"
output.append(previous_node.format(href_link=page_link, disabled=is_disabled))
@@ -297,30 +323,44 @@ def generate_pages(current_page, num_of_pages, search=None, status=None, tags=No
return page == current
for page in pages:
+ qs = get_params(
+ page=page,
+ search=search,
+ status=status,
+ tags=tags,
+ sorting_key=sorting_key,
+ sorting_direction=sorting_direction,
+ )
vals = {
"is_active": "active" if is_current(current_page, page) else "",
- "href_link": void_link
- if is_current(current_page, page)
- else f"?{get_params(page=page, search=search, status=status, tags=tags)}",
+ "href_link": void_link if is_current(current_page, page) else f"?{qs}",
"page_num": page + 1,
}
output.append(page_node.format(**vals))
is_disabled = "disabled" if current_page >= num_of_pages - 1 else ""
- page_link = (
- void_link
- if current_page >= num_of_pages - 1
- else f"?{get_params(page=current_page + 1, search=search, status=status, tags=tags)}"
+ qs = get_params(
+ page=current_page + 1,
+ search=search,
+ status=status,
+ tags=tags,
+ sorting_key=sorting_key,
+ sorting_direction=sorting_direction,
)
+ page_link = void_link if current_page >= num_of_pages - 1 else f"?{qs}"
output.append(next_node.format(href_link=page_link, disabled=is_disabled))
- last_node_link = (
- void_link
- if is_disabled
- else f"?{get_params(page=last_page, search=search, status=status, tags=tags)}"
+ qs = get_params(
+ page=last_page,
+ search=search,
+ status=status,
+ tags=tags,
+ sorting_key=sorting_key,
+ sorting_direction=sorting_direction,
)
+ last_node_link = void_link if is_disabled else f"?{qs}"
output.append(
last_node.format(
href_link=last_node_link,
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 15ab7fc327..880d447699 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -827,6 +827,8 @@ class Airflow(AirflowBaseView):
search=escape(arg_search_query) if arg_search_query else None,
status=arg_status_filter if arg_status_filter else None,
tags=arg_tags_filter if arg_tags_filter else None,
+ sorting_key=arg_sorting_key if arg_sorting_key else None,
+ sorting_direction=arg_sorting_direction if arg_sorting_direction else None,
),
num_runs=num_runs,
tags=tags,
@@ -3784,8 +3786,7 @@ class Airflow(AirflowBaseView):
audit_logs_count=audit_logs_count,
page_size=PAGE_SIZE,
paging=wwwutils.generate_pages(
- current_page,
- num_of_pages,
+ current_page, num_of_pages, arg_sorting_key, arg_sorting_direction
),
sorting_key=arg_sorting_key,
sorting_direction=arg_sorting_direction,
diff --git a/tests/www/test_utils.py b/tests/www/test_utils.py
index 33b0cc6487..c6f48ee360 100644
--- a/tests/www/test_utils.py
+++ b/tests/www/test_utils.py
@@ -28,10 +28,27 @@ from airflow.www.utils import wrapped_markdown
class TestUtils:
- def check_generate_pages_html(self, current_page, total_pages, window=7, check_middle=False):
+ def check_generate_pages_html(
+ self,
+ current_page,
+ total_pages,
+ window=7,
+ check_middle=False,
+ sorting_key=None,
+ sorting_direction=None,
+ ):
extra_links = 4 # first, prev, next, last
search = "'>\"/><img src=x onerror=alert(1)>"
- html_str = utils.generate_pages(current_page, total_pages, search=search)
+ if sorting_key and sorting_direction:
+ html_str = utils.generate_pages(
+ current_page,
+ total_pages,
+ search=search,
+ sorting_key=sorting_key,
+ sorting_direction=sorting_direction,
+ )
+ else:
+ html_str = utils.generate_pages(current_page, total_pages, search=search)
assert search not in html_str, "The raw search string shouldn't appear in the output"
assert "search=%27%3E%22%2F%3E%3Cimg+src%3Dx+onerror%3Dalert%281%29%3E" in html_str
@@ -47,10 +64,27 @@ class TestUtils:
page_items = ulist_items[2:-2]
mid = int(len(page_items) / 2)
+ all_nodes = []
+ pages = []
+
+ if sorting_key and sorting_direction:
+ last_page = total_pages - 1
+
+ if current_page <= mid or total_pages < window:
+ pages = list(range(0, min(total_pages, window)))
+ elif mid < current_page < last_page - mid:
+ pages = list(range(current_page - mid, current_page + mid + 1))
+ else:
+ pages = list(range(total_pages - window, last_page + 1))
+
+ pages.append(last_page + 1)
+ pages.sort(reverse=True if sorting_direction == "desc" else False)
+
for i, item in enumerate(page_items):
a_node = item.a
href_link = a_node["href"]
node_text = a_node.string
+ all_nodes.append(node_text)
if node_text == str(current_page + 1):
if check_middle:
assert mid == i
@@ -62,6 +96,13 @@ class TestUtils:
assert query["page"] == [str(int(node_text) - 1)]
assert query["search"] == [search]
+ if sorting_key and sorting_direction:
+ if pages[0] == 0:
+ pages = pages[1:]
+ pages = list(map(lambda x: str(x), pages))
+
+ assert pages == all_nodes
+
def test_generate_pager_current_start(self):
self.check_generate_pages_html(current_page=0, total_pages=6)
@@ -71,6 +112,11 @@ class TestUtils:
def test_generate_pager_current_end(self):
self.check_generate_pages_html(current_page=38, total_pages=39)
+ def test_generate_pager_current_start_with_sorting(self):
+ self.check_generate_pages_html(
+ current_page=0, total_pages=4, sorting_key="dag_id", sorting_direction="asc"
+ )
+
def test_params_no_values(self):
"""Should return an empty string if no params are passed"""
assert "" == utils.get_params()
[airflow] 07/23: Move extra links position in grid view (#29703)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3297c943bdc5fffbbdac93ad561bf4e97b0a98a3
Author: Brent Bovenzi <br...@astronomer.io>
AuthorDate: Wed Feb 22 15:33:35 2023 -0500
Move extra links position in grid view (#29703)
(cherry picked from commit 8449180f4818ecaa5c065c65152922335ffe8c5a)
---
airflow/www/static/js/dag/details/taskInstance/index.tsx | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow/www/static/js/dag/details/taskInstance/index.tsx b/airflow/www/static/js/dag/details/taskInstance/index.tsx
index 53e2a4b899..66bd8d74e7 100644
--- a/airflow/www/static/js/dag/details/taskInstance/index.tsx
+++ b/airflow/www/static/js/dag/details/taskInstance/index.tsx
@@ -185,7 +185,6 @@ const TaskInstance = ({
isGroup={isGroup}
/>
</Box>
- <Details instance={instance} group={group} dagId={dagId} />
{!isMapped && (
<ExtraLinks
taskId={taskId}
@@ -194,6 +193,7 @@ const TaskInstance = ({
extraLinks={group?.extraLinks || []}
/>
)}
+ <Details instance={instance} group={group} dagId={dagId} />
</Box>
</TabPanel>
[airflow] 01/23: Bump undici from 5.9.1 to 5.19.1 (#29583)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 60d3bdd73fab1e88bb3b693414fe24ac07f6f91d
Author: Dependabot [bot] <49...@users.noreply.github.com>
AuthorDate: Fri Feb 17 21:51:19 2023 +0400
Bump undici from 5.9.1 to 5.19.1 (#29583)
Bumps [undici](https://github.com/nodejs/undici) from 5.9.1 to 5.19.1.
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v5.9.1...v5.19.1)
---
updated-dependencies:
- dependency-name: undici
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
(cherry picked from commit 792416d4ad495f1e5562e6170f73f4d8f1fa2eff)
---
airflow/www/yarn.lock | 20 +++++++++++++++++---
1 file changed, 17 insertions(+), 3 deletions(-)
diff --git a/airflow/www/yarn.lock b/airflow/www/yarn.lock
index 6a3a6302f3..189ce02e56 100644
--- a/airflow/www/yarn.lock
+++ b/airflow/www/yarn.lock
@@ -3867,6 +3867,13 @@ buffer-from@^1.0.0:
resolved "https://registry.yarnpkg.com/buffer-from/-/buffer-from-1.1.2.tgz#2b146a6fd72e80b4f55d255f35ed59a3a9a41bd5"
integrity sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ==
+busboy@^1.6.0:
+ version "1.6.0"
+ resolved "https://registry.yarnpkg.com/busboy/-/busboy-1.6.0.tgz#966ea36a9502e43cdb9146962523b92f531f6893"
+ integrity sha512-8SFQbg/0hQ9xy3UNTB0YEnsNBbWfhf7RtnzpL7TkBiTBRfrQ9Fxcnz7VJsleJpyp6rVLvXiuORqjlHi5q+PYuA==
+ dependencies:
+ streamsearch "^1.1.0"
+
cacache@^15.0.5:
version "15.2.0"
resolved "https://registry.yarnpkg.com/cacache/-/cacache-15.2.0.tgz#73af75f77c58e72d8c630a7a2858cb18ef523389"
@@ -9574,6 +9581,11 @@ stickyfill@^1.1.1:
resolved "https://registry.yarnpkg.com/stickyfill/-/stickyfill-1.1.1.tgz#39413fee9d025c74a7e59ceecb23784cc0f17f02"
integrity sha1-OUE/7p0CXHSn5ZzuyyN4TMDxfwI=
+streamsearch@^1.1.0:
+ version "1.1.0"
+ resolved "https://registry.yarnpkg.com/streamsearch/-/streamsearch-1.1.0.tgz#404dd1e2247ca94af554e841a8ef0eaa238da764"
+ integrity sha512-Mcc5wHehp9aXz1ax6bZUyY5afg9u2rv5cqQI3mRrYkGC8rW2hM02jWuwjtL++LS5qinSyhj2QfLyNsuc+VsExg==
+
string-length@^4.0.1:
version "4.0.2"
resolved "https://registry.yarnpkg.com/string-length/-/string-length-4.0.2.tgz#a8a8dc7bd5c1a82b9b3c8b87e125f66871b6e57a"
@@ -10188,9 +10200,11 @@ unbox-primitive@^1.0.2:
which-boxed-primitive "^1.0.2"
undici@^5.4.0:
- version "5.9.1"
- resolved "https://registry.yarnpkg.com/undici/-/undici-5.9.1.tgz#fc9fd85dd488f965f153314a63d9426a11f3360b"
- integrity sha512-6fB3a+SNnWEm4CJbgo0/CWR8RGcOCQP68SF4X0mxtYTq2VNN8T88NYrWVBAeSX+zb7bny2dx2iYhP3XHi00omg==
+ version "5.19.1"
+ resolved "https://registry.yarnpkg.com/undici/-/undici-5.19.1.tgz#92b1fd3ab2c089b5a6bd3e579dcda8f1934ebf6d"
+ integrity sha512-YiZ61LPIgY73E7syxCDxxa3LV2yl3sN8spnIuTct60boiiRaE1J8mNWHO8Im2Zi/sFrPusjLlmRPrsyraSqX6A==
+ dependencies:
+ busboy "^1.6.0"
unicode-canonical-property-names-ecmascript@^2.0.0:
version "2.0.0"
[airflow] 10/23: some fixes in the issue triage process doc (#29730)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4bbc298703dce539ff84389f5ae7a19534f8e2e7
Author: Hussein Awala <ho...@gmail.com>
AuthorDate: Thu Feb 23 22:18:38 2023 +0100
some fixes in the issue triage process doc (#29730)
(cherry picked from commit 09654eee335977b0d3607dcf7bc98dc5eb7f8f7e)
---
ISSUE_TRIAGE_PROCESS.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/ISSUE_TRIAGE_PROCESS.rst b/ISSUE_TRIAGE_PROCESS.rst
index 9870c3ade3..a2ec4d9ec6 100644
--- a/ISSUE_TRIAGE_PROCESS.rst
+++ b/ISSUE_TRIAGE_PROCESS.rst
@@ -82,7 +82,7 @@ Being an active and helpful member of the "Issue Triage Team" is actually one of
becoming a committer. By actively helping the users, triaging the issues, responding to them and
involving others (when needed) shows that you are not only willing to help our users and the community,
but are also ready to learn about parts of the projects you are not actively contributing to - all of that
-are super valuable components of being eligible to `become a committer <COMMITTERS.md>`_.
+are super valuable components of being eligible to `become a committer <COMMITTERS.rst>`_.
If you are a member of the triage team and not able to make any commitment, it's best to ask to have yourself
removed from the triage team.
@@ -314,7 +314,7 @@ We use judgment about which Issues to convert to discussions, it's best to alway
Note that we can always convert discussions back to issues.
-**Stale Policy **
+**Stale Policy**
As time passes bug reports that have been accepted may be out dated.
Bot will scan older bug reports and if the report is inactive it will comment
[airflow] 22/23: Normalising the usage of generate_pages (#29898)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 09e90630f0c16beacda24e7d12d07b059f84b960
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Fri Mar 3 22:38:56 2023 +0530
Normalising the usage of generate_pages (#29898)
Co-authored-by: Amogh <ad...@cloudera.com>
(cherry picked from commit f0bd45389c7799884a26d2b9c79b0498683d3e03)
---
airflow/www/views.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 7b535484de..56e3a28a26 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -3786,7 +3786,10 @@ class Airflow(AirflowBaseView):
audit_logs_count=audit_logs_count,
page_size=PAGE_SIZE,
paging=wwwutils.generate_pages(
- current_page, num_of_pages, arg_sorting_key, arg_sorting_direction
+ current_page,
+ num_of_pages,
+ sorting_key=arg_sorting_key if arg_sorting_key else None,
+ sorting_direction=arg_sorting_direction if arg_sorting_direction else None,
),
sorting_key=arg_sorting_key,
sorting_direction=arg_sorting_direction,
[airflow] 16/23: Update ref anchor for env var link in Connection how-to doc (#29816)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9eac84439b692b0e5bc2ef7ca43c2691d3cf3c95
Author: Josh Fell <48...@users.noreply.github.com>
AuthorDate: Thu Mar 2 13:05:18 2023 -0500
Update ref anchor for env var link in Connection how-to doc (#29816)
In the "Managing Connections" how-to doc, the "environment variables" links in the doc point to the "Security of connections in the database" section rather than the "Storing connections in environment variables" section.
(cherry picked from commit b33b11bf9fdbbedbd31699428853896f4db478cb)
---
docs/apache-airflow-providers-dbt-cloud/connections.rst | 2 +-
docs/apache-airflow/cli-and-env-variables-ref.rst | 2 +-
docs/apache-airflow/howto/connection.rst | 7 ++++---
3 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/docs/apache-airflow-providers-dbt-cloud/connections.rst b/docs/apache-airflow-providers-dbt-cloud/connections.rst
index 0b4fd2e362..cf36833ddc 100644
--- a/docs/apache-airflow-providers-dbt-cloud/connections.rst
+++ b/docs/apache-airflow-providers-dbt-cloud/connections.rst
@@ -95,5 +95,5 @@ For example, to add a connection with the connection ID of "dbt_cloud_default":
export AIRFLOW_CONN_DBT_CLOUD_DEFAULT='dbt-cloud://:api_token@:/my-tenant'
You can refer to the documentation on
-:ref:`creating connections via environment variables <environment_variables_secrets_backend>` for more
+:ref:`creating connections via environment variables <environment_variables_connections>` for more
information.
diff --git a/docs/apache-airflow/cli-and-env-variables-ref.rst b/docs/apache-airflow/cli-and-env-variables-ref.rst
index 7593233585..e2a929f1f4 100644
--- a/docs/apache-airflow/cli-and-env-variables-ref.rst
+++ b/docs/apache-airflow/cli-and-env-variables-ref.rst
@@ -90,7 +90,7 @@ Environment Variables
For example, if you want to create a connection named ``PROXY_POSTGRES_TCP``, you can create
a key ``AIRFLOW_CONN_PROXY_POSTGRES_TCP`` with the connection URI as the value.
- For more information, see: :ref:`environment_variables_secrets_backend`.
+ For more information, see: :ref:`environment_variables_connections`.
.. envvar:: AIRFLOW_HOME
diff --git a/docs/apache-airflow/howto/connection.rst b/docs/apache-airflow/howto/connection.rst
index d1ca89e09a..ee9cb19eda 100644
--- a/docs/apache-airflow/howto/connection.rst
+++ b/docs/apache-airflow/howto/connection.rst
@@ -26,11 +26,13 @@ Airflow's :class:`~airflow.models.connection.Connection` object is used for stor
Connections may be defined in the following ways:
- - in :ref:`environment variables <environment_variables_secrets_backend>`
+ - in :ref:`environment variables <environment_variables_connections>`
- in an external :doc:`/administration-and-deployment/security/secrets/secrets-backend/index`
- in the :ref:`Airflow metadata database <connections-in-database>`
(using the :ref:`CLI <connection/cli>` or :ref:`web UI <creating_connection_ui>`)
+.. _environment_variables_connections:
+
Storing connections in environment variables
--------------------------------------------
@@ -92,7 +94,7 @@ Storing connections in the database
-----------------------------------
.. seealso::
- Connections can alternatively be stored in :ref:`environment variables <environment_variables_secrets_backend>` or an :doc:`external secrets backend </administration-and-deployment/security/secrets/secrets-backend/index>` such as HashiCorp Vault, AWS SSM Parameter Store, etc.
+ Connections can alternatively be stored in :ref:`environment variables <environment_variables_connections>` or an :doc:`external secrets backend </administration-and-deployment/security/secrets/secrets-backend/index>` such as HashiCorp Vault, AWS SSM Parameter Store, etc.
When storing connections in the database, you may manage them using either the web UI or the Airflow CLI.
@@ -179,7 +181,6 @@ Exporting connections to file
You can export to file connections stored in the database (e.g. for migrating connections from one environment to another). See :ref:`Exporting Connections <cli-export-connections>` for usage.
-.. _environment_variables_secrets_backend:
Security of connections in the database
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[airflow] 11/23: Description of dag_processing.last_duration (#29740)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3cba77fd15291d6cc2f58d4f362e94f9fddbe573
Author: Nikos Chasiotis <ha...@gmail.com>
AuthorDate: Sat Feb 25 18:38:35 2023 +0200
Description of dag_processing.last_duration (#29740)
Time metric reports dag_processing.last_duration.<dag_file> reposts seconds and not milliseconds acording to https://github.com/apache/airflow/blob/2.4.3/airflow/dag_processing/manager.py#L874
(cherry picked from commit 4e3b5ae7248c2327864f64b25dc7a5bd7705430c)
---
.../administration-and-deployment/logging-monitoring/metrics.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst b/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
index 00c3537ccf..b20c544ab3 100644
--- a/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
+++ b/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
@@ -157,7 +157,7 @@ Name Description
=================================================== ========================================================================
``dagrun.dependency-check.<dag_id>`` Milliseconds taken to check DAG dependencies
``dag.<dag_id>.<task_id>.duration`` Milliseconds taken to finish a task
-``dag_processing.last_duration.<dag_file>`` Milliseconds taken to load the given DAG file
+``dag_processing.last_duration.<dag_file>`` Seconds taken to load the given DAG file
``dagrun.duration.success.<dag_id>`` Seconds taken for a DagRun to reach success state
``dagrun.duration.failed.<dag_id>`` Milliseconds taken for a DagRun to reach failed state
``dagrun.schedule_delay.<dag_id>`` Seconds of delay between the scheduled DagRun
[airflow] 19/23: Correct link to best_practice(s) section (#29876)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1ed564fbee9a11a56b007151c8ad09a0d12c0932
Author: Mayeul Kauffmann <ma...@users.noreply.github.com>
AuthorDate: Thu Mar 2 18:37:13 2023 +0100
Correct link to best_practice(s) section (#29876)
Logical link <best_practices/airflow_variables> fails.
In similar file faq.rst, logical link `DAG writing best practices<best_practice:writing_a_dag>` (without "s" at practice) works.
(cherry picked from commit 43d48e17dc18d81d620c9e90758078f152762d74)
---
docs/apache-airflow/howto/dynamic-dag-generation.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow/howto/dynamic-dag-generation.rst b/docs/apache-airflow/howto/dynamic-dag-generation.rst
index f180183631..5d542a2932 100644
--- a/docs/apache-airflow/howto/dynamic-dag-generation.rst
+++ b/docs/apache-airflow/howto/dynamic-dag-generation.rst
@@ -40,7 +40,7 @@ If you want to use variables to configure your code, you should always use
`environment variables <https://wiki.archlinux.org/title/environment_variables>`_ in your
top-level code rather than :doc:`Airflow Variables </core-concepts/variables>`. Using Airflow Variables
in top-level code creates a connection to the metadata DB of Airflow to fetch the value, which can slow
-down parsing and place extra load on the DB. See the `best practices on Airflow Variables <best_practices/airflow_variables>`_
+down parsing and place extra load on the DB. See the `best practices on Airflow Variables <best_practice:airflow_variables>`_
to make the best use of Airflow Variables in your DAGs using Jinja templates.
For example you could set ``DEPLOYMENT`` variable differently for your production and development
[airflow] 15/23: Removed continue for not in (#29791)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit cc334d563bbbb99c288958618613c8d598d9e55b
Author: Jose Sanchez <34...@users.noreply.github.com>
AuthorDate: Tue Feb 28 06:15:11 2023 +0100
Removed continue for not in (#29791)
(cherry picked from commit 60d98a1bc2d54787fcaad5edac36ecfa484fb42b)
---
airflow/www/views.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 880d447699..7b535484de 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -2843,10 +2843,10 @@ class Airflow(AirflowBaseView):
form = GraphForm(data=dt_nr_dr_data)
form.execution_date.choices = dt_nr_dr_data["dr_choices"]
- task_instances = {
- ti.task_id: wwwutils.get_instance_with_map(ti, session)
- for ti in dag.get_task_instances(dttm, dttm)
- }
+ task_instances = {}
+ for ti in dag.get_task_instances(dttm, dttm):
+ if ti.task_id not in task_instances:
+ task_instances[ti.task_id] = wwwutils.get_instance_with_map(ti, session)
tasks = {
t.task_id: {
"dag_id": t.dag_id,
[airflow] 09/23: Update docs re: template_fields typing and subclasses (#29725)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 47fe890435740084f8d9d0f96fe2f3431bc8a058
Author: Josh Fell <48...@users.noreply.github.com>
AuthorDate: Thu Feb 23 23:47:37 2023 -0500
Update docs re: template_fields typing and subclasses (#29725)
(cherry picked from commit 28573c94ff9054aee76aee42dd1eec314e849016)
---
docs/apache-airflow/core-concepts/operators.rst | 3 +-
docs/apache-airflow/howto/custom-operator.rst | 72 +++++++++++++++++++++----
2 files changed, 65 insertions(+), 10 deletions(-)
diff --git a/docs/apache-airflow/core-concepts/operators.rst b/docs/apache-airflow/core-concepts/operators.rst
index cb278402de..03da9893b7 100644
--- a/docs/apache-airflow/core-concepts/operators.rst
+++ b/docs/apache-airflow/core-concepts/operators.rst
@@ -109,7 +109,8 @@ You can also use Jinja templating with nested fields, as long as these nested fi
)
-.. note:: The ``template_fields`` property can equally be a class variable or an instance variable.
+.. note:: The ``template_fields`` property is a class variable and guaranteed to be of a ``Sequence[str]``
+ type (i.e. a list or tuple of strings).
Deep nested fields can also be substituted, as long as all intermediate fields are marked as template fields:
diff --git a/docs/apache-airflow/howto/custom-operator.rst b/docs/apache-airflow/howto/custom-operator.rst
index 33b1ffb357..bd80913a25 100644
--- a/docs/apache-airflow/howto/custom-operator.rst
+++ b/docs/apache-airflow/howto/custom-operator.rst
@@ -94,7 +94,7 @@ logic of an operation is in one place - in the operator.
.. _custom-operator/hook:
Hooks
-^^^^^
+-----
Hooks act as an interface to communicate with the external shared resources in a DAG.
For example, multiple tasks in a DAG can require access to a MySQL database. Instead of
creating a connection per task, you can retrieve a connection from the hook and utilize it.
@@ -131,7 +131,7 @@ The ``execute`` gets called only during a DAG run.
User interface
-^^^^^^^^^^^^^^^
+--------------
Airflow also allows the developer to control how the operator shows up in the DAG UI.
Override ``ui_color`` to change the background color of the operator in UI.
Override ``ui_fgcolor`` to change the color of the label.
@@ -146,7 +146,7 @@ Override ``custom_operator_name`` to change the displayed name to something othe
# ...
Templating
-^^^^^^^^^^^
+----------
You can use :ref:`Jinja templates <concepts:jinja-templating>` to parameterize your operator.
Airflow considers the field names present in ``template_fields`` for templating while rendering
the operator.
@@ -157,12 +157,13 @@ the operator.
template_fields: Sequence[str] = ("name",)
- def __init__(self, name: str, **kwargs) -> None:
+ def __init__(self, name: str, world: str, **kwargs) -> None:
super().__init__(**kwargs)
self.name = name
+ self.world = world
def execute(self, context):
- message = f"Hello from {self.name}"
+ message = f"Hello {self.world} it's {self.name}!"
print(message)
return message
@@ -171,7 +172,11 @@ You can use the template as follows:
.. code-block:: python
with dag:
- hello_task = HelloOperator(task_id="task_id_1", dag=dag, name="{{ task_instance.task_id }}")
+ hello_task = HelloOperator(
+ task_id="task_id_1",
+ name="{{ task_instance.task_id }}",
+ world="Earth",
+ )
In this example, Jinja looks for the ``name`` parameter and substitutes ``{{ task_instance.task_id }}`` with
``task_id_1``.
@@ -232,7 +237,6 @@ Then using this template as follows:
config_task = MyConfigOperator(
task_id="task_id_1",
configuration={"query": {"job_id": "123", "sql": "select * from my_table"}},
- dag=dag,
)
This will result in the UI rendering ``configuration`` as json in addition to the value contained in the
@@ -267,15 +271,65 @@ Currently available lexers:
If you use a non-existing lexer then the value of the template field will be rendered as a pretty-printed object.
+Add template fields with subclassing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+A common use case for creating a custom operator is for simply augmenting existing ``template_fields``.
+There might be a situation is which an operator you wish to use doesn't define certain parameters as
+templated, but you'd like to be able to dynamically pass an argument as a Jinja expression. This can easily be
+achieved with a quick subclassing of the existing operator.
+
+Let's assume you want to use the ``HelloOperator`` defined earlier:
+
+.. code-block:: python
+
+ class HelloOperator(BaseOperator):
+
+ template_fields: Sequence[str] = ("name",)
+
+ def __init__(self, name: str, world: str, **kwargs) -> None:
+ super().__init__(**kwargs)
+ self.name = name
+ self.world = world
+
+ def execute(self, context):
+ message = f"Hello {self.world} it's {self.name}!"
+ print(message)
+ return message
+
+However, you'd like to dynamically parameterize ``world`` arguments. Because the ``template_fields`` property
+is guaranteed to be a ``Sequence[str]`` type (i.e. a list or tuple of strings), you can subclass the
+``HelloOperator`` to modify the ``template_fields`` as desired easily.
+
+.. code-block:: python
+
+ class MyHelloOperator(HelloOperator):
+
+ template_fields: Sequence[str] = (*HelloOperator.template_fields, "world")
+
+Now you can use ``MyHelloOperator`` like this:
+
+.. code-block:: python
+
+ with dag:
+ hello_task = MyHelloOperator(
+ task_id="task_id_1",
+ name="{{ task_instance.task_id }}",
+ world="{{ var.value.my_world }}",
+ )
+
+In this example, the ``world`` argument will be dynamically set to the value of an Airflow Variable named
+"my_world" via a Jinja expression.
+
+
Define an operator extra link
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+------------------------------
For your operator, you can :doc:`Define an extra link <define-extra-link>` that can
redirect users to external systems. For example, you can add a link that redirects
the user to the operator's manual.
Sensors
-^^^^^^^^
+-------
Airflow provides a primitive for a special kind of operator, whose purpose is to
poll some state (e.g. presence of a file) on a regular interval until a
success criteria is met.
[airflow] 02/23: Specific use-case: adding packages via requirements.txt in compose (#29598)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 46f9cb90d61c07f56e2ad9f9e4b95c6194f96b92
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Feb 19 19:09:03 2023 +0100
Specific use-case: adding packages via requirements.txt in compose (#29598)
The users of docker compose do not realise that docker compose
has built-in mechanism for rebuilding the image automatically when
needed and do not realise that this does not make their workflows
more complex. Apparently just describing how you can use custom
images is either scary or not clear enough, as users keep on for
dynamic way of adding the packages (which makes no sense if you
use what docker-compose provides out-of-the-box) so it is
worthwile to specifically mention "requirements.txt" and
explicitly add all the steps needed to make requirements.txt and
custom Docker image part of your regular workflow.
(cherry picked from commit a21c17bc07c1eeb733eca889a02396fab401b215)
---
docs/apache-airflow/howto/docker-compose/index.rst | 34 ++++++++++++++++++++++
1 file changed, 34 insertions(+)
diff --git a/docs/apache-airflow/howto/docker-compose/index.rst b/docs/apache-airflow/howto/docker-compose/index.rst
index 6e75ef8416..e4d169c0ea 100644
--- a/docs/apache-airflow/howto/docker-compose/index.rst
+++ b/docs/apache-airflow/howto/docker-compose/index.rst
@@ -271,6 +271,40 @@ to rebuild the images on-the-fly when you run other ``docker compose`` commands.
Examples of how you can extend the image with custom providers, python packages,
apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.
+Special case - adding dependencies via requirements.txt file
+============================================================
+
+Usual case for custom images, is when you want to add a set of requirements to it - usually stored in
+``requirements.txt`` file. For development, you might be tempted to add it dynamically when you are
+starting the original airflow image, but this has a number of side effects (for example your containers
+will start much slower - each additional dependency will further delay your containers start up time).
+Also it is completely unnecessary, because docker compose has the development workflow built-in.
+You can - following the previous chapter, automatically build and use your custom image when you
+iterate with docker compose locally. Specifically when you want to add your own requirement file,
+you should do those steps:
+
+1) Comment out the ``image: ...`` line and remove comment from the ``build: .`` line in the
+ ``docker-compose.yaml`` file. The relevant part of the docker-compose file of yours should look similar
+ to (use correct image tag):
+
+```
+#image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.5.1}
+build: .
+```
+
+2) Create ``Dockerfile`` in the same folder your ``docker-compose.yaml`` file is with content similar to:
+
+```
+FROM apache/airflow:2.5.1
+ADD requirements.txt .
+RUN pip install -r requirements.txt
+```
+
+3) Place ``requirements.txt`` file in the same directory.
+
+Run ``docker compose build`` to build the image, or add ``--build`` flag to ``docker compose up`` or
+``docker compose run`` commands to build the image automatically as needed.
+
Networking
==========
[airflow] 08/23: Correct version label for data interval variables in documentation (#29704)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 940464b991e1bf0ea2ac7ac6d5757ffd90594228
Author: Eric Pickup <ep...@brex.com>
AuthorDate: Sat Feb 25 06:19:39 2023 -0500
Correct version label for data interval variables in documentation (#29704)
* Correct date for data interval variables
* Update prev interval variable docs
(cherry picked from commit 717426ec40bbdd31c564c1d4bdc6ba6e0789d1a2)
---
docs/apache-airflow/templates-ref.rst | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/docs/apache-airflow/templates-ref.rst b/docs/apache-airflow/templates-ref.rst
index 68b38d9687..c84703a486 100644
--- a/docs/apache-airflow/templates-ref.rst
+++ b/docs/apache-airflow/templates-ref.rst
@@ -36,8 +36,8 @@ in all templates
=========================================== ===================== ===================================================================
Variable Type Description
=========================================== ===================== ===================================================================
-``{{ data_interval_start }}`` `pendulum.DateTime`_ Start of the data interval. Added in version 2.3.
-``{{ data_interval_end }}`` `pendulum.DateTime`_ End of the data interval. Added in version 2.3.
+``{{ data_interval_start }}`` `pendulum.DateTime`_ Start of the data interval. Added in version 2.2.
+``{{ data_interval_end }}`` `pendulum.DateTime`_ End of the data interval. Added in version 2.2.
``{{ ds }}`` str | The DAG run's logical date as ``YYYY-MM-DD``.
| Same as ``{{ dag_run.logical_date | ds }}``.
``{{ ds_nodash }}`` str Same as ``{{ dag_run.logical_date | ds_nodash }}``.
@@ -48,9 +48,9 @@ Variable Type Description
``{{ ts_nodash }}`` str | Same as ``{{ dag_run.logical_date | ts_nodash }}``.
| Example: ``20180101T000000``.
``{{ prev_data_interval_start_success }}`` `pendulum.DateTime`_ | Start of the data interval of the prior successful DAG run.
- | ``None`` | Added in version 2.3.
+ | ``None`` | Added in version 2.2.
``{{ prev_data_interval_end_success }}`` `pendulum.DateTime`_ | End of the data interval of the prior successful DAG run.
- | ``None`` | Added in version 2.3.
+ | ``None`` | Added in version 2.2.
``{{ prev_start_date_success }}`` `pendulum.DateTime`_ Start date from prior successful DAG run (if available).
| ``None``
``{{ dag }}`` DAG The currently running DAG.
[airflow] 21/23: Correct argument name of Workday timetable in timetable.rst (#29896)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c962688473044c6643d9391895ba5a7cd97dbb7e
Author: Christophoros Antoniades <36...@users.noreply.github.com>
AuthorDate: Fri Mar 3 20:24:58 2023 +0100
Correct argument name of Workday timetable in timetable.rst (#29896)
(cherry picked from commit c02c3010d83b3cc1cc3007ad56f51af6538c8424)
---
docs/apache-airflow/howto/timetable.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow/howto/timetable.rst b/docs/apache-airflow/howto/timetable.rst
index 0cbf817c78..fac296377f 100644
--- a/docs/apache-airflow/howto/timetable.rst
+++ b/docs/apache-airflow/howto/timetable.rst
@@ -123,7 +123,7 @@ Next is the implementation of ``next_dagrun_info``:
:start-after: [START howto_timetable_next_dagrun_info]
:end-before: [END howto_timetable_next_dagrun_info]
-This method accepts two arguments. ``last_automated_dagrun`` is a
+This method accepts two arguments. ``last_automated_data_interval`` is a
:class:`~airflow.timetables.base.DataInterval` instance indicating the data
interval of this DAG's previous non-manually-triggered run, or ``None`` if this
is the first time ever the DAG is being scheduled. ``restriction`` encapsulates
[airflow] 20/23: replace tree view with grid view in docs (#29879)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 995dffc584b7df42e5c7e5b588428bdf5bee5538
Author: eladkal <45...@users.noreply.github.com>
AuthorDate: Thu Mar 2 18:41:54 2023 +0200
replace tree view with grid view in docs (#29879)
(cherry picked from commit a71f908a0eeab0c0671c39bef394ee07e06039d9)
---
docs/apache-airflow/best-practices.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/apache-airflow/best-practices.rst b/docs/apache-airflow/best-practices.rst
index 65a3d29fa0..b3004ea5b4 100644
--- a/docs/apache-airflow/best-practices.rst
+++ b/docs/apache-airflow/best-practices.rst
@@ -75,7 +75,7 @@ result -
Deleting a task
----------------
-Be careful when deleting a task from a DAG. You would not be able to see the Task in Graph View, Tree View, etc making
+Be careful when deleting a task from a DAG. You would not be able to see the Task in Graph View, Grid View, etc making
it difficult to check the logs of that Task from the Webserver. If that is not desired, please create a new DAG.
[airflow] 05/23: Fix Scheduler crash when clear a previous run of a normal task that is now a mapped task (#29645)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3aa0c008fcbd445f9b3f277f3bf79f00da01425e
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Mon Feb 20 20:45:25 2023 +0100
Fix Scheduler crash when clear a previous run of a normal task that is now a mapped task (#29645)
The fix was to clear the db references of the taskinstances in XCom, RenderedTaskInstanceFields
and TaskFail. That way, we are able to run the mapped tasks
(cherry picked from commit a770edfac493f3972c10a43e45bcd0e7cfaea65f)
---
airflow/models/dagrun.py | 8 +++++
airflow/models/taskinstance.py | 19 +++++++++++
tests/models/test_dagrun.py | 67 +++++++++++++++++++++++++++++----------
tests/models/test_taskinstance.py | 16 ++++++++++
4 files changed, 93 insertions(+), 17 deletions(-)
diff --git a/airflow/models/dagrun.py b/airflow/models/dagrun.py
index e06d911c27..6f1505d489 100644
--- a/airflow/models/dagrun.py
+++ b/airflow/models/dagrun.py
@@ -778,6 +778,14 @@ class DagRun(Base, LoggingMixin):
"""
if ti.map_index >= 0: # Already expanded, we're good.
return None
+
+ from airflow.models.mappedoperator import MappedOperator
+
+ if isinstance(ti.task, MappedOperator):
+ # If we get here, it could be that we are moving from non-mapped to mapped
+ # after task instance clearing or this ti is not yet expanded. Safe to clear
+ # the db references.
+ ti.clear_db_references(session=session)
try:
expanded_tis, _ = ti.task.expand_mapped_task(self.run_id, session=session)
except NotMapped: # Not a mapped task, nothing needed.
diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
index 615a98f6cb..1eb778df55 100644
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -2658,6 +2658,25 @@ class TaskInstance(Base, LoggingMixin):
map_index_start = ancestor_map_index * further_count
return range(map_index_start, map_index_start + further_count)
+ def clear_db_references(self, session):
+ """
+ Clear DB references to XCom, TaskFail and RenderedTaskInstanceFields.
+
+ :param session: ORM Session
+
+ :meta private:
+ """
+ from airflow.models.renderedtifields import RenderedTaskInstanceFields
+
+ tables = [TaskFail, XCom, RenderedTaskInstanceFields]
+ for table in tables:
+ session.query(table).filter(
+ table.dag_id == self.dag_id,
+ table.task_id == self.task_id,
+ table.run_id == self.run_id,
+ table.map_index == self.map_index,
+ ).delete()
+
def _find_common_ancestor_mapped_group(node1: Operator, node2: Operator) -> MappedTaskGroup | None:
"""Given two operators, find their innermost common mapped task group."""
diff --git a/tests/models/test_dagrun.py b/tests/models/test_dagrun.py
index 3d368ae962..34295473c9 100644
--- a/tests/models/test_dagrun.py
+++ b/tests/models/test_dagrun.py
@@ -49,13 +49,7 @@ from airflow.utils.state import DagRunState, State, TaskInstanceState
from airflow.utils.trigger_rule import TriggerRule
from airflow.utils.types import DagRunType
from tests.models import DEFAULT_DATE as _DEFAULT_DATE
-from tests.test_utils.db import (
- clear_db_dags,
- clear_db_datasets,
- clear_db_pools,
- clear_db_runs,
- clear_db_variables,
-)
+from tests.test_utils import db
from tests.test_utils.mock_operators import MockOperator
DEFAULT_DATE = pendulum.instance(_DEFAULT_DATE)
@@ -64,19 +58,21 @@ DEFAULT_DATE = pendulum.instance(_DEFAULT_DATE)
class TestDagRun:
dagbag = DagBag(include_examples=True)
+ @staticmethod
+ def clean_db():
+ db.clear_db_runs()
+ db.clear_db_pools()
+ db.clear_db_dags()
+ db.clear_db_variables()
+ db.clear_db_datasets()
+ db.clear_db_xcom()
+ db.clear_db_task_fail()
+
def setup_class(self) -> None:
- clear_db_runs()
- clear_db_pools()
- clear_db_dags()
- clear_db_variables()
- clear_db_datasets()
+ self.clean_db()
def teardown_method(self) -> None:
- clear_db_runs()
- clear_db_pools()
- clear_db_dags()
- clear_db_variables()
- clear_db_datasets()
+ self.clean_db()
def create_dag_run(
self,
@@ -2222,3 +2218,40 @@ def test_mapped_task_depends_on_past(dag_maker, session):
assert len(decision.unfinished_tis) == 0
decision = dr2.task_instance_scheduling_decisions(session=session)
assert len(decision.unfinished_tis) == 0
+
+
+def test_clearing_task_and_moving_from_non_mapped_to_mapped(dag_maker, session):
+ """
+ Test that clearing a task and moving from non-mapped to mapped clears existing
+ references in XCom, TaskFail, and RenderedTaskInstanceFields
+ To be able to test this, RenderedTaskInstanceFields was not used in the test
+ since it would require that the task is expanded first.
+ """
+
+ from airflow.models.taskfail import TaskFail
+ from airflow.models.xcom import XCom
+
+ @task
+ def printx(x):
+ print(x)
+
+ with dag_maker() as dag:
+ printx.expand(x=[1])
+
+ dr1: DagRun = dag_maker.create_dagrun(run_type=DagRunType.SCHEDULED)
+ ti = dr1.get_task_instances()[0]
+ # mimicking a case where task moved from non-mapped to mapped
+ # in that case, it would have map_index of -1 even though mapped
+ ti.map_index = -1
+ session.merge(ti)
+ session.flush()
+ # Purposely omitted RenderedTaskInstanceFields because the ti need
+ # to be expanded but here we are mimicking and made it map_index -1
+ session.add(TaskFail(ti))
+ XCom.set(key="test", value="value", task_id=ti.task_id, dag_id=dag.dag_id, run_id=ti.run_id)
+ session.commit()
+ for table in [TaskFail, XCom]:
+ assert session.query(table).count() == 1
+ dr1.task_instance_scheduling_decisions(session)
+ for table in [TaskFail, XCom]:
+ assert session.query(table).count() == 0
diff --git a/tests/models/test_taskinstance.py b/tests/models/test_taskinstance.py
index 849358cc69..bb2b676a59 100644
--- a/tests/models/test_taskinstance.py
+++ b/tests/models/test_taskinstance.py
@@ -129,6 +129,7 @@ class TestTaskInstance:
db.clear_rendered_ti_fields()
db.clear_db_task_reschedule()
db.clear_db_datasets()
+ db.clear_db_xcom()
def setup_method(self):
self.clean_db()
@@ -2846,6 +2847,21 @@ class TestTaskInstance:
assert ser_ti.operator == "EmptyOperator"
assert ser_ti.task.operator_name == "EmptyOperator"
+ def test_clear_db_references(self, session, create_task_instance):
+ tables = [TaskFail, RenderedTaskInstanceFields, XCom]
+ ti = create_task_instance()
+ session.merge(ti)
+ session.commit()
+ for table in [TaskFail, RenderedTaskInstanceFields]:
+ session.add(table(ti))
+ XCom.set(key="key", value="value", task_id=ti.task_id, dag_id=ti.dag_id, run_id=ti.run_id)
+ session.commit()
+ for table in tables:
+ assert session.query(table).count() == 1
+ ti.clear_db_references(session)
+ for table in tables:
+ assert session.query(table).count() == 0
+
@pytest.mark.parametrize("pool_override", [None, "test_pool2"])
def test_refresh_from_task(pool_override):
[airflow] 17/23: Add a check for not templateable fields (#29821)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5add348e59c5ada75a3d73976090d4a6b6d43e6e
Author: Hussein Awala <ho...@gmail.com>
AuthorDate: Sat Mar 4 19:19:09 2023 +0100
Add a check for not templateable fields (#29821)
(cherry picked from commit 7963360b8d43a15791a6b7d4335f482fce1d82d2)
---
airflow/serialization/serialized_objects.py | 6 ++++++
tests/serialization/test_dag_serialization.py | 18 +++++++++++++++++-
2 files changed, 23 insertions(+), 1 deletion(-)
diff --git a/airflow/serialization/serialized_objects.py b/airflow/serialization/serialized_objects.py
index 6cb33cd203..e3bc785701 100644
--- a/airflow/serialization/serialized_objects.py
+++ b/airflow/serialization/serialized_objects.py
@@ -20,6 +20,7 @@ from __future__ import annotations
import collections.abc
import datetime
import enum
+import inspect
import logging
import warnings
import weakref
@@ -701,6 +702,7 @@ class SerializedBaseOperator(BaseOperator, BaseSerialization):
def task_type(self) -> str:
# Overwrites task_type of BaseOperator to use _task_type instead of
# __class__.__name__.
+
return self._task_type
@task_type.setter
@@ -769,8 +771,12 @@ class SerializedBaseOperator(BaseOperator, BaseSerialization):
# Store all template_fields as they are if there are JSON Serializable
# If not, store them as strings
+ # And raise an exception if the field is not templateable
+ forbidden_fields = set(inspect.signature(BaseOperator.__init__).parameters.keys())
if op.template_fields:
for template_field in op.template_fields:
+ if template_field in forbidden_fields:
+ raise AirflowException(f"Cannot template BaseOperator fields: {template_field}")
value = getattr(op, template_field, None)
if not cls._is_excluded(value, template_field, op):
serialize_op[template_field] = serialize_template_field(value)
diff --git a/tests/serialization/test_dag_serialization.py b/tests/serialization/test_dag_serialization.py
index ec07d60954..305efd2ea0 100644
--- a/tests/serialization/test_dag_serialization.py
+++ b/tests/serialization/test_dag_serialization.py
@@ -38,7 +38,7 @@ from kubernetes.client import models as k8s
import airflow
from airflow.datasets import Dataset
-from airflow.exceptions import SerializationError
+from airflow.exceptions import AirflowException, SerializationError
from airflow.hooks.base import BaseHook
from airflow.kubernetes.pod_generator import PodGenerator
from airflow.models import DAG, Connection, DagBag, Operator
@@ -1867,6 +1867,22 @@ class TestStringifiedDAGs:
assert param.description == "hello"
assert param.schema == {"type": "string"}
+ def test_not_templateable_fields_in_serialized_dag(
+ self,
+ ):
+ """
+ Test that when we use not templateable fields, an Airflow exception is raised.
+ """
+
+ class TestOperator(BaseOperator):
+ template_fields = ("execution_timeout",)
+
+ dag = DAG("test_not_templateable_fields", start_date=datetime(2019, 8, 1))
+ with dag:
+ TestOperator(task_id="test", execution_timeout=timedelta(seconds=10))
+ with pytest.raises(AirflowException, match="Cannot template BaseOperator fields: execution_timeout"):
+ SerializedDAG.to_dict(dag)
+
def test_kubernetes_optional():
"""Serialisation / deserialisation continues to work without kubernetes installed"""
[airflow] 18/23: POST /dagRuns API should 404 if dag not active (#29860)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 29b7bf204e41f89c8fcb51ce184a5dd01f93aa96
Author: Sam Wheating <sa...@gmail.com>
AuthorDate: Fri Mar 3 06:40:07 2023 -0800
POST /dagRuns API should 404 if dag not active (#29860)
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
(cherry picked from commit 751a995df55419068f11ebabe483dba3302916ed)
---
airflow/api_connexion/endpoints/dag_run_endpoint.py | 2 +-
tests/api_connexion/endpoints/test_dag_run_endpoint.py | 15 ++++++++++++++-
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/airflow/api_connexion/endpoints/dag_run_endpoint.py b/airflow/api_connexion/endpoints/dag_run_endpoint.py
index ea1c0af88c..b2a07f87b1 100644
--- a/airflow/api_connexion/endpoints/dag_run_endpoint.py
+++ b/airflow/api_connexion/endpoints/dag_run_endpoint.py
@@ -293,7 +293,7 @@ def get_dag_runs_batch(*, session: Session = NEW_SESSION) -> APIResponse:
)
def post_dag_run(*, dag_id: str, session: Session = NEW_SESSION) -> APIResponse:
"""Trigger a DAG."""
- dm = session.query(DagModel).filter(DagModel.dag_id == dag_id).first()
+ dm = session.query(DagModel).filter(DagModel.is_active, DagModel.dag_id == dag_id).first()
if not dm:
raise NotFound(title="DAG not found", detail=f"DAG with dag_id: '{dag_id}' not found")
if dm.has_import_errors:
diff --git a/tests/api_connexion/endpoints/test_dag_run_endpoint.py b/tests/api_connexion/endpoints/test_dag_run_endpoint.py
index 1ea4b8d1fe..93c9c6a1d0 100644
--- a/tests/api_connexion/endpoints/test_dag_run_endpoint.py
+++ b/tests/api_connexion/endpoints/test_dag_run_endpoint.py
@@ -120,6 +120,7 @@ class TestDagRunEndpoint:
def _create_dag(self, dag_id):
dag_instance = DagModel(dag_id=dag_id)
+ dag_instance.is_active = True
with create_session() as session:
session.add(dag_instance)
dag = DAG(dag_id=dag_id, schedule=None)
@@ -132,7 +133,7 @@ class TestDagRunEndpoint:
for i in range(idx_start, idx_start + 2):
if i == 1:
- dags.append(DagModel(dag_id="TEST_DAG_ID"))
+ dags.append(DagModel(dag_id="TEST_DAG_ID", is_active=True))
dagrun_model = DagRun(
dag_id="TEST_DAG_ID",
run_id="TEST_DAG_RUN_ID_" + str(i),
@@ -1073,6 +1074,18 @@ class TestPostDagRun(TestDagRunEndpoint):
}
_check_last_log(session, dag_id="TEST_DAG_ID", event="dag_run.create", execution_date=None)
+ def test_should_respond_404_if_a_dag_is_inactive(self, session):
+ dm = self._create_dag("TEST_INACTIVE_DAG_ID")
+ dm.is_active = False
+ session.add(dm)
+ session.flush()
+ response = self.client.post(
+ "api/v1/dags/TEST_INACTIVE_DAG_ID/dagRuns",
+ json={},
+ environ_overrides={"REMOTE_USER": "test"},
+ )
+ assert response.status_code == 404
+
def test_should_respond_400_if_a_dag_has_import_errors(self, session):
"""Test that if a dagmodel has import errors, dags won't be triggered"""
dm = self._create_dag("TEST_DAG_ID")
[airflow] 04/23: convert moment with timezone to UTC instead of raising an exception (#29606)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e716b7509c789549ee43f060ebea00676dd67b07
Author: Hussein Awala <ho...@gmail.com>
AuthorDate: Sun Feb 19 21:27:44 2023 +0100
convert moment with timezone to UTC instead of raising an exception (#29606)
* convert moments with timezone to utc instead of raising an exception
* test DateTimeTrigger with different timezones
(cherry picked from commit 79c07e3fc5d580aea271ff3f0887291ae9e4473f)
---
airflow/triggers/temporal.py | 4 +---
tests/triggers/test_temporal.py | 14 +++++++++++---
2 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/airflow/triggers/temporal.py b/airflow/triggers/temporal.py
index 3967940a7e..6293be84d8 100644
--- a/airflow/triggers/temporal.py
+++ b/airflow/triggers/temporal.py
@@ -41,10 +41,8 @@ class DateTimeTrigger(BaseTrigger):
# Make sure it's in UTC
elif moment.tzinfo is None:
raise ValueError("You cannot pass naive datetimes")
- elif not hasattr(moment.tzinfo, "offset") or moment.tzinfo.offset != 0: # type: ignore
- raise ValueError(f"The passed datetime must be using Pendulum's UTC, not {moment.tzinfo!r}")
else:
- self.moment = moment
+ self.moment = timezone.convert_to_utc(moment)
def serialize(self) -> tuple[str, dict[str, Any]]:
return ("airflow.triggers.temporal.DateTimeTrigger", {"moment": self.moment})
diff --git a/tests/triggers/test_temporal.py b/tests/triggers/test_temporal.py
index 21a419065d..655910394f 100644
--- a/tests/triggers/test_temporal.py
+++ b/tests/triggers/test_temporal.py
@@ -61,14 +61,22 @@ def test_timedelta_trigger_serialization():
assert -2 < (kwargs["moment"] - expected_moment).total_seconds() < 2
+@pytest.mark.parametrize(
+ "tz",
+ [
+ pendulum.tz.timezone("UTC"),
+ pendulum.tz.timezone("Europe/Paris"),
+ pendulum.tz.timezone("America/Toronto"),
+ ],
+)
@pytest.mark.asyncio
-async def test_datetime_trigger_timing():
+async def test_datetime_trigger_timing(tz):
"""
Tests that the DateTimeTrigger only goes off on or after the appropriate
time.
"""
- past_moment = timezone.utcnow() - datetime.timedelta(seconds=60)
- future_moment = timezone.utcnow() + datetime.timedelta(seconds=60)
+ past_moment = pendulum.instance((timezone.utcnow() - datetime.timedelta(seconds=60)).astimezone(tz))
+ future_moment = pendulum.instance((timezone.utcnow() + datetime.timedelta(seconds=60)).astimezone(tz))
# Create a task that runs the trigger for a short time then cancels it
trigger = DateTimeTrigger(future_moment)
[airflow] 03/23: fix clear dag run openapi spec responses by adding additional return type (#29600)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 195d47aa58fd0916f4b26aeeb1358b88ff543022
Author: Alexander Liotta <al...@astronomer.io>
AuthorDate: Mon Feb 20 15:53:17 2023 -0500
fix clear dag run openapi spec responses by adding additional return type (#29600)
(cherry picked from commit 54c4d590caa9399f9fb331811531e0ea8d56aa41)
---
airflow/api_connexion/openapi/v1.yaml | 4 +++-
airflow/www/static/js/types/api-generated.ts | 3 ++-
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/airflow/api_connexion/openapi/v1.yaml b/airflow/api_connexion/openapi/v1.yaml
index 0ee095fe4f..a32ac04986 100644
--- a/airflow/api_connexion/openapi/v1.yaml
+++ b/airflow/api_connexion/openapi/v1.yaml
@@ -887,7 +887,9 @@ paths:
content:
application/json:
schema:
- $ref: '#/components/schemas/DAGRun'
+ anyOf:
+ - $ref: '#/components/schemas/DAGRun'
+ - $ref: '#/components/schemas/TaskInstanceCollection'
'400':
$ref: '#/components/responses/BadRequest'
'401':
diff --git a/airflow/www/static/js/types/api-generated.ts b/airflow/www/static/js/types/api-generated.ts
index 52bb98e769..72fc939a38 100644
--- a/airflow/www/static/js/types/api-generated.ts
+++ b/airflow/www/static/js/types/api-generated.ts
@@ -2942,7 +2942,8 @@ export interface operations {
/** Success. */
200: {
content: {
- "application/json": components["schemas"]["DAGRun"];
+ "application/json": Partial<components["schemas"]["DAGRun"]> &
+ Partial<components["schemas"]["TaskInstanceCollection"]>;
};
};
400: components["responses"]["BadRequest"];
[airflow] 23/23: Update min version of python-deamon to fix containerd file limits (#29916)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c0cd3397f74bf0fd2ad233d5efbab51df4774ce0
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Mar 4 16:27:20 2023 +0100
Update min version of python-deamon to fix containerd file limits (#29916)
Recent change in the new containerd causes memory exhaution as
huge amount of memory were used by python-daemon when starting,
thus running Airflow in Docker for multiple OS-es using the new
containerd was impossible without implementing some workarounds.
Python daemon fix has been released in 3.0.0 version in response to
https://pagure.io/python-daemon/issue/72 and we should add min
version for the package to make sure the new version is used.
Fixes: #29841
(cherry picked from commit c8cc49af2d011f048ebea8a6559ddd5fca00f378)
---
setup.cfg | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/setup.cfg b/setup.cfg
index 6ae6b503c2..34fdf014df 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -119,7 +119,7 @@ install_requires =
psutil>=4.2.0
pygments>=2.0.1
pyjwt>=2.0.0
- python-daemon>=2.2.4
+ python-daemon>=3.0.0
python-dateutil>=2.3
python-nvd3>=0.15.0
python-slugify>=5.0
[airflow] 13/23: Fix compile www assets dev (#29769)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1d997fd381d4ad309baf002535fedb4660740830
Author: Pierre Jeambrun <pi...@gmail.com>
AuthorDate: Sun Feb 26 11:25:40 2023 +0100
Fix compile www assets dev (#29769)
* Fix compile www assets dev
* Add parents True
(cherry picked from commit a0e13370053452e992d45e7956ff33290563b3a0)
---
scripts/ci/pre_commit/pre_commit_compile_www_assets_dev.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/scripts/ci/pre_commit/pre_commit_compile_www_assets_dev.py b/scripts/ci/pre_commit/pre_commit_compile_www_assets_dev.py
index fb17c88f10..1b90c42ba6 100755
--- a/scripts/ci/pre_commit/pre_commit_compile_www_assets_dev.py
+++ b/scripts/ci/pre_commit/pre_commit_compile_www_assets_dev.py
@@ -35,6 +35,7 @@ WWW_ASSET_OUT_DEV_MODE_FILE = WWW_CACHE_DIR / "asset_compile_dev_mode.out"
if __name__ == "__main__":
www_directory = AIRFLOW_SOURCES_PATH / "airflow" / "www"
+ WWW_CACHE_DIR.mkdir(parents=True, exist_ok=True)
if WWW_HASH_FILE.exists():
# cleanup hash of www so that next compile-assets recompiles them
WWW_HASH_FILE.unlink()
[airflow] 14/23: better description for limit in api (#29773)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 9f8895f2d4efd7e984929fc2f18c4657ec1bc59f
Author: Bowrna <ma...@gmail.com>
AuthorDate: Sun Feb 26 21:49:51 2023 +0530
better description for limit in api (#29773)
(cherry picked from commit 228d79c1b3e11ecfbff5a27c900f9d49a84ad365)
---
airflow/config_templates/config.yml | 11 ++++++-----
airflow/config_templates/default_airflow.cfg | 11 ++++++-----
2 files changed, 12 insertions(+), 10 deletions(-)
diff --git a/airflow/config_templates/config.yml b/airflow/config_templates/config.yml
index 699142b708..dd825209b8 100644
--- a/airflow/config_templates/config.yml
+++ b/airflow/config_templates/config.yml
@@ -911,17 +911,18 @@
default: "airflow.api.auth.backend.session"
- name: maximum_page_limit
description: |
- Used to set the maximum page limit for API requests
+ Used to set the maximum page limit for API requests. If limit passed as param
+ is greater than maximum page limit, it will be ignored and maximum page limit value
+ will be set as the limit
version_added: 2.0.0
type: integer
example: ~
default: "100"
- name: fallback_page_limit
description: |
- Used to set the default page limit when limit is zero. A default limit
- of 100 is set on OpenApi spec. However, this particular default limit
- only work when limit is set equal to zero(0) from API requests.
- If no limit is supplied, the OpenApi spec default is used.
+ Used to set the default page limit when limit param is zero or not provided in API
+ requests. Otherwise if positive integer is passed in the API requests as limit, the
+ smallest number of user given limit or maximum page limit is taken as limit.
type: integer
example: ~
version_added: 2.0.0
diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg
index 2103d6be6d..cc5e82d3f5 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -492,13 +492,14 @@ enable_experimental_api = False
# ("airflow.api.auth.backend.default" allows all requests for historic reasons)
auth_backends = airflow.api.auth.backend.session
-# Used to set the maximum page limit for API requests
+# Used to set the maximum page limit for API requests. If limit passed as param
+# is greater than maximum page limit, it will be ignored and maximum page limit value
+# will be set as the limit
maximum_page_limit = 100
-# Used to set the default page limit when limit is zero. A default limit
-# of 100 is set on OpenApi spec. However, this particular default limit
-# only work when limit is set equal to zero(0) from API requests.
-# If no limit is supplied, the OpenApi spec default is used.
+# Used to set the default page limit when limit param is zero or not provided in API
+# requests. Otherwise if positive integer is passed in the API requests as limit, the
+# smallest number of user given limit or maximum page limit is taken as limit.
fallback_page_limit = 100
# The intended audience for JWT token credentials used for authorization. This value must match on the client and server sides. If empty, audience will not be tested.
[airflow] 06/23: FIx formatting of Dataset inlet/outlet note in TaskFlow concepts (#29678)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 18bda7a70f183c2e28b2781abc622faba4da7cf8
Author: Josh Fell <48...@users.noreply.github.com>
AuthorDate: Tue Feb 21 16:37:31 2023 -0500
FIx formatting of Dataset inlet/outlet note in TaskFlow concepts (#29678)
(cherry picked from commit 9de301da2a44385f57be5407e80e16ee376f3d39)
---
docs/apache-airflow/core-concepts/taskflow.rst | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/docs/apache-airflow/core-concepts/taskflow.rst b/docs/apache-airflow/core-concepts/taskflow.rst
index 97847455ab..40fe72dda0 100644
--- a/docs/apache-airflow/core-concepts/taskflow.rst
+++ b/docs/apache-airflow/core-concepts/taskflow.rst
@@ -91,9 +91,10 @@ need to be able to be serialized. Airflow out of the box supports all built-in t
supports objects that are decorated with ``@dataclass`` or ``@attr.define``. The following example shows the use of
a ``Dataset``, which is ``@attr.define`` decorated, together with TaskFlow.
-::
+.. note::
+
+ An additional benefit of using ``Dataset`` is that it automatically registers as an ``inlet`` in case it is used as an input argument. It also auto registers as an ``outlet`` if the return value of your task is a ``dataset`` or a ``list[Dataset]]``.
- Note: An additional benefit of using ``Dataset`` is that it automatically registers as an ``inlet`` in case it is used as an input argument. It also auto registers as an ``outlet`` if the return value of your task is a ``dataset`` or a ``list[Dataset]]``.
.. code-block:: python