You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2023/03/09 10:33:47 UTC
[airflow] branch v2-5-test updated (92a5d84212 -> 124179076a)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a change to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit 92a5d84212 Limit importlib-metadata backwport to < 5.0.0 (#29924)
omit 62476a708b Migrate remaining core sensors tests to `pytest` (#28204)
omit ba4ef7846c Move help message to the google auth code (#29888)
omit caef382367 Add Changelog for 2.5.2
new d6b92bfb25 Move help message to the google auth code (#29888)
new bb90f4afdc Migrate remaining core sensors tests to `pytest` (#28204)
new 955a5590b6 Limit importlib-metadata backwport to < 5.0.0 (#29924)
new 124179076a Add Changelog for 2.5.2
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (92a5d84212)
\
N -- N -- N refs/heads/v2-5-test (124179076a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
RELEASE_NOTES.rst | 22 +++++++++++-----------
docs/spelling_wordlist.txt | 1 +
2 files changed, 12 insertions(+), 11 deletions(-)
[airflow] 04/04: Add Changelog for 2.5.2
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 124179076a8369cb69f8f07ecf96abb64bed6478
Author: pierrejeambrun <pi...@gmail.com>
AuthorDate: Wed Mar 8 23:22:09 2023 +0100
Add Changelog for 2.5.2
---
RELEASE_NOTES.rst | 107 ++++++++++++++++++++++++++++++++++++
docs/spelling_wordlist.txt | 1 +
newsfragments/29395.significant.rst | 8 ---
newsfragments/29547.significant.rst | 3 -
4 files changed, 108 insertions(+), 11 deletions(-)
diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst
index ff777a75ed..a136e15303 100644
--- a/RELEASE_NOTES.rst
+++ b/RELEASE_NOTES.rst
@@ -21,6 +21,113 @@
.. towncrier release notes start
+Airflow 2.5.2 (2023-03-13)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+The date-time fields passed as API parameters or Params should be RFC3339-compliant (#29395)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+In case of API calls, it was possible that "+" passed as part of the date-time fields were not URL-encoded, and
+such date-time fields could pass validation. Such date-time parameters should now be URL-encoded (as ``%2B``).
+
+In case of parameters, we still allow IS8601-compliant date-time (so for example it is possible that
+' ' was used instead of ``T`` separating date from time and no timezone was specified) but we raise
+deprecation warning.
+
+Default for ``[webserver] expose_hostname`` changed to ``False`` (#29547)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The default for ``[webserver] expose_hostname`` has been set to ``False``, instead of ``True``. This means administrators must opt-in to expose webserver hostnames to end users.
+
+Bug Fixes
+^^^^^^^^^
+- Fix validation of date-time field in API and Parameter schemas (#29395)
+- Fix grid logs for large logs (#29390)
+- Fix on_failure_callback when task receives a SIGTERM (#29743)
+- Update min version of python-daemon to fix containerd file limits (#29916)
+- POST ``/dagRuns`` API should 404 if dag not active (#29860)
+- DAG list sorting lost when switching page (#29756)
+- Fix Scheduler crash when clear a previous run of a normal task that is now a mapped task (#29645)
+- Convert moment with timezone to UTC instead of raising an exception (#29606)
+- Fix clear dag run ``openapi`` spec responses by adding additional return type (#29600)
+- Don't display empty rendered attrs in Task Instance Details page (#29545)
+- Remove section check from get-value command (#29541)
+- Do not show version/node in UI traceback for unauthenticated user (#29501)
+- Make ``prev_logical_date`` variable offset-aware (#29454)
+- Fix nested fields rendering in mapped operators (#29451)
+- Datasets, next_run_datasets, remove unnecessary timestamp filter (#29441)
+- ``Edgemodifier`` refactoring w/ labels in TaskGroup edge case (#29410)
+- Fix Rest API update user output (#29409)
+- Ensure Serialized DAG is deleted (#29407)
+- Persist DAG and task doc values in TaskFlow API if explicitly set (#29399)
+- Redirect to the origin page with all the params (#29212)
+- Fixing Task Duration view in case of manual DAG runs only (#22015) (#29195)
+- Remove poke method to fall back to parent implementation (#29146)
+- PR: Introduced fix to run tasks on Windows systems (#29107)
+- Fix warning in migrations about old config. (#29092)
+- Emit dagrun failed duration when timeout (#29076)
+- Handling error on cluster policy itself (#29056)
+- Fix kerberos authentication for the REST API. (#29054)
+- Fix leak sensitive field via V1EnvVar on exception (#29016)
+- Sanitize url_for arguments before they are passed (#29039)
+- Fix dag run trigger with a note. (#29228)
+- Write action log to DB when DAG run is triggered via API (#28998)
+- Resolve all variables in pickled XCom iterator (#28982)
+- Allow URI without authority and host blocks in ``airflow connections add`` (#28922)
+- Be more selective when adopting pods with KubernetesExecutor (#28899)
+- KubenetesExecutor sends state even when successful (#28871)
+- Annotate KubernetesExecutor pods that we don't delete (#28844)
+- Throttle streaming log reads (#28818)
+- Introduce dag processor job (#28799)
+- Fix #28391 manual task trigger from UI fails for k8s executor (#28394)
+- Logging poke info when external dag is not none and task_id and task_ids are none (#28097)
+- Fix inconsistencies in checking edit permissions for a DAG (#20346)
+
+Misc/Internal
+^^^^^^^^^^^^^
+- Add a check for not templateable fields (#29821)
+- Removed continue for not in (#29791)
+- Move extra links position in grid view (#29703)
+- Bump ``undici`` from ``5.9.1`` to ``5.19.1`` (#29583)
+- Change expose_hostname default to false (#29547)
+- Change permissions of config/password files created by airflow (#29495)
+- Use newer setuptools ``v67.2.0`` (#29465)
+- Increase max height for grid view elements (#29367)
+- Clarify description of worker control config (#29247)
+- Bump ``ua-parser-js`` from ``0.7.31`` to ``0.7.33`` in ``/airflow/www`` (#29172)
+- Remove upper bound limitation for ``pytest`` (#29086)
+- Check for ``run_id`` url param when linking to ``graph/gantt`` views (#29066)
+- Clarify graph view dynamic task labels (#29042)
+- Fixing import error for dataset (#29007)
+- Update how PythonSensor returns values from ``python_callable`` (#28932)
+- Add dep context description for better log message (#28875)
+- Bump ``swagger-ui-dist`` from ``3.52.0`` to ``4.1.3`` in ``/airflow/www`` (#28824)
+
+Doc Only Changes
+^^^^^^^^^^^^^^^^
+- Update pipeline.rst - Fix query in ``merge_data()`` task (#29158)
+- Correct argument name of Workday timetable in timetable.rst (#29896)
+- Update ref anchor for env var link in Connection how-to doc (#29816)
+- Better description for limit in api (#29773)
+- Description of dag_processing.last_duration (#29740)
+- Update docs re: template_fields typing and subclasses (#29725)
+- Fix formatting of Dataset inlet/outlet note in TaskFlow concepts (#29678)
+- Specific use-case: adding packages via requirements.txt in compose (#29598)
+- Detect is 'docker-compose' existing (#29544)
+- Add Landing Times entry to UI docs (#29511)
+- Improve health checks in example docker-compose and clarify usage (#29408)
+- Remove ``notes`` param from TriggerDagRunOperator docstring (#29298)
+- Use ``schedule`` param rather than ``timetable`` in Timetables docs (#29255)
+- Add trigger process to Airflow Docker docs (#29203)
+- Update set-up-database.rst (#29104)
+- Several improvements to the Params doc (#29062)
+- Email Config docs more explicit env var examples (#28845)
+- Listener plugin example added (#27905)
+
+
Airflow 2.5.1 (2023-01-20)
--------------------------
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index 67cc6cfc65..b2ceb800d2 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -1350,6 +1350,7 @@ ssm
Stackdriver
stackdriver
stacktrace
+Standarization
starttls
StatefulSet
StatefulSets
diff --git a/newsfragments/29395.significant.rst b/newsfragments/29395.significant.rst
deleted file mode 100644
index 14f094397d..0000000000
--- a/newsfragments/29395.significant.rst
+++ /dev/null
@@ -1,8 +0,0 @@
-The date-time fields passed as API parameters or Params should be RFC3339-compliant.
-
-In case of API calls, it was possible that "+" passed as part of the date-time fields were not URL-encoded, and
-such date-time fields could pass validation. Such date-time parameters should now be URL-encoded (as ``%2B``).
-
-In case of parameters, we still allow IS8601-compliant date-time (so for example it is possible that
-' ' was used instead of ``T`` separating date from time and no timezone was specified) but we raise
-deprecation warning.
diff --git a/newsfragments/29547.significant.rst b/newsfragments/29547.significant.rst
deleted file mode 100644
index 2a9f4c45c3..0000000000
--- a/newsfragments/29547.significant.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Default for ``[webserver] expose_hostname`` changed to ``False``
-
-The default for ``[webserver] expose_hostname`` has been set to ``False``, instead of ``True``. This means administrators must opt-in to expose webserver hostnames to end users.
[airflow] 03/04: Limit importlib-metadata backwport to < 5.0.0 (#29924)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 955a5590b6adca0c4b07ddc4dc9b2d60a7d2d508
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Mar 5 06:11:04 2023 +0100
Limit importlib-metadata backwport to < 5.0.0 (#29924)
The Importlib-metadata 5 breaks importing Celery < 5.3.0. Until
Celery 5.3.0 gets released (it is pre-release/beta now) we should
workaround the problem by limiting importlib-metadata to < 5.0.0.
(cherry picked from commit 4ea457a5ca694eab150e02a26c0b3ef62858fbca)
---
setup.cfg | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/setup.cfg b/setup.cfg
index a304b64163..0e359f6674 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -99,7 +99,10 @@ install_requires =
graphviz>=0.12
gunicorn>=20.1.0
httpx
- importlib_metadata>=1.7;python_version<"3.9"
+ # Importlib-metadata 5 is breaking Celery import due to regression it introduced
+ # This was tracked and fixed in https://github.com/celery/celery/pull/7785 but it is not released yet
+ # We can remove the < 5.0.0 limitation hwne Celery 5.3.0 gets released and we bump celeryt o >= 5.3.0
+ importlib_metadata>=1.7,<5.0.0;python_version<"3.9"
importlib_resources>=5.2;python_version<"3.9"
itsdangerous>=2.0
jinja2>=3.0.0
[airflow] 02/04: Migrate remaining core sensors tests to `pytest` (#28204)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit bb90f4afdc55129dca73b3e412f49c3b533d5d17
Author: Andrey Anshin <An...@taragol.is>
AuthorDate: Thu Dec 8 16:55:27 2022 +0400
Migrate remaining core sensors tests to `pytest` (#28204)
(cherry picked from commit e93b669d841828b5f912edf40f255299b1833521)
---
tests/sensors/test_date_time.py | 13 ++--
tests/sensors/test_external_task_sensor.py | 79 +++++++++++-----------
tests/sensors/test_time_sensor.py | 6 +-
tests/task/__init__.py | 4 --
.../task/{__init__.py => task_runner/conftest.py} | 24 ++++++-
5 files changed, 69 insertions(+), 57 deletions(-)
diff --git a/tests/sensors/test_date_time.py b/tests/sensors/test_date_time.py
index b82fbf3f07..a298300c54 100644
--- a/tests/sensors/test_date_time.py
+++ b/tests/sensors/test_date_time.py
@@ -20,7 +20,6 @@ from __future__ import annotations
from unittest.mock import patch
import pytest
-from parameterized import parameterized
from airflow.models.dag import DAG
from airflow.sensors.date_time import DateTimeSensor
@@ -35,7 +34,8 @@ class TestDateTimeSensor:
args = {"owner": "airflow", "start_date": DEFAULT_DATE}
cls.dag = DAG("test_dag", default_args=args)
- @parameterized.expand(
+ @pytest.mark.parametrize(
+ "task_id, target_time, expected",
[
(
"valid_datetime",
@@ -52,7 +52,7 @@ class TestDateTimeSensor:
"{{ ds }}",
"{{ ds }}",
),
- ]
+ ],
)
def test_valid_input(self, task_id, target_time, expected):
"""target_time should be a string as it is a template field"""
@@ -71,7 +71,8 @@ class TestDateTimeSensor:
dag=self.dag,
)
- @parameterized.expand(
+ @pytest.mark.parametrize(
+ "task_id, target_time, expected",
[
(
"poke_datetime",
@@ -80,12 +81,12 @@ class TestDateTimeSensor:
),
("poke_str_extended", "2020-01-01T23:00:00.001+00:00", False),
("poke_str_basic_with_tz", "20200102T065959+8", True),
- ]
+ ],
)
@patch(
"airflow.sensors.date_time.timezone.utcnow",
return_value=timezone.datetime(2020, 1, 1, 23, 0, tzinfo=timezone.utc),
)
- def test_poke(self, task_id, target_time, expected, mock_utcnow):
+ def test_poke(self, mock_utcnow, task_id, target_time, expected):
op = DateTimeSensor(task_id=task_id, target_time=target_time, dag=self.dag)
assert op.poke(None) == expected
diff --git a/tests/sensors/test_external_task_sensor.py b/tests/sensors/test_external_task_sensor.py
index 1b5a5032cf..c02f9a8752 100644
--- a/tests/sensors/test_external_task_sensor.py
+++ b/tests/sensors/test_external_task_sensor.py
@@ -21,7 +21,6 @@ import hashlib
import logging
import os
import tempfile
-import unittest
import zipfile
from datetime import time, timedelta
@@ -91,8 +90,8 @@ def dag_zip_maker():
yield DagZipMaker()
-class TestExternalTaskSensor(unittest.TestCase):
- def setUp(self):
+class TestExternalTaskSensor:
+ def setup_method(self):
self.dagbag = DagBag(dag_folder=DEV_NULL, include_examples=True)
self.args = {"owner": "airflow", "start_date": DEFAULT_DATE}
self.dag = DAG(TEST_DAG_ID, default_args=self.args)
@@ -200,8 +199,9 @@ class TestExternalTaskSensor(unittest.TestCase):
task_id="test_external_task_sensor_check",
external_dag_id=TEST_DAG_ID,
external_task_group_id="fake-task-group",
- timeout=1,
+ timeout=0.001,
dag=self.dag,
+ poke_interval=0.1,
)
op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
@@ -266,7 +266,7 @@ class TestExternalTaskSensor(unittest.TestCase):
)
op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
- def test_external_task_sensor_failed_states_as_success(self):
+ def test_external_task_sensor_failed_states_as_success(self, caplog):
self.add_time_sensor()
op = ExternalTaskSensor(
task_id="test_external_task_sensor_check",
@@ -276,20 +276,16 @@ class TestExternalTaskSensor(unittest.TestCase):
failed_states=["success"],
dag=self.dag,
)
- with self.assertLogs(op.log, level=logging.INFO) as cm:
- with pytest.raises(AirflowException) as ctx:
+ error_message = rf"Some of the external tasks \['{TEST_TASK_ID}'\] in DAG {TEST_DAG_ID} failed\."
+ with pytest.raises(AirflowException, match=error_message):
+ with caplog.at_level(logging.INFO, logger=op.log.name):
+ caplog.clear()
op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
- assert (
- f"INFO:airflow.task.operators:Poking for tasks ['time_sensor_check'] "
- f"in dag unit_test_dag on {DEFAULT_DATE.isoformat()} ... " in cm.output
- )
- assert (
- str(ctx.value) == "Some of the external tasks "
- "['time_sensor_check'] in DAG "
- "unit_test_dag failed."
- )
+ assert (
+ f"Poking for tasks ['{TEST_TASK_ID}'] in dag {TEST_DAG_ID} on {DEFAULT_DATE.isoformat()} ... "
+ ) in caplog.messages
- def test_external_task_sensor_soft_fail_failed_states_as_skipped(self, session=None):
+ def test_external_task_sensor_soft_fail_failed_states_as_skipped(self):
self.add_time_sensor()
op = ExternalTaskSensor(
task_id="test_external_task_sensor_check",
@@ -311,7 +307,7 @@ class TestExternalTaskSensor(unittest.TestCase):
assert len(task_instances) == 1, "Unexpected number of task instances"
assert task_instances[0].state == State.SKIPPED, "Unexpected external task state"
- def test_external_task_sensor_external_task_id_param(self):
+ def test_external_task_sensor_external_task_id_param(self, caplog):
"""Test external_task_ids is set properly when external_task_id is passed as a template"""
self.add_time_sensor()
op = ExternalTaskSensor(
@@ -322,14 +318,15 @@ class TestExternalTaskSensor(unittest.TestCase):
dag=self.dag,
)
- with self.assertLogs(op.log, level=logging.INFO) as cm:
+ with caplog.at_level(logging.INFO, logger=op.log.name):
+ caplog.clear()
op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
assert (
- f"INFO:airflow.task.operators:Poking for tasks ['{TEST_TASK_ID}'] "
- f"in dag unit_test_dag on {DEFAULT_DATE.isoformat()} ... " in cm.output
- )
+ f"Poking for tasks ['{TEST_TASK_ID}'] "
+ f"in dag {TEST_DAG_ID} on {DEFAULT_DATE.isoformat()} ... "
+ ) in caplog.messages
- def test_external_task_sensor_external_task_ids_param(self):
+ def test_external_task_sensor_external_task_ids_param(self, caplog):
"""Test external_task_ids rendering when a template is passed."""
self.add_time_sensor()
op = ExternalTaskSensor(
@@ -340,14 +337,15 @@ class TestExternalTaskSensor(unittest.TestCase):
dag=self.dag,
)
- with self.assertLogs(op.log, level=logging.INFO) as cm:
+ with caplog.at_level(logging.INFO, logger=op.log.name):
+ caplog.clear()
op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
assert (
- f"INFO:airflow.task.operators:Poking for tasks ['{TEST_TASK_ID}'] "
- f"in dag unit_test_dag on {DEFAULT_DATE.isoformat()} ... " in cm.output
- )
+ f"Poking for tasks ['{TEST_TASK_ID}'] "
+ f"in dag {TEST_DAG_ID} on {DEFAULT_DATE.isoformat()} ... "
+ ) in caplog.messages
- def test_external_task_sensor_failed_states_as_success_mulitple_task_ids(self):
+ def test_external_task_sensor_failed_states_as_success_mulitple_task_ids(self, caplog):
self.add_time_sensor(task_id=TEST_TASK_ID)
self.add_time_sensor(task_id=TEST_TASK_ID_ALTERNATE)
op = ExternalTaskSensor(
@@ -358,19 +356,18 @@ class TestExternalTaskSensor(unittest.TestCase):
failed_states=["success"],
dag=self.dag,
)
- with self.assertLogs(op.log, level=logging.INFO) as cm:
- with pytest.raises(AirflowException) as ctx:
+ error_message = (
+ rf"Some of the external tasks \['{TEST_TASK_ID}'\, \'{TEST_TASK_ID_ALTERNATE}\'] "
+ rf"in DAG {TEST_DAG_ID} failed\."
+ )
+ with pytest.raises(AirflowException, match=error_message):
+ with caplog.at_level(logging.INFO, logger=op.log.name):
+ caplog.clear()
op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
- assert (
- f"INFO:airflow.task.operators:Poking for tasks "
- f"['time_sensor_check', 'time_sensor_check_alternate'] "
- f"in dag unit_test_dag on {DEFAULT_DATE.isoformat()} ... " in cm.output
- )
- assert (
- str(ctx.value) == "Some of the external tasks "
- "['time_sensor_check', 'time_sensor_check_alternate'] in DAG "
- "unit_test_dag failed."
- )
+ assert (
+ f"Poking for tasks ['{TEST_TASK_ID}', '{TEST_TASK_ID_ALTERNATE}'] "
+ f"in dag unit_test_dag on {DEFAULT_DATE.isoformat()} ... "
+ ) in caplog.messages
def test_external_dag_sensor(self):
other_dag = DAG("other_dag", default_args=self.args, end_date=DEFAULT_DATE, schedule="@once")
@@ -767,7 +764,7 @@ def test_external_task_sensor_templated(dag_maker, app):
assert f"/dags/dag_{DEFAULT_DATE.date()}/grid" in url
-class TestExternalTaskMarker(unittest.TestCase):
+class TestExternalTaskMarker:
def test_serialized_fields(self):
assert {"recursion_depth"}.issubset(ExternalTaskMarker.get_serialized_fields())
diff --git a/tests/sensors/test_time_sensor.py b/tests/sensors/test_time_sensor.py
index 06f2332625..70b875311c 100644
--- a/tests/sensors/test_time_sensor.py
+++ b/tests/sensors/test_time_sensor.py
@@ -23,7 +23,6 @@ from unittest.mock import patch
import freezegun
import pendulum
import pytest
-from parameterized import parameterized
from airflow.exceptions import TaskDeferred
from airflow.models.dag import DAG
@@ -41,12 +40,13 @@ DEFAULT_DATE_WITH_TZ = datetime(2015, 1, 1, tzinfo=pendulum.tz.timezone(DEFAULT_
return_value=timezone.datetime(2020, 1, 1, 23, 0).replace(tzinfo=timezone.utc),
)
class TestTimeSensor:
- @parameterized.expand(
+ @pytest.mark.parametrize(
+ "default_timezone, start_date, expected",
[
("UTC", DEFAULT_DATE_WO_TZ, True),
("UTC", DEFAULT_DATE_WITH_TZ, False),
(DEFAULT_TIMEZONE, DEFAULT_DATE_WO_TZ, False),
- ]
+ ],
)
def test_timezone(self, mock_utcnow, default_timezone, start_date, expected):
with patch("airflow.settings.TIMEZONE", pendulum.timezone(default_timezone)):
diff --git a/tests/task/__init__.py b/tests/task/__init__.py
index 865dd751b4..217e5db960 100644
--- a/tests/task/__init__.py
+++ b/tests/task/__init__.py
@@ -15,7 +15,3 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-# flake8: noqa
-from __future__ import annotations
-
-from .task_runner import *
diff --git a/tests/task/__init__.py b/tests/task/task_runner/conftest.py
similarity index 55%
copy from tests/task/__init__.py
copy to tests/task/task_runner/conftest.py
index 865dd751b4..3af87d1ed7 100644
--- a/tests/task/__init__.py
+++ b/tests/task/task_runner/conftest.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,7 +14,26 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-# flake8: noqa
+
from __future__ import annotations
-from .task_runner import *
+import logging
+from logging.config import dictConfig
+
+import pytest
+
+from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
+from airflow.listeners.listener import get_listener_manager
+
+
+@pytest.fixture(scope="module", autouse=True)
+def reset_to_default_logging():
+ """
+ Initialize ``BaseTaskRunner`` might have side effect to another tests.
+ This fixture reset back logging to default after execution of separate module in this test package.
+ """
+ yield
+ airflow_logger = logging.getLogger("airflow")
+ airflow_logger.handlers = []
+ dictConfig(DEFAULT_LOGGING_CONFIG)
+ get_listener_manager().clear()
[airflow] 01/04: Move help message to the google auth code (#29888)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-5-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d6b92bfb25ec0628569a171c9c906bbe04d69a19
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Mar 3 11:11:23 2023 +0100
Move help message to the google auth code (#29888)
The "google-auth" 2.16.2 just released, removed the _HELP_MESSAGE
that was imported by google provider auth util thus failing the
imports in canary builds of ours attempting to upgrade to
newver versions of released libraries.
This PR inlines the original help message into our code to make
it independent from google-auth package version used.
(cherry picked from commit 47ab0ca6295f301d393dbb29e134fa90be0ad7df)
---
...et_candidate_file_descriptor_ranges-to-us.patch | 76 ++++++++++++++++++++++
airflow/_vendor/daemon/module.mk | 27 ++++++++
.../google/common/utils/id_token_credentials.py | 26 +++++++-
.../common/utils/test_id_token_credentials.py | 7 +-
4 files changed, 128 insertions(+), 8 deletions(-)
diff --git a/airflow/_vendor/daemon/0001-Refactor-_get_candidate_file_descriptor_ranges-to-us.patch b/airflow/_vendor/daemon/0001-Refactor-_get_candidate_file_descriptor_ranges-to-us.patch
new file mode 100644
index 0000000000..75e9cb187d
--- /dev/null
+++ b/airflow/_vendor/daemon/0001-Refactor-_get_candidate_file_descriptor_ranges-to-us.patch
@@ -0,0 +1,76 @@
+From dd2f3d76c32ed12d56ae18b8ecb7086a8d9b1180 Mon Sep 17 00:00:00 2001
+From: Igor Kholopov <kh...@gmail.com>
+Date: Thu, 1 Dec 2022 18:34:30 +1100
+Subject: [PATCH 01/15] =?UTF-8?q?Refactor=20=E2=80=98=5Fget=5Fcandidate=5F?=
+ =?UTF-8?q?file=5Fdescriptor=5Franges=E2=80=99=20to=20use=20=E2=80=98range?=
+ =?UTF-8?q?=E2=80=99=20objects.?=
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+This avoids the requirement for a concrete set of all the actual file
+descriptors. When the range is very large, this can greatly improve the memory
+requirements and execution time of this function.
+---
+ daemon/daemon.py | 43 ++++++++++++++++++++++---------------------
+ 1 file changed, 22 insertions(+), 21 deletions(-)
+
+diff --git a/daemon/daemon.py b/daemon/daemon.py
+index 08245a4..d1673be 100644
+--- a/daemon/daemon.py
++++ b/daemon/daemon.py
+@@ -921,29 +921,30 @@ def _get_candidate_file_descriptor_ranges(exclude):
+ in this process, excluding those integers in the `exclude`
+ collection.
+ """
+- candidates_list = sorted(_get_candidate_file_descriptors(exclude))
++ _validate_fd_values(exclude)
+ ranges = []
+
+- def append_range_if_needed(candidate_range):
+- (low, high) = candidate_range
+- if (low < high):
+- # The range is not empty.
+- ranges.append(candidate_range)
+-
+- this_range = (
+- (min(candidates_list), (min(candidates_list) + 1))
+- if candidates_list else (0, 0))
+- for fd in candidates_list[1:]:
+- high = fd + 1
+- if this_range[1] == fd:
+- # This file descriptor extends the current range.
+- this_range = (this_range[0], high)
+- else:
+- # The previous range has ended at a gap.
+- append_range_if_needed(this_range)
+- # This file descriptor begins a new range.
+- this_range = (fd, high)
+- append_range_if_needed(this_range)
++ remaining_range = _total_file_descriptor_range
++ for exclude_fd in sorted(exclude):
++ if (exclude_fd > remaining_range.stop):
++ # This and all later exclusions are higher than the remaining
++ # range.
++ break
++ if (exclude_fd < remaining_range.start):
++ # The remaining range does not include the current exclusion.
++ continue
++ if (exclude_fd != remaining_range.start):
++ # There is a candidate range below the current exclusion.
++ ranges.append((remaining_range.start, exclude_fd))
++ # Narrow the remaining range to those above the current exclusion.
++ remaining_range = range(exclude_fd + 1, remaining_range.stop)
++
++ if (remaining_range.start < remaining_range.stop):
++ # After excluding all the specified file descriptors, there is a
++ # remaining range; append it as a candidate for closing file
++ # descriptors.
++ ranges.append((remaining_range.start, remaining_range.stop))
++
+ return ranges
+
+
+--
+2.34.1
+
diff --git a/airflow/_vendor/daemon/module.mk b/airflow/_vendor/daemon/module.mk
new file mode 100644
index 0000000000..063a7ca5a1
--- /dev/null
+++ b/airflow/_vendor/daemon/module.mk
@@ -0,0 +1,27 @@
+# daemon/module.mk
+# Part of ‘python-daemon’, an implementation of PEP 3143.
+#
+# This is free software, and you are welcome to redistribute it under
+# certain conditions; see the end of this file for copyright
+# information, grant of license, and disclaimer of warranty.
+
+# Makefile module for ‘daemon’ Python package.
+
+MODULE_DIR := $(CURDIR)/daemon
+
+CODE_MODULES += $(shell find ${MODULE_DIR}/ -name '*.py')
+
+
+# Copyright © 2006–2023 Ben Finney <be...@benfinney.id.au>
+#
+# This is free software: you may copy, modify, and/or distribute this work
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; version 3 of that license or any later version.
+# No warranty expressed or implied. See the file ‘LICENSE.GPL-3’ for details.
+
+
+# Local Variables:
+# mode: makefile
+# coding: utf-8
+# End:
+# vim: fileencoding=utf-8 filetype=make :
diff --git a/airflow/providers/google/common/utils/id_token_credentials.py b/airflow/providers/google/common/utils/id_token_credentials.py
index 6cac058992..73a3d588c3 100644
--- a/airflow/providers/google/common/utils/id_token_credentials.py
+++ b/airflow/providers/google/common/utils/id_token_credentials.py
@@ -36,9 +36,26 @@ import os
import google.auth.transport
import google.oauth2
from google.auth import credentials as google_auth_credentials, environment_vars, exceptions
-from google.auth._default import _AUTHORIZED_USER_TYPE, _HELP_MESSAGE, _SERVICE_ACCOUNT_TYPE, _VALID_TYPES
from google.oauth2 import credentials as oauth2_credentials, service_account
+# Valid types accepted for file-based credentials.
+# They are taken from "google.auth._default" and since they are all "protected" and the imports might
+# change any time and fail the whole Google provider functionality - we should inline them
+_AUTHORIZED_USER_TYPE = "authorized_user"
+_SERVICE_ACCOUNT_TYPE = "service_account"
+_EXTERNAL_ACCOUNT_TYPE = "external_account"
+_EXTERNAL_ACCOUNT_AUTHORIZED_USER_TYPE = "external_account_authorized_user"
+_IMPERSONATED_SERVICE_ACCOUNT_TYPE = "impersonated_service_account"
+_GDCH_SERVICE_ACCOUNT_TYPE = "gdch_service_account"
+_VALID_TYPES = (
+ _AUTHORIZED_USER_TYPE,
+ _SERVICE_ACCOUNT_TYPE,
+ _EXTERNAL_ACCOUNT_TYPE,
+ _EXTERNAL_ACCOUNT_AUTHORIZED_USER_TYPE,
+ _IMPERSONATED_SERVICE_ACCOUNT_TYPE,
+ _GDCH_SERVICE_ACCOUNT_TYPE,
+)
+
class IDTokenCredentialsAdapter(google_auth_credentials.Credentials):
"""Convert Credentials with ``openid`` scope to IDTokenCredentials."""
@@ -198,7 +215,12 @@ def get_default_id_token_credentials(
if current_credentials is not None:
return current_credentials
- raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
+ raise exceptions.DefaultCredentialsError(
+ f"""Could not automatically determine credentials. Please set {environment_vars.CREDENTIALS} or
+ explicitly create credentials and re-run the application. For more information, please see
+ https://cloud.google.com/docs/authentication/getting-started
+""".strip()
+ )
if __name__ == "__main__":
diff --git a/tests/providers/google/common/utils/test_id_token_credentials.py b/tests/providers/google/common/utils/test_id_token_credentials.py
index cf0d4c1f71..a3a4d88dcb 100644
--- a/tests/providers/google/common/utils/test_id_token_credentials.py
+++ b/tests/providers/google/common/utils/test_id_token_credentials.py
@@ -18,7 +18,6 @@ from __future__ import annotations
import json
import os
-import re
import unittest
from unittest import mock
@@ -61,11 +60,7 @@ class TestGetDefaultIdTokenCredentials(unittest.TestCase):
del os.environ[CREDENTIALS]
with pytest.raises(
exceptions.DefaultCredentialsError,
- match=re.escape(
- "Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS "
- "or explicitly create credentials and re-run the application. For more information, please "
- "see https://cloud.google.com/docs/authentication/getting-started"
- ),
+ match="Please set GOOGLE_APPLICATION_CREDENTIALS",
):
get_default_id_token_credentials(target_audience="example.org")