You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2022/07/06 10:48:55 UTC
[airflow] branch v2-3-test updated (7bbb14a12c -> 4d2ec8444b)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a change to branch v2-3-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 7bbb14a12c Fix exception in mini task scheduler. (#24865)
omit 21a37cfc41 Fix the implementation of UtcDateTime for MsSQL
omit 16c734d440 Update the release note
new 89f96a8830 Fix the implementation of UtcDateTime for MsSQL
new 77bec7ef0f Fix exception in mini task scheduler. (#24865)
new 4d2ec8444b Update the release note
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (7bbb14a12c)
\
N -- N -- N refs/heads/v2-3-test (4d2ec8444b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
RELEASE_NOTES.rst | 1 +
1 file changed, 1 insertion(+)
[airflow] 02/03: Fix exception in mini task scheduler. (#24865)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-3-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 77bec7ef0fb4b0a7b00004e292f9574c6c6e8c59
Author: Ash Berlin-Taylor <as...@apache.org>
AuthorDate: Wed Jul 6 11:34:48 2022 +0100
Fix exception in mini task scheduler. (#24865)
I introduced a bug in 2.3.0 as part of the dynamic task mapping work
that frequently made the mini scheduler fail for tasks involving
XComArgs.
The fix is to alter the logic in BaseOperator's deepcopy to not set the
`__instantiated` flag until all the other attributes are copied.
For background the `__instantiated` flag is use so that when you do
`task.some_attr = an_xcom_arg` the relationships are set appropriately,
but since we are copying all the existing attributes we don't need to do
that, as the relationships will already be set!
(cherry picked from commit c23b31cd786760da8a8e39ecbcf2c0d31e50e594)
---
airflow/models/baseoperator.py | 11 +++++++----
tests/models/test_baseoperator.py | 13 +++++++++++++
2 files changed, 20 insertions(+), 4 deletions(-)
diff --git a/airflow/models/baseoperator.py b/airflow/models/baseoperator.py
index 2fea760716..30990880dc 100644
--- a/airflow/models/baseoperator.py
+++ b/airflow/models/baseoperator.py
@@ -1140,11 +1140,10 @@ class BaseOperator(AbstractOperator, metaclass=BaseOperatorMeta):
"""
def __deepcopy__(self, memo):
- """
- Hack sorting double chained task lists by task_id to avoid hitting
- max_depth on deepcopy operations.
- """
+ # Hack sorting double chained task lists by task_id to avoid hitting
+ # max_depth on deepcopy operations.
sys.setrecursionlimit(5000) # TODO fix this in a better way
+
cls = self.__class__
result = cls.__new__(cls)
memo[id(self)] = result
@@ -1152,10 +1151,14 @@ class BaseOperator(AbstractOperator, metaclass=BaseOperatorMeta):
shallow_copy = cls.shallow_copy_attrs + cls._base_operator_shallow_copy_attrs
for k, v in self.__dict__.items():
+ if k == "_BaseOperator__instantiated":
+ # Don't set this until the _end_, as it changes behaviour of __setattr__
+ continue
if k not in shallow_copy:
setattr(result, k, copy.deepcopy(v, memo))
else:
setattr(result, k, copy.copy(v))
+ result.__instantiated = self.__instantiated
return result
def __getstate__(self):
diff --git a/tests/models/test_baseoperator.py b/tests/models/test_baseoperator.py
index 8cb8d96e81..40c228c961 100644
--- a/tests/models/test_baseoperator.py
+++ b/tests/models/test_baseoperator.py
@@ -15,6 +15,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+import copy
import logging
import uuid
from datetime import date, datetime, timedelta
@@ -994,3 +995,15 @@ def test_mapped_render_template_fields_validating_operator(dag_maker, session):
assert op.value == "{{ ds }}", "Should not be templated!"
assert op.arg1 == "{{ ds }}"
assert op.arg2 == "a"
+
+
+def test_deepcopy():
+ # Test bug when copying an operator attached to a DAG
+ with DAG("dag0", start_date=DEFAULT_DATE) as dag:
+
+ @dag.task
+ def task0():
+ pass
+
+ MockOperator(task_id="task1", arg1=task0())
+ copy.deepcopy(dag)
[airflow] 01/03: Fix the implementation of UtcDateTime for MsSQL
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-3-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 89f96a8830633f21f062e65d2836fde90416ff70
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Tue Jul 5 22:40:23 2022 +0000
Fix the implementation of UtcDateTime for MsSQL
---
airflow/utils/sqlalchemy.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/airflow/utils/sqlalchemy.py b/airflow/utils/sqlalchemy.py
index b10af0a74d..c94b4100dd 100644
--- a/airflow/utils/sqlalchemy.py
+++ b/airflow/utils/sqlalchemy.py
@@ -24,6 +24,7 @@ from typing import Any, Dict, Iterable, Tuple
import pendulum
from dateutil import relativedelta
from sqlalchemy import and_, event, false, nullsfirst, or_, tuple_
+from sqlalchemy.dialects import mssql
from sqlalchemy.exc import OperationalError
from sqlalchemy.orm.session import Session
from sqlalchemy.sql import ColumnElement
@@ -94,6 +95,11 @@ class UtcDateTime(TypeDecorator):
return value
+ def load_dialect_impl(self, dialect):
+ if dialect.name == 'mssql':
+ return mssql.DATETIME2(precision=6)
+ return super().load_dialect_impl(dialect)
+
class ExtendedJSON(TypeDecorator):
"""
[airflow] 03/03: Update the release note
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-3-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4d2ec8444be937f9e85e7f1ba3540f6f81694bf8
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Tue Jul 5 15:44:34 2022 +0100
Update the release note
---
RELEASE_NOTES.rst | 21 ++++++++++++++++++++-
docs/docker-stack/index.rst | 4 ++--
docs/spelling_wordlist.txt | 1 +
newsfragments/23647.bugfix.rst | 1 -
4 files changed, 23 insertions(+), 4 deletions(-)
diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst
index ed9ea874af..3791ac7dcd 100644
--- a/RELEASE_NOTES.rst
+++ b/RELEASE_NOTES.rst
@@ -64,8 +64,23 @@ Here is the list of breaking changes in dependencies that comes together with FA
Bug Fixes
^^^^^^^^^
+- Fix exception in mini task scheduler (#24865)
+- Fix cycle bug with attaching label to task group (#24847)
+- Fix timestamp defaults for ``sensorinstance`` (#24638)
+- Move fallible ``ti.task.dag`` assignment back inside ``try/except`` block (#24533) (#24592)
+- Add missing types to ``FSHook`` (#24470)
+- Mask secrets in ``stdout`` for ``airflow tasks test`` (#24362)
+- ``DebugExecutor`` use ``ti.run()`` instead of ``ti._run_raw_task`` (#24357)
+- Fix bugs in ``URI`` constructor for ``MySQL`` connection (#24320)
+- Missing ``scheduleinterval`` nullable true added in ``openapi`` (#24253)
+- Unify ``return_code`` interface for task runner (#24093)
+- Handle occasional deadlocks in trigger with retries (#24071)
+- Remove special serde logic for mapped ``op_kwargs`` (#23860)
+- ``ExternalTaskSensor`` respects ``soft_fail`` if the external task enters a ``failed_state`` (#23647)
+- Fix ``StatD`` timing metric units (#21106)
+- Add ``cache_ok`` flag to sqlalchemy TypeDecorators. (#24499)
- Allow for ``LOGGING_LEVEL=DEBUG`` (#23360)
-- Fix grid date ticks (#24738)
+- Fix grid date ticks (#24738, #24849)
- Debounce status highlighting in Grid view (#24710)
- Fix Grid vertical scrolling (#24684)
- don't try to render child rows for closed groups (#24637)
@@ -107,6 +122,7 @@ Bug Fixes
Doc only changes
^^^^^^^^^^^^^^^^
+- Fix doc description of ``[core]`` parallelism config setting (#23768)
- Update templates doc to mention ``extras`` and format Airflow ``Vars`` / ``Conns`` (#24735)
- Document built in Timetables (#23099)
- Alphabetizes two tables (#23923)
@@ -121,6 +137,9 @@ Doc only changes
Misc/Internal
^^^^^^^^^^^^^
+- Do not support ``MSSQL`` less than ``v2017`` in code (#24095)
+- Drop Python ``3.6`` compatibility objects/modules (#24048)
+- Remove upper-binding for SQLAlchemy (#24819)
- Remove internet explorer support (#24495)
- Removing magic status code numbers from ``api_connexion`` (#24050)
- Upgrade FAB to ``4.1.2`` (#24619)
diff --git a/docs/docker-stack/index.rst b/docs/docker-stack/index.rst
index ed52625031..87e1f4088c 100644
--- a/docs/docker-stack/index.rst
+++ b/docs/docker-stack/index.rst
@@ -59,7 +59,7 @@ Those are "reference" regular images. They contain the most common set of extras
often used by the users and they are good to "try-things-out" when you want to just take Airflow for a spin,
You can also use "slim" images that contain only core airflow and are about half the size of the "regular" images
-but you need to add all the :doc:`extra-packages-ref` and providers that you need separately
+but you need to add all the extra packages and providers that you need separately
via :ref:`Building the image <build:build_image>`.
* :subst-code:`apache/airflow:slim-latest` - the latest released Airflow image with default Python version (3.7 currently)
@@ -69,7 +69,7 @@ via :ref:`Building the image <build:build_image>`.
The Apache Airflow image provided as convenience package is optimized for size, and
it provides just a bare minimal set of the extras and dependencies installed and in most cases
-you want to either extend or customize the image. You can see all possible extras in :doc:`extra-packages-ref`.
+you want to either extend or customize the image. You can see all possible extras in :doc:`apache-airflow:extra-packages-ref`.
The set of extras used in Airflow Production image are available in the
`Dockerfile <https://github.com/apache/airflow/blob/2c6c7fdb2308de98e142618836bdf414df9768c8/Dockerfile#L37>`_.
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index fd7a57694d..33a6acdc50 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -1319,6 +1319,7 @@ seealso
seedlist
segmentGranularity
sendgrid
+serde
serialise
serializable
serverless
diff --git a/newsfragments/23647.bugfix.rst b/newsfragments/23647.bugfix.rst
deleted file mode 100644
index d12c1d7046..0000000000
--- a/newsfragments/23647.bugfix.rst
+++ /dev/null
@@ -1 +0,0 @@
-``ExternalTaskSensor`` now supports the ``soft_fail`` flag to skip if external task or DAG enters a failed state.