You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2021/09/15 21:20:34 UTC
[airflow] branch v2-1-test updated (4377be3 -> 614858f)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a change to branch v2-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 4377be3 Update CHANGELOG.txt
new 66dcbf4 Fix ``DagRunState`` enum query for ``MySQLdb`` driver (#17886)
new bb8abef Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)
new f0f4e51 Advises the kernel to not cache log files generated by Airflow (#18054)
new abc3280 Doc: Improve installing from sources (#18194)
new 84f8b99 Doc: Use ``closer.lua`` script for downloading sources (#18179)
new f0bccdc Improves installing from sources pages for all components (#18251)
new 614858f Update Changelog 2.1.4rc2
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGELOG.txt | 8 +-
airflow/executors/celery_executor.py | 4 +-
airflow/utils/cli.py | 3 +-
airflow/utils/log/file_processor_handler.py | 3 +-
airflow/utils/log/file_task_handler.py | 3 +-
airflow/utils/log/non_caching_file_handler.py | 46 +++++++
airflow/utils/state.py | 3 +
docs/apache-airflow-providers/index.rst | 10 +-
.../installing-from-sources.rst | 59 ++++-----
docs/apache-airflow/index.rst | 7 +-
docs/apache-airflow/installing-from-sources.rst | 81 ++++++++++---
docs/conf.py | 41 ++++++-
docs/helm-chart/index.rst | 6 +
.../installing-helm-chart-from-sources.rst | 132 +++++++++++++++++++++
tests/executors/test_celery_executor.py | 28 +++++
tests/utils/{test_types.py => test_state.py} | 19 ++-
16 files changed, 385 insertions(+), 68 deletions(-)
create mode 100644 airflow/utils/log/non_caching_file_handler.py
copy docs/{apache-airflow => apache-airflow-providers}/installing-from-sources.rst (54%)
create mode 100644 docs/helm-chart/installing-helm-chart-from-sources.rst
copy tests/utils/{test_types.py => test_state.py} (76%)
[airflow] 07/07: Update Changelog 2.1.4rc2
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v2-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 614858fb7d443880451e6111b27fdaf942f563a4
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Wed Sep 15 22:18:37 2021 +0100
Update Changelog 2.1.4rc2
---
CHANGELOG.txt | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/CHANGELOG.txt b/CHANGELOG.txt
index b5749d6..853e0c0 100644
--- a/CHANGELOG.txt
+++ b/CHANGELOG.txt
@@ -1,4 +1,4 @@
-Airflow 2.1.4, 2021-09-15
+Airflow 2.1.4, 2021-09-18
-------------------------
Bug Fixes
@@ -27,6 +27,9 @@ Bug Fixes
- Do not let ``create_dagrun`` overwrite explicit ``run_id`` (#17728)
- BugFix: Regression on pid reset to allow task start after heartbeat (#17333)
- Set task state to failed when pod is DELETED while running (#18095)
+- Advises the kernel to not cache log files generated by Airflow (#18054)
+- Sort adopted tasks in ``_check_for_stalled_adopted_tasks`` method (#18208)
+- Fix ``DagRunState`` enum query for ``MySQLdb`` driver (#17886)
Doc only changes
""""""""""""""""
@@ -40,6 +43,9 @@ Doc only changes
- Suggest to use secrets backend for variable when it contains sensitive data (#17319)
- Improves quick-start docker-compose warnings and documentation (#18164)
- Separate Installing from sources section and add more details (#18171)
+- Doc: Use ``closer.lua`` script for downloading sources (#18179)
+- Doc: Improve installing from sources (#18194)
+- Improves installing from sources pages for all components (#18251)
Airflow 2.1.3, 2021-08-21
-------------------------
[airflow] 03/07: Advises the kernel to not cache log files
generated by Airflow (#18054)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v2-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f0f4e51e1353bf26eb0cca93a31d1820d11513db
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Sep 9 12:50:45 2021 +0200
Advises the kernel to not cache log files generated by Airflow (#18054)
* Advises the kernel to not cache log files generated by Airflow
Extends the standard python logging.FileHandler with advise to the
Kernel to not cache the file in PageCache when it is written. While
there is nothing wrong with such cache (it will be cleaned when memory
is needed), it causes ever-growing memory usage when scheduler is
running as it keeps on writing new log files and the files are not
rotated later on. This might lead to confusion for our users, who are
monitoring memory usage of Scheduler - without realising that it is
harmless and expected in this case.
Adding the advice to Kernel might help with not generating the cache
memory growth in the first place.
Closes: #14924
(cherry picked from commit 43f595fe1b8cd6f325d8535c03ee219edbf4a559)
---
airflow/utils/cli.py | 3 +-
airflow/utils/log/file_processor_handler.py | 3 +-
airflow/utils/log/file_task_handler.py | 3 +-
airflow/utils/log/non_caching_file_handler.py | 46 +++++++++++++++++++++++++++
4 files changed, 52 insertions(+), 3 deletions(-)
diff --git a/airflow/utils/cli.py b/airflow/utils/cli.py
index acdb291..6415676 100644
--- a/airflow/utils/cli.py
+++ b/airflow/utils/cli.py
@@ -34,6 +34,7 @@ from typing import TYPE_CHECKING, Callable, Optional, TypeVar, cast
from airflow import settings
from airflow.exceptions import AirflowException
from airflow.utils import cli_action_loggers
+from airflow.utils.log.non_caching_file_handler import NonCachingFileHandler
from airflow.utils.platform import getuser, is_terminal_support_colors
from airflow.utils.session import provide_session
@@ -244,7 +245,7 @@ def setup_locations(process, pid=None, stdout=None, stderr=None, log=None):
def setup_logging(filename):
"""Creates log file handler for daemon process"""
root = logging.getLogger()
- handler = logging.FileHandler(filename)
+ handler = NonCachingFileHandler(filename)
formatter = logging.Formatter(settings.SIMPLE_LOG_FORMAT)
handler.setFormatter(formatter)
root.addHandler(handler)
diff --git a/airflow/utils/log/file_processor_handler.py b/airflow/utils/log/file_processor_handler.py
index 7792faf..e1ce491 100644
--- a/airflow/utils/log/file_processor_handler.py
+++ b/airflow/utils/log/file_processor_handler.py
@@ -23,6 +23,7 @@ from pathlib import Path
from airflow import settings
from airflow.utils.helpers import parse_template_string
+from airflow.utils.log.non_caching_file_handler import NonCachingFileHandler
class FileProcessorHandler(logging.Handler):
@@ -54,7 +55,7 @@ class FileProcessorHandler(logging.Handler):
:param filename: filename in which the dag is located
"""
local_loc = self._init_file(filename)
- self.handler = logging.FileHandler(local_loc)
+ self.handler = NonCachingFileHandler(local_loc)
self.handler.setFormatter(self.formatter)
self.handler.setLevel(self.level)
diff --git a/airflow/utils/log/file_task_handler.py b/airflow/utils/log/file_task_handler.py
index e15e375..8d545ed 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -26,6 +26,7 @@ from itsdangerous import TimedJSONWebSignatureSerializer
from airflow.configuration import AirflowConfigException, conf
from airflow.utils.helpers import parse_template_string
+from airflow.utils.log.non_caching_file_handler import NonCachingFileHandler
if TYPE_CHECKING:
from airflow.models import TaskInstance
@@ -55,7 +56,7 @@ class FileTaskHandler(logging.Handler):
:param ti: task instance object
"""
local_loc = self._init_file(ti)
- self.handler = logging.FileHandler(local_loc, encoding='utf-8')
+ self.handler = NonCachingFileHandler(local_loc, encoding='utf-8')
if self.formatter:
self.handler.setFormatter(self.formatter)
self.handler.setLevel(self.level)
diff --git a/airflow/utils/log/non_caching_file_handler.py b/airflow/utils/log/non_caching_file_handler.py
new file mode 100644
index 0000000..caf9f4f
--- /dev/null
+++ b/airflow/utils/log/non_caching_file_handler.py
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import logging
+import os
+
+
+class NonCachingFileHandler(logging.FileHandler):
+ """
+ This is an extension of the python FileHandler that advises the Kernel to not cache the file
+ in PageCache when it is written. While there is nothing wrong with such cache (it will be cleaned
+ when memory is needed), it causes ever-growing memory usage when scheduler is running as it keeps
+ on writing new log files and the files are not rotated later on. This might lead to confusion
+ for our users, who are monitoring memory usage of Scheduler - without realising that it is
+ harmless and expected in this case.
+
+ See https://github.com/apache/airflow/issues/14924
+
+ Adding the advice to Kernel might help with not generating the cache memory growth in the first place.
+ """
+
+ def _open(self):
+ wrapper = super()._open()
+ try:
+ fd = wrapper.fileno()
+ os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
+ except Exception:
+ # in case either file descriptor cannot be retrieved or fadvise is not available
+ # we should simply return the wrapper retrieved by FileHandler's open method
+ # the advise to the kernel is just an advise and if we cannot give it, we won't
+ pass
+ return wrapper
[airflow] 01/07: Fix ``DagRunState`` enum query for ``MySQLdb``
driver (#17886)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v2-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 66dcbf429aee0316e206a1d6ded089580fc94ddf
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Mon Aug 30 19:05:08 2021 +0100
Fix ``DagRunState`` enum query for ``MySQLdb`` driver (#17886)
same as https://github.com/apache/airflow/pull/13278 but for `DagRunState` introduced in https://github.com/apache/airflow/pull/16854
closes https://github.com/apache/airflow/issues/17879
(cherry picked from commit a3f9c690aa80d12ff1d5c42eaaff4fced07b9429)
---
airflow/utils/state.py | 3 +++
tests/utils/test_state.py | 58 +++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 61 insertions(+)
diff --git a/airflow/utils/state.py b/airflow/utils/state.py
index e95b409..00e518e 100644
--- a/airflow/utils/state.py
+++ b/airflow/utils/state.py
@@ -65,6 +65,9 @@ class DagRunState(str, Enum):
SUCCESS = "success"
FAILED = "failed"
+ def __str__(self) -> str:
+ return self.value
+
class State:
"""
diff --git a/tests/utils/test_state.py b/tests/utils/test_state.py
new file mode 100644
index 0000000..88e4a0d
--- /dev/null
+++ b/tests/utils/test_state.py
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.models import DAG
+from airflow.models.dagrun import DagRun
+from airflow.utils.session import create_session
+from airflow.utils.state import DagRunState
+from airflow.utils.types import DagRunType
+from tests.models import DEFAULT_DATE
+
+
+def test_dagrun_state_enum_escape():
+ """
+ Make sure DagRunState.QUEUED is converted to string 'queued' when
+ referenced in DB query
+ """
+ with create_session() as session:
+ dag = DAG(dag_id='test_dagrun_state_enum_escape', start_date=DEFAULT_DATE)
+ dag.create_dagrun(
+ run_type=DagRunType.SCHEDULED,
+ state=DagRunState.QUEUED,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session,
+ )
+
+ query = session.query(DagRun.dag_id, DagRun.state, DagRun.run_type,).filter(
+ DagRun.dag_id == dag.dag_id,
+ # make sure enum value can be used in filter queries
+ DagRun.state == DagRunState.QUEUED,
+ )
+ assert str(query.statement.compile(compile_kwargs={"literal_binds": True})) == (
+ 'SELECT dag_run.dag_id, dag_run.state, dag_run.run_type \n'
+ 'FROM dag_run \n'
+ "WHERE dag_run.dag_id = 'test_dagrun_state_enum_escape' AND dag_run.state = 'queued'"
+ )
+
+ rows = query.all()
+ assert len(rows) == 1
+ assert rows[0].dag_id == dag.dag_id
+ # make sure value in db is stored as `queued`, not `DagRunType.QUEUED`
+ assert rows[0].state == 'queued'
+
+ session.rollback()
[airflow] 06/07: Improves installing from sources pages for all
components (#18251)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v2-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f0bccdc1ccd06631932986be85463bbdd619a4b2
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Sep 15 08:32:06 2021 +0200
Improves installing from sources pages for all components (#18251)
* Shorter menu sections for installation page
* Added "installing from sources" for Helm Chart
* Added Providers summary page for all provider packages
* Added scripts to verify PyPI packages with gpg/sha
(cherry picked from commit 67fddbf644eb6b11810addaf17d1be29c1bc39f6)
---
docs/apache-airflow-providers/index.rst | 10 +-
.../installing-from-sources.rst | 45 +++----
docs/apache-airflow/index.rst | 7 +-
docs/apache-airflow/installing-from-sources.rst | 63 +++++++---
docs/conf.py | 36 +++++-
docs/helm-chart/index.rst | 6 +
.../installing-helm-chart-from-sources.rst | 132 +++++++++++++++++++++
7 files changed, 252 insertions(+), 47 deletions(-)
diff --git a/docs/apache-airflow-providers/index.rst b/docs/apache-airflow-providers/index.rst
index f0c3285..0b0b527 100644
--- a/docs/apache-airflow-providers/index.rst
+++ b/docs/apache-airflow-providers/index.rst
@@ -407,10 +407,9 @@ provider package will keep its own dependencies, and while we expect those provi
backwards-compatible, particular versions of particular providers might introduce dependencies on
specific Airflow versions.
-Contents
---------
.. toctree::
+ :hidden:
:maxdepth: 2
Providers <self>
@@ -418,3 +417,10 @@ Contents
Operators and hooks <operators-and-hooks-ref/index>
Core Extensions <core-extensions/index>
Update community providers <howto/create-update-providers>
+
+.. toctree::
+ :maxdepth: 2
+ :hidden:
+ :caption: Resources
+
+ Installing from sources <installing-from-sources>
diff --git a/docs/apache-airflow/installing-from-sources.rst b/docs/apache-airflow-providers/installing-from-sources.rst
similarity index 57%
copy from docs/apache-airflow/installing-from-sources.rst
copy to docs/apache-airflow-providers/installing-from-sources.rst
index 4e64a2a..00e2a2d 100644
--- a/docs/apache-airflow/installing-from-sources.rst
+++ b/docs/apache-airflow-providers/installing-from-sources.rst
@@ -16,35 +16,28 @@
under the License.
-Installing from Sources
------------------------
+Installing Providers from Sources
+---------------------------------
.. contents:: :local:
-Installing Airflow from released sources and packages
-'''''''''''''''''''''''''''''''''''''''''''''''''''''
-
-You can also install Airflow using the official sources and packages. Those sources and packages
-released are the "official" sources of installation that you can use if you want to verify the
-origin of the packages and want to verify checksums and signatures of the packages.
-
-The packages are available at the
-`Official Apache Software Foundations Downloads page <https://downloads.apache.org/airflow/>`_
-
-The |version| downloads are available at:
+Released packages
+'''''''''''''''''
.. jinja:: official_download_page
- * `Sdist package <{{ closer_lua_url }}/apache-airflow-{{ airflow_version }}.tar.gz>`_ (`asc <{{ base_url }}/apache-airflow-{{ airflow_version }}.tar.gz.asc>`__, `sha512 <{{ base_url }}/apache-airflow-{{ airflow_version }}.tar.gz.sha512>`__)
- * `Wheel package <{{ closer_lua_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl>`_ (`asc <{{ base_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.asc>`__, `sha512 <{{ base_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.sha512>`__)
- * `Sources <{{ closer_lua_url }}/apache-airflow-{{ airflow_version }}-source.tar.gz>`_ (`asc <{{ base_url }}/apache_airflow-{{ airflow_version }}-source.tar.gz.asc>`__, `sha512 <{{ base_url }}/apache-airflow-{{ airflow_version }}-source.tar.gz.sha512>`__)
+ .. raw:: html
+
+ <ul style="column-count: 2;">
+ {% for provider in all_providers %}
+ <li><a href="/docs/{{ provider['package-name'] }}/latest/installing-providers-from-sources.html"><code>{{ provider.name }}</code></a></li>
+ {% endfor %}
+ </ul>
-If you want to install from the source code, you can download from the sources link above, it will contain
-a ``INSTALL`` file containing details on how you can build and install Airflow.
-Release integrity & Verification of releases
-''''''''''''''''''''''''''''''''''''''''''''
+Release integrity
+'''''''''''''''''
`PGP signatures KEYS <https://downloads.apache.org/airflow/KEYS>`_
@@ -74,26 +67,26 @@ distribution directory and follow the below guide.
.. code-block:: bash
- gpg --verify apache-airflow-********.asc apache-airflow-*********
+ gpg --verify apache-airflow-providers-********.asc apache-airflow-providers-*********
or
.. code-block:: bash
- pgpv apache-airflow-********.asc
+ pgpv apache-airflow-providers-********.asc
or
.. code-block:: bash
- pgp apache-airflow-********.asc
+ pgp apache-airflow-providers-********.asc
Example:
.. code-block:: console
:substitutions:
- $ gpg --verify apache-airflow-|version|-source.tar.gz.asc apache-airflow-|version|-source.tar.gz
+ $ gpg --verify apache-airflow-providers-airbyte-1.0.0-source.tar.gz.asc apache-airflow-providers-airbyte-1.0.0-source.tar.gz
gpg: Signature made Sat 11 Sep 12:49:54 2021 BST
gpg: using RSA key CDE15C6E4D3A8EC4ECF4BA4B6674E08AD7DE406F
gpg: issuer "kaxilnaik@apache.org"
@@ -112,7 +105,7 @@ For SHA512 sum check, download the relevant ``sha512`` and run the following:
.. code-block:: bash
- shasum -a 512 apache-airflow--******** | diff - apache-airflow--********.sha512
+ shasum -a 512 apache-airflow-providers-******** | diff - apache-airflow-providers-********.sha512
The ``SHASUM`` of the file should match the one provided in ``.sha512`` file.
@@ -121,4 +114,4 @@ Example:
.. code-block:: bash
:substitutions:
- shasum -a 512 apache-airflow-|version|-source.tar.gz | diff - apache-airflow-|version|-source.tar.gz.sha512
+ shasum -a 512 apache-airflow-providers-airbyte-1.0.0-source.tar.gz | diff - apache-airflow-providers-airbyte-1.0.0-source.tar.gz.sha512
diff --git a/docs/apache-airflow/index.rst b/docs/apache-airflow/index.rst
index 6002dd0..62787ad 100644
--- a/docs/apache-airflow/index.rst
+++ b/docs/apache-airflow/index.rst
@@ -78,7 +78,6 @@ unit of work and continuity.
license
start/index
installation
- installing-from-sources
upgrading-to-2
upgrade-check
tutorial
@@ -118,3 +117,9 @@ unit of work and continuity.
Configurations <configurations-ref>
Extra packages <extra-packages-ref>
Database Migrations <migrations-ref>
+
+.. toctree::
+ :hidden:
+ :caption: Resources
+
+ Installing from sources <installing-from-sources>
diff --git a/docs/apache-airflow/installing-from-sources.rst b/docs/apache-airflow/installing-from-sources.rst
index 4e64a2a..a730078 100644
--- a/docs/apache-airflow/installing-from-sources.rst
+++ b/docs/apache-airflow/installing-from-sources.rst
@@ -15,38 +15,45 @@
specific language governing permissions and limitations
under the License.
-
Installing from Sources
-----------------------
.. contents:: :local:
-Installing Airflow from released sources and packages
-'''''''''''''''''''''''''''''''''''''''''''''''''''''
+Released packages
+'''''''''''''''''
+
+.. jinja:: official_download_page
+
+ This page describes downloading and verifying ``Apache Airflow`` version
+ ``{{ airflow_version }}`` using officially released packages.
+ You can also install ``Apache Airflow`` - as most Python packages - via
+ `PyPI <https://pypi.org/project/apache-airflow/{{ airflow_version }}>`__ .
+ You can choose different version of Airflow by selecting different version from the drop-down at
+ the top-left of the page.
-You can also install Airflow using the official sources and packages. Those sources and packages
-released are the "official" sources of installation that you can use if you want to verify the
-origin of the packages and want to verify checksums and signatures of the packages.
+The ``source``, ``sdist`` and ``whl`` packages released are the "official" sources of installation that you
+can use if you want to verify the origin of the packages and want to verify checksums and signatures of
+the packages. The packages are available via the
+`Official Apache Software Foundations Mirrors <http://ws.apache.org/mirrors.cgi>`_
-The packages are available at the
-`Official Apache Software Foundations Downloads page <https://downloads.apache.org/airflow/>`_
The |version| downloads are available at:
.. jinja:: official_download_page
- * `Sdist package <{{ closer_lua_url }}/apache-airflow-{{ airflow_version }}.tar.gz>`_ (`asc <{{ base_url }}/apache-airflow-{{ airflow_version }}.tar.gz.asc>`__, `sha512 <{{ base_url }}/apache-airflow-{{ airflow_version }}.tar.gz.sha512>`__)
- * `Wheel package <{{ closer_lua_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl>`_ (`asc <{{ base_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.asc>`__, `sha512 <{{ base_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.sha512>`__)
- * `Sources <{{ closer_lua_url }}/apache-airflow-{{ airflow_version }}-source.tar.gz>`_ (`asc <{{ base_url }}/apache_airflow-{{ airflow_version }}-source.tar.gz.asc>`__, `sha512 <{{ base_url }}/apache-airflow-{{ airflow_version }}-source.tar.gz.sha512>`__)
+ * `Sources package <{{ closer_lua_url }}/apache-airflow-{{ airflow_version }}-source.tar.gz>`__ (`asc <{{ base_url }}/apache_airflow-{{ airflow_version }}-source.tar.gz.asc>`__, `sha512 <{{ base_url }}/apache-airflow-{{ airflow_version }}-source.tar.gz.sha512>`__)
+ * `Sdist package <{{ closer_lua_url }}/apache-airflow-{{ airflow_version }}.tar.gz>`__ (`asc <{{ base_url }}/apache-airflow-{{ airflow_version }}.tar.gz.asc>`__, `sha512 <{{ base_url }}/apache-airflow-{{ airflow_version }}.tar.gz.sha512>`__)
+ * `Whl package <{{ closer_lua_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl>`__ (`asc <{{ base_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.asc>`__, `sha512 <{{ base_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.sha512>`__)
If you want to install from the source code, you can download from the sources link above, it will contain
a ``INSTALL`` file containing details on how you can build and install Airflow.
-Release integrity & Verification of releases
-''''''''''''''''''''''''''''''''''''''''''''
+Release integrity
+'''''''''''''''''
-`PGP signatures KEYS <https://downloads.apache.org/airflow/KEYS>`_
+`PGP signatures KEYS <https://downloads.apache.org/airflow/KEYS>`__
It is essential that you verify the integrity of the downloaded files using the PGP or SHA signatures.
The PGP signatures can be verified using GPG or PGP. Please download the KEYS as well as the asc
@@ -122,3 +129,31 @@ Example:
:substitutions:
shasum -a 512 apache-airflow-|version|-source.tar.gz | diff - apache-airflow-|version|-source.tar.gz.sha512
+
+
+Verifying PyPI releases
+'''''''''''''''''''''''
+
+You can verify the Airflow ``.whl`` packages from PyPI by locally downloading the package and signature
+and SHA sum files with the script below:
+
+
+.. jinja:: official_download_page
+
+ .. code-block:: bash
+
+ #!/bin/bash
+ AIRFLOW_VERSION="{{ airflow_version }}"
+ airflow_download_dir="$(mktemp -d)"
+ pip download --no-deps "apache-airflow==${AIRFLOW_VERSION}" --dest "${airflow_download_dir}"
+ curl "https://downloads.apache.org/airflow/${AIRFLOW_VERSION}/apache_airflow-${AIRFLOW_VERSION}-py3-none-any.whl.asc" \
+ -L -o "${airflow_download_dir}/apache_airflow-${AIRFLOW_VERSION}-py3-none-any.whl.asc"
+ curl "https://downloads.apache.org/airflow/${AIRFLOW_VERSION}/apache_airflow-${AIRFLOW_VERSION}-py3-none-any.whl.sha512" \
+ -L -o "${airflow_download_dir}/apache_airflow-${AIRFLOW_VERSION}-py3-none-any.whl.sha512"
+ echo
+ echo "Please verify files downloaded to ${airflow_download_dir}"
+ ls -la "${airflow_download_dir}"
+ echo
+
+Once you verify the files following the instructions from previous chapter you can remove the temporary
+folder created.
diff --git a/docs/conf.py b/docs/conf.py
index 4cee315..c3521ae 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -75,6 +75,12 @@ elif PACKAGE_NAME.startswith('apache-airflow-providers-'):
raise Exception(f"Could not find provider.yaml file for package: {PACKAGE_NAME}")
PACKAGE_DIR = CURRENT_PROVIDER['package-dir']
PACKAGE_VERSION = CURRENT_PROVIDER['versions'][0]
+elif PACKAGE_NAME == 'apache-airflow-providers':
+ from provider_yaml_utils import load_package_data
+
+ PACKAGE_DIR = os.path.join(ROOT_DIR, 'airflow', 'providers')
+ PACKAGE_VERSION = 'devel'
+ ALL_PROVIDER_YAMLS = load_package_data()
elif PACKAGE_NAME == 'helm-chart':
PACKAGE_DIR = os.path.join(ROOT_DIR, 'chart')
PACKAGE_VERSION = 'devel' # TODO do we care? probably
@@ -149,6 +155,7 @@ if PACKAGE_NAME == 'apache-airflow':
if PACKAGE_NAME == "apache-airflow-providers":
extensions.extend(
[
+ 'sphinxcontrib.jinja',
'operators_and_hooks_ref',
'providers_packages_ref',
]
@@ -386,9 +393,22 @@ elif PACKAGE_NAME.startswith('apache-airflow-providers-'):
return yaml.load(f, SafeLoader)
config = _load_config()
- if config:
- jinja_contexts = {'config_ctx': {"configs": config}}
- extensions.append('sphinxcontrib.jinja')
+ jinja_contexts = {
+ 'config_ctx': {"configs": config},
+ 'official_download_page': {
+ 'base_url': 'https://downloads.apache.org/airflow/providers',
+ 'closer_lua_url': 'https://www.apache.org/dyn/closer.lua/airflow/providers',
+ 'package_name': PACKAGE_NAME,
+ 'package_name_underscores': PACKAGE_NAME.replace('-', '_'),
+ 'package_version': PACKAGE_VERSION,
+ },
+ }
+elif PACKAGE_NAME == 'apache-airflow-providers':
+ jinja_contexts = {
+ 'official_download_page': {
+ 'all_providers': ALL_PROVIDER_YAMLS,
+ },
+ }
elif PACKAGE_NAME == 'helm-chart':
def _str_representer(dumper, data):
@@ -468,7 +488,15 @@ elif PACKAGE_NAME == 'helm-chart':
if sections:
raise ValueError(f"Found section(s) which were not in `section_order`: {list(sections.keys())}")
- jinja_contexts = {"params_ctx": {"sections": ordered_sections}}
+ jinja_contexts = {
+ "params_ctx": {"sections": ordered_sections},
+ 'official_download_page': {
+ 'base_url': 'https://downloads.apache.org/airflow/helm-chart',
+ 'closer_lua_url': 'https://www.apache.org/dyn/closer.lua/airflow/helm-chart',
+ 'package_name': PACKAGE_NAME,
+ 'package_version': PACKAGE_VERSION,
+ },
+ }
# -- Options for sphinx.ext.autodoc --------------------------------------------
diff --git a/docs/helm-chart/index.rst b/docs/helm-chart/index.rst
index 6b809fc..d43a70d 100644
--- a/docs/helm-chart/index.rst
+++ b/docs/helm-chart/index.rst
@@ -39,6 +39,12 @@ Helm Chart for Apache Airflow
Parameters <parameters-ref>
+.. toctree::
+ :hidden:
+ :caption: Resources
+
+ Installing from sources<installing-helm-chart-from-sources>
+
This chart will bootstrap an `Airflow <https://airflow.apache.org>`__
deployment on a `Kubernetes <http://kubernetes.io>`__ cluster using the
diff --git a/docs/helm-chart/installing-helm-chart-from-sources.rst b/docs/helm-chart/installing-helm-chart-from-sources.rst
new file mode 100644
index 0000000..c5bef74
--- /dev/null
+++ b/docs/helm-chart/installing-helm-chart-from-sources.rst
@@ -0,0 +1,132 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Installing Helm Chart from sources
+----------------------------------
+
+.. contents:: :local:
+
+
+Released packages
+'''''''''''''''''
+
+.. jinja:: official_download_page
+
+ This page describes downloading and verifying ``Apache Airflow Official Helm Chart`` version
+ ``{{ package_version}}`` using officially released source packages. You can also install the chart
+ directly from the ``airflow.apache.org`` repo as described in
+ `Installing the chart <index#installing-the-chart>`_.
+ You can choose different version of the chart by selecting different version from the drop-down at
+ the top-left of the page.
+
+
+The sources and packages released are the "official" sources of installation that you can use if
+you want to verify the origin of the packages and want to verify checksums and signatures of the packages.
+The packages are available via the
+`Official Apache Software Foundations Mirrors <http://ws.apache.org/mirrors.cgi>`_
+
+The downloads are available at:
+
+.. jinja:: official_download_page
+
+ * `Sources package <{{ closer_lua_url }}/{{ package_version }}/airflow-chart-{{ package_version }}-source.tar.gz>`__ (`asc <{{ base_url }}/{{ package_version }}/airflow-chart-{{ package_version }}-source.tar.gz.asc>`__, `sha512 <{{ base_url }}/{{ package_version }}/airflow-chart-{{ package_version }}-source.tar.gz.sha512>`__)
+ * `Installable package <{{ closer_lua_url }}/{{ package_version }}/airflow-chart-{{ package_version }}.tar.gz>`__ (`asc <{{ base_url }}/{{ package_version }}/airflow-chart-{{ package_version }}.tar.gz.asc>`__, `sha512 <{{ base_url }}/{{ package_version }}/airflow-chart-{{ package_version }}.tar.gz.sha512>`__)
+
+If you want to install from the source code, you can download from the sources link above, it will contain
+a ``INSTALL`` file containing details on how you can build and install the chart.
+
+Release integrity
+'''''''''''''''''
+
+`PGP signatures KEYS <https://downloads.apache.org/airflow/KEYS>`_
+
+It is essential that you verify the integrity of the downloaded files using the PGP or SHA signatures.
+The PGP signatures can be verified using GPG or PGP. Please download the KEYS as well as the asc
+signature files for relevant distribution. It is recommended to get these files from the
+main distribution directory and not from the mirrors.
+
+.. code-block:: bash
+
+ gpg -i KEYS
+
+or
+
+.. code-block:: bash
+
+ pgpk -a KEYS
+
+or
+
+.. code-block:: bash
+
+ pgp -ka KEYS
+
+To verify the binaries/sources you can download the relevant asc files for it from main
+distribution directory and follow the below guide.
+
+.. code-block:: bash
+
+ gpg --verify airflow-chart-********.asc airflow-chart-*********
+
+or
+
+.. code-block:: bash
+
+ pgpv airflow-chart-********.asc
+
+or
+
+.. code-block:: bash
+
+ pgp airflow-chart-********.asc
+
+Example:
+
+.. jinja:: official_download_page
+
+ .. code-block:: console
+ :substitutions:
+
+ $ gpg --verify airflow-chart-{{ package_version }}.tar.gz.asc airflow-chart-{{ package_version }}.tar.gz
+ gpg: Signature made Sat 11 Sep 12:49:54 2021 BST
+ gpg: using RSA key CDE15C6E4D3A8EC4ECF4BA4B6674E08AD7DE406F
+ gpg: issuer "kaxilnaik@apache.org"
+ gpg: Good signature from "Kaxil Naik <ka...@apache.org>" [unknown]
+ gpg: aka "Kaxil Naik <ka...@gmail.com>" [unknown]
+ gpg: WARNING: The key's User ID is not certified with a trusted signature!
+ gpg: There is no indication that the signature belongs to the owner.
+ Primary key fingerprint: CDE1 5C6E 4D3A 8EC4 ECF4 BA4B 6674 E08A D7DE 406F
+
+ The "Good signature from ..." is indication that the signatures are correct.
+ Do not worry about the "not certified with a trusted signature" warning. Most of the certificates used
+ by release managers are self signed, that's why you get this warning. By importing the server in the
+ previous step and importing it via ID from ``KEYS`` page, you know that this is a valid Key already.
+
+ For SHA512 sum check, download the relevant ``sha512`` and run the following:
+
+ .. code-block:: bash
+
+ shasum -a 512 airflow-chart-******** | diff - airflow-chart-********.sha512
+
+ The ``SHASUM`` of the file should match the one provided in ``.sha512`` file.
+
+ Example:
+
+ .. code-block:: bash
+ :substitutions:
+
+ shasum -a 512 airflow-chart-{{ package_version }}.tar.gz | diff - airflow-chart-{{ package_version }}.tar.gz.sha512
[airflow] 04/07: Doc: Improve installing from sources (#18194)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v2-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit abc328040e8018eb5245ac89c7f44f20441374ac
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Sun Sep 12 21:37:42 2021 +0100
Doc: Improve installing from sources (#18194)
Based on https://github.com/apache/airflow/pull/18187/#discussion_r706861500
(cherry picked from commit ec79da0118048d5d80ec3126a5d8fe1452522534)
---
docs/apache-airflow/installing-from-sources.rst | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/docs/apache-airflow/installing-from-sources.rst b/docs/apache-airflow/installing-from-sources.rst
index d555910..307d1c7 100644
--- a/docs/apache-airflow/installing-from-sources.rst
+++ b/docs/apache-airflow/installing-from-sources.rst
@@ -55,11 +55,15 @@ main distribution directory and not from the mirrors.
gpg -i KEYS
- or
+or
+
+.. code-block:: bash
pgpk -a KEYS
- or
+or
+
+.. code-block:: bash
pgp -ka KEYS
@@ -70,11 +74,15 @@ distribution directory and follow the below guide.
gpg --verify apache-airflow-********.asc apache-airflow-*********
- or
+or
+
+.. code-block:: bash
pgpv apache-airflow-********.asc
- or
+or
+
+.. code-block:: bash
pgp apache-airflow-********.asc
[airflow] 02/07: Sort adopted tasks in
_check_for_stalled_adopted_tasks method (#18208)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v2-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit bb8abef2c997067e4dc976fab8f1b45b1750e596
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Wed Sep 15 14:36:45 2021 +0100
Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)
This PR adds sorting in adopted_tasks_timeout to ensure we correctly
clear stalled adopted tasks
(cherry picked from commit 9a7243adb8ec4d3d9185bad74da22e861582ffbe)
---
airflow/executors/celery_executor.py | 4 +++-
tests/executors/test_celery_executor.py | 28 ++++++++++++++++++++++++++++
2 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/airflow/executors/celery_executor.py b/airflow/executors/celery_executor.py
index b2c5016..ec9052b 100644
--- a/airflow/executors/celery_executor.py
+++ b/airflow/executors/celery_executor.py
@@ -339,8 +339,10 @@ class CeleryExecutor(BaseExecutor):
"""
now = utcnow()
+ sorted_adopted_task_timeouts = sorted(self.adopted_task_timeouts.items(), key=lambda k: k[1])
+
timedout_keys = []
- for key, stalled_after in self.adopted_task_timeouts.items():
+ for key, stalled_after in sorted_adopted_task_timeouts:
if stalled_after > now:
# Since items are stored sorted, if we get to a stalled_after
# in the future then we can stop
diff --git a/tests/executors/test_celery_executor.py b/tests/executors/test_celery_executor.py
index 88ea95c..25773aa 100644
--- a/tests/executors/test_celery_executor.py
+++ b/tests/executors/test_celery_executor.py
@@ -381,6 +381,34 @@ class TestCeleryExecutor(unittest.TestCase):
assert executor.running == set()
assert executor.adopted_task_timeouts == {}
+ @pytest.mark.backend("mysql", "postgres")
+ def test_check_for_stalled_adopted_tasks_goes_in_ordered_fashion(self):
+ start_date = timezone.utcnow() - timedelta(days=2)
+ queued_dttm = timezone.utcnow() - timedelta(minutes=30)
+ queued_dttm_2 = timezone.utcnow() - timedelta(minutes=4)
+
+ try_number = 1
+
+ with DAG("test_check_for_stalled_adopted_tasks") as dag:
+ task_1 = BaseOperator(task_id="task_1", start_date=start_date)
+ task_2 = BaseOperator(task_id="task_2", start_date=start_date)
+
+ key_1 = TaskInstanceKey(dag.dag_id, task_1.task_id, "runid", try_number)
+ key_2 = TaskInstanceKey(dag.dag_id, task_2.task_id, "runid", try_number)
+
+ executor = celery_executor.CeleryExecutor()
+ executor.adopted_task_timeouts = {
+ key_2: queued_dttm_2 + executor.task_adoption_timeout,
+ key_1: queued_dttm + executor.task_adoption_timeout,
+ }
+ executor.running = {key_1, key_2}
+ executor.tasks = {key_1: AsyncResult("231"), key_2: AsyncResult("232")}
+ executor.sync()
+ assert executor.event_buffer == {key_1: (State.FAILED, None)}
+ assert executor.tasks == {key_2: AsyncResult('232')}
+ assert executor.running == {key_2}
+ assert executor.adopted_task_timeouts == {key_2: queued_dttm_2 + executor.task_adoption_timeout}
+
def test_operation_timeout_config():
assert celery_executor.OPERATION_TIMEOUT == 1
[airflow] 05/07: Doc: Use ``closer.lua`` script for downloading
sources (#18179)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v2-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 84f8b99de374edc58f4320d671540a096d8a2bd0
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Sun Sep 12 09:58:54 2021 +0100
Doc: Use ``closer.lua`` script for downloading sources (#18179)
- Follows first point of https://infra.apache.org/release-download-pages.html#download-page to use `https://www.apache.org/dyn/closer.lua/PROJECT/VERSION/SOURCE-RELEASE` for mirrors
- Fixes bug as the current version substitution does not work for Hyperlinks (open PR: https://github.com/adamtheturtle/sphinx-substitution-extensions/issues/178)
(cherry picked from commit 9d05b32de593d2d553cfc567e092941b3d30a8a7)
---
docs/apache-airflow/installing-from-sources.rst | 8 +++++---
docs/conf.py | 5 +++++
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/docs/apache-airflow/installing-from-sources.rst b/docs/apache-airflow/installing-from-sources.rst
index 307d1c7..4e64a2a 100644
--- a/docs/apache-airflow/installing-from-sources.rst
+++ b/docs/apache-airflow/installing-from-sources.rst
@@ -34,9 +34,11 @@ The packages are available at the
The |version| downloads are available at:
-* `Apache Airflow |version| sdist package <https://downloads.apache.org/airflow/|version|/apache-airflow-|version|.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/|version|/apache-airflow-|version|.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/|version|/apache-airflow-|version|.tar.gz.sha512>`__)
-* `Apache Airflow |version| wheel package <https://downloads.apache.org/airflow/|version|/apache_airflow-|version|-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/|version|/apache_airflow-|version|-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/|version|/apache_airflow-|version|-py3-none-any.whl.sha512>`__)
-* `Apache Airflow |version| sources <https://downloads.apache.org/airflow/|version|/apache-airflow-|version|-source.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/|version|/apache-airflow-|version|-source.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/|version|/apache-airflow-|version|-source.tar.gz.sha512>`__)
+.. jinja:: official_download_page
+
+ * `Sdist package <{{ closer_lua_url }}/apache-airflow-{{ airflow_version }}.tar.gz>`_ (`asc <{{ base_url }}/apache-airflow-{{ airflow_version }}.tar.gz.asc>`__, `sha512 <{{ base_url }}/apache-airflow-{{ airflow_version }}.tar.gz.sha512>`__)
+ * `Wheel package <{{ closer_lua_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl>`_ (`asc <{{ base_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.asc>`__, `sha512 <{{ base_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.sha512>`__)
+ * `Sources <{{ closer_lua_url }}/apache-airflow-{{ airflow_version }}-source.tar.gz>`_ (`asc <{{ base_url }}/apache_airflow-{{ airflow_version }}-source.tar.gz.asc>`__, `sha512 <{{ base_url }}/apache-airflow-{{ airflow_version }}-source.tar.gz.sha512>`__)
If you want to install from the source code, you can download from the sources link above, it will contain
a ``INSTALL`` file containing details on how you can build and install Airflow.
diff --git a/docs/conf.py b/docs/conf.py
index 0197e42..4cee315 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -368,6 +368,11 @@ if PACKAGE_NAME == 'apache-airflow':
'http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/'
)
},
+ 'official_download_page': {
+ 'base_url': f'https://downloads.apache.org/airflow/{PACKAGE_VERSION}',
+ 'closer_lua_url': f'https://www.apache.org/dyn/closer.lua/airflow/{PACKAGE_VERSION}',
+ 'airflow_version': PACKAGE_VERSION,
+ },
}
elif PACKAGE_NAME.startswith('apache-airflow-providers-'):