You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by je...@apache.org on 2022/02/16 01:07:17 UTC
[airflow] branch v2-2-test updated (a520845 -> c03b086)
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a change to branch v2-2-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from a520845 update tutorial_etl_dag notes (#21503)
new 045f6ce Fix slow DAG deletion due to missing ``dag_id`` index for job table (#20282)
new 2aec9d8 Reorder migrations to include bugfix in 2.2.4 (#21598)
new c03b086 Add note about Variable precedence with env vars (#21568)
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
airflow/jobs/base_job.py | 1 +
...> 587bdf053233_adding_index_for_dag_id_in_job.py} | 20 +++++++++++---------
docs/apache-airflow/howto/variable.rst | 3 ++-
docs/apache-airflow/migrations-ref.rst | 4 +++-
4 files changed, 17 insertions(+), 11 deletions(-)
copy airflow/migrations/versions/{561833c1c74b_add_password_column_to_user.py => 587bdf053233_adding_index_for_dag_id_in_job.py} (70%)
[airflow] 03/03: Add note about Variable precedence with env vars (#21568)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-2-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c03b086e488229857d6ca49a715f3120112ed3c8
Author: Madison Swain-Bowden <bo...@spu.edu>
AuthorDate: Tue Feb 15 13:56:00 2022 -0800
Add note about Variable precedence with env vars (#21568)
This PR updates some documentation regarding setting Airflow Variables using environment variables. Environment variables take precedence over variables defined in the UI/metastore based on this default search path list: https://github.dev/apache/airflow/blob/7864693e43c40fd8f0914c05f7e196a007d16d50/airflow/secrets/__init__.py#L29-L30
(cherry picked from commit 7a268cb3c9fc6bc03f2400c6632ff8dccf4e451e)
---
docs/apache-airflow/howto/variable.rst | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/docs/apache-airflow/howto/variable.rst b/docs/apache-airflow/howto/variable.rst
index 7cb9377..401dcb1 100644
--- a/docs/apache-airflow/howto/variable.rst
+++ b/docs/apache-airflow/howto/variable.rst
@@ -62,7 +62,8 @@ You can use them in your DAGs as:
Single underscores surround ``VAR``. This is in contrast with the way ``airflow.cfg``
parameters are stored, where double underscores surround the config section name.
Variables set using Environment Variables would not appear in the Airflow UI but you will
- be able to use them in your DAG file.
+ be able to use them in your DAG file. Variables set using Environment Variables will also
+ take precedence over variables defined in the Airflow UI.
Securing Variables
------------------
[airflow] 01/03: Fix slow DAG deletion due to missing ``dag_id`` index for job table (#20282)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-2-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 045f6ce79ae722c2f95d9980d664d1ff152f863f
Author: Kush <36...@users.noreply.github.com>
AuthorDate: Thu Dec 30 15:56:24 2021 +0530
Fix slow DAG deletion due to missing ``dag_id`` index for job table (#20282)
Fixes #20249
(cherry picked from commit ac9f29da200c208bb52d412186c5a1b936eb0b5a)
---
airflow/jobs/base_job.py | 1 +
.../587bdf053233_adding_index_for_dag_id_in_job.py | 43 ++++++++++++++++++++++
docs/apache-airflow/migrations-ref.rst | 4 +-
3 files changed, 47 insertions(+), 1 deletion(-)
diff --git a/airflow/jobs/base_job.py b/airflow/jobs/base_job.py
index 745f248..174e4d5 100644
--- a/airflow/jobs/base_job.py
+++ b/airflow/jobs/base_job.py
@@ -71,6 +71,7 @@ class BaseJob(Base, LoggingMixin):
__table_args__ = (
Index('job_type_heart', job_type, latest_heartbeat),
Index('idx_job_state_heartbeat', state, latest_heartbeat),
+ Index('idx_job_dag_id', dag_id),
)
task_instances_enqueued = relationship(
diff --git a/airflow/migrations/versions/587bdf053233_adding_index_for_dag_id_in_job.py b/airflow/migrations/versions/587bdf053233_adding_index_for_dag_id_in_job.py
new file mode 100644
index 0000000..3532fe9
--- /dev/null
+++ b/airflow/migrations/versions/587bdf053233_adding_index_for_dag_id_in_job.py
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""adding index for dag_id in job
+
+Revision ID: 587bdf053233
+Revises: f9da662e7089
+Create Date: 2021-12-14 10:20:12.482940
+
+"""
+
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = '587bdf053233'
+down_revision = 'f9da662e7089'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+ """Apply adding index for dag_id in job"""
+ op.create_index('idx_job_dag_id', 'job', ['dag_id'], unique=False)
+
+
+def downgrade():
+ """Unapply adding index for dag_id in job"""
+ op.drop_index('idx_job_dag_id', table_name='job')
diff --git a/docs/apache-airflow/migrations-ref.rst b/docs/apache-airflow/migrations-ref.rst
index 8dc1a55..0eac329 100644
--- a/docs/apache-airflow/migrations-ref.rst
+++ b/docs/apache-airflow/migrations-ref.rst
@@ -23,7 +23,9 @@ Here's the list of all the Database Migrations that are executed via when you ru
+--------------------------------+------------------+-----------------+---------------------------------------------------------------------------------------+
| Revision ID | Revises ID | Airflow Version | Description |
+--------------------------------+------------------+-----------------+---------------------------------------------------------------------------------------+
-| ``c381b21cb7e4`` (head) | ``be2bfac3da23`` | ``2.2.4`` | Create a ``session`` table to store web session data |
+| ``587bdf053233`` (head) | ``f9da662e7089`` | ``2.3.0`` | Add index for ``dag_id`` column in ``job`` table. |
++--------------------------------+------------------+-----------------+---------------------------------------------------------------------------------------+
+| ``c381b21cb7e4`` | ``be2bfac3da23`` | ``2.2.4`` | Create a ``session`` table to store web session data |
+--------------------------------+------------------+-----------------+---------------------------------------------------------------------------------------+
| ``be2bfac3da23`` | ``7b2661a43ba3`` | ``2.2.3`` | Add has_import_errors column to DagModel |
+--------------------------------+------------------+-----------------+---------------------------------------------------------------------------------------+
[airflow] 02/03: Reorder migrations to include bugfix in 2.2.4 (#21598)
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-2-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2aec9d86631621c2e76f202c9e5e66121d06cfef
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Tue Feb 15 16:38:56 2022 -0700
Reorder migrations to include bugfix in 2.2.4 (#21598)
(cherry picked from commit 005cef042bc4184c24ad03c1b4ee40cdbaf96cb5)
---
.../versions/587bdf053233_adding_index_for_dag_id_in_job.py | 4 ++--
docs/apache-airflow/migrations-ref.rst | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/airflow/migrations/versions/587bdf053233_adding_index_for_dag_id_in_job.py b/airflow/migrations/versions/587bdf053233_adding_index_for_dag_id_in_job.py
index 3532fe9..c643a62 100644
--- a/airflow/migrations/versions/587bdf053233_adding_index_for_dag_id_in_job.py
+++ b/airflow/migrations/versions/587bdf053233_adding_index_for_dag_id_in_job.py
@@ -19,7 +19,7 @@
"""adding index for dag_id in job
Revision ID: 587bdf053233
-Revises: f9da662e7089
+Revises: c381b21cb7e4
Create Date: 2021-12-14 10:20:12.482940
"""
@@ -28,7 +28,7 @@ from alembic import op
# revision identifiers, used by Alembic.
revision = '587bdf053233'
-down_revision = 'f9da662e7089'
+down_revision = 'c381b21cb7e4'
branch_labels = None
depends_on = None
diff --git a/docs/apache-airflow/migrations-ref.rst b/docs/apache-airflow/migrations-ref.rst
index 0eac329..cdaa447 100644
--- a/docs/apache-airflow/migrations-ref.rst
+++ b/docs/apache-airflow/migrations-ref.rst
@@ -23,7 +23,7 @@ Here's the list of all the Database Migrations that are executed via when you ru
+--------------------------------+------------------+-----------------+---------------------------------------------------------------------------------------+
| Revision ID | Revises ID | Airflow Version | Description |
+--------------------------------+------------------+-----------------+---------------------------------------------------------------------------------------+
-| ``587bdf053233`` (head) | ``f9da662e7089`` | ``2.3.0`` | Add index for ``dag_id`` column in ``job`` table. |
+| ``587bdf053233`` (head) | ``c381b21cb7e4`` | ``2.2.4`` | Add index for ``dag_id`` column in ``job`` table. |
+--------------------------------+------------------+-----------------+---------------------------------------------------------------------------------------+
| ``c381b21cb7e4`` | ``be2bfac3da23`` | ``2.2.4`` | Create a ``session`` table to store web session data |
+--------------------------------+------------------+-----------------+---------------------------------------------------------------------------------------+