You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2020/11/20 15:18:07 UTC
[airflow] branch v1-10-test updated (967d45a -> bbcbb71)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
discard 967d45a TimeSensor should respect DAG timezone (#9882)
discard 4f5e8da TimeSensor should respect the default_timezone config (#9699)
discard b5aa391 Show Generic Error for Charts & Query View in old UI (#12495)
omit eaf5491 Add 1.10.13 Changelog
omit 8c58b32 Sync FAB Permissions for all base views (#12162)
new 483386a Sync FAB Permissions for all base views (#12162)
new 21c08b8 Add 1.10.13 Changelog
new 0aeb51f Show Generic Error for Charts & Query View in old UI (#12495)
new e2fe0b7 TimeSensor should respect the default_timezone config (#9699)
new bbcbb71 TimeSensor should respect DAG timezone (#9882)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (967d45a)
\
N -- N -- N refs/heads/v1-10-test (bbcbb71)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
tests/core/test_core.py | 1 +
1 file changed, 1 insertion(+)
[airflow] 02/05: Add 1.10.13 Changelog
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 21c08b8642873623258b5a3142ec9dd5a96047a0
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Thu Nov 19 21:28:11 2020 +0000
Add 1.10.13 Changelog
---
CHANGELOG.txt | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 101 insertions(+)
diff --git a/CHANGELOG.txt b/CHANGELOG.txt
index 4fb12de..92695fb 100644
--- a/CHANGELOG.txt
+++ b/CHANGELOG.txt
@@ -1,3 +1,103 @@
+Airflow 1.10.13, 2020-11-24
+----------------------------
+
+New Features
+""""""""""""
+
+- Add "already checked" to failed pods in K8sPodOperator (#11368)
+- Pass SQLAlchemy engine options to FAB based UI (#11395)
+- [AIRFLOW-4438] Add Gzip compression to S3_hook (#8571)
+- Add permission "extra_links" for Viewer role and above (#10719)
+- Add generate_yaml command to easily test KubernetesExecutor before deploying pods (#10677)
+- Add Secrets backend for Microsoft Azure Key Vault (#10898)
+
+Bug Fixes
+"""""""""
+
+- SkipMixin: Handle empty branches (#11120)
+- [AIRFLOW-5274] dag loading duration metric name too long (#5890)
+- Handle no Dagrun in DagrunIdDep (#8389) (#11343)
+- Fix Kubernetes Executor logs for long dag names (#10942)
+- Add on_kill support for the KubernetesPodOperator (#10666)
+- KubernetesPodOperator template fix (#10963)
+- Fix displaying of add serialized_dag table migration
+- Fix Start Date tooltip on DAGs page (#10637)
+- URL encode execution date in the Last Run link (#10595)
+- Fixes issue with affinity backcompat in Airflow 1.10
+- Fix KubernetesExecutor import in views.py
+- Fix issues with Gantt View (#12419)
+- Fix Entrypoint and _CMD config variables (#12411)
+- Fix operator field update for SerializedBaseOperator (#10924)
+- Limited cryptography to < 3.2 for python 2.7
+- Install cattr on Python 3.7 - Fix docs build on RTD (#12045)
+- Limit version of marshmallow-sqlalchemy
+- Pin `kubernetes` to a max version of 11.0.0 (#11974)
+- Use snakebite-py3 for HDFS dependency for Python3 (#12340)
+- Removes snakebite kerberos dependency (#10865)
+- Fix failing dependencies for FAB and Celery (#10828)
+- Fix pod_mutation_hook for 1.10.13 (#10850)
+- Fix formatting of Host information
+- Fix Logout Google Auth issue in Non-RBAC UI (#11890)
+- Add missing imports to app.py (#10650)
+
+Improvements
+""""""""""""
+
+- Add XCom.deserialize_value to Airflow 1.10.13 (#12328)
+- Mount airflow.cfg to pod_template_file (#12311)
+- All k8s object must comply with JSON Schema (#12003)
+- Validate airflow chart values.yaml & values.schema.json (#11990)
+- Pod template file uses custom custom env variable (#11480)
+- Bump attrs and cattrs dependencies (#11969)
+- Bump attrs to > 20.0 (#11799)
+- [AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep (#4751)
+- Rename task with duplicate task_id
+- Manage Flask AppBuilder Tables using Alembic Migrations (#12352)
+- ``airflow test`` only works for tasks in 1.10, not whole dags (#11191)
+- Improve warning messaging for duplicate task_ids in a DAG (#11126)
+- Pins moto to 1.3.14 (#10986)
+- DbApiHook: Support kwargs in get_pandas_df (#9730)
+- Make grace_period_seconds option on K8sPodOperator (#10727)
+- Fix syntax error in Dockerfile 'maintainer' Label (#10899)
+- The entrypoints in Docker Image should be owned by Airflow (#10853)
+- Make dockerfiles Google Shell Guide Compliant (#10734)
+- clean-logs script for Dockerfile: trim logs before sleep (#10685)
+- When sending tasks to celery from a sub-process, reset signal handlers (#11278)
+- SkipMixin: Add missing session.commit() and test (#10421)
+- Webserver: Further Sanitize values passed to origin param (#12459)
+- Security upgrade lodash from 4.17.19 to 4.17.20 (#11095)
+- Log instead of raise an Error for unregistered OperatorLinks (#11959)
+- Mask Password in Log table when using the CLI (#11468)
+- [AIRFLOW-3607] Optimize dep checking when depends on past set and concurrency limit
+- Execute job cancel HTTPRequest in Dataproc Hook (#10361)
+- Use rst lexer to format airflow upgrade check output (#11259)
+- Remove deprecation warning from contrib/kubernetes/pod.py
+- adding body as templated field for CloudSqlImportOperator (#10510)
+- Deprecate importing Hooks from plugin-created module (#12133)
+- Deprecate adding Operators and Sensors via plugins (#12069)
+
+Doc only changes
+""""""""""""""""
+
+- [Doc] Correct description for macro task_instance_key_str (#11062)
+- Checks if all the libraries in setup.py are listed in installation.rst file (#12023)
+- Revise "Project Focus" copy (#12011)
+- Move Project focus and Principles higher in the README (#11973)
+- Remove archived link from README.md (#11945)
+- Update download url for Airflow Version (#11800)
+- Add Project URLs for PyPI page (#11801)
+- Move Backport Providers docs to our docsite (#11136)
+- Refactor rebase copy (#11030)
+- Add missing images for kubernetes executor docs (#11083)
+- Fix identation in executor_config example (#10467)
+- Enhanced the Kubernetes Executor doc (#10433)
+- Refactor content to a markdown table (#10863)
+- Rename "Beyond the Horizon" section and refactor content (#10802)
+- Refactor official source section to use bullets (#10801)
+- Add section for official source code (#10678)
+- Add redbubble link to Airflow merchandise (#10359)
+- README Doc: Link to Airflow directory in ASF Directory (#11137)
+
Airflow 1.10.12, 2020-08-25
----------------------------
@@ -36,6 +136,7 @@ Bug Fixes
- BugFix: K8s Executor Multinamespace mode is evaluated to true by default (#10410)
- Make KubernetesExecutor recognize kubernetes_labels (#10412)
- Fix broken Kubernetes PodRuntimeInfoEnv (#10478)
+- Sync FAB Permissions for all base views (#12162)
Improvements
""""""""""""
[airflow] 03/05: Show Generic Error for Charts & Query View in old
UI (#12495)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0aeb51fd5e68b3275481767756d7dbe974333b28
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Fri Nov 20 13:31:10 2020 +0000
Show Generic Error for Charts & Query View in old UI (#12495)
---
airflow/www/views.py | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/airflow/www/views.py b/airflow/www/views.py
index fdb674d..f11fb84 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -471,8 +471,9 @@ class Airflow(AirflowViewMixin, BaseView):
df = hook.get_pandas_df(
wwwutils.limit_sql(sql, CHART_LIMIT, conn_type=db.conn_type))
df = df.fillna(0)
- except Exception as e:
- payload['error'] += "SQL execution failed. Details: " + str(e)
+ except Exception:
+ log.exception("Chart SQL execution failed")
+ payload['error'] += "SQL execution failed. Contact your System Administrator for more details"
if csv:
return Response(
@@ -2398,8 +2399,9 @@ class QueryView(wwwutils.DataProfilingMixin, AirflowViewMixin, BaseView):
index=False,
na_rep='',
) if has_data else ''
- except Exception as e:
- flash(str(e), 'error')
+ except Exception:
+ log.exception("Query SQL execution failed")
+ flash("SQL execution failed. Contact your System Administrator for more details", "error")
error = True
if has_data and len(df) == QUERY_LIMIT:
[airflow] 05/05: TimeSensor should respect DAG timezone (#9882)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit bbcbb711d20d966fd62beb6f52118eabbe4a27e0
Author: zikun <33...@users.noreply.github.com>
AuthorDate: Tue Jul 21 00:19:08 2020 +0800
TimeSensor should respect DAG timezone (#9882)
(cherry picked from commit 9c518fe937f8dc5e35908be96bd075c4ff666755)
---
UPDATING.md | 6 ++---
airflow/sensors/time_sensor.py | 2 +-
tests/sensors/test_time_sensor.py | 52 +++++++++++++++++++++++++++++++++++++++
3 files changed, 56 insertions(+), 4 deletions(-)
diff --git a/UPDATING.md b/UPDATING.md
index faa75fd..e2285df 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -63,12 +63,12 @@ https://developers.google.com/style/inclusive-documentation
-->
## Airflow 1.10.13
-### TimeSensor will consider default_timezone setting.
+### TimeSensor is now timezone aware
Previously `TimeSensor` always compared the `target_time` with the current time in UTC.
-Now it will compare `target_time` with the current time in the timezone set by `default_timezone` under the `core`
-section of the config.
+Now it will compare `target_time` with the current time in the timezone of the DAG,
+defaulting to the `default_timezone` in the global config.
### Removed Kerberos support for HDFS hook
diff --git a/airflow/sensors/time_sensor.py b/airflow/sensors/time_sensor.py
index d26c32d..5c41c2c 100644
--- a/airflow/sensors/time_sensor.py
+++ b/airflow/sensors/time_sensor.py
@@ -37,4 +37,4 @@ class TimeSensor(BaseSensorOperator):
def poke(self, context):
self.log.info('Checking if the time (%s) has come', self.target_time)
- return timezone.make_naive(timezone.utcnow()).time() > self.target_time
+ return timezone.make_naive(timezone.utcnow(), self.dag.timezone).time() > self.target_time
diff --git a/tests/sensors/test_time_sensor.py b/tests/sensors/test_time_sensor.py
new file mode 100644
index 0000000..c08bdd1
--- /dev/null
+++ b/tests/sensors/test_time_sensor.py
@@ -0,0 +1,52 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from datetime import datetime, time
+
+import pendulum
+from parameterized import parameterized
+
+from airflow.models.dag import DAG
+from airflow.sensors.time_sensor import TimeSensor
+from airflow.utils import timezone
+from tests.compat import patch
+
+DEFAULT_TIMEZONE = "Asia/Singapore" # UTC+08:00
+DEFAULT_DATE_WO_TZ = datetime(2015, 1, 1)
+DEFAULT_DATE_WITH_TZ = datetime(
+ 2015, 1, 1, tzinfo=pendulum.tz.timezone(DEFAULT_TIMEZONE)
+)
+
+
+@patch(
+ "airflow.sensors.time_sensor.timezone.utcnow",
+ return_value=timezone.datetime(2020, 1, 1, 23, 0).replace(tzinfo=timezone.utc),
+)
+class TestTimeSensor:
+ @parameterized.expand(
+ [
+ ("UTC", DEFAULT_DATE_WO_TZ, True),
+ ("UTC", DEFAULT_DATE_WITH_TZ, False),
+ (DEFAULT_TIMEZONE, DEFAULT_DATE_WO_TZ, False),
+ ]
+ )
+ def test_timezone(self, mock_utcnow, default_timezone, start_date, expected):
+ with patch("airflow.settings.TIMEZONE", pendulum.timezone(default_timezone)):
+ dag = DAG("test", default_args={"start_date": start_date})
+ op = TimeSensor(task_id="test", target_time=time(10, 0), dag=dag)
+ assert op.poke(None) == expected
[airflow] 01/05: Sync FAB Permissions for all base views (#12162)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 483386af1abbaa5a2a122f0976aa5e2262e18d21
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Sat Nov 7 17:32:53 2020 +0000
Sync FAB Permissions for all base views (#12162)
If a user has set `[webserver] update_fab_perms = False` and runs `airflow sync-perm` command to sync all permissions, they will receive the following error:
```
webserver_1 | [2020-11-07 15:13:07,431] {decorators.py:113} WARNING - Access is Denied for: can_index on: Airflow
```
and if the user was created before and some perms were sync'd a user won't be able to find Security Menu & Configurations View
---
airflow/bin/cli.py | 1 +
tests/core/test_core.py | 2 ++
2 files changed, 3 insertions(+)
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 1fb3b88..37f70c4 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -1863,6 +1863,7 @@ def sync_perm(args): # noqa
appbuilder = cached_appbuilder()
print('Updating permission, view-menu for all existing roles')
appbuilder.sm.sync_roles()
+ appbuilder.add_permissions(update_perms=True)
print('Updating permission on all DAG views')
dags = DagBag(store_serialized_dags=settings.STORE_SERIALIZED_DAGS).dags.values()
for dag in dags:
diff --git a/tests/core/test_core.py b/tests/core/test_core.py
index b229d53..bf74945 100644
--- a/tests/core/test_core.py
+++ b/tests/core/test_core.py
@@ -1110,6 +1110,7 @@ class CliTests(unittest.TestCase):
DAG('no_access_control')
], dagbag_mock)
self.appbuilder.sm = mock.Mock()
+ self.appbuilder.add_permissions = mock.Mock()
args = self.parser.parse_args([
'sync_perm'
@@ -1127,6 +1128,7 @@ class CliTests(unittest.TestCase):
'no_access_control',
None,
)
+ self.appbuilder.add_permissions.assert_called_once_with(update_perms=True)
def expect_dagbag_contains(self, dags, dagbag_mock):
dagbag = mock.Mock()
[airflow] 04/05: TimeSensor should respect the default_timezone
config (#9699)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e2fe0b7a7101b24dd3354dc13f5dd1fdac9c0fec
Author: zikun <33...@users.noreply.github.com>
AuthorDate: Sun Jul 19 01:36:28 2020 +0800
TimeSensor should respect the default_timezone config (#9699)
(cherry picked from commit b34ba874452809495354f3012e0b1dcbf4209e09)
---
UPDATING.md | 7 +++++++
airflow/sensors/time_sensor.py | 2 +-
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/UPDATING.md b/UPDATING.md
index 2057491..faa75fd 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -63,6 +63,13 @@ https://developers.google.com/style/inclusive-documentation
-->
## Airflow 1.10.13
+### TimeSensor will consider default_timezone setting.
+
+Previously `TimeSensor` always compared the `target_time` with the current time in UTC.
+
+Now it will compare `target_time` with the current time in the timezone set by `default_timezone` under the `core`
+section of the config.
+
### Removed Kerberos support for HDFS hook
The HDFS hook's Kerberos support has been removed due to removed python-krbV dependency from PyPI
diff --git a/airflow/sensors/time_sensor.py b/airflow/sensors/time_sensor.py
index 0c39235..d26c32d 100644
--- a/airflow/sensors/time_sensor.py
+++ b/airflow/sensors/time_sensor.py
@@ -37,4 +37,4 @@ class TimeSensor(BaseSensorOperator):
def poke(self, context):
self.log.info('Checking if the time (%s) has come', self.target_time)
- return timezone.utcnow().time() > self.target_time
+ return timezone.make_naive(timezone.utcnow()).time() > self.target_time