You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2019/01/03 16:41:53 UTC
[GitHub] aliceabe closed pull request #4424: [AIRFLOW-3622] Add ability to
pass hive_conf to HiveToMysqlTransfer
aliceabe closed pull request #4424: [AIRFLOW-3622] Add ability to pass hive_conf to HiveToMysqlTransfer
URL: https://github.com/apache/incubator-airflow/pull/4424
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.codecov.yml b/.codecov.yml
deleted file mode 100644
index a496cb4ce0..0000000000
--- a/.codecov.yml
+++ /dev/null
@@ -1,14 +0,0 @@
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# keep default
diff --git a/.coveragerc b/.coveragerc
index 1d487e845c..3fa0521084 100644
--- a/.coveragerc
+++ b/.coveragerc
@@ -1,15 +1,20 @@
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
[report]
omit =
@@ -18,3 +23,4 @@ omit =
scripts/*
dev/*
airflow/migrations/*
+ airflow/www_rbac/node_modules/**
diff --git a/.editorconfig b/.editorconfig
index b4d5f0728d..3110c2c76b 100644
--- a/.editorconfig
+++ b/.editorconfig
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
root = true
[*]
diff --git a/.flake8 b/.flake8
new file mode 100644
index 0000000000..2723df1f10
--- /dev/null
+++ b/.flake8
@@ -0,0 +1,3 @@
+[flake8]
+max-line-length = 110
+ignore = E731
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 6103caa7b5..0a3e7016a7 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,25 +1,35 @@
Make sure you have checked _all_ steps below.
-### JIRA
-- [ ] My PR addresses the following [Airflow JIRA](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- - https://issues.apache.org/jira/browse/AIRFLOW-XXX
+### Jira
+- [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
+ - https://issues.apache.org/jira/browse/AIRFLOW-XXX
+ - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
-- [ ] Here are some details about my PR, including screenshots of any UI changes:
+- [ ] Here are some details about my PR, including screenshots of any UI changes:
### Tests
-- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
+- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
### Commits
-- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
- 1. Subject is separated from body by a blank line
- 2. Subject is limited to 50 characters
- 3. Subject does not end with a period
- 4. Subject uses the imperative mood ("add", not "adding")
- 5. Body wraps at 72 characters
- 6. Body explains "what" and "why", not "how"
-
-- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
+
+- [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
+ 1. Subject is separated from body by a blank line
+ 1. Subject is limited to 50 characters (not including Jira issue reference)
+ 1. Subject does not end with a period
+ 1. Subject uses the imperative mood ("add", not "adding")
+ 1. Body wraps at 72 characters
+ 1. Body explains "what" and "why", not "how"
+
+### Documentation
+
+- [ ] In case of new functionality, my PR adds documentation that describes how to use it.
+ - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
+ - All the public functions and the classes in the PR contain docstrings that explain what it does
+
+### Code Quality
+
+- [ ] Passes `flake8`
diff --git a/.github/stale.yml b/.github/stale.yml
new file mode 100644
index 0000000000..734dda6598
--- /dev/null
+++ b/.github/stale.yml
@@ -0,0 +1,55 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+
+# Number of days of inactivity before an Issue or Pull Request becomes stale
+daysUntilStale: 45
+
+# Number of days of inactivity before an Issue or Pull Request with the stale label is closed.
+# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale.
+daysUntilClose: 7
+
+# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable
+exemptLabels:
+ - pinned
+ - security
+ - "[Status] Maybe Later"
+
+# Set to true to ignore issues in a project (defaults to false)
+exemptProjects: false
+
+# Set to true to ignore issues in a milestone (defaults to false)
+exemptMilestones: false
+
+# Label to use when marking as stale
+staleLabel: stale
+
+# Comment to post when marking as stale. Set to `false` to disable
+markComment: >
+ This issue has been automatically marked as stale because it has not had
+ recent activity. It will be closed if no further activity occurs. Thank you
+ for your contributions.
+
+# Comment to post when removing the stale label.
+# unmarkComment: >
+# Your comment here.
+
+# Comment to post when closing a stale Issue or Pull Request.
+# closeComment: >
+# Your comment here.
+
+# Limit the number of actions per hour, from 1-30. Default is 30
+limitPerRun: 30
+
+# Limit to only `issues` or `pulls`
+# only: issues
+
+# Optionally, specify configuration settings that are specific to just 'issues' or 'pulls':
+# pulls:
+# daysUntilStale: 30
+# markComment: >
+# This pull request has been automatically marked as stale because it has not had
+# recent activity. It will be closed if no further activity occurs. Thank you
+# for your contributions.
+
+# issues:
+# exemptLabels:
+# - confirmed
diff --git a/.gitignore b/.gitignore
index 0a512df699..114236d467 100644
--- a/.gitignore
+++ b/.gitignore
@@ -14,11 +14,13 @@ unittests.db
airflow/git_version
airflow/www/static/coverage/
logs/
+airflow-webserver.pid
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
+.pytest_cache/
# C extensions
*.so
@@ -62,6 +64,7 @@ nosetests.xml
coverage.xml
*,cover
.hypothesis/
+.pytest_cache
# Translations
*.mo
@@ -128,3 +131,24 @@ ENV/
# Spark
rat-results.txt
+
+# Git stuff
+.gitattributes
+# Kubernetes generated templated files
+*.generated
+*.tar.gz
+scripts/ci/kubernetes/kube/.generated/airflow.yaml
+
+# Node & Webpack Stuff
+*.entry.js
+node_modules
+npm-debug.log*
+static/dist
+derby.log
+metastore_db
+
+# Airflow log files when airflow is run locally
+airflow-*.err
+airflow-*.out
+airflow-*.log
+airflow-*.pid
diff --git a/.rat-excludes b/.rat-excludes
index 25fe61e585..96e59ee43c 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -1,15 +1,21 @@
+# Note: these patterns are applied to single files or directories, not full paths
+# coverage/* will ignore any coverage dir, but airflow/www/static/coverage/* will match nothing
+
.gitignore
.gitattributes
+.airflowignore
.coverage
.coveragerc
.codecov.yml
+.eslintrc
+.eslintignore
+.flake8
.rat-excludes
requirements.txt
.*log
.travis.yml
.*pyc
-docs
-.*md
+.*lock
dist
build
airflow.egg-info
@@ -17,16 +23,23 @@ apache_airflow.egg-info
.idea
metastore_db
.*sql
+.*svg
.*csv
CHANGELOG.txt
.*zip
.*lock
+# Generated doc files
+.*html
+_build/*
+_static/*
+.buildinfo
+searchindex.js
+
# Apache Rat does not detect BSD-2 clause properly
# it is compatible according to http://www.apache.org/legal/resolved.html#category-a
kerberos_auth.py
airflow_api_auth_backend_kerberos_auth_py.html
licenses/*
-airflow/www/static/docs
parallel.js
underscore.js
jquery.dataTables.min.js
@@ -36,3 +49,13 @@ bootstrap-toggle.min.js
bootstrap-toggle.min.css
d3.v3.min.js
ace.js
+node_modules/*
+.*json
+coverage/*
+git_version
+flake8_diff.sh
+coverage.xml
+
+rat-results.txt
+apache-airflow-.*\+incubating-source.tar.gz.*
+apache-airflow-.*\+incubating-bin.tar.gz.*
diff --git a/.readthedocs.yml b/.readthedocs.yml
index ca5b56d9ef..dddded3636 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -1,20 +1,33 @@
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
python:
pip_install: true
extra_requirements:
+ - all_dbs
+ - databricks
- doc
- docker
+ - emr
- gcp_api
- - emr
+ - s3
+ - salesforce
+ - sendgrid
+ - ssh
+ - slack
+ - qds
diff --git a/.travis.yml b/.travis.yml
index dec9181fc6..934b0d749b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,103 +1,69 @@
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
sudo: true
dist: trusty
language: python
-jdk:
- - oraclejdk8
-services:
- - mysql
- - postgresql
- - rabbitmq
-addons:
- apt:
- packages:
- - slapd
- - ldap-utils
- - openssh-server
- - mysql-server-5.6
- - mysql-client-core-5.6
- - mysql-client-5.6
- - krb5-user
- - krb5-kdc
- - krb5-admin-server
- - oracle-java8-installer
- - python-selinux
- postgresql: "9.2"
-python:
- - "2.7"
- - "3.5"
env:
global:
+ - DOCKER_COMPOSE_VERSION=1.20.0
+ - SLUGIFY_USES_TEXT_UNIDECODE=yes
- TRAVIS_CACHE=$HOME/.travis_cache/
- - KRB5_CONFIG=/etc/krb5.conf
- - KRB5_KTNAME=/etc/airflow.keytab
- # Travis on google cloud engine has a global /etc/boto.cfg that
- # does not work with python 3
- - BOTO_CONFIG=/tmp/bogusvalue
matrix:
- - TOX_ENV=py27-backend_mysql
- - TOX_ENV=py27-backend_sqlite
- - TOX_ENV=py27-backend_postgres
- - TOX_ENV=py35-backend_mysql
- - TOX_ENV=py35-backend_sqlite
- - TOX_ENV=py35-backend_postgres
- TOX_ENV=flake8
- - TOX_ENV=py27-backend_postgres KUBERNETES_VERSION=v1.7.0
- - TOX_ENV=py27-backend_postgres KUBERNETES_VERSION=v1.8.0
-matrix:
- exclude:
- - python: "3.5"
- env: TOX_ENV=py27-backend_mysql
- - python: "3.5"
- env: TOX_ENV=py27-backend_sqlite
- - python: "3.5"
- env: TOX_ENV=py27-backend_postgres
- - python: "2.7"
- env: TOX_ENV=py35-backend_mysql
- - python: "2.7"
- env: TOX_ENV=py35-backend_sqlite
- - python: "2.7"
- env: TOX_ENV=py35-backend_postgres
- - python: "2.7"
- env: TOX_ENV=flake8
- - python: "3.5"
- env: TOX_ENV=py27-backend_postgres KUBERNETES_VERSION=v1.7.0
- - python: "3.5"
- env: TOX_ENV=py27-backend_postgres KUBERNETES_VERSION=v1.8.0
- allow_failures:
- - env: TOX_ENV=py27-backend_postgres KUBERNETES_VERSION=v1.7.0
- - env: TOX_ENV=py27-backend_postgres KUBERNETES_VERSION=v1.8.0
+ - TOX_ENV=py27-backend_mysql-env_docker
+ - TOX_ENV=py27-backend_sqlite-env_docker
+ - TOX_ENV=py27-backend_postgres-env_docker
+ - TOX_ENV=py35-backend_mysql-env_docker PYTHON_VERSION=3
+ - TOX_ENV=py35-backend_sqlite-env_docker PYTHON_VERSION=3
+ - TOX_ENV=py35-backend_postgres-env_docker PYTHON_VERSION=3
+ - TOX_ENV=py27-backend_postgres-env_kubernetes KUBERNETES_VERSION=v1.9.0
+ - TOX_ENV=py35-backend_postgres-env_kubernetes KUBERNETES_VERSION=v1.10.0 PYTHON_VERSION=3
+
cache:
directories:
- $HOME/.wheelhouse/
+ - $HOME/.cache/pip
- $HOME/.travis_cache/
before_install:
- - yes | ssh-keygen -t rsa -C your_email@youremail.com -P '' -f ~/.ssh/id_rsa
- - cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
- - ln -s ~/.ssh/authorized_keys ~/.ssh/authorized_keys2
- - chmod 600 ~/.ssh/*
- - jdk_switcher use oraclejdk8
+ # Required for K8s v1.10.x. See
+ # https://github.com/kubernetes/kubernetes/issues/61058#issuecomment-372764783
+ - if [ ! -z "$KUBERNETES_VERSION" ]; then sudo mount --make-shared / && sudo service docker restart; fi
install:
+ # Use recent docker-compose version
+ - sudo rm /usr/local/bin/docker-compose
+ - curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
+ - chmod +x docker-compose
+ - sudo mv docker-compose /usr/local/bin
- pip install --upgrade pip
- - pip install tox
- - pip install codecov
-before_script:
- - mysql -e 'drop database if exists airflow; create database airflow' -u root
- - psql -c 'create database airflow;' -U postgres
- - export PATH=${PATH}:/tmp/hive/bin
script:
- - ./scripts/ci/travis_script.sh
-after_success:
- - codecov
+ - if [ -z "$KUBERNETES_VERSION" ]; then docker-compose --log-level ERROR -f scripts/ci/docker-compose.yml run airflow-testing /app/scripts/ci/run-ci.sh; fi
+ - if [ ! -z "$KUBERNETES_VERSION" ]; then
+ ./scripts/ci/kubernetes/minikube/stop_minikube.sh &&
+ ./scripts/ci/kubernetes/setup_kubernetes.sh &&
+ ./scripts/ci/kubernetes/kube/deploy.sh -d persistent_mode &&
+ MINIKUBE_IP=$(minikube ip) docker-compose --log-level ERROR -f scripts/ci/docker-compose.yml -f scripts/ci/docker-compose-kubernetes.yml run airflow-testing /app/scripts/ci/run-ci.sh;
+ fi
+ - if [ ! -z "$KUBERNETES_VERSION" ]; then
+ ./scripts/ci/kubernetes/minikube/stop_minikube.sh &&
+ ./scripts/ci/kubernetes/setup_kubernetes.sh &&
+ ./scripts/ci/kubernetes/kube/deploy.sh -d git_mode &&
+ MINIKUBE_IP=$(minikube ip) docker-compose --log-level ERROR -f scripts/ci/docker-compose.yml -f scripts/ci/docker-compose-kubernetes.yml run airflow-testing /app/scripts/ci/run-ci.sh;
+ fi
+before_cache:
+ - sudo chown -R travis:travis $HOME/.cache/pip $HOME/.wheelhouse/
diff --git a/CHANGELOG.txt b/CHANGELOG.txt
index fa4e6547a7..98a1103792 100644
--- a/CHANGELOG.txt
+++ b/CHANGELOG.txt
@@ -1,3 +1,918 @@
+AIRFLOW 1.10.1, 2018-11-13
+
+New features:
+
+[AIRFLOW-2524] Airflow integration with AWS Sagemaker
+[AIRFLOW-2657] Add ability to delete DAG from web ui
+[AIRFLOW-2780] Adds IMAP Hook to interact with a mail server
+[AIRFLOW-2794] Add delete support for Azure blob
+[AIRFLOW-2912] Add operators for Google Cloud Functions
+[AIRFLOW-2974] Add Start/Restart/Terminate methods Databricks Hook
+[AIRFLOW-2989] No Parameter to change bootDiskType for DataprocClusterCreateOperator
+[AIRFLOW-3078] Basic operators for Google Compute Engine
+[AIRFLOW-3147] Update Flask-AppBuilder version
+[AIRFLOW-3231] Basic operators for Google Cloud SQL (deploy / patch / delete)
+[AIRFLOW-3276] Google Cloud SQL database create / patch / delete operators
+
+Improvements:
+
+[AIRFLOW-393] Add progress callbacks for FTP downloads
+[AIRFLOW-520] Show Airflow version on web page
+[AIRFLOW-843] Exceptions now available in context during on_failure_callback
+[AIRFLOW-2476] Update tabulate dependency to v0.8.2
+[AIRFLOW-2592] Bump Bleach dependency
+[AIRFLOW-2622] Add "confirm=False" option to SFTPOperator
+[AIRFLOW-2662] support affinity & nodeSelector policies for kubernetes executor/operator
+[AIRFLOW-2709] Improve error handling in Databricks hook
+[AIRFLOW-2723] Update lxml dependency to >= 4.0.
+[AIRFLOW-2763] No precheck mechanism in place during worker initialisation for the connection to metadata database
+[AIRFLOW-2789] Add ability to create single node cluster to DataprocClusterCreateOperator
+[AIRFLOW-2797] Add ability to create Google Dataproc cluster with custom image
+[AIRFLOW-2854] kubernetes_pod_operator add more configuration items
+[AIRFLOW-2855] Need to Check Validity of Cron Expression When Process DAG File/Zip File
+[AIRFLOW-2904] Clean an unnecessary line in airflow/executors/celery_executor.py
+[AIRFLOW-2921] A trivial incorrectness in CeleryExecutor()
+[AIRFLOW-2922] Potential deal-lock bug in CeleryExecutor()
+[AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file
+[AIRFLOW-2949] Syntax Highlight for Single Quote
+[AIRFLOW-2951] dag_run end_date Null after a dag is finished
+[AIRFLOW-2956] Kubernetes tolerations for pod operator
+[AIRFLOW-2997] Support for clustered tables in Bigquery hooks/operators
+[AIRFLOW-3006] Fix error when schedule_interval="None"
+[AIRFLOW-3008] Move Kubernetes related example DAGs to contrib/example_dags
+[AIRFLOW-3025] Allow to specify dns and dns-search parameters for DockerOperator
+[AIRFLOW-3067] (www_rbac) Flask flash messages are not displayed properly (no background color)
+[AIRFLOW-3069] Decode output of S3 file transform operator
+[AIRFLOW-3072] Assign permission get_logs_with_metadata to viewer role
+[AIRFLOW-3090] INFO logs are too verbose
+[AIRFLOW-3103] Update Flask-Login
+[AIRFLOW-3112] Align SFTP hook with SSH hook
+[AIRFLOW-3119] Enable loglevel on celery worker and inherit from airflow.cfg
+[AIRFLOW-3137] Make ProxyFix middleware optional
+[AIRFLOW-3173] Add _cmd options for more password config options
+[AIRFLOW-3177] Change scheduler_heartbeat metric from gauge to counter
+[AIRFLOW-3193] Pin docker requirement version to v3
+[AIRFLOW-3195] Druid Hook: Log ingestion spec and task id
+[AIRFLOW-3197] EMR Hook is missing some parameters to valid on the AWS API
+[AIRFLOW-3232] Make documentation for GCF Functions operator more readable
+[AIRFLOW-3262] Can't get log containing Response when using SimpleHttpOperator
+[AIRFLOW-3265] Add support for "unix_socket" in connection extra for Mysql Hook
+
+Doc-only changes:
+
+[AIRFLOW-1441] Tutorial Inconsistencies Between Example Pipeline Definition and Recap
+[AIRFLOW-2682] Add how-to guide(s) for how to use basic operators like BashOperator and PythonOperator
+[AIRFLOW-3104] .airflowignore feature is not mentioned at all in documentation
+[AIRFLOW-3237] Refactor example DAGs
+[AIRFLOW-3187] Update airflow.gif file with a slower version
+[AIRFLOW-3159] Update Airflow documentation on GCP Logging
+[AIRFLOW-3030] Command Line docs incorrect subdir
+[AIRFLOW-2990] Docstrings for Hooks/Operators are in incorrect format
+[AIRFLOW-3127] Celery SSL Documentation is out-dated
+[AIRFLOW-2779] Add license headers to doc files
+[AIRFLOW-2779] Add project version to license
+
+Bug fixes:
+
+[AIRFLOW-839] docker_operator.py attempts to log status key without first checking existence
+[AIRFLOW-1104] Concurrency check in scheduler should count queued tasks as well as running
+[AIRFLOW-1163] Add support for x-forwarded-* headers to support access behind AWS ELB
+[AIRFLOW-1195] Cleared tasks in SubDagOperator do not trigger Parent dag_runs
+[AIRFLOW-1508] Skipped state not part of State.task_states
+[AIRFLOW-1762] Use key_file in SSHHook.create_tunnel()
+[AIRFLOW-1837] Differing start_dates on tasks not respected by scheduler.
+[AIRFLOW-1874] Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators
+[AIRFLOW-1917] print() from python operators end up with extra new line
+[AIRFLOW-1970] Database cannot be initialized if an invalid fernet key is provided
+[AIRFLOW-2145] Deadlock after clearing a running task
+[AIRFLOW-2216] Cannot specify a profile for AWS Hook to load with s3 config file
+[AIRFLOW-2574] initdb fails when mysql password contains percent sign
+[AIRFLOW-2707] Error accessing log files from web UI
+[AIRFLOW-2716] Replace new Python 3.7 keywords
+[AIRFLOW-2744] RBAC app doesn't integrate plugins (blueprints etc)
+[AIRFLOW-2772] BigQuery hook does not allow specifying both the partition field name and table name at the same time
+[AIRFLOW-2778] Bad Import in collect_dag in DagBag
+[AIRFLOW-2786] Variables view fails to render if a variable has an empty key
+[AIRFLOW-2799] Filtering UI objects by datetime is broken
+[AIRFLOW-2800] Remove airflow/ low-hanging linting errors
+[AIRFLOW-2825] S3ToHiveTransfer operator may not may able to handle GZIP file with uppercase ext in S3
+[AIRFLOW-2848] dag_id is missing in metadata table "job" for LocalTaskJob
+[AIRFLOW-2860] DruidHook: time variable is not updated correctly when checking for timeout
+[AIRFLOW-2865] Race condition between on_success_callback and LocalTaskJob's cleanup
+[AIRFLOW-2893] Stuck dataflow job due to jobName mismatch.
+[AIRFLOW-2895] Prevent scheduler from spamming heartbeats/logs
+[AIRFLOW-2900] Code not visible for Packaged DAGs
+[AIRFLOW-2905] Switch to regional dataflow job service.
+[AIRFLOW-2907] Sendgrid - Attachments - ERROR - Object of type 'bytes' is not JSON serializable
+[AIRFLOW-2938] Invalid 'extra' field in connection can raise an AttributeError when attempting to edit
+[AIRFLOW-2979] Deprecated Celery Option not in Options list
+[AIRFLOW-2981] TypeError in dataflow operators when using GCS jar or py_file
+[AIRFLOW-2984] Cannot convert naive_datetime when task has a naive start_date/end_date
+[AIRFLOW-2994] flatten_results in BigQueryOperator/BigQueryHook should default to None
+[AIRFLOW-3002] ValueError in dataflow operators when using GCS jar or py_file
+[AIRFLOW-3012] Email on sla miss is send only to first address on the list
+[AIRFLOW-3046] ECS Operator mistakenly reports success when task is killed due to EC2 host termination
+[AIRFLOW-3064] No output from `airflow test` due to default logging config
+[AIRFLOW-3072] Only admin can view logs in RBAC UI
+[AIRFLOW-3079] Improve initdb to support MSSQL Server
+[AIRFLOW-3089] Google auth doesn't work under http
+[AIRFLOW-3099] Errors raised when some blocs are missing in airflow.cfg
+[AIRFLOW-3109] Default user permission should contain 'can_clear'
+[AIRFLOW-3111] Confusing comments and instructions for log templates in UPDATING.md and default_airflow.cfg
+[AIRFLOW-3124] Broken webserver debug mode (RBAC)
+[AIRFLOW-3136] Scheduler Failing the Task retries run while processing Executor Events
+[AIRFLOW-3138] Migration cc1e65623dc7 creates issues with postgres
+[AIRFLOW-3161] Log Url link does not link to task instance logs in RBAC UI
+[AIRFLOW-3162] HttpHook fails to parse URL when port is specified
+[AIRFLOW-3183] Potential Bug in utils/dag_processing/DagFileProcessorManager.max_runs_reached()
+[AIRFLOW-3203] Bugs in DockerOperator & Some operator test scripts were named incorrectly
+[AIRFLOW-3238] Dags, removed from the filesystem, are not deactivated on initdb
+[AIRFLOW-3268] Cannot pass SSL dictionary to mysql connection via URL
+[AIRFLOW-3277] Invalid timezone transition handling for cron schedules
+[AIRFLOW-3295] Require encryption in DaskExecutor when certificates are configured.
+[AIRFLOW-3297] EmrStepSensor marks cancelled step as successful
+
+AIRFLOW 1.10.0, 2018-08-03
+--------------------------
+[AIRFLOW-2870] Use abstract TaskInstance for migration
+[AIRFLOW-2859] Implement own UtcDateTime (#3708)
+[AIRFLOW-2140] Don't require kubernetes for the SparkSubmit hook
+[AIRFLOW-2869] Remove smart quote from default config
+[AIRFLOW-2857] Fix Read the Docs env
+[AIRFLOW-2817] Force explicit choice on GPL dependency
+[AIRFLOW-2716] Replace async and await py3.7 keywords
+[AIRFLOW-2810] Fix typo in Xcom model timestamp
+[AIRFLOW-2710] Clarify fernet key value in documentation
+[AIRFLOW-2606] Fix DB schema and SQLAlchemy model
+[AIRFLOW-2646] Fix setup.py not to install snakebite on Python3
+[AIRFLOW-2604] Add index to task_fail
+[AIRFLOW-2650] Mark SchedulerJob as succeed when hitting Ctrl-c
+[AIRFLOW-2678] Fix db schema unit test to remove checking fab models
+[AIRFLOW-2624] Fix webserver login as anonymous
+[AIRFLOW-2654] Fix incorret URL on refresh in Graph View of FAB UI
+[AIRFLOW-2668] Handle missing optional cryptography dependency
+[AIRFLOW-2681] Include last dag run of externally triggered DAGs in UI.
+[AIRFLOW-1840] Support back-compat on old celery config
+[AIRFLOW-2612][AIRFLOW-2534] Clean up Hive-related tests
+[AIRFLOW-2608] Implements/Standardize custom exceptions for experimental APIs
+[AIRFLOW-2607] Fix failing TestLocalClient
+[AIRFLOW-2638] dbapi_hook: support REPLACE INTO
+[AIRFLOW-2542][AIRFLOW-1790] Rename AWS Batch Operator queue to job_queue
+[AIRFLOW-2567] Extract result from the kubernetes pod as Xcom
+[AIRFLOW-XXX] Adding REA Group to readme
+[AIRFLOW-2601] Allow user to specify k8s config
+[AIRFLOW-2559] Azure Fileshare hook
+[AIRFLOW-1786] Enforce correct behavior for soft-fail sensors
+[AIRFLOW-2355] Airflow trigger tag parameters in subdag
+[AIRFLOW-2613] Fix Airflow searching .zip bug
+[AIRFLOW-2627] Add a sensor for Cassandra
+[AIRFLOW-2634][AIRFLOW-2534] Remove dependency for impyla
+[AIRFLOW-2611] Fix wrong dag volume mount path for kubernetes executor
+[AIRFLOW-2562] Add Google Kubernetes Engine Operators
+[AIRFLOW-2630] Fix classname in test_sql_sensor.py
+[AIRFLOW-2534] Fix bug in HiveServer2Hook
+[AIRFLOW-2586] Stop getting AIRFLOW_HOME value from config file in bash operator
+[AIRFLOW-2605] Fix autocommit for MySqlHook
+[AIRFLOW-2539][AIRFLOW-2359] Move remaing log config to configuration file
+[AIRFLOW-1656] Tree view dags query changed
+[AIRFLOW-2617] add imagePullPolicy config for kubernetes executor
+[AIRFLOW-2429] Fix security/task/sensors/ti_deps folders flake8 error
+[AIRFLOW-2550] Implements API endpoint to list DAG runs
+[AIRFLOW-2512][AIRFLOW-2522] Use google-auth instead of oauth2client
+[AIRFLOW-2429] Fix operators folder flake8 error
+[AIRFLOW-2585] Fix several bugs in CassandraHook and CassandraToGCSOperator
+[AIRFLOW-2597] Restore original dbapi.run() behavior
+[AIRFLOW-2590] Fix commit in DbApiHook.run() for no-autocommit DB
+[AIRFLOW-1115] fix github oauth api URL
+[AIRFLOW-2587] Add TIMESTAMP type mapping to MySqlToHiveTransfer
+[AIRFLOW-2591][AIRFLOW-2581] Set default value of autocommit to False in DbApiHook.run()
+[AIRFLOW-59] Implement bulk_dump and bulk_load for the Postgres hook
+[AIRFLOW-2533] Fix path to DAG's on kubernetes executor workers
+[AIRFLOW-2581] RFLOW-2581] Fix DbApiHook autocommit
+[AIRFLOW-2578] Add option to use proxies in JiraHook
+[AIRFLOW-2575] Make gcs to gcs operator work with large files
+[AIRFLOW-437] Send TI context in kill zombies
+[AIRFLOW-2566] Change backfill to rerun failed tasks
+[AIRFLOW-1021] Fix double login for new users with LDAP
+[AIRFLOW-XXX] Typo fix
+[AIRFLOW-2561] Fix typo in EmailOperator
+[AIRFLOW-2573] Cast BigQuery TIMESTAMP field to float
+[AIRFLOW-2560] Adding support for internalIpOnly to DataprocClusterCreateOperator
+[AIRFLOW-2565] templatize cluster_label
+[AIRFLOW-83] add mongo hook and operator
+[AIRFLOW-2558] Clear task/dag is clearing all executions
+[AIRFLOW-XXX] Fix doc typos
+[AIRFLOW-2513] Change `bql` to `sql` for BigQuery Hooks & Ops
+[AIRFLOW-2557] Fix pagination for s3
+[AIRFLOW-2545] Eliminate DeprecationWarning
+[AIRFLOW-2500] Fix MySqlToHiveTransfer to transfer unsigned type properly
+[AIRFLOW-2462] Change PasswordUser setter to correct syntax
+[AIRFLOW-2525] Fix a bug introduced by commit dabf1b9
+[AIRFLOW-2553] Add webserver.pid to .gitignore
+[AIRFLOW-1863][AIRFLOW-2529] Add dag run selection widgets to gantt view
+[AIRFLOW-2504] Log username correctly and add extra to search columns
+[AIRFLOW-2551] Encode binary data with base64 standard rather than base64 url
+[AIRFLOW-2537] Add reset-dagrun option to backfill command
+[AIRFLOW-2526] dag_run.conf can override params
+[AIRFLOW-2544][AIRFLOW-1967] Guard against next major release of Celery, Flower
+[AIRFLOW-XXX] Add Yieldr to who is using airflow
+[AIRFLOW-2547] Describe how to run tests using Docker
+[AIRFLOW-2538] Update faq doc on how to reduce airflow scheduler latency
+[AIRFLOW-2529] Improve graph view performance and usability
+[AIRFLOW-2517] backfill support passing key values through CLI
+[AIRFLOW-2532] Support logs_volume_subpath for KubernetesExecutor
+[AIRFLOW-2466] consider task_id in _change_state_for_tis_without_dagrun
+[AIRFLOW-2519] Fix CeleryExecutor with SQLAlchemy
+[AIRFLOW-2402] Fix RBAC task log
+[AIRFLOW-XXX] Add M4U to user list
+[AIRFLOW-2536] docs about how to deal with airflow initdb failure
+[AIRFLOW-2530] KubernetesOperator supports multiple clusters
+[AIRFLOW-1499] Eliminate duplicate and unneeded code
+[AIRFLOW-2521] backfill - make variable name and logging messages more acurate
+[AIRFLOW-2429] Fix hook, macros folder flake8 error
+[Airflow-XXX] add Prime to company list
+[AIRFLOW-2525] Fix PostgresHook.copy_expert to work with "COPY FROM"
+[AIRFLOW-2515] Add dependency on thrift_sasl to hive extra
+[AIRFLOW-2523] Add how-to for managing GCP connections
+[AIRFLOW-2510] Introduce new macros: prev_ds and next_ds
+[AIRFLOW-1730] Unpickle value of XCom queried from DB
+[AIRFLOW-2518] Fix broken ToC links in integration.rst
+[AIRFLOW-1472] Fix SLA misses triggering on skipped tasks.
+[AIRFLOW-2520] CLI - make backfill less verbose
+[AIRFLOW-2107] add time_partitioning to run_query on BigQueryBaseCursor
+[AIRFLOW-1057][AIRFLOW-1380][AIRFLOW-2362][2362] AIRFLOW Update DockerOperator to new API
+[AIRFLOW-2415] Make airflow DAG templating render numbers
+[AIRFLOW-2473] Fix wrong skip condition for TransferTests
+[AIRFLOW-2472] Implement MySqlHook.bulk_dump
+[AIRFLOW-2419] Use default view for subdag operator
+[AIRFLOW-2498] Fix Unexpected argument in SFTP Sensor
+[AIRFLOW-2509] Separate config docs into how-to guides
+[AIRFLOW-2429] Add BaseExecutor back
+[AIRFLOW-2429] Fix dag, example_dags, executors flake8 error
+[AIRFLOW-2502] Change Single triple quotes to double for docstrings
+[AIRFLOW-2503] Fix broken links in CONTRIBUTING.md
+[AIRFLOW-2501] Refer to devel instructions in docs contrib guide
+[AIRFLOW-2429] Fix contrib folder's flake8 errors
+[AIRFLOW-2471] Fix HiveCliHook.load_df to use unused parameters
+[AIRFLOW-2495] Update celery to 4.1.1
+[AIRFLOW-2429] Fix api, bin, config_templates folders flake8 error
+[AIRFLOW-2493] Mark template_fields of all Operators in the API document as "templated"
+[AIRFLOW-2489] Update FlaskAppBuilder to 1.11.1
+[AIRFLOW-2448] Enhance HiveCliHook.load_df to work with datetime
+[AIRFLOW-2487] Enhance druid ingestion hook
+[AIRFLOW-2397] Support affinity policies for Kubernetes executor/operator
+[AIRFLOW-2482] Add test for rewrite method in GCS Hook
+[AIRFLOW-2481] Fix flaky Kubernetes test
+[AIRFLOW-2479] Improve doc FAQ section
+[AIRFLOW-2485] Fix Incorrect logging for Qubole Sensor
+[AIRFLOW-2486] Remove unnecessary slash after port
+[AIRFLOW-2429] Make Airflow flake8 compliant
+[AIRFLOW-2491] Resolve flask version conflict
+[AIRFLOW-2484] Remove duplicate key in MySQL to GCS Op
+[AIRFLOW-2458] Add cassandra-to-gcs operator
+[AIRFLOW-2477] Improve time units for task duration and landing times charts for RBAC UI
+[AIRFLOW-2474] Only import snakebite if using py2
+[AIRFLOW-48] Parse connection uri querystring
+[AIRFLOW-2467][AIRFLOW-2] Update import direct warn message to use the module name
+[AIRFLOW-XXX] Fix order of companies
+[AIRFLOW-2452] Document field_dict must be OrderedDict
+[AIRFLOW-2420] Azure Data Lake Hook
+[AIRFLOW-2213] Add Quoble check operator
+[AIRFLOW-2465] Fix wrong module names in the doc
+[AIRFLOW-1929] Modifying TriggerDagRunOperator to accept execution_date
+[AIRFLOW-2460] Users can now use volume mounts and volumes
+[AIRFLOW-2110][AIRFLOW-2122] Enhance Http Hook
+[AIRFLOW-XXX] Updated contributors list
+[AIRFLOW-2435] Add launch_type to ECSOperator to allow FARGATE
+[AIRFLOW-2451] Remove extra slash ('/') char when using wildcard in gcs_to_gcs operator
+[AIRFLOW-2461] Add support for cluster scaling on dataproc operator
+[AIRFLOW-2376] Fix no hive section error
+[AIRFLOW-2425] Add lineage support
+[AIRFLOW-2430] Extend query batching to additional slow queries
+[AIRFLOW-2453] Add default nil value for kubernetes/git_subpath
+[AIRFLOW-2396] Add support for resources in kubernetes operator
+[AIRFLOW-2169] Encode binary data with base64 before importing to BigQuery
+[AIRFLOW-XXX] Add spotahome in user list
+[AIRFLOW-2457] Update FAB version requirement
+[AIRFLOW-2454][Airflow 2454] Support imagePullPolicy for k8s
+[AIRFLOW-2450] update supported k8s versions to 1.9 and 1.10
+[AIRFLOW-2333] Add Segment Hook and TrackEventOperator
+[AIRFLOW-2442][AIRFLOW-2] Airflow run command leaves database connections open
+[AIRFLOW-2016] assign template_fields for Dataproc Workflow Template sub-classes, not base class
+[AIRFLOW-2446] Add S3ToRedshiftTransfer into the "Integration" doc
+[AIRFLOW-2449] Fix operators.py to run all test cases
+[AIRFLOW-2424] Add dagrun status endpoint and increased k8s test coverage
+[AIRFLOW-2441] Fix bugs in HiveCliHook.load_df
+[AIRFLOW-2358][AIRFLOW-201804] Make the Kubernetes example optional
+[AIRFLOW-2436] Remove cli_logger in initdb
+[AIRFLOW-2444] Remove unused option(include_adhoc) in cli backfill command
+[AIRFLOW-2447] Fix TestHiveMetastoreHook to run all cases
+[AIRFLOW-2445] Allow templating in kubernetes operator
+[AIRFLOW-2086][AIRFLOW-2393] Customize default dagrun number in tree view
+[AIRFLOW-2437] Add PubNub to list of current airflow users
+[AIRFLOW-XXX] Add Quantopian to list of Airflow users
+[AIRFLOW-1978] Add WinRM windows operator and hook
+[AIRFLOW-2427] Add tests to named hive sensor
+[AIRFLOW-2412] Fix HiveCliHook.load_file to address HIVE-10541
+[AIRFLOW-2431] Add the navigation bar color parameter for RBAC UI
+[AIRFLOW-2407] Resolve Python undefined names
+[AIRFLOW-1952] Add the navigation bar color parameter
+[AIRFLOW-2222] Implement GoogleCloudStorageHook.rewrite
+[AIRFLOW-2426] Add Google Cloud Storage Hook tests
+[AIRFLOW-2418] Bump Flask-WTF
+[AIRFLOW-2417] Wait for pod is not running to end task
+[AIRFLOW-1914] Add other charset support to email utils
+[AIRFLOW-XXX] Update README.md with Craig@Work
+[AIRFLOW-1899] Fix Kubernetes tests
+[AIRFLOW-1812] Update logging example
+[AIRFLOW-2313] Add TTL parameters for Dataproc
+[AIRFLOW-2411] add dataproc_jars to templated_fields
+[AIRFLOW-XXX] Add Reddit to Airflow users
+[AIRFLOW-XXX] Fix wrong table header in scheduler.rst
+[AIRFLOW-2409] Supply password as a parameter
+[AIRFLOW-2410][AIRFLOW-75] Set the timezone in the RBAC Web UI
+[AIRFLOW-2394] default cmds and arguments in kubernetes operator
+[AIRFLOW-2406] Add Apache2 License Shield to Readme
+[AIRFLOW-2404] Add additional documentation for unqueued task
+[AIRFLOW-2400] Add Ability to set Environment Variables for K8s
+[AIRFLOW-XXX] Add Twine Labs as an Airflow user
+[AIRFLOW-1853] Show only the desired number of runs in tree view
+[AIRFLOW-2401] Document the use of variables in Jinja template
+[AIRFLOW-2403] Fix License Headers
+[AIRFLOW-1313] Fix license header
+[AIRFLOW-2398] Add BounceX to list of current airflow users
+[AIRFLOW-2363] Fix return type bug in TaskHandler
+[AIRFLOW-2389] Create a pinot db api hook
+[AIRFLOW-2390] Resolve FlaskWTFDeprecationWarning
+[AIRFLOW-1933] Fix some typos
+[AIRFLOW-1960] Add support for secrets in kubernetes operator
+[AIRFLOW-1313] Add vertica_to_mysql operator
+[AIRFLOW-1575] Add AWS Kinesis Firehose Hook for inserting batch records
+[AIRFLOW-2266][AIRFLOW-2343] Remove google-cloud-dataflow dependency
+[AIRFLOW-2370] Implement --use_random_password in create_user
+[AIRFLOW-2348] Strip path prefix from the destination_object when source_object contains a wildcard[]
+[AIRFLOW-2391] Fix to Flask 0.12.2
+[AIRFLOW-2381] Fix the flaky ApiPasswordTests test
+[AIRFLOW-2378] Add Groupon to list of current users
+[AIRFLOW-2382] Fix wrong description for delimiter
+[AIRFLOW-2380] Add support for environment variables in Spark submit operator.
+[AIRFLOW-2377] Improve Sendgrid sender support
+[AIRFLOW-2331] Support init action timeout on dataproc cluster create
+[AIRFLOW-1835] Update docs: Variable file is json
+[AIRFLOW-1781] Make search case-insensitive in LDAP group
+[AIRFLOW-2042] Fix browser menu appearing over the autocomplete menu
+[AIRFLOW-XXX] Remove wheelhouse files from travis not owned by travis
+[AIRFLOW-2336] Use hmsclient in hive_hook
+[AIRFLOW-2041] Correct Syntax in python examples
+[AIRFLOW-74] SubdagOperators can consume all celeryd worker processes
+[AIRFLOW-2369] Fix gcs tests
+[AIRFLOW-2365] Fix autocommit attribute check
+[AIRFLOW-2068] MesosExecutor allows optional Docker image
+[AIRFLOW-1652] Push DatabricksRunSubmitOperator metadata into XCOM
+[AIRFLOW-2234] Enable insert_rows for PrestoHook
+[AIRFLOW-2208][Airflow-22208] Link to same DagRun graph from TaskInstance view
+[AIRFLOW-1153] Allow HiveOperators to take hiveconfs
+[AIRFLOW-775] Fix autocommit settings with Jdbc hook
+[AIRFLOW-2364] Warn when setting autocommit on a connection which does not support it
+[AIRFLOW-2357] Add persistent volume for the logs
+[AIRFLOW-766] Skip conn.commit() when in Auto-commit
+[AIRFLOW-2351] Check for valid default_args start_date
+[AIRFLOW-1433] Set default rbac to initdb
+[AIRFLOW-2270] Handle removed tasks in backfill
+[AIRFLOW-2344] Fix `connections -l` to work with pipe/redirect
+[AIRFLOW-2300] Add S3 Select functionarity to S3ToHiveTransfer
+[AIRFLOW-1314] Cleanup the config
+[AIRFLOW-1314] Polish some of the Kubernetes docs/config
+[AIRFLOW-1314] Improve error handling
+[AIRFLOW-1999] Add per-task GCP service account support
+[AIRFLOW-1314] Rebasing against master
+[AIRFLOW-1314] Small cleanup to address PR comments (#24)
+[AIRFLOW-1314] Add executor_config and tests
+[AIRFLOW-1314] Improve k8s support
+[AIRFLOW-1314] Use VolumeClaim for transporting DAGs
+[AIRFLOW-1314] Create integration testing environment
+[AIRFLOW-1314] Git Mode to pull in DAGs for Kubernetes Executor
+[AIRFLOW-1314] Add support for volume mounts & Secrets in Kubernetes Executor
+[AIRFLOW=1314] Basic Kubernetes Mode
+[AIRFLOW-2326][AIRFLOW-2222] remove contrib.gcs_copy_operator
+[AIRFLOW-2328] Fix empty GCS blob in S3ToGoogleCloudStorageOperator
+[AIRFLOW-2350] Fix grammar in UPDATING.md
+[AIRFLOW-2302] Fix documentation
+[AIRFLOW-2345] pip is not used in this setup.py
+[AIRFLOW-2347] Add Banco de Formaturas to Readme
+[AIRFLOW-2346] Add Investorise as official user of Airflow
+[AIRFLOW-2330] Do not append destination prefix if not given
+[AIRFLOW-2240][DASK] Added TLS/SSL support for the dask-distributed scheduler.
+[AIRFLOW-2309] Fix duration calculation on TaskFail
+[AIRFLOW-2335] fix issue with jdk8 download for ci
+[AIRFLOW-2184] Add druid_checker_operator
+[AIRFLOW-2299] Add S3 Select functionarity to S3FileTransformOperator
+[AIRFLOW-2254] Put header as first row in unload
+[AIRFLOW-610] Respect _cmd option in config before defaults
+[AIRFLOW-2287] Fix incorrect ASF headers
+[AIRFLOW-XXX] Add Zego as an Apache Airflow user
+[AIRFLOW-952] fix save empty extra field in UI
+[AIRFLOW-1325] Add ElasticSearch log handler and reader
+[AIRFLOW-2301] Sync files of an S3 key with a GCS path
+[AIRFLOW-2293] Fix S3FileTransformOperator to work with boto3
+[AIRFLOW-3212][AIRFLOW-2314] Remove only leading slash in GCS path
+[AIRFLOW-1509][AIRFLOW-442] SFTP Sensor
+[AIRFLOW-2291] Add optional params to ML Engine
+[AIRFLOW-1774] Allow consistent templating of arguments in MLEngineBatchPredictionOperator
+[AIRFLOW-2302] Add missing operators and hooks
+[AIRFLOW-2312] Docs Typo Correction: Corresponding
+[AIRFLOW-1623] Trigger on_kill method in operators
+[AIRFLOW-2162] When impersonating another user, pass env variables to sudo
+[AIRFLOW-2304] Update quickstart doc to mention scheduler part
+[AIRFLOW-1633] docker_operator needs a way to set shm_size
+[AIRFLOW-1340] Add S3 to Redshift transfer operator
+[AIRFLOW-2303] Lists the keys inside an S3 bucket
+[AIRFLOW-2209] restore flask_login imports
+[AIRFLOW-2306] Add Bonnier Broadcasting to list of current users
+[AIRFLOW-2305][AIRFLOW-2027] Fix CI failure caused by []
+[AIRFLOW-2281] Add support for Sendgrid categories
+[AIRFLOW-2027] Only trigger sleep in scheduler after all files have parsed
+[AIRFLOW-2256] SparkOperator: Add Client Standalone mode and retry mechanism
+[AIRFLOW-2284] GCS to S3 operator
+[AIRFLOW-2287] Update license notices
+[AIRFLOW-2296] Add Cinimex DataLab to Readme
+[AIRFLOW-2298] Add Kalibrr to who uses airflow
+[AIRFLOW-2292] Fix docstring for S3Hook.get_wildcard_key
+[AIRFLOW-XXX] Update PR template
+[AIRFLOW-XXX] Remove outdated migrations.sql
+[AIRFLOW-2287] Add license header to docs/Makefile
+[AIRFLOW-2286] Add tokopedia to the readme
+[AIRFLOW-2273] Add Discord webhook operator/hook
+[AIRFLOW-2282] Fix grammar in UPDATING.md
+[AIRFLOW-2200] Add snowflake operator with tests
+[AIRFLOW-2178] Add handling on SLA miss errors
+[AIRFLOW-2169] Fix type 'bytes' is not JSON serializable in python3
+[AIRFLOW-2215] Pass environment to subproces.Popen in base_task_runner
+[AIRFLOW-2253] Add Airflow CLI instrumentation
+[AIRFLOW-2274] Fix Dataflow tests
+[AIRFLOW-2269] Add Custom Ink as an Airflow user
+[AIRFLOW-2259] Dataflow Hook Index out of range
+[AIRFLOW-2233] Update updating.md to include the info of hdfs_sensors renaming
+[AIRFLOW-2217] Add Slack webhook operator
+[AIRFLOW-1729] improve dagBag time
+[AIRFLOW-2264] Improve create_user cli help message
+[AIRFLOW-2260] [AIRFLOW-2260] SSHOperator add command template .sh files
+[AIRFLOW-2261] Check config/env for remote base log folder
+[AIRFLOW-2258] Allow import of Parquet-format files into BigQuery
+[AIRFLOW-1430] Include INSTALL instructions to avoid GPL
+[AIRFLOW-1430] Solve GPL dependency
+[AIRFLOW-2251] Add Thinknear as an Airflow user
+[AIRFLOW-2244] bugfix: remove legacy LongText code from models.py
+[AIRFLOW-2247] Fix RedshiftToS3Transfer not to fail with ValueError
+[AIRFLOW-2249] Add side-loading support for Zendesk Hook
+[AIRFLOW-XXX] Add Qplum to Airflow users
+[AIRFLOW-2228] Enhancements in ValueCheckOperator
+[AIRFLOW-1206] Typos
+[AIRFLOW-2060] Update pendulum version to 1.4.4
+[AIRFLOW-2248] Fix wrong param name in RedshiftToS3Transfer doc
+[AIRFLOW-1433][AIRFLOW-85] New Airflow Webserver UI with RBAC support
+[AIRFLOW-1235] Fix webserver's odd behaviour
+[AIRFLOW-1460] Allow restoration of REMOVED TI's
+[airflow-2235] Fix wrong docstrings in two operators
+[AIRFLOW-XXX] Fix chronological order for companies using Airflow
+[AIRFLOW-2124] Upload Python file to a bucket for Dataproc
+[AIRFLOW-2212] Fix ungenerated sensor API reference
+[AIRFLOW-2226] Rename google_cloud_storage_default to google_cloud_default
+[AIRFLOW-2211] Rename hdfs_sensors.py to hdfs_sensor.py for consistency
+[AIRFLOW-2225] Update document to include DruidDbApiHook
+[Airflow-2202] Add filter support in HiveMetastoreHook().max_partition()
+[AIRFLOW-2220] Remove duplicate numeric list entry in security.rst
+[AIRFLOW-XXX] Update tutorial documentation
+[AIRFLOW-2215] Update celery task to preserve environment variables and improve logging on exception
+[AIRFLOW-2185] Use state instead of query param
+[AIRFLOW-2183] Refactor DruidHook to enable sql
+[AIRFLOW-2203] Defer cycle detection
+[AIRFLOW-2203] Remove Useless Commands.
+[AIRFLOW-2203] Cache signature in apply_defaults
+[AIRFLOW-2203] Speed up Operator Resources
+[AIRFLOW-2203] Cache static rules (trigger/weight)
+[AIRFLOW-2203] Store task ids as sets not lists
+[AIRFLOW-2205] Remove unsupported args from JdbcHook doc
+[AIRFLOW-2207] Fix flaky test that uses app.cached_app()
+[AIRFLOW-2206] Remove unsupported args from JdbcOperator doc
+[AIRFLOW-2140] Add Kubernetes scheduler to SparkSubmitOperator
+[AIRFLOW-XXX] Add Xero to list of users
+[AIRFLOW-2204] Fix webserver debug mode
+[AIRFLOW-102] Fix test_complex_template always succeeds
+[AIRFLOW-442] Add SFTPHook
+[AIRFLOW-2169] Add schema to MySqlToGoogleCloudStorageOperator
+[AIRFLOW-2184][AIRFLOW-2138] Google Cloud Storage allow wildcards
+[AIRFLOW-1588] Cast Variable value to string
+[AIRFLOW-2199] Fix invalid reference to logger
+[AIRFLOW-2191] Change scheduler heartbeat logs from info to debug
+[AIRFLOW-2106] SalesForce hook sandbox option
+[AIRFLOW-2197] Silence hostname_callable config error message
+[AIRFLOW-2150] Use lighter call in HiveMetastoreHook().max_partition()
+[AIRFLOW-2186] Change the way logging is carried out in few ops
+[AIRFLOW-2181] Convert password_auth and test_password_endpoints from DOS to UNIX
+[AIRFLOW-2187] Fix Broken Travis CI due to AIRFLOW-2123
+[AIRFLOW-2175] Check that filepath is not None
+[AIRFLOW-2173] Don't check task IDs for concurrency reached check
+[AIRFLOW-2168] Remote logging for Azure Blob Storage
+[AIRFLOW-XXX] Add DocuTAP to list of users
+[AIRFLOW-2176] Change the way logging is carried out in BQ Get Data Operator
+[AIRFLOW-2177] Add mock test for GCS Download op
+[AIRFLOW-2123] Install CI dependencies from setup.py
+[AIRFLOW-2129] Presto hook calls _parse_exception_message but defines _get_pretty_exception_message
+[AIRFLOW-2174] Fix typos and wrongly rendered documents
+[AIRFLOW-2171] Store delegated credentials
+[AIRFLOW-2166] Restore BQ run_query dialect param
+[AIRFLOW-2163] Add HBC Digital to users of airflow
+[AIRFLOW-2065] Fix race-conditions when creating loggers
+[AIRFLOW-2147] Plugin manager: added 'sensors' attribute
+[AIRFLOW-2059] taskinstance query is awful, un-indexed, and does not scale
+[AIRFLOW-2159] Fix a few typos in salesforce_hook
+[AIRFLOW-2132] Add step to initialize database
+[AIRFLOW-2160] Fix bad rowid deserialization
+[AIRFLOW-2161] Add Vevo to list of companies using Airflow
+[AIRFLOW-2149] Add link to apache Beam documentation to create self executing Jar
+[AIRFLOW-2151] Allow getting the session from AwsHook
+[AIRFLOW-2097] tz referenced before assignment
+[AIRFLOW-2152] Add Multiply to list of companies using Airflow
+[AIRFLOW-1551] Add operator to trigger Jenkins job
+[AIRFLOW-2034] Fix mixup between %s and {} when using str.format Convention is to use .format for string formating oustide logging, else use lazy format See comment in related issue https://github.com/apache/incubator-airflow/pull/2823/files Identified problematic case using following command line .git/COMMIT_EDITMSG:`grep -r '%s'./* | grep '\.format('`
+[AIRFLOW-2102] Add custom_args to Sendgrid personalizations
+[AIRFLOW-1035][AIRFLOW-1053] import unicode_literals to parse Unicode in HQL
+[AIRFLOW-2127] Keep loggers during DB migrations
+[AIRFLOW-2146] Resolve issues with BQ using DbApiHook methods
+[AIRFLOW-2087] Scheduler Report shows incorrect Total task number
+[AIRFLOW-2139] Remove unncecessary boilerplate to get DataFrame using pandas_gbq
+[AIRFLOW-2125] Using binary package psycopg2-binary
+[AIRFLOW-2142] Include message on mkdir failure
+[AIRFLOW-1615] SSHHook: use port specified by Connection
+[AIRFLOW-2122] Handle boolean values in sshHook
+[AIRFLOW-XXX] Add Tile to the list of users
+[AIRFLOW-2130] Add missing Operators to API Reference docs
+[AIRFLOW-XXX] Add timeout units (seconds)
+[AIRFLOW-2134] Add Alan to the list of companies that use Airflow
+[AIRFLOW-2133] Remove references to GitHub issues in CONTRIBUTING
+[AIRFLOW-2131] Remove confusing AirflowImport docs
+[AIRFLOW-1852] Allow hostname to be overridable.
+[AIRFLOW-2126] Add Bluecore to active users
+[AIRFLOW-1618] Add feature to create GCS bucket
+[AIRFLOW-2108] Fix log indentation in BashOperator
+[AIRFLOW-2115] Fix doc links to PythonHosted
+[AIRFLOW-XXX] Add contributor from Easy company
+[AIRFLOW-1882] Add ignoreUnknownValues option to gcs_to_bq operator
+[AIRFLOW-2089] Add on kill for SparkSubmit in Standalone Cluster
+[AIRFLOW-2113] Address missing DagRun callbacks Given that the handle_callback method belongs to the DAG object, we are able to get the list of task directly with get_task and reduce the communication with the database, making airflow more lightweight.
+[AIRFLOW-2112] Fix svg width for Recent Tasks on UI.
+[AIRFLOW-2116] Set CI Cloudant version to <2.0
+[AIRFLOW-XXX] Add PMC to list of companies using Airflow
+[AIRFLOW-2100] Fix Broken Documentation Links
+[AIRFLOW-1404] Add 'flatten_results' & 'maximum_bytes_billed' to BQ Operator
+[AIRFLOW-800] Initialize valid Google BigQuery Connection
+[AIRFLOW-1319] Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
+[AIRFLOW-1983] Parse environment parameter as template
+[AIRFLOW-2095] Add operator to create External BigQuery Table
+[AIRFLOW-2085] Add SparkJdbc operator
+[AIRFLOW-1002] Add ability to clean all dependencies of removed DAG
+[AIRFLOW-2094] Jinjafied project_id, region & zone in DataProc{*} Operators
+[AIRFLOW-2092] Fixed incorrect parameter in docstring for FTPHook
+[AIRFLOW-XXX] Add SocialCops to Airflow users
+[AIRFLOW-2088] Fix duplicate keys in MySQL to GCS Helper function
+[AIRFLOW-2091] Fix incorrect docstring parameter in BigQuery Hook
+[AIRFLOW-2090] Fix typo in DataStore Hook
+[AIRFLOW-1157] Fix missing pools crashing the scheduler
+[AIRFLOW-713] Jinjafy {EmrCreateJobFlow,EmrAddSteps}Operator attributes
+[AIRFLOW-2083] Docs: Use "its" instead of "it's" where appropriate
+[AIRFLOW-2066] Add operator to create empty BQ table
+[AIRFLOW-XXX] add Karmic to list of companies
+[AIRFLOW-2073] Make FileSensor fail when the file doesn't exist
+[AIRFLOW-2078] Improve task_stats and dag_stats performance
+[AIRFLOW-2080] Use a log-out icon instead of a power button
+[AIRFLOW-2077] Fetch all pages of list_objects_v2 response
+[AIRFLOW-XXX] Add TM to list of companies
+[AIRFLOW-1985] Impersonation fixes for using `run_as_user`
+[AIRFLOW-2018][AIRFLOW-2] Make Sensors backward compatible
+[AIRFLOW-XXX] Fix typo in concepts doc (dag_md)
+[AIRFLOW-2069] Allow Bytes to be uploaded to S3
+[AIRFLOW-2074] Fix log var name in GHE auth
+[AIRFLOW-1927] Convert naive datetimes for TaskInstances
+[AIRFLOW-1760] Password auth for experimental API
+[AIRFLOW-2038] Add missing kubernetes dependency for dev
+[AIRFLOW-2040] Escape special chars in task instance logs URL
+[AIRFLOW-1968][AIRFLOW-1520] Add role_arn and aws_account_id/aws_iam_role support back to aws hook
+[AIRFLOW-2048] Fix task instance failure string formatting
+[AIRFLOW-2046] Fix kerberos error to work with python 3.x
+[AIRFLOW-2063] Add missing docs for GCP
+[AIRFLOW-XXX] Fix typo in docs
+[AIRFLOW-1793] Use docker_url instead of invalid base_url
+[AIRFLOW-2055] Elaborate on slightly ambiguous documentation
+[AIRFLOW-2039] BigQueryOperator supports priority property
+[AIRFLOW-2053] Fix quote character bug in BQ hook
+[AIRFLOW-2057] Add Overstock to list of companies
+[AIRFLOW-XXX] Add Plaid to Airflow users
+[AIRFLOW-2044] Add SparkSubmitOperator to documentation
+[AIRFLOW-2037] Add methods to get Hash values of a GCS object
+[AIRFLOW-2050] Fix Travis permission problem
+[AIRFLOW-2043] Add Intercom to list of companies
+[AIRFLOW-2023] Add debug logging around number of queued files
+[AIRFLOW-XXX] Add Pernod-ricard as a airflow user
+[AIRFLOW-1453] Add 'steps' into template_fields in EmrAddSteps
+[AIRFLOW-2015] Add flag for interactive runs
+[AIRFLOW-1895] Fix primary key integrity for mysql
+[AIRFLOW-2030] Fix KeyError:`i` in DbApiHook for insert
+[AIRFLOW-1943] Add External BigQuery Table feature
+[AIRFLOW-2033] Add Google Cloud Storage List Operator
+[AIRFLOW-2006] Add local log catching to kubernetes operator
+[AIRFLOW-2031] Add missing gcp_conn_id in the example in DataFlow docstrings
+[AIRFLOW-2029] Fix AttributeError in BigQueryPandasConnector
+[AIRFLOW-2028] Add JobTeaser to official users list
+[AIRFLOW-2016] Add support for Dataproc Workflow Templates
+[AIRFLOW-2025] Reduced Logging verbosity
+[AIRFLOW-1267][AIRFLOW-1874] Add dialect parameter to BigQueryHook
+[AIRFLOW-XXX] Fixed a typo
+[AIRFLOW-XXX] Typo node to nodes
+[AIRFLOW-2019] Update DataflowHook for updating Streaming type job
+[AIRFLOW-2017][Airflow 2017] adding query output to PostgresOperator
+[AIRFLOW-1889] Split sensors into separate files
+[AIRFLOW-1950] Optionally pass xcom_pull task_ids
+[AIRFLOW-1755] Allow mount below root
+[AIRFLOW-511][Airflow 511] add success/failure callbacks on dag level
+[AIRFLOW-192] Add weight_rule param to BaseOperator
+[AIRFLOW-2008] Use callable for python column defaults
+[AIRFLOW-1984] Fix to AWS Batch operator
+[AIRFLOW-2000] Support non-main dataflow job class
+[AIRFLOW-2003] Use flask-caching instead of flask-cache
+[AIRFLOW-2002] Do not swallow exception on logging import
+[AIRFLOW-2004] Import flash from flask not flask.login
+[AIRFLOW-1997] Fix GCP operator doc strings
+[AIRFLOW-1996] Update DataflowHook waitfordone for Streaming type job[]
+[AIRFLOW-1995][Airflow 1995] add on_kill method to SqoopOperator
+[AIRFLOW-1770] Allow HiveOperator to take in a file
+[AIRFLOW-1994] Change background color of Scheduled state Task Instances
+[AIRFLOW-1436][AIRFLOW-1475] EmrJobFlowSensor considers Cancelled step as Successful
+[AIRFLOW-1517] Kubernetes operator PR fixes
+[AIRFLOW-1517] addressed PR comments
+[AIRFLOW-1517] started documentation of k8s operator
+[AIRFLOW-1517] Restore authorship of resources
+[AIRFLOW-1517] Remove authorship of resources
+[AIRFLOW-1517] Add minikube for kubernetes integration tests
+[AIRFLOW-1517] Restore authorship of resources
+[AIRFLOW-1517] fixed license issues
+[AIRFLOW-1517] Created more accurate failures for kube cluster issues
+[AIRFLOW-1517] Remove authorship of resources
+[AIRFLOW-1517] Add minikube for kubernetes integration tests
+[AIRFLOW-1988] Change BG color of None state TIs
+[AIRFLOW-790] Clean up TaskInstances without DagRuns
+[AIRFLOW-1949] Fix var upload, str() produces "b'...'" which is not json
+[AIRFLOW-1930] Convert func.now() to timezone.utcnow()
+[AIRFLOW-1688] Support load.time_partitioning in bigquery_hook
+[AIRFLOW-1975] Make TriggerDagRunOperator callback optional
+[AIRFLOW-1480] Render template attributes for ExternalTaskSensor fields
+[AIRFLOW-1958] Add **kwargs to send_email
+[AIRFLOW-1976] Fix for missing log/logger attribute FileProcessHandler
+[AIRFLOW-1982] Fix Executor event log formatting
+[AIRFLOW-1971] Propagate hive config on impersonation
+[AIRFLOW-1969] Always use HTTPS URIs for Google OAuth2
+[AIRFLOW-1954] Add DataFlowTemplateOperator
+[AIRFLOW-1963] Add config for HiveOperator mapred_queue
+[AIRFLOW-1946][AIRFLOW-1855] Create a BigQuery Get Data Operator
+[AIRFLOW-1953] Add labels to dataflow operators
+[AIRFLOW-1967] Update Celery to 4.0.2
+[AIRFLOW-1964] Add Upsight to list of Airflow users
+[AIRFLOW-XXX] Changelog for 1.9.0
+[AIRFLOW-1470] Implement BashSensor operator
+[AIRFLOW-XXX] Pin sqlalchemy dependency
+[AIRFLOW-1955] Do not reference unassigned variable
+[AIRFLOW-1957] Add contributor to BalanceHero in Readme
+[AIRFLOW-1517] Restore authorship of secrets and init container
+[AIRFLOW-1517] Remove authorship of secrets and init container
+[AIRFLOW-1935] Add BalanceHero to readme
+[AIRFLOW-1939] add astronomer contributors
+[AIRFLOW-1517] Kubernetes Operator
+[AIRFLOW-1928] Fix @once with catchup=False
+[AIRFLOW-1937] Speed up scheduling by committing in batch
+[AIRFLOW-1821] Enhance default logging config by removing extra loggers
+[AIRFLOW-1904] Correct DAG fileloc to the right filepath
+[AIRFLOW-1909] Update docs with supported versions of MySQL server
+[AIRFLOW-1915] Relax flask-wtf dependency specification
+[AIRFLOW-1920] Update CONTRIBUTING.md to reflect enforced linting rules
+[AIRFLOW-1942] Update Sphinx docs to remove deprecated import structure
+[AIRFLOW-1846][AIRFLOW-1697] Hide Ad Hoc Query behind secure_mode config
+[AIRFLOW-1948] Include details for on_kill failure
+[AIRFLOW-1938] Clean up unused exception
+[AIRFLOW-1932] Add GCP Pub/Sub Pull and Ack
+[AIRFLOW-XXX] Purge coveralls
+[AIRFLOW-XXX] Remove unused coveralls token
+[AIRFLOW-1938] Remove tag version check in setup.py
+[AIRFLOW-1916] Don't upload logs to remote from `run --raw`
+[AIRFLOW-XXX] Fix failing PubSub tests on Python3
+[AIRFLOW-XXX] Upgrade to python 3.5 and disable dask tests
+[AIRFLOW-1913] Add new GCP PubSub operators
+[AIRFLOW-1525] Fix minor LICENSE and NOTICE issues
+[AIRFLOW-1687] fix fernet error without encryption
+[AIRFLOW-1912] airflow.processor should not propagate logging
+[AIRFLOW-1911] Rename celeryd_concurrency
+[AIRFLOW-1885] Fix IndexError in ready_prefix_on_cmdline
+[AIRFLOW-1854] Improve Spark Submit operator for standalone cluster mode
+[AIRFLOW-1908] Fix celery broker options config load
+[AIRFLOW-1907] Pass max_ingestion_time to Druid hook
+[AIRFLOW-1909] Add away to list of users
+[AIRFLOW-1893][AIRFLOW-1901] Propagate PYTHONPATH when using impersonation
+[AIRFLOW-1892] Modify BQ hook to extract data filtered by column
+[AIRFLOW-1829] Support for schema updates in query jobs
+[AIRFLOW-1840] Make celery configuration congruent with Celery 4
+[AIRFLOW-1878] Fix stderr/stdout redirection for tasks
+[AIRFLOW-1897][AIRFLOW-1873] Task Logs for running instance not visible in WebUI
+[AIRFLOW-1896] FIX bleach <> html5lib incompatibility
+[AIRFLOW-1884][AIRFLOW-1059] Reset orphaned task state for external dagruns
+[AIRFLOW-XXX] Fix typo in comment
+[AIRFLOW-1869] Do not emit spurious warning on missing logs
+[AIRFLOW-1888] Add AWS Redshift Cluster Sensor
+[AIRFLOW-1887] Renamed endpoint url variable
+[AIRFLOW-1873] Set TI.try_number to right value depending TI state
+[AIRFLOW-1891] Fix non-ascii typo in default configuration template
+[AIRFLOW-1879] Handle ti log entirely within ti
+[AIRFLOW-1869] Write more error messages into gcs and file logs
+[AIRFLOW-1876] Write subtask id to task log header
+[AIRFLOW-1554] Fix wrong DagFileProcessor termination method call
+[AIRFLOW-342] Do not use amqp, rpc as result backend
+[AIRFLOW-966] Make celery broker_transport_options configurable
+[AIRFLOW-1881] Make operator log in task log
+[AIRFLOW-XXX] Added DataReply to the list of Airflow Users
+[AIRFLOW-1883] Get File Size for objects in Google Cloud Storage
+[AIRFLOW-1872] Set context for all handlers including parents
+[AIRFLOW-1855][AIRFLOW-1866] Add GCS Copy Operator to copy multiple files
+[AIRFLOW-1870] Enable flake8 tests
+[AIRFLOW-1785] Enable Python 3 tests
+[AIRFLOW-1850] Copy cmd before masking
+[AIRFLOW-1665] Reconnect on database errors
+[AIRFLOW-1559] Dispose SQLAlchemy engines on exit
+[AIRFLOW-1559] Close file handles in subprocesses
+[AIRFLOW-1559] Make database pooling optional
+[AIRFLOW-1848][Airflow-1848] Fix DataFlowPythonOperator py_file extension doc comment
+[AIRFLOW-1843] Add Google Cloud Storage Sensor with prefix
+[AIRFLOW-1803] Time zone documentation
+[AIRFLOW-1826] Update views to use timezone aware objects
+[AIRFLOW-1827] Fix api endpoint date parsing
+[AIRFLOW-1806] Use naive datetime when using cron
+[AIRFLOW-1809] Update tests to use timezone aware objects
+[AIRFLOW-1806] Use naive datetime for cron scheduling
+[AIRFLOW-1807] Force use of time zone aware db fields
+[AIRFLOW-1808] Convert all utcnow() to time zone aware
+[AIRFLOW-1804] Add time zone configuration options
+[AIRFLOW-1802] Convert database fields to timezone aware
+[AIRFLOW-XXX] Add dask lock files to excludes
+[AIRFLOW-1790] Add support for AWS Batch operator
+[AIRFLOW-XXX] Update README.md
+[AIRFLOW-1820] Remove timestamp from metric name
+[AIRFLOW-1810] Remove unused mysql import in migrations.
+[AIRFLOW-1838] Properly log collect_dags exception
+[AIRFLOW-1842] Fixed Super class name for the gcs to gcs copy operator
+[AIRFLOW-1845] Modal background now covers long or tall pages
+[AIRFLOW-1229] Add link to Run Id, incl execution_date
+[AIRFLOW-1842] Add gcs to gcs copy operator with renaming if required
+[AIRFLOW-1841] change False to None in operator and hook
+[AIRFLOW-1839] Fix more bugs in S3Hook boto -> boto3 migration
+[AIRFLOW-1830] Support multiple domains in Google authentication backend
+[AIRFLOW-1831] Add driver-classpath spark submit
+[AIRFLOW-1795] Correctly call S3Hook after migration to boto3
+[AIRFLOW-1811] Fix render Druid operator
+[AIRFLOW-1819] Fix slack operator unittest bug
+[AIRFLOW-1805] Allow Slack token to be passed through connection
+[AIRFLOW-1816] Add region param to Dataproc operators
+[AIRFLOW-868] Add postgres_to_gcs operator and unittests
+[AIRFLOW-1613] make mysql_to_gcs_operator py3 compatible
+[AIRFLOW-1817] use boto3 for s3 dependency
+[AIRFLOW-1813] Bug SSH Operator empty buffer
+[AIRFLOW-1801][AIRFLOW-288] Url encode execution dates
+[AIRFLOW-1563] Catch OSError while symlinking the latest log directory
+[AIRFLOW-1794] Remove uses of Exception.message for Python 3
+[AIRFLOW-1799] Fix logging line which raises errors
+[AIRFLOW-1102] Upgrade Gunicorn >=19.4.0
+[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook
+[AIRFLOW-1797] S3Hook.load_string didn't work on Python3
+[AIRFLOW-646] Add docutils to setup_requires
+[AIRFLOW-1792] Missing intervals DruidOperator
+[AIRFLOW-1789][AIRFLOW-1712] Log SSHOperator stderr to log.warning
+[AIRFLOW-1787] Fix task instance batch clear and set state bugs
+[AIRFLOW-1780] Fix long output lines with unicode from hanging parent
+[AIRFLOW-387] Close SQLAlchemy sessions properly
+[AIRFLOW-1779] Add keepalive packets to ssh hook
+[AIRFLOW-1669] Fix Docker and pin Moto to 1.1.19
+[AIRFLOW-71] Add support for private Docker images
+[AIRFLOW-XXX] Give a clue what the 'ds' variable is
+[AIRFLOW-XXX] Correct typos in the faq docs page
+[AIRFLOW-1571] Add AWS Lambda Hook
+[AIRFLOW-1675] Fix docstrings for API docs
+[AIRFLOW-1712][AIRFLOW-756][AIRFLOW-751] Log SSHOperator output
+[AIRFLOW-1776] Capture stdout and stderr for logging
+[AIRFLOW-1765] Make experimental API securable without needing Kerberos.
+[AIRFLOW-1764] The web interface should not use the experimental API
+[AIRFLOW-1771] Rename heartbeat to avoid confusion
+[AIRFLOW-1769] Add support for templates in VirtualenvOperator
+[AIRFLOW-1763] Fix S3TaskHandler unit tests
+[AIRFLOW-1315] Add Qubole File & Partition Sensors
+[AIRFLOW-1018] Make processor use logging framework
+[AIRFLOW-1695] Add RedshiftHook using boto3
+[AIRFLOW-1706] Fix query error for MSSQL backend
+[AIRFLOW-1711] Use ldap3 dict for group membership
+[AIRFLOW-1723] Make sendgrid a plugin
+[AIRFLOW-1757] Add missing options to SparkSubmitOperator
+[AIRFLOW-1734][Airflow 1734] Sqoop hook/operator enhancements
+[AIRFLOW-1761] Fix type in scheduler.rst
+[AIRFLOW-1731] Set pythonpath for logging
+[AIRFLOW-1641] Handle executor events in the scheduler
+[AIRFLOW-1744] Make sure max_tries can be set
+[AIRFLOW-1732] Improve dataflow hook logging
+[AIRFLOW-1736] Add HotelQuickly to Who Uses Airflow
+[AIRFLOW-1657] Handle failing qubole operator
+[AIRFLOW-1677] Fix typo in example_qubole_operator
+[AIRFLOW-926] Fix JDBC Hook
+[AIRFLOW-1520] Boto3 S3Hook, S3Log
+[AIRFLOW-1716] Fix multiple __init__ def in SimpleDag
+[AIRFLOW-XXX] Fix DateTime in Tree View
+[AIRFLOW-1719] Fix small typo
+[AIRFLOW-1432] Charts label for Y axis not visible
+[AIRFLOW-1743] Verify ldap filters correctly
+[AIRFLOW-1745] Restore default signal disposition
+[AIRFLOW-1741] Correctly hide second chart on task duration page
+[AIRFLOW-1728] Add networkUri, subnet, tags to Dataproc operator
+[AIRFLOW-1726] Add copy_expert psycopg2 method to PostgresHook
+[AIRFLOW-1330] Add conn_type argument to CLI when adding connection
+[AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd
+[AIRFLOW-1694] Stop using itertools.izip
+[AIRFLOW-1692] Change test_views filename to support Windows
+[AIRFLOW-1722] Fix typo in scheduler autorestart output filename
+[AIRFLOW-1723] Support sendgrid in email backend
+[AIRFLOW-1718] Set num_retries on Dataproc job request execution
+[AIRFLOW-1727] Add unit tests for DataProcHook
+[AIRFLOW-1631] Fix timing issue in unit test
+[AIRFLOW-1631] Fix local executor unbound parallelism
+[AIRFLOW-1724] Add Fundera to Who uses Airflow?
+[AIRFLOW-1683] Cancel BigQuery job on timeout.
+[AIRFLOW-1714] Fix misspelling: s/seperate/separate/
+[AIRFLOW-1681] Add batch clear in task instance view
+[AIRFLOW-1696] Fix dataproc version label error
+[AIRFLOW-1613] Handle binary field in MySqlToGoogleCloudStorageOperator
+[AIRFLOW-1697] Mode to disable charts endpoint
+[AIRFLOW-1691] Add better Google cloud logging documentation
+[AIRFLOW-1690] Add detail to gcs error messages
+[AIRFLOW-1682] Make S3TaskHandler write to S3 on close
+[AIRFLOW-1634] Adds task_concurrency feature
+[AIRFLOW-1676] Make GCSTaskHandler write to GCS on close
+[AIRFLOW-1678] Fix erroneously repeated word in function docstrings
+[AIRFLOW-1323] Made Dataproc operator parameter names consistent
+[AIRFLOW-1590] fix unused module and variable
+[AIRFLOW-1671] Add @apply_defaults back to gcs download operator
+[AIRFLOW-988] Fix repeating SLA miss callbacks
+[AIRFLOW-1611] Customize logging
+[AIRFLOW-1668] Expose keepalives_idle for Postgres connections
+[AIRFLOW-1658] Kill Druid task on timeout
+[AIRFLOW-1669][AIRFLOW-1368] Fix Docker import
+[AIRFLOW-891] Make webserver clock include date
+[AIRFLOW-1560] Add AWS DynamoDB hook and operator for inserting batch items
+[AIRFLOW-1654] Show tooltips for link icons in DAGs view
+[AIRFLOW-1660] Change webpage width to full-width
+[AIRFLOW-1664] write file as binary instead of str
+[AIRFLOW-1659] Fix invalid obj attribute bug in file_task_handler.py
+[AIRFLOW-1635] Allow creating GCP connection without requiring a JSON file
+[AIRFLOW-1650] Fix custom celery config loading
+[AIRFLOW-1647] Fix Spark-sql hook
+[AIRFLOW-1587] Fix CeleryExecutor import error
+[Airflow-1640][AIRFLOW-1640] Add qubole default connection
+[AIRFLOW-1576] Added region param to Dataproc{*}Operators
+[AIRFLOW-1643] Add healthjump to officially using list
+[AIRFLOW-1626] Add Azri Solutions to Airflow users
+[AIRFLOW-1636] Add AWS and EMR connection type
+[AIRFLOW-1527] Refactor celery config
+[AIRFLOW-1639] Fix Fernet error handling
+[AIRFLOW-1637] Fix Travis CI build status link
+[AIRFLOW-1628] Fix docstring of sqlsensor
+[AIRFLOW-1331] add SparkSubmitOperator option
+[AIRFLOW-1627] Only query pool in SubDAG init when necessary
+[AIRFLOW-1629] Make extra a textarea in edit connections form
+[AIRFLOW-1368] Automatically remove Docker container on exit
+[AIRFLOW-289] Make airflow timezone independent
+[AIRFLOW-1356] Add `--celery_hostname` to `airflow worker`
+[AIRFLOW-1247] Fix ignore_all_dependencies argument ignored
+[AIRFLOW-1621] Add tests for server side paging
+[AIRFLOW-1591] Avoid attribute error when rendering logging filename
+[AIRFLOW-1031] Replace hard-code to DagRun.ID_PREFIX
+[AIRFLOW-1604] Rename logger to log
+[AIRFLOW-1512] Add PythonVirtualenvOperator
+[AIRFLOW-1617] Fix XSS vulnerability in Variable endpoint
+[AIRFLOW-1497] Reset hidden fields when changing connection type
+[AIRFLOW-1619] Add poll_sleep parameter to GCP dataflow operator
+[AIRFLOW-XXX] Remove landscape.io config
+[AIRFLOW-XXX] Remove non working service badges
+[AIRFLOW-1177] Fix Variable.setdefault w/existing JSON
+[AIRFLOW-1600] Fix exception handling in get_fernet
+[AIRFLOW-1614] Replace inspect.stack() with sys._getframe()
+[AIRFLOW-1519] Add server side paging in DAGs list
+[AIRFLOW-1309] Allow hive_to_druid to take tblproperties
+[AIRFLOW-1613] Make MySqlToGoogleCloudStorageOperator compaitible with python3
+[AIRFLOW-1603] add PAYMILL to companies list
+[AIRFLOW-1609] Fix gitignore to ignore all venvs
+[AIRFLOW-1601] Add configurable task cleanup time
+
AIRFLOW 1.9.0, 2018-01-02
-------------------------
@@ -589,7 +1504,7 @@ AIRFLOW 1.8.0, 2017-03-12
[AIRFLOW-784] Pin funcsigs to 1.0.0
[AIRFLOW-624] Fix setup.py to not import airflow.version as version
[AIRFLOW-779] Task should fail with specific message when deleted
-[AIRFLOW-778] Fix completey broken MetastorePartitionSensor
+[AIRFLOW-778] Fix completely broken MetastorePartitionSensor
[AIRFLOW-739] Set pickle_info log to debug
[AIRFLOW-771] Make S3 logs append instead of clobber
[AIRFLOW-773] Fix flaky datetime addition in api test
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index da82805cf8..d8dab9c281 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,12 +1,30 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
# Contributing
Contributions are welcome and are greatly appreciated! Every
little bit helps, and credit will always be given.
-
# Table of Contents
* [TOC](#table-of-contents)
- * [Types of Contributions](#types-of-contribution)
+ * [Types of Contributions](#types-of-contributions)
- [Report Bugs](#report-bugs)
- [Fix Bugs](#fix-bugs)
- [Implement Features](#implement-features)
@@ -15,11 +33,10 @@ little bit helps, and credit will always be given.
* [Documentation](#documentation)
* [Development and Testing](#development-and-testing)
- [Setting up a development environment](#setting-up-a-development-environment)
- - [Pull requests guidelines](#pull-requests-guidelines)
- - [Testing Locally](#testing-locally)
+ - [Running unit tests](#running-unit-tests)
+ * [Pull requests guidelines](#pull-request-guidelines)
* [Changing the Metadata Database](#changing-the-metadata-database)
-
## Types of Contributions
### Report Bugs
@@ -29,7 +46,6 @@ Report bugs through [Apache Jira](https://issues.apache.org/jira/browse/AIRFLOW)
Please report relevant information and preferably code that exhibits
the problem.
-
### Fix Bugs
Look through the Jira issues for bugs. Anything is open to whoever wants
@@ -37,12 +53,11 @@ to implement it.
### Implement Features
-Look through the GitHub issues for features. Anything tagged with
-"feature" is open to whoever wants to implement it.
+Look through the [Apache Jira](https://issues.apache.org/jira/browse/AIRFLOW) for features. Any unassigned "Improvement" issue is open to whoever wants to implement it.
We've created the operators, hooks, macros and executors we needed, but we
made sure that this part of Airflow is extensible. New operators,
-hooks and operators are very welcomed!
+hooks, macros and executors are very welcomed!
### Improve Documentation
@@ -53,116 +68,116 @@ articles.
### Submit Feedback
-The best way to send feedback is to file an issue on Github.
+The best way to send feedback is to open an issue on [Apache Jira](https://issues.apache.org/jira/browse/AIRFLOW)
If you are proposing a feature:
-- Explain in detail how it would work.
-- Keep the scope as narrow as possible, to make it easier to
- implement.
-- Remember that this is a volunteer-driven project, and that
- contributions are welcome :)
+- Explain in detail how it would work.
+- Keep the scope as narrow as possible, to make it easier to implement.
+- Remember that this is a volunteer-driven project, and that contributions are welcome :)
## Documentation
-The latest API documentation is usually available [here](http://pythonhosted.org/airflow).
-To generate a local version, you need to have installed airflow with
-the `doc` extra. In that case you can generate the doc by running:
+The latest API documentation is usually available
+[here](https://airflow.incubator.apache.org/). To generate a local version,
+you need to have set up an Airflow development environment (see below). Also
+install the `doc` extra.
+
+```
+pip install -e .[doc]
+```
- cd docs && ./build.sh
+Generate and serve the documentation by running:
+
+```
+cd docs
+./build.sh
+./start_doc_server.sh
+```
+
+Only a subset of the API reference documentation builds. Install additional
+extras to build the full API reference.
## Development and Testing
-### Setting up a development environment
+### Set up a development environment
-Please install python(2.7.x or 3.4.x), mysql, and libxml by using system-level package
-managers like yum, apt-get for Linux, or homebrew for Mac OS at first.
-It is usually best to work in a virtualenv and tox. Install development requirements:
+There are three ways to setup an Apache Airflow development environment.
- cd $AIRFLOW_HOME
- virtualenv env
- source env/bin/activate
- pip install -e .[devel]
- tox
+1. Using tools and libraries installed directly on your system.
-Feel free to customize based on the extras available in [setup.py](./setup.py)
+ Install Python (2.7.x or 3.5.x), MySQL, and libxml by using system-level package
+ managers like yum, apt-get for Linux, or Homebrew for Mac OS at first. Refer to the [base CI Dockerfile](https://github.com/apache/incubator-airflow-ci/blob/master/Dockerfile) for
+ a comprehensive list of required packages.
-### Pull Request Guidelines
+ Then install python development requirements. It is usually best to work in a virtualenv:
-Before you submit a pull request from your forked repo, check that it
-meets these guidelines:
+ ```bash
+ cd $AIRFLOW_HOME
+ virtualenv env
+ source env/bin/activate
+ pip install -e .[devel]
+ ```
+
+2. Using a Docker container
+
+ Go to your Airflow directory and start a new docker container. You can choose between Python 2 or 3, whatever you prefer.
+
+ ```
+ # Start docker in your Airflow directory
+ docker run -t -i -v `pwd`:/airflow/ -w /airflow/ -e SLUGIFY_USES_TEXT_UNIDECODE=yes python:3 bash
+
+ # Install Airflow with all the required dependencies,
+ # including the devel which will provide the development tools
+ pip install -e ".[hdfs,hive,druid,devel]"
+
+ # Init the database
+ airflow initdb
+
+ nosetests -v tests/hooks/test_druid_hook.py
+
+ test_get_first_record (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok
+ test_get_records (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok
+ test_get_uri (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok
+ test_get_conn_url (tests.hooks.test_druid_hook.TestDruidHook) ... ok
+ test_submit_gone_wrong (tests.hooks.test_druid_hook.TestDruidHook) ... ok
+ test_submit_ok (tests.hooks.test_druid_hook.TestDruidHook) ... ok
+ test_submit_timeout (tests.hooks.test_druid_hook.TestDruidHook) ... ok
+ test_submit_unknown_response (tests.hooks.test_druid_hook.TestDruidHook) ... ok
+
+ ----------------------------------------------------------------------
+ Ran 8 tests in 3.036s
+
+ OK
+ ```
+
+ The Airflow code is mounted inside of the Docker container, so if you change something using your favorite IDE, you can directly test is in the container.
+
+3. Using [Docker Compose](https://docs.docker.com/compose/) and Airflow's CI scripts.
+
+ Start a docker container through Compose for development to avoid installing the packages directly on your system. The following will give you a shell inside a container, run all required service containers (MySQL, PostgresSQL, krb5 and so on) and install all the dependencies:
-1. The pull request should include tests, either as doctests, unit tests, or
-both. The airflow repo uses [Travis CI](https://travis-ci.org/apache/incubator-airflow)
-to run the tests and [codecov](https://codecov.io/gh/apache/incubator-airflow)
-to track coverage. You can set up both for free on your fork. It will
-help you making sure you do not break the build with your PR and that you help
-increase coverage.
-2. Please [rebase your fork](http://stackoverflow.com/a/7244456/1110993),
-squash commits, and resolve all conflicts.
-3. Every pull request should have an associated
-[JIRA](https://issues.apache.org/jira/browse/AIRFLOW/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
-The JIRA link should also be contained in the PR description.
-4. Preface your commit's subject & PR's title with **[AIRFLOW-XXX]**
-where *XXX* is the JIRA number. We compose release notes (i.e. for Airflow releases) from all commit titles in a release.
-By placing the JIRA number in the commit title and hence in the release notes,
-Airflow users can look into JIRA and Github PRs for more details about a particular change.
-5. Add an [Apache License](http://www.apache.org/legal/src-headers.html)
- header to all new files
-6. If the pull request adds functionality, the docs should be updated as part
-of the same PR. Doc string are often sufficient. Make sure to follow the
-Sphinx compatible standards.
-7. The pull request should work for Python 2.7 and 3.4. If you need help
-writing code that works in both Python 2 and 3, see the documentation at the
-[Python-Future project](http://python-future.org) (the future package is an
-Airflow requirement and should be used where possible).
-8. As Airflow grows as a project, we try to enforce a more consistent
-style and try to follow the Python community guidelines. We track this
-using [landscape.io](https://landscape.io/github/apache/incubator-airflow/),
-which you can setup on your fork as well to check before you submit your
-PR. We currently enforce most [PEP8](https://www.python.org/dev/peps/pep-0008/)
-and a few other linting rules. It is usually a good idea to lint locally
-as well using [flake8](https://flake8.readthedocs.org/en/latest/)
-using `flake8 airflow tests`. `git diff upstream/master -u -- "*.py" | flake8 --diff` will return any changed files in your branch that require linting.
-9. Please read this excellent [article](http://chris.beams.io/posts/git-commit/) on
-commit messages and adhere to them. It makes the lives of those who
-come after you a lot easier.
-
-### Testing locally
-
-#### TL;DR
-Tests can then be run with (see also the [Running unit tests](#running-unit-tests) section below):
-
- ./run_unit_tests.sh
-
-Individual test files can be run with:
-
- nosetests [path to file]
-
-#### Running unit tests
-
-We *highly* recommend setting up [Travis CI](https://travis-ci.org/) on
-your repo to automate this. It is free for open source projects. If for
-some reason you cannot, you can use the steps below to run tests.
-
-Here are loose guidelines on how to get your environment to run the unit tests.
-We do understand that no one out there can run the full test suite since
-Airflow is meant to connect to virtually any external system and that you most
-likely have only a subset of these in your environment. You should run the
-CoreTests and tests related to things you touched in your PR.
-
-To set up a unit test environment, first take a look at `run_unit_tests.sh` and
-understand that your ``AIRFLOW_CONFIG`` points to an alternate config file
-while running the tests. You shouldn't have to alter this config file but
-you may if need be.
-
-From that point, you can actually export these same environment variables in
-your shell, start an Airflow webserver ``airflow webserver -d`` and go and
-configure your connection. Default connections that are used in the tests
-should already have been created, you just need to point them to the systems
-where you want your tests to run.
-
-Once your unit test environment is setup, you should be able to simply run
+ ```bash
+ docker-compose -f scripts/ci/docker-compose.yml run airflow-testing bash
+ # From the container
+ pip install -e .[devel]
+ # Run all the tests with python and mysql through tox
+ pip install tox
+ tox -e py35-backend_mysql
+ ```
+
+ If you wish to run individual tests inside of Docker environment you can do as follows:
+
+ ```bash
+ # From the container (with your desired environment) with druid hook
+ tox -e py35-backend_mysql -- tests/hooks/test_druid_hook.py
+ ```
+
+
+### Running unit tests
+
+To run tests locally, once your unit test environment is setup (directly on your
+system or through our Docker setup) you should be able to simply run
``./run_unit_tests.sh`` at will.
For example, in order to just execute the "core" unit tests, run the following:
@@ -177,11 +192,151 @@ or a single test method:
./run_unit_tests.sh tests.core:CoreTest.test_check_operators -s --logging-level=DEBUG
```
+To run the whole test suite with Docker Compose, do:
+
+```
+# Install Docker Compose first, then this will run the tests
+docker-compose -f scripts/ci/docker-compose.yml run airflow-testing /app/scripts/ci/run-ci.sh
+```
+
+Alternatively can also set up [Travis CI](https://travis-ci.org/) on your repo to automate this.
+It is free for open source projects.
+
+Another great way of automating linting and testing is to use [Git Hooks](https://git-scm.com/book/uz/v2/Customizing-Git-Git-Hooks). For example you could create a `pre-commit` file based on the Travis CI Pipeline so that before each commit a local pipeline will be triggered and if this pipeline fails (returns an exit code other than `0`) the commit does not come through.
+This "in theory" has the advantage that you can not commit any code that fails that again reduces the errors in the Travis CI Pipelines.
+
+Since there are a lot of tests the script would last very long so you propably only should test your new feature locally.
+
+The following example of a `pre-commit` file allows you..
+- to lint your code via flake8
+- to test your code via nosetests in a docker container based on python 2
+- to test your code via nosetests in a docker container based on python 3
+
+```
+#!/bin/sh
+
+GREEN='\033[0;32m'
+NO_COLOR='\033[0m'
+
+setup_python_env() {
+ local venv_path=${1}
+
+ echo -e "${GREEN}Activating python virtual environment ${venv_path}..${NO_COLOR}"
+ source ${venv_path}
+}
+run_linting() {
+ local project_dir=$(git rev-parse --show-toplevel)
+
+ echo -e "${GREEN}Running flake8 over directory ${project_dir}..${NO_COLOR}"
+ flake8 ${project_dir}
+}
+run_testing_in_docker() {
+ local feature_path=${1}
+ local airflow_py2_container=${2}
+ local airflow_py3_container=${3}
+
+ echo -e "${GREEN}Running tests in ${feature_path} in airflow python 2 docker container..${NO_COLOR}"
+ docker exec -i -w /airflow/ ${airflow_py2_container} nosetests -v ${feature_path}
+ echo -e "${GREEN}Running tests in ${feature_path} in airflow python 3 docker container..${NO_COLOR}"
+ docker exec -i -w /airflow/ ${airflow_py3_container} nosetests -v ${feature_path}
+}
+
+set -e
+# NOTE: Before running this make sure you have set the function arguments correctly.
+setup_python_env /Users/feluelle/venv/bin/activate
+run_linting
+run_testing_in_docker tests/contrib/hooks/test_imap_hook.py dazzling_chatterjee quirky_stallman
+
+```
+
For more information on how to run a subset of the tests, take a look at the
nosetests docs.
See also the list of test classes and methods in `tests/core.py`.
+Feel free to customize based on the extras available in [setup.py](./setup.py)
+
+## Pull Request Guidelines
+
+Before you submit a pull request from your forked repo, check that it
+meets these guidelines:
+
+1. The pull request should include tests, either as doctests, unit tests, or both. The airflow repo uses [Travis CI](https://travis-ci.org/apache/incubator-airflow) to run the tests and [codecov](https://codecov.io/gh/apache/incubator-airflow) to track coverage. You can set up both for free on your fork (see the "Testing on Travis CI" section below). It will help you making sure you do not break the build with your PR and that you help increase coverage.
+1. Please [rebase your fork](http://stackoverflow.com/a/7244456/1110993), squash commits, and resolve all conflicts.
+1. Every pull request should have an associated [JIRA](https://issues.apache.org/jira/browse/AIRFLOW/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). The JIRA link should also be contained in the PR description.
+1. Preface your commit's subject & PR's title with **[AIRFLOW-XXX]** where *XXX* is the JIRA number. We compose release notes (i.e. for Airflow releases) from all commit titles in a release. By placing the JIRA number in the commit title and hence in the release notes, Airflow users can look into JIRA and Github PRs for more details about a particular change.
+1. Add an [Apache License](http://www.apache.org/legal/src-headers.html) header to all new files
+1. If the pull request adds functionality, the docs should be updated as part of the same PR. Doc string are often sufficient. Make sure to follow the Sphinx compatible standards.
+1. The pull request should work for Python 2.7 and 3.5. If you need help writing code that works in both Python 2 and 3, see the documentation at the [Python-Future project](http://python-future.org) (the future package is an Airflow requirement and should be used where possible).
+1. As Airflow grows as a project, we try to enforce a more consistent style and try to follow the Python community guidelines. We track this using [landscape.io](https://landscape.io/github/apache/incubator-airflow/), which you can setup on your fork as well to check before you submit your PR. We currently enforce most [PEP8](https://www.python.org/dev/peps/pep-0008/) and a few other linting rules. It is usually a good idea to lint locally as well using [flake8](https://flake8.readthedocs.org/en/latest/) using `flake8 airflow tests`. `git diff upstream/master -u -- "*.py" | flake8 --diff` will return any changed files in your branch that require linting.
+1. Please read this excellent [article](http://chris.beams.io/posts/git-commit/) on commit messages and adhere to them. It makes the lives of those who come after you a lot easier.
+
+### Testing on Travis CI
+
+We currently rely heavily on Travis CI for running the full Airflow test suite
+as running all of the tests locally requires significant setup. You can setup
+Travis CI in your fork of Airflow by following the
+[Travis CI Getting Started guide][travis-ci-getting-started].
+
+There are two different options available for running Travis CI which are
+setup as separate components on GitHub:
+
+1. **Travis CI GitHub App** (new version)
+1. **Travis CI GitHub Services** (legacy version)
+
+#### Travis CI GitHub App (new version)
+
+1. Once installed, you can configure the Travis CI GitHub App at
+https://github.com/settings/installations -> Configure Travis CI.
+
+1. For the Travis CI GitHub App, you can set repository access to either "All
+repositories" for convenience, or "Only select repositories" and choose
+`<username>/incubator-airflow` in the dropdown.
+
+1. You can access Travis CI for your fork at
+`https://travis-ci.com/<username>/incubator-airflow`.
+
+#### Travis CI GitHub Services (legacy version)
+
+The Travis CI GitHub Services versions uses an Authorized OAuth App. Note
+that `apache/incubator-airflow` is currently still using the legacy version.
+
+1. Once installed, you can configure the Travis CI Authorized OAuth App at
+https://github.com/settings/connections/applications/88c5b97de2dbfc50f3ac.
+
+1. If you are a GitHub admin, click the "Grant" button next to your
+organization; otherwise, click the "Request" button.
+
+1. For the Travis CI Authorized OAuth App, you may have to grant access to the
+forked `<organization>/incubator-airflow` repo even though it is public.
+
+1. You can access Travis CI for your fork at
+`https://travis-ci.org/<organization>/incubator-airflow`.
+
+#### Prefer travis-ci.com over travis-ci.org
+
+The travis-ci.org site for open source projects is now legacy and new projects
+should instead be created on travis-ci.com for both private repos and open
+source.
+
+Note that there is a second Authorized OAuth App available called "Travis CI
+for Open Source" used for the
+[legacy travis-ci.org service][travis-ci-org-vs-com]. It should not be used
+for new projects.
+
+More information:
+
+- [Open Source on travis-ci.com][travis-ci-open-source]
+- [Legacy GitHub Services to GitHub Apps Migration Guide][travis-ci-migrating]
+- [Migrating Multiple Repositories to GitHub Apps Guide][travis-ci-migrating-2]
+
+[travis-ci-getting-started]: https://docs.travis-ci.com/user/getting-started/
+[travis-ci-migrating-2]: https://docs.travis-ci.com/user/travis-migrate-to-apps-gem-guide/
+[travis-ci-migrating]: https://docs.travis-ci.com/user/legacy-services-to-github-apps-migration-guide/
+[travis-ci-open-source]: https://docs.travis-ci.com/user/open-source-on-travis-ci-com/
+[travis-ci-org-vs-com]: https://devops.stackexchange.com/a/4305/8830
+
+
### Changing the Metadata Database
When developing features the need may arise to persist information to the the
@@ -199,3 +354,80 @@ $ alembic revision -m "add new field to db"
Generating
~/airflow/airflow/migrations/versions/12341123_add_new_field_to_db.py
```
+
+## Setting up the node / npm javascript environment (ONLY FOR www_rbac)
+
+`airflow/www_rbac/` contains all npm-managed, front end assets.
+Flask-Appbuilder itself comes bundled with jQuery and bootstrap.
+While these may be phased out over time, these packages are currently not
+managed with npm.
+
+### Node/npm versions
+
+Make sure you are using recent versions of node and npm. No problems have been found with node>=8.11.3 and npm>=6.1.3
+
+### Using npm to generate bundled files
+
+#### npm
+
+First, npm must be available in your environment. If it is not you can run the following commands
+(taken from [this source](https://gist.github.com/DanHerbert/9520689))
+
+```
+brew install node --without-npm
+echo prefix=~/.npm-packages >> ~/.npmrc
+curl -L https://www.npmjs.com/install.sh | sh
+```
+
+The final step is to add `~/.npm-packages/bin` to your `PATH` so commands you install globally are usable.
+Add something like this to your `.bashrc` file, then `source ~/.bashrc` to reflect the change.
+
+```
+export PATH="$HOME/.npm-packages/bin:$PATH"
+```
+
+#### npm packages
+
+To install third party libraries defined in `package.json`, run the
+following within the `airflow/www_rbac/` directory which will install them in a
+new `node_modules/` folder within `www_rbac/`.
+
+```bash
+# from the root of the repository, move to where our JS package.json lives
+cd airflow/www_rbac/
+# run npm install to fetch all the dependencies
+npm install
+```
+
+To parse and generate bundled files for airflow, run either of the
+following commands. The `dev` flag will keep the npm script running and
+re-run it upon any changes within the assets directory.
+
+```
+# Compiles the production / optimized js & css
+npm run prod
+
+# Start a web server that manages and updates your assets as you modify them
+npm run dev
+```
+
+#### Upgrading npm packages
+
+Should you add or upgrade a npm package, which involves changing `package.json`, you'll need to re-run `npm install`
+and push the newly generated `package-lock.json` file so we get the reproducible build.
+
+#### Javascript Style Guide
+
+We try to enforce a more consistent style and try to follow the JS community guidelines.
+Once you add or modify any javascript code in the project, please make sure it follows the guidelines
+defined in [Airbnb JavaScript Style Guide](https://github.com/airbnb/javascript).
+Apache Airflow uses [ESLint](https://eslint.org/) as a tool for identifying and reporting on patterns in JavaScript,
+which can be used by running any of the following commands.
+
+```bash
+# Check JS code in .js and .html files, and report any errors/warnings
+npm run lint
+
+# Check JS code in .js and .html files, report any errors/warnings and fix them if possible
+npm run lint:fix
+```
diff --git a/DISCLAIMER b/DISCLAIMER
index 8fe69887c3..2758508789 100644
--- a/DISCLAIMER
+++ b/DISCLAIMER
@@ -1 +1,6 @@
-Apache Airflow is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+Apache Airflow is an effort undergoing incubation at The Apache Software Foundation (ASF),
+sponsored by the Apache Incubator. Incubation is required of all newly accepted projects
+until a further review indicates that the infrastructure, communications, and decision
+making process have stabilized in a manner consistent with other successful ASF projects.
+While incubation status is not necessarily a reflection of the completeness or stability
+of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
diff --git a/INSTALL b/INSTALL
index 7e5bf50a60..b018839ab1 100644
--- a/INSTALL
+++ b/INSTALL
@@ -1,9 +1,30 @@
-# INSTALL / BUILD instruction for Apache Airflow (incubating)
-# fetch the tarball and untar the source
+# INSTALL / BUILD instructions for Apache Airflow (incubating)
+
+# [required] fetch the tarball and untar the source
+# change into the directory that was untarred.
# [optional] run Apache RAT (release audit tool) to validate license headers
-# RAT docs here: https://creadur.apache.org/rat/
+# RAT docs here: https://creadur.apache.org/rat/. Requires Java and Apache Rat
java -jar apache-rat.jar -E ./.rat-excludes -d .
-# install the release
+# [optional] Airflow pulls in quite a lot of dependencies in order
+# to connect to other services. You might want to test or run Airflow
+# from a virtual env to make sure those dependencies are separated
+# from your system wide versions
+python -m my_env
+source my_env/bin/activate
+
+# [required] by default one of Apache Airflow's dependencies pulls in a GPL
+# library. Airflow will not install (and upgrade) without an explicit choice.
+#
+# To make sure not to install the GPL dependency:
+# export SLUGIFY_USES_TEXT_UNIDECODE=yes
+# In case you do not mind:
+# export AIRFLOW_GPL_UNIDECODE=yes
+
+# [required] building and installing
+# by pip (preferred)
+pip install .
+
+# or directly
python setup.py install
diff --git a/LICENSE b/LICENSE
index d46809475e..debddc0d7b 100644
--- a/LICENSE
+++ b/LICENSE
@@ -209,33 +209,55 @@ limitations under the License.
licenses.
+========================================================================
+Third party Apache 2.0 licenses
+========================================================================
+
+The following components are provided under the Apache 2.0 License.
+See project link for details. The text of each license is also included
+at licenses/LICENSE-[project].txt.
+
+ (ALv2 License) hue v4.3.0 (https://github.com/cloudera/hue/)
+ (ALv2 License) jqclock v2.3.0 (https://github.com/JohnRDOrazio/jQuery-Clock-Plugin)
+ (ALv2 License) bootstrap3-typeahead v4.0.2 (https://github.com/bassjobsen/Bootstrap-3-Typeahead)
+ (ALv2 License) airflow.contrib.auth.backends.github_enterprise_auth
+
========================================================================
MIT licenses
========================================================================
The following components are provided under the MIT License. See project link for details.
-The text of each license is also included at licenses/LICENSE-[project]-[version].txt.
-
- (MIT License) jquery (https://jquery.org/license/)
- (MIT License) dagre-d3 (https://github.com/cpettitt/dagre-d3)
- (MIT License) bootstrap (https://github.com/twbs/bootstrap/blob/v3-dev/LICENSE)
- (MIT License) d3-tip (https://github.com/Caged/d3-tip)
- (MIT License) dataTables (https://datatables.net)
- (MIT License) Clock Plugin (https://github.com/Lwangaman/jQuery-Clock-Plugin)
- (MIT License) WebGL-2D (https://github.com/gameclosure/webgl-2d)
- (MIT License) Underscore.js (http://underscorejs.org)
- (MIT License) Bootstrap Toggle (http://www.bootstraptoggle.com)
- (MIT License) normalize.css (http://git.io/normalize)
+The text of each license is also included at licenses/LICENSE-[project].txt.
+
+ (MIT License) jquery v2.1.4 (https://jquery.org/license/)
+ (MIT License) dagre-d3 v0.6.1 (https://github.com/cpettitt/dagre-d3)
+ (MIT License) bootstrap v3.2 (https://github.com/twbs/bootstrap/)
+ (MIT License) d3-tip v0.6.3 (https://github.com/Caged/d3-tip)
+ (MIT License) dataTables v1.10.10 (https://datatables.net)
+ (MIT License) WebGL-2D (git-commit 9a7ec26) (https://github.com/gameclosure/webgl-2d)
+ (MIT License) Underscorejs v1.5.0 (http://underscorejs.org)
+ (MIT License) Bootstrap Toggle v2.2.0 (http://www.bootstraptoggle.com)
+ (MIT License) normalize.css v3.0.2 (http://necolas.github.io/normalize.css/)
+ (MIT License) ElasticMock v1.3.2 (https://github.com/vrcmarcos/elasticmock)
+ (MIT License) MomentJS v2.22.2 (http://momentjs.com/)
========================================================================
BSD 2-Clause licenses
========================================================================
The following components are provided under the BSD 2-Clause license.
See file headers and project links for details.
-The text of each license is also included at licenses/LICENSE-[project]-[version].txt.
+The text of each license is also included at licenses/LICENSE-[project].txt.
+
+ (BSD 2 License) flask-kerberos v1.0.4 (https://github.com/mkomitee/flask-kerberos)
+
+========================================================================
+BSD 3-Clause licenses
+========================================================================
+The following components are provided under the BSD 3-Clause license. See project links for details.
+The text of each license is also included at licenses/LICENSE-[project].txt.
- (BSD 2 License) flask-kerberos (https://github.com/mkomitee/flask-kerberos)
- (BSD 2 License) Ace (https://github.com/ajaxorg/ace)
- (BSD 2 License) d3js (https://d3js.org)
- (BSD 3 License) parallel.js (https://parallel.js.org/)
+ (BSD 3 License) Ace v1.1.8 (https://github.com/ajaxorg/ace)
+ (BSD 3 License) d3js v3.5.17 (https://d3js.org)
+ (BSD 3 License) parallel-coordinates v0.7.0 (http://syntagmatic.github.com/parallel-coordinates/)
+ (BSD 3 License) scikit-learn v0.19.1 (https://github.com/scikit-learn/scikit-learn)
diff --git a/MANIFEST.in b/MANIFEST.in
index 69ccafec80..ec99c1f6b2 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1,24 +1,35 @@
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
include NOTICE
include LICENSE
include DISCLAIMER
include CHANGELOG.txt
include README.md
+graft licenses/
graft airflow/www/templates
graft airflow/www/static
+graft airflow/www_rbac
+graft airflow/www_rbac/static
+graft airflow/www_rbac/templates
+graft airflow/www_rbac/translations
include airflow/alembic.ini
graft scripts/systemd
graft scripts/upstart
graft airflow/config_templates
+recursive-exclude airflow/www_rbac/node_modules *
diff --git a/NOTICE b/NOTICE
index b1e78ad9c3..a642546efb 100644
--- a/NOTICE
+++ b/NOTICE
@@ -1,6 +1,19 @@
Apache Airflow
-Copyright 2016 and onwards The Apache Software Foundation
-
+Copyright 2016-2018 The Apache Software Foundation
This product includes software developed at The Apache Software
Foundation (http://www.apache.org/).
+
+=======================================================================
+
+airflow.contrib.auth.backends.github_enterprise_auth:
+-----------------------------------------------------
+
+* Copyright 2015 Matthew Pelland (matt@pelland.io)
+
+hue:
+-----
+This product contains a modified portion of 'Hue' developed by Cloudera, Inc.
+(https://github.com/cloudera/hue/).
+
+* Copyright 2009-2017 Cloudera Inc.
diff --git a/README.md b/README.md
index 6659bf1deb..0f7c36fdc7 100644
--- a/README.md
+++ b/README.md
@@ -1,28 +1,44 @@
-# Airflow
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Apache Airflow (Incubating)
[](https://badge.fury.io/py/apache-airflow)
[](https://travis-ci.org/apache/incubator-airflow)
[](https://codecov.io/github/apache/incubator-airflow?branch=master)
-[](http://pythonhosted.org/airflow/)
-[](https://gitter.im/apache/incubator-airflow?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[](https://airflow.readthedocs.io/en/latest/?badge=latest)
+[](http://www.apache.org/licenses/LICENSE-2.0.txt)
+[](https://pypi.org/project/apache-airflow/)
+[](https://twitter.com/ApacheAirflow)
-_NOTE: The transition from 1.8.0 (or before) to 1.8.1 (or after) requires uninstalling Airflow before installing the new version. The package name was changed from `airflow` to `apache-airflow` as of version 1.8.1._
-
-Airflow is a platform to programmatically author, schedule, and monitor
-workflows.
+Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.
When workflows are defined as code, they become more maintainable,
versionable, testable, and collaborative.
-Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks.
-The Airflow scheduler executes your tasks on an array of workers while
-following the specified dependencies. Rich command line utilities make
-performing complex surgeries on DAGs a snap. The rich user interface
-makes it easy to visualize pipelines running in production,
-monitor progress, and troubleshoot issues when needed.
+Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.
## Getting started
-Please visit the Airflow Platform documentation for help with [installing Airflow](http://pythonhosted.org/airflow/installation.html), getting a [quick start](http://pythonhosted.org/airflow/start.html), or a more complete [tutorial](http://pythonhosted.org/airflow/tutorial.html).
+
+Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.incubator.apache.org/installation.html), getting a [quick start](https://airflow.incubator.apache.org/start.html), or a more complete [tutorial](https://airflow.incubator.apache.org/tutorial.html).
+
+Documentation of GitHub master (latest development branch): [ReadTheDocs Documentation](https://airflow.readthedocs.io/en/latest/)
For further information, please visit the [Airflow Wiki](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home).
@@ -46,137 +62,225 @@ unit of work and continuity.
- **Dynamic**: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.
- **Extensible**: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
- **Elegant**: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine.
-- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity.
+- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers.
## User Interface
- **DAGs**: Overview of all DAGs in your environment.
-
+
+ 
- **Tree View**: Tree representation of a DAG that spans across time.
-
+
+ 
- **Graph View**: Visualization of a DAG's dependencies and their current status for a specific run.
-
+
+ 
- **Task Duration**: Total time spent on different tasks over time.
-
+
+ 
- **Gantt View**: Duration and overlap of a DAG.
-
+
+ 
- **Code View**: Quick way to view source code of a DAG.
-
-## Who uses Airflow?
+ 
+
+
+## Contributing
+
+Want to help build Apache Airflow? Check out our [contributing documentation](https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md).
-As the Airflow community grows, we'd like to keep track of who is using
-the platform. Please send a PR with your company name and @githubhandle
-if you may.
-Committers:
+## Who uses Apache Airflow?
-* Refer to [Committers](https://cwiki.apache.org/confluence/display/AIRFLOW/Committers)
+As the Apache Airflow community grows, we'd like to keep track of who is using
+the platform. Please send a PR with your company name and @githubhandle
+if you may.
Currently **officially** using Airflow:
-1. [Airbnb](http://airbnb.io/) [[@mistercrunch](https://github.com/mistercrunch), [@artwr](https://github.com/artwr)]
+1. [6play](https://www.6play.fr) [[@lemourA](https://github.com/lemoura), [@achaussende](https://github.com/achaussende), [@d-nguyen](https://github.com/d-nguyen), [@julien-gm](https://github.com/julien-gm)]
+1. [8fit](https://8fit.com/) [[@nicor88](https://github.com/nicor88), [@frnzska](https://github.com/frnzska)]
+1. [90 Seconds](https://90seconds.tv/) [[@aaronmak](https://github.com/aaronmak)]
+1. [99](https://99taxis.com) [[@fbenevides](https://github.com/fbenevides), [@gustavoamigo](https://github.com/gustavoamigo) & [@mmmaia](https://github.com/mmmaia)]
+1. [AdBOOST](https://www.adboost.sk) [[AdBOOST](https://github.com/AdBOOST)]
1. [Agari](https://github.com/agaridata) [[@r39132](https://github.com/r39132)]
+1. [Airbnb](http://airbnb.io/) [[@mistercrunch](https://github.com/mistercrunch), [@artwr](https://github.com/artwr)]
+1. [AirDNA](https://www.airdna.co)
+1. [Airtel](https://www.airtel.in/) [[@harishbisht](https://github.com/harishbisht)]
+1. [Alan](https://alan.eu) [[@charles-go](https://github.com/charles-go)]
1. [allegro.pl](http://allegro.tech/) [[@kretes](https://github.com/kretes)]
1. [AltX](https://www.getaltx.com/about) [[@pedromduarte](https://github.com/pedromduarte)]
1. [Apigee](https://apigee.com) [[@btallman](https://github.com/btallman)]
1. [ARGO Labs](http://www.argolabs.org) [[California Data Collaborative](https://github.com/California-Data-Collaborative)]
+1. [ARMEDANGELS](https://www.armedangels.de) [[@swiffer](https://github.com/swiffer)]
+1. [Arquivei](https://www.arquivei.com.br/) [[@arquivei](https://github.com/arquivei)]
1. [Astronomer](http://www.astronomer.io) [[@schnie](https://github.com/schnie), [@andscoop](https://github.com/andscoop), [@tedmiston](https://github.com/tedmiston), [@benjamingregory](https://github.com/benjamingregory)]
1. [Auth0](https://auth0.com) [[@sicarul](https://github.com/sicarul)]
1. [Away](https://awaytravel.com) [[@trunsky](https://github.com/trunsky)]
-1. [BalanceHero](http://truebalance.io/) [[@swalloow](https://github.com/swalloow)]
1. [Azri Solutions](http://www.azrisolutions.com/) [[@userimack](https://github.com/userimack)]
+1. [BalanceHero](http://truebalance.io/) [[@swalloow](https://github.com/swalloow)]
+1. [Banco de Formaturas](https://www.bancodeformaturas.com.br) [[@guiligan](https://github.com/guiligan)]
1. [BandwidthX](http://www.bandwidthx.com) [[@dineshdsharma](https://github.com/dineshdsharma)]
+1. [BBM](https://www.bbm.com/)
1. [Bellhops](https://github.com/bellhops)
+1. [BelugaDB](https://belugadb.com) [[@fabio-nukui](https://github.com/fabio-nukui) & [@joao-sallaberry](http://github.com/joao-sallaberry) & [@lucianoviola](https://github.com/lucianoviola) & [@tmatuki](https://github.com/tmatuki)]
+1. [Betterment](https://www.betterment.com/) [[@betterment](https://github.com/Betterment)]
1. [BlaBlaCar](https://www.blablacar.com) [[@puckel](https://github.com/puckel) & [@wmorin](https://github.com/wmorin)]
1. [Bloc](https://www.bloc.io) [[@dpaola2](https://github.com/dpaola2)]
-1. [BlueApron](https://www.blueapron.com) [[@jasonjho](https://github.com/jasonjho) & [@matthewdavidhauser](https://github.com/matthewdavidhauser)]
1. [Blue Yonder](http://www.blue-yonder.com) [[@blue-yonder](https://github.com/blue-yonder)]
+1. [BlueApron](https://www.blueapron.com) [[@jasonjho](https://github.com/jasonjho) & [@matthewdavidhauser](https://github.com/matthewdavidhauser)]
+1. [Bluecore](https://www.bluecore.com) [[@JLDLaughlin](https://github.com/JLDLaughlin)]
+1. [Boda Telecom Suite - CE](https://github.com/bodastage/bts-ce) [[@erssebaggala](https://github.com/erssebaggala), [@bodastage](https://github.com/bodastage)]
+1. [Bodastage Solutions](http://bodastage.com) [[@erssebaggala](https://github.com/erssebaggala), [@bodastage](https://github.com/bodastage)]
+1. [Bombora Inc](https://bombora.com/) [[@jeffkpayne](https://github.com/jeffkpayne), [@TheOriginalAlex](https://github.com/TheOriginalAlex)]
+1. [Bonnier Broadcasting](http://www.bonnierbroadcasting.com) [[@wileeam](https://github.com/wileeam)]
+1. [BounceX](http://www.bouncex.com) [[@JoshFerge](https://github.com/JoshFerge), [@hudsonrio](https://github.com/hudsonrio), [@ronniekritou](https://github.com/ronniekritou)]
+1. [Branch](https://branch.io) [[@sdebarshi](https://github.com/sdebarshi), [@dmitrig01](https://github.com/dmitrig01)]
1. [California Data Collaborative](https://github.com/California-Data-Collaborative) powered by [ARGO Labs](http://www.argolabs.org)
1. [Carbonite](https://www.carbonite.com) [[@ajbosco](https://github.com/ajbosco)]
+1. [CarLabs](https://www.carlabs.ai/) [[@sganz](https://github.com/sganz) & [@odannyc](https://github.com/odannyc)]
+1. [CAVA](https://www.cava.com) [[@minh5](http://github.com/minh5)]
1. [Celect](http://www.celect.com) [[@superdosh](https://github.com/superdosh) & [@chadcelect](https://github.com/chadcelect)]
+1. [Censys](https://censys.io) [[@zakird](https://github.com/zakird), [@dadrian](https://github.com/dadrian), & [@andrewsardone](https://github.com/andrewsardone)]
1. [Change.org](https://www.change.org) [[@change](https://github.com/change), [@vijaykramesh](https://github.com/vijaykramesh)]
+1. [Chartboost](https://www.chartboost.com) [[@cgelman](https://github.com/cgelman) & [@dclubb](https://github.com/dclubb)]
1. [Checkr](https://checkr.com) [[@tongboh](https://github.com/tongboh)]
1. [Children's Hospital of Philadelphia Division of Genomic Diagnostics](http://www.chop.edu/centers-programs/division-genomic-diagnostics) [[@genomics-geek]](https://github.com/genomics-geek/)
+1. [Cinimex DataLab](http://cinimex.ru) [[@kdubovikov](https://github.com/kdubovikov)]
1. [City of San Diego](http://sandiego.gov) [[@MrMaksimize](https://github.com/mrmaksimize), [@andrell81](https://github.com/andrell81) & [@arnaudvedy](https://github.com/arnaudvedy)]
+1. [Civey](https://civey.com/) [[@WesleyBatista](https://github.com/WesleyBatista)]
1. [Clairvoyant](https://clairvoyantsoft.com) [@shekharv](https://github.com/shekharv)
1. [Clover Health](https://www.cloverhealth.com) [[@gwax](https://github.com/gwax) & [@vansivallab](https://github.com/vansivallab)]
-1. [Chartboost](https://www.chartboost.com) [[@cgelman](https://github.com/cgelman) & [@dclubb](https://github.com/dclubb)]
+1. [Collectivehealth Inc.](https://www.collectivehealth.com) [@retornam](https://github.com/retornam)
+1. [Compass](https://www.compass.com) [[@wdhorton](https://github.com/wdhorton)]
1. [ContaAzul](https://www.contaazul.com) [[@bern4rdelli](https://github.com/bern4rdelli), [@renanleme](https://github.com/renanleme) & [@sabino](https://github.com/sabino)]
1. [Cotap](https://github.com/cotap/) [[@maraca](https://github.com/maraca) & [@richardchew](https://github.com/richardchew)]
-1. [CreditCards.com](https://www.creditcards.com/)[[@vmAggies](https://github.com/vmAggies) & [@jay-wallaby](https://github.com/jay-wallaby)]
+1. [Craig@Work](https://www.craigatwork.com)
1. [Credit Karma](https://www.creditkarma.com/) [[@preete-dixit-ck](https://github.com/preete-dixit-ck) & [@harish-gaggar-ck](https://github.com/harish-gaggar-ck) & [@greg-finley-ck](https://github.com/greg-finley-ck)]
1. [Creditas](https://www.creditas.com.br) [[@dcassiano](https://github.com/dcassiano)]
-1. [DataFox](https://www.datafox.com/) [[@sudowork](https://github.com/sudowork)]
+1. [CreditCards.com](https://www.creditcards.com/)[[@vmAggies](https://github.com/vmAggies) & [@jay-wallaby](https://github.com/jay-wallaby)]
+1. [Custom Ink](https://www.customink.com/) [[@david-dalisay](https://github.com/david-dalisay), [@dmartin11](https://github.com/dmartin11) & [@mpeteuil](https://github.com/mpeteuil)]
+1. [Dailymotion](http://www.dailymotion.com/fr) [[@germaintanguy](https://github.com/germaintanguy) & [@hc](https://github.com/hc)]
+1. [Danamica](https://www.danamica.dk) [[@testvinder](https://github.com/testvinder)]
1. [Data Reply](https://www.datareply.co.uk/) [[@kaxil](https://github.com/kaxil)]
+1. [DataCamp](https://datacamp.com/) [[@dgrtwo](https://github.com/dgrtwo)]
+1. [DataFox](https://www.datafox.com/) [[@sudowork](https://github.com/sudowork)]
1. [Digital First Media](http://www.digitalfirstmedia.com/) [[@duffn](https://github.com/duffn) & [@mschmo](https://github.com/mschmo) & [@seanmuth](https://github.com/seanmuth)]
+1. [DigitalOcean](https://digitalocean.com/) [[@ajbosco](https://github.com/ajbosco)]
+1. [DocuTAP](https://www.docutap.com/) [[@jshvrsn](https://github.com/jshvrsn) & [@lhvphan](https://github.com/lhvphan) & [@cloneluke](https://github.com/cloneluke)]
+1. [DoorDash](https://www.doordash.com/)
+1. [Dotmodus](http://dotmodus.com) [[@dannylee12](https://github.com/dannylee12)]
1. [Drivy](https://www.drivy.com) [[@AntoineAugusti](https://github.com/AntoineAugusti)]
-1. [Easy Taxi](http://www.easytaxi.com/) [[@caique-lima](https://github.com/caique-lima) & [@WesleyBatista](https://github.com/WesleyBatista)]
+1. [Easy Taxi](http://www.easytaxi.com/) [[@caique-lima](https://github.com/caique-lima) & [@diraol](https://github.com/diraol)]
+1. [Enigma](https://www.enigma.com) [[@hydrosquall](https://github.com/hydrosquall)]
1. [eRevalue](https://www.datamaran.com) [[@hamedhsn](https://github.com/hamedhsn)]
+1. [Etsy](https://www.etsy.com) [[@mchalek](https://github.com/mchalek)]
1. [evo.company](https://evo.company/) [[@orhideous](https://github.com/orhideous)]
+1. [Fathom Health](https://www.fathomhealth.co/)
+1. [Flipp](https://www.flipp.com) [[@sethwilsonwishabi](https://github.com/sethwilsonwishabi)]
+1. [Format](https://www.format.com) [[@format](https://github.com/4ormat) & [@jasonicarter](https://github.com/jasonicarter)]
1. [FreshBooks](https://github.com/freshbooks) [[@DinoCow](https://github.com/DinoCow)]
1. [Fundera](https://fundera.com) [[@andyxhadji](https://github.com/andyxhadji)]
+1. [G Adventures](https://gadventures.com) [[@samuelmullin](https://github.com/samuelmullin)]
1. [GameWisp](https://gamewisp.com) [[@tjbiii](https://github.com/TJBIII) & [@theryanwalls](https://github.com/theryanwalls)]
+1. [GeneCards](https://www.genecards.org) [[@oferze](https://github.com/oferze)]
1. [Gentner Lab](http://github.com/gentnerlab) [[@neuromusic](https://github.com/neuromusic)]
-1. [Glassdoor](https://github.com/Glassdoor) [[@syvineckruyk](https://github.com/syvineckruyk)]
+1. [Get Simpl](https://getsimpl.com/) [[@rootcss](https://github.com/rootcss)]
+1. [Glassdoor](https://github.com/Glassdoor) [[@syvineckruyk](https://github.com/syvineckruyk) & [@sid88in](https://github.com/sid88in)]
1. [Global Fashion Group](http://global-fashion-group.com) [[@GFG](https://github.com/GFG)]
1. [GovTech GDS](https://gds-gov.tech) [[@chrissng](https://github.com/chrissng) & [@datagovsg](https://github.com/datagovsg)]
+1. [Gradeup](https://gradeup.co) [[@gradeup](https://github.com/gradeup)]
1. [Grand Rounds](https://www.grandrounds.com/) [[@richddr](https://github.com/richddr), [@timz1290](https://github.com/timz1290), [@wenever](https://github.com/@wenever), & [@runongirlrunon](https://github.com/runongirlrunon)]
1. [Groupalia](http://es.groupalia.com) [[@jesusfcr](https://github.com/jesusfcr)]
+1. [Groupon](https://groupon.com) [[@stevencasey](https://github.com/stevencasey)]
1. [Gusto](https://gusto.com) [[@frankhsu](https://github.com/frankhsu)]
1. [Handshake](https://joinhandshake.com/) [[@mhickman](https://github.com/mhickman)]
1. [Handy](http://www.handy.com/careers/73115?gh_jid=73115&gh_src=o5qcxn) [[@marcintustin](https://github.com/marcintustin) / [@mtustin-handy](https://github.com/mtustin-handy)]
-1. [Healthjump](http://www.healthjump.com/) [[@miscbits](https://github.com/miscbits)]
+1. [happn](https://www.happn.com) [[@PierreCORBEL](https://github.com/PierreCORBEL)]
+1. [HAVAN](https://www.havan.com.br) [[@botbiz](https://github.com/botbiz)]
+1. [HBC Digital](http://tech.hbc.com) [[@tmccartan](https://github.com/tmccartan) & [@dmateusp](https://github.com/dmateusp)]
1. [HBO](http://www.hbo.com/)[[@yiwang](https://github.com/yiwang)]
+1. [Healthjump](http://www.healthjump.com/) [[@miscbits](https://github.com/miscbits)]
1. [HelloFresh](https://www.hellofresh.com) [[@tammymendt](https://github.com/tammymendt) & [@davidsbatista](https://github.com/davidsbatista) & [@iuriinedostup](https://github.com/iuriinedostup)]
+1. [Hipages](https://www.hipages.com.au/) [[@arihantsurana](https://github.com/arihantsurana)]
1. [Holimetrix](http://holimetrix.com/) [[@thibault-ketterer](https://github.com/thibault-ketterer)]
1. [Hootsuite](https://github.com/hootsuite)
1. [Hostnfly](https://www.hostnfly.com/) [[@CyrilLeMat](https://github.com/CyrilLeMat) & [@pierrechopin](https://github.com/pierrechopin) & [@alexisrosuel](https://github.com/alexisrosuel)]
1. [HotelQuickly](https://github.com/HotelQuickly) [[@zinuzoid](https://github.com/zinuzoid)]
+1. [Iflix](https://piay.iflix.com) [[@ChaturvediSulabh](https://github.com/ChaturvediSulabh)]
1. [IFTTT](https://www.ifttt.com/) [[@apurvajoshi](https://github.com/apurvajoshi)]
1. [iHeartRadio](http://www.iheart.com/)[[@yiwang](https://github.com/yiwang)]
1. [imgix](https://www.imgix.com/) [[@dclubb](https://github.com/dclubb)]
1. [ING](http://www.ing.com/)
+1. [Intercom](http://www.intercom.com/) [[@fox](https://github.com/fox) & [@paulvic](https://github.com/paulvic)]
+1. [Investorise](https://investorise.com/) [[@svenvarkel](https://github.com/svenvarkel)]
1. [Jampp](https://github.com/jampp)
+1. [Jeitto](https://www.jeitto.com.br) [[@BrennerPablo](https://github.com/BrennerPablo) & [@ds-mauri](https://github.com/ds-mauri)]
+1. [Jetlore](http://www.jetlore.com/)
1. [JobTeaser](https://www.jobteaser.com) [[@stefani75](https://github.com/stefani75) & [@knil-sama](https://github.com/knil-sama)]
+1. [Kalibrr](https://www.kalibrr.com/) [[@charlesverdad](https://github.com/charlesverdad)]
+1. [Karmic](https://karmiclabs.com) [[@hyw](https://github.com/hyw)]
+1. [King](https://king.com) [[@nathadfield](https://github.com/nathadfield)]
1. [Kiwi.com](https://kiwi.com/) [[@underyx](https://github.com/underyx)]
1. [Kogan.com](https://github.com/kogan) [[@geeknam](https://github.com/geeknam)]
+1. [KPN B.V.](https://www.kpn.com/) [[@biyanisuraj](https://github.com/biyanisuraj) & [@gmic](https://github.com/gmic)]
1. [Lemann Foundation](http://fundacaolemann.org.br) [[@fernandosjp](https://github.com/fernandosjp)]
+1. [LeMans Corporation](https://www.parts-unlimited.com/) [[@alloydwhitlock](https://github.com/alloydwhitlock)] & [[@tinyrye](https://github.com/tinyrye)]
1. [LendUp](https://www.lendup.com/) [[@lendup](https://github.com/lendup)]
1. [LetsBonus](http://www.letsbonus.com) [[@jesusfcr](https://github.com/jesusfcr) & [@OpringaoDoTurno](https://github.com/OpringaoDoTurno)]
+1. [Liberty Global](https://www.libertyglobal.com/) [[@LibertyGlobal](https://github.com/LibertyGlobal/)]
1. [liligo](http://liligo.com/) [[@tromika](https://github.com/tromika)]
1. [LingoChamp](http://www.liulishuo.com/) [[@haitaoyao](https://github.com/haitaoyao)]
1. [Lucid](http://luc.id) [[@jbrownlucid](https://github.com/jbrownlucid) & [@kkourtchikov](https://github.com/kkourtchikov)]
1. [Lumos Labs](https://www.lumosity.com/) [[@rfroetscher](https://github.com/rfroetscher/) & [@zzztimbo](https://github.com/zzztimbo/)]
-1. [Lyft](https://www.lyft.com/)[[@SaurabhBajaj](https://github.com/SaurabhBajaj)]
+1. [Lyft](https://www.lyft.com/)[[@feng-tao](https://github.com/feng-tao)]
+1. [M4U](https://www.m4u.com.br/) [[@msantino](https://github.com/msantino)]
1. [Madrone](http://madroneco.com/) [[@mbreining](https://github.com/mbreining) & [@scotthb](https://github.com/scotthb)]
1. [Markovian](https://markovian.com/) [[@al-xv](https://github.com/al-xv), [@skogsbaeck](https://github.com/skogsbaeck), [@waltherg](https://github.com/waltherg)]
1. [Mercadoni](https://www.mercadoni.com.co) [[@demorenoc](https://github.com/demorenoc)]
1. [Mercari](http://www.mercari.com/) [[@yu-iskw](https://github.com/yu-iskw)]
-1. [MiNODES](https://www.minodes.com) [[@dice89](https://github.com/dice89), [@diazcelsa](https://github.com/diazcelsa)]
1. [MFG Labs](https://github.com/MfgLabs)
+1. [MiNODES](https://www.minodes.com) [[@dice89](https://github.com/dice89), [@diazcelsa](https://github.com/diazcelsa)]
+1. [Modernizing Medicine](https://www.modmed.com/)[[@kehv1n](https://github.com/kehv1n), [@dalupus](https://github.com/dalupus)]
+1. [Multiply](https://www.multiply.com) [[@nrhvyc](https://github.com/nrhvyc)]
1. [mytaxi](https://mytaxi.com) [[@mytaxi](https://github.com/mytaxi)]
+1. [Neoway](https://www.neoway.com.br/) [[@neowaylabs](https://github.com/orgs/NeowayLabs/people)]
1. [Nerdwallet](https://www.nerdwallet.com)
1. [New Relic](https://www.newrelic.com) [[@marcweil](https://github.com/marcweil)]
1. [Newzoo](https://www.newzoo.com) [[@newzoo-nexus](https://github.com/newzoo-nexus)]
+1. [NEXT Trucking](https://www.nexttrucking.com/) [[@earthmancash2](https://github.com/earthmancash2), [@kppullin](https://github.com/kppullin)]
1. [Nextdoor](https://nextdoor.com) [[@SivaPandeti](https://github.com/SivaPandeti), [@zshapiro](https://github.com/zshapiro) & [@jthomas123](https://github.com/jthomas123)]
+1. [OdysseyPrime](https://www.goprime.io/) [[@davideberdin](https://github.com/davideberdin)]
1. [OfferUp](https://offerupnow.com)
1. [OneFineStay](https://www.onefinestay.com) [[@slangwald](https://github.com/slangwald)]
1. [Open Knowledge International](https://okfn.org) [@vitorbaptista](https://github.com/vitorbaptista)
-1. [Pandora Media](https://www.pandora.com/) [[@Acehaidrey](https://github.com/Acehaidrey)]
+1. [Overstock](https://www.github.com/overstock) [[@mhousley](https://github.com/mhousley) & [@mct0006](https://github.com/mct0006)]
+1. [Pagar.me](https://pagar.me/) [[@pagarme](https://github.com/pagarme)
+1. [Pandora Media](https://www.pandora.com/) [[@Acehaidrey](https://github.com/Acehaidrey) & [@wolfier](https://github.com/wolfier)]
1. [PAYMILL](https://www.paymill.com/) [[@paymill](https://github.com/paymill) & [@matthiashuschle](https://github.com/matthiashuschle)]
1. [PayPal](https://www.paypal.com/) [[@r39132](https://github.com/r39132) & [@jhsenjaliya](https://github.com/jhsenjaliya)]
+1. [Pernod-Ricard](https://www.pernod-ricard.com/) [[@romain-nio](https://github.com/romain-nio)]
+1. [Plaid](https://www.plaid.com/) [[@plaid](https://github.com/plaid), [@AustinBGibbons](https://github.com/AustinBGibbons) & [@jeeyoungk](https://github.com/jeeyoungk)]
1. [Playbuzz](https://www.playbuzz.com/) [[@clintonboys](https://github.com/clintonboys) & [@dbn](https://github.com/dbn)]
+1. [PMC](https://pmc.com/) [[@andrewm4894](https://github.com/andrewm4894)]
+1. [Poshmark](https://www.poshmark.com)
1. [Postmates](http://www.postmates.com) [[@syeoryn](https://github.com/syeoryn)]
1. [Pronto Tools](http://www.prontotools.io/) [[@zkan](https://github.com/zkan) & [@mesodiar](https://github.com/mesodiar)]
+1. [Publicis Pixelpark](https://www.publicispixelpark.de/) [[@feluelle](https://github.com/feluelle)]
+1. [PubNub](https://pubnub.com) [[@jzucker2](https://github.com/jzucker2)]
+1. [Qplum](https://qplum.co) [[@manti](https://github.com/manti)]
+1. [Quantopian](https://www.quantopian.com/) [[@eronarn](http://github.com/eronarn)]
1. [Qubole](https://qubole.com) [[@msumit](https://github.com/msumit)]
1. [Quizlet](https://quizlet.com) [[@quizlet](https://github.com/quizlet)]
1. [Quora](https://www.quora.com/)
+1. [REA Group](https://www.rea-group.com/)
+1. [Reddit](https://www.reddit.com/) [[@reddit](https://github.com/reddit/)]
1. [Robinhood](https://robinhood.com) [[@vineet-rh](https://github.com/vineet-rh)]
1. [Scaleway](https://scaleway.com) [[@kdeldycke](https://github.com/kdeldycke)]
1. [Sense360](https://github.com/Sense360) [[@kamilmroczek](https://github.com/KamilMroczek)]
@@ -184,35 +288,66 @@ Currently **officially** using Airflow:
1. [Sidecar](https://hello.getsidecar.com/) [[@getsidecar](https://github.com/getsidecar)]
1. [SimilarWeb](https://www.similarweb.com/) [[@similarweb](https://github.com/similarweb)]
1. [SmartNews](https://www.smartnews.com/) [[@takus](https://github.com/takus)]
+1. [SocialCops](https://www.socialcops.com/) [[@vinayak-mehta](https://github.com/vinayak-mehta) & [@sharky93](https://github.com/sharky93)]
+1. [Société générale](https://www.societegenerale.fr/) [[@medmrgh](https://github.com/medmrgh) & [@s83](https://github.com/s83)]
+1. [Spotahome](https://www.spotahome.com/) [[@spotahome](https://github.com/spotahome)]
1. [Spotify](https://github.com/spotify) [[@znichols](https://github.com/znichols)]
+1. [Square](https://squareup.com/)
1. [Stackspace](https://beta.stackspace.io/)
+1. [Strava](https://strava.com) [[@strava](https://github.com/strava), [@dhuang](https://github.com/dhuang) & [@liamstewart](https://github.com/liamstewart)]
1. [Stripe](https://stripe.com) [[@jbalogh](https://github.com/jbalogh)]
+1. [Strongmind](https://www.strongmind.com) [[@tomchapin](https://github.com/tomchapin) & [@wongstein](https://github.com/wongstein)]
+1. [Surfline](https://www.surfline.com/) [[@jawang35](https://github.com/jawang35)]
+1. [T2 Systems](http://t2systems.com) [[@unclaimedpants](https://github.com/unclaimedpants)]
1. [Tails.com](https://tails.com/) [[@alanmcruickshank](https://github.com/alanmcruickshank)]
+1. [TEK](https://www.tek.fi/en) [[@telac](https://github.com/telac)]
+1. [Tesla](https://www.tesla.com/) [[@thoralf-gutierrez](https://github.com/thoralf-gutierrez)]
+1. [The Home Depot](https://www.homedepot.com/)[[@apekshithr](https://github.com/apekshithr)]
+1. [THE ICONIC](https://www.theiconic.com.au/) [[@revathijay](https://github.com/revathijay)] [[@ilikedata](https://github.com/ilikedata)]
+1. [Thinking Machines](https://thinkingmachin.es) [[@marksteve](https://github.com/marksteve)]
+1. [Thinknear](https://www.thinknear.com/) [[@d3cay1](https://github.com/d3cay1), [@ccson](https://github.com/ccson), & [@ababian](https://github.com/ababian)]
+1. [ThoughtWorks](https://www.thoughtworks.com/) [[@sann3](https://github.com/sann3)]
1. [Thumbtack](https://www.thumbtack.com/) [[@natekupp](https://github.com/natekupp)]
1. [Tictail](https://tictail.com/)
-1. [T2 Systems](http://t2systems.com) [[@unclaimedpants](https://github.com/unclaimedpants)]
+1. [Tile](https://tile.com/) [[@ranjanmanish](https://github.com/ranjanmanish)]
+1. [Tokopedia](https://www.tokopedia.com/) [@topedmaria](https://github.com/topedmaria)
+1. [Twine Labs](https://www.twinelabs.com/) [[@ivorpeles](https://github.com/ivorpeles)]
+1. [Twitter](https://www.twitter.com/) [[@aoen](https://github.com/aoen)]
1. [Ubisoft](https://www.ubisoft.com/) [[@Walkoss](https://github.com/Walkoss)]
1. [United Airlines](https://www.united.com/) [[@ilopezfr](https://github.com/ilopezfr)]
-1. [Upsight](https://www.upsight.com) [[@dhuang](https://github.com/dhuang)]
+1. [Upsight](https://www.upsight.com)
+1. [VeeR VR](https://veer.tv) [[@pishilong](https://github.com/pishilong)]
1. [Vente-Exclusive.com](http://www.vente-exclusive.com/) [[@alexvanboxel](https://github.com/alexvanboxel)]
+1. [Vevo](https://www.vevo.com/) [[@csetiawan](https://github.com/csetiawan) & [@jerrygillespie](https://github.com/jerrygillespie)]
+1. [Vidio](https://www.vidio.com/)
+1. [Ville de Montréal](http://ville.montreal.qc.ca/)[@VilledeMontreal](https://github.com/VilledeMontreal/)]
1. [Vnomics](https://github.com/vnomics) [[@lpalum](https://github.com/lpalum)]
1. [WePay](http://www.wepay.com) [[@criccomini](https://github.com/criccomini) & [@mtagle](https://github.com/mtagle)]
1. [WeTransfer](https://github.com/WeTransfer) [[@jochem](https://github.com/jochem)]
1. [Whistle Labs](http://www.whistle.com) [[@ananya77041](https://github.com/ananya77041)]
1. [WiseBanyan](https://wisebanyan.com/)
1. [Wooga](https://www.wooga.com/)
-1. [Xoom](https://www.xoom.com/india/send-money) [[@gepser](https://github.com/gepser) & [@omarvides](https://github.com/omarvides)]
+1. [Xero](https://www.xero.com/) [[@yan9yu](https://github.com/yan9yu)]
+1. [Xoom](https://www.xoom.com/)
1. [Yahoo!](https://www.yahoo.com/)
+1. [Yieldr](https://www.yieldr.com/) [[@ggeorgiadis](https://github.com/ggeorgiadis)]
1. [Zapier](https://www.zapier.com) [[@drknexus](https://github.com/drknexus) & [@statwonk](https://github.com/statwonk)]
+1. [Zego](https://www.zego.com/) [[@ruimffl](https://github.com/ruimffl)]
1. [Zendesk](https://www.github.com/zendesk)
1. [Zenly](https://zen.ly) [[@cerisier](https://github.com/cerisier) & [@jbdalido](https://github.com/jbdalido)]
1. [Zymergen](https://www.zymergen.com/)
-1. [99](https://99taxis.com) [[@fbenevides](https://github.com/fbenevides), [@gustavoamigo](https://github.com/gustavoamigo) & [@mmmaia](https://github.com/mmmaia)]
-## Links
+## Who Maintains Apache Airflow?
+Airflow is the work of the [community](https://github.com/apache/incubator-airflow/graphs/contributors),
+but the [core committers/maintainers](https://people.apache.org/committers-by-project.html#airflow)
+are responsible for reviewing and merging PRs as well as steering conversation around new feature requests.
+If you would like to become a maintainer, please review the Apache Airflow
+[committer requirements](https://cwiki.apache.org/confluence/display/AIRFLOW/Committers).
+
+## Links
-* [Documentation](http://airflow.incubator.apache.org/)
-* [Chat](https://gitter.im/apache/incubator-airflow)
-* [Apache Airflow Incubation Status](http://incubator.apache.org/projects/airflow.html)
-* [More](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links)
+- [Documentation](https://airflow.incubator.apache.org/)
+- [Chat](https://apache-airflow-slack.herokuapp.com/)
+- [Apache Airflow Incubation Status](http://incubator.apache.org/projects/airflow.html)
+- [More](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links)
diff --git a/TODO.md b/TODO.md
index cf19035e1b..f49d99ce6f 100644
--- a/TODO.md
+++ b/TODO.md
@@ -1,10 +1,30 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
#### Roadmap items
+
* UI page answering "Why isn't this task instance running?"
* Attempt removing DagBag caching for the web server
* Distributed scheduler (supervisors)
- * Get the supervisors to run sensors (as opposed to each sensor taking a slot)
- * Improve DagBag differential refresh
- * Pickle all the THINGS! supervisors maintains fresh, versionned pickles in the database as they monitor for change
+ * Get the supervisors to run sensors (as opposed to each sensor taking a slot)
+ * Improve DagBag differential refresh
+ * Pickle all the THINGS! supervisors maintains fresh, versioned pickles in the database as they monitor for change
* Pre-prod running off of master
* Containment / YarnExecutor / Docker?
* Get s3 logs
@@ -12,19 +32,23 @@
* Run Hive / Hadoop / HDFS tests in Travis-CI
#### UI
+
* Backfill form
* Better task filtering int duration and landing time charts (operator toggle, task regex, uncheck all button)
* Add templating to adhoc queries
#### Backend
+
* Add a run_only_latest flag to BaseOperator, runs only most recent task instance where deps are met
* Raise errors when setting dependencies on task in foreign DAGs
* Add an is_test flag to the run context
#### Wishlist
+
* Pause flag at the task level
* Increase unit test coverage
* Stats logging interface with support for stats and sqlalchemy to collect detailed information from the scheduler and dag processing times
#### Other
-* deprecate TimeSensor
+
+* Deprecate TimeSensor
diff --git a/UPDATING.md b/UPDATING.md
index 7a801e5741..7409a35b5f 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -1,41 +1,303 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
# Updating Airflow
This file documents any backwards-incompatible changes in Airflow and
-assists people when migrating to a new version.
+assists users migrating to a new version.
+
+## Airflow Master
+
+### Modification to config file discovery
+
+If the `AIRFLOW_CONFIG` environment variable was not set and the
+`~/airflow/airflow.cfg` file existed, airflow previously used
+`~/airflow/airflow.cfg` instead of `$AIRFLOW_HOME/airflow.cfg`. Now airflow
+will discover its config file using the `$AIRFLOW_CONFIG` and `$AIRFLOW_HOME`
+environment variables rather than checking for the presence of a file.
+
+### Modification to `ts_nodash` macro
+`ts_nodash` previously contained TimeZone information alongwith execution date. For Example: `20150101T000000+0000`. This is not user-friendly for file or folder names which was a popular use case for `ts_nodash`. Hence this behavior has been changed and using `ts_nodash` will no longer contain TimeZone information, restoring the pre-1.10 behavior of this macro. And a new macro `ts_nodash_with_tz` has been added which can be used to get a string with execution date and timezone info without dashes.
+
+Examples:
+ * `ts_nodash`: `20150101T000000`
+ * `ts_nodash_with_tz`: `20150101T000000+0000`
+
+### New `dag_processor_manager_log_location` config option
+
+The DAG parsing manager log now by default will be log into a file, where its location is
+controlled by the new `dag_processor_manager_log_location` config option in core section.
+
+### new `sync_parallelism` config option in celery section
+
+The new `sync_parallelism` config option will control how many processes CeleryExecutor will use to
+fetch celery task state in parallel. Default value is max(1, number of cores - 1)
+
+### Semantics of next_ds/prev_ds changed for manually triggered runs
+
+next_ds/prev_ds now map to execution_date instead of the next/previous schedule-aligned execution date for DAGs triggered in the UI.
+
+### Rename of BashTaskRunner to StandardTaskRunner
+
+BashTaskRunner has been renamed to StandardTaskRunner. It is the default task runner
+so you might need to update your config.
+
+`task_runner = StandardTaskRunner`
+
+### DAG level Access Control for new RBAC UI
+
+Extend and enhance new Airflow RBAC UI to support DAG level ACL. Each dag now has two permissions(one for write, one for read) associated('can_dag_edit', 'can_dag_read').
+The admin will create new role, associate the dag permission with the target dag and assign that role to users. That user can only access / view the certain dags on the UI
+that he has permissions on. If a new role wants to access all the dags, the admin could associate dag permissions on an artificial view(``all_dags``) with that role.
+
+We also provide a new cli command(``sync_perm``) to allow admin to auto sync permissions.
+
+
+### min_file_parsing_loop_time config option temporarily disabled
+
+The scheduler.min_file_parsing_loop_time config option has been temporarily removed due to
+some bugs.
+
+### CLI Changes
+
+The ability to manipulate users from the command line has been changed. 'airflow create_user' and 'airflow delete_user' and 'airflow list_users' has been grouped to a single command `airflow users` with optional flags `--create`, `--list` and `--delete`.
+
+Example Usage:
+
+To create a new user:
+```bash
+airflow users --create --username jondoe --lastname doe --firstname jon --email jdoe@apache.org --role Viewer --password test
+```
+
+To list users:
+```bash
+airflow users --list
+```
+
+To delete a user:
+```bash
+airflow users --delete --username jondoe
+```
+
+### User model changes
+This patch changes the `User.superuser` field from a hardcoded boolean to a `Boolean()` database column. `User.superuser` will default to `False`, which means that this privilege will have to be granted manually to any users that may require it.
+
+For example, open a Python shell and
+```python
+from airflow import models, settings
+
+session = settings.Session()
+users = session.query(models.User).all() # [admin, regular_user]
+
+users[1].superuser # False
+
+admin = users[0]
+admin.superuser = True
+session.add(admin)
+session.commit()
+```
+
+## Airflow 1.10.1
+
+### StatsD Metrics
+
+The `scheduler_heartbeat` metric has been changed from a gauge to a counter. Each loop of the scheduler will increment the counter by 1. This provides a higher degree of visibility and allows for better integration with Prometheus using the [StatsD Exporter](https://github.com/prometheus/statsd_exporter). The scheduler's activity status can be determined by graphing and alerting using a rate of change of the counter. If the scheduler goes down, the rate will drop to 0.
+
+### Custom auth backends interface change
+
+We have updated the version of flask-login we depend upon, and as a result any
+custom auth backends might need a small change: `is_active`,
+`is_authenticated`, and `is_anonymous` should now be properties. What this means is if
+previously you had this in your user class
+
+ def is_active(self):
+ return self.active
+
+then you need to change it like this
+
+ @property
+ def is_active(self):
+ return self.active
+
+### EMRHook now passes all of connection's extra to CreateJobFlow API
+
+EMRHook.create_job_flow has been changed to pass all keys to the create_job_flow API, rather than
+just specific known keys for greater flexibility.
+
+However prior to this release the "emr_default" sample connection that was created had invalid
+configuration, so creating EMR clusters might fail until your connection is updated. (Ec2KeyName,
+Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they
+should be inside the "Instances" dict)
+
+### LDAP Auth Backend now requires TLS
+
+Connecting to an LDAP server over plain text is not supported anymore. The
+certificate presented by the LDAP server must be signed by a trusted
+certificate, or you must provide the `cacert` option under `[ldap]` in the
+config file.
+
+If you want to use LDAP auth backend without TLS then you will have to create a
+custom-auth backend based on
+https://github.com/apache/incubator-airflow/blob/1.10.0/airflow/contrib/auth/backends/ldap_auth.py
+
+## Airflow 1.10
+
+Installation and upgrading requires setting `SLUGIFY_USES_TEXT_UNIDECODE=yes` in your environment or
+`AIRFLOW_GPL_UNIDECODE=yes`. In case of the latter a GPL runtime dependency will be installed due to a
+dependency (python-nvd3 -> python-slugify -> unidecode).
+
+### Replace DataProcHook.await calls to DataProcHook.wait
+
+The method name was changed to be compatible with the Python 3.7 async/await keywords
+
+### Setting UTF-8 as default mime_charset in email utils
+
+### Add a configuration variable(default_dag_run_display_number) to control numbers of dag run for display
+
+Add a configuration variable(default_dag_run_display_number) under webserver section to control the number of dag runs to show in UI.
+
+### Default executor for SubDagOperator is changed to SequentialExecutor
+
+### New Webserver UI with Role-Based Access Control
+
+The current webserver UI uses the Flask-Admin extension. The new webserver UI uses the [Flask-AppBuilder (FAB)](https://github.com/dpgaspar/Flask-AppBuilder) extension. FAB has built-in authentication support and Role-Based Access Control (RBAC), which provides configurable roles and permissions for individual users.
+
+To turn on this feature, in your airflow.cfg file (under [webserver]), set the configuration variable `rbac = True`, and then run `airflow` command, which will generate the `webserver_config.py` file in your $AIRFLOW_HOME.
+
+#### Setting up Authentication
+
+FAB has built-in authentication support for DB, OAuth, OpenID, LDAP, and REMOTE_USER. The default auth type is `AUTH_DB`.
-## Airflow 1.9.1
+For any other authentication type (OAuth, OpenID, LDAP, REMOTE_USER), see the [Authentication section of FAB docs](http://flask-appbuilder.readthedocs.io/en/latest/security.html#authentication-methods) for how to configure variables in webserver_config.py file.
+
+Once you modify your config file, run `airflow initdb` to generate new tables for RBAC support (these tables will have the prefix `ab_`).
+
+#### Creating an Admin Account
+
+Once configuration settings have been updated and new tables have been generated, create an admin account with `airflow create_user` command.
+
+#### Using your new UI
+
+Run `airflow webserver` to start the new UI. This will bring up a log in page, enter the recently created admin username and password.
+
+There are five roles created for Airflow by default: Admin, User, Op, Viewer, and Public. To configure roles/permissions, go to the `Security` tab and click `List Roles` in the new UI.
+
+#### Breaking changes
+
+- AWS Batch Operator renamed property queue to job_queue to prevent conflict with the internal queue from CeleryExecutor - AIRFLOW-2542
+- Users created and stored in the old users table will not be migrated automatically. FAB's built-in authentication support must be reconfigured.
+- Airflow dag home page is now `/home` (instead of `/admin`).
+- All ModelViews in Flask-AppBuilder follow a different pattern from Flask-Admin. The `/admin` part of the URL path will no longer exist. For example: `/admin/connection` becomes `/connection/list`, `/admin/connection/new` becomes `/connection/add`, `/admin/connection/edit` becomes `/connection/edit`, etc.
+- Due to security concerns, the new webserver will no longer support the features in the `Data Profiling` menu of old UI, including `Ad Hoc Query`, `Charts`, and `Known Events`.
+- HiveServer2Hook.get_results() always returns a list of tuples, even when a single column is queried, as per Python API 2.
+- **UTC is now the default timezone**: Either reconfigure your workflows scheduling in UTC or set `default_timezone` as explained in https://airflow.apache.org/timezone.html#default-time-zone
+
+### airflow.contrib.sensors.hdfs_sensors renamed to airflow.contrib.sensors.hdfs_sensor
+
+We now rename airflow.contrib.sensors.hdfs_sensors to airflow.contrib.sensors.hdfs_sensor for consistency purpose.
+
+### MySQL setting required
+
+We now rely on more strict ANSI SQL settings for MySQL in order to have sane defaults. Make sure
+to have specified `explicit_defaults_for_timestamp=1` in your my.cnf under `[mysqld]`
### Celery config
To make the config of Airflow compatible with Celery, some properties have been renamed:
+
```
celeryd_concurrency -> worker_concurrency
celery_result_backend -> result_backend
+celery_ssl_active -> ssl_active
+celery_ssl_cert -> ssl_cert
+celery_ssl_key -> ssl_key
```
-This will result in the same config parameters as Celery 4 and will make it more transparent.
+
+Resulting in the same config parameters as Celery 4, with more transparency.
### GCP Dataflow Operators
+
Dataflow job labeling is now supported in Dataflow{Java,Python}Operator with a default
"airflow-version" label, please upgrade your google-cloud-dataflow or apache-beam version
to 2.2.0 or greater.
+### BigQuery Hooks and Operator
+
+The `bql` parameter passed to `BigQueryOperator` and `BigQueryBaseCursor.run_query` has been deprecated and renamed to `sql` for consistency purposes. Using `bql` will still work (and raise a `DeprecationWarning`), but is no longer
+supported and will be removed entirely in Airflow 2.0
+
+### Redshift to S3 Operator
+
+With Airflow 1.9 or lower, Unload operation always included header row. In order to include header row,
+we need to turn off parallel unload. It is preferred to perform unload operation using all nodes so that it is
+faster for larger tables. So, parameter called `include_header` is added and default is set to False.
+Header row will be added only if this parameter is set True and also in that case parallel will be automatically turned off (`PARALLEL OFF`)
+
+### Google cloud connection string
+
+With Airflow 1.9 or lower, there were two connection strings for the Google Cloud operators, both `google_cloud_storage_default` and `google_cloud_default`. This can be confusing and therefore the `google_cloud_storage_default` connection id has been replaced with `google_cloud_default` to make the connection id consistent across Airflow.
+
+### Logging Configuration
+
+With Airflow 1.9 or lower, `FILENAME_TEMPLATE`, `PROCESSOR_FILENAME_TEMPLATE`, `LOG_ID_TEMPLATE`, `END_OF_LOG_MARK` were configured in `airflow_local_settings.py`. These have been moved into the configuration file, and hence if you were using a custom configuration file the following defaults need to be added.
+
+```
+[core]
+fab_logging_level = WARN
+log_filename_template = {{{{ ti.dag_id }}}}/{{{{ ti.task_id }}}}/{{{{ ts }}}}/{{{{ try_number }}}}.log
+log_processor_filename_template = {{{{ filename }}}}.log
+
+[elasticsearch]
+elasticsearch_log_id_template = {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}
+elasticsearch_end_of_log_mark = end_of_log
+```
+
+The previous setting of `log_task_reader` is not needed in many cases now when using the default logging config with remote storages. (Previously it needed to be set to `s3.task` or similar. This is not needed with the default config anymore)
+
+#### Change of per-task log path
+
+With the change to Airflow core to be timezone aware the default log path for task instances will now include timezone information. This will by default mean all previous task logs won't be found. You can get the old behaviour back by setting the following config options:
+
+```
+[core]
+log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ execution_date.strftime("%%Y-%%m-%%dT%%H:%%M:%%S") }}/{{ try_number }}.log
+```
+
## Airflow 1.9
### SSH Hook updates, along with new SSH Operator & SFTP Operator
-SSH Hook now uses Paramiko library to create ssh client connection, instead of sub-process based ssh command execution previously (<1.9.0), so this is backward incompatible.
- - update SSHHook constructor
- - use SSHOperator class in place of SSHExecuteOperator which is removed now. Refer test_ssh_operator.py for usage info.
- - SFTPOperator is added to perform secure file transfer from serverA to serverB. Refer test_sftp_operator.py.py for usage info.
- - No updates are required if you are using ftpHook, it will continue work as is.
+SSH Hook now uses the Paramiko library to create an ssh client connection, instead of the sub-process based ssh command execution previously (<1.9.0), so this is backward incompatible.
+
+- update SSHHook constructor
+- use SSHOperator class in place of SSHExecuteOperator which is removed now. Refer to test_ssh_operator.py for usage info.
+- SFTPOperator is added to perform secure file transfer from serverA to serverB. Refer to test_sftp_operator.py for usage info.
+- No updates are required if you are using ftpHook, it will continue to work as is.
### S3Hook switched to use Boto3
-The airflow.hooks.S3_hook.S3Hook has been switched to use boto3 instead of the older boto (a.k.a. boto2). This result in a few backwards incompatible changes to the following classes: S3Hook:
- - the constructors no longer accepts `s3_conn_id`. It is now called `aws_conn_id`.
- - the default conneciton is now "aws_default" instead of "s3_default"
- - the return type of objects returned by `get_bucket` is now boto3.s3.Bucket
- - the return type of `get_key`, and `get_wildcard_key` is now an boto3.S3.Object.
+The airflow.hooks.S3_hook.S3Hook has been switched to use boto3 instead of the older boto (a.k.a. boto2). This results in a few backwards incompatible changes to the following classes: S3Hook:
+
+- the constructors no longer accepts `s3_conn_id`. It is now called `aws_conn_id`.
+- the default connection is now "aws_default" instead of "s3_default"
+- the return type of objects returned by `get_bucket` is now boto3.s3.Bucket
+- the return type of `get_key`, and `get_wildcard_key` is now an boto3.S3.Object.
If you are using any of these in your DAGs and specify a connection ID you will need to update the parameter name for the connection to "aws_conn_id": S3ToHiveTransfer, S3PrefixSensor, S3KeySensor, RedshiftToS3Transfer.
@@ -47,7 +309,7 @@ The logging structure of Airflow has been rewritten to make configuration easier
A logger is the entry point into the logging system. Each logger is a named bucket to which messages can be written for processing. A logger is configured to have a log level. This log level describes the severity of the messages that the logger will handle. Python defines the following log levels: DEBUG, INFO, WARNING, ERROR or CRITICAL.
-Each message that is written to the logger is a Log Record. Each log record also has a log level indicating the severity of that specific message. A log record can also contain useful metadata that describes the event that is being logged. This can include details such as a stack trace or an error code.
+Each message that is written to the logger is a Log Record. Each log record contains a log level indicating the severity of that specific message. A log record can also contain useful metadata that describes the event that is being logged. This can include details such as a stack trace or an error code.
When a message is given to the logger, the log level of the message is compared to the log level of the logger. If the log level of the message meets or exceeds the log level of the logger itself, the message will undergo further processing. If it doesn’t, the message will be ignored.
@@ -55,7 +317,7 @@ Once a logger has determined that a message needs to be processed, it is passed
#### Changes in Airflow Logging
-Airflow's logging mechanism has been refactored to uses Python’s builtin `logging` module to perform logging of the application. By extending classes with the existing `LoggingMixin`, all the logging will go through a central logger. Also the `BaseHook` and `BaseOperator` already extends this class, so it is easily available to do logging.
+Airflow's logging mechanism has been refactored to use Python’s built-in `logging` module to perform logging of the application. By extending classes with the existing `LoggingMixin`, all the logging will go through a central logger. Also the `BaseHook` and `BaseOperator` already extend this class, so it is easily available to do logging.
The main benefit is easier configuration of the logging by setting a single centralized python file. Disclaimer; there is still some inline configuration, but this will be removed eventually. The new logging class is defined by setting the dotted classpath in your `~/airflow/airflow.cfg` file:
@@ -66,11 +328,120 @@ The main benefit is easier configuration of the logging by setting a single cent
logging_config_class = my.path.default_local_settings.LOGGING_CONFIG
```
-The logging configuration file needs to be on the `PYTHONPATH`, for example `$AIRFLOW_HOME/config`. This directory is loaded by default. Of course you are free to add any directory to the `PYTHONPATH`, this might be handy when you have the config in another directory or you mount a volume in case of Docker.
+The logging configuration file needs to be on the `PYTHONPATH`, for example `$AIRFLOW_HOME/config`. This directory is loaded by default. Any directory may be added to the `PYTHONPATH`, this might be handy when the config is in another directory or a volume is mounted in case of Docker.
-You can take the config from `airflow/config_templates/airflow_local_settings.py` as a starting point. Copy the contents to `${AIRFLOW_HOME}/config/airflow_local_settings.py`, and alter the config as you like.
+The config can be taken from `airflow/config_templates/airflow_local_settings.py` as a starting point. Copy the contents to `${AIRFLOW_HOME}/config/airflow_local_settings.py`, and alter the config as is preferred.
+
+```
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+
+from airflow import configuration as conf
+
+# TODO: Logging format and level should be configured
+# in this file instead of from airflow.cfg. Currently
+# there are other log format and level configurations in
+# settings.py and cli.py. Please see AIRFLOW-1455.
+
+LOG_LEVEL = conf.get('core', 'LOGGING_LEVEL').upper()
+LOG_FORMAT = conf.get('core', 'log_format')
+
+BASE_LOG_FOLDER = conf.get('core', 'BASE_LOG_FOLDER')
+PROCESSOR_LOG_FOLDER = conf.get('scheduler', 'child_process_log_directory')
+
+FILENAME_TEMPLATE = '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'
+PROCESSOR_FILENAME_TEMPLATE = '{{ filename }}.log'
+
+DEFAULT_LOGGING_CONFIG = {
+ 'version': 1,
+ 'disable_existing_loggers': False,
+ 'formatters': {
+ 'airflow.task': {
+ 'format': LOG_FORMAT,
+ },
+ 'airflow.processor': {
+ 'format': LOG_FORMAT,
+ },
+ },
+ 'handlers': {
+ 'console': {
+ 'class': 'logging.StreamHandler',
+ 'formatter': 'airflow.task',
+ 'stream': 'ext://sys.stdout'
+ },
+ 'file.task': {
+ 'class': 'airflow.utils.log.file_task_handler.FileTaskHandler',
+ 'formatter': 'airflow.task',
+ 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+ 'filename_template': FILENAME_TEMPLATE,
+ },
+ 'file.processor': {
+ 'class': 'airflow.utils.log.file_processor_handler.FileProcessorHandler',
+ 'formatter': 'airflow.processor',
+ 'base_log_folder': os.path.expanduser(PROCESSOR_LOG_FOLDER),
+ 'filename_template': PROCESSOR_FILENAME_TEMPLATE,
+ }
+ # When using s3 or gcs, provide a customized LOGGING_CONFIG
+ # in airflow_local_settings within your PYTHONPATH, see UPDATING.md
+ # for details
+ # 's3.task': {
+ # 'class': 'airflow.utils.log.s3_task_handler.S3TaskHandler',
+ # 'formatter': 'airflow.task',
+ # 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+ # 's3_log_folder': S3_LOG_FOLDER,
+ # 'filename_template': FILENAME_TEMPLATE,
+ # },
+ # 'gcs.task': {
+ # 'class': 'airflow.utils.log.gcs_task_handler.GCSTaskHandler',
+ # 'formatter': 'airflow.task',
+ # 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+ # 'gcs_log_folder': GCS_LOG_FOLDER,
+ # 'filename_template': FILENAME_TEMPLATE,
+ # },
+ },
+ 'loggers': {
+ '': {
+ 'handlers': ['console'],
+ 'level': LOG_LEVEL
+ },
+ 'airflow': {
+ 'handlers': ['console'],
+ 'level': LOG_LEVEL,
+ 'propagate': False,
+ },
+ 'airflow.processor': {
+ 'handlers': ['file.processor'],
+ 'level': LOG_LEVEL,
+ 'propagate': True,
+ },
+ 'airflow.task': {
+ 'handlers': ['file.task'],
+ 'level': LOG_LEVEL,
+ 'propagate': False,
+ },
+ 'airflow.task_runner': {
+ 'handlers': ['file.task'],
+ 'level': LOG_LEVEL,
+ 'propagate': True,
+ },
+ }
+}
+```
-If you want to customize the logging (for example, use logging rotate), you can do this by defining one or more of the logging handles that [Python has to offer](https://docs.python.org/3/library/logging.handlers.html). For more details about the Python logging, please refer to the [official logging documentation](https://docs.python.org/3/library/logging.html).
+To customize the logging (for example, use logging rotate), define one or more of the logging handles that [Python has to offer](https://docs.python.org/3/library/logging.handlers.html). For more details about the Python logging, please refer to the [official logging documentation](https://docs.python.org/3/library/logging.html).
Furthermore, this change also simplifies logging within the DAG itself:
@@ -95,17 +466,18 @@ Type "help", "copyright", "credits" or "license" for more information.
#### Template path of the file_task_handler
-The `file_task_handler` logger is more flexible. You can change the default format, `{dag_id}/{task_id}/{execution_date}/{try_number}.log` by supplying Jinja templating in the `FILENAME_TEMPLATE` configuration variable. See the `file_task_handler` for more information.
+The `file_task_handler` logger has been made more flexible. The default format can be changed, `{dag_id}/{task_id}/{execution_date}/{try_number}.log` by supplying Jinja templating in the `FILENAME_TEMPLATE` configuration variable. See the `file_task_handler` for more information.
#### I'm using S3Log or GCSLogs, what do I do!?
If you are logging to Google cloud storage, please see the [Google cloud platform documentation](https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform) for logging instructions.
If you are using S3, the instructions should be largely the same as the Google cloud platform instructions above. You will need a custom logging config. The `REMOTE_BASE_LOG_FOLDER` configuration key in your airflow config has been removed, therefore you will need to take the following steps:
- - Copy the logging configuration from [`airflow/config_templates/airflow_logging_settings.py`](https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py) and copy it.
- - Place it in a directory inside the Python import path `PYTHONPATH`. If you are using Python 2.7, ensuring that any `__init__.py` files exist so that it is importable.
- - Update the config by setting the path of `REMOTE_BASE_LOG_FOLDER` explicitly in the config. The `REMOTE_BASE_LOG_FOLDER` key is not used anymore.
- - Set the `logging_config_class` to the filename and dict. For example, if you place `custom_logging_config.py` on the base of your pythonpath, you will need to set `logging_config_class = custom_logging_config.LOGGING_CONFIG` in your config as Airflow 1.8.
+
+- Copy the logging configuration from [`airflow/config_templates/airflow_logging_settings.py`](https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py).
+- Place it in a directory inside the Python import path `PYTHONPATH`. If you are using Python 2.7, ensuring that any `__init__.py` files exist so that it is importable.
+- Update the config by setting the path of `REMOTE_BASE_LOG_FOLDER` explicitly in the config. The `REMOTE_BASE_LOG_FOLDER` key is not used anymore.
+- Set the `logging_config_class` to the filename and dict. For example, if you place `custom_logging_config.py` on the base of your `PYTHONPATH`, you will need to set `logging_config_class = custom_logging_config.LOGGING_CONFIG` in your config as Airflow 1.8.
### New Features
@@ -114,8 +486,10 @@ If you are using S3, the instructions should be largely the same as the Google c
A new DaskExecutor allows Airflow tasks to be run in Dask Distributed clusters.
### Deprecated Features
+
These features are marked for deprecation. They may still work (and raise a `DeprecationWarning`), but are no longer
supported and will be removed entirely in Airflow 2.0
+
- If you're using the `google_cloud_conn_id` or `dataproc_cluster` argument names explicitly in `contrib.operators.Dataproc{*}Operator`(s), be sure to rename them to `gcp_conn_id` or `cluster_name`, respectively. We've renamed these arguments for consistency. (AIRFLOW-1323)
- `post_execute()` hooks now take two arguments, `context` and `result`
@@ -129,68 +503,82 @@ supported and will be removed entirely in Airflow 2.0
Note that JSON serialization is stricter than pickling, so if you want to e.g. pass
raw bytes through XCom you must encode them using an encoding like base64.
By default pickling is still enabled until Airflow 2.0. To disable it
- Set enable_xcom_pickling = False in your Airflow config.
+ set enable_xcom_pickling = False in your Airflow config.
## Airflow 1.8.1
-The Airflow package name was changed from `airflow` to `apache-airflow` during this release. You must uninstall your
-previously installed version of Airflow before installing 1.8.1.
+The Airflow package name was changed from `airflow` to `apache-airflow` during this release. You must uninstall
+a previously installed version of Airflow before installing 1.8.1.
## Airflow 1.8
### Database
+
The database schema needs to be upgraded. Make sure to shutdown Airflow and make a backup of your database. To
upgrade the schema issue `airflow upgradedb`.
### Upgrade systemd unit files
+
Systemd unit files have been updated. If you use systemd please make sure to update these.
> Please note that the webserver does not detach properly, this will be fixed in a future version.
### Tasks not starting although dependencies are met due to stricter pool checking
+
Airflow 1.7.1 has issues with being able to over subscribe to a pool, ie. more slots could be used than were
available. This is fixed in Airflow 1.8.0, but due to past issue jobs may fail to start although their
dependencies are met after an upgrade. To workaround either temporarily increase the amount of slots above
-the the amount of queued tasks or use a new pool.
+the amount of queued tasks or use a new pool.
### Less forgiving scheduler on dynamic start_date
+
Using a dynamic start_date (e.g. `start_date = datetime.now()`) is not considered a best practice. The 1.8.0 scheduler
is less forgiving in this area. If you encounter DAGs not being scheduled you can try using a fixed start_date and
-renaming your dag. The last step is required to make sure you start with a clean slate, otherwise the old schedule can
+renaming your DAG. The last step is required to make sure you start with a clean slate, otherwise the old schedule can
interfere.
### New and updated scheduler options
-Please read through these options, defaults have changed since 1.7.1.
+
+Please read through the new scheduler options, defaults have changed since 1.7.1.
#### child_process_log_directory
-In order the increase the robustness of the scheduler, DAGS our now processed in their own process. Therefore each
-DAG has its own log file for the scheduler. These are placed in `child_process_log_directory` which defaults to
+
+In order to increase the robustness of the scheduler, DAGS are now processed in their own process. Therefore each
+DAG has its own log file for the scheduler. These log files are placed in `child_process_log_directory` which defaults to
`<AIRFLOW_HOME>/scheduler/latest`. You will need to make sure these log files are removed.
> DAG logs or processor logs ignore and command line settings for log file locations.
#### run_duration
+
Previously the command line option `num_runs` was used to let the scheduler terminate after a certain amount of
loops. This is now time bound and defaults to `-1`, which means run continuously. See also num_runs.
#### num_runs
+
Previously `num_runs` was used to let the scheduler terminate after a certain amount of loops. Now num_runs specifies
the number of times to try to schedule each DAG file within `run_duration` time. Defaults to `-1`, which means try
indefinitely. This is only available on the command line.
#### min_file_process_interval
+
After how much time should an updated DAG be picked up from the filesystem.
+#### min_file_parsing_loop_time
+CURRENTLY DISABLED DUE TO A BUG
+How many seconds to wait between file-parsing loops to prevent the logs from being spammed.
+
#### dag_dir_list_interval
-How often the scheduler should relist the contents of the DAG directory. If you experience that while developing your
-dags are not being picked up, have a look at this number and decrease it when necessary.
+
+The frequency with which the scheduler should relist the contents of the DAG directory. If while developing +dags, they are not being picked up, have a look at this number and decrease it when necessary.
#### catchup_by_default
+
By default the scheduler will fill any missing interval DAG Runs between the last execution date and the current date.
This setting changes that behavior to only execute the latest interval. This can also be specified per DAG as
`catchup = False / True`. Command line backfills will still work.
-### Faulty Dags do not show an error in the Web UI
+### Faulty DAGs do not show an error in the Web UI
Due to changes in the way Airflow processes DAGs the Web UI does not show an error when processing a faulty DAG. To
find processing errors go the `child_process_log_directory` which defaults to `<AIRFLOW_HOME>/scheduler/latest`.
@@ -207,7 +595,7 @@ dags_are_paused_at_creation = False
### Airflow Context variable are passed to Hive config if conf is specified
If you specify a hive conf to the run_cli command of the HiveHook, Airflow add some
-convenience variables to the config. In case your run a sceure Hadoop setup it might be
+convenience variables to the config. In case you run a secure Hadoop setup it might be
required to whitelist these variables by adding the following to your configuration:
```
@@ -216,6 +604,7 @@ required to whitelist these variables by adding the following to your configurat
<value>airflow\.ctx\..*</value>
</property>
```
+
### Google Cloud Operator and Hook alignment
All Google Cloud Operators and Hooks are aligned and use the same client library. Now you have a single connection
@@ -227,6 +616,7 @@ Also the old P12 key file type is not supported anymore and only the new JSON ke
account.
### Deprecated Features
+
These features are marked for deprecation. They may still work (and raise a `DeprecationWarning`), but are no longer
supported and will be removed entirely in Airflow 2.0
@@ -243,6 +633,7 @@ supported and will be removed entirely in Airflow 2.0
- The config value secure_mode will default to True which will disable some insecure endpoints/features
### Known Issues
+
There is a report that the default of "-1" for num_runs creates an issue where errors are reported while parsing tasks.
It was not confirmed, but a workaround was found by changing the default back to `None`.
@@ -255,7 +646,7 @@ To do this edit `cli.py`, find the following:
help="Set the number of runs to execute before exiting"),
```
-and change `default=-1` to `default=None`. Please report on the mailing list if you have this issue.
+and change `default=-1` to `default=None`. If you have this issue please report it on the mailing list.
## Airflow 1.7.1.2
@@ -269,7 +660,9 @@ To continue using the default smtp email backend, change the email_backend line
[email]
email_backend = airflow.utils.send_email_smtp
```
+
to:
+
```
[email]
email_backend = airflow.utils.email.send_email_smtp
@@ -282,7 +675,9 @@ To continue using S3 logging, update your config file so:
```
s3_log_folder = s3://my-airflow-log-bucket/logs
```
+
becomes:
+
```
remote_base_log_folder = s3://my-airflow-log-bucket/logs
remote_log_conn_id = <your desired s3 connection>
diff --git a/airflow/__init__.py b/airflow/__init__.py
index 3c5f24c218..d010fe4c74 100644
--- a/airflow/__init__.py
+++ b/airflow/__init__.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
"""
@@ -27,8 +32,8 @@
import sys
-from airflow import configuration as conf
-from airflow import settings
+# flake8: noqa: F401
+from airflow import settings, configuration as conf
from airflow.models import DAG
from flask_admin import BaseView
from importlib import import_module
@@ -75,12 +80,15 @@ class AirflowMacroPlugin(object):
def __init__(self, namespace):
self.namespace = namespace
-from airflow import operators
-from airflow import hooks
-from airflow import executors
-from airflow import macros
+
+from airflow import operators # noqa: E402
+from airflow import sensors # noqa: E402
+from airflow import hooks # noqa: E402
+from airflow import executors # noqa: E402
+from airflow import macros # noqa: E402
operators._integrate_plugins()
+sensors._integrate_plugins() # noqa: E402
hooks._integrate_plugins()
executors._integrate_plugins()
macros._integrate_plugins()
diff --git a/airflow/alembic.ini b/airflow/alembic.ini
index 6274d5ee86..a7cf974f2c 100644
--- a/airflow/alembic.ini
+++ b/airflow/alembic.ini
@@ -1,15 +1,20 @@
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
# A generic, single database configuration.
diff --git a/airflow/api/__init__.py b/airflow/api/__init__.py
index 31a303b9e0..b4a2f8f5bc 100644
--- a/airflow/api/__init__.py
+++ b/airflow/api/__init__.py
@@ -1,16 +1,22 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
from __future__ import print_function
from airflow.exceptions import AirflowException
diff --git a/airflow/api/auth/__init__.py b/airflow/api/auth/__init__.py
index 9d7677a99b..114d189da1 100644
--- a/airflow/api/auth/__init__.py
+++ b/airflow/api/auth/__init__.py
@@ -1,13 +1,18 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/api/auth/backend/__init__.py b/airflow/api/auth/backend/__init__.py
index 9d7677a99b..114d189da1 100644
--- a/airflow/api/auth/backend/__init__.py
+++ b/airflow/api/auth/backend/__init__.py
@@ -1,13 +1,18 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/api/auth/backend/default.py b/airflow/api/auth/backend/default.py
index 49453ea4bd..c3f4d8bdd7 100644
--- a/airflow/api/auth/backend/default.py
+++ b/airflow/api/auth/backend/default.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from functools import wraps
diff --git a/airflow/api/auth/backend/deny_all.py b/airflow/api/auth/backend/deny_all.py
index 1b15e87357..a38532d378 100644
--- a/airflow/api/auth/backend/deny_all.py
+++ b/airflow/api/auth/backend/deny_all.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from functools import wraps
from flask import Response
diff --git a/airflow/api/auth/backend/kerberos_auth.py b/airflow/api/auth/backend/kerberos_auth.py
index a904d59d3d..50a88106fc 100644
--- a/airflow/api/auth/backend/kerberos_auth.py
+++ b/airflow/api/auth/backend/kerberos_auth.py
@@ -1,3 +1,5 @@
+# -*- coding: utf-8 -*-
+#
# Copyright (c) 2013, Michael Komitee
# All rights reserved.
#
@@ -26,8 +28,6 @@
from airflow.utils.log.logging_mixin import LoggingMixin
-install_aliases()
-
import kerberos
import os
@@ -43,6 +43,8 @@
from requests_kerberos import HTTPKerberosAuth
from socket import getfqdn
+install_aliases()
+
client_auth = HTTPKerberosAuth(service='airflow')
_SERVICE_NAME = None
diff --git a/airflow/api/client/__init__.py b/airflow/api/client/__init__.py
index c82f5790fe..114d189da1 100644
--- a/airflow/api/client/__init__.py
+++ b/airflow/api/client/__init__.py
@@ -1,14 +1,18 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/api/client/api_client.py b/airflow/api/client/api_client.py
index f24d80945f..7a7d13c8f9 100644
--- a/airflow/api/client/api_client.py
+++ b/airflow/api/client/api_client.py
@@ -1,17 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# http://www.apache.org/licenses/LICENSE-2.0
#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
class Client(object):
@@ -32,6 +36,13 @@ def trigger_dag(self, dag_id, run_id=None, conf=None, execution_date=None):
"""
raise NotImplementedError()
+ def delete_dag(self, dag_id):
+ """Delete all DB records related to the specified dag.
+
+ :param dag_id:
+ """
+ raise NotImplementedError()
+
def get_pool(self, name):
"""Get pool.
diff --git a/airflow/api/client/json_client.py b/airflow/api/client/json_client.py
index 37e24d3c4e..e05913ebb4 100644
--- a/airflow/api/client/json_client.py
+++ b/airflow/api/client/json_client.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from future.moves.urllib.parse import urljoin
import requests
@@ -50,6 +55,12 @@ def trigger_dag(self, dag_id, run_id=None, conf=None, execution_date=None):
})
return data['message']
+ def delete_dag(self, dag_id):
+ endpoint = '/api/experimental/dags/{}/delete_dag'.format(dag_id)
+ url = urljoin(self._api_base_url, endpoint)
+ data = self._request(url, method='DELETE')
+ return data['message']
+
def get_pool(self, name):
endpoint = '/api/experimental/pools/{}'.format(name)
url = urljoin(self._api_base_url, endpoint)
diff --git a/airflow/api/client/local_client.py b/airflow/api/client/local_client.py
index 5bc7f76aaa..4b46921e64 100644
--- a/airflow/api/client/local_client.py
+++ b/airflow/api/client/local_client.py
@@ -1,20 +1,26 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from airflow.api.client import api_client
from airflow.api.common.experimental import pool
from airflow.api.common.experimental import trigger_dag
+from airflow.api.common.experimental import delete_dag
class Client(api_client.Client):
@@ -27,6 +33,10 @@ def trigger_dag(self, dag_id, run_id=None, conf=None, execution_date=None):
execution_date=execution_date)
return "Created {}".format(dr)
+ def delete_dag(self, dag_id):
+ count = delete_dag.delete_dag(dag_id)
+ return "Removed {} record(s)".format(count)
+
def get_pool(self, name):
p = pool.get_pool(name=name)
return p.pool, p.slots, p.description
diff --git a/airflow/api/common/__init__.py b/airflow/api/common/__init__.py
index 9d7677a99b..114d189da1 100644
--- a/airflow/api/common/__init__.py
+++ b/airflow/api/common/__init__.py
@@ -1,13 +1,18 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/api/common/experimental/__init__.py b/airflow/api/common/experimental/__init__.py
index 9d7677a99b..114d189da1 100644
--- a/airflow/api/common/experimental/__init__.py
+++ b/airflow/api/common/experimental/__init__.py
@@ -1,13 +1,18 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/api/common/experimental/delete_dag.py b/airflow/api/common/experimental/delete_dag.py
new file mode 100644
index 0000000000..fc211c016a
--- /dev/null
+++ b/airflow/api/common/experimental/delete_dag.py
@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+from sqlalchemy import or_
+
+from airflow import models, settings
+from airflow.exceptions import DagNotFound, DagFileExists
+
+
+def delete_dag(dag_id, keep_records_in_log=True):
+ """
+ :param dag_id: the dag_id of the DAG to delete
+ :type dag_id: str
+ :param keep_records_in_log: whether keep records of the given dag_id
+ in the Log table in the backend database (for reasons like auditing).
+ The default value is True.
+ :type keep_records_in_log: bool
+ """
+ session = settings.Session()
+
+ DM = models.DagModel
+ dag = session.query(DM).filter(DM.dag_id == dag_id).first()
+ if dag is None:
+ raise DagNotFound("Dag id {} not found".format(dag_id))
+
+ if dag.fileloc and not os.path.exists(dag.fileloc):
+ raise DagFileExists("Dag id {} is still in DagBag. "
+ "Remove the DAG file first: {}".format(dag_id, dag.fileloc))
+
+ count = 0
+
+ # noinspection PyUnresolvedReferences,PyProtectedMember
+ for m in models.base.Base._decl_class_registry.values():
+ if hasattr(m, "dag_id"):
+ if keep_records_in_log and m.__name__ == 'Log':
+ continue
+ cond = or_(m.dag_id == dag_id, m.dag_id.like(dag_id + ".%"))
+ count += session.query(m).filter(cond).delete(synchronize_session='fetch')
+
+ if dag.is_subdag:
+ p, c = dag_id.rsplit(".", 1)
+ for m in models.DagRun, models.TaskFail, models.TaskInstance:
+ count += session.query(m).filter(m.dag_id == p, m.task_id == c).delete()
+
+ session.commit()
+
+ return count
diff --git a/airflow/api/common/experimental/get_dag_run_state.py b/airflow/api/common/experimental/get_dag_run_state.py
new file mode 100644
index 0000000000..a37aff6abc
--- /dev/null
+++ b/airflow/api/common/experimental/get_dag_run_state.py
@@ -0,0 +1,44 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.exceptions import DagNotFound, DagRunNotFound
+from airflow.models import DagBag
+
+
+def get_dag_run_state(dag_id, execution_date):
+ """Return the task object identified by the given dag_id and task_id."""
+
+ dagbag = DagBag()
+
+ # Check DAG exists.
+ if dag_id not in dagbag.dags:
+ error_message = "Dag id {} not found".format(dag_id)
+ raise DagNotFound(error_message)
+
+ # Get DAG object and check Task Exists
+ dag = dagbag.get_dag(dag_id)
+
+ # Get DagRun object and check that it exists
+ dagrun = dag.get_dagrun(execution_date=execution_date)
+ if not dagrun:
+ error_message = ('Dag Run for date {} not found in dag {}'
+ .format(execution_date, dag_id))
+ raise DagRunNotFound(error_message)
+
+ return {'state': dagrun.get_state()}
diff --git a/airflow/api/common/experimental/get_dag_runs.py b/airflow/api/common/experimental/get_dag_runs.py
new file mode 100644
index 0000000000..63b1f993d3
--- /dev/null
+++ b/airflow/api/common/experimental/get_dag_runs.py
@@ -0,0 +1,55 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask import url_for
+
+from airflow.exceptions import AirflowException
+from airflow.models import DagBag, DagRun
+
+
+def get_dag_runs(dag_id, state=None):
+ """
+ Returns a list of Dag Runs for a specific DAG ID.
+ :param dag_id: String identifier of a DAG
+ :param state: queued|running|success...
+ :return: List of DAG runs of a DAG with requested state,
+ or all runs if the state is not specified
+ """
+ dagbag = DagBag()
+
+ # Check DAG exists.
+ if dag_id not in dagbag.dags:
+ error_message = "Dag id {} not found".format(dag_id)
+ raise AirflowException(error_message)
+
+ dag_runs = list()
+ state = state.lower() if state else None
+ for run in DagRun.find(dag_id=dag_id, state=state):
+ dag_runs.append({
+ 'id': run.id,
+ 'run_id': run.run_id,
+ 'state': run.state,
+ 'dag_id': run.dag_id,
+ 'execution_date': run.execution_date.isoformat(),
+ 'start_date': ((run.start_date or '') and
+ run.start_date.isoformat()),
+ 'dag_run_url': url_for('Airflow.graph', dag_id=run.dag_id,
+ execution_date=run.execution_date)
+ })
+
+ return dag_runs
diff --git a/airflow/api/common/experimental/get_task.py b/airflow/api/common/experimental/get_task.py
index 9023ad1f8d..96aa35a888 100644
--- a/airflow/api/common/experimental/get_task.py
+++ b/airflow/api/common/experimental/get_task.py
@@ -1,18 +1,23 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
-from airflow.exceptions import AirflowException
+from airflow.exceptions import DagNotFound, TaskNotFound
from airflow.models import DagBag
@@ -23,13 +28,13 @@ def get_task(dag_id, task_id):
# Check DAG exists.
if dag_id not in dagbag.dags:
error_message = "Dag id {} not found".format(dag_id)
- raise AirflowException(error_message)
+ raise DagNotFound(error_message)
# Get DAG object and check Task Exists
dag = dagbag.get_dag(dag_id)
if not dag.has_task(task_id):
error_message = 'Task {} not found in dag {}'.format(task_id, dag_id)
- raise AirflowException(error_message)
+ raise TaskNotFound(error_message)
# Return the task.
return dag.get_task(task_id)
diff --git a/airflow/api/common/experimental/get_task_instance.py b/airflow/api/common/experimental/get_task_instance.py
index 7ab5e6e20f..c9f59cd527 100644
--- a/airflow/api/common/experimental/get_task_instance.py
+++ b/airflow/api/common/experimental/get_task_instance.py
@@ -1,18 +1,24 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from airflow.exceptions import AirflowException
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.exceptions import (DagNotFound, TaskNotFound,
+ DagRunNotFound, TaskInstanceNotFound)
from airflow.models import DagBag
@@ -24,26 +30,26 @@ def get_task_instance(dag_id, task_id, execution_date):
# Check DAG exists.
if dag_id not in dagbag.dags:
error_message = "Dag id {} not found".format(dag_id)
- raise AirflowException(error_message)
+ raise DagNotFound(error_message)
# Get DAG object and check Task Exists
dag = dagbag.get_dag(dag_id)
if not dag.has_task(task_id):
error_message = 'Task {} not found in dag {}'.format(task_id, dag_id)
- raise AirflowException(error_message)
+ raise TaskNotFound(error_message)
# Get DagRun object and check that it exists
dagrun = dag.get_dagrun(execution_date=execution_date)
if not dagrun:
error_message = ('Dag Run for date {} not found in dag {}'
.format(execution_date, dag_id))
- raise AirflowException(error_message)
+ raise DagRunNotFound(error_message)
# Get task instance object and check that it exists
task_instance = dagrun.get_task_instance(task_id)
if not task_instance:
error_message = ('Task {} instance for date {} not found'
.format(task_id, execution_date))
- raise AirflowException(error_message)
+ raise TaskInstanceNotFound(error_message)
return task_instance
diff --git a/airflow/api/common/experimental/mark_tasks.py b/airflow/api/common/experimental/mark_tasks.py
index e9366e017f..2fac1254cd 100644
--- a/airflow/api/common/experimental/mark_tasks.py
+++ b/airflow/api/common/experimental/mark_tasks.py
@@ -1,26 +1,32 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from sqlalchemy import or_
from airflow.jobs import BackfillJob
from airflow.models import DagRun, TaskInstance
from airflow.operators.subdag_operator import SubDagOperator
from airflow.settings import Session
from airflow.utils import timezone
+from airflow.utils.db import provide_session
from airflow.utils.state import State
-from sqlalchemy import or_
-
def _create_dagruns(dag, execution_dates, state, run_id_template):
"""
@@ -152,7 +158,7 @@ def set_state(task, execution_date, upstream=False, downstream=False,
# get all tasks of the main dag that will be affected by a state change
qry_dag = session.query(TI).filter(
- TI.dag_id==dag.dag_id,
+ TI.dag_id == dag.dag_id,
TI.execution_date.in_(confirmed_dates),
TI.task_id.in_(task_ids)).filter(
or_(TI.state.is_(None),
@@ -186,15 +192,40 @@ def set_state(task, execution_date, upstream=False, downstream=False,
return tis_altered
-def set_dag_run_state(dag, execution_date, state=State.SUCCESS, commit=False):
+def _set_dag_run_state(dag_id, execution_date, state, session=None):
+ """
+ Helper method that set dag run state in the DB.
+ :param dag_id: dag_id of target dag run
+ :param execution_date: the execution date from which to start looking
+ :param state: target state
+ :param session: database session
+ """
+ DR = DagRun
+ dr = session.query(DR).filter(
+ DR.dag_id == dag_id,
+ DR.execution_date == execution_date
+ ).one()
+ dr.state = state
+ if state == State.RUNNING:
+ dr.start_date = timezone.utcnow()
+ dr.end_date = None
+ else:
+ dr.end_date = timezone.utcnow()
+ session.commit()
+
+
+@provide_session
+def set_dag_run_state_to_success(dag, execution_date, commit=False,
+ session=None):
"""
- Set the state of a dag run and all task instances associated with the dag
- run for a specific execution date.
+ Set the dag run for a specific execution date and its task instances
+ to success.
:param dag: the DAG of which to alter state
:param execution_date: the execution date from which to start looking
- :param state: the state to which the DAG need to be set
:param commit: commit DAG and tasks to be altered to the database
- :return: list of tasks that have been created and updated
+ :param session: database session
+ :return: If commit is true, list of tasks that have been updated,
+ otherwise list of tasks that will be updated
:raises: AssertionError if dag or execution_date is invalid
"""
res = []
@@ -202,18 +233,81 @@ def set_dag_run_state(dag, execution_date, state=State.SUCCESS, commit=False):
if not dag or not execution_date:
return res
- # Mark all task instances in the dag run
+ # Mark the dag run to success.
+ if commit:
+ _set_dag_run_state(dag.dag_id, execution_date, State.SUCCESS, session)
+
+ # Mark all task instances of the dag run to success.
for task in dag.tasks:
task.dag = dag
new_state = set_state(task=task, execution_date=execution_date,
- state=state, commit=commit)
+ state=State.SUCCESS, commit=commit)
res.extend(new_state)
- # Mark the dag run
+ return res
+
+
+@provide_session
+def set_dag_run_state_to_failed(dag, execution_date, commit=False,
+ session=None):
+ """
+ Set the dag run for a specific execution date and its running task instances
+ to failed.
+ :param dag: the DAG of which to alter state
+ :param execution_date: the execution date from which to start looking
+ :param commit: commit DAG and tasks to be altered to the database
+ :param session: database session
+ :return: If commit is true, list of tasks that have been updated,
+ otherwise list of tasks that will be updated
+ :raises: AssertionError if dag or execution_date is invalid
+ """
+ res = []
+
+ if not dag or not execution_date:
+ return res
+
+ # Mark the dag run to failed.
+ if commit:
+ _set_dag_run_state(dag.dag_id, execution_date, State.FAILED, session)
+
+ # Mark only RUNNING task instances.
+ TI = TaskInstance
+ task_ids = [task.task_id for task in dag.tasks]
+ tis = session.query(TI).filter(
+ TI.dag_id == dag.dag_id,
+ TI.execution_date == execution_date,
+ TI.task_id.in_(task_ids)).filter(TI.state == State.RUNNING)
+ task_ids_of_running_tis = [ti.task_id for ti in tis]
+ for task in dag.tasks:
+ if task.task_id not in task_ids_of_running_tis:
+ continue
+ task.dag = dag
+ new_state = set_state(task=task, execution_date=execution_date,
+ state=State.FAILED, commit=commit)
+ res.extend(new_state)
+
+ return res
+
+
+@provide_session
+def set_dag_run_state_to_running(dag, execution_date, commit=False,
+ session=None):
+ """
+ Set the dag run for a specific execution date to running.
+ :param dag: the DAG of which to alter state
+ :param execution_date: the execution date from which to start looking
+ :param commit: commit DAG and tasks to be altered to the database
+ :param session: database session
+ :return: If commit is true, list of tasks that have been updated,
+ otherwise list of tasks that will be updated
+ """
+ res = []
+ if not dag or not execution_date:
+ return res
+
+ # Mark the dag run to running.
if commit:
- drs = DagRun.find(dag.dag_id, execution_date=execution_date)
- for dr in drs:
- dr.dag = dag
- dr.update_state()
+ _set_dag_run_state(dag.dag_id, execution_date, State.RUNNING, session)
+ # To keep the return type consistent with the other similar functions.
return res
diff --git a/airflow/api/common/experimental/pool.py b/airflow/api/common/experimental/pool.py
index 6e963a2fd2..f125036188 100644
--- a/airflow/api/common/experimental/pool.py
+++ b/airflow/api/common/experimental/pool.py
@@ -1,35 +1,32 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from airflow.exceptions import AirflowException
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.exceptions import AirflowBadRequest, PoolNotFound
from airflow.models import Pool
from airflow.utils.db import provide_session
-class PoolBadRequest(AirflowException):
- status = 400
-
-
-class PoolNotFound(AirflowException):
- status = 404
-
-
@provide_session
def get_pool(name, session=None):
"""Get pool by a given name."""
if not (name and name.strip()):
- raise PoolBadRequest("Pool name shouldn't be empty")
+ raise AirflowBadRequest("Pool name shouldn't be empty")
pool = session.query(Pool).filter_by(pool=name).first()
if pool is None:
@@ -48,12 +45,12 @@ def get_pools(session=None):
def create_pool(name, slots, description, session=None):
"""Create a pool with a given parameters."""
if not (name and name.strip()):
- raise PoolBadRequest("Pool name shouldn't be empty")
+ raise AirflowBadRequest("Pool name shouldn't be empty")
try:
slots = int(slots)
except ValueError:
- raise PoolBadRequest("Bad value for `slots`: %s" % slots)
+ raise AirflowBadRequest("Bad value for `slots`: %s" % slots)
session.expire_on_commit = False
pool = session.query(Pool).filter_by(pool=name).first()
@@ -73,7 +70,7 @@ def create_pool(name, slots, description, session=None):
def delete_pool(name, session=None):
"""Delete pool by a given name."""
if not (name and name.strip()):
- raise PoolBadRequest("Pool name shouldn't be empty")
+ raise AirflowBadRequest("Pool name shouldn't be empty")
pool = session.query(Pool).filter_by(pool=name).first()
if pool is None:
diff --git a/airflow/api/common/experimental/trigger_dag.py b/airflow/api/common/experimental/trigger_dag.py
index 9d9934d2f4..0268752565 100644
--- a/airflow/api/common/experimental/trigger_dag.py
+++ b/airflow/api/common/experimental/trigger_dag.py
@@ -1,59 +1,107 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
import json
-from airflow.exceptions import AirflowException
-from airflow.models import DagRun, DagBag
+from airflow.exceptions import DagRunAlreadyExists, DagNotFound
+from airflow.models import DagRun, DagBag, DagModel
from airflow.utils import timezone
from airflow.utils.state import State
-def trigger_dag(dag_id, run_id=None, conf=None, execution_date=None):
- dagbag = DagBag()
+def _trigger_dag(
+ dag_id,
+ dag_bag,
+ dag_run,
+ run_id,
+ conf,
+ execution_date,
+ replace_microseconds,
+):
+ if dag_id not in dag_bag.dags:
+ raise DagNotFound("Dag id {} not found".format(dag_id))
- if dag_id not in dagbag.dags:
- raise AirflowException("Dag id {} not found".format(dag_id))
-
- dag = dagbag.get_dag(dag_id)
+ dag = dag_bag.get_dag(dag_id)
if not execution_date:
execution_date = timezone.utcnow()
assert timezone.is_localized(execution_date)
- execution_date = execution_date.replace(microsecond=0)
+
+ if replace_microseconds:
+ execution_date = execution_date.replace(microsecond=0)
if not run_id:
run_id = "manual__{0}".format(execution_date.isoformat())
- dr = DagRun.find(dag_id=dag_id, run_id=run_id)
+ dr = dag_run.find(dag_id=dag_id, run_id=run_id)
if dr:
- raise AirflowException("Run id {} already exists for dag id {}".format(
+ raise DagRunAlreadyExists("Run id {} already exists for dag id {}".format(
run_id,
dag_id
))
run_conf = None
if conf:
- run_conf = json.loads(conf)
+ if type(conf) is dict:
+ run_conf = conf
+ else:
+ run_conf = json.loads(conf)
+
+ triggers = list()
+ dags_to_trigger = list()
+ dags_to_trigger.append(dag)
+ while dags_to_trigger:
+ dag = dags_to_trigger.pop()
+ trigger = dag.create_dagrun(
+ run_id=run_id,
+ execution_date=execution_date,
+ state=State.RUNNING,
+ conf=run_conf,
+ external_trigger=True,
+ )
+ triggers.append(trigger)
+ if dag.subdags:
+ dags_to_trigger.extend(dag.subdags)
+ return triggers
+
- trigger = dag.create_dagrun(
+def trigger_dag(
+ dag_id,
+ run_id=None,
+ conf=None,
+ execution_date=None,
+ replace_microseconds=True,
+):
+ dag_model = DagModel.get_current(dag_id)
+ if dag_model is None:
+ raise DagNotFound("Dag id {} not found in DagModel".format(dag_id))
+ dagbag = DagBag(dag_folder=dag_model.fileloc)
+ dag_run = DagRun()
+ triggers = _trigger_dag(
+ dag_id=dag_id,
+ dag_run=dag_run,
+ dag_bag=dagbag,
run_id=run_id,
+ conf=conf,
execution_date=execution_date,
- state=State.RUNNING,
- conf=run_conf,
- external_trigger=True
+ replace_microseconds=replace_microseconds,
)
- return trigger
+ return triggers[0] if triggers else None
diff --git a/airflow/bin/__init__.py b/airflow/bin/__init__.py
index c82f5790fe..114d189da1 100644
--- a/airflow/bin/__init__.py
+++ b/airflow/bin/__init__.py
@@ -1,14 +1,18 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/bin/airflow b/airflow/bin/airflow
index 2c0024d240..d0b7db3bf5 100755
--- a/airflow/bin/airflow
+++ b/airflow/bin/airflow
@@ -1,26 +1,31 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
import os
from airflow import configuration
from airflow.bin.cli import CLIFactory
if __name__ == '__main__':
- if configuration.get("core", "security") == 'kerberos':
- os.environ['KRB5CCNAME'] = configuration.get('kerberos', 'ccache')
- os.environ['KRB5_KTNAME'] = configuration.get('kerberos', 'keytab')
+ if configuration.conf.get("core", "security") == 'kerberos':
+ os.environ['KRB5CCNAME'] = configuration.conf.get('kerberos', 'ccache')
+ os.environ['KRB5_KTNAME'] = configuration.conf.get('kerberos', 'keytab')
parser = CLIFactory.get_parser()
args = parser.parse_args()
diff --git a/airflow/bin/airflow_scheduler_autorestart.sh b/airflow/bin/airflow_scheduler_autorestart.sh
index 910404d39b..7bb7c1b4a1 100755
--- a/airflow/bin/airflow_scheduler_autorestart.sh
+++ b/airflow/bin/airflow_scheduler_autorestart.sh
@@ -1,15 +1,20 @@
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
while echo "Running"; do
airflow scheduler -n 5
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
old mode 100755
new mode 100644
index b032729433..877bf34e20
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -1,33 +1,40 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from __future__ import print_function
import logging
-import reprlib
-
import os
-import socket
import subprocess
import textwrap
+import random
+import string
from importlib import import_module
+import getpass
+import reprlib
import argparse
from builtins import input
from collections import namedtuple
-from dateutil.parser import parse as parsedate
+
+from airflow.utils.timezone import parse as parsedate
import json
from tabulate import tabulate
@@ -46,19 +53,22 @@
from airflow import api
from airflow import jobs, settings
from airflow import configuration as conf
-from airflow.exceptions import AirflowException
+from airflow.exceptions import AirflowException, AirflowWebServerTimeout
from airflow.executors import GetDefaultExecutor
-from airflow.models import (DagModel, DagBag, TaskInstance,
- DagPickle, DagRun, Variable, DagStat,
- Connection, DAG)
-
+from airflow.models import DagModel, DagBag, TaskInstance, DagRun, Variable, DAG
+from airflow.models.connection import Connection
+from airflow.models.dagpickle import DagPickle
from airflow.ti_deps.dep_context import (DepContext, SCHEDULER_DEPS)
+from airflow.utils import cli as cli_utils
from airflow.utils import db as db_utils
+from airflow.utils.net import get_hostname
from airflow.utils.log.logging_mixin import (LoggingMixin, redirect_stderr,
- redirect_stdout, set_context)
-from airflow.www.app import cached_app
+ redirect_stdout)
+from airflow.www.app import (cached_app, create_app)
+from airflow.www_rbac.app import cached_app as cached_app_rbac
+from airflow.www_rbac.app import create_app as create_app_rbac
+from airflow.www_rbac.app import cached_appbuilder
-from sqlalchemy import func
from sqlalchemy.orm import exc
api.load_auth()
@@ -68,6 +78,11 @@
log = LoggingMixin().log
+DAGS_FOLDER = settings.DAGS_FOLDER
+
+if "BUILDING_AIRFLOW_DOCS" in os.environ:
+ DAGS_FOLDER = '[AIRFLOW_HOME]/dags'
+
def sigint_handler(sig, frame):
sys.exit(0)
@@ -104,20 +119,24 @@ def setup_logging(filename):
def setup_locations(process, pid=None, stdout=None, stderr=None, log=None):
if not stderr:
- stderr = os.path.join(os.path.expanduser(settings.AIRFLOW_HOME), "airflow-{}.err".format(process))
+ stderr = os.path.join(os.path.expanduser(settings.AIRFLOW_HOME),
+ 'airflow-{}.err'.format(process))
if not stdout:
- stdout = os.path.join(os.path.expanduser(settings.AIRFLOW_HOME), "airflow-{}.out".format(process))
+ stdout = os.path.join(os.path.expanduser(settings.AIRFLOW_HOME),
+ 'airflow-{}.out'.format(process))
if not log:
- log = os.path.join(os.path.expanduser(settings.AIRFLOW_HOME), "airflow-{}.log".format(process))
+ log = os.path.join(os.path.expanduser(settings.AIRFLOW_HOME),
+ 'airflow-{}.log'.format(process))
if not pid:
- pid = os.path.join(os.path.expanduser(settings.AIRFLOW_HOME), "airflow-{}.pid".format(process))
+ pid = os.path.join(os.path.expanduser(settings.AIRFLOW_HOME),
+ 'airflow-{}.pid'.format(process))
return pid, stdout, stderr, log
def process_subdir(subdir):
if subdir:
- subdir = subdir.replace('DAGS_FOLDER', settings.DAGS_FOLDER)
+ subdir = subdir.replace('DAGS_FOLDER', DAGS_FOLDER)
subdir = os.path.abspath(os.path.expanduser(subdir))
return subdir
@@ -144,6 +163,7 @@ def get_dags(args):
return matched_dags
+@cli_utils.action_logging
def backfill(args, dag=None):
logging.basicConfig(
level=settings.LOGGING_LEVEL,
@@ -163,6 +183,10 @@ def backfill(args, dag=None):
task_regex=args.task_regex,
include_upstream=not args.ignore_dependencies)
+ run_conf = None
+ if args.conf:
+ run_conf = json.loads(args.conf)
+
if args.dry_run:
print("Dry run of DAG {0} on {1}".format(args.dag_id,
args.start_date))
@@ -171,20 +195,33 @@ def backfill(args, dag=None):
ti = TaskInstance(task, args.start_date)
ti.dry_run()
else:
+ if args.reset_dagruns:
+ DAG.clear_dags(
+ [dag],
+ start_date=args.start_date,
+ end_date=args.end_date,
+ confirm_prompt=True,
+ include_subdags=True,
+ )
+
dag.run(
start_date=args.start_date,
end_date=args.end_date,
mark_success=args.mark_success,
- include_adhoc=args.include_adhoc,
local=args.local,
donot_pickle=(args.donot_pickle or
conf.getboolean('core', 'donot_pickle')),
ignore_first_depends_on_past=args.ignore_first_depends_on_past,
ignore_task_deps=args.ignore_dependencies,
pool=args.pool,
- delay_on_limit_secs=args.delay_on_limit)
+ delay_on_limit_secs=args.delay_on_limit,
+ verbose=args.verbose,
+ conf=run_conf,
+ rerun_failed_tasks=args.rerun_failed_tasks,
+ )
+@cli_utils.action_logging
def trigger_dag(args):
"""
Creates a dag run for the specified dag
@@ -203,6 +240,28 @@ def trigger_dag(args):
log.info(message)
+@cli_utils.action_logging
+def delete_dag(args):
+ """
+ Deletes all DB records related to the specified dag
+ :param args:
+ :return:
+ """
+ log = LoggingMixin().log
+ if args.yes or input(
+ "This will drop all existing records related to the specified DAG. "
+ "Proceed? (y/n)").upper() == "Y":
+ try:
+ message = api_client.delete_dag(dag_id=args.dag_id)
+ except IOError as err:
+ log.error(err)
+ raise AirflowException(err)
+ log.info(message)
+ else:
+ print("Bail.")
+
+
+@cli_utils.action_logging
def pool(args):
log = LoggingMixin().log
@@ -211,6 +270,7 @@ def _tabulate(pools):
tablefmt="fancy_grid")
try:
+ imp = getattr(args, 'import')
if args.get is not None:
pools = [api_client.get_pool(name=args.get)]
elif args.set:
@@ -219,6 +279,14 @@ def _tabulate(pools):
description=args.set[2])]
elif args.delete:
pools = [api_client.delete_pool(name=args.delete)]
+ elif imp:
+ if os.path.exists(imp):
+ pools = pool_import_helper(imp)
+ else:
+ print("Missing pools file.")
+ pools = api_client.get_pools()
+ elif args.export:
+ pools = pool_export_helper(args.export)
else:
pools = api_client.get_pools()
except (AirflowException, IOError) as err:
@@ -227,6 +295,44 @@ def _tabulate(pools):
log.info(_tabulate(pools=pools))
+def pool_import_helper(filepath):
+ with open(filepath, 'r') as poolfile:
+ pl = poolfile.read()
+ try:
+ d = json.loads(pl)
+ except Exception as e:
+ print("Please check the validity of the json file: " + str(e))
+ else:
+ try:
+ pools = []
+ n = 0
+ for k, v in d.items():
+ if isinstance(v, dict) and len(v) == 2:
+ pools.append(api_client.create_pool(name=k,
+ slots=v["slots"],
+ description=v["description"]))
+ n += 1
+ else:
+ pass
+ except Exception:
+ pass
+ finally:
+ print("{} of {} pool(s) successfully updated.".format(n, len(d)))
+ return pools
+
+
+def pool_export_helper(filepath):
+ pool_dict = {}
+ pools = api_client.get_pools()
+ for pool in pools:
+ pool_dict[pool[0]] = {"slots": pool[1], "description": pool[2]}
+ with open(filepath, 'w') as poolfile:
+ poolfile.write(json.dumps(pool_dict, sort_keys=True, indent=4))
+ print("{} pools successfully exported to {}".format(len(pool_dict), filepath))
+ return pools
+
+
+@cli_utils.action_logging
def variables(args):
if args.get:
try:
@@ -303,10 +409,12 @@ def export_helper(filepath):
print("{} variables successfully exported to {}".format(len(var_dict), filepath))
+@cli_utils.action_logging
def pause(args, dag=None):
set_is_paused(True, args, dag)
+@cli_utils.action_logging
def unpause(args, dag=None):
set_is_paused(False, args, dag)
@@ -324,11 +432,60 @@ def set_is_paused(is_paused, args, dag=None):
print(msg)
-def run(args, dag=None):
- # Disable connection pooling to reduce the # of connections on the DB
- # while it's waiting for the task to finish.
- settings.configure_orm(disable_connection_pool=True)
+def _run(args, dag, ti):
+ if args.local:
+ run_job = jobs.LocalTaskJob(
+ task_instance=ti,
+ mark_success=args.mark_success,
+ pickle_id=args.pickle,
+ ignore_all_deps=args.ignore_all_dependencies,
+ ignore_depends_on_past=args.ignore_depends_on_past,
+ ignore_task_deps=args.ignore_dependencies,
+ ignore_ti_state=args.force,
+ pool=args.pool)
+ run_job.run()
+ elif args.raw:
+ ti._run_raw_task(
+ mark_success=args.mark_success,
+ job_id=args.job_id,
+ pool=args.pool,
+ )
+ else:
+ pickle_id = None
+ if args.ship_dag:
+ try:
+ # Running remotely, so pickling the DAG
+ session = settings.Session()
+ pickle = DagPickle(dag)
+ session.add(pickle)
+ session.commit()
+ pickle_id = pickle.id
+ # TODO: This should be written to a log
+ print('Pickled dag {dag} as pickle_id:{pickle_id}'
+ .format(**locals()))
+ except Exception as e:
+ print('Could not pickle the DAG')
+ print(e)
+ raise e
+
+ executor = GetDefaultExecutor()
+ executor.start()
+ print("Sending to executor.")
+ executor.queue_task_instance(
+ ti,
+ mark_success=args.mark_success,
+ pickle_id=pickle_id,
+ ignore_all_deps=args.ignore_all_dependencies,
+ ignore_depends_on_past=args.ignore_depends_on_past,
+ ignore_task_deps=args.ignore_dependencies,
+ ignore_ti_state=args.force,
+ pool=args.pool)
+ executor.heartbeat()
+ executor.end()
+
+@cli_utils.action_logging
+def run(args, dag=None):
if dag:
args.dag_id = dag.dag_id
@@ -342,11 +499,14 @@ def run(args, dag=None):
if os.path.exists(args.cfg_path):
os.remove(args.cfg_path)
- for section, config in conf_dict.items():
- for option, value in config.items():
- conf.set(section, option, value)
+ conf.conf.read_dict(conf_dict, source=args.cfg_path)
settings.configure_vars()
- settings.configure_orm()
+
+ # IMPORTANT, have to use the NullPool, otherwise, each "run" command may leave
+ # behind multiple open sleeping connections while heartbeating, which could
+ # easily exceed the database connection limit when
+ # processing hundreds of simultaneous tasks.
+ settings.configure_orm(disable_connection_pool=True)
if not args.pickle and not dag:
dag = get_dag(args)
@@ -365,74 +525,28 @@ def run(args, dag=None):
ti.init_run_context(raw=args.raw)
- hostname = socket.getfqdn()
+ hostname = get_hostname()
log.info("Running %s on host %s", ti, hostname)
- with redirect_stdout(ti.log, logging.INFO), redirect_stderr(ti.log, logging.WARN):
- if args.local:
- run_job = jobs.LocalTaskJob(
- task_instance=ti,
- mark_success=args.mark_success,
- pickle_id=args.pickle,
- ignore_all_deps=args.ignore_all_dependencies,
- ignore_depends_on_past=args.ignore_depends_on_past,
- ignore_task_deps=args.ignore_dependencies,
- ignore_ti_state=args.force,
- pool=args.pool)
- run_job.run()
- elif args.raw:
- ti._run_raw_task(
- mark_success=args.mark_success,
- job_id=args.job_id,
- pool=args.pool,
- )
- else:
- pickle_id = None
- if args.ship_dag:
- try:
- # Running remotely, so pickling the DAG
- session = settings.Session()
- pickle = DagPickle(dag)
- session.add(pickle)
- session.commit()
- pickle_id = pickle.id
- # TODO: This should be written to a log
- print((
- 'Pickled dag {dag} '
- 'as pickle_id:{pickle_id}').format(**locals()))
- except Exception as e:
- print('Could not pickle the DAG')
- print(e)
- raise e
-
- executor = GetDefaultExecutor()
- executor.start()
- print("Sending to executor.")
- executor.queue_task_instance(
- ti,
- mark_success=args.mark_success,
- pickle_id=pickle_id,
- ignore_all_deps=args.ignore_all_dependencies,
- ignore_depends_on_past=args.ignore_depends_on_past,
- ignore_task_deps=args.ignore_dependencies,
- ignore_ti_state=args.force,
- pool=args.pool)
- executor.heartbeat()
- executor.end()
-
+ if args.interactive:
+ _run(args, dag, ti)
+ else:
+ with redirect_stdout(ti.log, logging.INFO), redirect_stderr(ti.log, logging.WARN):
+ _run(args, dag, ti)
logging.shutdown()
+@cli_utils.action_logging
def task_failed_deps(args):
"""
Returns the unmet dependencies for a task instance from the perspective of the
scheduler (i.e. why a task instance doesn't get scheduled and then queued by the
scheduler, and then run by an executor).
-
>>> airflow task_failed_deps tutorial sleep 2015-01-01
Task instance dependencies not met:
Dagrun Running: Task instance's dagrun did not exist: Unknown reason
- Trigger Rule: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es).
+ Trigger Rule: Task's trigger rule 'all_success' requires all upstream tasks
+ to have succeeded, but found 1 non-success(es).
"""
dag = get_dag(args)
task = dag.get_task(task_id=args.task_id)
@@ -449,10 +563,10 @@ def task_failed_deps(args):
print("Task instance dependencies are all met.")
+@cli_utils.action_logging
def task_state(args):
"""
Returns the state of a TaskInstance at the command line.
-
>>> airflow task_state tutorial sleep 2015-01-01
success
"""
@@ -462,10 +576,10 @@ def task_state(args):
print(ti.current_state())
+@cli_utils.action_logging
def dag_state(args):
"""
Returns the state of a DagRun at the command line.
-
>>> airflow dag_state tutorial 2015-01-01T00:00:00.000000
running
"""
@@ -474,6 +588,32 @@ def dag_state(args):
print(dr[0].state if len(dr) > 0 else None)
+@cli_utils.action_logging
+def next_execution(args):
+ """
+ Returns the next execution datetime of a DAG at the command line.
+ >>> airflow next_execution tutorial
+ 2018-08-31 10:38:00
+ """
+ dag = get_dag(args)
+
+ if dag.is_paused:
+ print("[INFO] Please be reminded this DAG is PAUSED now.")
+
+ if dag.latest_execution_date:
+ next_execution_dttm = dag.following_schedule(dag.latest_execution_date)
+
+ if next_execution_dttm is None:
+ print("[WARN] No following schedule can be found. " +
+ "This DAG may have schedule interval '@once' or `None`.")
+
+ print(next_execution_dttm)
+ else:
+ print("[WARN] Only applicable when there is execution record found for the DAG.")
+ print(None)
+
+
+@cli_utils.action_logging
def list_dags(args):
dagbag = DagBag(process_subdir(args.subdir))
s = textwrap.dedent("""\n
@@ -488,6 +628,7 @@ def list_dags(args):
print(dagbag.dagbag_report())
+@cli_utils.action_logging
def list_tasks(args, dag=None):
dag = dag or get_dag(args)
if args.tree:
@@ -497,7 +638,13 @@ def list_tasks(args, dag=None):
print("\n".join(sorted(tasks)))
+@cli_utils.action_logging
def test(args, dag=None):
+ # We want log outout from operators etc to show up here. Normally
+ # airflow.task would redirect to a file, but here we want it to propagate
+ # up to the normal airflow handler.
+ logging.getLogger('airflow.task').propagate = True
+
dag = dag or get_dag(args)
task = dag.get_task(task_id=args.task_id)
@@ -513,6 +660,7 @@ def test(args, dag=None):
ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
+@cli_utils.action_logging
def render(args):
dag = get_dag(args)
task = dag.get_task(task_id=args.task_id)
@@ -527,6 +675,7 @@ def render(args):
""".format(attr, getattr(task, attr))))
+@cli_utils.action_logging
def clear(args):
logging.basicConfig(
level=settings.LOGGING_LEVEL,
@@ -547,7 +696,9 @@ def clear(args):
only_failed=args.only_failed,
only_running=args.only_running,
confirm_prompt=not args.no_confirm,
- include_subdags=not args.exclude_subdags)
+ include_subdags=not args.exclude_subdags,
+ include_parentdag=not args.exclude_parentdag,
+ )
def get_num_ready_workers_running(gunicorn_master_proc):
@@ -566,43 +717,45 @@ def ready_prefix_on_cmdline(proc):
return len(ready_workers)
-def restart_workers(gunicorn_master_proc, num_workers_expected):
+def get_num_workers_running(gunicorn_master_proc):
+ workers = psutil.Process(gunicorn_master_proc.pid).children()
+ return len(workers)
+
+
+def restart_workers(gunicorn_master_proc, num_workers_expected, master_timeout):
"""
Runs forever, monitoring the child processes of @gunicorn_master_proc and
restarting workers occasionally.
-
Each iteration of the loop traverses one edge of this state transition
diagram, where each state (node) represents
[ num_ready_workers_running / num_workers_running ]. We expect most time to
be spent in [n / n]. `bs` is the setting webserver.worker_refresh_batch_size.
-
The horizontal transition at ? happens after the new worker parses all the
dags (so it could take a while!)
-
V ────────────────────────────────────────────────────────────────────────┐
[n / n] ──TTIN──> [ [n, n+bs) / n + bs ] ────?───> [n + bs / n + bs] ──TTOU─┘
^ ^───────────────┘
│
│ ┌────────────────v
└──────┴────── [ [0, n) / n ] <─── start
-
We change the number of workers by sending TTIN and TTOU to the gunicorn
master process, which increases and decreases the number of child workers
respectively. Gunicorn guarantees that on TTOU workers are terminated
gracefully and that the oldest worker is terminated.
"""
- def wait_until_true(fn):
+ def wait_until_true(fn, timeout=0):
"""
Sleeps until fn is true
"""
+ t = time.time()
while not fn():
+ if 0 < timeout and timeout <= time.time() - t:
+ raise AirflowWebServerTimeout(
+ "No response from gunicorn master within {0} seconds"
+ .format(timeout))
time.sleep(0.1)
- def get_num_workers_running(gunicorn_master_proc):
- workers = psutil.Process(gunicorn_master_proc.pid).children()
- return len(workers)
-
def start_refresh(gunicorn_master_proc):
batch_size = conf.getint('webserver', 'worker_refresh_batch_size')
log.debug('%s doing a refresh of %s workers', state, batch_size)
@@ -614,61 +767,73 @@ def start_refresh(gunicorn_master_proc):
gunicorn_master_proc.send_signal(signal.SIGTTIN)
excess += 1
wait_until_true(lambda: num_workers_expected + excess ==
- get_num_workers_running(gunicorn_master_proc))
-
- wait_until_true(lambda: num_workers_expected ==
- get_num_workers_running(gunicorn_master_proc))
-
- while True:
- num_workers_running = get_num_workers_running(gunicorn_master_proc)
- num_ready_workers_running = get_num_ready_workers_running(gunicorn_master_proc)
-
- state = '[{0} / {1}]'.format(num_ready_workers_running, num_workers_running)
-
- # Whenever some workers are not ready, wait until all workers are ready
- if num_ready_workers_running < num_workers_running:
- log.debug('%s some workers are starting up, waiting...', state)
- sys.stdout.flush()
- time.sleep(1)
-
- # Kill a worker gracefully by asking gunicorn to reduce number of workers
- elif num_workers_running > num_workers_expected:
- excess = num_workers_running - num_workers_expected
- log.debug('%s killing %s workers', state, excess)
-
- for _ in range(excess):
- gunicorn_master_proc.send_signal(signal.SIGTTOU)
- excess -= 1
- wait_until_true(lambda: num_workers_expected + excess ==
- get_num_workers_running(gunicorn_master_proc))
-
- # Start a new worker by asking gunicorn to increase number of workers
- elif num_workers_running == num_workers_expected:
- refresh_interval = conf.getint('webserver', 'worker_refresh_interval')
- log.debug(
- '%s sleeping for %ss starting doing a refresh...',
- state, refresh_interval
- )
- time.sleep(refresh_interval)
- start_refresh(gunicorn_master_proc)
+ get_num_workers_running(gunicorn_master_proc),
+ master_timeout)
- else:
- # num_ready_workers_running == num_workers_running < num_workers_expected
- log.error((
- "%s some workers seem to have died and gunicorn"
- "did not restart them as expected"
- ), state)
- time.sleep(10)
- if len(
- psutil.Process(gunicorn_master_proc.pid).children()
- ) < num_workers_expected:
+ try:
+ wait_until_true(lambda: num_workers_expected ==
+ get_num_workers_running(gunicorn_master_proc),
+ master_timeout)
+ while True:
+ num_workers_running = get_num_workers_running(gunicorn_master_proc)
+ num_ready_workers_running = \
+ get_num_ready_workers_running(gunicorn_master_proc)
+
+ state = '[{0} / {1}]'.format(num_ready_workers_running, num_workers_running)
+
+ # Whenever some workers are not ready, wait until all workers are ready
+ if num_ready_workers_running < num_workers_running:
+ log.debug('%s some workers are starting up, waiting...', state)
+ sys.stdout.flush()
+ time.sleep(1)
+
+ # Kill a worker gracefully by asking gunicorn to reduce number of workers
+ elif num_workers_running > num_workers_expected:
+ excess = num_workers_running - num_workers_expected
+ log.debug('%s killing %s workers', state, excess)
+
+ for _ in range(excess):
+ gunicorn_master_proc.send_signal(signal.SIGTTOU)
+ excess -= 1
+ wait_until_true(lambda: num_workers_expected + excess ==
+ get_num_workers_running(gunicorn_master_proc),
+ master_timeout)
+
+ # Start a new worker by asking gunicorn to increase number of workers
+ elif num_workers_running == num_workers_expected:
+ refresh_interval = conf.getint('webserver', 'worker_refresh_interval')
+ log.debug(
+ '%s sleeping for %ss starting doing a refresh...',
+ state, refresh_interval
+ )
+ time.sleep(refresh_interval)
start_refresh(gunicorn_master_proc)
+ else:
+ # num_ready_workers_running == num_workers_running < num_workers_expected
+ log.error((
+ "%s some workers seem to have died and gunicorn"
+ "did not restart them as expected"
+ ), state)
+ time.sleep(10)
+ if len(
+ psutil.Process(gunicorn_master_proc.pid).children()
+ ) < num_workers_expected:
+ start_refresh(gunicorn_master_proc)
+ except (AirflowWebServerTimeout, OSError) as err:
+ log.error(err)
+ log.error("Shutting down webserver")
+ try:
+ gunicorn_master_proc.terminate()
+ gunicorn_master_proc.wait()
+ finally:
+ sys.exit(1)
+
+@cli_utils.action_logging
def webserver(args):
print(settings.HEADER)
- app = cached_app(conf)
access_logfile = args.access_logfile or conf.get('webserver', 'access_logfile')
error_logfile = args.error_logfile or conf.get('webserver', 'error_logfile')
num_workers = args.workers or conf.get('webserver', 'workers')
@@ -687,10 +852,19 @@ def webserver(args):
print(
"Starting the web server on port {0} and host {1}.".format(
args.port, args.hostname))
- app.run(debug=True, port=args.port, host=args.hostname,
+ if settings.RBAC:
+ app, _ = create_app_rbac(conf, testing=conf.get('core', 'unit_test_mode'))
+ else:
+ app = create_app(conf, testing=conf.get('core', 'unit_test_mode'))
+ app.run(debug=True, use_reloader=False if app.config['TESTING'] else True,
+ port=args.port, host=args.hostname,
ssl_context=(ssl_cert, ssl_key) if ssl_cert and ssl_key else None)
else:
- pid, stdout, stderr, log_file = setup_locations("webserver", args.pid, args.stdout, args.stderr, args.log_file)
+ os.environ['SKIP_DAGS_PARSING'] = 'True'
+ app = cached_app_rbac(conf) if settings.RBAC else cached_app(conf)
+ pid, stdout, stderr, log_file = setup_locations(
+ "webserver", args.pid, args.stdout, args.stderr, args.log_file)
+ os.environ.pop('SKIP_DAGS_PARSING')
if args.daemon:
handle = setup_logging(log_file)
stdout = open(stdout, 'w+')
@@ -714,7 +888,7 @@ def webserver(args):
'-b', args.hostname + ':' + str(args.port),
'-n', 'airflow-webserver',
'-p', str(pid),
- '-c', 'python:airflow.www.gunicorn_config'
+ '-c', 'python:airflow.www.gunicorn_config',
]
if args.access_logfile:
@@ -729,7 +903,8 @@ def webserver(args):
if ssl_cert:
run_args += ['--certfile', ssl_cert, '--keyfile', ssl_key]
- run_args += ["airflow.www.app:cached_app()"]
+ webserver_module = 'www_rbac' if settings.RBAC else 'www'
+ run_args += ["airflow." + webserver_module + ".app:cached_app()"]
gunicorn_master_proc = None
@@ -741,11 +916,14 @@ def kill_proc(dummy_signum, dummy_frame):
def monitor_gunicorn(gunicorn_master_proc):
# These run forever until SIG{INT, TERM, KILL, ...} signal is sent
if conf.getint('webserver', 'worker_refresh_interval') > 0:
- restart_workers(gunicorn_master_proc, num_workers)
+ master_timeout = conf.getint('webserver', 'web_server_master_timeout')
+ restart_workers(gunicorn_master_proc, num_workers, master_timeout)
else:
- while True:
+ while gunicorn_master_proc.poll() is None:
time.sleep(1)
+ sys.exit(gunicorn_master_proc.returncode)
+
if args.daemon:
base, ext = os.path.splitext(pid)
ctx = daemon.DaemonContext(
@@ -786,6 +964,7 @@ def monitor_gunicorn(gunicorn_master_proc):
monitor_gunicorn(gunicorn_master_proc)
+@cli_utils.action_logging
def scheduler(args):
print(settings.HEADER)
job = jobs.SchedulerJob(
@@ -796,7 +975,11 @@ def scheduler(args):
do_pickle=args.do_pickle)
if args.daemon:
- pid, stdout, stderr, log_file = setup_locations("scheduler", args.pid, args.stdout, args.stderr, args.log_file)
+ pid, stdout, stderr, log_file = setup_locations("scheduler",
+ args.pid,
+ args.stdout,
+ args.stderr,
+ args.log_file)
handle = setup_logging(log_file)
stdout = open(stdout, 'w+')
stderr = open(stderr, 'w+')
@@ -819,13 +1002,14 @@ def scheduler(args):
job.run()
+@cli_utils.action_logging
def serve_logs(args):
print("Starting flask")
import flask
flask_app = flask.Flask(__name__)
@flask_app.route('/log/<path:filename>')
- def serve_logs(filename): # noqa
+ def serve_logs(filename):
log = os.path.expanduser(conf.get('core', 'BASE_LOG_FOLDER'))
return flask.send_from_directory(
log,
@@ -839,25 +1023,43 @@ def serve_logs(filename): # noqa
host='0.0.0.0', port=WORKER_LOG_SERVER_PORT)
+@cli_utils.action_logging
def worker(args):
env = os.environ.copy()
env['AIRFLOW_HOME'] = settings.AIRFLOW_HOME
+ if not settings.validate_session():
+ log = LoggingMixin().log
+ log.error("Worker exiting... database connection precheck failed! ")
+ sys.exit(1)
+
# Celery worker
from airflow.executors.celery_executor import app as celery_app
from celery.bin import worker
+ autoscale = args.autoscale
+ if autoscale is None and conf.has_option("celery", "worker_autoscale"):
+ autoscale = conf.get("celery", "worker_autoscale")
worker = worker.worker(app=celery_app)
options = {
'optimization': 'fair',
'O': 'fair',
'queues': args.queues,
'concurrency': args.concurrency,
+ 'autoscale': autoscale,
'hostname': args.celery_hostname,
+ 'loglevel': conf.get('core', 'LOGGING_LEVEL'),
}
+ if conf.has_option("celery", "pool"):
+ options["pool"] = conf.get("celery", "pool")
+
if args.daemon:
- pid, stdout, stderr, log_file = setup_locations("worker", args.pid, args.stdout, args.stderr, args.log_file)
+ pid, stdout, stderr, log_file = setup_locations("worker",
+ args.pid,
+ args.stdout,
+ args.stderr,
+ args.log_file)
handle = setup_logging(log_file)
stdout = open(stdout, 'w+')
stderr = open(stderr, 'w+')
@@ -885,40 +1087,30 @@ def worker(args):
sp.kill()
-def initdb(args): # noqa
+def initdb(args):
print("DB: " + repr(settings.engine.url))
- db_utils.initdb()
+ db_utils.initdb(settings.RBAC)
print("Done.")
def resetdb(args):
print("DB: " + repr(settings.engine.url))
- if args.yes or input(
- "This will drop existing tables if they exist. "
- "Proceed? (y/n)").upper() == "Y":
- db_utils.resetdb()
+ if args.yes or input("This will drop existing tables "
+ "if they exist. Proceed? "
+ "(y/n)").upper() == "Y":
+ db_utils.resetdb(settings.RBAC)
else:
print("Bail.")
-def upgradedb(args): # noqa
+@cli_utils.action_logging
+def upgradedb(args):
print("DB: " + repr(settings.engine.url))
db_utils.upgradedb()
- # Populate DagStats table
- session = settings.Session()
- ds_rows = session.query(DagStat).count()
- if not ds_rows:
- qry = (
- session.query(DagRun.dag_id, DagRun.state, func.count('*'))
- .group_by(DagRun.dag_id, DagRun.state)
- )
- for dag_id, state, count in qry:
- session.add(DagStat(dag_id=dag_id, state=state, count=count))
- session.commit()
-
-def version(args): # noqa
+@cli_utils.action_logging
+def version(args):
print(settings.HEADER + " v" + airflow.__version__)
@@ -926,6 +1118,7 @@ def version(args): # noqa
'conn_login', 'conn_password', 'conn_schema', 'conn_port']
+@cli_utils.action_logging
def connections(args):
if args.list:
# Check that no other flags were passed to the command
@@ -947,9 +1140,12 @@ def connections(args):
Connection.is_extra_encrypted,
Connection.extra).all()
conns = [map(reprlib.repr, conn) for conn in conns]
- print(tabulate(conns, ['Conn Id', 'Conn Type', 'Host', 'Port',
+ msg = tabulate(conns, ['Conn Id', 'Conn Type', 'Host', 'Port',
'Is Encrypted', 'Is Extra Encrypted', 'Extra'],
- tablefmt="fancy_grid"))
+ tablefmt="fancy_grid")
+ if sys.version_info[0] < 3:
+ msg = msg.encode('utf-8')
+ print(msg)
return
if args.delete:
@@ -1023,20 +1219,31 @@ def connections(args):
if args.conn_uri:
new_conn = Connection(conn_id=args.conn_id, uri=args.conn_uri)
else:
- new_conn = Connection(conn_id=args.conn_id, conn_type=args.conn_type, host=args.conn_host,
- login=args.conn_login, password=args.conn_password, schema=args.conn_schema, port=args.conn_port)
+ new_conn = Connection(conn_id=args.conn_id,
+ conn_type=args.conn_type,
+ host=args.conn_host,
+ login=args.conn_login,
+ password=args.conn_password,
+ schema=args.conn_schema,
+ port=args.conn_port)
if args.conn_extra is not None:
new_conn.set_extra(args.conn_extra)
session = settings.Session()
- if not (session
- .query(Connection)
- .filter(Connection.conn_id == new_conn.conn_id).first()):
+ if not (session.query(Connection)
+ .filter(Connection.conn_id == new_conn.conn_id).first()):
session.add(new_conn)
session.commit()
msg = '\n\tSuccessfully added `conn_id`={conn_id} : {uri}\n'
- msg = msg.format(conn_id=new_conn.conn_id, uri=args.conn_uri or urlunparse((args.conn_type, '{login}:{password}@{host}:{port}'.format(
- login=args.conn_login or '', password=args.conn_password or '', host=args.conn_host or '', port=args.conn_port or ''), args.conn_schema or '', '', '', '')))
+ msg = msg.format(conn_id=new_conn.conn_id,
+ uri=args.conn_uri or
+ urlunparse((args.conn_type,
+ '{login}:{password}@{host}:{port}'
+ .format(login=args.conn_login or '',
+ password=args.conn_password or '',
+ host=args.conn_host or '',
+ port=args.conn_port or ''),
+ args.conn_schema or '', '', '', '')))
print(msg)
else:
msg = '\n\tA connection with `conn_id`={conn_id} already exists\n'
@@ -1046,6 +1253,7 @@ def connections(args):
return
+@cli_utils.action_logging
def flower(args):
broka = conf.get('celery', 'BROKER_URL')
address = '--address={}'.format(args.hostname)
@@ -1058,12 +1266,20 @@ def flower(args):
if args.url_prefix:
url_prefix = '--url-prefix=' + args.url_prefix
+ basic_auth = ''
+ if args.basic_auth:
+ basic_auth = '--basic_auth=' + args.basic_auth
+
flower_conf = ''
if args.flower_conf:
flower_conf = '--conf=' + args.flower_conf
if args.daemon:
- pid, stdout, stderr, log_file = setup_locations("flower", args.pid, args.stdout, args.stderr, args.log_file)
+ pid, stdout, stderr, log_file = setup_locations("flower",
+ args.pid,
+ args.stdout,
+ args.stderr,
+ args.log_file)
stdout = open(stdout, 'w+')
stderr = open(stderr, 'w+')
@@ -1075,7 +1291,7 @@ def flower(args):
with ctx:
os.execvp("flower", ['flower', '-b',
- broka, address, port, api, flower_conf, url_prefix])
+ broka, address, port, api, flower_conf, url_prefix, basic_auth])
stdout.close()
stderr.close()
@@ -1084,15 +1300,20 @@ def flower(args):
signal.signal(signal.SIGTERM, sigint_handler)
os.execvp("flower", ['flower', '-b',
- broka, address, port, api, flower_conf, url_prefix])
+ broka, address, port, api, flower_conf, url_prefix, basic_auth])
-def kerberos(args): # noqa
+@cli_utils.action_logging
+def kerberos(args):
print(settings.HEADER)
import airflow.security.kerberos
if args.daemon:
- pid, stdout, stderr, log_file = setup_locations("kerberos", args.pid, args.stdout, args.stderr, args.log_file)
+ pid, stdout, stderr, log_file = setup_locations("kerberos",
+ args.pid,
+ args.stdout,
+ args.stderr,
+ args.log_file)
stdout = open(stdout, 'w+')
stderr = open(stderr, 'w+')
@@ -1103,12 +1324,146 @@ def kerberos(args): # noqa
)
with ctx:
- airflow.security.kerberos.run()
+ airflow.security.kerberos.run(principal=args.principal, keytab=args.keytab)
stdout.close()
stderr.close()
else:
- airflow.security.kerberos.run()
+ airflow.security.kerberos.run(principal=args.principal, keytab=args.keytab)
+
+
+@cli_utils.action_logging
+def users(args):
+ if args.list:
+
+ appbuilder = cached_appbuilder()
+ users = appbuilder.sm.get_all_users()
+ fields = ['id', 'username', 'email', 'first_name', 'last_name', 'roles']
+ users = [[user.__getattribute__(field) for field in fields] for user in users]
+ msg = tabulate(users, [field.capitalize().replace('_', ' ') for field in fields],
+ tablefmt="fancy_grid")
+ if sys.version_info[0] < 3:
+ msg = msg.encode('utf-8')
+ print(msg)
+
+ return
+
+ if args.create:
+ fields = {
+ 'role': args.role,
+ 'username': args.username,
+ 'email': args.email,
+ 'firstname': args.firstname,
+ 'lastname': args.lastname,
+ }
+ empty_fields = [k for k, v in fields.items() if not v]
+ if empty_fields:
+ raise SystemExit('Required arguments are missing: {}.'.format(
+ ', '.join(empty_fields)))
+
+ appbuilder = cached_appbuilder()
+ role = appbuilder.sm.find_role(args.role)
+ if not role:
+ raise SystemExit('{} is not a valid role.'.format(args.role))
+
+ if args.use_random_password:
+ password = ''.join(random.choice(string.printable) for _ in range(16))
+ elif args.password:
+ password = args.password
+ else:
+ password = getpass.getpass('Password:')
+ password_confirmation = getpass.getpass('Repeat for confirmation:')
+ if password != password_confirmation:
+ raise SystemExit('Passwords did not match!')
+
+ if appbuilder.sm.find_user(args.username):
+ print('{} already exist in the db'.format(args.username))
+ return
+ user = appbuilder.sm.add_user(args.username, args.firstname, args.lastname,
+ args.email, role, password)
+ if user:
+ print('{} user {} created.'.format(args.role, args.username))
+ else:
+ raise SystemExit('Failed to create user.')
+
+ if args.delete:
+ if not args.username:
+ raise SystemExit('Required arguments are missing: username')
+
+ appbuilder = cached_appbuilder()
+
+ try:
+ u = next(u for u in appbuilder.sm.get_all_users()
+ if u.username == args.username)
+ except StopIteration:
+ raise SystemExit('{} is not a valid user.'.format(args.username))
+
+ if appbuilder.sm.del_register_user(u):
+ print('User {} deleted.'.format(args.username))
+ else:
+ raise SystemExit('Failed to delete user.')
+
+
+@cli_utils.action_logging
+def list_dag_runs(args, dag=None):
+ if dag:
+ args.dag_id = dag.dag_id
+
+ dagbag = DagBag()
+
+ if args.dag_id not in dagbag.dags:
+ error_message = "Dag id {} not found".format(args.dag_id)
+ raise AirflowException(error_message)
+
+ dag_runs = list()
+ state = args.state.lower() if args.state else None
+ for run in DagRun.find(dag_id=args.dag_id,
+ state=state,
+ no_backfills=args.no_backfill):
+ dag_runs.append({
+ 'id': run.id,
+ 'run_id': run.run_id,
+ 'state': run.state,
+ 'dag_id': run.dag_id,
+ 'execution_date': run.execution_date.isoformat(),
+ 'start_date': ((run.start_date or '') and
+ run.start_date.isoformat()),
+ })
+ if not dag_runs:
+ print('No dag runs for {dag_id}'.format(dag_id=args.dag_id))
+
+ s = textwrap.dedent("""\n
+ {line}
+ DAG RUNS
+ {line}
+ {dag_run_header}
+ """)
+
+ dag_runs.sort(key=lambda x: x['execution_date'], reverse=True)
+ dag_run_header = '%-3s | %-20s | %-10s | %-20s | %-20s |' % ('id',
+ 'run_id',
+ 'state',
+ 'execution_date',
+ 'state_date')
+ print(s.format(dag_run_header=dag_run_header,
+ line='-' * 120))
+ for dag_run in dag_runs:
+ record = '%-3s | %-20s | %-10s | %-20s | %-20s |' % (dag_run['id'],
+ dag_run['run_id'],
+ dag_run['state'],
+ dag_run['execution_date'],
+ dag_run['start_date'])
+ print(record)
+
+
+@cli_utils.action_logging
+def sync_perm(args):
+ if settings.RBAC:
+ appbuilder = cached_appbuilder()
+ print('Update permission, view-menu for all existing roles')
+ appbuilder.sm.sync_roles()
+ else:
+ print('The sync_perm command only works for rbac UI.')
Arg = namedtuple(
@@ -1129,8 +1484,10 @@ class CLIFactory(object):
"The regex to filter specific task_ids to backfill (optional)"),
'subdir': Arg(
("-sd", "--subdir"),
- "File location or directory from which to look for the dag",
- default=settings.DAGS_FOLDER),
+ "File location or directory from which to look for the dag. "
+ "Defaults to '[AIRFLOW_HOME]/dags' where [AIRFLOW_HOME] is the "
+ "value you set for 'AIRFLOW_HOME' config you set in 'airflow.cfg' ",
+ default=DAGS_FOLDER),
'start_date': Arg(
("-s", "--start_date"), "Override start_date YYYY-MM-DD",
type=parsedate),
@@ -1152,11 +1509,28 @@ class CLIFactory(object):
("--stdout",), "Redirect stdout to this file"),
'log_file': Arg(
("-l", "--log-file"), "Location of the log file"),
+ 'yes': Arg(
+ ("-y", "--yes"),
+ "Do not prompt to confirm reset. Use with care!",
+ "store_true",
+ default=False),
+
+ # list_dag_runs
+ 'no_backfill': Arg(
+ ("--no_backfill",),
+ "filter all the backfill dagruns given the dag id", "store_true"),
+ 'state': Arg(
+ ("--state",),
+ "Only list the dag runs corresponding to the state"
+ ),
# backfill
'mark_success': Arg(
("-m", "--mark_success"),
"Mark jobs as succeeded without running them", "store_true"),
+ 'verbose': Arg(
+ ("-v", "--verbose"),
+ "Make logging output more verbose", "store_true"),
'local': Arg(
("-l", "--local"),
"Run the task using the LocalExecutor", "store_true"),
@@ -1166,9 +1540,6 @@ class CLIFactory(object):
"to the workers, just tell the workers to run their version "
"of the code."),
"store_true"),
- 'include_adhoc': Arg(
- ("-a", "--include_adhoc"),
- "Include dags with the adhoc parameter.", "store_true"),
'bf_ignore_dependencies': Arg(
("-i", "--ignore_dependencies"),
(
@@ -1192,6 +1563,21 @@ class CLIFactory(object):
"again."),
type=float,
default=1.0),
+ 'reset_dag_run': Arg(
+ ("--reset_dagruns",),
+ (
+ "if set, the backfill will delete existing "
+ "backfill-related DAG runs and start "
+ "anew with fresh, running DAG runs"),
+ "store_true"),
+ 'rerun_failed_tasks': Arg(
+ ("--rerun_failed_tasks",),
+ (
+ "if set, the backfill will auto-rerun "
+ "all the failed tasks for the backfill date range "
+ "instead of throwing exceptions"),
+ "store_true"),
+
# list_tasks
'tree': Arg(("-t", "--tree"), "Tree view", "store_true"),
# list_dags
@@ -1212,6 +1598,10 @@ class CLIFactory(object):
'exclude_subdags': Arg(
("-x", "--exclude_subdags"),
"Exclude subdags", "store_true"),
+ 'exclude_parentdag': Arg(
+ ("-xp", "--exclude_parentdag"),
+ "Exclude ParentDAGS if the task cleared is a part of a SubDAG",
+ "store_true"),
'dag_regex': Arg(
("-dx", "--dag_regex"),
"Search dag_id as regex instead of exact string", "store_true"),
@@ -1237,6 +1627,14 @@ class CLIFactory(object):
("-x", "--delete"),
metavar="NAME",
help="Delete a pool"),
+ 'pool_import': Arg(
+ ("-i", "--import"),
+ metavar="FILEPATH",
+ help="Import pool from JSON file"),
+ 'pool_export': Arg(
+ ("-e", "--export"),
+ metavar="FILEPATH",
+ help="Export pool to JSON file"),
# variables
'set': Arg(
("-s", "--set"),
@@ -1270,8 +1668,7 @@ class CLIFactory(object):
help="Delete a variable"),
# kerberos
'principal': Arg(
- ("principal",), "kerberos principal",
- nargs='?', default=conf.get('kerberos', 'principal')),
+ ("principal",), "kerberos principal", nargs='?'),
'keytab': Arg(
("-kt", "--keytab"), "keytab",
nargs='?', default=conf.get('kerberos', 'keytab')),
@@ -1281,6 +1678,11 @@ class CLIFactory(object):
# dependency. This flag should be deprecated and renamed to 'ignore_ti_state' and
# the "ignore_all_dependencies" command should be called the"force" command
# instead.
+ 'interactive': Arg(
+ ('-int', '--interactive'),
+ help='Do not capture standard output and error streams '
+ '(useful for interactive debugging)',
+ action='store_true'),
'force': Arg(
("-f", "--force"),
"Ignore previous task instance state, rerun regardless if task already "
@@ -1364,12 +1766,6 @@ class CLIFactory(object):
default=conf.get('webserver', 'ERROR_LOGFILE'),
help="The logfile to store the webserver error log. Use '-' to print to "
"stderr."),
- # resetdb
- 'yes': Arg(
- ("-y", "--yes"),
- "Do not prompt to confirm reset. Use with care!",
- "store_true",
- default=False),
# scheduler
'dag_id_opt': Arg(("-d", "--dag_id"), help="The id of the dag to run"),
'run_duration': Arg(
@@ -1420,6 +1816,12 @@ class CLIFactory(object):
("-u", "--url_prefix"),
default=conf.get('celery', 'FLOWER_URL_PREFIX'),
help="URL prefix for Flower"),
+ 'flower_basic_auth': Arg(
+ ("-ba", "--basic_auth"),
+ default=conf.get('celery', 'FLOWER_BASIC_AUTH'),
+ help=("Securing Flower with Basic Authentication. "
+ "Accepts user:password pairs separated by a comma. "
+ "Example: flower_basic_auth = user1:password1,user2:password2")),
'task_params': Arg(
("-tp", "--task_params"),
help="Sends a JSON params dict to the task"),
@@ -1472,16 +1874,83 @@ class CLIFactory(object):
('--conn_extra',),
help='Connection `Extra` field, optional when adding a connection',
type=str),
+ # users
+ 'username': Arg(
+ ('--username',),
+ help='Username of the user, required to create/delete a user',
+ type=str),
+ 'firstname': Arg(
+ ('--firstname',),
+ help='First name of the user, required to create a user',
+ type=str),
+ 'lastname': Arg(
+ ('--lastname',),
+ help='Last name of the user, required to create a user',
+ type=str),
+ 'role': Arg(
+ ('--role',),
+ help='Role of the user. Existing roles include Admin, '
+ 'User, Op, Viewer, and Public. Required to create a user',
+ type=str),
+ 'email': Arg(
+ ('--email',),
+ help='Email of the user, required to create a user',
+ type=str),
+ 'password': Arg(
+ ('--password',),
+ help='Password of the user, required to create a user '
+ 'without --use_random_password',
+ type=str),
+ 'use_random_password': Arg(
+ ('--use_random_password',),
+ help='Do not prompt for password. Use random string instead.'
+ ' Required to create a user without --password ',
+ default=False,
+ action='store_true'),
+ 'list_users': Arg(
+ ('-l', '--list'),
+ help='List all users',
+ action='store_true'),
+ 'create_user': Arg(
+ ('-c', '--create'),
+ help='Create a user',
+ action='store_true'),
+ 'delete_user': Arg(
+ ('-d', '--delete'),
+ help='Delete a user',
+ action='store_true'),
+ 'autoscale': Arg(
+ ('-a', '--autoscale'),
+ help="Minimum and Maximum number of worker to autoscale"),
+
}
subparsers = (
{
'func': backfill,
- 'help': "Run subsections of a DAG for a specified date range",
+ 'help': "Run subsections of a DAG for a specified date range. "
+ "If reset_dag_run option is used,"
+ " backfill will first prompt users whether airflow "
+ "should clear all the previous dag_run and task_instances "
+ "within the backfill date range. "
+ "If rerun_failed_tasks is used, backfill "
+ "will auto re-run the previous failed task instances"
+ " within the backfill date range.",
'args': (
'dag_id', 'task_regex', 'start_date', 'end_date',
- 'mark_success', 'local', 'donot_pickle', 'include_adhoc',
+ 'mark_success', 'local', 'donot_pickle',
'bf_ignore_dependencies', 'bf_ignore_first_depends_on_past',
- 'subdir', 'pool', 'delay_on_limit', 'dry_run')
+ 'subdir', 'pool', 'delay_on_limit', 'dry_run', 'verbose', 'conf',
+ 'reset_dag_run', 'rerun_failed_tasks',
+ )
+ }, {
+ 'func': list_dag_runs,
+ 'help': "List dag runs given a DAG id. If state option is given, it will only"
+ "search for all the dagruns with the given state. "
+ "If no_backfill option is given, it will filter out"
+ "all backfill dagruns for given dag id.",
+ 'args': (
+ 'dag_id', 'no_backfill', 'state'
+ )
}, {
'func': list_tasks,
'help': "List the tasks within a DAG",
@@ -1492,7 +1961,7 @@ class CLIFactory(object):
'args': (
'dag_id', 'task_regex', 'start_date', 'end_date', 'subdir',
'upstream', 'downstream', 'no_confirm', 'only_failed',
- 'only_running', 'exclude_subdags', 'dag_regex'),
+ 'only_running', 'exclude_subdags', 'exclude_parentdag', 'dag_regex'),
}, {
'func': pause,
'help': "Pause a DAG",
@@ -1505,14 +1974,19 @@ class CLIFactory(object):
'func': trigger_dag,
'help': "Trigger a DAG run",
'args': ('dag_id', 'subdir', 'run_id', 'conf', 'exec_date'),
+ }, {
+ 'func': delete_dag,
+ 'help': "Delete all DB records related to the specified DAG",
+ 'args': ('dag_id', 'yes',),
}, {
'func': pool,
'help': "CRUD operations on pools",
- "args": ('pool_set', 'pool_get', 'pool_delete'),
+ "args": ('pool_set', 'pool_get', 'pool_delete', 'pool_import', 'pool_export'),
}, {
'func': variables,
'help': "CRUD operations on variables",
- "args": ('set', 'get', 'json', 'default', 'var_import', 'var_export', 'var_delete'),
+ "args": ('set', 'get', 'json', 'default',
+ 'var_import', 'var_export', 'var_delete'),
}, {
'func': kerberos,
'help': "Start a kerberos ticket renewer",
@@ -1529,7 +2003,7 @@ class CLIFactory(object):
'dag_id', 'task_id', 'execution_date', 'subdir',
'mark_success', 'force', 'pool', 'cfg_path',
'local', 'raw', 'ignore_all_dependencies', 'ignore_dependencies',
- 'ignore_depends_on_past', 'ship_dag', 'pickle', 'job_id'),
+ 'ignore_depends_on_past', 'ship_dag', 'pickle', 'job_id', 'interactive',),
}, {
'func': initdb,
'help': "Initialize the metadata database",
@@ -1562,7 +2036,7 @@ class CLIFactory(object):
'func': test,
'help': (
"Test a task instance. This will run a task without checking for "
- "dependencies or recording it's state in the database."),
+ "dependencies or recording its state in the database."),
'args': (
'dag_id', 'task_id', 'execution_date', 'subdir', 'dry_run',
'task_params'),
@@ -1590,12 +2064,12 @@ class CLIFactory(object):
'func': worker,
'help': "Start a Celery worker node",
'args': ('do_pickle', 'queues', 'concurrency', 'celery_hostname',
- 'pid', 'daemon', 'stdout', 'stderr', 'log_file'),
+ 'pid', 'daemon', 'stdout', 'stderr', 'log_file', 'autoscale'),
}, {
'func': flower,
'help': "Start a Celery Flower",
'args': ('flower_hostname', 'flower_port', 'flower_conf', 'flower_url_prefix',
- 'broker_api', 'pid', 'daemon', 'stdout', 'stderr', 'log_file'),
+ 'flower_basic_auth', 'broker_api', 'pid', 'daemon', 'stdout', 'stderr', 'log_file'),
}, {
'func': version,
'help': "Show the version",
@@ -1605,11 +2079,27 @@ class CLIFactory(object):
'help': "List/Add/Delete connections",
'args': ('list_connections', 'add_connection', 'delete_connection',
'conn_id', 'conn_uri', 'conn_extra') + tuple(alternative_conn_specs),
+ }, {
+ 'func': users,
+ 'help': "List/Create/Delete users",
+ 'args': ('list_users', 'create_user', 'delete_user',
+ 'username', 'email', 'firstname', 'lastname', 'role',
+ 'password', 'use_random_password'),
},
+ {
+ 'func': sync_perm,
+ 'help': "Update existing role's permissions.",
+ 'args': tuple(),
+ },
+ {
+ 'func': next_execution,
+ 'help': "Get the next execution datetime of a DAG.",
+ 'args': ('dag_id', 'subdir')
+ }
)
subparsers_dict = {sp['func'].__name__: sp for sp in subparsers}
dag_subparsers = (
- 'list_tasks', 'backfill', 'test', 'run', 'pause', 'unpause')
+ 'list_tasks', 'backfill', 'test', 'run', 'pause', 'unpause', 'list_dag_runs')
@classmethod
def get_parser(cls, dag_parser=False):
diff --git a/airflow/config_templates/__init__.py b/airflow/config_templates/__init__.py
index 9d7677a99b..114d189da1 100644
--- a/airflow/config_templates/__init__.py
+++ b/airflow/config_templates/__init__.py
@@ -1,13 +1,18 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/config_templates/airflow_local_settings.py b/airflow/config_templates/airflow_local_settings.py
index 899e815f2e..45a2f2923c 100644
--- a/airflow/config_templates/airflow_local_settings.py
+++ b/airflow/config_templates/airflow_local_settings.py
@@ -1,20 +1,26 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
import os
from airflow import configuration as conf
+from airflow.utils.file import mkdirs
# TODO: Logging format and level should be configured
# in this file instead of from airflow.cfg. Currently
@@ -22,20 +28,36 @@
# settings.py and cli.py. Please see AIRFLOW-1455.
LOG_LEVEL = conf.get('core', 'LOGGING_LEVEL').upper()
+
+# Flask appbuilder's info level log is very verbose,
+# so it's set to 'WARN' by default.
+FAB_LOG_LEVEL = conf.get('core', 'FAB_LOGGING_LEVEL').upper()
+
LOG_FORMAT = conf.get('core', 'LOG_FORMAT')
BASE_LOG_FOLDER = conf.get('core', 'BASE_LOG_FOLDER')
PROCESSOR_LOG_FOLDER = conf.get('scheduler', 'CHILD_PROCESS_LOG_DIRECTORY')
-FILENAME_TEMPLATE = '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'
+DAG_PROCESSOR_MANAGER_LOG_LOCATION = \
+ conf.get('core', 'DAG_PROCESSOR_MANAGER_LOG_LOCATION')
+
+FILENAME_TEMPLATE = conf.get('core', 'LOG_FILENAME_TEMPLATE')
-PROCESSOR_FILENAME_TEMPLATE = '{{ filename }}.log'
+PROCESSOR_FILENAME_TEMPLATE = conf.get('core', 'LOG_PROCESSOR_FILENAME_TEMPLATE')
# Storage bucket url for remote logging
# s3 buckets should start with "s3://"
# gcs buckets should start with "gs://"
-REMOTE_BASE_LOG_FOLDER = ''
+# wasb buckets should start with "wasb"
+# just to help Airflow select correct handler
+REMOTE_BASE_LOG_FOLDER = conf.get('core', 'REMOTE_BASE_LOG_FOLDER')
+
+ELASTICSEARCH_HOST = conf.get('elasticsearch', 'ELASTICSEARCH_HOST')
+
+LOG_ID_TEMPLATE = conf.get('elasticsearch', 'ELASTICSEARCH_LOG_ID_TEMPLATE')
+
+END_OF_LOG_MARK = conf.get('elasticsearch', 'ELASTICSEARCH_END_OF_LOG_MARK')
DEFAULT_LOGGING_CONFIG = {
'version': 1,
@@ -62,7 +84,7 @@
'formatter': 'airflow',
'base_log_folder': os.path.expanduser(PROCESSOR_LOG_FOLDER),
'filename_template': PROCESSOR_FILENAME_TEMPLATE,
- },
+ }
},
'loggers': {
'airflow.processor': {
@@ -75,6 +97,11 @@
'level': LOG_LEVEL,
'propagate': False,
},
+ 'flask_appbuilder': {
+ 'handler': ['console'],
+ 'level': FAB_LOG_LEVEL,
+ 'propagate': True,
+ }
},
'root': {
'handlers': ['console'],
@@ -82,6 +109,26 @@
}
}
+DEFAULT_DAG_PARSING_LOGGING_CONFIG = {
+ 'handlers': {
+ 'processor_manager': {
+ 'class': 'logging.handlers.RotatingFileHandler',
+ 'formatter': 'airflow',
+ 'filename': DAG_PROCESSOR_MANAGER_LOG_LOCATION,
+ 'mode': 'a',
+ 'maxBytes': 104857600, # 100MB
+ 'backupCount': 5
+ }
+ },
+ 'loggers': {
+ 'airflow.processor_manager': {
+ 'handlers': ['processor_manager'],
+ 'level': LOG_LEVEL,
+ 'propagate': False,
+ }
+ }
+}
+
REMOTE_HANDLERS = {
's3': {
'task': {
@@ -114,12 +161,63 @@
'gcs_log_folder': REMOTE_BASE_LOG_FOLDER,
'filename_template': PROCESSOR_FILENAME_TEMPLATE,
},
- }
+ },
+ 'wasb': {
+ 'task': {
+ 'class': 'airflow.utils.log.wasb_task_handler.WasbTaskHandler',
+ 'formatter': 'airflow',
+ 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+ 'wasb_log_folder': REMOTE_BASE_LOG_FOLDER,
+ 'wasb_container': 'airflow-logs',
+ 'filename_template': FILENAME_TEMPLATE,
+ 'delete_local_copy': False,
+ },
+ 'processor': {
+ 'class': 'airflow.utils.log.wasb_task_handler.WasbTaskHandler',
+ 'formatter': 'airflow',
+ 'base_log_folder': os.path.expanduser(PROCESSOR_LOG_FOLDER),
+ 'wasb_log_folder': REMOTE_BASE_LOG_FOLDER,
+ 'wasb_container': 'airflow-logs',
+ 'filename_template': PROCESSOR_FILENAME_TEMPLATE,
+ 'delete_local_copy': False,
+ },
+ },
+ 'elasticsearch': {
+ 'task': {
+ 'class': 'airflow.utils.log.es_task_handler.ElasticsearchTaskHandler',
+ 'formatter': 'airflow',
+ 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+ 'log_id_template': LOG_ID_TEMPLATE,
+ 'filename_template': FILENAME_TEMPLATE,
+ 'end_of_log_mark': END_OF_LOG_MARK,
+ 'host': ELASTICSEARCH_HOST,
+ },
+ },
}
REMOTE_LOGGING = conf.get('core', 'remote_logging')
+# Only update the handlers and loggers when CONFIG_PROCESSOR_MANAGER_LOGGER is set.
+# This is to avoid exceptions when initializing RotatingFileHandler multiple times
+# in multiple processes.
+if os.environ.get('CONFIG_PROCESSOR_MANAGER_LOGGER') == 'True':
+ DEFAULT_LOGGING_CONFIG['handlers'] \
+ .update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'])
+ DEFAULT_LOGGING_CONFIG['loggers'] \
+ .update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['loggers'])
+
+ # Manually create log directory for processor_manager handler as RotatingFileHandler
+ # will only create file but not the directory.
+ processor_manager_handler_config = DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'][
+ 'processor_manager']
+ directory = os.path.dirname(processor_manager_handler_config['filename'])
+ mkdirs(directory, 0o755)
+
if REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('s3://'):
DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['s3'])
elif REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('gs://'):
DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['gcs'])
+elif REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('wasb'):
+ DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['wasb'])
+elif REMOTE_LOGGING and ELASTICSEARCH_HOST:
+ DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['elasticsearch'])
diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg
index 59ff7401ff..960fe8ff5f 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -1,14 +1,21 @@
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# -*- coding: utf-8 -*-
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
# This is the template for Airflow's default configuration. When Airflow is
@@ -35,16 +42,18 @@ dags_folder = {AIRFLOW_HOME}/dags
# This path must be absolute
base_log_folder = {AIRFLOW_HOME}/logs
-# Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users
-# must supply an Airflow connection id that provides access to the storage
+# Airflow can store logs remotely in AWS S3, Google Cloud Storage or Elastic Search.
+# Users must supply an Airflow connection id that provides access to the storage
# location. If remote_logging is set to true, see UPDATING.md for additional
# configuration requirements.
remote_logging = False
remote_log_conn_id =
+remote_base_log_folder =
encrypt_s3_logs = False
# Logging level
logging_level = INFO
+fab_logging_level = WARN
# Logging class
# Specify the class that will specify the logging configuration
@@ -56,12 +65,20 @@ logging_config_class =
log_format = [%%(asctime)s] {{%%(filename)s:%%(lineno)d}} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
+# Log filename format
+log_filename_template = {{{{ ti.dag_id }}}}/{{{{ ti.task_id }}}}/{{{{ ts }}}}/{{{{ try_number }}}}.log
+log_processor_filename_template = {{{{ filename }}}}.log
+dag_processor_manager_log_location = {AIRFLOW_HOME}/logs/dag_processor_manager/dag_processor_manager.log
+
+# Hostname by providing a path to a callable, which will resolve the hostname
+hostname_callable = socket:getfqdn
+
# Default timezone in case supplied date times are naive
# can be utc (default), system, or any IANA timezone string (e.g. Europe/Amsterdam)
default_timezone = utc
# The executor class that airflow should use. Choices include
-# SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor
+# SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor, KubernetesExecutor
executor = SequentialExecutor
# The SqlAlchemy connection string to the metadata database.
@@ -69,6 +86,9 @@ executor = SequentialExecutor
# their website
sql_alchemy_conn = sqlite:///{AIRFLOW_HOME}/airflow.db
+# The encoding for the databases
+sql_engine_encoding = utf-8
+
# If SqlAlchemy should pool database connections.
sql_alchemy_pool_enabled = True
@@ -78,13 +98,18 @@ sql_alchemy_pool_size = 5
# The SqlAlchemy pool recycle is the number of seconds a connection
# can be idle in the pool before it is invalidated. This config does
-# not apply to sqlite.
-sql_alchemy_pool_recycle = 3600
+# not apply to sqlite. If the number of DB connections is ever exceeded,
+# a lower config value will allow the system to recover faster.
+sql_alchemy_pool_recycle = 1800
# How many seconds to retry re-establishing a DB connection after
# disconnects. Setting this to 0 disables retries.
sql_alchemy_reconnect_timeout = 300
+# The schema to use for the metadata database
+# SqlAlchemy supports databases with the concept of multiple schemas.
+sql_alchemy_schema =
+
# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
@@ -121,7 +146,7 @@ donot_pickle = False
dagbag_import_timeout = 30
# The class to use for running task instances in a subprocess
-task_runner = BashTaskRunner
+task_runner = StandardTaskRunner
# If set, tasks without a `run_as_user` argument will be run with this user
# Can be used to de-elevate a sudo user running Airflow when executing tasks
@@ -150,6 +175,13 @@ enable_xcom_pickling = True
# it has to cleanup after it is sent a SIGTERM, before it is SIGKILLED
killed_task_cleanup_time = 60
+# Whether to override params with dag_run.conf. If you pass some key-value pairs through `airflow backfill -c` or
+# `airflow trigger_dag -c`, the key-value pairs will override the existing ones in params.
+dag_run_conf_overrides_params = False
+
+# Worker initialisation check to validate Metadata Database connection
+worker_precheck = False
+
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
@@ -165,10 +197,21 @@ endpoint_url = http://localhost:8080
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
+[lineage]
+# what lineage backend to use
+backend =
+
+[atlas]
+sasl_enabled = False
+host =
+port = 21000
+username =
+password =
+
[operators]
# The default owner assigned to each new operator, unless
# provided explicitly or passed via `default_args`
-default_owner = Airflow
+default_owner = airflow
default_cpus = 1
default_ram = 512
default_disk = 512
@@ -177,6 +220,9 @@ default_gpus = 0
[hive]
# Default mapreduce queue for HiveOperator tasks
default_hive_mapred_queue =
+# Template for mapred_job_name in HiveOperator, supports the following named parameters:
+# hostname, dag_id, task_id, execution_date
+mapred_job_name_template = Airflow HiveOperator task for {{hostname}}.{{dag_id}}.{{task_id}}.{{execution_date}}
[webserver]
# The base url of your website as airflow cannot guess what domain or
@@ -195,6 +241,9 @@ web_server_port = 8080
web_server_ssl_cert =
web_server_ssl_key =
+# Number of seconds the webserver waits before killing gunicorn master that doesn't respond
+web_server_master_timeout = 120
+
# Number of seconds the gunicorn webserver waits before timing out on a worker
web_server_worker_timeout = 120
@@ -207,7 +256,8 @@ worker_refresh_batch_size = 1
worker_refresh_interval = 30
# Secret key used to run your flask app
-secret_key = temporary_key
+# It should be as random as possible
+secret_key = {SECRET_KEY}
# Number of workers to run the Gunicorn web server
workers = 4
@@ -221,10 +271,13 @@ access_logfile = -
error_logfile = -
# Expose the configuration file in the web server
+# This is only applicable for the flask-admin based web UI (non FAB-based).
+# In the FAB-based web UI with RBAC feature,
+# access to configuration is controlled by role permissions.
expose_config = False
# Set to true to turn on authentication:
-# http://pythonhosted.org/airflow/security.html#web-authentication
+# https://airflow.incubator.apache.org/security.html#web-authentication
authenticate = False
# Filter the list of dags by owner name (requires authentication to be enabled)
@@ -260,6 +313,19 @@ hide_paused_dags_by_default = False
# Consistent page size across all listing views in the UI
page_size = 100
+# Use FAB-based webserver with RBAC feature
+rbac = False
+
+# Define the color of navigation bar
+navbar_color = #007A87
+
+# Default dagrun to show in UI
+default_dag_run_display_number = 25
+
+# Enable werkzeug `ProxyFix` middleware
+enable_proxy_fix = False
+
+
[email]
email_backend = airflow.utils.email.send_email_smtp
@@ -291,6 +357,13 @@ celery_app_name = airflow.executors.celery_executor
# your worker box and the nature of your tasks
worker_concurrency = 16
+# The minimum and maximum concurrency that will be used when starting workers with the
+# "airflow worker" command. Pick these numbers based on resources on
+# worker box and the nature of the task. If autoscale option is available worker_concurrency
+# will be ignored.
+# http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-autoscale
+# worker_autoscale = 12,16
+
# When you start an airflow worker, airflow starts a tiny web server
# subprocess to serve the workers local log files to the airflow main
# web server, who then builds pages and sends them to users. This defines
@@ -323,31 +396,53 @@ flower_url_prefix =
# This defines the port that Celery Flower runs on
flower_port = 5555
+# Securing Flower with Basic Authentication
+# Accepts user:password pairs separated by a comma
+# Example: flower_basic_auth = user1:password1,user2:password2
+flower_basic_auth =
+
# Default queue that tasks get assigned to and that worker listen on.
default_queue = default
+# How many processes CeleryExecutor uses to sync task state.
+# 0 means to use max(1, number of cores - 1) processes.
+sync_parallelism = 0
+
# Import path for celery configuration options
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
-[celery_broker_transport_options]
-# The visibility timeout defines the number of seconds to wait for the worker
-# to acknowledge the task before the message is redelivered to another worker.
-# Make sure to increase the visibility timeout to match the time of the longest
-# ETA you're planning to use. Especially important in case of using Redis or SQS
-visibility_timeout = 21600
-
# In case of using SSL
ssl_active = False
ssl_key =
ssl_cert =
ssl_cacert =
+[celery_broker_transport_options]
+# This section is for specifying options which can be passed to the
+# underlying celery broker transport. See:
+# http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_transport_options
+
+# The visibility timeout defines the number of seconds to wait for the worker
+# to acknowledge the task before the message is redelivered to another worker.
+# Make sure to increase the visibility timeout to match the time of the longest
+# ETA you're planning to use.
+#
+# visibility_timeout is only supported for Redis and SQS celery brokers.
+# See:
+# http://docs.celeryproject.org/en/master/userguide/configuration.html#std:setting-broker_transport_options
+#
+#visibility_timeout = 21600
+
[dask]
# This section only applies if you are using the DaskExecutor in
# [core] section above
# The IP address and port of the Dask cluster's scheduler.
cluster_address = 127.0.0.1:8786
+# TLS/ SSL settings to access a secured Dask scheduler.
+tls_ca =
+tls_cert =
+tls_key =
[scheduler]
@@ -365,9 +460,10 @@ scheduler_heartbeat_sec = 5
# -1 indicates to run continuously (see also num_runs)
run_duration = -1
-# after how much time a new DAGs should be picked up from the filesystem
+# after how much time (seconds) a new DAGs should be picked up from the filesystem
min_file_process_interval = 0
+# How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
dag_dir_list_interval = 300
# How often should stats be printed to the logs
@@ -389,9 +485,16 @@ scheduler_zombie_task_threshold = 300
catchup_by_default = True
# This changes the batch size of queries in the scheduling main loop.
-# This depends on query length limits and how long you are willing to hold locks.
-# 0 for no limit
-max_tis_per_query = 0
+# If this is too high, SQL query performance may be impacted by one
+# or more of the following:
+# - reversion to full table scan
+# - complexity of query predicate
+# - excessive locking
+#
+# Additionally, you may hit the maximum allowable query length for your db.
+#
+# Set this to 0 for no limit (not advised)
+max_tis_per_query = 512
# Statsd (https://github.com/etsy/statsd) integration settings
statsd_on = False
@@ -405,6 +508,10 @@ max_threads = 2
authenticate = False
+# Turn off scheduler use of cron intervals by setting this to False.
+# DAGs submitted manually in the web UI or with trigger_dag will still run.
+use_job_schedule = True
+
[ldap]
# set this to ldaps://<your.ldap.server>:<port>
uri =
@@ -456,6 +563,10 @@ authenticate = False
# default_principal = admin
# default_secret = admin
+# Optional Docker Image to run on slave before running the command
+# This image should be accessible from mesos slave i.e mesos slave
+# should be able to pull this docker image before executing the command.
+# docker_image_slave = puckel/docker-airflow
[kerberos]
ccache = /tmp/airflow_krb5_ccache
@@ -472,3 +583,113 @@ api_rev = v3
[admin]
# UI to hide sensitive variable fields when set to True
hide_sensitive_variable_fields = True
+
+[elasticsearch]
+elasticsearch_host =
+elasticsearch_log_id_template = {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}
+elasticsearch_end_of_log_mark = end_of_log
+
+[kubernetes]
+# The repository, tag and imagePullPolicy of the Kubernetes Image for the Worker to Run
+worker_container_repository =
+worker_container_tag =
+worker_container_image_pull_policy = IfNotPresent
+
+# If True (default), worker pods will be deleted upon termination
+delete_worker_pods = True
+
+# The Kubernetes namespace where airflow workers should be created. Defaults to `default`
+namespace = default
+
+# The name of the Kubernetes ConfigMap Containing the Airflow Configuration (this file)
+airflow_configmap =
+
+# For docker image already contains DAGs, this is set to `True`, and the worker will search for dags in dags_folder,
+# otherwise use git sync or dags volume claim to mount DAGs
+dags_in_image = False
+
+# For either git sync or volume mounted DAGs, the worker will look in this subpath for DAGs
+dags_volume_subpath =
+
+# For DAGs mounted via a volume claim (mutually exclusive with git-sync and host path)
+dags_volume_claim =
+
+# For volume mounted logs, the worker will look in this subpath for logs
+logs_volume_subpath =
+
+# A shared volume claim for the logs
+logs_volume_claim =
+
+# For DAGs mounted via a hostPath volume (mutually exclusive with volume claim and git-sync)
+# Useful in local environment, discouraged in production
+dags_volume_host =
+
+# A hostPath volume for the logs
+# Useful in local environment, discouraged in production
+logs_volume_host =
+
+# Git credentials and repository for DAGs mounted via Git (mutually exclusive with volume claim)
+git_repo =
+git_branch =
+git_user =
+git_password =
+git_subpath =
+git_sync_root = /git
+git_sync_dest = repo
+# Mount point of the volume if git-sync is being used.
+# i.e. {AIRFLOW_HOME}/dags
+git_dags_folder_mount_point =
+
+# For cloning DAGs from git repositories into volumes: https://github.com/kubernetes/git-sync
+git_sync_container_repository = gcr.io/google-containers/git-sync-amd64
+git_sync_container_tag = v2.0.5
+git_sync_init_container_name = git-sync-clone
+
+# The name of the Kubernetes service account to be associated with airflow workers, if any.
+# Service accounts are required for workers that require access to secrets or cluster resources.
+# See the Kubernetes RBAC documentation for more:
+# https://kubernetes.io/docs/admin/authorization/rbac/
+worker_service_account_name =
+
+# Any image pull secrets to be given to worker pods, If more than one secret is
+# required, provide a comma separated list: secret_a,secret_b
+image_pull_secrets =
+
+# GCP Service Account Keys to be provided to tasks run on Kubernetes Executors
+# Should be supplied in the format: key-name-1:key-path-1,key-name-2:key-path-2
+gcp_service_account_keys =
+
+# Use the service account kubernetes gives to pods to connect to kubernetes cluster.
+# It's intended for clients that expect to be running inside a pod running on kubernetes.
+# It will raise an exception if called from a process not running in a kubernetes environment.
+in_cluster = True
+
+# Affinity configuration as a single line formatted JSON object.
+# See the affinity model for top-level key names (e.g. `nodeAffinity`, etc.):
+# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/#affinity-v1-core
+affinity =
+
+# A list of toleration objects as a single line formatted JSON array
+# See:
+# https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/#toleration-v1-core
+tolerations =
+
+[kubernetes_node_selectors]
+# The Key-value pairs to be given to worker pods.
+# The worker pods will be scheduled to the nodes of the specified key-value pairs.
+# Should be supplied in the format: key = value
+
+[kubernetes_secrets]
+# The scheduler mounts the following secrets into your workers as they are launched by the
+# scheduler. You may define as many secrets as needed and the kubernetes launcher will parse the
+# defined secrets and mount them as secret environment variables in the launched workers.
+# Secrets in this section are defined as follows
+# <environment_variable_mount> = <kubernetes_secret_object>:<kubernetes_secret_key>
+#
+# For example if you wanted to mount a kubernetes secret key named `postgres_password` from the
+# kubernetes secret object `airflow-secret` as the environment variable `POSTGRES_PASSWORD` into
+# your workers you would follow the following format:
+# POSTGRES_PASSWORD = airflow-secret:postgres_credentials
+#
+# Additionally you may override worker airflow settings with the AIRFLOW__<SECTION>__<KEY>
+# formatting as supported by airflow normally.
diff --git a/airflow/config_templates/default_celery.py b/airflow/config_templates/default_celery.py
index 57b9611fc9..5e72134de1 100644
--- a/airflow/config_templates/default_celery.py
+++ b/airflow/config_templates/default_celery.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
import ssl
@@ -18,36 +23,46 @@
from airflow.exceptions import AirflowConfigException, AirflowException
from airflow.utils.log.logging_mixin import LoggingMixin
+
+def _broker_supports_visibility_timeout(url):
+ return url.startswith("redis://") or url.startswith("sqs://")
+
+
log = LoggingMixin().log
-broker_transport_options = configuration.getsection('celery_broker_transport_options')
-if broker_transport_options is None:
- broker_transport_options = {'visibility_timeout': 21600}
+broker_url = configuration.conf.get('celery', 'BROKER_URL')
+
+broker_transport_options = configuration.conf.getsection(
+ 'celery_broker_transport_options'
+)
+if 'visibility_timeout' not in broker_transport_options:
+ if _broker_supports_visibility_timeout(broker_url):
+ broker_transport_options['visibility_timeout'] = 21600
DEFAULT_CELERY_CONFIG = {
'accept_content': ['json', 'pickle'],
'event_serializer': 'json',
'worker_prefetch_multiplier': 1,
'task_acks_late': True,
- 'task_default_queue': configuration.get('celery', 'DEFAULT_QUEUE'),
- 'task_default_exchange': configuration.get('celery', 'DEFAULT_QUEUE'),
- 'broker_url': configuration.get('celery', 'BROKER_URL'),
+ 'task_default_queue': configuration.conf.get('celery', 'DEFAULT_QUEUE'),
+ 'task_default_exchange': configuration.conf.get('celery', 'DEFAULT_QUEUE'),
+ 'broker_url': broker_url,
'broker_transport_options': broker_transport_options,
- 'result_backend': configuration.get('celery', 'RESULT_BACKEND'),
- 'worker_concurrency': configuration.getint('celery', 'WORKER_CONCURRENCY'),
+ 'result_backend': configuration.conf.get('celery', 'RESULT_BACKEND'),
+ 'worker_concurrency': configuration.conf.getint('celery', 'WORKER_CONCURRENCY'),
}
celery_ssl_active = False
try:
- celery_ssl_active = configuration.getboolean('celery', 'SSL_ACTIVE')
+ celery_ssl_active = configuration.conf.getboolean('celery', 'SSL_ACTIVE')
except AirflowConfigException as e:
log.warning("Celery Executor will run without SSL")
try:
if celery_ssl_active:
- broker_use_ssl = {'keyfile': configuration.get('celery', 'SSL_KEY'),
- 'certfile': configuration.get('celery', 'SSL_CERT'),
- 'ca_certs': configuration.get('celery', 'SSL_CACERT'),
+ broker_use_ssl = {'keyfile': configuration.conf.get('celery', 'SSL_KEY'),
+ 'certfile': configuration.conf.get('celery', 'SSL_CERT'),
+ 'ca_certs': configuration.conf.get('celery', 'SSL_CACERT'),
'cert_reqs': ssl.CERT_REQUIRED}
DEFAULT_CELERY_CONFIG['broker_use_ssl'] = broker_use_ssl
except AirflowConfigException as e:
diff --git a/airflow/config_templates/default_test.cfg b/airflow/config_templates/default_test.cfg
index eaf3d03694..f0a467894d 100644
--- a/airflow/config_templates/default_test.cfg
+++ b/airflow/config_templates/default_test.cfg
@@ -1,14 +1,21 @@
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# -*- coding: utf-8 -*-
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
# This is the template for Airflow's unit test configuration. When Airflow runs
@@ -29,6 +36,10 @@ dags_folder = {TEST_DAGS_FOLDER}
plugins_folder = {TEST_PLUGINS_FOLDER}
base_log_folder = {AIRFLOW_HOME}/logs
logging_level = INFO
+fab_logging_level = WARN
+log_filename_template = {{{{ ti.dag_id }}}}/{{{{ ti.task_id }}}}/{{{{ ts }}}}/{{{{ try_number }}}}.log
+log_processor_filename_template = {{{{ filename }}}}.log
+dag_processor_manager_log_location = {AIRFLOW_HOME}/logs/dag_processor_manager/dag_processor_manager.log
executor = SequentialExecutor
sql_alchemy_conn = sqlite:///{AIRFLOW_HOME}/unittests.db
load_examples = True
@@ -40,6 +51,8 @@ non_pooled_task_slot_count = 128
enable_xcom_pickling = False
killed_task_cleanup_time = 5
secure_mode = False
+hostname_callable = socket:getfqdn
+worker_precheck = False
[cli]
api_client = airflow.api.client.local_client
@@ -53,6 +66,7 @@ default_owner = airflow
[hive]
default_hive_mapred_queue = airflow
+mapred_job_name_template = Airflow HiveOperator task for {{hostname}}.{{dag_id}}.{{task_id}}.{{execution_date}}
[webserver]
base_url = http://localhost:8080
@@ -63,6 +77,7 @@ dag_default_view = tree
log_fetch_timeout_sec = 5
hide_paused_dags_by_default = False
page_size = 100
+rbac = False
[email]
email_backend = airflow.utils.email.send_email_smtp
@@ -83,6 +98,16 @@ result_backend = db+mysql://airflow:airflow@localhost:3306/airflow
flower_host = 0.0.0.0
flower_port = 5555
default_queue = default
+sync_parallelism = 0
+
+[mesos]
+master = localhost:5050
+framework_name = Airflow
+task_cpu = 1
+task_memory = 256
+checkpoint = False
+authenticate = False
+docker_image_slave = test/docker-airflow
[scheduler]
job_heartbeat_sec = 1
@@ -92,7 +117,15 @@ max_threads = 2
catchup_by_default = True
scheduler_zombie_task_threshold = 300
dag_dir_list_interval = 0
-max_tis_per_query = 0
+max_tis_per_query = 512
[admin]
hide_sensitive_variable_fields = True
+
+[elasticsearch]
+elasticsearch_host =
+elasticsearch_log_id_template = {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}
+elasticsearch_end_of_log_mark = end_of_log
+
+[kubernetes]
+dags_volume_claim = default
diff --git a/airflow/config_templates/default_webserver_config.py b/airflow/config_templates/default_webserver_config.py
new file mode 100644
index 0000000000..e61c7e1a45
--- /dev/null
+++ b/airflow/config_templates/default_webserver_config.py
@@ -0,0 +1,101 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from airflow import configuration as conf
+from flask_appbuilder.security.manager import AUTH_DB
+# from flask_appbuilder.security.manager import AUTH_LDAP
+# from flask_appbuilder.security.manager import AUTH_OAUTH
+# from flask_appbuilder.security.manager import AUTH_OID
+# from flask_appbuilder.security.manager import AUTH_REMOTE_USER
+basedir = os.path.abspath(os.path.dirname(__file__))
+
+# The SQLAlchemy connection string.
+SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')
+
+# Flask-WTF flag for CSRF
+CSRF_ENABLED = True
+
+# ----------------------------------------------------
+# AUTHENTICATION CONFIG
+# ----------------------------------------------------
+# For details on how to set up each of the following authentication, see
+# http://flask-appbuilder.readthedocs.io/en/latest/security.html# authentication-methods
+# for details.
+
+# The authentication type
+# AUTH_OID : Is for OpenID
+# AUTH_DB : Is for database
+# AUTH_LDAP : Is for LDAP
+# AUTH_REMOTE_USER : Is for using REMOTE_USER from web server
+# AUTH_OAUTH : Is for OAuth
+AUTH_TYPE = AUTH_DB
+
+# Uncomment to setup Full admin role name
+# AUTH_ROLE_ADMIN = 'Admin'
+
+# Uncomment to setup Public role name, no authentication needed
+# AUTH_ROLE_PUBLIC = 'Public'
+
+# Will allow user self registration
+# AUTH_USER_REGISTRATION = True
+
+# The default user self registration role
+# AUTH_USER_REGISTRATION_ROLE = "Public"
+
+# When using OAuth Auth, uncomment to setup provider(s) info
+# Google OAuth example:
+# OAUTH_PROVIDERS = [{
+# 'name':'google',
+# 'whitelist': ['@YOU_COMPANY_DOMAIN'], # optional
+# 'token_key':'access_token',
+# 'icon':'fa-google',
+# 'remote_app': {
+# 'base_url':'https://www.googleapis.com/oauth2/v2/',
+# 'request_token_params':{
+# 'scope': 'email profile'
+# },
+# 'access_token_url':'https://accounts.google.com/o/oauth2/token',
+# 'authorize_url':'https://accounts.google.com/o/oauth2/auth',
+# 'request_token_url': None,
+# 'consumer_key': CONSUMER_KEY,
+# 'consumer_secret': SECRET_KEY,
+# }
+# }]
+
+# When using LDAP Auth, setup the ldap server
+# AUTH_LDAP_SERVER = "ldap://ldapserver.new"
+
+# When using OpenID Auth, uncomment to setup OpenID providers.
+# example for OpenID authentication
+# OPENID_PROVIDERS = [
+# { 'name': 'Yahoo', 'url': 'https://me.yahoo.com' },
+# { 'name': 'AOL', 'url': 'http://openid.aol.com/<username>' },
+# { 'name': 'Flickr', 'url': 'http://www.flickr.com/<username>' },
+# { 'name': 'MyOpenID', 'url': 'https://www.myopenid.com' }]
+
+# ----------------------------------------------------
+# Theme CONFIG
+# ----------------------------------------------------
+# Flask App Builder comes up with a number of predefined themes
+# that you can use for Apache Airflow.
+# http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes
+# Please make sure to remove "navbar_color" configuration from airflow.cfg
+# in order to fully utilize the theme. (or use that property in conjunction with theme)
+# APP_THEME = "bootstrap-theme.css" # default bootstrap
diff --git a/airflow/configuration.py b/airflow/configuration.py
index 2bb2a49f91..332c069a3a 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -1,45 +1,49 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
+from base64 import b64encode
+from builtins import str
+from collections import OrderedDict
import copy
import errno
+from future import standard_library
import os
+import shlex
import six
+from six import iteritems
import subprocess
-import warnings
-import shlex
import sys
+import warnings
-from future import standard_library
-
-from six import iteritems
+from backports.configparser import ConfigParser
+from zope.deprecation import deprecated
+from airflow.exceptions import AirflowConfigException
from airflow.utils.log.logging_mixin import LoggingMixin
standard_library.install_aliases()
-from builtins import str
-from collections import OrderedDict
-from six.moves import configparser
-
-from airflow.exceptions import AirflowConfigException
-
log = LoggingMixin().log
# show Airflow's deprecation warnings
@@ -48,22 +52,14 @@
warnings.filterwarnings(
action='default', category=PendingDeprecationWarning, module='airflow')
-if six.PY3:
- ConfigParser = configparser.ConfigParser
-else:
- ConfigParser = configparser.SafeConfigParser
-
def generate_fernet_key():
try:
from cryptography.fernet import Fernet
except ImportError:
- pass
- try:
- key = Fernet.generate_key().decode()
- except NameError:
- key = "cryptography_not_found_storing_passwords_in_plain_text"
- return key
+ return ''
+ else:
+ return Fernet.generate_key().decode()
def expand_env_var(env_var):
@@ -102,11 +98,21 @@ def run_command(command):
return output
-_templates_dir = os.path.join(os.path.dirname(__file__), 'config_templates')
-with open(os.path.join(_templates_dir, 'default_airflow.cfg')) as f:
- DEFAULT_CONFIG = f.read()
-with open(os.path.join(_templates_dir, 'default_test.cfg')) as f:
- TEST_CONFIG = f.read()
+
+def _read_default_config_file(file_name):
+ templates_dir = os.path.join(os.path.dirname(__file__), 'config_templates')
+ file_path = os.path.join(templates_dir, file_name)
+ if six.PY2:
+ with open(file_path) as f:
+ config = f.read()
+ return config.decode('utf-8')
+ else:
+ with open(file_path, encoding='utf-8') as f:
+ return f.read()
+
+
+DEFAULT_CONFIG = _read_default_config_file('default_airflow.cfg')
+TEST_CONFIG = _read_default_config_file('default_test.cfg')
class AirflowConfigParser(ConfigParser):
@@ -118,28 +124,42 @@ class AirflowConfigParser(ConfigParser):
('core', 'sql_alchemy_conn'),
('core', 'fernet_key'),
('celery', 'broker_url'),
- ('celery', 'result_backend')
+ ('celery', 'result_backend'),
+ # Todo: remove this in Airflow 1.11
+ ('celery', 'celery_result_backend'),
+ ('atlas', 'password'),
+ ('smtp', 'smtp_password'),
+ ('ldap', 'bind_password'),
+ ('kubernetes', 'git_password'),
}
- def __init__(self, *args, **kwargs):
- ConfigParser.__init__(self, *args, **kwargs)
- self.read_string(parameterized_config(DEFAULT_CONFIG))
- self.is_validated = False
+ # A two-level mapping of (section -> new_name -> old_name). When reading
+ # new_name, the old_name will be checked to see if it exists. If it does a
+ # DeprecationWarning will be issued and the old name will be used instead
+ deprecated_options = {
+ 'celery': {
+ # Remove these keys in Airflow 1.11
+ 'worker_concurrency': 'celeryd_concurrency',
+ 'result_backend': 'celery_result_backend',
+ 'broker_url': 'celery_broker_url',
+ 'ssl_active': 'celery_ssl_active',
+ 'ssl_cert': 'celery_ssl_cert',
+ 'ssl_key': 'celery_ssl_key',
+ }
+ }
+ deprecation_format_string = (
+ 'The {old} option in [{section}] has been renamed to {new} - the old '
+ 'setting has been used, but please update your config.'
+ )
- def read_string(self, string, source='<string>'):
- """
- Read configuration from a string.
+ def __init__(self, default_config=None, *args, **kwargs):
+ super(AirflowConfigParser, self).__init__(*args, **kwargs)
- A backwards-compatible version of the ConfigParser.read_string()
- method that was introduced in Python 3.
- """
- # Python 3 added read_string() method
- if six.PY3:
- ConfigParser.read_string(self, string, source=source)
- # Python 2 requires StringIO buffer
- else:
- import StringIO
- self.readfp(StringIO.StringIO(string))
+ self.airflow_defaults = ConfigParser(*args, **kwargs)
+ if default_config is not None:
+ self.airflow_defaults.read_string(default_config)
+
+ self.is_validated = False
def _validate(self):
if (
@@ -178,33 +198,58 @@ def _get_env_var_option(self, section, key):
def _get_cmd_option(self, section, key):
fallback_key = key + '_cmd'
# if this is a valid command key...
- if (section, key) in AirflowConfigParser.as_command_stdout:
- # if the original key is present, return it no matter what
- if self.has_option(section, key):
- return ConfigParser.get(self, section, key)
- # otherwise, execute the fallback key
- elif self.has_option(section, fallback_key):
- command = self.get(section, fallback_key)
+ if (section, key) in self.as_command_stdout:
+ if super(AirflowConfigParser, self) \
+ .has_option(section, fallback_key):
+ command = super(AirflowConfigParser, self) \
+ .get(section, fallback_key)
return run_command(command)
def get(self, section, key, **kwargs):
section = str(section).lower()
key = str(key).lower()
+ deprecated_name = self.deprecated_options.get(section, {}).get(key, None)
+
# first check environment variables
option = self._get_env_var_option(section, key)
if option is not None:
return option
+ if deprecated_name:
+ option = self._get_env_var_option(section, deprecated_name)
+ if option is not None:
+ self._warn_deprecate(section, key, deprecated_name)
+ return option
# ...then the config file
- if self.has_option(section, key):
+ if super(AirflowConfigParser, self).has_option(section, key):
+ # Use the parent's methods to get the actual config here to be able to
+ # separate the config from default config.
return expand_env_var(
- ConfigParser.get(self, section, key, **kwargs))
+ super(AirflowConfigParser, self).get(section, key, **kwargs))
+ if deprecated_name:
+ if super(AirflowConfigParser, self).has_option(section, deprecated_name):
+ self._warn_deprecate(section, key, deprecated_name)
+ return expand_env_var(super(AirflowConfigParser, self).get(
+ section,
+ deprecated_name,
+ **kwargs
+ ))
# ...then commands
option = self._get_cmd_option(section, key)
if option:
return option
+ if deprecated_name:
+ option = self._get_cmd_option(section, deprecated_name)
+ if option:
+ self._warn_deprecate(section, key, deprecated_name)
+ return option
+
+ # ...then the default config
+ if self.airflow_defaults.has_option(section, key):
+ return expand_env_var(
+ self.airflow_defaults.get(section, key, **kwargs))
else:
log.warning(
@@ -235,9 +280,34 @@ def getfloat(self, section, key):
return float(self.get(section, key))
def read(self, filenames):
- ConfigParser.read(self, filenames)
+ super(AirflowConfigParser, self).read(filenames)
+ self._validate()
+
+ def read_dict(self, *args, **kwargs):
+ super(AirflowConfigParser, self).read_dict(*args, **kwargs)
self._validate()
+ def has_option(self, section, option):
+ try:
+ # Using self.get() to avoid reimplementing the priority order
+ # of config variables (env, config, cmd, defaults)
+ self.get(section, option)
+ return True
+ except AirflowConfigException:
+ return False
+
+ def remove_option(self, section, option, remove_default=True):
+ """
+ Remove an option if it exists in config from a file or
+ default config. If both of config have the same option, this removes
+ the option in both configs unless remove_default=False.
+ """
+ if super(AirflowConfigParser, self).has_option(section, option):
+ super(AirflowConfigParser, self).remove_option(section, option)
+
+ if self.airflow_defaults.has_option(section, option) and remove_default:
+ self.airflow_defaults.remove_option(section, option)
+
def getsection(self, section):
"""
Returns the section as a dict. Values are converted to int, float, bool
@@ -245,47 +315,64 @@ def getsection(self, section):
:param section: section from the config
:return: dict
"""
+ if (section not in self._sections and
+ section not in self.airflow_defaults._sections):
+ return None
+
+ _section = copy.deepcopy(self.airflow_defaults._sections[section])
+
if section in self._sections:
- _section = self._sections[section]
- for key, val in iteritems(self._sections[section]):
+ _section.update(copy.deepcopy(self._sections[section]))
+
+ section_prefix = 'AIRFLOW__{S}__'.format(S=section.upper())
+ for env_var in sorted(os.environ.keys()):
+ if env_var.startswith(section_prefix):
+ key = env_var.replace(section_prefix, '').lower()
+ _section[key] = self._get_env_var_option(section, key)
+
+ for key, val in iteritems(_section):
+ try:
+ val = int(val)
+ except ValueError:
try:
- val = int(val)
+ val = float(val)
except ValueError:
- try:
- val = float(val)
- except ValueError:
- if val.lower() in ('t', 'true'):
- val = True
- elif val.lower() in ('f', 'false'):
- val = False
- _section[key] = val
- return _section
-
- return None
-
- def as_dict(self, display_source=False, display_sensitive=False):
+ if val.lower() in ('t', 'true'):
+ val = True
+ elif val.lower() in ('f', 'false'):
+ val = False
+ _section[key] = val
+ return _section
+
+ def as_dict(
+ self, display_source=False, display_sensitive=False, raw=False):
"""
Returns the current configuration as an OrderedDict of OrderedDicts.
:param display_source: If False, the option value is returned. If True,
a tuple of (option_value, source) is returned. Source is either
- 'airflow.cfg' or 'default'.
+ 'airflow.cfg', 'default', 'env var', or 'cmd'.
:type display_source: bool
:param display_sensitive: If True, the values of options set by env
vars and bash commands will be displayed. If False, those options
are shown as '< hidden >'
:type display_sensitive: bool
+ :param raw: Should the values be output as interpolated values, or the
+ "raw" form that can be fed back in to ConfigParser
+ :type raw: bool
"""
- cfg = copy.deepcopy(self._sections)
-
- # remove __name__ (affects Python 2 only)
- for options in cfg.values():
- options.pop('__name__', None)
-
- # add source
- if display_source:
- for section in cfg:
- for k, v in cfg[section].items():
- cfg[section][k] = (v, 'airflow config')
+ cfg = {}
+ configs = [
+ ('default', self.airflow_defaults),
+ ('airflow.cfg', self),
+ ]
+
+ for (source_name, config) in configs:
+ for section in config.sections():
+ sect = cfg.setdefault(section, OrderedDict())
+ for (k, val) in config.items(section=section, raw=raw):
+ if display_source:
+ val = (val, source_name)
+ sect[k] = val
# add env vars and overwrite because they have priority
for ev in [ev for ev in os.environ if ev.startswith('AIRFLOW__')]:
@@ -293,26 +380,28 @@ def as_dict(self, display_source=False, display_sensitive=False):
_, section, key = ev.split('__')
opt = self._get_env_var_option(section, key)
except ValueError:
- opt = None
- if opt:
- if (
- not display_sensitive
- and ev != 'AIRFLOW__CORE__UNIT_TEST_MODE'):
- opt = '< hidden >'
- if display_source:
- opt = (opt, 'env var')
- cfg.setdefault(section.lower(), OrderedDict()).update(
- {key.lower(): opt})
+ continue
+ if (not display_sensitive and ev != 'AIRFLOW__CORE__UNIT_TEST_MODE'):
+ opt = '< hidden >'
+ elif raw:
+ opt = opt.replace('%', '%%')
+ if display_source:
+ opt = (opt, 'env var')
+ cfg.setdefault(section.lower(), OrderedDict()).update(
+ {key.lower(): opt})
# add bash commands
- for (section, key) in AirflowConfigParser.as_command_stdout:
+ for (section, key) in self.as_command_stdout:
opt = self._get_cmd_option(section, key)
if opt:
if not display_sensitive:
opt = '< hidden >'
if display_source:
- opt = (opt, 'bash cmd')
+ opt = (opt, 'cmd')
+ elif raw:
+ opt = opt.replace('%', '%%')
cfg.setdefault(section, OrderedDict()).update({key: opt})
+ del cfg[section][key + '_cmd']
return cfg
@@ -329,6 +418,17 @@ def load_test_config(self):
# then read any "custom" test settings
self.read(TEST_CONFIG_FILE)
+ def _warn_deprecate(self, section, key, deprecated_name):
+ warnings.warn(
+ self.deprecation_format_string.format(
+ old=deprecated_name,
+ new=key,
+ section=section,
+ ),
+ DeprecationWarning,
+ stacklevel=3,
+ )
+
def mkdir_p(path):
try:
@@ -337,26 +437,27 @@ def mkdir_p(path):
if exc.errno == errno.EEXIST and os.path.isdir(path):
pass
else:
- raise AirflowConfigException('Had trouble creating a directory')
+ raise AirflowConfigException(
+ 'Error creating {}: {}'.format(path, exc.strerror))
-# Setting AIRFLOW_HOME and AIRFLOW_CONFIG from environment variables, using
-# "~/airflow" and "~/airflow/airflow.cfg" respectively as defaults.
+def get_airflow_home():
+ return expand_env_var(os.environ.get('AIRFLOW_HOME', '~/airflow'))
-if 'AIRFLOW_HOME' not in os.environ:
- AIRFLOW_HOME = expand_env_var('~/airflow')
-else:
- AIRFLOW_HOME = expand_env_var(os.environ['AIRFLOW_HOME'])
+def get_airflow_config(airflow_home):
+ if 'AIRFLOW_CONFIG' not in os.environ:
+ return os.path.join(airflow_home, 'airflow.cfg')
+ return expand_env_var(os.environ['AIRFLOW_CONFIG'])
+
+
+# Setting AIRFLOW_HOME and AIRFLOW_CONFIG from environment variables, using
+# "~/airflow" and "$AIRFLOW_HOME/airflow.cfg" respectively as defaults.
+
+AIRFLOW_HOME = get_airflow_home()
+AIRFLOW_CONFIG = get_airflow_config(AIRFLOW_HOME)
mkdir_p(AIRFLOW_HOME)
-if 'AIRFLOW_CONFIG' not in os.environ:
- if os.path.isfile(expand_env_var('~/airflow.cfg')):
- AIRFLOW_CONFIG = expand_env_var('~/airflow.cfg')
- else:
- AIRFLOW_CONFIG = AIRFLOW_HOME + '/airflow.cfg'
-else:
- AIRFLOW_CONFIG = expand_env_var(os.environ['AIRFLOW_CONFIG'])
# Set up dags folder for unit tests
# this directory won't exist if users install via pip
@@ -398,6 +499,8 @@ def parameterized_config(template):
else:
FERNET_KEY = ''
+SECRET_KEY = b64encode(os.urandom(16)).decode('utf-8')
+
TEMPLATE_START = (
'# ----------------------- TEMPLATE BEGINS HERE -----------------------')
if not os.path.isfile(TEST_CONFIG_FILE):
@@ -414,59 +517,49 @@ def parameterized_config(template):
)
with open(AIRFLOW_CONFIG, 'w') as f:
cfg = parameterized_config(DEFAULT_CONFIG)
- f.write(cfg.split(TEMPLATE_START)[-1].strip())
+ cfg = cfg.split(TEMPLATE_START)[-1].strip()
+ if six.PY2:
+ cfg = cfg.encode('utf8')
+ f.write(cfg)
log.info("Reading the config from %s", AIRFLOW_CONFIG)
-conf = AirflowConfigParser()
-conf.read(AIRFLOW_CONFIG)
-
-
-def load_test_config():
- """
- Load the unit test configuration.
-
- Note: this is not reversible.
- """
- conf.load_test_config()
-
-if conf.getboolean('core', 'unit_test_mode'):
- load_test_config()
-
-
-def get(section, key, **kwargs):
- return conf.get(section, key, **kwargs)
-
+conf = AirflowConfigParser(default_config=parameterized_config(DEFAULT_CONFIG))
-def getboolean(section, key):
- return conf.getboolean(section, key)
-
-
-def getfloat(section, key):
- return conf.getfloat(section, key)
-
-
-def getint(section, key):
- return conf.getint(section, key)
-
-
-def getsection(section):
- return conf.getsection(section)
-
-
-def has_option(section, key):
- return conf.has_option(section, key)
+conf.read(AIRFLOW_CONFIG)
-def remove_option(section, option):
- return conf.remove_option(section, option)
+if conf.getboolean('webserver', 'rbac'):
+ DEFAULT_WEBSERVER_CONFIG = _read_default_config_file('default_webserver_config.py')
+ WEBSERVER_CONFIG = AIRFLOW_HOME + '/webserver_config.py'
-def as_dict(display_source=False, display_sensitive=False):
- return conf.as_dict(
- display_source=display_source, display_sensitive=display_sensitive)
-as_dict.__doc__ = conf.as_dict.__doc__
+ if not os.path.isfile(WEBSERVER_CONFIG):
+ log.info('Creating new FAB webserver config file in: %s', WEBSERVER_CONFIG)
+ with open(WEBSERVER_CONFIG, 'w') as f:
+ f.write(DEFAULT_WEBSERVER_CONFIG)
+if conf.getboolean('core', 'unit_test_mode'):
+ conf.load_test_config()
-def set(section, option, value): # noqa
- return conf.set(section, option, value)
+# Historical convenience functions to access config entries
+
+load_test_config = conf.load_test_config
+get = conf.get
+getboolean = conf.getboolean
+getfloat = conf.getfloat
+getint = conf.getint
+getsection = conf.getsection
+has_option = conf.has_option
+remove_option = conf.remove_option
+as_dict = conf.as_dict
+set = conf.set # noqa
+
+for func in [load_test_config, get, getboolean, getfloat, getint, has_option,
+ remove_option, as_dict, set]:
+ deprecated(
+ func,
+ "Accessing configuration method '{f.__name__}' directly from "
+ "the configuration module is deprecated. Please access the "
+ "configuration from the 'configuration.conf' object via "
+ "'conf.{f.__name__}'".format(f=func))
diff --git a/airflow/contrib/__init__.py b/airflow/contrib/__init__.py
index c82f5790fe..114d189da1 100644
--- a/airflow/contrib/__init__.py
+++ b/airflow/contrib/__init__.py
@@ -1,14 +1,18 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/contrib/auth/__init__.py b/airflow/contrib/auth/__init__.py
index c82f5790fe..114d189da1 100644
--- a/airflow/contrib/auth/__init__.py
+++ b/airflow/contrib/auth/__init__.py
@@ -1,14 +1,18 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/contrib/auth/backends/__init__.py b/airflow/contrib/auth/backends/__init__.py
index c82f5790fe..114d189da1 100644
--- a/airflow/contrib/auth/backends/__init__.py
+++ b/airflow/contrib/auth/backends/__init__.py
@@ -1,14 +1,18 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/contrib/auth/backends/github_enterprise_auth.py b/airflow/contrib/auth/backends/github_enterprise_auth.py
index e4665cb0c1..e1b8b13342 100644
--- a/airflow/contrib/auth/backends/github_enterprise_auth.py
+++ b/airflow/contrib/auth/backends/github_enterprise_auth.py
@@ -1,10 +1,13 @@
-# Copyright 2015 Matthew Pelland (matt@pelland.io)
+# -*- coding: utf-8 -*-
+#
+# See the NOTICE file distributed with this work for additional information
+# regarding copyright ownership.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
@@ -14,18 +17,14 @@
import flask_login
# Need to expose these downstream
-# pylint: disable=unused-import
-from flask_login import (current_user,
- logout_user,
- login_required,
- login_user)
-# pylint: enable=unused-import
+# flake8: noqa: F401
+from flask_login import current_user, logout_user, login_required, login_user
from flask import url_for, redirect, request
from flask_oauthlib.client import OAuth
-from airflow import models, configuration, settings
+from airflow import models, configuration
from airflow.configuration import AirflowConfigException
from airflow.utils.db import provide_session
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -34,7 +33,7 @@
def get_config_param(param):
- return str(configuration.get('github_enterprise', param))
+ return str(configuration.conf.get('github_enterprise', param))
class GHEUser(models.User):
@@ -42,28 +41,31 @@ class GHEUser(models.User):
def __init__(self, user):
self.user = user
+ @property
def is_active(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return True
+ @property
def is_authenticated(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return True
+ @property
def is_anonymous(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return False
def get_id(self):
- '''Returns the current user id as required by flask_login'''
+ """Returns the current user id as required by flask_login"""
return self.user.get_id()
def data_profiling(self):
- '''Provides access to data profiling tools'''
+ """Provides access to data profiling tools"""
return True
def is_superuser(self):
- '''Access all the things'''
+ """Access all the things"""
return True
@@ -79,17 +81,18 @@ def __init__(self):
self.login_manager.login_view = 'airflow.login'
self.flask_app = None
self.ghe_oauth = None
- self.api_rev = None
+ self.api_url = None
def ghe_api_route(self, leaf):
- if not self.api_rev:
- self.api_rev = get_config_param('api_rev')
-
- return '/'.join(['https:/',
- self.ghe_host,
- 'api',
- self.api_rev,
- leaf.strip('/')])
+ if not self.api_url:
+ self.api_url = (
+ 'https://api.github.com' if self.ghe_host == 'github.com'
+ else '/'.join(['https:/',
+ self.ghe_host,
+ 'api',
+ get_config_param('api_rev')])
+ )
+ return self.api_url + leaf
def init_app(self, flask_app):
self.flask_app = flask_app
@@ -119,11 +122,11 @@ def init_app(self, flask_app):
self.oauth_callback)
def login(self, request):
- _log.debug('Redirecting user to GHE login')
+ log.debug('Redirecting user to GHE login')
return self.ghe_oauth.authorize(callback=url_for(
'ghe_oauth_callback',
- _external=True,
- next=request.args.get('next') or request.referrer or None))
+ _external=True),
+ state=request.args.get('next') or request.referrer or None)
def get_ghe_user_profile_info(self, ghe_token):
resp = self.ghe_oauth.get(self.ghe_api_route('/user'),
@@ -145,8 +148,9 @@ def ghe_team_check(self, username, ghe_token):
get_config_param('allowed_teams').split(',')]
except ValueError:
# this is to deprecate using the string name for a team
- raise ValueError('it appears that you are using the string name for a team, '
- 'please use the id number instead')
+ raise ValueError(
+ 'it appears that you are using the string name for a team, '
+ 'please use the id number instead')
except AirflowConfigException:
# No allowed teams defined, let anyone in GHE in.
@@ -169,9 +173,9 @@ def ghe_team_check(self, username, ghe_token):
if team['id'] in allowed_teams:
return True
- _log.debug('Denying access for user "%s", not a member of "%s"',
- username,
- str(allowed_teams))
+ log.debug('Denying access for user "%s", not a member of "%s"',
+ username,
+ str(allowed_teams))
return False
@@ -186,9 +190,9 @@ def load_user(self, userid, session=None):
@provide_session
def oauth_callback(self, session=None):
- _log.debug('GHE OAuth callback called')
+ log.debug('GHE OAuth callback called')
- next_url = request.args.get('next') or url_for('admin.index')
+ next_url = request.args.get('state') or url_for('admin.index')
resp = self.ghe_oauth.authorized_response()
@@ -206,7 +210,7 @@ def oauth_callback(self, session=None):
return redirect(url_for('airflow.noaccess'))
except AuthenticationError:
- _log.exception('')
+ log.exception('')
return redirect(url_for('airflow.noaccess'))
user = session.query(models.User).filter(
@@ -225,6 +229,7 @@ def oauth_callback(self, session=None):
return redirect(next_url)
+
login_manager = GHEAuthBackend()
diff --git a/airflow/contrib/auth/backends/google_auth.py b/airflow/contrib/auth/backends/google_auth.py
index 65e0f3ab89..ddbcb1222f 100644
--- a/airflow/contrib/auth/backends/google_auth.py
+++ b/airflow/contrib/auth/backends/google_auth.py
@@ -1,31 +1,32 @@
-# Copyright 2016 Ananya Mishra (am747@cornell.edu)
+# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
import flask_login
# Need to expose these downstream
-# pylint: disable=unused-import
-from flask_login import (current_user,
- logout_user,
- login_required,
- login_user)
-# pylint: enable=unused-import
+# flake8: noqa: F401
+from flask_login import current_user, logout_user, login_required, login_user
from flask import url_for, redirect, request
from flask_oauthlib.client import OAuth
-from airflow import models, configuration, settings
+from airflow import models, configuration
from airflow.utils.db import provide_session
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -33,7 +34,7 @@
def get_config_param(param):
- return str(configuration.get('google', param))
+ return str(configuration.conf.get('google', param))
class GoogleUser(models.User):
@@ -41,28 +42,31 @@ class GoogleUser(models.User):
def __init__(self, user):
self.user = user
+ @property
def is_active(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return True
+ @property
def is_authenticated(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return True
+ @property
def is_anonymous(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return False
def get_id(self):
- '''Returns the current user id as required by flask_login'''
+ """Returns the current user id as required by flask_login"""
return self.user.get_id()
def data_profiling(self):
- '''Provides access to data profiling tools'''
+ """Provides access to data profiling tools"""
return True
def is_superuser(self):
- '''Access all the things'''
+ """Access all the things"""
return True
@@ -108,13 +112,13 @@ def login(self, request):
log.debug('Redirecting user to Google login')
return self.google_oauth.authorize(callback=url_for(
'google_oauth_callback',
- _external=True,
- _scheme='https',
- next=request.args.get('next') or request.referrer or None))
+ _external=True),
+ state=request.args.get('next') or request.referrer or None)
def get_google_user_profile_info(self, google_token):
- resp = self.google_oauth.get('https://www.googleapis.com/oauth2/v1/userinfo',
- token=(google_token, ''))
+ resp = self.google_oauth.get(
+ 'https://www.googleapis.com/oauth2/v1/userinfo',
+ token=(google_token, ''))
if not resp or resp.status != 200:
raise AuthenticationError(
@@ -143,7 +147,7 @@ def load_user(self, userid, session=None):
def oauth_callback(self, session=None):
log.debug('Google OAuth callback called')
- next_url = request.args.get('next') or url_for('admin.index')
+ next_url = request.args.get('state') or url_for('admin.index')
resp = self.google_oauth.authorized_response()
@@ -179,6 +183,7 @@ def oauth_callback(self, session=None):
return redirect(next_url)
+
login_manager = GoogleAuthBackend()
diff --git a/airflow/contrib/auth/backends/kerberos_auth.py b/airflow/contrib/auth/backends/kerberos_auth.py
index 21e0ffb7d7..08ae898737 100644
--- a/airflow/contrib/auth/backends/kerberos_auth.py
+++ b/airflow/contrib/auth/backends/kerberos_auth.py
@@ -1,32 +1,37 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import logging
import flask_login
-from flask_login import login_required, current_user, logout_user
+from airflow.exceptions import AirflowConfigException
+from flask_login import current_user
from flask import flash
-from wtforms import (
- Form, PasswordField, StringField)
+from wtforms import Form, PasswordField, StringField
from wtforms.validators import InputRequired
# pykerberos should be used as it verifies the KDC, the "kerberos" module does not do so
# and make it possible to spoof the KDC
import kerberos
-import airflow.security.utils as utils
+from airflow.security import utils
from flask import url_for, redirect
-from airflow import settings
from airflow import models
from airflow import configuration
from airflow.utils.db import provide_session
@@ -47,42 +52,58 @@ def __init__(self, user):
@staticmethod
def authenticate(username, password):
- service_principal = "%s/%s" % (configuration.get('kerberos', 'principal'), utils.get_fqdn())
- realm = configuration.get("kerberos", "default_realm")
- user_principal = utils.principal_from_username(username)
+ service_principal = "%s/%s" % (
+ configuration.conf.get('kerberos', 'principal'),
+ utils.get_fqdn()
+ )
+ realm = configuration.conf.get("kerberos", "default_realm")
+
+ try:
+ user_realm = configuration.conf.get("security", "default_realm")
+ except AirflowConfigException:
+ user_realm = realm
+
+ user_principal = utils.principal_from_username(username, user_realm)
try:
# this is pykerberos specific, verify = True is needed to prevent KDC spoofing
- if not kerberos.checkPassword(user_principal, password, service_principal, realm, True):
+ if not kerberos.checkPassword(user_principal,
+ password,
+ service_principal, realm, True):
raise AuthenticationError()
except kerberos.KrbError as e:
- logging.error('Password validation for principal %s failed %s', user_principal, e)
+ logging.error(
+ 'Password validation for user '
+ '%s in realm %s failed %s', user_principal, realm, e)
raise AuthenticationError(e)
return
+ @property
def is_active(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return True
+ @property
def is_authenticated(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return True
+ @property
def is_anonymous(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return False
def get_id(self):
- '''Returns the current user id as required by flask_login'''
+ """Returns the current user id as required by flask_login"""
return self.user.get_id()
def data_profiling(self):
- '''Provides access to data profiling tools'''
+ """Provides access to data profiling tools"""
return True
def is_superuser(self):
- '''Access all the things'''
+ """Access all the things"""
return True
@@ -98,7 +119,7 @@ def load_user(userid, session=None):
@provide_session
def login(self, request, session=None):
- if current_user.is_authenticated():
+ if current_user.is_authenticated:
flash("You are already logged in")
return redirect(url_for('index'))
diff --git a/airflow/contrib/auth/backends/ldap_auth.py b/airflow/contrib/auth/backends/ldap_auth.py
index 98c620e392..587fc2be16 100644
--- a/airflow/contrib/auth/backends/ldap_auth.py
+++ b/airflow/contrib/auth/backends/ldap_auth.py
@@ -1,26 +1,30 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from future.utils import native
import flask_login
-from flask_login import login_required, current_user, logout_user
+from flask_login import login_required, current_user, logout_user # noqa: F401
from flask import flash
-from wtforms import (
- Form, PasswordField, StringField)
+from wtforms import Form, PasswordField, StringField
from wtforms.validators import InputRequired
-from ldap3 import Server, Connection, Tls, LEVEL, SUBTREE, BASE
+from ldap3 import Server, Connection, Tls, LEVEL, SUBTREE
import ssl
from flask import url_for, redirect
@@ -51,16 +55,18 @@ class LdapException(Exception):
def get_ldap_connection(dn=None, password=None):
- tls_configuration = None
- use_ssl = False
try:
- cacert = configuration.get("ldap", "cacert")
- tls_configuration = Tls(validate=ssl.CERT_REQUIRED, ca_certs_file=cacert)
- use_ssl = True
- except:
+ cacert = configuration.conf.get("ldap", "cacert")
+ except AirflowConfigException:
pass
- server = Server(configuration.get("ldap", "uri"), use_ssl, tls_configuration)
+ tls_configuration = Tls(validate=ssl.CERT_REQUIRED,
+ ca_certs_file=cacert)
+
+ server = Server(configuration.conf.get("ldap", "uri"),
+ use_ssl=True,
+ tls=tls_configuration)
+
conn = Connection(server, native(dn), native(password))
if not conn.bind():
@@ -72,12 +78,14 @@ def get_ldap_connection(dn=None, password=None):
def group_contains_user(conn, search_base, group_filter, user_name_attr, username):
search_filter = '(&({0}))'.format(group_filter)
+
if not conn.search(native(search_base), native(search_filter),
attributes=[native(user_name_attr)]):
log.warning("Unable to find group for %s %s", search_base, search_filter)
else:
for entry in conn.entries:
- if username in getattr(entry, user_name_attr).values:
+ if username.lower() in map(lambda attr: attr.lower(),
+ getattr(entry, user_name_attr).values):
return True
return False
@@ -86,8 +94,8 @@ def group_contains_user(conn, search_base, group_filter, user_name_attr, usernam
def groups_user(conn, search_base, user_filter, user_name_att, username):
search_filter = "(&({0})({1}={2}))".format(user_filter, user_name_att, username)
try:
- memberof_attr = configuration.get("ldap", "group_member_attr")
- except:
+ memberof_attr = configuration.conf.get("ldap", "group_member_attr")
+ except Exception:
memberof_attr = "memberOf"
res = conn.search(native(search_base), native(search_filter),
attributes=[native(memberof_attr)])
@@ -122,13 +130,13 @@ def __init__(self, user):
self.ldap_groups = []
# Load and cache superuser and data_profiler settings.
- conn = get_ldap_connection(configuration.get("ldap", "bind_user"),
- configuration.get("ldap", "bind_password"))
+ conn = get_ldap_connection(configuration.conf.get("ldap", "bind_user"),
+ configuration.conf.get("ldap", "bind_password"))
superuser_filter = None
data_profiler_filter = None
try:
- superuser_filter = configuration.get("ldap", "superuser_filter")
+ superuser_filter = configuration.conf.get("ldap", "superuser_filter")
except AirflowConfigException:
pass
@@ -137,14 +145,14 @@ def __init__(self, user):
log.debug("Missing configuration for superuser settings or empty. Skipping.")
else:
self.superuser = group_contains_user(conn,
- configuration.get("ldap", "basedn"),
+ configuration.conf.get("ldap", "basedn"),
superuser_filter,
- configuration.get("ldap",
- "user_name_attr"),
+ configuration.conf.get("ldap",
+ "user_name_attr"),
user.username)
try:
- data_profiler_filter = configuration.get("ldap", "data_profiler_filter")
+ data_profiler_filter = configuration.conf.get("ldap", "data_profiler_filter")
except AirflowConfigException:
pass
@@ -153,47 +161,48 @@ def __init__(self, user):
log.debug("Missing configuration for data profiler settings or empty. "
"Skipping.")
else:
- self.data_profiler = group_contains_user(conn,
- configuration.get("ldap", "basedn"),
- data_profiler_filter,
- configuration.get("ldap",
- "user_name_attr"),
- user.username)
+ self.data_profiler = group_contains_user(
+ conn,
+ configuration.conf.get("ldap", "basedn"),
+ data_profiler_filter,
+ configuration.conf.get("ldap",
+ "user_name_attr"),
+ user.username
+ )
# Load the ldap group(s) a user belongs to
try:
- self.ldap_groups = groups_user(conn,
- configuration.get("ldap", "basedn"),
- configuration.get("ldap", "user_filter"),
- configuration.get("ldap", "user_name_attr"),
- user.username)
+ self.ldap_groups = groups_user(
+ conn,
+ configuration.conf.get("ldap", "basedn"),
+ configuration.conf.get("ldap", "user_filter"),
+ configuration.conf.get("ldap", "user_name_attr"),
+ user.username
+ )
except AirflowConfigException:
log.debug("Missing configuration for ldap settings. Skipping")
@staticmethod
def try_login(username, password):
- conn = get_ldap_connection(configuration.get("ldap", "bind_user"),
- configuration.get("ldap", "bind_password"))
+ conn = get_ldap_connection(configuration.conf.get("ldap", "bind_user"),
+ configuration.conf.get("ldap", "bind_password"))
search_filter = "(&({0})({1}={2}))".format(
- configuration.get("ldap", "user_filter"),
- configuration.get("ldap", "user_name_attr"),
+ configuration.conf.get("ldap", "user_filter"),
+ configuration.conf.get("ldap", "user_name_attr"),
username
)
- search_scopes = {
- "LEVEL": LEVEL,
- "SUBTREE": SUBTREE,
- "BASE": BASE
- }
-
search_scope = LEVEL
- if configuration.has_option("ldap", "search_scope"):
- search_scope = SUBTREE if configuration.get("ldap", "search_scope") == "SUBTREE" else LEVEL
+ if configuration.conf.has_option("ldap", "search_scope"):
+ if configuration.conf.get("ldap", "search_scope") == "SUBTREE":
+ search_scope = SUBTREE
+ else:
+ search_scope = LEVEL
# todo: BASE or ONELEVEL?
- res = conn.search(native(configuration.get("ldap", "basedn")),
+ res = conn.search(native(configuration.conf.get("ldap", "basedn")),
native(search_filter),
search_scope=native(search_scope))
@@ -213,39 +222,46 @@ def try_login(username, password):
try:
conn = get_ldap_connection(entry['dn'], password)
- except KeyError as e:
+ except KeyError:
log.error("""
- Unable to parse LDAP structure. If you're using Active Directory and not specifying an OU, you must set search_scope=SUBTREE in airflow.cfg.
+ Unable to parse LDAP structure. If you're using Active Directory
+ and not specifying an OU, you must set search_scope=SUBTREE in airflow.cfg.
%s
""" % traceback.format_exc())
- raise LdapException("Could not parse LDAP structure. Try setting search_scope in airflow.cfg, or check logs")
+ raise LdapException(
+ "Could not parse LDAP structure. "
+ "Try setting search_scope in airflow.cfg, or check logs"
+ )
if not conn:
log.info("Password incorrect for user %s", username)
raise AuthenticationError("Invalid username or password")
+ @property
def is_active(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return True
+ @property
def is_authenticated(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return True
+ @property
def is_anonymous(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return False
def get_id(self):
- '''Returns the current user id as required by flask_login'''
+ """Returns the current user id as required by flask_login"""
return self.user.get_id()
def data_profiling(self):
- '''Provides access to data profiling tools'''
+ """Provides access to data profiling tools"""
return self.data_profiler
def is_superuser(self):
- '''Access all the things'''
+ """Access all the things"""
return self.superuser
@@ -259,9 +275,10 @@ def load_user(userid, session=None):
user = session.query(models.User).filter(models.User.id == int(userid)).first()
return LdapUser(user)
+
@provide_session
def login(self, request, session=None):
- if current_user.is_authenticated():
+ if current_user.is_authenticated:
flash("You are already logged in")
return redirect(url_for('admin.index'))
@@ -290,9 +307,10 @@ def login(self, request, session=None):
user = models.User(
username=username,
is_superuser=False)
+ session.add(user)
- session.merge(user)
session.commit()
+ session.merge(user)
flask_login.login_user(LdapUser(user))
session.commit()
diff --git a/airflow/contrib/auth/backends/password_auth.py b/airflow/contrib/auth/backends/password_auth.py
index e380ec4b7b..dcdb1d1225 100644
--- a/airflow/contrib/auth/backends/password_auth.py
+++ b/airflow/contrib/auth/backends/password_auth.py
@@ -1,33 +1,38 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from __future__ import unicode_literals
from sys import version_info
+import base64
import flask_login
-from flask_login import login_required, current_user, logout_user
-from flask import flash
-from wtforms import (
- Form, PasswordField, StringField)
+from flask_login import login_required, current_user, logout_user # noqa: F401
+from flask import flash, Response
+from wtforms import Form, PasswordField, StringField
from wtforms.validators import InputRequired
+from functools import wraps
-from flask import url_for, redirect
+from flask import url_for, redirect, make_response
from flask_bcrypt import generate_password_hash, check_password_hash
-from sqlalchemy import (
- Column, String, DateTime)
+from sqlalchemy import Column, String
from sqlalchemy.ext.hybrid import hybrid_property
from airflow import settings
@@ -58,7 +63,7 @@ def password(self):
return self._password
@password.setter
- def _set_password(self, plaintext):
+ def password(self, plaintext):
self._password = generate_password_hash(plaintext, 12)
if PY3:
self._password = str(self._password, 'utf-8')
@@ -66,29 +71,31 @@ def _set_password(self, plaintext):
def authenticate(self, plaintext):
return check_password_hash(self._password, plaintext)
+ @property
def is_active(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return True
+ @property
def is_authenticated(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return True
+ @property
def is_anonymous(self):
- '''Required by flask_login'''
+ """Required by flask_login"""
return False
def get_id(self):
- '''Returns the current user id as required by flask_login'''
+ """Returns the current user id as required by flask_login"""
return str(self.id)
def data_profiling(self):
- '''Provides access to data profiling tools'''
+ """Provides access to data profiling tools"""
return True
def is_superuser(self):
- '''Access all the things'''
- return True
+ return hasattr(self, 'user') and self.user.is_superuser()
@login_manager.user_loader
@@ -102,9 +109,37 @@ def load_user(userid, session=None):
return PasswordUser(user)
+def authenticate(session, username, password):
+ """
+ Authenticate a PasswordUser with the specified
+ username/password.
+
+ :param session: An active SQLAlchemy session
+ :param username: The username
+ :param password: The password
+
+ :raise AuthenticationError: if an error occurred
+ :return: a PasswordUser
+ """
+ if not username or not password:
+ raise AuthenticationError()
+
+ user = session.query(PasswordUser).filter(
+ PasswordUser.username == username).first()
+
+ if not user:
+ raise AuthenticationError()
+
+ if not user.authenticate(password):
+ raise AuthenticationError()
+
+ log.info("User %s successfully authenticated", username)
+ return user
+
+
@provide_session
def login(self, request, session=None):
- if current_user.is_authenticated():
+ if current_user.is_authenticated:
flash("You are already logged in")
return redirect(url_for('admin.index'))
@@ -117,26 +152,9 @@ def login(self, request, session=None):
username = request.form.get("username")
password = request.form.get("password")
- if not username or not password:
- return self.render('airflow/login.html',
- title="Airflow - Login",
- form=form)
-
try:
- user = session.query(PasswordUser).filter(
- PasswordUser.username == username).first()
-
- if not user:
- session.close()
- raise AuthenticationError()
-
- if not user.authenticate(password):
- session.close()
- raise AuthenticationError()
- log.info("User %s successfully authenticated", username)
-
+ user = authenticate(session, username, password)
flask_login.login_user(user)
- session.commit()
return redirect(request.args.get("next") or url_for("admin.index"))
except AuthenticationError:
@@ -144,8 +162,55 @@ def login(self, request, session=None):
return self.render('airflow/login.html',
title="Airflow - Login",
form=form)
+ finally:
+ session.commit()
+ session.close()
class LoginForm(Form):
username = StringField('Username', [InputRequired()])
password = PasswordField('Password', [InputRequired()])
+
+
+def _unauthorized():
+ """
+ Indicate that authorization is required
+ :return:
+ """
+ return Response("Unauthorized", 401, {"WWW-Authenticate": "Basic"})
+
+
+def _forbidden():
+ return Response("Forbidden", 403)
+
+
+def init_app(app):
+ pass
+
+
+def requires_authentication(function):
+ @wraps(function)
+ def decorated(*args, **kwargs):
+ from flask import request
+
+ header = request.headers.get("Authorization")
+ if header:
+ userpass = ''.join(header.split()[1:])
+ username, password = base64.b64decode(userpass).decode("utf-8").split(":", 1)
+
+ session = settings.Session()
+ try:
+ authenticate(session, username, password)
+
+ response = function(*args, **kwargs)
+ response = make_response(response)
+ return response
+
+ except AuthenticationError:
+ return _forbidden()
+
+ finally:
+ session.commit()
+ session.close()
+ return _unauthorized()
+ return decorated
diff --git a/airflow/contrib/example_dags/__init__.py b/airflow/contrib/example_dags/__init__.py
new file mode 100644
index 0000000000..114d189da1
--- /dev/null
+++ b/airflow/contrib/example_dags/__init__.py
@@ -0,0 +1,18 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/contrib/example_dags/example_azure_container_instances_operator.py b/airflow/contrib/example_dags/example_azure_container_instances_operator.py
new file mode 100644
index 0000000000..181a30b50e
--- /dev/null
+++ b/airflow/contrib/example_dags/example_azure_container_instances_operator.py
@@ -0,0 +1,54 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow import DAG
+from airflow.contrib.operators.azure_container_instances_operator import AzureContainerInstancesOperator
+from datetime import datetime, timedelta
+
+default_args = {
+ 'owner': 'airflow',
+ 'depends_on_past': False,
+ 'start_date': datetime(2018, 11, 1),
+ 'email': ['airflow@example.com'],
+ 'email_on_failure': False,
+ 'email_on_retry': False,
+ 'retries': 1,
+ 'retry_delay': timedelta(minutes=5),
+}
+
+dag = DAG(
+ 'aci_example',
+ default_args=default_args,
+ schedule_interval=timedelta(1)
+)
+
+t1 = AzureContainerInstancesOperator(
+ ci_conn_id='azure_container_instances_default',
+ registry_conn_id=None,
+ resource_group='resource-group',
+ name='aci-test-{{ ds }}',
+ image='hello-world',
+ region='WestUS2',
+ environment_variables={},
+ volumes=[],
+ memory_in_gb=4.0,
+ cpu=1.0,
+ task_id='start_container',
+ dag=dag
+)
diff --git a/airflow/contrib/example_dags/example_azure_cosmosdb_sensor.py b/airflow/contrib/example_dags/example_azure_cosmosdb_sensor.py
new file mode 100644
index 0000000000..dd0b83e811
--- /dev/null
+++ b/airflow/contrib/example_dags/example_azure_cosmosdb_sensor.py
@@ -0,0 +1,64 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+This is only an example DAG to highlight usage of AzureCosmosDocumentSensor to detect
+if a document now exists.
+
+You can trigger this manually with `airflow trigger_dag example_cosmosdb_sensor`.
+
+*Note: Make sure that connection `azure_cosmos_default` is properly set before running
+this example.*
+"""
+
+from airflow import DAG
+from airflow.contrib.sensors.azure_cosmos_sensor import AzureCosmosDocumentSensor
+from airflow.contrib.operators.azure_cosmos_operator import AzureCosmosInsertDocumentOperator
+from airflow.utils import dates
+
+default_args = {
+ 'owner': 'airflow',
+ 'depends_on_past': False,
+ 'start_date': dates.days_ago(2),
+ 'email': ['airflow@example.com'],
+ 'email_on_failure': False,
+ 'email_on_retry': False
+}
+
+dag = DAG('example_azure_cosmosdb_sensor', default_args=default_args)
+
+dag.doc_md = __doc__
+
+t1 = AzureCosmosDocumentSensor(
+ task_id='check_cosmos_file',
+ database_name='airflow_example_db',
+ collection_name='airflow_example_coll',
+ document_id='airflow_checkid',
+ azure_cosmos_conn_id='azure_cosmos_default',
+ dag=dag)
+
+t2 = AzureCosmosInsertDocumentOperator(
+ task_id='insert_cosmos_file',
+ dag=dag,
+ database_name='airflow_example_db',
+ collection_name='new-collection',
+ document={"id": "someuniqueid", "param1": "value1", "param2": "value2"},
+ azure_cosmos_conn_id='azure_cosmos_default')
+
+t1 >> t2
diff --git a/airflow/contrib/example_dags/example_databricks_operator.py b/airflow/contrib/example_dags/example_databricks_operator.py
index abf6844d90..79f947ba1c 100644
--- a/airflow/contrib/example_dags/example_databricks_operator.py
+++ b/airflow/contrib/example_dags/example_databricks_operator.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
import airflow
@@ -27,7 +32,7 @@
# the spark jar task will NOT run until the notebook task completes
# successfully.
#
-# The definition of a succesful run is if the run has a result_state of "SUCCESS".
+# The definition of a successful run is if the run has a result_state of "SUCCESS".
# For more information about the state of a run refer to
# https://docs.databricks.com/api/latest/jobs.html#runstate
diff --git a/airflow/contrib/example_dags/example_emr_job_flow_automatic_steps.py b/airflow/contrib/example_dags/example_emr_job_flow_automatic_steps.py
index b03b36fb14..098a7a04ea 100644
--- a/airflow/contrib/example_dags/example_emr_job_flow_automatic_steps.py
+++ b/airflow/contrib/example_dags/example_emr_job_flow_automatic_steps.py
@@ -1,21 +1,27 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from datetime import timedelta
import airflow
from airflow import DAG
-from airflow.contrib.operators.emr_create_job_flow_operator import EmrCreateJobFlowOperator
+from airflow.contrib.operators.emr_create_job_flow_operator \
+ import EmrCreateJobFlowOperator
from airflow.contrib.sensors.emr_job_flow_sensor import EmrJobFlowSensor
DEFAULT_ARGS = {
diff --git a/airflow/contrib/example_dags/example_emr_job_flow_manual_steps.py b/airflow/contrib/example_dags/example_emr_job_flow_manual_steps.py
index d123be0b8e..48a178a2a9 100644
--- a/airflow/contrib/example_dags/example_emr_job_flow_manual_steps.py
+++ b/airflow/contrib/example_dags/example_emr_job_flow_manual_steps.py
@@ -1,25 +1,33 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from datetime import timedelta
import airflow
from airflow import DAG
-from airflow.contrib.operators.emr_create_job_flow_operator import EmrCreateJobFlowOperator
-from airflow.contrib.operators.emr_add_steps_operator import EmrAddStepsOperator
+from airflow.contrib.operators.emr_create_job_flow_operator \
+ import EmrCreateJobFlowOperator
+from airflow.contrib.operators.emr_add_steps_operator \
+ import EmrAddStepsOperator
from airflow.contrib.sensors.emr_step_sensor import EmrStepSensor
-from airflow.contrib.operators.emr_terminate_job_flow_operator import EmrTerminateJobFlowOperator
+from airflow.contrib.operators.emr_terminate_job_flow_operator \
+ import EmrTerminateJobFlowOperator
DEFAULT_ARGS = {
'owner': 'airflow',
diff --git a/airflow/contrib/example_dags/example_gcp_compute.py b/airflow/contrib/example_dags/example_gcp_compute.py
new file mode 100644
index 0000000000..928e9744b6
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_compute.py
@@ -0,0 +1,112 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that starts, stops and sets the machine type of a Google Compute
+Engine instance.
+
+This DAG relies on the following OS environment variables
+
+* PROJECT_ID - Google Cloud Platform project where the Compute Engine instance exists.
+* ZONE - Google Cloud Platform zone where the instance exists.
+* INSTANCE - Name of the Compute Engine instance.
+* SHORT_MACHINE_TYPE_NAME - Machine type resource name to set, e.g. 'n1-standard-1'.
+ See https://cloud.google.com/compute/docs/machine-types
+"""
+import os
+import datetime
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcp_compute_operator import GceInstanceStartOperator, \
+ GceInstanceStopOperator, GceSetMachineTypeOperator
+
+# [START howto_operator_gce_args_common]
+PROJECT_ID = os.environ.get('PROJECT_ID', 'example-project')
+ZONE = os.environ.get('ZONE', 'europe-west1-b')
+INSTANCE = os.environ.get('INSTANCE', 'testinstance')
+# [END howto_operator_gce_args_common]
+
+default_args = {
+ 'start_date': airflow.utils.dates.days_ago(1)
+}
+
+# [START howto_operator_gce_args_set_machine_type]
+SHORT_MACHINE_TYPE_NAME = os.environ.get('SHORT_MACHINE_TYPE_NAME', 'n1-standard-1')
+SET_MACHINE_TYPE_BODY = {
+ 'machineType': 'zones/{}/machineTypes/{}'.format(ZONE, SHORT_MACHINE_TYPE_NAME)
+}
+# [END howto_operator_gce_args_set_machine_type]
+
+
+with models.DAG(
+ 'example_gcp_compute',
+ default_args=default_args,
+ schedule_interval=datetime.timedelta(days=1)
+) as dag:
+ # [START howto_operator_gce_start]
+ gce_instance_start = GceInstanceStartOperator(
+ project_id=PROJECT_ID,
+ zone=ZONE,
+ resource_id=INSTANCE,
+ task_id='gcp_compute_start_task'
+ )
+ # [END howto_operator_gce_start]
+ # Duplicate start for idempotence testing
+ gce_instance_start2 = GceInstanceStartOperator(
+ project_id=PROJECT_ID,
+ zone=ZONE,
+ resource_id=INSTANCE,
+ task_id='gcp_compute_start_task2'
+ )
+ # [START howto_operator_gce_stop]
+ gce_instance_stop = GceInstanceStopOperator(
+ project_id=PROJECT_ID,
+ zone=ZONE,
+ resource_id=INSTANCE,
+ task_id='gcp_compute_stop_task'
+ )
+ # [END howto_operator_gce_stop]
+ # Duplicate stop for idempotence testing
+ gce_instance_stop2 = GceInstanceStopOperator(
+ project_id=PROJECT_ID,
+ zone=ZONE,
+ resource_id=INSTANCE,
+ task_id='gcp_compute_stop_task2'
+ )
+ # [START howto_operator_gce_set_machine_type]
+ gce_set_machine_type = GceSetMachineTypeOperator(
+ project_id=PROJECT_ID,
+ zone=ZONE,
+ resource_id=INSTANCE,
+ body=SET_MACHINE_TYPE_BODY,
+ task_id='gcp_compute_set_machine_type'
+ )
+ # [END howto_operator_gce_set_machine_type]
+ # Duplicate set machine type for idempotence testing
+ gce_set_machine_type2 = GceSetMachineTypeOperator(
+ project_id=PROJECT_ID,
+ zone=ZONE,
+ resource_id=INSTANCE,
+ body=SET_MACHINE_TYPE_BODY,
+ task_id='gcp_compute_set_machine_type2'
+ )
+
+ gce_instance_start >> gce_instance_start2 >> gce_instance_stop >> \
+ gce_instance_stop2 >> gce_set_machine_type >> gce_set_machine_type2
diff --git a/airflow/contrib/example_dags/example_gcp_compute_igm.py b/airflow/contrib/example_dags/example_gcp_compute_igm.py
new file mode 100644
index 0000000000..3e4543c60d
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_compute_igm.py
@@ -0,0 +1,143 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that uses IGM-type compute operations:
+* copy of Instance Template
+* update template in Instance Group Manager
+
+This DAG relies on the following OS environment variables
+
+* PROJECT_ID - the Google Cloud Platform project where the Compute Engine instance exists
+* ZONE - the zone where the Compute Engine instance exists
+
+Variables for copy template operator:
+* TEMPLATE_NAME - name of the template to copy
+* NEW_TEMPLATE_NAME - name of the new template
+* NEW_DESCRIPTION - description added to the template
+
+Variables for update template in Group Manager:
+
+* INSTANCE_GROUP_MANAGER_NAME - name of the Instance Group Manager
+* SOURCE_TEMPLATE_URL - url of the template to replace in the Instance Group Manager
+* DESTINATION_TEMPLATE_URL - url of the new template to set in the Instance Group Manager
+"""
+
+import os
+import datetime
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcp_compute_operator import \
+ GceInstanceTemplateCopyOperator, GceInstanceGroupManagerUpdateTemplateOperator
+
+# [START howto_operator_compute_igm_common_args]
+PROJECT_ID = os.environ.get('PROJECT_ID', 'example-project')
+ZONE = os.environ.get('ZONE', 'europe-west1-b')
+# [END howto_operator_compute_igm_common_args]
+
+default_args = {
+ 'start_date': airflow.utils.dates.days_ago(1)
+}
+
+# [START howto_operator_compute_template_copy_args]
+TEMPLATE_NAME = os.environ.get('TEMPLATE_NAME', 'instance-template-test')
+NEW_TEMPLATE_NAME = os.environ.get('NEW_TEMPLATE_NAME',
+ 'instance-template-test-new')
+NEW_DESCRIPTION = os.environ.get('NEW_DESCRIPTION', 'Test new description')
+GCE_INSTANCE_TEMPLATE_BODY_UPDATE = {
+ "name": NEW_TEMPLATE_NAME,
+ "description": NEW_DESCRIPTION,
+ "properties": {
+ "machineType": "n1-standard-2"
+ }
+}
+# [END howto_operator_compute_template_copy_args]
+
+# [START howto_operator_compute_igm_update_template_args]
+INSTANCE_GROUP_MANAGER_NAME = os.environ.get('INSTANCE_GROUP_MANAGER_NAME',
+ 'instance-group-test')
+
+SOURCE_TEMPLATE_URL = os.environ.get(
+ 'SOURCE_TEMPLATE_URL',
+ "https://www.googleapis.com/compute/beta/projects/"
+ "example-project/global/instanceTemplates/instance-template-test")
+
+DESTINATION_TEMPLATE_URL = os.environ.get(
+ 'DESTINATION_TEMPLATE_URL',
+ "https://www.googleapis.com/compute/beta/projects/"
+ "example-airflow/global/instanceTemplates/" + NEW_TEMPLATE_NAME)
+
+UPDATE_POLICY = {
+ "type": "OPPORTUNISTIC",
+ "minimalAction": "RESTART",
+ "maxSurge": {
+ "fixed": 1
+ },
+ "minReadySec": 1800
+}
+
+# [END howto_operator_compute_igm_update_template_args]
+
+
+with models.DAG(
+ 'example_gcp_compute_igm',
+ default_args=default_args,
+ schedule_interval=datetime.timedelta(days=1)
+) as dag:
+ # [START howto_operator_gce_igm_copy_template]
+ gce_instance_template_copy = GceInstanceTemplateCopyOperator(
+ project_id=PROJECT_ID,
+ resource_id=TEMPLATE_NAME,
+ body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
+ task_id='gcp_compute_igm_copy_template_task'
+ )
+ # [END howto_operator_gce_igm_copy_template]
+ # Added to check for idempotence
+ gce_instance_template_copy2 = GceInstanceTemplateCopyOperator(
+ project_id=PROJECT_ID,
+ resource_id=TEMPLATE_NAME,
+ body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
+ task_id='gcp_compute_igm_copy_template_task_2'
+ )
+ # [START howto_operator_gce_igm_update_template]
+ gce_instance_group_manager_update_template = \
+ GceInstanceGroupManagerUpdateTemplateOperator(
+ project_id=PROJECT_ID,
+ resource_id=INSTANCE_GROUP_MANAGER_NAME,
+ zone=ZONE,
+ source_template=SOURCE_TEMPLATE_URL,
+ destination_template=DESTINATION_TEMPLATE_URL,
+ update_policy=UPDATE_POLICY,
+ task_id='gcp_compute_igm_group_manager_update_template'
+ )
+ # [END howto_operator_gce_igm_update_template]
+ # Added to check for idempotence (and without UPDATE_POLICY)
+ gce_instance_group_manager_update_template2 = \
+ GceInstanceGroupManagerUpdateTemplateOperator(
+ project_id=PROJECT_ID,
+ resource_id=INSTANCE_GROUP_MANAGER_NAME,
+ zone=ZONE,
+ source_template=SOURCE_TEMPLATE_URL,
+ destination_template=DESTINATION_TEMPLATE_URL,
+ task_id='gcp_compute_igm_group_manager_update_template_2'
+ )
+ gce_instance_template_copy >> gce_instance_template_copy2 >> \
+ gce_instance_group_manager_update_template >> \
+ gce_instance_group_manager_update_template2
diff --git a/airflow/contrib/example_dags/example_gcp_function_delete.py b/airflow/contrib/example_dags/example_gcp_function_delete.py
new file mode 100644
index 0000000000..642e3a744c
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_function_delete.py
@@ -0,0 +1,59 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that deletes a Google Cloud Function.
+This DAG relies on the following OS environment variables
+* PROJECT_ID - Google Cloud Project where the Cloud Function exists.
+* LOCATION - Google Cloud Functions region where the function exists.
+* ENTRYPOINT - Name of the executable function in the source code.
+"""
+
+import os
+import datetime
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcp_function_operator import GcfFunctionDeleteOperator
+
+# [START howto_operator_gcf_delete_args]
+PROJECT_ID = os.environ.get('PROJECT_ID', 'example-project')
+LOCATION = os.environ.get('LOCATION', 'europe-west1')
+ENTRYPOINT = os.environ.get('ENTRYPOINT', 'helloWorld')
+# A fully-qualified name of the function to delete
+
+FUNCTION_NAME = 'projects/{}/locations/{}/functions/{}'.format(PROJECT_ID, LOCATION,
+ ENTRYPOINT)
+# [END howto_operator_gcf_delete_args]
+
+default_args = {
+ 'start_date': airflow.utils.dates.days_ago(1)
+}
+
+with models.DAG(
+ 'example_gcp_function_delete',
+ default_args=default_args,
+ schedule_interval=datetime.timedelta(days=1)
+) as dag:
+ # [START howto_operator_gcf_delete]
+ t1 = GcfFunctionDeleteOperator(
+ task_id="gcf_delete_task",
+ name=FUNCTION_NAME
+ )
+ # [END howto_operator_gcf_delete]
diff --git a/airflow/contrib/example_dags/example_gcp_function_deploy_delete.py b/airflow/contrib/example_dags/example_gcp_function_deploy_delete.py
new file mode 100644
index 0000000000..76563d7596
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_function_deploy_delete.py
@@ -0,0 +1,117 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that creates a Google Cloud Function and then deletes it.
+
+This DAG relies on the following OS environment variables
+https://airflow.apache.org/concepts.html#variables
+* PROJECT_ID - Google Cloud Project to use for the Cloud Function.
+* LOCATION - Google Cloud Functions region where the function should be
+ created.
+* SOURCE_ARCHIVE_URL - Path to the zipped source in Google Cloud Storage
+or
+ * SOURCE_UPLOAD_URL - Generated upload URL for the zipped source
+ or
+ * ZIP_PATH - Local path to the zipped source archive
+or
+* SOURCE_REPOSITORY - The URL pointing to the hosted repository where the function is
+defined in a supported Cloud Source Repository URL format
+https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions#SourceRepository
+* ENTRYPOINT - Name of the executable function in the source code.
+"""
+
+import os
+import datetime
+
+from airflow import models
+from airflow.contrib.operators.gcp_function_operator \
+ import GcfFunctionDeployOperator, GcfFunctionDeleteOperator
+from airflow.utils import dates
+
+# [START howto_operator_gcf_deploy_variables]
+PROJECT_ID = os.environ.get('PROJECT_ID', 'example-project')
+LOCATION = os.environ.get('LOCATION', 'europe-west1')
+SOURCE_ARCHIVE_URL = os.environ.get('SOURCE_ARCHIVE_URL', '')
+SOURCE_UPLOAD_URL = os.environ.get('SOURCE_UPLOAD_URL', '')
+SOURCE_REPOSITORY = os.environ.get(
+ 'SOURCE_REPOSITORY',
+ 'https://source.developers.google.com/'
+ 'projects/example-project/repos/hello-world/moveable-aliases/master')
+ZIP_PATH = os.environ.get('ZIP_PATH', '')
+ENTRYPOINT = os.environ.get('ENTRYPOINT', 'helloWorld')
+FUNCTION_NAME = 'projects/{}/locations/{}/functions/{}'.format(PROJECT_ID, LOCATION,
+ ENTRYPOINT)
+RUNTIME = 'nodejs6'
+VALIDATE_BODY = os.environ.get('VALIDATE_BODY', True)
+
+# [END howto_operator_gcf_deploy_variables]
+
+# [START howto_operator_gcf_deploy_body]
+body = {
+ "name": FUNCTION_NAME,
+ "entryPoint": ENTRYPOINT,
+ "runtime": RUNTIME,
+ "httpsTrigger": {}
+}
+# [END howto_operator_gcf_deploy_body]
+
+# [START howto_operator_gcf_default_args]
+default_args = {
+ 'start_date': dates.days_ago(1)
+}
+# [END howto_operator_gcf_default_args]
+
+# [START howto_operator_gcf_deploy_variants]
+if SOURCE_ARCHIVE_URL:
+ body['sourceArchiveUrl'] = SOURCE_ARCHIVE_URL
+elif SOURCE_REPOSITORY:
+ body['sourceRepository'] = {
+ 'url': SOURCE_REPOSITORY
+ }
+elif ZIP_PATH:
+ body['sourceUploadUrl'] = ''
+ default_args['zip_path'] = ZIP_PATH
+elif SOURCE_UPLOAD_URL:
+ body['sourceUploadUrl'] = SOURCE_UPLOAD_URL
+else:
+ raise Exception("Please provide one of the source_code parameters")
+# [END howto_operator_gcf_deploy_variants]
+
+
+with models.DAG(
+ 'example_gcp_function_deploy_delete',
+ default_args=default_args,
+ schedule_interval=datetime.timedelta(days=1)
+) as dag:
+ # [START howto_operator_gcf_deploy]
+ deploy_task = GcfFunctionDeployOperator(
+ task_id="gcf_deploy_task",
+ name=FUNCTION_NAME,
+ project_id=PROJECT_ID,
+ location=LOCATION,
+ body=body,
+ validate_body=VALIDATE_BODY
+ )
+ # [END howto_operator_gcf_deploy]
+ delete_task = GcfFunctionDeleteOperator(
+ task_id="gcf_delete_task",
+ name=FUNCTION_NAME
+ )
+ deploy_task >> delete_task
diff --git a/airflow/contrib/example_dags/example_gcp_spanner.py b/airflow/contrib/example_dags/example_gcp_spanner.py
new file mode 100644
index 0000000000..cec3dcb855
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_spanner.py
@@ -0,0 +1,113 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that creates, updates, queries and deletes a Cloud Spanner instance.
+
+This DAG relies on the following environment variables
+* SPANNER_PROJECT_ID - Google Cloud Platform project for the Cloud Spanner instance.
+* SPANNER_INSTANCE_ID - Cloud Spanner instance ID.
+* SPANNER_CONFIG_NAME - The name of the instance's configuration. Values are of the form
+ projects/<project>/instanceConfigs/<configuration>.
+ See also:
+ https://cloud.google.com/spanner/docs/reference/rest/v1/projects.instanceConfigs#InstanceConfig
+ https://cloud.google.com/spanner/docs/reference/rest/v1/projects.instanceConfigs/list#google.spanner.admin.instance.v1.InstanceAdmin.ListInstanceConfigs
+* SPANNER_NODE_COUNT - Number of nodes allocated to the instance.
+* SPANNER_DISPLAY_NAME - The descriptive name for this instance as it appears in UIs.
+ Must be unique per project and between 4 and 30 characters in length.
+"""
+
+import os
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcp_spanner_operator import \
+ CloudSpannerInstanceDeployOperator, CloudSpannerInstanceDatabaseQueryOperator, \
+ CloudSpannerInstanceDeleteOperator
+
+# [START howto_operator_spanner_arguments]
+PROJECT_ID = os.environ.get('SPANNER_PROJECT_ID', 'example-project')
+INSTANCE_ID = os.environ.get('SPANNER_INSTANCE_ID', 'testinstance')
+DB_ID = os.environ.get('SPANNER_DB_ID', 'db1')
+CONFIG_NAME = os.environ.get('SPANNER_CONFIG_NAME',
+ 'projects/example-project/instanceConfigs/eur3')
+NODE_COUNT = os.environ.get('SPANNER_NODE_COUNT', '1')
+DISPLAY_NAME = os.environ.get('SPANNER_DISPLAY_NAME', 'Test Instance')
+# [END howto_operator_spanner_arguments]
+
+default_args = {
+ 'start_date': airflow.utils.dates.days_ago(1)
+}
+
+with models.DAG(
+ 'example_gcp_spanner',
+ default_args=default_args,
+ schedule_interval=None # Override to match your needs
+) as dag:
+ # Create
+ # [START howto_operator_spanner_deploy]
+ spanner_instance_create_task = CloudSpannerInstanceDeployOperator(
+ project_id=PROJECT_ID,
+ instance_id=INSTANCE_ID,
+ configuration_name=CONFIG_NAME,
+ node_count=int(NODE_COUNT),
+ display_name=DISPLAY_NAME,
+ task_id='spanner_instance_create_task'
+ )
+ # [END howto_operator_spanner_deploy]
+
+ # Update
+ spanner_instance_update_task = CloudSpannerInstanceDeployOperator(
+ project_id=PROJECT_ID,
+ instance_id=INSTANCE_ID,
+ configuration_name=CONFIG_NAME,
+ node_count=int(NODE_COUNT) + 1,
+ display_name=DISPLAY_NAME + '_updated',
+ task_id='spanner_instance_update_task'
+ )
+
+ # [START howto_operator_spanner_query]
+ spanner_instance_query = CloudSpannerInstanceDatabaseQueryOperator(
+ project_id=PROJECT_ID,
+ instance_id=INSTANCE_ID,
+ database_id='db1',
+ query="DELETE FROM my_table2 WHERE true",
+ task_id='spanner_instance_query'
+ )
+ # [END howto_operator_spanner_query]
+
+ spanner_instance_query2 = CloudSpannerInstanceDatabaseQueryOperator(
+ project_id=PROJECT_ID,
+ instance_id=INSTANCE_ID,
+ database_id='db1',
+ query="example_gcp_spanner.sql",
+ task_id='spanner_instance_query2'
+ )
+
+ # [START howto_operator_spanner_delete]
+ spanner_instance_delete_task = CloudSpannerInstanceDeleteOperator(
+ project_id=PROJECT_ID,
+ instance_id=INSTANCE_ID,
+ task_id='spanner_instance_delete_task'
+ )
+ # [END howto_operator_spanner_delete]
+
+ spanner_instance_create_task >> spanner_instance_update_task \
+ >> spanner_instance_query >> spanner_instance_query2 \
+ >> spanner_instance_delete_task
diff --git a/airflow/contrib/example_dags/example_gcp_spanner.sql b/airflow/contrib/example_dags/example_gcp_spanner.sql
new file mode 100644
index 0000000000..5d5f238022
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_spanner.sql
@@ -0,0 +1,3 @@
+INSERT my_table2 (id, name) VALUES (7, 'Seven');
+INSERT my_table2 (id, name)
+ VALUES (8, 'Eight');
diff --git a/airflow/contrib/example_dags/example_gcp_sql.py b/airflow/contrib/example_dags/example_gcp_sql.py
new file mode 100644
index 0000000000..c6838a2baf
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_sql.py
@@ -0,0 +1,315 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that creates, patches and deletes a Cloud SQL instance, and also
+creates, patches and deletes a database inside the instance, in Google Cloud Platform.
+
+This DAG relies on the following OS environment variables
+https://airflow.apache.org/concepts.html#variables
+* PROJECT_ID - Google Cloud Platform project for the Cloud SQL instance.
+* INSTANCE_NAME - Name of the Cloud SQL instance.
+* DB_NAME - Name of the database inside a Cloud SQL instance.
+"""
+
+import os
+
+import re
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcp_sql_operator import CloudSqlInstanceCreateOperator, \
+ CloudSqlInstancePatchOperator, CloudSqlInstanceDeleteOperator, \
+ CloudSqlInstanceDatabaseCreateOperator, CloudSqlInstanceDatabasePatchOperator, \
+ CloudSqlInstanceDatabaseDeleteOperator, CloudSqlInstanceExportOperator, \
+ CloudSqlInstanceImportOperator
+from airflow.contrib.operators.gcs_acl_operator import \
+ GoogleCloudStorageBucketCreateAclEntryOperator, \
+ GoogleCloudStorageObjectCreateAclEntryOperator
+
+# [START howto_operator_cloudsql_arguments]
+PROJECT_ID = os.environ.get('PROJECT_ID', 'example-project')
+INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'test-mysql')
+INSTANCE_NAME2 = os.environ.get('INSTANCE_NAME2', 'test-mysql2')
+DB_NAME = os.environ.get('DB_NAME', 'testdb')
+# [END howto_operator_cloudsql_arguments]
+
+# [START howto_operator_cloudsql_export_import_arguments]
+EXPORT_URI = os.environ.get('EXPORT_URI', 'gs://bucketName/fileName')
+IMPORT_URI = os.environ.get('IMPORT_URI', 'gs://bucketName/fileName')
+# [END howto_operator_cloudsql_export_import_arguments]
+
+# Bodies below represent Cloud SQL instance resources:
+# https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances
+
+# [START howto_operator_cloudsql_create_body]
+body = {
+ "name": INSTANCE_NAME,
+ "settings": {
+ "tier": "db-n1-standard-1",
+ "backupConfiguration": {
+ "binaryLogEnabled": True,
+ "enabled": True,
+ "startTime": "05:00"
+ },
+ "activationPolicy": "ALWAYS",
+ "dataDiskSizeGb": 30,
+ "dataDiskType": "PD_SSD",
+ "databaseFlags": [],
+ "ipConfiguration": {
+ "ipv4Enabled": True,
+ "requireSsl": True,
+ },
+ "locationPreference": {
+ "zone": "europe-west4-a"
+ },
+ "maintenanceWindow": {
+ "hour": 5,
+ "day": 7,
+ "updateTrack": "canary"
+ },
+ "pricingPlan": "PER_USE",
+ "replicationType": "ASYNCHRONOUS",
+ "storageAutoResize": False,
+ "storageAutoResizeLimit": 0,
+ "userLabels": {
+ "my-key": "my-value"
+ }
+ },
+ "databaseVersion": "MYSQL_5_7",
+ "region": "europe-west4",
+}
+# [END howto_operator_cloudsql_create_body]
+
+body2 = {
+ "name": INSTANCE_NAME2,
+ "settings": {
+ "tier": "db-n1-standard-1",
+ },
+ "databaseVersion": "MYSQL_5_7",
+ "region": "europe-west4",
+}
+
+# [START howto_operator_cloudsql_patch_body]
+patch_body = {
+ "name": INSTANCE_NAME,
+ "settings": {
+ "dataDiskSizeGb": 35,
+ "maintenanceWindow": {
+ "hour": 3,
+ "day": 6,
+ "updateTrack": "canary"
+ },
+ "userLabels": {
+ "my-key-patch": "my-value-patch"
+ }
+ }
+}
+# [END howto_operator_cloudsql_patch_body]
+# [START howto_operator_cloudsql_export_body]
+export_body = {
+ "exportContext": {
+ "fileType": "sql",
+ "uri": EXPORT_URI,
+ "sqlExportOptions": {
+ "schemaOnly": False
+ }
+ }
+}
+# [END howto_operator_cloudsql_export_body]
+# [START howto_operator_cloudsql_import_body]
+import_body = {
+ "importContext": {
+ "fileType": "sql",
+ "uri": IMPORT_URI
+ }
+}
+# [END howto_operator_cloudsql_import_body]
+# [START howto_operator_cloudsql_db_create_body]
+db_create_body = {
+ "instance": INSTANCE_NAME,
+ "name": DB_NAME,
+ "project": PROJECT_ID
+}
+# [END howto_operator_cloudsql_db_create_body]
+# [START howto_operator_cloudsql_db_patch_body]
+db_patch_body = {
+ "charset": "utf16",
+ "collation": "utf16_general_ci"
+}
+# [END howto_operator_cloudsql_db_patch_body]
+
+default_args = {
+ 'start_date': airflow.utils.dates.days_ago(1)
+}
+
+with models.DAG(
+ 'example_gcp_sql',
+ default_args=default_args,
+ schedule_interval=None
+) as dag:
+ prev_task = None
+
+ def next_dep(task, prev):
+ prev >> task
+ return task
+
+ # ############################################## #
+ # ### INSTANCES SET UP ######################### #
+ # ############################################## #
+
+ # [START howto_operator_cloudsql_create]
+ sql_instance_create = CloudSqlInstanceCreateOperator(
+ project_id=PROJECT_ID,
+ body=body,
+ instance=INSTANCE_NAME,
+ task_id='sql_instance_create'
+ )
+ # [END howto_operator_cloudsql_create]
+ prev_task = sql_instance_create
+
+ sql_instance_create_2 = CloudSqlInstanceCreateOperator(
+ project_id=PROJECT_ID,
+ body=body2,
+ instance=INSTANCE_NAME2,
+ task_id='sql_instance_create_2'
+ )
+ prev_task = next_dep(sql_instance_create_2, prev_task)
+
+ # ############################################## #
+ # ### MODIFYING INSTANCE AND ITS DATABASE ###### #
+ # ############################################## #
+
+ # [START howto_operator_cloudsql_patch]
+ sql_instance_patch_task = CloudSqlInstancePatchOperator(
+ project_id=PROJECT_ID,
+ body=patch_body,
+ instance=INSTANCE_NAME,
+ task_id='sql_instance_patch_task'
+ )
+ # [END howto_operator_cloudsql_patch]
+ prev_task = next_dep(sql_instance_patch_task, prev_task)
+
+ # [START howto_operator_cloudsql_db_create]
+ sql_db_create_task = CloudSqlInstanceDatabaseCreateOperator(
+ project_id=PROJECT_ID,
+ body=db_create_body,
+ instance=INSTANCE_NAME,
+ task_id='sql_db_create_task'
+ )
+ # [END howto_operator_cloudsql_db_create]
+ prev_task = next_dep(sql_db_create_task, prev_task)
+
+ # [START howto_operator_cloudsql_db_patch]
+ sql_db_patch_task = CloudSqlInstanceDatabasePatchOperator(
+ project_id=PROJECT_ID,
+ body=db_patch_body,
+ instance=INSTANCE_NAME,
+ database=DB_NAME,
+ task_id='sql_db_patch_task'
+ )
+ # [END howto_operator_cloudsql_db_patch]
+ prev_task = next_dep(sql_db_patch_task, prev_task)
+
+ # ############################################## #
+ # ### EXPORTING SQL FROM INSTANCE 1 ############ #
+ # ############################################## #
+
+ # For export to work we need to add the Cloud SQL instance's Service Account
+ # write access to the destination GCS bucket.
+ # [START howto_operator_cloudsql_export_gcs_permissions]
+ sql_gcp_add_bucket_permission = GoogleCloudStorageBucketCreateAclEntryOperator(
+ entity="user-{{ task_instance.xcom_pull('sql_instance_create', key='service_account_email') }}",
+ role="WRITER",
+ bucket=re.match(r'gs:\/\/(\S*)\/', EXPORT_URI).group(1),
+ task_id='sql_gcp_add_bucket_permission'
+ )
+ # [END howto_operator_cloudsql_export_gcs_permissions]
+ prev_task = next_dep(sql_gcp_add_bucket_permission, prev_task)
+
+ # [START howto_operator_cloudsql_export]
+ sql_export_task = CloudSqlInstanceExportOperator(
+ project_id=PROJECT_ID,
+ body=export_body,
+ instance=INSTANCE_NAME,
+ task_id='sql_export_task'
+ )
+ # [END howto_operator_cloudsql_export]
+ prev_task = next_dep(sql_export_task, prev_task)
+
+ # ############################################## #
+ # ### IMPORTING SQL TO INSTANCE 2 ############## #
+ # ############################################## #
+
+ # For import to work we need to add the Cloud SQL instance's Service Account
+ # read access to the target GCS object.
+ # [START howto_operator_cloudsql_import_gcs_permissions]
+ sql_gcp_add_object_permission = GoogleCloudStorageObjectCreateAclEntryOperator(
+ entity="user-{{ task_instance.xcom_pull('sql_instance_create_2', key='service_account_email') }}",
+ role="READER",
+ bucket=re.match(r'gs:\/\/(\S*)\/', IMPORT_URI).group(1),
+ object_name=re.match(r'gs:\/\/[^\/]*\/(\S*)', IMPORT_URI).group(1),
+ task_id='sql_gcp_add_object_permission',
+ )
+ # [END howto_operator_cloudsql_import_gcs_permissions]
+ prev_task = next_dep(sql_gcp_add_object_permission, prev_task)
+
+ # [START howto_operator_cloudsql_import]
+ sql_import_task = CloudSqlInstanceImportOperator(
+ project_id=PROJECT_ID,
+ body=import_body,
+ instance=INSTANCE_NAME2,
+ task_id='sql_import_task'
+ )
+ # [END howto_operator_cloudsql_import]
+ prev_task = next_dep(sql_import_task, prev_task)
+
+ # ############################################## #
+ # ### DELETING A DATABASE FROM AN INSTANCE ##### #
+ # ############################################## #
+
+ # [START howto_operator_cloudsql_db_delete]
+ sql_db_delete_task = CloudSqlInstanceDatabaseDeleteOperator(
+ project_id=PROJECT_ID,
+ instance=INSTANCE_NAME,
+ database=DB_NAME,
+ task_id='sql_db_delete_task'
+ )
+ # [END howto_operator_cloudsql_db_delete]
+ prev_task = next_dep(sql_db_delete_task, prev_task)
+
+ # ############################################## #
+ # ### INSTANCES TEAR DOWN ###################### #
+ # ############################################## #
+
+ # [START howto_operator_cloudsql_delete]
+ sql_instance_delete_task = CloudSqlInstanceDeleteOperator(
+ project_id=PROJECT_ID,
+ instance=INSTANCE_NAME,
+ task_id='sql_instance_delete_task'
+ )
+ # [END howto_operator_cloudsql_delete]
+ prev_task = next_dep(sql_instance_delete_task, prev_task)
+
+ sql_instance_delete_task_2 = CloudSqlInstanceDeleteOperator(
+ project_id=PROJECT_ID,
+ instance=INSTANCE_NAME2,
+ task_id='sql_instance_delete_task_2'
+ )
+ prev_task = next_dep(sql_instance_delete_task_2, prev_task)
diff --git a/airflow/contrib/example_dags/example_gcp_sql_query.py b/airflow/contrib/example_dags/example_gcp_sql_query.py
new file mode 100644
index 0000000000..5439fb6afc
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_sql_query.py
@@ -0,0 +1,259 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that performs query in a Cloud SQL instance.
+
+This DAG relies on the following OS environment variables
+
+* PROJECT_ID - Google Cloud Platform project for the Cloud SQL instance
+* LOCATION - Google Cloud location where the database is created
+*
+* POSTGRES_INSTANCE_NAME - Name of the postgres Cloud SQL instance
+* POSTGRES_USER - Name of the postgres database user
+* POSTGRES_PASSWORD - Password of the postgres database user
+* POSTGRES_PROXY_PORT - Local port number for proxy connections for postgres
+* POSTGRES_PUBLIC_IP - Public IP of the Postgres database
+* POSTGRES_PUBLIC_PORT - Port of the postgres database
+*
+* MYSQL_INSTANCE_NAME - Name of the postgres Cloud SQL instance
+* MYSQL_USER - Name of the mysql database user
+* MYSQL_PASSWORD - Password of the mysql database user
+* MYSQL_PROXY_PORT - Local port number for proxy connections for mysql
+* MYSQL_PUBLIC_IP - Public IP of the mysql database
+* MYSQL_PUBLIC_PORT - Port of the mysql database
+"""
+
+import os
+import subprocess
+
+from six.moves.urllib.parse import quote_plus
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcp_sql_operator import CloudSqlQueryOperator
+
+# [START howto_operator_cloudsql_query_arguments]
+
+PROJECT_ID = os.environ.get('PROJECT_ID', 'example-project')
+LOCATION = os.environ.get('REGION', 'europe-west-1')
+
+POSTGRES_INSTANCE_NAME = os.environ.get('POSTGRES_INSTANCE_NAME', 'testpostgres')
+POSTGRES_DATABASE_NAME = os.environ.get('POSTGRES_DATABASE_NAME', 'postgresdb')
+POSTGRES_USER = os.environ.get('POSTGRES_USER', 'postgres_user')
+POSTGRES_PASSWORD = os.environ.get('POSTGRES_PASSWORD', 'password')
+POSTGRES_PUBLIC_IP = os.environ.get('POSTGRES_PUBLIC_IP', '0.0.0.0')
+POSTGRES_PUBLIC_PORT = os.environ.get('POSTGRES_PUBLIC_PORT', 5432)
+POSTGRES_CLIENT_CERT_FILE = os.environ.get('POSTGRES_CLIENT_CERT_FILE',
+ "/tmp/client-cert.pem")
+POSTGRES_CLIENT_KEY_FILE = os.environ.get('POSTGRES_CLIENT_KEY_FILE',
+ "/tmp/client-key.pem")
+POSTGRES_SERVER_CA_FILE = os.environ.get('POSTGRES_SERVER_CA_FILE',
+ "/tmp/server-ca.pem")
+
+MYSQL_INSTANCE_NAME = os.environ.get('MYSQL_INSTANCE_NAME', 'testmysql')
+MYSQL_DATABASE_NAME = os.environ.get('MYSQL_DATABASE_NAME', 'mysqldb')
+MYSQL_USER = os.environ.get('MYSQL_USER', 'mysql_user')
+MYSQL_PASSWORD = os.environ.get('MYSQL_PASSWORD', 'password')
+MYSQL_PUBLIC_IP = os.environ.get('MYSQL_PUBLIC_IP', '0.0.0.0')
+MYSQL_PUBLIC_PORT = os.environ.get('MYSQL_PUBLIC_PORT', 3306)
+MYSQL_CLIENT_CERT_FILE = os.environ.get('MYSQL_CLIENT_CERT_FILE',
+ "/tmp/client-cert.pem")
+MYSQL_CLIENT_KEY_FILE = os.environ.get('MYSQL_CLIENT_KEY_FILE',
+ "/tmp/client-key.pem")
+MYSQL_SERVER_CA_FILE = os.environ.get('MYSQL_SERVER_CA_FILE',
+ "/tmp/server-ca.pem")
+
+SQL = [
+ 'CREATE TABLE IF NOT EXISTS TABLE_TEST (I INTEGER)',
+ 'CREATE TABLE IF NOT EXISTS TABLE_TEST (I INTEGER)', # shows warnings logged
+ 'INSERT INTO TABLE_TEST VALUES (0)',
+ 'CREATE TABLE IF NOT EXISTS TABLE_TEST2 (I INTEGER)',
+ 'DROP TABLE TABLE_TEST',
+ 'DROP TABLE TABLE_TEST2',
+]
+
+# [END howto_operator_cloudsql_query_arguments]
+default_args = {
+ 'start_date': airflow.utils.dates.days_ago(1)
+}
+
+
+# [START howto_operator_cloudsql_query_connections]
+
+postgres_kwargs = dict(
+ user=quote_plus(POSTGRES_USER),
+ password=quote_plus(POSTGRES_PASSWORD),
+ public_port=POSTGRES_PUBLIC_PORT,
+ public_ip=quote_plus(POSTGRES_PUBLIC_IP),
+ project_id=quote_plus(PROJECT_ID),
+ location=quote_plus(LOCATION),
+ instance=quote_plus(POSTGRES_INSTANCE_NAME),
+ database=quote_plus(POSTGRES_DATABASE_NAME),
+ client_cert_file=quote_plus(POSTGRES_CLIENT_CERT_FILE),
+ client_key_file=quote_plus(POSTGRES_CLIENT_KEY_FILE),
+ server_ca_file=quote_plus(POSTGRES_SERVER_CA_FILE)
+)
+
+# The connections below are created using one of the standard approaches - via environment
+# variables named AIRFLOW_CONN_* . The connections can also be created in the database
+# of AIRFLOW (using command line or UI).
+
+# Postgres: connect via proxy over TCP
+os.environ['AIRFLOW_CONN_PROXY_POSTGRES_TCP'] = \
+ "gcpcloudsql://{user}:{password}@{public_ip}:{public_port}/{database}?" \
+ "database_type=postgres&" \
+ "project_id={project_id}&" \
+ "location={location}&" \
+ "instance={instance}&" \
+ "use_proxy=True&" \
+ "sql_proxy_use_tcp=True".format(**postgres_kwargs)
+
+# Postgres: connect via proxy over UNIX socket (specific proxy version)
+os.environ['AIRFLOW_CONN_PROXY_POSTGRES_SOCKET'] = \
+ "gcpcloudsql://{user}:{password}@{public_ip}:{public_port}/{database}?" \
+ "database_type=postgres&" \
+ "project_id={project_id}&" \
+ "location={location}&" \
+ "instance={instance}&" \
+ "use_proxy=True&" \
+ "sql_proxy_version=v1.13&" \
+ "sql_proxy_use_tcp=False".format(**postgres_kwargs)
+
+# Postgres: connect directly via TCP (non-SSL)
+os.environ['AIRFLOW_CONN_PUBLIC_POSTGRES_TCP'] = \
+ "gcpcloudsql://{user}:{password}@{public_ip}:{public_port}/{database}?" \
+ "database_type=postgres&" \
+ "project_id={project_id}&" \
+ "location={location}&" \
+ "instance={instance}&" \
+ "use_proxy=False&" \
+ "use_ssl=False".format(**postgres_kwargs)
+
+# Postgres: connect directly via TCP (SSL)
+os.environ['AIRFLOW_CONN_PUBLIC_POSTGRES_TCP_SSL'] = \
+ "gcpcloudsql://{user}:{password}@{public_ip}:{public_port}/{database}?" \
+ "database_type=postgres&" \
+ "project_id={project_id}&" \
+ "location={location}&" \
+ "instance={instance}&" \
+ "use_proxy=False&" \
+ "use_ssl=True&" \
+ "sslcert={client_cert_file}&" \
+ "sslkey={client_key_file}&" \
+ "sslrootcert={server_ca_file}"\
+ .format(**postgres_kwargs)
+
+mysql_kwargs = dict(
+ user=quote_plus(MYSQL_USER),
+ password=quote_plus(MYSQL_PASSWORD),
+ public_port=MYSQL_PUBLIC_PORT,
+ public_ip=quote_plus(MYSQL_PUBLIC_IP),
+ project_id=quote_plus(PROJECT_ID),
+ location=quote_plus(LOCATION),
+ instance=quote_plus(MYSQL_INSTANCE_NAME),
+ database=quote_plus(MYSQL_DATABASE_NAME),
+ client_cert_file=quote_plus(MYSQL_CLIENT_CERT_FILE),
+ client_key_file=quote_plus(MYSQL_CLIENT_KEY_FILE),
+ server_ca_file=quote_plus(MYSQL_SERVER_CA_FILE)
+)
+
+# MySQL: connect via proxy over TCP (specific proxy version)
+os.environ['AIRFLOW_CONN_PROXY_MYSQL_TCP'] = \
+ "gcpcloudsql://{user}:{password}@{public_ip}:{public_port}/{database}?" \
+ "database_type=mysql&" \
+ "project_id={project_id}&" \
+ "location={location}&" \
+ "instance={instance}&" \
+ "use_proxy=True&" \
+ "sql_proxy_version=v1.13&" \
+ "sql_proxy_use_tcp=True".format(**mysql_kwargs)
+
+# MySQL: connect via proxy over UNIX socket using pre-downloaded Cloud Sql Proxy binary
+try:
+ sql_proxy_binary_path = subprocess.check_output(
+ ['which', 'cloud_sql_proxy']).decode('utf-8').rstrip()
+except subprocess.CalledProcessError:
+ sql_proxy_binary_path = "/tmp/anyhow_download_cloud_sql_proxy"
+
+os.environ['AIRFLOW_CONN_PROXY_MYSQL_SOCKET'] = \
+ "gcpcloudsql://{user}:{password}@{public_ip}:{public_port}/{database}?" \
+ "database_type=mysql&" \
+ "project_id={project_id}&" \
+ "location={location}&" \
+ "instance={instance}&" \
+ "use_proxy=True&" \
+ "sql_proxy_binary_path={sql_proxy_binary_path}&" \
+ "sql_proxy_use_tcp=False".format(
+ sql_proxy_binary_path=quote_plus(sql_proxy_binary_path), **mysql_kwargs)
+
+# MySQL: connect directly via TCP (non-SSL)
+os.environ['AIRFLOW_CONN_PUBLIC_MYSQL_TCP'] = \
+ "gcpcloudsql://{user}:{password}@{public_ip}:{public_port}/{database}?" \
+ "database_type=mysql&" \
+ "project_id={project_id}&" \
+ "location={location}&" \
+ "instance={instance}&" \
+ "use_proxy=False&" \
+ "use_ssl=False".format(**mysql_kwargs)
+
+# MySQL: connect directly via TCP (SSL) and with fixed Cloud Sql Proxy binary path
+os.environ['AIRFLOW_CONN_PUBLIC_MYSQL_TCP_SSL'] = \
+ "gcpcloudsql://{user}:{password}@{public_ip}:{public_port}/{database}?" \
+ "database_type=mysql&" \
+ "project_id={project_id}&" \
+ "location={location}&" \
+ "instance={instance}&" \
+ "use_proxy=False&" \
+ "use_ssl=True&" \
+ "sslcert={client_cert_file}&" \
+ "sslkey={client_key_file}&" \
+ "sslrootcert={server_ca_file}".format(**mysql_kwargs)
+
+# [END howto_operator_cloudsql_query_connections]
+
+# [START howto_operator_cloudsql_query_operators]
+
+connection_names = [
+ "proxy_postgres_tcp",
+ "proxy_postgres_socket",
+ "public_postgres_tcp",
+ "public_postgres_tcp_ssl",
+ "proxy_mysql_tcp",
+ "proxy_mysql_socket",
+ "public_mysql_tcp",
+ "public_mysql_tcp_ssl"
+]
+
+tasks = []
+
+with models.DAG(
+ dag_id='example_gcp_sql_query',
+ default_args=default_args,
+ schedule_interval=None
+) as dag:
+ for connection_name in connection_names:
+ tasks.append(
+ CloudSqlQueryOperator(
+ gcp_cloudsql_conn_id=connection_name,
+ task_id="example_gcp_sql_task_" + connection_name,
+ sql=SQL
+ )
+ )
+# [END howto_operator_cloudsql_query_operators]
diff --git a/airflow/contrib/example_dags/example_gcs_acl.py b/airflow/contrib/example_dags/example_gcs_acl.py
new file mode 100644
index 0000000000..7247199a4f
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcs_acl.py
@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that creates a new ACL entry on the specified bucket and object.
+
+This DAG relies on the following OS environment variables
+
+* GCS_ACL_BUCKET - Name of a bucket.
+* GCS_ACL_OBJECT - Name of the object. For information about how to URL encode object
+ names to be path safe, see:
+ https://cloud.google.com/storage/docs/json_api/#encoding
+* GCS_ACL_ENTITY - The entity holding the permission.
+* GCS_ACL_BUCKET_ROLE - The access permission for the entity for the bucket.
+* GCS_ACL_OBJECT_ROLE - The access permission for the entity for the object.
+"""
+import os
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcs_acl_operator import \
+ GoogleCloudStorageBucketCreateAclEntryOperator, \
+ GoogleCloudStorageObjectCreateAclEntryOperator
+
+# [START howto_operator_gcs_acl_args_common]
+GCS_ACL_BUCKET = os.environ.get('GCS_ACL_BUCKET', 'example-bucket')
+GCS_ACL_OBJECT = os.environ.get('GCS_ACL_OBJECT', 'example-object')
+GCS_ACL_ENTITY = os.environ.get('GCS_ACL_ENTITY', 'example-entity')
+GCS_ACL_BUCKET_ROLE = os.environ.get('GCS_ACL_BUCKET_ROLE', 'example-bucket-role')
+GCS_ACL_OBJECT_ROLE = os.environ.get('GCS_ACL_OBJECT_ROLE', 'example-object-role')
+# [END howto_operator_gcs_acl_args_common]
+
+default_args = {
+ 'start_date': airflow.utils.dates.days_ago(1)
+}
+
+with models.DAG(
+ 'example_gcs_acl',
+ default_args=default_args,
+ schedule_interval=None # Change to match your use case
+) as dag:
+ # [START howto_operator_gcs_bucket_create_acl_entry_task]
+ gcs_bucket_create_acl_entry_task = GoogleCloudStorageBucketCreateAclEntryOperator(
+ bucket=GCS_ACL_BUCKET,
+ entity=GCS_ACL_ENTITY,
+ role=GCS_ACL_BUCKET_ROLE,
+ task_id="gcs_bucket_create_acl_entry_task"
+ )
+ # [END howto_operator_gcs_bucket_create_acl_entry_task]
+ # [START howto_operator_gcs_object_create_acl_entry_task]
+ gcs_object_create_acl_entry_task = GoogleCloudStorageObjectCreateAclEntryOperator(
+ bucket=GCS_ACL_BUCKET,
+ object_name=GCS_ACL_OBJECT,
+ entity=GCS_ACL_ENTITY,
+ role=GCS_ACL_OBJECT_ROLE,
+ task_id="gcs_object_create_acl_entry_task"
+ )
+ # [END howto_operator_gcs_object_create_acl_entry_task]
+
+ gcs_bucket_create_acl_entry_task >> gcs_object_create_acl_entry_task
diff --git a/airflow/contrib/example_dags/example_gcs_to_bq_operator.py b/airflow/contrib/example_dags/example_gcs_to_bq_operator.py
new file mode 100644
index 0000000000..ee9fe09391
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcs_to_bq_operator.py
@@ -0,0 +1,63 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import airflow
+try:
+ from airflow.contrib.operators import gcs_to_bq
+except ImportError:
+ gcs_to_bq = None
+from airflow import models
+from airflow.operators import bash_operator
+
+
+if gcs_to_bq is not None:
+ args = {
+ 'owner': 'airflow',
+ 'start_date': airflow.utils.dates.days_ago(2)
+ }
+
+ dag = models.DAG(
+ dag_id='example_gcs_to_bq_operator', default_args=args,
+ schedule_interval=None)
+
+ create_test_dataset = bash_operator.BashOperator(
+ task_id='create_airflow_test_dataset',
+ bash_command='bq mk airflow_test',
+ dag=dag)
+
+ # [START howto_operator_gcs_to_bq]
+ load_csv = gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
+ task_id='gcs_to_bq_example',
+ bucket='cloud-samples-data',
+ source_objects=['bigquery/us-states/us-states.csv'],
+ destination_project_dataset_table='airflow_test.gcs_to_bq_table',
+ schema_fields=[
+ {'name': 'name', 'type': 'STRING', 'mode': 'NULLABLE'},
+ {'name': 'post_abbr', 'type': 'STRING', 'mode': 'NULLABLE'},
+ ],
+ write_disposition='WRITE_TRUNCATE',
+ dag=dag)
+ # [END howto_operator_gcs_to_bq]
+
+ delete_test_dataset = bash_operator.BashOperator(
+ task_id='delete_airflow_test_dataset',
+ bash_command='bq rm -rf airflow_test',
+ dag=dag)
+
+ create_test_dataset >> load_csv >> delete_test_dataset
diff --git a/airflow/contrib/example_dags/example_jenkins_job_trigger_operator.py.notexecutable b/airflow/contrib/example_dags/example_jenkins_job_trigger_operator.py.notexecutable
new file mode 100644
index 0000000000..2d8906b0b3
--- /dev/null
+++ b/airflow/contrib/example_dags/example_jenkins_job_trigger_operator.py.notexecutable
@@ -0,0 +1,82 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+
+from airflow import DAG
+from airflow.contrib.operators.jenkins_job_trigger_operator import JenkinsJobTriggerOperator
+from airflow.operators.python_operator import PythonOperator
+from airflow.contrib.hooks.jenkins_hook import JenkinsHook
+
+from six.moves.urllib.request import Request
+
+import jenkins
+import datetime
+
+
+datetime_start_date = datetime(2017, 6, 1)
+default_args = {
+ "owner": "airflow",
+ "start_date": datetime_start_date,
+ "retries": 1,
+ "retry_delay": timedelta(minutes=5),
+ "depends_on_past": False,
+ "concurrency": 8,
+ "max_active_runs": 8
+
+}
+
+
+dag = DAG("test_jenkins", default_args=default_args, schedule_interval=None)
+#This DAG shouldn't be executed and is only here to provide example of how to use the JenkinsJobTriggerOperator
+#(it requires a jenkins server to be executed)
+
+job_trigger = JenkinsJobTriggerOperator(
+ dag=dag,
+ task_id="trigger_job",
+ job_name="generate-merlin-config",
+ parameters={"first_parameter":"a_value", "second_parameter":"18"},
+ #parameters="resources/paremeter.json", You can also pass a path to a json file containing your param
+ jenkins_connection_id="your_jenkins_connection" #The connection must be configured first
+ )
+
+def grabArtifactFromJenkins(**context):
+ """
+ Grab an artifact from the previous job
+ The python-jenkins library doesn't expose a method for that
+ But it's totally possible to build manually the request for that
+ """
+ hook = JenkinsHook("your_jenkins_connection")
+ jenkins_server = hook.get_jenkins_server()
+ url = context['task_instance'].xcom_pull(task_ids='trigger_job')
+ #The JenkinsJobTriggerOperator store the job url in the xcom variable corresponding to the task
+ #You can then use it to access things or to get the job number
+ #This url looks like : http://jenkins_url/job/job_name/job_number/
+ url = url + "artifact/myartifact.xml" #Or any other artifact name
+ request = Request(url)
+ response = jenkins_server.jenkins_open(request)
+ return response #We store the artifact content in a xcom variable for later use
+
+artifact_grabber = PythonOperator(
+ task_id='artifact_grabber',
+ provide_context=True,
+ python_callable=grabArtifactFromJenkins,
+ dag=dag)
+
+artifact_grabber.set_upstream(job_trigger)
diff --git a/airflow/contrib/example_dags/example_kubernetes_executor.py b/airflow/contrib/example_dags/example_kubernetes_executor.py
new file mode 100644
index 0000000000..d03e255ab3
--- /dev/null
+++ b/airflow/contrib/example_dags/example_kubernetes_executor.py
@@ -0,0 +1,97 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import print_function
+import airflow
+from airflow.operators.python_operator import PythonOperator
+from airflow.models import DAG
+import os
+
+args = {
+ 'owner': 'airflow',
+ 'start_date': airflow.utils.dates.days_ago(2)
+}
+
+dag = DAG(
+ dag_id='example_kubernetes_executor', default_args=args,
+ schedule_interval=None
+)
+
+affinity = {
+ 'podAntiAffinity': {
+ 'requiredDuringSchedulingIgnoredDuringExecution': [
+ {
+ 'topologyKey': 'kubernetes.io/hostname',
+ 'labelSelector': {
+ 'matchExpressions': [
+ {
+ 'key': 'app',
+ 'operator': 'In',
+ 'values': ['airflow']
+ }
+ ]
+ }
+ }
+ ]
+ }
+}
+
+tolerations = [{
+ 'key': 'dedicated',
+ 'operator': 'Equal',
+ 'value': 'airflow'
+}]
+
+
+def print_stuff():
+ print("stuff!")
+
+
+def use_zip_binary():
+ rc = os.system("zip")
+ assert rc == 0
+
+
+# You don't have to use any special KubernetesExecutor configuration if you don't want to
+start_task = PythonOperator(
+ task_id="start_task", python_callable=print_stuff, dag=dag
+)
+
+# But you can if you want to
+one_task = PythonOperator(
+ task_id="one_task", python_callable=print_stuff, dag=dag,
+ executor_config={"KubernetesExecutor": {"image": "airflow/ci:latest"}}
+)
+
+# Use the zip binary, which is only found in this special docker image
+two_task = PythonOperator(
+ task_id="two_task", python_callable=use_zip_binary, dag=dag,
+ executor_config={"KubernetesExecutor": {"image": "airflow/ci_zip:latest"}}
+)
+
+# Limit resources on this operator/task with node affinity & tolerations
+three_task = PythonOperator(
+ task_id="three_task", python_callable=print_stuff, dag=dag,
+ executor_config={
+ "KubernetesExecutor": {"request_memory": "128Mi",
+ "limit_memory": "128Mi",
+ "tolerations": tolerations,
+ "affinity": affinity}}
+)
+
+start_task.set_downstream([one_task, two_task, three_task])
diff --git a/airflow/contrib/example_dags/example_kubernetes_executor_config.py b/airflow/contrib/example_dags/example_kubernetes_executor_config.py
new file mode 100644
index 0000000000..1b194499b2
--- /dev/null
+++ b/airflow/contrib/example_dags/example_kubernetes_executor_config.py
@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import print_function
+import airflow
+from airflow.operators.python_operator import PythonOperator
+from libs.helper import print_stuff
+from airflow.models import DAG
+import os
+
+args = {
+ 'owner': 'airflow',
+ 'start_date': airflow.utils.dates.days_ago(2)
+}
+
+dag = DAG(
+ dag_id='example_kubernetes_executor_config', default_args=args,
+ schedule_interval=None
+)
+
+
+def test_volume_mount():
+ with open('/foo/volume_mount_test.txt', 'w') as foo:
+ foo.write('Hello')
+
+ rc = os.system("cat /foo/volume_mount_test.txt")
+ assert rc == 0
+
+
+# You can use annotations on your kubernetes pods!
+start_task = PythonOperator(
+ task_id="start_task", python_callable=print_stuff, dag=dag,
+ executor_config={
+ "KubernetesExecutor": {
+ "annotations": {"test": "annotation"}
+ }
+ }
+)
+
+# You can mount volume or secret to the worker pod
+second_task = PythonOperator(
+ task_id="four_task", python_callable=test_volume_mount, dag=dag,
+ executor_config={
+ "KubernetesExecutor": {
+ "volumes": [
+ {
+ "name": "example-kubernetes-test-volume",
+ "hostPath": {"path": "/tmp/"},
+ },
+ ],
+ "volume_mounts": [
+ {
+ "mountPath": "/foo/",
+ "name": "example-kubernetes-test-volume",
+ },
+ ]
+ }
+ }
+)
+
+start_task.set_downstream(second_task)
diff --git a/airflow/contrib/example_dags/example_kubernetes_operator.py b/airflow/contrib/example_dags/example_kubernetes_operator.py
new file mode 100644
index 0000000000..ba1f4433a7
--- /dev/null
+++ b/airflow/contrib/example_dags/example_kubernetes_operator.py
@@ -0,0 +1,67 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.utils.dates import days_ago
+from airflow.utils.log.logging_mixin import LoggingMixin
+from airflow.models import DAG
+
+log = LoggingMixin().log
+
+try:
+ # Kubernetes is optional, so not available in vanilla Airflow
+ # pip install apache-airflow[kubernetes]
+ from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
+
+ args = {
+ 'owner': 'airflow',
+ 'start_date': days_ago(2)
+ }
+
+ dag = DAG(
+ dag_id='example_kubernetes_operator',
+ default_args=args,
+ schedule_interval=None)
+
+ tolerations = [
+ {
+ 'key': "key",
+ 'operator': 'Equal',
+ 'value': 'value'
+ }
+ ]
+
+ k = KubernetesPodOperator(
+ namespace='default',
+ image="ubuntu:16.04",
+ cmds=["bash", "-cx"],
+ arguments=["echo", "10"],
+ labels={"foo": "bar"},
+ name="airflow-test-pod",
+ in_cluster=False,
+ task_id="task",
+ get_logs=True,
+ dag=dag,
+ is_delete_operator_pod=False,
+ tolerations=tolerations
+ )
+
+except ImportError as e:
+ log.warn("Could not import KubernetesPodOperator: " + str(e))
+ log.warn("Install kubernetes dependencies with: "
+ " pip install apache-airflow[kubernetes]")
diff --git a/airflow/contrib/example_dags/example_pubsub_flow.py b/airflow/contrib/example_dags/example_pubsub_flow.py
index c8843c86ae..6c13ec1daf 100644
--- a/airflow/contrib/example_dags/example_pubsub_flow.py
+++ b/airflow/contrib/example_dags/example_pubsub_flow.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
"""
This example DAG demonstrates how the PubSub*Operators and PubSubPullSensor
diff --git a/airflow/contrib/example_dags/example_qubole_operator.py b/airflow/contrib/example_dags/example_qubole_operator.py
index 6d9340d97a..826a50af99 100644
--- a/airflow/contrib/example_dags/example_qubole_operator.py
+++ b/airflow/contrib/example_dags/example_qubole_operator.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
"""
This is only an example DAG to highlight usage of QuboleOperator in various scenarios,
@@ -44,21 +49,31 @@
dag.doc_md = __doc__
+
def compare_result(ds, **kwargs):
ti = kwargs['ti']
r1 = t1.get_results(ti)
r2 = t2.get_results(ti)
return filecmp.cmp(r1, r2)
+
t1 = QuboleOperator(
task_id='hive_show_table',
command_type='hivecmd',
query='show tables',
- cluster_label='default',
- fetch_logs=True, # If true, will fetch qubole command logs and concatenate them into corresponding airflow task logs
- tags='aiflow_example_run', # To attach tags to qubole command, auto attach 3 tags - dag_id, task_id, run_id
- qubole_conn_id='qubole_default', # Connection id to submit commands inside QDS, if not set "qubole_default" is used
- dag=dag)
+ cluster_label='{{ params.cluster_label }}',
+ fetch_logs=True,
+ # If `fetch_logs`=true, will fetch qubole command logs and concatenate
+ # them into corresponding airflow task logs
+ tags='aiflow_example_run',
+ # To attach tags to qubole command, auto attach 3 tags - dag_id, task_id, run_id
+ qubole_conn_id='qubole_default',
+ # Connection id to submit commands inside QDS, if not set "qubole_default" is used
+ dag=dag,
+ params={
+ 'cluster_label': 'default',
+ }
+)
t2 = QuboleOperator(
task_id='hive_s3_location',
@@ -98,10 +113,19 @@ def compare_result(ds, **kwargs):
t4 = QuboleOperator(
task_id='hadoop_jar_cmd',
command_type='hadoopcmd',
- sub_command='jar s3://paid-qubole/HadoopAPIExamples/jars/hadoop-0.20.1-dev-streaming.jar -mapper wc -numReduceTasks 0 -input s3://paid-qubole/HadoopAPITests/data/3.tsv -output s3://paid-qubole/HadoopAPITests/data/3_wc',
- cluster_label='default',
+ sub_command='jar s3://paid-qubole/HadoopAPIExamples/'
+ 'jars/hadoop-0.20.1-dev-streaming.jar '
+ '-mapper wc '
+ '-numReduceTasks 0 -input s3://paid-qubole/HadoopAPITests/'
+ 'data/3.tsv -output '
+ 's3://paid-qubole/HadoopAPITests/data/3_wc',
+ cluster_label='{{ params.cluster_label }}',
fetch_logs=True,
- dag=dag)
+ dag=dag,
+ params={
+ 'cluster_label': 'default',
+ }
+)
t5 = QuboleOperator(
task_id='pig_cmd',
diff --git a/airflow/contrib/example_dags/example_qubole_sensor.py b/airflow/contrib/example_dags/example_qubole_sensor.py
index 7e06fd4a92..3922d3eca9 100644
--- a/airflow/contrib/example_dags/example_qubole_sensor.py
+++ b/airflow/contrib/example_dags/example_qubole_sensor.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
"""
This is only an example DAG to highlight usage of QuboleSensor in various scenarios,
@@ -45,11 +50,13 @@
qubole_conn_id='qubole_default',
poke_interval=60,
timeout=600,
- data={"files":
- ["s3://paid-qubole/HadoopAPIExamples/jars/hadoop-0.20.1-dev-streaming.jar",
- "s3://paid-qubole/HadoopAPITests/data/{{ ds.split('-')[2] }}.tsv"
- ] # will check for availability of all the files in array
- },
+ data={
+ "files":
+ [
+ "s3://paid-qubole/HadoopAPIExamples/jars/hadoop-0.20.1-dev-streaming.jar",
+ "s3://paid-qubole/HadoopAPITests/data/{{ ds.split('-')[2] }}.tsv"
+ ] # will check for availability of all the files in array
+ },
dag=dag
)
@@ -57,12 +64,14 @@
task_id='check_hive_partition',
poke_interval=10,
timeout=60,
- data={"schema":"default",
- "table":"my_partitioned_table",
- "columns":[
- {"column" : "month", "values" : ["{{ ds.split('-')[1] }}"]},
- {"column" : "day", "values" : ["{{ ds.split('-')[2] }}" , "{{ yesterday_ds.split('-')[2] }}"]}
- ]# will check for partitions like [month=12/day=12,month=12/day=13]
+ data={"schema": "default",
+ "table": "my_partitioned_table",
+ "columns": [
+ {"column": "month", "values":
+ ["{{ ds.split('-')[1] }}"]},
+ {"column": "day", "values":
+ ["{{ ds.split('-')[2] }}", "{{ yesterday_ds.split('-')[2] }}"]}
+ ] # will check for partitions like [month=12/day=12,month=12/day=13]
},
dag=dag
)
diff --git a/airflow/contrib/example_dags/example_twitter_README.md b/airflow/contrib/example_dags/example_twitter_README.md
index 319eac39f6..67e95581d7 100644
--- a/airflow/contrib/example_dags/example_twitter_README.md
+++ b/airflow/contrib/example_dags/example_twitter_README.md
@@ -1,3 +1,22 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
# Example Twitter DAG
***Introduction:*** This example dag depicts a typical ETL process and is a perfect use case automation scenario for Airflow. Please note that the main scripts associated with the tasks are returning None. The purpose of this DAG is to demonstrate how to write a functional DAG within Airflow.
@@ -7,11 +26,11 @@
***Overview:*** At first, we need tasks that will get the tweets of our interest and save them on the hard-disk. Then, we need subsequent tasks that will clean and analyze the tweets. Then we want to store these files into HDFS, and load them into a Data Warehousing platform like Hive or HBase. The main reason we have selected Hive here is because it gives us a familiar SQL like interface, and makes our life of writing different queries a lot easier. Finally, the DAG needs to store a summarized result to a traditional database, i.e. MySQL or PostgreSQL, which is used by a reporting or business intelligence application. In other words, we basically want to achieve the following steps:
1. Fetch Tweets
-2. Clean Tweets
-3. Analyze Tweets
-4. Put Tweets to HDFS
-5. Load data to Hive
-6. Save Summary to MySQL
+1. Clean Tweets
+1. Analyze Tweets
+1. Put Tweets to HDFS
+1. Load data to Hive
+1. Save Summary to MySQL
***Screenshot:***
<img src="http://i.imgur.com/rRpSO12.png" width="99%"/>
@@ -21,16 +40,18 @@
The python functions here are just placeholders. In case you are interested to actually make this DAG fully functional, first start with filling out the scripts as separate files and importing them into the DAG with absolute or relative import. My approach was to store the retrieved data in memory using Pandas dataframe first, and then use the built in method to save the CSV file on hard-disk.
The eight different CSV files are then put into eight different folders within HDFS. Each of the newly inserted files are then loaded into eight different external hive tables. Hive tables can be external or internal. In this case, we are inserting the data right into the table, and so we are making our tables internal. Each file is inserted into the respected Hive table named after the twitter channel, i.e. toTwitter_A or fromTwitter_A. It is also important to note that when we created the tables, we facilitated for partitioning by date using the variable dt and declared comma as the row deliminator. The partitioning is very handy and ensures our query execution time remains constant even with growing volume of data.
As most probably these folders and hive tables doesn't exist in your system, you will get an error for these tasks within the DAG. If you rebuild a function DAG from this example, make sure those folders and hive tables exists. When you create the table, keep the consideration of table partitioning and declaring comma as the row deliminator in your mind. Furthermore, you may also need to skip headers on each read and ensure that the user under which you have Airflow running has the right permission access. Below is a sample HQL snippet on creating such table:
+
```
CREATE TABLE toTwitter_A(id BIGINT, id_str STRING
- created_at STRING, text STRING)
- PARTITIONED BY (dt STRING)
- ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
- STORED AS TEXTFILE;
- alter table toTwitter_A SET serdeproperties ('skip.header.line.count' = '1');
+ created_at STRING, text STRING)
+ PARTITIONED BY (dt STRING)
+ ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
+ STORED AS TEXTFILE;
+ alter table toTwitter_A SET serdeproperties ('skip.header.line.count' = '1');
```
+
When you review the code for the DAG, you will notice that these tasks are generated using for loop. These two for loops could be combined into one loop. However, in most cases, you will be running different analysis on your incoming incoming and outgoing tweets, and hence they are kept separated in this example.
Final step is a running the broker script, brokerapi.py, which will run queries in Hive and store the summarized data to MySQL in our case. To connect to Hive, pyhs2 library is extremely useful and easy to use. To insert data into MySQL from Python, sqlalchemy is also a good one to use.
-I hope you find this tutorial useful. If you have question feel free to ask me on [Twitter](https://twitter.com/EkhtiarSyed) or via the live Airflow chatroom room in [Gitter](https://gitter.im/airbnb/airflow).<p>
+I hope you find this tutorial useful. If you have question feel free to ask me on [Twitter](https://twitter.com/EkhtiarSyed) or via the live Airflow chatroom room in [Gitter](https://gitter.im/apache/incubator-airflow).<p>
-Ekhtiar Syed
Last Update: 8-April-2016
diff --git a/airflow/contrib/example_dags/example_twitter_dag.py b/airflow/contrib/example_dags/example_twitter_dag.py
index 2205b9a108..c2c8f4dd07 100644
--- a/airflow/contrib/example_dags/example_twitter_dag.py
+++ b/airflow/contrib/example_dags/example_twitter_dag.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
# --------------------------------------------------------------------------------
# Written By: Ekhtiar Syed
# Last Update: 8th April 2016
diff --git a/airflow/contrib/example_dags/example_winrm_operator.py b/airflow/contrib/example_dags/example_winrm_operator.py
new file mode 100644
index 0000000000..195bf5d98d
--- /dev/null
+++ b/airflow/contrib/example_dags/example_winrm_operator.py
@@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# --------------------------------------------------------------------------------
+# Written By: Ekhtiar Syed
+# Last Update: 8th April 2016
+# Caveat: This Dag will not run because of missing scripts.
+# The purpose of this is to give you a sample of a real world example DAG!
+# --------------------------------------------------------------------------------
+
+# --------------------------------------------------------------------------------
+# Load The Dependencies
+# --------------------------------------------------------------------------------
+import airflow
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.models import DAG
+from datetime import timedelta
+
+from airflow.contrib.hooks import WinRMHook
+from airflow.contrib.operators.winrm_operator import WinRMOperator
+
+
+args = {
+ 'owner': 'airflow',
+ 'start_date': airflow.utils.dates.days_ago(2)
+}
+
+dag = DAG(
+ dag_id='POC_winrm_parallel', default_args=args,
+ schedule_interval='0 0 * * *',
+ dagrun_timeout=timedelta(minutes=60))
+
+cmd = 'ls -l'
+run_this_last = DummyOperator(task_id='run_this_last', dag=dag)
+
+winRMHook = WinRMHook(ssh_conn_id='ssh_POC1')
+
+t1 = WinRMOperator(
+ task_id="wintask1",
+ command='ls -altr',
+ winrm_hook=winRMHook,
+ dag=dag)
+
+t2 = WinRMOperator(
+ task_id="wintask2",
+ command='sleep 60',
+ winrm_hook=winRMHook,
+ dag=dag)
+
+t3 = WinRMOperator(
+ task_id="wintask3",
+ command='echo \'luke test\' ',
+ winrm_hook=winRMHook,
+ dag=dag)
+
+t1.set_downstream(run_this_last)
+t2.set_downstream(run_this_last)
+t3.set_downstream(run_this_last)
diff --git a/airflow/contrib/example_dags/libs/__init__.py b/airflow/contrib/example_dags/libs/__init__.py
new file mode 100644
index 0000000000..114d189da1
--- /dev/null
+++ b/airflow/contrib/example_dags/libs/__init__.py
@@ -0,0 +1,18 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/contrib/example_dags/libs/helper.py b/airflow/contrib/example_dags/libs/helper.py
new file mode 100644
index 0000000000..d7b62e65c7
--- /dev/null
+++ b/airflow/contrib/example_dags/libs/helper.py
@@ -0,0 +1,22 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+def print_stuff():
+ print("annotated!")
diff --git a/airflow/contrib/executors/__init__.py b/airflow/contrib/executors/__init__.py
index c82f5790fe..b7f8352944 100644
--- a/airflow/contrib/executors/__init__.py
+++ b/airflow/contrib/executors/__init__.py
@@ -1,14 +1,19 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
diff --git a/airflow/contrib/executors/kubernetes_executor.py b/airflow/contrib/executors/kubernetes_executor.py
new file mode 100644
index 0000000000..fa81cf3203
--- /dev/null
+++ b/airflow/contrib/executors/kubernetes_executor.py
@@ -0,0 +1,682 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import base64
+import json
+import multiprocessing
+from queue import Queue
+from dateutil import parser
+from uuid import uuid4
+import kubernetes
+from kubernetes import watch, client
+from kubernetes.client.rest import ApiException
+from airflow.configuration import conf
+from airflow.contrib.kubernetes.pod_launcher import PodLauncher
+from airflow.contrib.kubernetes.kube_client import get_kube_client
+from airflow.contrib.kubernetes.worker_configuration import WorkerConfiguration
+from airflow.executors.base_executor import BaseExecutor
+from airflow.executors import Executors
+from airflow.models import TaskInstance, KubeResourceVersion, KubeWorkerIdentifier
+from airflow.utils.state import State
+from airflow import configuration, settings
+from airflow.exceptions import AirflowConfigException, AirflowException
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class KubernetesExecutorConfig:
+ def __init__(self, image=None, image_pull_policy=None, request_memory=None,
+ request_cpu=None, limit_memory=None, limit_cpu=None,
+ gcp_service_account_key=None, node_selectors=None, affinity=None,
+ annotations=None, volumes=None, volume_mounts=None, tolerations=None):
+ self.image = image
+ self.image_pull_policy = image_pull_policy
+ self.request_memory = request_memory
+ self.request_cpu = request_cpu
+ self.limit_memory = limit_memory
+ self.limit_cpu = limit_cpu
+ self.gcp_service_account_key = gcp_service_account_key
+ self.node_selectors = node_selectors
+ self.affinity = affinity
+ self.annotations = annotations
+ self.volumes = volumes
+ self.volume_mounts = volume_mounts
+ self.tolerations = tolerations
+
+ def __repr__(self):
+ return "{}(image={}, image_pull_policy={}, request_memory={}, request_cpu={}, " \
+ "limit_memory={}, limit_cpu={}, gcp_service_account_key={}, " \
+ "node_selectors={}, affinity={}, annotations={}, volumes={}, " \
+ "volume_mounts={}, tolerations={})" \
+ .format(KubernetesExecutorConfig.__name__, self.image, self.image_pull_policy,
+ self.request_memory, self.request_cpu, self.limit_memory,
+ self.limit_cpu, self.gcp_service_account_key, self.node_selectors,
+ self.affinity, self.annotations, self.volumes, self.volume_mounts,
+ self.tolerations)
+
+ @staticmethod
+ def from_dict(obj):
+ if obj is None:
+ return KubernetesExecutorConfig()
+
+ if not isinstance(obj, dict):
+ raise TypeError(
+ 'Cannot convert a non-dictionary object into a KubernetesExecutorConfig')
+
+ namespaced = obj.get(Executors.KubernetesExecutor, {})
+
+ return KubernetesExecutorConfig(
+ image=namespaced.get('image', None),
+ image_pull_policy=namespaced.get('image_pull_policy', None),
+ request_memory=namespaced.get('request_memory', None),
+ request_cpu=namespaced.get('request_cpu', None),
+ limit_memory=namespaced.get('limit_memory', None),
+ limit_cpu=namespaced.get('limit_cpu', None),
+ gcp_service_account_key=namespaced.get('gcp_service_account_key', None),
+ node_selectors=namespaced.get('node_selectors', None),
+ affinity=namespaced.get('affinity', None),
+ annotations=namespaced.get('annotations', {}),
+ volumes=namespaced.get('volumes', []),
+ volume_mounts=namespaced.get('volume_mounts', []),
+ tolerations=namespaced.get('tolerations', None),
+ )
+
+ def as_dict(self):
+ return {
+ 'image': self.image,
+ 'image_pull_policy': self.image_pull_policy,
+ 'request_memory': self.request_memory,
+ 'request_cpu': self.request_cpu,
+ 'limit_memory': self.limit_memory,
+ 'limit_cpu': self.limit_cpu,
+ 'gcp_service_account_key': self.gcp_service_account_key,
+ 'node_selectors': self.node_selectors,
+ 'affinity': self.affinity,
+ 'annotations': self.annotations,
+ 'volumes': self.volumes,
+ 'volume_mounts': self.volume_mounts,
+ 'tolerations': self.tolerations,
+ }
+
+
+class KubeConfig:
+ core_section = 'core'
+ kubernetes_section = 'kubernetes'
+
+ def __init__(self):
+ configuration_dict = configuration.as_dict(display_sensitive=True)
+ self.core_configuration = configuration_dict['core']
+ self.kube_secrets = configuration_dict.get('kubernetes_secrets', {})
+ self.airflow_home = configuration.get(self.core_section, 'airflow_home')
+ self.dags_folder = configuration.get(self.core_section, 'dags_folder')
+ self.parallelism = configuration.getint(self.core_section, 'PARALLELISM')
+ self.worker_container_repository = configuration.get(
+ self.kubernetes_section, 'worker_container_repository')
+ self.worker_container_tag = configuration.get(
+ self.kubernetes_section, 'worker_container_tag')
+ self.kube_image = '{}:{}'.format(
+ self.worker_container_repository, self.worker_container_tag)
+ self.kube_image_pull_policy = configuration.get(
+ self.kubernetes_section, "worker_container_image_pull_policy"
+ )
+ self.kube_node_selectors = configuration_dict.get('kubernetes_node_selectors', {})
+ self.delete_worker_pods = conf.getboolean(
+ self.kubernetes_section, 'delete_worker_pods')
+
+ self.worker_service_account_name = conf.get(
+ self.kubernetes_section, 'worker_service_account_name')
+ self.image_pull_secrets = conf.get(self.kubernetes_section, 'image_pull_secrets')
+
+ # NOTE: user can build the dags into the docker image directly,
+ # this will set to True if so
+ self.dags_in_image = conf.getboolean(self.kubernetes_section, 'dags_in_image')
+
+ # NOTE: `git_repo` and `git_branch` must be specified together as a pair
+ # The http URL of the git repository to clone from
+ self.git_repo = conf.get(self.kubernetes_section, 'git_repo')
+ # The branch of the repository to be checked out
+ self.git_branch = conf.get(self.kubernetes_section, 'git_branch')
+ # Optionally, the directory in the git repository containing the dags
+ self.git_subpath = conf.get(self.kubernetes_section, 'git_subpath')
+ # Optionally, the root directory for git operations
+ self.git_sync_root = conf.get(self.kubernetes_section, 'git_sync_root')
+ # Optionally, the name at which to publish the checked-out files under --root
+ self.git_sync_dest = conf.get(self.kubernetes_section, 'git_sync_dest')
+ # Optionally, if git_dags_folder_mount_point is set the worker will use
+ # {git_dags_folder_mount_point}/{git_sync_dest}/{git_subpath} as dags_folder
+ self.git_dags_folder_mount_point = conf.get(self.kubernetes_section,
+ 'git_dags_folder_mount_point')
+
+ # Optionally a user may supply a `git_user` and `git_password` for private
+ # repositories
+ self.git_user = conf.get(self.kubernetes_section, 'git_user')
+ self.git_password = conf.get(self.kubernetes_section, 'git_password')
+
+ # NOTE: The user may optionally use a volume claim to mount a PV containing
+ # DAGs directly
+ self.dags_volume_claim = conf.get(self.kubernetes_section, 'dags_volume_claim')
+
+ # This prop may optionally be set for PV Claims and is used to write logs
+ self.logs_volume_claim = conf.get(self.kubernetes_section, 'logs_volume_claim')
+
+ # This prop may optionally be set for PV Claims and is used to locate DAGs
+ # on a SubPath
+ self.dags_volume_subpath = conf.get(
+ self.kubernetes_section, 'dags_volume_subpath')
+
+ # This prop may optionally be set for PV Claims and is used to locate logs
+ # on a SubPath
+ self.logs_volume_subpath = conf.get(
+ self.kubernetes_section, 'logs_volume_subpath')
+
+ # Optionally, hostPath volume containing DAGs
+ self.dags_volume_host = conf.get(self.kubernetes_section, 'dags_volume_host')
+
+ # Optionally, write logs to a hostPath Volume
+ self.logs_volume_host = conf.get(self.kubernetes_section, 'logs_volume_host')
+
+ # This prop may optionally be set for PV Claims and is used to write logs
+ self.base_log_folder = configuration.get(self.core_section, 'base_log_folder')
+
+ # The Kubernetes Namespace in which the Scheduler and Webserver reside. Note
+ # that if your
+ # cluster has RBAC enabled, your scheduler may need service account permissions to
+ # create, watch, get, and delete pods in this namespace.
+ self.kube_namespace = conf.get(self.kubernetes_section, 'namespace')
+ # The Kubernetes Namespace in which pods will be created by the executor. Note
+ # that if your
+ # cluster has RBAC enabled, your workers may need service account permissions to
+ # interact with cluster components.
+ self.executor_namespace = conf.get(self.kubernetes_section, 'namespace')
+ # Task secrets managed by KubernetesExecutor.
+ self.gcp_service_account_keys = conf.get(self.kubernetes_section,
+ 'gcp_service_account_keys')
+
+ # If the user is using the git-sync container to clone their repository via git,
+ # allow them to specify repository, tag, and pod name for the init container.
+ self.git_sync_container_repository = conf.get(
+ self.kubernetes_section, 'git_sync_container_repository')
+
+ self.git_sync_container_tag = conf.get(
+ self.kubernetes_section, 'git_sync_container_tag')
+ self.git_sync_container = '{}:{}'.format(
+ self.git_sync_container_repository, self.git_sync_container_tag)
+
+ self.git_sync_init_container_name = conf.get(
+ self.kubernetes_section, 'git_sync_init_container_name')
+
+ # The worker pod may optionally have a valid Airflow config loaded via a
+ # configmap
+ self.airflow_configmap = conf.get(self.kubernetes_section, 'airflow_configmap')
+
+ affinity_json = conf.get(self.kubernetes_section, 'affinity')
+ if affinity_json:
+ self.kube_affinity = json.loads(affinity_json)
+ else:
+ self.kube_affinity = None
+
+ tolerations_json = conf.get(self.kubernetes_section, 'tolerations')
+ if tolerations_json:
+ self.kube_tolerations = json.loads(tolerations_json)
+ else:
+ self.kube_tolerations = None
+
+ self._validate()
+
+ def _validate(self):
+ # TODO: use XOR for dags_volume_claim and git_dags_folder_mount_point
+ if not self.dags_volume_claim \
+ and not self.dags_volume_host \
+ and not self.dags_in_image \
+ and (not self.git_repo or not self.git_branch or not self.git_dags_folder_mount_point):
+ raise AirflowConfigException(
+ 'In kubernetes mode the following must be set in the `kubernetes` '
+ 'config section: `dags_volume_claim` '
+ 'or `dags_volume_host` '
+ 'or `dags_in_image` '
+ 'or `git_repo and git_branch and git_dags_folder_mount_point`')
+
+
+class KubernetesJobWatcher(multiprocessing.Process, LoggingMixin, object):
+ def __init__(self, namespace, watcher_queue, resource_version, worker_uuid):
+ multiprocessing.Process.__init__(self)
+ self.namespace = namespace
+ self.worker_uuid = worker_uuid
+ self.watcher_queue = watcher_queue
+ self.resource_version = resource_version
+
+ def run(self):
+ kube_client = get_kube_client()
+ while True:
+ try:
+ self.resource_version = self._run(kube_client, self.resource_version,
+ self.worker_uuid)
+ except Exception:
+ self.log.exception('Unknown error in KubernetesJobWatcher. Failing')
+ raise
+ else:
+ self.log.warn('Watch died gracefully, starting back up with: '
+ 'last resource_version: %s', self.resource_version)
+
+ def _run(self, kube_client, resource_version, worker_uuid):
+ self.log.info(
+ 'Event: and now my watch begins starting at resource_version: %s',
+ resource_version
+ )
+ watcher = watch.Watch()
+
+ kwargs = {'label_selector': 'airflow-worker={}'.format(worker_uuid)}
+ if resource_version:
+ kwargs['resource_version'] = resource_version
+
+ last_resource_version = None
+ for event in watcher.stream(kube_client.list_namespaced_pod, self.namespace,
+ **kwargs):
+ task = event['object']
+ self.log.info(
+ 'Event: %s had an event of type %s',
+ task.metadata.name, event['type']
+ )
+ if event['type'] == 'ERROR':
+ return self.process_error(event)
+ self.process_status(
+ task.metadata.name, task.status.phase, task.metadata.labels,
+ task.metadata.resource_version
+ )
+ last_resource_version = task.metadata.resource_version
+
+ return last_resource_version
+
+ def process_error(self, event):
+ self.log.error(
+ 'Encountered Error response from k8s list namespaced pod stream => %s',
+ event
+ )
+ raw_object = event['raw_object']
+ if raw_object['code'] == 410:
+ self.log.info(
+ 'Kubernetes resource version is too old, must reset to 0 => %s',
+ raw_object['message']
+ )
+ # Return resource version 0
+ return '0'
+ raise AirflowException(
+ 'Kubernetes failure for %s with code %s and message: %s',
+ raw_object['reason'], raw_object['code'], raw_object['message']
+ )
+
+ def process_status(self, pod_id, status, labels, resource_version):
+ if status == 'Pending':
+ self.log.info('Event: %s Pending', pod_id)
+ elif status == 'Failed':
+ self.log.info('Event: %s Failed', pod_id)
+ self.watcher_queue.put((pod_id, State.FAILED, labels, resource_version))
+ elif status == 'Succeeded':
+ self.log.info('Event: %s Succeeded', pod_id)
+ self.watcher_queue.put((pod_id, None, labels, resource_version))
+ elif status == 'Running':
+ self.log.info('Event: %s is Running', pod_id)
+ else:
+ self.log.warn(
+ 'Event: Invalid state: %s on pod: %s with labels: %s with '
+ 'resource_version: %s', status, pod_id, labels, resource_version
+ )
+
+
+class AirflowKubernetesScheduler(LoggingMixin):
+ def __init__(self, kube_config, task_queue, result_queue, session,
+ kube_client, worker_uuid):
+ self.log.debug("Creating Kubernetes executor")
+ self.kube_config = kube_config
+ self.task_queue = task_queue
+ self.result_queue = result_queue
+ self.namespace = self.kube_config.kube_namespace
+ self.log.debug("Kubernetes using namespace %s", self.namespace)
+ self.kube_client = kube_client
+ self.launcher = PodLauncher(kube_client=self.kube_client)
+ self.worker_configuration = WorkerConfiguration(kube_config=self.kube_config)
+ self.watcher_queue = multiprocessing.Queue()
+ self._session = session
+ self.worker_uuid = worker_uuid
+ self.kube_watcher = self._make_kube_watcher()
+
+ def _make_kube_watcher(self):
+ resource_version = KubeResourceVersion.get_current_resource_version(self._session)
+ watcher = KubernetesJobWatcher(self.namespace, self.watcher_queue,
+ resource_version, self.worker_uuid)
+ watcher.start()
+ return watcher
+
+ def _health_check_kube_watcher(self):
+ if self.kube_watcher.is_alive():
+ pass
+ else:
+ self.log.error(
+ 'Error while health checking kube watcher process. '
+ 'Process died for unknown reasons')
+ self.kube_watcher = self._make_kube_watcher()
+
+ def run_next(self, next_job):
+ """
+
+ The run_next command will check the task_queue for any un-run jobs.
+ It will then create a unique job-id, launch that job in the cluster,
+ and store relevant info in the current_jobs map so we can track the job's
+ status
+ """
+ self.log.info('Kubernetes job is %s', str(next_job))
+ key, command, kube_executor_config = next_job
+ dag_id, task_id, execution_date, try_number = key
+ self.log.debug("Kubernetes running for command %s", command)
+ self.log.debug("Kubernetes launching image %s", self.kube_config.kube_image)
+ pod = self.worker_configuration.make_pod(
+ namespace=self.namespace, worker_uuid=self.worker_uuid,
+ pod_id=self._create_pod_id(dag_id, task_id),
+ dag_id=dag_id, task_id=task_id,
+ execution_date=self._datetime_to_label_safe_datestring(execution_date),
+ airflow_command=command, kube_executor_config=kube_executor_config
+ )
+ # the watcher will monitor pods, so we do not block.
+ self.launcher.run_pod_async(pod)
+ self.log.debug("Kubernetes Job created!")
+
+ def delete_pod(self, pod_id):
+ if self.kube_config.delete_worker_pods:
+ try:
+ self.kube_client.delete_namespaced_pod(
+ pod_id, self.namespace, body=client.V1DeleteOptions())
+ except ApiException as e:
+ # If the pod is already deleted
+ if e.status != 404:
+ raise
+
+ def sync(self):
+ """
+ The sync function checks the status of all currently running kubernetes jobs.
+ If a job is completed, it's status is placed in the result queue to
+ be sent back to the scheduler.
+
+ :return:
+
+ """
+ self._health_check_kube_watcher()
+ while not self.watcher_queue.empty():
+ self.process_watcher_task()
+
+ def process_watcher_task(self):
+ pod_id, state, labels, resource_version = self.watcher_queue.get()
+ self.log.info(
+ 'Attempting to finish pod; pod_id: %s; state: %s; labels: %s',
+ pod_id, state, labels
+ )
+ key = self._labels_to_key(labels=labels)
+ if key:
+ self.log.debug('finishing job %s - %s (%s)', key, state, pod_id)
+ self.result_queue.put((key, state, pod_id, resource_version))
+
+ @staticmethod
+ def _strip_unsafe_kubernetes_special_chars(string):
+ """
+ Kubernetes only supports lowercase alphanumeric characters and "-" and "." in
+ the pod name
+ However, there are special rules about how "-" and "." can be used so let's
+ only keep
+ alphanumeric chars see here for detail:
+ https://kubernetes.io/docs/concepts/overview/working-with-objects/names/
+
+ :param string: The requested Pod name
+ :return: ``str`` Pod name stripped of any unsafe characters
+ """
+ return ''.join(ch.lower() for ind, ch in enumerate(string) if ch.isalnum())
+
+ @staticmethod
+ def _make_safe_pod_id(safe_dag_id, safe_task_id, safe_uuid):
+ """
+ Kubernetes pod names must be <= 253 chars and must pass the following regex for
+ validation
+ "^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$"
+
+ :param safe_dag_id: a dag_id with only alphanumeric characters
+ :param safe_task_id: a task_id with only alphanumeric characters
+ :param random_uuid: a uuid
+ :return: ``str`` valid Pod name of appropriate length
+ """
+ MAX_POD_ID_LEN = 253
+
+ safe_key = safe_dag_id + safe_task_id
+
+ safe_pod_id = safe_key[:MAX_POD_ID_LEN - len(safe_uuid) - 1] + "-" + safe_uuid
+
+ return safe_pod_id
+
+ @staticmethod
+ def _create_pod_id(dag_id, task_id):
+ safe_dag_id = AirflowKubernetesScheduler._strip_unsafe_kubernetes_special_chars(
+ dag_id)
+ safe_task_id = AirflowKubernetesScheduler._strip_unsafe_kubernetes_special_chars(
+ task_id)
+ safe_uuid = AirflowKubernetesScheduler._strip_unsafe_kubernetes_special_chars(
+ uuid4().hex)
+ return AirflowKubernetesScheduler._make_safe_pod_id(safe_dag_id, safe_task_id,
+ safe_uuid)
+
+ @staticmethod
+ def _label_safe_datestring_to_datetime(string):
+ """
+ Kubernetes doesn't permit ":" in labels. ISO datetime format uses ":" but not
+ "_", let's
+ replace ":" with "_"
+
+ :param string: str
+ :return: datetime.datetime object
+ """
+ return parser.parse(string.replace('_plus_', '+').replace("_", ":"))
+
+ @staticmethod
+ def _datetime_to_label_safe_datestring(datetime_obj):
+ """
+ Kubernetes doesn't like ":" in labels, since ISO datetime format uses ":" but
+ not "_" let's
+ replace ":" with "_"
+ :param datetime_obj: datetime.datetime object
+ :return: ISO-like string representing the datetime
+ """
+ return datetime_obj.isoformat().replace(":", "_").replace('+', '_plus_')
+
+ def _labels_to_key(self, labels):
+ try:
+ return (
+ labels['dag_id'], labels['task_id'],
+ self._label_safe_datestring_to_datetime(labels['execution_date']),
+ labels['try_number'])
+ except Exception as e:
+ self.log.warn(
+ 'Error while converting labels to key; labels: %s; exception: %s',
+ labels, e
+ )
+ return None
+
+
+class KubernetesExecutor(BaseExecutor, LoggingMixin):
+ def __init__(self):
+ self.kube_config = KubeConfig()
+ self.task_queue = None
+ self._session = None
+ self.result_queue = None
+ self.kube_scheduler = None
+ self.kube_client = None
+ self.worker_uuid = None
+ super(KubernetesExecutor, self).__init__(parallelism=self.kube_config.parallelism)
+
+ def clear_not_launched_queued_tasks(self):
+ """
+ If the airflow scheduler restarts with pending "Queued" tasks, the tasks may or
+ may not
+ have been launched Thus, on starting up the scheduler let's check every
+ "Queued" task to
+ see if it has been launched (ie: if there is a corresponding pod on kubernetes)
+
+ If it has been launched then do nothing, otherwise reset the state to "None" so
+ the task
+ will be rescheduled
+
+ This will not be necessary in a future version of airflow in which there is
+ proper support
+ for State.LAUNCHED
+ """
+ queued_tasks = self._session.query(
+ TaskInstance).filter(TaskInstance.state == State.QUEUED).all()
+ self.log.info(
+ 'When executor started up, found %s queued task instances',
+ len(queued_tasks)
+ )
+
+ for task in queued_tasks:
+ dict_string = "dag_id={},task_id={},execution_date={},airflow-worker={}" \
+ .format(task.dag_id, task.task_id,
+ AirflowKubernetesScheduler._datetime_to_label_safe_datestring(
+ task.execution_date), self.worker_uuid)
+ kwargs = dict(label_selector=dict_string)
+ pod_list = self.kube_client.list_namespaced_pod(
+ self.kube_config.kube_namespace, **kwargs)
+ if len(pod_list.items) == 0:
+ self.log.info(
+ 'TaskInstance: %s found in queued state but was not launched, '
+ 'rescheduling', task
+ )
+ self._session.query(TaskInstance).filter(
+ TaskInstance.dag_id == task.dag_id,
+ TaskInstance.task_id == task.task_id,
+ TaskInstance.execution_date == task.execution_date
+ ).update({TaskInstance.state: State.NONE})
+
+ self._session.commit()
+
+ def _inject_secrets(self):
+ def _create_or_update_secret(secret_name, secret_path):
+ try:
+ return self.kube_client.create_namespaced_secret(
+ self.kube_config.executor_namespace, kubernetes.client.V1Secret(
+ data={
+ 'key.json': base64.b64encode(open(secret_path, 'r').read())},
+ metadata=kubernetes.client.V1ObjectMeta(name=secret_name)))
+ except ApiException as e:
+ if e.status == 409:
+ return self.kube_client.replace_namespaced_secret(
+ secret_name, self.kube_config.executor_namespace,
+ kubernetes.client.V1Secret(
+ data={'key.json': base64.b64encode(
+ open(secret_path, 'r').read())},
+ metadata=kubernetes.client.V1ObjectMeta(name=secret_name)))
+ self.log.exception(
+ 'Exception while trying to inject secret. '
+ 'Secret name: %s, error details: %s',
+ secret_name, e
+ )
+ raise
+
+ # For each GCP service account key, inject it as a secret in executor
+ # namespace with the specific secret name configured in the airflow.cfg.
+ # We let exceptions to pass through to users.
+ if self.kube_config.gcp_service_account_keys:
+ name_path_pair_list = [
+ {'name': account_spec.strip().split('=')[0],
+ 'path': account_spec.strip().split('=')[1]}
+ for account_spec in self.kube_config.gcp_service_account_keys.split(',')]
+ for service_account in name_path_pair_list:
+ _create_or_update_secret(service_account['name'], service_account['path'])
+
+ def start(self):
+ self.log.info('Start Kubernetes executor')
+ self._session = settings.Session()
+ self.worker_uuid = KubeWorkerIdentifier.get_or_create_current_kube_worker_uuid(
+ self._session)
+ self.log.debug('Start with worker_uuid: %s', self.worker_uuid)
+ # always need to reset resource version since we don't know
+ # when we last started, note for behavior below
+ # https://github.com/kubernetes-client/python/blob/master/kubernetes/docs
+ # /CoreV1Api.md#list_namespaced_pod
+ KubeResourceVersion.reset_resource_version(self._session)
+ self.task_queue = Queue()
+ self.result_queue = Queue()
+ self.kube_client = get_kube_client()
+ self.kube_scheduler = AirflowKubernetesScheduler(
+ self.kube_config, self.task_queue, self.result_queue, self._session,
+ self.kube_client, self.worker_uuid
+ )
+ self._inject_secrets()
+ self.clear_not_launched_queued_tasks()
+
+ def execute_async(self, key, command, queue=None, executor_config=None):
+ self.log.info(
+ 'Add task %s with command %s with executor_config %s',
+ key, command, executor_config
+ )
+ kube_executor_config = KubernetesExecutorConfig.from_dict(executor_config)
+ self.task_queue.put((key, command, kube_executor_config))
+
+ def sync(self):
+ if self.running:
+ self.log.debug('self.running: %s', self.running)
+ if self.queued_tasks:
+ self.log.debug('self.queued: %s', self.queued_tasks)
+ self.kube_scheduler.sync()
+
+ last_resource_version = None
+ while not self.result_queue.empty():
+ results = self.result_queue.get()
+ key, state, pod_id, resource_version = results
+ last_resource_version = resource_version
+ self.log.info('Changing state of %s to %s', results, state)
+ self._change_state(key, state, pod_id)
+
+ KubeResourceVersion.checkpoint_resource_version(
+ last_resource_version, session=self._session)
+
+ if not self.task_queue.empty():
+ task = self.task_queue.get()
+
+ try:
+ self.kube_scheduler.run_next(task)
+ except ApiException:
+ self.log.exception('ApiException when attempting ' +
+ 'to run task, re-queueing.')
+ self.task_queue.put(task)
+
+ def _change_state(self, key, state, pod_id):
+ if state != State.RUNNING:
+ self.kube_scheduler.delete_pod(pod_id)
+ try:
+ self.log.info('Deleted pod: %s', str(key))
+ self.running.pop(key)
+ except KeyError:
+ self.log.debug('Could not find key: %s', str(key))
+ pass
+ self.event_buffer[key] = state
+ (dag_id, task_id, ex_time, try_number) = key
+ item = self._session.query(TaskInstance).filter_by(
+ dag_id=dag_id,
+ task_id=task_id,
+ execution_date=ex_time
+ ).one()
+ if state:
+ item.state = state
+ self._session.add(item)
+ self._session.commit()
+
+ def end(self):
+ self.log.info('Shutting down Kubernetes executor')
+ self.task_queue.join()
diff --git a/airflow/contrib/executors/mesos_executor.py b/airflow/contrib/executors/mesos_executor.py
index 87285664a2..7aae91e6d4 100644
--- a/airflow/contrib/executors/mesos_executor.py
+++ b/airflow/contrib/executors/mesos_executor.py
@@ -1,23 +1,28 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from future import standard_library
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.www.utils import LoginMixin
-standard_library.install_aliases()
+
from builtins import str
from queue import Queue
@@ -31,15 +36,15 @@
from airflow.utils.state import State
from airflow.exceptions import AirflowException
-
+standard_library.install_aliases()
DEFAULT_FRAMEWORK_NAME = 'Airflow'
FRAMEWORK_CONNID_PREFIX = 'mesos_framework_'
def get_framework_name():
- if not configuration.get('mesos', 'FRAMEWORK_NAME'):
+ if not configuration.conf.get('mesos', 'FRAMEWORK_NAME'):
return DEFAULT_FRAMEWORK_NAME
- return configuration.get('mesos', 'FRAMEWORK_NAME')
+ return configuration.conf.get('mesos', 'FRAMEWORK_NAME')
# AirflowMesosScheduler, implements Mesos Scheduler interface
@@ -63,13 +68,19 @@ def __init__(self,
self.task_mem = task_mem
self.task_counter = 0
self.task_key_map = {}
+ if configuration.get('mesos', 'DOCKER_IMAGE_SLAVE'):
+ self.mesos_slave_docker_image = configuration.get(
+ 'mesos', 'DOCKER_IMAGE_SLAVE'
+ )
def registered(self, driver, frameworkId, masterInfo):
- self.log.info("AirflowScheduler registered to Mesos with framework ID %s", frameworkId.value)
+ self.log.info("AirflowScheduler registered to Mesos with framework ID %s",
+ frameworkId.value)
- if configuration.getboolean('mesos', 'CHECKPOINT') and configuration.get('mesos', 'FAILOVER_TIMEOUT'):
+ if configuration.conf.getboolean('mesos', 'CHECKPOINT') and \
+ configuration.conf.get('mesos', 'FAILOVER_TIMEOUT'):
# Import here to work around a circular import error
- from airflow.models import Connection
+ from airflow.models.connection import Connection
# Update the Framework ID in the database.
session = Session()
@@ -118,14 +129,15 @@ def resourceOffers(self, driver, offers):
elif resource.name == "mem":
offerMem += resource.scalar.value
- self.log.info("Received offer %s with cpus: %s and mem: %s", offer.id.value, offerCpus, offerMem)
+ self.log.info("Received offer %s with cpus: %s and mem: %s",
+ offer.id.value, offerCpus, offerMem)
remainingCpus = offerCpus
remainingMem = offerMem
while (not self.task_queue.empty()) and \
- remainingCpus >= self.task_cpu and \
- remainingMem >= self.task_mem:
+ remainingCpus >= self.task_cpu and \
+ remainingMem >= self.task_mem:
key, cmd = self.task_queue.get()
tid = self.task_counter
self.task_counter += 1
@@ -150,9 +162,24 @@ def resourceOffers(self, driver, offers):
command = mesos_pb2.CommandInfo()
command.shell = True
- command.value = cmd
+ command.value = " ".join(cmd)
task.command.MergeFrom(command)
+ # If docker image for airflow is specified in config then pull that
+ # image before running the above airflow command
+ if self.mesos_slave_docker_image:
+ network = mesos_pb2.ContainerInfo.DockerInfo.Network.Value('BRIDGE')
+ docker = mesos_pb2.ContainerInfo.DockerInfo(
+ image=self.mesos_slave_docker_image,
+ force_pull_image=False,
+ network=network
+ )
+ container = mesos_pb2.ContainerInfo(
+ type=mesos_pb2.ContainerInfo.DOCKER,
+ docker=docker
+ )
+ task.container.MergeFrom(container)
+
tasks.append(task)
remainingCpus -= self.task_cpu
@@ -169,7 +196,8 @@ def statusUpdate(self, driver, update):
try:
key = self.task_key_map[update.task_id.value]
except KeyError:
- # The map may not contain an item if the framework re-registered after a failover.
+ # The map may not contain an item if the framework re-registered
+ # after a failover.
# Discard these tasks.
self.log.warning("Unrecognised task key %s", update.task_id.value)
return
@@ -202,66 +230,75 @@ def start(self):
framework = mesos_pb2.FrameworkInfo()
framework.user = ''
- if not configuration.get('mesos', 'MASTER'):
+ if not configuration.conf.get('mesos', 'MASTER'):
self.log.error("Expecting mesos master URL for mesos executor")
raise AirflowException("mesos.master not provided for mesos executor")
- master = configuration.get('mesos', 'MASTER')
+ master = configuration.conf.get('mesos', 'MASTER')
framework.name = get_framework_name()
- if not configuration.get('mesos', 'TASK_CPU'):
+ if not configuration.conf.get('mesos', 'TASK_CPU'):
task_cpu = 1
else:
- task_cpu = configuration.getint('mesos', 'TASK_CPU')
+ task_cpu = configuration.conf.getint('mesos', 'TASK_CPU')
- if not configuration.get('mesos', 'TASK_MEMORY'):
+ if not configuration.conf.get('mesos', 'TASK_MEMORY'):
task_memory = 256
else:
- task_memory = configuration.getint('mesos', 'TASK_MEMORY')
+ task_memory = configuration.conf.getint('mesos', 'TASK_MEMORY')
- if configuration.getboolean('mesos', 'CHECKPOINT'):
+ if configuration.conf.getboolean('mesos', 'CHECKPOINT'):
framework.checkpoint = True
- if configuration.get('mesos', 'FAILOVER_TIMEOUT'):
+ if configuration.conf.get('mesos', 'FAILOVER_TIMEOUT'):
# Import here to work around a circular import error
- from airflow.models import Connection
+ from airflow.models.connection import Connection
# Query the database to get the ID of the Mesos Framework, if available.
conn_id = FRAMEWORK_CONNID_PREFIX + framework.name
session = Session()
connection = session.query(Connection).filter_by(conn_id=conn_id).first()
if connection is not None:
- # Set the Framework ID to let the scheduler reconnect with running tasks.
+ # Set the Framework ID to let the scheduler reconnect
+ # with running tasks.
framework.id.value = connection.extra
- framework.failover_timeout = configuration.getint('mesos', 'FAILOVER_TIMEOUT')
+ framework.failover_timeout = configuration.conf.getint(
+ 'mesos', 'FAILOVER_TIMEOUT'
+ )
else:
framework.checkpoint = False
self.log.info(
'MesosFramework master : %s, name : %s, cpu : %s, mem : %s, checkpoint : %s',
- master, framework.name, str(task_cpu), str(task_memory), str(framework.checkpoint)
+ master, framework.name,
+ str(task_cpu), str(task_memory), str(framework.checkpoint)
)
implicit_acknowledgements = 1
- if configuration.getboolean('mesos', 'AUTHENTICATE'):
- if not configuration.get('mesos', 'DEFAULT_PRINCIPAL'):
+ if configuration.conf.getboolean('mesos', 'AUTHENTICATE'):
+ if not configuration.conf.get('mesos', 'DEFAULT_PRINCIPAL'):
self.log.error("Expecting authentication principal in the environment")
- raise AirflowException("mesos.default_principal not provided in authenticated mode")
- if not configuration.get('mesos', 'DEFAULT_SECRET'):
+ raise AirflowException(
+ "mesos.default_principal not provided in authenticated mode")
+ if not configuration.conf.get('mesos', 'DEFAULT_SECRET'):
self.log.error("Expecting authentication secret in the environment")
- raise AirflowException("mesos.default_secret not provided in authenticated mode")
+ raise AirflowException(
+ "mesos.default_secret not provided in authenticated mode")
credential = mesos_pb2.Credential()
- credential.principal = configuration.get('mesos', 'DEFAULT_PRINCIPAL')
- credential.secret = configuration.get('mesos', 'DEFAULT_SECRET')
+ credential.principal = configuration.conf.get('mesos', 'DEFAULT_PRINCIPAL')
+ credential.secret = configuration.conf.get('mesos', 'DEFAULT_SECRET')
framework.principal = credential.principal
driver = mesos.native.MesosSchedulerDriver(
- AirflowMesosScheduler(self.task_queue, self.result_queue, task_cpu, task_memory),
+ AirflowMesosScheduler(self.task_queue,
+ self.result_queue,
+ task_cpu,
+ task_memory),
framework,
master,
implicit_acknowledgements,
@@ -269,7 +306,10 @@ def start(self):
else:
framework.principal = 'Airflow'
driver = mesos.native.MesosSchedulerDriver(
- AirflowMesosScheduler(self.task_queue, self.result_queue, task_cpu, task_memory),
+ AirflowMesosScheduler(self.task_queue,
+ self.result_queue,
+ task_cpu,
+ task_memory),
framework,
master,
implicit_acknowledgements)
@@ -277,7 +317,7 @@ def start(self):
self.mesos_driver = driver
self.mesos_driver.start()
- def execute_async(self, key, command, queue=None):
+ def execute_async(self, key, command, queue=None, executor_config=None):
self.task_queue.put((key, command))
def sync(self):
diff --git a/airflow/contrib/hooks/__init__.py b/airflow/contrib/hooks/__init__.py
index 99a1746ee6..b7f8352944 100644
--- a/airflow/contrib/hooks/__init__.py
+++ b/airflow/contrib/hooks/__init__.py
@@ -1,59 +1,19 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-# Contrib hooks are not imported by default. They should be accessed
-# directly: from airflow.contrib.hooks.hook_module import Hook
-
-
-import sys
-
-
-# ------------------------------------------------------------------------
-#
-# #TODO #FIXME Airflow 2.0
-#
-# Old import machinary below.
-#
-# This is deprecated but should be kept until Airflow 2.0
-# for compatibility.
-#
-# ------------------------------------------------------------------------
-_hooks = {
- 'docker_hook': ['DockerHook'],
- 'ftp_hook': ['FTPHook'],
- 'ftps_hook': ['FTPSHook'],
- 'vertica_hook': ['VerticaHook'],
- 'ssh_hook': ['SSHHook'],
- 'bigquery_hook': ['BigQueryHook'],
- 'qubole_hook': ['QuboleHook'],
- 'gcs_hook': ['GoogleCloudStorageHook'],
- 'datastore_hook': ['DatastoreHook'],
- 'gcp_cloudml_hook': ['CloudMLHook'],
- 'redshift_hook': ['RedshiftHook'],
- 'gcp_dataproc_hook': ['DataProcHook'],
- 'gcp_dataflow_hook': ['DataFlowHook'],
- 'spark_submit_operator': ['SparkSubmitOperator'],
- 'cloudant_hook': ['CloudantHook'],
- 'fs_hook': ['FSHook'],
- 'wasb_hook': ['WasbHook'],
- 'gcp_pubsub_hook': ['PubSubHook'],
- 'aws_dynamodb_hook': ['AwsDynamoDBHook']
-}
-
-import os as _os
-if not _os.environ.get('AIRFLOW_USE_NEW_IMPORTS', False):
- from airflow.utils.helpers import AirflowImporter
- airflow_importer = AirflowImporter(sys.modules[__name__], _hooks)
diff --git a/airflow/contrib/hooks/aws_athena_hook.py b/airflow/contrib/hooks/aws_athena_hook.py
new file mode 100644
index 0000000000..f11ff23c51
--- /dev/null
+++ b/airflow/contrib/hooks/aws_athena_hook.py
@@ -0,0 +1,150 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from time import sleep
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AWSAthenaHook(AwsHook):
+ """
+ Interact with AWS Athena to run, poll queries and return query results
+
+ :param aws_conn_id: aws connection to use.
+ :type aws_conn_id: str
+ :param sleep_time: Time to wait between two consecutive call to check query status on athena
+ :type sleep_time: int
+ """
+
+ INTERMEDIATE_STATES = ('QUEUED', 'RUNNING',)
+ FAILURE_STATES = ('FAILED', 'CANCELLED',)
+ SUCCESS_STATES = ('SUCCEEDED',)
+
+ def __init__(self, aws_conn_id='aws_default', sleep_time=30, *args, **kwargs):
+ super(AWSAthenaHook, self).__init__(aws_conn_id, **kwargs)
+ self.sleep_time = sleep_time
+ self.conn = None
+
+ def get_conn(self):
+ """
+ check if aws conn exists already or create one and return it
+
+ :return: boto3 session
+ """
+ if not self.conn:
+ self.conn = self.get_client_type('athena')
+ return self.conn
+
+ def run_query(self, query, query_context, result_configuration, client_request_token=None):
+ """
+ Run Presto query on athena with provided config and return submitted query_execution_id
+
+ :param query: Presto query to run
+ :type query: str
+ :param query_context: Context in which query need to be run
+ :type query_context: dict
+ :param result_configuration: Dict with path to store results in and config related to encryption
+ :type result_configuration: dict
+ :param client_request_token: Unique token created by user to avoid multiple executions of same query
+ :type client_request_token: str
+ :return: str
+ """
+ response = self.conn.start_query_execution(QueryString=query,
+ ClientRequestToken=client_request_token,
+ QueryExecutionContext=query_context,
+ ResultConfiguration=result_configuration)
+ query_execution_id = response['QueryExecutionId']
+ return query_execution_id
+
+ def check_query_status(self, query_execution_id):
+ """
+ Fetch the status of submitted athena query. Returns None or one of valid query states.
+
+ :param query_execution_id: Id of submitted athena query
+ :type query_execution_id: str
+ :return: str
+ """
+ response = self.conn.get_query_execution(QueryExecutionId=query_execution_id)
+ state = None
+ try:
+ state = response['QueryExecution']['Status']['State']
+ except Exception as ex:
+ self.log.error('Exception while getting query state', ex)
+ finally:
+ return state
+
+ def get_query_results(self, query_execution_id):
+ """
+ Fetch submitted athena query results. returns none if query is in intermediate state or
+ failed/cancelled state else dict of query output
+
+ :param query_execution_id: Id of submitted athena query
+ :type query_execution_id: str
+ :return: dict
+ """
+ query_state = self.check_query_status(query_execution_id)
+ if query_state is None:
+ self.log.error('Invalid Query state')
+ return None
+ elif query_state in self.INTERMEDIATE_STATES or query_state in self.FAILURE_STATES:
+ self.log.error('Query is in {state} state. Cannot fetch results'.format(state=query_state))
+ return None
+ return self.conn.get_query_results(QueryExecutionId=query_execution_id)
+
+ def poll_query_status(self, query_execution_id, max_tries=None):
+ """
+ Poll the status of submitted athena query until query state reaches final state.
+ Returns one of the final states
+
+ :param query_execution_id: Id of submitted athena query
+ :type query_execution_id: str
+ :param max_tries: Number of times to poll for query state before function exits
+ :type max_tries: int
+ :return: str
+ """
+ try_number = 1
+ final_query_state = None # Query state when query reaches final state or max_tries reached
+ while True:
+ query_state = self.check_query_status(query_execution_id)
+ if query_state is None:
+ self.log.info('Trial {try_number}: Invalid query state. Retrying again'.format(
+ try_number=try_number))
+ elif query_state in self.INTERMEDIATE_STATES:
+ self.log.info('Trial {try_number}: Query is still in an intermediate state - {state}'
+ .format(try_number=try_number, state=query_state))
+ else:
+ self.log.info('Trial {try_number}: Query execution completed. Final state is {state}'
+ .format(try_number=try_number, state=query_state))
+ final_query_state = query_state
+ break
+ if max_tries and try_number >= max_tries: # Break loop if max_tries reached
+ final_query_state = query_state
+ break
+ try_number += 1
+ sleep(self.sleep_time)
+ return final_query_state
+
+ def stop_query(self, query_execution_id):
+ """
+ Cancel the submitted athena query
+
+ :param query_execution_id: Id of submitted athena query
+ :type query_execution_id: str
+ :return: dict
+ """
+ return self.conn.stop_query_execution(QueryExecutionId=query_execution_id)
diff --git a/airflow/contrib/hooks/aws_dynamodb_hook.py b/airflow/contrib/hooks/aws_dynamodb_hook.py
index bb50ada637..69324c6dba 100644
--- a/airflow/contrib/hooks/aws_dynamodb_hook.py
+++ b/airflow/contrib/hooks/aws_dynamodb_hook.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from airflow.exceptions import AirflowException
from airflow.contrib.hooks.aws_hook import AwsHook
@@ -28,7 +33,11 @@ class AwsDynamoDBHook(AwsHook):
:type region_name: str
"""
- def __init__(self, table_keys=None, table_name=None, region_name=None, *args, **kwargs):
+ def __init__(self,
+ table_keys=None,
+ table_name=None,
+ region_name=None,
+ *args, **kwargs):
self.table_keys = table_keys
self.table_name = table_name
self.region_name = region_name
diff --git a/airflow/contrib/hooks/aws_firehose_hook.py b/airflow/contrib/hooks/aws_firehose_hook.py
new file mode 100644
index 0000000000..b273e22b54
--- /dev/null
+++ b/airflow/contrib/hooks/aws_firehose_hook.py
@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AwsFirehoseHook(AwsHook):
+ """
+ Interact with AWS Kinesis Firehose.
+ :param delivery_stream: Name of the delivery stream
+ :type delivery_stream: str
+ :param region_name: AWS region name (example: us-east-1)
+ :type region_name: str
+ """
+
+ def __init__(self, delivery_stream, region_name=None, *args, **kwargs):
+ self.delivery_stream = delivery_stream
+ self.region_name = region_name
+ super(AwsFirehoseHook, self).__init__(*args, **kwargs)
+
+ def get_conn(self):
+ """
+ Returns AwsHook connection object.
+ """
+
+ self.conn = self.get_client_type('firehose', self.region_name)
+ return self.conn
+
+ def put_records(self, records):
+ """
+ Write batch records to Kinesis Firehose
+ """
+
+ firehose_conn = self.get_conn()
+
+ response = firehose_conn.put_record_batch(
+ DeliveryStreamName=self.delivery_stream,
+ Records=records
+ )
+
+ return response
diff --git a/airflow/contrib/hooks/aws_hook.py b/airflow/contrib/hooks/aws_hook.py
index fac0ab7401..9d4a73e1c0 100644
--- a/airflow/contrib/hooks/aws_hook.py
+++ b/airflow/contrib/hooks/aws_hook.py
@@ -1,20 +1,25 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
import boto3
import configparser
+import logging
from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
@@ -66,7 +71,7 @@ def _parse_s3_config(config_file_name, config_format='boto', profile=None):
try:
access_key = config.get(cred_section, key_id_option)
secret_key = config.get(cred_section, secret_key_option)
- except:
+ except Exception:
logging.warning("Option Error in parsing s3 config file")
raise
return access_key, secret_key
@@ -78,62 +83,123 @@ class AwsHook(BaseHook):
This class is a thin wrapper around the boto3 python library.
"""
- def __init__(self, aws_conn_id='aws_default'):
+ def __init__(self, aws_conn_id='aws_default', verify=None):
self.aws_conn_id = aws_conn_id
+ self.verify = verify
def _get_credentials(self, region_name):
aws_access_key_id = None
aws_secret_access_key = None
+ aws_session_token = None
endpoint_url = None
if self.aws_conn_id:
try:
connection_object = self.get_connection(self.aws_conn_id)
+ extra_config = connection_object.extra_dejson
if connection_object.login:
aws_access_key_id = connection_object.login
aws_secret_access_key = connection_object.password
- elif 'aws_secret_access_key' in connection_object.extra_dejson:
- aws_access_key_id = connection_object.extra_dejson['aws_access_key_id']
- aws_secret_access_key = connection_object.extra_dejson['aws_secret_access_key']
+ elif 'aws_secret_access_key' in extra_config:
+ aws_access_key_id = extra_config[
+ 'aws_access_key_id']
+ aws_secret_access_key = extra_config[
+ 'aws_secret_access_key']
- elif 's3_config_file' in connection_object.extra_dejson:
+ elif 's3_config_file' in extra_config:
aws_access_key_id, aws_secret_access_key = \
- _parse_s3_config(connection_object.extra_dejson['s3_config_file'],
- connection_object.extra_dejson.get('s3_config_format'))
+ _parse_s3_config(
+ extra_config['s3_config_file'],
+ extra_config.get('s3_config_format'),
+ extra_config.get('profile'))
if region_name is None:
- region_name = connection_object.extra_dejson.get('region_name')
-
- endpoint_url = connection_object.extra_dejson.get('host')
+ region_name = extra_config.get('region_name')
+
+ role_arn = extra_config.get('role_arn')
+ external_id = extra_config.get('external_id')
+ aws_account_id = extra_config.get('aws_account_id')
+ aws_iam_role = extra_config.get('aws_iam_role')
+
+ if role_arn is None and aws_account_id is not None and \
+ aws_iam_role is not None:
+ role_arn = "arn:aws:iam::{}:role/{}" \
+ .format(aws_account_id, aws_iam_role)
+
+ if role_arn is not None:
+ sts_session = boto3.session.Session(
+ aws_access_key_id=aws_access_key_id,
+ aws_secret_access_key=aws_secret_access_key,
+ region_name=region_name)
+
+ sts_client = sts_session.client('sts')
+
+ if external_id is None:
+ sts_response = sts_client.assume_role(
+ RoleArn=role_arn,
+ RoleSessionName='Airflow_' + self.aws_conn_id)
+ else:
+ sts_response = sts_client.assume_role(
+ RoleArn=role_arn,
+ RoleSessionName='Airflow_' + self.aws_conn_id,
+ ExternalId=external_id)
+
+ credentials = sts_response['Credentials']
+ aws_access_key_id = credentials['AccessKeyId']
+ aws_secret_access_key = credentials['SecretAccessKey']
+ aws_session_token = credentials['SessionToken']
+
+ endpoint_url = extra_config.get('host')
except AirflowException:
# No connection found: fallback on boto3 credential strategy
# http://boto3.readthedocs.io/en/latest/guide/configuration.html
pass
- return aws_access_key_id, aws_secret_access_key, region_name, endpoint_url
-
- def get_client_type(self, client_type, region_name=None):
- aws_access_key_id, aws_secret_access_key, region_name, endpoint_url = \
- self._get_credentials(region_name)
-
- return boto3.client(
- client_type,
- region_name=region_name,
+ return boto3.session.Session(
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
- endpoint_url=endpoint_url
- )
-
- def get_resource_type(self, resource_type, region_name=None):
- aws_access_key_id, aws_secret_access_key, region_name, endpoint_url = \
- self._get_credentials(region_name)
-
- return boto3.resource(
- resource_type,
- region_name=region_name,
- aws_access_key_id=aws_access_key_id,
- aws_secret_access_key=aws_secret_access_key,
- endpoint_url=endpoint_url
- )
+ aws_session_token=aws_session_token,
+ region_name=region_name), endpoint_url
+
+ def get_client_type(self, client_type, region_name=None, config=None):
+ session, endpoint_url = self._get_credentials(region_name)
+
+ return session.client(client_type, endpoint_url=endpoint_url,
+ config=config, verify=self.verify)
+
+ def get_resource_type(self, resource_type, region_name=None, config=None):
+ session, endpoint_url = self._get_credentials(region_name)
+
+ return session.resource(resource_type, endpoint_url=endpoint_url,
+ config=config, verify=self.verify)
+
+ def get_session(self, region_name=None):
+ """Get the underlying boto3.session."""
+ session, _ = self._get_credentials(region_name)
+ return session
+
+ def get_credentials(self, region_name=None):
+ """Get the underlying `botocore.Credentials` object.
+
+ This contains the following authentication attributes: access_key, secret_key and token.
+ """
+ session, _ = self._get_credentials(region_name)
+ # Credentials are refreshable, so accessing your access key and
+ # secret key separately can lead to a race condition.
+ # See https://stackoverflow.com/a/36291428/8283373
+ return session.get_credentials().get_frozen_credentials()
+
+ def expand_role(self, role):
+ """
+ If the IAM role is a role name, get the Amazon Resource Name (ARN) for the role.
+ If IAM role is already an IAM role ARN, no change is made.
+
+ :param role: IAM role name or ARN
+ :return: IAM role ARN
+ """
+ if '/' in role:
+ return role
+ else:
+ return self.get_client_type('iam').get_role(RoleName=role)['Role']['Arn']
diff --git a/airflow/contrib/hooks/aws_lambda_hook.py b/airflow/contrib/hooks/aws_lambda_hook.py
index bcd1c7fd70..57cf716fda 100644
--- a/airflow/contrib/hooks/aws_lambda_hook.py
+++ b/airflow/contrib/hooks/aws_lambda_hook.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from airflow.contrib.hooks.aws_hook import AwsHook
@@ -31,7 +36,8 @@ class AwsLambdaHook(AwsHook):
:type invocation_type: str
"""
- def __init__(self, function_name, region_name=None, log_type='None', qualifier='$LATEST',
+ def __init__(self, function_name, region_name=None,
+ log_type='None', qualifier='$LATEST',
invocation_type='RequestResponse', *args, **kwargs):
self.function_name = function_name
self.region_name = region_name
diff --git a/airflow/contrib/hooks/aws_sns_hook.py b/airflow/contrib/hooks/aws_sns_hook.py
new file mode 100644
index 0000000000..4308b493ce
--- /dev/null
+++ b/airflow/contrib/hooks/aws_sns_hook.py
@@ -0,0 +1,60 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AwsSnsHook(AwsHook):
+ """
+ Interact with Amazon Simple Notification Service.
+ """
+
+ def __init__(self, *args, **kwargs):
+ super(AwsSnsHook, self).__init__(*args, **kwargs)
+
+ def get_conn(self):
+ """
+ Get an SNS connection
+ """
+ self.conn = self.get_client_type('sns')
+ return self.conn
+
+ def publish_to_target(self, target_arn, message):
+ """
+ Publish a message to a topic or an endpoint.
+
+ :param target_arn: either a TopicArn or an EndpointArn
+ :type target_arn: str
+ :param message: the default message you want to send
+ :param message: str
+ """
+
+ conn = self.get_conn()
+
+ messages = {
+ 'default': message
+ }
+
+ return conn.publish(
+ TargetArn=target_arn,
+ Message=json.dumps(messages),
+ MessageStructure='json'
+ )
diff --git a/airflow/contrib/hooks/azure_container_instance_hook.py b/airflow/contrib/hooks/azure_container_instance_hook.py
new file mode 100644
index 0000000000..5ad64de6d7
--- /dev/null
+++ b/airflow/contrib/hooks/azure_container_instance_hook.py
@@ -0,0 +1,167 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+
+from airflow.hooks.base_hook import BaseHook
+from airflow.exceptions import AirflowException
+
+from azure.common.client_factory import get_client_from_auth_file
+from azure.common.credentials import ServicePrincipalCredentials
+
+from azure.mgmt.containerinstance import ContainerInstanceManagementClient
+
+
+class AzureContainerInstanceHook(BaseHook):
+ """
+ A hook to communicate with Azure Container Instances.
+
+ This hook requires a service principal in order to work.
+ After creating this service principal
+ (Azure Active Directory/App Registrations), you need to fill in the
+ client_id (Application ID) as login, the generated password as password,
+ and tenantId and subscriptionId in the extra's field as a json.
+
+ :param conn_id: connection id of a service principal which will be used
+ to start the container instance
+ :type conn_id: str
+ """
+
+ def __init__(self, conn_id='azure_default'):
+ self.conn_id = conn_id
+ self.connection = self.get_conn()
+
+ def get_conn(self):
+ conn = self.get_connection(self.conn_id)
+ key_path = conn.extra_dejson.get('key_path', False)
+ if key_path:
+ if key_path.endswith('.json'):
+ self.log.info('Getting connection using a JSON key file.')
+ return get_client_from_auth_file(ContainerInstanceManagementClient,
+ key_path)
+ else:
+ raise AirflowException('Unrecognised extension for key file.')
+
+ if os.environ.get('AZURE_AUTH_LOCATION'):
+ key_path = os.environ.get('AZURE_AUTH_LOCATION')
+ if key_path.endswith('.json'):
+ self.log.info('Getting connection using a JSON key file.')
+ return get_client_from_auth_file(ContainerInstanceManagementClient,
+ key_path)
+ else:
+ raise AirflowException('Unrecognised extension for key file.')
+
+ credentials = ServicePrincipalCredentials(
+ client_id=conn.login,
+ secret=conn.password,
+ tenant=conn.extra_dejson['tenantId']
+ )
+
+ subscription_id = conn.extra_dejson['subscriptionId']
+ return ContainerInstanceManagementClient(credentials, str(subscription_id))
+
+ def create_or_update(self, resource_group, name, container_group):
+ """
+ Create a new container group
+
+ :param resource_group: the name of the resource group
+ :type resource_group: str
+ :param name: the name of the container group
+ :type name: str
+ :param container_group: the properties of the container group
+ :type container_group: azure.mgmt.containerinstance.models.ContainerGroup
+ """
+ self.connection.container_groups.create_or_update(resource_group,
+ name,
+ container_group)
+
+ def get_state_exitcode_details(self, resource_group, name):
+ """
+ Get the state and exitcode of a container group
+
+ :param resource_group: the name of the resource group
+ :type resource_group: str
+ :param name: the name of the container group
+ :type name: str
+ :return: A tuple with the state, exitcode, and details.
+ If the exitcode is unknown 0 is returned.
+ :rtype: tuple(state,exitcode,details)
+ """
+ current_state = self._get_instance_view(resource_group, name).current_state
+ return (current_state.state,
+ current_state.exit_code,
+ current_state.detail_status)
+
+ def _get_instance_view(self, resource_group, name):
+ response = self.connection.container_groups.get(resource_group,
+ name,
+ raw=False)
+ return response.containers[0].instance_view.current_state
+
+ def get_messages(self, resource_group, name):
+ """
+ Get the messages of a container group
+
+ :param resource_group: the name of the resource group
+ :type resource_group: str
+ :param name: the name of the container group
+ :type name: str
+ :return: A list of the event messages
+ :rtype: list<str>
+ """
+ instance_view = self._get_instance_view(resource_group, name)
+
+ return [event.message for event in instance_view.events]
+
+ def get_logs(self, resource_group, name, tail=1000):
+ """
+ Get the tail from logs of a container group
+
+ :param resource_group: the name of the resource group
+ :type resource_group: str
+ :param name: the name of the container group
+ :type name: str
+ :param tail: the size of the tail
+ :type tail: int
+ :return: A list of log messages
+ :rtype: list<str>
+ """
+ logs = self.connection.container.list_logs(resource_group, name, name, tail=tail)
+ return logs.content.splitlines(True)
+
+ def delete(self, resource_group, name):
+ """
+ Delete a container group
+
+ :param resource_group: the name of the resource group
+ :type resource_group: str
+ :param name: the name of the container group
+ :type name: str
+ """
+ self.connection.container_groups.delete(resource_group, name)
+
+ def exists(self, resource_group, name):
+ """
+ Test if a container group exists
+
+ :param resource_group: the name of the resource group
+ :type resource_group: str
+ :param name: the name of the container group
+ :type name: str
+ """
+ for container in self.connection.container_groups.list_by_resource_group(resource_group):
+ if container.name == name:
+ return True
+ return False
diff --git a/airflow/contrib/hooks/azure_container_registry_hook.py b/airflow/contrib/hooks/azure_container_registry_hook.py
new file mode 100644
index 0000000000..af38c1a943
--- /dev/null
+++ b/airflow/contrib/hooks/azure_container_registry_hook.py
@@ -0,0 +1,39 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.hooks.base_hook import BaseHook
+from azure.mgmt.containerinstance.models import ImageRegistryCredential
+
+
+class AzureContainerRegistryHook(BaseHook):
+ """
+ A hook to communicate with a Azure Container Registry.
+
+ :param conn_id: connection id of a service principal which will be used
+ to start the container instance
+ :type conn_id: str
+ """
+
+ def __init__(self, conn_id='azure_registry'):
+ self.conn_id = conn_id
+ self.connection = self.get_conn()
+
+ def get_conn(self):
+ conn = self.get_connection(self.conn_id)
+ return ImageRegistryCredential(server=conn.host, username=conn.login, password=conn.password)
diff --git a/airflow/contrib/hooks/azure_container_volume_hook.py b/airflow/contrib/hooks/azure_container_volume_hook.py
new file mode 100644
index 0000000000..5bf3491064
--- /dev/null
+++ b/airflow/contrib/hooks/azure_container_volume_hook.py
@@ -0,0 +1,54 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.hooks.base_hook import BaseHook
+from azure.mgmt.containerinstance.models import (Volume,
+ AzureFileVolume)
+
+
+class AzureContainerVolumeHook(BaseHook):
+ """
+ A hook which wraps an Azure Volume.
+
+ :param wasb_conn_id: connection id of a Azure storage account of
+ which file shares should be mounted
+ :type wasb_conn_id: str
+ """
+
+ def __init__(self, wasb_conn_id='wasb_default'):
+ self.conn_id = wasb_conn_id
+
+ def get_storagekey(self):
+ conn = self.get_connection(self.conn_id)
+ service_options = conn.extra_dejson
+
+ if 'connection_string' in service_options:
+ for keyvalue in service_options['connection_string'].split(";"):
+ key, value = keyvalue.split("=", 1)
+ if key == "AccountKey":
+ return value
+ return conn.password
+
+ def get_file_volume(self, mount_name, share_name,
+ storage_account_name, read_only=False):
+ return Volume(name=mount_name,
+ azure_file=AzureFileVolume(share_name=share_name,
+ storage_account_name=storage_account_name,
+ read_only=read_only,
+ storage_account_key=self.get_storagekey()))
diff --git a/airflow/contrib/hooks/azure_cosmos_hook.py b/airflow/contrib/hooks/azure_cosmos_hook.py
new file mode 100644
index 0000000000..01b4007b03
--- /dev/null
+++ b/airflow/contrib/hooks/azure_cosmos_hook.py
@@ -0,0 +1,287 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import azure.cosmos.cosmos_client as cosmos_client
+from azure.cosmos.errors import HTTPFailure
+import uuid
+
+from airflow.exceptions import AirflowBadRequest
+from airflow.hooks.base_hook import BaseHook
+
+
+class AzureCosmosDBHook(BaseHook):
+ """
+ Interacts with Azure CosmosDB.
+
+ login should be the endpoint uri, password should be the master key
+ optionally, you can use the following extras to default these values
+ {"database_name": "<DATABASE_NAME>", "collection_name": "COLLECTION_NAME"}.
+
+ :param azure_cosmos_conn_id: Reference to the Azure CosmosDB connection.
+ :type azure_cosmos_conn_id: str
+ """
+
+ def __init__(self, azure_cosmos_conn_id='azure_cosmos_default'):
+ self.conn_id = azure_cosmos_conn_id
+ self.connection = self.get_connection(self.conn_id)
+ self.extras = self.connection.extra_dejson
+
+ self.endpoint_uri = self.connection.login
+ self.master_key = self.connection.password
+ self.default_database_name = self.extras.get('database_name')
+ self.default_collection_name = self.extras.get('collection_name')
+ self.cosmos_client = None
+
+ def get_conn(self):
+ """
+ Return a cosmos db client.
+ """
+ if self.cosmos_client is not None:
+ return self.cosmos_client
+
+ # Initialize the Python Azure Cosmos DB client
+ self.cosmos_client = cosmos_client.CosmosClient(self.endpoint_uri, {'masterKey': self.master_key})
+
+ return self.cosmos_client
+
+ def __get_database_name(self, database_name=None):
+ db_name = database_name
+ if db_name is None:
+ db_name = self.default_database_name
+
+ if db_name is None:
+ raise AirflowBadRequest("Database name must be specified")
+
+ return db_name
+
+ def __get_collection_name(self, collection_name=None):
+ coll_name = collection_name
+ if coll_name is None:
+ coll_name = self.default_collection_name
+
+ if coll_name is None:
+ raise AirflowBadRequest("Collection name must be specified")
+
+ return coll_name
+
+ def does_collection_exist(self, collection_name, database_name=None):
+ """
+ Checks if a collection exists in CosmosDB.
+ """
+ if collection_name is None:
+ raise AirflowBadRequest("Collection name cannot be None.")
+
+ existing_container = list(self.get_conn().QueryContainers(
+ get_database_link(self.__get_database_name(database_name)), {
+ "query": "SELECT * FROM r WHERE r.id=@id",
+ "parameters": [
+ {"name": "@id", "value": collection_name}
+ ]
+ }))
+ if len(existing_container) == 0:
+ return False
+
+ return True
+
+ def create_collection(self, collection_name, database_name=None):
+ """
+ Creates a new collection in the CosmosDB database.
+ """
+ if collection_name is None:
+ raise AirflowBadRequest("Collection name cannot be None.")
+
+ # We need to check to see if this container already exists so we don't try
+ # to create it twice
+ existing_container = list(self.get_conn().QueryContainers(
+ get_database_link(self.__get_database_name(database_name)), {
+ "query": "SELECT * FROM r WHERE r.id=@id",
+ "parameters": [
+ {"name": "@id", "value": collection_name}
+ ]
+ }))
+
+ # Only create if we did not find it already existing
+ if len(existing_container) == 0:
+ self.get_conn().CreateContainer(
+ get_database_link(self.__get_database_name(database_name)),
+ {"id": collection_name})
+
+ def does_database_exist(self, database_name):
+ """
+ Checks if a database exists in CosmosDB.
+ """
+ if database_name is None:
+ raise AirflowBadRequest("Database name cannot be None.")
+
+ existing_database = list(self.get_conn().QueryDatabases({
+ "query": "SELECT * FROM r WHERE r.id=@id",
+ "parameters": [
+ {"name": "@id", "value": database_name}
+ ]
+ }))
+ if len(existing_database) == 0:
+ return False
+
+ return True
+
+ def create_database(self, database_name):
+ """
+ Creates a new database in CosmosDB.
+ """
+ if database_name is None:
+ raise AirflowBadRequest("Database name cannot be None.")
+
+ # We need to check to see if this database already exists so we don't try
+ # to create it twice
+ existing_database = list(self.get_conn().QueryDatabases({
+ "query": "SELECT * FROM r WHERE r.id=@id",
+ "parameters": [
+ {"name": "@id", "value": database_name}
+ ]
+ }))
+
+ # Only create if we did not find it already existing
+ if len(existing_database) == 0:
+ self.get_conn().CreateDatabase({"id": database_name})
+
+ def delete_database(self, database_name):
+ """
+ Deletes an existing database in CosmosDB.
+ """
+ if database_name is None:
+ raise AirflowBadRequest("Database name cannot be None.")
+
+ self.get_conn().DeleteDatabase(get_database_link(database_name))
+
+ def delete_collection(self, collection_name, database_name=None):
+ """
+ Deletes an existing collection in the CosmosDB database.
+ """
+ if collection_name is None:
+ raise AirflowBadRequest("Collection name cannot be None.")
+
+ self.get_conn().DeleteContainer(
+ get_collection_link(self.__get_database_name(database_name), collection_name))
+
+ def upsert_document(self, document, database_name=None, collection_name=None, document_id=None):
+ """
+ Inserts a new document (or updates an existing one) into an existing
+ collection in the CosmosDB database.
+ """
+ # Assign unique ID if one isn't provided
+ if document_id is None:
+ document_id = str(uuid.uuid4())
+
+ if document is None:
+ raise AirflowBadRequest("You cannot insert a None document")
+
+ # Add document id if isn't found
+ if 'id' in document:
+ if document['id'] is None:
+ document['id'] = document_id
+ else:
+ document['id'] = document_id
+
+ created_document = self.get_conn().CreateItem(
+ get_collection_link(
+ self.__get_database_name(database_name),
+ self.__get_collection_name(collection_name)),
+ document)
+
+ return created_document
+
+ def insert_documents(self, documents, database_name=None, collection_name=None):
+ """
+ Insert a list of new documents into an existing collection in the CosmosDB database.
+ """
+ if documents is None:
+ raise AirflowBadRequest("You cannot insert empty documents")
+
+ created_documents = []
+ for single_document in documents:
+ created_documents.append(
+ self.get_conn().CreateItem(
+ get_collection_link(
+ self.__get_database_name(database_name),
+ self.__get_collection_name(collection_name)),
+ single_document))
+
+ return created_documents
+
+ def delete_document(self, document_id, database_name=None, collection_name=None):
+ """
+ Delete an existing document out of a collection in the CosmosDB database.
+ """
+ if document_id is None:
+ raise AirflowBadRequest("Cannot delete a document without an id")
+
+ self.get_conn().DeleteItem(
+ get_document_link(
+ self.__get_database_name(database_name),
+ self.__get_collection_name(collection_name),
+ document_id))
+
+ def get_document(self, document_id, database_name=None, collection_name=None):
+ """
+ Get a document from an existing collection in the CosmosDB database.
+ """
+ if document_id is None:
+ raise AirflowBadRequest("Cannot get a document without an id")
+
+ try:
+ return self.get_conn().ReadItem(
+ get_document_link(
+ self.__get_database_name(database_name),
+ self.__get_collection_name(collection_name),
+ document_id))
+ except HTTPFailure:
+ return None
+
+ def get_documents(self, sql_string, database_name=None, collection_name=None, partition_key=None):
+ """
+ Get a list of documents from an existing collection in the CosmosDB database via SQL query.
+ """
+ if sql_string is None:
+ raise AirflowBadRequest("SQL query string cannot be None")
+
+ # Query them in SQL
+ query = {'query': sql_string}
+
+ try:
+ result_iterable = self.get_conn().QueryItems(
+ get_collection_link(
+ self.__get_database_name(database_name),
+ self.__get_collection_name(collection_name)),
+ query,
+ partition_key)
+
+ return list(result_iterable)
+ except HTTPFailure:
+ return None
+
+
+def get_database_link(database_id):
+ return "dbs/" + database_id
+
+
+def get_collection_link(database_id, collection_id):
+ return get_database_link(database_id) + "/colls/" + collection_id
+
+
+def get_document_link(database_id, collection_id, document_id):
+ return get_collection_link(database_id, collection_id) + "/docs/" + document_id
diff --git a/airflow/contrib/hooks/azure_data_lake_hook.py b/airflow/contrib/hooks/azure_data_lake_hook.py
new file mode 100644
index 0000000000..2178738220
--- /dev/null
+++ b/airflow/contrib/hooks/azure_data_lake_hook.py
@@ -0,0 +1,153 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from airflow.hooks.base_hook import BaseHook
+from azure.datalake.store import core, lib, multithread
+
+
+class AzureDataLakeHook(BaseHook):
+ """
+ Interacts with Azure Data Lake.
+
+ Client ID and client secret should be in user and password parameters.
+ Tenant and account name should be extra field as
+ {"tenant": "<TENANT>", "account_name": "ACCOUNT_NAME"}.
+
+ :param azure_data_lake_conn_id: Reference to the Azure Data Lake connection.
+ :type azure_data_lake_conn_id: str
+ """
+
+ def __init__(self, azure_data_lake_conn_id='azure_data_lake_default'):
+ self.conn_id = azure_data_lake_conn_id
+ self.connection = self.get_conn()
+
+ def get_conn(self):
+ """Return a AzureDLFileSystem object."""
+ conn = self.get_connection(self.conn_id)
+ service_options = conn.extra_dejson
+ self.account_name = service_options.get('account_name')
+
+ adlCreds = lib.auth(tenant_id=service_options.get('tenant'),
+ client_secret=conn.password,
+ client_id=conn.login)
+ adlsFileSystemClient = core.AzureDLFileSystem(adlCreds,
+ store_name=self.account_name)
+ adlsFileSystemClient.connect()
+ return adlsFileSystemClient
+
+ def check_for_file(self, file_path):
+ """
+ Check if a file exists on Azure Data Lake.
+
+ :param file_path: Path and name of the file.
+ :type file_path: str
+ :return: True if the file exists, False otherwise.
+ :rtype: bool
+ """
+ try:
+ files = self.connection.glob(file_path, details=False, invalidate_cache=True)
+ return len(files) == 1
+ except FileNotFoundError:
+ return False
+
+ def upload_file(self, local_path, remote_path, nthreads=64, overwrite=True,
+ buffersize=4194304, blocksize=4194304):
+ """
+ Upload a file to Azure Data Lake.
+
+ :param local_path: local path. Can be single file, directory (in which case,
+ upload recursively) or glob pattern. Recursive glob patterns using `**`
+ are not supported.
+ :type local_path: str
+ :param remote_path: Remote path to upload to; if multiple files, this is the
+ dircetory root to write within.
+ :type remote_path: str
+ :param nthreads: Number of threads to use. If None, uses the number of cores.
+ :type nthreads: int
+ :param overwrite: Whether to forcibly overwrite existing files/directories.
+ If False and remote path is a directory, will quit regardless if any files
+ would be overwritten or not. If True, only matching filenames are actually
+ overwritten.
+ :type overwrite: bool
+ :param buffersize: int [2**22]
+ Number of bytes for internal buffer. This block cannot be bigger than
+ a chunk and cannot be smaller than a block.
+ :type buffersize: int
+ :param blocksize: int [2**22]
+ Number of bytes for a block. Within each chunk, we write a smaller
+ block for each API call. This block cannot be bigger than a chunk.
+ :type blocksize: int
+ """
+ multithread.ADLUploader(self.connection,
+ lpath=local_path,
+ rpath=remote_path,
+ nthreads=nthreads,
+ overwrite=overwrite,
+ buffersize=buffersize,
+ blocksize=blocksize)
+
+ def download_file(self, local_path, remote_path, nthreads=64, overwrite=True,
+ buffersize=4194304, blocksize=4194304):
+ """
+ Download a file from Azure Blob Storage.
+
+ :param local_path: local path. If downloading a single file, will write to this
+ specific file, unless it is an existing directory, in which case a file is
+ created within it. If downloading multiple files, this is the root
+ directory to write within. Will create directories as required.
+ :type local_path: str
+ :param remote_path: remote path/globstring to use to find remote files.
+ Recursive glob patterns using `**` are not supported.
+ :type remote_path: str
+ :param nthreads: Number of threads to use. If None, uses the number of cores.
+ :type nthreads: int
+ :param overwrite: Whether to forcibly overwrite existing files/directories.
+ If False and remote path is a directory, will quit regardless if any files
+ would be overwritten or not. If True, only matching filenames are actually
+ overwritten.
+ :type overwrite: bool
+ :param buffersize: int [2**22]
+ Number of bytes for internal buffer. This block cannot be bigger than
+ a chunk and cannot be smaller than a block.
+ :type buffersize: int
+ :param blocksize: int [2**22]
+ Number of bytes for a block. Within each chunk, we write a smaller
+ block for each API call. This block cannot be bigger than a chunk.
+ :type blocksize: int
+ """
+ multithread.ADLDownloader(self.connection,
+ lpath=local_path,
+ rpath=remote_path,
+ nthreads=nthreads,
+ overwrite=overwrite,
+ buffersize=buffersize,
+ blocksize=blocksize)
+
+ def list(self, path):
+ """
+ List files in Azure Data Lake Storage
+
+ :param path: full path/globstring to use to list files in ADLS
+ :type path: str
+ """
+ if "*" in path:
+ return self.connection.glob(path)
+ else:
+ return self.connection.walk(path)
diff --git a/airflow/contrib/hooks/azure_fileshare_hook.py b/airflow/contrib/hooks/azure_fileshare_hook.py
new file mode 100644
index 0000000000..8afa1540d7
--- /dev/null
+++ b/airflow/contrib/hooks/azure_fileshare_hook.py
@@ -0,0 +1,212 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from airflow.hooks.base_hook import BaseHook
+from azure.storage.file import FileService
+
+
+class AzureFileShareHook(BaseHook):
+ """
+ Interacts with Azure FileShare Storage.
+
+ Additional options passed in the 'extra' field of the connection will be
+ passed to the `FileService()` constructor.
+
+ :param wasb_conn_id: Reference to the wasb connection.
+ :type wasb_conn_id: str
+ """
+
+ def __init__(self, wasb_conn_id='wasb_default'):
+ self.conn_id = wasb_conn_id
+ self.connection = self.get_conn()
+
+ def get_conn(self):
+ """Return the FileService object."""
+ conn = self.get_connection(self.conn_id)
+ service_options = conn.extra_dejson
+ return FileService(account_name=conn.login,
+ account_key=conn.password, **service_options)
+
+ def check_for_directory(self, share_name, directory_name, **kwargs):
+ """
+ Check if a directory exists on Azure File Share.
+
+ :param share_name: Name of the share.
+ :type share_name: str
+ :param directory_name: Name of the directory.
+ :type directory_name: str
+ :param kwargs: Optional keyword arguments that
+ `FileService.exists()` takes.
+ :type kwargs: object
+ :return: True if the file exists, False otherwise.
+ :rtype: bool
+ """
+ return self.connection.exists(share_name, directory_name,
+ **kwargs)
+
+ def check_for_file(self, share_name, directory_name, file_name, **kwargs):
+ """
+ Check if a file exists on Azure File Share.
+
+ :param share_name: Name of the share.
+ :type share_name: str
+ :param directory_name: Name of the directory.
+ :type directory_name: str
+ :param file_name: Name of the file.
+ :type file_name: str
+ :param kwargs: Optional keyword arguments that
+ `FileService.exists()` takes.
+ :type kwargs: object
+ :return: True if the file exists, False otherwise.
+ :rtype: bool
+ """
+ return self.connection.exists(share_name, directory_name,
+ file_name, **kwargs)
+
+ def list_directories_and_files(self, share_name, directory_name=None, **kwargs):
+ """
+ Return the list of directories and files stored on a Azure File Share.
+
+ :param share_name: Name of the share.
+ :type share_name: str
+ :param directory_name: Name of the directory.
+ :type directory_name: str
+ :param kwargs: Optional keyword arguments that
+ `FileService.list_directories_and_files()` takes.
+ :type kwargs: object
+ :return: A list of files and directories
+ :rtype: list
+ """
+ return self.connection.list_directories_and_files(share_name,
+ directory_name,
+ **kwargs)
+
+ def create_directory(self, share_name, directory_name, **kwargs):
+ """
+ Create a new directory on a Azure File Share.
+
+ :param share_name: Name of the share.
+ :type share_name: str
+ :param directory_name: Name of the directory.
+ :type directory_name: str
+ :param kwargs: Optional keyword arguments that
+ `FileService.create_directory()` takes.
+ :type kwargs: object
+ :return: A list of files and directories
+ :rtype: list
+ """
+ return self.connection.create_directory(share_name, directory_name, **kwargs)
+
+ def get_file(self, file_path, share_name, directory_name, file_name, **kwargs):
+ """
+ Download a file from Azure File Share.
+
+ :param file_path: Where to store the file.
+ :type file_path: str
+ :param share_name: Name of the share.
+ :type share_name: str
+ :param directory_name: Name of the directory.
+ :type directory_name: str
+ :param file_name: Name of the file.
+ :type file_name: str
+ :param kwargs: Optional keyword arguments that
+ `FileService.get_file_to_path()` takes.
+ :type kwargs: object
+ """
+ self.connection.get_file_to_path(share_name, directory_name,
+ file_name, file_path, **kwargs)
+
+ def get_file_to_stream(self, stream, share_name, directory_name, file_name, **kwargs):
+ """
+ Download a file from Azure File Share.
+
+ :param stream: A filehandle to store the file to.
+ :type stream: file-like object
+ :param share_name: Name of the share.
+ :type share_name: str
+ :param directory_name: Name of the directory.
+ :type directory_name: str
+ :param file_name: Name of the file.
+ :type file_name: str
+ :param kwargs: Optional keyword arguments that
+ `FileService.get_file_to_stream()` takes.
+ :type kwargs: object
+ """
+ self.connection.get_file_to_stream(share_name, directory_name,
+ file_name, stream, **kwargs)
+
+ def load_file(self, file_path, share_name, directory_name, file_name, **kwargs):
+ """
+ Upload a file to Azure File Share.
+
+ :param file_path: Path to the file to load.
+ :type file_path: str
+ :param share_name: Name of the share.
+ :type share_name: str
+ :param directory_name: Name of the directory.
+ :type directory_name: str
+ :param file_name: Name of the file.
+ :type file_name: str
+ :param kwargs: Optional keyword arguments that
+ `FileService.create_file_from_path()` takes.
+ :type kwargs: object
+ """
+ self.connection.create_file_from_path(share_name, directory_name,
+ file_name, file_path, **kwargs)
+
+ def load_string(self, string_data, share_name, directory_name, file_name, **kwargs):
+ """
+ Upload a string to Azure File Share.
+
+ :param string_data: String to load.
+ :type string_data: str
+ :param share_name: Name of the share.
+ :type share_name: str
+ :param directory_name: Name of the directory.
+ :type directory_name: str
+ :param file_name: Name of the file.
+ :type file_name: str
+ :param kwargs: Optional keyword arguments that
+ `FileService.create_file_from_text()` takes.
+ :type kwargs: object
+ """
+ self.connection.create_file_from_text(share_name, directory_name,
+ file_name, string_data, **kwargs)
+
+ def load_stream(self, stream, share_name, directory_name, file_name, count, **kwargs):
+ """
+ Upload a stream to Azure File Share.
+
+ :param stream: Opened file/stream to upload as the file content.
+ :type stream: file-like
+ :param share_name: Name of the share.
+ :type share_name: str
+ :param directory_name: Name of the directory.
+ :type directory_name: str
+ :param file_name: Name of the file.
+ :type file_name: str
+ :param count: Size of the stream in bytes
+ :type count: int
+ :param kwargs: Optional keyword arguments that
+ `FileService.create_file_from_stream()` takes.
+ :type kwargs: object
+ """
+ self.connection.create_file_from_stream(share_name, directory_name,
+ file_name, stream, count, **kwargs)
diff --git a/airflow/contrib/hooks/bigquery_hook.py b/airflow/contrib/hooks/bigquery_hook.py
index 4ab4ac07a9..6dabd3ea4a 100644
--- a/airflow/contrib/hooks/bigquery_hook.py
+++ b/airflow/contrib/hooks/bigquery_hook.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
"""
This module contains a BigQuery Hook, as well as a very basic PEP 249
@@ -18,7 +23,10 @@
"""
import time
+import six
from builtins import range
+from copy import deepcopy
+from six import iteritems
from past.builtins import basestring
@@ -28,10 +36,9 @@
from airflow.utils.log.logging_mixin import LoggingMixin
from apiclient.discovery import HttpError, build
from googleapiclient import errors
-from pandas.tools.merge import concat
from pandas_gbq.gbq import \
_check_google_client_version as gbq_check_google_client_version
-from pandas_gbq.gbq import _parse_data as gbq_parse_data
+from pandas_gbq import read_gbq
from pandas_gbq.gbq import \
_test_google_api_imports as gbq_test_google_api_imports
from pandas_gbq.gbq import GbqConnector
@@ -47,10 +54,12 @@ class BigQueryHook(GoogleCloudBaseHook, DbApiHook, LoggingMixin):
def __init__(self,
bigquery_conn_id='bigquery_default',
delegate_to=None,
- use_legacy_sql=True):
+ use_legacy_sql=True,
+ location=None):
super(BigQueryHook, self).__init__(
- conn_id=bigquery_conn_id, delegate_to=delegate_to)
+ gcp_conn_id=bigquery_conn_id, delegate_to=delegate_to)
self.use_legacy_sql = use_legacy_sql
+ self.location = location
def get_conn(self):
"""
@@ -61,14 +70,17 @@ def get_conn(self):
return BigQueryConnection(
service=service,
project_id=project,
- use_legacy_sql=self.use_legacy_sql)
+ use_legacy_sql=self.use_legacy_sql,
+ location=self.location,
+ )
def get_service(self):
"""
Returns a BigQuery service object.
"""
http_authorized = self._authorize()
- return build('bigquery', 'v2', http=http_authorized)
+ return build(
+ 'bigquery', 'v2', http=http_authorized, cache_discovery=False)
def insert_rows(self, table, rows, target_fields=None, commit_every=1000):
"""
@@ -78,7 +90,7 @@ def insert_rows(self, table, rows, target_fields=None, commit_every=1000):
"""
raise NotImplementedError()
- def get_pandas_df(self, bql, parameters=None, dialect=None):
+ def get_pandas_df(self, sql, parameters=None, dialect=None):
"""
Returns a Pandas DataFrame for the results produced by a BigQuery
query. The DbApiHook method must be overridden because Pandas
@@ -87,33 +99,22 @@ def get_pandas_df(self, bql, parameters=None, dialect=None):
https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L447
https://github.com/pydata/pandas/issues/6900
- :param bql: The BigQuery SQL to execute.
- :type bql: string
+ :param sql: The BigQuery SQL to execute.
+ :type sql: str
:param parameters: The parameters to render the SQL query with (not
used, leave to override superclass method)
:type parameters: mapping or iterable
:param dialect: Dialect of BigQuery SQL – legacy SQL or standard SQL
defaults to use `self.use_legacy_sql` if not specified
- :type dialect: string in {'legacy', 'standard'}
+ :type dialect: str in {'legacy', 'standard'}
"""
- service = self.get_service()
- project = self._get_field('project')
-
if dialect is None:
dialect = 'legacy' if self.use_legacy_sql else 'standard'
- connector = BigQueryPandasConnector(project, service, dialect=dialect)
- schema, pages = connector.run_query(bql)
- dataframe_list = []
-
- while len(pages) > 0:
- page = pages.pop()
- dataframe_list.append(gbq_parse_data(schema, page))
-
- if len(dataframe_list) > 0:
- return concat(dataframe_list, ignore_index=True)
- else:
- return gbq_parse_data(schema, [])
+ return read_gbq(sql,
+ project_id=self._get_field('project'),
+ dialect=dialect,
+ verbose=False)
def table_exists(self, project_id, dataset_id, table_id):
"""
@@ -122,12 +123,12 @@ def table_exists(self, project_id, dataset_id, table_id):
:param project_id: The Google cloud project in which to look for the
table. The connection supplied to the hook must provide access to
the specified project.
- :type project_id: string
+ :type project_id: str
:param dataset_id: The name of the dataset in which to look for the
table.
- :type dataset_id: string
+ :type dataset_id: str
:param table_id: The name of the table to check the existence of.
- :type table_id: string
+ :type table_id: str
"""
service = self.get_service()
try:
@@ -201,22 +202,325 @@ class BigQueryBaseCursor(LoggingMixin):
PEP 249 cursor isn't needed.
"""
- def __init__(self, service, project_id, use_legacy_sql=True):
+ def __init__(self,
+ service,
+ project_id,
+ use_legacy_sql=True,
+ api_resource_configs=None,
+ location=None):
+
self.service = service
self.project_id = project_id
self.use_legacy_sql = use_legacy_sql
+ if api_resource_configs:
+ _validate_value("api_resource_configs", api_resource_configs, dict)
+ self.api_resource_configs = api_resource_configs \
+ if api_resource_configs else {}
self.running_job_id = None
+ self.location = location
+
+ def create_empty_table(self,
+ project_id,
+ dataset_id,
+ table_id,
+ schema_fields=None,
+ time_partitioning=None,
+ labels=None,
+ view=None):
+ """
+ Creates a new, empty table in the dataset.
+ To create a view, which is defined by a SQL query, parse a dictionary to 'view' kwarg
+
+ :param project_id: The project to create the table into.
+ :type project_id: str
+ :param dataset_id: The dataset to create the table into.
+ :type dataset_id: str
+ :param table_id: The Name of the table to be created.
+ :type table_id: str
+ :param schema_fields: If set, the schema field list as defined here:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema
+ :type schema_fields: list
+ :param labels: a dictionary containing labels for the table, passed to BigQuery
+ :type labels: dict
+
+ **Example**: ::
+
+ schema_fields=[{"name": "emp_name", "type": "STRING", "mode": "REQUIRED"},
+ {"name": "salary", "type": "INTEGER", "mode": "NULLABLE"}]
+
+ :param time_partitioning: configure optional time partitioning fields i.e.
+ partition by field, type and expiration as per API specifications.
+
+ .. seealso::
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#timePartitioning
+ :type time_partitioning: dict
+ :param view: [Optional] A dictionary containing definition for the view.
+ If set, it will create a view instead of a table:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#view
+ :type view: dict
+
+ **Example**: ::
+
+ view = {
+ "query": "SELECT * FROM `test-project-id.test_dataset_id.test_table_prefix*` LIMIT 1000",
+ "useLegacySql": False
+ }
+
+ :return:
+ """
+
+ project_id = project_id if project_id is not None else self.project_id
+
+ table_resource = {
+ 'tableReference': {
+ 'tableId': table_id
+ }
+ }
+
+ if schema_fields:
+ table_resource['schema'] = {'fields': schema_fields}
+
+ if time_partitioning:
+ table_resource['timePartitioning'] = time_partitioning
+
+ if labels:
+ table_resource['labels'] = labels
+
+ if view:
+ table_resource['view'] = view
+
+ self.log.info('Creating Table %s:%s.%s',
+ project_id, dataset_id, table_id)
+
+ try:
+ self.service.tables().insert(
+ projectId=project_id,
+ datasetId=dataset_id,
+ body=table_resource).execute()
+
+ self.log.info('Table created successfully: %s:%s.%s',
+ project_id, dataset_id, table_id)
+
+ except HttpError as err:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(err.content)
+ )
+
+ def create_external_table(self,
+ external_project_dataset_table,
+ schema_fields,
+ source_uris,
+ source_format='CSV',
+ autodetect=False,
+ compression='NONE',
+ ignore_unknown_values=False,
+ max_bad_records=0,
+ skip_leading_rows=0,
+ field_delimiter=',',
+ quote_character=None,
+ allow_quoted_newlines=False,
+ allow_jagged_rows=False,
+ src_fmt_configs=None,
+ labels=None
+ ):
+ """
+ Creates a new external table in the dataset with the data in Google
+ Cloud Storage. See here:
+
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#resource
+
+ for more details about these parameters.
+
+ :param external_project_dataset_table:
+ The dotted (<project>.|<project>:)<dataset>.<table>($<partition>) BigQuery
+ table name to create external table.
+ If <project> is not included, project will be the
+ project defined in the connection json.
+ :type external_project_dataset_table: str
+ :param schema_fields: The schema field list as defined here:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#resource
+ :type schema_fields: list
+ :param source_uris: The source Google Cloud
+ Storage URI (e.g. gs://some-bucket/some-file.txt). A single wild
+ per-object name can be used.
+ :type source_uris: list
+ :param source_format: File format to export.
+ :type source_format: str
+ :param autodetect: Try to detect schema and format options automatically.
+ Any option specified explicitly will be honored.
+ :type autodetect: bool
+ :param compression: [Optional] The compression type of the data source.
+ Possible values include GZIP and NONE.
+ The default value is NONE.
+ This setting is ignored for Google Cloud Bigtable,
+ Google Cloud Datastore backups and Avro formats.
+ :type compression: str
+ :param ignore_unknown_values: [Optional] Indicates if BigQuery should allow
+ extra values that are not represented in the table schema.
+ If true, the extra values are ignored. If false, records with extra columns
+ are treated as bad records, and if there are too many bad records, an
+ invalid error is returned in the job result.
+ :type ignore_unknown_values: bool
+ :param max_bad_records: The maximum number of bad records that BigQuery can
+ ignore when running the job.
+ :type max_bad_records: int
+ :param skip_leading_rows: Number of rows to skip when loading from a CSV.
+ :type skip_leading_rows: int
+ :param field_delimiter: The delimiter to use when loading from a CSV.
+ :type field_delimiter: str
+ :param quote_character: The value that is used to quote data sections in a CSV
+ file.
+ :type quote_character: str
+ :param allow_quoted_newlines: Whether to allow quoted newlines (true) or not
+ (false).
+ :type allow_quoted_newlines: bool
+ :param allow_jagged_rows: Accept rows that are missing trailing optional columns.
+ The missing values are treated as nulls. If false, records with missing
+ trailing columns are treated as bad records, and if there are too many bad
+ records, an invalid error is returned in the job result. Only applicable when
+ soure_format is CSV.
+ :type allow_jagged_rows: bool
+ :param src_fmt_configs: configure optional fields specific to the source format
+ :type src_fmt_configs: dict
+ :param labels: a dictionary containing labels for the table, passed to BigQuery
+ :type labels: dict
+ """
+
+ if src_fmt_configs is None:
+ src_fmt_configs = {}
+ project_id, dataset_id, external_table_id = \
+ _split_tablename(table_input=external_project_dataset_table,
+ default_project_id=self.project_id,
+ var_name='external_project_dataset_table')
+
+ # bigquery only allows certain source formats
+ # we check to make sure the passed source format is valid
+ # if it's not, we raise a ValueError
+ # Refer to this link for more details:
+ # https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#externalDataConfiguration.sourceFormat
+
+ source_format = source_format.upper()
+ allowed_formats = [
+ "CSV", "NEWLINE_DELIMITED_JSON", "AVRO", "GOOGLE_SHEETS",
+ "DATASTORE_BACKUP", "PARQUET"
+ ]
+ if source_format not in allowed_formats:
+ raise ValueError("{0} is not a valid source format. "
+ "Please use one of the following types: {1}"
+ .format(source_format, allowed_formats))
+
+ compression = compression.upper()
+ allowed_compressions = ['NONE', 'GZIP']
+ if compression not in allowed_compressions:
+ raise ValueError("{0} is not a valid compression format. "
+ "Please use one of the following types: {1}"
+ .format(compression, allowed_compressions))
+
+ table_resource = {
+ 'externalDataConfiguration': {
+ 'autodetect': autodetect,
+ 'sourceFormat': source_format,
+ 'sourceUris': source_uris,
+ 'compression': compression,
+ 'ignoreUnknownValues': ignore_unknown_values
+ },
+ 'tableReference': {
+ 'projectId': project_id,
+ 'datasetId': dataset_id,
+ 'tableId': external_table_id,
+ }
+ }
+
+ if schema_fields:
+ table_resource['externalDataConfiguration'].update({
+ 'schema': {
+ 'fields': schema_fields
+ }
+ })
+
+ self.log.info('Creating external table: %s', external_project_dataset_table)
+
+ if max_bad_records:
+ table_resource['externalDataConfiguration']['maxBadRecords'] = max_bad_records
+
+ # if following fields are not specified in src_fmt_configs,
+ # honor the top-level params for backward-compatibility
+ if 'skipLeadingRows' not in src_fmt_configs:
+ src_fmt_configs['skipLeadingRows'] = skip_leading_rows
+ if 'fieldDelimiter' not in src_fmt_configs:
+ src_fmt_configs['fieldDelimiter'] = field_delimiter
+ if 'quote_character' not in src_fmt_configs:
+ src_fmt_configs['quote'] = quote_character
+ if 'allowQuotedNewlines' not in src_fmt_configs:
+ src_fmt_configs['allowQuotedNewlines'] = allow_quoted_newlines
+ if 'allowJaggedRows' not in src_fmt_configs:
+ src_fmt_configs['allowJaggedRows'] = allow_jagged_rows
+
+ src_fmt_to_param_mapping = {
+ 'CSV': 'csvOptions',
+ 'GOOGLE_SHEETS': 'googleSheetsOptions'
+ }
+
+ src_fmt_to_configs_mapping = {
+ 'csvOptions': [
+ 'allowJaggedRows', 'allowQuotedNewlines',
+ 'fieldDelimiter', 'skipLeadingRows',
+ 'quote'
+ ],
+ 'googleSheetsOptions': ['skipLeadingRows']
+ }
+
+ if source_format in src_fmt_to_param_mapping.keys():
+
+ valid_configs = src_fmt_to_configs_mapping[
+ src_fmt_to_param_mapping[source_format]
+ ]
+
+ src_fmt_configs = {
+ k: v
+ for k, v in src_fmt_configs.items() if k in valid_configs
+ }
+
+ table_resource['externalDataConfiguration'][src_fmt_to_param_mapping[
+ source_format]] = src_fmt_configs
+
+ if labels:
+ table_resource['labels'] = labels
+
+ try:
+ self.service.tables().insert(
+ projectId=project_id,
+ datasetId=dataset_id,
+ body=table_resource
+ ).execute()
+
+ self.log.info('External table created successfully: %s',
+ external_project_dataset_table)
+
+ except HttpError as err:
+ raise Exception(
+ 'BigQuery job failed. Error was: {}'.format(err.content)
+ )
def run_query(self,
- bql,
- destination_dataset_table=False,
+ sql,
+ destination_dataset_table=None,
write_disposition='WRITE_EMPTY',
allow_large_results=False,
- udf_config=False,
+ flatten_results=None,
+ udf_config=None,
+ use_legacy_sql=None,
maximum_billing_tier=None,
+ maximum_bytes_billed=None,
create_disposition='CREATE_IF_NEEDED',
query_params=None,
- schema_update_options=()):
+ labels=None,
+ schema_update_options=(),
+ priority='INTERACTIVE',
+ time_partitioning=None,
+ api_resource_configs=None,
+ cluster_fields=None,
+ location=None):
"""
Executes a BigQuery SQL query. Optionally persists results in a BigQuery
table. See here:
@@ -225,99 +529,197 @@ def run_query(self,
For more details about these parameters.
- :param bql: The BigQuery SQL to execute.
- :type bql: string
+ :param sql: The BigQuery SQL to execute.
+ :type sql: str
:param destination_dataset_table: The dotted <dataset>.<table>
BigQuery table to save the query results.
+ :type destination_dataset_table: str
:param write_disposition: What to do if the table already exists in
BigQuery.
- :type write_disposition: string
+ :type write_disposition: str
:param allow_large_results: Whether to allow large results.
- :type allow_large_results: boolean
+ :type allow_large_results: bool
+ :param flatten_results: If true and query uses legacy SQL dialect, flattens
+ all nested and repeated fields in the query results. ``allowLargeResults``
+ must be true if this is set to false. For standard SQL queries, this
+ flag is ignored and results are never flattened.
+ :type flatten_results: bool
:param udf_config: The User Defined Function configuration for the query.
See https://cloud.google.com/bigquery/user-defined-functions for details.
:type udf_config: list
+ :param use_legacy_sql: Whether to use legacy SQL (true) or standard SQL (false).
+ If `None`, defaults to `self.use_legacy_sql`.
+ :type use_legacy_sql: bool
+ :param api_resource_configs: a dictionary that contain params
+ 'configuration' applied for Google BigQuery Jobs API:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs
+ for example, {'query': {'useQueryCache': False}}. You could use it
+ if you need to provide some params that are not supported by the
+ BigQueryHook like args.
+ :type api_resource_configs: dict
:param maximum_billing_tier: Positive integer that serves as a
multiplier of the basic price.
- :type maximum_billing_tier: integer
+ :type maximum_billing_tier: int
+ :param maximum_bytes_billed: Limits the bytes billed for this job.
+ Queries that will have bytes billed beyond this limit will fail
+ (without incurring a charge). If unspecified, this will be
+ set to your project default.
+ :type maximum_bytes_billed: float
:param create_disposition: Specifies whether the job is allowed to
create new tables.
- :type create_disposition: string
+ :type create_disposition: str
:param query_params a dictionary containing query parameter types and
values, passed to BigQuery
:type query_params: dict
- :param schema_update_options: Allows the schema of the desitination
+ :param labels a dictionary containing labels for the job/query,
+ passed to BigQuery
+ :type labels: dict
+ :param schema_update_options: Allows the schema of the destination
table to be updated as a side effect of the query job.
:type schema_update_options: tuple
+ :param priority: Specifies a priority for the query.
+ Possible values include INTERACTIVE and BATCH.
+ The default value is INTERACTIVE.
+ :type priority: str
+ :param time_partitioning: configure optional time partitioning fields i.e.
+ partition by field, type and expiration as per API specifications.
+ :type time_partitioning: dict
+ :param cluster_fields: Request that the result of this query be stored sorted
+ by one or more columns. This is only available in combination with
+ time_partitioning. The order of columns given determines the sort order.
+ :type cluster_fields: list of str
+ :param location: The geographic location of the job. Required except for
+ US and EU. See details at
+ https://cloud.google.com/bigquery/docs/locations#specifying_your_location
+ :type location: str
"""
+ if time_partitioning is None:
+ time_partitioning = {}
+
+ if location:
+ self.location = location
+
+ if not api_resource_configs:
+ api_resource_configs = self.api_resource_configs
+ else:
+ _validate_value('api_resource_configs',
+ api_resource_configs, dict)
+ configuration = deepcopy(api_resource_configs)
+ if 'query' not in configuration:
+ configuration['query'] = {}
+
+ else:
+ _validate_value("api_resource_configs['query']",
+ configuration['query'], dict)
+
+ if sql is None and not configuration['query'].get('query', None):
+ raise TypeError('`BigQueryBaseCursor.run_query` '
+ 'missing 1 required positional argument: `sql`')
+
# BigQuery also allows you to define how you want a table's schema to change
# as a side effect of a query job
# for more details:
# https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.schemaUpdateOptions
+
allowed_schema_update_options = [
'ALLOW_FIELD_ADDITION', "ALLOW_FIELD_RELAXATION"
]
- if not set(allowed_schema_update_options).issuperset(
- set(schema_update_options)):
- raise ValueError(
- "{0} contains invalid schema update options. "
- "Please only use one or more of the following options: {1}"
- .format(schema_update_options, allowed_schema_update_options))
- configuration = {
- 'query': {
- 'query': bql,
- 'useLegacySql': self.use_legacy_sql,
- 'maximumBillingTier': maximum_billing_tier
- }
- }
+ if not set(allowed_schema_update_options
+ ).issuperset(set(schema_update_options)):
+ raise ValueError("{0} contains invalid schema update options. "
+ "Please only use one or more of the following "
+ "options: {1}"
+ .format(schema_update_options,
+ allowed_schema_update_options))
+
+ if schema_update_options:
+ if write_disposition not in ["WRITE_APPEND", "WRITE_TRUNCATE"]:
+ raise ValueError("schema_update_options is only "
+ "allowed if write_disposition is "
+ "'WRITE_APPEND' or 'WRITE_TRUNCATE'.")
if destination_dataset_table:
- assert '.' in destination_dataset_table, (
- 'Expected destination_dataset_table in the format of '
- '<dataset>.<table>. Got: {}').format(destination_dataset_table)
destination_project, destination_dataset, destination_table = \
_split_tablename(table_input=destination_dataset_table,
default_project_id=self.project_id)
- configuration['query'].update({
- 'allowLargeResults':
- allow_large_results,
- 'writeDisposition':
- write_disposition,
- 'createDisposition':
- create_disposition,
- 'destinationTable': {
- 'projectId': destination_project,
- 'datasetId': destination_dataset,
- 'tableId': destination_table,
- }
- })
- if udf_config:
- assert isinstance(udf_config, list)
- configuration['query'].update({
- 'userDefinedFunctionResources':
- udf_config
- })
- if query_params:
- if self.use_legacy_sql:
- raise ValueError("Query paramaters are not allowed when using "
- "legacy SQL")
- else:
- configuration['query']['queryParameters'] = query_params
+ destination_dataset_table = {
+ 'projectId': destination_project,
+ 'datasetId': destination_dataset,
+ 'tableId': destination_table,
+ }
- if schema_update_options:
- if write_disposition not in ["WRITE_APPEND", "WRITE_TRUNCATE"]:
- raise ValueError("schema_update_options is only "
- "allowed if write_disposition is "
- "'WRITE_APPEND' or 'WRITE_TRUNCATE'.")
- else:
- self.log.info(
- "Adding experimental "
- "'schemaUpdateOptions': {0}".format(schema_update_options))
- configuration['query'][
- 'schemaUpdateOptions'] = schema_update_options
+ if cluster_fields:
+ cluster_fields = {'fields': cluster_fields}
+
+ query_param_list = [
+ (sql, 'query', None, six.string_types),
+ (priority, 'priority', 'INTERACTIVE', six.string_types),
+ (use_legacy_sql, 'useLegacySql', self.use_legacy_sql, bool),
+ (query_params, 'queryParameters', None, dict),
+ (udf_config, 'userDefinedFunctionResources', None, list),
+ (maximum_billing_tier, 'maximumBillingTier', None, int),
+ (maximum_bytes_billed, 'maximumBytesBilled', None, float),
+ (time_partitioning, 'timePartitioning', {}, dict),
+ (schema_update_options, 'schemaUpdateOptions', None, tuple),
+ (destination_dataset_table, 'destinationTable', None, dict),
+ (cluster_fields, 'clustering', None, dict),
+ ]
+
+ for param_tuple in query_param_list:
+
+ param, param_name, param_default, param_type = param_tuple
+
+ if param_name not in configuration['query'] and param in [None, {}, ()]:
+ if param_name == 'timePartitioning':
+ param_default = _cleanse_time_partitioning(
+ destination_dataset_table, time_partitioning)
+ param = param_default
+
+ if param not in [None, {}, ()]:
+ _api_resource_configs_duplication_check(
+ param_name, param, configuration['query'])
+
+ configuration['query'][param_name] = param
+
+ # check valid type of provided param,
+ # it last step because we can get param from 2 sources,
+ # and first of all need to find it
+
+ _validate_value(param_name, configuration['query'][param_name],
+ param_type)
+
+ if param_name == 'schemaUpdateOptions' and param:
+ self.log.info("Adding experimental 'schemaUpdateOptions': "
+ "{0}".format(schema_update_options))
+
+ if param_name == 'destinationTable':
+ for key in ['projectId', 'datasetId', 'tableId']:
+ if key not in configuration['query']['destinationTable']:
+ raise ValueError(
+ "Not correct 'destinationTable' in "
+ "api_resource_configs. 'destinationTable' "
+ "must be a dict with {'projectId':'', "
+ "'datasetId':'', 'tableId':''}")
+
+ configuration['query'].update({
+ 'allowLargeResults': allow_large_results,
+ 'flattenResults': flatten_results,
+ 'writeDisposition': write_disposition,
+ 'createDisposition': create_disposition,
+ })
+
+ if 'useLegacySql' in configuration['query'] and \
+ 'queryParameters' in configuration['query']:
+ raise ValueError("Query parameters are not allowed "
+ "when using legacy SQL")
+
+ if labels:
+ _api_resource_configs_duplication_check(
+ 'labels', labels, configuration)
+ configuration['labels'] = labels
return self.run_with_configuration(configuration)
@@ -328,7 +730,8 @@ def run_extract( # noqa
compression='NONE',
export_format='CSV',
field_delimiter=',',
- print_header=True):
+ print_header=True,
+ labels=None):
"""
Executes a BigQuery extract command to copy data from BigQuery to
Google Cloud Storage. See here:
@@ -339,20 +742,23 @@ def run_extract( # noqa
:param source_project_dataset_table: The dotted <dataset>.<table>
BigQuery table to use as the source data.
- :type source_project_dataset_table: string
+ :type source_project_dataset_table: str
:param destination_cloud_storage_uris: The destination Google Cloud
Storage URI (e.g. gs://some-bucket/some-file.txt). Follows
convention defined here:
https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple
:type destination_cloud_storage_uris: list
:param compression: Type of compression to use.
- :type compression: string
+ :type compression: str
:param export_format: File format to export.
- :type export_format: string
+ :type export_format: str
:param field_delimiter: The delimiter to use when extracting to a CSV.
- :type field_delimiter: string
+ :type field_delimiter: str
:param print_header: Whether to print a header for a CSV file extract.
- :type print_header: boolean
+ :type print_header: bool
+ :param labels: a dictionary containing labels for the job/query,
+ passed to BigQuery
+ :type labels: dict
"""
source_project, source_dataset, source_table = \
@@ -373,6 +779,9 @@ def run_extract( # noqa
}
}
+ if labels:
+ configuration['labels'] = labels
+
if export_format == 'CSV':
# Only set fieldDelimiter and printHeader fields if using CSV.
# Google does not like it if you set these fields for other export
@@ -386,7 +795,8 @@ def run_copy(self,
source_project_dataset_tables,
destination_project_dataset_table,
write_disposition='WRITE_EMPTY',
- create_disposition='CREATE_IF_NEEDED'):
+ create_disposition='CREATE_IF_NEEDED',
+ labels=None):
"""
Executes a BigQuery copy command to copy data from one BigQuery table
to another. See here:
@@ -404,11 +814,14 @@ def run_copy(self,
:type source_project_dataset_tables: list|string
:param destination_project_dataset_table: The destination BigQuery
table. Format is: (project:|project.)<dataset>.<table>
- :type destination_project_dataset_table: string
+ :type destination_project_dataset_table: str
:param write_disposition: The write disposition if the table already exists.
- :type write_disposition: string
+ :type write_disposition: str
:param create_disposition: The create disposition if the table doesn't exist.
- :type create_disposition: string
+ :type create_disposition: str
+ :param labels a dictionary containing labels for the job/query,
+ passed to BigQuery
+ :type labels: dict
"""
source_project_dataset_tables = ([
source_project_dataset_tables
@@ -446,12 +859,15 @@ def run_copy(self,
}
}
+ if labels:
+ configuration['labels'] = labels
+
return self.run_with_configuration(configuration)
def run_load(self,
destination_project_dataset_table,
- schema_fields,
source_uris,
+ schema_fields=None,
source_format='CSV',
create_disposition='CREATE_IF_NEEDED',
skip_leading_rows=0,
@@ -459,11 +875,14 @@ def run_load(self,
field_delimiter=',',
max_bad_records=0,
quote_character=None,
+ ignore_unknown_values=False,
allow_quoted_newlines=False,
allow_jagged_rows=False,
schema_update_options=(),
- src_fmt_configs={},
- time_partitioning={}):
+ src_fmt_configs=None,
+ time_partitioning=None,
+ cluster_fields=None,
+ autodetect=False):
"""
Executes a BigQuery load command to load data from Google Cloud Storage
to BigQuery. See here:
@@ -478,33 +897,43 @@ def run_load(self,
project defined in the connection json. If a partition is specified the
operator will automatically append the data, create a new partition or create
a new DAY partitioned table.
- :type destination_project_dataset_table: string
+ :type destination_project_dataset_table: str
:param schema_fields: The schema field list as defined here:
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load
+ Required if autodetect=False; optional if autodetect=True.
:type schema_fields: list
+ :param autodetect: Attempt to autodetect the schema for CSV and JSON
+ source files.
+ :type autodetect: bool
:param source_uris: The source Google Cloud
Storage URI (e.g. gs://some-bucket/some-file.txt). A single wild
per-object name can be used.
:type source_uris: list
:param source_format: File format to export.
- :type source_format: string
+ :type source_format: str
:param create_disposition: The create disposition if the table doesn't exist.
- :type create_disposition: string
+ :type create_disposition: str
:param skip_leading_rows: Number of rows to skip when loading from a CSV.
:type skip_leading_rows: int
:param write_disposition: The write disposition if the table already exists.
- :type write_disposition: string
+ :type write_disposition: str
:param field_delimiter: The delimiter to use when loading from a CSV.
- :type field_delimiter: string
+ :type field_delimiter: str
:param max_bad_records: The maximum number of bad records that BigQuery can
ignore when running the job.
:type max_bad_records: int
:param quote_character: The value that is used to quote data sections in a CSV
file.
- :type quote_character: string
+ :type quote_character: str
+ :param ignore_unknown_values: [Optional] Indicates if BigQuery should allow
+ extra values that are not represented in the table schema.
+ If true, the extra values are ignored. If false, records with extra columns
+ are treated as bad records, and if there are too many bad records, an
+ invalid error is returned in the job result.
+ :type ignore_unknown_values: bool
:param allow_quoted_newlines: Whether to allow quoted newlines (true) or not
(false).
- :type allow_quoted_newlines: boolean
+ :type allow_quoted_newlines: bool
:param allow_jagged_rows: Accept rows that are missing trailing optional columns.
The missing values are treated as nulls. If false, records with missing
trailing columns are treated as bad records, and if there are too many bad
@@ -517,10 +946,12 @@ def run_load(self,
:param src_fmt_configs: configure optional fields specific to the source format
:type src_fmt_configs: dict
:param time_partitioning: configure optional time partitioning fields i.e.
- partition by field, type and
- expiration as per API specifications. Note that 'field' is not available in
- concurrency with dataset.table$partition.
+ partition by field, type and expiration as per API specifications.
:type time_partitioning: dict
+ :param cluster_fields: Request that the result of this load be stored sorted
+ by one or more columns. This is only available in combination with
+ time_partitioning. The order of columns given determines the sort order.
+ :type cluster_fields: list of str
"""
# bigquery only allows certain source formats
@@ -528,10 +959,18 @@ def run_load(self,
# if it's not, we raise a ValueError
# Refer to this link for more details:
# https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).sourceFormat
+
+ if schema_fields is None and not autodetect:
+ raise ValueError(
+ 'You must either pass a schema or autodetect=True.')
+
+ if src_fmt_configs is None:
+ src_fmt_configs = {}
+
source_format = source_format.upper()
allowed_formats = [
"CSV", "NEWLINE_DELIMITED_JSON", "AVRO", "GOOGLE_SHEETS",
- "DATASTORE_BACKUP"
+ "DATASTORE_BACKUP", "PARQUET"
]
if source_format not in allowed_formats:
raise ValueError("{0} is not a valid source format. "
@@ -548,7 +987,7 @@ def run_load(self,
if not set(allowed_schema_update_options).issuperset(
set(schema_update_options)):
raise ValueError(
- "{0} contains invalid schema update options. "
+ "{0} contains invalid schema update options."
"Please only use one or more of the following options: {1}"
.format(schema_update_options, allowed_schema_update_options))
@@ -559,6 +998,7 @@ def run_load(self,
configuration = {
'load': {
+ 'autodetect': autodetect,
'createDisposition': create_disposition,
'destinationTable': {
'projectId': destination_project,
@@ -568,24 +1008,21 @@ def run_load(self,
'sourceFormat': source_format,
'sourceUris': source_uris,
'writeDisposition': write_disposition,
+ 'ignoreUnknownValues': ignore_unknown_values
}
}
- # if it is a partitioned table ($ is in the table name) add partition load option
- if '$' in destination_project_dataset_table:
- if time_partitioning.get('field'):
- raise AirflowException(
- "Cannot specify field partition and partition name "
- "(dataset.table$partition) at the same time"
- )
- configuration['load']['timePartitioning'] = dict(type='DAY')
-
- # can specify custom time partitioning options based on a field, or adding
- # expiration
+ time_partitioning = _cleanse_time_partitioning(
+ destination_project_dataset_table,
+ time_partitioning
+ )
if time_partitioning:
- if not configuration.get('load', {}).get('timePartitioning'):
- configuration['load']['timePartitioning'] = {}
- configuration['load']['timePartitioning'].update(time_partitioning)
+ configuration['load'].update({
+ 'timePartitioning': time_partitioning
+ })
+
+ if cluster_fields:
+ configuration['load'].update({'clustering': {'fields': cluster_fields}})
if schema_fields:
configuration['load']['schema'] = {'fields': schema_fields}
@@ -611,7 +1048,9 @@ def run_load(self,
src_fmt_configs['skipLeadingRows'] = skip_leading_rows
if 'fieldDelimiter' not in src_fmt_configs:
src_fmt_configs['fieldDelimiter'] = field_delimiter
- if quote_character:
+ if 'ignoreUnknownValues' not in src_fmt_configs:
+ src_fmt_configs['ignoreUnknownValues'] = ignore_unknown_values
+ if quote_character is not None:
src_fmt_configs['quote'] = quote_character
if allow_quoted_newlines:
src_fmt_configs['allowQuotedNewlines'] = allow_quoted_newlines
@@ -624,6 +1063,7 @@ def run_load(self,
],
'DATASTORE_BACKUP': ['projectionFields'],
'NEWLINE_DELIMITED_JSON': ['autodetect', 'ignoreUnknownValues'],
+ 'PARQUET': ['autodetect', 'ignoreUnknownValues'],
'AVRO': [],
}
valid_configs = src_fmt_to_configs_mapping[source_format]
@@ -662,12 +1102,18 @@ def run_with_configuration(self, configuration):
# Wait for query to finish.
keep_polling_job = True
- while (keep_polling_job):
+ while keep_polling_job:
try:
- job = jobs.get(
- projectId=self.project_id,
- jobId=self.running_job_id).execute()
- if (job['status']['state'] == 'DONE'):
+ if self.location:
+ job = jobs.get(
+ projectId=self.project_id,
+ jobId=self.running_job_id,
+ location=self.location).execute()
+ else:
+ job = jobs.get(
+ projectId=self.project_id,
+ jobId=self.running_job_id).execute()
+ if job['status']['state'] == 'DONE':
keep_polling_job = False
# Check if job had errors.
if 'errorResult' in job['status']:
@@ -695,8 +1141,14 @@ def run_with_configuration(self, configuration):
def poll_job_complete(self, job_id):
jobs = self.service.jobs()
try:
- job = jobs.get(projectId=self.project_id, jobId=job_id).execute()
- if (job['status']['state'] == 'DONE'):
+ if self.location:
+ job = jobs.get(projectId=self.project_id,
+ jobId=job_id,
+ location=self.location).execute()
+ else:
+ job = jobs.get(projectId=self.project_id,
+ jobId=job_id).execute()
+ if job['status']['state'] == 'DONE':
return True
except HttpError as err:
if err.resp.status in [500, 503]:
@@ -718,9 +1170,15 @@ def cancel_query(self):
not self.poll_job_complete(self.running_job_id)):
self.log.info('Attempting to cancel job : %s, %s', self.project_id,
self.running_job_id)
- jobs.cancel(
- projectId=self.project_id,
- jobId=self.running_job_id).execute()
+ if self.location:
+ jobs.cancel(
+ projectId=self.project_id,
+ jobId=self.running_job_id,
+ location=self.location).execute()
+ else:
+ jobs.cancel(
+ projectId=self.project_id,
+ jobId=self.running_job_id).execute()
else:
self.log.info('No running BigQuery jobs to cancel.')
return
@@ -730,13 +1188,13 @@ def cancel_query(self):
polling_attempts = 0
job_complete = False
- while (polling_attempts < max_polling_attempts and not job_complete):
+ while polling_attempts < max_polling_attempts and not job_complete:
polling_attempts = polling_attempts + 1
job_complete = self.poll_job_complete(self.running_job_id)
- if (job_complete):
+ if job_complete:
self.log.info('Job successfully canceled: %s, %s',
self.project_id, self.running_job_id)
- elif (polling_attempts == max_polling_attempts):
+ elif polling_attempts == max_polling_attempts:
self.log.info(
"Stopping polling due to timeout. Job with id %s "
"has not completed cancel and may or may not finish.",
@@ -805,13 +1263,9 @@ def run_table_delete(self, deletion_dataset_table,
:type deletion_dataset_table: str
:param ignore_if_missing: if True, then return success even if the
requested table does not exist.
- :type ignore_if_missing: boolean
+ :type ignore_if_missing: bool
:return:
"""
-
- assert '.' in deletion_dataset_table, (
- 'Expected deletion_dataset_table in the format of '
- '<dataset>.<table>. Got: {}').format(deletion_dataset_table)
deletion_project, deletion_dataset, deletion_table = \
_split_tablename(table_input=deletion_dataset_table,
default_project_id=self.project_id)
@@ -939,10 +1393,253 @@ def run_grant_dataset_view_access(self,
# if view is already in access, do nothing.
self.log.info(
'Table %s:%s.%s already has authorized view access to %s:%s dataset.',
- view_project, view_dataset, view_table, source_project,
- source_dataset)
+ view_project, view_dataset, view_table, source_project, source_dataset)
return source_dataset_resource
+ def create_empty_dataset(self, dataset_id="", project_id="",
+ dataset_reference=None):
+ """
+ Create a new empty dataset:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert
+
+ :param project_id: The name of the project where we want to create
+ an empty a dataset. Don't need to provide, if projectId in dataset_reference.
+ :type project_id: str
+ :param dataset_id: The id of dataset. Don't need to provide,
+ if datasetId in dataset_reference.
+ :type dataset_id: str
+ :param dataset_reference: Dataset reference that could be provided
+ with request body. More info:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
+ :type dataset_reference: dict
+ """
+
+ if dataset_reference:
+ _validate_value('dataset_reference', dataset_reference, dict)
+ else:
+ dataset_reference = {}
+
+ if "datasetReference" not in dataset_reference:
+ dataset_reference["datasetReference"] = {}
+
+ if not dataset_reference["datasetReference"].get("datasetId") and not dataset_id:
+ raise ValueError(
+ "{} not provided datasetId. Impossible to create dataset")
+
+ dataset_required_params = [(dataset_id, "datasetId", ""),
+ (project_id, "projectId", self.project_id)]
+ for param_tuple in dataset_required_params:
+ param, param_name, param_default = param_tuple
+ if param_name not in dataset_reference['datasetReference']:
+ if param_default and not param:
+ self.log.info("{} was not specified. Will be used default "
+ "value {}.".format(param_name,
+ param_default))
+ param = param_default
+ dataset_reference['datasetReference'].update(
+ {param_name: param})
+ elif param:
+ _api_resource_configs_duplication_check(
+ param_name, param,
+ dataset_reference['datasetReference'], 'dataset_reference')
+
+ dataset_id = dataset_reference.get("datasetReference").get("datasetId")
+ dataset_project_id = dataset_reference.get("datasetReference").get(
+ "projectId")
+
+ self.log.info('Creating Dataset: %s in project: %s ', dataset_id,
+ dataset_project_id)
+
+ try:
+ self.service.datasets().insert(
+ projectId=dataset_project_id,
+ body=dataset_reference).execute()
+ self.log.info('Dataset created successfully: In project %s '
+ 'Dataset %s', dataset_project_id, dataset_id)
+
+ except HttpError as err:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(err.content)
+ )
+
+ def delete_dataset(self, project_id, dataset_id):
+ """
+ Delete a dataset of Big query in your project.
+ :param project_id: The name of the project where we have the dataset .
+ :type project_id: str
+ :param dataset_id: The dataset to be delete.
+ :type dataset_id: str
+ :return:
+ """
+ project_id = project_id if project_id is not None else self.project_id
+ self.log.info('Deleting from project: %s Dataset:%s',
+ project_id, dataset_id)
+
+ try:
+ self.service.datasets().delete(
+ projectId=project_id,
+ datasetId=dataset_id).execute()
+ self.log.info('Dataset deleted successfully: In project %s '
+ 'Dataset %s', project_id, dataset_id)
+
+ except HttpError as err:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(err.content)
+ )
+
+ def get_dataset(self, dataset_id, project_id=None):
+ """
+ Method returns dataset_resource if dataset exist
+ and raised 404 error if dataset does not exist
+
+ :param dataset_id: The BigQuery Dataset ID
+ :type dataset_id: str
+ :param project_id: The GCP Project ID
+ :type project_id: str
+ :return: dataset_resource
+
+ .. seealso::
+ For more information, see Dataset Resource content:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
+ """
+
+ if not dataset_id or not isinstance(dataset_id, str):
+ raise ValueError("dataset_id argument must be provided and has "
+ "a type 'str'. You provided: {}".format(dataset_id))
+
+ dataset_project_id = project_id if project_id else self.project_id
+
+ try:
+ dataset_resource = self.service.datasets().get(
+ datasetId=dataset_id, projectId=dataset_project_id).execute()
+ self.log.info("Dataset Resource: {}".format(dataset_resource))
+ except HttpError as err:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(err.content))
+
+ return dataset_resource
+
+ def get_datasets_list(self, project_id=None):
+ """
+ Method returns full list of BigQuery datasets in the current project
+
+ .. seealso::
+ For more information, see:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/list
+
+ :param project_id: Google Cloud Project for which you
+ try to get all datasets
+ :type project_id: str
+ :return: datasets_list
+
+ Example of returned datasets_list: ::
+
+ {
+ "kind":"bigquery#dataset",
+ "location":"US",
+ "id":"your-project:dataset_2_test",
+ "datasetReference":{
+ "projectId":"your-project",
+ "datasetId":"dataset_2_test"
+ }
+ },
+ {
+ "kind":"bigquery#dataset",
+ "location":"US",
+ "id":"your-project:dataset_1_test",
+ "datasetReference":{
+ "projectId":"your-project",
+ "datasetId":"dataset_1_test"
+ }
+ }
+ ]
+ """
+ dataset_project_id = project_id if project_id else self.project_id
+
+ try:
+ datasets_list = self.service.datasets().list(
+ projectId=dataset_project_id).execute()['datasets']
+ self.log.info("Datasets List: {}".format(datasets_list))
+
+ except HttpError as err:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(err.content))
+
+ return datasets_list
+
+ def insert_all(self, project_id, dataset_id, table_id,
+ rows, ignore_unknown_values=False,
+ skip_invalid_rows=False, fail_on_error=False):
+ """
+ Method to stream data into BigQuery one record at a time without needing
+ to run a load job
+
+ .. seealso::
+ For more information, see:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll
+
+ :param project_id: The name of the project where we have the table
+ :type project_id: str
+ :param dataset_id: The name of the dataset where we have the table
+ :type dataset_id: str
+ :param table_id: The name of the table
+ :type table_id: str
+ :param rows: the rows to insert
+ :type rows: list
+
+ **Example or rows**:
+ rows=[{"json": {"a_key": "a_value_0"}}, {"json": {"a_key": "a_value_1"}}]
+
+ :param ignore_unknown_values: [Optional] Accept rows that contain values
+ that do not match the schema. The unknown values are ignored.
+ The default value is false, which treats unknown values as errors.
+ :type ignore_unknown_values: bool
+ :param skip_invalid_rows: [Optional] Insert all valid rows of a request,
+ even if invalid rows exist. The default value is false, which causes
+ the entire request to fail if any invalid rows exist.
+ :type skip_invalid_rows: bool
+ :param fail_on_error: [Optional] Force the task to fail if any errors occur.
+ The default value is false, which indicates the task should not fail
+ even if any insertion errors occur.
+ :type fail_on_error: bool
+ """
+
+ dataset_project_id = project_id if project_id else self.project_id
+
+ body = {
+ "rows": rows,
+ "ignoreUnknownValues": ignore_unknown_values,
+ "kind": "bigquery#tableDataInsertAllRequest",
+ "skipInvalidRows": skip_invalid_rows,
+ }
+
+ try:
+ self.log.info('Inserting {} row(s) into Table {}:{}.{}'.format(
+ len(rows), dataset_project_id,
+ dataset_id, table_id))
+
+ resp = self.service.tabledata().insertAll(
+ projectId=dataset_project_id, datasetId=dataset_id,
+ tableId=table_id, body=body
+ ).execute()
+
+ if 'insertErrors' not in resp:
+ self.log.info('All row(s) inserted successfully: {}:{}.{}'.format(
+ dataset_project_id, dataset_id, table_id))
+ else:
+ error_msg = '{} insert error(s) occurred: {}:{}.{}. Details: {}'.format(
+ len(resp['insertErrors']),
+ dataset_project_id, dataset_id, table_id, resp['insertErrors'])
+ if fail_on_error:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(error_msg)
+ )
+ self.log.info(error_msg)
+ except HttpError as err:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(err.content)
+ )
+
class BigQueryCursor(BigQueryBaseCursor):
"""
@@ -953,11 +1650,13 @@ class BigQueryCursor(BigQueryBaseCursor):
https://github.com/dropbox/PyHive/blob/master/pyhive/common.py
"""
- def __init__(self, service, project_id, use_legacy_sql=True):
+ def __init__(self, service, project_id, use_legacy_sql=True, location=None):
super(BigQueryCursor, self).__init__(
service=service,
project_id=project_id,
- use_legacy_sql=use_legacy_sql)
+ use_legacy_sql=use_legacy_sql,
+ location=location,
+ )
self.buffersize = None
self.page_token = None
self.job_id = None
@@ -983,23 +1682,23 @@ def execute(self, operation, parameters=None):
Executes a BigQuery query, and returns the job ID.
:param operation: The query to execute.
- :type operation: string
+ :type operation: str
:param parameters: Parameters to substitute into the query.
:type parameters: dict
"""
- bql = _bind_parameters(operation,
+ sql = _bind_parameters(operation,
parameters) if parameters else operation
- self.job_id = self.run_query(bql)
+ self.job_id = self.run_query(sql)
def executemany(self, operation, seq_of_parameters):
"""
Execute a BigQuery query multiple times with different parameters.
:param operation: The query to execute.
- :type operation: string
- :param parameters: List of dictionary parameters to substitute into the
+ :type operation: str
+ :param seq_of_parameters: List of dictionary parameters to substitute into the
query.
- :type parameters: list
+ :type seq_of_parameters: list
"""
for parameters in seq_of_parameters:
self.execute(operation, parameters)
@@ -1111,7 +1810,7 @@ def _bind_parameters(operation, parameters):
""" Helper method that binds parameters to a SQL query. """
# inspired by MySQL Python Connector (conversion.py)
string_parameters = {}
- for (name, value) in parameters.iteritems():
+ for (name, value) in iteritems(parameters):
if value is None:
string_parameters[name] = 'NULL'
elif isinstance(value, basestring):
@@ -1139,19 +1838,28 @@ def _bq_cast(string_field, bq_type):
"""
if string_field is None:
return None
- elif bq_type == 'INTEGER' or bq_type == 'TIMESTAMP':
+ elif bq_type == 'INTEGER':
return int(string_field)
- elif bq_type == 'FLOAT':
+ elif bq_type == 'FLOAT' or bq_type == 'TIMESTAMP':
return float(string_field)
elif bq_type == 'BOOLEAN':
- assert string_field in set(['true', 'false'])
+ if string_field not in ['true', 'false']:
+ raise ValueError("{} must have value 'true' or 'false'".format(
+ string_field))
return string_field == 'true'
else:
return string_field
def _split_tablename(table_input, default_project_id, var_name=None):
- assert default_project_id is not None, "INTERNAL: No default project is specified"
+
+ if '.' not in table_input:
+ raise ValueError(
+ 'Expected target table name in the format of '
+ '<dataset>.<table>. Got: {}'.format(table_input))
+
+ if not default_project_id:
+ raise ValueError("INTERNAL: No default project is specified")
def var_print(var_name):
if var_name is None:
@@ -1163,7 +1871,6 @@ def var_print(var_name):
raise Exception(('{var}Use either : or . to specify project '
'got {input}').format(
var=var_print(var_name), input=table_input))
-
cmpt = table_input.rsplit(':', 1)
project_id = None
rest = table_input
@@ -1181,8 +1888,10 @@ def var_print(var_name):
cmpt = rest.split('.')
if len(cmpt) == 3:
- assert project_id is None, ("{var}Use either : or . to specify project"
- ).format(var=var_print(var_name))
+ if project_id:
+ raise ValueError(
+ "{var}Use either : or . to specify project".format(
+ var=var_print(var_name)))
project_id = cmpt[0]
dataset_id = cmpt[1]
table_id = cmpt[2]
@@ -1206,3 +1915,34 @@ def var_print(var_name):
project_id = default_project_id
return project_id, dataset_id, table_id
+
+
+def _cleanse_time_partitioning(destination_dataset_table, time_partitioning_in):
+ # if it is a partitioned table ($ is in the table name) add partition load option
+
+ if time_partitioning_in is None:
+ time_partitioning_in = {}
+
+ time_partitioning_out = {}
+ if destination_dataset_table and '$' in destination_dataset_table:
+ time_partitioning_out['type'] = 'DAY'
+ time_partitioning_out.update(time_partitioning_in)
+ return time_partitioning_out
+
+
+def _validate_value(key, value, expected_type):
+ """ function to check expected type and raise
+ error if type is not correct """
+ if not isinstance(value, expected_type):
+ raise TypeError("{} argument must have a type {} not {}".format(
+ key, expected_type, type(value)))
+
+
+def _api_resource_configs_duplication_check(key, value, config_dict,
+ config_dict_name='api_resource_configs'):
+ if key in config_dict and value != config_dict[key]:
+ raise ValueError("Values of {param_name} param are duplicated. "
+ "{dict_name} contained {param_name} param "
+ "in `query` config and {param_name} was also provided "
+ "with arg to run_query() method. Please remove duplicates."
+ .format(param_name=key, dict_name=config_dict_name))
diff --git a/airflow/contrib/hooks/cassandra_hook.py b/airflow/contrib/hooks/cassandra_hook.py
new file mode 100644
index 0000000000..2c744fe9b1
--- /dev/null
+++ b/airflow/contrib/hooks/cassandra_hook.py
@@ -0,0 +1,197 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from cassandra.cluster import Cluster
+from cassandra.policies import (RoundRobinPolicy, DCAwareRoundRobinPolicy,
+ TokenAwarePolicy, WhiteListRoundRobinPolicy)
+from cassandra.auth import PlainTextAuthProvider
+
+from airflow.hooks.base_hook import BaseHook
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+
+class CassandraHook(BaseHook, LoggingMixin):
+ """
+ Hook used to interact with Cassandra
+
+ Contact points can be specified as a comma-separated string in the 'hosts'
+ field of the connection.
+
+ Port can be specified in the port field of the connection.
+
+ If SSL is enabled in Cassandra, pass in a dict in the extra field as kwargs for
+ ``ssl.wrap_socket()``. For example:
+ {
+ 'ssl_options' : {
+ 'ca_certs' : PATH_TO_CA_CERTS
+ }
+ }
+
+ Default load balancing policy is RoundRobinPolicy. To specify a different LB policy:
+ - DCAwareRoundRobinPolicy
+ {
+ 'load_balancing_policy': 'DCAwareRoundRobinPolicy',
+ 'load_balancing_policy_args': {
+ 'local_dc': LOCAL_DC_NAME, // optional
+ 'used_hosts_per_remote_dc': SOME_INT_VALUE, // optional
+ }
+ }
+ - WhiteListRoundRobinPolicy
+ {
+ 'load_balancing_policy': 'WhiteListRoundRobinPolicy',
+ 'load_balancing_policy_args': {
+ 'hosts': ['HOST1', 'HOST2', 'HOST3']
+ }
+ }
+ - TokenAwarePolicy
+ {
+ 'load_balancing_policy': 'TokenAwarePolicy',
+ 'load_balancing_policy_args': {
+ 'child_load_balancing_policy': CHILD_POLICY_NAME, // optional
+ 'child_load_balancing_policy_args': { ... } // optional
+ }
+ }
+
+ For details of the Cluster config, see cassandra.cluster.
+ """
+ def __init__(self, cassandra_conn_id='cassandra_default'):
+ conn = self.get_connection(cassandra_conn_id)
+
+ conn_config = {}
+ if conn.host:
+ conn_config['contact_points'] = conn.host.split(',')
+
+ if conn.port:
+ conn_config['port'] = int(conn.port)
+
+ if conn.login:
+ conn_config['auth_provider'] = PlainTextAuthProvider(
+ username=conn.login, password=conn.password)
+
+ policy_name = conn.extra_dejson.get('load_balancing_policy', None)
+ policy_args = conn.extra_dejson.get('load_balancing_policy_args', {})
+ lb_policy = self.get_lb_policy(policy_name, policy_args)
+ if lb_policy:
+ conn_config['load_balancing_policy'] = lb_policy
+
+ cql_version = conn.extra_dejson.get('cql_version', None)
+ if cql_version:
+ conn_config['cql_version'] = cql_version
+
+ ssl_options = conn.extra_dejson.get('ssl_options', None)
+ if ssl_options:
+ conn_config['ssl_options'] = ssl_options
+
+ self.cluster = Cluster(**conn_config)
+ self.keyspace = conn.schema
+ self.session = None
+
+ def get_conn(self):
+ """
+ Returns a cassandra Session object
+ """
+ if self.session and not self.session.is_shutdown:
+ return self.session
+ self.session = self.cluster.connect(self.keyspace)
+ return self.session
+
+ def get_cluster(self):
+ return self.cluster
+
+ def shutdown_cluster(self):
+ """
+ Closes all sessions and connections associated with this Cluster.
+ """
+ if not self.cluster.is_shutdown:
+ self.cluster.shutdown()
+
+ @staticmethod
+ def get_lb_policy(policy_name, policy_args):
+ policies = {
+ 'RoundRobinPolicy': RoundRobinPolicy,
+ 'DCAwareRoundRobinPolicy': DCAwareRoundRobinPolicy,
+ 'WhiteListRoundRobinPolicy': WhiteListRoundRobinPolicy,
+ 'TokenAwarePolicy': TokenAwarePolicy,
+ }
+
+ if not policies.get(policy_name) or policy_name == 'RoundRobinPolicy':
+ return RoundRobinPolicy()
+
+ if policy_name == 'DCAwareRoundRobinPolicy':
+ local_dc = policy_args.get('local_dc', '')
+ used_hosts_per_remote_dc = int(policy_args.get('used_hosts_per_remote_dc', 0))
+ return DCAwareRoundRobinPolicy(local_dc, used_hosts_per_remote_dc)
+
+ if policy_name == 'WhiteListRoundRobinPolicy':
+ hosts = policy_args.get('hosts')
+ if not hosts:
+ raise Exception('Hosts must be specified for WhiteListRoundRobinPolicy')
+ return WhiteListRoundRobinPolicy(hosts)
+
+ if policy_name == 'TokenAwarePolicy':
+ allowed_child_policies = ('RoundRobinPolicy',
+ 'DCAwareRoundRobinPolicy',
+ 'WhiteListRoundRobinPolicy',)
+ child_policy_name = policy_args.get('child_load_balancing_policy',
+ 'RoundRobinPolicy')
+ child_policy_args = policy_args.get('child_load_balancing_policy_args', {})
+ if child_policy_name not in allowed_child_policies:
+ return TokenAwarePolicy(RoundRobinPolicy())
+ else:
+ child_policy = CassandraHook.get_lb_policy(child_policy_name,
+ child_policy_args)
+ return TokenAwarePolicy(child_policy)
+
+ def table_exists(self, table):
+ """
+ Checks if a table exists in Cassandra
+
+ :param table: Target Cassandra table.
+ Use dot notation to target a specific keyspace.
+ :type table: str
+ """
+ keyspace = self.keyspace
+ if '.' in table:
+ keyspace, table = table.split('.', 1)
+ cluster_metadata = self.get_conn().cluster.metadata
+ return (keyspace in cluster_metadata.keyspaces and
+ table in cluster_metadata.keyspaces[keyspace].tables)
+
+ def record_exists(self, table, keys):
+ """
+ Checks if a record exists in Cassandra
+
+ :param table: Target Cassandra table.
+ Use dot notation to target a specific keyspace.
+ :type table: str
+ :param keys: The keys and their values to check the existence.
+ :type keys: dict
+ """
+ keyspace = self.keyspace
+ if '.' in table:
+ keyspace, table = table.split('.', 1)
+ ks = " AND ".join("{}=%({})s".format(key, key) for key in keys.keys())
+ cql = "SELECT * FROM {keyspace}.{table} WHERE {keys}".format(
+ keyspace=keyspace, table=table, keys=ks)
+
+ try:
+ rs = self.get_conn().execute(cql, keys)
+ return rs.one() is not None
+ except Exception:
+ return False
diff --git a/airflow/contrib/hooks/cloudant_hook.py b/airflow/contrib/hooks/cloudant_hook.py
index cbb0ccad43..5d39f3fa8a 100644
--- a/airflow/contrib/hooks/cloudant_hook.py
+++ b/airflow/contrib/hooks/cloudant_hook.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from past.builtins import unicode
@@ -37,8 +42,8 @@ def _str(s):
if isinstance(s, unicode):
log = LoggingMixin().log
log.debug(
- 'cloudant-python does not support unicode. Encoding %s as ascii using "ignore".',
- s
+ 'cloudant-python does not support unicode. Encoding %s as '
+ 'ascii using "ignore".', s
)
return s.encode('ascii', 'ignore')
diff --git a/airflow/contrib/hooks/databricks_hook.py b/airflow/contrib/hooks/databricks_hook.py
index cd9dc547c0..f4a890ac7b 100644
--- a/airflow/contrib/hooks/databricks_hook.py
+++ b/airflow/contrib/hooks/databricks_hook.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
import requests
@@ -19,6 +24,7 @@
from airflow.hooks.base_hook import BaseHook
from requests import exceptions as requests_exceptions
from requests.auth import AuthBase
+from time import sleep
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -27,7 +33,11 @@
except ImportError:
import urlparse
+RESTART_CLUSTER_ENDPOINT = ("POST", "api/2.0/clusters/restart")
+START_CLUSTER_ENDPOINT = ("POST", "api/2.0/clusters/start")
+TERMINATE_CLUSTER_ENDPOINT = ("POST", "api/2.0/clusters/delete")
+RUN_NOW_ENDPOINT = ('POST', 'api/2.0/jobs/run-now')
SUBMIT_RUN_ENDPOINT = ('POST', 'api/2.0/jobs/runs/submit')
GET_RUN_ENDPOINT = ('GET', 'api/2.0/jobs/runs/get')
CANCEL_RUN_ENDPOINT = ('POST', 'api/2.0/jobs/runs/cancel')
@@ -42,24 +52,31 @@ def __init__(
self,
databricks_conn_id='databricks_default',
timeout_seconds=180,
- retry_limit=3):
+ retry_limit=3,
+ retry_delay=1.0):
"""
:param databricks_conn_id: The name of the databricks connection to use.
- :type databricks_conn_id: string
+ :type databricks_conn_id: str
:param timeout_seconds: The amount of time in seconds the requests library
will wait before timing-out.
:type timeout_seconds: int
:param retry_limit: The number of times to retry the connection in case of
service outages.
:type retry_limit: int
+ :param retry_delay: The number of seconds to wait between retries (it
+ might be a floating point number).
+ :type retry_delay: float
"""
self.databricks_conn_id = databricks_conn_id
self.databricks_conn = self.get_connection(databricks_conn_id)
self.timeout_seconds = timeout_seconds
- assert retry_limit >= 1, 'Retry limit must be greater than equal to 1'
+ if retry_limit < 1:
+ raise ValueError('Retry limit must be greater than equal to 1')
self.retry_limit = retry_limit
+ self.retry_delay = retry_delay
- def _parse_host(self, host):
+ @staticmethod
+ def _parse_host(host):
"""
The purpose of this function is to be robust to improper connections
settings provided by users, specifically in the host field.
@@ -112,7 +129,8 @@ def _do_api_call(self, endpoint_info, json):
else:
raise AirflowException('Unexpected HTTP Method: ' + method)
- for attempt_num in range(1, self.retry_limit+1):
+ attempt_num = 1
+ while True:
try:
response = request_func(
url,
@@ -120,21 +138,41 @@ def _do_api_call(self, endpoint_info, json):
auth=auth,
headers=USER_AGENT_HEADER,
timeout=self.timeout_seconds)
- if response.status_code == requests.codes.ok:
- return response.json()
- else:
+ response.raise_for_status()
+ return response.json()
+ except requests_exceptions.RequestException as e:
+ if not _retryable_error(e):
# In this case, the user probably made a mistake.
# Don't retry.
raise AirflowException('Response: {0}, Status Code: {1}'.format(
- response.content, response.status_code))
- except (requests_exceptions.ConnectionError,
- requests_exceptions.Timeout) as e:
- self.log.error(
- 'Attempt %s API Request to Databricks failed with reason: %s',
- attempt_num, e
- )
- raise AirflowException(('API requests to Databricks failed {} times. ' +
- 'Giving up.').format(self.retry_limit))
+ e.response.content, e.response.status_code))
+
+ self._log_request_error(attempt_num, e)
+
+ if attempt_num == self.retry_limit:
+ raise AirflowException(('API requests to Databricks failed {} times. ' +
+ 'Giving up.').format(self.retry_limit))
+
+ attempt_num += 1
+ sleep(self.retry_delay)
+
+ def _log_request_error(self, attempt_num, error):
+ self.log.error(
+ 'Attempt %s API Request to Databricks failed with reason: %s',
+ attempt_num, error
+ )
+
+ def run_now(self, json):
+ """
+ Utility function to call the ``api/2.0/jobs/run-now`` endpoint.
+
+ :param json: The data used in the body of the request to the ``run-now`` endpoint.
+ :type json: dict
+ :return: the run_id as a string
+ :rtype: str
+ """
+ response = self._do_api_call(RUN_NOW_ENDPOINT, json)
+ return response['run_id']
def submit_run(self, json):
"""
@@ -143,7 +181,7 @@ def submit_run(self, json):
:param json: The data used in the body of the request to the ``submit`` endpoint.
:type json: dict
:return: the run_id as a string
- :rtype: string
+ :rtype: str
"""
response = self._do_api_call(SUBMIT_RUN_ENDPOINT, json)
return response['run_id']
@@ -167,6 +205,21 @@ def cancel_run(self, run_id):
json = {'run_id': run_id}
self._do_api_call(CANCEL_RUN_ENDPOINT, json)
+ def restart_cluster(self, json):
+ self._do_api_call(RESTART_CLUSTER_ENDPOINT, json)
+
+ def start_cluster(self, json):
+ self._do_api_call(START_CLUSTER_ENDPOINT, json)
+
+ def terminate_cluster(self, json):
+ self._do_api_call(TERMINATE_CLUSTER_ENDPOINT, json)
+
+
+def _retryable_error(exception):
+ return isinstance(exception, requests_exceptions.ConnectionError) \
+ or isinstance(exception, requests_exceptions.Timeout) \
+ or exception.response is not None and exception.response.status_code >= 500
+
RUN_LIFE_CYCLE_STATES = [
'PENDING',
@@ -190,10 +243,11 @@ def __init__(self, life_cycle_state, result_state, state_message):
@property
def is_terminal(self):
if self.life_cycle_state not in RUN_LIFE_CYCLE_STATES:
- raise AirflowException(('Unexpected life cycle state: {}: If the state has '
- 'been introduced recently, please check the Databricks user '
- 'guide for troubleshooting information').format(
- self.life_cycle_state))
+ raise AirflowException(
+ ('Unexpected life cycle state: {}: If the state has '
+ 'been introduced recently, please check the Databricks user '
+ 'guide for troubleshooting information').format(
+ self.life_cycle_state))
return self.life_cycle_state in ('TERMINATED', 'SKIPPED', 'INTERNAL_ERROR')
@property
diff --git a/airflow/contrib/hooks/datadog_hook.py b/airflow/contrib/hooks/datadog_hook.py
index 6caf611adb..f00e49b097 100644
--- a/airflow/contrib/hooks/datadog_hook.py
+++ b/airflow/contrib/hooks/datadog_hook.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
import time
from airflow.hooks.base_hook import BaseHook
@@ -30,7 +35,7 @@ class DatadogHook(BaseHook, LoggingMixin):
Airflow runs.
:param datadog_conn_id: The connection to datadog, containing metadata for api keys.
- :param datadog_conn_id: string
+ :param datadog_conn_id: str
"""
def __init__(self, datadog_conn_id='datadog_default'):
conn = self.get_connection(datadog_conn_id)
@@ -43,9 +48,11 @@ def __init__(self, datadog_conn_id='datadog_default'):
self.host = conn.host
if self.api_key is None:
- raise AirflowException("api_key must be specified in the Datadog connection details")
+ raise AirflowException("api_key must be specified in the "
+ "Datadog connection details")
if self.app_key is None:
- raise AirflowException("app_key must be specified in the Datadog connection details")
+ raise AirflowException("app_key must be specified in the "
+ "Datadog connection details")
self.log.info("Setting up api keys for Datadog")
options = {
@@ -64,9 +71,9 @@ def send_metric(self, metric_name, datapoint, tags=None):
Sends a single datapoint metric to DataDog
:param metric_name: The name of the metric
- :type metric_name: string
+ :type metric_name: str
:param datapoint: A single integer or float related to the metric
- :type datapoint: integer or float
+ :type datapoint: int or float
:param tags: A list of tags associated with the metric
:type tags: list
"""
@@ -84,11 +91,11 @@ def query_metric(self,
from_seconds_ago,
to_seconds_ago):
"""
- Queries datadog for a specific metric, potentially with some function applied to it
- and returns the results.
+ Queries datadog for a specific metric, potentially with some
+ function applied to it and returns the results.
:param query: The datadog query to execute (see datadog docs)
- :type query: string
+ :type query: str
:param from_seconds_ago: How many seconds ago to start querying for.
:type from_seconds_ago: int
:param to_seconds_ago: Up to how many seconds ago to query for.
@@ -107,20 +114,20 @@ def query_metric(self,
def post_event(self, title, text, tags=None, alert_type=None, aggregation_key=None):
"""
Posts an event to datadog (processing finished, potentially alerts, other issues)
- Think about this as a means to maintain persistence of alerts, rather than alerting
- itself.
+ Think about this as a means to maintain persistence of alerts, rather than
+ alerting itself.
:param title: The title of the event
- :type title: string
+ :type title: str
:param text: The body of the event (more information)
- :type text: string
+ :type text: str
:param tags: List of string tags to apply to the event
:type tags: list
:param alert_type: The alert type for the event, one of
["error", "warning", "info", "success"]
- :type alert_type: string
+ :type alert_type: str
:param aggregation_key: Key that can be used to aggregate this event in a stream
- :type aggregation_key: string
+ :type aggregation_key: str
"""
response = api.Event.create(
title=title,
diff --git a/airflow/contrib/hooks/datastore_hook.py b/airflow/contrib/hooks/datastore_hook.py
index cf98dc749f..c055e347d6 100644
--- a/airflow/contrib/hooks/datastore_hook.py
+++ b/airflow/contrib/hooks/datastore_hook.py
@@ -1,19 +1,23 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
-import json
import time
from apiclient.discovery import build
from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
@@ -25,7 +29,7 @@ class DatastoreHook(GoogleCloudBaseHook):
connection.
This object is not threads safe. If you want to make multiple requests
- simultaniously, you will need to create a hook per thread.
+ simultaneously, you will need to create a hook per thread.
"""
def __init__(self,
@@ -37,10 +41,11 @@ def __init__(self,
def get_conn(self, version='v1'):
"""
- Returns a Google Cloud Storage service object.
+ Returns a Google Cloud Datastore service object.
"""
http_authorized = self._authorize()
- return build('datastore', version, http=http_authorized)
+ return build(
+ 'datastore', version, http=http_authorized, cache_discovery=False)
def allocate_ids(self, partialKeys):
"""
@@ -50,34 +55,45 @@ def allocate_ids(self, partialKeys):
:param partialKeys: a list of partial keys
:return: a list of full keys.
"""
- resp = self.connection.projects().allocateIds(projectId=self.project_id, body={'keys': partialKeys}).execute()
+ resp = self.connection.projects().allocateIds(
+ projectId=self.project_id, body={'keys': partialKeys}
+ ).execute()
return resp['keys']
def begin_transaction(self):
"""
Get a new transaction handle
- see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/beginTransaction
+
+ .. seealso::
+ https://cloud.google.com/datastore/docs/reference/rest/v1/projects/beginTransaction
:return: a transaction handle
"""
- resp = self.connection.projects().beginTransaction(projectId=self.project_id, body={}).execute()
+ resp = self.connection.projects().beginTransaction(
+ projectId=self.project_id, body={}).execute()
return resp['transaction']
def commit(self, body):
"""
Commit a transaction, optionally creating, deleting or modifying some entities.
- see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/commit
+
+ .. seealso::
+ https://cloud.google.com/datastore/docs/reference/rest/v1/projects/commit
:param body: the body of the commit request
:return: the response body of the commit request
"""
- resp = self.connection.projects().commit(projectId=self.project_id, body=body).execute()
+ resp = self.connection.projects().commit(
+ projectId=self.project_id, body=body).execute()
return resp
def lookup(self, keys, read_consistency=None, transaction=None):
"""
Lookup some entities by key
- see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/lookup
+
+ .. seealso::
+ https://cloud.google.com/datastore/docs/reference/rest/v1/projects/lookup
+
:param keys: the keys to lookup
:param read_consistency: the read consistency to use. default, strong or eventual.
Cannot be used with a transaction.
@@ -89,25 +105,34 @@ def lookup(self, keys, read_consistency=None, transaction=None):
body['readConsistency'] = read_consistency
if transaction:
body['transaction'] = transaction
- return self.connection.projects().lookup(projectId=self.project_id, body=body).execute()
+ return self.connection.projects().lookup(
+ projectId=self.project_id, body=body).execute()
def rollback(self, transaction):
"""
Roll back a transaction
- see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/rollback
+
+ .. seealso::
+ https://cloud.google.com/datastore/docs/reference/rest/v1/projects/rollback
+
:param transaction: the transaction to roll back
"""
- self.connection.projects().rollback(projectId=self.project_id, body={'transaction': transaction})\
+ self.connection.projects().rollback(
+ projectId=self.project_id, body={'transaction': transaction})\
.execute()
def run_query(self, body):
"""
Run a query for entities.
- see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/runQuery
+
+ .. seealso::
+ https://cloud.google.com/datastore/docs/reference/rest/v1/projects/runQuery
+
:param body: the body of the query request
:return: the batch of query results.
"""
- resp = self.connection.projects().runQuery(projectId=self.project_id, body=body).execute()
+ resp = self.connection.projects().runQuery(
+ projectId=self.project_id, body=body).execute()
return resp['batch']
def get_operation(self, name):
@@ -142,11 +167,12 @@ def poll_operation_until_done(self, name, polling_interval_in_seconds):
else:
return result
- def export_to_storage_bucket(self, bucket, namespace=None, entity_filter=None, labels=None):
+ def export_to_storage_bucket(self, bucket, namespace=None,
+ entity_filter=None, labels=None):
"""
Export entities from Cloud Datastore to Cloud Storage for backup
"""
- output_uri_prefix = 'gs://' + ('/').join(filter(None, [bucket, namespace]))
+ output_uri_prefix = 'gs://' + '/'.join(filter(None, [bucket, namespace]))
if not entity_filter:
entity_filter = {}
if not labels:
@@ -156,14 +182,16 @@ def export_to_storage_bucket(self, bucket, namespace=None, entity_filter=None, l
'entityFilter': entity_filter,
'labels': labels,
}
- resp = self.admin_connection.projects().export(projectId=self.project_id, body=body).execute()
+ resp = self.admin_connection.projects().export(
+ projectId=self.project_id, body=body).execute()
return resp
- def import_from_storage_bucket(self, bucket, file, namespace=None, entity_filter=None, labels=None):
+ def import_from_storage_bucket(self, bucket, file,
+ namespace=None, entity_filter=None, labels=None):
"""
Import a backup from Cloud Storage to Cloud Datastore
"""
- input_url = 'gs://' + ('/').join(filter(None, [bucket, namespace, file]))
+ input_url = 'gs://' + '/'.join(filter(None, [bucket, namespace, file]))
if not entity_filter:
entity_filter = {}
if not labels:
@@ -173,5 +201,6 @@ def import_from_storage_bucket(self, bucket, file, namespace=None, entity_filter
'entityFilter': entity_filter,
'labels': labels,
}
- resp = self.admin_connection.projects().import_(projectId=self.project_id, body=body).execute()
+ resp = self.admin_connection.projects().import_(
+ projectId=self.project_id, body=body).execute()
return resp
diff --git a/airflow/contrib/hooks/discord_webhook_hook.py b/airflow/contrib/hooks/discord_webhook_hook.py
new file mode 100644
index 0000000000..731d9d575a
--- /dev/null
+++ b/airflow/contrib/hooks/discord_webhook_hook.py
@@ -0,0 +1,140 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+import json
+import re
+
+from airflow.hooks.http_hook import HttpHook
+from airflow.exceptions import AirflowException
+
+
+class DiscordWebhookHook(HttpHook):
+ """
+ This hook allows you to post messages to Discord using incoming webhooks.
+ Takes a Discord connection ID with a default relative webhook endpoint. The
+ default endpoint can be overridden using the webhook_endpoint parameter
+ (https://discordapp.com/developers/docs/resources/webhook).
+
+ Each Discord webhook can be pre-configured to use a specific username and
+ avatar_url. You can override these defaults in this hook.
+
+ :param http_conn_id: Http connection ID with host as "https://discord.com/api/" and
+ default webhook endpoint in the extra field in the form of
+ {"webhook_endpoint": "webhooks/{webhook.id}/{webhook.token}"}
+ :type http_conn_id: str
+ :param webhook_endpoint: Discord webhook endpoint in the form of
+ "webhooks/{webhook.id}/{webhook.token}"
+ :type webhook_endpoint: str
+ :param message: The message you want to send to your Discord channel
+ (max 2000 characters)
+ :type message: str
+ :param username: Override the default username of the webhook
+ :type username: str
+ :param avatar_url: Override the default avatar of the webhook
+ :type avatar_url: str
+ :param tts: Is a text-to-speech message
+ :type tts: bool
+ :param proxy: Proxy to use to make the Discord webhook call
+ :type proxy: str
+ """
+
+ def __init__(self,
+ http_conn_id=None,
+ webhook_endpoint=None,
+ message="",
+ username=None,
+ avatar_url=None,
+ tts=False,
+ proxy=None,
+ *args,
+ **kwargs):
+ super(DiscordWebhookHook, self).__init__(*args, **kwargs)
+ self.http_conn_id = http_conn_id
+ self.webhook_endpoint = self._get_webhook_endpoint(http_conn_id, webhook_endpoint)
+ self.message = message
+ self.username = username
+ self.avatar_url = avatar_url
+ self.tts = tts
+ self.proxy = proxy
+
+ def _get_webhook_endpoint(self, http_conn_id, webhook_endpoint):
+ """
+ Given a Discord http_conn_id, return the default webhook endpoint or override if a
+ webhook_endpoint is manually supplied.
+
+ :param http_conn_id: The provided connection ID
+ :param webhook_endpoint: The manually provided webhook endpoint
+ :return: Webhook endpoint (str) to use
+ """
+ if webhook_endpoint:
+ endpoint = webhook_endpoint
+ elif http_conn_id:
+ conn = self.get_connection(http_conn_id)
+ extra = conn.extra_dejson
+ endpoint = extra.get('webhook_endpoint', '')
+ else:
+ raise AirflowException('Cannot get webhook endpoint: No valid Discord '
+ 'webhook endpoint or http_conn_id supplied.')
+
+ # make sure endpoint matches the expected Discord webhook format
+ if not re.match('^webhooks/[0-9]+/[a-zA-Z0-9_-]+$', endpoint):
+ raise AirflowException('Expected Discord webhook endpoint in the form '
+ 'of "webhooks/{webhook.id}/{webhook.token}".')
+
+ return endpoint
+
+ def _build_discord_payload(self):
+ """
+ Construct the Discord JSON payload. All relevant parameters are combined here
+ to a valid Discord JSON payload.
+
+ :return: Discord payload (str) to send
+ """
+ payload = {}
+
+ if self.username:
+ payload['username'] = self.username
+ if self.avatar_url:
+ payload['avatar_url'] = self.avatar_url
+
+ payload['tts'] = self.tts
+
+ if len(self.message) <= 2000:
+ payload['content'] = self.message
+ else:
+ raise AirflowException('Discord message length must be 2000 or fewer '
+ 'characters.')
+
+ return json.dumps(payload)
+
+ def execute(self):
+ """
+ Execute the Discord webhook call
+ """
+ proxies = {}
+ if self.proxy:
+ # we only need https proxy for Discord
+ proxies = {'https': self.proxy}
+
+ discord_payload = self._build_discord_payload()
+
+ self.run(endpoint=self.webhook_endpoint,
+ data=discord_payload,
+ headers={'Content-type': 'application/json'},
+ extra_options={'proxies': proxies})
diff --git a/airflow/contrib/hooks/emr_hook.py b/airflow/contrib/hooks/emr_hook.py
index cee239867c..fcdf4ac848 100644
--- a/airflow/contrib/hooks/emr_hook.py
+++ b/airflow/contrib/hooks/emr_hook.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from airflow.exceptions import AirflowException
from airflow.contrib.hooks.aws_hook import AwsHook
@@ -18,7 +23,8 @@
class EmrHook(AwsHook):
"""
- Interact with AWS EMR. emr_conn_id is only neccessary for using the create_job_flow method.
+ Interact with AWS EMR. emr_conn_id is only necessary for using the
+ create_job_flow method.
"""
def __init__(self, emr_conn_id=None, *args, **kwargs):
@@ -32,7 +38,8 @@ def get_conn(self):
def create_job_flow(self, job_flow_overrides):
"""
Creates a job flow using the config from the EMR connection.
- Keys of the json extra hash may have the arguments of the boto3 run_job_flow method.
+ Keys of the json extra hash may have the arguments of the boto3
+ run_job_flow method.
Overrides for this config may be passed as the job_flow_overrides.
"""
@@ -44,19 +51,6 @@ def create_job_flow(self, job_flow_overrides):
config = emr_conn.extra_dejson.copy()
config.update(job_flow_overrides)
- response = self.get_conn().run_job_flow(
- Name=config.get('Name'),
- LogUri=config.get('LogUri'),
- ReleaseLabel=config.get('ReleaseLabel'),
- Instances=config.get('Instances'),
- Steps=config.get('Steps', []),
- BootstrapActions=config.get('BootstrapActions', []),
- Applications=config.get('Applications'),
- Configurations=config.get('Configurations', []),
- VisibleToAllUsers=config.get('VisibleToAllUsers'),
- JobFlowRole=config.get('JobFlowRole'),
- ServiceRole=config.get('ServiceRole'),
- Tags=config.get('Tags')
- )
+ response = self.get_conn().run_job_flow(**config)
return response
diff --git a/airflow/contrib/hooks/fs_hook.py b/airflow/contrib/hooks/fs_hook.py
index bee60cedba..6832f20c22 100644
--- a/airflow/contrib/hooks/fs_hook.py
+++ b/airflow/contrib/hooks/fs_hook.py
@@ -1,23 +1,28 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
from airflow.hooks.base_hook import BaseHook
class FSHook(BaseHook):
- '''
+ """
Allows for interaction with an file server.
Connection should have a name and a path specified under extra:
@@ -27,7 +32,7 @@ class FSHook(BaseHook):
Conn Type: File (path)
Host, Shchema, Login, Password, Port: empty
Extra: {"path": "/tmp"}
- '''
+ """
def __init__(self, conn_id='fs_default'):
conn = self.get_connection(conn_id)
diff --git a/airflow/contrib/hooks/ftp_hook.py b/airflow/contrib/hooks/ftp_hook.py
index b1e224db6d..03849012a3 100644
--- a/airflow/contrib/hooks/ftp_hook.py
+++ b/airflow/contrib/hooks/ftp_hook.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
import datetime
@@ -91,6 +96,7 @@ def close_conn(self):
"""
conn = self.conn
conn.quit()
+ self.conn = None
def describe_directory(self, path):
"""
@@ -142,7 +148,11 @@ def delete_directory(self, path):
conn = self.get_conn()
conn.rmd(path)
- def retrieve_file(self, remote_full_path, local_full_path_or_buffer):
+ def retrieve_file(
+ self,
+ remote_full_path,
+ local_full_path_or_buffer,
+ callback=None):
"""
Transfers the remote file to a local location.
@@ -154,24 +164,60 @@ def retrieve_file(self, remote_full_path, local_full_path_or_buffer):
:type remote_full_path: str
:param local_full_path_or_buffer: full path to the local file or a
file-like buffer
- :type local_full_path: str or file-like buffer
+ :type local_full_path_or_buffer: str or file-like buffer
+ :param callback: callback which is called each time a block of data
+ is read. if you do not use a callback, these blocks will be written
+ to the file or buffer passed in. if you do pass in a callback, note
+ that writing to a file or buffer will need to be handled inside the
+ callback.
+ [default: output_handle.write()]
+ :type callback: callable
+
+ Example::
+ hook = FTPHook(ftp_conn_id='my_conn')
+
+ remote_path = '/path/to/remote/file'
+ local_path = '/path/to/local/file'
+
+ # with a custom callback (in this case displaying progress on each read)
+ def print_progress(percent_progress):
+ self.log.info('Percent Downloaded: %s%%' % percent_progress)
+
+ total_downloaded = 0
+ total_file_size = hook.get_size(remote_path)
+ output_handle = open(local_path, 'wb')
+ def write_to_file_with_progress(data):
+ total_downloaded += len(data)
+ output_handle.write(data)
+ percent_progress = (total_downloaded / total_file_size) * 100
+ print_progress(percent_progress)
+ hook.retrieve_file(remote_path, None, callback=write_to_file_with_progress)
+
+ # without a custom callback data is written to the local_path
+ hook.retrieve_file(remote_path, local_path)
"""
conn = self.get_conn()
is_path = isinstance(local_full_path_or_buffer, basestring)
- if is_path:
- output_handle = open(local_full_path_or_buffer, 'wb')
+ # without a callback, default to writing to a user-provided file or
+ # file-like buffer
+ if not callback:
+ if is_path:
+ output_handle = open(local_full_path_or_buffer, 'wb')
+ else:
+ output_handle = local_full_path_or_buffer
+ callback = output_handle.write
else:
- output_handle = local_full_path_or_buffer
+ output_handle = None
remote_path, remote_file_name = os.path.split(remote_full_path)
conn.cwd(remote_path)
self.log.info('Retrieving file from FTP: %s', remote_full_path)
- conn.retrbinary('RETR %s' % remote_file_name, output_handle.write)
+ conn.retrbinary('RETR %s' % remote_file_name, callback)
self.log.info('Finished retrieving file from FTP: %s', remote_full_path)
- if is_path:
+ if is_path and output_handle:
output_handle.close()
def store_file(self, remote_full_path, local_full_path_or_buffer):
@@ -224,6 +270,12 @@ def rename(self, from_name, to_name):
return conn.rename(from_name, to_name)
def get_mod_time(self, path):
+ """
+ Returns a datetime object representing the last time the file was modified
+
+ :param path: remote file path
+ :type path: string
+ """
conn = self.get_conn()
ftp_mdtm = conn.sendcmd('MDTM ' + path)
time_val = ftp_mdtm[4:]
@@ -233,6 +285,16 @@ def get_mod_time(self, path):
except ValueError:
return datetime.datetime.strptime(time_val, '%Y%m%d%H%M%S')
+ def get_size(self, path):
+ """
+ Returns the size of a file (in bytes)
+
+ :param path: remote file path
+ :type path: string
+ """
+ conn = self.get_conn()
+ return conn.size(path)
+
class FTPSHook(FTPHook):
@@ -244,7 +306,7 @@ def get_conn(self):
params = self.get_connection(self.ftp_conn_id)
if params.port:
- ftplib.FTP_TLS.port=params.port
+ ftplib.FTP_TLS.port = params.port
self.conn = ftplib.FTP_TLS(
params.host, params.login, params.password
diff --git a/airflow/contrib/hooks/gcp_api_base_hook.py b/airflow/contrib/hooks/gcp_api_base_hook.py
index e6ca2402c4..696f00da6a 100644
--- a/airflow/contrib/hooks/gcp_api_base_hook.py
+++ b/airflow/contrib/hooks/gcp_api_base_hook.py
@@ -1,28 +1,37 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
import json
import httplib2
-from oauth2client.client import GoogleCredentials
-from oauth2client.service_account import ServiceAccountCredentials
+import google.auth
+import google_auth_httplib2
+import google.oauth2.service_account
from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
from airflow.utils.log.logging_mixin import LoggingMixin
+_DEFAULT_SCOPES = ('https://www.googleapis.com/auth/cloud-platform',)
+
+
class GoogleCloudBaseHook(BaseHook, LoggingMixin):
"""
A base hook for Google cloud-related hooks. Google cloud has a shared REST
@@ -36,25 +45,28 @@ class GoogleCloudBaseHook(BaseHook, LoggingMixin):
All hook derived from this base hook use the 'Google Cloud Platform' connection
type. Two ways of authentication are supported:
- Default credentials: Only specify 'Project Id'. Then you need to have executed
- ``gcloud auth`` on the Airflow worker machine.
+ Default credentials: Only the 'Project Id' is required. You'll need to
+ have set up default credentials, such as by the
+ ``GOOGLE_APPLICATION_DEFAULT`` environment variable or from the metadata
+ server on Google Compute Engine.
JSON key file: Specify 'Project Id', 'Key Path' and 'Scope'.
Legacy P12 key files are not supported.
"""
- def __init__(self, conn_id, delegate_to=None):
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
"""
- :param conn_id: The connection ID to use when fetching connection info.
- :type conn_id: string
+ :param gcp_conn_id: The connection ID to use when fetching connection info.
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
- self.conn_id = conn_id
+ self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
- self.extras = self.get_connection(conn_id).extra_dejson
+ self.extras = self.get_connection(self.gcp_conn_id).extra_dejson
def _get_credentials(self):
"""
@@ -62,36 +74,30 @@ def _get_credentials(self):
"""
key_path = self._get_field('key_path', False)
keyfile_dict = self._get_field('keyfile_dict', False)
- scope = self._get_field('scope', False)
-
- kwargs = {}
- if self.delegate_to:
- kwargs['sub'] = self.delegate_to
+ scope = self._get_field('scope', None)
+ if scope:
+ scopes = [s.strip() for s in scope.split(',')]
+ else:
+ scopes = _DEFAULT_SCOPES
if not key_path and not keyfile_dict:
- self.log.info('Getting connection using `gcloud auth` user, since no key file '
- 'is defined for hook.')
- credentials = GoogleCredentials.get_application_default()
+ self.log.info('Getting connection using `google.auth.default()` '
+ 'since no key file is defined for hook.')
+ credentials, _ = google.auth.default(scopes=scopes)
elif key_path:
- if not scope:
- raise AirflowException('Scope should be defined when using a key file.')
- scopes = [s.strip() for s in scope.split(',')]
-
# Get credentials from a JSON file.
if key_path.endswith('.json'):
- self.log.info('Getting connection using a JSON key file.')
- credentials = ServiceAccountCredentials\
- .from_json_keyfile_name(key_path, scopes)
+ self.log.debug('Getting connection using JSON key file %s' % key_path)
+ credentials = (
+ google.oauth2.service_account.Credentials.from_service_account_file(
+ key_path, scopes=scopes)
+ )
elif key_path.endswith('.p12'):
raise AirflowException('Legacy P12 key file are not supported, '
'use a JSON key file.')
else:
raise AirflowException('Unrecognised extension for key file.')
else:
- if not scope:
- raise AirflowException('Scope should be defined when using key JSON.')
- scopes = [s.strip() for s in scope.split(',')]
-
# Get credentials from JSON data provided in the UI.
try:
keyfile_dict = json.loads(keyfile_dict)
@@ -101,17 +107,21 @@ def _get_credentials(self):
keyfile_dict['private_key'] = keyfile_dict['private_key'].replace(
'\\n', '\n')
- credentials = ServiceAccountCredentials\
- .from_json_keyfile_dict(keyfile_dict, scopes)
+ credentials = (
+ google.oauth2.service_account.Credentials.from_service_account_info(
+ keyfile_dict, scopes=scopes)
+ )
except json.decoder.JSONDecodeError:
raise AirflowException('Invalid key JSON.')
- return credentials
+
+ return credentials.with_subject(self.delegate_to) \
+ if self.delegate_to else credentials
def _get_access_token(self):
"""
Returns a valid access token from Google API Credentials
"""
- return self._get_credentials().get_access_token().access_token
+ return self._get_credentials().token
def _authorize(self):
"""
@@ -120,7 +130,9 @@ def _authorize(self):
"""
credentials = self._get_credentials()
http = httplib2.Http()
- return credentials.authorize(http)
+ authed_http = google_auth_httplib2.AuthorizedHttp(
+ credentials, http=http)
+ return authed_http
def _get_field(self, f, default=None):
"""
diff --git a/airflow/contrib/hooks/gcp_compute_hook.py b/airflow/contrib/hooks/gcp_compute_hook.py
new file mode 100644
index 0000000000..617e39cb40
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_compute_hook.py
@@ -0,0 +1,307 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import time
+from googleapiclient.discovery import build
+
+from airflow import AirflowException
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+# Number of retries - used by googleapiclient method calls to perform retries
+# For requests that are "retriable"
+NUM_RETRIES = 5
+
+# Time to sleep between active checks of the operation results
+TIME_TO_SLEEP_IN_SECONDS = 1
+
+
+class GceOperationStatus:
+ PENDING = "PENDING"
+ RUNNING = "RUNNING"
+ DONE = "DONE"
+
+
+# noinspection PyAbstractClass
+class GceHook(GoogleCloudBaseHook):
+ """
+ Hook for Google Compute Engine APIs.
+ """
+ _conn = None
+
+ def __init__(self,
+ api_version,
+ gcp_conn_id='google_cloud_default',
+ delegate_to=None):
+ super(GceHook, self).__init__(gcp_conn_id, delegate_to)
+ self.api_version = api_version
+
+ def get_conn(self):
+ """
+ Retrieves connection to Google Compute Engine.
+
+ :return: Google Compute Engine services object
+ :rtype: dict
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build('compute', self.api_version,
+ http=http_authorized, cache_discovery=False)
+ return self._conn
+
+ def start_instance(self, project_id, zone, resource_id):
+ """
+ Starts an existing instance defined by project_id, zone and resource_id.
+
+ :param project_id: Google Cloud Platform project ID where the Compute Engine
+ Instance exists
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the instance exists
+ :type zone: str
+ :param resource_id: Name of the Compute Engine instance resource
+ :type resource_id: str
+ :return: True if the operation succeeded, raises an error otherwise.
+ :rtype: bool
+ """
+ response = self.get_conn().instances().start(
+ project=project_id,
+ zone=zone,
+ instance=resource_id
+ ).execute(num_retries=NUM_RETRIES)
+ try:
+ operation_name = response["name"]
+ except KeyError:
+ raise AirflowException(
+ "Wrong response '{}' returned - it should contain "
+ "'name' field".format(response))
+ return self._wait_for_operation_to_complete(project_id, operation_name, zone)
+
+ def stop_instance(self, project_id, zone, resource_id):
+ """
+ Stops an instance defined by project_id, zone and resource_id
+
+ :param project_id: Google Cloud Platform project ID where the Compute Engine
+ Instance exists
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the instance exists
+ :type zone: str
+ :param resource_id: Name of the Compute Engine instance resource
+ :type resource_id: str
+ :return: True if the operation succeeded, raises an error otherwise.
+ :rtype: bool
+ """
+ response = self.get_conn().instances().stop(
+ project=project_id,
+ zone=zone,
+ instance=resource_id
+ ).execute(num_retries=NUM_RETRIES)
+ try:
+ operation_name = response["name"]
+ except KeyError:
+ raise AirflowException(
+ "Wrong response '{}' returned - it should contain "
+ "'name' field".format(response))
+ return self._wait_for_operation_to_complete(project_id, operation_name, zone)
+
+ def set_machine_type(self, project_id, zone, resource_id, body):
+ """
+ Sets machine type of an instance defined by project_id, zone and resource_id.
+
+ :param project_id: Google Cloud Platform project ID where the Compute Engine
+ Instance exists
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the instance exists.
+ :type zone: str
+ :param resource_id: Name of the Compute Engine instance resource
+ :type resource_id: str
+ :param body: Body required by the Compute Engine setMachineType API,
+ as described in
+ https://cloud.google.com/compute/docs/reference/rest/v1/instances/setMachineType
+ :type body: dict
+ :return: True if the operation succeeded, raises an error otherwise.
+ :rtype: bool
+ """
+ response = self._execute_set_machine_type(project_id, zone, resource_id, body)
+ try:
+ operation_name = response["name"]
+ except KeyError:
+ raise AirflowException(
+ "Wrong response '{}' returned - it should contain "
+ "'name' field".format(response))
+ return self._wait_for_operation_to_complete(project_id, operation_name, zone)
+
+ def _execute_set_machine_type(self, project_id, zone, resource_id, body):
+ return self.get_conn().instances().setMachineType(
+ project=project_id, zone=zone, instance=resource_id, body=body)\
+ .execute(num_retries=NUM_RETRIES)
+
+ def get_instance_template(self, project_id, resource_id):
+ """
+ Retrieves instance template by project_id and resource_id.
+
+ :param project_id: Google Cloud Platform project ID where the Compute Engine
+ Instance template exists
+ :type project_id: str
+ :param resource_id: Name of the instance template
+ :type resource_id: str
+ :return: Instance template representation as object according to
+ https://cloud.google.com/compute/docs/reference/rest/v1/instanceTemplates
+ :rtype: dict
+ """
+ response = self.get_conn().instanceTemplates().get(
+ project=project_id,
+ instanceTemplate=resource_id
+ ).execute(num_retries=NUM_RETRIES)
+ return response
+
+ def insert_instance_template(self, project_id, body, request_id=None):
+ """
+ Inserts instance template using body specified
+
+ :param project_id: Google Cloud Platform project ID where the Compute Engine
+ Instance exists
+ :type project_id: str
+ :param body: Instance template representation as object according to
+ https://cloud.google.com/compute/docs/reference/rest/v1/instanceTemplates
+ :type body: dict
+ :param request_id: Optional, unique request_id that you might add to achieve
+ full idempotence (for example when client call times out repeating the request
+ with the same request id will not create a new instance template again)
+ It should be in UUID format as defined in RFC 4122
+ :type request_id: str
+ :return: True if the operation succeeded
+ :rtype: bool
+ """
+ response = self.get_conn().instanceTemplates().insert(
+ project=project_id,
+ body=body,
+ requestId=request_id
+ ).execute(num_retries=NUM_RETRIES)
+ try:
+ operation_name = response["name"]
+ except KeyError:
+ raise AirflowException(
+ "Wrong response '{}' returned - it should contain "
+ "'name' field".format(response))
+ return self._wait_for_operation_to_complete(project_id, operation_name)
+
+ def get_instance_group_manager(self, project_id, zone, resource_id):
+ """
+ Retrieves Instance Group Manager by project_id, zone and resource_id.
+
+ :param project_id: Google Cloud Platform project ID where the Compute Engine
+ Instance Group Manager exists
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the Instance Group Manager exists
+ :type zone: str
+ :param resource_id: Name of the Instance Group Manager
+ :type resource_id: str
+ :return: Instance group manager representation as object according to
+ https://cloud.google.com/compute/docs/reference/rest/beta/instanceGroupManagers
+ :rtype: dict
+ """
+ response = self.get_conn().instanceGroupManagers().get(
+ project=project_id,
+ zone=zone,
+ instanceGroupManager=resource_id
+ ).execute(num_retries=NUM_RETRIES)
+ return response
+
+ def patch_instance_group_manager(self, project_id, zone, resource_id,
+ body, request_id=None):
+ """
+ Patches Instance Group Manager with the specified body.
+
+ :param project_id: Google Cloud Platform project ID where the Compute Engine
+ Instance Group Manager exists
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the Instance Group Manager exists
+ :type zone: str
+ :param resource_id: Name of the Instance Group Manager
+ :type resource_id: str
+ :param body: Instance Group Manager representation as json-merge-patch object
+ according to
+ https://cloud.google.com/compute/docs/reference/rest/beta/instanceTemplates/patch
+ :type body: dict
+ :param request_id: Optional, unique request_id that you might add to achieve
+ full idempotence (for example when client call times out repeating the request
+ with the same request id will not create a new instance template again).
+ It should be in UUID format as defined in RFC 4122
+ :type request_id: str
+ :return: True if the operation succeeded
+ :rtype: bool
+ """
+ response = self.get_conn().instanceGroupManagers().patch(
+ project=project_id,
+ zone=zone,
+ instanceGroupManager=resource_id,
+ body=body,
+ requestId=request_id
+ ).execute(num_retries=NUM_RETRIES)
+ try:
+ operation_name = response["name"]
+ except KeyError:
+ raise AirflowException(
+ "Wrong response '{}' returned - it should contain "
+ "'name' field".format(response))
+ return self._wait_for_operation_to_complete(project_id, operation_name, zone)
+
+ def _wait_for_operation_to_complete(self, project_id, operation_name, zone=None):
+ """
+ Waits for the named operation to complete - checks status of the async call.
+
+ :param operation_name: name of the operation
+ :type operation_name: str
+ :param zone: optional region of the request (might be None for global operations)
+ :type zone: str
+ :return: True if the operation succeeded, raises an error otherwise
+ :rtype: bool
+ """
+ service = self.get_conn()
+ while True:
+ if zone is None:
+ # noinspection PyTypeChecker
+ operation_response = self._check_global_operation_status(
+ service, operation_name, project_id)
+ else:
+ # noinspection PyTypeChecker
+ operation_response = self._check_zone_operation_status(
+ service, operation_name, project_id, zone)
+ if operation_response.get("status") == GceOperationStatus.DONE:
+ error = operation_response.get("error")
+ if error:
+ code = operation_response.get("httpErrorStatusCode")
+ msg = operation_response.get("httpErrorMessage")
+ # Extracting the errors list as string and trimming square braces
+ error_msg = str(error.get("errors"))[1:-1]
+ raise AirflowException("{} {}: ".format(code, msg) + error_msg)
+ # No meaningful info to return from the response in case of success
+ return True
+ time.sleep(TIME_TO_SLEEP_IN_SECONDS)
+
+ @staticmethod
+ def _check_zone_operation_status(service, operation_name, project_id, zone):
+ return service.zoneOperations().get(
+ project=project_id, zone=zone, operation=operation_name).execute(
+ num_retries=NUM_RETRIES)
+
+ @staticmethod
+ def _check_global_operation_status(service, operation_name, project_id):
+ return service.globalOperations().get(
+ project=project_id, operation=operation_name).execute(
+ num_retries=NUM_RETRIES)
diff --git a/airflow/contrib/hooks/gcp_container_hook.py b/airflow/contrib/hooks/gcp_container_hook.py
new file mode 100644
index 0000000000..4a610e56c9
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_container_hook.py
@@ -0,0 +1,243 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+import json
+import time
+
+from airflow import AirflowException, version
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+from google.api_core.exceptions import AlreadyExists, NotFound
+from google.api_core.gapic_v1.method import DEFAULT
+from google.cloud import container_v1, exceptions
+from google.cloud.container_v1.gapic.enums import Operation
+from google.cloud.container_v1.types import Cluster
+from google.protobuf import json_format
+from google.api_core.gapic_v1.client_info import ClientInfo
+
+OPERATIONAL_POLL_INTERVAL = 15
+
+
+class GKEClusterHook(GoogleCloudBaseHook):
+
+ def __init__(self,
+ gcp_conn_id='google_cloud_default',
+ delegate_to=None,
+ location=None):
+ super(GKEClusterHook, self).__init__(
+ gcp_conn_id=gcp_conn_id, delegate_to=delegate_to)
+ self._client = None
+ self.location = location
+
+ def get_client(self):
+ if self._client is None:
+ credentials = self._get_credentials()
+ # Add client library info for better error tracking
+ client_info = ClientInfo(client_library_version='airflow_v' + version.version)
+ self._client = container_v1.ClusterManagerClient(credentials=credentials, client_info=client_info)
+ return self._client
+
+ @staticmethod
+ def _dict_to_proto(py_dict, proto):
+ """
+ Converts a python dictionary to the proto supplied
+
+ :param py_dict: The dictionary to convert
+ :type py_dict: dict
+ :param proto: The proto object to merge with dictionary
+ :type proto: protobuf
+ :return: A parsed python dictionary in provided proto format
+ :raises:
+ ParseError: On JSON parsing problems.
+ """
+ dict_json_str = json.dumps(py_dict)
+ return json_format.Parse(dict_json_str, proto)
+
+ def wait_for_operation(self, operation, project_id=None):
+ """
+ Given an operation, continuously fetches the status from Google Cloud until either
+ completion or an error occurring
+
+ :param operation: The Operation to wait for
+ :type operation: A google.cloud.container_V1.gapic.enums.Operator
+ :param project_id: Google Cloud Platform project ID
+ :type project_id: str
+ :return: A new, updated operation fetched from Google Cloud
+ """
+ self.log.info("Waiting for OPERATION_NAME %s" % operation.name)
+ time.sleep(OPERATIONAL_POLL_INTERVAL)
+ while operation.status != Operation.Status.DONE:
+ if operation.status == Operation.Status.RUNNING or operation.status == \
+ Operation.Status.PENDING:
+ time.sleep(OPERATIONAL_POLL_INTERVAL)
+ else:
+ raise exceptions.GoogleCloudError(
+ "Operation has failed with status: %s" % operation.status)
+ # To update status of operation
+ operation = self.get_operation(operation.name, project_id=project_id or self.project_id)
+ return operation
+
+ def get_operation(self, operation_name, project_id=None):
+ """
+ Fetches the operation from Google Cloud
+
+ :param operation_name: Name of operation to fetch
+ :type operation_name: str
+ :param project_id: Google Cloud Platform project ID
+ :type project_id: str
+ :return: The new, updated operation from Google Cloud
+ """
+ return self.get_client().get_operation(project_id=project_id or self.project_id,
+ zone=self.location,
+ operation_id=operation_name)
+
+ @staticmethod
+ def _append_label(cluster_proto, key, val):
+ """
+ Append labels to provided Cluster Protobuf
+
+ Labels must fit the regex [a-z]([-a-z0-9]*[a-z0-9])? (current airflow version
+ string follows semantic versioning spec: x.y.z).
+
+ :param cluster_proto: The proto to append resource_label airflow version to
+ :type cluster_proto: google.cloud.container_v1.types.Cluster
+ :param key: The key label
+ :type key: str
+ :param val:
+ :type val: str
+ :return: The cluster proto updated with new label
+ """
+ val = val.replace('.', '-').replace('+', '-')
+ cluster_proto.resource_labels.update({key: val})
+ return cluster_proto
+
+ def delete_cluster(self, name, project_id=None, retry=DEFAULT, timeout=DEFAULT):
+ """
+ Deletes the cluster, including the Kubernetes endpoint and all
+ worker nodes. Firewalls and routes that were configured during
+ cluster creation are also deleted. Other Google Compute Engine
+ resources that might be in use by the cluster (e.g. load balancer
+ resources) will not be deleted if they weren’t present at the
+ initial create time.
+
+ :param name: The name of the cluster to delete
+ :type name: str
+ :param project_id: Google Cloud Platform project ID
+ :type project_id: str
+ :param retry: Retry object used to determine when/if to retry requests.
+ If None is specified, requests will not be retried.
+ :type retry: google.api_core.retry.Retry
+ :param timeout: The amount of time, in seconds, to wait for the request to
+ complete. Note that if retry is specified, the timeout applies to each
+ individual attempt.
+ :type timeout: float
+ :return: The full url to the delete operation if successful, else None
+ """
+
+ self.log.info("Deleting (project_id={}, zone={}, cluster_id={})".format(
+ self.project_id, self.location, name))
+
+ try:
+ op = self.get_client().delete_cluster(project_id=project_id or self.project_id,
+ zone=self.location,
+ cluster_id=name,
+ retry=retry,
+ timeout=timeout)
+ op = self.wait_for_operation(op)
+ # Returns server-defined url for the resource
+ return op.self_link
+ except NotFound as error:
+ self.log.info('Assuming Success: ' + error.message)
+
+ def create_cluster(self, cluster, project_id=None, retry=DEFAULT, timeout=DEFAULT):
+ """
+ Creates a cluster, consisting of the specified number and type of Google Compute
+ Engine instances.
+
+ :param cluster: A Cluster protobuf or dict. If dict is provided, it must be of
+ the same form as the protobuf message google.cloud.container_v1.types.Cluster
+ :type cluster: dict or google.cloud.container_v1.types.Cluster
+ :param project_id: Google Cloud Platform project ID
+ :type project_id: str
+ :param retry: A retry object (google.api_core.retry.Retry) used to retry requests.
+ If None is specified, requests will not be retried.
+ :type retry: google.api_core.retry.Retry
+ :param timeout: The amount of time, in seconds, to wait for the request to
+ complete. Note that if retry is specified, the timeout applies to each
+ individual attempt.
+ :type timeout: float
+ :return: The full url to the new, or existing, cluster
+ :raises
+ ParseError: On JSON parsing problems when trying to convert dict
+ AirflowException: cluster is not dict type nor Cluster proto type
+ """
+
+ if isinstance(cluster, dict):
+ cluster_proto = Cluster()
+ cluster = self._dict_to_proto(py_dict=cluster, proto=cluster_proto)
+ elif not isinstance(cluster, Cluster):
+ raise AirflowException(
+ "cluster is not instance of Cluster proto or python dict")
+
+ self._append_label(cluster, 'airflow-version', 'v' + version.version)
+
+ self.log.info("Creating (project_id={}, zone={}, cluster_name={})".format(
+ self.project_id,
+ self.location,
+ cluster.name))
+ try:
+ op = self.get_client().create_cluster(project_id=project_id or self.project_id,
+ zone=self.location,
+ cluster=cluster,
+ retry=retry,
+ timeout=timeout)
+ op = self.wait_for_operation(op)
+
+ return op.target_link
+ except AlreadyExists as error:
+ self.log.info('Assuming Success: ' + error.message)
+ return self.get_cluster(name=cluster.name).self_link
+
+ def get_cluster(self, name, project_id=None, retry=DEFAULT, timeout=DEFAULT):
+ """
+ Gets details of specified cluster
+
+ :param name: The name of the cluster to retrieve
+ :type name: str
+ :param project_id: Google Cloud Platform project ID
+ :type project_id: str
+ :param retry: A retry object used to retry requests. If None is specified,
+ requests will not be retried.
+ :type retry: google.api_core.retry.Retry
+ :param timeout: The amount of time, in seconds, to wait for the request to
+ complete. Note that if retry is specified, the timeout applies to each
+ individual attempt.
+ :type timeout: float
+ :return: A google.cloud.container_v1.types.Cluster instance
+ """
+ self.log.info("Fetching cluster (project_id={}, zone={}, cluster_name={})".format(
+ project_id or self.project_id,
+ self.location,
+ name))
+
+ return self.get_client().get_cluster(project_id=project_id or self.project_id,
+ zone=self.location,
+ cluster_id=name,
+ retry=retry,
+ timeout=timeout).self_link
diff --git a/airflow/contrib/hooks/gcp_dataflow_hook.py b/airflow/contrib/hooks/gcp_dataflow_hook.py
index dc03f05a5e..0eee769d61 100644
--- a/airflow/contrib/hooks/gcp_dataflow_hook.py
+++ b/airflow/contrib/hooks/gcp_dataflow_hook.py
@@ -1,17 +1,23 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
import json
+import re
import select
import subprocess
import time
@@ -22,20 +28,27 @@
from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
from airflow.utils.log.logging_mixin import LoggingMixin
+# This is the default location
+# https://cloud.google.com/dataflow/pipelines/specifying-exec-params
+DEFAULT_DATAFLOW_LOCATION = 'us-central1'
+
class _DataflowJob(LoggingMixin):
- def __init__(self, dataflow, project_number, name, poll_sleep=10):
+ def __init__(self, dataflow, project_number, name, location, poll_sleep=10,
+ job_id=None):
self._dataflow = dataflow
self._project_number = project_number
self._job_name = name
- self._job_id = None
+ self._job_location = location
+ self._job_id = job_id
self._job = self._get_job()
self._poll_sleep = poll_sleep
def _get_job_id_from_name(self):
- jobs = self._dataflow.projects().jobs().list(
- projectId=self._project_number
- ).execute()
+ jobs = self._dataflow.projects().locations().jobs().list(
+ projectId=self._project_number,
+ location=self._job_location
+ ).execute(num_retries=5)
for job in jobs['jobs']:
if job['name'] == self._job_name:
self._job_id = job['id']
@@ -43,26 +56,36 @@ def _get_job_id_from_name(self):
return None
def _get_job(self):
- if self._job_id is None:
+ if self._job_id:
+ job = self._dataflow.projects().locations().jobs().get(
+ projectId=self._project_number,
+ location=self._job_location,
+ jobId=self._job_id).execute(num_retries=5)
+ elif self._job_name:
job = self._get_job_id_from_name()
else:
- job = self._dataflow.projects().jobs().get(projectId=self._project_number,
- jobId=self._job_id).execute()
- if 'currentState' in job:
+ raise Exception('Missing both dataflow job ID and name.')
+
+ if job and 'currentState' in job:
self.log.info(
'Google Cloud DataFlow job %s is %s',
job['name'], job['currentState']
)
- else:
+ elif job:
self.log.info(
'Google Cloud DataFlow with job_id %s has name %s',
self._job_id, job['name']
)
+ else:
+ self.log.info(
+ 'Google Cloud DataFlow job not available yet..'
+ )
+
return job
def wait_for_done(self):
while True:
- if 'currentState' in self._job:
+ if self._job and 'currentState' in self._job:
if 'JOB_STATE_DONE' == self._job['currentState']:
return True
elif 'JOB_STATE_RUNNING' == self._job['currentState'] and \
@@ -104,35 +127,50 @@ def __init__(self, cmd):
def _line(self, fd):
if fd == self._proc.stderr.fileno():
- lines = self._proc.stderr.readlines()
- for line in lines:
- self.log.warning(line[:-1])
- line = lines[-1][:-1]
+ line = b''.join(self._proc.stderr.readlines())
+ if line:
+ self.log.warning(line[:-1])
return line
if fd == self._proc.stdout.fileno():
- line = self._proc.stdout.readline()
+ line = b''.join(self._proc.stdout.readlines())
+ if line:
+ self.log.info(line[:-1])
return line
@staticmethod
def _extract_job(line):
- if line is not None:
- if line.startswith("Submitted job: "):
- return line[15:-1]
+ # Job id info: https://goo.gl/SE29y9.
+ job_id_pattern = re.compile(
+ b'.*console.cloud.google.com/dataflow.*/jobs/([a-z|0-9|A-Z|\-|\_]+).*')
+ matched_job = job_id_pattern.search(line or '')
+ if matched_job:
+ return matched_job.group(1).decode()
def wait_for_done(self):
reads = [self._proc.stderr.fileno(), self._proc.stdout.fileno()]
self.log.info("Start waiting for DataFlow process to complete.")
- while self._proc.poll() is None:
+ job_id = None
+ # Make sure logs are processed regardless whether the subprocess is
+ # terminated.
+ process_ends = False
+ while True:
ret = select.select(reads, [], [], 5)
if ret is not None:
for fd in ret[0]:
line = self._line(fd)
- self.log.debug(line[:-1])
+ if line:
+ job_id = job_id or self._extract_job(line)
else:
self.log.info("Waiting for DataFlow process to complete.")
+ if process_ends:
+ break
+ if self._proc.poll() is not None:
+ # Mark process completion but allows its outputs to be consumed.
+ process_ends = True
if self._proc.returncode is not 0:
raise Exception("DataFlow failed with return code {}".format(
self._proc.returncode))
+ return job_id
class DataFlowHook(GoogleCloudBaseHook):
@@ -146,60 +184,77 @@ def __init__(self,
def get_conn(self):
"""
- Returns a Google Cloud Storage service object.
+ Returns a Google Cloud Dataflow service object.
"""
http_authorized = self._authorize()
- return build('dataflow', 'v1b3', http=http_authorized)
+ return build(
+ 'dataflow', 'v1b3', http=http_authorized, cache_discovery=False)
+
+ def _start_dataflow(self, variables, name, command_prefix, label_formatter):
+ variables = self._set_variables(variables)
+ cmd = command_prefix + self._build_cmd(variables, label_formatter)
+ job_id = _Dataflow(cmd).wait_for_done()
+ _DataflowJob(self.get_conn(), variables['project'], name,
+ variables['region'],
+ self.poll_sleep, job_id).wait_for_done()
- def _start_dataflow(self, task_id, variables, name,
- command_prefix, label_formatter):
- cmd = command_prefix + self._build_cmd(task_id, variables,
- label_formatter)
- _Dataflow(cmd).wait_for_done()
- _DataflowJob(self.get_conn(), variables['project'],
- name, self.poll_sleep).wait_for_done()
+ @staticmethod
+ def _set_variables(variables):
+ if variables['project'] is None:
+ raise Exception('Project not specified')
+ if 'region' not in variables.keys():
+ variables['region'] = DEFAULT_DATAFLOW_LOCATION
+ return variables
- def start_java_dataflow(self, task_id, variables, dataflow, job_class=None,
+ def start_java_dataflow(self, job_name, variables, dataflow, job_class=None,
append_job_name=True):
- if append_job_name:
- name = task_id + "-" + str(uuid.uuid1())[:8]
- else:
- name = task_id
+ name = self._build_dataflow_job_name(job_name, append_job_name)
variables['jobName'] = name
def label_formatter(labels_dict):
return ['--labels={}'.format(
- json.dumps(labels_dict).replace(' ', ''))]
+ json.dumps(labels_dict).replace(' ', ''))]
command_prefix = (["java", "-cp", dataflow, job_class] if job_class
else ["java", "-jar", dataflow])
- self._start_dataflow(task_id, variables, name,
- command_prefix, label_formatter)
+ self._start_dataflow(variables, name, command_prefix, label_formatter)
- def start_template_dataflow(self, task_id, variables, parameters, dataflow_template,
+ def start_template_dataflow(self, job_name, variables, parameters, dataflow_template,
append_job_name=True):
- if append_job_name:
- name = task_id + "-" + str(uuid.uuid1())[:8]
- else:
- name = task_id
+ variables = self._set_variables(variables)
+ name = self._build_dataflow_job_name(job_name, append_job_name)
self._start_template_dataflow(
name, variables, parameters, dataflow_template)
- def start_python_dataflow(self, task_id, variables, dataflow, py_options,
+ def start_python_dataflow(self, job_name, variables, dataflow, py_options,
append_job_name=True):
- if append_job_name:
- name = task_id + "-" + str(uuid.uuid1())[:8]
- else:
- name = task_id
- variables["job_name"] = name
+ name = self._build_dataflow_job_name(job_name, append_job_name)
+ variables['job_name'] = name
def label_formatter(labels_dict):
return ['--labels={}={}'.format(key, value)
for key, value in labels_dict.items()]
- self._start_dataflow(task_id, variables, name,
- ["python"] + py_options + [dataflow],
+ self._start_dataflow(variables, name, ["python2"] + py_options + [dataflow],
label_formatter)
- def _build_cmd(self, task_id, variables, label_formatter):
+ @staticmethod
+ def _build_dataflow_job_name(job_name, append_job_name=True):
+ base_job_name = str(job_name).replace('_', '-')
+
+ if not re.match(r"^[a-z]([-a-z0-9]*[a-z0-9])?$", base_job_name):
+ raise ValueError(
+ 'Invalid job_name ({}); the name must consist of'
+ 'only the characters [-a-z0-9], starting with a '
+ 'letter and ending with a letter or number '.format(base_job_name))
+
+ if append_job_name:
+ safe_job_name = base_job_name + "-" + str(uuid.uuid4())[:8]
+ else:
+ safe_job_name = base_job_name
+
+ return safe_job_name
+
+ @staticmethod
+ def _build_cmd(variables, label_formatter):
command = ["--runner=DataflowRunner"]
if variables is not None:
for attr, value in variables.items():
@@ -211,7 +266,8 @@ def _build_cmd(self, task_id, variables, label_formatter):
command.append("--" + attr + "=" + value)
return command
- def _start_template_dataflow(self, name, variables, parameters, dataflow_template):
+ def _start_template_dataflow(self, name, variables, parameters,
+ dataflow_template):
# Builds RuntimeEnvironment from variables dictionary
# https://cloud.google.com/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment
environment = {}
@@ -223,13 +279,14 @@ def _start_template_dataflow(self, name, variables, parameters, dataflow_templat
"parameters": parameters,
"environment": environment}
service = self.get_conn()
- if variables['project'] is None:
- raise Exception(
- 'Project not specified')
- request = service.projects().templates().launch(projectId=variables['project'],
- gcsPath=dataflow_template,
- body=body)
+ request = service.projects().locations().templates().launch(
+ projectId=variables['project'],
+ location=variables['region'],
+ gcsPath=dataflow_template,
+ body=body
+ )
response = request.execute()
- _DataflowJob(
- self.get_conn(), variables['project'], name, self.poll_sleep).wait_for_done()
+ variables = self._set_variables(variables)
+ _DataflowJob(self.get_conn(), variables['project'], name, variables['region'],
+ self.poll_sleep).wait_for_done()
return response
diff --git a/airflow/contrib/hooks/gcp_dataproc_hook.py b/airflow/contrib/hooks/gcp_dataproc_hook.py
index bc5aa83ac9..e068d65cbe 100644
--- a/airflow/contrib/hooks/gcp_dataproc_hook.py
+++ b/airflow/contrib/hooks/gcp_dataproc_hook.py
@@ -1,28 +1,35 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
import time
import uuid
from apiclient.discovery import build
+from zope.deprecation import deprecation
from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
from airflow.utils.log.logging_mixin import LoggingMixin
class _DataProcJob(LoggingMixin):
- def __init__(self, dataproc_api, project_id, job, region='global'):
+ def __init__(self, dataproc_api, project_id, job, region='global',
+ job_error_states=None):
self.dataproc_api = dataproc_api
self.project_id = project_id
self.region = region
@@ -31,6 +38,7 @@ def __init__(self, dataproc_api, project_id, job, region='global'):
region=self.region,
body=job).execute()
self.job_id = self.job['reference']['jobId']
+ self.job_error_states = job_error_states
self.log.info(
'DataProc job %s is %s',
self.job_id, str(self.job['status']['state'])
@@ -43,19 +51,23 @@ def wait_for_done(self):
region=self.region,
jobId=self.job_id).execute(num_retries=5)
if 'ERROR' == self.job['status']['state']:
- print(str(self.job))
self.log.error('DataProc job %s has errors', self.job_id)
self.log.error(self.job['status']['details'])
self.log.debug(str(self.job))
+ self.log.info('Driver output location: %s',
+ self.job['driverOutputResourceUri'])
return False
if 'CANCELLED' == self.job['status']['state']:
- print(str(self.job))
self.log.warning('DataProc job %s is cancelled', self.job_id)
if 'details' in self.job['status']:
self.log.warning(self.job['status']['details'])
self.log.debug(str(self.job))
+ self.log.info('Driver output location: %s',
+ self.job['driverOutputResourceUri'])
return False
if 'DONE' == self.job['status']['state']:
+ self.log.info('Driver output location: %s',
+ self.job['driverOutputResourceUri'])
return True
self.log.debug(
'DataProc job %s is %s',
@@ -64,10 +76,14 @@ def wait_for_done(self):
time.sleep(5)
def raise_error(self, message=None):
- if 'ERROR' == self.job['status']['state']:
- if message is None:
- message = "Google DataProc job has error"
- raise Exception(message + ": " + str(self.job['status']['details']))
+ job_state = self.job['status']['state']
+ # We always consider ERROR to be an error state.
+ if (self.job_error_states and job_state in self.job_error_states) or 'ERROR' == job_state:
+ ex_message = message or ("Google DataProc job has state: %s" % job_state)
+ ex_details = (str(self.job['status']['details'])
+ if 'details' in self.job['status']
+ else "No details available")
+ raise Exception(ex_message + ": " + ex_details)
def get(self):
return self.job
@@ -75,7 +91,7 @@ def get(self):
class _DataProcJobBuilder:
def __init__(self, project_id, task_id, cluster_name, job_type, properties):
- name = task_id + "_" + str(uuid.uuid1())[:8]
+ name = task_id + "_" + str(uuid.uuid4())[:8]
self.job_type = job_type
self.job = {
"job": {
@@ -135,7 +151,7 @@ def set_python_main(self, main):
self.job["job"][self.job_type]["mainPythonFileUri"] = main
def set_job_name(self, name):
- self.job["job"]["reference"]["jobId"] = name + "_" + str(uuid.uuid1())[:8]
+ self.job["job"]["reference"]["jobId"] = name + "_" + str(uuid.uuid4())[:8]
def build(self):
return self.job
@@ -192,25 +208,44 @@ class DataProcHook(GoogleCloudBaseHook):
def __init__(self,
gcp_conn_id='google_cloud_default',
delegate_to=None,
- api_version='v1'):
+ api_version='v1beta2'):
super(DataProcHook, self).__init__(gcp_conn_id, delegate_to)
self.api_version = api_version
def get_conn(self):
"""Returns a Google Cloud Dataproc service object."""
http_authorized = self._authorize()
- return build('dataproc', self.api_version, http=http_authorized)
-
- def submit(self, project_id, job, region='global'):
- submitted = _DataProcJob(self.get_conn(), project_id, job, region)
+ return build(
+ 'dataproc', self.api_version, http=http_authorized,
+ cache_discovery=False)
+
+ def get_cluster(self, project_id, region, cluster_name):
+ return self.get_conn().projects().regions().clusters().get(
+ projectId=project_id,
+ region=region,
+ clusterName=cluster_name
+ ).execute(num_retries=5)
+
+ def submit(self, project_id, job, region='global', job_error_states=None):
+ submitted = _DataProcJob(self.get_conn(), project_id, job, region,
+ job_error_states=job_error_states)
if not submitted.wait_for_done():
- submitted.raise_error('DataProcTask has errors')
+ submitted.raise_error()
def create_job_template(self, task_id, cluster_name, job_type, properties):
return _DataProcJobBuilder(self.project_id, task_id, cluster_name,
job_type, properties)
- def await(self, operation):
+ def wait(self, operation):
"""Awaits for Google Cloud Dataproc Operation to complete."""
submitted = _DataProcOperation(self.get_conn(), operation)
submitted.wait_for_done()
+
+
+setattr(
+ DataProcHook,
+ "await",
+ deprecation.deprecated(
+ DataProcHook.wait, "renamed to 'wait' for Python3.7 compatibility"
+ ),
+)
diff --git a/airflow/contrib/hooks/gcp_function_hook.py b/airflow/contrib/hooks/gcp_function_hook.py
new file mode 100644
index 0000000000..29cef1716c
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_function_hook.py
@@ -0,0 +1,195 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import time
+import requests
+from googleapiclient.discovery import build
+
+from airflow import AirflowException
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+# Number of retries - used by googleapiclient method calls to perform retries
+# For requests that are "retriable"
+NUM_RETRIES = 5
+
+# Time to sleep between active checks of the operation results
+TIME_TO_SLEEP_IN_SECONDS = 1
+
+
+# noinspection PyAbstractClass
+class GcfHook(GoogleCloudBaseHook):
+ """
+ Hook for the Google Cloud Functions APIs.
+ """
+ _conn = None
+
+ def __init__(self,
+ api_version,
+ gcp_conn_id='google_cloud_default',
+ delegate_to=None):
+ super(GcfHook, self).__init__(gcp_conn_id, delegate_to)
+ self.api_version = api_version
+
+ def get_conn(self):
+ """
+ Retrieves the connection to Cloud Functions.
+
+ :return: Google Cloud Function services object
+ :rtype: dict
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build('cloudfunctions', self.api_version,
+ http=http_authorized, cache_discovery=False)
+ return self._conn
+
+ def get_function(self, name):
+ """
+ Returns the Cloud Function with the given name.
+
+ :param name: name of the function
+ :type name: str
+ :return: a Cloud Functions object representing the function
+ :rtype: dict
+ """
+ return self.get_conn().projects().locations().functions().get(
+ name=name).execute(num_retries=NUM_RETRIES)
+
+ def list_functions(self, full_location):
+ """
+ Lists all Cloud Functions created in the location.
+
+ :param full_location: full location including the project in the form of
+ of /projects/<PROJECT>/location/<LOCATION>
+ :type full_location: str
+ :return: array of Cloud Functions objects - representing functions in the location
+ :rtype: [dict]
+ """
+ list_response = self.get_conn().projects().locations().functions().list(
+ parent=full_location).execute(num_retries=NUM_RETRIES)
+ return list_response.get("functions", [])
+
+ def create_new_function(self, full_location, body):
+ """
+ Creates a new function in Cloud Function in the location specified in the body.
+
+ :param full_location: full location including the project in the form of
+ of /projects/<PROJECT>/location/<LOCATION>
+ :type full_location: str
+ :param body: body required by the Cloud Functions insert API
+ :type body: dict
+ :return: response returned by the operation
+ :rtype: dict
+ """
+ response = self.get_conn().projects().locations().functions().create(
+ location=full_location,
+ body=body
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(operation_name)
+
+ def update_function(self, name, body, update_mask):
+ """
+ Updates Cloud Functions according to the specified update mask.
+
+ :param name: name of the function
+ :type name: str
+ :param body: body required by the cloud function patch API
+ :type body: str
+ :param update_mask: update mask - array of fields that should be patched
+ :type update_mask: [str]
+ :return: response returned by the operation
+ :rtype: dict
+ """
+ response = self.get_conn().projects().locations().functions().patch(
+ updateMask=",".join(update_mask),
+ name=name,
+ body=body
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(operation_name)
+
+ def upload_function_zip(self, parent, zip_path):
+ """
+ Uploads zip file with sources.
+
+ :param parent: Google Cloud Platform project id and region where zip file should
+ be uploaded in the form of /projects/<PROJECT>/location/<LOCATION>
+ :type parent: str
+ :param zip_path: path of the valid .zip file to upload
+ :type zip_path: str
+ :return: Upload URL that was returned by generateUploadUrl method
+ """
+ response = self.get_conn().projects().locations().functions().generateUploadUrl(
+ parent=parent
+ ).execute(num_retries=NUM_RETRIES)
+ upload_url = response.get('uploadUrl')
+ with open(zip_path, 'rb') as fp:
+ requests.put(
+ url=upload_url,
+ data=fp.read(),
+ # Those two headers needs to be specified according to:
+ # https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions/generateUploadUrl
+ # nopep8
+ headers={
+ 'Content-type': 'application/zip',
+ 'x-goog-content-length-range': '0,104857600',
+ }
+ )
+ return upload_url
+
+ def delete_function(self, name):
+ """
+ Deletes the specified Cloud Function.
+
+ :param name: name of the function
+ :type name: str
+ :return: response returned by the operation
+ :rtype: dict
+ """
+ response = self.get_conn().projects().locations().functions().delete(
+ name=name).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(operation_name)
+
+ def _wait_for_operation_to_complete(self, operation_name):
+ """
+ Waits for the named operation to complete - checks status of the
+ asynchronous call.
+
+ :param operation_name: name of the operation
+ :type operation_name: str
+ :return: response returned by the operation
+ :rtype: dict
+ :exception: AirflowException in case error is returned
+ """
+ service = self.get_conn()
+ while True:
+ operation_response = service.operations().get(
+ name=operation_name,
+ ).execute(num_retries=NUM_RETRIES)
+ if operation_response.get("done"):
+ response = operation_response.get("response")
+ error = operation_response.get("error")
+ # Note, according to documentation always either response or error is
+ # set when "done" == True
+ if error:
+ raise AirflowException(str(error))
+ return response
+ time.sleep(TIME_TO_SLEEP_IN_SECONDS)
diff --git a/airflow/contrib/hooks/gcp_kms_hook.py b/airflow/contrib/hooks/gcp_kms_hook.py
new file mode 100644
index 0000000000..6f2b3aedff
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_kms_hook.py
@@ -0,0 +1,108 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import base64
+
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+from apiclient.discovery import build
+
+
+def _b64encode(s):
+ """ Base 64 encodes a bytes object to a string """
+ return base64.b64encode(s).decode('ascii')
+
+
+def _b64decode(s):
+ """ Base 64 decodes a string to bytes. """
+ return base64.b64decode(s.encode('utf-8'))
+
+
+class GoogleCloudKMSHook(GoogleCloudBaseHook):
+ """
+ Interact with Google Cloud KMS. This hook uses the Google Cloud Platform
+ connection.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleCloudKMSHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a KMS service object.
+
+ :rtype: apiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build(
+ 'cloudkms', 'v1', http=http_authorized, cache_discovery=False)
+
+ def encrypt(self, key_name, plaintext, authenticated_data=None):
+ """
+ Encrypts a plaintext message using Google Cloud KMS.
+
+ :param key_name: The Resource Name for the key (or key version)
+ to be used for encyption. Of the form
+ ``projects/*/locations/*/keyRings/*/cryptoKeys/**``
+ :type key_name: str
+ :param plaintext: The message to be encrypted.
+ :type plaintext: bytes
+ :param authenticated_data: Optional additional authenticated data that
+ must also be provided to decrypt the message.
+ :type authenticated_data: bytes
+ :return: The base 64 encoded ciphertext of the original message.
+ :rtype: str
+ """
+ keys = self.get_conn().projects().locations().keyRings().cryptoKeys()
+ body = {'plaintext': _b64encode(plaintext)}
+ if authenticated_data:
+ body['additionalAuthenticatedData'] = _b64encode(authenticated_data)
+
+ request = keys.encrypt(name=key_name, body=body)
+ response = request.execute()
+
+ ciphertext = response['ciphertext']
+ return ciphertext
+
+ def decrypt(self, key_name, ciphertext, authenticated_data=None):
+ """
+ Decrypts a ciphertext message using Google Cloud KMS.
+
+ :param key_name: The Resource Name for the key to be used for decyption.
+ Of the form ``projects/*/locations/*/keyRings/*/cryptoKeys/**``
+ :type key_name: str
+ :param ciphertext: The message to be decrypted.
+ :type ciphertext: str
+ :param authenticated_data: Any additional authenticated data that was
+ provided when encrypting the message.
+ :type authenticated_data: bytes
+ :return: The original message.
+ :rtype: bytes
+ """
+ keys = self.get_conn().projects().locations().keyRings().cryptoKeys()
+ body = {'ciphertext': ciphertext}
+ if authenticated_data:
+ body['additionalAuthenticatedData'] = _b64encode(authenticated_data)
+
+ request = keys.decrypt(name=key_name, body=body)
+ response = request.execute()
+
+ plaintext = _b64decode(response['plaintext'])
+ return plaintext
diff --git a/airflow/contrib/hooks/gcp_mlengine_hook.py b/airflow/contrib/hooks/gcp_mlengine_hook.py
index c17b614a4a..9c255dbe80 100644
--- a/airflow/contrib/hooks/gcp_mlengine_hook.py
+++ b/airflow/contrib/hooks/gcp_mlengine_hook.py
@@ -17,7 +17,6 @@
import time
from apiclient import errors
from apiclient.discovery import build
-from oauth2client.client import GoogleCredentials
from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -55,8 +54,8 @@ def get_conn(self):
"""
Returns a Google MLEngine service object.
"""
- credentials = GoogleCredentials.get_application_default()
- return build('ml', 'v1', credentials=credentials)
+ authed_http = self._authorize()
+ return build('ml', 'v1', http=authed_http, cache_discovery=False)
def create_job(self, project_id, job, use_existing_job_fn=None):
"""
@@ -64,17 +63,17 @@ def create_job(self, project_id, job, use_existing_job_fn=None):
:param project_id: The Google Cloud project id within which MLEngine
job will be launched.
- :type project_id: string
+ :type project_id: str
:param job: MLEngine Job object that should be provided to the MLEngine
- API, such as:
- {
- 'jobId': 'my_job_id',
- 'trainingInput': {
- 'scaleTier': 'STANDARD_1',
- ...
- }
- }
+ API, such as: ::
+ {
+ 'jobId': 'my_job_id',
+ 'trainingInput': {
+ 'scaleTier': 'STANDARD_1',
+ ...
+ }
+ }
:type job: dict
:param use_existing_job_fn: In case that a MLEngine job with the same
@@ -153,7 +152,8 @@ def _wait_for_job_done(self, project_id, job_id, interval=30):
apiclient.errors.HttpError: if HTTP error is returned when getting
the job
"""
- assert interval > 0
+ if interval <= 0:
+ raise ValueError("Interval must be > 0")
while True:
job = self._get_job(project_id, job_id)
if job['state'] in ['SUCCEEDED', 'FAILED', 'CANCELLED']:
@@ -243,7 +243,9 @@ def create_model(self, project_id, model):
"""
Create a Model. Blocks until finished.
"""
- assert model['name'] is not None and model['name'] is not ''
+ if not model['name']:
+ raise ValueError("Model name must be provided and "
+ "could not be an empty string")
project = 'projects/{}'.format(project_id)
request = self._mlengine.projects().models().create(
@@ -254,7 +256,9 @@ def get_model(self, project_id, model_name):
"""
Gets a Model. Blocks until finished.
"""
- assert model_name is not None and model_name is not ''
+ if not model_name:
+ raise ValueError("Model name must be provided and "
+ "it could not be an empty string")
full_model_name = 'projects/{}/models/{}'.format(
project_id, model_name)
request = self._mlengine.projects().models().get(name=full_model_name)
diff --git a/airflow/contrib/hooks/gcp_pubsub_hook.py b/airflow/contrib/hooks/gcp_pubsub_hook.py
index e45ad63c6a..bf33a9b6bc 100644
--- a/airflow/contrib/hooks/gcp_pubsub_hook.py
+++ b/airflow/contrib/hooks/gcp_pubsub_hook.py
@@ -1,16 +1,21 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
from uuid import uuid4
@@ -48,16 +53,17 @@ def get_conn(self):
:rtype: apiclient.discovery.Resource
"""
http_authorized = self._authorize()
- return build('pubsub', 'v1', http=http_authorized)
+ return build(
+ 'pubsub', 'v1', http=http_authorized, cache_discovery=False)
def publish(self, project, topic, messages):
"""Publishes messages to a Pub/Sub topic.
:param project: the GCP project ID in which to publish
- :type project: string
+ :type project: str
:param topic: the Pub/Sub topic to which to publish; do not
include the ``projects/{project}/topics/`` prefix.
- :type topic: string
+ :type topic: str
:param messages: messages to publish; if the data field in a
message is set, it should already be base64 encoded.
:type messages: list of PubSub messages; see
@@ -78,10 +84,10 @@ def create_topic(self, project, topic, fail_if_exists=False):
:param project: the GCP project ID in which to create
the topic
- :type project: string
+ :type project: str
:param topic: the Pub/Sub topic name to create; do not
include the ``projects/{project}/topics/`` prefix.
- :type topic: string
+ :type topic: str
:param fail_if_exists: if set, raise an exception if the topic
already exists
:type fail_if_exists: bool
@@ -106,10 +112,10 @@ def delete_topic(self, project, topic, fail_if_not_exists=False):
"""Deletes a Pub/Sub topic if it exists.
:param project: the GCP project ID in which to delete the topic
- :type project: string
+ :type project: str
:param topic: the Pub/Sub topic name to delete; do not
include the ``projects/{project}/topics/`` prefix.
- :type topic: string
+ :type topic: str
:param fail_if_not_exists: if set, raise an exception if the topic
does not exist
:type fail_if_not_exists: bool
@@ -136,17 +142,17 @@ def create_subscription(self, topic_project, topic, subscription=None,
:param topic_project: the GCP project ID of the topic that the
subscription will be bound to.
- :type topic_project: string
+ :type topic_project: str
:param topic: the Pub/Sub topic name that the subscription will be bound
to create; do not include the ``projects/{project}/subscriptions/``
prefix.
- :type topic: string
+ :type topic: str
:param subscription: the Pub/Sub subscription name. If empty, a random
name will be generated using the uuid module
- :type subscription: string
+ :type subscription: str
:param subscription_project: the GCP project ID where the subscription
will be created. If unspecified, ``topic_project`` will be used.
- :type subscription_project: string
+ :type subscription_project: str
:param ack_deadline_secs: Number of seconds that a subscriber has to
acknowledge each message pulled from the subscription
:type ack_deadline_secs: int
@@ -155,7 +161,7 @@ def create_subscription(self, topic_project, topic, subscription=None,
:type fail_if_exists: bool
:return: subscription name which will be the system-generated value if
the ``subscription`` parameter is not supplied
- :rtype: string
+ :rtype: str
"""
service = self.get_conn()
full_topic = _format_topic(topic_project, topic)
@@ -191,10 +197,10 @@ def delete_subscription(self, project, subscription,
"""Deletes a Pub/Sub subscription, if it exists.
:param project: the GCP project ID where the subscription exists
- :type project: string
+ :type project: str
:param subscription: the Pub/Sub subscription name to delete; do not
include the ``projects/{project}/subscriptions/`` prefix.
- :type subscription: string
+ :type subscription: str
:param fail_if_not_exists: if set, raise an exception if the topic
does not exist
:type fail_if_not_exists: bool
@@ -222,10 +228,10 @@ def pull(self, project, subscription, max_messages,
"""Pulls up to ``max_messages`` messages from Pub/Sub subscription.
:param project: the GCP project ID where the subscription exists
- :type project: string
+ :type project: str
:param subscription: the Pub/Sub subscription name to pull from; do not
include the 'projects/{project}/topics/' prefix.
- :type subscription: string
+ :type subscription: str
:param max_messages: The maximum number of messages to return from
the Pub/Sub API.
:type max_messages: int
@@ -259,10 +265,10 @@ def acknowledge(self, project, subscription, ack_ids):
:param project: the GCP project name or ID in which to create
the topic
- :type project: string
+ :type project: str
:param subscription: the Pub/Sub subscription name to delete; do not
include the 'projects/{project}/topics/' prefix.
- :type subscription: string
+ :type subscription: str
:param ack_ids: List of ReceivedMessage ackIds from a previous pull
response
:type ack_ids: list
diff --git a/airflow/contrib/hooks/gcp_spanner_hook.py b/airflow/contrib/hooks/gcp_spanner_hook.py
new file mode 100644
index 0000000000..96e8bcb71c
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_spanner_hook.py
@@ -0,0 +1,208 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from google.api_core.exceptions import GoogleAPICallError
+from google.cloud.spanner_v1.client import Client
+from google.cloud.spanner_v1.database import Database
+from google.cloud.spanner_v1.instance import Instance # noqa: F401
+from google.longrunning.operations_grpc_pb2 import Operation # noqa: F401
+from typing import Optional, Callable # noqa: F401
+
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+
+# noinspection PyAbstractClass
+class CloudSpannerHook(GoogleCloudBaseHook):
+ """
+ Hook for Google Cloud Spanner APIs.
+ """
+ _client = None
+
+ def __init__(self,
+ gcp_conn_id='google_cloud_default',
+ delegate_to=None):
+ super(CloudSpannerHook, self).__init__(gcp_conn_id, delegate_to)
+
+ def get_client(self, project_id):
+ # type: (str) -> Client
+ """
+ Provides a client for interacting with Cloud Spanner API.
+
+ :param project_id: The ID of the project which owns the instances, tables and data.
+ :type project_id: str
+ :return: Client for interacting with Cloud Spanner API. See:
+ https://googleapis.github.io/google-cloud-python/latest/spanner/client-api.html#google.cloud.spanner_v1.client.Client
+ :rtype: object
+ """
+ if not self._client:
+ self._client = Client(project=project_id, credentials=self._get_credentials())
+ return self._client
+
+ def get_instance(self, project_id, instance_id):
+ # type: (str, str) -> Optional[Instance]
+ """
+ Gets information about a particular instance.
+
+ :param project_id: The ID of the project which owns the instances, tables and data.
+ :type project_id: str
+ :param instance_id: The ID of the instance.
+ :type instance_id: str
+ :return: Representation of a Cloud Spanner Instance. See:
+ https://googleapis.github.io/google-cloud-python/latest/spanner/instance-api.html#google.cloud.spanner_v1.instance.Instance
+ :rtype: object
+ """
+ client = self.get_client(project_id)
+ instance = client.instance(instance_id)
+ if not instance.exists():
+ return None
+ return instance
+
+ def create_instance(self, project_id, instance_id, configuration_name, node_count,
+ display_name):
+ # type: (str, str, str, int, str) -> bool
+ """
+ Creates a new Cloud Spanner instance.
+
+ :param project_id: The ID of the project which owns the instances, tables and
+ data.
+ :type project_id: str
+ :param instance_id: The ID of the instance.
+ :type instance_id: str
+ :param configuration_name: Name of the instance configuration defining how the
+ instance will be created. Required for instances which do not yet exist.
+ :type configuration_name: str
+ :param node_count: (Optional) Number of nodes allocated to the instance.
+ :type node_count: int
+ :param display_name: (Optional) The display name for the instance in the Cloud
+ Console UI. (Must be between 4 and 30 characters.) If this value is not set
+ in the constructor, will fall back to the instance ID.
+ :type display_name: str
+ :return: True if the operation succeeded, raises an exception otherwise.
+ :rtype: bool
+ """
+ return self._apply_to_instance(project_id, instance_id, configuration_name,
+ node_count, display_name, lambda x: x.create())
+
+ def update_instance(self, project_id, instance_id, configuration_name, node_count,
+ display_name):
+ # type: (str, str, str, int, str) -> bool
+ """
+ Updates an existing Cloud Spanner instance.
+
+ :param project_id: The ID of the project which owns the instances, tables and
+ data.
+ :type project_id: str
+ :param instance_id: The ID of the instance.
+ :type instance_id: str
+ :param configuration_name: Name of the instance configuration defining how the
+ instance will be created. Required for instances which do not yet exist.
+ :type configuration_name: str
+ :param node_count: (Optional) Number of nodes allocated to the instance.
+ :type node_count: int
+ :param display_name: (Optional) The display name for the instance in the Cloud
+ Console UI. (Must be between 4 and 30 characters.) If this value is not set
+ in the constructor, will fall back to the instance ID.
+ :type display_name: str
+ :return: True if the operation succeeded, raises an exception otherwise.
+ :rtype: bool
+ """
+ return self._apply_to_instance(project_id, instance_id, configuration_name,
+ node_count, display_name, lambda x: x.update())
+
+ def _apply_to_instance(self, project_id, instance_id, configuration_name, node_count,
+ display_name, func):
+ # type: (str, str, str, int, str, Callable[[Instance], Operation]) -> bool
+ """
+ Invokes a method on a given instance by applying a specified Callable.
+
+ :param project_id: The ID of the project which owns the instances, tables and
+ data.
+ :type project_id: str
+ :param instance_id: The ID of the instance.
+ :type instance_id: str
+ :param configuration_name: Name of the instance configuration defining how the
+ instance will be created. Required for instances which do not yet exist.
+ :type configuration_name: str
+ :param node_count: (Optional) Number of nodes allocated to the instance.
+ :type node_count: int
+ :param display_name: (Optional) The display name for the instance in the Cloud
+ Console UI. (Must be between 4 and 30 characters.) If this value is not set
+ in the constructor, will fall back to the instance ID.
+ :type display_name: str
+ :param func: Method of the instance to be called.
+ :type func: Callable
+ """
+ client = self.get_client(project_id)
+ instance = client.instance(instance_id,
+ configuration_name=configuration_name,
+ node_count=node_count,
+ display_name=display_name)
+ try:
+ operation = func(instance) # type: Operation
+ except GoogleAPICallError as e:
+ self.log.error('An error occurred: %s. Aborting.', e.message)
+ raise e
+
+ if operation:
+ result = operation.result()
+ self.log.info(result)
+ return True
+
+ def delete_instance(self, project_id, instance_id):
+ # type: (str, str) -> bool
+ """
+ Deletes an existing Cloud Spanner instance.
+
+ :param project_id: The ID of the project which owns the instances, tables and data.
+ :type project_id: str
+ :param instance_id: The ID of the instance.
+ :type instance_id: str
+ """
+ client = self.get_client(project_id)
+ instance = client.instance(instance_id)
+ try:
+ instance.delete()
+ return True
+ except GoogleAPICallError as e:
+ self.log.error('An error occurred: %s. Aborting.', e.message)
+ raise e
+
+ def execute_dml(self, project_id, instance_id, database_id, queries):
+ # type: (str, str, str, str) -> None
+ """
+ Executes an arbitrary DML query (INSERT, UPDATE, DELETE).
+
+ :param project_id: The ID of the project which owns the instances, tables and data.
+ :type project_id: str
+ :param instance_id: The ID of the instance.
+ :type instance_id: str
+ :param database_id: The ID of the database.
+ :type database_id: str
+ :param queries: The queries to be executed.
+ :type queries: str
+ """
+ client = self.get_client(project_id)
+ instance = client.instance(instance_id)
+ database = Database(database_id, instance)
+ database.run_in_transaction(lambda transaction:
+ self._execute_sql_in_transaction(transaction, queries))
+
+ @staticmethod
+ def _execute_sql_in_transaction(transaction, queries):
+ for sql in queries:
+ transaction.execute_update(sql)
diff --git a/airflow/contrib/hooks/gcp_sql_hook.py b/airflow/contrib/hooks/gcp_sql_hook.py
new file mode 100644
index 0000000000..9872746b7b
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_sql_hook.py
@@ -0,0 +1,910 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import errno
+import json
+import os
+import re
+import shutil
+import socket
+import platform
+import subprocess
+import time
+import uuid
+from os.path import isfile
+from googleapiclient import errors
+from subprocess import Popen, PIPE
+from six.moves.urllib.parse import quote_plus
+
+import requests
+from googleapiclient.discovery import build
+
+from airflow import AirflowException, LoggingMixin
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+# Number of retries - used by googleapiclient method calls to perform retries
+# For requests that are "retriable"
+from airflow.hooks.base_hook import BaseHook
+from airflow.hooks.mysql_hook import MySqlHook
+from airflow.hooks.postgres_hook import PostgresHook
+from airflow.models.connection import Connection
+from airflow.utils.db import provide_session
+
+NUM_RETRIES = 5
+
+# Time to sleep between active checks of the operation results
+TIME_TO_SLEEP_IN_SECONDS = 1
+
+
+class CloudSqlOperationStatus:
+ PENDING = "PENDING"
+ RUNNING = "RUNNING"
+ DONE = "DONE"
+ UNKNOWN = "UNKNOWN"
+
+
+# noinspection PyAbstractClass
+class CloudSqlHook(GoogleCloudBaseHook):
+ """
+ Hook for Google Cloud SQL APIs.
+ """
+ _conn = None
+
+ def __init__(self,
+ api_version,
+ gcp_conn_id='google_cloud_default',
+ delegate_to=None):
+ super(CloudSqlHook, self).__init__(gcp_conn_id, delegate_to)
+ self.api_version = api_version
+
+ def get_conn(self):
+ """
+ Retrieves connection to Cloud SQL.
+
+ :return: Google Cloud SQL services object.
+ :rtype: dict
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build('sqladmin', self.api_version,
+ http=http_authorized, cache_discovery=False)
+ return self._conn
+
+ def get_instance(self, project_id, instance):
+ """
+ Retrieves a resource containing information about a Cloud SQL instance.
+
+ :param project_id: Project ID of the project that contains the instance.
+ :type project_id: str
+ :param instance: Database instance ID. This does not include the project ID.
+ :type instance: str
+ :return: A Cloud SQL instance resource.
+ :rtype: dict
+ """
+ return self.get_conn().instances().get(
+ project=project_id,
+ instance=instance
+ ).execute(num_retries=NUM_RETRIES)
+
+ def create_instance(self, project_id, body):
+ """
+ Creates a new Cloud SQL instance.
+
+ :param project_id: Project ID of the project to which the newly created
+ Cloud SQL instances should belong.
+ :type project_id: str
+ :param body: Body required by the Cloud SQL insert API, as described in
+ https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/insert#request-body.
+ :type body: dict
+ :return: True if the operation succeeded otherwise raises an error.
+ :rtype: bool
+ """
+ response = self.get_conn().instances().insert(
+ project=project_id,
+ body=body
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project_id, operation_name)
+
+ def patch_instance(self, project_id, body, instance):
+ """
+ Updates settings of a Cloud SQL instance.
+
+ Caution: This is not a partial update, so you must include values for
+ all the settings that you want to retain.
+
+ :param project_id: Project ID of the project that contains the instance.
+ :type project_id: str
+ :param body: Body required by the Cloud SQL patch API, as described in
+ https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/patch#request-body.
+ :type body: dict
+ :param instance: Cloud SQL instance ID. This does not include the project ID.
+ :type instance: str
+ :return: True if the operation succeeded otherwise raises an error.
+ :rtype: bool
+ """
+ response = self.get_conn().instances().patch(
+ project=project_id,
+ instance=instance,
+ body=body
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project_id, operation_name)
+
+ def delete_instance(self, project_id, instance):
+ """
+ Deletes a Cloud SQL instance.
+
+ :param project_id: Project ID of the project that contains the instance.
+ :type project_id: str
+ :param instance: Cloud SQL instance ID. This does not include the project ID.
+ :type instance: str
+ :return: True if the operation succeeded otherwise raises an error.
+ :rtype: bool
+ """
+ response = self.get_conn().instances().delete(
+ project=project_id,
+ instance=instance,
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project_id, operation_name)
+
+ def get_database(self, project_id, instance, database):
+ """
+ Retrieves a database resource from a Cloud SQL instance.
+
+ :param project_id: Project ID of the project that contains the instance.
+ :type project_id: str
+ :param instance: Database instance ID. This does not include the project ID.
+ :type instance: str
+ :param database: Name of the database in the instance.
+ :type database: str
+ :return: A Cloud SQL database resource, as described in
+ https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases#resource.
+ :rtype: dict
+ """
+ return self.get_conn().databases().get(
+ project=project_id,
+ instance=instance,
+ database=database
+ ).execute(num_retries=NUM_RETRIES)
+
+ def create_database(self, project, instance, body):
+ """
+ Creates a new database inside a Cloud SQL instance.
+
+ :param project: Project ID of the project that contains the instance.
+ :type project: str
+ :param instance: Database instance ID. This does not include the project ID.
+ :type instance: str
+ :param body: The request body, as described in
+ https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body.
+ :type body: dict
+ :return: True if the operation succeeded otherwise raises an error.
+ :rtype: bool
+ """
+ response = self.get_conn().databases().insert(
+ project=project,
+ instance=instance,
+ body=body
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project, operation_name)
+
+ def patch_database(self, project, instance, database, body):
+ """
+ Updates a database resource inside a Cloud SQL instance.
+
+ This method supports patch semantics.
+ See https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch.
+
+ :param project: Project ID of the project that contains the instance.
+ :type project: str
+ :param instance: Database instance ID. This does not include the project ID.
+ :type instance: str
+ :param database: Name of the database to be updated in the instance.
+ :type database: str
+ :param body: The request body, as described in
+ https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body.
+ :type body: dict
+ :return: True if the operation succeeded otherwise raises an error.
+ :rtype: bool
+ """
+ response = self.get_conn().databases().patch(
+ project=project,
+ instance=instance,
+ database=database,
+ body=body
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project, operation_name)
+
+ def delete_database(self, project, instance, database):
+ """
+ Deletes a database from a Cloud SQL instance.
+
+ :param project: Project ID of the project that contains the instance.
+ :type project: str
+ :param instance: Database instance ID. This does not include the project ID.
+ :type instance: str
+ :param database: Name of the database to be deleted in the instance.
+ :type database: str
+ :return: True if the operation succeeded otherwise raises an error.
+ :rtype: bool
+ """
+ response = self.get_conn().databases().delete(
+ project=project,
+ instance=instance,
+ database=database
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project, operation_name)
+
+ def export_instance(self, project_id, instance_id, body):
+ """
+ Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump
+ or CSV file.
+
+ :param project_id: Project ID of the project where the instance exists.
+ :type project_id: str
+ :param instance_id: Name of the Cloud SQL instance. This does not include the
+ project ID.
+ :type instance_id: str
+ :param body: The request body, as described in
+ https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body
+ :type body: dict
+ :return: True if the operation succeeded, raises an error otherwise
+ :rtype: bool
+ """
+ try:
+ response = self.get_conn().instances().export(
+ project=project_id,
+ instance=instance_id,
+ body=body
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project_id, operation_name)
+ except errors.HttpError as ex:
+ raise AirflowException(
+ 'Exporting instance {} failed: {}'.format(instance_id, ex.content)
+ )
+
+ def import_instance(self, project_id, instance_id, body):
+ """
+ Imports data into a Cloud SQL instance from a SQL dump or CSV file in
+ Cloud Storage.
+
+ :param project_id: Project ID of the project where the instance exists.
+ :type project_id: str
+ :param instance_id: Name of the Cloud SQL instance. This does not include the
+ project ID.
+ :type instance_id: str
+ :param body: The request body, as described in
+ https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body
+ :type body: dict
+ :return: True if the operation succeeded, raises an error otherwise
+ :rtype: bool
+ """
+ try:
+ response = self.get_conn().instances().import_(
+ project=project_id,
+ instance=instance_id,
+ body=body
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project_id, operation_name)
+ except errors.HttpError as ex:
+ raise AirflowException(
+ 'Importing instance {} failed: {}'.format(instance_id, ex.content)
+ )
+
+ def _wait_for_operation_to_complete(self, project_id, operation_name):
+ """
+ Waits for the named operation to complete - checks status of the
+ asynchronous call.
+
+ :param project_id: Project ID of the project that contains the instance.
+ :type project_id: str
+ :param operation_name: Name of the operation.
+ :type operation_name: str
+ :return: Response returned by the operation.
+ :rtype: dict
+ """
+ service = self.get_conn()
+ while True:
+ operation_response = service.operations().get(
+ project=project_id,
+ operation=operation_name,
+ ).execute(num_retries=NUM_RETRIES)
+ if operation_response.get("status") == CloudSqlOperationStatus.DONE:
+ error = operation_response.get("error")
+ if error:
+ # Extracting the errors list as string and trimming square braces
+ error_msg = str(error.get("errors"))[1:-1]
+ raise AirflowException(error_msg)
+ # No meaningful info to return from the response in case of success
+ return True
+ time.sleep(TIME_TO_SLEEP_IN_SECONDS)
+
+
+CLOUD_SQL_PROXY_DOWNLOAD_URL = "https://dl.google.com/cloudsql/cloud_sql_proxy.{}.{}"
+CLOUD_SQL_PROXY_VERSION_DOWNLOAD_URL = \
+ "https://storage.googleapis.com/cloudsql-proxy/{}/cloud_sql_proxy.{}.{}"
+
+GCP_CREDENTIALS_KEY_PATH = "extra__google_cloud_platform__key_path"
+GCP_CREDENTIALS_KEYFILE_DICT = "extra__google_cloud_platform__keyfile_dict"
+
+
+class CloudSqlProxyRunner(LoggingMixin):
+ """
+ Downloads and runs cloud-sql-proxy as subprocess of the Python process.
+
+ The cloud-sql-proxy needs to be downloaded and started before we can connect
+ to the Google Cloud SQL instance via database connection. It establishes
+ secure tunnel connection to the database. It authorizes using the
+ GCP credentials that are passed by the configuration.
+
+ More details about the proxy can be found here:
+ https://cloud.google.com/sql/docs/mysql/sql-proxy
+
+ """
+ def __init__(self,
+ path_prefix,
+ instance_specification,
+ gcp_conn_id='google_cloud_default',
+ project_id=None,
+ sql_proxy_version=None,
+ sql_proxy_binary_path=None):
+ """
+ Creates the proxy runner class.
+
+ :param path_prefix: Unique path prefix where proxy will be downloaded and
+ directories created for unix sockets.
+ :type path_prefix: str
+ :param instance_specification: Specification of the instance to connect the
+ proxy to. It should be specified in the form that is described in
+ https://cloud.google.com/sql/docs/mysql/sql-proxy#multiple-instances in
+ -instances parameter (typically in the form of <project>:<region>:<instance>
+ for UNIX socket connections and in the form of
+ <project>:<region>:<instance>=tcp:<port> for TCP connections.
+ :type instance_specification: str
+ :param gcp_conn_id: Id of Google Cloud Platform connection to use for
+ authentication
+ :type: str
+ :param project_id: Optional id of the GCP project to connect to - it overwrites
+ default project id taken from the GCP connection
+ :type project_id: str
+ :param sql_proxy_version: Specific version of SQL proxy to download
+ (for example 'v1.13'). By default latest version is downloaded.
+ :type sql_proxy_version: str
+ :param sql_proxy_binary_path: If specified, then proxy will be
+ used from the path specified rather than dynamically generated. This means
+ that if the binary is not present in that path it will also be downloaded.
+ :type sql_proxy_binary_path: str
+ """
+ super(CloudSqlProxyRunner, self).__init__()
+ self.path_prefix = path_prefix
+ if not self.path_prefix:
+ raise AirflowException("The path_prefix must not be empty!")
+ self.sql_proxy_was_downloaded = False
+ self.sql_proxy_version = sql_proxy_version
+ self.download_sql_proxy_dir = None
+ self.sql_proxy_process = None
+ self.instance_specification = instance_specification
+ self.project_id = project_id
+ self.gcp_conn_id = gcp_conn_id
+ self.command_line_parameters = []
+ self.cloud_sql_proxy_socket_directory = self.path_prefix
+ self.sql_proxy_path = sql_proxy_binary_path if sql_proxy_binary_path \
+ else self.path_prefix + "_cloud_sql_proxy"
+ self.credentials_path = self.path_prefix + "_credentials.json"
+ self._build_command_line_parameters()
+
+ def _build_command_line_parameters(self):
+ self.command_line_parameters.extend(
+ ['-dir', self.cloud_sql_proxy_socket_directory])
+ self.command_line_parameters.extend(
+ ['-instances', self.instance_specification])
+
+ @staticmethod
+ def _is_os_64bit():
+ return platform.machine().endswith('64')
+
+ def _download_sql_proxy_if_needed(self):
+ if os.path.isfile(self.sql_proxy_path):
+ self.log.info("cloud-sql-proxy is already present")
+ return
+ system = platform.system().lower()
+ processor = "amd64" if CloudSqlProxyRunner._is_os_64bit() else "386"
+ if not self.sql_proxy_version:
+ download_url = CLOUD_SQL_PROXY_DOWNLOAD_URL.format(system, processor)
+ else:
+ download_url = CLOUD_SQL_PROXY_VERSION_DOWNLOAD_URL.format(
+ self.sql_proxy_version, system, processor)
+ proxy_path_tmp = self.sql_proxy_path + ".tmp"
+ self.log.info("Downloading cloud_sql_proxy from {} to {}".
+ format(download_url, proxy_path_tmp))
+ r = requests.get(download_url, allow_redirects=True)
+ # Downloading to .tmp file first to avoid case where partially downloaded
+ # binary is used by parallel operator which uses the same fixed binary path
+ with open(proxy_path_tmp, 'wb') as f:
+ f.write(r.content)
+ if r.status_code != 200:
+ raise AirflowException(
+ "The cloud-sql-proxy could not be downloaded. Status code = {}. "
+ "Reason = {}".format(r.status_code, r.reason))
+ self.log.info("Moving sql_proxy binary from {} to {}".format(
+ proxy_path_tmp, self.sql_proxy_path
+ ))
+ shutil.move(proxy_path_tmp, self.sql_proxy_path)
+ os.chmod(self.sql_proxy_path, 0o744) # Set executable bit
+ self.sql_proxy_was_downloaded = True
+
+ @provide_session
+ def _get_credential_parameters(self, session):
+ connection = session.query(Connection). \
+ filter(Connection.conn_id == self.gcp_conn_id).first()
+ session.expunge_all()
+ if GCP_CREDENTIALS_KEY_PATH in connection.extra_dejson:
+ credential_params = [
+ '-credential_file',
+ connection.extra_dejson[GCP_CREDENTIALS_KEY_PATH]
+ ]
+ elif GCP_CREDENTIALS_KEYFILE_DICT in connection.extra_dejson:
+ credential_file_content = json.loads(
+ connection.extra_dejson[GCP_CREDENTIALS_KEYFILE_DICT])
+ self.log.info("Saving credentials to {}".format(self.credentials_path))
+ with open(self.credentials_path, "w") as f:
+ json.dump(credential_file_content, f)
+ credential_params = [
+ '-credential_file',
+ self.credentials_path
+ ]
+ else:
+ self.log.info(
+ "The credentials are not supplied by neither key_path nor "
+ "keyfile_dict of the gcp connection {}. Falling back to "
+ "default activated account".format(self.gcp_conn_id))
+ credential_params = []
+
+ if not self.instance_specification:
+ project_id = connection.extra_dejson.get(
+ 'extra__google_cloud_platform__project')
+ if self.project_id:
+ project_id = self.project_id
+ if not project_id:
+ raise AirflowException("For forwarding all instances, the project id "
+ "for GCP should be provided either "
+ "by project_id extra in the GCP connection or by "
+ "project_id provided in the operator.")
+ credential_params.extend(['-projects', project_id])
+ return credential_params
+
+ def start_proxy(self):
+ """
+ Starts Cloud SQL Proxy.
+
+ You have to remember to stop the proxy if you started it!
+ """
+ self._download_sql_proxy_if_needed()
+ if self.sql_proxy_process:
+ raise AirflowException("The sql proxy is already running: {}".format(
+ self.sql_proxy_process))
+ else:
+ command_to_run = [self.sql_proxy_path]
+ command_to_run.extend(self.command_line_parameters)
+ try:
+ self.log.info("Creating directory {}".format(
+ self.cloud_sql_proxy_socket_directory))
+ os.makedirs(self.cloud_sql_proxy_socket_directory)
+ except OSError:
+ # Needed for python 2 compatibility (exists_ok missing)
+ pass
+ command_to_run.extend(self._get_credential_parameters())
+ self.log.info("Running the command: `{}`".format(" ".join(command_to_run)))
+ self.sql_proxy_process = Popen(command_to_run,
+ stdin=PIPE, stdout=PIPE, stderr=PIPE)
+ self.log.info("The pid of cloud_sql_proxy: {}".format(
+ self.sql_proxy_process.pid))
+ while True:
+ line = self.sql_proxy_process.stderr.readline().decode('utf-8')
+ return_code = self.sql_proxy_process.poll()
+ if line == '' and return_code is not None:
+ self.sql_proxy_process = None
+ raise AirflowException(
+ "The cloud_sql_proxy finished early with return code {}!".format(
+ return_code))
+ if line != '':
+ self.log.info(line)
+ if "googleapi: Error" in line or "invalid instance name:" in line:
+ self.stop_proxy()
+ raise AirflowException(
+ "Error when starting the cloud_sql_proxy {}!".format(
+ line))
+ if "Ready for new connections" in line:
+ return
+
+ def stop_proxy(self):
+ """
+ Stops running proxy.
+
+ You should stop the proxy after you stop using it.
+ """
+ if not self.sql_proxy_process:
+ raise AirflowException("The sql proxy is not started yet")
+ else:
+ self.log.info("Stopping the cloud_sql_proxy pid: {}".format(
+ self.sql_proxy_process.pid))
+ self.sql_proxy_process.kill()
+ self.sql_proxy_process = None
+ # Cleanup!
+ self.log.info("Removing the socket directory: {}".
+ format(self.cloud_sql_proxy_socket_directory))
+ shutil.rmtree(self.cloud_sql_proxy_socket_directory, ignore_errors=True)
+ if self.sql_proxy_was_downloaded:
+ self.log.info("Removing downloaded proxy: {}".format(self.sql_proxy_path))
+ # Silently ignore if the file has already been removed (concurrency)
+ try:
+ os.remove(self.sql_proxy_path)
+ except OSError as e:
+ if not e.errno == errno.ENOENT:
+ raise
+ else:
+ self.log.info("Skipped removing proxy - it was not downloaded: {}".
+ format(self.sql_proxy_path))
+ if isfile(self.credentials_path):
+ self.log.info("Removing generated credentials file {}".
+ format(self.credentials_path))
+ # Here file cannot be delete by concurrent task (each task has its own copy)
+ os.remove(self.credentials_path)
+
+ def get_proxy_version(self):
+ """
+ Returns version of the Cloud SQL Proxy.
+ """
+ self._download_sql_proxy_if_needed()
+ command_to_run = [self.sql_proxy_path]
+ command_to_run.extend(['--version'])
+ command_to_run.extend(self._get_credential_parameters())
+ result = subprocess.check_output(command_to_run).decode('utf-8')
+ pattern = re.compile("^.*[V|v]ersion ([^;]*);.*$")
+ m = pattern.match(result)
+ if m:
+ return m.group(1)
+ else:
+ return None
+
+ def get_socket_path(self):
+ """
+ Retrieves UNIX socket path used by Cloud SQL Proxy.
+
+ :return: The dynamically generated path for the socket created by the proxy.
+ :rtype: str
+ """
+ return self.cloud_sql_proxy_socket_directory + "/" + self.instance_specification
+
+
+CONNECTION_URIS = {
+ "postgres": {
+ "proxy": {
+ "tcp":
+ "postgresql://{user}:{password}@127.0.0.1:{proxy_port}/{database}",
+ "socket":
+ "postgresql://{user}:{password}@{socket_path}/{database}"
+ },
+ "public": {
+ "ssl":
+ "postgresql://{user}:{password}@{public_ip}:{public_port}/{database}?"
+ "sslmode=verify-ca&"
+ "sslcert={client_cert_file}&"
+ "sslkey={client_key_file}&"
+ "sslrootcert={server_ca_file}",
+ "non-ssl":
+ "postgresql://{user}:{password}@{public_ip}:{public_port}/{database}"
+ }
+ },
+ "mysql": {
+ "proxy": {
+ "tcp":
+ "mysql://{user}:{password}@127.0.0.1:{proxy_port}/{database}",
+ "socket":
+ "mysql://{user}:{password}@localhost/{database}?"
+ "unix_socket={socket_path}"
+ },
+ "public": {
+ "ssl":
+ "mysql://{user}:{password}@{public_ip}:{public_port}/{database}?"
+ "ssl={ssl_spec}",
+ "non-ssl":
+ "mysql://{user}:{password}@{public_ip}:{public_port}/{database}"
+ }
+ }
+}
+
+CLOUD_SQL_VALID_DATABASE_TYPES = ['postgres', 'mysql']
+
+
+# noinspection PyAbstractClass
+class CloudSqlDatabaseHook(BaseHook):
+ """
+ Serves DB connection configuration for Google Cloud SQL (Connections
+ of *gcpcloudsql://* type).
+
+ The hook is a "meta" one. It does not perform an actual connection.
+ It is there to retrieve all the parameters configured in gcpcloudsql:// connection,
+ start/stop Cloud SQL Proxy if needed, dynamically generate Postgres or MySQL
+ connection in the database and return an actual Postgres or MySQL hook.
+ The returned Postgres/MySQL hooks are using direct connection or Cloud SQL
+ Proxy socket/TCP as configured.
+
+ Main parameters of the hook are retrieved from the standard URI components:
+
+ * **user** - User name to authenticate to the database (from login of the URI).
+ * **password** - Password to authenticate to the database (from password of the URI).
+ * **public_ip** - IP to connect to for public connection (from host of the URI).
+ * **public_port** - Port to connect to for public connection (from port of the URI).
+ * **database** - Database to connect to (from schema of the URI).
+
+ Remaining parameters are retrieved from the extras (URI query parameters):
+
+ * **project_id** - Google Cloud Platform project where the Cloud SQL instance exists.
+ * **instance** - Name of the instance of the Cloud SQL database instance.
+ * **location** - The location of the Cloud SQL instance (for example europe-west1).
+ * **database_type** - The type of the database instance (MySQL or Postgres).
+ * **use_proxy** - (default False) Whether SQL proxy should be used to connect to Cloud
+ SQL DB.
+ * **use_ssl** - (default False) Whether SSL should be used to connect to Cloud SQL DB.
+ You cannot use proxy and SSL together.
+ * **sql_proxy_use_tcp** - (default False) If set to true, TCP is used to connect via
+ proxy, otherwise UNIX sockets are used.
+ * **sql_proxy_binary_path** - Optional path to Cloud SQL Proxy binary. If the binary
+ is not specified or the binary is not present, it is automatically downloaded.
+ * **sql_proxy_version** - Specific version of the proxy to download (for example
+ v1.13). If not specified, the latest version is downloaded.
+ * **sslcert** - Path to client certificate to authenticate when SSL is used.
+ * **sslkey** - Path to client private key to authenticate when SSL is used.
+ * **sslrootcert** - Path to server's certificate to authenticate when SSL is used.
+ """
+ _conn = None
+
+ def __init__(self, gcp_cloudsql_conn_id='google_cloud_sql_default'):
+ super(CloudSqlDatabaseHook, self).__init__(source=None)
+ self.gcp_cloudsql_conn_id = gcp_cloudsql_conn_id
+ self.cloudsql_connection = self.get_connection(self.gcp_cloudsql_conn_id)
+ self.extras = self.cloudsql_connection.extra_dejson
+ self.project_id = self.extras.get('project_id')
+ self.instance = self.extras.get('instance')
+ self.database = self.cloudsql_connection.schema
+ self.location = self.extras.get('location')
+ self.database_type = self.extras.get('database_type')
+ self.use_proxy = self._get_bool(self.extras.get('use_proxy', 'False'))
+ self.use_ssl = self._get_bool(self.extras.get('use_ssl', 'False'))
+ self.sql_proxy_use_tcp = self._get_bool(
+ self.extras.get('sql_proxy_use_tcp', 'False'))
+ self.sql_proxy_version = self.extras.get('sql_proxy_version')
+ self.sql_proxy_binary_path = self.extras.get('sql_proxy_binary_path')
+ self.user = self.cloudsql_connection.login
+ self.password = self.cloudsql_connection.password
+ self.public_ip = self.cloudsql_connection.host
+ self.public_port = self.cloudsql_connection.port
+ self.sslcert = self.extras.get('sslcert')
+ self.sslkey = self.extras.get('sslkey')
+ self.sslrootcert = self.extras.get('sslrootcert')
+ # Port and socket path and db_hook are automatically generated
+ self.sql_proxy_tcp_port = None
+ self.sql_proxy_unique_path = None
+ self.db_hook = None
+ self.reserved_tcp_socket = None
+ # Generated based on clock + clock sequence. Unique per host (!).
+ # This is important as different hosts share the database
+ self.db_conn_id = str(uuid.uuid1())
+ self._validate_inputs()
+
+ @staticmethod
+ def _get_bool(val):
+ if val == 'False':
+ return False
+ return val
+
+ @staticmethod
+ def _check_ssl_file(file_to_check, name):
+ if not file_to_check:
+ raise AirflowException("SSL connections requires {name} to be set".
+ format(name=name))
+ if not isfile(file_to_check):
+ raise AirflowException("The {file_to_check} must be a readable file".
+ format(file_to_check=file_to_check))
+
+ def _validate_inputs(self):
+ if not self.project_id:
+ raise AirflowException("The required extra 'project_id' is empty")
+ if not self.location:
+ raise AirflowException("The required extra 'location' is empty")
+ if not self.instance:
+ raise AirflowException("The required extra 'instance' is empty")
+ if self.database_type not in CLOUD_SQL_VALID_DATABASE_TYPES:
+ raise AirflowException("Invalid database type '{}'. Must be one of {}".format(
+ self.database_type, CLOUD_SQL_VALID_DATABASE_TYPES
+ ))
+ if self.use_proxy and self.use_ssl:
+ raise AirflowException("Cloud SQL Proxy does not support SSL connections."
+ " SSL is not needed as Cloud SQL Proxy "
+ "provides encryption on its own")
+ if self.use_ssl:
+ self._check_ssl_file(self.sslcert, "sslcert")
+ self._check_ssl_file(self.sslkey, "sslkey")
+ self._check_ssl_file(self.sslrootcert, "sslrootcert")
+
+ def _generate_unique_path(self):
+ # We are not using mkdtemp here as the path generated with mkdtemp
+ # can be close to 60 characters and there is a limitation in
+ # length of socket path to around 100 characters in total.
+ # We append project/location/instance to it later and postgres
+ # appends its own prefix, so we chose a shorter "/tmp/{uuid1}" - based
+ # on host name and clock + clock sequence. This should be fairly
+ # sufficient for our needs and should even work if the time is set back.
+ # We are using db_conn_id generated with uuid1 so that connection
+ # id matches the folder - for easier debugging.
+ return "/tmp/" + self.db_conn_id
+
+ @staticmethod
+ def _quote(value):
+ return quote_plus(value) if value else None
+
+ def _generate_connection_uri(self):
+ if self.use_proxy:
+ if self.sql_proxy_use_tcp:
+ if not self.sql_proxy_tcp_port:
+ self.reserve_free_tcp_port()
+ if not self.sql_proxy_unique_path:
+ self.sql_proxy_unique_path = self._generate_unique_path()
+
+ database_uris = CONNECTION_URIS[self.database_type]
+ ssl_spec = None
+ socket_path = None
+ if self.use_proxy:
+ proxy_uris = database_uris['proxy']
+ if self.sql_proxy_use_tcp:
+ format_string = proxy_uris['tcp']
+ else:
+ format_string = proxy_uris['socket']
+ socket_path = \
+ "{sql_proxy_socket_path}/{instance_socket_name}".format(
+ sql_proxy_socket_path=self.sql_proxy_unique_path,
+ instance_socket_name=self._get_instance_socket_name()
+ )
+ else:
+ public_uris = database_uris['public']
+ if self.use_ssl:
+ format_string = public_uris['ssl']
+ ssl_spec = {
+ 'cert': self.sslcert,
+ 'key': self.sslkey,
+ 'ca': self.sslrootcert
+ }
+ else:
+ format_string = public_uris['non-ssl']
+
+ connection_uri = format_string.format(
+ user=quote_plus(self.user),
+ password=quote_plus(self.password),
+ database=quote_plus(self.database),
+ public_ip=self.public_ip,
+ public_port=self.public_port,
+ proxy_port=self.sql_proxy_tcp_port,
+ socket_path=self._quote(socket_path),
+ ssl_spec=self._quote(json.dumps(ssl_spec)) if ssl_spec else None,
+ client_cert_file=self._quote(self.sslcert),
+ client_key_file=self._quote(self.sslkey),
+ server_ca_file=self._quote(self.sslrootcert)
+ )
+ self.log.info("DB connection URI {}".format(connection_uri.replace(
+ quote_plus(self.password), 'XXXXXXXXXXXX')))
+ return connection_uri
+
+ def _get_instance_socket_name(self):
+ return self.project_id + ":" + self.location + ":" + self.instance
+
+ def _get_sqlproxy_instance_specification(self):
+ instance_specification = self._get_instance_socket_name()
+ if self.sql_proxy_use_tcp:
+ instance_specification += "=tcp:" + str(self.sql_proxy_tcp_port)
+ return instance_specification
+
+ @provide_session
+ def create_connection(self, session=None):
+ """
+ Create connection in the Connection table, according to whether it uses
+ proxy, TCP, UNIX sockets, SSL. Connection ID will be randomly generated.
+
+ :param session: Session of the SQL Alchemy ORM (automatically generated with
+ decorator).
+ """
+ connection = Connection(conn_id=self.db_conn_id)
+ uri = self._generate_connection_uri()
+ self.log.info("Creating connection {}".format(self.db_conn_id))
+ connection.parse_from_uri(uri)
+ session.add(connection)
+ session.commit()
+
+ @provide_session
+ def delete_connection(self, session=None):
+ """
+ Delete the dynamically created connection from the Connection table.
+
+ :param session: Session of the SQL Alchemy ORM (automatically generated with
+ decorator).
+ """
+ self.log.info("Deleting connection {}".format(self.db_conn_id))
+ connection = session.query(Connection).filter(
+ Connection.conn_id == self.db_conn_id)[0]
+ session.delete(connection)
+ session.commit()
+
+ def get_sqlproxy_runner(self):
+ """
+ Retrieve Cloud SQL Proxy runner. It is used to manage the proxy
+ lifecycle per task.
+
+ :return: The Cloud SQL Proxy runner.
+ :rtype: CloudSqlProxyRunner
+ """
+ return CloudSqlProxyRunner(
+ path_prefix=self.sql_proxy_unique_path,
+ instance_specification=self._get_sqlproxy_instance_specification(),
+ project_id=self.project_id,
+ sql_proxy_version=self.sql_proxy_version,
+ sql_proxy_binary_path=self.sql_proxy_binary_path
+ )
+
+ def get_database_hook(self):
+ """
+ Retrieve database hook. This is the actual Postgres or MySQL database hook
+ that uses proxy or connects directly to the Google Cloud SQL database.
+ """
+ if self.database_type == 'postgres':
+ self.db_hook = PostgresHook(postgres_conn_id=self.db_conn_id,
+ schema=self.database)
+ else:
+ self.db_hook = MySqlHook(mysql_conn_id=self.db_conn_id,
+ schema=self.database)
+ return self.db_hook
+
+ def cleanup_database_hook(self):
+ """
+ Clean up database hook after it was used.
+ """
+ if self.database_type == 'postgres':
+ for output in self.db_hook.conn.notices:
+ self.log.info(output)
+
+ def reserve_free_tcp_port(self):
+ """
+ Reserve free TCP port to be used by Cloud SQL Proxy
+ """
+ self.reserved_tcp_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ self.reserved_tcp_socket.bind(('127.0.0.1', 0))
+ self.sql_proxy_tcp_port = self.reserved_tcp_socket.getsockname()[1]
+
+ def free_reserved_port(self):
+ """
+ Free TCP port. Makes it immediately ready to be used by Cloud SQL Proxy.
+ """
+ if self.reserved_tcp_socket:
+ self.reserved_tcp_socket.close()
+ self.reserved_tcp_socket = None
diff --git a/airflow/contrib/hooks/gcp_transfer_hook.py b/airflow/contrib/hooks/gcp_transfer_hook.py
new file mode 100644
index 0000000000..906dba786f
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_transfer_hook.py
@@ -0,0 +1,107 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+import time
+import datetime
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+# Time to sleep between active checks of the operation results
+TIME_TO_SLEEP_IN_SECONDS = 1
+
+
+# noinspection PyAbstractClass
+class GCPTransferServiceHook(GoogleCloudBaseHook):
+ """
+ Hook for GCP Storage Transfer Service.
+ """
+ _conn = None
+
+ def __init__(self,
+ api_version='v1',
+ gcp_conn_id='google_cloud_default',
+ delegate_to=None):
+ super(GCPTransferServiceHook, self).__init__(gcp_conn_id, delegate_to)
+ self.api_version = api_version
+
+ def get_conn(self):
+ """
+ Retrieves connection to Google Storage Transfer service.
+
+ :return: Google Storage Transfer service object
+ :rtype: dict
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build('storagetransfer', self.api_version,
+ http=http_authorized, cache_discovery=False)
+ return self._conn
+
+ def create_transfer_job(self, project_id, description, schedule, transfer_spec):
+ transfer_job = {
+ 'status': 'ENABLED',
+ 'projectId': project_id,
+ 'description': description,
+ 'transferSpec': transfer_spec,
+ 'schedule': schedule or self._schedule_once_now(),
+ }
+ return self.get_conn().transferJobs().create(body=transfer_job).execute()
+
+ def wait_for_transfer_job(self, job):
+ while True:
+ result = self.get_conn().transferOperations().list(
+ name='transferOperations',
+ filter=json.dumps({
+ 'project_id': job['projectId'],
+ 'job_names': [job['name']],
+ }),
+ ).execute()
+ if self._check_operations_result(result):
+ return True
+ time.sleep(TIME_TO_SLEEP_IN_SECONDS)
+
+ def _check_operations_result(self, result):
+ operations = result.get('operations', [])
+ if len(operations) == 0:
+ return False
+ for operation in operations:
+ if operation['metadata']['status'] in {'FAILED', 'ABORTED'}:
+ raise AirflowException('Operation {} {}'.format(
+ operation['name'], operation['metadata']['status']))
+ if operation['metadata']['status'] != 'SUCCESS':
+ return False
+ return True
+
+ def _schedule_once_now(self):
+ now = datetime.datetime.utcnow()
+ return {
+ 'scheduleStartDate': {
+ 'day': now.day,
+ 'month': now.month,
+ 'year': now.year,
+ },
+ 'scheduleEndDate': {
+ 'day': now.day,
+ 'month': now.month,
+ 'year': now.year,
+ }
+ }
diff --git a/airflow/contrib/hooks/gcs_hook.py b/airflow/contrib/hooks/gcs_hook.py
index 3103a5af98..6499ea38b5 100644
--- a/airflow/contrib/hooks/gcs_hook.py
+++ b/airflow/contrib/hooks/gcs_hook.py
@@ -1,22 +1,33 @@
# -*- coding: utf-8 -*-
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
#
from apiclient.discovery import build
from apiclient.http import MediaFileUpload
from googleapiclient import errors
from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+from airflow.exceptions import AirflowException
+
+import gzip as gz
+import shutil
+import re
+import os
class GoogleCloudStorageHook(GoogleCloudBaseHook):
@@ -26,7 +37,7 @@ class GoogleCloudStorageHook(GoogleCloudBaseHook):
"""
def __init__(self,
- google_cloud_storage_conn_id='google_cloud_storage_default',
+ google_cloud_storage_conn_id='google_cloud_default',
delegate_to=None):
super(GoogleCloudStorageHook, self).__init__(google_cloud_storage_conn_id,
delegate_to)
@@ -36,7 +47,8 @@ def get_conn(self):
Returns a Google Cloud Storage service object.
"""
http_authorized = self._authorize()
- return build('storage', 'v1', http=http_authorized)
+ return build(
+ 'storage', 'v1', http=http_authorized, cache_discovery=False)
# pylint:disable=redefined-builtin
def copy(self, source_bucket, source_object, destination_bucket=None,
@@ -48,19 +60,21 @@ def copy(self, source_bucket, source_object, destination_bucket=None,
source bucket/object is used, but not both.
:param source_bucket: The bucket of the object to copy from.
- :type source_bucket: string
+ :type source_bucket: str
:param source_object: The object to copy.
- :type source_object: string
+ :type source_object: str
:param destination_bucket: The destination of the object to copied to.
Can be omitted; then the same bucket is used.
- :type destination_bucket: string
+ :type destination_bucket: str
:param destination_object: The (renamed) path of the object if given.
Can be omitted; then the same name is used.
+ :type destination_object: str
"""
destination_bucket = destination_bucket or source_bucket
destination_object = destination_object or source_object
- if (source_bucket == destination_bucket and
- source_object == destination_object):
+ if source_bucket == destination_bucket and \
+ source_object == destination_object:
+
raise ValueError(
'Either source/destination bucket or source/destination object '
'must be different, not both the same: bucket=%s, object=%s' %
@@ -82,6 +96,58 @@ def copy(self, source_bucket, source_object, destination_bucket=None,
return False
raise
+ def rewrite(self, source_bucket, source_object, destination_bucket,
+ destination_object=None):
+ """
+ Has the same functionality as copy, except that will work on files
+ over 5 TB, as well as when copying between locations and/or storage
+ classes.
+
+ destination_object can be omitted, in which case source_object is used.
+
+ :param source_bucket: The bucket of the object to copy from.
+ :type source_bucket: str
+ :param source_object: The object to copy.
+ :type source_object: str
+ :param destination_bucket: The destination of the object to copied to.
+ :type destination_bucket: str
+ :param destination_object: The (renamed) path of the object if given.
+ Can be omitted; then the same name is used.
+ :type destination_object: str
+ """
+ destination_object = destination_object or source_object
+ if (source_bucket == destination_bucket and
+ source_object == destination_object):
+ raise ValueError(
+ 'Either source/destination bucket or source/destination object '
+ 'must be different, not both the same: bucket=%s, object=%s' %
+ (source_bucket, source_object))
+ if not source_bucket or not source_object:
+ raise ValueError('source_bucket and source_object cannot be empty.')
+
+ service = self.get_conn()
+ request_count = 1
+ try:
+ result = service.objects() \
+ .rewrite(sourceBucket=source_bucket, sourceObject=source_object,
+ destinationBucket=destination_bucket,
+ destinationObject=destination_object, body='') \
+ .execute()
+ self.log.info('Rewrite request #%s: %s', request_count, result)
+ while not result['done']:
+ request_count += 1
+ result = service.objects() \
+ .rewrite(sourceBucket=source_bucket, sourceObject=source_object,
+ destinationBucket=destination_bucket,
+ destinationObject=destination_object,
+ rewriteToken=result['rewriteToken'], body='') \
+ .execute()
+ self.log.info('Rewrite request #%s: %s', request_count, result)
+ return True
+ except errors.HttpError as ex:
+ if ex.resp['status'] == '404':
+ return False
+ raise
# pylint:disable=redefined-builtin
def download(self, bucket, object, filename=None):
@@ -89,11 +155,11 @@ def download(self, bucket, object, filename=None):
Get a file from Google Cloud Storage.
:param bucket: The bucket to fetch from.
- :type bucket: string
+ :type bucket: str
:param object: The object to fetch.
- :type object: string
+ :type object: str
:param filename: If set, a local file path where the file should be written to.
- :type filename: string
+ :type filename: str
"""
service = self.get_conn()
downloaded_file_bytes = service \
@@ -110,25 +176,79 @@ def download(self, bucket, object, filename=None):
return downloaded_file_bytes
# pylint:disable=redefined-builtin
- def upload(self, bucket, object, filename, mime_type='application/octet-stream'):
+ def upload(self, bucket, object, filename,
+ mime_type='application/octet-stream', gzip=False,
+ multipart=False, num_retries=0):
"""
Uploads a local file to Google Cloud Storage.
:param bucket: The bucket to upload to.
- :type bucket: string
+ :type bucket: str
:param object: The object name to set when uploading the local file.
- :type object: string
+ :type object: str
:param filename: The local file path to the file to be uploaded.
- :type filename: string
+ :type filename: str
:param mime_type: The MIME type to set when uploading the file.
- :type mime_type: string
+ :type mime_type: str
+ :param gzip: Option to compress file for upload
+ :type gzip: bool
+ :param multipart: If True, the upload will be split into multiple HTTP requests. The
+ default size is 256MiB per request. Pass a number instead of True to
+ specify the request size, which must be a multiple of 262144 (256KiB).
+ :type multipart: bool or int
+ :param num_retries: The number of times to attempt to re-upload the file (or individual
+ chunks, in the case of multipart uploads). Retries are attempted
+ with exponential backoff.
+ :type num_retries: int
"""
service = self.get_conn()
- media = MediaFileUpload(filename, mime_type)
- response = service \
- .objects() \
- .insert(bucket=bucket, name=object, media_body=media) \
- .execute()
+
+ if gzip:
+ filename_gz = filename + '.gz'
+
+ with open(filename, 'rb') as f_in:
+ with gz.open(filename_gz, 'wb') as f_out:
+ shutil.copyfileobj(f_in, f_out)
+ filename = filename_gz
+
+ try:
+ if multipart:
+ if multipart is True:
+ chunksize = 256 * 1024 * 1024
+ else:
+ chunksize = multipart
+
+ if chunksize % (256 * 1024) > 0 or chunksize < 0:
+ raise ValueError("Multipart size is not a multiple of 262144 (256KiB)")
+
+ media = MediaFileUpload(filename, mimetype=mime_type,
+ chunksize=chunksize, resumable=True)
+
+ request = service.objects().insert(bucket=bucket, name=object, media_body=media)
+ response = None
+ while response is None:
+ status, response = request.next_chunk(num_retries=num_retries)
+ if status:
+ self.log.info("Upload progress %.1f%%", status.progress() * 100)
+
+ else:
+ media = MediaFileUpload(filename, mime_type)
+
+ service \
+ .objects() \
+ .insert(bucket=bucket, name=object, media_body=media) \
+ .execute(num_retries=num_retries)
+
+ except errors.HttpError as ex:
+ if ex.resp['status'] == '404':
+ return False
+ raise
+
+ finally:
+ if gzip:
+ os.remove(filename)
+
+ return True
# pylint:disable=redefined-builtin
def exists(self, bucket, object):
@@ -136,10 +256,10 @@ def exists(self, bucket, object):
Checks for the existence of a file in Google Cloud Storage.
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
+ :type bucket: str
:param object: The name of the object to check in the Google cloud
storage bucket.
- :type object: string
+ :type object: str
"""
service = self.get_conn()
try:
@@ -159,10 +279,10 @@ def is_updated_after(self, bucket, object, ts):
Checks if an object is updated in Google Cloud Storage.
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
+ :type bucket: str
:param object: The name of the object to check in the Google cloud
storage bucket.
- :type object: string
+ :type object: str
:param ts: The timestamp to check against.
:type ts: datetime
"""
@@ -198,11 +318,11 @@ def delete(self, bucket, object, generation=None):
parameter is used.
:param bucket: name of the bucket, where the object resides
- :type bucket: string
+ :type bucket: str
:param object: name of the object to delete
- :type object: string
+ :type object: str
:param generation: if present, permanently delete the object of this generation
- :type generation: string
+ :type generation: str
:return: True if succeeded
"""
service = self.get_conn()
@@ -223,22 +343,23 @@ def list(self, bucket, versions=None, maxResults=None, prefix=None, delimiter=No
List all objects from the bucket with the give string prefix in name
:param bucket: bucket name
- :type bucket: string
+ :type bucket: str
:param versions: if true, list all versions of the objects
- :type versions: boolean
+ :type versions: bool
:param maxResults: max count of items to return in a single page of responses
- :type maxResults: integer
- :param prefix: prefix string which filters objects whose name begin with this prefix
- :type prefix: string
+ :type maxResults: int
+ :param prefix: prefix string which filters objects whose name begin with
+ this prefix
+ :type prefix: str
:param delimiter: filters objects based on the delimiter (for e.g '.csv')
- :type delimiter:string
+ :type delimiter: str
:return: a stream of object names matching the filtering criteria
"""
service = self.get_conn()
ids = list()
pageToken = None
- while(True):
+ while True:
response = service.objects().list(
bucket=bucket,
versions=versions,
@@ -273,13 +394,16 @@ def list(self, bucket, versions=None, maxResults=None, prefix=None, delimiter=No
def get_size(self, bucket, object):
"""
Gets the size of a file in Google Cloud Storage.
+
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
- :param object: The name of the object to check in the Google cloud
- storage bucket.
- :type object: string
+ :type bucket: str
+ :param object: The name of the object to check in the Google cloud storage bucket.
+ :type object: str
+
"""
- self.log.info('Checking the file size of object: %s in bucket: %s', object, bucket)
+ self.log.info('Checking the file size of object: %s in bucket: %s',
+ object,
+ bucket)
service = self.get_conn()
try:
response = service.objects().get(
@@ -290,10 +414,266 @@ def get_size(self, bucket, object):
if 'name' in response and response['name'][-1] != '/':
# Remove Directories & Just check size of files
size = response['size']
- self.log.info('The file size of %s is %s', object, size)
+ self.log.info('The file size of %s is %s bytes.', object, size)
return size
else:
raise ValueError('Object is not a file')
except errors.HttpError as ex:
if ex.resp['status'] == '404':
raise ValueError('Object Not Found')
+
+ def get_crc32c(self, bucket, object):
+ """
+ Gets the CRC32c checksum of an object in Google Cloud Storage.
+
+ :param bucket: The Google cloud storage bucket where the object is.
+ :type bucket: str
+ :param object: The name of the object to check in the Google cloud
+ storage bucket.
+ :type object: str
+ """
+ self.log.info('Retrieving the crc32c checksum of '
+ 'object: %s in bucket: %s', object, bucket)
+ service = self.get_conn()
+ try:
+ response = service.objects().get(
+ bucket=bucket,
+ object=object
+ ).execute()
+
+ crc32c = response['crc32c']
+ self.log.info('The crc32c checksum of %s is %s', object, crc32c)
+ return crc32c
+
+ except errors.HttpError as ex:
+ if ex.resp['status'] == '404':
+ raise ValueError('Object Not Found')
+
+ def get_md5hash(self, bucket, object):
+ """
+ Gets the MD5 hash of an object in Google Cloud Storage.
(This diff was longer than 20,000 lines, and has been truncated...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services