You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2018/10/12 01:33:32 UTC
[GitHub] danmactough closed pull request #3938: [AIRFLOW-3046] Fix ECS
Operator mistakenly reports success
danmactough closed pull request #3938: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success
URL: https://github.com/apache/incubator-airflow/pull/3938
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 6000d0e5ff..90452d954b 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,33 +1,34 @@
Make sure you have checked _all_ steps below.
-### JIRA
-- [ ] My PR addresses the following [Airflow JIRA](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- - https://issues.apache.org/jira/browse/AIRFLOW-XXX
- - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a JIRA issue.
+### Jira
+- [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
+ - https://issues.apache.org/jira/browse/AIRFLOW-XXX
+ - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
-- [ ] Here are some details about my PR, including screenshots of any UI changes:
+- [ ] Here are some details about my PR, including screenshots of any UI changes:
### Tests
-- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
+- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
### Commits
-- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
- 1. Subject is separated from body by a blank line
- 2. Subject is limited to 50 characters
- 3. Subject does not end with a period
- 4. Subject uses the imperative mood ("add", not "adding")
- 5. Body wraps at 72 characters
- 6. Body explains "what" and "why", not "how"
+- [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
+ 1. Subject is separated from body by a blank line
+ 1. Subject is limited to 50 characters (not including Jira issue reference)
+ 1. Subject does not end with a period
+ 1. Subject uses the imperative mood ("add", not "adding")
+ 1. Body wraps at 72 characters
+ 1. Body explains "what" and "why", not "how"
### Documentation
-- [ ] In case of new functionality, my PR adds documentation that describes how to use it.
- - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
+- [ ] In case of new functionality, my PR adds documentation that describes how to use it.
+ - When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
### Code Quality
+
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
diff --git a/.github/stale.yml b/.github/stale.yml
new file mode 100644
index 0000000000..734dda6598
--- /dev/null
+++ b/.github/stale.yml
@@ -0,0 +1,55 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+
+# Number of days of inactivity before an Issue or Pull Request becomes stale
+daysUntilStale: 45
+
+# Number of days of inactivity before an Issue or Pull Request with the stale label is closed.
+# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale.
+daysUntilClose: 7
+
+# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable
+exemptLabels:
+ - pinned
+ - security
+ - "[Status] Maybe Later"
+
+# Set to true to ignore issues in a project (defaults to false)
+exemptProjects: false
+
+# Set to true to ignore issues in a milestone (defaults to false)
+exemptMilestones: false
+
+# Label to use when marking as stale
+staleLabel: stale
+
+# Comment to post when marking as stale. Set to `false` to disable
+markComment: >
+ This issue has been automatically marked as stale because it has not had
+ recent activity. It will be closed if no further activity occurs. Thank you
+ for your contributions.
+
+# Comment to post when removing the stale label.
+# unmarkComment: >
+# Your comment here.
+
+# Comment to post when closing a stale Issue or Pull Request.
+# closeComment: >
+# Your comment here.
+
+# Limit the number of actions per hour, from 1-30. Default is 30
+limitPerRun: 30
+
+# Limit to only `issues` or `pulls`
+# only: issues
+
+# Optionally, specify configuration settings that are specific to just 'issues' or 'pulls':
+# pulls:
+# daysUntilStale: 30
+# markComment: >
+# This pull request has been automatically marked as stale because it has not had
+# recent activity. It will be closed if no further activity occurs. Thank you
+# for your contributions.
+
+# issues:
+# exemptLabels:
+# - confirmed
diff --git a/.gitignore b/.gitignore
index 0e53aaded0..d005437890 100644
--- a/.gitignore
+++ b/.gitignore
@@ -138,3 +138,11 @@ rat-results.txt
*.generated
*.tar.gz
scripts/ci/kubernetes/kube/.generated/airflow.yaml
+
+# Node & Webpack Stuff
+*.entry.js
+node_modules
+npm-debug.log*
+static/dist
+derby.log
+metastore_db
diff --git a/.rat-excludes b/.rat-excludes
index e5373aacd5..b13c1648c4 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -3,6 +3,8 @@
.coverage
.coveragerc
.codecov.yml
+.eslintrc
+.eslintignore
.rat-excludes
requirements.txt
.*log
@@ -37,3 +39,5 @@ bootstrap-toggle.min.js
bootstrap-toggle.min.css
d3.v3.min.js
ace.js
+airflow/www_rbac/node_modules
+.*json
diff --git a/.readthedocs.yml b/.readthedocs.yml
index 643548c03e..87fb227d81 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -6,9 +6,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -19,7 +19,14 @@
python:
pip_install: true
extra_requirements:
+ - all_dbs
+ - databricks
- doc
- docker
+ - emr
- gcp_api
- - emr
+ - s3
+ - salesforce
+ - sendgrid
+ - ssh
+ - slack
diff --git a/.travis.yml b/.travis.yml
index 01c08d9537..dd493363ab 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -19,93 +19,44 @@
sudo: true
dist: trusty
language: python
-jdk:
- - oraclejdk8
-services:
- - cassandra
- - mysql
- - postgresql
- - rabbitmq
-addons:
- apt:
- packages:
- - slapd
- - ldap-utils
- - openssh-server
- - mysql-server-5.6
- - mysql-client-core-5.6
- - mysql-client-5.6
- - krb5-user
- - krb5-kdc
- - krb5-admin-server
- - oracle-java8-installer
- - python-selinux
- postgresql: "9.2"
-python:
- - "2.7"
- - "3.5"
env:
global:
+ - DOCKER_COMPOSE_VERSION=1.20.0
+ - SLUGIFY_USES_TEXT_UNIDECODE=yes
- TRAVIS_CACHE=$HOME/.travis_cache/
- - KRB5_CONFIG=/etc/krb5.conf
- - KRB5_KTNAME=/etc/airflow.keytab
- # Travis on google cloud engine has a global /etc/boto.cfg that
- # does not work with python 3
- - BOTO_CONFIG=/tmp/bogusvalue
matrix:
- - TOX_ENV=py27-backend_mysql
- - TOX_ENV=py27-backend_sqlite
- - TOX_ENV=py27-backend_postgres
- - TOX_ENV=py35-backend_mysql
- - TOX_ENV=py35-backend_sqlite
- - TOX_ENV=py35-backend_postgres
- TOX_ENV=flake8
- - TOX_ENV=py27-backend_postgres KUBERNETES_VERSION=v1.9.0
- - TOX_ENV=py35-backend_postgres KUBERNETES_VERSION=v1.10.0
-matrix:
- exclude:
- - python: "3.5"
- env: TOX_ENV=py27-backend_mysql
- - python: "3.5"
- env: TOX_ENV=py27-backend_sqlite
- - python: "3.5"
- env: TOX_ENV=py27-backend_postgres
- - python: "2.7"
- env: TOX_ENV=py35-backend_mysql
- - python: "2.7"
- env: TOX_ENV=py35-backend_sqlite
- - python: "2.7"
- env: TOX_ENV=py35-backend_postgres
- - python: "2.7"
- env: TOX_ENV=flake8
- - python: "3.5"
- env: TOX_ENV=py27-backend_postgres KUBERNETES_VERSION=v1.9.0
- - python: "2.7"
- env: TOX_ENV=py35-backend_postgres KUBERNETES_VERSION=v1.10.0
+ - TOX_ENV=py27-backend_mysql-env_docker
+ - TOX_ENV=py27-backend_sqlite-env_docker
+ - TOX_ENV=py27-backend_postgres-env_docker
+ - TOX_ENV=py35-backend_mysql-env_docker PYTHON_VERSION=3
+ - TOX_ENV=py35-backend_sqlite-env_docker PYTHON_VERSION=3
+ - TOX_ENV=py35-backend_postgres-env_docker PYTHON_VERSION=3
+ - TOX_ENV=py27-backend_postgres-env_kubernetes KUBERNETES_VERSION=v1.9.0
+ - TOX_ENV=py35-backend_postgres-env_kubernetes KUBERNETES_VERSION=v1.10.0 PYTHON_VERSION=3
+
cache:
directories:
- $HOME/.wheelhouse/
+ - $HOME/.cache/pip
- $HOME/.travis_cache/
before_install:
- - yes | ssh-keygen -t rsa -C your_email@youremail.com -P '' -f ~/.ssh/id_rsa
- - cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
- - ln -s ~/.ssh/authorized_keys ~/.ssh/authorized_keys2
- - chmod 600 ~/.ssh/*
- - jdk_switcher use oraclejdk8
-install:
- - pip install --upgrade pip
- - pip install tox
- - pip install codecov
-before_script:
- - cat "$TRAVIS_BUILD_DIR/scripts/ci/my.cnf" | sudo tee -a /etc/mysql/my.cnf
- - mysql -e 'drop database if exists airflow; create database airflow' -u root
- - sudo service mysql restart
- - psql -c 'create database airflow;' -U postgres
- - export PATH=${PATH}:/tmp/hive/bin
# Required for K8s v1.10.x. See
# https://github.com/kubernetes/kubernetes/issues/61058#issuecomment-372764783
- - sudo mount --make-shared / && sudo service docker restart
+ - if [ ! -z "$KUBERNETES_VERSION" ]; then sudo mount --make-shared / && sudo service docker restart; fi
+install:
+ # Use recent docker-compose version
+ - sudo rm /usr/local/bin/docker-compose
+ - curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
+ - chmod +x docker-compose
+ - sudo mv docker-compose /usr/local/bin
+ - pip install --upgrade pip
+ - if [ ! -z "$KUBERNETES_VERSION" ]; then ./scripts/ci/kubernetes/setup_kubernetes.sh; fi
script:
- - ./scripts/ci/travis_script.sh
-after_success:
- - codecov
+ - if [ -z "$KUBERNETES_VERSION" ]; then docker-compose --log-level ERROR -f scripts/ci/docker-compose.yml run airflow-testing /app/scripts/ci/run-ci.sh; fi
+ - if [ ! -z "$KUBERNETES_VERSION" ]; then
+ ./scripts/ci/kubernetes/kube/deploy.sh &&
+ MINIKUBE_IP=$(minikube ip) docker-compose --log-level ERROR -f scripts/ci/docker-compose.yml -f scripts/ci/docker-compose-kubernetes.yml run airflow-testing /app/scripts/ci/run-ci.sh;
+ fi
+before_cache:
+ - sudo chown -R travis:travis $HOME/.cache/pip $HOME/.wheelhouse/
diff --git a/CHANGELOG.txt b/CHANGELOG.txt
index fa4e6547a7..b4ee1755b4 100644
--- a/CHANGELOG.txt
+++ b/CHANGELOG.txt
@@ -1,3 +1,785 @@
+AIRFLOW 1.10.0, 2018-08-03
+--------------------------
+[AIRFLOW-2870] Use abstract TaskInstance for migration
+[AIRFLOW-2859] Implement own UtcDateTime (#3708)
+[AIRFLOW-2140] Don't require kubernetes for the SparkSubmit hook
+[AIRFLOW-2869] Remove smart quote from default config
+[AIRFLOW-2857] Fix Read the Docs env
+[AIRFLOW-2817] Force explicit choice on GPL dependency
+[AIRFLOW-2716] Replace async and await py3.7 keywords
+[AIRFLOW-2810] Fix typo in Xcom model timestamp
+[AIRFLOW-2710] Clarify fernet key value in documentation
+[AIRFLOW-2606] Fix DB schema and SQLAlchemy model
+[AIRFLOW-2646] Fix setup.py not to install snakebite on Python3
+[AIRFLOW-2512][AIRFLOW-2522] Use google-auth instead of oauth2client
+[AIRFLOW-2604] Add index to task_fail
+[AIRFLOW-2650] Mark SchedulerJob as succeed when hitting Ctrl-c
+[AIRFLOW-2678] Fix db schema unit test to remove checking fab models
+[AIRFLOW-2624] Fix webserver login as anonymous
+[AIRFLOW-2654] Fix incorret URL on refresh in Graph View of FAB UI
+[AIRFLOW-2668] Handle missing optional cryptography dependency
+[AIRFLOW-2681] Include last dag run of externally triggered DAGs in UI.
+[AIRFLOW-1840] Support back-compat on old celery config
+[AIRFLOW-2612][AIRFLOW-2534] Clean up Hive-related tests
+[AIRFLOW-2608] Implements/Standardize custom exceptions for experimental APIs
+[AIRFLOW-2607] Fix failing TestLocalClient
+[AIRFLOW-2638] dbapi_hook: support REPLACE INTO
+[AIRFLOW-2542][AIRFLOW-1790] Rename AWS Batch Operator queue to job_queue
+[AIRFLOW-2567] Extract result from the kubernetes pod as Xcom
+[AIRFLOW-XXX] Adding REA Group to readme
+[AIRFLOW-2601] Allow user to specify k8s config
+[AIRFLOW-2559] Azure Fileshare hook
+[AIRFLOW-1786] Enforce correct behavior for soft-fail sensors
+[AIRFLOW-2355] Airflow trigger tag parameters in subdag
+[AIRFLOW-2613] Fix Airflow searching .zip bug
+[AIRFLOW-2627] Add a sensor for Cassandra
+[AIRFLOW-2634][AIRFLOW-2534] Remove dependency for impyla
+[AIRFLOW-2611] Fix wrong dag volume mount path for kubernetes executor
+[AIRFLOW-2562] Add Google Kubernetes Engine Operators
+[AIRFLOW-2630] Fix classname in test_sql_sensor.py
+[AIRFLOW-2534] Fix bug in HiveServer2Hook
+[AIRFLOW-2586] Stop getting AIRFLOW_HOME value from config file in bash operator
+[AIRFLOW-2605] Fix autocommit for MySqlHook
+[AIRFLOW-2539][AIRFLOW-2359] Move remaing log config to configuration file
+[AIRFLOW-1656] Tree view dags query changed
+[AIRFLOW-2617] add imagePullPolicy config for kubernetes executor
+[AIRFLOW-2429] Fix security/task/sensors/ti_deps folders flake8 error
+[AIRFLOW-2550] Implements API endpoint to list DAG runs
+[AIRFLOW-2512][AIRFLOW-2522] Use google-auth instead of oauth2client
+[AIRFLOW-2429] Fix operators folder flake8 error
+[AIRFLOW-2585] Fix several bugs in CassandraHook and CassandraToGCSOperator
+[AIRFLOW-2597] Restore original dbapi.run() behavior
+[AIRFLOW-2590] Fix commit in DbApiHook.run() for no-autocommit DB
+[AIRFLOW-1115] fix github oauth api URL
+[AIRFLOW-2587] Add TIMESTAMP type mapping to MySqlToHiveTransfer
+[AIRFLOW-2591][AIRFLOW-2581] Set default value of autocommit to False in DbApiHook.run()
+[AIRFLOW-59] Implement bulk_dump and bulk_load for the Postgres hook
+[AIRFLOW-2533] Fix path to DAG's on kubernetes executor workers
+[AIRFLOW-2581] RFLOW-2581] Fix DbApiHook autocommit
+[AIRFLOW-2578] Add option to use proxies in JiraHook
+[AIRFLOW-2575] Make gcs to gcs operator work with large files
+[AIRFLOW-437] Send TI context in kill zombies
+[AIRFLOW-2566] Change backfill to rerun failed tasks
+[AIRFLOW-1021] Fix double login for new users with LDAP
+[AIRFLOW-XXX] Typo fix
+[AIRFLOW-2561] Fix typo in EmailOperator
+[AIRFLOW-2573] Cast BigQuery TIMESTAMP field to float
+[AIRFLOW-2560] Adding support for internalIpOnly to DataprocClusterCreateOperator
+[AIRFLOW-2565] templatize cluster_label
+[AIRFLOW-83] add mongo hook and operator
+[AIRFLOW-2558] Clear task/dag is clearing all executions
+[AIRFLOW-XXX] Fix doc typos
+[AIRFLOW-2513] Change `bql` to `sql` for BigQuery Hooks & Ops
+[AIRFLOW-2557] Fix pagination for s3
+[AIRFLOW-2545] Eliminate DeprecationWarning
+[AIRFLOW-2500] Fix MySqlToHiveTransfer to transfer unsigned type properly
+[AIRFLOW-2462] Change PasswordUser setter to correct syntax
+[AIRFLOW-2525] Fix a bug introduced by commit dabf1b9
+[AIRFLOW-2553] Add webserver.pid to .gitignore
+[AIRFLOW-1863][AIRFLOW-2529] Add dag run selection widgets to gantt view
+[AIRFLOW-2504] Log username correctly and add extra to search columns
+[AIRFLOW-2551] Encode binary data with base64 standard rather than base64 url
+[AIRFLOW-2537] Add reset-dagrun option to backfill command
+[AIRFLOW-2526] dag_run.conf can override params
+[AIRFLOW-2544][AIRFLOW-1967] Guard against next major release of Celery, Flower
+[AIRFLOW-XXX] Add Yieldr to who is using airflow
+[AIRFLOW-2547] Describe how to run tests using Docker
+[AIRFLOW-2538] Update faq doc on how to reduce airflow scheduler latency
+[AIRFLOW-2529] Improve graph view performance and usability
+[AIRFLOW-2517] backfill support passing key values through CLI
+[AIRFLOW-2532] Support logs_volume_subpath for KubernetesExecutor
+[AIRFLOW-2466] consider task_id in _change_state_for_tis_without_dagrun
+[AIRFLOW-2519] Fix CeleryExecutor with SQLAlchemy
+[AIRFLOW-2402] Fix RBAC task log
+[AIRFLOW-XXX] Add M4U to user list
+[AIRFLOW-2536] docs about how to deal with airflow initdb failure
+[AIRFLOW-2530] KubernetesOperator supports multiple clusters
+[AIRFLOW-1499] Eliminate duplicate and unneeded code
+[AIRFLOW-2521] backfill - make variable name and logging messages more acurate
+[AIRFLOW-2429] Fix hook, macros folder flake8 error
+[Airflow-XXX] add Prime to company list
+[AIRFLOW-2525] Fix PostgresHook.copy_expert to work with "COPY FROM"
+[AIRFLOW-2515] Add dependency on thrift_sasl to hive extra
+[AIRFLOW-2523] Add how-to for managing GCP connections
+[AIRFLOW-2510] Introduce new macros: prev_ds and next_ds
+[AIRFLOW-1730] Unpickle value of XCom queried from DB
+[AIRFLOW-2518] Fix broken ToC links in integration.rst
+[AIRFLOW-1472] Fix SLA misses triggering on skipped tasks.
+[AIRFLOW-2520] CLI - make backfill less verbose
+[AIRFLOW-2107] add time_partitioning to run_query on BigQueryBaseCursor
+[AIRFLOW-1057][AIRFLOW-1380][AIRFLOW-2362][2362] AIRFLOW Update DockerOperator to new API
+[AIRFLOW-2415] Make airflow DAG templating render numbers
+[AIRFLOW-2473] Fix wrong skip condition for TransferTests
+[AIRFLOW-2472] Implement MySqlHook.bulk_dump
+[AIRFLOW-2419] Use default view for subdag operator
+[AIRFLOW-2498] Fix Unexpected argument in SFTP Sensor
+[AIRFLOW-2509] Separate config docs into how-to guides
+[AIRFLOW-2429] Add BaseExecutor back
+[AIRFLOW-2429] Fix dag, example_dags, executors flake8 error
+[AIRFLOW-2502] Change Single triple quotes to double for docstrings
+[AIRFLOW-2503] Fix broken links in CONTRIBUTING.md
+[AIRFLOW-2501] Refer to devel instructions in docs contrib guide
+[AIRFLOW-2429] Fix contrib folder's flake8 errors
+[AIRFLOW-2471] Fix HiveCliHook.load_df to use unused parameters
+[AIRFLOW-2495] Update celery to 4.1.1
+[AIRFLOW-2429] Fix api, bin, config_templates folders flake8 error
+[AIRFLOW-2493] Mark template_fields of all Operators in the API document as "templated"
+[AIRFLOW-2489] Update FlaskAppBuilder to 1.11.1
+[AIRFLOW-2448] Enhance HiveCliHook.load_df to work with datetime
+[AIRFLOW-2487] Enhance druid ingestion hook
+[AIRFLOW-2397] Support affinity policies for Kubernetes executor/operator
+[AIRFLOW-2482] Add test for rewrite method in GCS Hook
+[AIRFLOW-2481] Fix flaky Kubernetes test
+[AIRFLOW-2479] Improve doc FAQ section
+[AIRFLOW-2485] Fix Incorrect logging for Qubole Sensor
+[AIRFLOW-2486] Remove unnecessary slash after port
+[AIRFLOW-2429] Make Airflow flake8 compliant
+[AIRFLOW-2491] Resolve flask version conflict
+[AIRFLOW-2484] Remove duplicate key in MySQL to GCS Op
+[ARIFLOW-2458] Add cassandra-to-gcs operator
+[AIRFLOW-2477] Improve time units for task duration and landing times charts for RBAC UI
+[AIRFLOW-2474] Only import snakebite if using py2
+[AIRFLOW-48] Parse connection uri querystring
+[AIRFLOW-2467][AIRFLOW-2] Update import direct warn message to use the module name
+[AIRFLOW-XXX] Fix order of companies
+[AIRFLOW-2452] Document field_dict must be OrderedDict
+[AIRFLOW-2420] Azure Data Lake Hook
+[AIRFLOW-2213] Add Quoble check operator
+[AIRFLOW-2465] Fix wrong module names in the doc
+[AIRFLOW-1929] Modifying TriggerDagRunOperator to accept execution_date
+[AIRFLOW-2460] Users can now use volume mounts and volumes
+[AIRFLOW-2110][AIRFLOW-2122] Enhance Http Hook
+[AIRFLOW-XXX] Updated contributors list
+[AIRFLOW-2435] Add launch_type to ECSOperator to allow FARGATE
+[AIRFLOW-2451] Remove extra slash ('/') char when using wildcard in gcs_to_gcs operator
+[AIRFLOW-2461] Add support for cluster scaling on dataproc operator
+[AIRFLOW-2376] Fix no hive section error
+[AIRFLOW-2425] Add lineage support
+[AIRFLOW-2430] Extend query batching to additional slow queries
+[AIRFLOW-2453] Add default nil value for kubernetes/git_subpath
+[AIRFLOW-2396] Add support for resources in kubernetes operator
+[AIRFLOW-2169] Encode binary data with base64 before importing to BigQuery
+[AIRFLOW-XXX] Add spotahome in user list
+[AIRFLOW-2457] Update FAB version requirement
+[AIRFLOW-2454][Airflow 2454] Support imagePullPolicy for k8s
+[AIRFLOW-2450] update supported k8s versions to 1.9 and 1.10
+[AIRFLOW-2333] Add Segment Hook and TrackEventOperator
+[AIRFLOW-2442][AIRFLOW-2] Airflow run command leaves database connections open
+[AIRFLOW-2016] assign template_fields for Dataproc Workflow Template sub-classes, not base class
+[AIRFLOW-2446] Add S3ToRedshiftTransfer into the "Integration" doc
+[AIRFLOW-2449] Fix operators.py to run all test cases
+[AIRFLOW-2424] Add dagrun status endpoint and increased k8s test coverage
+[AIRFLOW-2441] Fix bugs in HiveCliHook.load_df
+[AIRFLOW-2358][AIRFLOW-201804] Make the Kubernetes example optional
+[AIRFLOW-2436] Remove cli_logger in initdb
+[AIRFLOW-2444] Remove unused option(include_adhoc) in cli backfill command
+[AIRFLOW-2447] Fix TestHiveMetastoreHook to run all cases
+[AIRFLOW-2445] Allow templating in kubernetes operator
+[AIRFLOW-2086][AIRFLOW-2393] Customize default dagrun number in tree view
+[AIRFLOW-2437] Add PubNub to list of current airflow users
+[AIRFLOW-XXX] Add Quantopian to list of Airflow users
+[AIRFLOW-1978] Add WinRM windows operator and hook
+[AIRFLOW-2427] Add tests to named hive sensor
+[AIRFLOW-2412] Fix HiveCliHook.load_file to address HIVE-10541
+[AIRFLOW-2431] Add the navigation bar color parameter for RBAC UI
+[AIRFLOW-2407] Resolve Python undefined names
+[AIRFLOW-1952] Add the navigation bar color parameter
+[AIRFLOW-2222] Implement GoogleCloudStorageHook.rewrite
+[AIRFLOW-2426] Add Google Cloud Storage Hook tests
+[AIRFLOW-2418] Bump Flask-WTF
+[AIRFLOW-2417] Wait for pod is not running to end task
+[AIRFLOW-1914] Add other charset support to email utils
+[AIRFLOW-XXX] Update README.md with Craig@Work
+[AIRFLOW-1899] Fix Kubernetes tests
+[AIRFLOW-1812] Update logging example
+[AIRFLOW-2313] Add TTL parameters for Dataproc
+[AIRFLOW-2411] add dataproc_jars to templated_fields
+[AIRFLOW-XXX] Add Reddit to Airflow users
+[AIRFLOW-XXX] Fix wrong table header in scheduler.rst
+[AIRFLOW-2409] Supply password as a parameter
+[AIRFLOW-2410][AIRFLOW-75] Set the timezone in the RBAC Web UI
+[AIRFLOW-2394] default cmds and arguments in kubernetes operator
+[AIRFLOW-2406] Add Apache2 License Shield to Readme
+[AIRFLOW-2404] Add additional documentation for unqueued task
+[AIRFLOW-2400] Add Ability to set Environment Variables for K8s
+[AIRFLOW-XXX] Add Twine Labs as an Airflow user
+[AIRFLOW-1853] Show only the desired number of runs in tree view
+[AIRFLOW-2401] Document the use of variables in Jinja template
+[AIRFLOW-2403] Fix License Headers
+[AIRFLOW-1313] Fix license header
+[AIRFLOW-2398] Add BounceX to list of current airflow users
+[AIRFLOW-2363] Fix return type bug in TaskHandler
+[AIRFLOW-2389] Create a pinot db api hook
+[AIRFLOW-2390] Resolve FlaskWTFDeprecationWarning
+[AIRFLOW-1933] Fix some typos
+[AIRFLOW-1960] Add support for secrets in kubernetes operator
+[AIRFLOW-1313] Add vertica_to_mysql operator
+[AIRFLOW-1575] Add AWS Kinesis Firehose Hook for inserting batch records
+[AIRFLOW-2266][AIRFLOW-2343] Remove google-cloud-dataflow dependency
+[AIRFLOW-2370] Implement --use_random_password in create_user
+[AIRFLOW-2348] Strip path prefix from the destination_object when source_object contains a wildcard[]
+[AIRFLOW-2391] Fix to Flask 0.12.2
+[AIRFLOW-2381] Fix the flaky ApiPasswordTests test
+[AIRFLOW-2378] Add Groupon to list of current users
+[AIRFLOW-2382] Fix wrong description for delimiter
+[AIRFLOW-2380] Add support for environment variables in Spark submit operator.
+[AIRFLOW-2377] Improve Sendgrid sender support
+[AIRFLOW-2331] Support init action timeout on dataproc cluster create
+[AIRFLOW-1835] Update docs: Variable file is json
+[AIRFLOW-1781] Make search case-insensitive in LDAP group
+[AIRFLOW-2042] Fix browser menu appearing over the autocomplete menu
+[AIRFLOW-XXX] Remove wheelhouse files from travis not owned by travis
+[AIRFLOW-2336] Use hmsclient in hive_hook
+[AIRFLOW-2041] Correct Syntax in python examples
+[AIRFLOW-74] SubdagOperators can consume all celeryd worker processes
+[AIRFLOW-2369] Fix gcs tests
+[AIRFLOW-2365] Fix autocommit attribute check
+[AIRFLOW-2068] MesosExecutor allows optional Docker image
+[AIRFLOW-1652] Push DatabricksRunSubmitOperator metadata into XCOM
+[AIRFLOW-2234] Enable insert_rows for PrestoHook
+[AIRFLOW-2208][Airflow-22208] Link to same DagRun graph from TaskInstance view
+[AIRFLOW-1153] Allow HiveOperators to take hiveconfs
+[AIRFLOW-775] Fix autocommit settings with Jdbc hook
+[AIRFLOW-2364] Warn when setting autocommit on a connection which does not support it
+[AIRFLOW-2357] Add persistent volume for the logs
+[AIRFLOW-766] Skip conn.commit() when in Auto-commit
+[AIRFLOW-2351] Check for valid default_args start_date
+[AIRFLOW-1433] Set default rbac to initdb
+[AIRFLOW-2270] Handle removed tasks in backfill
+[AIRFLOW-2344] Fix `connections -l` to work with pipe/redirect
+[AIRFLOW-2300] Add S3 Select functionarity to S3ToHiveTransfer
+[AIRFLOW-1314] Cleanup the config
+[AIRFLOW-1314] Polish some of the Kubernetes docs/config
+[AIRFLOW-1314] Improve error handling
+[AIRFLOW-1999] Add per-task GCP service account support
+[AIRFLOW-1314] Rebasing against master
+[AIRFLOW-1314] Small cleanup to address PR comments (#24)
+[AIRFLOW-1314] Add executor_config and tests
+[AIRFLOW-1314] Improve k8s support
+[AIRFLOW-1314] Use VolumeClaim for transporting DAGs
+[AIRFLOW-1314] Create integration testing environment
+[AIRFLOW-1314] Git Mode to pull in DAGs for Kubernetes Executor
+[AIRFLOW-1314] Add support for volume mounts & Secrets in Kubernetes Executor
+[AIRFLOW=1314] Basic Kubernetes Mode
+[AIRFLOW-2326][AIRFLOW-2222] remove contrib.gcs_copy_operator
+[AIRFLOW-2328] Fix empty GCS blob in S3ToGoogleCloudStorageOperator
+[AIRFLOW-2350] Fix grammar in UPDATING.md
+[AIRFLOW-2302] Fix documentation
+[AIRFLOW-2345] pip is not used in this setup.py
+[AIRFLOW-2347] Add Banco de Formaturas to Readme
+[AIRFLOW-2346] Add Investorise as official user of Airflow
+[AIRFLOW-2330] Do not append destination prefix if not given
+[AIRFLOW-2240][DASK] Added TLS/SSL support for the dask-distributed scheduler.
+[AIRFLOW-2309] Fix duration calculation on TaskFail
+[AIRFLOW-2335] fix issue with jdk8 download for ci
+[AIRFLOW-2184] Add druid_checker_operator
+[AIRFLOW-2299] Add S3 Select functionarity to S3FileTransformOperator
+[AIRFLOW-2254] Put header as first row in unload
+[AIRFLOW-610] Respect _cmd option in config before defaults
+[AIRFLOW-2287] Fix incorrect ASF headers
+[AIRFLOW-XXX] Add Zego as an Apache Airflow user
+[AIRFLOW-952] fix save empty extra field in UI
+[AIRFLOW-1325] Add ElasticSearch log handler and reader
+[AIRFLOW-2301] Sync files of an S3 key with a GCS path
+[AIRFLOW-2293] Fix S3FileTransformOperator to work with boto3
+[AIRFLOW-3212][AIRFLOW-2314] Remove only leading slash in GCS path
+[AIRFLOW-1509][AIRFLOW-442] SFTP Sensor
+[AIRFLOW-2291] Add optional params to ML Engine
+[AIRFLOW-1774] Allow consistent templating of arguments in MLEngineBatchPredictionOperator
+[AIRFLOW-2302] Add missing operators and hooks
+[AIRFLOW-2312] Docs Typo Correction: Corresponding
+[AIRFLOW-1623] Trigger on_kill method in operators
+[AIRFLOW-2162] When impersonating another user, pass env variables to sudo
+[AIRFLOW-2304] Update quickstart doc to mention scheduler part
+[AIRFLOW-1633] docker_operator needs a way to set shm_size
+[AIRFLOW-1340] Add S3 to Redshift transfer operator
+[AIRFLOW-2303] Lists the keys inside an S3 bucket
+[AIRFLOW-2209] restore flask_login imports
+[AIRFLOW-2306] Add Bonnier Broadcasting to list of current users
+[AIRFLOW-2305][AIRFLOW-2027] Fix CI failure caused by []
+[AIRFLOW-2281] Add support for Sendgrid categories
+[AIRFLOW-2027] Only trigger sleep in scheduler after all files have parsed
+[AIRFLOW-2256] SparkOperator: Add Client Standalone mode and retry mechanism
+[AIRFLOW-2284] GCS to S3 operator
+[AIRFLOW-2287] Update license notices
+[AIRFLOW-2296] Add Cinimex DataLab to Readme
+[AIRFLOW-2298] Add Kalibrr to who uses airflow
+[AIRFLOW-2292] Fix docstring for S3Hook.get_wildcard_key
+[AIRFLOW-XXX] Update PR template
+[AIRFLOW-XXX] Remove outdated migrations.sql
+[AIRFLOW-2287] Add license header to docs/Makefile
+[AIRFLOW-2286] Add tokopedia to the readme
+[AIRFLOW-2273] Add Discord webhook operator/hook
+[AIRFLOW-2282] Fix grammar in UPDATING.md
+[AIRFLOW-2200] Add snowflake operator with tests
+[AIRFLOW-2178] Add handling on SLA miss errors
+[AIRFLOW-2169] Fix type 'bytes' is not JSON serializable in python3
+[AIRFLOW-2215] Pass environment to subproces.Popen in base_task_runner
+[AIRFLOW-2253] Add Airflow CLI instrumentation
+[AIRFLOW-2274] Fix Dataflow tests
+[AIRFLOW-2269] Add Custom Ink as an Airflow user
+[AIRFLOW-2259] Dataflow Hook Index out of range
+[AIRFLOW-2233] Update updating.md to include the info of hdfs_sensors renaming
+[AIRFLOW-2217] Add Slack webhook operator
+[AIRFLOW-1729] improve dagBag time
+[AIRFLOW-2264] Improve create_user cli help message
+[AIRFLOW-2260] [AIRFLOW-2260] SSHOperator add command template .sh files
+[AIRFLOW-2261] Check config/env for remote base log folder
+[AIRFLOW-2258] Allow import of Parquet-format files into BigQuery
+[AIRFLOW-1430] Include INSTALL instructions to avoid GPL
+[AIRFLOW-1430] Solve GPL dependency
+[AIRFLOW-2251] Add Thinknear as an Airflow user
+[AIRFLOW-2244] bugfix: remove legacy LongText code from models.py
+[AIRFLOW-2247] Fix RedshiftToS3Transfer not to fail with ValueError
+[AIRFLOW-2249] Add side-loading support for Zendesk Hook
+[AIRFLOW-XXX] Add Qplum to Airflow users
+[AIRFLOW-2228] Enhancements in ValueCheckOperator
+[AIRFLOW-1206] Typos
+[AIRFLOW-2060] Update pendulum version to 1.4.4
+[AIRFLOW-2248] Fix wrong param name in RedshiftToS3Transfer doc
+[AIRFLOW-1433][AIRFLOW-85] New Airflow Webserver UI with RBAC support
+[AIRFLOW-1235] Fix webserver's odd behaviour
+[AIRFLOW-1460] Allow restoration of REMOVED TI's
+[airflow-2235] Fix wrong docstrings in two operators
+[AIRFLOW-XXX] Fix chronological order for companies using Airflow
+[AIRFLOW-2124] Upload Python file to a bucket for Dataproc
+[AIRFLOW-2212] Fix ungenerated sensor API reference
+[AIRFLOW-2226] Rename google_cloud_storage_default to google_cloud_default
+[AIRFLOW-2211] Rename hdfs_sensors.py to hdfs_sensor.py for consistency
+[AIRFLOW-2225] Update document to include DruidDbApiHook
+[Airflow-2202] Add filter support in HiveMetastoreHook().max_partition()
+[AIRFLOW-2220] Remove duplicate numeric list entry in security.rst
+[AIRFLOW-XXX] Update tutorial documentation
+[AIRFLOW-2215] Update celery task to preserve environment variables and improve logging on exception
+[AIRFLOW-2185] Use state instead of query param
+[AIRFLOW-2183] Refactor DruidHook to enable sql
+[AIRFLOW-2203] Defer cycle detection
+[AIRFLOW-2203] Remove Useless Commands.
+[AIRFLOW-2203] Cache signature in apply_defaults
+[AIRFLOW-2203] Speed up Operator Resources
+[AIRFLOW-2203] Cache static rules (trigger/weight)
+[AIRFLOW-2203] Store task ids as sets not lists
+[AIRFLOW-2205] Remove unsupported args from JdbcHook doc
+[AIRFLOW-2207] Fix flaky test that uses app.cached_app()
+[AIRFLOW-2206] Remove unsupported args from JdbcOperator doc
+[AIRFLOW-2140] Add Kubernetes scheduler to SparkSubmitOperator
+[AIRFLOW-XXX] Add Xero to list of users
+[AIRFLOW-2204] Fix webserver debug mode
+[AIRFLOW-102] Fix test_complex_template always succeeds
+[AIRFLOW-442] Add SFTPHook
+[AIRFLOW-2169] Add schema to MySqlToGoogleCloudStorageOperator
+[AIRFLOW-2184][AIRFLOW-2138] Google Cloud Storage allow wildcards
+[AIRFLOW-1588] Cast Variable value to string
+[AIRFLOW-2199] Fix invalid reference to logger
+[AIRFLOW-2191] Change scheduler heartbeat logs from info to debug
+[AIRFLOW-2106] SalesForce hook sandbox option
+[AIRFLOW-2197] Silence hostname_callable config error message
+[AIRFLOW-2150] Use lighter call in HiveMetastoreHook().max_partition()
+[AIRFLOW-2186] Change the way logging is carried out in few ops
+[AIRFLOW-2181] Convert password_auth and test_password_endpoints from DOS to UNIX
+[AIRFLOW-2187] Fix Broken Travis CI due to AIRFLOW-2123
+[AIRFLOW-2175] Check that filepath is not None
+[AIRFLOW-2173] Don't check task IDs for concurrency reached check
+[AIRFLOW-2168] Remote logging for Azure Blob Storage
+[AIRFLOW-XXX] Add DocuTAP to list of users
+[AIRFLOW-2176] Change the way logging is carried out in BQ Get Data Operator
+[AIRFLOW-2177] Add mock test for GCS Download op
+[AIRFLOW-2123] Install CI dependencies from setup.py
+[AIRFLOW-2129] Presto hook calls _parse_exception_message but defines _get_pretty_exception_message
+[AIRFLOW-2174] Fix typos and wrongly rendered documents
+[AIRFLOW-2171] Store delegated credentials
+[AIRFLOW-2166] Restore BQ run_query dialect param
+[AIRFLOW-2163] Add HBC Digital to users of airflow
+[AIRFLOW-2065] Fix race-conditions when creating loggers
+[AIRFLOW-2147] Plugin manager: added 'sensors' attribute
+[AIRFLOW-2059] taskinstance query is awful, un-indexed, and does not scale
+[AIRFLOW-2159] Fix a few typos in salesforce_hook
+[AIRFLOW-2132] Add step to initialize database
+[AIRFLOW-2160] Fix bad rowid deserialization
+[AIRFLOW-2161] Add Vevo to list of companies using Airflow
+[AIRFLOW-2149] Add link to apache Beam documentation to create self executing Jar
+[AIRFLOW-2151] Allow getting the session from AwsHook
+[AIRFLOW-2097] tz referenced before assignment
+[AIRFLOW-2152] Add Multiply to list of companies using Airflow
+[AIRFLOW-1551] Add operator to trigger Jenkins job
+[AIRFLOW-2034] Fix mixup between %s and {} when using str.format Convention is to use .format for string formating oustide logging, else use lazy format See comment in related issue https://github.com/apache/incubator-airflow/pull/2823/files Identified problematic case using following command line .git/COMMIT_EDITMSG:`grep -r '%s'./* | grep '\.format('`
+[AIRFLOW-2102] Add custom_args to Sendgrid personalizations
+[AIRFLOW-1035][AIRFLOW-1053] import unicode_literals to parse Unicode in HQL
+[AIRFLOW-2127] Keep loggers during DB migrations
+[AIRFLOW-2146] Resolve issues with BQ using DbApiHook methods
+[AIRFLOW-2087] Scheduler Report shows incorrect Total task number
+[AIRFLOW-2139] Remove unncecessary boilerplate to get DataFrame using pandas_gbq
+[AIRFLOW-2125] Using binary package psycopg2-binary
+[AIRFLOW-2142] Include message on mkdir failure
+[AIRFLOW-1615] SSHHook: use port specified by Connection
+[AIRFLOW-2122] Handle boolean values in sshHook
+[AIRFLOW-XXX] Add Tile to the list of users
+[AIRFLOW-2130] Add missing Operators to API Reference docs
+[AIRFLOW-XXX] Add timeout units (seconds)
+[AIRFLOW-2134] Add Alan to the list of companies that use Airflow
+[AIRFLOW-2133] Remove references to GitHub issues in CONTRIBUTING
+[AIRFLOW-2131] Remove confusing AirflowImport docs
+[AIRFLOW-1852] Allow hostname to be overridable.
+[AIRFLOW-2126] Add Bluecore to active users
+[AIRFLOW-1618] Add feature to create GCS bucket
+[AIRFLOW-2108] Fix log indentation in BashOperator
+[AIRFLOW-2115] Fix doc links to PythonHosted
+[AIRFLOW-XXX] Add contributor from Easy company
+[AIRFLOW-1882] Add ignoreUnknownValues option to gcs_to_bq operator
+[AIRFLOW-2089] Add on kill for SparkSubmit in Standalone Cluster
+[AIRFLOW-2113] Address missing DagRun callbacks Given that the handle_callback method belongs to the DAG object, we are able to get the list of task directly with get_task and reduce the communication with the database, making airflow more lightweight.
+[AIRFLOW-2112] Fix svg width for Recent Tasks on UI.
+[AIRFLOW-2116] Set CI Cloudant version to <2.0
+[AIRFLOW-XXX] Add PMC to list of companies using Airflow
+[AIRFLOW-2100] Fix Broken Documentation Links
+[AIRFLOW-1404] Add 'flatten_results' & 'maximum_bytes_billed' to BQ Operator
+[AIRFLOW-800] Initialize valid Google BigQuery Connection
+[AIRFLOW-1319] Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
+[AIRFLOW-1983] Parse environment parameter as template
+[AIRFLOW-2095] Add operator to create External BigQuery Table
+[AIRFLOW-2085] Add SparkJdbc operator
+[AIRFLOW-1002] Add ability to clean all dependencies of removed DAG
+[AIRFLOW-2094] Jinjafied project_id, region & zone in DataProc{*} Operators
+[AIRFLOW-2092] Fixed incorrect parameter in docstring for FTPHook
+[AIRFLOW-XXX] Add SocialCops to Airflow users
+[AIRFLOW-2088] Fix duplicate keys in MySQL to GCS Helper function
+[AIRFLOW-2091] Fix incorrect docstring parameter in BigQuery Hook
+[AIRFLOW-2090] Fix typo in DataStore Hook
+[AIRFLOW-1157] Fix missing pools crashing the scheduler
+[AIRFLOW-713] Jinjafy {EmrCreateJobFlow,EmrAddSteps}Operator attributes
+[AIRFLOW-2083] Docs: Use "its" instead of "it's" where appropriate
+[AIRFLOW-2066] Add operator to create empty BQ table
+[AIRFLOW-XXX] add Karmic to list of companies
+[AIRFLOW-2073] Make FileSensor fail when the file doesn't exist
+[AIRFLOW-2078] Improve task_stats and dag_stats performance
+[AIRFLOW-2080] Use a log-out icon instead of a power button
+[AIRFLOW-2077] Fetch all pages of list_objects_v2 response
+[AIRFLOW-XXX] Add TM to list of companies
+[AIRFLOW-1985] Impersonation fixes for using `run_as_user`
+[AIRFLOW-2018][AIRFLOW-2] Make Sensors backward compatible
+[AIRFLOW-XXX] Fix typo in concepts doc (dag_md)
+[AIRFLOW-2069] Allow Bytes to be uploaded to S3
+[AIRFLOW-2074] Fix log var name in GHE auth
+[AIRFLOW-1927] Convert naive datetimes for TaskInstances
+[AIRFLOW-1760] Password auth for experimental API
+[AIRFLOW-2038] Add missing kubernetes dependency for dev
+[AIRFLOW-2040] Escape special chars in task instance logs URL
+[AIRFLOW-1968][AIRFLOW-1520] Add role_arn and aws_account_id/aws_iam_role support back to aws hook
+[AIRFLOW-2048] Fix task instance failure string formatting
+[AIRFLOW-2046] Fix kerberos error to work with python 3.x
+[AIRFLOW-2063] Add missing docs for GCP
+[AIRFLOW-XXX] Fix typo in docs
+[AIRFLOW-1793] Use docker_url instead of invalid base_url
+[AIRFLOW-2055] Elaborate on slightly ambiguous documentation
+[AIRFLOW-2039] BigQueryOperator supports priority property
+[AIRFLOW-2053] Fix quote character bug in BQ hook
+[AIRFLOW-2057] Add Overstock to list of companies
+[AIRFLOW-XXX] Add Plaid to Airflow users
+[AIRFLOW-2044] Add SparkSubmitOperator to documentation
+[AIRFLOW-2037] Add methods to get Hash values of a GCS object
+[AIRFLOW-2050] Fix Travis permission problem
+[AIRFLOW-2043] Add Intercom to list of companies
+[AIRFLOW-2023] Add debug logging around number of queued files
+[AIRFLOW-XXX] Add Pernod-ricard as a airflow user
+[AIRFLOW-1453] Add 'steps' into template_fields in EmrAddSteps
+[AIRFLOW-2015] Add flag for interactive runs
+[AIRFLOW-1895] Fix primary key integrity for mysql
+[AIRFLOW-2030] Fix KeyError:`i` in DbApiHook for insert
+[AIRFLOW-1943] Add External BigQuery Table feature
+[AIRFLOW-2033] Add Google Cloud Storage List Operator
+[AIRFLOW-2006] Add local log catching to kubernetes operator
+[AIRFLOW-2031] Add missing gcp_conn_id in the example in DataFlow docstrings
+[AIRFLOW-2029] Fix AttributeError in BigQueryPandasConnector
+[AIRFLOW-2028] Add JobTeaser to official users list
+[AIRFLOW-2016] Add support for Dataproc Workflow Templates
+[AIRFLOW-2025] Reduced Logging verbosity
+[AIRFLOW-1267][AIRFLOW-1874] Add dialect parameter to BigQueryHook
+[AIRFLOW-XXX] Fixed a typo
+[AIRFLOW-XXX] Typo node to nodes
+[AIRFLOW-2019] Update DataflowHook for updating Streaming type job
+[AIRFLOW-2017][Airflow 2017] adding query output to PostgresOperator
+[AIRFLOW-1889] Split sensors into separate files
+[AIRFLOW-1950] Optionally pass xcom_pull task_ids
+[AIRFLOW-1755] Allow mount below root
+[AIRFLOW-511][Airflow 511] add success/failure callbacks on dag level
+[AIRFLOW-192] Add weight_rule param to BaseOperator
+[AIRFLOW-2008] Use callable for python column defaults
+[AIRFLOW-1984] Fix to AWS Batch operator
+[AIRFLOW-2000] Support non-main dataflow job class
+[AIRFLOW-2003] Use flask-caching instead of flask-cache
+[AIRFLOW-2002] Do not swallow exception on logging import
+[AIRFLOW-2004] Import flash from flask not flask.login
+[AIRFLOW-1997] Fix GCP operator doc strings
+[AIRFLOW-1996] Update DataflowHook waitfordone for Streaming type job[]
+[AIRFLOW-1995][Airflow 1995] add on_kill method to SqoopOperator
+[AIRFLOW-1770] Allow HiveOperator to take in a file
+[AIRFLOW-1994] Change background color of Scheduled state Task Instances
+[AIRFLOW-1436][AIRFLOW-1475] EmrJobFlowSensor considers Cancelled step as Successful
+[AIRFLOW-1517] Kubernetes operator PR fixes
+[AIRFLOW-1517] addressed PR comments
+[AIRFLOW-1517] started documentation of k8s operator
+[AIRFLOW-1517] Restore authorship of resources
+[AIRFLOW-1517] Remove authorship of resources
+[AIRFLOW-1517] Add minikube for kubernetes integration tests
+[AIRFLOW-1517] Restore authorship of resources
+[AIRFLOW-1517] fixed license issues
+[AIRFLOW-1517] Created more accurate failures for kube cluster issues
+[AIRFLOW-1517] Remove authorship of resources
+[AIRFLOW-1517] Add minikube for kubernetes integration tests
+[AIRFLOW-1988] Change BG color of None state TIs
+[AIRFLOW-790] Clean up TaskInstances without DagRuns
+[AIRFLOW-1949] Fix var upload, str() produces "b'...'" which is not json
+[AIRFLOW-1930] Convert func.now() to timezone.utcnow()
+[AIRFLOW-1688] Support load.time_partitioning in bigquery_hook
+[AIRFLOW-1975] Make TriggerDagRunOperator callback optional
+[AIRFLOW-1480] Render template attributes for ExternalTaskSensor fields
+[AIRFLOW-1958] Add **kwargs to send_email
+[AIRFLOW-1976] Fix for missing log/logger attribute FileProcessHandler
+[AIRFLOW-1982] Fix Executor event log formatting
+[AIRFLOW-1971] Propagate hive config on impersonation
+[AIRFLOW-1969] Always use HTTPS URIs for Google OAuth2
+[AIRFLOW-1954] Add DataFlowTemplateOperator
+[AIRFLOW-1963] Add config for HiveOperator mapred_queue
+[AIRFLOW-1946][AIRFLOW-1855] Create a BigQuery Get Data Operator
+[AIRFLOW-1953] Add labels to dataflow operators
+[AIRFLOW-1967] Update Celery to 4.0.2
+[AIRFLOW-1964] Add Upsight to list of Airflow users
+[AIRFLOW-XXX] Changelog for 1.9.0
+[AIRFLOW-1470] Implement BashSensor operator
+[AIRFLOW-XXX] Pin sqlalchemy dependency
+[AIRFLOW-1955] Do not reference unassigned variable
+[AIRFLOW-1957] Add contributor to BalanceHero in Readme
+[AIRFLOW-1517] Restore authorship of secrets and init container
+[AIRFLOW-1517] Remove authorship of secrets and init container
+[AIRFLOW-1935] Add BalanceHero to readme
+[AIRFLOW-1939] add astronomer contributors
+[AIRFLOW-1517] Kubernetes Operator
+[AIRFLOW-1928] Fix @once with catchup=False
+[AIRFLOW-1937] Speed up scheduling by committing in batch
+[AIRFLOW-1821] Enhance default logging config by removing extra loggers
+[AIRFLOW-1904] Correct DAG fileloc to the right filepath
+[AIRFLOW-1909] Update docs with supported versions of MySQL server
+[AIRFLOW-1915] Relax flask-wtf dependency specification
+[AIRFLOW-1920] Update CONTRIBUTING.md to reflect enforced linting rules
+[AIRFLOW-1942] Update Sphinx docs to remove deprecated import structure
+[AIRFLOW-1846][AIRFLOW-1697] Hide Ad Hoc Query behind secure_mode config
+[AIRFLOW-1948] Include details for on_kill failure
+[AIRFLOW-1938] Clean up unused exception
+[AIRFLOW-1932] Add GCP Pub/Sub Pull and Ack
+[AIRFLOW-XXX] Purge coveralls
+[AIRFLOW-XXX] Remove unused coveralls token
+[AIRFLOW-1938] Remove tag version check in setup.py
+[AIRFLOW-1916] Don't upload logs to remote from `run --raw`
+[AIRFLOW-XXX] Fix failing PubSub tests on Python3
+[AIRFLOW-XXX] Upgrade to python 3.5 and disable dask tests
+[AIRFLOW-1913] Add new GCP PubSub operators
+[AIRFLOW-1525] Fix minor LICENSE and NOTICE issues
+[AIRFLOW-1687] fix fernet error without encryption
+[AIRFLOW-1912] airflow.processor should not propagate logging
+[AIRFLOW-1911] Rename celeryd_concurrency
+[AIRFLOW-1885] Fix IndexError in ready_prefix_on_cmdline
+[AIRFLOW-1854] Improve Spark Submit operator for standalone cluster mode
+[AIRFLOW-1908] Fix celery broker options config load
+[AIRFLOW-1907] Pass max_ingestion_time to Druid hook
+[AIRFLOW-1909] Add away to list of users
+[AIRFLOW-1893][AIRFLOW-1901] Propagate PYTHONPATH when using impersonation
+[AIRFLOW-1892] Modify BQ hook to extract data filtered by column
+[AIRFLOW-1829] Support for schema updates in query jobs
+[AIRFLOW-1840] Make celery configuration congruent with Celery 4
+[AIRFLOW-1878] Fix stderr/stdout redirection for tasks
+[AIRFLOW-1897][AIRFLOW-1873] Task Logs for running instance not visible in WebUI
+[AIRFLOW-1896] FIX bleach <> html5lib incompatibility
+[AIRFLOW-1884][AIRFLOW-1059] Reset orphaned task state for external dagruns
+[AIRFLOW-XXX] Fix typo in comment
+[AIRFLOW-1869] Do not emit spurious warning on missing logs
+[AIRFLOW-1888] Add AWS Redshift Cluster Sensor
+[AIRFLOW-1887] Renamed endpoint url variable
+[AIRFLOW-1873] Set TI.try_number to right value depending TI state
+[AIRFLOW-1891] Fix non-ascii typo in default configuration template
+[AIRFLOW-1879] Handle ti log entirely within ti
+[AIRFLOW-1869] Write more error messages into gcs and file logs
+[AIRFLOW-1876] Write subtask id to task log header
+[AIRFLOW-1554] Fix wrong DagFileProcessor termination method call
+[AIRFLOW-342] Do not use amqp, rpc as result backend
+[AIRFLOW-966] Make celery broker_transport_options configurable
+[AIRFLOW-1881] Make operator log in task log
+[AIRFLOW-XXX] Added DataReply to the list of Airflow Users
+[AIRFLOW-1883] Get File Size for objects in Google Cloud Storage
+[AIRFLOW-1872] Set context for all handlers including parents
+[AIRFLOW-1855][AIRFLOW-1866] Add GCS Copy Operator to copy multiple files
+[AIRFLOW-1870] Enable flake8 tests
+[AIRFLOW-1785] Enable Python 3 tests
+[AIRFLOW-1850] Copy cmd before masking
+[AIRFLOW-1665] Reconnect on database errors
+[AIRFLOW-1559] Dispose SQLAlchemy engines on exit
+[AIRFLOW-1559] Close file handles in subprocesses
+[AIRFLOW-1559] Make database pooling optional
+[AIRFLOW-1848][Airflow-1848] Fix DataFlowPythonOperator py_file extension doc comment
+[AIRFLOW-1843] Add Google Cloud Storage Sensor with prefix
+[AIRFLOW-1803] Time zone documentation
+[AIRFLOW-1826] Update views to use timezone aware objects
+[AIRFLOW-1827] Fix api endpoint date parsing
+[AIRFLOW-1806] Use naive datetime when using cron
+[AIRFLOW-1809] Update tests to use timezone aware objects
+[AIRFLOW-1806] Use naive datetime for cron scheduling
+[AIRFLOW-1807] Force use of time zone aware db fields
+[AIRFLOW-1808] Convert all utcnow() to time zone aware
+[AIRFLOW-1804] Add time zone configuration options
+[AIRFLOW-1802] Convert database fields to timezone aware
+[AIRFLOW-XXX] Add dask lock files to excludes
+[AIRFLOW-1790] Add support for AWS Batch operator
+[AIRFLOW-XXX] Update README.md
+[AIRFLOW-1820] Remove timestamp from metric name
+[AIRFLOW-1810] Remove unused mysql import in migrations.
+[AIRFLOW-1838] Properly log collect_dags exception
+[AIRFLOW-1842] Fixed Super class name for the gcs to gcs copy operator
+[AIRFLOW-1845] Modal background now covers long or tall pages
+[AIRFLOW-1229] Add link to Run Id, incl execution_date
+[AIRFLOW-1842] Add gcs to gcs copy operator with renaming if required
+[AIRFLOW-1841] change False to None in operator and hook
+[AIRFLOW-1839] Fix more bugs in S3Hook boto -> boto3 migration
+[AIRFLOW-1830] Support multiple domains in Google authentication backend
+[AIRFLOW-1831] Add driver-classpath spark submit
+[AIRFLOW-1795] Correctly call S3Hook after migration to boto3
+[AIRFLOW-1811] Fix render Druid operator
+[AIRFLOW-1819] Fix slack operator unittest bug
+[AIRFLOW-1805] Allow Slack token to be passed through connection
+[AIRFLOW-1816] Add region param to Dataproc operators
+[AIRFLOW-868] Add postgres_to_gcs operator and unittests
+[AIRFLOW-1613] make mysql_to_gcs_operator py3 compatible
+[AIRFLOW-1817] use boto3 for s3 dependency
+[AIRFLOW-1813] Bug SSH Operator empty buffer
+[AIRFLOW-1801][AIRFLOW-288] Url encode execution dates
+[AIRFLOW-1563] Catch OSError while symlinking the latest log directory
+[AIRFLOW-1794] Remove uses of Exception.message for Python 3
+[AIRFLOW-1799] Fix logging line which raises errors
+[AIRFLOW-1102] Upgrade Gunicorn >=19.4.0
+[AIRFLOW-1756] Fix S3TaskHandler to work with Boto3-based S3Hook
+[AIRFLOW-1797] S3Hook.load_string didn't work on Python3
+[AIRFLOW-646] Add docutils to setup_requires
+[AIRFLOW-1792] Missing intervals DruidOperator
+[AIRFLOW-1789][AIRFLOW-1712] Log SSHOperator stderr to log.warning
+[AIRFLOW-1787] Fix task instance batch clear and set state bugs
+[AIRFLOW-1780] Fix long output lines with unicode from hanging parent
+[AIRFLOW-387] Close SQLAlchemy sessions properly
+[AIRFLOW-1779] Add keepalive packets to ssh hook
+[AIRFLOW-1669] Fix Docker and pin Moto to 1.1.19
+[AIRFLOW-71] Add support for private Docker images
+[AIRFLOW-XXX] Give a clue what the 'ds' variable is
+[AIRFLOW-XXX] Correct typos in the faq docs page
+[AIRFLOW-1571] Add AWS Lambda Hook
+[AIRFLOW-1675] Fix docstrings for API docs
+[AIRFLOW-1712][AIRFLOW-756][AIRFLOW-751] Log SSHOperator output
+[AIRFLOW-1776] Capture stdout and stderr for logging
+[AIRFLOW-1765] Make experimental API securable without needing Kerberos.
+[AIRFLOW-1764] The web interface should not use the experimental API
+[AIRFLOW-1771] Rename heartbeat to avoid confusion
+[AIRFLOW-1769] Add support for templates in VirtualenvOperator
+[AIRFLOW-1763] Fix S3TaskHandler unit tests
+[AIRFLOW-1315] Add Qubole File & Partition Sensors
+[AIRFLOW-1018] Make processor use logging framework
+[AIRFLOW-1695] Add RedshiftHook using boto3
+[AIRFLOW-1706] Fix query error for MSSQL backend
+[AIRFLOW-1711] Use ldap3 dict for group membership
+[AIRFLOW-1723] Make sendgrid a plugin
+[AIRFLOW-1757] Add missing options to SparkSubmitOperator
+[AIRFLOW-1734][Airflow 1734] Sqoop hook/operator enhancements
+[AIRFLOW-1761] Fix type in scheduler.rst
+[AIRFLOW-1731] Set pythonpath for logging
+[AIRFLOW-1641] Handle executor events in the scheduler
+[AIRFLOW-1744] Make sure max_tries can be set
+[AIRFLOW-1732] Improve dataflow hook logging
+[AIRFLOW-1736] Add HotelQuickly to Who Uses Airflow
+[AIRFLOW-1657] Handle failing qubole operator
+[AIRFLOW-1677] Fix typo in example_qubole_operator
+[AIRFLOW-926] Fix JDBC Hook
+[AIRFLOW-1520] Boto3 S3Hook, S3Log
+[AIRFLOW-1716] Fix multiple __init__ def in SimpleDag
+[AIRFLOW-XXX] Fix DateTime in Tree View
+[AIRFLOW-1719] Fix small typo
+[AIRFLOW-1432] Charts label for Y axis not visible
+[AIRFLOW-1743] Verify ldap filters correctly
+[AIRFLOW-1745] Restore default signal disposition
+[AIRFLOW-1741] Correctly hide second chart on task duration page
+[AIRFLOW-1728] Add networkUri, subnet, tags to Dataproc operator
+[AIRFLOW-1726] Add copy_expert psycopg2 method to PostgresHook
+[AIRFLOW-1330] Add conn_type argument to CLI when adding connection
+[AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd
+[AIRFLOW-1694] Stop using itertools.izip
+[AIRFLOW-1692] Change test_views filename to support Windows
+[AIRFLOW-1722] Fix typo in scheduler autorestart output filename
+[AIRFLOW-1723] Support sendgrid in email backend
+[AIRFLOW-1718] Set num_retries on Dataproc job request execution
+[AIRFLOW-1727] Add unit tests for DataProcHook
+[AIRFLOW-1631] Fix timing issue in unit test
+[AIRFLOW-1631] Fix local executor unbound parallelism
+[AIRFLOW-1724] Add Fundera to Who uses Airflow?
+[AIRFLOW-1683] Cancel BigQuery job on timeout.
+[AIRFLOW-1714] Fix misspelling: s/seperate/separate/
+[AIRFLOW-1681] Add batch clear in task instance view
+[AIRFLOW-1696] Fix dataproc version label error
+[AIRFLOW-1613] Handle binary field in MySqlToGoogleCloudStorageOperator
+[AIRFLOW-1697] Mode to disable charts endpoint
+[AIRFLOW-1691] Add better Google cloud logging documentation
+[AIRFLOW-1690] Add detail to gcs error messages
+[AIRFLOW-1682] Make S3TaskHandler write to S3 on close
+[AIRFLOW-1634] Adds task_concurrency feature
+[AIRFLOW-1676] Make GCSTaskHandler write to GCS on close
+[AIRFLOW-1678] Fix erroneously repeated word in function docstrings
+[AIRFLOW-1323] Made Dataproc operator parameter names consistent
+[AIRFLOW-1590] fix unused module and variable
+[AIRFLOW-1671] Add @apply_defaults back to gcs download operator
+[AIRFLOW-988] Fix repeating SLA miss callbacks
+[AIRFLOW-1611] Customize logging
+[AIRFLOW-1668] Expose keepalives_idle for Postgres connections
+[AIRFLOW-1658] Kill Druid task on timeout
+[AIRFLOW-1669][AIRFLOW-1368] Fix Docker import
+[AIRFLOW-891] Make webserver clock include date
+[AIRFLOW-1560] Add AWS DynamoDB hook and operator for inserting batch items
+[AIRFLOW-1654] Show tooltips for link icons in DAGs view
+[AIRFLOW-1660] Change webpage width to full-width
+[AIRFLOW-1664] write file as binary instead of str
+[AIRFLOW-1659] Fix invalid obj attribute bug in file_task_handler.py
+[AIRFLOW-1635] Allow creating GCP connection without requiring a JSON file
+[AIRFLOW-1650] Fix custom celery config loading
+[AIRFLOW-1647] Fix Spark-sql hook
+[AIRFLOW-1587] Fix CeleryExecutor import error
+[Airflow-1640][AIRFLOW-1640] Add qubole default connection
+[AIRFLOW-1576] Added region param to Dataproc{*}Operators
+[AIRFLOW-1643] Add healthjump to officially using list
+[AIRFLOW-1626] Add Azri Solutions to Airflow users
+[AIRFLOW-1636] Add AWS and EMR connection type
+[AIRFLOW-1527] Refactor celery config
+[AIRFLOW-1639] Fix Fernet error handling
+[AIRFLOW-1637] Fix Travis CI build status link
+[AIRFLOW-1628] Fix docstring of sqlsensor
+[AIRFLOW-1331] add SparkSubmitOperator option
+[AIRFLOW-1627] Only query pool in SubDAG init when necessary
+[AIRFLOW-1629] Make extra a textarea in edit connections form
+[AIRFLOW-1368] Automatically remove Docker container on exit
+[AIRFLOW-289] Make airflow timezone independent
+[AIRFLOW-1356] Add `--celery_hostname` to `airflow worker`
+[AIRFLOW-1247] Fix ignore_all_dependencies argument ignored
+[AIRFLOW-1621] Add tests for server side paging
+[AIRFLOW-1591] Avoid attribute error when rendering logging filename
+[AIRFLOW-1031] Replace hard-code to DagRun.ID_PREFIX
+[AIRFLOW-1604] Rename logger to log
+[AIRFLOW-1512] Add PythonVirtualenvOperator
+[AIRFLOW-1617] Fix XSS vulnerability in Variable endpoint
+[AIRFLOW-1497] Reset hidden fields when changing connection type
+[AIRFLOW-1619] Add poll_sleep parameter to GCP dataflow operator
+[AIRFLOW-XXX] Remove landscape.io config
+[AIRFLOW-XXX] Remove non working service badges
+[AIRFLOW-1177] Fix Variable.setdefault w/existing JSON
+[AIRFLOW-1600] Fix exception handling in get_fernet
+[AIRFLOW-1614] Replace inspect.stack() with sys._getframe()
+[AIRFLOW-1519] Add server side paging in DAGs list
+[AIRFLOW-1309] Allow hive_to_druid to take tblproperties
+[AIRFLOW-1613] Make MySqlToGoogleCloudStorageOperator compaitible with python3
+[AIRFLOW-1603] add PAYMILL to companies list
+[AIRFLOW-1609] Fix gitignore to ignore all venvs
+[AIRFLOW-1601] Add configurable task cleanup time
+
AIRFLOW 1.9.0, 2018-01-02
-------------------------
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index e45060bfe7..f114c66585 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -3,7 +3,6 @@
Contributions are welcome and are greatly appreciated! Every
little bit helps, and credit will always be given.
-
# Table of Contents
* [TOC](#table-of-contents)
* [Types of Contributions](#types-of-contributions)
@@ -15,11 +14,10 @@ little bit helps, and credit will always be given.
* [Documentation](#documentation)
* [Development and Testing](#development-and-testing)
- [Setting up a development environment](#setting-up-a-development-environment)
- - [Pull requests guidelines](#pull-request-guidelines)
- - [Testing Locally](#testing-locally)
+ - [Running unit tests](#running-unit-tests)
+ * [Pull requests guidelines](#pull-request-guidelines)
* [Changing the Metadata Database](#changing-the-metadata-database)
-
## Types of Contributions
### Report Bugs
@@ -55,157 +53,115 @@ The best way to send feedback is to open an issue on [Apache Jira](https://issue
If you are proposing a feature:
-- Explain in detail how it would work.
-- Keep the scope as narrow as possible, to make it easier to
- implement.
-- Remember that this is a volunteer-driven project, and that
- contributions are welcome :)
+- Explain in detail how it would work.
+- Keep the scope as narrow as possible, to make it easier to implement.
+- Remember that this is a volunteer-driven project, and that contributions are welcome :)
## Documentation
The latest API documentation is usually available
[here](https://airflow.incubator.apache.org/). To generate a local version,
-you need to have set up an Airflow development environemnt (see below). Also
+you need to have set up an Airflow development environment (see below). Also
install the `doc` extra.
- pip install -e .[doc]
+```
+pip install -e .[doc]
+```
-Generate the documentation by running:
+Generate and serve the documentation by running:
- cd docs && ./build.sh
+```
+cd docs
+./build.sh
+./start_doc_server.sh
+```
Only a subset of the API reference documentation builds. Install additional
extras to build the full API reference.
## Development and Testing
-### Set up a development env using Docker
+### Set up a development environment
-Go to your Airflow directory and start a new docker container. You can choose between Python 2 or 3, whatever you prefer.
+There are three ways to setup an Apache Airflow development environment.
-```
-# Start docker in your Airflow directory
-docker run -t -i -v `pwd`:/airflow/ python:2 bash
+1. Using tools and libraries installed directly on your system.
-# Go to the Airflow directory
-cd /airflow/
+ Install Python (2.7.x or 3.4.x), MySQL, and libxml by using system-level package
+ managers like yum, apt-get for Linux, or Homebrew for Mac OS at first. Refer to the [base CI Dockerfile](https://github.com/apache/incubator-airflow-ci/blob/master/Dockerfile) for
+ a comprehensive list of required packages.
-# Install Airflow with all the required dependencies,
-# including the devel which will provide the development tools
-pip install -e ".[hdfs,hive,druid,devel]"
+ Then install python development requirements. It is usually best to work in a virtualenv:
-# Init the database
-airflow initdb
+ ```bash
+ cd $AIRFLOW_HOME
+ virtualenv env
+ source env/bin/activate
+ pip install -e .[devel]
+ ```
-nosetests -v tests/hooks/test_druid_hook.py
+2. Using a Docker container
- test_get_first_record (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok
- test_get_records (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok
- test_get_uri (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok
- test_get_conn_url (tests.hooks.test_druid_hook.TestDruidHook) ... ok
- test_submit_gone_wrong (tests.hooks.test_druid_hook.TestDruidHook) ... ok
- test_submit_ok (tests.hooks.test_druid_hook.TestDruidHook) ... ok
- test_submit_timeout (tests.hooks.test_druid_hook.TestDruidHook) ... ok
- test_submit_unknown_response (tests.hooks.test_druid_hook.TestDruidHook) ... ok
+ Go to your Airflow directory and start a new docker container. You can choose between Python 2 or 3, whatever you prefer.
- ----------------------------------------------------------------------
- Ran 8 tests in 3.036s
+ ```
+ # Start docker in your Airflow directory
+ docker run -t -i -v `pwd`:/airflow/ -w /airflow/ -e SLUGIFY_USES_TEXT_UNIDECODE=yes python:2 bash
- OK
-```
+ # Go to the Airflow directory
+ cd /airflow/
-The Airflow code is mounted inside of the Docker container, so if you change something using your favorite IDE, you can directly test is in the container.
+ # Install Airflow with all the required dependencies,
+ # including the devel which will provide the development tools
+ pip install -e ".[hdfs,hive,druid,devel]"
-### Set up a development env using Virtualenv
+ # Init the database
+ airflow initdb
-Please install python(2.7.x or 3.4.x), mysql, and libxml by using system-level package
-managers like yum, apt-get for Linux, or homebrew for Mac OS at first.
-It is usually best to work in a virtualenv and tox. Install development requirements:
+ nosetests -v tests/hooks/test_druid_hook.py
- cd $AIRFLOW_HOME
- virtualenv env
- source env/bin/activate
- pip install -e .[devel]
- tox
+ test_get_first_record (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok
+ test_get_records (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok
+ test_get_uri (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok
+ test_get_conn_url (tests.hooks.test_druid_hook.TestDruidHook) ... ok
+ test_submit_gone_wrong (tests.hooks.test_druid_hook.TestDruidHook) ... ok
+ test_submit_ok (tests.hooks.test_druid_hook.TestDruidHook) ... ok
+ test_submit_timeout (tests.hooks.test_druid_hook.TestDruidHook) ... ok
+ test_submit_unknown_response (tests.hooks.test_druid_hook.TestDruidHook) ... ok
-Feel free to customize based on the extras available in [setup.py](./setup.py)
+ ----------------------------------------------------------------------
+ Ran 8 tests in 3.036s
-### Pull Request Guidelines
+ OK
+ ```
-Before you submit a pull request from your forked repo, check that it
-meets these guidelines:
+ The Airflow code is mounted inside of the Docker container, so if you change something using your favorite IDE, you can directly test is in the container.
-1. The pull request should include tests, either as doctests, unit tests, or
-both. The airflow repo uses [Travis CI](https://travis-ci.org/apache/incubator-airflow)
-to run the tests and [codecov](https://codecov.io/gh/apache/incubator-airflow)
-to track coverage. You can set up both for free on your fork. It will
-help you making sure you do not break the build with your PR and that you help
-increase coverage.
-2. Please [rebase your fork](http://stackoverflow.com/a/7244456/1110993),
-squash commits, and resolve all conflicts.
-3. Every pull request should have an associated
-[JIRA](https://issues.apache.org/jira/browse/AIRFLOW/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
-The JIRA link should also be contained in the PR description.
-4. Preface your commit's subject & PR's title with **[AIRFLOW-XXX]**
-where *XXX* is the JIRA number. We compose release notes (i.e. for Airflow releases) from all commit titles in a release.
-By placing the JIRA number in the commit title and hence in the release notes,
-Airflow users can look into JIRA and Github PRs for more details about a particular change.
-5. Add an [Apache License](http://www.apache.org/legal/src-headers.html)
- header to all new files
-6. If the pull request adds functionality, the docs should be updated as part
-of the same PR. Doc string are often sufficient. Make sure to follow the
-Sphinx compatible standards.
-7. The pull request should work for Python 2.7 and 3.4. If you need help
-writing code that works in both Python 2 and 3, see the documentation at the
-[Python-Future project](http://python-future.org) (the future package is an
-Airflow requirement and should be used where possible).
-8. As Airflow grows as a project, we try to enforce a more consistent
-style and try to follow the Python community guidelines. We track this
-using [landscape.io](https://landscape.io/github/apache/incubator-airflow/),
-which you can setup on your fork as well to check before you submit your
-PR. We currently enforce most [PEP8](https://www.python.org/dev/peps/pep-0008/)
-and a few other linting rules. It is usually a good idea to lint locally
-as well using [flake8](https://flake8.readthedocs.org/en/latest/)
-using `flake8 airflow tests`. `git diff upstream/master -u -- "*.py" | flake8 --diff` will return any changed files in your branch that require linting.
-9. Please read this excellent [article](http://chris.beams.io/posts/git-commit/) on
-commit messages and adhere to them. It makes the lives of those who
-come after you a lot easier.
-
-### Testing locally
-
-#### TL;DR
-Tests can then be run with (see also the [Running unit tests](#running-unit-tests) section below):
-
- ./run_unit_tests.sh
-
-Individual test files can be run with:
-
- nosetests [path to file]
-
-#### Running unit tests
-
-We *highly* recommend setting up [Travis CI](https://travis-ci.org/) on
-your repo to automate this. It is free for open source projects. If for
-some reason you cannot, you can use the steps below to run tests.
-
-Here are loose guidelines on how to get your environment to run the unit tests.
-We do understand that no one out there can run the full test suite since
-Airflow is meant to connect to virtually any external system and that you most
-likely have only a subset of these in your environment. You should run the
-CoreTests and tests related to things you touched in your PR.
-
-To set up a unit test environment, first take a look at `run_unit_tests.sh` and
-understand that your ``AIRFLOW_CONFIG`` points to an alternate config file
-while running the tests. You shouldn't have to alter this config file but
-you may if need be.
-
-From that point, you can actually export these same environment variables in
-your shell, start an Airflow webserver ``airflow webserver -d`` and go and
-configure your connection. Default connections that are used in the tests
-should already have been created, you just need to point them to the systems
-where you want your tests to run.
-
-Once your unit test environment is setup, you should be able to simply run
+3. Using [Docker Compose](https://docs.docker.com/compose/) and Airflow's CI scripts.
+
+ Start a docker container through Compose for development to avoid installing the packages directly on your system. The following will give you a shell inside a container, run all required service containers (MySQL, PostgresSQL, krb5 and so on) and install all the dependencies:
+
+ ```bash
+ docker-compose -f scripts/ci/docker-compose.yml run airflow-testing bash
+ # From the container
+ pip install -e .[devel]
+ # Run all the tests with python and mysql through tox
+ pip install tox
+ tox -e py35-backend_mysql
+ ```
+
+ If you wish to run individual tests inside of docker enviroment you can do as follows:
+
+ ```bash
+ # From the container (with your desired enviroment) with druid hook
+ tox -e py35-backend_mysql -- tests/hooks/test_druid_hook.py
+ ```
+
+
+### Running unit tests
+
+To run tests locally, once your unit test environment is setup (directly on your
+system or through our Docker setup) you should be able to simply run
``./run_unit_tests.sh`` at will.
For example, in order to just execute the "core" unit tests, run the following:
@@ -220,11 +176,104 @@ or a single test method:
./run_unit_tests.sh tests.core:CoreTest.test_check_operators -s --logging-level=DEBUG
```
+To run the whole test suite with Docker Compose, do:
+
+```
+# Install Docker Compose first, then this will run the tests
+docker-compose -f scripts/ci/docker-compose.yml run airflow-testing /app/scripts/ci/run-ci.sh
+```
+
+Alternatively can also set up [Travis CI](https://travis-ci.org/) on your repo to automate this.
+It is free for open source projects.
+
For more information on how to run a subset of the tests, take a look at the
nosetests docs.
See also the list of test classes and methods in `tests/core.py`.
+Feel free to customize based on the extras available in [setup.py](./setup.py)
+
+## Pull Request Guidelines
+
+Before you submit a pull request from your forked repo, check that it
+meets these guidelines:
+
+1. The pull request should include tests, either as doctests, unit tests, or both. The airflow repo uses [Travis CI](https://travis-ci.org/apache/incubator-airflow) to run the tests and [codecov](https://codecov.io/gh/apache/incubator-airflow) to track coverage. You can set up both for free on your fork (see the "Testing on Travis CI" section below). It will help you making sure you do not break the build with your PR and that you help increase coverage.
+1. Please [rebase your fork](http://stackoverflow.com/a/7244456/1110993), squash commits, and resolve all conflicts.
+1. Every pull request should have an associated [JIRA](https://issues.apache.org/jira/browse/AIRFLOW/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). The JIRA link should also be contained in the PR description.
+1. Preface your commit's subject & PR's title with **[AIRFLOW-XXX]** where *XXX* is the JIRA number. We compose release notes (i.e. for Airflow releases) from all commit titles in a release. By placing the JIRA number in the commit title and hence in the release notes, Airflow users can look into JIRA and Github PRs for more details about a particular change.
+1. Add an [Apache License](http://www.apache.org/legal/src-headers.html) header to all new files
+1. If the pull request adds functionality, the docs should be updated as part of the same PR. Doc string are often sufficient. Make sure to follow the Sphinx compatible standards.
+1. The pull request should work for Python 2.7 and 3.4. If you need help writing code that works in both Python 2 and 3, see the documentation at the [Python-Future project](http://python-future.org) (the future package is an Airflow requirement and should be used where possible).
+1. As Airflow grows as a project, we try to enforce a more consistent style and try to follow the Python community guidelines. We track this using [landscape.io](https://landscape.io/github/apache/incubator-airflow/), which you can setup on your fork as well to check before you submit your PR. We currently enforce most [PEP8](https://www.python.org/dev/peps/pep-0008/) and a few other linting rules. It is usually a good idea to lint locally as well using [flake8](https://flake8.readthedocs.org/en/latest/) using `flake8 airflow tests`. `git diff upstream/master -u -- "*.py" | flake8 --diff` will return any changed files in your branch that require linting.
+1. Please read this excellent [article](http://chris.beams.io/posts/git-commit/) on commit messages and adhere to them. It makes the lives of those who come after you a lot easier.
+
+### Testing on Travis CI
+
+We currently rely heavily on Travis CI for running the full Airflow test suite
+as running all of the tests locally requires significant setup. You can setup
+Travis CI in your fork of Airflow by following the
+[Travis CI Getting Started guide][travis-ci-getting-started].
+
+There are two different options available for running Travis CI which are
+setup as separate components on GitHub:
+
+1. **Travis CI GitHub App** (new version)
+1. **Travis CI GitHub Services** (legacy version)
+
+#### Travis CI GitHub App (new version)
+
+1. Once installed, you can configure the Travis CI GitHub App at
+https://github.com/settings/installations/169040.
+
+1. For the Travis CI GitHub App, you can set repository access to either "all
+repositories" for convenience, or "only select repositories" and choose
+`<username>/incubator-airflow` in the dropdown.
+
+1. You can access Travis CI for your fork at
+`https://travis-ci.com/<username>/incubator-airflow`.
+
+#### Travis CI GitHub Services (legacy version)
+
+The Travis CI GitHub Services versions uses an Authorized OAuth App. Note
+that `apache/incubator-airflow` is currently still using the legacy version.
+
+1. Once installed, you can configure the Travis CI Authorized OAuth App at
+https://github.com/settings/connections/applications/88c5b97de2dbfc50f3ac.
+
+1. If you are a GitHub admin, click the "Grant" button next to your
+organization; otherwise, click the "Request" button.
+
+1. For the Travis CI Authorized OAuth App, you may have to grant access to the
+forked `<organization>/incubator-airflow` repo even though it is public.
+
+1. You can access Travis CI for your fork at
+`https://travis-ci.org/<organization>/incubator-airflow`.
+
+#### Prefer travis-ci.com over travis-ci.org
+
+The travis-ci.org site for open source projects is now legacy and new projects
+should instead be created on travis-ci.com for both private repos and open
+source.
+
+Note that there is a second Authorized OAuth App available called "Travis CI
+for Open Source" used for the
+[legacy travis-ci.org service][travis-ci-org-vs-com]. It should not be used
+for new projects.
+
+More information:
+
+- [Open Source on travis-ci.com][travis-ci-open-source]
+- [Legacy GitHub Services to GitHub Apps Migration Guide][travis-ci-migrating]
+- [Migrating Multiple Repositories to GitHub Apps Guide][travis-ci-migrating-2]
+
+[travis-ci-getting-started]: https://docs.travis-ci.com/user/getting-started/
+[travis-ci-migrating-2]: https://docs.travis-ci.com/user/travis-migrate-to-apps-gem-guide/
+[travis-ci-migrating]: https://docs.travis-ci.com/user/legacy-services-to-github-apps-migration-guide/
+[travis-ci-open-source]: https://docs.travis-ci.com/user/open-source-on-travis-ci-com/
+[travis-ci-org-vs-com]: https://devops.stackexchange.com/a/4305/8830
+
+
### Changing the Metadata Database
When developing features the need may arise to persist information to the the
@@ -242,3 +291,80 @@ $ alembic revision -m "add new field to db"
Generating
~/airflow/airflow/migrations/versions/12341123_add_new_field_to_db.py
```
+
+## Setting up the node / npm javascript environment (ONLY FOR www_rbac)
+
+`airflow/www_rbac/` contains all npm-managed, front end assets.
+Flask-Appbuilder itself comes bundled with jQuery and bootstrap.
+While these may be phased out over time, these packages are currently not
+managed with npm.
+
+### Node/npm versions
+
+Make sure you are using recent versions of node and npm. No problems have been found with node>=8.11.3 and npm>=6.1.3
+
+### Using npm to generate bundled files
+
+#### npm
+
+First, npm must be available in your environment. If it is not you can run the following commands
+(taken from [this source](https://gist.github.com/DanHerbert/9520689))
+
+```
+brew install node --without-npm
+echo prefix=~/.npm-packages >> ~/.npmrc
+curl -L https://www.npmjs.com/install.sh | sh
+```
+
+The final step is to add `~/.npm-packages/bin` to your `PATH` so commands you install globally are usable.
+Add something like this to your `.bashrc` file, then `source ~/.bashrc` to reflect the change.
+
+```
+export PATH="$HOME/.npm-packages/bin:$PATH"
+```
+
+#### npm packages
+
+To install third party libraries defined in `package.json`, run the
+following within the `airflow/www_rbac/` directory which will install them in a
+new `node_modules/` folder within `www_rbac/`.
+
+```bash
+# from the root of the repository, move to where our JS package.json lives
+cd airflow/www_rbac/
+# run npm install to fetch all the dependencies
+npm install
+```
+
+To parse and generate bundled files for airflow, run either of the
+following commands. The `dev` flag will keep the npm script running and
+re-run it upon any changes within the assets directory.
+
+```
+# Compiles the production / optimized js & css
+npm run prod
+
+# Start a web server that manages and updates your assets as you modify them
+npm run dev
+```
+
+#### Upgrading npm packages
+
+Should you add or upgrade a npm package, which involves changing `package.json`, you'll need to re-run `npm install`
+and push the newly generated `package-lock.json` file so we get the reproducible build.
+
+#### Javascript Style Guide
+
+We try to enforce a more consistent style and try to follow the JS community guidelines.
+Once you add or modify any javascript code in the project, please make sure it follows the guidelines
+defined in [Airbnb JavaScript Style Guide](https://github.com/airbnb/javascript).
+Apache Airflow uses [ESLint](https://eslint.org/) as a tool for identifying and reporting on patterns in JavaScript,
+which can be used by running any of the following commands.
+
+```bash
+# Check JS code in .js and .html files, and report any errors/warnings
+npm run lint
+
+# Check JS code in .js and .html files, report any errors/warnings and fix them if possible
+npm run lint:fix
+```
diff --git a/DISCLAIMER b/DISCLAIMER
index 8fe69887c3..2758508789 100644
--- a/DISCLAIMER
+++ b/DISCLAIMER
@@ -1 +1,6 @@
-Apache Airflow is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
+Apache Airflow is an effort undergoing incubation at The Apache Software Foundation (ASF),
+sponsored by the Apache Incubator. Incubation is required of all newly accepted projects
+until a further review indicates that the infrastructure, communications, and decision
+making process have stabilized in a manner consistent with other successful ASF projects.
+While incubation status is not necessarily a reflection of the completeness or stability
+of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
diff --git a/INSTALL b/INSTALL
index 5c8f03eb66..b018839ab1 100644
--- a/INSTALL
+++ b/INSTALL
@@ -1,13 +1,30 @@
-# INSTALL / BUILD instruction for Apache Airflow (incubating)
-# fetch the tarball and untar the source
+# INSTALL / BUILD instructions for Apache Airflow (incubating)
+
+# [required] fetch the tarball and untar the source
+# change into the directory that was untarred.
# [optional] run Apache RAT (release audit tool) to validate license headers
-# RAT docs here: https://creadur.apache.org/rat/
+# RAT docs here: https://creadur.apache.org/rat/. Requires Java and Apache Rat
java -jar apache-rat.jar -E ./.rat-excludes -d .
-# [optional] by default one of Apache Airflow's dependencies pulls in a GPL
-# library. If this is a concern issue (also every upgrade):
-# export SLUGIFY_USES_TEXT_UNIDECODE=yes
+# [optional] Airflow pulls in quite a lot of dependencies in order
+# to connect to other services. You might want to test or run Airflow
+# from a virtual env to make sure those dependencies are separated
+# from your system wide versions
+python -m my_env
+source my_env/bin/activate
+
+# [required] by default one of Apache Airflow's dependencies pulls in a GPL
+# library. Airflow will not install (and upgrade) without an explicit choice.
+#
+# To make sure not to install the GPL dependency:
+# export SLUGIFY_USES_TEXT_UNIDECODE=yes
+# In case you do not mind:
+# export AIRFLOW_GPL_UNIDECODE=yes
+
+# [required] building and installing
+# by pip (preferred)
+pip install .
-# install the release
+# or directly
python setup.py install
diff --git a/LICENSE b/LICENSE
index 405540c64f..68ca6b6117 100644
--- a/LICENSE
+++ b/LICENSE
@@ -209,6 +209,19 @@ limitations under the License.
licenses.
+========================================================================
+Third party Apache 2.0 licenses
+========================================================================
+
+The following components are provided under the Apache 2.0 License.
+See project link for details. The text of each license is also included
+at licenses/LICENSE-[project].txt.
+
+ (ALv2 License) hue (https://github.com/cloudera/hue/)
+ (ALv2 License) jqclock (https://github.com/JohnRDOrazio/jQuery-Clock-Plugin)
+ (ALv2 License) bootstrap3-typeahead (https://github.com/bassjobsen/Bootstrap-3-Typeahead)
+ (ALv2 License) airflow.contrib.auth.backends.github_enterprise_auth
+
========================================================================
MIT licenses
========================================================================
@@ -226,6 +239,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
(MIT License) Bootstrap Toggle (http://www.bootstraptoggle.com)
(MIT License) normalize.css (http://necolas.github.io/normalize.css/)
(MIT License) ElasticMock (https://github.com/vrcmarcos/elasticmock)
+ (MIT License) MomentJS (http://momentjs.com/)
========================================================================
BSD 2-Clause licenses
@@ -239,10 +253,11 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
========================================================================
BSD 3-Clause licenses
========================================================================
-The following components are provided under the BSD 2-Clause license.
-See file headers and project links for details.
+The following components are provided under the BSD 3-Clause license. See project links for details.
The text of each license is also included at licenses/LICENSE-[project].txt.
(BSD 3 License) Ace (https://github.com/ajaxorg/ace)
(BSD 3 License) d3js (https://d3js.org)
(BSD 3 License) parallel-coordinates (http://syntagmatic.github.com/parallel-coordinates/)
+ (BSD 3 License) scikit-learn (https://github.com/scikit-learn/scikit-learn)
+
diff --git a/MANIFEST.in b/MANIFEST.in
index c04b80617d..ec99c1f6b2 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -6,9 +6,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -21,8 +21,10 @@ include LICENSE
include DISCLAIMER
include CHANGELOG.txt
include README.md
+graft licenses/
graft airflow/www/templates
graft airflow/www/static
+graft airflow/www_rbac
graft airflow/www_rbac/static
graft airflow/www_rbac/templates
graft airflow/www_rbac/translations
@@ -30,3 +32,4 @@ include airflow/alembic.ini
graft scripts/systemd
graft scripts/upstart
graft airflow/config_templates
+recursive-exclude airflow/www_rbac/node_modules *
diff --git a/NOTICE b/NOTICE
index b1e78ad9c3..99069f7a40 100644
--- a/NOTICE
+++ b/NOTICE
@@ -1,6 +1,26 @@
Apache Airflow
-Copyright 2016 and onwards The Apache Software Foundation
-
+Copyright 2016-2018 The Apache Software Foundation
This product includes software developed at The Apache Software
Foundation (http://www.apache.org/).
+
+=======================================================================
+
+Apache Airflow contains subcomponents with separate copyright notices and
+license terms. Your use of the source code for the these subcomponents
+is subject to the terms and conditions of their respective licenses.
+
+See the LICENSE file for a list of subcomponents and dependencies and
+their respective licenses.
+
+airflow.contrib.auth.backends.github_enterprise_auth:
+-----------------------------------------------------
+
+* Copyright 2015 Matthew Pelland (matt@pelland.io)
+
+hue:
+-----
+This product contains a modified portion of 'Hue' developed by Cloudera, Inc.
+(https://github.com/cloudera/hue/).
+
+* Copyright 2009-2017 Cloudera Inc.
diff --git a/README.md b/README.md
index e68d26cc8c..64740ca27a 100644
--- a/README.md
+++ b/README.md
@@ -1,28 +1,24 @@
-# Airflow
+# Apache Airflow (Incubating)
[](https://badge.fury.io/py/apache-airflow)
[](https://travis-ci.org/apache/incubator-airflow)
[](https://codecov.io/github/apache/incubator-airflow?branch=master)
[](https://airflow.readthedocs.io/en/latest/?badge=latest)
[](http://www.apache.org/licenses/LICENSE-2.0.txt)
-[](https://gitter.im/apache/incubator-airflow?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[](https://pypi.org/project/apache-airflow/)
+[](https://twitter.com/ApacheAirflow)
-_NOTE: The transition from 1.8.0 (or before) to 1.8.1 (or after) requires uninstalling Airflow before installing the new version. The package name was changed from `airflow` to `apache-airflow` as of version 1.8.1._
+_NOTE: The transition from 1.8.0 (or before) to 1.8.1 (or after) requires uninstalling Apache Airflow before installing the new version. The package name was changed from `airflow` to `apache-airflow` as of version 1.8.1._
-Airflow is a platform to programmatically author, schedule, and monitor
-workflows.
+Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.
When workflows are defined as code, they become more maintainable,
versionable, testable, and collaborative.
-Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks.
-The Airflow scheduler executes your tasks on an array of workers while
-following the specified dependencies. Rich command line utilities make
-performing complex surgeries on DAGs a snap. The rich user interface
-makes it easy to visualize pipelines running in production,
-monitor progress, and troubleshoot issues when needed.
+Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.
## Getting started
+
Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.incubator.apache.org/installation.html), getting a [quick start](https://airflow.incubator.apache.org/start.html), or a more complete [tutorial](https://airflow.incubator.apache.org/tutorial.html).
Documentation of GitHub master (latest development branch): [ReadTheDocs Documentation](https://airflow.readthedocs.io/en/latest/)
@@ -49,40 +45,51 @@ unit of work and continuity.
- **Dynamic**: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.
- **Extensible**: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
- **Elegant**: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine.
-- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity.
+- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers.
## User Interface
- **DAGs**: Overview of all DAGs in your environment.
-
+
+ 
- **Tree View**: Tree representation of a DAG that spans across time.
-
+
+ 
- **Graph View**: Visualization of a DAG's dependencies and their current status for a specific run.
-
+
+ 
- **Task Duration**: Total time spent on different tasks over time.
-
+
+ 
- **Gantt View**: Duration and overlap of a DAG.
-
+
+ 
- **Code View**: Quick way to view source code of a DAG.
-
-## Who uses Airflow?
+ 
-As the Airflow community grows, we'd like to keep track of who is using
-the platform. Please send a PR with your company name and @githubhandle
-if you may.
-Committers:
+## Contributing
-* Refer to [Committers](https://cwiki.apache.org/confluence/display/AIRFLOW/Committers)
+Want to help build Apache Airflow? Check out our [contributing documentation](https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md).
+
+
+## Who uses Apache Airflow?
+
+As the Apache Airflow community grows, we'd like to keep track of who is using
+the platform. Please send a PR with your company name and @githubhandle
+if you may.
Currently **officially** using Airflow:
+1. [6play](https://www.6play.fr) [[@lemourA](https://github.com/lemoura), [@achaussende](https://github.com/achaussende), [@d-nguyen](https://github.com/d-nguyen), [@julien-gm](https://github.com/julien-gm)]
+1. [8fit](https://8fit.com/) [[@nicor88](https://github.com/nicor88), [@frnzska](https://github.com/frnzska)]
+1. [90 Seconds](https://90seconds.tv/) [[@aaronmak](https://github.com/aaronmak)]
1. [AdBOOST](https://www.adboost.sk) [[AdBOOST](https://github.com/AdBOOST)]
1. [Agari](https://github.com/agaridata) [[@r39132](https://github.com/r39132)]
1. [Airbnb](http://airbnb.io/) [[@mistercrunch](https://github.com/mistercrunch), [@artwr](https://github.com/artwr)]
@@ -92,6 +99,7 @@ Currently **officially** using Airflow:
1. [AltX](https://www.getaltx.com/about) [[@pedromduarte](https://github.com/pedromduarte)]
1. [Apigee](https://apigee.com) [[@btallman](https://github.com/btallman)]
1. [ARGO Labs](http://www.argolabs.org) [[California Data Collaborative](https://github.com/California-Data-Collaborative)]
+1. [Arquivei](https://www.arquivei.com.br/) [[@arquivei](https://github.com/arquivei)]
1. [Astronomer](http://www.astronomer.io) [[@schnie](https://github.com/schnie), [@andscoop](https://github.com/andscoop), [@tedmiston](https://github.com/tedmiston), [@benjamingregory](https://github.com/benjamingregory)]
1. [Auth0](https://auth0.com) [[@sicarul](https://github.com/sicarul)]
1. [Away](https://awaytravel.com) [[@trunsky](https://github.com/trunsky)]
@@ -99,6 +107,7 @@ Currently **officially** using Airflow:
1. [BalanceHero](http://truebalance.io/) [[@swalloow](https://github.com/swalloow)]
1. [Banco de Formaturas](https://www.bancodeformaturas.com.br) [[@guiligan](https://github.com/guiligan)]
1. [BandwidthX](http://www.bandwidthx.com) [[@dineshdsharma](https://github.com/dineshdsharma)]
+1. [BBM](https://www.bbm.com/)
1. [Bellhops](https://github.com/bellhops)
1. [BelugaDB](https://belugadb.com) [[@fabio-nukui](https://github.com/fabio-nukui) & [@joao-sallaberry](http://github.com/joao-sallaberry) & [@lucianoviola](https://github.com/lucianoviola) & [@tmatuki](https://github.com/tmatuki)]
1. [BlaBlaCar](https://www.blablacar.com) [[@puckel](https://github.com/puckel) & [@wmorin](https://github.com/wmorin)]
@@ -108,11 +117,16 @@ Currently **officially** using Airflow:
1. [Bluecore](https://www.bluecore.com) [[@JLDLaughlin](https://github.com/JLDLaughlin)]
1. [Boda Telecom Suite - CE](https://github.com/bodastage/bts-ce) [[@erssebaggala](https://github.com/erssebaggala), [@bodastage](https://github.com/bodastage)]
1. [Bodastage Solutions](http://bodastage.com) [[@erssebaggala](https://github.com/erssebaggala), [@bodastage](https://github.com/bodastage)]
+1. [Bombora Inc](https://bombora.com/) [[@jeffkpayne](https://github.com/jeffkpayne), [@TheOriginalAlex](https://github.com/TheOriginalAlex)]
1. [Bonnier Broadcasting](http://www.bonnierbroadcasting.com) [[@wileeam](https://github.com/wileeam)]
1. [BounceX](http://www.bouncex.com) [[@JoshFerge](https://github.com/JoshFerge), [@hudsonrio](https://github.com/hudsonrio), [@ronniekritou](https://github.com/ronniekritou)]
+1. [Branch](https://branch.io) [[@sdebarshi](https://github.com/sdebarshi), [@dmitrig01](https://github.com/dmitrig01)]
1. [California Data Collaborative](https://github.com/California-Data-Collaborative) powered by [ARGO Labs](http://www.argolabs.org)
1. [Carbonite](https://www.carbonite.com) [[@ajbosco](https://github.com/ajbosco)]
+1. [CarLabs](https://www.carlabs.ai/) [[@sganz](https://github.com/sganz) & [@odannyc](https://github.com/odannyc)]
+1. [CAVA](https://www.cava.com) [[@minh5](http://github.com/minh5)]
1. [Celect](http://www.celect.com) [[@superdosh](https://github.com/superdosh) & [@chadcelect](https://github.com/chadcelect)]
+1. [Censys](https://censys.io) [[@zakird](https://github.com/zakird), [@dadrian](https://github.com/dadrian), & [@andrewsardone](https://github.com/andrewsardone)]
1. [Change.org](https://www.change.org) [[@change](https://github.com/change), [@vijaykramesh](https://github.com/vijaykramesh)]
1. [Checkr](https://checkr.com) [[@tongboh](https://github.com/tongboh)]
1. [Children's Hospital of Philadelphia Division of Genomic Diagnostics](http://www.chop.edu/centers-programs/division-genomic-diagnostics) [[@genomics-geek]](https://github.com/genomics-geek/)
@@ -121,6 +135,8 @@ Currently **officially** using Airflow:
1. [Clairvoyant](https://clairvoyantsoft.com) [@shekharv](https://github.com/shekharv)
1. [Clover Health](https://www.cloverhealth.com) [[@gwax](https://github.com/gwax) & [@vansivallab](https://github.com/vansivallab)]
1. [Chartboost](https://www.chartboost.com) [[@cgelman](https://github.com/cgelman) & [@dclubb](https://github.com/dclubb)]
+1. [Collectivehealth Inc.](https://www.collectivehealth.com) [@retornam](https://github.com/retornam)
+1. [Compass](https://www.compass.com) [[@wdhorton](https://github.com/wdhorton)]
1. [ContaAzul](https://www.contaazul.com) [[@bern4rdelli](https://github.com/bern4rdelli), [@renanleme](https://github.com/renanleme) & [@sabino](https://github.com/sabino)]
1. [Cotap](https://github.com/cotap/) [[@maraca](https://github.com/maraca) & [@richardchew](https://github.com/richardchew)]
1. [Craig@Work](https://www.craigatwork.com)
@@ -128,32 +144,44 @@ Currently **officially** using Airflow:
1. [CreditCards.com](https://www.creditcards.com/)[[@vmAggies](https://github.com/vmAggies) & [@jay-wallaby](https://github.com/jay-wallaby)]
1. [Creditas](https://www.creditas.com.br) [[@dcassiano](https://github.com/dcassiano)]
1. [Custom Ink](https://www.customink.com/) [[@david-dalisay](https://github.com/david-dalisay), [@dmartin11](https://github.com/dmartin11) & [@mpeteuil](https://github.com/mpeteuil)]
+1. [Dailymotion](http://www.dailymotion.com/fr) [[@germaintanguy](https://github.com/germaintanguy) & [@hc](https://github.com/hc)]
+1. [Danamica](https://www.danamica.dk) [[@testvinder](https://github.com/testvinder)]
1. [Data Reply](https://www.datareply.co.uk/) [[@kaxil](https://github.com/kaxil)]
+1. [DataCamp](https://datacamp.com/) [[@dgrtwo](https://github.com/dgrtwo)]
1. [DataFox](https://www.datafox.com/) [[@sudowork](https://github.com/sudowork)]
1. [Digital First Media](http://www.digitalfirstmedia.com/) [[@duffn](https://github.com/duffn) & [@mschmo](https://github.com/mschmo) & [@seanmuth](https://github.com/seanmuth)]
1. [DocuTAP](https://www.docutap.com/) [[@jshvrsn](https://github.com/jshvrsn) & [@lhvphan](https://github.com/lhvphan) & [@cloneluke](https://github.com/cloneluke)]
+1. [DoorDash](https://www.doordash.com/)
1. [Dotmodus](http://dotmodus.com) [[@dannylee12](https://github.com/dannylee12)]
1. [Drivy](https://www.drivy.com) [[@AntoineAugusti](https://github.com/AntoineAugusti)]
1. [Easy Taxi](http://www.easytaxi.com/) [[@caique-lima](https://github.com/caique-lima) & [@WesleyBatista](https://github.com/WesleyBatista) & [@diraol](https://github.com/diraol)]
+1. [Enigma](https://www.enigma.com) [[@hydrosquall](https://github.com/hydrosquall)]
1. [eRevalue](https://www.datamaran.com) [[@hamedhsn](https://github.com/hamedhsn)]
1. [evo.company](https://evo.company/) [[@orhideous](https://github.com/orhideous)]
+1. [Fathom Health](https://www.fathomhealth.co/)
+1. [Flipp](https://www.flipp.com) [[@sethwilsonwishabi](https://github.com/sethwilsonwishabi)]
+1. [Format](https://www.format.com) [[@format](https://github.com/4ormat) & [@jasonicarter](https://github.com/jasonicarter)]
1. [FreshBooks](https://github.com/freshbooks) [[@DinoCow](https://github.com/DinoCow)]
1. [Fundera](https://fundera.com) [[@andyxhadji](https://github.com/andyxhadji)]
+1. [G Adventures](https://gadventures.com) [[@samuelmullin](https://github.com/samuelmullin)]
1. [GameWisp](https://gamewisp.com) [[@tjbiii](https://github.com/TJBIII) & [@theryanwalls](https://github.com/theryanwalls)]
1. [Gentner Lab](http://github.com/gentnerlab) [[@neuromusic](https://github.com/neuromusic)]
1. [Glassdoor](https://github.com/Glassdoor) [[@syvineckruyk](https://github.com/syvineckruyk)]
1. [Global Fashion Group](http://global-fashion-group.com) [[@GFG](https://github.com/GFG)]
1. [GovTech GDS](https://gds-gov.tech) [[@chrissng](https://github.com/chrissng) & [@datagovsg](https://github.com/datagovsg)]
+1. [Gradeup](https://gradeup.co) [[@gradeup](https://github.com/gradeup)]
1. [Grand Rounds](https://www.grandrounds.com/) [[@richddr](https://github.com/richddr), [@timz1290](https://github.com/timz1290), [@wenever](https://github.com/@wenever), & [@runongirlrunon](https://github.com/runongirlrunon)]
1. [Groupalia](http://es.groupalia.com) [[@jesusfcr](https://github.com/jesusfcr)]
1. [Groupon](https://groupon.com) [[@stevencasey](https://github.com/stevencasey)]
1. [Gusto](https://gusto.com) [[@frankhsu](https://github.com/frankhsu)]
1. [Handshake](https://joinhandshake.com/) [[@mhickman](https://github.com/mhickman)]
1. [Handy](http://www.handy.com/careers/73115?gh_jid=73115&gh_src=o5qcxn) [[@marcintustin](https://github.com/marcintustin) / [@mtustin-handy](https://github.com/mtustin-handy)]
+1. [happn](https://www.happn.com) [[@PierreCORBEL](https://github.com/PierreCORBEL)]
1. [HBC Digital](http://tech.hbc.com) [[@tmccartan](https://github.com/tmccartan) & [@dmateusp](https://github.com/dmateusp)]
1. [HBO](http://www.hbo.com/)[[@yiwang](https://github.com/yiwang)]
1. [Healthjump](http://www.healthjump.com/) [[@miscbits](https://github.com/miscbits)]
1. [HelloFresh](https://www.hellofresh.com) [[@tammymendt](https://github.com/tammymendt) & [@davidsbatista](https://github.com/davidsbatista) & [@iuriinedostup](https://github.com/iuriinedostup)]
+1. [Hipages](https://www.hipages.com.au/) [[@arihantsurana](https://github.com/arihantsurana)]
1. [Holimetrix](http://holimetrix.com/) [[@thibault-ketterer](https://github.com/thibault-ketterer)]
1. [Hootsuite](https://github.com/hootsuite)
1. [Hostnfly](https://www.hostnfly.com/) [[@CyrilLeMat](https://github.com/CyrilLeMat) & [@pierrechopin](https://github.com/pierrechopin) & [@alexisrosuel](https://github.com/alexisrosuel)]
@@ -165,14 +193,18 @@ Currently **officially** using Airflow:
1. [Intercom](http://www.intercom.com/) [[@fox](https://github.com/fox) & [@paulvic](https://github.com/paulvic)]
1. [Investorise](https://investorise.com/) [[@svenvarkel](https://github.com/svenvarkel)]
1. [Jampp](https://github.com/jampp)
+1. [Jeitto](https://www.jeitto.com.br) [[@BrennerPablo](https://github.com/BrennerPablo) & [@ds-mauri](https://github.com/ds-mauri)]
1. [JobTeaser](https://www.jobteaser.com) [[@stefani75](https://github.com/stefani75) & [@knil-sama](https://github.com/knil-sama)]
1. [Kalibrr](https://www.kalibrr.com/) [[@charlesverdad](https://github.com/charlesverdad)]
1. [Karmic](https://karmiclabs.com) [[@hyw](https://github.com/hyw)]
+1. [King](https://king.com) [[@nathadfield](https://github.com/nathadfield)]
1. [Kiwi.com](https://kiwi.com/) [[@underyx](https://github.com/underyx)]
1. [Kogan.com](https://github.com/kogan) [[@geeknam](https://github.com/geeknam)]
+1. [KPN B.V.](https://www.kpn.com/) [[@biyanisuraj](https://github.com/biyanisuraj) & [@gmic](https://github.com/gmic)]
1. [Lemann Foundation](http://fundacaolemann.org.br) [[@fernandosjp](https://github.com/fernandosjp)]
1. [LendUp](https://www.lendup.com/) [[@lendup](https://github.com/lendup)]
1. [LetsBonus](http://www.letsbonus.com) [[@jesusfcr](https://github.com/jesusfcr) & [@OpringaoDoTurno](https://github.com/OpringaoDoTurno)]
+1. [Liberty Global](https://www.libertyglobal.com/) [[@LibertyGlobal](https://github.com/LibertyGlobal/)]
1. [liligo](http://liligo.com/) [[@tromika](https://github.com/tromika)]
1. [LingoChamp](http://www.liulishuo.com/) [[@haitaoyao](https://github.com/haitaoyao)]
1. [Lucid](http://luc.id) [[@jbrownlucid](https://github.com/jbrownlucid) & [@kkourtchikov](https://github.com/kkourtchikov)]
@@ -185,6 +217,7 @@ Currently **officially** using Airflow:
1. [Mercari](http://www.mercari.com/) [[@yu-iskw](https://github.com/yu-iskw)]
1. [MFG Labs](https://github.com/MfgLabs)
1. [MiNODES](https://www.minodes.com) [[@dice89](https://github.com/dice89), [@diazcelsa](https://github.com/diazcelsa)]
+1. [Modernizing Medicine](https://www.modmed.com/)[[@kehv1n](https://github.com/kehv1n), [@dalupus](https://github.com/dalupus)]
1. [Multiply](https://www.multiply.com) [[@nrhvyc](https://github.com/nrhvyc)]
1. [mytaxi](https://mytaxi.com) [[@mytaxi](https://github.com/mytaxi)]
1. [Nerdwallet](https://www.nerdwallet.com)
@@ -196,6 +229,7 @@ Currently **officially** using Airflow:
1. [OneFineStay](https://www.onefinestay.com) [[@slangwald](https://github.com/slangwald)]
1. [Open Knowledge International](https://okfn.org) [@vitorbaptista](https://github.com/vitorbaptista)
1. [Overstock](https://www.github.com/overstock) [[@mhousley](https://github.com/mhousley) & [@mct0006](https://github.com/mct0006)]
+1. [Pagar.me](https://pagar.me/) [[@pagarme](https://github.com/pagarme)
1. [Pandora Media](https://www.pandora.com/) [[@Acehaidrey](https://github.com/Acehaidrey) & [@wolfier](https://github.com/wolfier)]
1. [PAYMILL](https://www.paymill.com/) [[@paymill](https://github.com/paymill) & [@matthiashuschle](https://github.com/matthiashuschle)]
1. [PayPal](https://www.paypal.com/) [[@r39132](https://github.com/r39132) & [@jhsenjaliya](https://github.com/jhsenjaliya)]
@@ -205,6 +239,7 @@ Currently **officially** using Airflow:
1. [PMC](https://pmc.com/) [[@andrewm4894](https://github.com/andrewm4894)]
1. [Postmates](http://www.postmates.com) [[@syeoryn](https://github.com/syeoryn)]
1. [Pronto Tools](http://www.prontotools.io/) [[@zkan](https://github.com/zkan) & [@mesodiar](https://github.com/mesodiar)]
+1. [Publicis Pixelpark](https://www.publicispixelpark.de/) [[@feluelle](https://github.com/feluelle)]
1. [PubNub](https://pubnub.com) [[@jzucker2](https://github.com/jzucker2)]
1. [Qplum](https://qplum.co) [[@manti](https://github.com/manti)]
1. [Quantopian](https://www.quantopian.com/) [[@eronarn](http://github.com/eronarn)]
@@ -225,20 +260,29 @@ Currently **officially** using Airflow:
1. [Spotify](https://github.com/spotify) [[@znichols](https://github.com/znichols)]
1. [Stackspace](https://beta.stackspace.io/)
1. [Stripe](https://stripe.com) [[@jbalogh](https://github.com/jbalogh)]
+1. [Strongmind](https://www.strongmind.com) [[@tomchapin](https://github.com/tomchapin) & [@wongstein](https://github.com/wongstein)]
+1. [Square](https://squareup.com/)
1. [Tails.com](https://tails.com/) [[@alanmcruickshank](https://github.com/alanmcruickshank)]
+1. [Tesla](https://www.tesla.com/) [[@thoralf-gutierrez](https://github.com/thoralf-gutierrez)]
+1. [The Home Depot](https://www.homedepot.com/)[[@apekshithr](https://github.com/apekshithr)]
+1. [THE ICONIC](https://www.theiconic.com.au/) [[@revathijay](https://github.com/revathijay)] [[@ilikedata](https://github.com/ilikedata)]
1. [Thinking Machines](https://thinkingmachin.es) [[@marksteve](https://github.com/marksteve)]
1. [Thinknear](https://www.thinknear.com/) [[@d3cay1](https://github.com/d3cay1), [@ccson](https://github.com/ccson), & [@ababian](https://github.com/ababian)]
+1. [ThoughtWorks](https://www.thoughtworks.com/) [[@sann3](https://github.com/sann3)]
1. [Thumbtack](https://www.thumbtack.com/) [[@natekupp](https://github.com/natekupp)]
1. [Tictail](https://tictail.com/)
1. [Tile](https://tile.com/) [[@ranjanmanish](https://github.com/ranjanmanish)]
1. [Tokopedia](https://www.tokopedia.com/) [@topedmaria](https://github.com/topedmaria)
1. [Twine Labs](https://www.twinelabs.com/) [[@ivorpeles](https://github.com/ivorpeles)]
+1. [Twitter](https://www.twitter.com/) [[@aoen](https://github.com/aoen)]
1. [T2 Systems](http://t2systems.com) [[@unclaimedpants](https://github.com/unclaimedpants)]
1. [Ubisoft](https://www.ubisoft.com/) [[@Walkoss](https://github.com/Walkoss)]
1. [United Airlines](https://www.united.com/) [[@ilopezfr](https://github.com/ilopezfr)]
1. [Upsight](https://www.upsight.com) [[@dhuang](https://github.com/dhuang)]
1. [Vente-Exclusive.com](http://www.vente-exclusive.com/) [[@alexvanboxel](https://github.com/alexvanboxel)]
1. [Vevo](https://www.vevo.com/) [[@csetiawan](https://github.com/csetiawan) & [@jerrygillespie](https://github.com/jerrygillespie)]
+1. [Vidio](https://www.vidio.com/)
+1. [Ville de Montréal](http://ville.montreal.qc.ca/)[@VilledeMontreal](https://github.com/VilledeMontreal/)]
1. [Vnomics](https://github.com/vnomics) [[@lpalum](https://github.com/lpalum)]
1. [WePay](http://www.wepay.com) [[@criccomini](https://github.com/criccomini) & [@mtagle](https://github.com/mtagle)]
1. [WeTransfer](https://github.com/WeTransfer) [[@jochem](https://github.com/jochem)]
@@ -256,10 +300,17 @@ Currently **officially** using Airflow:
1. [Zymergen](https://www.zymergen.com/)
1. [99](https://99taxis.com) [[@fbenevides](https://github.com/fbenevides), [@gustavoamigo](https://github.com/gustavoamigo) & [@mmmaia](https://github.com/mmmaia)]
-## Links
+## Who Maintains Apache Airflow?
+Airflow is the work of the [community](https://github.com/apache/incubator-airflow/graphs/contributors),
+but the [core committers/maintainers](https://people.apache.org/committers-by-project.html#airflow)
+are responsible for reviewing and merging PRs as well as steering conversation around new feature requests.
+If you would like to become a maintainer, please review the Apache Airflow
+[committer requirements](https://cwiki.apache.org/confluence/display/AIRFLOW/Committers).
+
+## Links
-* [Documentation](https://airflow.incubator.apache.org/)
-* [Chat](https://gitter.im/apache/incubator-airflow)
-* [Apache Airflow Incubation Status](http://incubator.apache.org/projects/airflow.html)
-* [More](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links)
+- [Documentation](https://airflow.incubator.apache.org/)
+- [Chat](https://apache-airflow-slack.herokuapp.com/)
+- [Apache Airflow Incubation Status](http://incubator.apache.org/projects/airflow.html)
+- [More](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links)
diff --git a/TODO.md b/TODO.md
index cf19035e1b..780ca20722 100644
--- a/TODO.md
+++ b/TODO.md
@@ -1,10 +1,11 @@
#### Roadmap items
+
* UI page answering "Why isn't this task instance running?"
* Attempt removing DagBag caching for the web server
* Distributed scheduler (supervisors)
- * Get the supervisors to run sensors (as opposed to each sensor taking a slot)
- * Improve DagBag differential refresh
- * Pickle all the THINGS! supervisors maintains fresh, versionned pickles in the database as they monitor for change
+ * Get the supervisors to run sensors (as opposed to each sensor taking a slot)
+ * Improve DagBag differential refresh
+ * Pickle all the THINGS! supervisors maintains fresh, versioned pickles in the database as they monitor for change
* Pre-prod running off of master
* Containment / YarnExecutor / Docker?
* Get s3 logs
@@ -12,19 +13,23 @@
* Run Hive / Hadoop / HDFS tests in Travis-CI
#### UI
+
* Backfill form
* Better task filtering int duration and landing time charts (operator toggle, task regex, uncheck all button)
* Add templating to adhoc queries
#### Backend
+
* Add a run_only_latest flag to BaseOperator, runs only most recent task instance where deps are met
* Raise errors when setting dependencies on task in foreign DAGs
* Add an is_test flag to the run context
#### Wishlist
+
* Pause flag at the task level
* Increase unit test coverage
* Stats logging interface with support for stats and sqlalchemy to collect detailed information from the scheduler and dag processing times
#### Other
-* deprecate TimeSensor
+
+* Deprecate TimeSensor
diff --git a/UPDATING.md b/UPDATING.md
index 3a66e735c3..74337f3fe8 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -5,7 +5,83 @@ assists users migrating to a new version.
## Airflow Master
+### Rename of BashTaskRunner to StandardTaskRunner
+
+BashTaskRunner has been renamed to StandardTaskRunner. It is the default task runner
+so you might need to update your config.
+
+`task_runner = StandardTaskRunner`
+
+### DAG level Access Control for new RBAC UI
+
+Extend and enhance new Airflow RBAC UI to support DAG level ACL. Each dag now has two permissions(one for write, one for read) associated('can_dag_edit', 'can_dag_read').
+The admin will create new role, associate the dag permission with the target dag and assign that role to users. That user can only access / view the certain dags on the UI
+that he has permissions on. If a new role wants to access all the dags, the admin could associate dag permissions on an artificial view(``all_dags``) with that role.
+
+We also provide a new cli command(``sync_perm``) to allow admin to auto sync permissions.
+
+
+### min_file_parsing_loop_time config option temporarily disabled
+
+The scheduler.min_file_parsing_loop_time config option has been temporarily removed due to
+some bugs.
+
+### new `sync_parallelism` config option in celery section
+
+The new `sync_parallelism` config option will control how many processes CeleryExecutor will use to
+fetch celery task state in parallel. Default value is max(1, number of cores - 1)
+
+### CLI Changes
+
+The ability to manipulate users from the command line has been changed. 'airflow create_user' and 'airflow delete_user' and 'airflow list_users' has been grouped to a single command `airflow users` with optional flags `--create`, `--list` and `--delete`.
+
+Example Usage:
+
+To create a new user:
+```bash
+airflow users --create --username jondoe --lastname doe --firstname jon --email jdoe@apache.org --role Viewer --password test
+```
+
+To list users:
+```bash
+airflow users --list
+```
+
+To delete a user:
+```bash
+airflow users --delete --username jondoe
+```
+
+### Custom auth backends interface change
+
+We have updated the version of flask-login we depend upon, and as a result any
+custom auth backends might need a small change: `is_active`,
+`is_authenticated`, and `is_anonymous` should now be properties. What this means is if
+previously you had this in your user class
+
+ def is_active(self):
+ return self.active
+
+then you need to change it like this
+
+ @property
+ def is_active(self):
+ return self.active
+
+## Airflow 1.10
+
+Installation and upgrading requires setting `SLUGIFY_USES_TEXT_UNIDECODE=yes` in your environment or
+`AIRFLOW_GPL_UNIDECODE=yes`. In case of the latter a GPL runtime dependency will be installed due to a
+dependency (python-nvd3 -> python-slugify -> unidecode).
+
+### Replace DataProcHook.await calls to DataProcHook.wait
+
+The method name was changed to be compatible with the Python 3.7 async/await keywords
+
+### Setting UTF-8 as default mime_charset in email utils
+
### Add a configuration variable(default_dag_run_display_number) to control numbers of dag run for display
+
Add a configuration variable(default_dag_run_display_number) under webserver section to control num of dag run to show in UI.
### Default executor for SubDagOperator is changed to SequentialExecutor
@@ -35,11 +111,13 @@ Run `airflow webserver` to start the new UI. This will bring up a log in page, e
There are five roles created for Airflow by default: Admin, User, Op, Viewer, and Public. To configure roles/permissions, go to the `Security` tab and click `List Roles` in the new UI.
#### Breaking changes
+
- AWS Batch Operator renamed property queue to job_queue to prevent conflict with the internal queue from CeleryExecutor - AIRFLOW-2542
- Users created and stored in the old users table will not be migrated automatically. FAB's built-in authentication support must be reconfigured.
- Airflow dag home page is now `/home` (instead of `/admin`).
- All ModelViews in Flask-AppBuilder follow a different pattern from Flask-Admin. The `/admin` part of the url path will no longer exist. For example: `/admin/connection` becomes `/connection/list`, `/admin/connection/new` becomes `/connection/add`, `/admin/connection/edit` becomes `/connection/edit`, etc.
- Due to security concerns, the new webserver will no longer support the features in the `Data Profiling` menu of old UI, including `Ad Hoc Query`, `Charts`, and `Known Events`.
+- HiveServer2Hook.get_results() always returns a list of tuples, even when a single column is queried, as per Python API 2.
### airflow.contrib.sensors.hdfs_sensors renamed to airflow.contrib.sensors.hdfs_sensor
@@ -53,61 +131,84 @@ to have specified `explicit_defaults_for_timestamp=1` in your my.cnf under `[mys
### Celery config
To make the config of Airflow compatible with Celery, some properties have been renamed:
+
```
celeryd_concurrency -> worker_concurrency
celery_result_backend -> result_backend
+celery_ssl_active -> ssl_active
+celery_ssl_cert -> ssl_cert
+celery_ssl_key -> ssl_key
```
+
Resulting in the same config parameters as Celery 4, with more transparency.
### GCP Dataflow Operators
+
Dataflow job labeling is now supported in Dataflow{Java,Python}Operator with a default
"airflow-version" label, please upgrade your google-cloud-dataflow or apache-beam version
to 2.2.0 or greater.
### BigQuery Hooks and Operator
+
The `bql` parameter passed to `BigQueryOperator` and `BigQueryBaseCursor.run_query` has been deprecated and renamed to `sql` for consistency purposes. Using `bql` will still work (and raise a `DeprecationWarning`), but is no longer
supported and will be removed entirely in Airflow 2.0
### Redshift to S3 Operator
+
With Airflow 1.9 or lower, Unload operation always included header row. In order to include header row,
we need to turn off parallel unload. It is preferred to perform unload operation using all nodes so that it is
faster for larger tables. So, parameter called `include_header` is added and default is set to False.
-Header row will be added only if this parameter is set True and also in that case parallel will be automatically turned off (`PARALLEL OFF`)
+Header row will be added only if this parameter is set True and also in that case parallel will be automatically turned off (`PARALLEL OFF`)
### Google cloud connection string
With Airflow 1.9 or lower, there were two connection strings for the Google Cloud operators, both `google_cloud_storage_default` and `google_cloud_default`. This can be confusing and therefore the `google_cloud_storage_default` connection id has been replaced with `google_cloud_default` to make the connection id consistent across Airflow.
### Logging Configuration
+
With Airflow 1.9 or lower, `FILENAME_TEMPLATE`, `PROCESSOR_FILENAME_TEMPLATE`, `LOG_ID_TEMPLATE`, `END_OF_LOG_MARK` were configured in `airflow_local_settings.py`. These have been moved into the configuration file, and hence if you were using a custom configuration file the following defaults need to be added.
+
```
[core]
fab_logging_level = WARN
-log_filename_template = {{{{ ti.dag_id }}}}/{{{{ ti.task_id }}}}/{{{{ ts }}}}/{{{{ try_number }}}}.log
-log_processor_filename_template = {{{{ filename }}}}.log
+log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
+log_processor_filename_template = {{ filename }}.log
[elasticsearch]
-elasticsearch_log_id_template = {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}
+elasticsearch_log_id_template = {dag_id}-{task_id}-{execution_date}-{try_number}
elasticsearch_end_of_log_mark = end_of_log
```
+The previous setting of `log_task_reader` is not needed in many cases now when using the default logging config with remote storages. (Previously it needed to be set to `s3.task` or similar. This is not needed with the default config anymore)
+
+#### Change of per-task log path
+
+With the change to Airflow core to be timezone aware the default log path for task instances will now include timezone information. This will by default mean all previous task logs won't be found. You can get the old behaviour back by setting the following config options:
+
+```
+[core]
+log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ execution_date.strftime("%%Y-%%m-%%dT%%H:%%M:%%S") }}/{{ try_number }}.log
+```
+
## Airflow 1.9
### SSH Hook updates, along with new SSH Operator & SFTP Operator
SSH Hook now uses the Paramiko library to create an ssh client connection, instead of the sub-process based ssh command execution previously (<1.9.0), so this is backward incompatible.
- - update SSHHook constructor
- - use SSHOperator class in place of SSHExecuteOperator which is removed now. Refer to test_ssh_operator.py for usage info.
- - SFTPOperator is added to perform secure file transfer from serverA to serverB. Refer to test_sftp_operator.py.py for usage info.
- - No updates are required if you are using ftpHook, it will continue to work as is.
+
+- update SSHHook constructor
+- use SSHOperator class in place of SSHExecuteOperator which is removed now. Refer to test_ssh_operator.py for usage info.
+- SFTPOperator is added to perform secure file transfer from serverA to serverB. Refer to test_sftp_operator.py.py for usage info.
+- No updates are required if you are using ftpHook, it will continue to work as is.
### S3Hook switched to use Boto3
The airflow.hooks.S3_hook.S3Hook has been switched to use boto3 instead of the older boto (a.k.a. boto2). This results in a few backwards incompatible changes to the following classes: S3Hook:
- - the constructors no longer accepts `s3_conn_id`. It is now called `aws_conn_id`.
- - the default connection is now "aws_default" instead of "s3_default"
- - the return type of objects returned by `get_bucket` is now boto3.s3.Bucket
- - the return type of `get_key`, and `get_wildcard_key` is now an boto3.S3.Object.
+
+- the constructors no longer accepts `s3_conn_id`. It is now called `aws_conn_id`.
+- the default connection is now "aws_default" instead of "s3_default"
+- the return type of objects returned by `get_bucket` is now boto3.s3.Bucket
+- the return type of `get_key`, and `get_wildcard_key` is now an boto3.S3.Object.
If you are using any of these in your DAGs and specify a connection ID you will need to update the parameter name for the connection to "aws_conn_id": S3ToHiveTransfer, S3PrefixSensor, S3KeySensor, RedshiftToS3Transfer.
@@ -283,10 +384,11 @@ The `file_task_handler` logger has been made more flexible. The default format c
If you are logging to Google cloud storage, please see the [Google cloud platform documentation](https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform) for logging instructions.
If you are using S3, the instructions should be largely the same as the Google cloud platform instructions above. You will need a custom logging config. The `REMOTE_BASE_LOG_FOLDER` configuration key in your airflow config has been removed, therefore you will need to take the following steps:
- - Copy the logging configuration from [`airflow/config_templates/airflow_logging_settings.py`](https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py).
- - Place it in a directory inside the Python import path `PYTHONPATH`. If you are using Python 2.7, ensuring that any `__init__.py` files exist so that it is importable.
- - Update the config by setting the path of `REMOTE_BASE_LOG_FOLDER` explicitly in the config. The `REMOTE_BASE_LOG_FOLDER` key is not used anymore.
- - Set the `logging_config_class` to the filename and dict. For example, if you place `custom_logging_config.py` on the base of your pythonpath, you will need to set `logging_config_class = custom_logging_config.LOGGING_CONFIG` in your config as Airflow 1.8.
+
+- Copy the logging configuration from [`airflow/config_templates/airflow_logging_settings.py`](https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py).
+- Place it in a directory inside the Python import path `PYTHONPATH`. If you are using Python 2.7, ensuring that any `__init__.py` files exist so that it is importable.
+- Update the config by setting the path of `REMOTE_BASE_LOG_FOLDER` explicitly in the config. The `REMOTE_BASE_LOG_FOLDER` key is not used anymore.
+- Set the `logging_config_class` to the filename and dict. For example, if you place `custom_logging_config.py` on the base of your pythonpath, you will need to set `logging_config_class = custom_logging_config.LOGGING_CONFIG` in your config as Airflow 1.8.
### New Features
@@ -295,8 +397,10 @@ If you are using S3, the instructions should be largely the same as the Google c
A new DaskExecutor allows Airflow tasks to be run in Dask Distributed clusters.
### Deprecated Features
+
These features are marked for deprecation. They may still work (and raise a `DeprecationWarning`), but are no longer
supported and will be removed entirely in Airflow 2.0
+
- If you're using the `google_cloud_conn_id` or `dataproc_cluster` argument names explicitly in `contrib.operators.Dataproc{*}Operator`(s), be sure to rename them to `gcp_conn_id` or `cluster_name`, respectively. We've renamed these arguments for consistency. (AIRFLOW-1323)
- `post_execute()` hooks now take two arguments, `context` and `result`
@@ -320,30 +424,36 @@ a previously installed version of Airflow before installing 1.8.1.
## Airflow 1.8
### Database
+
The database schema needs to be upgraded. Make sure to shutdown Airflow and make a backup of your database. To
upgrade the schema issue `airflow upgradedb`.
### Upgrade systemd unit files
+
Systemd unit files have been updated. If you use systemd please make sure to update these.
> Please note that the webserver does not detach properly, this will be fixed in a future version.
### Tasks not starting although dependencies are met due to stricter pool checking
+
Airflow 1.7.1 has issues with being able to over subscribe to a pool, ie. more slots could be used than were
available. This is fixed in Airflow 1.8.0, but due to past issue jobs may fail to start although their
dependencies are met after an upgrade. To workaround either temporarily increase the amount of slots above
the amount of queued tasks or use a new pool.
### Less forgiving scheduler on dynamic start_date
+
Using a dynamic start_date (e.g. `start_date = datetime.now()`) is not considered a best practice. The 1.8.0 scheduler
is less forgiving in this area. If you encounter DAGs not being scheduled you can try using a fixed start_date and
renaming your DAG. The last step is required to make sure you start with a clean slate, otherwise the old schedule can
interfere.
### New and updated scheduler options
+
Please read through the new scheduler options, defaults have changed since 1.7.1.
#### child_process_log_directory
+
In order to increase the robustness of the scheduler, DAGS are now processed in their own process. Therefore each
DAG has its own log file for the scheduler. These log files are placed in `child_process_log_directory` which defaults to
`<AIRFLOW_HOME>/scheduler/latest`. You will need to make sure these log files are removed.
@@ -351,24 +461,30 @@ DAG has its own log file for the scheduler. These log files are placed in `child
> DAG logs or processor logs ignore and command line settings for log file locations.
#### run_duration
+
Previously the command line option `num_runs` was used to let the scheduler terminate after a certain amount of
loops. This is now time bound and defaults to `-1`, which means run continuously. See also num_runs.
#### num_runs
+
Previously `num_runs` was used to let the scheduler terminate after a certain amount of loops. Now num_runs specifies
the number of times to try to schedule each DAG file within `run_duration` time. Defaults to `-1`, which means try
indefinitely. This is only available on the command line.
#### min_file_process_interval
+
After how much time should an updated DAG be picked up from the filesystem.
#### min_file_parsing_loop_time
+CURRENTLY DISABLED DUE TO A BUG
How many seconds to wait between file-parsing loops to prevent the logs from being spammed.
#### dag_dir_list_interval
+
The frequency with which the scheduler should relist the contents of the DAG directory. If while developing +dags, they are not being picked up, have a look at this number and decrease it when necessary.
#### catchup_by_default
+
By default the scheduler will fill any missing interval DAG Runs between the last execution date and the current date.
This setting changes that behavior to only execute the latest interval. This can also be specified per DAG as
`catchup = False / True`. Command line backfills will still work.
@@ -399,6 +515,7 @@ required to whitelist these variables by adding the following to your configurat
<value>airflow\.ctx\..*</value>
</property>
```
+
### Google Cloud Operator and Hook alignment
All Google Cloud Operators and Hooks are aligned and use the same client library. Now you have a single connection
@@ -410,6 +527,7 @@ Also the old P12 key file type is not supported anymore and only the new JSON ke
account.
### Deprecated Features
+
These features are marked for deprecation. They may still work (and raise a `DeprecationWarning`), but are no longer
supported and will be removed entirely in Airflow 2.0
@@ -426,6 +544,7 @@ supported and will be removed entirely in Airflow 2.0
- The config value secure_mode will default to True which will disable some insecure endpoints/features
### Known Issues
+
There is a report that the default of "-1" for num_runs creates an issue where errors are reported while parsing tasks.
It was not confirmed, but a workaround was found by changing the default back to `None`.
@@ -452,7 +571,9 @@ To continue using the default smtp email backend, change the email_backend line
[email]
email_backend = airflow.utils.send_email_smtp
```
+
to:
+
```
[email]
email_backend = airflow.utils.email.send_email_smtp
@@ -465,7 +586,9 @@ To continue using S3 logging, update your config file so:
```
s3_log_folder = s3://my-airflow-log-bucket/logs
```
+
becomes:
+
```
remote_base_log_folder = s3://my-airflow-log-bucket/logs
remote_log_conn_id = <your desired s3 connection>
diff --git a/airflow/__init__.py b/airflow/__init__.py
index f40b08aab5..9d071a347c 100644
--- a/airflow/__init__.py
+++ b/airflow/__init__.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -32,8 +32,8 @@
import sys
-from airflow import configuration as conf
-from airflow import settings
+# flake8: noqa: F401
+from airflow import settings, configuration as conf
from airflow.models import DAG
from flask_admin import BaseView
from importlib import import_module
@@ -79,15 +79,3 @@ class AirflowViewPlugin(BaseView):
class AirflowMacroPlugin(object):
def __init__(self, namespace):
self.namespace = namespace
-
-from airflow import operators
-from airflow import sensors # noqa: E402
-from airflow import hooks
-from airflow import executors
-from airflow import macros
-
-operators._integrate_plugins()
-sensors._integrate_plugins() # noqa: E402
-hooks._integrate_plugins()
-executors._integrate_plugins()
-macros._integrate_plugins()
diff --git a/airflow/api/auth/backend/kerberos_auth.py b/airflow/api/auth/backend/kerberos_auth.py
index 7e560fb296..50a88106fc 100644
--- a/airflow/api/auth/backend/kerberos_auth.py
+++ b/airflow/api/auth/backend/kerberos_auth.py
@@ -1,21 +1,28 @@
# -*- coding: utf-8 -*-
#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
+# Copyright (c) 2013, Michael Komitee
+# All rights reserved.
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# Redistribution and use in source and binary forms, with or without modification,
+# are permitted provided that the following conditions are met:
#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
+# 1. Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
from future.standard_library import install_aliases
diff --git a/airflow/api/common/experimental/mark_tasks.py b/airflow/api/common/experimental/mark_tasks.py
index e9e4fec223..2fac1254cd 100644
--- a/airflow/api/common/experimental/mark_tasks.py
+++ b/airflow/api/common/experimental/mark_tasks.py
@@ -17,15 +17,16 @@
# specific language governing permissions and limitations
# under the License.
+from sqlalchemy import or_
+
from airflow.jobs import BackfillJob
from airflow.models import DagRun, TaskInstance
from airflow.operators.subdag_operator import SubDagOperator
from airflow.settings import Session
from airflow.utils import timezone
+from airflow.utils.db import provide_session
from airflow.utils.state import State
-from sqlalchemy import or_
-
def _create_dagruns(dag, execution_dates, state, run_id_template):
"""
@@ -191,15 +192,73 @@ def set_state(task, execution_date, upstream=False, downstream=False,
return tis_altered
-def set_dag_run_state(dag, execution_date, state=State.SUCCESS, commit=False):
+def _set_dag_run_state(dag_id, execution_date, state, session=None):
+ """
+ Helper method that set dag run state in the DB.
+ :param dag_id: dag_id of target dag run
+ :param execution_date: the execution date from which to start looking
+ :param state: target state
+ :param session: database session
+ """
+ DR = DagRun
+ dr = session.query(DR).filter(
+ DR.dag_id == dag_id,
+ DR.execution_date == execution_date
+ ).one()
+ dr.state = state
+ if state == State.RUNNING:
+ dr.start_date = timezone.utcnow()
+ dr.end_date = None
+ else:
+ dr.end_date = timezone.utcnow()
+ session.commit()
+
+
+@provide_session
+def set_dag_run_state_to_success(dag, execution_date, commit=False,
+ session=None):
"""
- Set the state of a dag run and all task instances associated with the dag
- run for a specific execution date.
+ Set the dag run for a specific execution date and its task instances
+ to success.
:param dag: the DAG of which to alter state
:param execution_date: the execution date from which to start looking
- :param state: the state to which the DAG need to be set
:param commit: commit DAG and tasks to be altered to the database
- :return: list of tasks that have been created and updated
+ :param session: database session
+ :return: If commit is true, list of tasks that have been updated,
+ otherwise list of tasks that will be updated
+ :raises: AssertionError if dag or execution_date is invalid
+ """
+ res = []
+
+ if not dag or not execution_date:
+ return res
+
+ # Mark the dag run to success.
+ if commit:
+ _set_dag_run_state(dag.dag_id, execution_date, State.SUCCESS, session)
+
+ # Mark all task instances of the dag run to success.
+ for task in dag.tasks:
+ task.dag = dag
+ new_state = set_state(task=task, execution_date=execution_date,
+ state=State.SUCCESS, commit=commit)
+ res.extend(new_state)
+
+ return res
+
+
+@provide_session
+def set_dag_run_state_to_failed(dag, execution_date, commit=False,
+ session=None):
+ """
+ Set the dag run for a specific execution date and its running task instances
+ to failed.
+ :param dag: the DAG of which to alter state
+ :param execution_date: the execution date from which to start looking
+ :param commit: commit DAG and tasks to be altered to the database
+ :param session: database session
+ :return: If commit is true, list of tasks that have been updated,
+ otherwise list of tasks that will be updated
:raises: AssertionError if dag or execution_date is invalid
"""
res = []
@@ -207,18 +266,48 @@ def set_dag_run_state(dag, execution_date, state=State.SUCCESS, commit=False):
if not dag or not execution_date:
return res
- # Mark all task instances in the dag run
+ # Mark the dag run to failed.
+ if commit:
+ _set_dag_run_state(dag.dag_id, execution_date, State.FAILED, session)
+
+ # Mark only RUNNING task instances.
+ TI = TaskInstance
+ task_ids = [task.task_id for task in dag.tasks]
+ tis = session.query(TI).filter(
+ TI.dag_id == dag.dag_id,
+ TI.execution_date == execution_date,
+ TI.task_id.in_(task_ids)).filter(TI.state == State.RUNNING)
+ task_ids_of_running_tis = [ti.task_id for ti in tis]
for task in dag.tasks:
+ if task.task_id not in task_ids_of_running_tis:
+ continue
task.dag = dag
new_state = set_state(task=task, execution_date=execution_date,
- state=state, commit=commit)
+ state=State.FAILED, commit=commit)
res.extend(new_state)
- # Mark the dag run
+ return res
+
+
+@provide_session
+def set_dag_run_state_to_running(dag, execution_date, commit=False,
+ session=None):
+ """
+ Set the dag run for a specific execution date to running.
+ :param dag: the DAG of which to alter state
+ :param execution_date: the execution date from which to start looking
+ :param commit: commit DAG and tasks to be altered to the database
+ :param session: database session
+ :return: If commit is true, list of tasks that have been updated,
+ otherwise list of tasks that will be updated
+ """
+ res = []
+ if not dag or not execution_date:
+ return res
+
+ # Mark the dag run to running.
if commit:
- drs = DagRun.find(dag.dag_id, execution_date=execution_date)
- for dr in drs:
- dr.dag = dag
- dr.update_state()
+ _set_dag_run_state(dag.dag_id, execution_date, State.RUNNING, session)
+ # To keep the return type consistent with the other similar functions.
return res
diff --git a/airflow/api/common/experimental/trigger_dag.py b/airflow/api/common/experimental/trigger_dag.py
index 86be6aa544..d7353f66d1 100644
--- a/airflow/api/common/experimental/trigger_dag.py
+++ b/airflow/api/common/experimental/trigger_dag.py
@@ -20,7 +20,7 @@
import json
from airflow.exceptions import DagRunAlreadyExists, DagNotFound
-from airflow.models import DagRun, DagBag
+from airflow.models import DagRun, DagBag, DagModel
from airflow.utils import timezone
from airflow.utils.state import State
@@ -86,7 +86,10 @@ def trigger_dag(
execution_date=None,
replace_microseconds=True,
):
- dagbag = DagBag()
+ dag_model = DagModel.get_current(dag_id)
+ if dag_model is None:
+ raise DagNotFound("Dag id {} not found in DagModel".format(dag_id))
+ dagbag = DagBag(dag_folder=dag_model.fileloc)
dag_run = DagRun()
triggers = _trigger_dag(
dag_id=dag_id,
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index b56e325327..6cbda705f1 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -80,6 +80,11 @@
log = LoggingMixin().log
+DAGS_FOLDER = settings.DAGS_FOLDER
+
+if "BUILDING_AIRFLOW_DOCS" in os.environ:
+ DAGS_FOLDER = '[AIRFLOW_HOME]/dags'
+
def sigint_handler(sig, frame):
sys.exit(0)
@@ -133,7 +138,7 @@ def setup_locations(process, pid=None, stdout=None, stderr=None, log=None):
def process_subdir(subdir):
if subdir:
- subdir = subdir.replace('DAGS_FOLDER', settings.DAGS_FOLDER)
+ subdir = subdir.replace('DAGS_FOLDER', DAGS_FOLDER)
subdir = os.path.abspath(os.path.expanduser(subdir))
return subdir
@@ -198,7 +203,7 @@ def backfill(args, dag=None):
start_date=args.start_date,
end_date=args.end_date,
confirm_prompt=True,
- include_subdags=False,
+ include_subdags=True,
)
dag.run(
@@ -267,6 +272,7 @@ def _tabulate(pools):
tablefmt="fancy_grid")
try:
+ imp = getattr(args, 'import')
if args.get is not None:
pools = [api_client.get_pool(name=args.get)]
elif args.set:
@@ -275,6 +281,14 @@ def _tabulate(pools):
description=args.set[2])]
elif args.delete:
pools = [api_client.delete_pool(name=args.delete)]
+ elif imp:
+ if os.path.exists(imp):
+ pools = pool_import_helper(imp)
+ else:
+ print("Missing pools file.")
+ pools = api_client.get_pools()
+ elif args.export:
+ pools = pool_export_helper(args.export)
else:
pools = api_client.get_pools()
except (AirflowException, IOError) as err:
@@ -283,6 +297,43 @@ def _tabulate(pools):
log.info(_tabulate(pools=pools))
+def pool_import_helper(filepath):
+ with open(filepath, 'r') as poolfile:
+ pl = poolfile.read()
+ try:
+ d = json.loads(pl)
+ except Exception as e:
+ print("Please check the validity of the json file: " + str(e))
+ else:
+ try:
+ pools = []
+ n = 0
+ for k, v in d.items():
+ if isinstance(v, dict) and len(v) == 2:
+ pools.append(api_client.create_pool(name=k,
+ slots=v["slots"],
+ description=v["description"]))
+ n += 1
+ else:
+ pass
+ except Exception:
+ pass
+ finally:
+ print("{} of {} pool(s) successfully updated.".format(n, len(d)))
+ return pools
+
+
+def pool_export_helper(filepath):
+ pool_dict = {}
+ pools = api_client.get_pools()
+ for pool in pools:
+ pool_dict[pool[0]] = {"slots": pool[1], "description": pool[2]}
+ with open(filepath, 'w') as poolfile:
+ poolfile.write(json.dumps(pool_dict, sort_keys=True, indent=4))
+ print("{} pools successfully exported to {}".format(len(pool_dict), filepath))
+ return pools
+
+
@cli_utils.action_logging
def variables(args):
if args.get:
@@ -437,6 +488,24 @@ def _run(args, dag, ti):
@cli_utils.action_logging
def run(args, dag=None):
+ # Optional sections won't log an error if they're missing in airflow.cfg.
+ OPTIONAL_AIRFLOW_CFG_SECTIONS = [
+ 'atlas',
+ 'celery',
+ 'celery_broker_transport_options',
+ 'dask',
+ 'elasticsearch',
+ 'github_enterprise',
+ 'hive',
+ 'kerberos',
+ 'kubernetes',
+ 'kubernetes_node_selectors',
+ 'kubernetes_secrets',
+ 'ldap',
+ 'lineage',
+ 'mesos',
+ ]
+
if dag:
args.dag_id = dag.dag_id
@@ -459,9 +528,15 @@ def run(args, dag=None):
try:
conf.set(section, option, value)
except NoSectionError:
- log.error('Section {section} Option {option} '
- 'does not exist in the config!'.format(section=section,
- option=option))
+ no_section_msg = (
+ 'Section {section} Option {option} '
+ 'does not exist in the config!'
+ ).format(section=section, option=option)
+
+ if section in OPTIONAL_AIRFLOW_CFG_SECTIONS:
+ log.debug(no_section_msg)
+ else:
+ log.error(no_section_msg)
settings.configure_vars()
@@ -551,6 +626,31 @@ def dag_state(args):
print(dr[0].state if len(dr) > 0 else None)
+@cli_utils.action_logging
+def next_execution(args):
+ """
+ Returns the next execution datetime of a DAG at the command line.
+ >>> airflow next_execution tutorial
+ 2018-08-31 10:38:00
+ """
+ dag = get_dag(args)
+
+ if dag.is_paused:
+ print("[INFO] Please be reminded this DAG is PAUSED now.")
+
+ if dag.latest_execution_date:
+ next_execution_dttm = dag.following_schedule(dag.latest_execution_date)
+
+ if next_execution_dttm is None:
+ print("[WARN] No following schedule can be found. " +
+ "This DAG may have schedule interval '@once' or `None`.")
+
+ print(next_execution_dttm)
+ else:
+ print("[WARN] Only applicable when there is execution record found for the DAG.")
+ print(None)
+
+
@cli_utils.action_logging
def list_dags(args):
dagbag = DagBag(process_subdir(args.subdir))
@@ -629,7 +729,9 @@ def clear(args):
only_failed=args.only_failed,
only_running=args.only_running,
confirm_prompt=not args.no_confirm,
- include_subdags=not args.exclude_subdags)
+ include_subdags=not args.exclude_subdags,
+ include_parentdag=not args.exclude_parentdag,
+ )
def get_num_ready_workers_running(gunicorn_master_proc):
@@ -783,13 +885,19 @@ def webserver(args):
print(
"Starting the web server on port {0} and host {1}.".format(
args.port, args.hostname))
- app = create_app_rbac(conf) if settings.RBAC else create_app(conf)
- app.run(debug=True, port=args.port, host=args.hostname,
+ if settings.RBAC:
+ app, _ = create_app_rbac(conf, testing=conf.get('core', 'unit_test_mode'))
+ else:
+ app = create_app(conf, testing=conf.get('core', 'unit_test_mode'))
+ app.run(debug=True, use_reloader=False if app.config['TESTING'] else True,
+ port=args.port, host=args.hostname,
ssl_context=(ssl_cert, ssl_key) if ssl_cert and ssl_key else None)
else:
+ os.environ['SKIP_DAGS_PARSING'] = 'True'
app = cached_app_rbac(conf) if settings.RBAC else cached_app(conf)
pid, stdout, stderr, log_file = setup_locations(
"webserver", args.pid, args.stdout, args.stderr, args.log_file)
+ os.environ.pop('SKIP_DAGS_PARSING')
if args.daemon:
handle = setup_logging(log_file)
stdout = open(stdout, 'w+')
@@ -844,9 +952,11 @@ def monitor_gunicorn(gunicorn_master_proc):
master_timeout = conf.getint('webserver', 'web_server_master_timeout')
restart_workers(gunicorn_master_proc, num_workers, master_timeout)
else:
- while True:
+ while gunicorn_master_proc.poll() is None:
time.sleep(1)
+ sys.exit(gunicorn_master_proc.returncode)
+
if args.daemon:
base, ext = os.path.splitext(pid)
ctx = daemon.DaemonContext(
@@ -951,6 +1061,11 @@ def worker(args):
env = os.environ.copy()
env['AIRFLOW_HOME'] = settings.AIRFLOW_HOME
+ if not settings.validate_session():
+ log = LoggingMixin().log
+ log.error("Worker exiting... database connection precheck failed! ")
+ sys.exit(1)
+
# Celery worker
from airflow.executors.celery_executor import app as celery_app
from celery.bin import worker
@@ -1003,7 +1118,6 @@ def initdb(args): # noqa
print("Done.")
-@cli_utils.action_logging
def resetdb(args):
print("DB: " + repr(settings.engine.url))
if args.yes or input("This will drop existing tables "
@@ -1252,40 +1366,137 @@ def kerberos(args): # noqa
@cli_utils.action_logging
-def create_user(args):
- fields = {
- 'role': args.role,
- 'username': args.username,
- 'email': args.email,
- 'firstname': args.firstname,
- 'lastname': args.lastname,
- }
- empty_fields = [k for k, v in fields.items() if not v]
- if empty_fields:
- raise SystemExit('Required arguments are missing: {}.'.format(
- ', '.join(empty_fields)))
-
- appbuilder = cached_appbuilder()
- role = appbuilder.sm.find_role(args.role)
- if not role:
- raise SystemExit('{} is not a valid role.'.format(args.role))
-
- if args.use_random_password:
- password = ''.join(random.choice(string.printable) for _ in range(16))
- elif args.password:
- password = args.password
- else:
- password = getpass.getpass('Password:')
- password_confirmation = getpass.getpass('Repeat for confirmation:')
- if password != password_confirmation:
- raise SystemExit('Passwords did not match!')
-
- user = appbuilder.sm.add_user(args.username, args.firstname, args.lastname,
- args.email, role, password)
- if user:
- print('{} user {} created.'.format(args.role, args.username))
+def users(args):
+ if args.list:
+
+ appbuilder = cached_appbuilder()
+ users = appbuilder.sm.get_all_users()
+ fields = ['id', 'username', 'email', 'first_name', 'last_name', 'roles']
+ users = [[user.__getattribute__(field) for field in fields] for user in users]
+ msg = tabulate(users, [field.capitalize().replace('_', ' ') for field in fields],
+ tablefmt="fancy_grid")
+ if sys.version_info[0] < 3:
+ msg = msg.encode('utf-8')
+ print(msg)
+
+ return
+
+ if args.create:
+ fields = {
+ 'role': args.role,
+ 'username': args.username,
+ 'email': args.email,
+ 'firstname': args.firstname,
+ 'lastname': args.lastname,
+ }
+ empty_fields = [k for k, v in fields.items() if not v]
+ if empty_fields:
+ raise SystemExit('Required arguments are missing: {}.'.format(
+ ', '.join(empty_fields)))
+
+ appbuilder = cached_appbuilder()
+ role = appbuilder.sm.find_role(args.role)
+ if not role:
+ raise SystemExit('{} is not a valid role.'.format(args.role))
+
+ if args.use_random_password:
+ password = ''.join(random.choice(string.printable) for _ in range(16))
+ elif args.password:
+ password = args.password
+ else:
+ password = getpass.getpass('Password:')
+ password_confirmation = getpass.getpass('Repeat for confirmation:')
+ if password != password_confirmation:
+ raise SystemExit('Passwords did not match!')
+
+ if appbuilder.sm.find_user(args.username):
+ print('{} already exist in the db'.format(args.username))
+ return
+ user = appbuilder.sm.add_user(args.username, args.firstname, args.lastname,
+ args.email, role, password)
+ if user:
+ print('{} user {} created.'.format(args.role, args.username))
+ else:
+ raise SystemExit('Failed to create user.')
+
+ if args.delete:
+ if not args.username:
+ raise SystemExit('Required arguments are missing: username')
+
+ appbuilder = cached_appbuilder()
+
+ try:
+ u = next(u for u in appbuilder.sm.get_all_users()
+ if u.username == args.username)
+ except StopIteration:
+ raise SystemExit('{} is not a valid user.'.format(args.username))
+
+ if appbuilder.sm.del_register_user(u):
+ print('User {} deleted.'.format(args.username))
+ else:
+ raise SystemExit('Failed to delete user.')
+
+
+@cli_utils.action_logging
+def list_dag_runs(args, dag=None):
+ if dag:
+ args.dag_id = dag.dag_id
+
+ dagbag = DagBag()
+
+ if args.dag_id not in dagbag.dags:
+ error_message = "Dag id {} not found".format(args.dag_id)
+ raise AirflowException(error_message)
+
+ dag_runs = list()
+ state = args.state.lower() if args.state else None
+ for run in DagRun.find(dag_id=args.dag_id,
+ state=state,
+ no_backfills=args.no_backfill):
+ dag_runs.append({
+ 'id': run.id,
+ 'run_id': run.run_id,
+ 'state': run.state,
+ 'dag_id': run.dag_id,
+ 'execution_date': run.execution_date.isoformat(),
+ 'start_date': ((run.start_date or '') and
+ run.start_date.isoformat()),
+ })
+ if not dag_runs:
+ print('No dag runs for {dag_id}'.format(dag_id=args.dag_id))
+
+ s = textwrap.dedent("""\n
+ {line}
+ DAG RUNS
+ {line}
+ {dag_run_header}
+ """)
+
+ dag_runs.sort(key=lambda x: x['execution_date'], reverse=True)
+ dag_run_header = '%-3s | %-20s | %-10s | %-20s | %-20s |' % ('id',
+ 'run_id',
+ 'state',
+ 'execution_date',
+ 'state_date')
+ print(s.format(dag_run_header=dag_run_header,
+ line='-' * 120))
+ for dag_run in dag_runs:
+ record = '%-3s | %-20s | %-10s | %-20s | %-20s |' % (dag_run['id'],
+ dag_run['run_id'],
+ dag_run['state'],
+ dag_run['execution_date'],
+ dag_run['start_date'])
+ print(record)
+
+
+@cli_utils.action_logging
+def sync_perm(args): # noqa
+ if settings.RBAC:
+ appbuilder = cached_appbuilder()
+ print('Update permission, view-menu for all existing roles')
+ appbuilder.sm.sync_roles()
else:
- raise SystemExit('Failed to create user.')
+ print('The sync_perm command only works for rbac UI.')
Arg = namedtuple(
@@ -1306,8 +1517,10 @@ class CLIFactory(object):
"The regex to filter specific task_ids to backfill (optional)"),
'subdir': Arg(
("-sd", "--subdir"),
- "File location or directory from which to look for the dag",
- default=settings.DAGS_FOLDER),
+ "File location or directory from which to look for the dag. "
+ "Defaults to '[AIRFLOW_HOME]/dags' where [AIRFLOW_HOME] is the "
+ "value you set for 'AIRFLOW_HOME' config you set in 'airflow.cfg' ",
+ default=DAGS_FOLDER),
'start_date': Arg(
("-s", "--start_date"), "Override start_date YYYY-MM-DD",
type=parsedate),
@@ -1335,6 +1548,15 @@ class CLIFactory(object):
"store_true",
default=False),
+ # list_dag_runs
+ 'no_backfill': Arg(
+ ("--no_backfill",),
+ "filter all the backfill dagruns given the dag id", "store_true"),
+ 'state': Arg(
+ ("--state",),
+ "Only list the dag runs corresponding to the state"
+ ),
+
# backfill
'mark_success': Arg(
("-m", "--mark_success"),
@@ -1409,6 +1631,10 @@ class CLIFactory(object):
'exclude_subdags': Arg(
("-x", "--exclude_subdags"),
"Exclude subdags", "store_true"),
+ 'exclude_parentdag': Arg(
+ ("-xp", "--exclude_parentdag"),
+ "Exclude ParentDAGS if the task cleared is a part of a SubDAG",
+ "store_true"),
'dag_regex': Arg(
("-dx", "--dag_regex"),
"Search dag_id as regex instead of exact string", "store_true"),
@@ -1434,6 +1660,14 @@ class CLIFactory(object):
("-x", "--delete"),
metavar="NAME",
help="Delete a pool"),
+ 'pool_import': Arg(
+ ("-i", "--import"),
+ metavar="FILEPATH",
+ help="Import pool from JSON file"),
+ 'pool_export': Arg(
+ ("-e", "--export"),
+ metavar="FILEPATH",
+ help="Export pool to JSON file"),
# variables
'set': Arg(
("-s", "--set"),
@@ -1668,37 +1902,52 @@ class CLIFactory(object):
('--conn_extra',),
help='Connection `Extra` field, optional when adding a connection',
type=str),
- # create_user
- 'role': Arg(
- ('-r', '--role',),
- help='Role of the user. Existing roles include Admin, '
- 'User, Op, Viewer, and Public',
+ # users
+ 'username': Arg(
+ ('--username',),
+ help='Username of the user, required to create/delete a user',
type=str),
'firstname': Arg(
- ('-f', '--firstname',),
- help='First name of the user',
+ ('--firstname',),
+ help='First name of the user, required to create a user',
type=str),
'lastname': Arg(
- ('-l', '--lastname',),
- help='Last name of the user',
+ ('--lastname',),
+ help='Last name of the user, required to create a user',
type=str),
- 'email': Arg(
- ('-e', '--email',),
- help='Email of the user',
+ 'role': Arg(
+ ('--role',),
+ help='Role of the user. Existing roles include Admin, '
+ 'User, Op, Viewer, and Public. Required to create a user',
type=str),
- 'username': Arg(
- ('-u', '--username',),
- help='Username of the user',
+ 'email': Arg(
+ ('--email',),
+ help='Email of the user, required to create a user',
type=str),
'password': Arg(
- ('-p', '--password',),
- help='Password of the user',
+ ('--password',),
+ help='Password of the user, required to create a user '
+ 'without --use_random_password',
type=str),
'use_random_password': Arg(
('--use_random_password',),
- help='Do not prompt for password. Use random string instead',
+ help='Do not prompt for password. Use random string instead.'
+ ' Required to create a user without --password ',
default=False,
action='store_true'),
+ 'list_users': Arg(
+ ('-l', '--list'),
+ help='List all users',
+ action='store_true'),
+ 'create_user': Arg(
+ ('-c', '--create'),
+ help='Create a user',
+ action='store_true'),
+ 'delete_user': Arg(
+ ('-d', '--delete'),
+ help='Delete a user',
+ action='store_true'),
+
}
subparsers = (
{
@@ -1707,7 +1956,7 @@ class CLIFactory(object):
"If reset_dag_run option is used,"
" backfill will first prompt users whether airflow "
"should clear all the previous dag_run and task_instances "
- "within the backfill date range."
+ "within the backfill date range. "
"If rerun_failed_tasks is used, backfill "
"will auto re-run the previous failed task instances"
" within the backfill date range.",
@@ -1718,6 +1967,15 @@ class CLIFactory(object):
'subdir', 'pool', 'delay_on_limit', 'dry_run', 'verbose', 'conf',
'reset_dag_run', 'rerun_failed_tasks',
)
+ }, {
+ 'func': list_dag_runs,
+ 'help': "List dag runs given a DAG id. If state option is given, it will only"
+ "search for all the dagruns with the given state. "
+ "If no_backfill option is given, it will filter out"
+ "all backfill dagruns for given dag id.",
+ 'args': (
+ 'dag_id', 'no_backfill', 'state'
+ )
}, {
'func': list_tasks,
'help': "List the tasks within a DAG",
@@ -1728,7 +1986,7 @@ class CLIFactory(object):
'args': (
'dag_id', 'task_regex', 'start_date', 'end_date', 'subdir',
'upstream', 'downstream', 'no_confirm', 'only_failed',
- 'only_running', 'exclude_subdags', 'dag_regex'),
+ 'only_running', 'exclude_subdags', 'exclude_parentdag', 'dag_regex'),
}, {
'func': pause,
'help': "Pause a DAG",
@@ -1748,7 +2006,7 @@ class CLIFactory(object):
}, {
'func': pool,
'help': "CRUD operations on pools",
- "args": ('pool_set', 'pool_get', 'pool_delete'),
+ "args": ('pool_set', 'pool_get', 'pool_delete', 'pool_import', 'pool_export'),
}, {
'func': variables,
'help': "CRUD operations on variables",
@@ -1847,15 +2105,26 @@ class CLIFactory(object):
'args': ('list_connections', 'add_connection', 'delete_connection',
'conn_id', 'conn_uri', 'conn_extra') + tuple(alternative_conn_specs),
}, {
- 'func': create_user,
- 'help': "Create an admin account",
- 'args': ('role', 'username', 'email', 'firstname', 'lastname',
+ 'func': users,
+ 'help': "List/Create/Delete users",
+ 'args': ('list_users', 'create_user', 'delete_user',
+ 'username', 'email', 'firstname', 'lastname', 'role',
'password', 'use_random_password'),
},
+ {
+ 'func': sync_perm,
+ 'help': "Update existing role's permissions.",
+ 'args': tuple(),
+ },
+ {
+ 'func': next_execution,
+ 'help': "Get the next execution datetime of a DAG.",
+ 'args': ('dag_id', 'subdir')
+ }
)
subparsers_dict = {sp['func'].__name__: sp for sp in subparsers}
dag_subparsers = (
- 'list_tasks', 'backfill', 'test', 'run', 'pause', 'unpause')
+ 'list_tasks', 'backfill', 'test', 'run', 'pause', 'unpause', 'list_dag_runs')
@classmethod
def get_parser(cls, dag_parser=False):
diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg
index 2f6130c897..b572dbb2f7 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -62,12 +62,10 @@ fab_logging_level = WARN
logging_config_class =
# Log format
-# we need to escape the curly braces by adding an additional curly brace
log_format = [%%(asctime)s] {{%%(filename)s:%%(lineno)d}} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
# Log filename format
-# we need to escape the curly braces by adding an additional curly brace
log_filename_template = {{{{ ti.dag_id }}}}/{{{{ ti.task_id }}}}/{{{{ ts }}}}/{{{{ try_number }}}}.log
log_processor_filename_template = {{{{ filename }}}}.log
@@ -140,7 +138,7 @@ donot_pickle = False
dagbag_import_timeout = 30
# The class to use for running task instances in a subprocess
-task_runner = BashTaskRunner
+task_runner = StandardTaskRunner
# If set, tasks without a `run_as_user` argument will be run with this user
# Can be used to de-elevate a sudo user running Airflow when executing tasks
@@ -173,6 +171,9 @@ killed_task_cleanup_time = 60
# `airflow trigger_dag -c`, the key-value pairs will override the existing ones in params.
dag_run_conf_overrides_params = False
+# Worker initialisation check to validate Metadata Database connection
+worker_precheck = False
+
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
@@ -211,6 +212,9 @@ default_gpus = 0
[hive]
# Default mapreduce queue for HiveOperator tasks
default_hive_mapred_queue =
+# Template for mapred_job_name in HiveOperator, supports the following named parameters:
+# hostname, dag_id, task_id, execution_date
+mapred_job_name_template = Airflow HiveOperator task for {{hostname}}.{{dag_id}}.{{task_id}}.{{execution_date}}
[webserver]
# The base url of your website as airflow cannot guess what domain or
@@ -244,7 +248,8 @@ worker_refresh_batch_size = 1
worker_refresh_interval = 30
# Secret key used to run your flask app
-secret_key = temporary_key
+# It should be as random as possible
+secret_key = {SECRET_KEY}
# Number of workers to run the Gunicorn web server
workers = 4
@@ -258,6 +263,9 @@ access_logfile = -
error_logfile = -
# Expose the configuration file in the web server
+# This is only applicable for the flask-admin based web UI (non FAB-based).
+# In the FAB-based web UI with RBAC feature,
+# access to configuration is controlled by role permissions.
expose_config = False
# Set to true to turn on authentication:
@@ -306,6 +314,9 @@ navbar_color = #007A87
# Default dagrun to show in UI
default_dag_run_display_number = 25
+# Enable werkzeug `ProxyFix` middleware
+enable_proxy_fix = False
+
[email]
email_backend = airflow.utils.email.send_email_smtp
@@ -373,6 +384,10 @@ flower_port = 5555
# Default queue that tasks get assigned to and that worker listen on.
default_queue = default
+# How many processes CeleryExecutor uses to sync task state.
+# 0 means to use max(1, number of cores - 1) processes.
+sync_parallelism = 0
+
# Import path for celery configuration options
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
@@ -425,12 +440,10 @@ scheduler_heartbeat_sec = 5
# -1 indicates to run continuously (see also num_runs)
run_duration = -1
-# after how much time a new DAGs should be picked up from the filesystem
+# after how much time (seconds) a new DAGs should be picked up from the filesystem
min_file_process_interval = 0
-# How many seconds to wait between file-parsing loops to prevent the logs from being spammed.
-min_file_parsing_loop_time = 1
-
+# How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
dag_dir_list_interval = 300
# How often should stats be printed to the logs
@@ -475,6 +488,10 @@ max_threads = 2
authenticate = False
+# Turn off scheduler use of cron intervals by setting this to False.
+# DAGs submitted manually in the web UI or with trigger_dag will still run.
+use_job_schedule = True
+
[ldap]
# set this to ldaps://<your.ldap.server>:<port>
uri =
@@ -549,14 +566,15 @@ hide_sensitive_variable_fields = True
[elasticsearch]
elasticsearch_host =
-# we need to escape the curly braces by adding an additional curly brace
elasticsearch_log_id_template = {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}
elasticsearch_end_of_log_mark = end_of_log
[kubernetes]
-# The repository and tag of the Kubernetes Image for the Worker to Run
+# The repository, tag and imagePullPolicy of the Kubernetes Image for the Worker to Run
worker_container_repository =
worker_container_tag =
+worker_container_image_pull_policy = IfNotPresent
+worker_dags_folder =
# If True (default), worker pods will be deleted upon termination
delete_worker_pods = True
@@ -606,10 +624,15 @@ image_pull_secrets =
gcp_service_account_keys =
# Use the service account kubernetes gives to pods to connect to kubernetes cluster.
-# It’s intended for clients that expect to be running inside a pod running on kubernetes.
+# It's intended for clients that expect to be running inside a pod running on kubernetes.
# It will raise an exception if called from a process not running in a kubernetes environment.
in_cluster = True
+[kubernetes_node_selectors]
+# The Key-value pairs to be given to worker pods.
+# The worker pods will be scheduled to the nodes of the specified key-value pairs.
+# Should be supplied in the format: key = value
+
[kubernetes_secrets]
# The scheduler mounts the following secrets into your workers as they are launched by the
# scheduler. You may define as many secrets as needed and the kubernetes launcher will parse the
diff --git a/airflow/config_templates/default_test.cfg b/airflow/config_templates/default_test.cfg
index cd4bd32e68..f9279cce54 100644
--- a/airflow/config_templates/default_test.cfg
+++ b/airflow/config_templates/default_test.cfg
@@ -51,6 +51,7 @@ enable_xcom_pickling = False
killed_task_cleanup_time = 5
secure_mode = False
hostname_callable = socket:getfqdn
+worker_precheck = False
[cli]
api_client = airflow.api.client.local_client
@@ -64,6 +65,7 @@ default_owner = airflow
[hive]
default_hive_mapred_queue = airflow
+mapred_job_name_template = Airflow HiveOperator task for {{hostname}}.{{dag_id}}.{{task_id}}.{{execution_date}}
[webserver]
base_url = http://localhost:8080
@@ -95,6 +97,7 @@ result_backend = db+mysql://airflow:airflow@localhost:3306/airflow
flower_host = 0.0.0.0
flower_port = 5555
default_queue = default
+sync_parallelism = 0
[mesos]
master = localhost:5050
diff --git a/airflow/config_templates/default_webserver_config.py b/airflow/config_templates/default_webserver_config.py
index ba33d81dec..e61c7e1a45 100644
--- a/airflow/config_templates/default_webserver_config.py
+++ b/airflow/config_templates/default_webserver_config.py
@@ -89,3 +89,13 @@
# { 'name': 'AOL', 'url': 'http://openid.aol.com/<username>' },
# { 'name': 'Flickr', 'url': 'http://www.flickr.com/<username>' },
# { 'name': 'MyOpenID', 'url': 'https://www.myopenid.com' }]
+
+# ----------------------------------------------------
+# Theme CONFIG
+# ----------------------------------------------------
+# Flask App Builder comes up with a number of predefined themes
+# that you can use for Apache Airflow.
+# http://flask-appbuilder.readthedocs.io/en/latest/customizing.html#changing-themes
+# Please make sure to remove "navbar_color" configuration from airflow.cfg
+# in order to fully utilize the theme. (or use that property in conjunction with theme)
+# APP_THEME = "bootstrap-theme.css" # default bootstrap
diff --git a/airflow/configuration.py b/airflow/configuration.py
index 2ee453fd7d..8768d70f23 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -22,6 +22,7 @@
from __future__ import print_function
from __future__ import unicode_literals
+from base64 import b64encode
from builtins import str
from collections import OrderedDict
import copy
@@ -36,7 +37,7 @@
import warnings
from backports.configparser import ConfigParser
-from zope.deprecation import deprecated as _deprecated
+from zope.deprecation import deprecated
from airflow.exceptions import AirflowConfigException
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -101,15 +102,20 @@ def run_command(command):
return output
-_templates_dir = os.path.join(os.path.dirname(__file__), 'config_templates')
-with open(os.path.join(_templates_dir, 'default_airflow.cfg')) as f:
- DEFAULT_CONFIG = f.read()
+def _read_default_config_file(file_name):
+ templates_dir = os.path.join(os.path.dirname(__file__), 'config_templates')
+ file_path = os.path.join(templates_dir, file_name)
if six.PY2:
- DEFAULT_CONFIG = DEFAULT_CONFIG.decode('utf-8')
-with open(os.path.join(_templates_dir, 'default_test.cfg')) as f:
- TEST_CONFIG = f.read()
- if six.PY2:
- TEST_CONFIG = TEST_CONFIG.decode('utf-8')
+ with open(file_path) as f:
+ config = f.read()
+ return config.decode('utf-8')
+ else:
+ with open(file_path, encoding='utf-8') as f:
+ return f.read()
+
+
+DEFAULT_CONFIG = _read_default_config_file('default_airflow.cfg')
+TEST_CONFIG = _read_default_config_file('default_test.cfg')
class AirflowConfigParser(ConfigParser):
@@ -121,9 +127,34 @@ class AirflowConfigParser(ConfigParser):
('core', 'sql_alchemy_conn'),
('core', 'fernet_key'),
('celery', 'broker_url'),
- ('celery', 'result_backend')
+ ('celery', 'result_backend'),
+ # Todo: remove this in Airflow 1.11
+ ('celery', 'celery_result_backend'),
+ ('atlas', 'password'),
+ ('smtp', 'smtp_password'),
+ ('ldap', 'bind_password'),
+ ('kubernetes', 'git_password'),
}
+ # A two-level mapping of (section -> new_name -> old_name). When reading
+ # new_name, the old_name will be checked to see if it exists. If it does a
+ # DeprecationWarning will be issued and the old name will be used instead
+ deprecated_options = {
+ 'celery': {
+ # Remove these keys in Airflow 1.11
+ 'worker_concurrency': 'celeryd_concurrency',
+ 'result_backend': 'celery_result_backend',
+ 'broker_url': 'celery_broker_url',
+ 'ssl_active': 'celery_ssl_active',
+ 'ssl_cert': 'celery_ssl_cert',
+ 'ssl_key': 'celery_ssl_key',
+ }
+ }
+ deprecation_format_string = (
+ 'The {old} option in [{section}] has been renamed to {new} - the old '
+ 'setting has been used, but please update your config.'
+ )
+
def __init__(self, default_config=None, *args, **kwargs):
super(AirflowConfigParser, self).__init__(*args, **kwargs)
@@ -181,10 +212,17 @@ def get(self, section, key, **kwargs):
section = str(section).lower()
key = str(key).lower()
+ deprecated_name = self.deprecated_options.get(section, {}).get(key, None)
+
# first check environment variables
option = self._get_env_var_option(section, key)
if option is not None:
return option
+ if deprecated_name:
+ option = self._get_env_var_option(section, deprecated_name)
+ if option is not None:
+ self._warn_deprecate(section, key, deprecated_name)
+ return option
# ...then the config file
if super(AirflowConfigParser, self).has_option(section, key):
@@ -192,11 +230,24 @@ def get(self, section, key, **kwargs):
# separate the config from default config.
return expand_env_var(
super(AirflowConfigParser, self).get(section, key, **kwargs))
+ if deprecated_name:
+ if super(AirflowConfigParser, self).has_option(section, deprecated_name):
+ self._warn_deprecate(section, key, deprecated_name)
+ return expand_env_var(super(AirflowConfigParser, self).get(
+ section,
+ deprecated_name,
+ **kwargs
+ ))
# ...then commands
option = self._get_cmd_option(section, key)
if option:
return option
+ if deprecated_name:
+ option = self._get_cmd_option(section, deprecated_name)
+ if option:
+ self._warn_deprecate(section, key, deprecated_name)
+ return option
# ...then the default config
if self.defaults.has_option(section, key):
@@ -352,6 +403,17 @@ def load_test_config(self):
# then read any "custom" test settings
self.read(TEST_CONFIG_FILE)
+ def _warn_deprecate(self, section, key, deprecated_name):
+ warnings.warn(
+ self.deprecation_format_string.format(
+ old=deprecated_name,
+ new=key,
+ section=section,
+ ),
+ DeprecationWarning,
+ stacklevel=3,
+ )
+
def mkdir_p(path):
try:
@@ -422,6 +484,8 @@ def parameterized_config(template):
else:
FERNET_KEY = ''
+SECRET_KEY = b64encode(os.urandom(16)).decode('utf-8')
+
TEMPLATE_START = (
'# ----------------------- TEMPLATE BEGINS HERE -----------------------')
if not os.path.isfile(TEST_CONFIG_FILE):
@@ -451,8 +515,7 @@ def parameterized_config(template):
if conf.getboolean('webserver', 'rbac'):
- with open(os.path.join(_templates_dir, 'default_webserver_config.py')) as f:
- DEFAULT_WEBSERVER_CONFIG = f.read()
+ DEFAULT_WEBSERVER_CONFIG = _read_default_config_file('default_webserver_config.py')
WEBSERVER_CONFIG = AIRFLOW_HOME + '/webserver_config.py'
@@ -479,7 +542,7 @@ def parameterized_config(template):
for func in [load_test_config, get, getboolean, getfloat, getint, has_option,
remove_option, as_dict, set]:
- _deprecated(
+ deprecated(
func,
"Accessing configuration method '{f.__name__}' directly from "
"the configuration module is deprecated. Please access the "
diff --git a/airflow/contrib/auth/backends/github_enterprise_auth.py b/airflow/contrib/auth/backends/github_enterprise_auth.py
index 5196725156..e1b8b13342 100644
--- a/airflow/contrib/auth/backends/github_enterprise_auth.py
+++ b/airflow/contrib/auth/backends/github_enterprise_auth.py
@@ -1,36 +1,30 @@
-# Copyright 2015 Matthew Pelland (matt@pelland.io)
+# -*- coding: utf-8 -*-
#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
+# See the NOTICE file distributed with this work for additional information
+# regarding copyright ownership.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
import flask_login
# Need to expose these downstream
-# pylint: disable=unused-import
-from flask_login import (current_user,
- logout_user,
- login_required,
- login_user)
-# pylint: enable=unused-import
+# flake8: noqa: F401
+from flask_login import current_user, logout_user, login_required, login_user
from flask import url_for, redirect, request
from flask_oauthlib.client import OAuth
-from airflow import models, configuration, settings
+from airflow import models, configuration
from airflow.configuration import AirflowConfigException
from airflow.utils.db import provide_session
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -47,14 +41,17 @@ class GHEUser(models.User):
def __init__(self, user):
self.user = user
+ @property
def is_active(self):
"""Required by flask_login"""
return True
+ @property
def is_authenticated(self):
"""Required by flask_login"""
return True
+ @property
def is_anonymous(self):
"""Required by flask_login"""
return False
diff --git a/airflow/contrib/auth/backends/google_auth.py b/airflow/contrib/auth/backends/google_auth.py
index d1a35791db..bc7d552f59 100644
--- a/airflow/contrib/auth/backends/google_auth.py
+++ b/airflow/contrib/auth/backends/google_auth.py
@@ -1,4 +1,4 @@
-# Copyright 2016 Ananya Mishra (am747@cornell.edu)
+# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -19,18 +19,14 @@
import flask_login
# Need to expose these downstream
-# pylint: disable=unused-import
-from flask_login import (current_user,
- logout_user,
- login_required,
- login_user)
-# pylint: enable=unused-import
+# flake8: noqa: F401
+from flask_login import current_user, logout_user, login_required, login_user
from flask import url_for, redirect, request
from flask_oauthlib.client import OAuth
-from airflow import models, configuration, settings
+from airflow import models, configuration
from airflow.utils.db import provide_session
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -46,14 +42,17 @@ class GoogleUser(models.User):
def __init__(self, user):
self.user = user
+ @property
def is_active(self):
"""Required by flask_login"""
return True
+ @property
def is_authenticated(self):
"""Required by flask_login"""
return True
+ @property
def is_anonymous(self):
"""Required by flask_login"""
return False
diff --git a/airflow/contrib/auth/backends/kerberos_auth.py b/airflow/contrib/auth/backends/kerberos_auth.py
index 08be299a19..e3500c9317 100644
--- a/airflow/contrib/auth/backends/kerberos_auth.py
+++ b/airflow/contrib/auth/backends/kerberos_auth.py
@@ -21,18 +21,16 @@
import flask_login
from flask_login import current_user
from flask import flash
-from wtforms import (
- Form, PasswordField, StringField)
+from wtforms import Form, PasswordField, StringField
from wtforms.validators import InputRequired
# pykerberos should be used as it verifies the KDC, the "kerberos" module does not do so
# and make it possible to spoof the KDC
import kerberos
-import airflow.security.utils as utils
+from airflow.security import utils
from flask import url_for, redirect
-from airflow import settings
from airflow import models
from airflow import configuration
from airflow.utils.db import provide_session
@@ -73,14 +71,17 @@ def authenticate(username, password):
return
+ @property
def is_active(self):
"""Required by flask_login"""
return True
+ @property
def is_authenticated(self):
"""Required by flask_login"""
return True
+ @property
def is_anonymous(self):
"""Required by flask_login"""
return False
@@ -110,7 +111,7 @@ def load_user(userid, session=None):
@provide_session
def login(self, request, session=None):
- if current_user.is_authenticated():
+ if current_user.is_authenticated:
flash("You are already logged in")
return redirect(url_for('index'))
diff --git a/airflow/contrib/auth/backends/ldap_auth.py b/airflow/contrib/auth/backends/ldap_auth.py
index eefaa1263b..fefc389e48 100644
--- a/airflow/contrib/auth/backends/ldap_auth.py
+++ b/airflow/contrib/auth/backends/ldap_auth.py
@@ -19,13 +19,12 @@
from future.utils import native
import flask_login
-from flask_login import login_required, current_user, logout_user
+from flask_login import login_required, current_user, logout_user # noqa: F401
from flask import flash
-from wtforms import (
- Form, PasswordField, StringField)
+from wtforms import Form, PasswordField, StringField
from wtforms.validators import InputRequired
-from ldap3 import Server, Connection, Tls, LEVEL, SUBTREE, BASE
+from ldap3 import Server, Connection, Tls, LEVEL, SUBTREE
import ssl
from flask import url_for, redirect
@@ -62,7 +61,7 @@ def get_ldap_connection(dn=None, password=None):
cacert = configuration.conf.get("ldap", "cacert")
tls_configuration = Tls(validate=ssl.CERT_REQUIRED, ca_certs_file=cacert)
use_ssl = True
- except:
+ except Exception:
pass
server = Server(configuration.conf.get("ldap", "uri"), use_ssl, tls_configuration)
@@ -94,7 +93,7 @@ def groups_user(conn, search_base, user_filter, user_name_att, username):
search_filter = "(&({0})({1}={2}))".format(user_filter, user_name_att, username)
try:
memberof_attr = configuration.conf.get("ldap", "group_member_attr")
- except:
+ except Exception:
memberof_attr = "memberOf"
res = conn.search(native(search_base), native(search_filter),
attributes=[native(memberof_attr)])
@@ -236,14 +235,17 @@ def try_login(username, password):
log.info("Password incorrect for user %s", username)
raise AuthenticationError("Invalid username or password")
+ @property
def is_active(self):
"""Required by flask_login"""
return True
+ @property
def is_authenticated(self):
"""Required by flask_login"""
return True
+ @property
def is_anonymous(self):
"""Required by flask_login"""
return False
@@ -274,7 +276,7 @@ def load_user(userid, session=None):
@provide_session
def login(self, request, session=None):
- if current_user.is_authenticated():
+ if current_user.is_authenticated:
flash("You are already logged in")
return redirect(url_for('admin.index'))
diff --git a/airflow/contrib/auth/backends/password_auth.py b/airflow/contrib/auth/backends/password_auth.py
index 879aaa142a..55f5daf8fd 100644
--- a/airflow/contrib/auth/backends/password_auth.py
+++ b/airflow/contrib/auth/backends/password_auth.py
@@ -71,14 +71,17 @@ def password(self, plaintext):
def authenticate(self, plaintext):
return check_password_hash(self._password, plaintext)
+ @property
def is_active(self):
"""Required by flask_login"""
return True
+ @property
def is_authenticated(self):
"""Required by flask_login"""
return True
+ @property
def is_anonymous(self):
"""Required by flask_login"""
return False
@@ -137,7 +140,7 @@ def authenticate(session, username, password):
@provide_session
def login(self, request, session=None):
- if current_user.is_authenticated():
+ if current_user.is_authenticated:
flash("You are already logged in")
return redirect(url_for('admin.index'))
diff --git a/.codecov.yml b/airflow/contrib/example_dags/__init__.py
similarity index 96%
rename from .codecov.yml
rename to airflow/contrib/example_dags/__init__.py
index 4094f35dcc..114d189da1 100644
--- a/.codecov.yml
+++ b/airflow/contrib/example_dags/__init__.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -6,14 +7,12 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-# keep default
diff --git a/airflow/contrib/example_dags/example_databricks_operator.py b/airflow/contrib/example_dags/example_databricks_operator.py
index bc827d465b..79f947ba1c 100644
--- a/airflow/contrib/example_dags/example_databricks_operator.py
+++ b/airflow/contrib/example_dags/example_databricks_operator.py
@@ -32,7 +32,7 @@
# the spark jar task will NOT run until the notebook task completes
# successfully.
#
-# The definition of a succesful run is if the run has a result_state of "SUCCESS".
+# The definition of a successful run is if the run has a result_state of "SUCCESS".
# For more information about the state of a run refer to
# https://docs.databricks.com/api/latest/jobs.html#runstate
diff --git a/airflow/contrib/example_dags/example_gcp_compute.py b/airflow/contrib/example_dags/example_gcp_compute.py
new file mode 100644
index 0000000000..e4abe2e152
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_compute.py
@@ -0,0 +1,108 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that starts, stops and sets the machine type of a Google Compute
+Engine instance.
+
+This DAG relies on the following Airflow variables
+https://airflow.apache.org/concepts.html#variables
+* PROJECT_ID - Google Cloud Platform project where the Compute Engine instance exists.
+* LOCATION - Google Cloud Platform zone where the instance exists.
+* INSTANCE - Name of the Compute Engine instance.
+* SHORT_MACHINE_TYPE_NAME - Machine type resource name to set, e.g. 'n1-standard-1'.
+ See https://cloud.google.com/compute/docs/machine-types
+"""
+
+import datetime
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcp_compute_operator import GceInstanceStartOperator, \
+ GceInstanceStopOperator, GceSetMachineTypeOperator
+
+# [START howto_operator_gce_args]
+PROJECT_ID = models.Variable.get('PROJECT_ID', '')
+LOCATION = models.Variable.get('LOCATION', '')
+INSTANCE = models.Variable.get('INSTANCE', '')
+SHORT_MACHINE_TYPE_NAME = models.Variable.get('SHORT_MACHINE_TYPE_NAME', '')
+SET_MACHINE_TYPE_BODY = {
+ 'machineType': 'zones/{}/machineTypes/{}'.format(LOCATION, SHORT_MACHINE_TYPE_NAME)
+}
+
+default_args = {
+ 'start_date': airflow.utils.dates.days_ago(1)
+}
+# [END howto_operator_gce_args]
+
+with models.DAG(
+ 'example_gcp_compute',
+ default_args=default_args,
+ schedule_interval=datetime.timedelta(days=1)
+) as dag:
+ # [START howto_operator_gce_start]
+ gce_instance_start = GceInstanceStartOperator(
+ project_id=PROJECT_ID,
+ zone=LOCATION,
+ resource_id=INSTANCE,
+ task_id='gcp_compute_start_task'
+ )
+ # [END howto_operator_gce_start]
+ # Duplicate start for idempotence testing
+ gce_instance_start2 = GceInstanceStartOperator(
+ project_id=PROJECT_ID,
+ zone=LOCATION,
+ resource_id=INSTANCE,
+ task_id='gcp_compute_start_task2'
+ )
+ # [START howto_operator_gce_stop]
+ gce_instance_stop = GceInstanceStopOperator(
+ project_id=PROJECT_ID,
+ zone=LOCATION,
+ resource_id=INSTANCE,
+ task_id='gcp_compute_stop_task'
+ )
+ # [END howto_operator_gce_stop]
+ # Duplicate stop for idempotence testing
+ gce_instance_stop2 = GceInstanceStopOperator(
+ project_id=PROJECT_ID,
+ zone=LOCATION,
+ resource_id=INSTANCE,
+ task_id='gcp_compute_stop_task2'
+ )
+ # [START howto_operator_gce_set_machine_type]
+ gce_set_machine_type = GceSetMachineTypeOperator(
+ project_id=PROJECT_ID,
+ zone=LOCATION,
+ resource_id=INSTANCE,
+ body=SET_MACHINE_TYPE_BODY,
+ task_id='gcp_compute_set_machine_type'
+ )
+ # [END howto_operator_gce_set_machine_type]
+ # Duplicate set machine type for idempotence testing
+ gce_set_machine_type2 = GceSetMachineTypeOperator(
+ project_id=PROJECT_ID,
+ zone=LOCATION,
+ resource_id=INSTANCE,
+ body=SET_MACHINE_TYPE_BODY,
+ task_id='gcp_compute_set_machine_type2'
+ )
+
+ gce_instance_start >> gce_instance_start2 >> gce_instance_stop >> \
+ gce_instance_stop2 >> gce_set_machine_type >> gce_set_machine_type2
diff --git a/airflow/contrib/example_dags/example_gcp_function_delete.py b/airflow/contrib/example_dags/example_gcp_function_delete.py
new file mode 100644
index 0000000000..30f5369af6
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_function_delete.py
@@ -0,0 +1,58 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that deletes a Google Cloud Function.
+This DAG relies on the following Airflow variables
+https://airflow.apache.org/concepts.html#variables
+* PROJECT_ID - Google Cloud Project where the Cloud Function exists.
+* LOCATION - Google Cloud Functions region where the function exists.
+* ENTRYPOINT - Name of the executable function in the source code.
+"""
+
+import datetime
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcp_function_operator import GcfFunctionDeleteOperator
+
+# [START howto_operator_gcf_delete_args]
+PROJECT_ID = models.Variable.get('PROJECT_ID', '')
+LOCATION = models.Variable.get('LOCATION', '')
+ENTRYPOINT = models.Variable.get('ENTRYPOINT', '')
+# A fully-qualified name of the function to delete
+
+FUNCTION_NAME = 'projects/{}/locations/{}/functions/{}'.format(PROJECT_ID, LOCATION,
+ ENTRYPOINT)
+default_args = {
+ 'start_date': airflow.utils.dates.days_ago(1)
+}
+# [END howto_operator_gcf_delete_args]
+
+with models.DAG(
+ 'example_gcp_function_delete',
+ default_args=default_args,
+ schedule_interval=datetime.timedelta(days=1)
+) as dag:
+ # [START howto_operator_gcf_delete]
+ t1 = GcfFunctionDeleteOperator(
+ task_id="gcf_delete_task",
+ name=FUNCTION_NAME
+ )
+ # [END howto_operator_gcf_delete]
diff --git a/airflow/contrib/example_dags/example_gcp_function_deploy_delete.py b/airflow/contrib/example_dags/example_gcp_function_deploy_delete.py
new file mode 100644
index 0000000000..a0e44957b9
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_function_deploy_delete.py
@@ -0,0 +1,113 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that creates a Google Cloud Function and then deletes it.
+
+This DAG relies on the following Airflow variables
+https://airflow.apache.org/concepts.html#variables
+* PROJECT_ID - Google Cloud Project to use for the Cloud Function.
+* LOCATION - Google Cloud Functions region where the function should be
+ created.
+* SOURCE_ARCHIVE_URL - Path to the zipped source in Google Cloud Storage
+or
+ * SOURCE_UPLOAD_URL - Generated upload URL for the zipped source
+ or
+ * ZIP_PATH - Local path to the zipped source archive
+or
+* SOURCE_REPOSITORY - The URL pointing to the hosted repository where the function is
+defined in a supported Cloud Source Repository URL format
+https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions#SourceRepository
+* ENTRYPOINT - Name of the executable function in the source code.
+"""
+
+import datetime
+
+from airflow import models
+from airflow.contrib.operators.gcp_function_operator \
+ import GcfFunctionDeployOperator, GcfFunctionDeleteOperator
+from airflow.utils import dates
+
+# [START howto_operator_gcf_deploy_variables]
+PROJECT_ID = models.Variable.get('PROJECT_ID', '')
+LOCATION = models.Variable.get('LOCATION', '')
+SOURCE_ARCHIVE_URL = models.Variable.get('SOURCE_ARCHIVE_URL', '')
+SOURCE_UPLOAD_URL = models.Variable.get('SOURCE_UPLOAD_URL', '')
+SOURCE_REPOSITORY = models.Variable.get('SOURCE_REPOSITORY', '')
+ZIP_PATH = models.Variable.get('ZIP_PATH', '')
+ENTRYPOINT = models.Variable.get('ENTRYPOINT', '')
+FUNCTION_NAME = 'projects/{}/locations/{}/functions/{}'.format(PROJECT_ID, LOCATION,
+ ENTRYPOINT)
+RUNTIME = 'nodejs6'
+VALIDATE_BODY = models.Variable.get('VALIDATE_BODY', True)
+
+# [END howto_operator_gcf_deploy_variables]
+
+# [START howto_operator_gcf_deploy_body]
+body = {
+ "name": FUNCTION_NAME,
+ "entryPoint": ENTRYPOINT,
+ "runtime": RUNTIME,
+ "httpsTrigger": {}
+}
+# [END howto_operator_gcf_deploy_body]
+
+# [START howto_operator_gcf_deploy_args]
+default_args = {
+ 'start_date': dates.days_ago(1),
+ 'project_id': PROJECT_ID,
+ 'location': LOCATION,
+ 'body': body,
+ 'validate_body': VALIDATE_BODY
+}
+# [END howto_operator_gcf_deploy_args]
+
+# [START howto_operator_gcf_deploy_variants]
+if SOURCE_ARCHIVE_URL:
+ body['sourceArchiveUrl'] = SOURCE_ARCHIVE_URL
+elif SOURCE_REPOSITORY:
+ body['sourceRepository'] = {
+ 'url': SOURCE_REPOSITORY
+ }
+elif ZIP_PATH:
+ body['sourceUploadUrl'] = ''
+ default_args['zip_path'] = ZIP_PATH
+elif SOURCE_UPLOAD_URL:
+ body['sourceUploadUrl'] = SOURCE_UPLOAD_URL
+else:
+ raise Exception("Please provide one of the source_code parameters")
+# [END howto_operator_gcf_deploy_variants]
+
+
+with models.DAG(
+ 'example_gcp_function_deploy_delete',
+ default_args=default_args,
+ schedule_interval=datetime.timedelta(days=1)
+) as dag:
+ # [START howto_operator_gcf_deploy]
+ deploy_task = GcfFunctionDeployOperator(
+ task_id="gcf_deploy_task",
+ name=FUNCTION_NAME
+ )
+ # [END howto_operator_gcf_deploy]
+ delete_task = GcfFunctionDeleteOperator(
+ task_id="gcf_delete_task",
+ name=FUNCTION_NAME
+ )
+ deploy_task >> delete_task
diff --git a/airflow/contrib/example_dags/example_gcs_to_bq_operator.py b/airflow/contrib/example_dags/example_gcs_to_bq_operator.py
new file mode 100644
index 0000000000..ee9fe09391
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcs_to_bq_operator.py
@@ -0,0 +1,63 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import airflow
+try:
+ from airflow.contrib.operators import gcs_to_bq
+except ImportError:
+ gcs_to_bq = None
+from airflow import models
+from airflow.operators import bash_operator
+
+
+if gcs_to_bq is not None:
+ args = {
+ 'owner': 'airflow',
+ 'start_date': airflow.utils.dates.days_ago(2)
+ }
+
+ dag = models.DAG(
+ dag_id='example_gcs_to_bq_operator', default_args=args,
+ schedule_interval=None)
+
+ create_test_dataset = bash_operator.BashOperator(
+ task_id='create_airflow_test_dataset',
+ bash_command='bq mk airflow_test',
+ dag=dag)
+
+ # [START howto_operator_gcs_to_bq]
+ load_csv = gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
+ task_id='gcs_to_bq_example',
+ bucket='cloud-samples-data',
+ source_objects=['bigquery/us-states/us-states.csv'],
+ destination_project_dataset_table='airflow_test.gcs_to_bq_table',
+ schema_fields=[
+ {'name': 'name', 'type': 'STRING', 'mode': 'NULLABLE'},
+ {'name': 'post_abbr', 'type': 'STRING', 'mode': 'NULLABLE'},
+ ],
+ write_disposition='WRITE_TRUNCATE',
+ dag=dag)
+ # [END howto_operator_gcs_to_bq]
+
+ delete_test_dataset = bash_operator.BashOperator(
+ task_id='delete_airflow_test_dataset',
+ bash_command='bq rm -rf airflow_test',
+ dag=dag)
+
+ create_test_dataset >> load_csv >> delete_test_dataset
diff --git a/scripts/ci/run_tests.sh b/airflow/contrib/example_dags/example_kubernetes_annotation.py
old mode 100755
new mode 100644
similarity index 54%
rename from scripts/ci/run_tests.sh
rename to airflow/contrib/example_dags/example_kubernetes_annotation.py
index d5e7655803..058baf6990
--- a/scripts/ci/run_tests.sh
+++ b/airflow/contrib/example_dags/example_kubernetes_annotation.py
@@ -1,5 +1,4 @@
-#!/usr/bin/env bash
-
+# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -8,40 +7,41 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-set -o verbose
-
-if [ -z "$HADOOP_HOME" ]; then
- echo "HADOOP_HOME not set - abort" >&2
- exit 1
-fi
-
-echo "Using ${HADOOP_DISTRO} distribution of Hadoop from ${HADOOP_HOME}"
-
-pwd
-
-mkdir ~/airflow/
-
-if [ "${TRAVIS}" ]; then
- echo "Using travis airflow.cfg"
- DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
- cp -f ${DIR}/airflow_travis.cfg ~/airflow/unittests.cfg
-
- ROOTDIR="$(dirname $(dirname $DIR))"
- export AIRFLOW__CORE__DAGS_FOLDER="$ROOTDIR/tests/dags"
-
- # kdc init happens in setup_kdc.sh
- kinit -kt ${KRB5_KTNAME} airflow
-fi
-
-echo Backend: $AIRFLOW__CORE__SQL_ALCHEMY_CONN
-./run_unit_tests.sh $@
+from __future__ import print_function
+import airflow
+from airflow.operators.python_operator import PythonOperator
+from airflow.models import DAG
+
+args = {
+ 'owner': 'airflow',
+ 'start_date': airflow.utils.dates.days_ago(2)
+}
+
+dag = DAG(
+ dag_id='example_kubernetes_annotation', default_args=args,
+ schedule_interval=None
+)
+
+
+def print_stuff():
+ print("annotated!")
+
+
+# You can use annotations on your kubernetes pods!
+start_task = PythonOperator(
+ task_id="start_task", python_callable=print_stuff, dag=dag,
+ executor_config={
+ "KubernetesExecutor": {
+ "annotations": {"test": "annotation"}
+ }
+ }
+)
diff --git a/airflow/example_dags/example_kubernetes_executor.py b/airflow/contrib/example_dags/example_kubernetes_executor.py
similarity index 100%
rename from airflow/example_dags/example_kubernetes_executor.py
rename to airflow/contrib/example_dags/example_kubernetes_executor.py
diff --git a/airflow/example_dags/example_kubernetes_operator.py b/airflow/contrib/example_dags/example_kubernetes_operator.py
similarity index 91%
rename from airflow/example_dags/example_kubernetes_operator.py
rename to airflow/contrib/example_dags/example_kubernetes_operator.py
index 92d73c5d33..4b3f54bd04 100644
--- a/airflow/example_dags/example_kubernetes_operator.py
+++ b/airflow/contrib/example_dags/example_kubernetes_operator.py
@@ -25,7 +25,7 @@
try:
# Kubernetes is optional, so not available in vanilla Airflow
- # pip install airflow[kubernetes]
+ # pip install apache-airflow[kubernetes]
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
args = {
@@ -48,9 +48,10 @@
in_cluster=False,
task_id="task",
get_logs=True,
- dag=dag)
+ dag=dag,
+ is_delete_operator_pod=False)
except ImportError as e:
log.warn("Could not import KubernetesPodOperator: " + str(e))
log.warn("Install kubernetes dependencies with: "
- " pip install airflow['kubernetes']")
+ " pip install apache-airflow[kubernetes]")
diff --git a/airflow/contrib/example_dags/example_twitter_README.md b/airflow/contrib/example_dags/example_twitter_README.md
index 319eac39f6..0f3aededd4 100644
--- a/airflow/contrib/example_dags/example_twitter_README.md
+++ b/airflow/contrib/example_dags/example_twitter_README.md
@@ -7,11 +7,11 @@
***Overview:*** At first, we need tasks that will get the tweets of our interest and save them on the hard-disk. Then, we need subsequent tasks that will clean and analyze the tweets. Then we want to store these files into HDFS, and load them into a Data Warehousing platform like Hive or HBase. The main reason we have selected Hive here is because it gives us a familiar SQL like interface, and makes our life of writing different queries a lot easier. Finally, the DAG needs to store a summarized result to a traditional database, i.e. MySQL or PostgreSQL, which is used by a reporting or business intelligence application. In other words, we basically want to achieve the following steps:
1. Fetch Tweets
-2. Clean Tweets
-3. Analyze Tweets
-4. Put Tweets to HDFS
-5. Load data to Hive
-6. Save Summary to MySQL
+1. Clean Tweets
+1. Analyze Tweets
+1. Put Tweets to HDFS
+1. Load data to Hive
+1. Save Summary to MySQL
***Screenshot:***
<img src="http://i.imgur.com/rRpSO12.png" width="99%"/>
@@ -21,16 +21,18 @@
The python functions here are just placeholders. In case you are interested to actually make this DAG fully functional, first start with filling out the scripts as separate files and importing them into the DAG with absolute or relative import. My approach was to store the retrieved data in memory using Pandas dataframe first, and then use the built in method to save the CSV file on hard-disk.
The eight different CSV files are then put into eight different folders within HDFS. Each of the newly inserted files are then loaded into eight different external hive tables. Hive tables can be external or internal. In this case, we are inserting the data right into the table, and so we are making our tables internal. Each file is inserted into the respected Hive table named after the twitter channel, i.e. toTwitter_A or fromTwitter_A. It is also important to note that when we created the tables, we facilitated for partitioning by date using the variable dt and declared comma as the row deliminator. The partitioning is very handy and ensures our query execution time remains constant even with growing volume of data.
As most probably these folders and hive tables doesn't exist in your system, you will get an error for these tasks within the DAG. If you rebuild a function DAG from this example, make sure those folders and hive tables exists. When you create the table, keep the consideration of table partitioning and declaring comma as the row deliminator in your mind. Furthermore, you may also need to skip headers on each read and ensure that the user under which you have Airflow running has the right permission access. Below is a sample HQL snippet on creating such table:
+
```
CREATE TABLE toTwitter_A(id BIGINT, id_str STRING
- created_at STRING, text STRING)
- PARTITIONED BY (dt STRING)
- ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
- STORED AS TEXTFILE;
- alter table toTwitter_A SET serdeproperties ('skip.header.line.count' = '1');
+ created_at STRING, text STRING)
+ PARTITIONED BY (dt STRING)
+ ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
+ STORED AS TEXTFILE;
+ alter table toTwitter_A SET serdeproperties ('skip.header.line.count' = '1');
```
+
When you review the code for the DAG, you will notice that these tasks are generated using for loop. These two for loops could be combined into one loop. However, in most cases, you will be running different analysis on your incoming incoming and outgoing tweets, and hence they are kept separated in this example.
Final step is a running the broker script, brokerapi.py, which will run queries in Hive and store the summarized data to MySQL in our case. To connect to Hive, pyhs2 library is extremely useful and easy to use. To insert data into MySQL from Python, sqlalchemy is also a good one to use.
-I hope you find this tutorial useful. If you have question feel free to ask me on [Twitter](https://twitter.com/EkhtiarSyed) or via the live Airflow chatroom room in [Gitter](https://gitter.im/airbnb/airflow).<p>
+I hope you find this tutorial useful. If you have question feel free to ask me on [Twitter](https://twitter.com/EkhtiarSyed) or via the live Airflow chatroom room in [Gitter](https://gitter.im/apache/incubator-airflow).<p>
-Ekhtiar Syed
Last Update: 8-April-2016
diff --git a/airflow/contrib/executors/kubernetes_executor.py b/airflow/contrib/executors/kubernetes_executor.py
index 65053bd56b..de1f9f4235 100644
--- a/airflow/contrib/executors/kubernetes_executor.py
+++ b/airflow/contrib/executors/kubernetes_executor.py
@@ -39,7 +39,8 @@
class KubernetesExecutorConfig:
def __init__(self, image=None, image_pull_policy=None, request_memory=None,
request_cpu=None, limit_memory=None, limit_cpu=None,
- gcp_service_account_key=None):
+ gcp_service_account_key=None, node_selectors=None, affinity=None,
+ annotations=None):
self.image = image
self.image_pull_policy = image_pull_policy
self.request_memory = request_memory
@@ -47,13 +48,18 @@ def __init__(self, image=None, image_pull_policy=None, request_memory=None,
self.limit_memory = limit_memory
self.limit_cpu = limit_cpu
self.gcp_service_account_key = gcp_service_account_key
+ self.node_selectors = node_selectors
+ self.affinity = affinity
+ self.annotations = annotations
def __repr__(self):
return "{}(image={}, image_pull_policy={}, request_memory={}, request_cpu={}, " \
- "limit_memory={}, limit_cpu={}, gcp_service_account_key={})" \
+ "limit_memory={}, limit_cpu={}, gcp_service_account_key={}, " \
+ "node_selectors={}, affinity={}, annotations={})" \
.format(KubernetesExecutorConfig.__name__, self.image, self.image_pull_policy,
self.request_memory, self.request_cpu, self.limit_memory,
- self.limit_cpu, self.gcp_service_account_key)
+ self.limit_cpu, self.gcp_service_account_key, self.node_selectors,
+ self.affinity, self.annotations)
@staticmethod
def from_dict(obj):
@@ -73,7 +79,10 @@ def from_dict(obj):
request_cpu=namespaced.get('request_cpu', None),
limit_memory=namespaced.get('limit_memory', None),
limit_cpu=namespaced.get('limit_cpu', None),
- gcp_service_account_key=namespaced.get('gcp_service_account_key', None)
+ gcp_service_account_key=namespaced.get('gcp_service_account_key', None),
+ node_selectors=namespaced.get('node_selectors', None),
+ affinity=namespaced.get('affinity', None),
+ annotations=namespaced.get('annotations', {}),
)
def as_dict(self):
@@ -84,7 +93,10 @@ def as_dict(self):
'request_cpu': self.request_cpu,
'limit_memory': self.limit_memory,
'limit_cpu': self.limit_cpu,
- 'gcp_service_account_key': self.gcp_service_account_key
+ 'gcp_service_account_key': self.gcp_service_account_key,
+ 'node_selectors': self.node_selectors,
+ 'affinity': self.affinity,
+ 'annotations': self.annotations,
}
@@ -103,11 +115,14 @@ def __init__(self):
self.kubernetes_section, 'worker_container_repository')
self.worker_container_tag = configuration.get(
self.kubernetes_section, 'worker_container_tag')
+ self.worker_dags_folder = configuration.get(
+ self.kubernetes_section, 'worker_dags_folder')
self.kube_image = '{}:{}'.format(
self.worker_container_repository, self.worker_container_tag)
self.kube_image_pull_policy = configuration.get(
self.kubernetes_section, "worker_container_image_pull_policy"
)
+ self.kube_node_selectors = configuration_dict.get('kubernetes_node_selectors', {})
self.delete_worker_pods = conf.getboolean(
self.kubernetes_section, 'delete_worker_pods')
@@ -418,7 +433,7 @@ def _label_safe_datestring_to_datetime(string):
"_", let's
replace ":" with "_"
- :param string: string
+ :param string: str
:return: datetime.datetime object
"""
return parser.parse(string.replace('_plus_', '+').replace("_", ":"))
diff --git a/airflow/contrib/executors/mesos_executor.py b/airflow/contrib/executors/mesos_executor.py
index ff974ffc3c..0609d71cf2 100644
--- a/airflow/contrib/executors/mesos_executor.py
+++ b/airflow/contrib/executors/mesos_executor.py
@@ -162,7 +162,7 @@ def resourceOffers(self, driver, offers):
command = mesos_pb2.CommandInfo()
command.shell = True
- command.value = cmd
+ command.value = " ".join(cmd)
task.command.MergeFrom(command)
# If docker image for airflow is specified in config then pull that
diff --git a/airflow/contrib/hooks/__init__.py b/airflow/contrib/hooks/__init__.py
index 95f9892d89..b7f8352944 100644
--- a/airflow/contrib/hooks/__init__.py
+++ b/airflow/contrib/hooks/__init__.py
@@ -17,54 +17,3 @@
# specific language governing permissions and limitations
# under the License.
#
-
-
-# Contrib hooks are not imported by default. They should be accessed
-# directly: from airflow.contrib.hooks.hook_module import Hook
-
-
-import sys
-import os as _os
-
-# ------------------------------------------------------------------------
-#
-# #TODO #FIXME Airflow 2.0
-#
-# Old import machinary below.
-#
-# This is deprecated but should be kept until Airflow 2.0
-# for compatibility.
-#
-# ------------------------------------------------------------------------
-_hooks = {
- 'docker_hook': ['DockerHook'],
- 'ftp_hook': ['FTPHook'],
- 'ftps_hook': ['FTPSHook'],
- 'vertica_hook': ['VerticaHook'],
- 'ssh_hook': ['SSHHook'],
- 'winrm_hook': ['WinRMHook'],
- 'sftp_hook': ['SFTPHook'],
- 'bigquery_hook': ['BigQueryHook'],
- 'qubole_hook': ['QuboleHook'],
- 'gcs_hook': ['GoogleCloudStorageHook'],
- 'datastore_hook': ['DatastoreHook'],
- 'gcp_cloudml_hook': ['CloudMLHook'],
- 'redshift_hook': ['RedshiftHook'],
- 'gcp_dataproc_hook': ['DataProcHook'],
- 'gcp_dataflow_hook': ['DataFlowHook'],
- 'spark_submit_operator': ['SparkSubmitOperator'],
- 'cloudant_hook': ['CloudantHook'],
- 'fs_hook': ['FSHook'],
- 'wasb_hook': ['WasbHook'],
- 'gcp_pubsub_hook': ['PubSubHook'],
- 'jenkins_hook': ['JenkinsHook'],
- 'aws_dynamodb_hook': ['AwsDynamoDBHook'],
- 'azure_data_lake_hook': ['AzureDataLakeHook'],
- 'azure_fileshare_hook': ['AzureFileShareHook'],
-}
-
-
-if not _os.environ.get('AIRFLOW_USE_NEW_IMPORTS', False):
- from airflow.utils.helpers import AirflowImporter
-
- airflow_importer = AirflowImporter(sys.modules[__name__], _hooks)
diff --git a/airflow/contrib/hooks/aws_hook.py b/airflow/contrib/hooks/aws_hook.py
index c712d2de79..448de63ffe 100644
--- a/airflow/contrib/hooks/aws_hook.py
+++ b/airflow/contrib/hooks/aws_hook.py
@@ -72,7 +72,7 @@ def _parse_s3_config(config_file_name, config_format='boto', profile=None):
try:
access_key = config.get(cred_section, key_id_option)
secret_key = config.get(cred_section, secret_key_option)
- except:
+ except Exception:
logging.warning("Option Error in parsing s3 config file")
raise
return access_key, secret_key
@@ -84,8 +84,9 @@ class AwsHook(BaseHook):
This class is a thin wrapper around the boto3 python library.
"""
- def __init__(self, aws_conn_id='aws_default'):
+ def __init__(self, aws_conn_id='aws_default', verify=None):
self.aws_conn_id = aws_conn_id
+ self.verify = verify
def _get_credentials(self, region_name):
aws_access_key_id = None
@@ -116,6 +117,7 @@ def _get_credentials(self, region_name):
region_name = connection_object.extra_dejson.get('region_name')
role_arn = connection_object.extra_dejson.get('role_arn')
+ external_id = connection_object.extra_dejson.get('external_id')
aws_account_id = connection_object.extra_dejson.get('aws_account_id')
aws_iam_role = connection_object.extra_dejson.get('aws_iam_role')
@@ -130,9 +132,17 @@ def _get_credentials(self, region_name):
region_name=region_name)
sts_client = sts_session.client('sts')
- sts_response = sts_client.assume_role(
- RoleArn=role_arn,
- RoleSessionName='Airflow_' + self.aws_conn_id)
+
+ if external_id is None:
+ sts_response = sts_client.assume_role(
+ RoleArn=role_arn,
+ RoleSessionName='Airflow_' + self.aws_conn_id)
+ else:
+ sts_response = sts_client.assume_role(
+ RoleArn=role_arn,
+ RoleSessionName='Airflow_' + self.aws_conn_id,
+ ExternalId=external_id)
+
aws_access_key_id = sts_response['Credentials']['AccessKeyId']
aws_secret_access_key = sts_response['Credentials']['SecretAccessKey']
aws_session_token = sts_response['Credentials']['SessionToken']
@@ -153,12 +163,14 @@ def _get_credentials(self, region_name):
def get_client_type(self, client_type, region_name=None):
session, endpoint_url = self._get_credentials(region_name)
- return session.client(client_type, endpoint_url=endpoint_url)
+ return session.client(client_type, endpoint_url=endpoint_url,
+ verify=self.verify)
def get_resource_type(self, resource_type, region_name=None):
session, endpoint_url = self._get_credentials(region_name)
- return session.resource(resource_type, endpoint_url=endpoint_url)
+ return session.resource(resource_type, endpoint_url=endpoint_url,
+ verify=self.verify)
def get_session(self, region_name=None):
"""Get the underlying boto3.session."""
diff --git a/airflow/contrib/hooks/azure_data_lake_hook.py b/airflow/contrib/hooks/azure_data_lake_hook.py
index 1a02d78f27..74ae194809 100644
--- a/airflow/contrib/hooks/azure_data_lake_hook.py
+++ b/airflow/contrib/hooks/azure_data_lake_hook.py
@@ -59,7 +59,7 @@ def check_for_file(self, file_path):
:param file_path: Path and name of the file.
:type file_path: str
:return: True if the file exists, False otherwise.
- :rtype bool
+ :rtype: bool
"""
try:
files = self.connection.glob(file_path, details=False, invalidate_cache=True)
diff --git a/airflow/contrib/hooks/azure_fileshare_hook.py b/airflow/contrib/hooks/azure_fileshare_hook.py
index edabc17293..8afa1540d7 100644
--- a/airflow/contrib/hooks/azure_fileshare_hook.py
+++ b/airflow/contrib/hooks/azure_fileshare_hook.py
@@ -56,7 +56,7 @@ def check_for_directory(self, share_name, directory_name, **kwargs):
`FileService.exists()` takes.
:type kwargs: object
:return: True if the file exists, False otherwise.
- :rtype bool
+ :rtype: bool
"""
return self.connection.exists(share_name, directory_name,
**kwargs)
@@ -75,7 +75,7 @@ def check_for_file(self, share_name, directory_name, file_name, **kwargs):
`FileService.exists()` takes.
:type kwargs: object
:return: True if the file exists, False otherwise.
- :rtype bool
+ :rtype: bool
"""
return self.connection.exists(share_name, directory_name,
file_name, **kwargs)
@@ -92,7 +92,7 @@ def list_directories_and_files(self, share_name, directory_name=None, **kwargs):
`FileService.list_directories_and_files()` takes.
:type kwargs: object
:return: A list of files and directories
- :rtype list
+ :rtype: list
"""
return self.connection.list_directories_and_files(share_name,
directory_name,
@@ -100,7 +100,7 @@ def list_directories_and_files(self, share_name, directory_name=None, **kwargs):
def create_directory(self, share_name, directory_name, **kwargs):
"""
- Create a new direcotry on a Azure File Share.
+ Create a new directory on a Azure File Share.
:param share_name: Name of the share.
:type share_name: str
@@ -110,7 +110,7 @@ def create_directory(self, share_name, directory_name, **kwargs):
`FileService.create_directory()` takes.
:type kwargs: object
:return: A list of files and directories
- :rtype list
+ :rtype: list
"""
return self.connection.create_directory(share_name, directory_name, **kwargs)
diff --git a/airflow/contrib/hooks/bigquery_hook.py b/airflow/contrib/hooks/bigquery_hook.py
index b452f51c22..dba4618e35 100644
--- a/airflow/contrib/hooks/bigquery_hook.py
+++ b/airflow/contrib/hooks/bigquery_hook.py
@@ -24,6 +24,7 @@
import time
from builtins import range
+from copy import deepcopy
from past.builtins import basestring
@@ -93,13 +94,13 @@ def get_pandas_df(self, sql, parameters=None, dialect=None):
https://github.com/pydata/pandas/issues/6900
:param sql: The BigQuery SQL to execute.
- :type sql: string
+ :type sql: str
:param parameters: The parameters to render the SQL query with (not
used, leave to override superclass method)
:type parameters: mapping or iterable
:param dialect: Dialect of BigQuery SQL – legacy SQL or standard SQL
defaults to use `self.use_legacy_sql` if not specified
- :type dialect: string in {'legacy', 'standard'}
+ :type dialect: str in {'legacy', 'standard'}
"""
if dialect is None:
dialect = 'legacy' if self.use_legacy_sql else 'standard'
@@ -116,12 +117,12 @@ def table_exists(self, project_id, dataset_id, table_id):
:param project_id: The Google cloud project in which to look for the
table. The connection supplied to the hook must provide access to
the specified project.
- :type project_id: string
+ :type project_id: str
:param dataset_id: The name of the dataset in which to look for the
table.
- :type dataset_id: string
+ :type dataset_id: str
:param table_id: The name of the table to check the existence of.
- :type table_id: string
+ :type table_id: str
"""
service = self.get_service()
try:
@@ -195,10 +196,19 @@ class BigQueryBaseCursor(LoggingMixin):
PEP 249 cursor isn't needed.
"""
- def __init__(self, service, project_id, use_legacy_sql=True):
+ def __init__(self,
+ service,
+ project_id,
+ use_legacy_sql=True,
+ api_resource_configs=None):
+
self.service = service
self.project_id = project_id
self.use_legacy_sql = use_legacy_sql
+ if api_resource_configs:
+ _validate_value("api_resource_configs", api_resource_configs, dict)
+ self.api_resource_configs = api_resource_configs \
+ if api_resource_configs else {}
self.running_job_id = None
def create_empty_table(self,
@@ -206,7 +216,8 @@ def create_empty_table(self,
dataset_id,
table_id,
schema_fields=None,
- time_partitioning={}
+ time_partitioning=None,
+ labels=None
):
"""
Creates a new, empty table in the dataset.
@@ -219,6 +230,8 @@ def create_empty_table(self,
:type table_id: str
:param schema_fields: If set, the schema field list as defined here:
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema
+ :param labels: a dictionary containing labels for the table, passed to BigQuery
+ :type labels: dict
**Example**: ::
@@ -235,6 +248,7 @@ def create_empty_table(self,
:return:
"""
+
project_id = project_id if project_id is not None else self.project_id
table_resource = {
@@ -249,6 +263,9 @@ def create_empty_table(self,
if time_partitioning:
table_resource['timePartitioning'] = time_partitioning
+ if labels:
+ table_resource['labels'] = labels
+
self.log.info('Creating Table %s:%s.%s',
project_id, dataset_id, table_id)
@@ -280,7 +297,8 @@ def create_external_table(self,
quote_character=None,
allow_quoted_newlines=False,
allow_jagged_rows=False,
- src_fmt_configs={}
+ src_fmt_configs=None,
+ labels=None
):
"""
Creates a new external table in the dataset with the data in Google
@@ -295,7 +313,7 @@ def create_external_table(self,
table name to create external table.
If <project> is not included, project will be the
project defined in the connection json.
- :type external_project_dataset_table: string
+ :type external_project_dataset_table: str
:param schema_fields: The schema field list as defined here:
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#resource
:type schema_fields: list
@@ -304,7 +322,7 @@ def create_external_table(self,
per-object name can be used.
:type source_uris: list
:param source_format: File format to export.
- :type source_format: string
+ :type source_format: str
:param autodetect: Try to detect schema and format options automatically.
Any option specified explicitly will be honored.
:type autodetect: bool
@@ -313,7 +331,7 @@ def create_external_table(self,
The default value is NONE.
This setting is ignored for Google Cloud Bigtable,
Google Cloud Datastore backups and Avro formats.
- :type compression: string
+ :type compression: str
:param ignore_unknown_values: [Optional] Indicates if BigQuery should allow
extra values that are not represented in the table schema.
If true, the extra values are ignored. If false, records with extra columns
@@ -326,13 +344,13 @@ def create_external_table(self,
:param skip_leading_rows: Number of rows to skip when loading from a CSV.
:type skip_leading_rows: int
:param field_delimiter: The delimiter to use when loading from a CSV.
- :type field_delimiter: string
+ :type field_delimiter: str
:param quote_character: The value that is used to quote data sections in a CSV
file.
- :type quote_character: string
+ :type quote_character: str
:param allow_quoted_newlines: Whether to allow quoted newlines (true) or not
(false).
- :type allow_quoted_newlines: boolean
+ :type allow_quoted_newlines: bool
:param allow_jagged_rows: Accept rows that are missing trailing optional columns.
The missing values are treated as nulls. If false, records with missing
trailing columns are treated as bad records, and if there are too many bad
@@ -341,8 +359,12 @@ def create_external_table(self,
:type allow_jagged_rows: bool
:param src_fmt_configs: configure optional fields specific to the source format
:type src_fmt_configs: dict
+ :param labels: a dictionary containing labels for the table, passed to BigQuery
+ :type labels: dict
"""
+ if src_fmt_configs is None:
+ src_fmt_configs = {}
project_id, dataset_id, external_table_id = \
_split_tablename(table_input=external_project_dataset_table,
default_project_id=self.project_id,
@@ -439,6 +461,9 @@ def create_external_table(self,
table_resource['externalDataConfiguration'][src_fmt_to_param_mapping[
source_format]] = src_fmt_configs
+ if labels:
+ table_resource['labels'] = labels
+
try:
self.service.tables().insert(
projectId=project_id,
@@ -457,19 +482,22 @@ def create_external_table(self,
def run_query(self,
bql=None,
sql=None,
- destination_dataset_table=False,
+ destination_dataset_table=None,
write_disposition='WRITE_EMPTY',
allow_large_results=False,
- flatten_results=False,
- udf_config=False,
+ flatten_results=None,
+ udf_config=None,
use_legacy_sql=None,
maximum_billing_tier=None,
maximum_bytes_billed=None,
create_disposition='CREATE_IF_NEEDED',
query_params=None,
+ labels=None,
schema_update_options=(),
priority='INTERACTIVE',
- time_partitioning={}):
+ time_partitioning=None,
+ api_resource_configs=None,
+ cluster_fields=None):
"""
Executes a BigQuery SQL query. Optionally persists results in a BigQuery
table. See here:
@@ -480,31 +508,38 @@ def run_query(self,
:param bql: (Deprecated. Use `sql` parameter instead) The BigQuery SQL
to execute.
- :type bql: string
+ :type bql: str
:param sql: The BigQuery SQL to execute.
- :type sql: string
+ :type sql: str
:param destination_dataset_table: The dotted <dataset>.<table>
BigQuery table to save the query results.
- :type destination_dataset_table: string
+ :type destination_dataset_table: str
:param write_disposition: What to do if the table already exists in
BigQuery.
- :type write_disposition: string
+ :type write_disposition: str
:param allow_large_results: Whether to allow large results.
- :type allow_large_results: boolean
+ :type allow_large_results: bool
:param flatten_results: If true and query uses legacy SQL dialect, flattens
all nested and repeated fields in the query results. ``allowLargeResults``
must be true if this is set to false. For standard SQL queries, this
flag is ignored and results are never flattened.
- :type flatten_results: boolean
+ :type flatten_results: bool
:param udf_config: The User Defined Function configuration for the query.
See https://cloud.google.com/bigquery/user-defined-functions for details.
:param use_legacy_sql: Whether to use legacy SQL (true) or standard SQL (false).
If `None`, defaults to `self.use_legacy_sql`.
- :type use_legacy_sql: boolean
+ :type use_legacy_sql: bool
+ :param api_resource_configs: a dictionary that contain params
+ 'configuration' applied for Google BigQuery Jobs API:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs
+ for example, {'query': {'useQueryCache': False}}. You could use it
+ if you need to provide some params that are not supported by the
+ BigQueryHook like args.
+ :type api_resource_configs: dict
:type udf_config: list
:param maximum_billing_tier: Positive integer that serves as a
multiplier of the basic price.
- :type maximum_billing_tier: integer
+ :type maximum_billing_tier: int
:param maximum_bytes_billed: Limits the bytes billed for this job.
Queries that will have bytes billed beyond this limit will fail
(without incurring a charge). If unspecified, this will be
@@ -512,28 +547,45 @@ def run_query(self,
:type maximum_bytes_billed: float
:param create_disposition: Specifies whether the job is allowed to
create new tables.
- :type create_disposition: string
+ :type create_disposition: str
:param query_params a dictionary containing query parameter types and
values, passed to BigQuery
:type query_params: dict
+ :param labels a dictionary containing labels for the job/query,
+ passed to BigQuery
+ :type labels: dict
:param schema_update_options: Allows the schema of the desitination
table to be updated as a side effect of the query job.
:type schema_update_options: tuple
:param priority: Specifies a priority for the query.
Possible values include INTERACTIVE and BATCH.
The default value is INTERACTIVE.
- :type priority: string
+ :type priority: str
:param time_partitioning: configure optional time partitioning fields i.e.
- partition by field, type and
- expiration as per API specifications. Note that 'field' is not available in
- conjunction with dataset.table$partition.
+ partition by field, type and expiration as per API specifications.
:type time_partitioning: dict
-
+ :param cluster_fields: Request that the result of this query be stored sorted
+ by one or more columns. This is only available in combination with
+ time_partitioning. The order of columns given determines the sort order.
+ :type cluster_fields: list of str
"""
- # TODO remove `bql` in Airflow 2.0 - Jira: [AIRFLOW-2513]
+ if not api_resource_configs:
+ api_resource_configs = self.api_resource_configs
+ else:
+ _validate_value('api_resource_configs',
+ api_resource_configs, dict)
+ configuration = deepcopy(api_resource_configs)
+ if 'query' not in configuration:
+ configuration['query'] = {}
+
+ else:
+ _validate_value("api_resource_configs['query']",
+ configuration['query'], dict)
+
sql = bql if sql is None else sql
+ # TODO remove `bql` in Airflow 2.0 - Jira: [AIRFLOW-2513]
if bql:
import warnings
warnings.warn('Deprecated parameter `bql` used in '
@@ -544,88 +596,113 @@ def run_query(self,
'Airflow.',
category=DeprecationWarning)
- if sql is None:
- raise TypeError('`BigQueryBaseCursor.run_query` missing 1 required '
- 'positional argument: `sql`')
+ if sql is None and not configuration['query'].get('query', None):
+ raise TypeError('`BigQueryBaseCursor.run_query` '
+ 'missing 1 required positional argument: `sql`')
# BigQuery also allows you to define how you want a table's schema to change
# as a side effect of a query job
# for more details:
# https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.schemaUpdateOptions
+
allowed_schema_update_options = [
'ALLOW_FIELD_ADDITION', "ALLOW_FIELD_RELAXATION"
]
- if not set(allowed_schema_update_options).issuperset(
- set(schema_update_options)):
- raise ValueError(
- "{0} contains invalid schema update options. "
- "Please only use one or more of the following options: {1}"
- .format(schema_update_options, allowed_schema_update_options))
- if use_legacy_sql is None:
- use_legacy_sql = self.use_legacy_sql
+ if not set(allowed_schema_update_options
+ ).issuperset(set(schema_update_options)):
+ raise ValueError("{0} contains invalid schema update options. "
+ "Please only use one or more of the following "
+ "options: {1}"
+ .format(schema_update_options,
+ allowed_schema_update_options))
- configuration = {
- 'query': {
- 'query': sql,
- 'useLegacySql': use_legacy_sql,
- 'maximumBillingTier': maximum_billing_tier,
- 'maximumBytesBilled': maximum_bytes_billed,
- 'priority': priority
- }
- }
+ if schema_update_options:
+ if write_disposition not in ["WRITE_APPEND", "WRITE_TRUNCATE"]:
+ raise ValueError("schema_update_options is only "
+ "allowed if write_disposition is "
+ "'WRITE_APPEND' or 'WRITE_TRUNCATE'.")
if destination_dataset_table:
- assert '.' in destination_dataset_table, (
- 'Expected destination_dataset_table in the format of '
- '<dataset>.<table>. Got: {}').format(destination_dataset_table)
destination_project, destination_dataset, destination_table = \
_split_tablename(table_input=destination_dataset_table,
default_project_id=self.project_id)
- configuration['query'].update({
- 'allowLargeResults': allow_large_results,
- 'flattenResults': flatten_results,
- 'writeDisposition': write_disposition,
- 'createDisposition': create_disposition,
- 'destinationTable': {
- 'projectId': destination_project,
- 'datasetId': destination_dataset,
- 'tableId': destination_table,
- }
- })
- if udf_config:
- assert isinstance(udf_config, list)
- configuration['query'].update({
- 'userDefinedFunctionResources': udf_config
- })
- if query_params:
- if self.use_legacy_sql:
- raise ValueError("Query paramaters are not allowed when using "
- "legacy SQL")
- else:
- configuration['query']['queryParameters'] = query_params
+ destination_dataset_table = {
+ 'projectId': destination_project,
+ 'datasetId': destination_dataset,
+ 'tableId': destination_table,
+ }
- time_partitioning = _cleanse_time_partitioning(
- destination_dataset_table,
- time_partitioning
- )
- if time_partitioning:
- configuration['query'].update({
- 'timePartitioning': time_partitioning
- })
+ if cluster_fields:
+ cluster_fields = {'fields': cluster_fields}
+
+ query_param_list = [
+ (sql, 'query', None, str),
+ (priority, 'priority', 'INTERACTIVE', str),
+ (use_legacy_sql, 'useLegacySql', self.use_legacy_sql, bool),
+ (query_params, 'queryParameters', None, dict),
+ (udf_config, 'userDefinedFunctionResources', None, list),
+ (maximum_billing_tier, 'maximumBillingTier', None, int),
+ (maximum_bytes_billed, 'maximumBytesBilled', None, float),
+ (time_partitioning, 'timePartitioning', {}, dict),
+ (schema_update_options, 'schemaUpdateOptions', None, tuple),
+ (destination_dataset_table, 'destinationTable', None, dict),
+ (cluster_fields, 'clustering', None, dict),
+ ]
- if schema_update_options:
- if write_disposition not in ["WRITE_APPEND", "WRITE_TRUNCATE"]:
- raise ValueError("schema_update_options is only "
- "allowed if write_disposition is "
- "'WRITE_APPEND' or 'WRITE_TRUNCATE'.")
- else:
- self.log.info(
- "Adding experimental "
- "'schemaUpdateOptions': {0}".format(schema_update_options))
- configuration['query'][
- 'schemaUpdateOptions'] = schema_update_options
+ for param_tuple in query_param_list:
+
+ param, param_name, param_default, param_type = param_tuple
+
+ if param_name not in configuration['query'] and param in [None, {}, ()]:
+ if param_name == 'timePartitioning':
+ param_default = _cleanse_time_partitioning(
+ destination_dataset_table, time_partitioning)
+ param = param_default
+
+ if param not in [None, {}, ()]:
+ _api_resource_configs_duplication_check(
+ param_name, param, configuration['query'])
+
+ configuration['query'][param_name] = param
+
+ # check valid type of provided param,
+ # it last step because we can get param from 2 sources,
+ # and first of all need to find it
+
+ _validate_value(param_name, configuration['query'][param_name],
+ param_type)
+
+ if param_name == 'schemaUpdateOptions' and param:
+ self.log.info("Adding experimental 'schemaUpdateOptions': "
+ "{0}".format(schema_update_options))
+
+ if param_name == 'destinationTable':
+ for key in ['projectId', 'datasetId', 'tableId']:
+ if key not in configuration['query']['destinationTable']:
+ raise ValueError(
+ "Not correct 'destinationTable' in "
+ "api_resource_configs. 'destinationTable' "
+ "must be a dict with {'projectId':'', "
+ "'datasetId':'', 'tableId':''}")
+
+ configuration['query'].update({
+ 'allowLargeResults': allow_large_results,
+ 'flattenResults': flatten_results,
+ 'writeDisposition': write_disposition,
+ 'createDisposition': create_disposition,
+ })
+
+ if 'useLegacySql' in configuration['query'] and \
+ 'queryParameters' in configuration['query']:
+ raise ValueError("Query parameters are not allowed "
+ "when using legacy SQL")
+
+ if labels:
+ _api_resource_configs_duplication_check(
+ 'labels', labels, configuration)
+ configuration['labels'] = labels
return self.run_with_configuration(configuration)
@@ -636,7 +713,8 @@ def run_extract( # noqa
compression='NONE',
export_format='CSV',
field_delimiter=',',
- print_header=True):
+ print_header=True,
+ labels=None):
"""
Executes a BigQuery extract command to copy data from BigQuery to
Google Cloud Storage. See here:
@@ -647,20 +725,23 @@ def run_extract( # noqa
:param source_project_dataset_table: The dotted <dataset>.<table>
BigQuery table to use as the source data.
- :type source_project_dataset_table: string
+ :type source_project_dataset_table: str
:param destination_cloud_storage_uris: The destination Google Cloud
Storage URI (e.g. gs://some-bucket/some-file.txt). Follows
convention defined here:
https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple
:type destination_cloud_storage_uris: list
:param compression: Type of compression to use.
- :type compression: string
+ :type compression: str
:param export_format: File format to export.
- :type export_format: string
+ :type export_format: str
:param field_delimiter: The delimiter to use when extracting to a CSV.
- :type field_delimiter: string
+ :type field_delimiter: str
:param print_header: Whether to print a header for a CSV file extract.
- :type print_header: boolean
+ :type print_header: bool
+ :param labels: a dictionary containing labels for the job/query,
+ passed to BigQuery
+ :type labels: dict
"""
source_project, source_dataset, source_table = \
@@ -681,6 +762,9 @@ def run_extract( # noqa
}
}
+ if labels:
+ configuration['labels'] = labels
+
if export_format == 'CSV':
# Only set fieldDelimiter and printHeader fields if using CSV.
# Google does not like it if you set these fields for other export
@@ -694,7 +778,8 @@ def run_copy(self,
source_project_dataset_tables,
destination_project_dataset_table,
write_disposition='WRITE_EMPTY',
- create_disposition='CREATE_IF_NEEDED'):
+ create_disposition='CREATE_IF_NEEDED',
+ labels=None):
"""
Executes a BigQuery copy command to copy data from one BigQuery table
to another. See here:
@@ -712,11 +797,14 @@ def run_copy(self,
:type source_project_dataset_tables: list|string
:param destination_project_dataset_table: The destination BigQuery
table. Format is: (project:|project.)<dataset>.<table>
- :type destination_project_dataset_table: string
+ :type destination_project_dataset_table: str
:param write_disposition: The write disposition if the table already exists.
- :type write_disposition: string
+ :type write_disposition: str
:param create_disposition: The create disposition if the table doesn't exist.
- :type create_disposition: string
+ :type create_disposition: str
+ :param labels a dictionary containing labels for the job/query,
+ passed to BigQuery
+ :type labels: dict
"""
source_project_dataset_tables = ([
source_project_dataset_tables
@@ -754,6 +842,9 @@ def run_copy(self,
}
}
+ if labels:
+ configuration['labels'] = labels
+
return self.run_with_configuration(configuration)
def run_load(self,
@@ -771,8 +862,9 @@ def run_load(self,
allow_quoted_newlines=False,
allow_jagged_rows=False,
schema_update_options=(),
- src_fmt_configs={},
- time_partitioning={}):
+ src_fmt_configs=None,
+ time_partitioning=None,
+ cluster_fields=None):
"""
Executes a BigQuery load command to load data from Google Cloud Storage
to BigQuery. See here:
@@ -787,7 +879,7 @@ def run_load(self,
project defined in the connection json. If a partition is specified the
operator will automatically append the data, create a new partition or create
a new DAY partitioned table.
- :type destination_project_dataset_table: string
+ :type destination_project_dataset_table: str
:param schema_fields: The schema field list as defined here:
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load
:type schema_fields: list
@@ -796,21 +888,21 @@ def run_load(self,
per-object name can be used.
:type source_uris: list
:param source_format: File format to export.
- :type source_format: string
+ :type source_format: str
:param create_disposition: The create disposition if the table doesn't exist.
- :type create_disposition: string
+ :type create_disposition: str
:param skip_leading_rows: Number of rows to skip when loading from a CSV.
:type skip_leading_rows: int
:param write_disposition: The write disposition if the table already exists.
- :type write_disposition: string
+ :type write_disposition: str
:param field_delimiter: The delimiter to use when loading from a CSV.
- :type field_delimiter: string
+ :type field_delimiter: str
:param max_bad_records: The maximum number of bad records that BigQuery can
ignore when running the job.
:type max_bad_records: int
:param quote_character: The value that is used to quote data sections in a CSV
file.
- :type quote_character: string
+ :type quote_character: str
:param ignore_unknown_values: [Optional] Indicates if BigQuery should allow
extra values that are not represented in the table schema.
If true, the extra values are ignored. If false, records with extra columns
@@ -819,7 +911,7 @@ def run_load(self,
:type ignore_unknown_values: bool
:param allow_quoted_newlines: Whether to allow quoted newlines (true) or not
(false).
- :type allow_quoted_newlines: boolean
+ :type allow_quoted_newlines: bool
:param allow_jagged_rows: Accept rows that are missing trailing optional columns.
The missing values are treated as nulls. If false, records with missing
trailing columns are treated as bad records, and if there are too many bad
@@ -832,10 +924,12 @@ def run_load(self,
:param src_fmt_configs: configure optional fields specific to the source format
:type src_fmt_configs: dict
:param time_partitioning: configure optional time partitioning fields i.e.
- partition by field, type and
- expiration as per API specifications. Note that 'field' is not available in
- conjunction with dataset.table$partition.
+ partition by field, type and expiration as per API specifications.
:type time_partitioning: dict
+ :param cluster_fields: Request that the result of this load be stored sorted
+ by one or more columns. This is only available in combination with
+ time_partitioning. The order of columns given determines the sort order.
+ :type cluster_fields: list of str
"""
# bigquery only allows certain source formats
@@ -843,6 +937,9 @@ def run_load(self,
# if it's not, we raise a ValueError
# Refer to this link for more details:
# https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).sourceFormat
+ if src_fmt_configs is None:
+ src_fmt_configs = {}
+
source_format = source_format.upper()
allowed_formats = [
"CSV", "NEWLINE_DELIMITED_JSON", "AVRO", "GOOGLE_SHEETS",
@@ -863,7 +960,7 @@ def run_load(self,
if not set(allowed_schema_update_options).issuperset(
set(schema_update_options)):
raise ValueError(
- "{0} contains invalid schema update options. "
+ "{0} contains invalid schema update options."
"Please only use one or more of the following options: {1}"
.format(schema_update_options, allowed_schema_update_options))
@@ -896,6 +993,9 @@ def run_load(self,
'timePartitioning': time_partitioning
})
+ if cluster_fields:
+ configuration['load'].update({'clustering': {'fields': cluster_fields}})
+
if schema_fields:
configuration['load']['schema'] = {'fields': schema_fields}
@@ -974,12 +1074,12 @@ def run_with_configuration(self, configuration):
# Wait for query to finish.
keep_polling_job = True
- while (keep_polling_job):
+ while keep_polling_job:
try:
job = jobs.get(
projectId=self.project_id,
jobId=self.running_job_id).execute()
- if (job['status']['state'] == 'DONE'):
+ if job['status']['state'] == 'DONE':
keep_polling_job = False
# Check if job had errors.
if 'errorResult' in job['status']:
@@ -1008,7 +1108,7 @@ def poll_job_complete(self, job_id):
jobs = self.service.jobs()
try:
job = jobs.get(projectId=self.project_id, jobId=job_id).execute()
- if (job['status']['state'] == 'DONE'):
+ if job['status']['state'] == 'DONE':
return True
except HttpError as err:
if err.resp.status in [500, 503]:
@@ -1042,13 +1142,13 @@ def cancel_query(self):
polling_attempts = 0
job_complete = False
- while (polling_attempts < max_polling_attempts and not job_complete):
+ while polling_attempts < max_polling_attempts and not job_complete:
polling_attempts = polling_attempts + 1
job_complete = self.poll_job_complete(self.running_job_id)
- if (job_complete):
+ if job_complete:
self.log.info('Job successfully canceled: %s, %s',
self.project_id, self.running_job_id)
- elif (polling_attempts == max_polling_attempts):
+ elif polling_attempts == max_polling_attempts:
self.log.info(
"Stopping polling due to timeout. Job with id %s "
"has not completed cancel and may or may not finish.",
@@ -1117,13 +1217,9 @@ def run_table_delete(self, deletion_dataset_table,
:type deletion_dataset_table: str
:param ignore_if_missing: if True, then return success even if the
requested table does not exist.
- :type ignore_if_missing: boolean
+ :type ignore_if_missing: bool
:return:
"""
-
- assert '.' in deletion_dataset_table, (
- 'Expected deletion_dataset_table in the format of '
- '<dataset>.<table>. Got: {}').format(deletion_dataset_table)
deletion_project, deletion_dataset, deletion_table = \
_split_tablename(table_input=deletion_dataset_table,
default_project_id=self.project_id)
@@ -1251,10 +1347,180 @@ def run_grant_dataset_view_access(self,
# if view is already in access, do nothing.
self.log.info(
'Table %s:%s.%s already has authorized view access to %s:%s dataset.',
- view_project, view_dataset, view_table, source_project,
- source_dataset)
+ view_project, view_dataset, view_table, source_project, source_dataset)
return source_dataset_resource
+ def create_empty_dataset(self, dataset_id="", project_id="",
+ dataset_reference=None):
+ """
+ Create a new empty dataset:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert
+
+ :param project_id: The name of the project where we want to create
+ an empty a dataset. Don't need to provide, if projectId in dataset_reference.
+ :type project_id: str
+ :param dataset_id: The id of dataset. Don't need to provide,
+ if datasetId in dataset_reference.
+ :type dataset_id: str
+ :param dataset_reference: Dataset reference that could be provided
+ with request body. More info:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
+ :type dataset_reference: dict
+ """
+
+ if dataset_reference:
+ _validate_value('dataset_reference', dataset_reference, dict)
+ else:
+ dataset_reference = {}
+
+ if "datasetReference" not in dataset_reference:
+ dataset_reference["datasetReference"] = {}
+
+ if not dataset_reference["datasetReference"].get("datasetId") and not dataset_id:
+ raise ValueError(
+ "{} not provided datasetId. Impossible to create dataset")
+
+ dataset_required_params = [(dataset_id, "datasetId", ""),
+ (project_id, "projectId", self.project_id)]
+ for param_tuple in dataset_required_params:
+ param, param_name, param_default = param_tuple
+ if param_name not in dataset_reference['datasetReference']:
+ if param_default and not param:
+ self.log.info("{} was not specified. Will be used default "
+ "value {}.".format(param_name,
+ param_default))
+ param = param_default
+ dataset_reference['datasetReference'].update(
+ {param_name: param})
+ elif param:
+ _api_resource_configs_duplication_check(
+ param_name, param,
+ dataset_reference['datasetReference'], 'dataset_reference')
+
+ dataset_id = dataset_reference.get("datasetReference").get("datasetId")
+ dataset_project_id = dataset_reference.get("datasetReference").get(
+ "projectId")
+
+ self.log.info('Creating Dataset: %s in project: %s ', dataset_id,
+ dataset_project_id)
+
+ try:
+ self.service.datasets().insert(
+ projectId=dataset_project_id,
+ body=dataset_reference).execute()
+ self.log.info('Dataset created successfully: In project %s '
+ 'Dataset %s', dataset_project_id, dataset_id)
+
+ except HttpError as err:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(err.content)
+ )
+
+ def delete_dataset(self, project_id, dataset_id):
+ """
+ Delete a dataset of Big query in your project.
+ :param project_id: The name of the project where we have the dataset .
+ :type project_id: str
+ :param dataset_id: The dataset to be delete.
+ :type dataset_id: str
+ :return:
+ """
+ project_id = project_id if project_id is not None else self.project_id
+ self.log.info('Deleting from project: %s Dataset:%s',
+ project_id, dataset_id)
+
+ try:
+ self.service.datasets().delete(
+ projectId=project_id,
+ datasetId=dataset_id).execute()
+ self.log.info('Dataset deleted successfully: In project %s '
+ 'Dataset %s', project_id, dataset_id)
+
+ except HttpError as err:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(err.content)
+ )
+
+ def get_dataset(self, dataset_id, project_id=None):
+ """
+ Method returns dataset_resource if dataset exist
+ and raised 404 error if dataset does not exist
+
+ :param dataset_id: The BigQuery Dataset ID
+ :type dataset_id: str
+ :param project_id: The GCP Project ID
+ :type project_id: str
+ :return: dataset_resource
+
+ .. seealso::
+ For more information, see Dataset Resource content:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
+ """
+
+ if not dataset_id or not isinstance(dataset_id, str):
+ raise ValueError("dataset_id argument must be provided and has "
+ "a type 'str'. You provided: {}".format(dataset_id))
+
+ dataset_project_id = project_id if project_id else self.project_id
+
+ try:
+ dataset_resource = self.service.datasets().get(
+ datasetId=dataset_id, projectId=dataset_project_id).execute()
+ self.log.info("Dataset Resource: {}".format(dataset_resource))
+ except HttpError as err:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(err.content))
+
+ return dataset_resource
+
+ def get_datasets_list(self, project_id=None):
+ """
+ Method returns full list of BigQuery datasets in the current project
+
+ .. seealso::
+ For more information, see:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/list
+
+ :param project_id: Google Cloud Project for which you
+ try to get all datasets
+ :type project_id: str
+ :return: datasets_list
+
+ Example of returned datasets_list: ::
+
+ {
+ "kind":"bigquery#dataset",
+ "location":"US",
+ "id":"your-project:dataset_2_test",
+ "datasetReference":{
+ "projectId":"your-project",
+ "datasetId":"dataset_2_test"
+ }
+ },
+ {
+ "kind":"bigquery#dataset",
+ "location":"US",
+ "id":"your-project:dataset_1_test",
+ "datasetReference":{
+ "projectId":"your-project",
+ "datasetId":"dataset_1_test"
+ }
+ }
+ ]
+ """
+ dataset_project_id = project_id if project_id else self.project_id
+
+ try:
+ datasets_list = self.service.datasets().list(
+ projectId=dataset_project_id).execute()['datasets']
+ self.log.info("Datasets List: {}".format(datasets_list))
+
+ except HttpError as err:
+ raise AirflowException(
+ 'BigQuery job failed. Error was: {}'.format(err.content))
+
+ return datasets_list
+
class BigQueryCursor(BigQueryBaseCursor):
"""
@@ -1295,7 +1561,7 @@ def execute(self, operation, parameters=None):
Executes a BigQuery query, and returns the job ID.
:param operation: The query to execute.
- :type operation: string
+ :type operation: str
:param parameters: Parameters to substitute into the query.
:type parameters: dict
"""
@@ -1308,7 +1574,7 @@ def executemany(self, operation, seq_of_parameters):
Execute a BigQuery query multiple times with different parameters.
:param operation: The query to execute.
- :type operation: string
+ :type operation: str
:param seq_of_parameters: List of dictionary parameters to substitute into the
query.
:type seq_of_parameters: list
@@ -1456,14 +1722,23 @@ def _bq_cast(string_field, bq_type):
elif bq_type == 'FLOAT' or bq_type == 'TIMESTAMP':
return float(string_field)
elif bq_type == 'BOOLEAN':
- assert string_field in set(['true', 'false'])
+ if string_field not in ['true', 'false']:
+ raise ValueError("{} must have value 'true' or 'false'".format(
+ string_field))
return string_field == 'true'
else:
return string_field
def _split_tablename(table_input, default_project_id, var_name=None):
- assert default_project_id is not None, "INTERNAL: No default project is specified"
+
+ if '.' not in table_input:
+ raise ValueError(
+ 'Expected deletion_dataset_table name in the format of '
+ '<dataset>.<table>. Got: {}'.format(table_input))
+
+ if not default_project_id:
+ raise ValueError("INTERNAL: No default project is specified")
def var_print(var_name):
if var_name is None:
@@ -1475,7 +1750,6 @@ def var_print(var_name):
raise Exception(('{var}Use either : or . to specify project '
'got {input}').format(
var=var_print(var_name), input=table_input))
-
cmpt = table_input.rsplit(':', 1)
project_id = None
rest = table_input
@@ -1493,8 +1767,10 @@ def var_print(var_name):
cmpt = rest.split('.')
if len(cmpt) == 3:
- assert project_id is None, ("{var}Use either : or . to specify project"
- ).format(var=var_print(var_name))
+ if project_id:
+ raise ValueError(
+ "{var}Use either : or . to specify project".format(
+ var=var_print(var_name)))
project_id = cmpt[0]
dataset_id = cmpt[1]
table_id = cmpt[2]
@@ -1522,13 +1798,30 @@ def var_print(var_name):
def _cleanse_time_partitioning(destination_dataset_table, time_partitioning_in):
# if it is a partitioned table ($ is in the table name) add partition load option
+
+ if time_partitioning_in is None:
+ time_partitioning_in = {}
+
time_partitioning_out = {}
if destination_dataset_table and '$' in destination_dataset_table:
- assert not time_partitioning_in.get('field'), (
- "Cannot specify field partition and partition name "
- "(dataset.table$partition) at the same time"
- )
time_partitioning_out['type'] = 'DAY'
-
time_partitioning_out.update(time_partitioning_in)
return time_partitioning_out
+
+
+def _validate_value(key, value, expected_type):
+ """ function to check expected type and raise
+ error if type is not correct """
+ if not isinstance(value, expected_type):
+ raise TypeError("{} argument must have a type {} not {}".format(
+ key, expected_type, type(value)))
+
+
+def _api_resource_configs_duplication_check(key, value, config_dict,
+ config_dict_name='api_resource_configs'):
+ if key in config_dict and value != config_dict[key]:
+ raise ValueError("Values of {param_name} param are duplicated. "
+ "{dict_name} contained {param_name} param "
+ "in `query` config and {param_name} was also provided "
+ "with arg to run_query() method. Please remove duplicates."
+ .format(param_name=key, dict_name=config_dict_name))
diff --git a/airflow/contrib/hooks/cassandra_hook.py b/airflow/contrib/hooks/cassandra_hook.py
index 704ba0d8d0..2c744fe9b1 100644
--- a/airflow/contrib/hooks/cassandra_hook.py
+++ b/airflow/contrib/hooks/cassandra_hook.py
@@ -158,22 +158,37 @@ def get_lb_policy(policy_name, policy_args):
child_policy_args)
return TokenAwarePolicy(child_policy)
+ def table_exists(self, table):
+ """
+ Checks if a table exists in Cassandra
+
+ :param table: Target Cassandra table.
+ Use dot notation to target a specific keyspace.
+ :type table: str
+ """
+ keyspace = self.keyspace
+ if '.' in table:
+ keyspace, table = table.split('.', 1)
+ cluster_metadata = self.get_conn().cluster.metadata
+ return (keyspace in cluster_metadata.keyspaces and
+ table in cluster_metadata.keyspaces[keyspace].tables)
+
def record_exists(self, table, keys):
"""
Checks if a record exists in Cassandra
:param table: Target Cassandra table.
Use dot notation to target a specific keyspace.
- :type table: string
+ :type table: str
:param keys: The keys and their values to check the existence.
:type keys: dict
"""
- keyspace = None
+ keyspace = self.keyspace
if '.' in table:
keyspace, table = table.split('.', 1)
ks = " AND ".join("{}=%({})s".format(key, key) for key in keys.keys())
cql = "SELECT * FROM {keyspace}.{table} WHERE {keys}".format(
- keyspace=(keyspace or self.keyspace), table=table, keys=ks)
+ keyspace=keyspace, table=table, keys=ks)
try:
rs = self.get_conn().execute(cql, keys)
diff --git a/airflow/contrib/hooks/databricks_hook.py b/airflow/contrib/hooks/databricks_hook.py
index 1443ff4740..f4a890ac7b 100644
--- a/airflow/contrib/hooks/databricks_hook.py
+++ b/airflow/contrib/hooks/databricks_hook.py
@@ -24,6 +24,7 @@
from airflow.hooks.base_hook import BaseHook
from requests import exceptions as requests_exceptions
from requests.auth import AuthBase
+from time import sleep
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -32,7 +33,11 @@
except ImportError:
import urlparse
+RESTART_CLUSTER_ENDPOINT = ("POST", "api/2.0/clusters/restart")
+START_CLUSTER_ENDPOINT = ("POST", "api/2.0/clusters/start")
+TERMINATE_CLUSTER_ENDPOINT = ("POST", "api/2.0/clusters/delete")
+RUN_NOW_ENDPOINT = ('POST', 'api/2.0/jobs/run-now')
SUBMIT_RUN_ENDPOINT = ('POST', 'api/2.0/jobs/runs/submit')
GET_RUN_ENDPOINT = ('GET', 'api/2.0/jobs/runs/get')
CANCEL_RUN_ENDPOINT = ('POST', 'api/2.0/jobs/runs/cancel')
@@ -47,24 +52,31 @@ def __init__(
self,
databricks_conn_id='databricks_default',
timeout_seconds=180,
- retry_limit=3):
+ retry_limit=3,
+ retry_delay=1.0):
"""
:param databricks_conn_id: The name of the databricks connection to use.
- :type databricks_conn_id: string
+ :type databricks_conn_id: str
:param timeout_seconds: The amount of time in seconds the requests library
will wait before timing-out.
:type timeout_seconds: int
:param retry_limit: The number of times to retry the connection in case of
service outages.
:type retry_limit: int
+ :param retry_delay: The number of seconds to wait between retries (it
+ might be a floating point number).
+ :type retry_delay: float
"""
self.databricks_conn_id = databricks_conn_id
self.databricks_conn = self.get_connection(databricks_conn_id)
self.timeout_seconds = timeout_seconds
- assert retry_limit >= 1, 'Retry limit must be greater than equal to 1'
+ if retry_limit < 1:
+ raise ValueError('Retry limit must be greater than equal to 1')
self.retry_limit = retry_limit
+ self.retry_delay = retry_delay
- def _parse_host(self, host):
+ @staticmethod
+ def _parse_host(host):
"""
The purpose of this function is to be robust to improper connections
settings provided by users, specifically in the host field.
@@ -117,7 +129,8 @@ def _do_api_call(self, endpoint_info, json):
else:
raise AirflowException('Unexpected HTTP Method: ' + method)
- for attempt_num in range(1, self.retry_limit + 1):
+ attempt_num = 1
+ while True:
try:
response = request_func(
url,
@@ -125,21 +138,41 @@ def _do_api_call(self, endpoint_info, json):
auth=auth,
headers=USER_AGENT_HEADER,
timeout=self.timeout_seconds)
- if response.status_code == requests.codes.ok:
- return response.json()
- else:
+ response.raise_for_status()
+ return response.json()
+ except requests_exceptions.RequestException as e:
+ if not _retryable_error(e):
# In this case, the user probably made a mistake.
# Don't retry.
raise AirflowException('Response: {0}, Status Code: {1}'.format(
- response.content, response.status_code))
- except (requests_exceptions.ConnectionError,
- requests_exceptions.Timeout) as e:
- self.log.error(
- 'Attempt %s API Request to Databricks failed with reason: %s',
- attempt_num, e
- )
- raise AirflowException(('API requests to Databricks failed {} times. ' +
- 'Giving up.').format(self.retry_limit))
+ e.response.content, e.response.status_code))
+
+ self._log_request_error(attempt_num, e)
+
+ if attempt_num == self.retry_limit:
+ raise AirflowException(('API requests to Databricks failed {} times. ' +
+ 'Giving up.').format(self.retry_limit))
+
+ attempt_num += 1
+ sleep(self.retry_delay)
+
+ def _log_request_error(self, attempt_num, error):
+ self.log.error(
+ 'Attempt %s API Request to Databricks failed with reason: %s',
+ attempt_num, error
+ )
+
+ def run_now(self, json):
+ """
+ Utility function to call the ``api/2.0/jobs/run-now`` endpoint.
+
+ :param json: The data used in the body of the request to the ``run-now`` endpoint.
+ :type json: dict
+ :return: the run_id as a string
+ :rtype: str
+ """
+ response = self._do_api_call(RUN_NOW_ENDPOINT, json)
+ return response['run_id']
def submit_run(self, json):
"""
@@ -148,7 +181,7 @@ def submit_run(self, json):
:param json: The data used in the body of the request to the ``submit`` endpoint.
:type json: dict
:return: the run_id as a string
- :rtype: string
+ :rtype: str
"""
response = self._do_api_call(SUBMIT_RUN_ENDPOINT, json)
return response['run_id']
@@ -172,6 +205,21 @@ def cancel_run(self, run_id):
json = {'run_id': run_id}
self._do_api_call(CANCEL_RUN_ENDPOINT, json)
+ def restart_cluster(self, json):
+ self._do_api_call(RESTART_CLUSTER_ENDPOINT, json)
+
+ def start_cluster(self, json):
+ self._do_api_call(START_CLUSTER_ENDPOINT, json)
+
+ def terminate_cluster(self, json):
+ self._do_api_call(TERMINATE_CLUSTER_ENDPOINT, json)
+
+
+def _retryable_error(exception):
+ return isinstance(exception, requests_exceptions.ConnectionError) \
+ or isinstance(exception, requests_exceptions.Timeout) \
+ or exception.response is not None and exception.response.status_code >= 500
+
RUN_LIFE_CYCLE_STATES = [
'PENDING',
diff --git a/airflow/contrib/hooks/datadog_hook.py b/airflow/contrib/hooks/datadog_hook.py
index 3dfeb781ae..f00e49b097 100644
--- a/airflow/contrib/hooks/datadog_hook.py
+++ b/airflow/contrib/hooks/datadog_hook.py
@@ -35,7 +35,7 @@ class DatadogHook(BaseHook, LoggingMixin):
Airflow runs.
:param datadog_conn_id: The connection to datadog, containing metadata for api keys.
- :param datadog_conn_id: string
+ :param datadog_conn_id: str
"""
def __init__(self, datadog_conn_id='datadog_default'):
conn = self.get_connection(datadog_conn_id)
@@ -71,9 +71,9 @@ def send_metric(self, metric_name, datapoint, tags=None):
Sends a single datapoint metric to DataDog
:param metric_name: The name of the metric
- :type metric_name: string
+ :type metric_name: str
:param datapoint: A single integer or float related to the metric
- :type datapoint: integer or float
+ :type datapoint: int or float
:param tags: A list of tags associated with the metric
:type tags: list
"""
@@ -95,7 +95,7 @@ def query_metric(self,
function applied to it and returns the results.
:param query: The datadog query to execute (see datadog docs)
- :type query: string
+ :type query: str
:param from_seconds_ago: How many seconds ago to start querying for.
:type from_seconds_ago: int
:param to_seconds_ago: Up to how many seconds ago to query for.
@@ -118,16 +118,16 @@ def post_event(self, title, text, tags=None, alert_type=None, aggregation_key=No
alerting itself.
:param title: The title of the event
- :type title: string
+ :type title: str
:param text: The body of the event (more information)
- :type text: string
+ :type text: str
:param tags: List of string tags to apply to the event
:type tags: list
:param alert_type: The alert type for the event, one of
["error", "warning", "info", "success"]
- :type alert_type: string
+ :type alert_type: str
:param aggregation_key: Key that can be used to aggregate this event in a stream
- :type aggregation_key: string
+ :type aggregation_key: str
"""
response = api.Event.create(
title=title,
diff --git a/airflow/contrib/hooks/datastore_hook.py b/airflow/contrib/hooks/datastore_hook.py
index 5e54cf2a65..b8c3ca00a0 100644
--- a/airflow/contrib/hooks/datastore_hook.py
+++ b/airflow/contrib/hooks/datastore_hook.py
@@ -172,7 +172,7 @@ def export_to_storage_bucket(self, bucket, namespace=None,
"""
Export entities from Cloud Datastore to Cloud Storage for backup
"""
- output_uri_prefix = 'gs://' + ('/').join(filter(None, [bucket, namespace]))
+ output_uri_prefix = 'gs://' + '/'.join(filter(None, [bucket, namespace]))
if not entity_filter:
entity_filter = {}
if not labels:
@@ -191,7 +191,7 @@ def import_from_storage_bucket(self, bucket, file,
"""
Import a backup from Cloud Storage to Cloud Datastore
"""
- input_url = 'gs://' + ('/').join(filter(None, [bucket, namespace, file]))
+ input_url = 'gs://' + '/'.join(filter(None, [bucket, namespace, file]))
if not entity_filter:
entity_filter = {}
if not labels:
diff --git a/airflow/contrib/hooks/emr_hook.py b/airflow/contrib/hooks/emr_hook.py
index 6cd92c6d85..d116f2275f 100644
--- a/airflow/contrib/hooks/emr_hook.py
+++ b/airflow/contrib/hooks/emr_hook.py
@@ -23,7 +23,7 @@
class EmrHook(AwsHook):
"""
- Interact with AWS EMR. emr_conn_id is only neccessary for using the
+ Interact with AWS EMR. emr_conn_id is only necessary for using the
create_job_flow method.
"""
diff --git a/airflow/contrib/hooks/gcp_api_base_hook.py b/airflow/contrib/hooks/gcp_api_base_hook.py
index 053494743f..72d8e483a7 100644
--- a/airflow/contrib/hooks/gcp_api_base_hook.py
+++ b/airflow/contrib/hooks/gcp_api_base_hook.py
@@ -58,11 +58,11 @@ class GoogleCloudBaseHook(BaseHook, LoggingMixin):
def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
"""
:param gcp_conn_id: The connection ID to use when fetching connection info.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
@@ -87,7 +87,7 @@ def _get_credentials(self):
elif key_path:
# Get credentials from a JSON file.
if key_path.endswith('.json'):
- self.log.info('Getting connection using a JSON key file.')
+ self.log.debug('Getting connection using JSON key file %s' % key_path)
credentials = (
google.oauth2.service_account.Credentials.from_service_account_file(
key_path, scopes=scopes)
diff --git a/airflow/contrib/hooks/gcp_compute_hook.py b/airflow/contrib/hooks/gcp_compute_hook.py
new file mode 100644
index 0000000000..5fa088942b
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_compute_hook.py
@@ -0,0 +1,167 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import time
+from googleapiclient.discovery import build
+
+from airflow import AirflowException
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+# Number of retries - used by googleapiclient method calls to perform retries
+# For requests that are "retriable"
+NUM_RETRIES = 5
+
+# Time to sleep between active checks of the operation results
+TIME_TO_SLEEP_IN_SECONDS = 1
+
+
+class GceOperationStatus:
+ PENDING = "PENDING"
+ RUNNING = "RUNNING"
+ DONE = "DONE"
+
+
+# noinspection PyAbstractClass
+class GceHook(GoogleCloudBaseHook):
+ """
+ Hook for Google Compute Engine APIs.
+ """
+ _conn = None
+
+ def __init__(self,
+ api_version,
+ gcp_conn_id='google_cloud_default',
+ delegate_to=None):
+ super(GceHook, self).__init__(gcp_conn_id, delegate_to)
+ self.api_version = api_version
+
+ def get_conn(self):
+ """
+ Retrieves connection to Google Compute Engine.
+
+ :return: Google Compute Engine services object
+ :rtype: dict
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build('compute', self.api_version,
+ http=http_authorized, cache_discovery=False)
+ return self._conn
+
+ def start_instance(self, project_id, zone, resource_id):
+ """
+ Starts an existing instance defined by project_id, zone and resource_id.
+
+ :param project_id: Google Cloud Platform project where the Compute Engine
+ instance exists.
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the instance exists.
+ :type zone: str
+ :param resource_id: Name of the Compute Engine instance resource.
+ :type resource_id: str
+ :return: True if the operation succeeded, raises an error otherwise
+ :rtype: bool
+ """
+ response = self.get_conn().instances().start(
+ project=project_id,
+ zone=zone,
+ instance=resource_id
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project_id, zone, operation_name)
+
+ def stop_instance(self, project_id, zone, resource_id):
+ """
+ Stops an instance defined by project_id, zone and resource_id.
+
+ :param project_id: Google Cloud Platform project where the Compute Engine
+ instance exists.
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the instance exists.
+ :type zone: str
+ :param resource_id: Name of the Compute Engine instance resource.
+ :type resource_id: str
+ :return: True if the operation succeeded, raises an error otherwise
+ :rtype: bool
+ """
+ response = self.get_conn().instances().stop(
+ project=project_id,
+ zone=zone,
+ instance=resource_id
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project_id, zone, operation_name)
+
+ def set_machine_type(self, project_id, zone, resource_id, body):
+ """
+ Sets machine type of an instance defined by project_id, zone and resource_id.
+
+ :param project_id: Google Cloud Platform project where the Compute Engine
+ instance exists.
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the instance exists.
+ :type zone: str
+ :param resource_id: Name of the Compute Engine instance resource.
+ :type resource_id: str
+ :param body: Body required by the Compute Engine setMachineType API,
+ as described in
+ https://cloud.google.com/compute/docs/reference/rest/v1/instances/setMachineType
+ :type body: dict
+ :return: True if the operation succeeded, raises an error otherwise
+ :rtype: bool
+ """
+ response = self._execute_set_machine_type(project_id, zone, resource_id, body)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(project_id, zone, operation_name)
+
+ def _execute_set_machine_type(self, project_id, zone, resource_id, body):
+ return self.get_conn().instances().setMachineType(
+ project=project_id, zone=zone, instance=resource_id, body=body)\
+ .execute(num_retries=NUM_RETRIES)
+
+ def _wait_for_operation_to_complete(self, project_id, zone, operation_name):
+ """
+ Waits for the named operation to complete - checks status of the
+ asynchronous call.
+
+ :param operation_name: name of the operation
+ :type operation_name: str
+ :return: True if the operation succeeded, raises an error otherwise
+ :rtype: bool
+ """
+ service = self.get_conn()
+ while True:
+ operation_response = self._check_operation_status(
+ service, operation_name, project_id, zone)
+ if operation_response.get("status") == GceOperationStatus.DONE:
+ error = operation_response.get("error")
+ if error:
+ code = operation_response.get("httpErrorStatusCode")
+ msg = operation_response.get("httpErrorMessage")
+ # Extracting the errors list as string and trimming square braces
+ error_msg = str(error.get("errors"))[1:-1]
+ raise AirflowException("{} {}: ".format(code, msg) + error_msg)
+ # No meaningful info to return from the response in case of success
+ return True
+ time.sleep(TIME_TO_SLEEP_IN_SECONDS)
+
+ def _check_operation_status(self, service, operation_name, project_id, zone):
+ return service.zoneOperations().get(
+ project=project_id, zone=zone, operation=operation_name).execute(
+ num_retries=NUM_RETRIES)
diff --git a/airflow/contrib/hooks/gcp_container_hook.py b/airflow/contrib/hooks/gcp_container_hook.py
index d36d796d76..3934f07a95 100644
--- a/airflow/contrib/hooks/gcp_container_hook.py
+++ b/airflow/contrib/hooks/gcp_container_hook.py
@@ -23,7 +23,7 @@
from airflow import AirflowException, version
from airflow.hooks.base_hook import BaseHook
-from google.api_core.exceptions import AlreadyExists
+from google.api_core.exceptions import AlreadyExists, NotFound
from google.api_core.gapic_v1.method import DEFAULT
from google.cloud import container_v1, exceptions
from google.cloud.container_v1.gapic.enums import Operation
@@ -44,9 +44,11 @@ def __init__(self, project_id, location):
client_info = ClientInfo(client_library_version='airflow_v' + version.version)
self.client = container_v1.ClusterManagerClient(client_info=client_info)
- def _dict_to_proto(self, py_dict, proto):
+ @staticmethod
+ def _dict_to_proto(py_dict, proto):
"""
Converts a python dictionary to the proto supplied
+
:param py_dict: The dictionary to convert
:type py_dict: dict
:param proto: The proto object to merge with dictionary
@@ -62,6 +64,7 @@ def wait_for_operation(self, operation):
"""
Given an operation, continuously fetches the status from Google Cloud until either
completion or an error occurring
+
:param operation: The Operation to wait for
:type operation: A google.cloud.container_V1.gapic.enums.Operator
:return: A new, updated operation fetched from Google Cloud
@@ -82,6 +85,7 @@ def wait_for_operation(self, operation):
def get_operation(self, operation_name):
"""
Fetches the operation from Google Cloud
+
:param operation_name: Name of operation to fetch
:type operation_name: str
:return: The new, updated operation from Google Cloud
@@ -90,12 +94,14 @@ def get_operation(self, operation_name):
zone=self.location,
operation_id=operation_name)
- def _append_label(self, cluster_proto, key, val):
+ @staticmethod
+ def _append_label(cluster_proto, key, val):
"""
Append labels to provided Cluster Protobuf
Labels must fit the regex [a-z]([-a-z0-9]*[a-z0-9])? (current airflow version
string follows semantic versioning spec: x.y.z).
+
:param cluster_proto: The proto to append resource_label airflow version to
:type cluster_proto: google.cloud.container_v1.types.Cluster
:param key: The key label
@@ -141,7 +147,7 @@ def delete_cluster(self, name, retry=DEFAULT, timeout=DEFAULT):
op = self.wait_for_operation(op)
# Returns server-defined url for the resource
return op.self_link
- except exceptions.NotFound as error:
+ except NotFound as error:
self.log.info('Assuming Success: ' + error.message)
def create_cluster(self, cluster, retry=DEFAULT, timeout=DEFAULT):
@@ -194,6 +200,7 @@ def create_cluster(self, cluster, retry=DEFAULT, timeout=DEFAULT):
def get_cluster(self, name, retry=DEFAULT, timeout=DEFAULT):
"""
Gets details of specified cluster
+
:param name: The name of the cluster to retrieve
:type name: str
:param retry: A retry object used to retry requests. If None is specified,
diff --git a/airflow/contrib/hooks/gcp_dataflow_hook.py b/airflow/contrib/hooks/gcp_dataflow_hook.py
index 7abb413a1a..4fdb07c74d 100644
--- a/airflow/contrib/hooks/gcp_dataflow_hook.py
+++ b/airflow/contrib/hooks/gcp_dataflow_hook.py
@@ -17,6 +17,7 @@
# specific language governing permissions and limitations
# under the License.
import json
+import re
import select
import subprocess
import time
@@ -33,12 +34,13 @@
class _DataflowJob(LoggingMixin):
- def __init__(self, dataflow, project_number, name, location, poll_sleep=10):
+ def __init__(self, dataflow, project_number, name, location, poll_sleep=10,
+ job_id=None):
self._dataflow = dataflow
self._project_number = project_number
self._job_name = name
self._job_location = location
- self._job_id = None
+ self._job_id = job_id
self._job = self._get_job()
self._poll_sleep = poll_sleep
@@ -46,7 +48,7 @@ def _get_job_id_from_name(self):
jobs = self._dataflow.projects().locations().jobs().list(
projectId=self._project_number,
location=self._job_location
- ).execute()
+ ).execute(num_retries=5)
for job in jobs['jobs']:
if job['name'] == self._job_name:
self._job_id = job['id']
@@ -54,13 +56,15 @@ def _get_job_id_from_name(self):
return None
def _get_job(self):
- if self._job_name:
+ if self._job_id:
+ job = self._dataflow.projects().locations().jobs().get(
+ projectId=self._project_number,
+ location=self._job_location,
+ jobId=self._job_id).execute(num_retries=5)
+ elif self._job_name:
job = self._get_job_id_from_name()
else:
- job = self._dataflow.projects().jobs().get(
- projectId=self._project_number,
- jobId=self._job_id
- ).execute()
+ raise Exception('Missing both dataflow job ID and name.')
if job and 'currentState' in job:
self.log.info(
@@ -123,36 +127,50 @@ def __init__(self, cmd):
def _line(self, fd):
if fd == self._proc.stderr.fileno():
- lines = self._proc.stderr.readlines()
- for line in lines:
+ line = b''.join(self._proc.stderr.readlines())
+ if line:
self.log.warning(line[:-1])
- if lines:
- return lines[-1]
+ return line
if fd == self._proc.stdout.fileno():
- line = self._proc.stdout.readline()
+ line = b''.join(self._proc.stdout.readlines())
+ if line:
+ self.log.info(line[:-1])
return line
@staticmethod
def _extract_job(line):
- if line is not None:
- if line.startswith("Submitted job: "):
- return line[15:-1]
+ # Job id info: https://goo.gl/SE29y9.
+ job_id_pattern = re.compile(
+ b'.*console.cloud.google.com/dataflow.*/jobs/([a-z|0-9|A-Z|\-|\_]+).*')
+ matched_job = job_id_pattern.search(line or '')
+ if matched_job:
+ return matched_job.group(1).decode()
def wait_for_done(self):
reads = [self._proc.stderr.fileno(), self._proc.stdout.fileno()]
self.log.info("Start waiting for DataFlow process to complete.")
- while self._proc.poll() is None:
+ job_id = None
+ # Make sure logs are processed regardless whether the subprocess is
+ # terminated.
+ process_ends = False
+ while True:
ret = select.select(reads, [], [], 5)
if ret is not None:
for fd in ret[0]:
line = self._line(fd)
if line:
- self.log.debug(line[:-1])
+ job_id = job_id or self._extract_job(line)
else:
self.log.info("Waiting for DataFlow process to complete.")
+ if process_ends:
+ break
+ if self._proc.poll() is not None:
+ # Mark process completion but allows its outputs to be consumed.
+ process_ends = True
if self._proc.returncode is not 0:
raise Exception("DataFlow failed with return code {}".format(
self._proc.returncode))
+ return job_id
class DataFlowHook(GoogleCloudBaseHook):
@@ -166,20 +184,19 @@ def __init__(self,
def get_conn(self):
"""
- Returns a Google Cloud Storage service object.
+ Returns a Google Cloud Dataflow service object.
"""
http_authorized = self._authorize()
return build(
'dataflow', 'v1b3', http=http_authorized, cache_discovery=False)
- def _start_dataflow(self, task_id, variables, name,
- command_prefix, label_formatter):
+ def _start_dataflow(self, variables, name, command_prefix, label_formatter):
variables = self._set_variables(variables)
- cmd = command_prefix + self._build_cmd(task_id, variables,
- label_formatter)
- _Dataflow(cmd).wait_for_done()
+ cmd = command_prefix + self._build_cmd(variables, label_formatter)
+ job_id = _Dataflow(cmd).wait_for_done()
_DataflowJob(self.get_conn(), variables['project'], name,
- variables['region'], self.poll_sleep).wait_for_done()
+ variables['region'],
+ self.poll_sleep, job_id).wait_for_done()
@staticmethod
def _set_variables(variables):
@@ -189,12 +206,9 @@ def _set_variables(variables):
variables['region'] = DEFAULT_DATAFLOW_LOCATION
return variables
- def start_java_dataflow(self, task_id, variables, dataflow, job_class=None,
+ def start_java_dataflow(self, job_name, variables, dataflow, job_class=None,
append_job_name=True):
- if append_job_name:
- name = task_id + "-" + str(uuid.uuid1())[:8]
- else:
- name = task_id
+ name = self._build_dataflow_job_name(job_name, append_job_name)
variables['jobName'] = name
def label_formatter(labels_dict):
@@ -202,34 +216,44 @@ def label_formatter(labels_dict):
json.dumps(labels_dict).replace(' ', ''))]
command_prefix = (["java", "-cp", dataflow, job_class] if job_class
else ["java", "-jar", dataflow])
- self._start_dataflow(task_id, variables, name,
- command_prefix, label_formatter)
+ self._start_dataflow(variables, name, command_prefix, label_formatter)
- def start_template_dataflow(self, task_id, variables, parameters, dataflow_template,
+ def start_template_dataflow(self, job_name, variables, parameters, dataflow_template,
append_job_name=True):
- if append_job_name:
- name = task_id + "-" + str(uuid.uuid1())[:8]
- else:
- name = task_id
+ name = self._build_dataflow_job_name(job_name, append_job_name)
self._start_template_dataflow(
name, variables, parameters, dataflow_template)
- def start_python_dataflow(self, task_id, variables, dataflow, py_options,
+ def start_python_dataflow(self, job_name, variables, dataflow, py_options,
append_job_name=True):
- if append_job_name:
- name = task_id + "-" + str(uuid.uuid1())[:8]
- else:
- name = task_id
+ name = self._build_dataflow_job_name(job_name, append_job_name)
variables['job_name'] = name
def label_formatter(labels_dict):
return ['--labels={}={}'.format(key, value)
for key, value in labels_dict.items()]
- self._start_dataflow(task_id, variables, name,
- ["python"] + py_options + [dataflow],
+ self._start_dataflow(variables, name, ["python2"] + py_options + [dataflow],
label_formatter)
- def _build_cmd(self, task_id, variables, label_formatter):
+ @staticmethod
+ def _build_dataflow_job_name(job_name, append_job_name=True):
+ base_job_name = str(job_name).replace('_', '-')
+
+ if not re.match(r"^[a-z]([-a-z0-9]*[a-z0-9])?$", base_job_name):
+ raise ValueError(
+ 'Invalid job_name ({}); the name must consist of'
+ 'only the characters [-a-z0-9], starting with a '
+ 'letter and ending with a letter or number '.format(base_job_name))
+
+ if append_job_name:
+ safe_job_name = base_job_name + "-" + str(uuid.uuid4())[:8]
+ else:
+ safe_job_name = base_job_name
+
+ return safe_job_name
+
+ @staticmethod
+ def _build_cmd(variables, label_formatter):
command = ["--runner=DataflowRunner"]
if variables is not None:
for attr, value in variables.items():
@@ -241,7 +265,8 @@ def _build_cmd(self, task_id, variables, label_formatter):
command.append("--" + attr + "=" + value)
return command
- def _start_template_dataflow(self, name, variables, parameters, dataflow_template):
+ def _start_template_dataflow(self, name, variables, parameters,
+ dataflow_template):
# Builds RuntimeEnvironment from variables dictionary
# https://cloud.google.com/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment
environment = {}
@@ -253,9 +278,11 @@ def _start_template_dataflow(self, name, variables, parameters, dataflow_templat
"parameters": parameters,
"environment": environment}
service = self.get_conn()
- request = service.projects().templates().launch(projectId=variables['project'],
- gcsPath=dataflow_template,
- body=body)
+ request = service.projects().templates().launch(
+ projectId=variables['project'],
+ gcsPath=dataflow_template,
+ body=body
+ )
response = request.execute()
variables = self._set_variables(variables)
_DataflowJob(self.get_conn(), variables['project'], name, variables['region'],
diff --git a/airflow/contrib/hooks/gcp_dataproc_hook.py b/airflow/contrib/hooks/gcp_dataproc_hook.py
index fc15137cbf..ca86f08795 100644
--- a/airflow/contrib/hooks/gcp_dataproc_hook.py
+++ b/airflow/contrib/hooks/gcp_dataproc_hook.py
@@ -21,13 +21,15 @@
import uuid
from apiclient.discovery import build
+from zope.deprecation import deprecation
from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
from airflow.utils.log.logging_mixin import LoggingMixin
class _DataProcJob(LoggingMixin):
- def __init__(self, dataproc_api, project_id, job, region='global'):
+ def __init__(self, dataproc_api, project_id, job, region='global',
+ job_error_states=None):
self.dataproc_api = dataproc_api
self.project_id = project_id
self.region = region
@@ -36,6 +38,7 @@ def __init__(self, dataproc_api, project_id, job, region='global'):
region=self.region,
body=job).execute()
self.job_id = self.job['reference']['jobId']
+ self.job_error_states = job_error_states
self.log.info(
'DataProc job %s is %s',
self.job_id, str(self.job['status']['state'])
@@ -48,19 +51,23 @@ def wait_for_done(self):
region=self.region,
jobId=self.job_id).execute(num_retries=5)
if 'ERROR' == self.job['status']['state']:
- print(str(self.job))
self.log.error('DataProc job %s has errors', self.job_id)
self.log.error(self.job['status']['details'])
self.log.debug(str(self.job))
+ self.log.info('Driver output location: %s',
+ self.job['driverOutputResourceUri'])
return False
if 'CANCELLED' == self.job['status']['state']:
- print(str(self.job))
self.log.warning('DataProc job %s is cancelled', self.job_id)
if 'details' in self.job['status']:
self.log.warning(self.job['status']['details'])
self.log.debug(str(self.job))
+ self.log.info('Driver output location: %s',
+ self.job['driverOutputResourceUri'])
return False
if 'DONE' == self.job['status']['state']:
+ self.log.info('Driver output location: %s',
+ self.job['driverOutputResourceUri'])
return True
self.log.debug(
'DataProc job %s is %s',
@@ -69,10 +76,15 @@ def wait_for_done(self):
time.sleep(5)
def raise_error(self, message=None):
- if 'ERROR' == self.job['status']['state']:
- if message is None:
- message = "Google DataProc job has error"
- raise Exception(message + ": " + str(self.job['status']['details']))
+ job_state = self.job['status']['state']
+ # We always consider ERROR to be an error state.
+ if ((self.job_error_states and job_state in self.job_error_states)
+ or 'ERROR' == job_state):
+ ex_message = message or ("Google DataProc job has state: %s" % job_state)
+ ex_details = (str(self.job['status']['details'])
+ if 'details' in self.job['status']
+ else "No details available")
+ raise Exception(ex_message + ": " + ex_details)
def get(self):
return self.job
@@ -80,7 +92,7 @@ def get(self):
class _DataProcJobBuilder:
def __init__(self, project_id, task_id, cluster_name, job_type, properties):
- name = task_id + "_" + str(uuid.uuid1())[:8]
+ name = task_id + "_" + str(uuid.uuid4())[:8]
self.job_type = job_type
self.job = {
"job": {
@@ -140,7 +152,7 @@ def set_python_main(self, main):
self.job["job"][self.job_type]["mainPythonFileUri"] = main
def set_job_name(self, name):
- self.job["job"]["reference"]["jobId"] = name + "_" + str(uuid.uuid1())[:8]
+ self.job["job"]["reference"]["jobId"] = name + "_" + str(uuid.uuid4())[:8]
def build(self):
return self.job
@@ -215,16 +227,26 @@ def get_cluster(self, project_id, region, cluster_name):
clusterName=cluster_name
).execute(num_retries=5)
- def submit(self, project_id, job, region='global'):
- submitted = _DataProcJob(self.get_conn(), project_id, job, region)
+ def submit(self, project_id, job, region='global', job_error_states=None):
+ submitted = _DataProcJob(self.get_conn(), project_id, job, region,
+ job_error_states=job_error_states)
if not submitted.wait_for_done():
- submitted.raise_error('DataProcTask has errors')
+ submitted.raise_error()
def create_job_template(self, task_id, cluster_name, job_type, properties):
return _DataProcJobBuilder(self.project_id, task_id, cluster_name,
job_type, properties)
- def await(self, operation):
+ def wait(self, operation):
"""Awaits for Google Cloud Dataproc Operation to complete."""
submitted = _DataProcOperation(self.get_conn(), operation)
submitted.wait_for_done()
+
+
+setattr(
+ DataProcHook,
+ "await",
+ deprecation.deprecated(
+ DataProcHook.wait, "renamed to 'wait' for Python3.7 compatibility"
+ ),
+)
diff --git a/airflow/contrib/hooks/gcp_function_hook.py b/airflow/contrib/hooks/gcp_function_hook.py
new file mode 100644
index 0000000000..d2da19285a
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_function_hook.py
@@ -0,0 +1,195 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import time
+import requests
+from googleapiclient.discovery import build
+
+from airflow import AirflowException
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+# Number of retries - used by googleapiclient method calls to perform retries
+# For requests that are "retriable"
+NUM_RETRIES = 5
+
+# Time to sleep between active checks of the operation results
+TIME_TO_SLEEP_IN_SECONDS = 1
+
+
+# noinspection PyAbstractClass
+class GcfHook(GoogleCloudBaseHook):
+ """
+ Hook for Google Cloud Functions APIs.
+ """
+ _conn = None
+
+ def __init__(self,
+ api_version,
+ gcp_conn_id='google_cloud_default',
+ delegate_to=None):
+ super(GcfHook, self).__init__(gcp_conn_id, delegate_to)
+ self.api_version = api_version
+
+ def get_conn(self):
+ """
+ Retrieves connection to cloud functions.
+
+ :return: Google Cloud Function services object
+ :rtype: dict
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build('cloudfunctions', self.api_version,
+ http=http_authorized, cache_discovery=False)
+ return self._conn
+
+ def get_function(self, name):
+ """
+ Returns the function with a given name.
+
+ :param name: name of the function
+ :type name: str
+ :return: a CloudFunction object representing the function
+ :rtype: dict
+ """
+ return self.get_conn().projects().locations().functions().get(
+ name=name).execute(num_retries=NUM_RETRIES)
+
+ def list_functions(self, full_location):
+ """
+ Lists all functions created in the location.
+
+ :param full_location: full location including project. On the form
+ of /projects/<PROJECT>/location/<LOCATION>
+ :type full_location: str
+ :return: array of CloudFunction objects - representing functions in the location
+ :rtype: [dict]
+ """
+ list_response = self.get_conn().projects().locations().functions().list(
+ parent=full_location).execute(num_retries=NUM_RETRIES)
+ return list_response.get("functions", [])
+
+ def create_new_function(self, full_location, body):
+ """
+ Creates new cloud function in location given with body specified.
+
+ :param full_location: full location including project. On the form
+ of /projects/<PROJECT>/location/<LOCATION>
+ :type full_location: str
+ :param body: body required by the cloud function insert API
+ :type body: dict
+ :return: response returned by the operation
+ :rtype: dict
+ """
+ response = self.get_conn().projects().locations().functions().create(
+ location=full_location,
+ body=body
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(operation_name)
+
+ def update_function(self, name, body, update_mask):
+ """
+ Updates cloud function according to the update mask specified.
+
+ :param name: name of the function
+ :type name: str
+ :param body: body required by the cloud function patch API
+ :type body: str
+ :param update_mask: update mask - array of fields that should be patched
+ :type update_mask: [str]
+ :return: response returned by the operation
+ :rtype: dict
+ """
+ response = self.get_conn().projects().locations().functions().patch(
+ updateMask=",".join(update_mask),
+ name=name,
+ body=body
+ ).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(operation_name)
+
+ def upload_function_zip(self, parent, zip_path):
+ """
+ Uploads zip file with sources.
+
+ :param parent: project and location in which signed upload URL should be generated
+ in the form of /projects/<PROJECT>/location/<LOCATION>
+ :type parent: str
+ :param zip_path: path of the file to upload (should point to valid .zip file)
+ :type zip_path: str
+ :return: Upload URL that was returned by generateUploadUrl method
+ """
+ response = self.get_conn().projects().locations().functions().generateUploadUrl(
+ parent=parent
+ ).execute(num_retries=NUM_RETRIES)
+ upload_url = response.get('uploadUrl')
+ with open(zip_path, 'rb') as fp:
+ requests.put(
+ url=upload_url,
+ data=fp.read(),
+ # Those two headers needs to be specified according to:
+ # https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions/generateUploadUrl
+ # nopep8
+ headers={
+ 'Content-type': 'application/zip',
+ 'x-goog-content-length-range': '0,104857600',
+ }
+ )
+ return upload_url
+
+ def delete_function(self, name):
+ """
+ Deletes cloud function specified by name.
+
+ :param name: name of the function
+ :type name: str
+ :return: response returned by the operation
+ :rtype: dict
+ """
+ response = self.get_conn().projects().locations().functions().delete(
+ name=name).execute(num_retries=NUM_RETRIES)
+ operation_name = response["name"]
+ return self._wait_for_operation_to_complete(operation_name)
+
+ def _wait_for_operation_to_complete(self, operation_name):
+ """
+ Waits for the named operation to complete - checks status of the
+ asynchronous call.
+
+ :param operation_name: name of the operation
+ :type operation_name: str
+ :return: response returned by the operation
+ :rtype: dict
+ :exception: AirflowException in case error is returned
+ """
+ service = self.get_conn()
+ while True:
+ operation_response = service.operations().get(
+ name=operation_name,
+ ).execute(num_retries=NUM_RETRIES)
+ if operation_response.get("done"):
+ response = operation_response.get("response")
+ error = operation_response.get("error")
+ # Note, according to documentation always either response or error is
+ # set when "done" == True
+ if error:
+ raise AirflowException(str(error))
+ return response
+ time.sleep(TIME_TO_SLEEP_IN_SECONDS)
diff --git a/airflow/contrib/hooks/gcp_kms_hook.py b/airflow/contrib/hooks/gcp_kms_hook.py
new file mode 100644
index 0000000000..6f2b3aedff
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_kms_hook.py
@@ -0,0 +1,108 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import base64
+
+from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook
+
+from apiclient.discovery import build
+
+
+def _b64encode(s):
+ """ Base 64 encodes a bytes object to a string """
+ return base64.b64encode(s).decode('ascii')
+
+
+def _b64decode(s):
+ """ Base 64 decodes a string to bytes. """
+ return base64.b64decode(s.encode('utf-8'))
+
+
+class GoogleCloudKMSHook(GoogleCloudBaseHook):
+ """
+ Interact with Google Cloud KMS. This hook uses the Google Cloud Platform
+ connection.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleCloudKMSHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a KMS service object.
+
+ :rtype: apiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build(
+ 'cloudkms', 'v1', http=http_authorized, cache_discovery=False)
+
+ def encrypt(self, key_name, plaintext, authenticated_data=None):
+ """
+ Encrypts a plaintext message using Google Cloud KMS.
+
+ :param key_name: The Resource Name for the key (or key version)
+ to be used for encyption. Of the form
+ ``projects/*/locations/*/keyRings/*/cryptoKeys/**``
+ :type key_name: str
+ :param plaintext: The message to be encrypted.
+ :type plaintext: bytes
+ :param authenticated_data: Optional additional authenticated data that
+ must also be provided to decrypt the message.
+ :type authenticated_data: bytes
+ :return: The base 64 encoded ciphertext of the original message.
+ :rtype: str
+ """
+ keys = self.get_conn().projects().locations().keyRings().cryptoKeys()
+ body = {'plaintext': _b64encode(plaintext)}
+ if authenticated_data:
+ body['additionalAuthenticatedData'] = _b64encode(authenticated_data)
+
+ request = keys.encrypt(name=key_name, body=body)
+ response = request.execute()
+
+ ciphertext = response['ciphertext']
+ return ciphertext
+
+ def decrypt(self, key_name, ciphertext, authenticated_data=None):
+ """
+ Decrypts a ciphertext message using Google Cloud KMS.
+
+ :param key_name: The Resource Name for the key to be used for decyption.
+ Of the form ``projects/*/locations/*/keyRings/*/cryptoKeys/**``
+ :type key_name: str
+ :param ciphertext: The message to be decrypted.
+ :type ciphertext: str
+ :param authenticated_data: Any additional authenticated data that was
+ provided when encrypting the message.
+ :type authenticated_data: bytes
+ :return: The original message.
+ :rtype: bytes
+ """
+ keys = self.get_conn().projects().locations().keyRings().cryptoKeys()
+ body = {'ciphertext': ciphertext}
+ if authenticated_data:
+ body['additionalAuthenticatedData'] = _b64encode(authenticated_data)
+
+ request = keys.decrypt(name=key_name, body=body)
+ response = request.execute()
+
+ plaintext = _b64decode(response['plaintext'])
+ return plaintext
diff --git a/airflow/contrib/hooks/gcp_mlengine_hook.py b/airflow/contrib/hooks/gcp_mlengine_hook.py
index 66f392b156..9c255dbe80 100644
--- a/airflow/contrib/hooks/gcp_mlengine_hook.py
+++ b/airflow/contrib/hooks/gcp_mlengine_hook.py
@@ -63,7 +63,7 @@ def create_job(self, project_id, job, use_existing_job_fn=None):
:param project_id: The Google Cloud project id within which MLEngine
job will be launched.
- :type project_id: string
+ :type project_id: str
:param job: MLEngine Job object that should be provided to the MLEngine
API, such as: ::
@@ -152,7 +152,8 @@ def _wait_for_job_done(self, project_id, job_id, interval=30):
apiclient.errors.HttpError: if HTTP error is returned when getting
the job
"""
- assert interval > 0
+ if interval <= 0:
+ raise ValueError("Interval must be > 0")
while True:
job = self._get_job(project_id, job_id)
if job['state'] in ['SUCCEEDED', 'FAILED', 'CANCELLED']:
@@ -242,7 +243,9 @@ def create_model(self, project_id, model):
"""
Create a Model. Blocks until finished.
"""
- assert model['name'] is not None and model['name'] is not ''
+ if not model['name']:
+ raise ValueError("Model name must be provided and "
+ "could not be an empty string")
project = 'projects/{}'.format(project_id)
request = self._mlengine.projects().models().create(
@@ -253,7 +256,9 @@ def get_model(self, project_id, model_name):
"""
Gets a Model. Blocks until finished.
"""
- assert model_name is not None and model_name is not ''
+ if not model_name:
+ raise ValueError("Model name must be provided and "
+ "it could not be an empty string")
full_model_name = 'projects/{}/models/{}'.format(
project_id, model_name)
request = self._mlengine.projects().models().get(name=full_model_name)
diff --git a/airflow/contrib/hooks/gcp_pubsub_hook.py b/airflow/contrib/hooks/gcp_pubsub_hook.py
index 1d55d5d487..bf33a9b6bc 100644
--- a/airflow/contrib/hooks/gcp_pubsub_hook.py
+++ b/airflow/contrib/hooks/gcp_pubsub_hook.py
@@ -60,10 +60,10 @@ def publish(self, project, topic, messages):
"""Publishes messages to a Pub/Sub topic.
:param project: the GCP project ID in which to publish
- :type project: string
+ :type project: str
:param topic: the Pub/Sub topic to which to publish; do not
include the ``projects/{project}/topics/`` prefix.
- :type topic: string
+ :type topic: str
:param messages: messages to publish; if the data field in a
message is set, it should already be base64 encoded.
:type messages: list of PubSub messages; see
@@ -84,10 +84,10 @@ def create_topic(self, project, topic, fail_if_exists=False):
:param project: the GCP project ID in which to create
the topic
- :type project: string
+ :type project: str
:param topic: the Pub/Sub topic name to create; do not
include the ``projects/{project}/topics/`` prefix.
- :type topic: string
+ :type topic: str
:param fail_if_exists: if set, raise an exception if the topic
already exists
:type fail_if_exists: bool
@@ -112,10 +112,10 @@ def delete_topic(self, project, topic, fail_if_not_exists=False):
"""Deletes a Pub/Sub topic if it exists.
:param project: the GCP project ID in which to delete the topic
- :type project: string
+ :type project: str
:param topic: the Pub/Sub topic name to delete; do not
include the ``projects/{project}/topics/`` prefix.
- :type topic: string
+ :type topic: str
:param fail_if_not_exists: if set, raise an exception if the topic
does not exist
:type fail_if_not_exists: bool
@@ -142,17 +142,17 @@ def create_subscription(self, topic_project, topic, subscription=None,
:param topic_project: the GCP project ID of the topic that the
subscription will be bound to.
- :type topic_project: string
+ :type topic_project: str
:param topic: the Pub/Sub topic name that the subscription will be bound
to create; do not include the ``projects/{project}/subscriptions/``
prefix.
- :type topic: string
+ :type topic: str
:param subscription: the Pub/Sub subscription name. If empty, a random
name will be generated using the uuid module
- :type subscription: string
+ :type subscription: str
:param subscription_project: the GCP project ID where the subscription
will be created. If unspecified, ``topic_project`` will be used.
- :type subscription_project: string
+ :type subscription_project: str
:param ack_deadline_secs: Number of seconds that a subscriber has to
acknowledge each message pulled from the subscription
:type ack_deadline_secs: int
@@ -161,7 +161,7 @@ def create_subscription(self, topic_project, topic, subscription=None,
:type fail_if_exists: bool
:return: subscription name which will be the system-generated value if
the ``subscription`` parameter is not supplied
- :rtype: string
+ :rtype: str
"""
service = self.get_conn()
full_topic = _format_topic(topic_project, topic)
@@ -197,10 +197,10 @@ def delete_subscription(self, project, subscription,
"""Deletes a Pub/Sub subscription, if it exists.
:param project: the GCP project ID where the subscription exists
- :type project: string
+ :type project: str
:param subscription: the Pub/Sub subscription name to delete; do not
include the ``projects/{project}/subscriptions/`` prefix.
- :type subscription: string
+ :type subscription: str
:param fail_if_not_exists: if set, raise an exception if the topic
does not exist
:type fail_if_not_exists: bool
@@ -228,10 +228,10 @@ def pull(self, project, subscription, max_messages,
"""Pulls up to ``max_messages`` messages from Pub/Sub subscription.
:param project: the GCP project ID where the subscription exists
- :type project: string
+ :type project: str
:param subscription: the Pub/Sub subscription name to pull from; do not
include the 'projects/{project}/topics/' prefix.
- :type subscription: string
+ :type subscription: str
:param max_messages: The maximum number of messages to return from
the Pub/Sub API.
:type max_messages: int
@@ -265,10 +265,10 @@ def acknowledge(self, project, subscription, ack_ids):
:param project: the GCP project name or ID in which to create
the topic
- :type project: string
+ :type project: str
:param subscription: the Pub/Sub subscription name to delete; do not
include the 'projects/{project}/topics/' prefix.
- :type subscription: string
+ :type subscription: str
:param ack_ids: List of ReceivedMessage ackIds from a previous pull
response
:type ack_ids: list
diff --git a/airflow/contrib/hooks/gcs_hook.py b/airflow/contrib/hooks/gcs_hook.py
index c5e356f41c..6cfa1cf565 100644
--- a/airflow/contrib/hooks/gcs_hook.py
+++ b/airflow/contrib/hooks/gcs_hook.py
@@ -57,14 +57,15 @@ def copy(self, source_bucket, source_object, destination_bucket=None,
source bucket/object is used, but not both.
:param source_bucket: The bucket of the object to copy from.
- :type source_bucket: string
+ :type source_bucket: str
:param source_object: The object to copy.
- :type source_object: string
+ :type source_object: str
:param destination_bucket: The destination of the object to copied to.
Can be omitted; then the same bucket is used.
- :type destination_bucket: string
+ :type destination_bucket: str
:param destination_object: The (renamed) path of the object if given.
Can be omitted; then the same name is used.
+ :type destination_object: str
"""
destination_bucket = destination_bucket or source_bucket
destination_object = destination_object or source_object
@@ -102,11 +103,11 @@ def rewrite(self, source_bucket, source_object, destination_bucket,
destination_object can be omitted, in which case source_object is used.
:param source_bucket: The bucket of the object to copy from.
- :type source_bucket: string
+ :type source_bucket: str
:param source_object: The object to copy.
- :type source_object: string
+ :type source_object: str
:param destination_bucket: The destination of the object to copied to.
- :type destination_bucket: string
+ :type destination_bucket: str
:param destination_object: The (renamed) path of the object if given.
Can be omitted; then the same name is used.
"""
@@ -150,11 +151,11 @@ def download(self, bucket, object, filename=None):
Get a file from Google Cloud Storage.
:param bucket: The bucket to fetch from.
- :type bucket: string
+ :type bucket: str
:param object: The object to fetch.
- :type object: string
+ :type object: str
:param filename: If set, a local file path where the file should be written to.
- :type filename: string
+ :type filename: str
"""
service = self.get_conn()
downloaded_file_bytes = service \
@@ -176,13 +177,13 @@ def upload(self, bucket, object, filename, mime_type='application/octet-stream')
Uploads a local file to Google Cloud Storage.
:param bucket: The bucket to upload to.
- :type bucket: string
+ :type bucket: str
:param object: The object name to set when uploading the local file.
- :type object: string
+ :type object: str
:param filename: The local file path to the file to be uploaded.
- :type filename: string
+ :type filename: str
:param mime_type: The MIME type to set when uploading the file.
- :type mime_type: string
+ :type mime_type: str
"""
service = self.get_conn()
media = MediaFileUpload(filename, mime_type)
@@ -203,10 +204,10 @@ def exists(self, bucket, object):
Checks for the existence of a file in Google Cloud Storage.
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
+ :type bucket: str
:param object: The name of the object to check in the Google cloud
storage bucket.
- :type object: string
+ :type object: str
"""
service = self.get_conn()
try:
@@ -226,10 +227,10 @@ def is_updated_after(self, bucket, object, ts):
Checks if an object is updated in Google Cloud Storage.
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
+ :type bucket: str
:param object: The name of the object to check in the Google cloud
storage bucket.
- :type object: string
+ :type object: str
:param ts: The timestamp to check against.
:type ts: datetime
"""
@@ -265,11 +266,11 @@ def delete(self, bucket, object, generation=None):
parameter is used.
:param bucket: name of the bucket, where the object resides
- :type bucket: string
+ :type bucket: str
:param object: name of the object to delete
- :type object: string
+ :type object: str
:param generation: if present, permanently delete the object of this generation
- :type generation: string
+ :type generation: str
:return: True if succeeded
"""
service = self.get_conn()
@@ -290,23 +291,23 @@ def list(self, bucket, versions=None, maxResults=None, prefix=None, delimiter=No
List all objects from the bucket with the give string prefix in name
:param bucket: bucket name
- :type bucket: string
+ :type bucket: str
:param versions: if true, list all versions of the objects
- :type versions: boolean
+ :type versions: bool
:param maxResults: max count of items to return in a single page of responses
- :type maxResults: integer
+ :type maxResults: int
:param prefix: prefix string which filters objects whose name begin with
this prefix
- :type prefix: string
+ :type prefix: str
:param delimiter: filters objects based on the delimiter (for e.g '.csv')
- :type delimiter: string
+ :type delimiter: str
:return: a stream of object names matching the filtering criteria
"""
service = self.get_conn()
ids = list()
pageToken = None
- while(True):
+ while True:
response = service.objects().list(
bucket=bucket,
versions=versions,
@@ -343,9 +344,9 @@ def get_size(self, bucket, object):
Gets the size of a file in Google Cloud Storage.
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
+ :type bucket: str
:param object: The name of the object to check in the Google cloud storage bucket.
- :type object: string
+ :type object: str
"""
self.log.info('Checking the file size of object: %s in bucket: %s',
@@ -374,10 +375,10 @@ def get_crc32c(self, bucket, object):
Gets the CRC32c checksum of an object in Google Cloud Storage.
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
+ :type bucket: str
:param object: The name of the object to check in the Google cloud
storage bucket.
- :type object: string
+ :type object: str
"""
self.log.info('Retrieving the crc32c checksum of '
'object: %s in bucket: %s', object, bucket)
@@ -401,10 +402,10 @@ def get_md5hash(self, bucket, object):
Gets the MD5 hash of an object in Google Cloud Storage.
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
+ :type bucket: str
:param object: The name of the object to check in the Google cloud
storage bucket.
- :type object: string
+ :type object: str
"""
self.log.info('Retrieving the MD5 hash of '
'object: %s in bucket: %s', object, bucket)
@@ -439,7 +440,7 @@ def create_bucket(self,
https://cloud.google.com/storage/docs/bucketnaming.html#requirements
:param bucket_name: The name of the bucket.
- :type bucket_name: string
+ :type bucket_name: str
:param storage_class: This defines how objects in the bucket are stored
and determines the SLA and the cost of storage. Values include
@@ -450,7 +451,7 @@ def create_bucket(self,
- ``COLDLINE``.
If this value is not specified when the bucket is
created, it will default to STANDARD.
- :type storage_class: string
+ :type storage_class: str
:param location: The location of the bucket.
Object data for objects in the bucket resides in physical storage
within this region. Defaults to US.
@@ -458,9 +459,9 @@ def create_bucket(self,
.. seealso::
https://developers.google.com/storage/docs/bucket-locations
- :type location: string
+ :type location: str
:param project_id: The ID of the GCP Project.
- :type project_id: string
+ :type project_id: str
:param labels: User-provided labels, in key/value pairs.
:type labels: dict
:return: If successful, it returns the ``id`` of the bucket.
@@ -477,15 +478,16 @@ def create_bucket(self,
self.log.info('Creating Bucket: %s; Location: %s; Storage Class: %s',
bucket_name, location, storage_class)
- assert storage_class in storage_classes, \
- 'Invalid value ({}) passed to storage_class. Value should be ' \
- 'one of {}'.format(storage_class, storage_classes)
+ if storage_class not in storage_classes:
+ raise ValueError(
+ 'Invalid value ({}) passed to storage_class. Value should be '
+ 'one of {}'.format(storage_class, storage_classes))
- assert re.match('[a-zA-Z0-9]+', bucket_name[0]), \
- 'Bucket names must start with a number or letter.'
+ if not re.match('[a-zA-Z0-9]+', bucket_name[0]):
+ raise ValueError('Bucket names must start with a number or letter.')
- assert re.match('[a-zA-Z0-9]+', bucket_name[-1]), \
- 'Bucket names must end with a number or letter.'
+ if not re.match('[a-zA-Z0-9]+', bucket_name[-1]):
+ raise ValueError('Bucket names must end with a number or letter.')
service = self.get_conn()
bucket_resource = {
diff --git a/airflow/contrib/hooks/jira_hook.py b/airflow/contrib/hooks/jira_hook.py
index c6806d935a..a56c414dce 100644
--- a/airflow/contrib/hooks/jira_hook.py
+++ b/airflow/contrib/hooks/jira_hook.py
@@ -29,7 +29,7 @@ class JiraHook(BaseHook, LoggingMixin):
Jira interaction hook, a Wrapper around JIRA Python SDK.
:param jira_conn_id: reference to a pre-defined Jira Connection
- :type jira_conn_id: string
+ :type jira_conn_id: str
"""
def __init__(self,
jira_conn_id='jira_default',
diff --git a/airflow/contrib/hooks/mongo_hook.py b/airflow/contrib/hooks/mongo_hook.py
index 6ae71a8c8d..80ceddec14 100644
--- a/airflow/contrib/hooks/mongo_hook.py
+++ b/airflow/contrib/hooks/mongo_hook.py
@@ -37,6 +37,13 @@ def __init__(self, conn_id='mongo_default', *args, **kwargs):
self.extras = self.connection.extra_dejson
self.client = None
+ def __enter__(self):
+ return self
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ if self.client is not None:
+ self.close_conn()
+
def get_conn(self):
"""
Fetches PyMongo Client
@@ -67,6 +74,12 @@ def get_conn(self):
return self.client
+ def close_conn(self):
+ client = self.client
+ if client is not None:
+ client.close()
+ self.client = None
+
def get_collection(self, mongo_collection, mongo_db=None):
"""
Fetches a mongo collection object for querying.
diff --git a/airflow/contrib/hooks/qubole_hook.py b/airflow/contrib/hooks/qubole_hook.py
index fb45862e88..3df77d3a1f 100755
--- a/airflow/contrib/hooks/qubole_hook.py
+++ b/airflow/contrib/hooks/qubole_hook.py
@@ -66,10 +66,12 @@
'user_program_arguments'],
'dbexportcmd': ['mode', 'hive_table', 'partition_spec', 'dbtap_id', 'db_table',
'db_update_mode', 'db_update_keys', 'export_dir',
- 'fields_terminated_by', 'tags', 'name'],
+ 'fields_terminated_by', 'tags', 'name', 'customer_cluster_label',
+ 'use_customer_cluster', 'additional_options'],
'dbimportcmd': ['mode', 'hive_table', 'dbtap_id', 'db_table', 'where_clause',
'parallelism', 'extract_query', 'boundary_query', 'split_column',
- 'tags', 'name']
+ 'tags', 'name', 'hive_serde', 'customer_cluster_label',
+ 'use_customer_cluster', 'schema', 'additional_options']
}
@@ -123,7 +125,7 @@ def execute(self, context):
def kill(self, ti):
"""
- Kill (cancel) a Qubole commmand
+ Kill (cancel) a Qubole command
:param ti: Task Instance of the dag, used to determine the Quboles command id
:return: response from Qubole
"""
diff --git a/airflow/contrib/hooks/redis_hook.py b/airflow/contrib/hooks/redis_hook.py
index 1de75dbca9..650cc9308b 100644
--- a/airflow/contrib/hooks/redis_hook.py
+++ b/airflow/contrib/hooks/redis_hook.py
@@ -88,6 +88,6 @@ def key_exists(self, key):
Checks if a key exists in Redis database
:param key: The key to check the existence.
- :type key: string
+ :type key: str
"""
return self.get_conn().exists(key)
diff --git a/airflow/contrib/hooks/sagemaker_hook.py b/airflow/contrib/hooks/sagemaker_hook.py
new file mode 100644
index 0000000000..bc096ff55f
--- /dev/null
+++ b/airflow/contrib/hooks/sagemaker_hook.py
@@ -0,0 +1,314 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import copy
+import time
+from botocore.exceptions import ClientError
+
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.aws_hook import AwsHook
+from airflow.hooks.S3_hook import S3Hook
+
+
+class SageMakerHook(AwsHook):
+ """
+ Interact with Amazon SageMaker.
+ sagemaker_conn_id is required for using
+ the config stored in db for training/tuning
+ """
+ non_terminal_states = {'InProgress', 'Stopping', 'Stopped'}
+ failed_states = {'Failed'}
+
+ def __init__(self,
+ sagemaker_conn_id=None,
+ use_db_config=False,
+ region_name=None,
+ check_interval=5,
+ max_ingestion_time=None,
+ *args, **kwargs):
+ super(SageMakerHook, self).__init__(*args, **kwargs)
+ self.sagemaker_conn_id = sagemaker_conn_id
+ self.use_db_config = use_db_config
+ self.region_name = region_name
+ self.check_interval = check_interval
+ self.max_ingestion_time = max_ingestion_time
+ self.conn = self.get_conn()
+
+ def check_for_url(self, s3url):
+ """
+ check if the s3url exists
+ :param s3url: S3 url
+ :type s3url:str
+ :return: bool
+ """
+ bucket, key = S3Hook.parse_s3_url(s3url)
+ s3hook = S3Hook(aws_conn_id=self.aws_conn_id)
+ if not s3hook.check_for_bucket(bucket_name=bucket):
+ raise AirflowException(
+ "The input S3 Bucket {} does not exist ".format(bucket))
+ if key and not s3hook.check_for_key(key=key, bucket_name=bucket)\
+ and not s3hook.check_for_prefix(
+ prefix=key, bucket_name=bucket, delimiter='/'):
+ # check if s3 key exists in the case user provides a single file
+ # or if s3 prefix exists in the case user provides a prefix for files
+ raise AirflowException("The input S3 Key "
+ "or Prefix {} does not exist in the Bucket {}"
+ .format(s3url, bucket))
+ return True
+
+ def check_valid_training_input(self, training_config):
+ """
+ Run checks before a training starts
+ :param training_config: training_config
+ :type training_config: dict
+ :return: None
+ """
+ for channel in training_config['InputDataConfig']:
+ self.check_for_url(channel['DataSource']
+ ['S3DataSource']['S3Uri'])
+
+ def check_valid_tuning_input(self, tuning_config):
+ """
+ Run checks before a tuning job starts
+ :param tuning_config: tuning_config
+ :type tuning_config: dict
+ :return: None
+ """
+ for channel in tuning_config['TrainingJobDefinition']['InputDataConfig']:
+ self.check_for_url(channel['DataSource']
+ ['S3DataSource']['S3Uri'])
+
+ def check_status(self, non_terminal_states,
+ failed_state, key,
+ describe_function, *args):
+ """
+ :param non_terminal_states: the set of non_terminal states
+ :type non_terminal_states: set
+ :param failed_state: the set of failed states
+ :type failed_state: set
+ :param key: the key of the response dict
+ that points to the state
+ :type key: str
+ :param describe_function: the function used to retrieve the status
+ :type describe_function: python callable
+ :param args: the arguments for the function
+ :return: None
+ """
+ sec = 0
+ running = True
+
+ while running:
+
+ sec = sec + self.check_interval
+
+ if self.max_ingestion_time and sec > self.max_ingestion_time:
+ # ensure that the job gets killed if the max ingestion time is exceeded
+ raise AirflowException("SageMaker job took more than "
+ "%s seconds", self.max_ingestion_time)
+
+ time.sleep(self.check_interval)
+ try:
+ response = describe_function(*args)
+ status = response[key]
+ self.log.info("Job still running for %s seconds... "
+ "current status is %s" % (sec, status))
+ except KeyError:
+ raise AirflowException("Could not get status of the SageMaker job")
+ except ClientError:
+ raise AirflowException("AWS request failed, check log for more info")
+
+ if status in non_terminal_states:
+ running = True
+ elif status in failed_state:
+ raise AirflowException("SageMaker job failed because %s"
+ % response['FailureReason'])
+ else:
+ running = False
+
+ self.log.info('SageMaker Job Compeleted')
+
+ def get_conn(self):
+ """
+ Establish an AWS connection
+ :return: a boto3 SageMaker client
+ """
+ return self.get_client_type('sagemaker', region_name=self.region_name)
+
+ def list_training_job(self, name_contains=None, status_equals=None):
+ """
+ List the training jobs associated with the given input
+ :param name_contains: A string in the training job name
+ :type name_contains: str
+ :param status_equals: 'InProgress'|'Completed'
+ |'Failed'|'Stopping'|'Stopped'
+ :return:dict
+ """
+ return self.conn.list_training_jobs(
+ NameContains=name_contains, StatusEquals=status_equals)
+
+ def list_tuning_job(self, name_contains=None, status_equals=None):
+ """
+ List the tuning jobs associated with the given input
+ :param name_contains: A string in the training job name
+ :type name_contains: str
+ :param status_equals: 'InProgress'|'Completed'
+ |'Failed'|'Stopping'|'Stopped'
+ :return:dict
+ """
+ return self.conn.list_hyper_parameter_tuning_job(
+ NameContains=name_contains, StatusEquals=status_equals)
+
+ def create_training_job(self, training_job_config, wait_for_completion=True):
+ """
+ Create a training job
+ :param training_job_config: the config for training
+ :type training_job_config: dict
+ :param wait_for_completion: if the program should keep running until job finishes
+ :type wait_for_completion: bool
+ :return: A dict that contains ARN of the training job.
+ """
+ if self.use_db_config:
+ if not self.sagemaker_conn_id:
+ raise AirflowException("SageMaker connection id must be present to read \
+ SageMaker training jobs configuration.")
+ sagemaker_conn = self.get_connection(self.sagemaker_conn_id)
+
+ config = copy.deepcopy(sagemaker_conn.extra_dejson)
+ training_job_config.update(config)
+
+ self.check_valid_training_input(training_job_config)
+
+ response = self.conn.create_training_job(
+ **training_job_config)
+ if wait_for_completion:
+ self.check_status(SageMakerHook.non_terminal_states,
+ SageMakerHook.failed_states,
+ 'TrainingJobStatus',
+ self.describe_training_job,
+ training_job_config['TrainingJobName'])
+ return response
+
+ def create_tuning_job(self, tuning_job_config, wait_for_completion=True):
+ """
+ Create a tuning job
+ :param tuning_job_config: the config for tuning
+ :type tuning_job_config: dict
+ :param wait_for_completion: if the program should keep running until job finishes
+ :param wait_for_completion: bool
+ :return: A dict that contains ARN of the tuning job.
+ """
+ if self.use_db_config:
+ if not self.sagemaker_conn_id:
+ raise AirflowException(
+ "SageMaker connection id must be present to \
+ read SageMaker tunning job configuration.")
+
+ sagemaker_conn = self.get_connection(self.sagemaker_conn_id)
+
+ config = sagemaker_conn.extra_dejson.copy()
+ tuning_job_config.update(config)
+
+ self.check_valid_tuning_input(tuning_job_config)
+
+ response = self.conn.create_hyper_parameter_tuning_job(
+ **tuning_job_config)
+ if wait_for_completion:
+ self.check_status(SageMakerHook.non_terminal_states,
+ SageMakerHook.failed_states,
+ 'HyperParameterTuningJobStatus',
+ self.describe_tuning_job,
+ tuning_job_config['HyperParameterTuningJobName'])
+ return response
+
+ def create_transform_job(self, transform_job_config, wait_for_completion=True):
+ """
+ Create a transform job
+ :param transform_job_config: the config for transform job
+ :type transform_job_config: dict
+ :param wait_for_completion:
+ if the program should keep running until job finishes
+ :type wait_for_completion: bool
+ :return: A dict that contains ARN of the transform job.
+ """
+ if self.use_db_config:
+ if not self.sagemaker_conn_id:
+ raise AirflowException(
+ "SageMaker connection id must be present to \
+ read SageMaker transform job configuration.")
+
+ sagemaker_conn = self.get_connection(self.sagemaker_conn_id)
+
+ config = sagemaker_conn.extra_dejson.copy()
+ transform_job_config.update(config)
+
+ self.check_for_url(transform_job_config
+ ['TransformInput']['DataSource']
+ ['S3DataSource']['S3Uri'])
+
+ response = self.conn.create_transform_job(
+ **transform_job_config)
+ if wait_for_completion:
+ self.check_status(SageMakerHook.non_terminal_states,
+ SageMakerHook.failed_states,
+ 'TransformJobStatus',
+ self.describe_transform_job,
+ transform_job_config['TransformJobName'])
+ return response
+
+ def create_model(self, model_config):
+ """
+ Create a model job
+ :param model_config: the config for model
+ :type model_config: dict
+ :return: A dict that contains ARN of the model.
+ """
+
+ return self.conn.create_model(
+ **model_config)
+
+ def describe_training_job(self, training_job_name):
+ """
+ :param training_job_name: the name of the training job
+ :type training_job_name: str
+ Return the training job info associated with the current job_name
+ :return: A dict contains all the training job info
+ """
+ return self.conn\
+ .describe_training_job(TrainingJobName=training_job_name)
+
+ def describe_tuning_job(self, tuning_job_name):
+ """
+ :param tuning_job_name: the name of the tuning job
+ :type tuning_job_name: string
+ Return the tuning job info associated with the current job_name
+ :return: A dict contains all the tuning job info
+ """
+ return self.conn\
+ .describe_hyper_parameter_tuning_job(
+ HyperParameterTuningJobName=tuning_job_name)
+
+ def describe_transform_job(self, transform_job_name):
+ """
+ :param transform_job_name: the name of the transform job
+ :type transform_job_name: string
+ Return the transform job info associated with the current job_name
+ :return: A dict contains all the transform job info
+ """
+ return self.conn\
+ .describe_transform_job(
+ TransformJobName=transform_job_name)
diff --git a/airflow/contrib/hooks/salesforce_hook.py b/airflow/contrib/hooks/salesforce_hook.py
index ee18b353d2..efc819e543 100644
--- a/airflow/contrib/hooks/salesforce_hook.py
+++ b/airflow/contrib/hooks/salesforce_hook.py
@@ -53,14 +53,14 @@ def __init__(
:param conn_id: the name of the connection that has the parameters
we need to connect to Salesforce.
- The conenction shoud be type `http` and include a
+ The connection shoud be type `http` and include a
user's security token in the `Extras` field.
.. note::
For the HTTP connection type, you can include a
JSON structure in the `Extras` field.
We need a user's security token to connect to Salesforce.
So we define it in the `Extras` field as:
- `{"security_token":"YOUR_SECRUITY_TOKEN"}`
+ `{"security_token":"YOUR_SECURITY_TOKEN"}`
"""
self.conn_id = conn_id
self._args = args
@@ -135,7 +135,8 @@ def get_available_fields(self, obj):
return [f['name'] for f in desc['fields']]
- def _build_field_list(self, fields):
+ @staticmethod
+ def _build_field_list(fields):
# join all of the fields in a comma separated list
return ",".join(fields)
diff --git a/airflow/contrib/hooks/segment_hook.py b/airflow/contrib/hooks/segment_hook.py
index 874d35d074..01613f2ade 100644
--- a/airflow/contrib/hooks/segment_hook.py
+++ b/airflow/contrib/hooks/segment_hook.py
@@ -54,7 +54,7 @@ def __init__(
:type segment_conn_id: str
:param segment_debug_mode: Determines whether Segment should run in debug mode.
Defaults to False
- :type segment_debug_mode: boolean
+ :type segment_debug_mode: bool
.. note::
You must include a JSON structure in the `Extras` field.
We need a user's security token to connect to Segment.
diff --git a/airflow/contrib/hooks/spark_submit_hook.py b/airflow/contrib/hooks/spark_submit_hook.py
index 0185cab283..65bb6134e6 100644
--- a/airflow/contrib/hooks/spark_submit_hook.py
+++ b/airflow/contrib/hooks/spark_submit_hook.py
@@ -26,7 +26,6 @@
from airflow.exceptions import AirflowException
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.contrib.kubernetes import kube_client
-from kubernetes.client.rest import ApiException
class SparkSubmitHook(BaseHook, LoggingMixin):
@@ -136,6 +135,10 @@ def __init__(self,
self._connection = self._resolve_connection()
self._is_yarn = 'yarn' in self._connection['master']
self._is_kubernetes = 'k8s' in self._connection['master']
+ if self._is_kubernetes and kube_client is None:
+ raise RuntimeError(
+ "{master} specified by kubernetes dependencies are not installed!".format(
+ self._connection['master']))
self._should_track_driver_status = self._resolve_should_track_driver_status()
self._driver_id = None
@@ -559,6 +562,6 @@ def on_kill(self):
self.log.info("Spark on K8s killed with response: %s", api_response)
- except ApiException as e:
+ except kube_client.ApiException as e:
self.log.info("Exception when attempting to kill Spark on K8s:")
self.log.exception(e)
diff --git a/airflow/contrib/hooks/ssh_hook.py b/airflow/contrib/hooks/ssh_hook.py
index f51f0fbd11..c496762187 100755
--- a/airflow/contrib/hooks/ssh_hook.py
+++ b/airflow/contrib/hooks/ssh_hook.py
@@ -1,8 +1,5 @@
# -*- coding: utf-8 -*-
#
-# Copyright 2012-2015 Spotify AB
-# Ported to Airflow by Bolke de Bruin
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -22,11 +19,12 @@
import getpass
import os
+import warnings
import paramiko
from paramiko.config import SSH_PORT
+from sshtunnel import SSHTunnelForwarder
-from contextlib import contextmanager
from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -65,7 +63,7 @@ def __init__(self,
username=None,
password=None,
key_file=None,
- port=SSH_PORT,
+ port=None,
timeout=10,
keepalive_interval=30
):
@@ -75,162 +73,167 @@ def __init__(self,
self.username = username
self.password = password
self.key_file = key_file
+ self.port = port
self.timeout = timeout
self.keepalive_interval = keepalive_interval
+
# Default values, overridable from Connection
self.compress = True
self.no_host_key_check = True
+ self.host_proxy = None
+
+ # Placeholder for deprecated __enter__
self.client = None
- self.port = port
+
+ # Use connection to override defaults
+ if self.ssh_conn_id is not None:
+ conn = self.get_connection(self.ssh_conn_id)
+ if self.username is None:
+ self.username = conn.login
+ if self.password is None:
+ self.password = conn.password
+ if self.remote_host is None:
+ self.remote_host = conn.host
+ if self.port is None:
+ self.port = conn.port
+ if conn.extra is not None:
+ extra_options = conn.extra_dejson
+ self.key_file = extra_options.get("key_file")
+
+ if "timeout" in extra_options:
+ self.timeout = int(extra_options["timeout"], 10)
+
+ if "compress" in extra_options\
+ and str(extra_options["compress"]).lower() == 'false':
+ self.compress = False
+ if "no_host_key_check" in extra_options\
+ and\
+ str(extra_options["no_host_key_check"]).lower() == 'false':
+ self.no_host_key_check = False
+
+ if not self.remote_host:
+ raise AirflowException("Missing required param: remote_host")
+
+ # Auto detecting username values from system
+ if not self.username:
+ self.log.debug(
+ "username to ssh to host: %s is not specified for connection id"
+ " %s. Using system's default provided by getpass.getuser()",
+ self.remote_host, self.ssh_conn_id
+ )
+ self.username = getpass.getuser()
+
+ user_ssh_config_filename = os.path.expanduser('~/.ssh/config')
+ if os.path.isfile(user_ssh_config_filename):
+ ssh_conf = paramiko.SSHConfig()
+ ssh_conf.parse(open(user_ssh_config_filename))
+ host_info = ssh_conf.lookup(self.remote_host)
+ if host_info and host_info.get('proxycommand'):
+ self.host_proxy = paramiko.ProxyCommand(host_info.get('proxycommand'))
+
+ if not (self.password or self.key_file):
+ if host_info and host_info.get('identityfile'):
+ self.key_file = host_info.get('identityfile')[0]
+
+ self.port = self.port or SSH_PORT
def get_conn(self):
- if not self.client:
- self.log.debug('Creating SSH client for conn_id: %s', self.ssh_conn_id)
- if self.ssh_conn_id is not None:
- conn = self.get_connection(self.ssh_conn_id)
- if self.username is None:
- self.username = conn.login
- if self.password is None:
- self.password = conn.password
- if self.remote_host is None:
- self.remote_host = conn.host
- if conn.port is not None:
- self.port = conn.port
- if conn.extra is not None:
- extra_options = conn.extra_dejson
- self.key_file = extra_options.get("key_file")
-
- if "timeout" in extra_options:
- self.timeout = int(extra_options["timeout"], 10)
-
- if "compress" in extra_options \
- and str(extra_options["compress"]).lower() == 'false':
- self.compress = False
- if "no_host_key_check" in extra_options \
- and \
- str(extra_options["no_host_key_check"]).lower() == 'false':
- self.no_host_key_check = False
-
- if not self.remote_host:
- raise AirflowException("Missing required param: remote_host")
-
- # Auto detecting username values from system
- if not self.username:
- self.log.debug(
- "username to ssh to host: %s is not specified for connection id"
- " %s. Using system's default provided by getpass.getuser()",
- self.remote_host, self.ssh_conn_id
- )
- self.username = getpass.getuser()
-
- host_proxy = None
- user_ssh_config_filename = os.path.expanduser('~/.ssh/config')
- if os.path.isfile(user_ssh_config_filename):
- ssh_conf = paramiko.SSHConfig()
- ssh_conf.parse(open(user_ssh_config_filename))
- host_info = ssh_conf.lookup(self.remote_host)
- if host_info and host_info.get('proxycommand'):
- host_proxy = paramiko.ProxyCommand(host_info.get('proxycommand'))
-
- if not (self.password or self.key_file):
- if host_info and host_info.get('identityfile'):
- self.key_file = host_info.get('identityfile')[0]
-
- try:
- client = paramiko.SSHClient()
- client.load_system_host_keys()
- if self.no_host_key_check:
- # Default is RejectPolicy
- client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
-
- if self.password and self.password.strip():
- client.connect(hostname=self.remote_host,
- username=self.username,
- password=self.password,
- timeout=self.timeout,
- compress=self.compress,
- port=self.port,
- sock=host_proxy)
- else:
- client.connect(hostname=self.remote_host,
- username=self.username,
- key_filename=self.key_file,
- timeout=self.timeout,
- compress=self.compress,
- port=self.port,
- sock=host_proxy)
-
- if self.keepalive_interval:
- client.get_transport().set_keepalive(self.keepalive_interval)
-
- self.client = client
- except paramiko.AuthenticationException as auth_error:
- self.log.error(
- "Auth failed while connecting to host: %s, error: %s",
- self.remote_host, auth_error
- )
- except paramiko.SSHException as ssh_error:
- self.log.error(
- "Failed connecting to host: %s, error: %s",
- self.remote_host, ssh_error
- )
- except Exception as error:
- self.log.error(
- "Error connecting to host: %s, error: %s",
- self.remote_host, error
- )
- return self.client
-
- @contextmanager
- def create_tunnel(self, local_port, remote_port=None, remote_host="localhost"):
"""
- Creates a tunnel between two hosts. Like ssh -L <LOCAL_PORT>:host:<REMOTE_PORT>.
- Remember to close() the returned "tunnel" object in order to clean up
- after yourself when you are done with the tunnel.
+ Opens a ssh connection to the remote host.
- :param local_port:
- :type local_port: int
- :param remote_port:
- :type remote_port: int
- :param remote_host:
- :type remote_host: str
- :return:
+ :return paramiko.SSHClient object
"""
- import subprocess
- # this will ensure the connection to the ssh.remote_host from where the tunnel
- # is getting created
- self.get_conn()
-
- tunnel_host = "{0}:{1}:{2}".format(local_port, remote_host, remote_port)
-
- ssh_cmd = ["ssh", "{0}@{1}".format(self.username, self.remote_host),
- "-o", "ControlMaster=no",
- "-o", "UserKnownHostsFile=/dev/null",
- "-o", "StrictHostKeyChecking=no"]
-
- ssh_tunnel_cmd = ["-L", tunnel_host,
- "echo -n ready && cat"
- ]
-
- ssh_cmd += ssh_tunnel_cmd
- self.log.debug("Creating tunnel with cmd: %s", ssh_cmd)
-
- proc = subprocess.Popen(ssh_cmd,
- stdin=subprocess.PIPE,
- stdout=subprocess.PIPE,
- close_fds=True)
- ready = proc.stdout.read(5)
- assert ready == b"ready", \
- "Did not get 'ready' from remote, got '{0}' instead".format(ready)
- yield
- proc.communicate()
- assert proc.returncode == 0, \
- "Tunnel process did unclean exit (returncode {}".format(proc.returncode)
+ self.log.debug('Creating SSH client for conn_id: %s', self.ssh_conn_id)
+ client = paramiko.SSHClient()
+ client.load_system_host_keys()
+ if self.no_host_key_check:
+ # Default is RejectPolicy
+ client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+ if self.password and self.password.strip():
+ client.connect(hostname=self.remote_host,
+ username=self.username,
+ password=self.password,
+ key_filename=self.key_file,
+ timeout=self.timeout,
+ compress=self.compress,
+ port=self.port,
+ sock=self.host_proxy)
+ else:
+ client.connect(hostname=self.remote_host,
+ username=self.username,
+ key_filename=self.key_file,
+ timeout=self.timeout,
+ compress=self.compress,
+ port=self.port,
+ sock=self.host_proxy)
+
+ if self.keepalive_interval:
+ client.get_transport().set_keepalive(self.keepalive_interval)
+
+ self.client = client
+ return client
def __enter__(self):
+ warnings.warn('The contextmanager of SSHHook is deprecated.'
+ 'Please use get_conn() as a contextmanager instead.'
+ 'This method will be removed in Airflow 2.0',
+ category=DeprecationWarning)
return self
def __exit__(self, exc_type, exc_val, exc_tb):
if self.client is not None:
self.client.close()
+ self.client = None
+
+ def get_tunnel(self, remote_port, remote_host="localhost", local_port=None):
+ """
+ Creates a tunnel between two hosts. Like ssh -L <LOCAL_PORT>:host:<REMOTE_PORT>.
+
+ :param remote_port: The remote port to create a tunnel to
+ :type remote_port: int
+ :param remote_host: The remote host to create a tunnel to (default localhost)
+ :type remote_host: str
+ :param local_port: The local port to attach the tunnel to
+ :type local_port: int
+
+ :return: sshtunnel.SSHTunnelForwarder object
+ """
+
+ if local_port:
+ local_bind_address = ('localhost', local_port)
+ else:
+ local_bind_address = ('localhost',)
+
+ if self.password and self.password.strip():
+ client = SSHTunnelForwarder(self.remote_host,
+ ssh_port=self.port,
+ ssh_username=self.username,
+ ssh_password=self.password,
+ ssh_pkey=self.key_file,
+ ssh_proxy=self.host_proxy,
+ local_bind_address=local_bind_address,
+ remote_bind_address=(remote_host, remote_port),
+ logger=self.log)
+ else:
+ client = SSHTunnelForwarder(self.remote_host,
+ ssh_port=self.port,
+ ssh_username=self.username,
+ ssh_pkey=self.key_file,
+ ssh_proxy=self.host_proxy,
+ local_bind_address=local_bind_address,
+ remote_bind_address=(remote_host, remote_port),
+ host_pkey_directories=[],
+ logger=self.log)
+
+ return client
+
+ def create_tunnel(self, local_port, remote_port=None, remote_host="localhost"):
+ warnings.warn('SSHHook.create_tunnel is deprecated, Please'
+ 'use get_tunnel() instead. But please note that the'
+ 'order of the parameters have changed'
+ 'This method will be removed in Airflow 2.0',
+ category=DeprecationWarning)
+
+ return self.get_tunnel(remote_port, remote_host, local_port)
diff --git a/airflow/contrib/hooks/vertica_hook.py b/airflow/contrib/hooks/vertica_hook.py
index f3411de994..e6b36b51d5 100644
--- a/airflow/contrib/hooks/vertica_hook.py
+++ b/airflow/contrib/hooks/vertica_hook.py
@@ -41,9 +41,9 @@ def get_conn(self):
"user": conn.login,
"password": conn.password or '',
"database": conn.schema,
+ "host": conn.host or 'localhost'
}
- conn_config["host"] = conn.host or 'localhost'
if not conn.port:
conn_config["port"] = 5433
else:
diff --git a/airflow/contrib/hooks/wasb_hook.py b/airflow/contrib/hooks/wasb_hook.py
index 1d73abd78b..130c19469b 100644
--- a/airflow/contrib/hooks/wasb_hook.py
+++ b/airflow/contrib/hooks/wasb_hook.py
@@ -58,7 +58,7 @@ def check_for_blob(self, container_name, blob_name, **kwargs):
`BlockBlobService.exists()` takes.
:type kwargs: object
:return: True if the blob exists, False otherwise.
- :rtype bool
+ :rtype: bool
"""
return self.connection.exists(container_name, blob_name, **kwargs)
@@ -74,7 +74,7 @@ def check_for_prefix(self, container_name, prefix, **kwargs):
`BlockBlobService.list_blobs()` takes.
:type kwargs: object
:return: True if blobs matching the prefix exist, False otherwise.
- :rtype bool
+ :rtype: bool
"""
matches = self.connection.list_blobs(container_name, prefix,
num_results=1, **kwargs)
diff --git a/airflow/contrib/hooks/winrm_hook.py b/airflow/contrib/hooks/winrm_hook.py
index 0be904b9bc..5cc24cb7ad 100644
--- a/airflow/contrib/hooks/winrm_hook.py
+++ b/airflow/contrib/hooks/winrm_hook.py
@@ -17,121 +17,235 @@
# specific language governing permissions and limitations
# under the License.
#
+
import getpass
+
from winrm.protocol import Protocol
+
from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
from airflow.utils.log.logging_mixin import LoggingMixin
class WinRMHook(BaseHook, LoggingMixin):
-
"""
Hook for winrm remote execution using pywinrm.
+ :seealso: https://github.com/diyan/pywinrm/blob/master/winrm/protocol.py
+
:param ssh_conn_id: connection id from airflow Connections from where all
- the required parameters can be fetched like username, password or key_file.
+ the required parameters can be fetched like username and password.
Thought the priority is given to the param passed during init
:type ssh_conn_id: str
- :param remote_host: remote host to connect
+ :param endpoint: When set to `None`, endpoint will be constructed like this:
+ 'http://{remote_host}:{remote_port}/wsman'
+ :type endpoint: str
+ :param remote_host: Remote host to connect to.
+ Ignored if `endpoint` is not `None`.
:type remote_host: str
+ :param remote_port: Remote port to connect to.
+ Ignored if `endpoint` is not `None`.
+ :type remote_port: int
+ :param transport: transport type, one of 'plaintext' (default), 'kerberos', 'ssl',
+ 'ntlm', 'credssp'
+ :type transport: str
:param username: username to connect to the remote_host
:type username: str
:param password: password of the username to connect to the remote_host
:type password: str
- :param key_file: key file to use to connect to the remote_host.
- :type key_file: str
- :param timeout: timeout for the attempt to connect to the remote_host.
- :type timeout: int
- :param keepalive_interval: send a keepalive packet to remote host
- every keepalive_interval seconds
- :type keepalive_interval: int
+ :param service: the service name, default is HTTP
+ :type service: str
+ :param keytab: the path to a keytab file if you are using one
+ :type keytab: str
+ :param ca_trust_path: Certification Authority trust path
+ :type ca_trust_path: str
+ :param cert_pem: client authentication certificate file path in PEM format
+ :type cert_pem: str
+ :param cert_key_pem: client authentication certificate key file path in PEM format
+ :type cert_key_pem: str
+ :param server_cert_validation: whether server certificate should be validated on
+ Python versions that suppport it; one of 'validate' (default), 'ignore'
+ :type server_cert_validation: str
+ :param kerberos_delegation: if True, TGT is sent to target server to
+ allow multiple hops
+ :type kerberos_delegation: bool
+ :param read_timeout_sec: maximum seconds to wait before an HTTP connect/read times out
+ (default 30). This value should be slightly higher than operation_timeout_sec,
+ as the server can block *at least* that long.
+ :type read_timeout_sec: int
+ :param operation_timeout_sec: maximum allowed time in seconds for any single wsman
+ HTTP operation (default 20). Note that operation timeouts while receiving output
+ (the only wsman operation that should take any significant time, and where these
+ timeouts are expected) will be silently retried indefinitely.
+ :type operation_timeout_sec: int
+ :param kerberos_hostname_override: the hostname to use for the kerberos exchange
+ (defaults to the hostname in the endpoint URL)
+ :type kerberos_hostname_override: str
+ :param message_encryption_enabled: Will encrypt the WinRM messages if set to True and
+ the transport auth supports message encryption (Default True).
+ :type message_encryption_enabled: bool
+ :param credssp_disable_tlsv1_2: Whether to disable TLSv1.2 support and work with older
+ protocols like TLSv1.0, default is False
+ :type credssp_disable_tlsv1_2: bool
+ :param send_cbt: Will send the channel bindings over a HTTPS channel (Default: True)
+ :type send_cbt: bool
"""
-
def __init__(self,
ssh_conn_id=None,
+ endpoint=None,
remote_host=None,
+ remote_port=5985,
+ transport='plaintext',
username=None,
password=None,
- key_file=None,
- timeout=10,
- keepalive_interval=30
+ service='HTTP',
+ keytab=None,
+ ca_trust_path=None,
+ cert_pem=None,
+ cert_key_pem=None,
+ server_cert_validation='validate',
+ kerberos_delegation=False,
+ read_timeout_sec=30,
+ operation_timeout_sec=20,
+ kerberos_hostname_override=None,
+ message_encryption='auto',
+ credssp_disable_tlsv1_2=False,
+ send_cbt=True,
):
super(WinRMHook, self).__init__(ssh_conn_id)
- # TODO make new win rm connection class
self.ssh_conn_id = ssh_conn_id
+ self.endpoint = endpoint
self.remote_host = remote_host
+ self.remote_port = remote_port
+ self.transport = transport
self.username = username
self.password = password
- self.key_file = key_file
- self.timeout = timeout
- self.keepalive_interval = keepalive_interval
- # Default values, overridable from Connection
- self.compress = True
- self.no_host_key_check = True
+ self.service = service
+ self.keytab = keytab
+ self.ca_trust_path = ca_trust_path
+ self.cert_pem = cert_pem
+ self.cert_key_pem = cert_key_pem
+ self.server_cert_validation = server_cert_validation
+ self.kerberos_delegation = kerberos_delegation
+ self.read_timeout_sec = read_timeout_sec
+ self.operation_timeout_sec = operation_timeout_sec
+ self.kerberos_hostname_override = kerberos_hostname_override
+ self.message_encryption = message_encryption
+ self.credssp_disable_tlsv1_2 = credssp_disable_tlsv1_2
+ self.send_cbt = send_cbt
+
self.client = None
self.winrm_protocol = None
def get_conn(self):
- if not self.client:
- self.log.debug('Creating WinRM client for conn_id: %s', self.ssh_conn_id)
- if self.ssh_conn_id is not None:
- conn = self.get_connection(self.ssh_conn_id)
- if self.username is None:
- self.username = conn.login
- if self.password is None:
- self.password = conn.password
- if self.remote_host is None:
- self.remote_host = conn.host
- if conn.extra is not None:
- extra_options = conn.extra_dejson
- self.key_file = extra_options.get("key_file")
-
- if "timeout" in extra_options:
- self.timeout = int(extra_options["timeout"], 10)
-
- if "compress" in extra_options \
- and extra_options["compress"].lower() == 'false':
- self.compress = False
- if "no_host_key_check" in extra_options \
- and extra_options["no_host_key_check"].lower() == 'false':
- self.no_host_key_check = False
-
- if not self.remote_host:
- raise AirflowException("Missing required param: remote_host")
-
- # Auto detecting username values from system
- if not self.username:
- self.log.debug(
- "username to ssh to host: %s is not specified for connection id"
- " %s. Using system's default provided by getpass.getuser()",
- self.remote_host, self.ssh_conn_id
- )
- self.username = getpass.getuser()
-
- try:
-
- if self.password and self.password.strip():
- self.winrm_protocol = Protocol(
- # TODO pass in port from ssh conn
- endpoint='http://' + self.remote_host + ':5985/wsman',
- # TODO get cert transport working
- # transport='certificate',
- transport='plaintext',
- # cert_pem=r'publickey.pem',
- # cert_key_pem=r'dev.pem',
- read_timeout_sec=70,
- operation_timeout_sec=60,
- username=self.username,
- password=self.password,
- server_cert_validation='ignore')
-
- self.log.info("Opening WinRM shell")
- self.client = self.winrm_protocol.open_shell()
-
- except Exception as error:
- self.log.error(
- "Error connecting to host: %s, error: %s",
- self.remote_host, error
+ if self.client:
+ return self.client
+
+ self.log.debug('Creating WinRM client for conn_id: %s', self.ssh_conn_id)
+ if self.ssh_conn_id is not None:
+ conn = self.get_connection(self.ssh_conn_id)
+
+ if self.username is None:
+ self.username = conn.login
+ if self.password is None:
+ self.password = conn.password
+ if self.remote_host is None:
+ self.remote_host = conn.host
+
+ if conn.extra is not None:
+ extra_options = conn.extra_dejson
+
+ if "endpoint" in extra_options:
+ self.endpoint = str(extra_options["endpoint"])
+ if "remote_port" in extra_options:
+ self.remote_port = int(extra_options["remote_port"])
+ if "transport" in extra_options:
+ self.transport = str(extra_options["transport"])
+ if "service" in extra_options:
+ self.service = str(extra_options["service"])
+ if "keytab" in extra_options:
+ self.keytab = str(extra_options["keytab"])
+ if "ca_trust_path" in extra_options:
+ self.ca_trust_path = str(extra_options["ca_trust_path"])
+ if "cert_pem" in extra_options:
+ self.cert_pem = str(extra_options["cert_pem"])
+ if "cert_key_pem" in extra_options:
+ self.cert_key_pem = str(extra_options["cert_key_pem"])
+ if "server_cert_validation" in extra_options:
+ self.server_cert_validation = \
+ str(extra_options["server_cert_validation"])
+ if "kerberos_delegation" in extra_options:
+ self.kerberos_delegation = \
+ str(extra_options["kerberos_delegation"]).lower() == 'true'
+ if "read_timeout_sec" in extra_options:
+ self.read_timeout_sec = int(extra_options["read_timeout_sec"])
+ if "operation_timeout_sec" in extra_options:
+ self.operation_timeout_sec = \
+ int(extra_options["operation_timeout_sec"])
+ if "kerberos_hostname_override" in extra_options:
+ self.kerberos_hostname_override = \
+ str(extra_options["kerberos_hostname_override"])
+ if "message_encryption" in extra_options:
+ self.message_encryption = str(extra_options["message_encryption"])
+ if "credssp_disable_tlsv1_2" in extra_options:
+ self.credssp_disable_tlsv1_2 = \
+ str(extra_options["credssp_disable_tlsv1_2"]).lower() == 'true'
+ if "send_cbt" in extra_options:
+ self.send_cbt = str(extra_options["send_cbt"]).lower() == 'true'
+
+ if not self.remote_host:
+ raise AirflowException("Missing required param: remote_host")
+
+ # Auto detecting username values from system
+ if not self.username:
+ self.log.debug(
+ "username to WinRM to host: %s is not specified for connection id"
+ " %s. Using system's default provided by getpass.getuser()",
+ self.remote_host, self.ssh_conn_id
+ )
+ self.username = getpass.getuser()
+
+ # If endpoint is not set, then build a standard wsman endpoint from host and port.
+ if not self.endpoint:
+ self.endpoint = 'http://{0}:{1}/wsman'.format(
+ self.remote_host,
+ self.remote_port
+ )
+
+ try:
+ if self.password and self.password.strip():
+ self.winrm_protocol = Protocol(
+ endpoint=self.endpoint,
+ transport=self.transport,
+ username=self.username,
+ password=self.password,
+ service=self.service,
+ keytab=self.keytab,
+ ca_trust_path=self.ca_trust_path,
+ cert_pem=self.cert_pem,
+ cert_key_pem=self.cert_key_pem,
+ server_cert_validation=self.server_cert_validation,
+ kerberos_delegation=self.kerberos_delegation,
+ read_timeout_sec=self.read_timeout_sec,
+ operation_timeout_sec=self.operation_timeout_sec,
+ kerberos_hostname_override=self.kerberos_hostname_override,
+ message_encryption=self.message_encryption,
+ credssp_disable_tlsv1_2=self.credssp_disable_tlsv1_2,
+ send_cbt=self.send_cbt
)
+
+ self.log.info(
+ "Establishing WinRM connection to host: %s",
+ self.remote_host
+ )
+ self.client = self.winrm_protocol.open_shell()
+
+ except Exception as error:
+ error_msg = "Error connecting to host: {0}, error: {1}".format(
+ self.remote_host,
+ error
+ )
+ self.log.error(error_msg)
+ raise AirflowException(error_msg)
+
return self.client
diff --git a/airflow/contrib/kubernetes/kube_client.py b/airflow/contrib/kubernetes/kube_client.py
index 8b71f41242..4b8fa17155 100644
--- a/airflow/contrib/kubernetes/kube_client.py
+++ b/airflow/contrib/kubernetes/kube_client.py
@@ -17,9 +17,21 @@
from airflow.configuration import conf
from six import PY2
+try:
+ from kubernetes import config, client
+ from kubernetes.client.rest import ApiException
+ has_kubernetes = True
+except ImportError as e:
+ # We need an exception class to be able to use it in ``except`` elsewhere
+ # in the code base
+ ApiException = BaseException
+ has_kubernetes = False
+ _import_err = e
+
def _load_kube_config(in_cluster, cluster_context, config_file):
- from kubernetes import config, client
+ if not has_kubernetes:
+ raise _import_err
if in_cluster:
config.load_incluster_config()
else:
diff --git a/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py b/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py
index 7133125ab1..97bcdf2abc 100644
--- a/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py
+++ b/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py
@@ -85,8 +85,9 @@ def extract_args(pod, req):
@staticmethod
def extract_node_selector(pod, req):
- if len(pod.node_selectors) > 0:
- req['spec']['nodeSelector'] = pod.node_selectors
+ req['spec']['nodeSelector'] = req['spec'].get('nodeSelector', {})
+ for k, v in six.iteritems(pod.node_selectors):
+ req['spec']['nodeSelector'][k] = v
@staticmethod
def attach_volumes(pod, req):
@@ -174,6 +175,11 @@ def extract_service_account_name(pod, req):
if pod.service_account_name:
req['spec']['serviceAccountName'] = pod.service_account_name
+ @staticmethod
+ def extract_hostnetwork(pod, req):
+ if pod.hostnetwork:
+ req['spec']['hostNetwork'] = pod.hostnetwork
+
@staticmethod
def extract_image_pull_secrets(pod, req):
if pod.image_pull_secrets:
diff --git a/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py b/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py
index 95d6c829de..877d7aafe2 100644
--- a/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py
+++ b/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py
@@ -59,6 +59,7 @@ def create(self, pod):
self.extract_image_pull_secrets(pod, req)
self.extract_annotations(pod, req)
self.extract_affinity(pod, req)
+ self.extract_hostnetwork(pod, req)
return req
@@ -116,4 +117,5 @@ def create(self, pod):
self.extract_image_pull_secrets(pod, req)
self.extract_annotations(pod, req)
self.extract_affinity(pod, req)
+ self.extract_hostnetwork(pod, req)
return req
diff --git a/airflow/contrib/kubernetes/pod.py b/airflow/contrib/kubernetes/pod.py
index c42221482f..221c8f4180 100644
--- a/airflow/contrib/kubernetes/pod.py
+++ b/airflow/contrib/kubernetes/pod.py
@@ -77,7 +77,8 @@ def __init__(
service_account_name=None,
resources=None,
annotations=None,
- affinity=None
+ affinity=None,
+ hostnetwork=False
):
self.image = image
self.envs = envs or {}
@@ -89,7 +90,7 @@ def __init__(
self.name = name
self.volumes = volumes or []
self.volume_mounts = volume_mounts or []
- self.node_selectors = node_selectors or []
+ self.node_selectors = node_selectors or {}
self.namespace = namespace
self.image_pull_policy = image_pull_policy
self.image_pull_secrets = image_pull_secrets
@@ -98,3 +99,4 @@ def __init__(
self.resources = resources or Resources()
self.annotations = annotations or {}
self.affinity = affinity or {}
+ self.hostnetwork = hostnetwork or False
diff --git a/airflow/contrib/kubernetes/pod_generator.py b/airflow/contrib/kubernetes/pod_generator.py
index 6d8d83ef05..bee7f5b957 100644
--- a/airflow/contrib/kubernetes/pod_generator.py
+++ b/airflow/contrib/kubernetes/pod_generator.py
@@ -149,7 +149,7 @@ def make_pod(self, namespace, image, pod_id, cmds, arguments, labels):
return Pod(
namespace=namespace,
- name=pod_id + "-" + str(uuid.uuid1())[:8],
+ name=pod_id + "-" + str(uuid.uuid4())[:8],
image=image,
cmds=cmds,
args=arguments,
diff --git a/airflow/contrib/kubernetes/pod_launcher.py b/airflow/contrib/kubernetes/pod_launcher.py
index 8ac5108507..8c8d949107 100644
--- a/airflow/contrib/kubernetes/pod_launcher.py
+++ b/airflow/contrib/kubernetes/pod_launcher.py
@@ -22,7 +22,7 @@
from datetime import datetime as dt
from airflow.contrib.kubernetes.kubernetes_request_factory import \
pod_request_factory as pod_factory
-from kubernetes import watch
+from kubernetes import watch, client
from kubernetes.client.rest import ApiException
from kubernetes.stream import stream as kubernetes_stream
from airflow import AirflowException
@@ -59,6 +59,15 @@ def run_pod_async(self, pod):
raise
return resp
+ def delete_pod(self, pod):
+ try:
+ self._client.delete_namespaced_pod(
+ pod.name, pod.namespace, body=client.V1DeleteOptions())
+ except ApiException as e:
+ # If the pod is already deleted
+ if e.status != 404:
+ raise
+
def run_pod(self, pod, startup_timeout=120, get_logs=True):
# type: (Pod) -> (State, result)
"""
@@ -104,7 +113,7 @@ def _monitor_pod(self, pod, get_logs):
while self.pod_is_running(pod):
self.log.info('Pod %s has state %s', pod.name, State.RUNNING)
time.sleep(2)
- return (self._task_status(self.read_pod(pod)), result)
+ return self._task_status(self.read_pod(pod)), result
def _task_status(self, event):
self.log.info(
diff --git a/airflow/contrib/kubernetes/worker_configuration.py b/airflow/contrib/kubernetes/worker_configuration.py
index 26540856f9..74658e384a 100644
--- a/airflow/contrib/kubernetes/worker_configuration.py
+++ b/airflow/contrib/kubernetes/worker_configuration.py
@@ -19,6 +19,7 @@
import os
import six
+from airflow.configuration import conf
from airflow.contrib.kubernetes.pod import Pod, Resources
from airflow.contrib.kubernetes.secret import Secret
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -80,11 +81,16 @@ def _get_init_containers(self, volume_mounts):
def _get_environment(self):
"""Defines any necessary environment variables for the pod executor"""
env = {
- 'AIRFLOW__CORE__DAGS_FOLDER': '/tmp/dags',
- 'AIRFLOW__CORE__EXECUTOR': 'LocalExecutor'
+ "AIRFLOW__CORE__EXECUTOR": "LocalExecutor",
}
+
if self.kube_config.airflow_configmap:
env['AIRFLOW__CORE__AIRFLOW_HOME'] = self.worker_airflow_home
+ if self.kube_config.worker_dags_folder:
+ env['AIRFLOW__CORE__DAGS_FOLDER'] = self.kube_config.worker_dags_folder
+ if (not self.kube_config.airflow_configmap and
+ 'AIRFLOW__CORE__SQL_ALCHEMY_CONN' not in self.kube_config.kube_secrets):
+ env['AIRFLOW__CORE__SQL_ALCHEMY_CONN'] = conf.get("core", "SQL_ALCHEMY_CONN")
return env
def _get_secrets(self):
@@ -106,34 +112,31 @@ def init_volumes_and_mounts(self):
dags_volume_name = 'airflow-dags'
logs_volume_name = 'airflow-logs'
- def _construct_volume(name, claim, subpath=None):
- vo = {
+ def _construct_volume(name, claim):
+ volume = {
'name': name
}
if claim:
- vo['persistentVolumeClaim'] = {
+ volume['persistentVolumeClaim'] = {
'claimName': claim
}
- if subpath:
- vo['subPath'] = subpath
else:
- vo['emptyDir'] = {}
- return vo
+ volume['emptyDir'] = {}
+ return volume
volumes = [
_construct_volume(
dags_volume_name,
- self.kube_config.dags_volume_claim,
- self.kube_config.dags_volume_subpath
+ self.kube_config.dags_volume_claim
),
_construct_volume(
logs_volume_name,
- self.kube_config.logs_volume_claim,
- self.kube_config.logs_volume_subpath
+ self.kube_config.logs_volume_claim
)
]
dag_volume_mount_path = ""
+
if self.kube_config.dags_volume_claim:
dag_volume_mount_path = self.worker_airflow_dags
else:
@@ -141,15 +144,25 @@ def _construct_volume(name, claim, subpath=None):
self.worker_airflow_dags,
self.kube_config.git_subpath
)
-
- volume_mounts = [{
+ dags_volume_mount = {
'name': dags_volume_name,
'mountPath': dag_volume_mount_path,
- 'readOnly': True
- }, {
+ 'readOnly': True,
+ }
+ if self.kube_config.dags_volume_subpath:
+ dags_volume_mount['subPath'] = self.kube_config.dags_volume_subpath
+
+ logs_volume_mount = {
'name': logs_volume_name,
- 'mountPath': self.worker_airflow_logs
- }]
+ 'mountPath': self.worker_airflow_logs,
+ }
+ if self.kube_config.logs_volume_subpath:
+ logs_volume_mount['subPath'] = self.kube_config.logs_volume_subpath
+
+ volume_mounts = [
+ dags_volume_mount,
+ logs_volume_mount
+ ]
# Mount the airflow.cfg file via a configmap the user has specified
if self.kube_config.airflow_configmap:
@@ -182,9 +195,9 @@ def make_pod(self, namespace, worker_uuid, pod_id, dag_id, task_id, execution_da
limit_cpu=kube_executor_config.limit_cpu
)
gcp_sa_key = kube_executor_config.gcp_service_account_key
- annotations = {
- 'iam.cloud.google.com/service-account': gcp_sa_key
- } if gcp_sa_key else {}
+ annotations = kube_executor_config.annotations.copy()
+ if gcp_sa_key:
+ annotations['iam.cloud.google.com/service-account'] = gcp_sa_key
return Pod(
namespace=namespace,
@@ -192,8 +205,7 @@ def make_pod(self, namespace, worker_uuid, pod_id, dag_id, task_id, execution_da
image=kube_executor_config.image or self.kube_config.kube_image,
image_pull_policy=(kube_executor_config.image_pull_policy or
self.kube_config.kube_image_pull_policy),
- cmds=['bash', '-cx', '--'],
- args=[airflow_command],
+ cmds=airflow_command,
labels={
'airflow-worker': worker_uuid,
'dag_id': dag_id,
@@ -208,5 +220,8 @@ def make_pod(self, namespace, worker_uuid, pod_id, dag_id, task_id, execution_da
volumes=volumes,
volume_mounts=volume_mounts,
resources=resources,
- annotations=annotations
+ annotations=annotations,
+ node_selectors=(kube_executor_config.node_selectors or
+ self.kube_config.kube_node_selectors),
+ affinity=kube_executor_config.affinity
)
diff --git a/airflow/contrib/operators/__init__.py b/airflow/contrib/operators/__init__.py
index 3485aaa7fd..b7f8352944 100644
--- a/airflow/contrib/operators/__init__.py
+++ b/airflow/contrib/operators/__init__.py
@@ -17,37 +17,3 @@
# specific language governing permissions and limitations
# under the License.
#
-
-
-# Contrib operators are not imported by default. They should be accessed
-# directly: from airflow.contrib.operators.operator_module import Operator
-
-
-import sys
-import os as _os
-
-# ------------------------------------------------------------------------
-#
-# #TODO #FIXME Airflow 2.0
-#
-# Old import machinary below.
-#
-# This is deprecated but should be kept until Airflow 2.0
-# for compatibility.
-#
-# ------------------------------------------------------------------------
-_operators = {
- 'ssh_operator': ['SSHOperator'],
- 'winrm_operator': ['WinRMOperator'],
- 'vertica_operator': ['VerticaOperator'],
- 'vertica_to_hive': ['VerticaToHiveTransfer'],
- 'qubole_operator': ['QuboleOperator'],
- 'spark_submit_operator': ['SparkSubmitOperator'],
- 'file_to_wasb': ['FileToWasbOperator'],
- 'fs_operator': ['FileSensor'],
- 'hive_to_dynamodb': ['HiveToDynamoDBTransferOperator']
-}
-
-if not _os.environ.get('AIRFLOW_USE_NEW_IMPORTS', False):
- from airflow.utils.helpers import AirflowImporter
- airflow_importer = AirflowImporter(sys.modules[__name__], _operators)
diff --git a/airflow/contrib/operators/awsbatch_operator.py b/airflow/contrib/operators/awsbatch_operator.py
index d23b44e0d4..3c778e6e68 100644
--- a/airflow/contrib/operators/awsbatch_operator.py
+++ b/airflow/contrib/operators/awsbatch_operator.py
@@ -24,7 +24,7 @@
from airflow.exceptions import AirflowException
from airflow.models import BaseOperator
-from airflow.utils import apply_defaults
+from airflow.utils.decorators import apply_defaults
from airflow.contrib.hooks.aws_hook import AwsHook
@@ -36,30 +36,32 @@ class AWSBatchOperator(BaseOperator):
.. warning: the queue parameter was renamed to job_queue to segreggate the
internal CeleryExecutor queue from the AWS Batch internal queue.
- :param job_name: the name for the job that will run on AWS Batch
+ :param job_name: the name for the job that will run on AWS Batch (templated)
:type job_name: str
:param job_definition: the job definition name on AWS Batch
:type job_definition: str
:param job_queue: the queue name on AWS Batch
:type job_queue: str
- :param: overrides: the same parameter that boto3 will receive on
- containerOverrides (templated):
- http://boto3.readthedocs.io/en/latest/reference/services/batch.html#submit_job
- :type: overrides: dict
- :param max_retries: exponential backoff retries while waiter is not merged, 4200 = 48 hours
+ :param overrides: the same parameter that boto3 will receive on
+ containerOverrides (templated):
+ http://boto3.readthedocs.io/en/latest/reference/services/batch.html#submit_job
+ :type overrides: dict
+ :param max_retries: exponential backoff retries while waiter is not
+ merged, 4200 = 48 hours
:type max_retries: int
:param aws_conn_id: connection id of AWS credentials / region name. If None,
- credential boto3 strategy will be used
- (http://boto3.readthedocs.io/en/latest/guide/configuration.html).
+ credential boto3 strategy will be used
+ (http://boto3.readthedocs.io/en/latest/guide/configuration.html).
:type aws_conn_id: str
:param region_name: region name to use in AWS Hook.
Override the region_name in connection (if provided)
+ :type region_name: str
"""
ui_color = '#c3dae0'
client = None
arn = None
- template_fields = ('overrides',)
+ template_fields = ('job_name', 'overrides',)
@apply_defaults
def __init__(self, job_name, job_definition, job_queue, overrides, max_retries=4200,
@@ -139,7 +141,7 @@ def _wait_for_task_ended(self):
if response['jobs'][-1]['status'] in ['SUCCEEDED', 'FAILED']:
retry = False
- sleep( 1 + pow(retries * 0.1, 2))
+ sleep(1 + pow(retries * 0.1, 2))
retries += 1
def _check_success_task(self):
@@ -152,17 +154,19 @@ def _check_success_task(self):
raise AirflowException('No job found for {}'.format(response))
for job in response['jobs']:
- if 'attempts' in job:
- containers = job['attempts']
- for container in containers:
- if (job['status'] == 'FAILED' or
- container['container']['exitCode'] != 0):
- raise AirflowException(
- 'This containers encounter an error during '
- 'execution {}'.format(job))
- elif job['status'] is not 'SUCCEEDED':
+ job_status = job['status']
+ if job_status == 'FAILED':
+ reason = job['statusReason']
+ raise AirflowException('Job failed with status {}'.format(reason))
+ elif job_status in [
+ 'SUBMITTED',
+ 'PENDING',
+ 'RUNNABLE',
+ 'STARTING',
+ 'RUNNING'
+ ]:
raise AirflowException(
- 'This task is still pending {}'.format(job['status']))
+ 'This task is still pending {}'.format(job_status))
def get_hook(self):
return AwsHook(
diff --git a/airflow/contrib/operators/bigquery_check_operator.py b/airflow/contrib/operators/bigquery_check_operator.py
index 59ef5d377d..247a1ae7fb 100644
--- a/airflow/contrib/operators/bigquery_check_operator.py
+++ b/airflow/contrib/operators/bigquery_check_operator.py
@@ -52,24 +52,28 @@ class BigQueryCheckOperator(CheckOperator):
without stopping the progress of the DAG.
:param sql: the sql to be executed
- :type sql: string
+ :type sql: str
:param bigquery_conn_id: reference to the BigQuery database
- :type bigquery_conn_id: string
+ :type bigquery_conn_id: str
+ :param use_legacy_sql: Whether to use legacy SQL (true)
+ or standard SQL (false).
+ :type use_legacy_sql: bool
"""
@apply_defaults
- def __init__(
- self,
- sql,
- bigquery_conn_id='bigquery_default',
- *args,
- **kwargs):
+ def __init__(self,
+ sql,
+ bigquery_conn_id='bigquery_default',
+ use_legacy_sql=True,
+ *args, **kwargs):
super(BigQueryCheckOperator, self).__init__(sql=sql, *args, **kwargs)
self.bigquery_conn_id = bigquery_conn_id
self.sql = sql
+ self.use_legacy_sql = use_legacy_sql
def get_db_hook(self):
- return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id)
+ return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+ use_legacy_sql=self.use_legacy_sql)
class BigQueryValueCheckOperator(ValueCheckOperator):
@@ -77,21 +81,28 @@ class BigQueryValueCheckOperator(ValueCheckOperator):
Performs a simple value check using sql code.
:param sql: the sql to be executed
- :type sql: string
+ :type sql: str
+ :param use_legacy_sql: Whether to use legacy SQL (true)
+ or standard SQL (false).
+ :type use_legacy_sql: bool
"""
@apply_defaults
- def __init__(
- self, sql, pass_value, tolerance=None,
- bigquery_conn_id='bigquery_default',
- *args, **kwargs):
+ def __init__(self, sql,
+ pass_value,
+ tolerance=None,
+ bigquery_conn_id='bigquery_default',
+ use_legacy_sql=True,
+ *args, **kwargs):
super(BigQueryValueCheckOperator, self).__init__(
sql=sql, pass_value=pass_value, tolerance=tolerance,
*args, **kwargs)
self.bigquery_conn_id = bigquery_conn_id
+ self.use_legacy_sql = use_legacy_sql
def get_db_hook(self):
- return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id)
+ return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+ use_legacy_sql=self.use_legacy_sql)
class BigQueryIntervalCheckOperator(IntervalCheckOperator):
@@ -102,7 +113,7 @@ class BigQueryIntervalCheckOperator(IntervalCheckOperator):
This method constructs a query like so ::
SELECT {metrics_threshold_dict_key} FROM {table}
- WHERE {date_filter_column}=<date>
+ WHERE {date_filter_column}=<date>
:param table: the table name
:type table: str
@@ -113,19 +124,22 @@ class BigQueryIntervalCheckOperator(IntervalCheckOperator):
example 'COUNT(*)': 1.5 would require a 50 percent or less difference
between the current day, and the prior days_back.
:type metrics_threshold: dict
+ :param use_legacy_sql: Whether to use legacy SQL (true)
+ or standard SQL (false).
+ :type use_legacy_sql: bool
"""
@apply_defaults
- def __init__(
- self, table, metrics_thresholds,
- date_filter_column='ds', days_back=-7,
- bigquery_conn_id='bigquery_default',
- *args, **kwargs):
+ def __init__(self, table, metrics_thresholds, date_filter_column='ds',
+ days_back=-7, bigquery_conn_id='bigquery_default',
+ use_legacy_sql=True, *args, **kwargs):
super(BigQueryIntervalCheckOperator, self).__init__(
table=table, metrics_thresholds=metrics_thresholds,
date_filter_column=date_filter_column, days_back=days_back,
*args, **kwargs)
self.bigquery_conn_id = bigquery_conn_id
+ self.use_legacy_sql = use_legacy_sql
def get_db_hook(self):
- return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id)
+ return BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+ use_legacy_sql=self.use_legacy_sql)
diff --git a/airflow/contrib/operators/bigquery_get_data.py b/airflow/contrib/operators/bigquery_get_data.py
index ab8f71b717..f5e6e50f06 100644
--- a/airflow/contrib/operators/bigquery_get_data.py
+++ b/airflow/contrib/operators/bigquery_get_data.py
@@ -51,21 +51,21 @@ class BigQueryGetDataOperator(BaseOperator):
)
:param dataset_id: The dataset ID of the requested table. (templated)
- :type destination_dataset_table: string
+ :type dataset_id: str
:param table_id: The table ID of the requested table. (templated)
- :type table_id: string
+ :type table_id: str
:param max_results: The maximum number of records (rows) to be fetched
from the table. (templated)
- :type max_results: string
+ :type max_results: str
:param selected_fields: List of fields to return (comma-separated). If
unspecified, all fields are returned.
- :type selected_fields: string
+ :type selected_fields: str
:param bigquery_conn_id: reference to a specific BigQuery hook.
- :type bigquery_conn_id: string
+ :type bigquery_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ('dataset_id', 'table_id', 'max_results')
ui_color = '#e4f0e8'
diff --git a/airflow/contrib/operators/bigquery_operator.py b/airflow/contrib/operators/bigquery_operator.py
index b36efbd6bf..9386e57c07 100644
--- a/airflow/contrib/operators/bigquery_operator.py
+++ b/airflow/contrib/operators/bigquery_operator.py
@@ -41,58 +41,70 @@ class BigQueryOperator(BaseOperator):
:param destination_dataset_table: A dotted
(<project>.|<project>:)<dataset>.<table> that, if set, will store the results
of the query. (templated)
- :type destination_dataset_table: string
+ :type destination_dataset_table: str
:param write_disposition: Specifies the action that occurs if the destination table
already exists. (default: 'WRITE_EMPTY')
- :type write_disposition: string
+ :type write_disposition: str
:param create_disposition: Specifies whether the job is allowed to create new tables.
(default: 'CREATE_IF_NEEDED')
- :type create_disposition: string
+ :type create_disposition: str
:param allow_large_results: Whether to allow large results.
- :type allow_large_results: boolean
+ :type allow_large_results: bool
:param flatten_results: If true and query uses legacy SQL dialect, flattens
all nested and repeated fields in the query results. ``allow_large_results``
must be ``true`` if this is set to ``false``. For standard SQL queries, this
flag is ignored and results are never flattened.
- :type flatten_results: boolean
+ :type flatten_results: bool
:param bigquery_conn_id: reference to a specific BigQuery hook.
- :type bigquery_conn_id: string
+ :type bigquery_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param udf_config: The User Defined Function configuration for the query.
See https://cloud.google.com/bigquery/user-defined-functions for details.
:type udf_config: list
:param use_legacy_sql: Whether to use legacy SQL (true) or standard SQL (false).
- :type use_legacy_sql: boolean
+ :type use_legacy_sql: bool
:param maximum_billing_tier: Positive integer that serves as a multiplier
of the basic price.
Defaults to None, in which case it uses the value set in the project.
- :type maximum_billing_tier: integer
+ :type maximum_billing_tier: int
:param maximum_bytes_billed: Limits the bytes billed for this job.
Queries that will have bytes billed beyond this limit will fail
(without incurring a charge). If unspecified, this will be
set to your project default.
:type maximum_bytes_billed: float
+ :param api_resource_configs: a dictionary that contain params
+ 'configuration' applied for Google BigQuery Jobs API:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs
+ for example, {'query': {'useQueryCache': False}}. You could use it
+ if you need to provide some params that are not supported by BigQueryOperator
+ like args.
+ :type api_resource_configs: dict
:param schema_update_options: Allows the schema of the destination
table to be updated as a side effect of the load job.
:type schema_update_options: tuple
:param query_params: a dictionary containing query parameter types and
values, passed to BigQuery.
:type query_params: dict
+ :param labels: a dictionary containing labels for the job/query,
+ passed to BigQuery
+ :type labels: dict
:param priority: Specifies a priority for the query.
Possible values include INTERACTIVE and BATCH.
The default value is INTERACTIVE.
- :type priority: string
+ :type priority: str
:param time_partitioning: configure optional time partitioning fields i.e.
- partition by field, type and
- expiration as per API specifications. Note that 'field' is not available in
- conjunction with dataset.table$partition.
+ partition by field, type and expiration as per API specifications.
:type time_partitioning: dict
+ :param cluster_fields: Request that the result of this query be stored sorted
+ by one or more columns. This is only available in conjunction with
+ time_partitioning. The order of columns given determines the sort order.
+ :type cluster_fields: list of str
"""
- template_fields = ('bql', 'sql', 'destination_dataset_table')
+ template_fields = ('bql', 'sql', 'destination_dataset_table', 'labels')
template_ext = ('.sql', )
ui_color = '#e4f0e8'
@@ -103,7 +115,7 @@ def __init__(self,
destination_dataset_table=False,
write_disposition='WRITE_EMPTY',
allow_large_results=False,
- flatten_results=False,
+ flatten_results=None,
bigquery_conn_id='bigquery_default',
delegate_to=None,
udf_config=False,
@@ -113,8 +125,11 @@ def __init__(self,
create_disposition='CREATE_IF_NEEDED',
schema_update_options=(),
query_params=None,
+ labels=None,
priority='INTERACTIVE',
- time_partitioning={},
+ time_partitioning=None,
+ api_resource_configs=None,
+ cluster_fields=None,
*args,
**kwargs):
super(BigQueryOperator, self).__init__(*args, **kwargs)
@@ -133,9 +148,14 @@ def __init__(self,
self.maximum_bytes_billed = maximum_bytes_billed
self.schema_update_options = schema_update_options
self.query_params = query_params
+ self.labels = labels
self.bq_cursor = None
self.priority = priority
- self.time_partitioning = time_partitioning
+ if time_partitioning is None:
+ self.time_partitioning = {}
+ if api_resource_configs is None:
+ self.api_resource_configs = {}
+ self.cluster_fields = cluster_fields
# TODO remove `bql` in Airflow 2.0
if self.bql:
@@ -171,9 +191,12 @@ def execute(self, context):
maximum_bytes_billed=self.maximum_bytes_billed,
create_disposition=self.create_disposition,
query_params=self.query_params,
+ labels=self.labels,
schema_update_options=self.schema_update_options,
priority=self.priority,
- time_partitioning=self.time_partitioning
+ time_partitioning=self.time_partitioning,
+ api_resource_configs=self.api_resource_configs,
+ cluster_fields=self.cluster_fields,
)
def on_kill(self):
@@ -195,11 +218,11 @@ class BigQueryCreateEmptyTableOperator(BaseOperator):
You can also create a table without schema.
:param project_id: The project to create the table into. (templated)
- :type project_id: string
+ :type project_id: str
:param dataset_id: The dataset to create the table into. (templated)
- :type dataset_id: string
+ :type dataset_id: str
:param table_id: The Name of the table to be created. (templated)
- :type table_id: string
+ :type table_id: str
:param schema_fields: If set, the schema field list as defined here:
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema
@@ -212,7 +235,7 @@ class BigQueryCreateEmptyTableOperator(BaseOperator):
:param gcs_schema_object: Full path to the JSON file containing
schema (templated). For
example: ``gs://test-bucket/dir1/dir2/employee_schema.json``
- :type gcs_schema_object: string
+ :type gcs_schema_object: str
:param time_partitioning: configure optional time partitioning fields i.e.
partition by field, type and expiration as per API specifications.
@@ -220,57 +243,60 @@ class BigQueryCreateEmptyTableOperator(BaseOperator):
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#timePartitioning
:type time_partitioning: dict
:param bigquery_conn_id: Reference to a specific BigQuery hook.
- :type bigquery_conn_id: string
+ :type bigquery_conn_id: str
:param google_cloud_storage_conn_id: Reference to a specific Google
cloud storage hook.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any. For this to
work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
-
- **Example (with schema JSON in GCS)**: ::
-
- CreateTable = BigQueryCreateEmptyTableOperator(
- task_id='BigQueryCreateEmptyTableOperator_task',
- dataset_id='ODS',
- table_id='Employees',
- project_id='internal-gcp-project',
- gcs_schema_object='gs://schema-bucket/employee_schema.json',
- bigquery_conn_id='airflow-service-account',
- google_cloud_storage_conn_id='airflow-service-account'
- )
-
- **Corresponding Schema file** (``employee_schema.json``): ::
-
- [
- {
- "mode": "NULLABLE",
- "name": "emp_name",
- "type": "STRING"
- },
- {
- "mode": "REQUIRED",
- "name": "salary",
- "type": "INTEGER"
- }
- ]
-
- **Example (with schema in the DAG)**: ::
-
- CreateTable = BigQueryCreateEmptyTableOperator(
- task_id='BigQueryCreateEmptyTableOperator_task',
- dataset_id='ODS',
- table_id='Employees',
- project_id='internal-gcp-project',
- schema_fields=[{"name": "emp_name", "type": "STRING", "mode": "REQUIRED"},
- {"name": "salary", "type": "INTEGER", "mode": "NULLABLE"}],
- bigquery_conn_id='airflow-service-account',
- google_cloud_storage_conn_id='airflow-service-account'
- )
+ :type delegate_to: str
+ :param labels: a dictionary containing labels for the table, passed to BigQuery
+
+ **Example (with schema JSON in GCS)**: ::
+
+ CreateTable = BigQueryCreateEmptyTableOperator(
+ task_id='BigQueryCreateEmptyTableOperator_task',
+ dataset_id='ODS',
+ table_id='Employees',
+ project_id='internal-gcp-project',
+ gcs_schema_object='gs://schema-bucket/employee_schema.json',
+ bigquery_conn_id='airflow-service-account',
+ google_cloud_storage_conn_id='airflow-service-account'
+ )
+
+ **Corresponding Schema file** (``employee_schema.json``): ::
+
+ [
+ {
+ "mode": "NULLABLE",
+ "name": "emp_name",
+ "type": "STRING"
+ },
+ {
+ "mode": "REQUIRED",
+ "name": "salary",
+ "type": "INTEGER"
+ }
+ ]
+
+ **Example (with schema in the DAG)**: ::
+
+ CreateTable = BigQueryCreateEmptyTableOperator(
+ task_id='BigQueryCreateEmptyTableOperator_task',
+ dataset_id='ODS',
+ table_id='Employees',
+ project_id='internal-gcp-project',
+ schema_fields=[{"name": "emp_name", "type": "STRING", "mode": "REQUIRED"},
+ {"name": "salary", "type": "INTEGER", "mode": "NULLABLE"}],
+ bigquery_conn_id='airflow-service-account',
+ google_cloud_storage_conn_id='airflow-service-account'
+ )
+ :type labels: dict
"""
- template_fields = ('dataset_id', 'table_id', 'project_id', 'gcs_schema_object')
+ template_fields = ('dataset_id', 'table_id', 'project_id',
+ 'gcs_schema_object', 'labels')
ui_color = '#f0eee4'
@apply_defaults
@@ -284,6 +310,7 @@ def __init__(self,
bigquery_conn_id='bigquery_default',
google_cloud_storage_conn_id='google_cloud_default',
delegate_to=None,
+ labels=None,
*args, **kwargs):
super(BigQueryCreateEmptyTableOperator, self).__init__(*args, **kwargs)
@@ -297,6 +324,7 @@ def __init__(self,
self.google_cloud_storage_conn_id = google_cloud_storage_conn_id
self.delegate_to = delegate_to
self.time_partitioning = time_partitioning
+ self.labels = labels
def execute(self, context):
bq_hook = BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
@@ -323,7 +351,8 @@ def execute(self, context):
dataset_id=self.dataset_id,
table_id=self.table_id,
schema_fields=schema_fields,
- time_partitioning=self.time_partitioning
+ time_partitioning=self.time_partitioning,
+ labels=self.labels
)
@@ -338,15 +367,15 @@ class BigQueryCreateExternalTableOperator(BaseOperator):
Google cloud storage must be a JSON file with the schema fields in it.
:param bucket: The bucket to point the external table to. (templated)
- :type bucket: string
+ :type bucket: str
:param source_objects: List of Google cloud storage URIs to point
table to. (templated)
If source_format is 'DATASTORE_BACKUP', the list must only contain a single URI.
- :type object: list
+ :type source_objects: list
:param destination_project_dataset_table: The dotted (<project>.)<dataset>.<table>
BigQuery table to load data into (templated). If <project> is not included,
project will be the project defined in the connection json.
- :type destination_project_dataset_table: string
+ :type destination_project_dataset_table: str
:param schema_fields: If set, the schema field list as defined here:
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema
@@ -359,26 +388,26 @@ class BigQueryCreateExternalTableOperator(BaseOperator):
:type schema_fields: list
:param schema_object: If set, a GCS object path pointing to a .json file that
contains the schema for the table. (templated)
- :param schema_object: string
+ :type schema_object: str
:param source_format: File format of the data.
- :type source_format: string
+ :type source_format: str
:param compression: [Optional] The compression type of the data source.
Possible values include GZIP and NONE.
The default value is NONE.
This setting is ignored for Google Cloud Bigtable,
Google Cloud Datastore backups and Avro formats.
- :type compression: string
+ :type compression: str
:param skip_leading_rows: Number of rows to skip when loading from a CSV.
:type skip_leading_rows: int
:param field_delimiter: The delimiter to use for the CSV.
- :type field_delimiter: string
+ :type field_delimiter: str
:param max_bad_records: The maximum number of bad records that BigQuery can
ignore when running the job.
:type max_bad_records: int
:param quote_character: The value that is used to quote data sections in a CSV file.
- :type quote_character: string
+ :type quote_character: str
:param allow_quoted_newlines: Whether to allow quoted newlines (true) or not (false).
- :type allow_quoted_newlines: boolean
+ :type allow_quoted_newlines: bool
:param allow_jagged_rows: Accept rows that are missing trailing optional columns.
The missing values are treated as nulls. If false, records with missing trailing
columns are treated as bad records, and if there are too many bad records, an
@@ -386,19 +415,21 @@ class BigQueryCreateExternalTableOperator(BaseOperator):
for other formats.
:type allow_jagged_rows: bool
:param bigquery_conn_id: Reference to a specific BigQuery hook.
- :type bigquery_conn_id: string
+ :type bigquery_conn_id: str
:param google_cloud_storage_conn_id: Reference to a specific Google
cloud storage hook.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any. For this to
work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param src_fmt_configs: configure optional fields specific to the source format
:type src_fmt_configs: dict
+ :param labels a dictionary containing labels for the table, passed to BigQuery
+ :type labels: dict
"""
template_fields = ('bucket', 'source_objects',
- 'schema_object', 'destination_project_dataset_table')
+ 'schema_object', 'destination_project_dataset_table', 'labels')
ui_color = '#f0eee4'
@apply_defaults
@@ -420,6 +451,7 @@ def __init__(self,
google_cloud_storage_conn_id='google_cloud_default',
delegate_to=None,
src_fmt_configs={},
+ labels=None,
*args, **kwargs):
super(BigQueryCreateExternalTableOperator, self).__init__(*args, **kwargs)
@@ -446,6 +478,7 @@ def __init__(self,
self.delegate_to = delegate_to
self.src_fmt_configs = src_fmt_configs
+ self.labels = labels
def execute(self, context):
bq_hook = BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
@@ -479,5 +512,120 @@ def execute(self, context):
quote_character=self.quote_character,
allow_quoted_newlines=self.allow_quoted_newlines,
allow_jagged_rows=self.allow_jagged_rows,
- src_fmt_configs=self.src_fmt_configs
+ src_fmt_configs=self.src_fmt_configs,
+ labels=self.labels
+ )
+
+
+class BigQueryDeleteDatasetOperator(BaseOperator):
+ """"
+ This operator deletes an existing dataset from your Project in Big query.
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/delete
+ :param project_id: The project id of the dataset.
+ :type project_id: str
+ :param dataset_id: The dataset to be deleted.
+ :type dataset_id: str
+
+ **Example**: ::
+
+ delete_temp_data = BigQueryDeleteDatasetOperator(dataset_id = 'temp-dataset',
+ project_id = 'temp-project',
+ bigquery_conn_id='_my_gcp_conn_',
+ task_id='Deletetemp',
+ dag=dag)
+ """
+
+ template_fields = ('dataset_id', 'project_id')
+ ui_color = '#f00004'
+
+ @apply_defaults
+ def __init__(self,
+ dataset_id,
+ project_id=None,
+ bigquery_conn_id='bigquery_default',
+ delegate_to=None,
+ *args, **kwargs):
+ self.dataset_id = dataset_id
+ self.project_id = project_id
+ self.bigquery_conn_id = bigquery_conn_id
+ self.delegate_to = delegate_to
+
+ self.log.info('Dataset id: %s', self.dataset_id)
+ self.log.info('Project id: %s', self.project_id)
+
+ super(BigQueryDeleteDatasetOperator, self).__init__(*args, **kwargs)
+
+ def execute(self, context):
+ bq_hook = BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+ delegate_to=self.delegate_to)
+
+ conn = bq_hook.get_conn()
+ cursor = conn.cursor()
+
+ cursor.delete_dataset(
+ project_id=self.project_id,
+ dataset_id=self.dataset_id
)
+
+
+class BigQueryCreateEmptyDatasetOperator(BaseOperator):
+ """"
+ This operator is used to create new dataset for your Project in Big query.
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
+
+ :param project_id: The name of the project where we want to create the dataset.
+ Don't need to provide, if projectId in dataset_reference.
+ :type project_id: str
+ :param dataset_id: The id of dataset. Don't need to provide,
+ if datasetId in dataset_reference.
+ :type dataset_id: str
+ :param dataset_reference: Dataset reference that could be provided with request body.
+ More info:
+ https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
+ :type dataset_reference: dict
+
+ **Example**: ::
+
+ create_new_dataset = BigQueryCreateEmptyDatasetOperator(
+ dataset_id = 'new-dataset',
+ project_id = 'my-project',
+ dataset_reference = {"friendlyName": "New Dataset"}
+ bigquery_conn_id='_my_gcp_conn_',
+ task_id='newDatasetCreator',
+ dag=dag)
+
+ """
+
+ template_fields = ('dataset_id', 'project_id')
+ ui_color = '#f0eee4'
+
+ @apply_defaults
+ def __init__(self,
+ dataset_id,
+ project_id=None,
+ dataset_reference=None,
+ bigquery_conn_id='bigquery_default',
+ delegate_to=None,
+ *args, **kwargs):
+ self.dataset_id = dataset_id
+ self.project_id = project_id
+ self.bigquery_conn_id = bigquery_conn_id
+ self.dataset_reference = dataset_reference if dataset_reference else {}
+ self.delegate_to = delegate_to
+
+ self.log.info('Dataset id: %s', self.dataset_id)
+ self.log.info('Project id: %s', self.project_id)
+
+ super(BigQueryCreateEmptyDatasetOperator, self).__init__(*args, **kwargs)
+
+ def execute(self, context):
+ bq_hook = BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
+ delegate_to=self.delegate_to)
+
+ conn = bq_hook.get_conn()
+ cursor = conn.cursor()
+
+ cursor.create_empty_dataset(
+ project_id=self.project_id,
+ dataset_id=self.dataset_id,
+ dataset_reference=self.dataset_reference)
diff --git a/airflow/contrib/operators/bigquery_table_delete_operator.py b/airflow/contrib/operators/bigquery_table_delete_operator.py
index a16107d8c4..45c481454e 100644
--- a/airflow/contrib/operators/bigquery_table_delete_operator.py
+++ b/airflow/contrib/operators/bigquery_table_delete_operator.py
@@ -29,16 +29,16 @@ class BigQueryTableDeleteOperator(BaseOperator):
:param deletion_dataset_table: A dotted
(<project>.|<project>:)<dataset>.<table> that indicates which table
will be deleted. (templated)
- :type deletion_dataset_table: string
+ :type deletion_dataset_table: str
:param bigquery_conn_id: reference to a specific BigQuery hook.
- :type bigquery_conn_id: string
+ :type bigquery_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param ignore_if_missing: if True, then return success even if the
requested table does not exist.
- :type ignore_if_missing: boolean
+ :type ignore_if_missing: bool
"""
template_fields = ('deletion_dataset_table',)
ui_color = '#ffd1dc'
diff --git a/airflow/contrib/operators/bigquery_to_bigquery.py b/airflow/contrib/operators/bigquery_to_bigquery.py
index 93a52b3102..2073cadef1 100644
--- a/airflow/contrib/operators/bigquery_to_bigquery.py
+++ b/airflow/contrib/operators/bigquery_to_bigquery.py
@@ -38,20 +38,23 @@ class BigQueryToBigQueryOperator(BaseOperator):
:type source_project_dataset_tables: list|string
:param destination_project_dataset_table: The destination BigQuery
table. Format is: (project:|project.)<dataset>.<table> (templated)
- :type destination_project_dataset_table: string
+ :type destination_project_dataset_table: str
:param write_disposition: The write disposition if the table already exists.
- :type write_disposition: string
+ :type write_disposition: str
:param create_disposition: The create disposition if the table doesn't exist.
- :type create_disposition: string
+ :type create_disposition: str
:param bigquery_conn_id: reference to a specific BigQuery hook.
- :type bigquery_conn_id: string
+ :type bigquery_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
+ :param labels: a dictionary containing labels for the job/query,
+ passed to BigQuery
+ :type labels: dict
"""
template_fields = ('source_project_dataset_tables',
- 'destination_project_dataset_table')
+ 'destination_project_dataset_table', 'labels')
template_ext = ('.sql',)
ui_color = '#e6f0e4'
@@ -63,6 +66,7 @@ def __init__(self,
create_disposition='CREATE_IF_NEEDED',
bigquery_conn_id='bigquery_default',
delegate_to=None,
+ labels=None,
*args,
**kwargs):
super(BigQueryToBigQueryOperator, self).__init__(*args, **kwargs)
@@ -72,6 +76,7 @@ def __init__(self,
self.create_disposition = create_disposition
self.bigquery_conn_id = bigquery_conn_id
self.delegate_to = delegate_to
+ self.labels = labels
def execute(self, context):
self.log.info(
@@ -86,4 +91,5 @@ def execute(self, context):
self.source_project_dataset_tables,
self.destination_project_dataset_table,
self.write_disposition,
- self.create_disposition)
+ self.create_disposition,
+ self.labels)
diff --git a/airflow/contrib/operators/bigquery_to_gcs.py b/airflow/contrib/operators/bigquery_to_gcs.py
index e2ce93068f..ec6b937d21 100644
--- a/airflow/contrib/operators/bigquery_to_gcs.py
+++ b/airflow/contrib/operators/bigquery_to_gcs.py
@@ -31,31 +31,35 @@ class BigQueryToCloudStorageOperator(BaseOperator):
https://cloud.google.com/bigquery/docs/reference/v2/jobs
:param source_project_dataset_table: The dotted
- (<project>.|<project>:)<dataset>.<table> BigQuery table to use as the source
- data. If <project> is not included, project will be the project
+ ``(<project>.|<project>:)<dataset>.<table>`` BigQuery table to use as the
+ source data. If <project> is not included, project will be the project
defined in the connection json. (templated)
- :type source_project_dataset_table: string
+ :type source_project_dataset_table: str
:param destination_cloud_storage_uris: The destination Google Cloud
Storage URI (e.g. gs://some-bucket/some-file.txt). (templated) Follows
convention defined here:
https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple
:type destination_cloud_storage_uris: list
:param compression: Type of compression to use.
- :type compression: string
+ :type compression: str
:param export_format: File format to export.
- :type field_delimiter: string
+ :type export_format: str
:param field_delimiter: The delimiter to use when extracting to a CSV.
- :type field_delimiter: string
+ :type field_delimiter: str
:param print_header: Whether to print a header for a CSV file extract.
- :type print_header: boolean
+ :type print_header: bool
:param bigquery_conn_id: reference to a specific BigQuery hook.
- :type bigquery_conn_id: string
+ :type bigquery_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
+ :param labels: a dictionary containing labels for the job/query,
+ passed to BigQuery
+ :type labels: dict
"""
- template_fields = ('source_project_dataset_table', 'destination_cloud_storage_uris')
+ template_fields = ('source_project_dataset_table',
+ 'destination_cloud_storage_uris', 'labels')
template_ext = ('.sql',)
ui_color = '#e4e6f0'
@@ -69,6 +73,7 @@ def __init__(self,
print_header=True,
bigquery_conn_id='bigquery_default',
delegate_to=None,
+ labels=None,
*args,
**kwargs):
super(BigQueryToCloudStorageOperator, self).__init__(*args, **kwargs)
@@ -80,6 +85,7 @@ def __init__(self,
self.print_header = print_header
self.bigquery_conn_id = bigquery_conn_id
self.delegate_to = delegate_to
+ self.labels = labels
def execute(self, context):
self.log.info('Executing extract of %s into: %s',
@@ -95,4 +101,5 @@ def execute(self, context):
self.compression,
self.export_format,
self.field_delimiter,
- self.print_header)
+ self.print_header,
+ self.labels)
diff --git a/airflow/contrib/operators/cassandra_to_gcs.py b/airflow/contrib/operators/cassandra_to_gcs.py
index 211444b96d..95107a497f 100644
--- a/airflow/contrib/operators/cassandra_to_gcs.py
+++ b/airflow/contrib/operators/cassandra_to_gcs.py
@@ -60,18 +60,18 @@ def __init__(self,
**kwargs):
"""
:param cql: The CQL to execute on the Cassandra table.
- :type cql: string
+ :type cql: str
:param bucket: The bucket to upload to.
- :type bucket: string
+ :type bucket: str
:param filename: The filename to use as the object name when uploading
to Google cloud storage. A {} should be specified in the filename
to allow the operator to inject file numbers in cases where the
file is split due to size.
- :type filename: string
+ :type filename: str
:param schema_filename: If set, the filename to use as the object name
when uploading a .json file containing the BigQuery schema fields
for the table that was dumped from MySQL.
- :type schema_filename: string
+ :type schema_filename: str
:param approx_max_file_size_bytes: This operator supports the ability
to split large table dumps into multiple files (see notes in the
filenamed param docs above). Google cloud storage allows for files
@@ -79,14 +79,14 @@ def __init__(self,
file size of the splits.
:type approx_max_file_size_bytes: long
:param cassandra_conn_id: Reference to a specific Cassandra hook.
- :type cassandra_conn_id: string
+ :type cassandra_conn_id: str
:param google_cloud_storage_conn_id: Reference to a specific Google
cloud storage hook.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any. For this to
work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
super(CassandraToGoogleCloudStorageOperator, self).__init__(*args, **kwargs)
self.cql = cql
diff --git a/airflow/contrib/operators/databricks_operator.py b/airflow/contrib/operators/databricks_operator.py
index 7b8d522dba..3ebc729f78 100644
--- a/airflow/contrib/operators/databricks_operator.py
+++ b/airflow/contrib/operators/databricks_operator.py
@@ -30,9 +30,69 @@
XCOM_RUN_PAGE_URL_KEY = 'run_page_url'
+def _deep_string_coerce(content, json_path='json'):
+ """
+ Coerces content or all values of content if it is a dict to a string. The
+ function will throw if content contains non-string or non-numeric types.
+
+ The reason why we have this function is because the ``self.json`` field must be a
+ dict with only string values. This is because ``render_template`` will fail
+ for numerical values.
+ """
+ c = _deep_string_coerce
+ if isinstance(content, six.string_types):
+ return content
+ elif isinstance(content, six.integer_types + (float,)):
+ # Databricks can tolerate either numeric or string types in the API backend.
+ return str(content)
+ elif isinstance(content, (list, tuple)):
+ return [c(e, '{0}[{1}]'.format(json_path, i)) for i, e in enumerate(content)]
+ elif isinstance(content, dict):
+ return {k: c(v, '{0}[{1}]'.format(json_path, k))
+ for k, v in list(content.items())}
+ else:
+ param_type = type(content)
+ msg = 'Type {0} used for parameter {1} is not a number or a string' \
+ .format(param_type, json_path)
+ raise AirflowException(msg)
+
+
+def _handle_databricks_operator_execution(operator, hook, log, context):
+ """
+ Handles the Airflow + Databricks lifecycle logic for a Databricks operator
+ :param operator: Databricks operator being handled
+ :param context: Airflow context
+ """
+ if operator.do_xcom_push:
+ context['ti'].xcom_push(key=XCOM_RUN_ID_KEY, value=operator.run_id)
+ log.info('Run submitted with run_id: %s', operator.run_id)
+ run_page_url = hook.get_run_page_url(operator.run_id)
+ if operator.do_xcom_push:
+ context['ti'].xcom_push(key=XCOM_RUN_PAGE_URL_KEY, value=run_page_url)
+
+ log.info('View run status, Spark UI, and logs at %s', run_page_url)
+ while True:
+ run_state = hook.get_run_state(operator.run_id)
+ if run_state.is_terminal:
+ if run_state.is_successful:
+ log.info('%s completed successfully.', operator.task_id)
+ log.info('View run status, Spark UI, and logs at %s', run_page_url)
+ return
+ else:
+ error_message = '{t} failed with terminal state: {s}'.format(
+ t=operator.task_id,
+ s=run_state)
+ raise AirflowException(error_message)
+ else:
+ log.info('%s in run state: %s', operator.task_id, run_state)
+ log.info('View run status, Spark UI, and logs at %s', run_page_url)
+ log.info('Sleeping for %s seconds.', operator.polling_period_seconds)
+ time.sleep(operator.polling_period_seconds)
+
+
class DatabricksSubmitRunOperator(BaseOperator):
"""
- Submits an Spark job run to Databricks using the
+ Submits a Spark job run to Databricks using the
`api/2.0/jobs/runs/submit
<https://docs.databricks.com/api/latest/jobs.html#runs-submit>`_
API endpoint.
@@ -119,7 +179,7 @@ class DatabricksSubmitRunOperator(BaseOperator):
:param existing_cluster_id: ID for existing cluster on which to run this task.
*EITHER* ``new_cluster`` *OR* ``existing_cluster_id`` should be specified.
This field will be templated.
- :type existing_cluster_id: string
+ :type existing_cluster_id: str
:param libraries: Libraries which this run will use.
This field will be templated.
@@ -130,7 +190,7 @@ class DatabricksSubmitRunOperator(BaseOperator):
By default this will be set to the Airflow ``task_id``. This ``task_id`` is a
required parameter of the superclass ``BaseOperator``.
This field will be templated.
- :type run_name: string
+ :type run_name: str
:param timeout_seconds: The timeout for this run. By default a value of 0 is used
which means to have no timeout.
This field will be templated.
@@ -139,15 +199,18 @@ class DatabricksSubmitRunOperator(BaseOperator):
By default and in the common case this will be ``databricks_default``. To use
token based authentication, provide the key ``token`` in the extra field for the
connection.
- :type databricks_conn_id: string
+ :type databricks_conn_id: str
:param polling_period_seconds: Controls the rate which we poll for the result of
this run. By default the operator will poll every 30 seconds.
:type polling_period_seconds: int
:param databricks_retry_limit: Amount of times retry if the Databricks backend is
unreachable. Its value must be greater than or equal to 1.
:type databricks_retry_limit: int
+ :param databricks_retry_delay: Number of seconds to wait between retries (it
+ might be a floating point number).
+ :type databricks_retry_delay: float
:param do_xcom_push: Whether we should push run_id and run_page_url to xcom.
- :type do_xcom_push: boolean
+ :type do_xcom_push: bool
"""
# Used in airflow.models.BaseOperator
template_fields = ('json',)
@@ -168,6 +231,7 @@ def __init__(
databricks_conn_id='databricks_default',
polling_period_seconds=30,
databricks_retry_limit=3,
+ databricks_retry_delay=1,
do_xcom_push=False,
**kwargs):
"""
@@ -178,6 +242,7 @@ def __init__(
self.databricks_conn_id = databricks_conn_id
self.polling_period_seconds = polling_period_seconds
self.databricks_retry_limit = databricks_retry_limit
+ self.databricks_retry_delay = databricks_retry_delay
if spark_jar_task is not None:
self.json['spark_jar_task'] = spark_jar_task
if notebook_task is not None:
@@ -195,72 +260,213 @@ def __init__(
if 'run_name' not in self.json:
self.json['run_name'] = run_name or kwargs['task_id']
- self.json = self._deep_string_coerce(self.json)
+ self.json = _deep_string_coerce(self.json)
# This variable will be used in case our task gets killed.
self.run_id = None
self.do_xcom_push = do_xcom_push
- def _deep_string_coerce(self, content, json_path='json'):
- """
- Coerces content or all values of content if it is a dict to a string. The
- function will throw if content contains non-string or non-numeric types.
+ def get_hook(self):
+ return DatabricksHook(
+ self.databricks_conn_id,
+ retry_limit=self.databricks_retry_limit,
+ retry_delay=self.databricks_retry_delay)
+
+ def execute(self, context):
+ hook = self.get_hook()
+ self.run_id = hook.submit_run(self.json)
+ _handle_databricks_operator_execution(self, hook, self.log, context)
+
+ def on_kill(self):
+ hook = self.get_hook()
+ hook.cancel_run(self.run_id)
+ self.log.info(
+ 'Task: %s with run_id: %s was requested to be cancelled.',
+ self.task_id, self.run_id
+ )
+
+
+class DatabricksRunNowOperator(BaseOperator):
+ """
+ Runs an existing Spark job run to Databricks using the
+ `api/2.0/jobs/run-now
+ <https://docs.databricks.com/api/latest/jobs.html#run-now>`_
+ API endpoint.
+
+ There are two ways to instantiate this operator.
+
+ In the first way, you can take the JSON payload that you typically use
+ to call the ``api/2.0/jobs/run-now`` endpoint and pass it directly
+ to our ``DatabricksRunNowOperator`` through the ``json`` parameter.
+ For example ::
+ json = {
+ "job_id": 42,
+ "notebook_params": {
+ "dry-run": "true",
+ "oldest-time-to-consider": "1457570074236"
+ }
+ }
+
+ notebook_run = DatabricksRunNowOperator(task_id='notebook_run', json=json)
+
+ Another way to accomplish the same thing is to use the named parameters
+ of the ``DatabricksRunNowOperator`` directly. Note that there is exactly
+ one named parameter for each top level parameter in the ``run-now``
+ endpoint. In this method, your code would look like this: ::
+
+ job_id=42
+
+ notebook_params = {
+ "dry-run": "true",
+ "oldest-time-to-consider": "1457570074236"
+ }
+
+ python_params = ["douglas adams", "42"]
+
+ spark_submit_params = ["--class", "org.apache.spark.examples.SparkPi"]
+
+ notebook_run = DatabricksRunNowOperator(
+ job_id=job_id,
+ notebook_params=notebook_params,
+ python_params=python_params,
+ spark_submit_params=spark_submit_params
+ )
+
+ In the case where both the json parameter **AND** the named parameters
+ are provided, they will be merged together. If there are conflicts during the merge,
+ the named parameters will take precedence and override the top level ``json`` keys.
+
+ Currently the named parameters that ``DatabricksRunNowOperator`` supports are
+ - ``job_id``
+ - ``json``
+ - ``notebook_params``
+ - ``python_params``
+ - ``spark_submit_params``
+
+
+ :param job_id: the job_id of the existing Databricks job.
+ This field will be templated.
+ .. seealso::
+ https://docs.databricks.com/api/latest/jobs.html#run-now
+ :type job_id: str
+ :param json: A JSON object containing API parameters which will be passed
+ directly to the ``api/2.0/jobs/run-now`` endpoint. The other named parameters
+ (i.e. ``notebook_params``, ``spark_submit_params``..) to this operator will
+ be merged with this json dictionary if they are provided.
+ If there are conflicts during the merge, the named parameters will
+ take precedence and override the top level json keys. (templated)
+
+ .. seealso::
+ For more information about templating see :ref:`jinja-templating`.
+ https://docs.databricks.com/api/latest/jobs.html#run-now
+ :type json: dict
+ :param notebook_params: A dict from keys to values for jobs with notebook task,
+ e.g. "notebook_params": {"name": "john doe", "age": "35"}.
+ The map is passed to the notebook and will be accessible through the
+ dbutils.widgets.get function. See Widgets for more information.
+ If not specified upon run-now, the triggered run will use the
+ job’s base parameters. notebook_params cannot be
+ specified in conjunction with jar_params. The json representation
+ of this field (i.e. {"notebook_params":{"name":"john doe","age":"35"}})
+ cannot exceed 10,000 bytes.
+ This field will be templated.
+
+ .. seealso::
+ https://docs.databricks.com/user-guide/notebooks/widgets.html
+ :type notebook_params: dict
+ :param python_params: A list of parameters for jobs with python tasks,
+ e.g. "python_params": ["john doe", "35"].
+ The parameters will be passed to python file as command line parameters.
+ If specified upon run-now, it would overwrite the parameters specified in
+ job setting.
+ The json representation of this field (i.e. {"python_params":["john doe","35"]})
+ cannot exceed 10,000 bytes.
+ This field will be templated.
+
+ .. seealso::
+ https://docs.databricks.com/api/latest/jobs.html#run-now
+ :type python_params: array of strings
+ :param spark_submit_params: A list of parameters for jobs with spark submit task,
+ e.g. "spark_submit_params": ["--class", "org.apache.spark.examples.SparkPi"].
+ The parameters will be passed to spark-submit script as command line parameters.
+ If specified upon run-now, it would overwrite the parameters specified
+ in job setting.
+ The json representation of this field cannot exceed 10,000 bytes.
+ This field will be templated.
+ .. seealso::
+ https://docs.databricks.com/api/latest/jobs.html#run-now
+ :type spark_submit_params: array of strings
+ :param timeout_seconds: The timeout for this run. By default a value of 0 is used
+ which means to have no timeout.
+ This field will be templated.
+ :type timeout_seconds: int32
+ :param databricks_conn_id: The name of the Airflow connection to use.
+ By default and in the common case this will be ``databricks_default``. To use
+ token based authentication, provide the key ``token`` in the extra field for the
+ connection.
+ :type databricks_conn_id: str
+ :param polling_period_seconds: Controls the rate which we poll for the result of
+ this run. By default the operator will poll every 30 seconds.
+ :type polling_period_seconds: int
+ :param databricks_retry_limit: Amount of times retry if the Databricks backend is
+ unreachable. Its value must be greater than or equal to 1.
+ :type databricks_retry_limit: int
+ :param do_xcom_push: Whether we should push run_id and run_page_url to xcom.
+ :type do_xcom_push: bool
+ """
+ # Used in airflow.models.BaseOperator
+ template_fields = ('json',)
+ # Databricks brand color (blue) under white text
+ ui_color = '#1CB1C2'
+ ui_fgcolor = '#fff'
+
+ def __init__(
+ self,
+ job_id,
+ json=None,
+ notebook_params=None,
+ python_params=None,
+ spark_submit_params=None,
+ databricks_conn_id='databricks_default',
+ polling_period_seconds=30,
+ databricks_retry_limit=3,
+ databricks_retry_delay=1,
+ do_xcom_push=False,
+ **kwargs):
- The reason why we have this function is because the ``self.json`` field must be a
- dict with only string values. This is because ``render_template`` will fail
- for numerical values.
"""
- c = self._deep_string_coerce
- if isinstance(content, six.string_types):
- return content
- elif isinstance(content, six.integer_types + (float,)):
- # Databricks can tolerate either numeric or string types in the API backend.
- return str(content)
- elif isinstance(content, (list, tuple)):
- return [c(e, '{0}[{1}]'.format(json_path, i)) for i, e in enumerate(content)]
- elif isinstance(content, dict):
- return {k: c(v, '{0}[{1}]'.format(json_path, k))
- for k, v in list(content.items())}
- else:
- param_type = type(content)
- msg = 'Type {0} used for parameter {1} is not a number or a string'\
- .format(param_type, json_path)
- raise AirflowException(msg)
+ Creates a new ``DatabricksRunNowOperator``.
+ """
+ super(DatabricksRunNowOperator, self).__init__(**kwargs)
+ self.json = json or {}
+ self.databricks_conn_id = databricks_conn_id
+ self.polling_period_seconds = polling_period_seconds
+ self.databricks_retry_limit = databricks_retry_limit
+ self.databricks_retry_delay = databricks_retry_delay
- def _log_run_page_url(self, url):
- self.log.info('View run status, Spark UI, and logs at %s', url)
+ if job_id is not None:
+ self.json['job_id'] = job_id
+ if notebook_params is not None:
+ self.json['notebook_params'] = notebook_params
+ if python_params is not None:
+ self.json['python_params'] = python_params
+ if spark_submit_params is not None:
+ self.json['spark_submit_params'] = spark_submit_params
+
+ self.json = _deep_string_coerce(self.json)
+ # This variable will be used in case our task gets killed.
+ self.run_id = None
+ self.do_xcom_push = do_xcom_push
def get_hook(self):
return DatabricksHook(
self.databricks_conn_id,
- retry_limit=self.databricks_retry_limit)
+ retry_limit=self.databricks_retry_limit,
+ retry_delay=self.databricks_retry_delay)
def execute(self, context):
hook = self.get_hook()
- self.run_id = hook.submit_run(self.json)
- if self.do_xcom_push:
- context['ti'].xcom_push(key=XCOM_RUN_ID_KEY, value=self.run_id)
- self.log.info('Run submitted with run_id: %s', self.run_id)
- run_page_url = hook.get_run_page_url(self.run_id)
- if self.do_xcom_push:
- context['ti'].xcom_push(key=XCOM_RUN_PAGE_URL_KEY, value=run_page_url)
- self._log_run_page_url(run_page_url)
- while True:
- run_state = hook.get_run_state(self.run_id)
- if run_state.is_terminal:
- if run_state.is_successful:
- self.log.info('%s completed successfully.', self.task_id)
- self._log_run_page_url(run_page_url)
- return
- else:
- error_message = '{t} failed with terminal state: {s}'.format(
- t=self.task_id,
- s=run_state)
- raise AirflowException(error_message)
- else:
- self.log.info('%s in run state: %s', self.task_id, run_state)
- self._log_run_page_url(run_page_url)
- self.log.info('Sleeping for %s seconds.', self.polling_period_seconds)
- time.sleep(self.polling_period_seconds)
+ self.run_id = hook.run_now(self.json)
+ _handle_databricks_operator_execution(self, hook, self.log, context)
def on_kill(self):
hook = self.get_hook()
diff --git a/airflow/contrib/operators/dataflow_operator.py b/airflow/contrib/operators/dataflow_operator.py
index e3c8c1fff1..5378735f94 100644
--- a/airflow/contrib/operators/dataflow_operator.py
+++ b/airflow/contrib/operators/dataflow_operator.py
@@ -16,7 +16,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+import os
import re
import uuid
import copy
@@ -33,6 +33,43 @@ class DataFlowJavaOperator(BaseOperator):
Start a Java Cloud DataFlow batch job. The parameters of the operation
will be passed to the job.
+ .. seealso::
+ For more detail on job submission have a look at the reference:
+ https://cloud.google.com/dataflow/pipelines/specifying-exec-params
+
+ :param jar: The reference to a self executing DataFlow jar (templated).
+ :type jar: str
+ :param job_name: The 'jobName' to use when executing the DataFlow job
+ (templated). This ends up being set in the pipeline options, so any entry
+ with key ``'jobName'`` in ``options`` will be overwritten.
+ :type job_name: str
+ :param dataflow_default_options: Map of default job options.
+ :type dataflow_default_options: dict
+ :param options: Map of job specific options.
+ :type options: dict
+ :param gcp_conn_id: The connection ID to use connecting to Google Cloud
+ Platform.
+ :type gcp_conn_id: str
+ :param delegate_to: The account to impersonate, if any.
+ For this to work, the service account making the request must have
+ domain-wide delegation enabled.
+ :type delegate_to: str
+ :param poll_sleep: The time in seconds to sleep between polling Google
+ Cloud Platform for the dataflow job status while the job is in the
+ JOB_STATE_RUNNING state.
+ :type poll_sleep: int
+ :param job_class: The name of the dataflow job class to be executued, it
+ is often not the main class configured in the dataflow jar file.
+ :type job_class: str
+
+ ``jar``, ``options``, and ``job_name`` are templated so you can use variables in them.
+
+ Note that both
+ ``dataflow_default_options`` and ``options`` will be merged to specify pipeline
+ execution parameter, and ``dataflow_default_options`` is expected to save
+ high-level options, for instances, project and zone information, which
+ apply to all dataflow operators in the DAG.
+
It's a good practice to define dataflow_* parameters in the default_args of the dag
like the project, zone and staging location.
@@ -53,7 +90,7 @@ class DataFlowJavaOperator(BaseOperator):
.. code-block:: python
- t1 = DataFlowOperation(
+ t1 = DataFlowJavaOperator(
task_id='datapflow_example',
jar='{{var.value.gcp_dataflow_base}}pipeline/build/libs/pipeline-example-1.0.jar',
options={
@@ -66,15 +103,15 @@ class DataFlowJavaOperator(BaseOperator):
gcp_conn_id='gcp-airflow-service-account',
dag=my-dag)
- Both ``jar`` and ``options`` are templated so you can use variables in them.
"""
- template_fields = ['options', 'jar']
+ template_fields = ['options', 'jar', 'job_name']
ui_color = '#0273d4'
@apply_defaults
def __init__(
self,
jar,
+ job_name='{{task.task_id}}',
dataflow_default_options=None,
options=None,
gcp_conn_id='google_cloud_default',
@@ -83,39 +120,6 @@ def __init__(
job_class=None,
*args,
**kwargs):
- """
- Create a new DataFlowJavaOperator. Note that both
- dataflow_default_options and options will be merged to specify pipeline
- execution parameter, and dataflow_default_options is expected to save
- high-level options, for instances, project and zone information, which
- apply to all dataflow operators in the DAG.
-
-
- .. seealso::
- For more detail on job submission have a look at the reference:
- https://cloud.google.com/dataflow/pipelines/specifying-exec-params
-
- :param jar: The reference to a self executing DataFlow jar.
- :type jar: string
- :param dataflow_default_options: Map of default job options.
- :type dataflow_default_options: dict
- :param options: Map of job specific options.
- :type options: dict
- :param gcp_conn_id: The connection ID to use connecting to Google Cloud
- Platform.
- :type gcp_conn_id: string
- :param delegate_to: The account to impersonate, if any.
- For this to work, the service account making the request must have
- domain-wide delegation enabled.
- :type delegate_to: string
- :param poll_sleep: The time in seconds to sleep between polling Google
- Cloud Platform for the dataflow job status while the job is in the
- JOB_STATE_RUNNING state.
- :type poll_sleep: int
- :param job_class: The name of the dataflow job class to be executued, it
- is often not the main class configured in the dataflow jar file.
- :type job_class: string
- """
super(DataFlowJavaOperator, self).__init__(*args, **kwargs)
dataflow_default_options = dataflow_default_options or {}
@@ -125,6 +129,7 @@ def __init__(
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
self.jar = jar
+ self.job_name = job_name
self.dataflow_default_options = dataflow_default_options
self.options = options
self.poll_sleep = poll_sleep
@@ -141,7 +146,7 @@ def execute(self, context):
dataflow_options = copy.copy(self.dataflow_default_options)
dataflow_options.update(self.options)
- hook.start_java_dataflow(self.task_id, dataflow_options,
+ hook.start_java_dataflow(self.job_name, dataflow_options,
self.jar, self.job_class)
@@ -149,6 +154,27 @@ class DataflowTemplateOperator(BaseOperator):
"""
Start a Templated Cloud DataFlow batch job. The parameters of the operation
will be passed to the job.
+
+ :param template: The reference to the DataFlow template.
+ :type template: str
+ :param job_name: The 'jobName' to use when executing the DataFlow template
+ (templated).
+ :param dataflow_default_options: Map of default job environment options.
+ :type dataflow_default_options: dict
+ :param parameters: Map of job specific parameters for the template.
+ :type parameters: dict
+ :param gcp_conn_id: The connection ID to use connecting to Google Cloud
+ Platform.
+ :type gcp_conn_id: str
+ :param delegate_to: The account to impersonate, if any.
+ For this to work, the service account making the request must have
+ domain-wide delegation enabled.
+ :type delegate_to: str
+ :param poll_sleep: The time in seconds to sleep between polling Google
+ Cloud Platform for the dataflow job status while the job is in the
+ JOB_STATE_RUNNING state.
+ :type poll_sleep: int
+
It's a good practice to define dataflow_* parameters in the default_args of the dag
like the project, zone and staging location.
@@ -160,9 +186,9 @@ class DataflowTemplateOperator(BaseOperator):
default_args = {
'dataflow_default_options': {
- 'project': 'my-gcp-project'
+ 'project': 'my-gcp-project',
'zone': 'europe-west1-d',
- 'tempLocation': 'gs://my-staging-bucket/staging/'
+ 'tempLocation': 'gs://my-staging-bucket/staging/',
}
}
}
@@ -183,16 +209,27 @@ class DataflowTemplateOperator(BaseOperator):
gcp_conn_id='gcp-airflow-service-account',
dag=my-dag)
- ``template``, ``dataflow_default_options`` and ``parameters`` are templated so you can
- use variables in them.
+ ``template``, ``dataflow_default_options``, ``parameters``, and ``job_name`` are
+ templated so you can use variables in them.
+
+ Note that ``dataflow_default_options`` is expected to save high-level options
+ for project information, which apply to all dataflow operators in the DAG.
+
+ .. seealso::
+ https://cloud.google.com/dataflow/docs/reference/rest/v1b3
+ /LaunchTemplateParameters
+ https://cloud.google.com/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment
+ For more detail on job template execution have a look at the reference:
+ https://cloud.google.com/dataflow/docs/templates/executing-templates
"""
- template_fields = ['parameters', 'dataflow_default_options', 'template']
+ template_fields = ['parameters', 'dataflow_default_options', 'template', 'job_name']
ui_color = '#0273d4'
@apply_defaults
def __init__(
self,
template,
+ job_name='{{task.task_id}}',
dataflow_default_options=None,
parameters=None,
gcp_conn_id='google_cloud_default',
@@ -200,36 +237,6 @@ def __init__(
poll_sleep=10,
*args,
**kwargs):
- """
- Create a new DataflowTemplateOperator. Note that
- dataflow_default_options is expected to save high-level options
- for project information, which apply to all dataflow operators in the DAG.
-
- .. seealso::
- https://cloud.google.com/dataflow/docs/reference/rest/v1b3
- /LaunchTemplateParameters
- https://cloud.google.com/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment
- For more detail on job template execution have a look at the reference:
- https://cloud.google.com/dataflow/docs/templates/executing-templates
-
- :param template: The reference to the DataFlow template.
- :type template: string
- :param dataflow_default_options: Map of default job environment options.
- :type dataflow_default_options: dict
- :param parameters: Map of job specific parameters for the template.
- :type parameters: dict
- :param gcp_conn_id: The connection ID to use connecting to Google Cloud
- Platform.
- :type gcp_conn_id: string
- :param delegate_to: The account to impersonate, if any.
- For this to work, the service account making the request must have
- domain-wide delegation enabled.
- :type delegate_to: string
- :param poll_sleep: The time in seconds to sleep between polling Google
- Cloud Platform for the dataflow job status while the job is in the
- JOB_STATE_RUNNING state.
- :type poll_sleep: int
- """
super(DataflowTemplateOperator, self).__init__(*args, **kwargs)
dataflow_default_options = dataflow_default_options or {}
@@ -240,6 +247,7 @@ def __init__(
self.dataflow_default_options = dataflow_default_options
self.poll_sleep = poll_sleep
self.template = template
+ self.job_name = job_name
self.parameters = parameters
def execute(self, context):
@@ -247,18 +255,54 @@ def execute(self, context):
delegate_to=self.delegate_to,
poll_sleep=self.poll_sleep)
- hook.start_template_dataflow(self.task_id, self.dataflow_default_options,
+ hook.start_template_dataflow(self.job_name, self.dataflow_default_options,
self.parameters, self.template)
class DataFlowPythonOperator(BaseOperator):
+ """
+ Launching Cloud Dataflow jobs written in python. Note that both
+ dataflow_default_options and options will be merged to specify pipeline
+ execution parameter, and dataflow_default_options is expected to save
+ high-level options, for instances, project and zone information, which
+ apply to all dataflow operators in the DAG.
- template_fields = ['options', 'dataflow_default_options']
+ .. seealso::
+ For more detail on job submission have a look at the reference:
+ https://cloud.google.com/dataflow/pipelines/specifying-exec-params
+
+ :param py_file: Reference to the python dataflow pipleline file.py, e.g.,
+ /some/local/file/path/to/your/python/pipeline/file.
+ :type py_file: str
+ :param job_name: The 'job_name' to use when executing the DataFlow job
+ (templated). This ends up being set in the pipeline options, so any entry
+ with key ``'jobName'`` or ``'job_name'`` in ``options`` will be overwritten.
+ :type job_name: str
+ :param py_options: Additional python options.
+ :type pyt_options: list of strings, e.g., ["-m", "-v"].
+ :param dataflow_default_options: Map of default job options.
+ :type dataflow_default_options: dict
+ :param options: Map of job specific options.
+ :type options: dict
+ :param gcp_conn_id: The connection ID to use connecting to Google Cloud
+ Platform.
+ :type gcp_conn_id: str
+ :param delegate_to: The account to impersonate, if any.
+ For this to work, the service account making the request must have
+ domain-wide delegation enabled.
+ :type delegate_to: str
+ :param poll_sleep: The time in seconds to sleep between polling Google
+ Cloud Platform for the dataflow job status while the job is in the
+ JOB_STATE_RUNNING state.
+ :type poll_sleep: int
+ """
+ template_fields = ['options', 'dataflow_default_options', 'job_name']
@apply_defaults
def __init__(
self,
py_file,
+ job_name='{{task.task_id}}',
py_options=None,
dataflow_default_options=None,
options=None,
@@ -267,41 +311,11 @@ def __init__(
poll_sleep=10,
*args,
**kwargs):
- """
- Create a new DataFlowPythonOperator. Note that both
- dataflow_default_options and options will be merged to specify pipeline
- execution parameter, and dataflow_default_options is expected to save
- high-level options, for instances, project and zone information, which
- apply to all dataflow operators in the DAG.
- .. seealso::
- For more detail on job submission have a look at the reference:
- https://cloud.google.com/dataflow/pipelines/specifying-exec-params
-
- :param py_file: Reference to the python dataflow pipleline file.py, e.g.,
- /some/local/file/path/to/your/python/pipeline/file.
- :type py_file: string
- :param py_options: Additional python options.
- :type pyt_options: list of strings, e.g., ["-m", "-v"].
- :param dataflow_default_options: Map of default job options.
- :type dataflow_default_options: dict
- :param options: Map of job specific options.
- :type options: dict
- :param gcp_conn_id: The connection ID to use connecting to Google Cloud
- Platform.
- :type gcp_conn_id: string
- :param delegate_to: The account to impersonate, if any.
- For this to work, the service account making the request must have
- domain-wide delegation enabled.
- :type delegate_to: string
- :param poll_sleep: The time in seconds to sleep between polling Google
- Cloud Platform for the dataflow job status while the job is in the
- JOB_STATE_RUNNING state.
- :type poll_sleep: int
- """
super(DataFlowPythonOperator, self).__init__(*args, **kwargs)
self.py_file = py_file
+ self.job_name = job_name
self.py_options = py_options or []
self.dataflow_default_options = dataflow_default_options or {}
self.options = options or {}
@@ -327,11 +341,11 @@ def execute(self, context):
formatted_options = {camel_to_snake(key): dataflow_options[key]
for key in dataflow_options}
hook.start_python_dataflow(
- self.task_id, formatted_options,
+ self.job_name, formatted_options,
self.py_file, self.py_options)
-class GoogleCloudBucketHelper():
+class GoogleCloudBucketHelper(object):
"""GoogleCloudStorageHook helper class to download GCS object."""
GCS_PREFIX_LENGTH = 5
@@ -348,9 +362,9 @@ def google_cloud_to_local(self, file_name):
will be returned immediately.
:param file_name: The full path of input file.
- :type file_name: string
+ :type file_name: str
:return: The full path of local file.
- :type: string
+ :type: str
"""
if not file_name.startswith('gs://'):
return file_name
@@ -358,19 +372,19 @@ def google_cloud_to_local(self, file_name):
# Extracts bucket_id and object_id by first removing 'gs://' prefix and
# then split the remaining by path delimiter '/'.
path_components = file_name[self.GCS_PREFIX_LENGTH:].split('/')
- if path_components < 2:
+ if len(path_components) < 2:
raise Exception(
- 'Invalid Google Cloud Storage (GCS) object path: {}.'
+ 'Invalid Google Cloud Storage (GCS) object path: {}'
.format(file_name))
bucket_id = path_components[0]
object_id = '/'.join(path_components[1:])
- local_file = '/tmp/dataflow{}-{}'.format(str(uuid.uuid1())[:8],
+ local_file = '/tmp/dataflow{}-{}'.format(str(uuid.uuid4())[:8],
path_components[-1])
- file_size = self._gcs_hook.download(bucket_id, object_id, local_file)
+ self._gcs_hook.download(bucket_id, object_id, local_file)
- if file_size > 0:
+ if os.stat(local_file).st_size > 0:
return local_file
raise Exception(
- 'Failed to download Google Cloud Storage GCS object: {}'
+ 'Failed to download Google Cloud Storage (GCS) object: {}'
.format(file_name))
diff --git a/airflow/contrib/operators/dataproc_operator.py b/airflow/contrib/operators/dataproc_operator.py
index 5d59f7fb6e..49e24a3df2 100644
--- a/airflow/contrib/operators/dataproc_operator.py
+++ b/airflow/contrib/operators/dataproc_operator.py
@@ -48,37 +48,49 @@ class DataprocClusterCreateOperator(BaseOperator):
parameters detailed in the link are available as a parameter to this operator.
:param cluster_name: The name of the DataProc cluster to create. (templated)
- :type cluster_name: string
+ :type cluster_name: str
:param project_id: The ID of the google cloud project in which
to create the cluster. (templated)
- :type project_id: string
+ :type project_id: str
:param num_workers: The # of workers to spin up
:type num_workers: int
:param storage_bucket: The storage bucket to use, setting to None lets dataproc
generate a custom one for you
- :type storage_bucket: string
+ :type storage_bucket: str
:param init_actions_uris: List of GCS uri's containing
dataproc initialization scripts
:type init_actions_uris: list[string]
:param init_action_timeout: Amount of time executable scripts in
init_actions_uris has to complete
- :type init_action_timeout: string
+ :type init_action_timeout: str
:param metadata: dict of key-value google compute engine metadata entries
to add to all instances
:type metadata: dict
:param image_version: the version of software inside the Dataproc cluster
- :type image_version: string
+ :type image_version: str
+ :param custom_image: custom Dataproc image for more info see
+ https://cloud.google.com/dataproc/docs/guides/dataproc-images
+ :type: custom_image: str
:param properties: dict of properties to set on
config files (e.g. spark-defaults.conf), see
- https://cloud.google.com/dataproc/docs/reference/rest/v1/ \
- projects.regions.clusters#SoftwareConfig
+ https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters#SoftwareConfig
:type properties: dict
:param master_machine_type: Compute engine machine type to use for the master node
- :type master_machine_type: string
+ :type master_machine_type: str
+ :param master_disk_type: Type of the boot disk for the master node
+ (default is ``pd-standard``).
+ Valid values: ``pd-ssd`` (Persistent Disk Solid State Drive) or
+ ``pd-standard`` (Persistent Disk Hard Disk Drive).
+ :type master_disk_type: str
:param master_disk_size: Disk size for the master node
:type master_disk_size: int
:param worker_machine_type: Compute engine machine type to use for the worker nodes
- :type worker_machine_type: string
+ :type worker_machine_type: str
+ :param worker_disk_type: Type of the boot disk for the worker node
+ (default is ``pd-standard``).
+ Valid values: ``pd-ssd`` (Persistent Disk Solid State Drive) or
+ ``pd-standard`` (Persistent Disk Hard Disk Drive).
+ :type worker_disk_type: str
:param worker_disk_size: Disk size for the worker nodes
:type worker_disk_size: int
:param num_preemptible_workers: The # of preemptible worker nodes to spin up
@@ -86,13 +98,13 @@ class DataprocClusterCreateOperator(BaseOperator):
:param labels: dict of labels to add to the cluster
:type labels: dict
:param zone: The zone where the cluster will be located. (templated)
- :type zone: string
+ :type zone: str
:param network_uri: The network uri to be used for machine communication, cannot be
specified with subnetwork_uri
- :type network_uri: string
+ :type network_uri: str
:param subnetwork_uri: The subnetwork uri to be used for machine communication,
cannot be specified with network_uri
- :type subnetwork_uri: string
+ :type subnetwork_uri: str
:param internal_ip_only: If true, all instances in the cluster will only
have internal IP addresses. This can only be enabled for subnetwork
enabled networks
@@ -100,14 +112,15 @@ class DataprocClusterCreateOperator(BaseOperator):
:param tags: The GCE tags to add to all instances
:type tags: list[string]
:param region: leave as 'global', might become relevant in the future. (templated)
+ :type region: str
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param service_account: The service account of the dataproc instances.
- :type service_account: string
+ :type service_account: str
:param service_account_scopes: The URIs of service account scopes to be included.
:type service_account_scopes: list[string]
:param idle_delete_ttl: The longest duration that cluster would keep alive while
@@ -138,11 +151,14 @@ def __init__(self,
init_actions_uris=None,
init_action_timeout="10m",
metadata=None,
+ custom_image=None,
image_version=None,
properties=None,
master_machine_type='n1-standard-4',
+ master_disk_type='pd-standard',
master_disk_size=500,
worker_machine_type='n1-standard-4',
+ worker_disk_type='pd-standard',
worker_disk_size=500,
num_preemptible_workers=0,
labels=None,
@@ -168,11 +184,14 @@ def __init__(self,
self.init_actions_uris = init_actions_uris
self.init_action_timeout = init_action_timeout
self.metadata = metadata
+ self.custom_image = custom_image
self.image_version = image_version
self.properties = properties
self.master_machine_type = master_machine_type
+ self.master_disk_type = master_disk_type
self.master_disk_size = master_disk_size
self.worker_machine_type = worker_machine_type
+ self.worker_disk_type = worker_disk_type
self.worker_disk_size = worker_disk_size
self.labels = labels
self.zone = zone
@@ -187,6 +206,9 @@ def __init__(self,
self.auto_delete_time = auto_delete_time
self.auto_delete_ttl = auto_delete_ttl
+ assert not (self.custom_image and self.image_version), \
+ "custom_image and image_version can't be both set"
+
def _get_cluster_list_for_project(self, service):
result = service.projects().regions().clusters().list(
projectId=self.project_id,
@@ -272,6 +294,7 @@ def _build_cluster_data(self):
'numInstances': 1,
'machineTypeUri': master_type_uri,
'diskConfig': {
+ 'bootDiskType': self.master_disk_type,
'bootDiskSizeGb': self.master_disk_size
}
},
@@ -279,6 +302,7 @@ def _build_cluster_data(self):
'numInstances': self.num_workers,
'machineTypeUri': worker_type_uri,
'diskConfig': {
+ 'bootDiskType': self.worker_disk_type,
'bootDiskSizeGb': self.worker_disk_size
}
},
@@ -292,6 +316,7 @@ def _build_cluster_data(self):
'numInstances': self.num_preemptible_workers,
'machineTypeUri': worker_type_uri,
'diskConfig': {
+ 'bootDiskType': self.worker_disk_type,
'bootDiskSizeGb': self.worker_disk_size
},
'isPreemptible': True
@@ -321,6 +346,12 @@ def _build_cluster_data(self):
cluster_data['config']['gceClusterConfig']['tags'] = self.tags
if self.image_version:
cluster_data['config']['softwareConfig']['imageVersion'] = self.image_version
+ elif self.custom_image:
+ custom_image_url = 'https://www.googleapis.com/compute/beta/projects/' \
+ '{}/global/images/{}'.format(self.project_id,
+ self.custom_image)
+ cluster_data['config']['masterConfig']['imageUri'] = custom_image_url
+ cluster_data['config']['workerConfig']['imageUri'] = custom_image_url
if self.properties:
cluster_data['config']['softwareConfig']['properties'] = self.properties
if self.idle_delete_ttl:
@@ -395,39 +426,39 @@ class DataprocClusterScaleOperator(BaseOperator):
**Example**: ::
- t1 = DataprocClusterScaleOperator(
- task_id='dataproc_scale',
- project_id='my-project',
- cluster_name='cluster-1',
- num_workers=10,
- num_preemptible_workers=10,
- graceful_decommission_timeout='1h'
- dag=dag)
+ t1 = DataprocClusterScaleOperator(
+ task_id='dataproc_scale',
+ project_id='my-project',
+ cluster_name='cluster-1',
+ num_workers=10,
+ num_preemptible_workers=10,
+ graceful_decommission_timeout='1h',
+ dag=dag)
.. seealso::
For more detail on about scaling clusters have a look at the reference:
https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/scaling-clusters
:param cluster_name: The name of the cluster to scale. (templated)
- :type cluster_name: string
+ :type cluster_name: str
:param project_id: The ID of the google cloud project in which
the cluster runs. (templated)
- :type project_id: string
+ :type project_id: str
:param region: The region for the dataproc cluster. (templated)
- :type region: string
+ :type region: str
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param num_workers: The new number of workers
:type num_workers: int
:param num_preemptible_workers: The new number of preemptible workers
:type num_preemptible_workers: int
:param graceful_decommission_timeout: Timeout for graceful YARN decomissioning.
Maximum value is 1d
- :type graceful_decommission_timeout: string
+ :type graceful_decommission_timeout: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ['cluster_name', 'project_id', 'region']
@@ -491,7 +522,8 @@ def _build_scale_cluster_data(self):
}
return scale_data
- def _get_graceful_decommission_timeout(self, timeout):
+ @staticmethod
+ def _get_graceful_decommission_timeout(timeout):
match = re.match(r"^(\d+)(s|m|h|d)$", timeout)
if match:
if match.group(2) == "s":
@@ -542,18 +574,18 @@ class DataprocClusterDeleteOperator(BaseOperator):
cluster is destroyed.
:param cluster_name: The name of the cluster to create. (templated)
- :type cluster_name: string
+ :type cluster_name: str
:param project_id: The ID of the google cloud project in which
the cluster runs. (templated)
- :type project_id: string
+ :type project_id: str
:param region: leave as 'global', might become relevant in the future. (templated)
- :type region: string
+ :type region: str
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ['cluster_name', 'project_id', 'region']
@@ -575,7 +607,8 @@ def __init__(self,
self.project_id = project_id
self.region = region
- def _wait_for_done(self, service, operation_name):
+ @staticmethod
+ def _wait_for_done(service, operation_name):
time.sleep(15)
while True:
response = service.projects().regions().operations().get(
@@ -643,18 +676,18 @@ class DataProcPigOperator(BaseOperator):
:param query: The query or reference to the query
file (pg or pig extension). (templated)
- :type query: string
+ :type query: str
:param query_uri: The uri of a pig script on Cloud Storage.
- :type query_uri: string
+ :type query_uri: str
:param variables: Map of named parameters for the query. (templated)
:type variables: dict
:param job_name: The job name used in the DataProc cluster. This
name by default is the task_id appended with the execution data, but can
be templated. The name will always be appended with a random number to
avoid name clashes. (templated)
- :type job_name: string
+ :type job_name: str
:param cluster_name: The name of the DataProc cluster. (templated)
- :type cluster_name: string
+ :type cluster_name: str
:param dataproc_pig_properties: Map for the Pig properties. Ideal to put in
default arguments
:type dataproc_pig_properties: dict
@@ -662,13 +695,25 @@ class DataProcPigOperator(BaseOperator):
UDFs and libs) and are ideal to put in default arguments.
:type dataproc_pig_jars: list
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param region: The specified region where the dataproc cluster is created.
- :type region: string
+ :type region: str
+ :param job_error_states: Job states that should be considered error states.
+ Any states in this list will result in an error being raised and failure of the
+ task. Eg, if the ``CANCELLED`` state should also be considered a task failure,
+ pass in ``['ERROR', 'CANCELLED']``. Possible values are currently only
+ ``'ERROR'`` and ``'CANCELLED'``, but could change in the future. Defaults to
+ ``['ERROR']``.
+ :type job_error_states: list
+ :var dataproc_job_id: The actual "jobId" as submitted to the Dataproc API.
+ This is useful for identifying or linking to the job in the Google Cloud Console
+ Dataproc UI, as the actual "jobId" submitted to the Dataproc API is appended with
+ an 8 character random string.
+ :vartype dataproc_job_id: str
"""
template_fields = ['query', 'variables', 'job_name', 'cluster_name', 'dataproc_jars']
template_ext = ('.pg', '.pig',)
@@ -687,6 +732,7 @@ def __init__(
gcp_conn_id='google_cloud_default',
delegate_to=None,
region='global',
+ job_error_states=['ERROR'],
*args,
**kwargs):
@@ -701,6 +747,7 @@ def __init__(
self.dataproc_properties = dataproc_pig_properties
self.dataproc_jars = dataproc_pig_jars
self.region = region
+ self.job_error_states = job_error_states
def execute(self, context):
hook = DataProcHook(gcp_conn_id=self.gcp_conn_id,
@@ -716,7 +763,10 @@ def execute(self, context):
job.add_jar_file_uris(self.dataproc_jars)
job.set_job_name(self.job_name)
- hook.submit(hook.project_id, job.build(), self.region)
+ job_to_submit = job.build()
+ self.dataproc_job_id = job_to_submit["job"]["reference"]["jobId"]
+
+ hook.submit(hook.project_id, job_to_submit, self.region, self.job_error_states)
class DataProcHiveOperator(BaseOperator):
@@ -724,17 +774,17 @@ class DataProcHiveOperator(BaseOperator):
Start a Hive query Job on a Cloud DataProc cluster.
:param query: The query or reference to the query file (q extension).
- :type query: string
+ :type query: str
:param query_uri: The uri of a hive script on Cloud Storage.
- :type query_uri: string
+ :type query_uri: str
:param variables: Map of named parameters for the query.
:type variables: dict
:param job_name: The job name used in the DataProc cluster. This name by default
is the task_id appended with the execution data, but can be templated. The
name will always be appended with a random number to avoid name clashes.
- :type job_name: string
+ :type job_name: str
:param cluster_name: The name of the DataProc cluster.
- :type cluster_name: string
+ :type cluster_name: str
:param dataproc_hive_properties: Map for the Pig properties. Ideal to put in
default arguments
:type dataproc_hive_properties: dict
@@ -742,13 +792,25 @@ class DataProcHiveOperator(BaseOperator):
UDFs and libs) and are ideal to put in default arguments.
:type dataproc_hive_jars: list
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param region: The specified region where the dataproc cluster is created.
- :type region: string
+ :type region: str
+ :param job_error_states: Job states that should be considered error states.
+ Any states in this list will result in an error being raised and failure of the
+ task. Eg, if the ``CANCELLED`` state should also be considered a task failure,
+ pass in ``['ERROR', 'CANCELLED']``. Possible values are currently only
+ ``'ERROR'`` and ``'CANCELLED'``, but could change in the future. Defaults to
+ ``['ERROR']``.
+ :type job_error_states: list
+ :var dataproc_job_id: The actual "jobId" as submitted to the Dataproc API.
+ This is useful for identifying or linking to the job in the Google Cloud Console
+ Dataproc UI, as the actual "jobId" submitted to the Dataproc API is appended with
+ an 8 character random string.
+ :vartype dataproc_job_id: str
"""
template_fields = ['query', 'variables', 'job_name', 'cluster_name', 'dataproc_jars']
template_ext = ('.q',)
@@ -767,6 +829,7 @@ def __init__(
gcp_conn_id='google_cloud_default',
delegate_to=None,
region='global',
+ job_error_states=['ERROR'],
*args,
**kwargs):
@@ -781,6 +844,7 @@ def __init__(
self.dataproc_properties = dataproc_hive_properties
self.dataproc_jars = dataproc_hive_jars
self.region = region
+ self.job_error_states = job_error_states
def execute(self, context):
hook = DataProcHook(gcp_conn_id=self.gcp_conn_id,
@@ -797,7 +861,10 @@ def execute(self, context):
job.add_jar_file_uris(self.dataproc_jars)
job.set_job_name(self.job_name)
- hook.submit(hook.project_id, job.build(), self.region)
+ job_to_submit = job.build()
+ self.dataproc_job_id = job_to_submit["job"]["reference"]["jobId"]
+
+ hook.submit(hook.project_id, job_to_submit, self.region, self.job_error_states)
class DataProcSparkSqlOperator(BaseOperator):
@@ -805,18 +872,18 @@ class DataProcSparkSqlOperator(BaseOperator):
Start a Spark SQL query Job on a Cloud DataProc cluster.
:param query: The query or reference to the query file (q extension). (templated)
- :type query: string
+ :type query: str
:param query_uri: The uri of a spark sql script on Cloud Storage.
- :type query_uri: string
+ :type query_uri: str
:param variables: Map of named parameters for the query. (templated)
:type variables: dict
:param job_name: The job name used in the DataProc cluster. This
name by default is the task_id appended with the execution data, but can
be templated. The name will always be appended with a random number to
avoid name clashes. (templated)
- :type job_name: string
+ :type job_name: str
:param cluster_name: The name of the DataProc cluster. (templated)
- :type cluster_name: string
+ :type cluster_name: str
:param dataproc_spark_properties: Map for the Pig properties. Ideal to put in
default arguments
:type dataproc_spark_properties: dict
@@ -824,13 +891,25 @@ class DataProcSparkSqlOperator(BaseOperator):
for UDFs and libs) and are ideal to put in default arguments.
:type dataproc_spark_jars: list
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param region: The specified region where the dataproc cluster is created.
- :type region: string
+ :type region: str
+ :param job_error_states: Job states that should be considered error states.
+ Any states in this list will result in an error being raised and failure of the
+ task. Eg, if the ``CANCELLED`` state should also be considered a task failure,
+ pass in ``['ERROR', 'CANCELLED']``. Possible values are currently only
+ ``'ERROR'`` and ``'CANCELLED'``, but could change in the future. Defaults to
+ ``['ERROR']``.
+ :type job_error_states: list
+ :var dataproc_job_id: The actual "jobId" as submitted to the Dataproc API.
+ This is useful for identifying or linking to the job in the Google Cloud Console
+ Dataproc UI, as the actual "jobId" submitted to the Dataproc API is appended with
+ an 8 character random string.
+ :vartype dataproc_job_id: str
"""
template_fields = ['query', 'variables', 'job_name', 'cluster_name', 'dataproc_jars']
template_ext = ('.q',)
@@ -849,6 +928,7 @@ def __init__(
gcp_conn_id='google_cloud_default',
delegate_to=None,
region='global',
+ job_error_states=['ERROR'],
*args,
**kwargs):
@@ -863,6 +943,7 @@ def __init__(
self.dataproc_properties = dataproc_spark_properties
self.dataproc_jars = dataproc_spark_jars
self.region = region
+ self.job_error_states = job_error_states
def execute(self, context):
hook = DataProcHook(gcp_conn_id=self.gcp_conn_id,
@@ -879,7 +960,10 @@ def execute(self, context):
job.add_jar_file_uris(self.dataproc_jars)
job.set_job_name(self.job_name)
- hook.submit(hook.project_id, job.build(), self.region)
+ job_to_submit = job.build()
+ self.dataproc_job_id = job_to_submit["job"]["reference"]["jobId"]
+
+ hook.submit(hook.project_id, job_to_submit, self.region, self.job_error_states)
class DataProcSparkOperator(BaseOperator):
@@ -888,10 +972,10 @@ class DataProcSparkOperator(BaseOperator):
:param main_jar: URI of the job jar provisioned on Cloud Storage. (use this or
the main_class, not both together).
- :type main_jar: string
+ :type main_jar: str
:param main_class: Name of the job class. (use this or the main_jar, not both
together).
- :type main_class: string
+ :type main_class: str
:param arguments: Arguments for the job. (templated)
:type arguments: list
:param archives: List of archived files that will be unpacked in the work
@@ -903,9 +987,9 @@ class DataProcSparkOperator(BaseOperator):
name by default is the task_id appended with the execution data, but can
be templated. The name will always be appended with a random number to
avoid name clashes. (templated)
- :type job_name: string
+ :type job_name: str
:param cluster_name: The name of the DataProc cluster. (templated)
- :type cluster_name: string
+ :type cluster_name: str
:param dataproc_spark_properties: Map for the Pig properties. Ideal to put in
default arguments
:type dataproc_spark_properties: dict
@@ -913,13 +997,25 @@ class DataProcSparkOperator(BaseOperator):
for UDFs and libs) and are ideal to put in default arguments.
:type dataproc_spark_jars: list
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param region: The specified region where the dataproc cluster is created.
- :type region: string
+ :type region: str
+ :param job_error_states: Job states that should be considered error states.
+ Any states in this list will result in an error being raised and failure of the
+ task. Eg, if the ``CANCELLED`` state should also be considered a task failure,
+ pass in ``['ERROR', 'CANCELLED']``. Possible values are currently only
+ ``'ERROR'`` and ``'CANCELLED'``, but could change in the future. Defaults to
+ ``['ERROR']``.
+ :type job_error_states: list
+ :var dataproc_job_id: The actual "jobId" as submitted to the Dataproc API.
+ This is useful for identifying or linking to the job in the Google Cloud Console
+ Dataproc UI, as the actual "jobId" submitted to the Dataproc API is appended with
+ an 8 character random string.
+ :vartype dataproc_job_id: str
"""
template_fields = ['arguments', 'job_name', 'cluster_name', 'dataproc_jars']
@@ -940,6 +1036,7 @@ def __init__(
gcp_conn_id='google_cloud_default',
delegate_to=None,
region='global',
+ job_error_states=['ERROR'],
*args,
**kwargs):
@@ -956,6 +1053,7 @@ def __init__(
self.dataproc_properties = dataproc_spark_properties
self.dataproc_jars = dataproc_spark_jars
self.region = region
+ self.job_error_states = job_error_states
def execute(self, context):
hook = DataProcHook(gcp_conn_id=self.gcp_conn_id,
@@ -970,7 +1068,10 @@ def execute(self, context):
job.add_file_uris(self.files)
job.set_job_name(self.job_name)
- hook.submit(hook.project_id, job.build(), self.region)
+ job_to_submit = job.build()
+ self.dataproc_job_id = job_to_submit["job"]["reference"]["jobId"]
+
+ hook.submit(hook.project_id, job_to_submit, self.region, self.job_error_states)
class DataProcHadoopOperator(BaseOperator):
@@ -979,10 +1080,10 @@ class DataProcHadoopOperator(BaseOperator):
:param main_jar: URI of the job jar provisioned on Cloud Storage. (use this or
the main_class, not both together).
- :type main_jar: string
+ :type main_jar: str
:param main_class: Name of the job class. (use this or the main_jar, not both
together).
- :type main_class: string
+ :type main_class: str
:param arguments: Arguments for the job. (templated)
:type arguments: list
:param archives: List of archived files that will be unpacked in the work
@@ -994,9 +1095,9 @@ class DataProcHadoopOperator(BaseOperator):
name by default is the task_id appended with the execution data, but can
be templated. The name will always be appended with a random number to
avoid name clashes. (templated)
- :type job_name: string
+ :type job_name: str
:param cluster_name: The name of the DataProc cluster. (templated)
- :type cluster_name: string
+ :type cluster_name: str
:param dataproc_hadoop_properties: Map for the Pig properties. Ideal to put in
default arguments
:type dataproc_hadoop_properties: dict
@@ -1004,13 +1105,25 @@ class DataProcHadoopOperator(BaseOperator):
for UDFs and libs) and are ideal to put in default arguments.
:type dataproc_hadoop_jars: list
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param region: The specified region where the dataproc cluster is created.
- :type region: string
+ :type region: str
+ :param job_error_states: Job states that should be considered error states.
+ Any states in this list will result in an error being raised and failure of the
+ task. Eg, if the ``CANCELLED`` state should also be considered a task failure,
+ pass in ``['ERROR', 'CANCELLED']``. Possible values are currently only
+ ``'ERROR'`` and ``'CANCELLED'``, but could change in the future. Defaults to
+ ``['ERROR']``.
+ :type job_error_states: list
+ :var dataproc_job_id: The actual "jobId" as submitted to the Dataproc API.
+ This is useful for identifying or linking to the job in the Google Cloud Console
+ Dataproc UI, as the actual "jobId" submitted to the Dataproc API is appended with
+ an 8 character random string.
+ :vartype dataproc_job_id: str
"""
template_fields = ['arguments', 'job_name', 'cluster_name', 'dataproc_jars']
@@ -1031,6 +1144,7 @@ def __init__(
gcp_conn_id='google_cloud_default',
delegate_to=None,
region='global',
+ job_error_states=['ERROR'],
*args,
**kwargs):
@@ -1047,6 +1161,7 @@ def __init__(
self.dataproc_properties = dataproc_hadoop_properties
self.dataproc_jars = dataproc_hadoop_jars
self.region = region
+ self.job_error_states = job_error_states
def execute(self, context):
hook = DataProcHook(gcp_conn_id=self.gcp_conn_id,
@@ -1061,7 +1176,10 @@ def execute(self, context):
job.add_file_uris(self.files)
job.set_job_name(self.job_name)
- hook.submit(hook.project_id, job.build(), self.region)
+ job_to_submit = job.build()
+ self.dataproc_job_id = job_to_submit["job"]["reference"]["jobId"]
+
+ hook.submit(hook.project_id, job_to_submit, self.region, self.job_error_states)
class DataProcPySparkOperator(BaseOperator):
@@ -1070,7 +1188,7 @@ class DataProcPySparkOperator(BaseOperator):
:param main: [Required] The Hadoop Compatible Filesystem (HCFS) URI of the main
Python file to use as the driver. Must be a .py file.
- :type main: string
+ :type main: str
:param arguments: Arguments for the job. (templated)
:type arguments: list
:param archives: List of archived files that will be unpacked in the work
@@ -1085,9 +1203,9 @@ class DataProcPySparkOperator(BaseOperator):
name by default is the task_id appended with the execution data, but can
be templated. The name will always be appended with a random number to
avoid name clashes. (templated)
- :type job_name: string
+ :type job_name: str
:param cluster_name: The name of the DataProc cluster.
- :type cluster_name: string
+ :type cluster_name: str
:param dataproc_pyspark_properties: Map for the Pig properties. Ideal to put in
default arguments
:type dataproc_pyspark_properties: dict
@@ -1095,13 +1213,25 @@ class DataProcPySparkOperator(BaseOperator):
for UDFs and libs) and are ideal to put in default arguments.
:type dataproc_pyspark_jars: list
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param region: The specified region where the dataproc cluster is created.
- :type region: string
+ :type region: str
+ :param job_error_states: Job states that should be considered error states.
+ Any states in this list will result in an error being raised and failure of the
+ task. Eg, if the ``CANCELLED`` state should also be considered a task failure,
+ pass in ``['ERROR', 'CANCELLED']``. Possible values are currently only
+ ``'ERROR'`` and ``'CANCELLED'``, but could change in the future. Defaults to
+ ``['ERROR']``.
+ :type job_error_states: list
+ :var dataproc_job_id: The actual "jobId" as submitted to the Dataproc API.
+ This is useful for identifying or linking to the job in the Google Cloud Console
+ Dataproc UI, as the actual "jobId" submitted to the Dataproc API is appended with
+ an 8 character random string.
+ :vartype dataproc_job_id: str
"""
template_fields = ['arguments', 'job_name', 'cluster_name', 'dataproc_jars']
@@ -1110,7 +1240,7 @@ class DataProcPySparkOperator(BaseOperator):
@staticmethod
def _generate_temp_filename(filename):
dt = time.strftime('%Y%m%d%H%M%S')
- return "{}_{}_{}".format(dt, str(uuid.uuid1())[:8], ntpath.basename(filename))
+ return "{}_{}_{}".format(dt, str(uuid.uuid4())[:8], ntpath.basename(filename))
"""
Upload a local file to a Google Cloud Storage bucket
@@ -1149,6 +1279,7 @@ def __init__(
gcp_conn_id='google_cloud_default',
delegate_to=None,
region='global',
+ job_error_states=['ERROR'],
*args,
**kwargs):
@@ -1165,6 +1296,7 @@ def __init__(
self.dataproc_properties = dataproc_pyspark_properties
self.dataproc_jars = dataproc_pyspark_jars
self.region = region
+ self.job_error_states = job_error_states
def execute(self, context):
hook = DataProcHook(
@@ -1192,7 +1324,10 @@ def execute(self, context):
job.add_python_file_uris(self.pyfiles)
job.set_job_name(self.job_name)
- hook.submit(hook.project_id, job.build(), self.region)
+ job_to_submit = job.build()
+ self.dataproc_job_id = job_to_submit["job"]["reference"]["jobId"]
+
+ hook.submit(hook.project_id, job_to_submit, self.region, self.job_error_states)
class DataprocWorkflowTemplateBaseOperator(BaseOperator):
@@ -1216,7 +1351,7 @@ def __init__(self,
)
def execute(self, context):
- self.hook.await(self.start())
+ self.hook.wait(self.start())
def start(self, context):
raise AirflowException('plese start a workflow operation')
@@ -1232,18 +1367,18 @@ class DataprocWorkflowTemplateInstantiateOperator(DataprocWorkflowTemplateBaseOp
https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.workflowTemplates/instantiate
:param template_id: The id of the template. (templated)
- :type template_id: string
+ :type template_id: str
:param project_id: The ID of the google cloud project in which
the template runs
- :type project_id: string
+ :type project_id: str
:param region: leave as 'global', might become relevant in the future
- :type region: string
+ :type region: str
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ['template_id']
@@ -1261,7 +1396,7 @@ def start(self):
.instantiate(
name=('projects/%s/regions/%s/workflowTemplates/%s' %
(self.project_id, self.region, self.template_id)),
- body={'instanceId': str(uuid.uuid1())})
+ body={'instanceId': str(uuid.uuid4())})
.execute())
@@ -1279,15 +1414,15 @@ class DataprocWorkflowTemplateInstantiateInlineOperator(
:type template: map
:param project_id: The ID of the google cloud project in which
the template runs
- :type project_id: string
+ :type project_id: str
:param region: leave as 'global', might become relevant in the future
- :type region: string
+ :type region: str
:param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ['template']
@@ -1304,6 +1439,6 @@ def start(self):
self.hook.get_conn().projects().regions().workflowTemplates()
.instantiateInline(
parent='projects/%s/regions/%s' % (self.project_id, self.region),
- instanceId=str(uuid.uuid1()),
+ instanceId=str(uuid.uuid4()),
body=self.template)
.execute())
diff --git a/airflow/contrib/operators/datastore_export_operator.py b/airflow/contrib/operators/datastore_export_operator.py
index f6dc7cc571..9d95eadc74 100644
--- a/airflow/contrib/operators/datastore_export_operator.py
+++ b/airflow/contrib/operators/datastore_export_operator.py
@@ -29,19 +29,19 @@ class DatastoreExportOperator(BaseOperator):
Export entities from Google Cloud Datastore to Cloud Storage
:param bucket: name of the cloud storage bucket to backup data
- :type bucket: string
+ :type bucket: str
:param namespace: optional namespace path in the specified Cloud Storage bucket
to backup data. If this namespace does not exist in GCS, it will be created.
:type namespace: str
:param datastore_conn_id: the name of the Datastore connection id to use
- :type datastore_conn_id: string
+ :type datastore_conn_id: str
:param cloud_storage_conn_id: the name of the cloud storage connection id to
force-write backup
- :type cloud_storage_conn_id: string
+ :type cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param entity_filter: description of what data from the project is included in the
export, refer to
https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter
diff --git a/airflow/contrib/operators/datastore_import_operator.py b/airflow/contrib/operators/datastore_import_operator.py
index 401d36e05b..c79767f35e 100644
--- a/airflow/contrib/operators/datastore_import_operator.py
+++ b/airflow/contrib/operators/datastore_import_operator.py
@@ -28,10 +28,10 @@ class DatastoreImportOperator(BaseOperator):
Import entities from Cloud Storage to Google Cloud Datastore
:param bucket: container in Cloud Storage to store data
- :type bucket: string
+ :type bucket: str
:param file: path of the backup metadata file in the specified Cloud Storage bucket.
It should have the extension .overall_export_metadata
- :type file: string
+ :type file: str
:param namespace: optional namespace of the backup metadata file in
the specified Cloud Storage bucket.
:type namespace: str
@@ -42,11 +42,11 @@ class DatastoreImportOperator(BaseOperator):
:param labels: client-assigned labels for cloud storage
:type labels: dict
:param datastore_conn_id: the name of the connection id to use
- :type datastore_conn_id: string
+ :type datastore_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param polling_interval_in_seconds: number of seconds to wait before polling for
execution status again
:type polling_interval_in_seconds: int
diff --git a/airflow/contrib/operators/ecs_operator.py b/airflow/contrib/operators/ecs_operator.py
index 60540f5e28..b58930e3b6 100644
--- a/airflow/contrib/operators/ecs_operator.py
+++ b/airflow/contrib/operators/ecs_operator.py
@@ -17,10 +17,11 @@
# specific language governing permissions and limitations
# under the License.
import sys
+import re
from airflow.exceptions import AirflowException
from airflow.models import BaseOperator
-from airflow.utils import apply_defaults
+from airflow.utils.decorators import apply_defaults
from airflow.contrib.hooks.aws_hook import AwsHook
@@ -33,17 +34,27 @@ class ECSOperator(BaseOperator):
:type task_definition: str
:param cluster: the cluster name on EC2 Container Service
:type cluster: str
- :param: overrides: the same parameter that boto3 will receive (templated):
- http://boto3.readthedocs.org/en/latest/reference/services/ecs.html#ECS.Client.run_task
- :type: overrides: dict
+ :param overrides: the same parameter that boto3 will receive (templated):
+ http://boto3.readthedocs.org/en/latest/reference/services/ecs.html#ECS.Client.run_task
+ :type overrides: dict
:param aws_conn_id: connection id of AWS credentials / region name. If None,
- credential boto3 strategy will be used
- (http://boto3.readthedocs.io/en/latest/guide/configuration.html).
+ credential boto3 strategy will be used
+ (http://boto3.readthedocs.io/en/latest/guide/configuration.html).
:type aws_conn_id: str
:param region_name: region name to use in AWS Hook.
Override the region_name in connection (if provided)
+ :type region_name: str
:param launch_type: the launch type on which to run your task ('EC2' or 'FARGATE')
- :type: launch_type: str
+ :type launch_type: str
+ :param group: the name of the task group associated with the task
+ :type group: str
+ :param placement_constraints: an array of placement constraint objects to use for
+ the task
+ :type placement_constraints: list
+ :param platform_version: the platform version on which your task is running
+ :type platform_version: str
+ :param network_configuration: the network configuration for the task
+ :type network_configuration: dict
"""
ui_color = '#f0ede4'
@@ -53,7 +64,9 @@ class ECSOperator(BaseOperator):
@apply_defaults
def __init__(self, task_definition, cluster, overrides,
- aws_conn_id=None, region_name=None, launch_type='EC2', **kwargs):
+ aws_conn_id=None, region_name=None, launch_type='EC2',
+ group=None, placement_constraints=None, platform_version='LATEST',
+ network_configuration=None, **kwargs):
super(ECSOperator, self).__init__(**kwargs)
self.aws_conn_id = aws_conn_id
@@ -62,6 +75,10 @@ def __init__(self, task_definition, cluster, overrides,
self.cluster = cluster
self.overrides = overrides
self.launch_type = launch_type
+ self.group = group
+ self.placement_constraints = placement_constraints
+ self.platform_version = platform_version
+ self.network_configuration = network_configuration
self.hook = self.get_hook()
@@ -77,13 +94,21 @@ def execute(self, context):
region_name=self.region_name
)
- response = self.client.run_task(
- cluster=self.cluster,
- taskDefinition=self.task_definition,
- overrides=self.overrides,
- startedBy=self.owner,
- launchType=self.launch_type
- )
+ run_opts = {
+ 'cluster': self.cluster,
+ 'taskDefinition': self.task_definition,
+ 'overrides': self.overrides,
+ 'startedBy': self.owner,
+ 'launchType': self.launch_type,
+ 'platformVersion': self.platform_version,
+ }
+ if self.group is not None:
+ run_opts['group'] = self.group
+ if self.placement_constraints is not None:
+ run_opts['placementConstraints'] = self.placement_constraints
+ if self.network_configuration is not None:
+ run_opts['networkConfiguration'] = self.network_configuration
+ response = self.client.run_task(**run_opts)
failures = response['failures']
if len(failures) > 0:
@@ -115,6 +140,10 @@ def _check_success_task(self):
raise AirflowException(response)
for task in response['tasks']:
+ if re.match(r'Host .+?\(.+?\) terminated\.', task.get('stoppedReason', '')):
+ raise AirflowException(
+ 'The task was stopped because the host instance terminated: {}'.
+ format(task.get('stoppedReason', '')))
containers = task['containers']
for container in containers:
if container.get('lastStatus') == 'STOPPED' and \
diff --git a/airflow/contrib/operators/emr_add_steps_operator.py b/airflow/contrib/operators/emr_add_steps_operator.py
index 643ffe9c1b..959543e617 100644
--- a/airflow/contrib/operators/emr_add_steps_operator.py
+++ b/airflow/contrib/operators/emr_add_steps_operator.py
@@ -17,7 +17,7 @@
# specific language governing permissions and limitations
# under the License.
from airflow.models import BaseOperator
-from airflow.utils import apply_defaults
+from airflow.utils.decorators import apply_defaults
from airflow.exceptions import AirflowException
from airflow.contrib.hooks.emr_hook import EmrHook
@@ -27,7 +27,7 @@ class EmrAddStepsOperator(BaseOperator):
An operator that adds steps to an existing EMR job_flow.
:param job_flow_id: id of the JobFlow to add steps to. (templated)
- :type job_flow_name: str
+ :type job_flow_id: str
:param aws_conn_id: aws connection to uses
:type aws_conn_id: str
:param steps: boto3 style steps to be added to the jobflow. (templated)
diff --git a/airflow/contrib/operators/emr_create_job_flow_operator.py b/airflow/contrib/operators/emr_create_job_flow_operator.py
index 89be12f065..42886d6006 100644
--- a/airflow/contrib/operators/emr_create_job_flow_operator.py
+++ b/airflow/contrib/operators/emr_create_job_flow_operator.py
@@ -18,7 +18,7 @@
# under the License.
from airflow.contrib.hooks.emr_hook import EmrHook
from airflow.models import BaseOperator
-from airflow.utils import apply_defaults
+from airflow.utils.decorators import apply_defaults
from airflow.exceptions import AirflowException
@@ -33,7 +33,7 @@ class EmrCreateJobFlowOperator(BaseOperator):
:param emr_conn_id: emr connection to use
:type emr_conn_id: str
:param job_flow_overrides: boto3 style arguments to override
- emr_connection extra. (templated)
+ emr_connection extra. (templated)
:type steps: dict
"""
template_fields = ['job_flow_overrides']
diff --git a/airflow/contrib/operators/emr_terminate_job_flow_operator.py b/airflow/contrib/operators/emr_terminate_job_flow_operator.py
index 50407a14ac..8e4c0687cc 100644
--- a/airflow/contrib/operators/emr_terminate_job_flow_operator.py
+++ b/airflow/contrib/operators/emr_terminate_job_flow_operator.py
@@ -17,7 +17,7 @@
# specific language governing permissions and limitations
# under the License.
from airflow.models import BaseOperator
-from airflow.utils import apply_defaults
+from airflow.utils.decorators import apply_defaults
from airflow.exceptions import AirflowException
from airflow.contrib.hooks.emr_hook import EmrHook
@@ -27,7 +27,7 @@ class EmrTerminateJobFlowOperator(BaseOperator):
Operator to terminate EMR JobFlows.
:param job_flow_id: id of the JobFlow to terminate. (templated)
- :type job_flow_name: str
+ :type job_flow_id: str
:param aws_conn_id: aws connection to uses
:type aws_conn_id: str
"""
diff --git a/airflow/contrib/operators/file_to_gcs.py b/airflow/contrib/operators/file_to_gcs.py
index 807385b43c..a392a16891 100644
--- a/airflow/contrib/operators/file_to_gcs.py
+++ b/airflow/contrib/operators/file_to_gcs.py
@@ -28,17 +28,17 @@ class FileToGoogleCloudStorageOperator(BaseOperator):
Uploads a file to Google Cloud Storage
:param src: Path to the local file. (templated)
- :type src: string
+ :type src: str
:param dst: Destination path within the specified bucket. (templated)
- :type dst: string
+ :type dst: str
:param bucket: The bucket to upload to. (templated)
- :type bucket: string
+ :type bucket: str
:param google_cloud_storage_conn_id: The Airflow connection ID to upload with
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param mime_type: The mime-type string
- :type mime_type: string
+ :type mime_type: str
:param delegate_to: The account to impersonate, if any
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ('src', 'dst', 'bucket')
diff --git a/airflow/contrib/operators/gcp_compute_operator.py b/airflow/contrib/operators/gcp_compute_operator.py
new file mode 100644
index 0000000000..a2fd545294
--- /dev/null
+++ b/airflow/contrib/operators/gcp_compute_operator.py
@@ -0,0 +1,183 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow import AirflowException
+from airflow.contrib.hooks.gcp_compute_hook import GceHook
+from airflow.contrib.utils.gcp_field_validator import GcpBodyFieldValidator
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class GceBaseOperator(BaseOperator):
+ """
+ Abstract base operator for Google Compute Engine operators to inherit from.
+ """
+ @apply_defaults
+ def __init__(self,
+ project_id,
+ zone,
+ resource_id,
+ gcp_conn_id='google_cloud_default',
+ api_version='v1',
+ *args, **kwargs):
+ self.project_id = project_id
+ self.zone = zone
+ self.full_location = 'projects/{}/zones/{}'.format(self.project_id,
+ self.zone)
+ self.resource_id = resource_id
+ self.gcp_conn_id = gcp_conn_id
+ self.api_version = api_version
+ self._validate_inputs()
+ self._hook = GceHook(gcp_conn_id=self.gcp_conn_id, api_version=self.api_version)
+ super(GceBaseOperator, self).__init__(*args, **kwargs)
+
+ def _validate_inputs(self):
+ if not self.project_id:
+ raise AirflowException("The required parameter 'project_id' is missing")
+ if not self.zone:
+ raise AirflowException("The required parameter 'zone' is missing")
+ if not self.resource_id:
+ raise AirflowException("The required parameter 'resource_id' is missing")
+
+ def execute(self, context):
+ pass
+
+
+class GceInstanceStartOperator(GceBaseOperator):
+ """
+ Start an instance in Google Compute Engine.
+
+ :param project_id: Google Cloud Platform project where the Compute Engine
+ instance exists.
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the instance exists.
+ :type zone: str
+ :param resource_id: Name of the Compute Engine instance resource.
+ :type resource_id: str
+ :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform.
+ :type gcp_conn_id: str
+ :param api_version: API version used (e.g. v1).
+ :type api_version: str
+ """
+ template_fields = ('project_id', 'zone', 'resource_id', 'gcp_conn_id', 'api_version')
+
+ @apply_defaults
+ def __init__(self,
+ project_id,
+ zone,
+ resource_id,
+ gcp_conn_id='google_cloud_default',
+ api_version='v1',
+ *args, **kwargs):
+ super(GceInstanceStartOperator, self).__init__(
+ project_id=project_id, zone=zone, resource_id=resource_id,
+ gcp_conn_id=gcp_conn_id, api_version=api_version, *args, **kwargs)
+
+ def execute(self, context):
+ return self._hook.start_instance(self.project_id, self.zone, self.resource_id)
+
+
+class GceInstanceStopOperator(GceBaseOperator):
+ """
+ Stop an instance in Google Compute Engine.
+
+ :param project_id: Google Cloud Platform project where the Compute Engine
+ instance exists.
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the instance exists.
+ :type zone: str
+ :param resource_id: Name of the Compute Engine instance resource.
+ :type resource_id: str
+ :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform.
+ :type gcp_conn_id: str
+ :param api_version: API version used (e.g. v1).
+ :type api_version: str
+ """
+ template_fields = ('project_id', 'zone', 'resource_id', 'gcp_conn_id', 'api_version')
+
+ @apply_defaults
+ def __init__(self,
+ project_id,
+ zone,
+ resource_id,
+ gcp_conn_id='google_cloud_default',
+ api_version='v1',
+ *args, **kwargs):
+ super(GceInstanceStopOperator, self).__init__(
+ project_id=project_id, zone=zone, resource_id=resource_id,
+ gcp_conn_id=gcp_conn_id, api_version=api_version, *args, **kwargs)
+
+ def execute(self, context):
+ return self._hook.stop_instance(self.project_id, self.zone, self.resource_id)
+
+
+SET_MACHINE_TYPE_VALIDATION_SPECIFICATION = [
+ dict(name="machineType", regexp="^.+$"),
+]
+
+
+class GceSetMachineTypeOperator(GceBaseOperator):
+ """
+ Changes the machine type for a stopped instance to the machine type specified in
+ the request.
+
+ :param project_id: Google Cloud Platform project where the Compute Engine
+ instance exists.
+ :type project_id: str
+ :param zone: Google Cloud Platform zone where the instance exists.
+ :type zone: str
+ :param resource_id: Name of the Compute Engine instance resource.
+ :type resource_id: str
+ :param body: Body required by the Compute Engine setMachineType API, as described in
+ https://cloud.google.com/compute/docs/reference/rest/v1/instances/setMachineType#request-body
+ :type body: dict
+ :param gcp_conn_id: The connection ID used to connect to Google Cloud Platform.
+ :type gcp_conn_id: str
+ :param api_version: API version used (e.g. v1).
+ :type api_version: str
+ """
+ template_fields = ('project_id', 'zone', 'resource_id', 'gcp_conn_id', 'api_version')
+
+ @apply_defaults
+ def __init__(self,
+ project_id,
+ zone,
+ resource_id,
+ body,
+ gcp_conn_id='google_cloud_default',
+ api_version='v1',
+ validate_body=True,
+ *args, **kwargs):
+ self.body = body
+ self._field_validator = None
+ if validate_body:
+ self._field_validator = GcpBodyFieldValidator(
+ SET_MACHINE_TYPE_VALIDATION_SPECIFICATION, api_version=api_version)
+ super(GceSetMachineTypeOperator, self).__init__(
+ project_id=project_id, zone=zone, resource_id=resource_id,
+ gcp_conn_id=gcp_conn_id, api_version=api_version, *args, **kwargs)
+
+ def _validate_all_body_fields(self):
+ if self._field_validator:
+ self._field_validator.validate(self.body)
+
+ def execute(self, context):
+ self._validate_all_body_fields()
+ return self._hook.set_machine_type(self.project_id, self.zone,
+ self.resource_id, self.body)
diff --git a/airflow/contrib/operators/gcp_container_operator.py b/airflow/contrib/operators/gcp_container_operator.py
index 5648b4d8a0..fda4d44b9d 100644
--- a/airflow/contrib/operators/gcp_container_operator.py
+++ b/airflow/contrib/operators/gcp_container_operator.py
@@ -17,13 +17,48 @@
# specific language governing permissions and limitations
# under the License.
#
+import os
+import subprocess
+import tempfile
+
from airflow import AirflowException
from airflow.contrib.hooks.gcp_container_hook import GKEClusterHook
+from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.models import BaseOperator
from airflow.utils.decorators import apply_defaults
class GKEClusterDeleteOperator(BaseOperator):
+ """
+ Deletes the cluster, including the Kubernetes endpoint and all worker nodes.
+
+ To delete a certain cluster, you must specify the ``project_id``, the ``name``
+ of the cluster, the ``location`` that the cluster is in, and the ``task_id``.
+
+ **Operator Creation**: ::
+
+ operator = GKEClusterDeleteOperator(
+ task_id='cluster_delete',
+ project_id='my-project',
+ location='cluster-location'
+ name='cluster-name')
+
+ .. seealso::
+ For more detail about deleting clusters have a look at the reference:
+ https://google-cloud-python.readthedocs.io/en/latest/container/gapic/v1/api.html#google.cloud.container_v1.ClusterManagerClient.delete_cluster
+
+ :param project_id: The Google Developers Console [project ID or project number]
+ :type project_id: str
+ :param name: The name of the resource to delete, in this case cluster name
+ :type name: str
+ :param location: The name of the Google Compute Engine zone in which the cluster
+ resides.
+ :type location: str
+ :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
+ :type gcp_conn_id: str
+ :param api_version: The api version to use
+ :type api_version: str
+ """
template_fields = ['project_id', 'gcp_conn_id', 'name', 'location', 'api_version']
@apply_defaults
@@ -35,37 +70,6 @@ def __init__(self,
api_version='v2',
*args,
**kwargs):
- """
- Deletes the cluster, including the Kubernetes endpoint and all worker nodes.
-
-
- To delete a certain cluster, you must specify the ``project_id``, the ``name``
- of the cluster, the ``location`` that the cluster is in, and the ``task_id``.
-
- **Operator Creation**: ::
-
- operator = GKEClusterDeleteOperator(
- task_id='cluster_delete',
- project_id='my-project',
- location='cluster-location'
- name='cluster-name')
-
- .. seealso::
- For more detail about deleting clusters have a look at the reference:
- https://google-cloud-python.readthedocs.io/en/latest/container/gapic/v1/api.html#google.cloud.container_v1.ClusterManagerClient.delete_cluster
-
- :param project_id: The Google Developers Console [project ID or project number]
- :type project_id: str
- :param name: The name of the resource to delete, in this case cluster name
- :type name: str
- :param location: The name of the Google Compute Engine zone in which the cluster
- resides.
- :type location: str
- :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: str
- :param api_version: The api version to use
- :type api_version: str
- """
super(GKEClusterDeleteOperator, self).__init__(*args, **kwargs)
self.project_id = project_id
@@ -88,61 +92,63 @@ def execute(self, context):
class GKEClusterCreateOperator(BaseOperator):
+ """
+ Create a Google Kubernetes Engine Cluster of specified dimensions
+ The operator will wait until the cluster is created.
+
+ The **minimum** required to define a cluster to create is:
+
+ ``dict()`` ::
+ cluster_def = {'name': 'my-cluster-name',
+ 'initial_node_count': 1}
+
+ or
+
+ ``Cluster`` proto ::
+ from google.cloud.container_v1.types import Cluster
+
+ cluster_def = Cluster(name='my-cluster-name', initial_node_count=1)
+
+ **Operator Creation**: ::
+
+ operator = GKEClusterCreateOperator(
+ task_id='cluster_create',
+ project_id='my-project',
+ location='my-location'
+ body=cluster_def)
+
+ .. seealso::
+ For more detail on about creating clusters have a look at the reference:
+ https://google-cloud-python.readthedocs.io/en/latest/container/gapic/v1/types.html#google.cloud.container_v1.types.Cluster
+
+ :param project_id: The Google Developers Console [project ID or project number]
+ :type project_id: str
+ :param location: The name of the Google Compute Engine zone in which the cluster
+ resides.
+ :type location: str
+ :param body: The Cluster definition to create, can be protobuf or python dict, if
+ dict it must match protobuf message Cluster
+ :type body: dict or google.cloud.container_v1.types.Cluster
+ :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
+ :type gcp_conn_id: str
+ :param api_version: The api version to use
+ :type api_version: str
+ """
template_fields = ['project_id', 'gcp_conn_id', 'location', 'api_version', 'body']
@apply_defaults
def __init__(self,
project_id,
location,
- body={},
+ body=None,
gcp_conn_id='google_cloud_default',
api_version='v2',
*args,
**kwargs):
- """
- Create a Google Kubernetes Engine Cluster of specified dimensions
- The operator will wait until the cluster is created.
-
- The **minimum** required to define a cluster to create is:
-
- ``dict()`` ::
- cluster_def = {'name': 'my-cluster-name',
- 'initial_node_count': 1}
-
- or
-
- ``Cluster`` proto ::
- from google.cloud.container_v1.types import Cluster
-
- cluster_def = Cluster(name='my-cluster-name', initial_node_count=1)
-
- **Operator Creation**: ::
-
- operator = GKEClusterCreateOperator(
- task_id='cluster_create',
- project_id='my-project',
- location='my-location'
- body=cluster_def)
-
- .. seealso::
- For more detail on about creating clusters have a look at the reference:
- https://google-cloud-python.readthedocs.io/en/latest/container/gapic/v1/types.html#google.cloud.container_v1.types.Cluster
-
- :param project_id: The Google Developers Console [project ID or project number]
- :type project_id: str
- :param location: The name of the Google Compute Engine zone in which the cluster
- resides.
- :type location: str
- :param body: The Cluster definition to create, can be protobuf or python dict, if
- dict it must match protobuf message Cluster
- :type body: dict or google.cloud.container_v1.types.Cluster
- :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
- :type gcp_conn_id: str
- :param api_version: The api version to use
- :type api_version: str
- """
super(GKEClusterCreateOperator, self).__init__(*args, **kwargs)
+ if body is None:
+ body = {}
self.project_id = project_id
self.gcp_conn_id = gcp_conn_id
self.location = location
@@ -170,3 +176,147 @@ def execute(self, context):
hook = GKEClusterHook(self.project_id, self.location)
create_op = hook.create_cluster(cluster=self.body)
return create_op
+
+
+KUBE_CONFIG_ENV_VAR = "KUBECONFIG"
+G_APP_CRED = "GOOGLE_APPLICATION_CREDENTIALS"
+
+
+class GKEPodOperator(KubernetesPodOperator):
+ """
+ Executes a task in a Kubernetes pod in the specified Google Kubernetes
+ Engine cluster
+
+ This Operator assumes that the system has gcloud installed and either
+ has working default application credentials or has configured a
+ connection id with a service account.
+
+ The **minimum** required to define a cluster to create are the variables
+ ``task_id``, ``project_id``, ``location``, ``cluster_name``, ``name``,
+ ``namespace``, and ``image``
+
+ **Operator Creation**: ::
+
+ operator = GKEPodOperator(task_id='pod_op',
+ project_id='my-project',
+ location='us-central1-a',
+ cluster_name='my-cluster-name',
+ name='task-name',
+ namespace='default',
+ image='perl')
+
+ .. seealso::
+ For more detail about application authentication have a look at the reference:
+ https://cloud.google.com/docs/authentication/production#providing_credentials_to_your_application
+
+ :param project_id: The Google Developers Console project id
+ :type project_id: str
+ :param location: The name of the Google Kubernetes Engine zone in which the
+ cluster resides, e.g. 'us-central1-a'
+ :type location: str
+ :param cluster_name: The name of the Google Kubernetes Engine cluster the pod
+ should be spawned in
+ :type cluster_name: str
+ :param gcp_conn_id: The google cloud connection id to use. This allows for
+ users to specify a service account.
+ :type gcp_conn_id: str
+ """
+ template_fields = ('project_id', 'location',
+ 'cluster_name') + KubernetesPodOperator.template_fields
+
+ @apply_defaults
+ def __init__(self,
+ project_id,
+ location,
+ cluster_name,
+ gcp_conn_id='google_cloud_default',
+ *args,
+ **kwargs):
+ super(GKEPodOperator, self).__init__(*args, **kwargs)
+ self.project_id = project_id
+ self.location = location
+ self.cluster_name = cluster_name
+ self.gcp_conn_id = gcp_conn_id
+
+ def execute(self, context):
+ # Specifying a service account file allows the user to using non default
+ # authentication for creating a Kubernetes Pod. This is done by setting the
+ # environment variable `GOOGLE_APPLICATION_CREDENTIALS` that gcloud looks at.
+ key_file = None
+
+ # If gcp_conn_id is not specified gcloud will use the default
+ # service account credentials.
+ if self.gcp_conn_id:
+ from airflow.hooks.base_hook import BaseHook
+ # extras is a deserialized json object
+ extras = BaseHook.get_connection(self.gcp_conn_id).extra_dejson
+ # key_file only gets set if a json file is created from a JSON string in
+ # the web ui, else none
+ key_file = self._set_env_from_extras(extras=extras)
+
+ # Write config to a temp file and set the environment variable to point to it.
+ # This is to avoid race conditions of reading/writing a single file
+ with tempfile.NamedTemporaryFile() as conf_file:
+ os.environ[KUBE_CONFIG_ENV_VAR] = conf_file.name
+ # Attempt to get/update credentials
+ # We call gcloud directly instead of using google-cloud-python api
+ # because there is no way to write kubernetes config to a file, which is
+ # required by KubernetesPodOperator.
+ # The gcloud command looks at the env variable `KUBECONFIG` for where to save
+ # the kubernetes config file.
+ subprocess.check_call(
+ ["gcloud", "container", "clusters", "get-credentials",
+ self.cluster_name,
+ "--zone", self.location,
+ "--project", self.project_id])
+
+ # Since the key file is of type mkstemp() closing the file will delete it from
+ # the file system so it cannot be accessed after we don't need it anymore
+ if key_file:
+ key_file.close()
+
+ # Tell `KubernetesPodOperator` where the config file is located
+ self.config_file = os.environ[KUBE_CONFIG_ENV_VAR]
+ super(GKEPodOperator, self).execute(context)
+
+ def _set_env_from_extras(self, extras):
+ """
+ Sets the environment variable `GOOGLE_APPLICATION_CREDENTIALS` with either:
+
+ - The path to the keyfile from the specified connection id
+ - A generated file's path if the user specified JSON in the connection id. The
+ file is assumed to be deleted after the process dies due to how mkstemp()
+ works.
+
+ The environment variable is used inside the gcloud command to determine correct
+ service account to use.
+ """
+ key_path = self._get_field(extras, 'key_path', False)
+ keyfile_json_str = self._get_field(extras, 'keyfile_dict', False)
+
+ if not key_path and not keyfile_json_str:
+ self.log.info('Using gcloud with application default credentials.')
+ elif key_path:
+ os.environ[G_APP_CRED] = key_path
+ else:
+ # Write service account JSON to secure file for gcloud to reference
+ service_key = tempfile.NamedTemporaryFile(delete=False)
+ service_key.write(keyfile_json_str)
+ os.environ[G_APP_CRED] = service_key.name
+ # Return file object to have a pointer to close after use,
+ # thus deleting from file system.
+ return service_key
+
+ def _get_field(self, extras, field, default=None):
+ """
+ Fetches a field from extras, and returns it. This is some Airflow
+ magic. The google_cloud_platform hook type adds custom UI elements
+ to the hook page, which allow admins to specify service_account,
+ key_path, etc. They get formatted as shown below.
+ """
+ long_f = 'extra__google_cloud_platform__{}'.format(field)
+ if long_f in extras:
+ return extras[long_f]
+ else:
+ self.log.info('Field {} not found in extras.'.format(field))
+ return default
diff --git a/airflow/contrib/operators/gcp_function_operator.py b/airflow/contrib/operators/gcp_function_operator.py
new file mode 100644
index 0000000000..8207b9d084
--- /dev/null
+++ b/airflow/contrib/operators/gcp_function_operator.py
@@ -0,0 +1,310 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import re
+
+from googleapiclient.errors import HttpError
+
+from airflow import AirflowException
+from airflow.contrib.utils.gcp_field_validator import GcpBodyFieldValidator, \
+ GcpFieldValidationException
+from airflow.version import version
+from airflow.models import BaseOperator
+from airflow.contrib.hooks.gcp_function_hook import GcfHook
+from airflow.utils.decorators import apply_defaults
+
+
+def _validate_available_memory_in_mb(value):
+ if int(value) <= 0:
+ raise GcpFieldValidationException("The available memory has to be greater than 0")
+
+
+def _validate_max_instances(value):
+ if int(value) <= 0:
+ raise GcpFieldValidationException(
+ "The max instances parameter has to be greater than 0")
+
+
+CLOUD_FUNCTION_VALIDATION = [
+ dict(name="name", regexp="^.+$"),
+ dict(name="description", regexp="^.+$", optional=True),
+ dict(name="entryPoint", regexp=r'^.+$', optional=True),
+ dict(name="runtime", regexp=r'^.+$', optional=True),
+ dict(name="timeout", regexp=r'^.+$', optional=True),
+ dict(name="availableMemoryMb", custom_validation=_validate_available_memory_in_mb,
+ optional=True),
+ dict(name="labels", optional=True),
+ dict(name="environmentVariables", optional=True),
+ dict(name="network", regexp=r'^.+$', optional=True),
+ dict(name="maxInstances", optional=True, custom_validation=_validate_max_instances),
+
+ dict(name="source_code", type="union", fields=[
+ dict(name="sourceArchiveUrl", regexp=r'^.+$'),
+ dict(name="sourceRepositoryUrl", regexp=r'^.+$', api_version='v1beta2'),
+ dict(name="sourceRepository", type="dict", fields=[
+ dict(name="url", regexp=r'^.+$')
+ ]),
+ dict(name="sourceUploadUrl")
+ ]),
+
+ dict(name="trigger", type="union", fields=[
+ dict(name="httpsTrigger", type="dict", fields=[
+ # This dict should be empty at input (url is added at output)
+ ]),
+ dict(name="eventTrigger", type="dict", fields=[
+ dict(name="eventType", regexp=r'^.+$'),
+ dict(name="resource", regexp=r'^.+$'),
+ dict(name="service", regexp=r'^.+$', optional=True),
+ dict(name="failurePolicy", type="dict", optional=True, fields=[
+ dict(name="retry", type="dict", optional=True)
+ ])
+ ])
+ ]),
+]
+
+
+class GcfFunctionDeployOperator(BaseOperator):
+ """
+ Create a function in Google Cloud Functions.
+
+ :param project_id: Project ID that the operator works on
+ :type project_id: str
+ :param location: Region where the operator operates on
+ :type location: str
+ :param body: Body of the cloud function definition. The body must be a CloudFunction
+ dictionary as described in:
+ https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions
+ (note that different API versions require different
+ variants of the CloudFunction dictionary)
+ :type body: dict or google.cloud.functions.v1.CloudFunction
+ :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
+ :type gcp_conn_id: str
+ :param api_version: Version of the API used (for example v1).
+ :type api_version: str
+ :param zip_path: Path to zip file containing source code of the function. If it is
+ set, then sourceUploadUrl should not be specified in the body (or it should
+ be empty), then the zip file will be uploaded using upload URL generated
+ via generateUploadUrl from cloud functions API
+ :type zip_path: str
+ :param validate_body: If set to False, no body validation is performed.
+ :type validate_body: bool
+ """
+
+ @apply_defaults
+ def __init__(self,
+ project_id,
+ location,
+ body,
+ gcp_conn_id='google_cloud_default',
+ api_version='v1',
+ zip_path=None,
+ validate_body=True,
+ *args, **kwargs):
+ self.project_id = project_id
+ self.location = location
+ self.full_location = 'projects/{}/locations/{}'.format(self.project_id,
+ self.location)
+ self.body = body
+ self.gcp_conn_id = gcp_conn_id
+ self.api_version = api_version
+ self.zip_path = zip_path
+ self.zip_path_preprocessor = ZipPathPreprocessor(body, zip_path)
+ self._field_validator = None
+ if validate_body:
+ self._field_validator = GcpBodyFieldValidator(CLOUD_FUNCTION_VALIDATION,
+ api_version=api_version)
+ self._hook = GcfHook(gcp_conn_id=self.gcp_conn_id, api_version=self.api_version)
+ self._validate_inputs()
+ super(GcfFunctionDeployOperator, self).__init__(*args, **kwargs)
+
+ def _validate_inputs(self):
+ if not self.project_id:
+ raise AirflowException("The required parameter 'project_id' is missing")
+ if not self.location:
+ raise AirflowException("The required parameter 'location' is missing")
+ if not self.body:
+ raise AirflowException("The required parameter 'body' is missing")
+ self.zip_path_preprocessor.preprocess_body()
+
+ def _validate_all_body_fields(self):
+ if self._field_validator:
+ self._field_validator.validate(self.body)
+
+ def _create_new_function(self):
+ self._hook.create_new_function(self.full_location, self.body)
+
+ def _update_function(self):
+ self._hook.update_function(self.body['name'], self.body, self.body.keys())
+
+ def _check_if_function_exists(self):
+ name = self.body.get('name')
+ if not name:
+ raise GcpFieldValidationException("The 'name' field should be present in "
+ "body: '{}'.".format(self.body))
+ try:
+ self._hook.get_function(name)
+ except HttpError as e:
+ status = e.resp.status
+ if status == 404:
+ return False
+ raise e
+ return True
+
+ def _upload_source_code(self):
+ return self._hook.upload_function_zip(parent=self.full_location,
+ zip_path=self.zip_path)
+
+ def _set_airflow_version_label(self):
+ if 'labels' not in self.body.keys():
+ self.body['labels'] = {}
+ self.body['labels'].update(
+ {'airflow-version': 'v' + version.replace('.', '-').replace('+', '-')})
+
+ def execute(self, context):
+ if self.zip_path_preprocessor.should_upload_function():
+ self.body[SOURCE_UPLOAD_URL] = self._upload_source_code()
+ self._validate_all_body_fields()
+ self._set_airflow_version_label()
+ if not self._check_if_function_exists():
+ self._create_new_function()
+ else:
+ self._update_function()
+
+
+SOURCE_ARCHIVE_URL = 'sourceArchiveUrl'
+SOURCE_UPLOAD_URL = 'sourceUploadUrl'
+SOURCE_REPOSITORY = 'sourceRepository'
+ZIP_PATH = 'zip_path'
+
+
+class ZipPathPreprocessor:
+ """
+ Pre-processes zip path parameter.
+
+ Responsible for checking if the zip path parameter is correctly specified in
+ relation with source_code body fields. Non empty zip path parameter is special because
+ it is mutually exclusive with sourceArchiveUrl and sourceRepository body fields.
+ It is also mutually exclusive with non-empty sourceUploadUrl.
+ The pre-process modifies sourceUploadUrl body field in special way when zip_path
+ is not empty. An extra step is run when execute method is called and sourceUploadUrl
+ field value is set in the body with the value returned by generateUploadUrl Cloud
+ Function API method.
+
+ :param body: Body passed to the create/update method calls.
+ :type body: dict
+ :param zip_path: path to the zip file containing source code.
+ :type body: dict
+
+ """
+ upload_function = None
+
+ def __init__(self, body, zip_path):
+ self.body = body
+ self.zip_path = zip_path
+
+ @staticmethod
+ def _is_present_and_empty(dictionary, field):
+ return field in dictionary and not dictionary[field]
+
+ def _verify_upload_url_and_no_zip_path(self):
+ if self._is_present_and_empty(self.body, SOURCE_UPLOAD_URL):
+ if not self.zip_path:
+ raise AirflowException(
+ "Parameter '{}' is empty in the body and argument '{}' "
+ "is missing or empty. You need to have non empty '{}' "
+ "when '{}' is present and empty.".
+ format(SOURCE_UPLOAD_URL, ZIP_PATH, ZIP_PATH, SOURCE_UPLOAD_URL))
+
+ def _verify_upload_url_and_zip_path(self):
+ if SOURCE_UPLOAD_URL in self.body and self.zip_path:
+ if not self.body[SOURCE_UPLOAD_URL]:
+ self.upload_function = True
+ else:
+ raise AirflowException("Only one of '{}' in body or '{}' argument "
+ "allowed. Found both."
+ .format(SOURCE_UPLOAD_URL, ZIP_PATH))
+
+ def _verify_archive_url_and_zip_path(self):
+ if SOURCE_ARCHIVE_URL in self.body and self.zip_path:
+ raise AirflowException("Only one of '{}' in body or '{}' argument "
+ "allowed. Found both."
+ .format(SOURCE_ARCHIVE_URL, ZIP_PATH))
+
+ def should_upload_function(self):
+ if self.upload_function is None:
+ raise AirflowException('validate() method has to be invoked before '
+ 'should_upload_function')
+ return self.upload_function
+
+ def preprocess_body(self):
+ self._verify_archive_url_and_zip_path()
+ self._verify_upload_url_and_zip_path()
+ self._verify_upload_url_and_no_zip_path()
+ if self.upload_function is None:
+ self.upload_function = False
+
+
+FUNCTION_NAME_PATTERN = '^projects/[^/]+/locations/[^/]+/functions/[^/]+$'
+FUNCTION_NAME_COMPILED_PATTERN = re.compile(FUNCTION_NAME_PATTERN)
+
+
+class GcfFunctionDeleteOperator(BaseOperator):
+ """
+ Delete a function with specified name from Google Cloud Functions.
+
+ :param name: A fully-qualified function name, matching
+ the pattern: `^projects/[^/]+/locations/[^/]+/functions/[^/]+$`
+ :type name: str
+ :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.
+ :type gcp_conn_id: str
+ :param api_version: Version of the API used (for example v1).
+ :type api_version: str
+ """
+
+ @apply_defaults
+ def __init__(self,
+ name,
+ gcp_conn_id='google_cloud_default',
+ api_version='v1',
+ *args, **kwargs):
+ self.name = name
+ self.gcp_conn_id = gcp_conn_id
+ self.api_version = api_version
+ self._validate_inputs()
+ self.hook = GcfHook(gcp_conn_id=self.gcp_conn_id, api_version=self.api_version)
+ super(GcfFunctionDeleteOperator, self).__init__(*args, **kwargs)
+
+ def _validate_inputs(self):
+ if not self.name:
+ raise AttributeError('Empty parameter: name')
+ else:
+ pattern = FUNCTION_NAME_COMPILED_PATTERN
+ if not pattern.match(self.name):
+ raise AttributeError(
+ 'Parameter name must match pattern: {}'.format(FUNCTION_NAME_PATTERN))
+
+ def execute(self, context):
+ try:
+ return self.hook.delete_function(self.name)
+ except HttpError as e:
+ status = e.resp.status
+ if status == 404:
+ self.log.info('The function does not exist in this project')
+ else:
+ self.log.error('An error occurred. Exiting.')
+ raise e
diff --git a/airflow/contrib/operators/gcs_download_operator.py b/airflow/contrib/operators/gcs_download_operator.py
index ce272aedd5..1d168d4660 100644
--- a/airflow/contrib/operators/gcs_download_operator.py
+++ b/airflow/contrib/operators/gcs_download_operator.py
@@ -29,26 +29,26 @@ class GoogleCloudStorageDownloadOperator(BaseOperator):
Downloads a file from Google Cloud Storage.
:param bucket: The Google cloud storage bucket where the object is. (templated)
- :type bucket: string
+ :type bucket: str
:param object: The name of the object to download in the Google cloud
storage bucket. (templated)
- :type object: string
+ :type object: str
:param filename: The file path on the local file system (where the
operator is being executed) that the file should be downloaded to. (templated)
If no filename passed, the downloaded data will not be stored on the local file
system.
- :type filename: string
+ :type filename: str
:param store_to_xcom_key: If this param is set, the operator will push
the contents of the downloaded file to XCom with the key set in this
parameter. If not set, the downloaded data will not be pushed to XCom. (templated)
- :type store_to_xcom_key: string
+ :type store_to_xcom_key: str
:param google_cloud_storage_conn_id: The connection ID to use when
connecting to Google cloud storage.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ('bucket', 'object', 'filename', 'store_to_xcom_key',)
ui_color = '#f0eee4'
diff --git a/airflow/contrib/operators/gcs_list_operator.py b/airflow/contrib/operators/gcs_list_operator.py
index 6474453afa..7b37b269a6 100644
--- a/airflow/contrib/operators/gcs_list_operator.py
+++ b/airflow/contrib/operators/gcs_list_operator.py
@@ -30,21 +30,21 @@ class GoogleCloudStorageListOperator(BaseOperator):
`xcom` in the downstream task.
:param bucket: The Google cloud storage bucket to find the objects. (templated)
- :type bucket: string
+ :type bucket: str
:param prefix: Prefix string which filters objects whose name begin with
this prefix. (templated)
- :type prefix: string
+ :type prefix: str
:param delimiter: The delimiter by which you want to filter the objects. (templated)
For e.g to lists the CSV files from in a directory in GCS you would use
delimiter='.csv'.
- :type delimiter: string
+ :type delimiter: str
:param google_cloud_storage_conn_id: The connection ID to use when
connecting to Google cloud storage.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
**Example**:
The following Operator would list all the Avro files from ``sales/sales-2017``
diff --git a/airflow/contrib/operators/gcs_operator.py b/airflow/contrib/operators/gcs_operator.py
index ef5e8de504..c685722185 100644
--- a/airflow/contrib/operators/gcs_operator.py
+++ b/airflow/contrib/operators/gcs_operator.py
@@ -33,7 +33,7 @@ class GoogleCloudStorageCreateBucketOperator(BaseOperator):
https://cloud.google.com/storage/docs/bucketnaming.html#requirements
:param bucket_name: The name of the bucket. (templated)
- :type bucket_name: string
+ :type bucket_name: str
:param storage_class: This defines how objects in the bucket are stored
and determines the SLA and the cost of storage (templated). Values include
@@ -44,7 +44,7 @@ class GoogleCloudStorageCreateBucketOperator(BaseOperator):
- ``COLDLINE``.
If this value is not specified when the bucket is
created, it will default to STANDARD.
- :type storage_class: string
+ :type storage_class: str
:param location: The location of the bucket. (templated)
Object data for objects in the bucket resides in physical storage
within this region. Defaults to US.
@@ -52,18 +52,18 @@ class GoogleCloudStorageCreateBucketOperator(BaseOperator):
.. seealso::
https://developers.google.com/storage/docs/bucket-locations
- :type location: string
+ :type location: str
:param project_id: The ID of the GCP Project. (templated)
- :type project_id: string
+ :type project_id: str
:param labels: User-provided labels, in key/value pairs.
:type labels: dict
:param google_cloud_storage_conn_id: The connection ID to use when
connecting to Google cloud storage.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must
have domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
**Example**:
The following Operator would create a new bucket ``test-bucket``
diff --git a/airflow/contrib/operators/gcs_to_bq.py b/airflow/contrib/operators/gcs_to_bq.py
index 3a7798030c..39dff21606 100644
--- a/airflow/contrib/operators/gcs_to_bq.py
+++ b/airflow/contrib/operators/gcs_to_bq.py
@@ -35,42 +35,42 @@ class GoogleCloudStorageToBigQueryOperator(BaseOperator):
Google cloud storage must be a JSON file with the schema fields in it.
:param bucket: The bucket to load from. (templated)
- :type bucket: string
+ :type bucket: str
:param source_objects: List of Google cloud storage URIs to load from. (templated)
If source_format is 'DATASTORE_BACKUP', the list must only contain a single URI.
- :type object: list
+ :type source_objects: list of str
:param destination_project_dataset_table: The dotted (<project>.)<dataset>.<table>
BigQuery table to load data into. If <project> is not included,
project will be the project defined in the connection json. (templated)
- :type destination_project_dataset_table: string
+ :type destination_project_dataset_table: str
:param schema_fields: If set, the schema field list as defined here:
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load
Should not be set when source_format is 'DATASTORE_BACKUP'.
:type schema_fields: list
:param schema_object: If set, a GCS object path pointing to a .json file that
contains the schema for the table. (templated)
- :param schema_object: string
+ :type schema_object: str
:param source_format: File format to export.
- :type source_format: string
+ :type source_format: str
:param compression: [Optional] The compression type of the data source.
Possible values include GZIP and NONE.
The default value is NONE.
This setting is ignored for Google Cloud Bigtable,
Google Cloud Datastore backups and Avro formats.
- :type compression: string
+ :type compression: str
:param create_disposition: The create disposition if the table doesn't exist.
- :type create_disposition: string
+ :type create_disposition: str
:param skip_leading_rows: Number of rows to skip when loading from a CSV.
:type skip_leading_rows: int
:param write_disposition: The write disposition if the table already exists.
- :type write_disposition: string
+ :type write_disposition: str
:param field_delimiter: The delimiter to use when loading from a CSV.
- :type field_delimiter: string
+ :type field_delimiter: str
:param max_bad_records: The maximum number of bad records that BigQuery can
ignore when running the job.
:type max_bad_records: int
:param quote_character: The value that is used to quote data sections in a CSV file.
- :type quote_character: string
+ :type quote_character: str
:param ignore_unknown_values: [Optional] Indicates if BigQuery should allow
extra values that are not represented in the table schema.
If true, the extra values are ignored. If false, records with extra columns
@@ -78,7 +78,7 @@ class GoogleCloudStorageToBigQueryOperator(BaseOperator):
invalid error is returned in the job result.
:type ignore_unknown_values: bool
:param allow_quoted_newlines: Whether to allow quoted newlines (true) or not (false).
- :type allow_quoted_newlines: boolean
+ :type allow_quoted_newlines: bool
:param allow_jagged_rows: Accept rows that are missing trailing optional columns.
The missing values are treated as nulls. If false, records with missing trailing
columns are treated as bad records, and if there are too many bad records, an
@@ -86,21 +86,21 @@ class GoogleCloudStorageToBigQueryOperator(BaseOperator):
for other formats.
:type allow_jagged_rows: bool
:param max_id_key: If set, the name of a column in the BigQuery table
- that's to be loaded. Thsi will be used to select the MAX value from
+ that's to be loaded. This will be used to select the MAX value from
BigQuery after the load occurs. The results will be returned by the
execute() command, which in turn gets stored in XCom for future
operators to use. This can be helpful with incremental loads--during
future executions, you can pick up from the max ID.
- :type max_id_key: string
+ :type max_id_key: str
:param bigquery_conn_id: Reference to a specific BigQuery hook.
- :type bigquery_conn_id: string
+ :type bigquery_conn_id: str
:param google_cloud_storage_conn_id: Reference to a specific Google
cloud storage hook.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any. For this to
work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param schema_update_options: Allows the schema of the destination
table to be updated as a side effect of the load job.
:type schema_update_options: list
@@ -114,6 +114,11 @@ class GoogleCloudStorageToBigQueryOperator(BaseOperator):
Note that 'field' is not available in concurrency with
dataset.table$partition.
:type time_partitioning: dict
+ :param cluster_fields: Request that the result of this load be stored sorted
+ by one or more columns. This is only available in conjunction with
+ time_partitioning. The order of columns given determines the sort order.
+ Not applicable for external tables.
+ :type cluster_fields: list of str
"""
template_fields = ('bucket', 'source_objects',
'schema_object', 'destination_project_dataset_table')
@@ -143,14 +148,19 @@ def __init__(self,
google_cloud_storage_conn_id='google_cloud_default',
delegate_to=None,
schema_update_options=(),
- src_fmt_configs={},
+ src_fmt_configs=None,
external_table=False,
- time_partitioning={},
+ time_partitioning=None,
+ cluster_fields=None,
*args, **kwargs):
super(GoogleCloudStorageToBigQueryOperator, self).__init__(*args, **kwargs)
# GCS config
+ if src_fmt_configs is None:
+ src_fmt_configs = {}
+ if time_partitioning is None:
+ time_partitioning = {}
self.bucket = bucket
self.source_objects = source_objects
self.schema_object = schema_object
@@ -179,6 +189,7 @@ def __init__(self,
self.schema_update_options = schema_update_options
self.src_fmt_configs = src_fmt_configs
self.time_partitioning = time_partitioning
+ self.cluster_fields = cluster_fields
def execute(self, context):
bq_hook = BigQueryHook(bigquery_conn_id=self.bigquery_conn_id,
@@ -234,7 +245,8 @@ def execute(self, context):
allow_jagged_rows=self.allow_jagged_rows,
schema_update_options=self.schema_update_options,
src_fmt_configs=self.src_fmt_configs,
- time_partitioning=self.time_partitioning)
+ time_partitioning=self.time_partitioning,
+ cluster_fields=self.cluster_fields)
if self.max_id_key:
cursor.execute('SELECT MAX({}) FROM {}'.format(
diff --git a/airflow/contrib/operators/gcs_to_gcs.py b/airflow/contrib/operators/gcs_to_gcs.py
index 256685f90b..12fbff5276 100644
--- a/airflow/contrib/operators/gcs_to_gcs.py
+++ b/airflow/contrib/operators/gcs_to_gcs.py
@@ -28,7 +28,7 @@ class GoogleCloudStorageToGoogleCloudStorageOperator(BaseOperator):
:param source_bucket: The source Google cloud storage bucket where the
object is. (templated)
- :type source_bucket: string
+ :type source_bucket: str
:param source_object: The source name of the object to copy in the Google cloud
storage bucket. (templated)
If wildcards are used in this argument:
@@ -36,33 +36,32 @@ class GoogleCloudStorageToGoogleCloudStorageOperator(BaseOperator):
bucket. The wildcard can appear inside the object name or at the
end of the object name. Appending a wildcard to the bucket name is
unsupported.
- :type source_object: string
+ :type source_object: str
:param destination_bucket: The destination Google cloud storage bucket
- where the object should be. (templated)
- :type destination_bucket: string
+ where the object should be. (templated)
+ :type destination_bucket: str
:param destination_object: The destination name of the object in the
destination Google cloud storage bucket. (templated)
If a wildcard is supplied in the source_object argument, this is the
prefix that will be prepended to the final destination objects' paths.
Note that the source path's part before the wildcard will be removed;
if it needs to be retained it should be appended to destination_object.
- For example, with prefix ``foo/*`` and destination_object `'blah/``, the
+ For example, with prefix ``foo/*`` and destination_object ``blah/``, the
file ``foo/baz`` will be copied to ``blah/baz``; to retain the prefix write
the destination_object as e.g. ``blah/foo``, in which case the copied file
will be named ``blah/foo/baz``.
- :type destination_object: string
+ :type destination_object: str
:param move_object: When move object is True, the object is moved instead
- of copied to the new location.
- This is the equivalent of a mv command as opposed to a
- cp command.
+ of copied to the new location. This is the equivalent of a mv command
+ as opposed to a cp command.
:type move_object: bool
:param google_cloud_storage_conn_id: The connection ID to use when
connecting to Google cloud storage.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
**Examples**:
The following Operator would copy a single file named
diff --git a/airflow/contrib/operators/gcs_to_s3.py b/airflow/contrib/operators/gcs_to_s3.py
index a87aa3af5c..d8b180c81a 100644
--- a/airflow/contrib/operators/gcs_to_s3.py
+++ b/airflow/contrib/operators/gcs_to_s3.py
@@ -28,25 +28,35 @@ class GoogleCloudStorageToS3Operator(GoogleCloudStorageListOperator):
Synchronizes a Google Cloud Storage bucket with an S3 bucket.
:param bucket: The Google Cloud Storage bucket to find the objects. (templated)
- :type bucket: string
+ :type bucket: str
:param prefix: Prefix string which filters objects whose name begin with
this prefix. (templated)
- :type prefix: string
+ :type prefix: str
:param delimiter: The delimiter by which you want to filter the objects. (templated)
For e.g to lists the CSV files from in a directory in GCS you would use
delimiter='.csv'.
- :type delimiter: string
+ :type delimiter: str
:param google_cloud_storage_conn_id: The connection ID to use when
connecting to Google Cloud Storage.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param dest_aws_conn_id: The destination S3 connection
:type dest_aws_conn_id: str
:param dest_s3_key: The base S3 key to be used to store the files. (templated)
:type dest_s3_key: str
+ :parame dest_verify: Whether or not to verify SSL certificates for S3 connection.
+ By default SSL certificates are verified.
+ You can provide the following values:
+ - False: do not validate SSL certificates. SSL will still be used
+ (unless use_ssl is False), but SSL certificates will not be
+ verified.
+ - path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.
+ You can specify this argument if you want to use a different
+ CA cert bundle than the one used by botocore.
+ :type dest_verify: bool or str
"""
template_fields = ('bucket', 'prefix', 'delimiter', 'dest_s3_key')
ui_color = '#f0eee4'
@@ -60,6 +70,7 @@ def __init__(self,
delegate_to=None,
dest_aws_conn_id=None,
dest_s3_key=None,
+ dest_verify=None,
replace=False,
*args,
**kwargs):
@@ -75,12 +86,13 @@ def __init__(self,
)
self.dest_aws_conn_id = dest_aws_conn_id
self.dest_s3_key = dest_s3_key
+ self.dest_verify = dest_verify
self.replace = replace
def execute(self, context):
# use the super to list all files in an Google Cloud Storage bucket
files = super(GoogleCloudStorageToS3Operator, self).execute(context)
- s3_hook = S3Hook(aws_conn_id=self.dest_aws_conn_id)
+ s3_hook = S3Hook(aws_conn_id=self.dest_aws_conn_id, verify=self.dest_verify)
if not self.replace:
# if we are not replacing -> list all files in the S3 bucket
diff --git a/airflow/contrib/operators/hipchat_operator.py b/airflow/contrib/operators/hipchat_operator.py
index 381cd72cdf..adeca23079 100644
--- a/airflow/contrib/operators/hipchat_operator.py
+++ b/airflow/contrib/operators/hipchat_operator.py
@@ -99,24 +99,23 @@ class HipChatAPISendRoomNotificationOperator(HipChatAPIOperator):
:param card: HipChat-defined card object
:type card: dict
"""
- template_fields = ('token', 'room_id', 'message')
+ template_fields = ('token', 'room_id', 'message', 'message_format',
+ 'color', 'frm', 'attach_to', 'notify', 'card')
ui_color = '#2980b9'
@apply_defaults
- def __init__(self, room_id, message, *args, **kwargs):
+ def __init__(self, room_id, message, message_format='html',
+ color='yellow', frm='airflow', attach_to=None,
+ notify=False, card=None, *args, **kwargs):
super(HipChatAPISendRoomNotificationOperator, self).__init__(*args, **kwargs)
self.room_id = room_id
self.message = message
- default_options = {
- 'message_format': 'html',
- 'color': 'yellow',
- 'frm': 'airflow',
- 'attach_to': None,
- 'notify': False,
- 'card': None
- }
- for (prop, default) in default_options.items():
- setattr(self, prop, kwargs.get(prop, default))
+ self.message_format = message_format
+ self.color = color
+ self.frm = frm
+ self.attach_to = attach_to
+ self.notify = notify
+ self.card = card
def prepare_request(self):
params = {
diff --git a/airflow/contrib/operators/jenkins_job_trigger_operator.py b/airflow/contrib/operators/jenkins_job_trigger_operator.py
index 37185f3e13..3e1aba0c1b 100644
--- a/airflow/contrib/operators/jenkins_job_trigger_operator.py
+++ b/airflow/contrib/operators/jenkins_job_trigger_operator.py
@@ -95,11 +95,11 @@ class JenkinsJobTriggerOperator(BaseOperator):
version >= 0.4.15 to communicate with jenkins server.
You'll also need to configure a Jenkins connection in the connections screen.
:param jenkins_connection_id: The jenkins connection to use for this job
- :type jenkins_connection_id: string
+ :type jenkins_connection_id: str
:param job_name: The name of the job to trigger
- :type job_name: string
+ :type job_name: str
:param parameters: The parameters block to provide to jenkins. (templated)
- :type parameters: string
+ :type parameters: str
:param sleep_time: How long will the operator sleep between each status
request for the job (min 1, default 10)
:type sleep_time: int
diff --git a/airflow/contrib/operators/kubernetes_pod_operator.py b/airflow/contrib/operators/kubernetes_pod_operator.py
index bf656f12fb..d3c396e668 100644
--- a/airflow/contrib/operators/kubernetes_pod_operator.py
+++ b/airflow/contrib/operators/kubernetes_pod_operator.py
@@ -65,11 +65,13 @@ class KubernetesPodOperator(BaseOperator):
:type in_cluster: bool
:param cluster_context: context that points to kubernetes cluster.
Ignored when in_cluster is True. If None, current-context is used.
- :type cluster_context: string
+ :type cluster_context: str
:param get_logs: get the stdout of the container as logs of the tasks
:type get_logs: bool
:param affinity: A dict containing a group of affinity scheduling rules
:type affinity: dict
+ :param node_selectors: A dict containing a group of scheduling rules
+ :type node_selectors: dict
:param config_file: The path to the Kubernetes config file
:type config_file: str
:param xcom_push: If xcom_push is True, the content of the file
@@ -100,12 +102,15 @@ def execute(self, context):
labels=self.labels,
)
+ pod.service_account_name = self.service_account_name
pod.secrets = self.secrets
pod.envs = self.env_vars
pod.image_pull_policy = self.image_pull_policy
pod.annotations = self.annotations
pod.resources = self.resources
pod.affinity = self.affinity
+ pod.node_selectors = self.node_selectors
+ pod.hostnetwork = self.hostnetwork
launcher = pod_launcher.PodLauncher(kube_client=client,
extract_xcom=self.xcom_push)
@@ -113,6 +118,10 @@ def execute(self, context):
pod,
startup_timeout=self.startup_timeout_seconds,
get_logs=self.get_logs)
+
+ if self.is_delete_operator_pod:
+ launcher.delete_pod(pod)
+
if final_state != State.SUCCESS:
raise AirflowException(
'Pod returned a failure: {state}'.format(state=final_state)
@@ -144,6 +153,11 @@ def __init__(self,
affinity=None,
config_file=None,
xcom_push=False,
+ node_selectors=None,
+ image_pull_secrets=None,
+ service_account_name="default",
+ is_delete_operator_pod=False,
+ hostnetwork=False,
*args,
**kwargs):
super(KubernetesPodOperator, self).__init__(*args, **kwargs)
@@ -162,8 +176,13 @@ def __init__(self,
self.cluster_context = cluster_context
self.get_logs = get_logs
self.image_pull_policy = image_pull_policy
+ self.node_selectors = node_selectors or {}
self.annotations = annotations or {}
self.affinity = affinity or {}
self.xcom_push = xcom_push
self.resources = resources or Resources()
self.config_file = config_file
+ self.image_pull_secrets = image_pull_secrets
+ self.service_account_name = service_account_name
+ self.is_delete_operator_pod = is_delete_operator_pod
+ self.hostnetwork = hostnetwork
diff --git a/airflow/contrib/operators/mlengine_operator.py b/airflow/contrib/operators/mlengine_operator.py
index 9fe966d387..8091ceefff 100644
--- a/airflow/contrib/operators/mlengine_operator.py
+++ b/airflow/contrib/operators/mlengine_operator.py
@@ -19,7 +19,7 @@
from airflow.contrib.hooks.gcp_mlengine_hook import MLEngineHook
from airflow.exceptions import AirflowException
-from airflow.operators import BaseOperator
+from airflow.models import BaseOperator
from airflow.utils.decorators import apply_defaults
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -94,16 +94,16 @@ class MLEngineBatchPredictionOperator(BaseOperator):
:param project_id: The Google Cloud project name where the
prediction job is submitted. (templated)
- :type project_id: string
+ :type project_id: str
:param job_id: A unique id for the prediction job on Google Cloud
ML Engine. (templated)
- :type job_id: string
+ :type job_id: str
:param data_format: The format of the input data.
It will default to 'DATA_FORMAT_UNSPECIFIED' if is not provided
or is not one of ["TEXT", "TF_RECORD", "TF_RECORD_GZIP"].
- :type data_format: string
+ :type data_format: str
:param input_paths: A list of GCS paths of input data for batch
prediction. Accepting wildcard operator *, but only at the end. (templated)
@@ -111,28 +111,28 @@ class MLEngineBatchPredictionOperator(BaseOperator):
:param output_path: The GCS path where the prediction results are
written to. (templated)
- :type output_path: string
+ :type output_path: str
:param region: The Google Compute Engine region to run the
prediction job in. (templated)
- :type region: string
+ :type region: str
:param model_name: The Google Cloud ML Engine model to use for prediction.
If version_name is not provided, the default version of this
model will be used.
Should not be None if version_name is provided.
Should be None if uri is provided. (templated)
- :type model_name: string
+ :type model_name: str
:param version_name: The Google Cloud ML Engine model version to use for
prediction.
Should be None if uri is provided. (templated)
- :type version_name: string
+ :type version_name: str
:param uri: The GCS path of the saved model to use for prediction.
Should be None if model_name is provided.
It should be a GCS path pointing to a tensorflow SavedModel. (templated)
- :type uri: string
+ :type uri: str
:param max_worker_count: The maximum number of workers to be used
for parallel processing. Defaults to 10 if not specified.
@@ -140,16 +140,16 @@ class MLEngineBatchPredictionOperator(BaseOperator):
:param runtime_version: The Google Cloud ML Engine runtime version to use
for batch prediction.
- :type runtime_version: string
+ :type runtime_version: str
:param gcp_conn_id: The connection ID used for connection to Google
Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must
have doamin-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
Raises:
``ValueError``: if a unique model/version origin cannot be determined.
@@ -281,8 +281,7 @@ class MLEngineModelOperator(BaseOperator):
:param project_id: The Google Cloud project name to which MLEngine
model belongs. (templated)
- :type project_id: string
-
+ :type project_id: str
:param model: A dictionary containing the information about the model.
If the `operation` is `create`, then the `model` parameter should
contain all the information about this model such as `name`.
@@ -290,19 +289,17 @@ class MLEngineModelOperator(BaseOperator):
If the `operation` is `get`, the `model` parameter
should contain the `name` of the model.
:type model: dict
-
:param operation: The operation to perform. Available operations are:
* ``create``: Creates a new model as provided by the `model` parameter.
* ``get``: Gets a particular model where the name is specified in `model`.
-
+ :type operation: str
:param gcp_conn_id: The connection ID to use when fetching connection info.
- :type gcp_conn_id: string
-
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = [
@@ -342,17 +339,17 @@ class MLEngineVersionOperator(BaseOperator):
:param project_id: The Google Cloud project name to which MLEngine
model belongs.
- :type project_id: string
+ :type project_id: str
:param model_name: The name of the Google Cloud ML Engine model that the version
belongs to. (templated)
- :type model_name: string
+ :type model_name: str
:param version_name: A name to use for the version being operated upon.
If not None and the `version` argument is None or does not have a value for
the `name` key, then this will be populated in the payload for the
`name` key. (templated)
- :type version_name: string
+ :type version_name: str
:param version: A dictionary containing the information about the version.
If the `operation` is `create`, `version` should contain all the
@@ -381,15 +378,15 @@ class MLEngineVersionOperator(BaseOperator):
model specified by `model_name`).
The name of the version should be specified in the `version`
parameter.
- :type operation: string
+ :type operation: str
:param gcp_conn_id: The connection ID to use when fetching connection info.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = [
@@ -427,7 +424,9 @@ def execute(self, context):
gcp_conn_id=self._gcp_conn_id, delegate_to=self._delegate_to)
if self._operation == 'create':
- assert self._version is not None
+ if not self._version:
+ raise ValueError("version attribute of {} could not "
+ "be empty".format(self.__class__.__name__))
return hook.create_version(self._project_id, self._model_name,
self._version)
elif self._operation == 'set_default':
@@ -448,56 +447,56 @@ class MLEngineTrainingOperator(BaseOperator):
:param project_id: The Google Cloud project name within which MLEngine
training job should run (templated).
- :type project_id: string
+ :type project_id: str
:param job_id: A unique templated id for the submitted Google MLEngine
training job. (templated)
- :type job_id: string
+ :type job_id: str
:param package_uris: A list of package locations for MLEngine training job,
which should include the main training program + any additional
dependencies. (templated)
- :type package_uris: string
+ :type package_uris: str
:param training_python_module: The Python module name to run within MLEngine
training job after installing 'package_uris' packages. (templated)
- :type training_python_module: string
+ :type training_python_module: str
:param training_args: A list of templated command line arguments to pass to
the MLEngine training program. (templated)
- :type training_args: string
+ :type training_args: str
:param region: The Google Compute Engine region to run the MLEngine training
job in (templated).
- :type region: string
+ :type region: str
:param scale_tier: Resource tier for MLEngine training job. (templated)
- :type scale_tier: string
+ :type scale_tier: str
:param runtime_version: The Google Cloud ML runtime version to use for
training. (templated)
- :type runtime_version: string
+ :type runtime_version: str
:param python_version: The version of Python used in training. (templated)
- :type python_version: string
+ :type python_version: str
:param job_dir: A Google Cloud Storage path in which to store training
outputs and other data needed for training. (templated)
- :type job_dir: string
+ :type job_dir: str
:param gcp_conn_id: The connection ID to use when fetching connection info.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param mode: Can be one of 'DRY_RUN'/'CLOUD'. In 'DRY_RUN' mode, no real
training job will be launched, but the MLEngine training job request
will be printed out. In 'CLOUD' mode, a real MLEngine training job
creation request will be issued.
- :type mode: string
+ :type mode: str
"""
template_fields = [
diff --git a/airflow/contrib/operators/mlengine_operator_utils.py b/airflow/contrib/operators/mlengine_operator_utils.py
index 7ce784ebb4..c2951b715d 100644
--- a/airflow/contrib/operators/mlengine_operator_utils.py
+++ b/airflow/contrib/operators/mlengine_operator_utils.py
@@ -108,16 +108,16 @@ def validate_err_and_count(summary):
:param task_prefix: a prefix for the tasks. Only alphanumeric characters and
hyphen are allowed (no underscores), since this will be used as dataflow
job name, which doesn't allow other characters.
- :type task_prefix: string
+ :type task_prefix: str
:param data_format: either of 'TEXT', 'TF_RECORD', 'TF_RECORD_GZIP'
- :type data_format: string
+ :type data_format: str
:param input_paths: a list of input paths to be sent to BatchPrediction.
:type input_paths: list of strings
:param prediction_path: GCS path to put the prediction results in.
- :type prediction_path: string
+ :type prediction_path: str
:param metric_fn_and_keys: a tuple of metric_fn and metric_keys:
- metric_fn is a function that accepts a dictionary (for an instance),
@@ -132,17 +132,17 @@ def validate_err_and_count(summary):
:param batch_prediction_job_id: the id to use for the Cloud ML Batch
prediction job. Passed directly to the MLEngineBatchPredictionOperator as
the job_id argument.
- :type batch_prediction_job_id: string
+ :type batch_prediction_job_id: str
:param project_id: the Google Cloud Platform project id in which to execute
Cloud ML Batch Prediction and Dataflow jobs. If None, then the `dag`'s
`default_args['project_id']` will be used.
- :type project_id: string
+ :type project_id: str
:param region: the Google Cloud Platform region in which to execute Cloud ML
Batch Prediction and Dataflow jobs. If None, then the `dag`'s
`default_args['region']` will be used.
- :type region: string
+ :type region: str
:param dataflow_options: options to run Dataflow jobs. If None, then the
`dag`'s `default_args['dataflow_default_options']` will be used.
@@ -152,19 +152,19 @@ def validate_err_and_count(summary):
tensorflow.estimator.export_savedmodel(). It cannot be used with
model_name or version_name below. See MLEngineBatchPredictionOperator for
more detail.
- :type model_uri: string
+ :type model_uri: str
:param model_name: Used to indicate a model to use for prediction. Can be
used in combination with version_name, but cannot be used together with
model_uri. See MLEngineBatchPredictionOperator for more detail. If None,
then the `dag`'s `default_args['model_name']` will be used.
- :type model_name: string
+ :type model_name: str
- :param version_name: Used to indicate a model version to use for prediciton,
+ :param version_name: Used to indicate a model version to use for prediction,
in combination with model_name. Cannot be used together with model_uri.
See MLEngineBatchPredictionOperator for more detail. If None, then the
`dag`'s `default_args['version_name']` will be used.
- :type version_name: string
+ :type version_name: str
:param dag: The `DAG` to use for all Operators.
:type dag: airflow.DAG
diff --git a/airflow/contrib/operators/mlengine_prediction_summary.py b/airflow/contrib/operators/mlengine_prediction_summary.py
index 17fc2c0903..def793c1be 100644
--- a/airflow/contrib/operators/mlengine_prediction_summary.py
+++ b/airflow/contrib/operators/mlengine_prediction_summary.py
@@ -1,3 +1,4 @@
+# flake8: noqa: F841
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
@@ -102,24 +103,26 @@ def metric_fn(inst):
class JsonCoder(object):
- def encode(self, x):
+ @staticmethod
+ def encode(x):
return json.dumps(x)
- def decode(self, x):
+ @staticmethod
+ def decode(x):
return json.loads(x)
@beam.ptransform_fn
def MakeSummary(pcoll, metric_fn, metric_keys): # pylint: disable=invalid-name
return (
- pcoll
- | "ApplyMetricFnPerInstance" >> beam.Map(metric_fn)
- | "PairWith1" >> beam.Map(lambda tup: tup + (1,))
- | "SumTuple" >> beam.CombineGlobally(beam.combiners.TupleCombineFn(
- *([sum] * (len(metric_keys) + 1))))
- | "AverageAndMakeDict" >> beam.Map(
+ pcoll |
+ "ApplyMetricFnPerInstance" >> beam.Map(metric_fn) |
+ "PairWith1" >> beam.Map(lambda tup: tup + (1,)) |
+ "SumTuple" >> beam.CombineGlobally(beam.combiners.TupleCombineFn(
+ *([sum] * (len(metric_keys) + 1)))) |
+ "AverageAndMakeDict" >> beam.Map(
lambda tup: dict(
- [(name, tup[i]/tup[-1]) for i, name in enumerate(metric_keys)] +
+ [(name, tup[i] / tup[-1]) for i, name in enumerate(metric_keys)] +
[("count", tup[-1])])))
diff --git a/airflow/contrib/operators/mongo_to_s3.py b/airflow/contrib/operators/mongo_to_s3.py
index 43b5d8b6c3..8bfa7a52f8 100644
--- a/airflow/contrib/operators/mongo_to_s3.py
+++ b/airflow/contrib/operators/mongo_to_s3.py
@@ -96,7 +96,8 @@ def execute(self, context):
return True
- def _stringify(self, iterable, joinable='\n'):
+ @staticmethod
+ def _stringify(iterable, joinable='\n'):
"""
Takes an iterable (pymongo Cursor or Array) containing dictionaries and
returns a stringified version using python join
@@ -105,7 +106,8 @@ def _stringify(self, iterable, joinable='\n'):
[json.dumps(doc, default=json_util.default) for doc in iterable]
)
- def transform(self, docs):
+ @staticmethod
+ def transform(docs):
"""
Processes pyMongo cursor and returns an iterable with each element being
a JSON serializable dictionary
diff --git a/airflow/contrib/operators/mysql_to_gcs.py b/airflow/contrib/operators/mysql_to_gcs.py
index 4d1bb7b329..d9fc932ff5 100644
--- a/airflow/contrib/operators/mysql_to_gcs.py
+++ b/airflow/contrib/operators/mysql_to_gcs.py
@@ -58,18 +58,18 @@ def __init__(self,
**kwargs):
"""
:param sql: The SQL to execute on the MySQL table.
- :type sql: string
+ :type sql: str
:param bucket: The bucket to upload to.
- :type bucket: string
+ :type bucket: str
:param filename: The filename to use as the object name when uploading
to Google cloud storage. A {} should be specified in the filename
to allow the operator to inject file numbers in cases where the
file is split due to size.
- :type filename: string
+ :type filename: str
:param schema_filename: If set, the filename to use as the object name
when uploading a .json file containing the BigQuery schema fields
for the table that was dumped from MySQL.
- :type schema_filename: string
+ :type schema_filename: str
:param approx_max_file_size_bytes: This operator supports the ability
to split large table dumps into multiple files (see notes in the
filenamed param docs above). Google cloud storage allows for files
@@ -77,10 +77,10 @@ def __init__(self,
file size of the splits.
:type approx_max_file_size_bytes: long
:param mysql_conn_id: Reference to a specific MySQL hook.
- :type mysql_conn_id: string
+ :type mysql_conn_id: str
:param google_cloud_storage_conn_id: Reference to a specific Google
cloud storage hook.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param schema: The schema to use, if any. Should be a list of dict or
a str. Pass a string if using Jinja template, otherwise, pass a list of
dict. Examples could be seen: https://cloud.google.com/bigquery/docs
@@ -218,7 +218,8 @@ def _upload_to_gcs(self, files_to_upload):
for object, tmp_file_handle in files_to_upload.items():
hook.upload(self.bucket, object, tmp_file_handle.name, 'application/json')
- def _convert_types(self, schema, col_type_dict, row):
+ @staticmethod
+ def _convert_types(schema, col_type_dict, row):
"""
Takes a value from MySQLdb, and converts it to a value that's safe for
JSON/Google cloud storage/BigQuery. Dates are converted to UTC seconds.
diff --git a/airflow/contrib/operators/oracle_to_azure_data_lake_transfer.py b/airflow/contrib/operators/oracle_to_azure_data_lake_transfer.py
new file mode 100644
index 0000000000..80cec8f462
--- /dev/null
+++ b/airflow/contrib/operators/oracle_to_azure_data_lake_transfer.py
@@ -0,0 +1,115 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.hooks.oracle_hook import OracleHook
+from airflow.contrib.hooks.azure_data_lake_hook import AzureDataLakeHook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.utils.file import TemporaryDirectory
+
+import unicodecsv as csv
+import os
+
+
+class OracleToAzureDataLakeTransfer(BaseOperator):
+ """
+ Moves data from Oracle to Azure Data Lake. The operator runs the query against
+ Oracle and stores the file locally before loading it into Azure Data Lake.
+
+
+ :param filename: file name to be used by the csv file.
+ :type filename: str
+ :param azure_data_lake_conn_id: destination azure data lake connection.
+ :type azure_data_lake_conn_id: str
+ :param azure_data_lake_path: destination path in azure data lake to put the file.
+ :type azure_data_lake_path: str
+ :param oracle_conn_id: source Oracle connection.
+ :type oracle_conn_id: str
+ :param sql: SQL query to execute against the Oracle database. (templated)
+ :type sql: str
+ :param sql_params: Parameters to use in sql query. (templated)
+ :type sql_params: str
+ :param delimiter: field delimiter in the file.
+ :type delimiter: str
+ :param encoding: enconding type for the file.
+ :type encoding: str
+ :param quotechar: Character to use in quoting.
+ :type quotechar: str
+ :param quoting: Quoting strategy. See unicodecsv quoting for more information.
+ :type quoting: str
+ """
+
+ template_fields = ('filename', 'sql', 'sql_params')
+ ui_color = '#e08c8c'
+
+ @apply_defaults
+ def __init__(
+ self,
+ filename,
+ azure_data_lake_conn_id,
+ azure_data_lake_path,
+ oracle_conn_id,
+ sql,
+ sql_params=None,
+ delimiter=",",
+ encoding="utf-8",
+ quotechar='"',
+ quoting=csv.QUOTE_MINIMAL,
+ *args, **kwargs):
+ super(OracleToAzureDataLakeTransfer, self).__init__(*args, **kwargs)
+ if sql_params is None:
+ sql_params = {}
+ self.filename = filename
+ self.oracle_conn_id = oracle_conn_id
+ self.sql = sql
+ self.sql_params = sql_params
+ self.azure_data_lake_conn_id = azure_data_lake_conn_id
+ self.azure_data_lake_path = azure_data_lake_path
+ self.delimiter = delimiter
+ self.encoding = encoding
+ self.quotechar = quotechar
+ self.quoting = quoting
+
+ def _write_temp_file(self, cursor, path_to_save):
+ with open(path_to_save, 'wb') as csvfile:
+ csv_writer = csv.writer(csvfile, delimiter=self.delimiter,
+ encoding=self.encoding, quotechar=self.quotechar,
+ quoting=self.quoting)
+ csv_writer.writerow(map(lambda field: field[0], cursor.description))
+ csv_writer.writerows(cursor)
+ csvfile.flush()
+
+ def execute(self, context):
+ oracle_hook = OracleHook(oracle_conn_id=self.oracle_conn_id)
+ azure_data_lake_hook = AzureDataLakeHook(
+ azure_data_lake_conn_id=self.azure_data_lake_conn_id)
+
+ self.log.info("Dumping Oracle query results to local file")
+ conn = oracle_hook.get_conn()
+ cursor = conn.cursor()
+ cursor.execute(self.sql, self.sql_params)
+
+ with TemporaryDirectory(prefix='airflow_oracle_to_azure_op_') as temp:
+ self._write_temp_file(cursor, os.path.join(temp, self.filename))
+ self.log.info("Uploading local file to Azure Data Lake")
+ azure_data_lake_hook.upload_file(os.path.join(temp, self.filename),
+ os.path.join(self.azure_data_lake_path,
+ self.filename))
+ cursor.close()
+ conn.close()
diff --git a/airflow/contrib/operators/oracle_to_oracle_transfer.py b/airflow/contrib/operators/oracle_to_oracle_transfer.py
new file mode 100644
index 0000000000..1db95f7520
--- /dev/null
+++ b/airflow/contrib/operators/oracle_to_oracle_transfer.py
@@ -0,0 +1,92 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.hooks.oracle_hook import OracleHook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class OracleToOracleTransfer(BaseOperator):
+ """
+ Moves data from Oracle to Oracle.
+
+
+ :param oracle_destination_conn_id: destination Oracle connection.
+ :type oracle_destination_conn_id: str
+ :param destination_table: destination table to insert rows.
+ :type destination_table: str
+ :param oracle_source_conn_id: source Oracle connection.
+ :type oracle_source_conn_id: str
+ :param source_sql: SQL query to execute against the source Oracle
+ database. (templated)
+ :type source_sql: str
+ :param source_sql_params: Parameters to use in sql query. (templated)
+ :type source_sql_params: dict
+ :param rows_chunk: number of rows per chunk to commit.
+ :type rows_chunk: int
+ """
+
+ template_fields = ('source_sql', 'source_sql_params')
+ ui_color = '#e08c8c'
+
+ @apply_defaults
+ def __init__(
+ self,
+ oracle_destination_conn_id,
+ destination_table,
+ oracle_source_conn_id,
+ source_sql,
+ source_sql_params=None,
+ rows_chunk=5000,
+ *args, **kwargs):
+ super(OracleToOracleTransfer, self).__init__(*args, **kwargs)
+ if source_sql_params is None:
+ source_sql_params = {}
+ self.oracle_destination_conn_id = oracle_destination_conn_id
+ self.destination_table = destination_table
+ self.oracle_source_conn_id = oracle_source_conn_id
+ self.source_sql = source_sql
+ self.source_sql_params = source_sql_params
+ self.rows_chunk = rows_chunk
+
+ def _execute(self, src_hook, dest_hook, context):
+ with src_hook.get_conn() as src_conn:
+ cursor = src_conn.cursor()
+ self.log.info("Querying data from source: {0}".format(
+ self.oracle_source_conn_id))
+ cursor.execute(self.source_sql, self.source_sql_params)
+ target_fields = list(map(lambda field: field[0], cursor.description))
+
+ rows_total = 0
+ rows = cursor.fetchmany(self.rows_chunk)
+ while len(rows) > 0:
+ rows_total = rows_total + len(rows)
+ dest_hook.bulk_insert_rows(self.destination_table, rows,
+ target_fields=target_fields,
+ commit_every=self.rows_chunk)
+ rows = cursor.fetchmany(self.rows_chunk)
+ self.log.info("Total inserted: {0} rows".format(rows_total))
+
+ self.log.info("Finished data transfer.")
+ cursor.close()
+
+ def execute(self, context):
+ src_hook = OracleHook(oracle_conn_id=self.oracle_source_conn_id)
+ dest_hook = OracleHook(oracle_conn_id=self.oracle_destination_conn_id)
+ self._execute(src_hook, dest_hook, context)
diff --git a/airflow/contrib/operators/postgres_to_gcs_operator.py b/airflow/contrib/operators/postgres_to_gcs_operator.py
index 88b4d00e39..78da78ee2f 100644
--- a/airflow/contrib/operators/postgres_to_gcs_operator.py
+++ b/airflow/contrib/operators/postgres_to_gcs_operator.py
@@ -56,18 +56,18 @@ def __init__(self,
**kwargs):
"""
:param sql: The SQL to execute on the Postgres table.
- :type sql: string
+ :type sql: str
:param bucket: The bucket to upload to.
- :type bucket: string
+ :type bucket: str
:param filename: The filename to use as the object name when uploading
to Google Cloud Storage. A {} should be specified in the filename
to allow the operator to inject file numbers in cases where the
file is split due to size.
- :type filename: string
+ :type filename: str
:param schema_filename: If set, the filename to use as the object name
when uploading a .json file containing the BigQuery schema fields
for the table that was dumped from Postgres.
- :type schema_filename: string
+ :type schema_filename: str
:param approx_max_file_size_bytes: This operator supports the ability
to split large table dumps into multiple files (see notes in the
filenamed param docs above). Google Cloud Storage allows for files
@@ -75,10 +75,10 @@ def __init__(self,
file size of the splits.
:type approx_max_file_size_bytes: long
:param postgres_conn_id: Reference to a specific Postgres hook.
- :type postgres_conn_id: string
+ :type postgres_conn_id: str
:param google_cloud_storage_conn_id: Reference to a specific Google
cloud storage hook.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any. For this to
work, the service account making the request must have domain-wide
delegation enabled.
@@ -133,28 +133,38 @@ def _write_local_data_files(self, cursor):
contain the data for the GCS objects.
"""
schema = list(map(lambda schema_tuple: schema_tuple[0], cursor.description))
- file_no = 0
- tmp_file_handle = NamedTemporaryFile(delete=True)
- tmp_file_handles = {self.filename.format(file_no): tmp_file_handle}
-
- for row in cursor:
- # Convert datetime objects to utc seconds, and decimals to floats
- row = map(self.convert_types, row)
- row_dict = dict(zip(schema, row))
-
- s = json.dumps(row_dict, sort_keys=True)
- if PY3:
- s = s.encode('utf-8')
- tmp_file_handle.write(s)
-
- # Append newline to make dumps BigQuery compatible.
- tmp_file_handle.write(b'\n')
-
- # Stop if the file exceeds the file size limit.
- if tmp_file_handle.tell() >= self.approx_max_file_size_bytes:
- file_no += 1
- tmp_file_handle = NamedTemporaryFile(delete=True)
- tmp_file_handles[self.filename.format(file_no)] = tmp_file_handle
+ tmp_file_handles = {}
+ row_no = 0
+
+ def _create_new_file():
+ handle = NamedTemporaryFile(delete=True)
+ filename = self.filename.format(len(tmp_file_handles))
+ tmp_file_handles[filename] = handle
+ return handle
+
+ # Don't create a file if there is nothing to write
+ if cursor.rowcount > 0:
+ tmp_file_handle = _create_new_file()
+
+ for row in cursor:
+ # Convert datetime objects to utc seconds, and decimals to floats
+ row = map(self.convert_types, row)
+ row_dict = dict(zip(schema, row))
+
+ s = json.dumps(row_dict, sort_keys=True)
+ if PY3:
+ s = s.encode('utf-8')
+ tmp_file_handle.write(s)
+
+ # Append newline to make dumps BigQuery compatible.
+ tmp_file_handle.write(b'\n')
+
+ # Stop if the file exceeds the file size limit.
+ if tmp_file_handle.tell() >= self.approx_max_file_size_bytes:
+ tmp_file_handle = _create_new_file()
+ row_no += 1
+
+ self.log.info('Received %s rows over %s files', row_no, len(tmp_file_handles))
return tmp_file_handles
diff --git a/airflow/contrib/operators/pubsub_operator.py b/airflow/contrib/operators/pubsub_operator.py
index 2d55b19f85..e40828bf92 100644
--- a/airflow/contrib/operators/pubsub_operator.py
+++ b/airflow/contrib/operators/pubsub_operator.py
@@ -67,19 +67,19 @@ def __init__(
**kwargs):
"""
:param project: the GCP project ID where the topic will be created
- :type project: string
+ :type project: str
:param topic: the topic to create. Do not include the
full topic path. In other words, instead of
``projects/{project}/topics/{topic}``, provide only
``{topic}``. (templated)
- :type topic: string
+ :type topic: str
:param gcp_conn_id: The connection ID to use connecting to
Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request
must have domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
super(PubSubTopicCreateOperator, self).__init__(*args, **kwargs)
@@ -163,28 +163,28 @@ def __init__(
**kwargs):
"""
:param topic_project: the GCP project ID where the topic exists
- :type topic_project: string
+ :type topic_project: str
:param topic: the topic to create. Do not include the
full topic path. In other words, instead of
``projects/{project}/topics/{topic}``, provide only
``{topic}``. (templated)
- :type topic: string
+ :type topic: str
:param subscription: the Pub/Sub subscription name. If empty, a random
name will be generated using the uuid module
- :type subscription: string
+ :type subscription: str
:param subscription_project: the GCP project ID where the subscription
will be created. If empty, ``topic_project`` will be used.
- :type subscription_project: string
+ :type subscription_project: str
:param ack_deadline_secs: Number of seconds that a subscriber has to
acknowledge each message pulled from the subscription
:type ack_deadline_secs: int
:param gcp_conn_id: The connection ID to use connecting to
Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request
must have domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
super(PubSubSubscriptionCreateOperator, self).__init__(*args, **kwargs)
@@ -248,22 +248,22 @@ def __init__(
**kwargs):
"""
:param project: the GCP project ID in which to work (templated)
- :type project: string
+ :type project: str
:param topic: the topic to delete. Do not include the
full topic path. In other words, instead of
``projects/{project}/topics/{topic}``, provide only
``{topic}``. (templated)
- :type topic: string
+ :type topic: str
:param fail_if_not_exists: If True and the topic does not exist, fail
the task
:type fail_if_not_exists: bool
:param gcp_conn_id: The connection ID to use connecting to
Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request
must have domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
super(PubSubTopicDeleteOperator, self).__init__(*args, **kwargs)
@@ -324,22 +324,22 @@ def __init__(
**kwargs):
"""
:param project: the GCP project ID in which to work (templated)
- :type project: string
+ :type project: str
:param subscription: the subscription to delete. Do not include the
full subscription path. In other words, instead of
``projects/{project}/subscription/{subscription}``, provide only
``{subscription}``. (templated)
- :type subscription: string
+ :type subscription: str
:param fail_if_not_exists: If True and the subscription does not exist,
fail the task
:type fail_if_not_exists: bool
:param gcp_conn_id: The connection ID to use connecting to
Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request
must have domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
super(PubSubSubscriptionDeleteOperator, self).__init__(*args, **kwargs)
@@ -396,12 +396,12 @@ def __init__(
**kwargs):
"""
:param project: the GCP project ID in which to work (templated)
- :type project: string
+ :type project: str
:param topic: the topic to which to publish. Do not include the
full topic path. In other words, instead of
``projects/{project}/topics/{topic}``, provide only
``{topic}``. (templated)
- :type topic: string
+ :type topic: str
:param messages: a list of messages to be published to the
topic. Each message is a dict with one or more of the
following keys-value mappings:
@@ -413,11 +413,11 @@ def __init__(
:type messages: list
:param gcp_conn_id: The connection ID to use connecting to
Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request
must have domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
super(PubSubPublishOperator, self).__init__(*args, **kwargs)
diff --git a/airflow/contrib/operators/qubole_check_operator.py b/airflow/contrib/operators/qubole_check_operator.py
index 0e8d75e167..8b6b5d351c 100644
--- a/airflow/contrib/operators/qubole_check_operator.py
+++ b/airflow/contrib/operators/qubole_check_operator.py
@@ -28,7 +28,7 @@ class QuboleCheckOperator(CheckOperator, QuboleOperator):
"""
Performs checks against Qubole Commands. ``QuboleCheckOperator`` expects
a command that will be executed on QDS.
- By default, each value on first row of the result of this Qubole Commmand
+ By default, each value on first row of the result of this Qubole Command
is evaluated using python ``bool`` casting. If any of the
values return ``False``, the check is failed and errors out.
@@ -215,11 +215,11 @@ def get_sql_from_qbol_cmd(params):
def handle_airflow_exception(airflow_exception, hook):
cmd = hook.cmd
if cmd is not None:
- if cmd.is_success:
+ if cmd.is_success(cmd.status):
qubole_command_results = hook.get_query_results()
qubole_command_id = cmd.id
exception_message = '\nQubole Command Id: {qubole_command_id}' \
'\nQubole Command Results:' \
'\n{qubole_command_results}'.format(**locals())
raise AirflowException(str(airflow_exception) + exception_message)
- raise AirflowException(airflow_exception.message)
+ raise AirflowException(str(airflow_exception))
diff --git a/airflow/contrib/operators/s3_copy_object_operator.py b/airflow/contrib/operators/s3_copy_object_operator.py
new file mode 100644
index 0000000000..330138ed27
--- /dev/null
+++ b/airflow/contrib/operators/s3_copy_object_operator.py
@@ -0,0 +1,93 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.hooks.S3_hook import S3Hook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class S3CopyObjectOperator(BaseOperator):
+ """
+ Creates a copy of an object that is already stored in S3.
+
+ Note: the S3 connection used here needs to have access to both
+ source and destination bucket/key.
+
+ :param source_bucket_key: The key of the source object.
+
+ It can be either full s3:// style url or relative path from root level.
+
+ When it's specified as a full s3:// url, please omit source_bucket_name.
+ :type source_bucket_key: str
+ :param dest_bucket_key: The key of the object to copy to.
+
+ The convention to specify `dest_bucket_key` is the same as `source_bucket_key`.
+ :type dest_bucket_key: str
+ :param source_bucket_name: Name of the S3 bucket where the source object is in.
+
+ It should be omitted when `source_bucket_key` is provided as a full s3:// url.
+ :type source_bucket_name: str
+ :param dest_bucket_name: Name of the S3 bucket to where the object is copied.
+
+ It should be omitted when `dest_bucket_key` is provided as a full s3:// url.
+ :type dest_bucket_name: str
+ :param source_version_id: Version ID of the source object (OPTIONAL)
+ :type source_version_id: str
+ :param aws_conn_id: Connection id of the S3 connection to use
+ :type aws_conn_id: str
+ :param verify: Whether or not to verify SSL certificates for S3 connection.
+ By default SSL certificates are verified.
+
+ You can provide the following values:
+
+ - False: do not validate SSL certificates. SSL will still be used,
+ but SSL certificates will not be
+ verified.
+ - path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.
+ You can specify this argument if you want to use a different
+ CA cert bundle than the one used by botocore.
+ :type verify: bool or str
+ """
+
+ @apply_defaults
+ def __init__(
+ self,
+ source_bucket_key,
+ dest_bucket_key,
+ source_bucket_name=None,
+ dest_bucket_name=None,
+ source_version_id=None,
+ aws_conn_id='aws_default',
+ verify=None,
+ *args, **kwargs):
+ super(S3CopyObjectOperator, self).__init__(*args, **kwargs)
+
+ self.source_bucket_key = source_bucket_key
+ self.dest_bucket_key = dest_bucket_key
+ self.source_bucket_name = source_bucket_name
+ self.dest_bucket_name = dest_bucket_name
+ self.source_version_id = source_version_id
+ self.aws_conn_id = aws_conn_id
+ self.verify = verify
+
+ def execute(self, context):
+ s3_hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
+ s3_hook.copy_object(self.source_bucket_key, self.dest_bucket_key,
+ self.source_bucket_name, self.dest_bucket_name,
+ self.source_version_id)
diff --git a/airflow/contrib/operators/s3_delete_objects_operator.py b/airflow/contrib/operators/s3_delete_objects_operator.py
new file mode 100644
index 0000000000..1aa1b3901e
--- /dev/null
+++ b/airflow/contrib/operators/s3_delete_objects_operator.py
@@ -0,0 +1,85 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.S3_hook import S3Hook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class S3DeleteObjectsOperator(BaseOperator):
+ """
+ To enable users to delete single object or multiple objects from
+ a bucket using a single HTTP request.
+
+ Users may specify up to 1000 keys to delete.
+
+ :param bucket: Name of the bucket in which you are going to delete object(s)
+ :type bucket: str
+ :param keys: The key(s) to delete from S3 bucket.
+
+ When ``keys`` is a string, it's supposed to be the key name of
+ the single object to delete.
+
+ When ``keys`` is a list, it's supposed to be the list of the
+ keys to delete.
+
+ You may specify up to 1000 keys.
+ :type keys: str or list
+ :param aws_conn_id: Connection id of the S3 connection to use
+ :type aws_conn_id: str
+ :param verify: Whether or not to verify SSL certificates for S3 connection.
+ By default SSL certificates are verified.
+
+ You can provide the following values:
+
+ - False: do not validate SSL certificates. SSL will still be used,
+ but SSL certificates will not be
+ verified.
+ - path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.
+ You can specify this argument if you want to use a different
+ CA cert bundle than the one used by botocore.
+ :type verify: bool or str
+ """
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket,
+ keys,
+ aws_conn_id='aws_default',
+ verify=None,
+ *args, **kwargs):
+ super(S3DeleteObjectsOperator, self).__init__(*args, **kwargs)
+ self.bucket = bucket
+ self.keys = keys
+ self.aws_conn_id = aws_conn_id
+ self.verify = verify
+
+ def execute(self, context):
+ s3_hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
+
+ response = s3_hook.delete_objects(bucket=self.bucket, keys=self.keys)
+
+ deleted_keys = [x['Key'] for x in response.get("Deleted", [])]
+ self.log.info("Deleted: {}".format(deleted_keys))
+
+ if "Errors" in response:
+ errors_keys = [x['Key'] for x in response.get("Errors", [])]
+ raise AirflowException("Errors when deleting: {}".format(errors_keys))
diff --git a/airflow/contrib/operators/s3_list_operator.py b/airflow/contrib/operators/s3_list_operator.py
index b85691b005..3ca22d5932 100644
--- a/airflow/contrib/operators/s3_list_operator.py
+++ b/airflow/contrib/operators/s3_list_operator.py
@@ -30,14 +30,24 @@ class S3ListOperator(BaseOperator):
used by `xcom` in the downstream task.
:param bucket: The S3 bucket where to find the objects. (templated)
- :type bucket: string
+ :type bucket: str
:param prefix: Prefix string to filters the objects whose name begin with
such prefix. (templated)
- :type prefix: string
+ :type prefix: str
:param delimiter: the delimiter marks key hierarchy. (templated)
- :type delimiter: string
+ :type delimiter: str
:param aws_conn_id: The connection ID to use when connecting to S3 storage.
- :type aws_conn_id: string
+ :type aws_conn_id: str
+ :parame verify: Whether or not to verify SSL certificates for S3 connection.
+ By default SSL certificates are verified.
+ You can provide the following values:
+ - False: do not validate SSL certificates. SSL will still be used
+ (unless use_ssl is False), but SSL certificates will not be
+ verified.
+ - path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.
+ You can specify this argument if you want to use a different
+ CA cert bundle than the one used by botocore.
+ :type verify: bool or str
**Example**:
The following operator would list all the files
@@ -61,6 +71,7 @@ def __init__(self,
prefix='',
delimiter='',
aws_conn_id='aws_default',
+ verify=None,
*args,
**kwargs):
super(S3ListOperator, self).__init__(*args, **kwargs)
@@ -68,9 +79,10 @@ def __init__(self,
self.prefix = prefix
self.delimiter = delimiter
self.aws_conn_id = aws_conn_id
+ self.verify = verify
def execute(self, context):
- hook = S3Hook(aws_conn_id=self.aws_conn_id)
+ hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
self.log.info(
'Getting the list of files from bucket: {0} in prefix: {1} (Delimiter {2})'.
diff --git a/airflow/contrib/operators/s3_to_gcs_operator.py b/airflow/contrib/operators/s3_to_gcs_operator.py
index 2898af1071..5dd355a6fd 100644
--- a/airflow/contrib/operators/s3_to_gcs_operator.py
+++ b/airflow/contrib/operators/s3_to_gcs_operator.py
@@ -33,31 +33,43 @@ class S3ToGoogleCloudStorageOperator(S3ListOperator):
destination path.
:param bucket: The S3 bucket where to find the objects. (templated)
- :type bucket: string
+ :type bucket: str
:param prefix: Prefix string which filters objects whose name begin with
such prefix. (templated)
- :type prefix: string
+ :type prefix: str
:param delimiter: the delimiter marks key hierarchy. (templated)
- :type delimiter: string
+ :type delimiter: str
:param aws_conn_id: The source S3 connection
- :type aws_conn_id: string
+ :type aws_conn_id: str
+ :parame verify: Whether or not to verify SSL certificates for S3 connection.
+ By default SSL certificates are verified.
+ You can provide the following values:
+ - False: do not validate SSL certificates. SSL will still be used
+ (unless use_ssl is False), but SSL certificates will not be
+ verified.
+ - path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.
+ You can specify this argument if you want to use a different
+ CA cert bundle than the one used by botocore.
+ :type verify: bool or str
:param dest_gcs_conn_id: The destination connection ID to use
when connecting to Google Cloud Storage.
- :type dest_gcs_conn_id: string
+ :type dest_gcs_conn_id: str
:param dest_gcs: The destination Google Cloud Storage bucket and prefix
where you want to store the files. (templated)
- :type dest_gcs: string
+ :type dest_gcs: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
:param replace: Whether you want to replace existing destination files
or not.
:type replace: bool
**Example**:
+
.. code-block:: python
+
s3_to_gcs_op = S3ToGoogleCloudStorageOperator(
task_id='s3_to_gcs_example',
bucket='my-s3-bucket',
@@ -80,6 +92,7 @@ def __init__(self,
prefix='',
delimiter='',
aws_conn_id='aws_default',
+ verify=None,
dest_gcs_conn_id=None,
dest_gcs=None,
delegate_to=None,
@@ -98,6 +111,7 @@ def __init__(self,
self.dest_gcs = dest_gcs
self.delegate_to = delegate_to
self.replace = replace
+ self.verify = verify
if dest_gcs and not self._gcs_object_is_directory(self.dest_gcs):
self.log.info(
@@ -146,7 +160,7 @@ def execute(self, context):
'There are no new files to sync. Have a nice day!')
if files:
- hook = S3Hook(aws_conn_id=self.aws_conn_id)
+ hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
for file in files:
# GCS hook builds its own in-memory file so we have to create
@@ -184,7 +198,8 @@ def execute(self, context):
# Following functionality may be better suited in
# airflow/contrib/hooks/gcs_hook.py
- def _gcs_object_is_directory(self, object):
+ @staticmethod
+ def _gcs_object_is_directory(object):
bucket, blob = _parse_gcs_url(object)
return len(blob) == 0 or blob.endswith('/')
diff --git a/airflow/contrib/operators/sagemaker_create_training_job_operator.py b/airflow/contrib/operators/sagemaker_create_training_job_operator.py
new file mode 100644
index 0000000000..2792208679
--- /dev/null
+++ b/airflow/contrib/operators/sagemaker_create_training_job_operator.py
@@ -0,0 +1,119 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.sagemaker_hook import SageMakerHook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.exceptions import AirflowException
+
+
+class SageMakerCreateTrainingJobOperator(BaseOperator):
+
+ """
+ Initiate a SageMaker training
+
+ This operator returns The ARN of the model created in Amazon SageMaker
+
+ :param training_job_config:
+ The configuration necessary to start a training job (templated)
+ :type training_job_config: dict
+ :param region_name: The AWS region_name
+ :type region_name: str
+ :param sagemaker_conn_id: The SageMaker connection ID to use.
+ :type sagemaker_conn_id: str
+ :param use_db_config: Whether or not to use db config
+ associated with sagemaker_conn_id.
+ If set to true, will automatically update the training config
+ with what's in db, so the db config doesn't need to
+ included everything, but what's there does replace the ones
+ in the training_job_config, so be careful
+ :type use_db_config: bool
+ :param aws_conn_id: The AWS connection ID to use.
+ :type aws_conn_id: str
+ :param wait_for_completion: if the operator should block
+ until training job finishes
+ :type wait_for_completion: bool
+ :param check_interval: if wait is set to be true, this is the time interval
+ in seconds which the operator will check the status of the training job
+ :type check_interval: int
+ :param max_ingestion_time: if wait is set to be true, the operator will fail
+ if the training job hasn't finish within the max_ingestion_time
+ (Caution: be careful to set this parameters because training can take very long)
+ :type max_ingestion_time: int
+
+ **Example**:
+ The following operator would start a training job when executed
+
+ sagemaker_training =
+ SageMakerCreateTrainingJobOperator(
+ task_id='sagemaker_training',
+ training_job_config=config,
+ region_name='us-west-2'
+ sagemaker_conn_id='sagemaker_customers_conn',
+ use_db_config=True,
+ aws_conn_id='aws_customers_conn'
+ )
+ """
+
+ template_fields = ['training_job_config']
+ template_ext = ()
+ ui_color = '#ededed'
+
+ @apply_defaults
+ def __init__(self,
+ training_job_config=None,
+ region_name=None,
+ sagemaker_conn_id=None,
+ use_db_config=False,
+ wait_for_completion=True,
+ check_interval=5,
+ max_ingestion_time=None,
+ *args, **kwargs):
+ super(SageMakerCreateTrainingJobOperator, self).__init__(*args, **kwargs)
+
+ self.sagemaker_conn_id = sagemaker_conn_id
+ self.training_job_config = training_job_config
+ self.use_db_config = use_db_config
+ self.region_name = region_name
+ self.wait_for_completion = wait_for_completion
+ self.check_interval = check_interval
+ self.max_ingestion_time = max_ingestion_time
+
+ def execute(self, context):
+ sagemaker = SageMakerHook(
+ sagemaker_conn_id=self.sagemaker_conn_id,
+ use_db_config=self.use_db_config,
+ region_name=self.region_name,
+ check_interval=self.check_interval,
+ max_ingestion_time=self.max_ingestion_time
+ )
+
+ self.log.info(
+ "Creating SageMaker Training Job %s."
+ % self.training_job_config['TrainingJobName']
+ )
+ response = sagemaker.create_training_job(
+ self.training_job_config,
+ wait_for_completion=self.wait_for_completion)
+ if not response['ResponseMetadata']['HTTPStatusCode'] \
+ == 200:
+ raise AirflowException(
+ 'Sagemaker Training Job creation failed: %s' % response)
+ else:
+ return response
diff --git a/airflow/contrib/operators/sagemaker_create_transform_job_operator.py b/airflow/contrib/operators/sagemaker_create_transform_job_operator.py
new file mode 100644
index 0000000000..22c8c2b4ba
--- /dev/null
+++ b/airflow/contrib/operators/sagemaker_create_transform_job_operator.py
@@ -0,0 +1,132 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.sagemaker_hook import SageMakerHook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.exceptions import AirflowException
+
+
+class SageMakerCreateTransformJobOperator(BaseOperator):
+ """
+ Initiate a SageMaker transform
+
+ This operator returns The ARN of the model created in Amazon SageMaker
+
+ :param sagemaker_conn_id: The SageMaker connection ID to use.
+ :type sagemaker_conn_id: string
+ :param transform_job_config:
+ The configuration necessary to start a transform job (templated)
+ :type transform_job_config: dict
+ :param model_config:
+ The configuration necessary to create a model, the default is none
+ which means that user should provide a created model in transform_job_config
+ If given, will be used to create a model before creating transform job
+ :type model_config: dict
+ :param use_db_config: Whether or not to use db config
+ associated with sagemaker_conn_id.
+ If set to true, will automatically update the transform config
+ with what's in db, so the db config doesn't need to
+ included everything, but what's there does replace the ones
+ in the transform_job_config, so be careful
+ :type use_db_config: bool
+ :param region_name: The AWS region_name
+ :type region_name: string
+ :param wait_for_completion: if the program should keep running until job finishes
+ :type wait_for_completion: bool
+ :param check_interval: if wait is set to be true, this is the time interval
+ in seconds which the operator will check the status of the transform job
+ :type check_interval: int
+ :param max_ingestion_time: if wait is set to be true, the operator will fail
+ if the transform job hasn't finish within the max_ingestion_time
+ (Caution: be careful to set this parameters because transform can take very long)
+ :type max_ingestion_time: int
+ :param aws_conn_id: The AWS connection ID to use.
+ :type aws_conn_id: string
+
+ **Example**:
+ The following operator would start a transform job when executed
+
+ sagemaker_transform =
+ SageMakerCreateTransformJobOperator(
+ task_id='sagemaker_transform',
+ transform_job_config=config_transform,
+ model_config=config_model,
+ region_name='us-west-2'
+ sagemaker_conn_id='sagemaker_customers_conn',
+ use_db_config=True,
+ aws_conn_id='aws_customers_conn'
+ )
+ """
+
+ template_fields = ['transform_job_config']
+ template_ext = ()
+ ui_color = '#ededed'
+
+ @apply_defaults
+ def __init__(self,
+ sagemaker_conn_id=None,
+ transform_job_config=None,
+ model_config=None,
+ use_db_config=False,
+ region_name=None,
+ wait_for_completion=True,
+ check_interval=2,
+ max_ingestion_time=None,
+ *args, **kwargs):
+ super(SageMakerCreateTransformJobOperator, self).__init__(*args, **kwargs)
+
+ self.sagemaker_conn_id = sagemaker_conn_id
+ self.transform_job_config = transform_job_config
+ self.model_config = model_config
+ self.use_db_config = use_db_config
+ self.region_name = region_name
+ self.wait_for_completion = wait_for_completion
+ self.check_interval = check_interval
+ self.max_ingestion_time = max_ingestion_time
+
+ def execute(self, context):
+ sagemaker = SageMakerHook(
+ sagemaker_conn_id=self.sagemaker_conn_id,
+ use_db_config=self.use_db_config,
+ region_name=self.region_name,
+ check_interval=self.check_interval,
+ max_ingestion_time=self.max_ingestion_time
+ )
+
+ if self.model_config:
+ self.log.info(
+ "Creating SageMaker Model %s for transform job"
+ % self.model_config['ModelName']
+ )
+ sagemaker.create_model(self.model_config)
+
+ self.log.info(
+ "Creating SageMaker transform Job %s."
+ % self.transform_job_config['TransformJobName']
+ )
+ response = sagemaker.create_transform_job(
+ self.transform_job_config,
+ wait_for_completion=self.wait_for_completion)
+ if not response['ResponseMetadata']['HTTPStatusCode'] \
+ == 200:
+ raise AirflowException(
+ 'Sagemaker transform Job creation failed: %s' % response)
+ else:
+ return response
diff --git a/airflow/contrib/operators/sagemaker_create_tuning_job_operator.py b/airflow/contrib/operators/sagemaker_create_tuning_job_operator.py
new file mode 100644
index 0000000000..d5f4396375
--- /dev/null
+++ b/airflow/contrib/operators/sagemaker_create_tuning_job_operator.py
@@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.sagemaker_hook import SageMakerHook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.exceptions import AirflowException
+
+
+class SageMakerCreateTuningJobOperator(BaseOperator):
+
+ """
+ Initiate a SageMaker HyperParameter Tuning Job
+
+ This operator returns The ARN of the model created in Amazon SageMaker
+
+ :param sagemaker_conn_id: The SageMaker connection ID to use.
+ :type sagemaker_conn_id: str
+ :param region_name: The AWS region_name
+ :type region_name: str
+ :param tuning_job_config:
+ The configuration necessary to start a tuning job (templated)
+ :type tuning_job_config: dict
+ :param use_db_config: Whether or not to use db config
+ associated with sagemaker_conn_id.
+ If set to true, will automatically update the tuning config
+ with what's in db, so the db config doesn't need to
+ included everything, but what's there does replace the ones
+ in the tuning_job_config, so be careful
+ :type use_db_config: bool
+ :param wait_for_completion: if the operator should block
+ until tuning job finishes
+ :type wait_for_completion: bool
+ :param check_interval: if wait is set to be true, this is the time interval
+ in seconds which the operator will check the status of the tuning job
+ :type check_interval: int
+ :param max_ingestion_time: if wait is set to be true, the operator will fail
+ if the tuning job hasn't finish within the max_ingestion_time
+ (Caution: be careful to set this parameters because tuning can take very long)
+ :type max_ingestion_time: int
+ :param aws_conn_id: The AWS connection ID to use.
+ :type aws_conn_id: str
+
+ **Example**:
+ The following operator would start a tuning job when executed
+
+ sagemaker_tuning =
+ SageMakerCreateTuningJobOperator(
+ task_id='sagemaker_tuning',
+ sagemaker_conn_id='sagemaker_customers_conn',
+ tuning_job_config=config,
+ check_interval=2,
+ max_ingestion_time=3600,
+ aws_conn_id='aws_customers_conn',
+ )
+ """
+
+ template_fields = ['tuning_job_config']
+ template_ext = ()
+ ui_color = '#ededed'
+
+ @apply_defaults
+ def __init__(self,
+ sagemaker_conn_id=None,
+ region_name=None,
+ tuning_job_config=None,
+ use_db_config=False,
+ wait_for_completion=True,
+ check_interval=5,
+ max_ingestion_time=None,
+ *args, **kwargs):
+ super(SageMakerCreateTuningJobOperator, self)\
+ .__init__(*args, **kwargs)
+
+ self.sagemaker_conn_id = sagemaker_conn_id
+ self.region_name = region_name
+ self.tuning_job_config = tuning_job_config
+ self.use_db_config = use_db_config
+ self.wait_for_completion = wait_for_completion
+ self.check_interval = check_interval
+ self.max_ingestion_time = max_ingestion_time
+
+ def execute(self, context):
+ sagemaker = SageMakerHook(sagemaker_conn_id=self.sagemaker_conn_id,
+ region_name=self.region_name,
+ use_db_config=self.use_db_config,
+ check_interval=self.check_interval,
+ max_ingestion_time=self.max_ingestion_time
+ )
+
+ self.log.info(
+ "Creating SageMaker Hyper Parameter Tunning Job %s"
+ % self.tuning_job_config['HyperParameterTuningJobName']
+ )
+
+ response = sagemaker.create_tuning_job(
+ self.tuning_job_config,
+ wait_for_completion=self.wait_for_completion
+ )
+ if not response['ResponseMetadata']['HTTPStatusCode'] \
+ == 200:
+ raise AirflowException(
+ "Sagemaker Tuning Job creation failed: %s" % response)
+ else:
+ return response
diff --git a/airflow/contrib/operators/segment_track_event_operator.py b/airflow/contrib/operators/segment_track_event_operator.py
index 04f6ae6d41..5d2f2db539 100644
--- a/airflow/contrib/operators/segment_track_event_operator.py
+++ b/airflow/contrib/operators/segment_track_event_operator.py
@@ -27,16 +27,16 @@ class SegmentTrackEventOperator(BaseOperator):
Send Track Event to Segment for a specified user_id and event
:param user_id: The ID for this user in your database. (templated)
- :type user_id: string
+ :type user_id: str
:param event: The name of the event you're tracking. (templated)
- :type event: string
+ :type event: str
:param properties: A dictionary of properties for the event. (templated)
:type properties: dict
:param segment_conn_id: The connection ID to use when connecting to Segment.
- :type segment_conn_id: string
+ :type segment_conn_id: str
:param segment_debug_mode: Determines whether Segment should run in debug mode.
Defaults to False
- :type segment_debug_mode: boolean
+ :type segment_debug_mode: bool
"""
template_fields = ('user_id', 'event', 'properties')
ui_color = '#ffd700'
diff --git a/airflow/contrib/operators/sftp_operator.py b/airflow/contrib/operators/sftp_operator.py
index 153f440d41..620d875f89 100644
--- a/airflow/contrib/operators/sftp_operator.py
+++ b/airflow/contrib/operators/sftp_operator.py
@@ -30,23 +30,29 @@ class SFTPOperation(object):
class SFTPOperator(BaseOperator):
"""
SFTPOperator for transferring files from remote host to local or vice a versa.
- This operator uses ssh_hook to open sftp trasport channel that serve as basis
+ This operator uses ssh_hook to open sftp transport channel that serve as basis
for file transfer.
- :param ssh_hook: predefined ssh_hook to use for remote execution
+ :param ssh_hook: predefined ssh_hook to use for remote execution.
+ Either `ssh_hook` or `ssh_conn_id` needs to be provided.
:type ssh_hook: :class:`SSHHook`
- :param ssh_conn_id: connection id from airflow Connections
+ :param ssh_conn_id: connection id from airflow Connections.
+ `ssh_conn_id` will be ingored if `ssh_hook` is provided.
:type ssh_conn_id: str
- :param remote_host: remote host to connect
+ :param remote_host: remote host to connect (templated)
+ Nullable. If provided, it will replace the `remote_host` which was
+ defined in `ssh_hook` or predefined in the connection of `ssh_conn_id`.
:type remote_host: str
:param local_filepath: local file path to get or put. (templated)
:type local_filepath: str
:param remote_filepath: remote file path to get or put. (templated)
:type remote_filepath: str
- :param operation: specify operation 'get' or 'put', defaults to get
+ :param operation: specify operation 'get' or 'put', defaults to put
:type get: bool
+ :param confirm: specify if the SFTP operation should be confirmed, defaults to True
+ :type confirm: bool
"""
- template_fields = ('local_filepath', 'remote_filepath')
+ template_fields = ('local_filepath', 'remote_filepath', 'remote_host')
@apply_defaults
def __init__(self,
@@ -56,6 +62,7 @@ def __init__(self,
local_filepath=None,
remote_filepath=None,
operation=SFTPOperation.PUT,
+ confirm=True,
*args,
**kwargs):
super(SFTPOperator, self).__init__(*args, **kwargs)
@@ -65,6 +72,7 @@ def __init__(self,
self.local_filepath = local_filepath
self.remote_filepath = remote_filepath
self.operation = operation
+ self.confirm = confirm
if not (self.operation.lower() == SFTPOperation.GET or
self.operation.lower() == SFTPOperation.PUT):
raise TypeError("unsupported operation value {0}, expected {1} or {2}"
@@ -73,27 +81,37 @@ def __init__(self,
def execute(self, context):
file_msg = None
try:
- if self.ssh_conn_id and not self.ssh_hook:
- self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
+ if self.ssh_conn_id:
+ if self.ssh_hook and isinstance(self.ssh_hook, SSHHook):
+ self.log.info("ssh_conn_id is ignored when ssh_hook is provided.")
+ else:
+ self.log.info("ssh_hook is not provided or invalid. " +
+ "Trying ssh_conn_id to create SSHHook.")
+ self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
if not self.ssh_hook:
- raise AirflowException("can not operate without ssh_hook or ssh_conn_id")
+ raise AirflowException("Cannot operate without ssh_hook or ssh_conn_id.")
if self.remote_host is not None:
+ self.log.info("remote_host is provided explicitly. " +
+ "It will replace the remote_host which was defined " +
+ "in ssh_hook or predefined in connection of ssh_conn_id.")
self.ssh_hook.remote_host = self.remote_host
- ssh_client = self.ssh_hook.get_conn()
- sftp_client = ssh_client.open_sftp()
- if self.operation.lower() == SFTPOperation.GET:
- file_msg = "from {0} to {1}".format(self.remote_filepath,
- self.local_filepath)
- self.log.debug("Starting to transfer %s", file_msg)
- sftp_client.get(self.remote_filepath, self.local_filepath)
- else:
- file_msg = "from {0} to {1}".format(self.local_filepath,
- self.remote_filepath)
- self.log.debug("Starting to transfer file %s", file_msg)
- sftp_client.put(self.local_filepath, self.remote_filepath)
+ with self.ssh_hook.get_conn() as ssh_client:
+ sftp_client = ssh_client.open_sftp()
+ if self.operation.lower() == SFTPOperation.GET:
+ file_msg = "from {0} to {1}".format(self.remote_filepath,
+ self.local_filepath)
+ self.log.debug("Starting to transfer %s", file_msg)
+ sftp_client.get(self.remote_filepath, self.local_filepath)
+ else:
+ file_msg = "from {0} to {1}".format(self.local_filepath,
+ self.remote_filepath)
+ self.log.debug("Starting to transfer file %s", file_msg)
+ sftp_client.put(self.local_filepath,
+ self.remote_filepath,
+ confirm=self.confirm)
except Exception as e:
raise AirflowException("Error while transferring {0}, error: {1}"
diff --git a/airflow/contrib/operators/slack_webhook_operator.py b/airflow/contrib/operators/slack_webhook_operator.py
index bab3e90856..64f2448a55 100644
--- a/airflow/contrib/operators/slack_webhook_operator.py
+++ b/airflow/contrib/operators/slack_webhook_operator.py
@@ -31,8 +31,8 @@ class SlackWebhookOperator(SimpleHttpOperator):
Each Slack webhook token can be pre-configured to use a specific channel, username and
icon. You can override these defaults in this hook.
- :param conn_id: connection that has Slack webhook token in the extra field
- :type conn_id: str
+ :param http_conn_id: connection that has Slack webhook token in the extra field
+ :type http_conn_id: str
:param webhook_token: Slack webhook token
:type webhook_token: str
:param message: The message you want to send on Slack
@@ -77,7 +77,7 @@ def __init__(self,
def execute(self, context):
"""
- Call the SparkSqlHook to run the provided sql query
+ Call the SlackWebhookHook to post the provided Slack message
"""
self.hook = SlackWebhookHook(
self.http_conn_id,
diff --git a/airflow/contrib/operators/snowflake_operator.py b/airflow/contrib/operators/snowflake_operator.py
index 39d7d496ea..6c3f403912 100644
--- a/airflow/contrib/operators/snowflake_operator.py
+++ b/airflow/contrib/operators/snowflake_operator.py
@@ -26,16 +26,16 @@ class SnowflakeOperator(BaseOperator):
Executes sql code in a Snowflake database
:param snowflake_conn_id: reference to specific snowflake connection id
- :type snowflake_conn_id: string
+ :type snowflake_conn_id: str
:param sql: the sql code to be executed. (templated)
:type sql: Can receive a str representing a sql statement,
a list of str (sql statements), or reference to a template file.
Template reference are recognized by str ending in '.sql'
:param warehouse: name of warehouse which overwrite defined
one in connection
- :type warehouse: string
+ :type warehouse: str
:param database: name of database which overwrite defined one in connection
- :type database: string
+ :type database: str
"""
template_fields = ('sql',)
diff --git a/airflow/contrib/operators/ssh_operator.py b/airflow/contrib/operators/ssh_operator.py
index d246800953..2bf342935d 100644
--- a/airflow/contrib/operators/ssh_operator.py
+++ b/airflow/contrib/operators/ssh_operator.py
@@ -31,11 +31,15 @@ class SSHOperator(BaseOperator):
"""
SSHOperator to execute commands on given remote host using the ssh_hook.
- :param ssh_hook: predefined ssh_hook to use for remote execution
+ :param ssh_hook: predefined ssh_hook to use for remote execution.
+ Either `ssh_hook` or `ssh_conn_id` needs to be provided.
:type ssh_hook: :class:`SSHHook`
- :param ssh_conn_id: connection id from airflow Connections
+ :param ssh_conn_id: connection id from airflow Connections.
+ `ssh_conn_id` will be ingored if `ssh_hook` is provided.
:type ssh_conn_id: str
- :param remote_host: remote host to connect
+ :param remote_host: remote host to connect (templated)
+ Nullable. If provided, it will replace the `remote_host` which was
+ defined in `ssh_hook` or predefined in the connection of `ssh_conn_id`.
:type remote_host: str
:param command: command to execute on remote host. (templated)
:type command: str
@@ -45,7 +49,7 @@ class SSHOperator(BaseOperator):
:type do_xcom_push: bool
"""
- template_fields = ('command',)
+ template_fields = ('command', 'remote_host')
template_ext = ('.sh',)
@apply_defaults
@@ -68,88 +72,96 @@ def __init__(self,
def execute(self, context):
try:
- if self.ssh_conn_id and not self.ssh_hook:
- self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id)
+ if self.ssh_conn_id:
+ if self.ssh_hook and isinstance(self.ssh_hook, SSHHook):
+ self.log.info("ssh_conn_id is ignored when ssh_hook is provided.")
+ else:
+ self.log.info("ssh_hook is not provided or invalid. " +
+ "Trying ssh_conn_id to create SSHHook.")
+ self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id,
+ timeout=self.timeout)
if not self.ssh_hook:
- raise AirflowException("can not operate without ssh_hook or ssh_conn_id")
+ raise AirflowException("Cannot operate without ssh_hook or ssh_conn_id.")
if self.remote_host is not None:
+ self.log.info("remote_host is provided explicitly. " +
+ "It will replace the remote_host which was defined " +
+ "in ssh_hook or predefined in connection of ssh_conn_id.")
self.ssh_hook.remote_host = self.remote_host
- ssh_client = self.ssh_hook.get_conn()
-
if not self.command:
- raise AirflowException("no command specified so nothing to execute here.")
-
- # Auto apply tty when its required in case of sudo
- get_pty = False
- if self.command.startswith('sudo'):
- get_pty = True
-
- # set timeout taken as params
- stdin, stdout, stderr = ssh_client.exec_command(command=self.command,
- get_pty=get_pty,
- timeout=self.timeout
- )
- # get channels
- channel = stdout.channel
-
- # closing stdin
- stdin.close()
- channel.shutdown_write()
-
- agg_stdout = b''
- agg_stderr = b''
-
- # capture any initial output in case channel is closed already
- stdout_buffer_length = len(stdout.channel.in_buffer)
-
- if stdout_buffer_length > 0:
- agg_stdout += stdout.channel.recv(stdout_buffer_length)
-
- # read from both stdout and stderr
- while not channel.closed or \
- channel.recv_ready() or \
- channel.recv_stderr_ready():
- readq, _, _ = select([channel], [], [], self.timeout)
- for c in readq:
- if c.recv_ready():
- line = stdout.channel.recv(len(c.in_buffer))
- line = line
- agg_stdout += line
- self.log.info(line.decode('utf-8').strip('\n'))
- if c.recv_stderr_ready():
- line = stderr.channel.recv_stderr(len(c.in_stderr_buffer))
- line = line
- agg_stderr += line
- self.log.warning(line.decode('utf-8').strip('\n'))
- if stdout.channel.exit_status_ready()\
- and not stderr.channel.recv_stderr_ready()\
- and not stdout.channel.recv_ready():
- stdout.channel.shutdown_read()
- stdout.channel.close()
- break
-
- stdout.close()
- stderr.close()
-
- exit_status = stdout.channel.recv_exit_status()
- if exit_status is 0:
- # returning output if do_xcom_push is set
- if self.do_xcom_push:
- enable_pickling = configuration.conf.getboolean(
- 'core', 'enable_xcom_pickling'
- )
- if enable_pickling:
- return agg_stdout
- else:
- return b64encode(agg_stdout).decode('utf-8')
-
- else:
- error_msg = agg_stderr.decode('utf-8')
- raise AirflowException("error running cmd: {0}, error: {1}"
- .format(self.command, error_msg))
+ raise AirflowException("SSH command not specified. Aborting.")
+
+ with self.ssh_hook.get_conn() as ssh_client:
+ # Auto apply tty when its required in case of sudo
+ get_pty = False
+ if self.command.startswith('sudo'):
+ get_pty = True
+
+ # set timeout taken as params
+ stdin, stdout, stderr = ssh_client.exec_command(command=self.command,
+ get_pty=get_pty,
+ timeout=self.timeout
+ )
+ # get channels
+ channel = stdout.channel
+
+ # closing stdin
+ stdin.close()
+ channel.shutdown_write()
+
+ agg_stdout = b''
+ agg_stderr = b''
+
+ # capture any initial output in case channel is closed already
+ stdout_buffer_length = len(stdout.channel.in_buffer)
+
+ if stdout_buffer_length > 0:
+ agg_stdout += stdout.channel.recv(stdout_buffer_length)
+
+ # read from both stdout and stderr
+ while not channel.closed or \
+ channel.recv_ready() or \
+ channel.recv_stderr_ready():
+ readq, _, _ = select([channel], [], [], self.timeout)
+ for c in readq:
+ if c.recv_ready():
+ line = stdout.channel.recv(len(c.in_buffer))
+ line = line
+ agg_stdout += line
+ self.log.info(line.decode('utf-8').strip('\n'))
+ if c.recv_stderr_ready():
+ line = stderr.channel.recv_stderr(len(c.in_stderr_buffer))
+ line = line
+ agg_stderr += line
+ self.log.warning(line.decode('utf-8').strip('\n'))
+ if stdout.channel.exit_status_ready()\
+ and not stderr.channel.recv_stderr_ready()\
+ and not stdout.channel.recv_ready():
+ stdout.channel.shutdown_read()
+ stdout.channel.close()
+ break
+
+ stdout.close()
+ stderr.close()
+
+ exit_status = stdout.channel.recv_exit_status()
+ if exit_status is 0:
+ # returning output if do_xcom_push is set
+ if self.do_xcom_push:
+ enable_pickling = configuration.conf.getboolean(
+ 'core', 'enable_xcom_pickling'
+ )
+ if enable_pickling:
+ return agg_stdout
+ else:
+ return b64encode(agg_stdout).decode('utf-8')
+
+ else:
+ error_msg = agg_stderr.decode('utf-8')
+ raise AirflowException("error running cmd: {0}, error: {1}"
+ .format(self.command, error_msg))
except Exception as e:
raise AirflowException("SSH operator error: {0}".format(str(e)))
diff --git a/airflow/contrib/operators/vertica_operator.py b/airflow/contrib/operators/vertica_operator.py
index 41072ff82b..37ccd134e8 100644
--- a/airflow/contrib/operators/vertica_operator.py
+++ b/airflow/contrib/operators/vertica_operator.py
@@ -26,7 +26,7 @@ class VerticaOperator(BaseOperator):
Executes sql code in a specific Vertica database
:param vertica_conn_id: reference to a specific Vertica database
- :type vertica_conn_id: string
+ :type vertica_conn_id: str
:param sql: the sql code to be executed. (templated)
:type sql: Can receive a str representing a sql statement,
a list of str (sql statements), or reference to a template file.
diff --git a/airflow/contrib/operators/winrm_operator.py b/airflow/contrib/operators/winrm_operator.py
index fcd2328d9a..c81acac44f 100644
--- a/airflow/contrib/operators/winrm_operator.py
+++ b/airflow/contrib/operators/winrm_operator.py
@@ -17,14 +17,24 @@
# specific language governing permissions and limitations
# under the License.
+from base64 import b64encode
+import logging
+
+from winrm.exceptions import WinRMOperationTimeoutError
+
+from airflow import configuration
from airflow.contrib.hooks.winrm_hook import WinRMHook
from airflow.exceptions import AirflowException
from airflow.models import BaseOperator
from airflow.utils.decorators import apply_defaults
+# Hide the following error message in urllib3 when making WinRM connections:
+# requests.packages.urllib3.exceptions.HeaderParsingError: [StartBoundaryNotFoundDefect(),
+# MultipartInvariantViolationDefect()], unparsed data: ''
+logging.getLogger('requests.packages.urllib3.connectionpool').setLevel(logging.CRITICAL)
-class WinRMOperator(BaseOperator):
+class WinRMOperator(BaseOperator):
"""
WinRMOperator to execute commands on given remote host using the winrm_hook.
@@ -41,7 +51,6 @@ class WinRMOperator(BaseOperator):
:param do_xcom_push: return the stdout which also get set in xcom by airflow platform
:type do_xcom_push: bool
"""
-
template_fields = ('command',)
@apply_defaults
@@ -63,48 +72,78 @@ def __init__(self,
self.do_xcom_push = do_xcom_push
def execute(self, context):
- try:
- if self.ssh_conn_id and not self.winrm_hook:
- self.log.info("hook not found, creating")
- self.winrm_hook = WinRMHook(ssh_conn_id=self.ssh_conn_id)
+ if self.ssh_conn_id and not self.winrm_hook:
+ self.log.info("Hook not found, creating...")
+ self.winrm_hook = WinRMHook(ssh_conn_id=self.ssh_conn_id)
- if not self.winrm_hook:
- raise AirflowException("can not operate without ssh_hook or ssh_conn_id")
+ if not self.winrm_hook:
+ raise AirflowException("Cannot operate without winrm_hook or ssh_conn_id.")
- if self.remote_host is not None:
- self.winrm_hook.remote_host = self.remote_host
+ if self.remote_host is not None:
+ self.winrm_hook.remote_host = self.remote_host
- winrm_client = self.winrm_hook.get_conn()
- self.log.info("Established WinRM connection")
+ if not self.command:
+ raise AirflowException("No command specified so nothing to execute here.")
- if not self.command:
- raise AirflowException("no command specified so nothing to execute here.")
+ winrm_client = self.winrm_hook.get_conn()
- self.log.info(
- "Starting command: '{command}' on remote host: {remotehost}".
- format(command=self.command, remotehost=self.winrm_hook.remote_host)
+ try:
+ self.log.info("Running command: '{command}'...".format(command=self.command))
+ command_id = self.winrm_hook.winrm_protocol.run_command(
+ winrm_client,
+ self.command
)
- command_id = self.winrm_hook.winrm_protocol. \
- run_command(winrm_client, self.command)
- std_out, std_err, status_code = self.winrm_hook.winrm_protocol. \
- get_command_output(winrm_client, command_id)
-
- self.log.info("std out: " + std_out.decode())
- self.log.info("std err: " + std_err.decode())
- self.log.info("exit code: " + str(status_code))
- self.log.info("Cleaning up WinRM command")
+
+ # See: https://github.com/diyan/pywinrm/blob/master/winrm/protocol.py
+ stdout_buffer = []
+ stderr_buffer = []
+ command_done = False
+ while not command_done:
+ try:
+ stdout, stderr, return_code, command_done = \
+ self.winrm_hook.winrm_protocol._raw_get_command_output(
+ winrm_client,
+ command_id
+ )
+
+ # Only buffer stdout if we need to so that we minimize memory usage.
+ if self.do_xcom_push:
+ stdout_buffer.append(stdout)
+ stderr_buffer.append(stderr)
+
+ for line in stdout.decode('utf-8').splitlines():
+ self.log.info(line)
+ for line in stderr.decode('utf-8').splitlines():
+ self.log.warning(line)
+ except WinRMOperationTimeoutError as e:
+ # this is an expected error when waiting for a
+ # long-running process, just silently retry
+ pass
+
self.winrm_hook.winrm_protocol.cleanup_command(winrm_client, command_id)
- self.log.info("Cleaning up WinRM protocol shell")
self.winrm_hook.winrm_protocol.close_shell(winrm_client)
- if status_code is 0:
- return std_out.decode()
-
- else:
- error_msg = std_err.decode()
- raise AirflowException("error running cmd: {0}, error: {1}"
- .format(self.command, error_msg))
except Exception as e:
raise AirflowException("WinRM operator error: {0}".format(str(e)))
+ if return_code is 0:
+ # returning output if do_xcom_push is set
+ if self.do_xcom_push:
+ enable_pickling = configuration.conf.getboolean(
+ 'core', 'enable_xcom_pickling'
+ )
+ if enable_pickling:
+ return stdout_buffer
+ else:
+ return b64encode(b''.join(stdout_buffer)).decode('utf-8')
+ else:
+ error_msg = "Error running cmd: {0}, return code: {1}, error: {2}".format(
+ self.command,
+ return_code,
+ b''.join(stderr_buffer).decode('utf-8')
+ )
+ raise AirflowException(error_msg)
+
+ self.log.info("Finished!")
+
return True
diff --git a/airflow/contrib/plugins/metastore_browser/templates/metastore_browser/table.html b/airflow/contrib/plugins/metastore_browser/templates/metastore_browser/table.html
index 3a9d7ca671..cbb7acff6d 100644
--- a/airflow/contrib/plugins/metastore_browser/templates/metastore_browser/table.html
+++ b/airflow/contrib/plugins/metastore_browser/templates/metastore_browser/table.html
@@ -28,7 +28,7 @@ <h4>
<li role="presentation" class="active"><a href="#home" aria-controls="fields" role="tab" data-toggle="tab">Fields</a></li>
<li role="presentation"><a href="#data" aria-controls="data" role="tab" data-toggle="tab">Sample Data</a></li>
<li role="presentation"><a href="#partitions" aria-controls="partitions" role="tab" data-toggle="tab">Partitions</a></li>
- <li role="presentation"><a href="#attributes" aria-controls="attributes" role="tab" data-toggle="tab">Atributes</a></li>
+ <li role="presentation"><a href="#attributes" aria-controls="attributes" role="tab" data-toggle="tab">Attributes</a></li>
<li role="presentation"><a href="#parameters" aria-controls="parameters" role="tab" data-toggle="tab">Parameters</a></li>
<li role="presentation"><a href="#ddl" aria-controls="ddl" role="tab" data-toggle="tab">DDL</a></li>
</ul>
diff --git a/airflow/contrib/sensors/bash_sensor.py b/airflow/contrib/sensors/bash_sensor.py
index 26fbb06c18..7372a419b6 100644
--- a/airflow/contrib/sensors/bash_sensor.py
+++ b/airflow/contrib/sensors/bash_sensor.py
@@ -33,7 +33,7 @@ class BashSensor(BaseSensorOperator):
:param bash_command: The command, set of commands or reference to a
bash script (must be '.sh') to be executed.
- :type bash_command: string
+ :type bash_command: str
:param env: If env is not None, it must be a mapping that defines the
environment variables for the new process; these are used instead
@@ -41,7 +41,7 @@ class BashSensor(BaseSensorOperator):
behavior. (templated)
:type env: dict
:param output_encoding: output encoding of bash command.
- :type output_encoding: string
+ :type output_encoding: str
"""
template_fields = ('bash_command', 'env')
diff --git a/airflow/contrib/sensors/bigquery_sensor.py b/airflow/contrib/sensors/bigquery_sensor.py
index 2e496f6889..89ec476d9d 100644
--- a/airflow/contrib/sensors/bigquery_sensor.py
+++ b/airflow/contrib/sensors/bigquery_sensor.py
@@ -28,19 +28,19 @@ class BigQueryTableSensor(BaseSensorOperator):
:param project_id: The Google cloud project in which to look for the table.
The connection supplied to the hook must provide
access to the specified project.
- :type project_id: string
+ :type project_id: str
:param dataset_id: The name of the dataset in which to look for the table.
storage bucket.
- :type dataset_id: string
+ :type dataset_id: str
:param table_id: The name of the table to check the existence of.
- :type table_id: string
+ :type table_id: str
:param bigquery_conn_id: The connection ID to use when connecting to
Google BigQuery.
- :type bigquery_conn_id: string
+ :type bigquery_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must
have domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ('project_id', 'dataset_id', 'table_id',)
ui_color = '#f0eee4'
diff --git a/airflow/contrib/sensors/cassandra_sensor.py b/airflow/contrib/sensors/cassandra_record_sensor.py
similarity index 85%
rename from airflow/contrib/sensors/cassandra_sensor.py
rename to airflow/contrib/sensors/cassandra_record_sensor.py
index aef66122e9..bbb2f50990 100644
--- a/airflow/contrib/sensors/cassandra_sensor.py
+++ b/airflow/contrib/sensors/cassandra_record_sensor.py
@@ -29,9 +29,10 @@ class CassandraRecordSensor(BaseSensorOperator):
primary keys 'p1' and 'p2' to be populated in keyspace 'k' and table 't',
instantiate it as follows:
- >>> CassandraRecordSensor(table="k.t", keys={"p1": "v1", "p2": "v2"},
- ... cassandra_conn_id="cassandra_default", task_id="cassandra_sensor")
- <Task(CassandraRecordSensor): cassandra_sensor>
+ >>> cassandra_sensor = CassandraRecordSensor(table="k.t",
+ ... keys={"p1": "v1", "p2": "v2"},
+ ... cassandra_conn_id="cassandra_default",
+ ... task_id="cassandra_sensor")
"""
template_fields = ('table', 'keys')
@@ -42,12 +43,12 @@ def __init__(self, table, keys, cassandra_conn_id, *args, **kwargs):
:param table: Target Cassandra table.
Use dot notation to target a specific keyspace.
- :type table: string
+ :type table: str
:param keys: The keys and their values to be monitored
:type keys: dict
:param cassandra_conn_id: The connection ID to use
when connecting to Cassandra cluster
- :type cassandra_conn_id: string
+ :type cassandra_conn_id: str
"""
super(CassandraRecordSensor, self).__init__(*args, **kwargs)
self.cassandra_conn_id = cassandra_conn_id
diff --git a/airflow/contrib/sensors/cassandra_table_sensor.py b/airflow/contrib/sensors/cassandra_table_sensor.py
new file mode 100644
index 0000000000..088c82bdd3
--- /dev/null
+++ b/airflow/contrib/sensors/cassandra_table_sensor.py
@@ -0,0 +1,56 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from airflow.contrib.hooks.cassandra_hook import CassandraHook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class CassandraTableSensor(BaseSensorOperator):
+ """
+ Checks for the existence of a table in a Cassandra cluster.
+
+ For example, if you want to wait for a table called 't' to be created
+ in a keyspace 'k', instantiate it as follows:
+
+ >>> cassandra_sensor = CassandraTableSensor(table="k.t",
+ ... cassandra_conn_id="cassandra_default",
+ ... task_id="cassandra_sensor")
+ """
+ template_fields = ('table',)
+
+ @apply_defaults
+ def __init__(self, table, cassandra_conn_id, *args, **kwargs):
+ """
+ Create a new CassandraTableSensor
+
+ :param table: Target Cassandra table.
+ Use dot notation to target a specific keyspace.
+ :type table: str
+ :param cassandra_conn_id: The connection ID to use
+ when connecting to Cassandra cluster
+ :type cassandra_conn_id: str
+ """
+ super(CassandraTableSensor, self).__init__(*args, **kwargs)
+ self.cassandra_conn_id = cassandra_conn_id
+ self.table = table
+
+ def poke(self, context):
+ self.log.info('Sensor check existence of table: %s', self.table)
+ hook = CassandraHook(self.cassandra_conn_id)
+ return hook.table_exists(self.table)
diff --git a/airflow/contrib/sensors/datadog_sensor.py b/airflow/contrib/sensors/datadog_sensor.py
index fb455d6cd7..01b7b9fc51 100644
--- a/airflow/contrib/sensors/datadog_sensor.py
+++ b/airflow/contrib/sensors/datadog_sensor.py
@@ -18,7 +18,7 @@
# under the License.
from airflow.sensors.base_sensor_operator import BaseSensorOperator
from airflow.contrib.hooks.datadog_hook import DatadogHook
-from airflow.utils import apply_defaults
+from airflow.utils.decorators import apply_defaults
from airflow.exceptions import AirflowException
from datadog import api
@@ -32,7 +32,7 @@ class DatadogSensor(BaseSensorOperator):
Airflow runs.
:param datadog_conn_id: The connection to datadog, containing metadata for api keys.
- :param datadog_conn_id: string
+ :param datadog_conn_id: str
"""
ui_color = '#66c3dd'
diff --git a/airflow/contrib/sensors/emr_base_sensor.py b/airflow/contrib/sensors/emr_base_sensor.py
index abeb421b9d..6dcb9fc5f3 100644
--- a/airflow/contrib/sensors/emr_base_sensor.py
+++ b/airflow/contrib/sensors/emr_base_sensor.py
@@ -17,7 +17,7 @@
# specific language governing permissions and limitations
# under the License.
from airflow.sensors.base_sensor_operator import BaseSensorOperator
-from airflow.utils import apply_defaults
+from airflow.utils.decorators import apply_defaults
from airflow.exceptions import AirflowException
diff --git a/airflow/contrib/sensors/emr_job_flow_sensor.py b/airflow/contrib/sensors/emr_job_flow_sensor.py
index 31d16a066b..ba8a018134 100644
--- a/airflow/contrib/sensors/emr_job_flow_sensor.py
+++ b/airflow/contrib/sensors/emr_job_flow_sensor.py
@@ -18,7 +18,7 @@
# under the License.
from airflow.contrib.hooks.emr_hook import EmrHook
from airflow.contrib.sensors.emr_base_sensor import EmrBaseSensor
-from airflow.utils import apply_defaults
+from airflow.utils.decorators import apply_defaults
class EmrJobFlowSensor(EmrBaseSensor):
@@ -27,7 +27,7 @@ class EmrJobFlowSensor(EmrBaseSensor):
If it fails the sensor errors, failing the task.
:param job_flow_id: job_flow_id to check the state of
- :type job_flow_id: string
+ :type job_flow_id: str
"""
NON_TERMINAL_STATES = ['STARTING', 'BOOTSTRAPPING', 'RUNNING',
@@ -50,5 +50,6 @@ def get_emr_response(self):
self.log.info('Poking cluster %s', self.job_flow_id)
return emr.describe_cluster(ClusterId=self.job_flow_id)
- def state_from_response(self, response):
+ @staticmethod
+ def state_from_response(response):
return response['Cluster']['Status']['State']
diff --git a/airflow/contrib/sensors/emr_step_sensor.py b/airflow/contrib/sensors/emr_step_sensor.py
index 3dddf01bd8..65bb7c95b1 100644
--- a/airflow/contrib/sensors/emr_step_sensor.py
+++ b/airflow/contrib/sensors/emr_step_sensor.py
@@ -18,7 +18,7 @@
# under the License.
from airflow.contrib.hooks.emr_hook import EmrHook
from airflow.contrib.sensors.emr_base_sensor import EmrBaseSensor
-from airflow.utils import apply_defaults
+from airflow.utils.decorators import apply_defaults
class EmrStepSensor(EmrBaseSensor):
@@ -27,9 +27,9 @@ class EmrStepSensor(EmrBaseSensor):
If it fails the sensor errors, failing the task.
:param job_flow_id: job_flow_id which contains the step check the state of
- :type job_flow_id: string
+ :type job_flow_id: str
:param step_id: step to check the state of
- :type step_id: string
+ :type step_id: str
"""
NON_TERMINAL_STATES = ['PENDING', 'RUNNING', 'CONTINUE']
@@ -53,5 +53,6 @@ def get_emr_response(self):
self.log.info('Poking step %s on cluster %s', self.step_id, self.job_flow_id)
return emr.describe_step(ClusterId=self.job_flow_id, StepId=self.step_id)
- def state_from_response(self, response):
+ @staticmethod
+ def state_from_response(response):
return response['Step']['Status']['State']
diff --git a/airflow/contrib/sensors/file_sensor.py b/airflow/contrib/sensors/file_sensor.py
index 3f7bb24e08..048dd06258 100644
--- a/airflow/contrib/sensors/file_sensor.py
+++ b/airflow/contrib/sensors/file_sensor.py
@@ -35,10 +35,10 @@ class FileSensor(BaseSensorOperator):
:param fs_conn_id: reference to the File (path)
connection id
- :type fs_conn_id: string
+ :type fs_conn_id: str
:param filepath: File or folder name (relative to
the base path set within the connection)
- :type fs_conn_id: string
+ :type fs_conn_id: str
"""
template_fields = ('filepath',)
ui_color = '#91818a'
@@ -46,7 +46,7 @@ class FileSensor(BaseSensorOperator):
@apply_defaults
def __init__(self,
filepath,
- fs_conn_id='fs_default2',
+ fs_conn_id='fs_default',
*args,
**kwargs):
super(FileSensor, self).__init__(*args, **kwargs)
@@ -56,7 +56,7 @@ def __init__(self,
def poke(self, context):
hook = FSHook(self.fs_conn_id)
basepath = hook.get_path()
- full_path = "/".join([basepath, self.filepath])
+ full_path = os.path.join(basepath, self.filepath)
self.log.info('Poking for file {full_path}'.format(**locals()))
try:
if stat.S_ISDIR(os.stat(full_path).st_mode):
diff --git a/airflow/contrib/sensors/gcs_sensor.py b/airflow/contrib/sensors/gcs_sensor.py
index 23cd760a41..5a4f73e61e 100644
--- a/airflow/contrib/sensors/gcs_sensor.py
+++ b/airflow/contrib/sensors/gcs_sensor.py
@@ -27,17 +27,17 @@ class GoogleCloudStorageObjectSensor(BaseSensorOperator):
Create a new GoogleCloudStorageObjectSensor.
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
+ :type bucket: str
:param object: The name of the object to check in the Google cloud
storage bucket.
- :type object: string
+ :type object: str
:param google_cloud_storage_conn_id: The connection ID to use when
connecting to Google cloud storage.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ('bucket', 'object')
ui_color = '#f0eee4'
@@ -79,21 +79,21 @@ class GoogleCloudStorageObjectUpdatedSensor(BaseSensorOperator):
Create a new GoogleCloudStorageObjectUpdatedSensor.
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
+ :type bucket: str
:param object: The name of the object to download in the Google cloud
storage bucket.
- :type object: string
+ :type object: str
:param ts_func: Callback for defining the update condition. The default callback
returns execution_date + schedule_interval. The callback takes the context
as parameter.
:type ts_func: function
:param google_cloud_storage_conn_id: The connection ID to use when
connecting to Google cloud storage.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ('bucket', 'object')
template_ext = ('.sql',)
@@ -129,17 +129,17 @@ class GoogleCloudStoragePrefixSensor(BaseSensorOperator):
Create a new GoogleCloudStorageObjectSensor.
:param bucket: The Google cloud storage bucket where the object is.
- :type bucket: string
+ :type bucket: str
:param prefix: The name of the prefix to check in the Google cloud
storage bucket.
- :type prefix: string
+ :type prefix: str
:param google_cloud_storage_conn_id: The connection ID to use when
connecting to Google cloud storage.
- :type google_cloud_storage_conn_id: string
+ :type google_cloud_storage_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
template_fields = ('bucket', 'prefix')
ui_color = '#f0eee4'
diff --git a/airflow/contrib/sensors/mongo_sensor.py b/airflow/contrib/sensors/mongo_sensor.py
new file mode 100644
index 0000000000..8fd32a0775
--- /dev/null
+++ b/airflow/contrib/sensors/mongo_sensor.py
@@ -0,0 +1,58 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from airflow.contrib.hooks.mongo_hook import MongoHook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class MongoSensor(BaseSensorOperator):
+ """
+ Checks for the existence of a document which
+ matches the given query in MongoDB. Example:
+
+ >>> mongo_sensor = MongoSensor(collection="coll",
+ ... query={"key": "value"},
+ ... mongo_conn_id="mongo_default",
+ ... task_id="mongo_sensor")
+ """
+ template_fields = ('collection', 'query')
+
+ @apply_defaults
+ def __init__(self, collection, query, mongo_conn_id="mongo_default", *args, **kwargs):
+ """
+ Create a new MongoSensor
+
+ :param collection: Target MongoDB collection.
+ :type collection: str
+ :param query: The query to find the target document.
+ :type query: dict
+ :param mongo_conn_id: The connection ID to use
+ when connecting to MongoDB.
+ :type mongo_conn_id: str
+ """
+ super(MongoSensor, self).__init__(*args, **kwargs)
+ self.mongo_conn_id = mongo_conn_id
+ self.collection = collection
+ self.query = query
+
+ def poke(self, context):
+ self.log.info("Sensor check existence of the document "
+ "that matches the following query: %s", self.query)
+ hook = MongoHook(self.mongo_conn_id)
+ return hook.find(self.collection, self.query, find_one=True) is not None
diff --git a/airflow/contrib/sensors/pubsub_sensor.py b/airflow/contrib/sensors/pubsub_sensor.py
index 7d1721570d..8104fb5699 100644
--- a/airflow/contrib/sensors/pubsub_sensor.py
+++ b/airflow/contrib/sensors/pubsub_sensor.py
@@ -54,10 +54,10 @@ def __init__(
**kwargs):
"""
:param project: the GCP project ID for the subscription (templated)
- :type project: string
+ :type project: str
:param subscription: the Pub/Sub subscription name. Do not include the
full subscription path.
- :type subscription: string
+ :type subscription: str
:param max_messages: The maximum number of messages to retrieve per
PubSub pull request
:type max_messages: int
@@ -69,11 +69,11 @@ def __init__(
:type ack_messages: bool
:param gcp_conn_id: The connection ID to use connecting to
Google Cloud Platform.
- :type gcp_conn_id: string
+ :type gcp_conn_id: str
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request
must have domain-wide delegation enabled.
- :type delegate_to: string
+ :type delegate_to: str
"""
super(PubSubPullSensor, self).__init__(*args, **kwargs)
diff --git a/airflow/contrib/sensors/qubole_sensor.py b/airflow/contrib/sensors/qubole_sensor.py
index d67fa8557e..c0291b1521 100644
--- a/airflow/contrib/sensors/qubole_sensor.py
+++ b/airflow/contrib/sensors/qubole_sensor.py
@@ -29,14 +29,6 @@
class QuboleSensor(BaseSensorOperator):
"""
Base class for all Qubole Sensors
-
- :param qubole_conn_id: The qubole connection to run the sensor against
- :type qubole_conn_id: string
- :param data: a JSON object containing payload, whose presence needs to be checked
- :type data: a JSON object
-
- .. note:: Both ``data`` and ``qubole_conn_id`` fields are template-supported. You can
- also use ``.txt`` files for template driven use cases.
"""
template_fields = ('data', 'qubole_conn_id')
@@ -75,6 +67,22 @@ def poke(self, context):
class QuboleFileSensor(QuboleSensor):
+ """
+ Wait for a file or folder to be present in cloud storage
+ and check for its presence via QDS APIs
+
+ :param qubole_conn_id: Connection id which consists of qds auth_token
+ :type qubole_conn_id: str
+ :param data: a JSON object containing payload, whose presence needs to be checked
+ Check this `example <https://github.com/apache/incubator-airflow/blob/master\
+ /airflow/contrib/example_dags/example_qubole_sensor.py>`_ for sample payload
+ structure.
+ :type data: a JSON object
+
+ .. note:: Both ``data`` and ``qubole_conn_id`` fields support templating. You can
+ also use ``.txt`` files for template-driven use cases.
+ """
+
@apply_defaults
def __init__(self, *args, **kwargs):
self.sensor_class = FileSensor
@@ -82,6 +90,22 @@ def __init__(self, *args, **kwargs):
class QubolePartitionSensor(QuboleSensor):
+ """
+ Wait for a Hive partition to show up in QHS (Qubole Hive Service)
+ and check for its presence via QDS APIs
+
+ :param qubole_conn_id: Connection id which consists of qds auth_token
+ :type qubole_conn_id: str
+ :param data: a JSON object containing payload, whose presence needs to be checked.
+ Check this `example <https://github.com/apache/incubator-airflow/blob/master\
+ /airflow/contrib/example_dags/example_qubole_sensor.py>`_ for sample payload
+ structure.
+ :type data: a JSON object
+
+ .. note:: Both ``data`` and ``qubole_conn_id`` fields support templating. You can
+ also use ``.txt`` files for template-driven use cases.
+ """
+
@apply_defaults
def __init__(self, *args, **kwargs):
self.sensor_class = PartitionSensor
diff --git a/airflow/contrib/sensors/redis_key_sensor.py b/airflow/contrib/sensors/redis_key_sensor.py
index baf3e161f5..a2d190baae 100644
--- a/airflow/contrib/sensors/redis_key_sensor.py
+++ b/airflow/contrib/sensors/redis_key_sensor.py
@@ -34,9 +34,9 @@ def __init__(self, key, redis_conn_id, *args, **kwargs):
Create a new RedisKeySensor
:param key: The key to be monitored
- :type key: string
+ :type key: str
:param redis_conn_id: The connection ID to use when connecting to Redis DB.
- :type redis_conn_id: string
+ :type redis_conn_id: str
"""
super(RedisKeySensor, self).__init__(*args, **kwargs)
self.redis_conn_id = redis_conn_id
diff --git a/airflow/contrib/sensors/sagemaker_base_sensor.py b/airflow/contrib/sensors/sagemaker_base_sensor.py
new file mode 100644
index 0000000000..149c2a1aab
--- /dev/null
+++ b/airflow/contrib/sensors/sagemaker_base_sensor.py
@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.exceptions import AirflowException
+
+
+class SageMakerBaseSensor(BaseSensorOperator):
+ """
+ Contains general sensor behavior for SageMaker.
+ Subclasses should implement get_sagemaker_response()
+ and state_from_response() methods.
+ Subclasses should also implement NON_TERMINAL_STATES and FAILED_STATE methods.
+ """
+ ui_color = '#66c3ff'
+
+ @apply_defaults
+ def __init__(
+ self,
+ aws_conn_id='aws_default',
+ *args, **kwargs):
+ super(SageMakerBaseSensor, self).__init__(*args, **kwargs)
+ self.aws_conn_id = aws_conn_id
+
+ def poke(self, context):
+ response = self.get_sagemaker_response()
+
+ if not response['ResponseMetadata']['HTTPStatusCode'] == 200:
+ self.log.info('Bad HTTP response: %s', response)
+ return False
+
+ state = self.state_from_response(response)
+
+ self.log.info('Job currently %s', state)
+
+ if state in self.non_terminal_states():
+ return False
+
+ if state in self.failed_states():
+ failed_reason = self.get_failed_reason_from_response(response)
+ raise AirflowException("Sagemaker job failed for the following reason: %s"
+ % failed_reason)
+ return True
+
+ def non_terminal_states(self):
+ raise AirflowException("Non Terminal States need to be specified in subclass")
+
+ def failed_states(self):
+ raise AirflowException("Failed States need to be specified in subclass")
+
+ def get_sagemaker_response(self):
+ raise AirflowException(
+ "Method get_sagemaker_response()not implemented.")
+
+ def get_failed_reason_from_response(self, response):
+ return 'Unknown'
+
+ def state_from_response(self, response):
+ raise AirflowException(
+ "Method state_from_response()not implemented.")
diff --git a/airflow/contrib/sensors/sagemaker_training_sensor.py b/airflow/contrib/sensors/sagemaker_training_sensor.py
new file mode 100644
index 0000000000..5e83e846ef
--- /dev/null
+++ b/airflow/contrib/sensors/sagemaker_training_sensor.py
@@ -0,0 +1,66 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.sagemaker_hook import SageMakerHook
+from airflow.contrib.sensors.sagemaker_base_sensor import SageMakerBaseSensor
+from airflow.utils.decorators import apply_defaults
+
+
+class SageMakerTrainingSensor(SageMakerBaseSensor):
+ """
+ Asks for the state of the training state until it reaches a terminal state.
+ If it fails the sensor errors, failing the task.
+
+ :param job_name: job_name of the training instance to check the state of
+ :type job_name: str
+ """
+
+ template_fields = ['job_name']
+ template_ext = ()
+
+ @apply_defaults
+ def __init__(self,
+ job_name,
+ region_name=None,
+ *args,
+ **kwargs):
+ super(SageMakerTrainingSensor, self).__init__(*args, **kwargs)
+ self.job_name = job_name
+ self.region_name = region_name
+
+ def non_terminal_states(self):
+ return SageMakerHook.non_terminal_states
+
+ def failed_states(self):
+ return SageMakerHook.failed_states
+
+ def get_sagemaker_response(self):
+ sagemaker = SageMakerHook(
+ aws_conn_id=self.aws_conn_id,
+ region_name=self.region_name
+ )
+
+ self.log.info('Poking Sagemaker Training Job %s', self.job_name)
+ return sagemaker.describe_training_job(self.job_name)
+
+ def get_failed_reason_from_response(self, response):
+ return response['FailureReason']
+
+ def state_from_response(self, response):
+ return response['TrainingJobStatus']
diff --git a/airflow/contrib/sensors/sagemaker_transform_sensor.py b/airflow/contrib/sensors/sagemaker_transform_sensor.py
new file mode 100644
index 0000000000..68ef1d8dd7
--- /dev/null
+++ b/airflow/contrib/sensors/sagemaker_transform_sensor.py
@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.sagemaker_hook import SageMakerHook
+from airflow.contrib.sensors.sagemaker_base_sensor import SageMakerBaseSensor
+from airflow.utils.decorators import apply_defaults
+
+
+class SageMakerTransformSensor(SageMakerBaseSensor):
+ """
+ Asks for the state of the transform state until it reaches a terminal state.
+ The sensor will error if the job errors, throwing a AirflowException
+ containing the failure reason.
+
+ :param job_name: job_name of the transform job instance to check the state of
+ :type job_name: string
+ :param region_name: The AWS region_name
+ :type region_name: string
+ """
+
+ template_fields = ['job_name']
+ template_ext = ()
+
+ @apply_defaults
+ def __init__(self,
+ job_name,
+ region_name=None,
+ *args,
+ **kwargs):
+ super(SageMakerTransformSensor, self).__init__(*args, **kwargs)
+ self.job_name = job_name
+ self.region_name = region_name
+
+ def non_terminal_states(self):
+ return SageMakerHook.non_terminal_states
+
+ def failed_states(self):
+ return SageMakerHook.failed_states
+
+ def get_sagemaker_response(self):
+ sagemaker = SageMakerHook(
+ aws_conn_id=self.aws_conn_id,
+ region_name=self.region_name
+ )
+
+ self.log.info('Poking Sagemaker Transform Job %s', self.job_name)
+ return sagemaker.describe_transform_job(self.job_name)
+
+ def get_failed_reason_from_response(self, response):
+ return response['FailureReason']
+
+ def state_from_response(self, response):
+ return response['TransformJobStatus']
diff --git a/airflow/contrib/sensors/sagemaker_tuning_sensor.py b/airflow/contrib/sensors/sagemaker_tuning_sensor.py
new file mode 100644
index 0000000000..ec90964266
--- /dev/null
+++ b/airflow/contrib/sensors/sagemaker_tuning_sensor.py
@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.sagemaker_hook import SageMakerHook
+from airflow.contrib.sensors.sagemaker_base_sensor import SageMakerBaseSensor
+from airflow.utils.decorators import apply_defaults
+
+
+class SageMakerTuningSensor(SageMakerBaseSensor):
+ """
+ Asks for the state of the tuning state until it reaches a terminal state.
+ The sensor will error if the job errors, throwing a AirflowException
+ containing the failure reason.
+
+ :param job_name: job_name of the tuning instance to check the state of
+ :type job_name: str
+ :param region_name: The AWS region_name
+ :type region_name: str
+ """
+
+ template_fields = ['job_name']
+ template_ext = ()
+
+ @apply_defaults
+ def __init__(self,
+ job_name,
+ region_name=None,
+ *args,
+ **kwargs):
+ super(SageMakerTuningSensor, self).__init__(*args, **kwargs)
+ self.job_name = job_name
+ self.region_name = region_name
+
+ def non_terminal_states(self):
+ return SageMakerHook.non_terminal_states
+
+ def failed_states(self):
+ return SageMakerHook.failed_states
+
+ def get_sagemaker_response(self):
+ sagemaker = SageMakerHook(
+ aws_conn_id=self.aws_conn_id,
+ region_name=self.region_name
+ )
+
+ self.log.info('Poking Sagemaker Tuning Job %s', self.job_name)
+ return sagemaker.describe_tuning_job(self.job_name)
+
+ def get_failed_reason_from_response(self, response):
+ return response['FailureReason']
+
+ def state_from_response(self, response):
+ return response['HyperParameterTuningJobStatus']
diff --git a/airflow/contrib/sensors/sftp_sensor.py b/airflow/contrib/sensors/sftp_sensor.py
index 51ad11ddbf..088e3e54b4 100644
--- a/airflow/contrib/sensors/sftp_sensor.py
+++ b/airflow/contrib/sensors/sftp_sensor.py
@@ -20,7 +20,7 @@
import logging
from paramiko import SFTP_NO_SUCH_FILE
from airflow.contrib.hooks.sftp_hook import SFTPHook
-from airflow.operators.sensors import BaseSensorOperator
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
diff --git a/airflow/contrib/task_runner/cgroup_task_runner.py b/airflow/contrib/task_runner/cgroup_task_runner.py
index a97eac2af8..4662b0fe82 100644
--- a/airflow/contrib/task_runner/cgroup_task_runner.py
+++ b/airflow/contrib/task_runner/cgroup_task_runner.py
@@ -117,13 +117,13 @@ def start(self):
"creating another one",
cgroups.get("cpu"), cgroups.get("memory")
)
- self.process = self.run_command(['bash', '-c'], join_args=True)
+ self.process = self.run_command()
return
# Create a unique cgroup name
cgroup_name = "airflow/{}/{}".format(datetime.datetime.utcnow().
strftime("%Y-%m-%d"),
- str(uuid.uuid1()))
+ str(uuid.uuid4()))
self.mem_cgroup_name = "memory/{}".format(cgroup_name)
self.cpu_cgroup_name = "cpu/{}".format(cgroup_name)
@@ -193,7 +193,8 @@ def on_finish(self):
if self._created_cpu_cgroup:
self._delete_cgroup(self.cpu_cgroup_name)
- def _get_cgroup_names(self):
+ @staticmethod
+ def _get_cgroup_names():
"""
:return: a mapping between the subsystem name to the cgroup name
:rtype: dict[str, str]
diff --git a/airflow/contrib/utils/gcp_field_validator.py b/airflow/contrib/utils/gcp_field_validator.py
new file mode 100644
index 0000000000..20f72d94b8
--- /dev/null
+++ b/airflow/contrib/utils/gcp_field_validator.py
@@ -0,0 +1,417 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Validator for body fields sent via GCP API.
+
+The validator performs validation of the body (being dictionary of fields) that
+is sent in the API request to Google Cloud (via googleclient API usually).
+
+Context
+-------
+The specification mostly focuses on helping Airflow DAG developers in the development
+phase. You can build your own GCP operator (such as GcfDeployOperator for example) which
+can have built-in validation specification for the particular API. It's super helpful
+when developer plays with different fields and their values at the initial phase of
+DAG development. Most of the Google Cloud APIs perform their own validation on the
+server side, but most of the requests are asynchronous and you need to wait for result
+of the operation. This takes precious times and slows
+down iteration over the API. BodyFieldValidator is meant to be used on the client side
+and it should therefore provide an instant feedback to the developer on misspelled or
+wrong type of parameters.
+
+The validation should be performed in "execute()" method call in order to allow
+template parameters to be expanded before validation is performed.
+
+Types of fields
+---------------
+
+Specification is an array of dictionaries - each dictionary describes field, its type,
+validation, optionality, api_version supported and nested fields (for unions and dicts).
+
+Typically (for clarity and in order to aid syntax highlighting) the array of
+dicts should be defined as series of dict() executions. Fragment of example
+specification might look as follows:
+
+```
+SPECIFICATION =[
+ dict(name="an_union", type="union", optional=True, fields=[
+ dict(name="variant_1", type="dict"),
+ dict(name="variant_2", regexp=r'^.+$', api_version='v1beta2'),
+ ),
+ dict(name="an_union", type="dict", fields=[
+ dict(name="field_1", type="dict"),
+ dict(name="field_2", regexp=r'^.+$'),
+ ),
+ ...
+]
+```
+
+Each field should have key = "name" indicating field name. The field can be of one of the
+following types:
+
+* Dict fields: (key = "type", value="dict"):
+ Field of this type should contain nested fields in form of an array of dicts.
+ Each of the fields in the array is then expected (unless marked as optional)
+ and validated recursively. If an extra field is present in the dictionary, warning is
+ printed in log file (but the validation succeeds - see the Forward-compatibility notes)
+* Union fields (key = "type", value="union"): field of this type should contain nested
+ fields in form of an array of dicts. One of the fields (and only one) should be
+ present (unless the union is marked as optional). If more than one union field is
+ present, FieldValidationException is raised. If none of the union fields is
+ present - warning is printed in the log (see below Forward-compatibility notes).
+* Regexp-validated fields: (key = "regexp") - fields of this type are assumed to be
+ strings and they are validated with the regexp specified. Remember that the regexps
+ should ideally contain ^ at the beginning and $ at the end to make sure that
+ the whole field content is validated. Typically such regexp
+ validations should be used carefully and sparingly (see Forward-compatibility
+ notes below). Most of regexp validation should be at most r'^.+$'.
+* Custom-validated fields: (key = "custom_validation") - fields of this type are validated
+ using method specified via custom_validation field. Any exception thrown in the custom
+ validation will be turned into FieldValidationException and will cause validation to
+ fail. Such custom validations might be used to check numeric fields (including
+ ranges of values), booleans or any other types of fields.
+* API version: (key="api_version") if API version is specified, then the field will only
+ be validated when api_version used at field validator initialization matches exactly the
+ the version specified. If you want to declare fields that are available in several
+ versions of the APIs, you should specify the field as many times as many API versions
+ should be supported (each time with different API version).
+* if none of the keys ("type", "regexp", "custom_validation" - the field is not validated
+
+You can see some of the field examples in EXAMPLE_VALIDATION_SPECIFICATION.
+
+
+Forward-compatibility notes
+---------------------------
+Certain decisions are crucial to allow the client APIs to work also with future API
+versions. Since body attached is passed to the API’s call, this is entirely
+possible to pass-through any new fields in the body (for future API versions) -
+albeit without validation on the client side - they can and will still be validated
+on the server side usually.
+
+Here are the guidelines that you should follow to make validation forward-compatible:
+
+* most of the fields are not validated for their content. It's possible to use regexp
+ in some specific cases that are guaranteed not to change in the future, but for most
+ fields regexp validation should be r'^.+$' indicating check for non-emptiness
+* api_version is not validated - user can pass any future version of the api here. The API
+ version is only used to filter parameters that are marked as present in this api version
+ any new (not present in the specification) fields in the body are allowed (not verified)
+ For dictionaries, new fields can be added to dictionaries by future calls. However if an
+ unknown field in dictionary is added, a warning is logged by the client (but validation
+ remains successful). This is very nice feature to protect against typos in names.
+* For unions, newly added union variants can be added by future calls and they will
+ pass validation, however the content or presence of those fields will not be validated.
+ This means that it’s possible to send a new non-validated union field together with an
+ old validated field and this problem will not be detected by the client. In such case
+ warning will be printed.
+* When you add validator to an operator, you should also add ``validate_body`` parameter
+ (default = True) to __init__ of such operators - when it is set to False,
+ no validation should be performed. This is a safeguard for totally unpredicted and
+ backwards-incompatible changes that might sometimes occur in the APIs.
+
+"""
+
+import re
+
+from airflow import LoggingMixin, AirflowException
+
+COMPOSITE_FIELD_TYPES = ['union', 'dict']
+
+
+class GcpFieldValidationException(AirflowException):
+ """Thrown when validation finds dictionary field not valid according to specification.
+ """
+
+ def __init__(self, message):
+ super(GcpFieldValidationException, self).__init__(message)
+
+
+class GcpValidationSpecificationException(AirflowException):
+ """Thrown when validation specification is wrong.
+
+ This should only happen during development as ideally
+ specification itself should not be invalid ;) .
+ """
+
+ def __init__(self, message):
+ super(GcpValidationSpecificationException, self).__init__(message)
+
+
+def _int_greater_than_zero(value):
+ if int(value) <= 0:
+ raise GcpFieldValidationException("The available memory has to be greater than 0")
+
+
+EXAMPLE_VALIDATION_SPECIFICATION = [
+ dict(name="name", regexp="^.+$"),
+ dict(name="description", regexp="^.+$", optional=True),
+ dict(name="availableMemoryMb", custom_validation=_int_greater_than_zero,
+ optional=True),
+ dict(name="labels", optional=True, type="dict"),
+ dict(name="an_union", type="union", fields=[
+ dict(name="variant_1", regexp=r'^.+$'),
+ dict(name="variant_2", regexp=r'^.+$', api_version='v1beta2'),
+ dict(name="variant_3", type="dict", fields=[
+ dict(name="url", regexp=r'^.+$')
+ ]),
+ dict(name="variant_4")
+ ]),
+]
+
+
+class GcpBodyFieldValidator(LoggingMixin):
+ """Validates correctness of request body according to specification.
+
+ The specification can describe various type of
+ fields including custom validation, and union of fields. This validator is
+ to be reusable by various operators. See the EXAMPLE_VALIDATION_SPECIFICATION
+ for some examples and explanations of how to create specification.
+
+ :param validation_specs: dictionary describing validation specification
+ :type validation_specs: [dict]
+ :param api_version: Version of the api used (for example v1)
+ :type api_version: str
+
+ """
+ def __init__(self, validation_specs, api_version):
+ # type: ([dict], str) -> None
+ super(GcpBodyFieldValidator, self).__init__()
+ self._validation_specs = validation_specs
+ self._api_version = api_version
+
+ @staticmethod
+ def _get_field_name_with_parent(field_name, parent):
+ if parent:
+ return parent + '.' + field_name
+ return field_name
+
+ @staticmethod
+ def _sanity_checks(children_validation_specs, field_type, full_field_path,
+ regexp, custom_validation, value):
+ # type: (dict, str, str, str, function, object) -> None
+ if value is None and field_type != 'union':
+ raise GcpFieldValidationException(
+ "The required body field '{}' is missing. Please add it.".
+ format(full_field_path))
+ if regexp and field_type:
+ raise GcpValidationSpecificationException(
+ "The validation specification entry '{}' has both type and regexp. "
+ "The regexp is only allowed without type (i.e. assume type is 'str' "
+ "that can be validated with regexp)".format(full_field_path))
+ if children_validation_specs and field_type not in COMPOSITE_FIELD_TYPES:
+ raise GcpValidationSpecificationException(
+ "Nested fields are specified in field '{}' of type '{}'. "
+ "Nested fields are only allowed for fields of those types: ('{}').".
+ format(full_field_path, field_type, COMPOSITE_FIELD_TYPES))
+ if custom_validation and field_type:
+ raise GcpValidationSpecificationException(
+ "The validation specification field '{}' has both type and "
+ "custom_validation. Custom validation is only allowed without type.".
+ format(full_field_path))
+
+ @staticmethod
+ def _validate_regexp(full_field_path, regexp, value):
+ # type: (str, str, str) -> None
+ if not re.match(regexp, value):
+ # Note matching of only the beginning as we assume the regexps all-or-nothing
+ raise GcpFieldValidationException(
+ "The body field '{}' of value '{}' does not match the field "
+ "specification regexp: '{}'.".
+ format(full_field_path, value, regexp))
+
+ def _validate_dict(self, children_validation_specs, full_field_path, value):
+ # type: (dict, str, dict) -> None
+ for child_validation_spec in children_validation_specs:
+ self._validate_field(validation_spec=child_validation_spec,
+ dictionary_to_validate=value,
+ parent=full_field_path)
+ all_dict_keys = [spec['name'] for spec in children_validation_specs]
+ for field_name in value.keys():
+ if field_name not in all_dict_keys:
+ self.log.warning(
+ "The field '{}' is in the body, but is not specified in the "
+ "validation specification '{}'. "
+ "This might be because you are using newer API version and "
+ "new field names defined for that version. Then the warning "
+ "can be safely ignored, or you might want to upgrade the operator"
+ "to the version that supports the new API version.".format(
+ self._get_field_name_with_parent(field_name, full_field_path),
+ children_validation_specs))
+
+ def _validate_union(self, children_validation_specs, full_field_path,
+ dictionary_to_validate):
+ # type: (dict, str, dict) -> None
+ field_found = False
+ found_field_name = None
+ for child_validation_spec in children_validation_specs:
+ # Forcing optional so that we do not have to type optional = True
+ # in specification for all union fields
+ new_field_found = self._validate_field(
+ validation_spec=child_validation_spec,
+ dictionary_to_validate=dictionary_to_validate,
+ parent=full_field_path,
+ force_optional=True)
+ field_name = child_validation_spec['name']
+ if new_field_found and field_found:
+ raise GcpFieldValidationException(
+ "The mutually exclusive fields '{}' and '{}' belonging to the "
+ "union '{}' are both present. Please remove one".
+ format(field_name, found_field_name, full_field_path))
+ if new_field_found:
+ field_found = True
+ found_field_name = field_name
+ if not field_found:
+ self.log.warning(
+ "There is no '{}' union defined in the body {}. "
+ "Validation expected one of '{}' but could not find any. It's possible "
+ "that you are using newer API version and there is another union variant "
+ "defined for that version. Then the warning can be safely ignored, "
+ "or you might want to upgrade the operator to the version that "
+ "supports the new API version.".format(
+ full_field_path,
+ dictionary_to_validate,
+ [field['name'] for field in children_validation_specs]))
+
+ def _validate_field(self, validation_spec, dictionary_to_validate, parent=None,
+ force_optional=False):
+ """
+ Validates if field is OK.
+ :param validation_spec: specification of the field
+ :type validation_spec: dict
+ :param dictionary_to_validate: dictionary where the field should be present
+ :type dictionary_to_validate: dict
+ :param parent: full path of parent field
+ :type parent: str
+ :param force_optional: forces the field to be optional
+ (all union fields have force_optional set to True)
+ :type force_optional: bool
+ :return: True if the field is present
+ """
+ field_name = validation_spec['name']
+ field_type = validation_spec.get('type')
+ optional = validation_spec.get('optional')
+ regexp = validation_spec.get('regexp')
+ children_validation_specs = validation_spec.get('fields')
+ required_api_version = validation_spec.get('api_version')
+ custom_validation = validation_spec.get('custom_validation')
+
+ full_field_path = self._get_field_name_with_parent(field_name=field_name,
+ parent=parent)
+ if required_api_version and required_api_version != self._api_version:
+ self.log.debug(
+ "Skipping validation of the field '{}' for API version '{}' "
+ "as it is only valid for API version '{}'".
+ format(field_name, self._api_version, required_api_version))
+ return False
+ value = dictionary_to_validate.get(field_name)
+
+ if (optional or force_optional) and value is None:
+ self.log.debug("The optional field '{}' is missing. That's perfectly OK.".
+ format(full_field_path))
+ return False
+
+ # Certainly down from here the field is present (value is not None)
+ # so we should only return True from now on
+
+ self._sanity_checks(children_validation_specs=children_validation_specs,
+ field_type=field_type,
+ full_field_path=full_field_path,
+ regexp=regexp,
+ custom_validation=custom_validation,
+ value=value)
+
+ if regexp:
+ self._validate_regexp(full_field_path, regexp, value)
+ elif field_type == 'dict':
+ if not isinstance(value, dict):
+ raise GcpFieldValidationException(
+ "The field '{}' should be dictionary type according to "
+ "specification '{}' but it is '{}'".
+ format(full_field_path, validation_spec, value))
+ if children_validation_specs is None:
+ self.log.debug(
+ "The dict field '{}' has no nested fields defined in the "
+ "specification '{}'. That's perfectly ok - it's content will "
+ "not be validated."
+ .format(full_field_path, validation_spec))
+ else:
+ self._validate_dict(children_validation_specs, full_field_path, value)
+ elif field_type == 'union':
+ if not children_validation_specs:
+ raise GcpValidationSpecificationException(
+ "The union field '{}' has no nested fields "
+ "defined in specification '{}'. Unions should have at least one "
+ "nested field defined.".format(full_field_path, validation_spec))
+ self._validate_union(children_validation_specs, full_field_path,
+ dictionary_to_validate)
+ elif custom_validation:
+ try:
+ custom_validation(value)
+ except Exception as e:
+ raise GcpFieldValidationException(
+ "Error while validating custom field '{}' specified by '{}': '{}'".
+ format(full_field_path, validation_spec, e))
+ elif field_type is None:
+ self.log.debug("The type of field '{}' is not specified in '{}'. "
+ "Not validating its content.".
+ format(full_field_path, validation_spec))
+ else:
+ raise GcpValidationSpecificationException(
+ "The field '{}' is of type '{}' in specification '{}'."
+ "This type is unknown to validation!".format(
+ full_field_path, field_type, validation_spec))
+ return True
+
+ def validate(self, body_to_validate):
+ """
+ Validates if the body (dictionary) follows specification that the validator was
+ instantiated with. Raises ValidationSpecificationException or
+ ValidationFieldException in case of problems with specification or the
+ body not conforming to the specification respectively.
+ :param body_to_validate: body that must follow the specification
+ :type body_to_validate: dict
+ :return: None
+ """
+ try:
+ for validation_spec in self._validation_specs:
+ self._validate_field(validation_spec=validation_spec,
+ dictionary_to_validate=body_to_validate)
+ except GcpFieldValidationException as e:
+ raise GcpFieldValidationException(
+ "There was an error when validating: body '{}': '{}'".
+ format(body_to_validate, e))
+ all_field_names = [spec['name'] for spec in self._validation_specs
+ if spec.get('type') != 'union' and
+ spec.get('api_version') != self._api_version]
+ all_union_fields = [spec for spec in self._validation_specs
+ if spec.get('type') == 'union']
+ for union_field in all_union_fields:
+ all_field_names.extend(
+ [nested_union_spec['name'] for nested_union_spec in union_field['fields']
+ if nested_union_spec.get('type') != 'union' and
+ nested_union_spec.get('api_version') != self._api_version])
+ for field_name in body_to_validate.keys():
+ if field_name not in all_field_names:
+ self.log.warning(
+ "The field '{}' is in the body, but is not specified in the "
+ "validation specification '{}'. "
+ "This might be because you are using newer API version and "
+ "new field names defined for that version. Then the warning "
+ "can be safely ignored, or you might want to upgrade the operator"
+ "to the version that supports the new API version.".format(
+ field_name, self._validation_specs))
diff --git a/airflow/contrib/utils/sendgrid.py b/airflow/contrib/utils/sendgrid.py
index 9055c97879..ceb3718ff5 100644
--- a/airflow/contrib/utils/sendgrid.py
+++ b/airflow/contrib/utils/sendgrid.py
@@ -42,7 +42,7 @@ def send_email(to, subject, html_content, files=None,
To use this plugin:
0. include sendgrid subpackage as part of your Airflow installation, e.g.,
- pip install airflow[sendgrid]
+ pip install apache-airflow[sendgrid]
1. update [email] backend in airflow.cfg, i.e.,
[email]
email_backend = airflow.contrib.utils.sendgrid.send_email
@@ -88,7 +88,7 @@ def send_email(to, subject, html_content, files=None,
basename = os.path.basename(fname)
attachment = Attachment()
with open(fname, "rb") as f:
- attachment.content = base64.b64encode(f.read())
+ attachment.content = str(base64.b64encode(f.read()), 'utf-8')
attachment.type = mimetypes.guess_type(basename)[0]
attachment.filename = basename
attachment.disposition = "attachment"
diff --git a/airflow/default_login.py b/airflow/default_login.py
index d44dbf39ea..e423199fe0 100644
--- a/airflow/default_login.py
+++ b/airflow/default_login.py
@@ -25,11 +25,11 @@
"""
import flask_login
-from flask_login import login_required, current_user, logout_user
+from flask_login import login_required, current_user, logout_user # noqa: F401
from flask import url_for, redirect
-from airflow import settings
+from airflow import settings # noqa: F401
from airflow import models
from airflow.utils.db import provide_session
@@ -44,14 +44,17 @@ class DefaultUser(object):
def __init__(self, user):
self.user = user
+ @property
def is_active(self):
"""Required by flask_login"""
return True
+ @property
def is_authenticated(self):
"""Required by flask_login"""
return True
+ @property
def is_anonymous(self):
"""Required by flask_login"""
return False
@@ -64,9 +67,6 @@ def is_superuser(self):
"""Access all the things"""
return True
-# models.User = User # hack!
-# del User
-
@login_manager.user_loader
@provide_session
diff --git a/airflow/example_dags/example_bash_operator.py b/airflow/example_dags/example_bash_operator.py
index b2d9d14610..a5ff651d6a 100644
--- a/airflow/example_dags/example_bash_operator.py
+++ b/airflow/example_dags/example_bash_operator.py
@@ -38,8 +38,10 @@
cmd = 'ls -l'
run_this_last = DummyOperator(task_id='run_this_last', dag=dag)
+# [START howto_operator_bash]
run_this = BashOperator(
task_id='run_after_loop', bash_command='echo 1', dag=dag)
+# [END howto_operator_bash]
run_this.set_downstream(run_this_last)
for i in range(3):
@@ -50,10 +52,12 @@
dag=dag)
task.set_downstream(run_this)
+# [START howto_operator_bash_template]
task = BashOperator(
task_id='also_run_this',
bash_command='echo "run_id={{ run_id }} | dag_run={{ dag_run }}"',
dag=dag)
+# [END howto_operator_bash_template]
task.set_downstream(run_this_last)
if __name__ == "__main__":
diff --git a/airflow/example_dags/example_python_operator.py b/airflow/example_dags/example_python_operator.py
index 0ecf099e7b..6c4874fc87 100644
--- a/airflow/example_dags/example_python_operator.py
+++ b/airflow/example_dags/example_python_operator.py
@@ -26,6 +26,7 @@
import time
from pprint import pprint
+
args = {
'owner': 'airflow',
'start_date': airflow.utils.dates.days_ago(2)
@@ -36,11 +37,7 @@
schedule_interval=None)
-def my_sleeping_function(random_base):
- """This is a function that will run within the DAG execution"""
- time.sleep(random_base)
-
-
+# [START howto_operator_python]
def print_context(ds, **kwargs):
pprint(kwargs)
print(ds)
@@ -52,6 +49,14 @@ def print_context(ds, **kwargs):
provide_context=True,
python_callable=print_context,
dag=dag)
+# [END howto_operator_python]
+
+
+# [START howto_operator_python_kwargs]
+def my_sleeping_function(random_base):
+ """This is a function that will run within the DAG execution"""
+ time.sleep(random_base)
+
# Generate 10 sleeping tasks, sleeping from 0 to 4 seconds respectively
for i in range(5):
@@ -62,3 +67,4 @@ def print_context(ds, **kwargs):
dag=dag)
task.set_upstream(run_this)
+# [END howto_operator_python_kwargs]
diff --git a/airflow/exceptions.py b/airflow/exceptions.py
index 89f3d0e048..d4098c4a32 100644
--- a/airflow/exceptions.py
+++ b/airflow/exceptions.py
@@ -47,6 +47,17 @@ class AirflowSensorTimeout(AirflowException):
pass
+class AirflowRescheduleException(AirflowException):
+ """
+ Raise when the task should be re-scheduled at a later time.
+
+ :param reschedule_date: The date when the task should be rescheduled
+ :type reschedule: datetime
+ """
+ def __init__(self, reschedule_date):
+ self.reschedule_date = reschedule_date
+
+
class AirflowTaskTimeout(AirflowException):
pass
diff --git a/airflow/executors/base_executor.py b/airflow/executors/base_executor.py
index 04c9088401..a989dc4408 100644
--- a/airflow/executors/base_executor.py
+++ b/airflow/executors/base_executor.py
@@ -75,7 +75,7 @@ def queue_task_instance(
# cfg_path is needed to propagate the config values if using impersonation
# (run_as_user), given that there are different code paths running tasks.
# For a long term solution we need to address AIRFLOW-1986
- command = task_instance.command(
+ command = task_instance.command_as_list(
local=True,
mark_success=mark_success,
ignore_all_deps=ignore_all_deps,
@@ -150,7 +150,7 @@ def heartbeat(self):
self.sync()
def change_state(self, key, state):
- print("popping: {}".format(key))
+ self.log.debug("Changing state: {}".format(key))
self.running.pop(key)
self.event_buffer[key] = state
@@ -193,7 +193,7 @@ def execute_async(self,
def end(self): # pragma: no cover
"""
- This method is called when the caller is done submitting job and is
+ This method is called when the caller is done submitting job and
wants to wait synchronously for the job submitted previously to be
all done.
"""
diff --git a/airflow/executors/celery_executor.py b/airflow/executors/celery_executor.py
index 6cfd2d3769..0de48b4d39 100644
--- a/airflow/executors/celery_executor.py
+++ b/airflow/executors/celery_executor.py
@@ -17,21 +17,25 @@
# specific language governing permissions and limitations
# under the License.
+import math
+import os
import subprocess
import time
-import os
+import traceback
+from multiprocessing import Pool, cpu_count
from celery import Celery
from celery import states as celery_states
+from airflow import configuration
from airflow.config_templates.default_celery import DEFAULT_CELERY_CONFIG
from airflow.exceptions import AirflowException
from airflow.executors.base_executor import BaseExecutor
-from airflow import configuration
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.utils.module_loading import import_string
-PARALLELISM = configuration.conf.get('core', 'PARALLELISM')
+# Make it constant for unit test.
+CELERY_FETCH_ERR_MSG_HEADER = 'Error fetching Celery task state'
'''
To start the celery worker, run the command:
@@ -56,7 +60,7 @@ def execute_command(command):
log.info("Executing command in Celery: %s", command)
env = os.environ.copy()
try:
- subprocess.check_call(command, shell=True, stderr=subprocess.STDOUT,
+ subprocess.check_call(command, stderr=subprocess.STDOUT,
close_fds=True, env=env)
except subprocess.CalledProcessError as e:
log.exception('execute_command encountered a CalledProcessError')
@@ -65,6 +69,42 @@ def execute_command(command):
raise AirflowException('Celery command failed')
+class ExceptionWithTraceback(object):
+ """
+ Wrapper class used to propogate exceptions to parent processes from subprocesses.
+ :param exception: The exception to wrap
+ :type exception: Exception
+ :param traceback: The stacktrace to wrap
+ :type traceback: str
+ """
+
+ def __init__(self, exception, exception_traceback):
+ self.exception = exception
+ self.traceback = exception_traceback
+
+
+def fetch_celery_task_state(celery_task):
+ """
+ Fetch and return the state of the given celery task. The scope of this function is
+ global so that it can be called by subprocesses in the pool.
+ :param celery_task: a tuple of the Celery task key and the async Celery object used
+ to fetch the task's state
+ :type celery_task: (str, celery.result.AsyncResult)
+ :return: a tuple of the Celery task key and the Celery state of the task
+ :rtype: (str, str)
+ """
+
+ try:
+ # Accessing state property of celery task will make actual network request
+ # to get the current state of the task.
+ res = (celery_task[0], celery_task[1].state)
+ except Exception as e:
+ exception_traceback = "Celery Task ID: {}\n{}".format(celery_task[0],
+ traceback.format_exc())
+ res = ExceptionWithTraceback(e, exception_traceback)
+ return res
+
+
class CeleryExecutor(BaseExecutor):
"""
CeleryExecutor is recommended for production use of Airflow. It allows
@@ -74,10 +114,27 @@ class CeleryExecutor(BaseExecutor):
vast amounts of messages, while providing operations with the tools
required to maintain such a system.
"""
- def start(self):
+
+ def __init__(self):
+ super(CeleryExecutor, self).__init__()
+
+ # Celery doesn't support querying the state of multiple tasks in parallel
+ # (which can become a bottleneck on bigger clusters) so we use
+ # a multiprocessing pool to speed this up.
+ # How many worker processes are created for checking celery task state.
+ self._sync_parallelism = configuration.getint('celery', 'SYNC_PARALLELISM')
+ if self._sync_parallelism == 0:
+ self._sync_parallelism = max(1, cpu_count() - 1)
+
+ self._sync_pool = None
self.tasks = {}
self.last_state = {}
+ def start(self):
+ self.log.debug(
+ 'Starting Celery Executor using {} processes for syncing'.format(
+ self._sync_parallelism))
+
def execute_async(self, key, command,
queue=DEFAULT_CELERY_CONFIG['task_default_queue'],
executor_config=None):
@@ -87,11 +144,48 @@ def execute_async(self, key, command,
args=[command], queue=queue)
self.last_state[key] = celery_states.PENDING
+ def _num_tasks_per_process(self):
+ """
+ How many Celery tasks should be sent to each worker process.
+ :return: Number of tasks that should be used per process
+ :rtype: int
+ """
+ return max(1,
+ int(math.ceil(1.0 * len(self.tasks) / self._sync_parallelism)))
+
def sync(self):
- self.log.debug("Inquiring about %s celery task(s)", len(self.tasks))
- for key, async in list(self.tasks.items()):
+ num_processes = min(len(self.tasks), self._sync_parallelism)
+ if num_processes == 0:
+ self.log.debug("No task to query celery, skipping sync")
+ return
+
+ self.log.debug("Inquiring about %s celery task(s) using %s processes",
+ len(self.tasks), num_processes)
+
+ # Recreate the process pool each sync in case processes in the pool die
+ self._sync_pool = Pool(processes=num_processes)
+
+ # Use chunking instead of a work queue to reduce context switching since tasks are
+ # roughly uniform in size
+ chunksize = self._num_tasks_per_process()
+
+ self.log.debug("Waiting for inquiries to complete...")
+ task_keys_to_states = self._sync_pool.map(
+ fetch_celery_task_state,
+ self.tasks.items(),
+ chunksize=chunksize)
+ self._sync_pool.close()
+ self._sync_pool.join()
+ self.log.debug("Inquiries completed.")
+
+ for key_and_state in task_keys_to_states:
+ if isinstance(key_and_state, ExceptionWithTraceback):
+ self.log.error(
+ CELERY_FETCH_ERR_MSG_HEADER + ", ignoring it:{}\n{}\n".format(
+ key_and_state.exception, key_and_state.traceback))
+ continue
+ key, state = key_and_state
try:
- state = async.state
if self.last_state[key] != state:
if state == celery_states.SUCCESS:
self.success(key)
@@ -106,16 +200,15 @@ def sync(self):
del self.tasks[key]
del self.last_state[key]
else:
- self.log.info("Unexpected state: %s", async.state)
- self.last_state[key] = async.state
- except Exception as e:
- self.log.error("Error syncing the celery executor, ignoring it:")
- self.log.exception(e)
+ self.log.info("Unexpected state: " + state)
+ self.last_state[key] = state
+ except Exception:
+ self.log.exception("Error syncing the Celery executor, ignoring it.")
def end(self, synchronous=False):
if synchronous:
while any([
- async.state not in celery_states.READY_STATES
- for async in self.tasks.values()]):
+ task.state not in celery_states.READY_STATES
+ for task in self.tasks.values()]):
time.sleep(5)
self.sync()
diff --git a/airflow/executors/dask_executor.py b/airflow/executors/dask_executor.py
index a6ba677f8b..23c0b636b6 100644
--- a/airflow/executors/dask_executor.py
+++ b/airflow/executors/dask_executor.py
@@ -43,7 +43,7 @@ def __init__(self, cluster_address=None):
super(DaskExecutor, self).__init__(parallelism=0)
def start(self):
- if (self.tls_ca) or (self.tls_key) or (self.tls_cert):
+ if self.tls_ca or self.tls_key or self.tls_cert:
from distributed.security import Security
security = Security(
tls_client_key=self.tls_key,
diff --git a/airflow/executors/local_executor.py b/airflow/executors/local_executor.py
index 0c85262324..3e622da053 100644
--- a/airflow/executors/local_executor.py
+++ b/airflow/executors/local_executor.py
@@ -77,14 +77,13 @@ def execute_work(self, key, command):
:param key: the key to identify the TI
:type key: Tuple(dag_id, task_id, execution_date)
:param command: the command to execute
- :type command: string
+ :type command: str
"""
if key is None:
return
self.log.info("%s running %s", self.__class__.__name__, command)
- command = "exec bash -c '{0}'".format(command)
try:
- subprocess.check_call(command, shell=True, close_fds=True)
+ subprocess.check_call(command, close_fds=True)
state = State.SUCCESS
except subprocess.CalledProcessError as e:
state = State.FAILED
@@ -147,7 +146,7 @@ def execute_async(self, key, command):
:param key: the key to identify the TI
:type key: Tuple(dag_id, task_id, execution_date)
:param command: the command to execute
- :type command: string
+ :type command: str
"""
local_worker = LocalWorker(self.executor.result_queue)
local_worker.key = key
@@ -192,7 +191,7 @@ def execute_async(self, key, command):
:param key: the key to identify the TI
:type key: Tuple(dag_id, task_id, execution_date)
:param command: the command to execute
- :type command: string
+ :type command: str
"""
self.executor.queue.put((key, command))
diff --git a/airflow/executors/sequential_executor.py b/airflow/executors/sequential_executor.py
index 9c0d8ecf0c..1542e3318e 100644
--- a/airflow/executors/sequential_executor.py
+++ b/airflow/executors/sequential_executor.py
@@ -45,7 +45,7 @@ def sync(self):
self.log.info("Executing command: %s", command)
try:
- subprocess.check_call(command, shell=True, close_fds=True)
+ subprocess.check_call(command, close_fds=True)
self.change_state(key, State.SUCCESS)
except subprocess.CalledProcessError as e:
self.change_state(key, State.FAILED)
diff --git a/airflow/hooks/S3_hook.py b/airflow/hooks/S3_hook.py
index b4f3ac3b03..35d26fd2b5 100644
--- a/airflow/hooks/S3_hook.py
+++ b/airflow/hooks/S3_hook.py
@@ -16,6 +16,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+from botocore.exceptions import ClientError
from airflow.exceptions import AirflowException
from airflow.contrib.hooks.aws_hook import AwsHook
@@ -42,7 +43,7 @@ def parse_s3_url(s3url):
else:
bucket_name = parsed_url.netloc
key = parsed_url.path.strip('/')
- return (bucket_name, key)
+ return bucket_name, key
def check_for_bucket(self, bucket_name):
"""
@@ -54,7 +55,8 @@ def check_for_bucket(self, bucket_name):
try:
self.get_conn().head_bucket(Bucket=bucket_name)
return True
- except:
+ except ClientError as e:
+ self.log.info(e.response["Error"]["Message"])
return False
def get_bucket(self, bucket_name):
@@ -70,6 +72,13 @@ def get_bucket(self, bucket_name):
def check_for_prefix(self, bucket_name, prefix, delimiter):
"""
Checks that a prefix exists in a bucket
+
+ :param bucket_name: the name of the bucket
+ :type bucket_name: str
+ :param prefix: a key prefix
+ :type prefix: str
+ :param delimiter: the delimiter marks key hierarchy.
+ :type delimiter: str
"""
prefix = prefix + delimiter if prefix[-1] != delimiter else prefix
prefix_split = re.split(r'(\w+[{d}])$'.format(d=delimiter), prefix, 1)
@@ -168,7 +177,8 @@ def check_for_key(self, key, bucket_name=None):
try:
self.get_conn().head_object(Bucket=bucket_name, Key=key)
return True
- except:
+ except ClientError as e:
+ self.log.info(e.response["Error"]["Message"])
return False
def get_key(self, key, bucket_name=None):
@@ -203,8 +213,8 @@ def read_key(self, key, bucket_name=None):
def select_key(self, key, bucket_name=None,
expression='SELECT * FROM S3Object',
expression_type='SQL',
- input_serialization={'CSV': {}},
- output_serialization={'CSV': {}}):
+ input_serialization=None,
+ output_serialization=None):
"""
Reads a key with S3 Select.
@@ -227,6 +237,10 @@ def select_key(self, key, bucket_name=None,
For more details about S3 Select parameters:
http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.select_object_content
"""
+ if input_serialization is None:
+ input_serialization = {'CSV': {}}
+ if output_serialization is None:
+ output_serialization = {'CSV': {}}
if not bucket_name:
(bucket_name, key) = self.parse_s3_url(key)
@@ -246,6 +260,13 @@ def check_for_wildcard_key(self,
wildcard_key, bucket_name=None, delimiter=''):
"""
Checks that a key matching a wildcard expression exists in a bucket
+
+ :param wildcard_key: the path to the key
+ :type wildcard_key: str
+ :param bucket_name: the name of the bucket
+ :type bucket_name: str
+ :param delimiter: the delimiter marks key hierarchy
+ :type delimiter: str
"""
return self.get_wildcard_key(wildcard_key=wildcard_key,
bucket_name=bucket_name,
@@ -259,6 +280,8 @@ def get_wildcard_key(self, wildcard_key, bucket_name=None, delimiter=''):
:type wildcard_key: str
:param bucket_name: the name of the bucket
:type bucket_name: str
+ :param delimiter: the delimiter marks key hierarchy
+ :type delimiter: str
"""
if not bucket_name:
(bucket_name, wildcard_key) = self.parse_s3_url(wildcard_key)
@@ -319,7 +342,7 @@ def load_string(self,
This is provided as a convenience to drop a string in S3. It uses the
boto infrastructure to ship a file to s3.
- :param string_data: string to set as content for the key.
+ :param string_data: str to set as content for the key.
:type string_data: str
:param key: S3 key that will point to the file
:type key: str
@@ -377,3 +400,89 @@ def load_bytes(self,
client = self.get_conn()
client.upload_fileobj(filelike_buffer, bucket_name, key, ExtraArgs=extra_args)
+
+ def copy_object(self,
+ source_bucket_key,
+ dest_bucket_key,
+ source_bucket_name=None,
+ dest_bucket_name=None,
+ source_version_id=None):
+ """
+ Creates a copy of an object that is already stored in S3.
+
+ Note: the S3 connection used here needs to have access to both
+ source and destination bucket/key.
+
+ :param source_bucket_key: The key of the source object.
+
+ It can be either full s3:// style url or relative path from root level.
+
+ When it's specified as a full s3:// url, please omit source_bucket_name.
+ :type source_bucket_key: str
+ :param dest_bucket_key: The key of the object to copy to.
+
+ The convention to specify `dest_bucket_key` is the same
+ as `source_bucket_key`.
+ :type dest_bucket_key: str
+ :param source_bucket_name: Name of the S3 bucket where the source object is in.
+
+ It should be omitted when `source_bucket_key` is provided as a full s3:// url.
+ :type source_bucket_name: str
+ :param dest_bucket_name: Name of the S3 bucket to where the object is copied.
+
+ It should be omitted when `dest_bucket_key` is provided as a full s3:// url.
+ :type dest_bucket_name: str
+ :param source_version_id: Version ID of the source object (OPTIONAL)
+ :type source_version_id: str
+ """
+
+ if dest_bucket_name is None:
+ dest_bucket_name, dest_bucket_key = self.parse_s3_url(dest_bucket_key)
+ else:
+ parsed_url = urlparse(dest_bucket_key)
+ if parsed_url.scheme != '' or parsed_url.netloc != '':
+ raise AirflowException('If dest_bucket_name is provided, ' +
+ 'dest_bucket_key should be relative path ' +
+ 'from root level, rather than a full s3:// url')
+
+ if source_bucket_name is None:
+ source_bucket_name, source_bucket_key = self.parse_s3_url(source_bucket_key)
+ else:
+ parsed_url = urlparse(source_bucket_key)
+ if parsed_url.scheme != '' or parsed_url.netloc != '':
+ raise AirflowException('If source_bucket_name is provided, ' +
+ 'source_bucket_key should be relative path ' +
+ 'from root level, rather than a full s3:// url')
+
+ CopySource = {'Bucket': source_bucket_name,
+ 'Key': source_bucket_key,
+ 'VersionId': source_version_id}
+ response = self.get_conn().copy_object(Bucket=dest_bucket_name,
+ Key=dest_bucket_key,
+ CopySource=CopySource)
+ return response
+
+ def delete_objects(self,
+ bucket,
+ keys):
+ """
+ :param bucket: Name of the bucket in which you are going to delete object(s)
+ :type bucket: str
+ :param keys: The key(s) to delete from S3 bucket.
+
+ When ``keys`` is a string, it's supposed to be the key name of
+ the single object to delete.
+
+ When ``keys`` is a list, it's supposed to be the list of the
+ keys to delete.
+ :type keys: str or list
+ """
+ if isinstance(keys, list):
+ keys = keys
+ else:
+ keys = [keys]
+
+ delete_dict = {"Objects": [{"Key": k} for k in keys]}
+ response = self.get_conn().delete_objects(Bucket=bucket,
+ Delete=delete_dict)
+ return response
diff --git a/airflow/hooks/__init__.py b/airflow/hooks/__init__.py
index 1f1c40f16a..44a434a07e 100644
--- a/airflow/hooks/__init__.py
+++ b/airflow/hooks/__init__.py
@@ -18,79 +18,15 @@
# under the License.
-import os as _os
import sys
-# ------------------------------------------------------------------------
-#
-# #TODO #FIXME Airflow 2.0
-#
-# Old import machinary below.
-#
-# This is deprecated but should be kept until Airflow 2.0
-# for compatibility.
-#
-# ------------------------------------------------------------------------
-
# Imports the hooks dynamically while keeping the package API clean,
# abstracting the underlying modules
-
-_hooks = {
- 'base_hook': ['BaseHook'],
- 'hive_hooks': [
- 'HiveCliHook',
- 'HiveMetastoreHook',
- 'HiveServer2Hook',
- ],
- 'hdfs_hook': ['HDFSHook'],
- 'webhdfs_hook': ['WebHDFSHook'],
- 'pig_hook': ['PigCliHook'],
- 'mysql_hook': ['MySqlHook'],
- 'postgres_hook': ['PostgresHook'],
- 'presto_hook': ['PrestoHook'],
- 'samba_hook': ['SambaHook'],
- 'sqlite_hook': ['SqliteHook'],
- 'S3_hook': ['S3Hook'],
- 'zendesk_hook': ['ZendeskHook'],
- 'http_hook': ['HttpHook'],
- 'druid_hook': [
- 'DruidHook',
- 'DruidDbApiHook',
- ],
- 'jdbc_hook': ['JdbcHook'],
- 'dbapi_hook': ['DbApiHook'],
- 'mssql_hook': ['MsSqlHook'],
- 'oracle_hook': ['OracleHook'],
- 'slack_hook': ['SlackHook'],
-}
-
-
-if not _os.environ.get('AIRFLOW_USE_NEW_IMPORTS', False):
- from airflow.utils.helpers import AirflowImporter
- airflow_importer = AirflowImporter(sys.modules[__name__], _hooks)
-
-
def _integrate_plugins():
"""Integrate plugins to the context"""
from airflow.plugins_manager import hooks_modules
for hooks_module in hooks_modules:
sys.modules[hooks_module.__name__] = hooks_module
globals()[hooks_module._name] = hooks_module
-
- ##########################################################
- # TODO FIXME Remove in Airflow 2.0
-
- if not _os.environ.get('AIRFLOW_USE_NEW_IMPORTS', False):
- from zope.deprecation import deprecated as _deprecated
- for _hook in hooks_module._objects:
- hook_name = _hook.__name__
- globals()[hook_name] = _hook
- _deprecated(
- hook_name,
- "Importing plugin hook '{i}' directly from "
- "'airflow.hooks' has been deprecated. Please "
- "import from 'airflow.hooks.[plugin_module]' "
- "instead. Support for direct imports will be dropped "
- "entirely in Airflow 2.0.".format(i=hook_name))
diff --git a/airflow/hooks/dbapi_hook.py b/airflow/hooks/dbapi_hook.py
index 5b50ade34e..bb2c20c188 100644
--- a/airflow/hooks/dbapi_hook.py
+++ b/airflow/hooks/dbapi_hook.py
@@ -163,10 +163,11 @@ def run(self, sql, autocommit=False, parameters=None):
for s in sql:
if sys.version_info[0] < 3:
s = s.encode('utf-8')
- self.log.info(s)
if parameters is not None:
+ self.log.info("{} with parameters {}".format(s, parameters))
cur.execute(s, parameters)
else:
+ self.log.info(s)
cur.execute(s)
# If autocommit was set to False for db that supports autocommit,
diff --git a/airflow/hooks/druid_hook.py b/airflow/hooks/druid_hook.py
index ef4f2338d2..75111fad2e 100644
--- a/airflow/hooks/druid_hook.py
+++ b/airflow/hooks/druid_hook.py
@@ -35,9 +35,10 @@ class DruidHook(BaseHook):
:param druid_ingest_conn_id: The connection id to the Druid overlord machine
which accepts index jobs
- :type druid_ingest_conn_id: string
+ :type druid_ingest_conn_id: str
:param timeout: The interval between polling
- the Druid job for the status of the ingestion job
+ the Druid job for the status of the ingestion job.
+ Must be greater than or equal to 1
:type timeout: int
:param max_ingestion_time: The maximum ingestion time before assuming the job failed
:type max_ingestion_time: int
@@ -53,6 +54,9 @@ def __init__(
self.max_ingestion_time = max_ingestion_time
self.header = {'content-type': 'application/json'}
+ if self.timeout < 1:
+ raise ValueError("Druid timeout should be equal or greater than 1")
+
def get_conn_url(self):
conn = self.get_connection(self.druid_ingest_conn_id)
host = conn.host
@@ -64,8 +68,8 @@ def get_conn_url(self):
def submit_indexing_job(self, json_index_spec):
url = self.get_conn_url()
- req_index = requests.post(url, data=json_index_spec, headers=self.header)
- if (req_index.status_code != 200):
+ req_index = requests.post(url, json=json_index_spec, headers=self.header)
+ if req_index.status_code != 200:
raise AirflowException('Did not get 200 when '
'submitting the Druid job to {}'.format(url))
@@ -81,8 +85,6 @@ def submit_indexing_job(self, json_index_spec):
self.log.info("Job still running for %s seconds...", sec)
- sec = sec + 1
-
if self.max_ingestion_time and sec > self.max_ingestion_time:
# ensure that the job gets killed if the max ingestion time is exceeded
requests.post("{0}/{1}/shutdown".format(url, druid_task_id))
@@ -91,6 +93,8 @@ def submit_indexing_job(self, json_index_spec):
time.sleep(self.timeout)
+ sec = sec + self.timeout
+
status = req_status.json()['status']['status']
if status == 'RUNNING':
running = True
diff --git a/airflow/hooks/hdfs_hook.py b/airflow/hooks/hdfs_hook.py
index 3c9136b1fd..b00bab42ce 100644
--- a/airflow/hooks/hdfs_hook.py
+++ b/airflow/hooks/hdfs_hook.py
@@ -39,9 +39,9 @@ class HDFSHook(BaseHook):
Interact with HDFS. This class is a wrapper around the snakebite library.
:param hdfs_conn_id: Connection id to fetch connection info
- :type conn_id: string
+ :type conn_id: str
:param proxy_user: effective user for HDFS operations
- :type proxy_user: string
+ :type proxy_user: str
:param autoconfig: use snakebite's automatically configured client
:type autoconfig: bool
"""
diff --git a/airflow/hooks/hive_hooks.py b/airflow/hooks/hive_hooks.py
index 93e1f45fe6..178606aa4e 100644
--- a/airflow/hooks/hive_hooks.py
+++ b/airflow/hooks/hive_hooks.py
@@ -21,30 +21,40 @@
import contextlib
import os
-
-from six.moves import zip
-from past.builtins import basestring, unicode
-
-import unicodecsv as csv
import re
-import six
import subprocess
import time
from collections import OrderedDict
from tempfile import NamedTemporaryFile
+
import hmsclient
+import six
+import unicodecsv as csv
+from past.builtins import basestring
+from past.builtins import unicode
+from six.moves import zip
-from airflow import configuration as conf
+from airflow import configuration
from airflow.exceptions import AirflowException
from airflow.hooks.base_hook import BaseHook
-from airflow.utils.helpers import as_flattened_list
+from airflow.security import utils
from airflow.utils.file import TemporaryDirectory
-from airflow import configuration
-import airflow.security.utils as utils
+from airflow.utils.helpers import as_flattened_list
+from airflow.utils.operator_helpers import AIRFLOW_VAR_NAME_FORMAT_MAPPING
HIVE_QUEUE_PRIORITIES = ['VERY_HIGH', 'HIGH', 'NORMAL', 'LOW', 'VERY_LOW']
+def get_context_from_env_var():
+ """
+ Extract context from env variable, e.g. dag_id, task_id and execution_date,
+ so that they can be used inside BashOperator and PythonOperator.
+ :return: The context of interest.
+ """
+ return {format_map['default']: os.environ.get(format_map['env_var_format'], '')
+ for format_map in AIRFLOW_VAR_NAME_FORMAT_MAPPING.values()}
+
+
class HiveCliHook(BaseHook):
"""Simple wrapper around the hive CLI.
@@ -62,13 +72,13 @@ class HiveCliHook(BaseHook):
connection string as is.
:param mapred_queue: queue used by the Hadoop Scheduler (Capacity or Fair)
- :type mapred_queue: string
+ :type mapred_queue: str
:param mapred_queue_priority: priority within the job queue.
Possible settings include: VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW
- :type mapred_queue_priority: string
+ :type mapred_queue_priority: str
:param mapred_job_name: This name will appear in the jobtracker.
This can make monitoring easier.
- :type mapred_job_name: string
+ :type mapred_job_name: str
"""
def __init__(
@@ -92,8 +102,8 @@ def __init__(
"Invalid Mapred Queue Priority. Valid values are: "
"{}".format(', '.join(HIVE_QUEUE_PRIORITIES)))
- self.mapred_queue = mapred_queue or conf.get('hive',
- 'default_hive_mapred_queue')
+ self.mapred_queue = mapred_queue or configuration.get('hive',
+ 'default_hive_mapred_queue')
self.mapred_queue_priority = mapred_queue_priority
self.mapred_job_name = mapred_job_name
@@ -126,6 +136,7 @@ def _prepare_cli_cmd(self):
jdbc_url += ";auth=" + self.auth
jdbc_url = jdbc_url.format(**locals())
+ jdbc_url = '"{}"'.format(jdbc_url)
cmd_extra += ['-u', jdbc_url]
if conn.login:
@@ -137,7 +148,8 @@ def _prepare_cli_cmd(self):
return [hive_bin] + cmd_extra + hive_params_list
- def _prepare_hiveconf(self, d):
+ @staticmethod
+ def _prepare_hiveconf(d):
"""
This function prepares a list of hiveconf params
from a dictionary of key value pairs.
@@ -184,10 +196,15 @@ def run_cli(self, hql, schema=None, verbose=True, hive_conf=None):
with TemporaryDirectory(prefix='airflow_hiveop_') as tmp_dir:
with NamedTemporaryFile(dir=tmp_dir) as f:
+ hql = hql + '\n'
f.write(hql.encode('UTF-8'))
f.flush()
hive_cmd = self._prepare_cli_cmd()
- hive_conf_params = self._prepare_hiveconf(hive_conf)
+ env_context = get_context_from_env_var()
+ # Only extend the hive_conf if it is defined.
+ if hive_conf:
+ env_context.update(hive_conf)
+ hive_conf_params = self._prepare_hiveconf(env_context)
if self.mapred_queue:
hive_conf_params.extend(
['-hiveconf',
@@ -307,7 +324,7 @@ def load_df(
:type field_dict: OrderedDict
:param delimiter: field delimiter in the file
:type delimiter: str
- :param encoding: string encoding to use when writing DataFrame to file
+ :param encoding: str encoding to use when writing DataFrame to file
:type encoding: str
:param pandas_kwargs: passed to DataFrame.to_csv
:type pandas_kwargs: dict
@@ -519,13 +536,13 @@ def check_for_partition(self, schema, table, partition):
Checks whether a partition exists
:param schema: Name of hive schema (database) @table belongs to
- :type schema: string
+ :type schema: str
:param table: Name of hive table @partition belongs to
- :type schema: string
+ :type schema: str
:partition: Expression that matches the partitions to check for
(eg `a = 'b' AND c = 'd'`)
- :type schema: string
- :rtype: boolean
+ :type schema: str
+ :rtype: bool
>>> hh = HiveMetastoreHook()
>>> t = 'static_babynames_partitioned'
@@ -546,12 +563,12 @@ def check_for_named_partition(self, schema, table, partition_name):
Checks whether a partition with a given name exists
:param schema: Name of hive schema (database) @table belongs to
- :type schema: string
+ :type schema: str
:param table: Name of hive table @partition belongs to
- :type schema: string
+ :type schema: str
:partition: Name of the partitions to check for (eg `a=b/c=d`)
- :type schema: string
- :rtype: boolean
+ :type schema: str
+ :rtype: bool
>>> hh = HiveMetastoreHook()
>>> t = 'static_babynames_partitioned'
@@ -635,7 +652,7 @@ def _get_max_partition_from_part_specs(part_specs, partition_key, filter_map):
:param part_specs: list of partition specs.
:type part_specs: list
:param partition_key: partition key name.
- :type partition_key: string
+ :type partition_key: str
:param filter_map: partition_key:partition_value map used for partition filtering,
e.g. {'key1': 'value1', 'key2': 'value2'}.
Only partitions matching all partition_key:partition_value
@@ -675,11 +692,11 @@ def max_partition(self, schema, table_name, field=None, filter_map=None):
filter out partitions.
:param schema: schema name.
- :type schema: string
+ :type schema: str
:param table_name: table name.
- :type table_name: string
+ :type table_name: str
:param field: partition key to get max partition from.
- :type field: string
+ :type field: str
:param filter_map: partition_key:partition_value map used for partition filtering.
:type filter_map: map
@@ -772,7 +789,7 @@ def get_conn(self, schema=None):
username=db.login or username,
database=schema or db.schema or 'default')
- def _get_results(self, hql, schema='default', fetch_size=None):
+ def _get_results(self, hql, schema='default', fetch_size=None, hive_conf=None):
from pyhive.exc import ProgrammingError
if isinstance(hql, basestring):
hql = [hql]
@@ -780,12 +797,22 @@ def _get_results(self, hql, schema='default', fetch_size=None):
with contextlib.closing(self.get_conn(schema)) as conn, \
contextlib.closing(conn.cursor()) as cur:
cur.arraysize = fetch_size or 1000
+
+ env_context = get_context_from_env_var()
+ if hive_conf:
+ env_context.update(hive_conf)
+ for k, v in env_context.items():
+ cur.execute("set {}={}".format(k, v))
+
for statement in hql:
cur.execute(statement)
# we only get results of statements that returns
lowered_statement = statement.lower().strip()
if (lowered_statement.startswith('select') or
- lowered_statement.startswith('with')):
+ lowered_statement.startswith('with') or
+ lowered_statement.startswith('show') or
+ (lowered_statement.startswith('set') and
+ '=' not in lowered_statement)):
description = [c for c in cur.description]
if previous_description and previous_description != description:
message = '''The statements are producing different descriptions:
@@ -805,8 +832,17 @@ def _get_results(self, hql, schema='default', fetch_size=None):
except ProgrammingError:
self.log.debug("get_results returned no records")
- def get_results(self, hql, schema='default', fetch_size=None):
- results_iter = self._get_results(hql, schema, fetch_size=fetch_size)
+ def get_results(self, hql, schema='default', fetch_size=None, hive_conf=None):
+ """
+ Get results of the provided hql in target schema.
+ :param hql: hql to be executed.
+ :param schema: target schema, default to 'default'.
+ :param fetch_size max size of result to fetch.
+ :param hive_conf: hive_conf to execute alone with the hql.
+ :return: results of hql execution.
+ """
+ results_iter = self._get_results(hql, schema,
+ fetch_size=fetch_size, hive_conf=hive_conf)
header = next(results_iter)
results = {
'data': list(results_iter),
@@ -822,9 +858,23 @@ def to_csv(
delimiter=',',
lineterminator='\r\n',
output_header=True,
- fetch_size=1000):
-
- results_iter = self._get_results(hql, schema, fetch_size=fetch_size)
+ fetch_size=1000,
+ hive_conf=None):
+ """
+ Execute hql in target schema and write results to a csv file.
+ :param hql: hql to be executed.
+ :param csv_filepath: filepath of csv to write results into.
+ :param schema: target schema, , default to 'default'.
+ :param delimiter: delimiter of the csv file.
+ :param lineterminator: lineterminator of the csv file.
+ :param output_header: header of the csv file.
+ :param fetch_size: number of result rows to write into the csv file.
+ :param hive_conf: hive_conf to execute alone with the hql.
+ :return:
+ """
+
+ results_iter = self._get_results(hql, schema,
+ fetch_size=fetch_size, hive_conf=hive_conf)
header = next(results_iter)
message = None
diff --git a/airflow/hooks/http_hook.py b/airflow/hooks/http_hook.py
index c449fe0c15..1e0c7b3058 100644
--- a/airflow/hooks/http_hook.py
+++ b/airflow/hooks/http_hook.py
@@ -70,7 +70,11 @@ def get_conn(self, headers=None):
if conn.login:
session.auth = (conn.login, conn.password)
if conn.extra:
- session.headers.update(conn.extra_dejson)
+ try:
+ session.headers.update(conn.extra_dejson)
+ except TypeError:
+ self.log.warn('Connection to {} has invalid extra field.'.format(
+ conn.host))
if headers:
session.headers.update(headers)
@@ -94,7 +98,11 @@ def run(self, endpoint, data=None, headers=None, extra_options=None):
session = self.get_conn(headers)
- url = self.base_url + endpoint
+ if not self.base_url.endswith('/') and not endpoint.startswith('/'):
+ url = self.base_url + '/' + endpoint
+ else:
+ url = self.base_url + endpoint
+
req = None
if self.method == 'GET':
# GET uses params
diff --git a/airflow/hooks/mysql_hook.py b/airflow/hooks/mysql_hook.py
index c02c0f43b5..51b53c1f8f 100644
--- a/airflow/hooks/mysql_hook.py
+++ b/airflow/hooks/mysql_hook.py
@@ -121,7 +121,7 @@ def bulk_dump(self, table, tmp_file):
def _serialize_cell(cell, conn):
"""
MySQLdb converts an argument to a literal
- when passing those seperately to execute. Hence, this method does nothing.
+ when passing those separately to execute. Hence, this method does nothing.
:param cell: The cell to insert into the table
:type cell: object
diff --git a/airflow/hooks/pig_hook.py b/airflow/hooks/pig_hook.py
index a3836b1ba3..fcfcc7fef2 100644
--- a/airflow/hooks/pig_hook.py
+++ b/airflow/hooks/pig_hook.py
@@ -55,7 +55,7 @@ def run_cli(self, pig, verbose=True):
with TemporaryDirectory(prefix='airflow_pigop_') as tmp_dir:
with NamedTemporaryFile(dir=tmp_dir) as f:
- f.write(pig)
+ f.write(pig.encode('utf-8'))
f.flush()
fname = f.name
pig_bin = 'pig'
@@ -76,8 +76,8 @@ def run_cli(self, pig, verbose=True):
close_fds=True)
self.sp = sp
stdout = ''
- for line in iter(sp.stdout.readline, ''):
- stdout += line
+ for line in iter(sp.stdout.readline, b''):
+ stdout += line.decode('utf-8')
if verbose:
self.log.info(line.strip())
sp.wait()
diff --git a/airflow/hooks/presto_hook.py b/airflow/hooks/presto_hook.py
index d6b5293fc0..c7ebd8f5dc 100644
--- a/airflow/hooks/presto_hook.py
+++ b/airflow/hooks/presto_hook.py
@@ -56,7 +56,8 @@ def get_conn(self):
def _strip_sql(sql):
return sql.strip().rstrip(';')
- def _get_pretty_exception_message(self, e):
+ @staticmethod
+ def _get_pretty_exception_message(e):
"""
Parses some DatabaseError to provide a better error message
"""
diff --git a/airflow/hooks/slack_hook.py b/airflow/hooks/slack_hook.py
index 17006fc9a5..953ce319b1 100644
--- a/airflow/hooks/slack_hook.py
+++ b/airflow/hooks/slack_hook.py
@@ -34,9 +34,9 @@ def __init__(self, token=None, slack_conn_id=None):
If both supplied, Slack API token will be used.
:param token: Slack API token
- :type token: string
+ :type token: str
:param slack_conn_id: connection that has Slack API token in the password field
- :type slack_conn_id: string
+ :type slack_conn_id: str
"""
self.token = self.__get_token(token, slack_conn_id)
diff --git a/airflow/jobs.py b/airflow/jobs.py
index ad114abda3..b224f75545 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -40,7 +40,6 @@
Column, Integer, String, func, Index, or_, and_, not_)
from sqlalchemy.exc import OperationalError
from sqlalchemy.orm.session import make_transient
-from sqlalchemy_utc import UtcDateTime
from tabulate import tabulate
from time import sleep
@@ -52,17 +51,18 @@
from airflow.task.task_runner import get_task_runner
from airflow.ti_deps.dep_context import DepContext, QUEUE_DEPS, RUN_DEPS
from airflow.utils import asciiart, helpers, timezone
+from airflow.utils.configuration import tmp_configuration_copy
from airflow.utils.dag_processing import (AbstractDagFileProcessor,
DagFileProcessorManager,
SimpleDag,
SimpleDagBag,
list_py_file_paths)
from airflow.utils.db import create_session, provide_session
-from airflow.utils.email import send_email
+from airflow.utils.email import send_email, get_email_address_list
from airflow.utils.log.logging_mixin import LoggingMixin, set_context, StreamLogWriter
-from airflow.utils.state import State
-from airflow.utils.configuration import tmp_configuration_copy
from airflow.utils.net import get_hostname
+from airflow.utils.state import State
+from airflow.utils.sqlalchemy import UtcDateTime
Base = models.Base
ID_LEN = models.ID_LEN
@@ -96,6 +96,7 @@ class BaseJob(Base, LoggingMixin):
__table_args__ = (
Index('job_type_heart', job_type, latest_heartbeat),
+ Index('idx_job_state_heartbeat', state, latest_heartbeat),
)
def __init__(
@@ -172,7 +173,8 @@ def heartbeat(self):
if job.latest_heartbeat:
sleep_for = max(
0,
- self.heartrate - (timezone.utcnow() - job.latest_heartbeat).total_seconds())
+ self.heartrate - (
+ timezone.utcnow() - job.latest_heartbeat).total_seconds())
sleep(sleep_for)
@@ -197,14 +199,20 @@ def run(self):
make_transient(self)
self.id = id_
- # Run
- self._execute()
-
- # Marking the success in the DB
- self.end_date = timezone.utcnow()
- self.state = State.SUCCESS
- session.merge(self)
- session.commit()
+ try:
+ self._execute()
+ # In case of max runs or max duration
+ self.state = State.SUCCESS
+ except SystemExit as e:
+ # In case of ^C or SIGTERM
+ self.state = State.SUCCESS
+ except Exception as e:
+ self.state = State.FAILED
+ raise
+ finally:
+ self.end_date = timezone.utcnow()
+ session.merge(self)
+ session.commit()
Stats.incr(self.__class__.__name__.lower() + '_end', 1, 1)
@@ -341,10 +349,10 @@ def _launch_process(result_queue,
:param file_path: the file to process
:type file_path: unicode
:param pickle_dags: whether to pickle the DAGs found in the file and
- save them to the DB
+ save them to the DB
:type pickle_dags: bool
:param dag_id_white_list: if specified, only examine DAG ID's that are
- in this list
+ in this list
:type dag_id_white_list: list[unicode]
:param thread_name: the name to use for the process that is launched
:type thread_name: unicode
@@ -384,7 +392,7 @@ def helper():
log.info(
"Processing %s took %.3f seconds", file_path, end_time - start_time
)
- except:
+ except Exception:
# Log exceptions through the logging framework.
log.exception("Got an exception! Propagating...")
raise
@@ -416,6 +424,7 @@ def start(self):
def terminate(self, sigkill=False):
"""
Terminate (and then kill) the process launched to process the file.
+
:param sigkill: whether to issue a SIGKILL if SIGTERM doesn't work.
:type sigkill: bool
"""
@@ -444,6 +453,7 @@ def pid(self):
def exit_code(self):
"""
After the process is finished, this can be called to get the return code
+
:return: the exit code of the process
:rtype: int
"""
@@ -455,6 +465,7 @@ def exit_code(self):
def done(self):
"""
Check if the process launched to process this file is done.
+
:return: whether the process is finished running
:rtype: bool
"""
@@ -525,8 +536,7 @@ def __init__(
num_runs=-1,
file_process_interval=conf.getint('scheduler',
'min_file_process_interval'),
- min_file_parsing_loop_time=conf.getint('scheduler',
- 'min_file_parsing_loop_time'),
+ processor_poll_interval=1.0,
run_duration=None,
do_pickle=False,
log=None,
@@ -537,14 +547,18 @@ def __init__(
:param dag_ids: if specified, only schedule tasks with these DAG IDs
:type dag_ids: list[unicode]
:param subdir: directory containing Python files with Airflow DAG
- definitions, or a specific path to a file
+ definitions, or a specific path to a file
:type subdir: unicode
:param num_runs: The number of times to try to schedule each DAG file.
- -1 for unlimited within the run_duration.
+ -1 for unlimited within the run_duration.
+ :type num_runs: int
+ :param processor_poll_interval: The number of seconds to wait between
+ polls of running processors
+ :type processor_poll_interval: int
:param run_duration: how long to run (in seconds) before exiting
:type run_duration: int
:param do_pickle: once a DAG object is obtained by executing the Python
- file, whether to serialize the DAG object to the DB
+ file, whether to serialize the DAG object to the DB
:type do_pickle: bool
"""
# for BaseJob compatibility
@@ -557,6 +571,7 @@ def __init__(
self.num_runs = num_runs
self.run_duration = run_duration
+ self._processor_poll_interval = processor_poll_interval
self.do_pickle = do_pickle
super(SchedulerJob, self).__init__(*args, **kwargs)
@@ -581,14 +596,10 @@ def __init__(
# 30 seconds.
self.print_stats_interval = conf.getint('scheduler',
'print_stats_interval')
- # Parse and schedule each file no faster than this interval. Default
- # to 3 minutes.
- self.file_process_interval = file_process_interval
- # Wait until at least this many seconds have passed before parsing files once all
- # files have finished parsing.
- self.min_file_parsing_loop_time = min_file_parsing_loop_time
+ self.file_process_interval = file_process_interval
+ self.max_tis_per_query = conf.getint('scheduler', 'max_tis_per_query')
if run_duration is None:
self.run_duration = conf.getint('scheduler',
'run_duration')
@@ -651,7 +662,7 @@ def manage_slas(self, dag, session=None):
slas = (
session
.query(SlaMiss)
- .filter(SlaMiss.notification_sent == False)
+ .filter(SlaMiss.notification_sent == False) # noqa: E712
.filter(SlaMiss.dag_id == dag.dag_id)
.all()
)
@@ -701,16 +712,13 @@ def manage_slas(self, dag, session=None):
Blocking tasks:
<pre><code>{blocking_task_list}\n{bug}<code></pre>
""".format(bug=asciiart.bug, **locals())
- emails = []
- for t in dag.tasks:
- if t.email:
- if isinstance(t.email, basestring):
- l = [t.email]
- elif isinstance(t.email, (list, tuple)):
- l = t.email
- for email in l:
- if email not in emails:
- emails.append(email)
+ emails = set()
+ for task in dag.tasks:
+ if task.email:
+ if isinstance(task.email, basestring):
+ emails |= set(get_email_address_list(task.email))
+ elif isinstance(task.email, (list, tuple)):
+ emails |= set(task.email)
if emails and len(slas):
try:
send_email(
@@ -779,10 +787,10 @@ def update_import_errors(session, dagbag):
def create_dag_run(self, dag, session=None):
"""
This method checks whether a new DagRun needs to be created
- for a DAG based on scheduling interval
+ for a DAG based on scheduling interval.
Returns DagRun if one is scheduled. Otherwise returns None.
"""
- if dag.schedule_interval:
+ if dag.schedule_interval and conf.getboolean('scheduler', 'USE_JOB_SCHEDULE'):
active_runs = DagRun.find(
dag_id=dag.dag_id,
state=State.RUNNING,
@@ -811,7 +819,7 @@ def create_dag_run(self, dag, session=None):
session.query(func.max(DagRun.execution_date))
.filter_by(dag_id=dag.dag_id)
.filter(or_(
- DagRun.external_trigger == False,
+ DagRun.external_trigger == False, # noqa: E712
# add % as a wildcard for the like query
DagRun.run_id.like(DagRun.ID_PREFIX + '%')
))
@@ -987,7 +995,7 @@ def _change_state_for_tis_without_dagrun(self,
:param new_state: set TaskInstances to this state
:type new_state: State
:param simple_dag_bag: TaskInstances associated with DAGs in the
- simple_dag_bag and with states in the old_state will be examined
+ simple_dag_bag and with states in the old_state will be examined
:type simple_dag_bag: SimpleDagBag
"""
tis_changed = 0
@@ -1058,7 +1066,7 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None):
dag concurrency, executor state, and priority.
:param simple_dag_bag: TaskInstances associated with DAGs in the
- simple_dag_bag will be fetched from the DB and executed
+ simple_dag_bag will be fetched from the DB and executed
:type simple_dag_bag: SimpleDagBag
:param executor: the executor that runs task instances
:type executor: BaseExecutor
@@ -1066,9 +1074,6 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None):
:type states: Tuple[State]
:return: List[TaskInstance]
"""
- # TODO(saguziel): Change this to include QUEUED, for concurrency
- # purposes we may want to count queued tasks
- states_to_count_as_running = [State.RUNNING]
executable_tis = []
# Get all the queued task instances from associated with scheduled
@@ -1081,24 +1086,27 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None):
session
.query(TI)
.filter(TI.dag_id.in_(simple_dag_bag.dag_ids))
- .outerjoin(DR,
- and_(DR.dag_id == TI.dag_id,
- DR.execution_date == TI.execution_date))
- .filter(or_(DR.run_id == None,
+ .outerjoin(
+ DR,
+ and_(DR.dag_id == TI.dag_id, DR.execution_date == TI.execution_date)
+ )
+ .filter(or_(DR.run_id == None, # noqa: E711
not_(DR.run_id.like(BackfillJob.ID_PREFIX + '%'))))
- .outerjoin(DM, DM.dag_id==TI.dag_id)
- .filter(or_(DM.dag_id == None,
+ .outerjoin(DM, DM.dag_id == TI.dag_id)
+ .filter(or_(DM.dag_id == None, # noqa: E711
not_(DM.is_paused)))
)
if None in states:
- ti_query = ti_query.filter(or_(TI.state == None, TI.state.in_(states)))
+ ti_query = ti_query.filter(
+ or_(TI.state == None, TI.state.in_(states)) # noqa: E711
+ )
else:
ti_query = ti_query.filter(TI.state.in_(states))
task_instances_to_examine = ti_query.all()
if len(task_instances_to_examine) == 0:
- self.log.info("No tasks to consider for execution.")
+ self.log.debug("No tasks to consider for execution.")
return executable_tis
# Put one task instance on each line
@@ -1113,7 +1121,9 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None):
for task_instance in task_instances_to_examine:
pool_to_task_instances[task_instance.pool].append(task_instance)
- task_concurrency_map = self.__get_task_concurrency_map(states=states_to_count_as_running, session=session)
+ states_to_count_as_running = [State.RUNNING, State.QUEUED]
+ task_concurrency_map = self.__get_task_concurrency_map(
+ states=states_to_count_as_running, session=session)
# Go through each pool, and queue up a task for execution if there are
# any open slots in the pool.
@@ -1162,7 +1172,6 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None):
simple_dag = simple_dag_bag.get_dag(dag_id)
if dag_id not in dag_id_to_possibly_running_task_count:
- # TODO(saguziel): also check against QUEUED state, see AIRFLOW-1104
dag_id_to_possibly_running_task_count[dag_id] = \
DAG.get_num_task_instances(
dag_id,
@@ -1178,15 +1187,20 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None):
)
if current_task_concurrency >= task_concurrency_limit:
self.log.info(
- "Not executing %s since the number of tasks running or queued from DAG %s"
- " is >= to the DAG's task concurrency limit of %s",
+ "Not executing %s since the number of tasks running or queued "
+ "from DAG %s is >= to the DAG's task concurrency limit of %s",
task_instance, dag_id, task_concurrency_limit
)
continue
- task_concurrency = simple_dag.get_task_special_arg(task_instance.task_id, 'task_concurrency')
+ task_concurrency = simple_dag.get_task_special_arg(
+ task_instance.task_id,
+ 'task_concurrency')
if task_concurrency is not None:
- num_running = task_concurrency_map[((task_instance.dag_id, task_instance.task_id))]
+ num_running = task_concurrency_map[
+ (task_instance.dag_id, task_instance.task_id)
+ ]
+
if num_running >= task_concurrency:
self.log.info("Not executing %s since the task concurrency for"
" this task has been reached.", task_instance)
@@ -1206,7 +1220,8 @@ def _find_executable_task_instances(self, simple_dag_bag, states, session=None):
task_instance_str = "\n\t".join(
["{}".format(x) for x in executable_tis])
- self.log.info("Setting the follow tasks to queued state:\n\t%s", task_instance_str)
+ self.log.info(
+ "Setting the follow tasks to queued state:\n\t%s", task_instance_str)
# so these dont expire on commit
for ti in executable_tis:
copy_dag_id = ti.dag_id
@@ -1248,7 +1263,9 @@ def _change_state_for_executable_task_instances(self, task_instances,
.filter(or_(*filter_for_ti_state_change)))
if None in acceptable_states:
- ti_query = ti_query.filter(or_(TI.state == None, TI.state.in_(acceptable_states)))
+ ti_query = ti_query.filter(
+ or_(TI.state == None, TI.state.in_(acceptable_states)) # noqa: E711
+ )
else:
ti_query = ti_query.filter(TI.state.in_(acceptable_states))
@@ -1311,7 +1328,7 @@ def _enqueue_task_instances_with_queued_state(self, simple_dag_bag, task_instanc
# actually enqueue them
for task_instance in task_instances:
simple_dag = simple_dag_bag.get_dag(task_instance.dag_id)
- command = " ".join(TI.generate_command(
+ command = TI.generate_command(
task_instance.dag_id,
task_instance.task_id,
task_instance.execution_date,
@@ -1323,7 +1340,7 @@ def _enqueue_task_instances_with_queued_state(self, simple_dag_bag, task_instanc
ignore_ti_state=False,
pool=task_instance.pool,
file_path=simple_dag.full_filepath,
- pickle_id=simple_dag.pickle_id))
+ pickle_id=simple_dag.pickle_id)
priority = task_instance.priority_weight
queue = task_instance.queue
@@ -1362,7 +1379,7 @@ def _execute_task_instances(self,
3. Enqueue the TIs in the executor.
:param simple_dag_bag: TaskInstances associated with DAGs in the
- simple_dag_bag will be fetched from the DB and executed
+ simple_dag_bag will be fetched from the DB and executed
:type simple_dag_bag: SimpleDagBag
:param states: Execute TaskInstances in these states
:type states: Tuple[State]
@@ -1471,7 +1488,7 @@ def _log_file_processing_stats(self,
Print out stats about how files are getting processed.
:param known_file_paths: a list of file paths that may contain Airflow
- DAG definitions
+ DAG definitions
:type known_file_paths: list[unicode]
:param processor_manager: manager for the file processors
:type stats: DagFileProcessorManager
@@ -1543,16 +1560,18 @@ def _execute(self):
# DAGs in parallel. By processing them in separate processes,
# we can get parallelism and isolation from potentially harmful
# user code.
- self.log.info("Processing files using up to %s processes at a time",
- self.max_threads)
+ self.log.info(
+ "Processing files using up to %s processes at a time",
+ self.max_threads)
self.log.info("Running execute loop for %s seconds", self.run_duration)
self.log.info("Processing each file at most %s times", self.num_runs)
- self.log.info("Process each file at most once every %s seconds",
- self.file_process_interval)
- self.log.info("Wait until at least %s seconds have passed between file parsing "
- "loops", self.min_file_parsing_loop_time)
- self.log.info("Checking for new files in %s every %s seconds",
- self.subdir, self.dag_dir_list_interval)
+ self.log.info(
+ "Process each file at most once every %s seconds",
+ self.file_process_interval)
+ self.log.info(
+ "Checking for new files in %s every %s seconds",
+ self.subdir,
+ self.dag_dir_list_interval)
# Build up a list of Python files that could contain DAGs
self.log.info("Searching for files in %s", self.subdir)
@@ -1568,7 +1587,6 @@ def processor_factory(file_path):
known_file_paths,
self.max_threads,
self.file_process_interval,
- self.min_file_parsing_loop_time,
self.num_runs,
processor_factory)
@@ -1593,7 +1611,8 @@ def processor_factory(file_path):
child.terminate()
# TODO: Remove magic number
timeout = 5
- self.log.info("Waiting up to %s seconds for processes to exit...", timeout)
+ self.log.info(
+ "Waiting up to %s seconds for processes to exit...", timeout)
try:
psutil.wait_procs(
child_processes, timeout=timeout,
@@ -1650,7 +1669,9 @@ def _execute_helper(self, processor_manager):
self.log.info("Searching for files in %s", self.subdir)
known_file_paths = list_py_file_paths(self.subdir)
last_dag_dir_refresh_time = timezone.utcnow()
- self.log.info("There are %s files in %s", len(known_file_paths), self.subdir)
+ self.log.info(
+ "There are %s files in %s", len(known_file_paths), self.subdir)
+
processor_manager.set_file_paths(known_file_paths)
self.log.debug("Removing old import errors")
@@ -1663,7 +1684,9 @@ def _execute_helper(self, processor_manager):
if self.using_sqlite:
# For the sqlite case w/ 1 thread, wait until the processor
# is finished to avoid concurrent access to the DB.
- self.log.debug("Waiting for processors to finish since we're using sqlite")
+ self.log.debug(
+ "Waiting for processors to finish since we're using sqlite")
+
processor_manager.wait_until_finished()
# Send tasks for execution if available
@@ -1715,13 +1738,17 @@ def _execute_helper(self, processor_manager):
last_stat_print_time = timezone.utcnow()
loop_end_time = time.time()
- self.log.debug("Ran scheduling loop in %.2f seconds",
- loop_end_time - loop_start_time)
+ self.log.debug(
+ "Ran scheduling loop in %.2f seconds",
+ loop_end_time - loop_start_time)
+ self.log.debug("Sleeping for %.2f seconds", self._processor_poll_interval)
+ time.sleep(self._processor_poll_interval)
# Exit early for a test mode
if processor_manager.max_runs_reached():
- self.log.info("Exiting loop as all files have been processed %s times",
- self.num_runs)
+ self.log.info(
+ "Exiting loop as all files have been processed %s times",
+ self.num_runs)
break
# Stop any processors
@@ -1767,7 +1794,7 @@ def process_file(self, file_path, pickle_dags=False, session=None):
:param file_path: the path to the Python file that should be executed
:type file_path: unicode
:param pickle_dags: whether serialize the DAGs found in the file and
- save them to the db
+ save them to the db
:type pickle_dags: bool
:return: a list of SimpleDags made from the Dags found in the file
:rtype: list[SimpleDag]
@@ -2006,6 +2033,7 @@ def _update_counters(self, ti_status):
"""
Updates the counters per state of the tasks that were running. Can re-add
to tasks to run in case required.
+
:param ti_status: the internal status of the backfill job tasks
:type ti_status: BackfillJob._DagRunTaskStatus
"""
@@ -2050,6 +2078,7 @@ def _manage_executor_state(self, running):
"""
Checks if the executor agrees with the state of task instances
that are running
+
:param running: dict of key, task to verify
"""
executor = self.executor
@@ -2081,6 +2110,7 @@ def _get_dag_run(self, run_date, session=None):
Returns a dag run for the given run date, which will be matched to an existing
dag run if available or create a new dag run otherwise. If the max_active_runs
limit is reached, this function will return None.
+
:param run_date: the execution date for the dag run
:type run_date: datetime
:param session: the database session object
@@ -2140,6 +2170,7 @@ def _task_instances_for_dag_run(self, dag_run, session=None):
"""
Returns a map of task instance key to task instance object for the tasks to
run in the given dag run.
+
:param dag_run: the dag run to get the tasks from
:type dag_run: models.DagRun
:param session: the database session object
@@ -2205,6 +2236,7 @@ def _process_backfill_task_instances(self,
Process a set of task instances from a set of dag runs. Special handling is done
to account for different task instance states that could be present when running
them in a backfill process.
+
:param ti_status: the internal status of the job
:type ti_status: BackfillJob._DagRunTaskStatus
:param executor: the executor to run the task instances
@@ -2442,6 +2474,7 @@ def _execute_for_run_dates(self, run_dates, ti_status, executor, pickle_id,
Computes the dag runs and their respective task instances for
the given run dates and executes the task instances.
Returns a list of execution dates of the dag runs that were executed.
+
:param run_dates: Execution dates for dag runs
:type run_dates: list
:param ti_status: internal BackfillJob status structure to tis track progress
@@ -2557,6 +2590,7 @@ def __init__(
pool=None,
*args, **kwargs):
self.task_instance = task_instance
+ self.dag_id = task_instance.dag_id
self.ignore_all_deps = ignore_all_deps
self.ignore_depends_on_past = ignore_depends_on_past
self.ignore_task_deps = ignore_task_deps
diff --git a/airflow/logging_config.py b/airflow/logging_config.py
index 33c2dc82e1..84850b4f82 100644
--- a/airflow/logging_config.py
+++ b/airflow/logging_config.py
@@ -18,6 +18,7 @@
# under the License.
#
import logging
+import warnings
from logging.config import dictConfig
from airflow import configuration as conf
@@ -70,4 +71,36 @@ def configure_logging():
# otherwise Airflow would silently fall back on the default config
raise e
+ validate_logging_config(logging_config)
+
return logging_config
+
+
+def validate_logging_config(logging_config):
+ # Now lets validate the other logging-related settings
+ task_log_reader = conf.get('core', 'task_log_reader')
+
+ logger = logging.getLogger('airflow.task')
+
+ def _get_handler(name):
+ return next((h for h in logger.handlers if h.name == name), None)
+
+ if _get_handler(task_log_reader) is None:
+ # Check for pre 1.10 setting that might be in deployed airflow.cfg files
+ if task_log_reader == "file.task" and _get_handler("task"):
+ warnings.warn(
+ "task_log_reader setting in [core] has a deprecated value of "
+ "{!r}, but no handler with this name was found. Please update "
+ "your config to use {!r}. Running config has been adjusted to "
+ "match".format(
+ task_log_reader,
+ "task",
+ ),
+ DeprecationWarning,
+ )
+ conf.set('core', 'task_log_reader', 'task')
+ else:
+ raise AirflowConfigException(
+ "Configured task_log_reader {!r} was not a handler of the 'airflow.task' "
+ "logger.".format(task_log_reader)
+ )
diff --git a/airflow/macros/__init__.py b/airflow/macros/__init__.py
index 3c33acda5d..fdd64f88a6 100644
--- a/airflow/macros/__init__.py
+++ b/airflow/macros/__init__.py
@@ -74,20 +74,3 @@ def _integrate_plugins():
for macros_module in macros_modules:
sys.modules[macros_module.__name__] = macros_module
globals()[macros_module._name] = macros_module
-
- ##########################################################
- # TODO FIXME Remove in Airflow 2.0
-
- import os as _os
- if not _os.environ.get('AIRFLOW_USE_NEW_IMPORTS', False):
- from zope.deprecation import deprecated as _deprecated
- for _macro in macros_module._objects:
- macro_name = _macro.__name__
- globals()[macro_name] = _macro
- _deprecated(
- macro_name,
- "Importing plugin macro '{i}' directly from "
- "'airflow.macros' has been deprecated. Please "
- "import from 'airflow.macros.[plugin_module]' "
- "instead. Support for direct imports will be dropped "
- "entirely in Airflow 2.0.".format(i=macro_name))
diff --git a/airflow/macros/hive.py b/airflow/macros/hive.py
index bb60203e95..f8e57bb515 100644
--- a/airflow/macros/hive.py
+++ b/airflow/macros/hive.py
@@ -27,14 +27,14 @@ def max_partition(
Gets the max partition for a table.
:param schema: The hive schema the table lives in
- :type schema: string
+ :type schema: str
:param table: The hive table you are interested in, supports the dot
notation as in "my_database.my_table", if a dot is found,
the schema param is disregarded
- :type table: string
+ :type table: str
:param metastore_conn_id: The hive connection you are interested in.
If your default is set you don't need to use this parameter.
- :type metastore_conn_id: string
+ :type metastore_conn_id: str
:param filter_map: partition_key:partition_value map used for partition filtering,
e.g. {'key1': 'value1', 'key2': 'value2'}.
Only partitions matching all partition_key:partition_value
diff --git a/airflow/migrations/__init__.py b/airflow/migrations/__init__.py
index f0f8b68337..114d189da1 100644
--- a/airflow/migrations/__init__.py
+++ b/airflow/migrations/__init__.py
@@ -7,13 +7,12 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
diff --git a/airflow/migrations/env.py b/airflow/migrations/env.py
index 828cacf0e0..4e1977635a 100644
--- a/airflow/migrations/env.py
+++ b/airflow/migrations/env.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -85,6 +85,7 @@ def run_migrations_online():
with context.begin_transaction():
context.run_migrations()
+
if context.is_offline_mode():
run_migrations_offline()
else:
diff --git a/init.sh b/airflow/migrations/versions/05f30312d566_merge_heads.py
similarity index 69%
rename from init.sh
rename to airflow/migrations/versions/05f30312d566_merge_heads.py
index 6f4adcad06..f869cb8c08 100644
--- a/init.sh
+++ b/airflow/migrations/versions/05f30312d566_merge_heads.py
@@ -1,5 +1,4 @@
-#!/usr/bin/env bash
-
+# flake8: noqa
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -18,4 +17,24 @@
# specific language governing permissions and limitations
# under the License.
-source $AIRFLOW_HOME/env/bin/activate
+"""merge heads
+
+Revision ID: 05f30312d566
+Revises: 86770d1215c0, 0e2a74e0fc9f
+Create Date: 2018-06-17 10:47:23.339972
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '05f30312d566'
+down_revision = ('86770d1215c0', '0e2a74e0fc9f')
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+ pass
+
+
+def downgrade():
+ pass
diff --git a/airflow/migrations/versions/0a2a5b66e19d_add_task_reschedule_table.py b/airflow/migrations/versions/0a2a5b66e19d_add_task_reschedule_table.py
new file mode 100644
index 0000000000..643a1ca81b
--- /dev/null
+++ b/airflow/migrations/versions/0a2a5b66e19d_add_task_reschedule_table.py
@@ -0,0 +1,91 @@
+# flake8: noqa
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""add task_reschedule table
+
+Revision ID: 0a2a5b66e19d
+Revises: 9635ae0956e7
+Create Date: 2018-06-17 22:50:00.053620
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '0a2a5b66e19d'
+down_revision = '9635ae0956e7'
+branch_labels = None
+depends_on = None
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import mysql
+
+
+TABLE_NAME = 'task_reschedule'
+INDEX_NAME = 'idx_' + TABLE_NAME + '_dag_task_date'
+
+# For Microsoft SQL Server, TIMESTAMP is a row-id type,
+# having nothing to do with date-time. DateTime() will
+# be sufficient.
+def mssql_timestamp():
+ return sa.DateTime()
+
+def mysql_timestamp():
+ return mysql.TIMESTAMP(fsp=6)
+
+def sa_timestamp():
+ return sa.TIMESTAMP(timezone=True)
+
+def upgrade():
+ # See 0e2a74e0fc9f_add_time_zone_awareness
+ conn = op.get_bind()
+ if conn.dialect.name == 'mysql':
+ timestamp = mysql_timestamp
+ elif conn.dialect.name == 'mssql':
+ timestamp = mssql_timestamp
+ else:
+ timestamp = sa_timestamp
+
+ op.create_table(
+ TABLE_NAME,
+ sa.Column('id', sa.Integer(), nullable=False),
+ sa.Column('task_id', sa.String(length=250), nullable=False),
+ sa.Column('dag_id', sa.String(length=250), nullable=False),
+ # use explicit server_default=None otherwise mysql implies defaults for first timestamp column
+ sa.Column('execution_date', timestamp(), nullable=False, server_default=None),
+ sa.Column('try_number', sa.Integer(), nullable=False),
+ sa.Column('start_date', timestamp(), nullable=False),
+ sa.Column('end_date', timestamp(), nullable=False),
+ sa.Column('duration', sa.Integer(), nullable=False),
+ sa.Column('reschedule_date', timestamp(), nullable=False),
+ sa.PrimaryKeyConstraint('id'),
+ sa.ForeignKeyConstraint(['task_id', 'dag_id', 'execution_date'],
+ ['task_instance.task_id', 'task_instance.dag_id','task_instance.execution_date'],
+ name='task_reschedule_dag_task_date_fkey')
+ )
+ op.create_index(
+ INDEX_NAME,
+ TABLE_NAME,
+ ['dag_id', 'task_id', 'execution_date'],
+ unique=False
+ )
+
+
+def downgrade():
+ op.drop_index(INDEX_NAME, table_name=TABLE_NAME)
+ op.drop_table(TABLE_NAME)
diff --git a/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py b/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py
index 64ee41c44d..ac7824fd9e 100644
--- a/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py
+++ b/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py
@@ -25,16 +25,16 @@
"""
+from alembic import op
+from sqlalchemy.dialects import mysql
+import sqlalchemy as sa
+
# revision identifiers, used by Alembic.
revision = '0e2a74e0fc9f'
down_revision = 'd2ae31099d61'
branch_labels = None
depends_on = None
-from alembic import op
-from sqlalchemy.dialects import mysql
-import sqlalchemy as sa
-
def upgrade():
conn = op.get_bind()
@@ -69,14 +69,16 @@ def upgrade():
op.alter_column(table_name='log', column_name='dttm', type_=mysql.TIMESTAMP(fsp=6))
op.alter_column(table_name='log', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6))
- op.alter_column(table_name='sla_miss', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6), nullable=False)
+ op.alter_column(table_name='sla_miss', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6),
+ nullable=False)
op.alter_column(table_name='sla_miss', column_name='timestamp', type_=mysql.TIMESTAMP(fsp=6))
op.alter_column(table_name='task_fail', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6))
op.alter_column(table_name='task_fail', column_name='start_date', type_=mysql.TIMESTAMP(fsp=6))
op.alter_column(table_name='task_fail', column_name='end_date', type_=mysql.TIMESTAMP(fsp=6))
- op.alter_column(table_name='task_instance', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6), nullable=False)
+ op.alter_column(table_name='task_instance', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6),
+ nullable=False)
op.alter_column(table_name='task_instance', column_name='start_date', type_=mysql.TIMESTAMP(fsp=6))
op.alter_column(table_name='task_instance', column_name='end_date', type_=mysql.TIMESTAMP(fsp=6))
op.alter_column(table_name='task_instance', column_name='queued_dttm', type_=mysql.TIMESTAMP(fsp=6))
@@ -84,8 +86,8 @@ def upgrade():
op.alter_column(table_name='xcom', column_name='timestamp', type_=mysql.TIMESTAMP(fsp=6))
op.alter_column(table_name='xcom', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6))
else:
- # sqlite datetime is fine as is not converting
- if conn.dialect.name == 'sqlite':
+ # sqlite and mssql datetime are fine as is. Therefore, not converting
+ if conn.dialect.name in ('sqlite', 'mssql'):
return
# we try to be database agnostic, but not every db (e.g. sqlserver)
@@ -117,14 +119,16 @@ def upgrade():
op.alter_column(table_name='log', column_name='dttm', type_=sa.TIMESTAMP(timezone=True))
op.alter_column(table_name='log', column_name='execution_date', type_=sa.TIMESTAMP(timezone=True))
- op.alter_column(table_name='sla_miss', column_name='execution_date', type_=sa.TIMESTAMP(timezone=True), nullable=False)
+ op.alter_column(table_name='sla_miss', column_name='execution_date', type_=sa.TIMESTAMP(timezone=True),
+ nullable=False)
op.alter_column(table_name='sla_miss', column_name='timestamp', type_=sa.TIMESTAMP(timezone=True))
op.alter_column(table_name='task_fail', column_name='execution_date', type_=sa.TIMESTAMP(timezone=True))
op.alter_column(table_name='task_fail', column_name='start_date', type_=sa.TIMESTAMP(timezone=True))
op.alter_column(table_name='task_fail', column_name='end_date', type_=sa.TIMESTAMP(timezone=True))
- op.alter_column(table_name='task_instance', column_name='execution_date', type_=sa.TIMESTAMP(timezone=True), nullable=False)
+ op.alter_column(table_name='task_instance', column_name='execution_date', type_=sa.TIMESTAMP(timezone=True),
+ nullable=False)
op.alter_column(table_name='task_instance', column_name='start_date', type_=sa.TIMESTAMP(timezone=True))
op.alter_column(table_name='task_instance', column_name='end_date', type_=sa.TIMESTAMP(timezone=True))
op.alter_column(table_name='task_instance', column_name='queued_dttm', type_=sa.TIMESTAMP(timezone=True))
@@ -161,14 +165,16 @@ def downgrade():
op.alter_column(table_name='log', column_name='dttm', type_=mysql.DATETIME(fsp=6))
op.alter_column(table_name='log', column_name='execution_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='sla_miss', column_name='execution_date', type_=mysql.DATETIME(fsp=6), nullable=False)
+ op.alter_column(table_name='sla_miss', column_name='execution_date', type_=mysql.DATETIME(fsp=6),
+ nullable=False)
op.alter_column(table_name='sla_miss', column_name='DATETIME', type_=mysql.DATETIME(fsp=6))
op.alter_column(table_name='task_fail', column_name='execution_date', type_=mysql.DATETIME(fsp=6))
op.alter_column(table_name='task_fail', column_name='start_date', type_=mysql.DATETIME(fsp=6))
op.alter_column(table_name='task_fail', column_name='end_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='task_instance', column_name='execution_date', type_=mysql.DATETIME(fsp=6), nullable=False)
+ op.alter_column(table_name='task_instance', column_name='execution_date', type_=mysql.DATETIME(fsp=6),
+ nullable=False)
op.alter_column(table_name='task_instance', column_name='start_date', type_=mysql.DATETIME(fsp=6))
op.alter_column(table_name='task_instance', column_name='end_date', type_=mysql.DATETIME(fsp=6))
op.alter_column(table_name='task_instance', column_name='queued_dttm', type_=mysql.DATETIME(fsp=6))
@@ -176,7 +182,7 @@ def downgrade():
op.alter_column(table_name='xcom', column_name='DATETIME', type_=mysql.DATETIME(fsp=6))
op.alter_column(table_name='xcom', column_name='execution_date', type_=mysql.DATETIME(fsp=6))
else:
- if conn.dialect.name == 'sqlite':
+ if conn.dialect.name in ('sqlite', 'mssql'):
return
# we try to be database agnostic, but not every db (e.g. sqlserver)
diff --git a/airflow/migrations/versions/127d2bf2dfa7_add_dag_id_state_index_on_dag_run_table.py b/airflow/migrations/versions/127d2bf2dfa7_add_dag_id_state_index_on_dag_run_table.py
index 5851725694..288a0b60aa 100644
--- a/airflow/migrations/versions/127d2bf2dfa7_add_dag_id_state_index_on_dag_run_table.py
+++ b/airflow/migrations/versions/127d2bf2dfa7_add_dag_id_state_index_on_dag_run_table.py
@@ -6,9 +6,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -23,6 +23,7 @@
Create Date: 2017-01-25 11:43:51.635667
"""
+from alembic import op
# revision identifiers, used by Alembic.
revision = '127d2bf2dfa7'
@@ -30,8 +31,6 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
def upgrade():
op.create_index('dag_id_state', 'dag_run', ['dag_id', 'state'], unique=False)
@@ -39,4 +38,3 @@ def upgrade():
def downgrade():
op.drop_index('dag_id_state', table_name='dag_run')
-
diff --git a/airflow/migrations/versions/13eb55f81627_for_compatibility.py b/airflow/migrations/versions/13eb55f81627_for_compatibility.py
index a14e195bbe..b9e88bd01b 100644
--- a/airflow/migrations/versions/13eb55f81627_for_compatibility.py
+++ b/airflow/migrations/versions/13eb55f81627_for_compatibility.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
diff --git a/airflow/migrations/versions/1507a7289a2f_create_is_encrypted.py b/airflow/migrations/versions/1507a7289a2f_create_is_encrypted.py
index 47bdf13d67..fe84254c38 100644
--- a/airflow/migrations/versions/1507a7289a2f_create_is_encrypted.py
+++ b/airflow/migrations/versions/1507a7289a2f_create_is_encrypted.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,6 +24,9 @@
Create Date: 2015-08-18 18:57:51.927315
"""
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.engine.reflection import Inspector
# revision identifiers, used by Alembic.
revision = '1507a7289a2f'
@@ -31,10 +34,6 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-from sqlalchemy.engine.reflection import Inspector
-
connectionhelper = sa.Table(
'connection',
sa.MetaData(),
diff --git a/airflow/migrations/versions/1968acfc09e3_add_is_encrypted_column_to_variable_.py b/airflow/migrations/versions/1968acfc09e3_add_is_encrypted_column_to_variable_.py
index 57ee27aeb4..16ab349563 100644
--- a/airflow/migrations/versions/1968acfc09e3_add_is_encrypted_column_to_variable_.py
+++ b/airflow/migrations/versions/1968acfc09e3_add_is_encrypted_column_to_variable_.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,6 +24,8 @@
Create Date: 2016-02-02 17:20:55.692295
"""
+from alembic import op
+import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '1968acfc09e3'
@@ -31,12 +33,9 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
- op.add_column('variable', sa.Column('is_encrypted', sa.Boolean,default=False))
+ op.add_column('variable', sa.Column('is_encrypted', sa.Boolean, default=False))
def downgrade():
diff --git a/airflow/migrations/versions/1b38cef5b76e_add_dagrun.py b/airflow/migrations/versions/1b38cef5b76e_add_dagrun.py
index a0af7fa7ad..50d53652c4 100644
--- a/airflow/migrations/versions/1b38cef5b76e_add_dagrun.py
+++ b/airflow/migrations/versions/1b38cef5b76e_add_dagrun.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -25,28 +25,27 @@
"""
+from alembic import op
+import sqlalchemy as sa
+
# revision identifiers, used by Alembic.
revision = '1b38cef5b76e'
down_revision = '502898887f84'
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
op.create_table('dag_run',
- sa.Column('id', sa.Integer(), nullable=False),
- sa.Column('dag_id', sa.String(length=250), nullable=True),
- sa.Column('execution_date', sa.DateTime(), nullable=True),
- sa.Column('state', sa.String(length=50), nullable=True),
- sa.Column('run_id', sa.String(length=250), nullable=True),
- sa.Column('external_trigger', sa.Boolean(), nullable=True),
- sa.PrimaryKeyConstraint('id'),
- sa.UniqueConstraint('dag_id', 'execution_date'),
- sa.UniqueConstraint('dag_id', 'run_id'),
- )
+ sa.Column('id', sa.Integer(), nullable=False),
+ sa.Column('dag_id', sa.String(length=250), nullable=True),
+ sa.Column('execution_date', sa.DateTime(), nullable=True),
+ sa.Column('state', sa.String(length=50), nullable=True),
+ sa.Column('run_id', sa.String(length=250), nullable=True),
+ sa.Column('external_trigger', sa.Boolean(), nullable=True),
+ sa.PrimaryKeyConstraint('id'),
+ sa.UniqueConstraint('dag_id', 'execution_date'),
+ sa.UniqueConstraint('dag_id', 'run_id'))
def downgrade():
diff --git a/airflow/migrations/versions/211e584da130_add_ti_state_index.py b/airflow/migrations/versions/211e584da130_add_ti_state_index.py
index 93347a2649..b17f390e0b 100644
--- a/airflow/migrations/versions/211e584da130_add_ti_state_index.py
+++ b/airflow/migrations/versions/211e584da130_add_ti_state_index.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,6 +24,7 @@
Create Date: 2016-06-30 10:54:24.323588
"""
+from alembic import op
# revision identifiers, used by Alembic.
revision = '211e584da130'
@@ -31,9 +32,6 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
op.create_index('ti_state', 'task_instance', ['state'], unique=False)
diff --git a/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py b/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
index b7213a3031..a757d27709 100644
--- a/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
+++ b/airflow/migrations/versions/27c6a30d7c24_add_executor_config_to_task_instance.py
@@ -1,16 +1,22 @@
# flake8: noqa
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
"""kubernetes_resource_checkpointing
@@ -20,18 +26,16 @@
"""
+from alembic import op
+import sqlalchemy as sa
+import dill
+
# revision identifiers, used by Alembic.
revision = '27c6a30d7c24'
down_revision = '33ae817a1ff4'
branch_labels = None
depends_on = None
-
-from alembic import op
-import sqlalchemy as sa
-import dill
-
-
TASK_INSTANCE_TABLE = "task_instance"
NEW_COLUMN = "executor_config"
@@ -42,4 +46,3 @@ def upgrade():
def downgrade():
op.drop_column(TASK_INSTANCE_TABLE, NEW_COLUMN)
-
diff --git a/airflow/migrations/versions/2e541a1dcfed_task_duration.py b/airflow/migrations/versions/2e541a1dcfed_task_duration.py
index 7b540430a5..595a5774a6 100644
--- a/airflow/migrations/versions/2e541a1dcfed_task_duration.py
+++ b/airflow/migrations/versions/2e541a1dcfed_task_duration.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -25,16 +25,16 @@
"""
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import mysql
+
# revision identifiers, used by Alembic.
revision = '2e541a1dcfed'
down_revision = '1b38cef5b76e'
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-from sqlalchemy.dialects import mysql
-
def upgrade():
# use batch_alter_table to support SQLite workaround
diff --git a/airflow/migrations/versions/2e82aab8ef20_rename_user_table.py b/airflow/migrations/versions/2e82aab8ef20_rename_user_table.py
index 0326667927..fc8a1aab20 100644
--- a/airflow/migrations/versions/2e82aab8ef20_rename_user_table.py
+++ b/airflow/migrations/versions/2e82aab8ef20_rename_user_table.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,6 +24,7 @@
Create Date: 2016-04-02 19:28:15.211915
"""
+from alembic import op
# revision identifiers, used by Alembic.
revision = '2e82aab8ef20'
@@ -31,9 +32,6 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
op.rename_table('user', 'users')
@@ -41,4 +39,3 @@ def upgrade():
def downgrade():
op.rename_table('users', 'user')
-
diff --git a/airflow/migrations/versions/338e90f54d61_more_logging_into_task_isntance.py b/airflow/migrations/versions/338e90f54d61_more_logging_into_task_isntance.py
index c101e4a477..473f76778b 100644
--- a/airflow/migrations/versions/338e90f54d61_more_logging_into_task_isntance.py
+++ b/airflow/migrations/versions/338e90f54d61_more_logging_into_task_isntance.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -17,13 +17,17 @@
# specific language governing permissions and limitations
# under the License.
-"""More logging into task_isntance
+"""More logging into task_instance
Revision ID: 338e90f54d61
Revises: 13eb55f81627
Create Date: 2015-08-25 06:09:20.460147
"""
+# flake8: noqa: E266
+
+from alembic import op
+import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '338e90f54d61'
@@ -31,9 +35,6 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
### commands auto generated by Alembic - please adjust! ###
diff --git a/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py b/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
index 4347bae92a..88a4199d32 100644
--- a/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
+++ b/airflow/migrations/versions/33ae817a1ff4_add_kubernetes_resource_checkpointing.py
@@ -1,16 +1,22 @@
# flake8: noqa
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
"""kubernetes_resource_checkpointing
@@ -19,6 +25,8 @@
Create Date: 2017-09-11 15:26:47.598494
"""
+from alembic import op
+import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '33ae817a1ff4'
@@ -26,20 +34,27 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
-
RESOURCE_TABLE = "kube_resource_version"
def upgrade():
+
+ columns_and_constraints = [
+ sa.Column("one_row_id", sa.Boolean, server_default=sa.true(), primary_key=True),
+ sa.Column("resource_version", sa.String(255))
+ ]
+
+ conn = op.get_bind()
+
+ # alembic creates an invalid SQL for mssql dialect
+ if conn.dialect.name not in ('mssql'):
+ columns_and_constraints.append(sa.CheckConstraint("one_row_id", name="kube_resource_version_one_row_id"))
+
table = op.create_table(
RESOURCE_TABLE,
- sa.Column("one_row_id", sa.Boolean, server_default=sa.true(), primary_key=True),
- sa.Column("resource_version", sa.String(255)),
- sa.CheckConstraint("one_row_id", name="kube_resource_version_one_row_id")
+ *columns_and_constraints
)
+
op.bulk_insert(table, [
{"resource_version": ""}
])
@@ -47,4 +62,3 @@ def upgrade():
def downgrade():
op.drop_table(RESOURCE_TABLE)
-
diff --git a/airflow/migrations/versions/40e67319e3a9_dagrun_config.py b/airflow/migrations/versions/40e67319e3a9_dagrun_config.py
index ecfe58cfc3..3da4d5f543 100644
--- a/airflow/migrations/versions/40e67319e3a9_dagrun_config.py
+++ b/airflow/migrations/versions/40e67319e3a9_dagrun_config.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,6 +24,8 @@
Create Date: 2015-10-29 08:36:31.726728
"""
+from alembic import op
+import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '40e67319e3a9'
@@ -31,9 +33,6 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
op.add_column('dag_run', sa.Column('conf', sa.PickleType(), nullable=True))
diff --git a/airflow/migrations/versions/4446e08588_dagrun_start_end.py b/airflow/migrations/versions/4446e08588_dagrun_start_end.py
index dc546f8c77..29932c9206 100644
--- a/airflow/migrations/versions/4446e08588_dagrun_start_end.py
+++ b/airflow/migrations/versions/4446e08588_dagrun_start_end.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -25,15 +25,15 @@
"""
+from alembic import op
+import sqlalchemy as sa
+
# revision identifiers, used by Alembic.
revision = '4446e08588'
down_revision = '561833c1c74b'
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
op.add_column('dag_run', sa.Column('end_date', sa.DateTime(), nullable=True))
diff --git a/airflow/migrations/versions/4addfa1236f1_add_fractional_seconds_to_mysql_tables.py b/airflow/migrations/versions/4addfa1236f1_add_fractional_seconds_to_mysql_tables.py
index 80af578367..655ff61042 100644
--- a/airflow/migrations/versions/4addfa1236f1_add_fractional_seconds_to_mysql_tables.py
+++ b/airflow/migrations/versions/4addfa1236f1_add_fractional_seconds_to_mysql_tables.py
@@ -6,9 +6,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,93 +24,147 @@
"""
+from alembic import op
+from sqlalchemy.dialects import mysql
+from alembic import context
+
# revision identifiers, used by Alembic.
revision = '4addfa1236f1'
down_revision = 'f2ca10b85618'
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-from sqlalchemy.dialects import mysql
-from alembic import context
-
def upgrade():
if context.config.get_main_option('sqlalchemy.url').startswith('mysql'):
- op.alter_column(table_name='dag', column_name='last_scheduler_run', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='dag', column_name='last_pickled', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='dag', column_name='last_expired', type_=mysql.DATETIME(fsp=6))
-
- op.alter_column(table_name='dag_pickle', column_name='created_dttm', type_=mysql.DATETIME(fsp=6))
-
- op.alter_column(table_name='dag_run', column_name='execution_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='dag_run', column_name='start_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='dag_run', column_name='end_date', type_=mysql.DATETIME(fsp=6))
-
- op.alter_column(table_name='import_error', column_name='timestamp', type_=mysql.DATETIME(fsp=6))
-
- op.alter_column(table_name='job', column_name='start_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='job', column_name='end_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='job', column_name='latest_heartbeat', type_=mysql.DATETIME(fsp=6))
-
- op.alter_column(table_name='known_event', column_name='start_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='known_event', column_name='end_date', type_=mysql.DATETIME(fsp=6))
-
- op.alter_column(table_name='log', column_name='dttm', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='log', column_name='execution_date', type_=mysql.DATETIME(fsp=6))
-
- op.alter_column(table_name='sla_miss', column_name='execution_date', type_=mysql.DATETIME(fsp=6), nullable=False)
- op.alter_column(table_name='sla_miss', column_name='timestamp', type_=mysql.DATETIME(fsp=6))
-
- op.alter_column(table_name='task_fail', column_name='execution_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='task_fail', column_name='start_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='task_fail', column_name='end_date', type_=mysql.DATETIME(fsp=6))
-
- op.alter_column(table_name='task_instance', column_name='execution_date', type_=mysql.DATETIME(fsp=6), nullable=False)
- op.alter_column(table_name='task_instance', column_name='start_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='task_instance', column_name='end_date', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='task_instance', column_name='queued_dttm', type_=mysql.DATETIME(fsp=6))
-
- op.alter_column(table_name='xcom', column_name='timestamp', type_=mysql.DATETIME(fsp=6))
- op.alter_column(table_name='xcom', column_name='execution_date', type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='dag', column_name='last_scheduler_run',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='dag', column_name='last_pickled',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='dag', column_name='last_expired',
+ type_=mysql.DATETIME(fsp=6))
+
+ op.alter_column(table_name='dag_pickle', column_name='created_dttm',
+ type_=mysql.DATETIME(fsp=6))
+
+ op.alter_column(table_name='dag_run', column_name='execution_date',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='dag_run', column_name='start_date',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='dag_run', column_name='end_date',
+ type_=mysql.DATETIME(fsp=6))
+
+ op.alter_column(table_name='import_error', column_name='timestamp',
+ type_=mysql.DATETIME(fsp=6))
+
+ op.alter_column(table_name='job', column_name='start_date',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='job', column_name='end_date',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='job', column_name='latest_heartbeat',
+ type_=mysql.DATETIME(fsp=6))
+
+ op.alter_column(table_name='known_event', column_name='start_date',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='known_event', column_name='end_date',
+ type_=mysql.DATETIME(fsp=6))
+
+ op.alter_column(table_name='log', column_name='dttm',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='log', column_name='execution_date',
+ type_=mysql.DATETIME(fsp=6))
+
+ op.alter_column(table_name='sla_miss', column_name='execution_date',
+ type_=mysql.DATETIME(fsp=6),
+ nullable=False)
+ op.alter_column(table_name='sla_miss', column_name='timestamp',
+ type_=mysql.DATETIME(fsp=6))
+
+ op.alter_column(table_name='task_fail', column_name='execution_date',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='task_fail', column_name='start_date',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='task_fail', column_name='end_date',
+ type_=mysql.DATETIME(fsp=6))
+
+ op.alter_column(table_name='task_instance', column_name='execution_date',
+ type_=mysql.DATETIME(fsp=6),
+ nullable=False)
+ op.alter_column(table_name='task_instance', column_name='start_date',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='task_instance', column_name='end_date',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='task_instance', column_name='queued_dttm',
+ type_=mysql.DATETIME(fsp=6))
+
+ op.alter_column(table_name='xcom', column_name='timestamp',
+ type_=mysql.DATETIME(fsp=6))
+ op.alter_column(table_name='xcom', column_name='execution_date',
+ type_=mysql.DATETIME(fsp=6))
def downgrade():
if context.config.get_main_option('sqlalchemy.url').startswith('mysql'):
- op.alter_column(table_name='dag', column_name='last_scheduler_run', type_=mysql.DATETIME())
- op.alter_column(table_name='dag', column_name='last_pickled', type_=mysql.DATETIME())
- op.alter_column(table_name='dag', column_name='last_expired', type_=mysql.DATETIME())
-
- op.alter_column(table_name='dag_pickle', column_name='created_dttm', type_=mysql.DATETIME())
-
- op.alter_column(table_name='dag_run', column_name='execution_date', type_=mysql.DATETIME())
- op.alter_column(table_name='dag_run', column_name='start_date', type_=mysql.DATETIME())
- op.alter_column(table_name='dag_run', column_name='end_date', type_=mysql.DATETIME())
-
- op.alter_column(table_name='import_error', column_name='timestamp', type_=mysql.DATETIME())
-
- op.alter_column(table_name='job', column_name='start_date', type_=mysql.DATETIME())
- op.alter_column(table_name='job', column_name='end_date', type_=mysql.DATETIME())
- op.alter_column(table_name='job', column_name='latest_heartbeat', type_=mysql.DATETIME())
-
- op.alter_column(table_name='known_event', column_name='start_date', type_=mysql.DATETIME())
- op.alter_column(table_name='known_event', column_name='end_date', type_=mysql.DATETIME())
-
- op.alter_column(table_name='log', column_name='dttm', type_=mysql.DATETIME())
- op.alter_column(table_name='log', column_name='execution_date', type_=mysql.DATETIME())
-
- op.alter_column(table_name='sla_miss', column_name='execution_date', type_=mysql.DATETIME(), nullable=False)
- op.alter_column(table_name='sla_miss', column_name='timestamp', type_=mysql.DATETIME())
-
- op.alter_column(table_name='task_fail', column_name='execution_date', type_=mysql.DATETIME())
- op.alter_column(table_name='task_fail', column_name='start_date', type_=mysql.DATETIME())
- op.alter_column(table_name='task_fail', column_name='end_date', type_=mysql.DATETIME())
-
- op.alter_column(table_name='task_instance', column_name='execution_date', type_=mysql.DATETIME(), nullable=False)
- op.alter_column(table_name='task_instance', column_name='start_date', type_=mysql.DATETIME())
- op.alter_column(table_name='task_instance', column_name='end_date', type_=mysql.DATETIME())
- op.alter_column(table_name='task_instance', column_name='queued_dttm', type_=mysql.DATETIME())
-
- op.alter_column(table_name='xcom', column_name='timestamp', type_=mysql.DATETIME())
- op.alter_column(table_name='xcom', column_name='execution_date', type_=mysql.DATETIME())
+ op.alter_column(table_name='dag', column_name='last_scheduler_run',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='dag', column_name='last_pickled',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='dag', column_name='last_expired',
+ type_=mysql.DATETIME())
+
+ op.alter_column(table_name='dag_pickle', column_name='created_dttm',
+ type_=mysql.DATETIME())
+
+ op.alter_column(table_name='dag_run', column_name='execution_date',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='dag_run', column_name='start_date',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='dag_run', column_name='end_date',
+ type_=mysql.DATETIME())
+
+ op.alter_column(table_name='import_error', column_name='timestamp',
+ type_=mysql.DATETIME())
+
+ op.alter_column(table_name='job', column_name='start_date',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='job', column_name='end_date',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='job', column_name='latest_heartbeat',
+ type_=mysql.DATETIME())
+
+ op.alter_column(table_name='known_event', column_name='start_date',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='known_event', column_name='end_date',
+ type_=mysql.DATETIME())
+
+ op.alter_column(table_name='log', column_name='dttm',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='log', column_name='execution_date',
+ type_=mysql.DATETIME())
+
+ op.alter_column(table_name='sla_miss', column_name='execution_date',
+ type_=mysql.DATETIME(), nullable=False)
+ op.alter_column(table_name='sla_miss', column_name='timestamp',
+ type_=mysql.DATETIME())
+
+ op.alter_column(table_name='task_fail', column_name='execution_date',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='task_fail', column_name='start_date',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='task_fail', column_name='end_date',
+ type_=mysql.DATETIME())
+
+ op.alter_column(table_name='task_instance', column_name='execution_date',
+ type_=mysql.DATETIME(),
+ nullable=False)
+ op.alter_column(table_name='task_instance', column_name='start_date',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='task_instance', column_name='end_date',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='task_instance', column_name='queued_dttm',
+ type_=mysql.DATETIME())
+
+ op.alter_column(table_name='xcom', column_name='timestamp',
+ type_=mysql.DATETIME())
+ op.alter_column(table_name='xcom', column_name='execution_date',
+ type_=mysql.DATETIME())
diff --git a/airflow/migrations/versions/502898887f84_adding_extra_to_log.py b/airflow/migrations/versions/502898887f84_adding_extra_to_log.py
index f5cded7ea5..632720a4e2 100644
--- a/airflow/migrations/versions/502898887f84_adding_extra_to_log.py
+++ b/airflow/migrations/versions/502898887f84_adding_extra_to_log.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,6 +24,8 @@
Create Date: 2015-11-03 22:50:49.794097
"""
+from alembic import op
+import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '502898887f84'
@@ -31,9 +33,6 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
op.add_column('log', sa.Column('extra', sa.Text(), nullable=True))
diff --git a/airflow/migrations/versions/52d714495f0_job_id_indices.py b/airflow/migrations/versions/52d714495f0_job_id_indices.py
index 43893dba8a..94374cb68a 100644
--- a/airflow/migrations/versions/52d714495f0_job_id_indices.py
+++ b/airflow/migrations/versions/52d714495f0_job_id_indices.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,6 +24,7 @@
Create Date: 2015-10-20 03:17:01.962542
"""
+from alembic import op
# revision identifiers, used by Alembic.
revision = '52d714495f0'
@@ -31,12 +32,10 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
- op.create_index('idx_job_state_heartbeat', 'job', ['state', 'latest_heartbeat'], unique=False)
+ op.create_index('idx_job_state_heartbeat', 'job',
+ ['state', 'latest_heartbeat'], unique=False)
def downgrade():
diff --git a/airflow/migrations/versions/561833c1c74b_add_password_column_to_user.py b/airflow/migrations/versions/561833c1c74b_add_password_column_to_user.py
index ad82c9378b..a26a105ac8 100644
--- a/airflow/migrations/versions/561833c1c74b_add_password_column_to_user.py
+++ b/airflow/migrations/versions/561833c1c74b_add_password_column_to_user.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,6 +24,8 @@
Create Date: 2015-11-30 06:51:25.872557
"""
+from alembic import op
+import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '561833c1c74b'
@@ -31,9 +33,6 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
op.add_column('user', sa.Column('password', sa.String(255)))
diff --git a/airflow/migrations/versions/5e7d17757c7a_add_pid_field_to_taskinstance.py b/airflow/migrations/versions/5e7d17757c7a_add_pid_field_to_taskinstance.py
index 5e2963f060..77a35db48f 100644
--- a/airflow/migrations/versions/5e7d17757c7a_add_pid_field_to_taskinstance.py
+++ b/airflow/migrations/versions/5e7d17757c7a_add_pid_field_to_taskinstance.py
@@ -6,9 +6,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,15 +24,15 @@
"""
+from alembic import op
+import sqlalchemy as sa
+
# revision identifiers, used by Alembic.
revision = '5e7d17757c7a'
down_revision = '8504051e801b'
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
op.add_column('task_instance', sa.Column('pid', sa.Integer))
diff --git a/airflow/migrations/versions/64de9cddf6c9_add_task_fails_journal_table.py b/airflow/migrations/versions/64de9cddf6c9_add_task_fails_journal_table.py
index be74738328..2def57e904 100644
--- a/airflow/migrations/versions/64de9cddf6c9_add_task_fails_journal_table.py
+++ b/airflow/migrations/versions/64de9cddf6c9_add_task_fails_journal_table.py
@@ -6,9 +6,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -23,6 +23,8 @@
Create Date: 2016-08-03 14:02:59.203021
"""
+from alembic import op
+import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '64de9cddf6c9'
@@ -30,9 +32,6 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
op.create_table(
@@ -47,5 +46,6 @@ def upgrade():
sa.PrimaryKeyConstraint('id'),
)
+
def downgrade():
op.drop_table('task_fail')
diff --git a/airflow/migrations/versions/8504051e801b_xcom_dag_task_indices.py b/airflow/migrations/versions/8504051e801b_xcom_dag_task_indices.py
index 47473e318d..fdcbc59df8 100644
--- a/airflow/migrations/versions/8504051e801b_xcom_dag_task_indices.py
+++ b/airflow/migrations/versions/8504051e801b_xcom_dag_task_indices.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -25,18 +25,18 @@
"""
+from alembic import op
+
# revision identifiers, used by Alembic.
revision = '8504051e801b'
down_revision = '4addfa1236f1'
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
def upgrade():
- op.create_index('idx_xcom_dag_task_date', 'xcom', ['dag_id', 'task_id', 'execution_date'], unique=False)
+ op.create_index('idx_xcom_dag_task_date', 'xcom',
+ ['dag_id', 'task_id', 'execution_date'], unique=False)
def downgrade():
diff --git a/airflow/migrations/versions/856955da8476_fix_sqlite_foreign_key.py b/airflow/migrations/versions/856955da8476_fix_sqlite_foreign_key.py
new file mode 100644
index 0000000000..52a817081b
--- /dev/null
+++ b/airflow/migrations/versions/856955da8476_fix_sqlite_foreign_key.py
@@ -0,0 +1,88 @@
+# flake8: noqa
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""fix sqlite foreign key
+
+Revision ID: 856955da8476
+Revises: f23433877c24
+Create Date: 2018-06-17 15:54:53.844230
+
+"""
+
+from alembic import op
+import sqlalchemy as sa
+
+# revision identifiers, used by Alembic.
+revision = '856955da8476'
+down_revision = 'f23433877c24'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+ conn = op.get_bind()
+ if conn.dialect.name == 'sqlite':
+ # Fix broken foreign-key constraint for existing SQLite DBs.
+ #
+ # Re-define tables and use copy_from to avoid reflection
+ # which would fail because referenced user table doesn't exist.
+ #
+ # Use batch_alter_table to support SQLite workaround.
+ chart_table = sa.Table('chart',
+ sa.MetaData(),
+ sa.Column('id', sa.Integer(), nullable=False),
+ sa.Column('label', sa.String(length=200), nullable=True),
+ sa.Column('conn_id', sa.String(length=250), nullable=False),
+ sa.Column('user_id', sa.Integer(), nullable=True),
+ sa.Column('chart_type', sa.String(length=100), nullable=True),
+ sa.Column('sql_layout', sa.String(length=50), nullable=True),
+ sa.Column('sql', sa.Text(), nullable=True),
+ sa.Column('y_log_scale', sa.Boolean(), nullable=True),
+ sa.Column('show_datatable', sa.Boolean(), nullable=True),
+ sa.Column('show_sql', sa.Boolean(), nullable=True),
+ sa.Column('height', sa.Integer(), nullable=True),
+ sa.Column('default_params', sa.String(length=5000), nullable=True),
+ sa.Column('x_is_date', sa.Boolean(), nullable=True),
+ sa.Column('iteration_no', sa.Integer(), nullable=True),
+ sa.Column('last_modified', sa.DateTime(), nullable=True),
+ sa.PrimaryKeyConstraint('id'))
+ with op.batch_alter_table('chart', copy_from=chart_table) as batch_op:
+ batch_op.create_foreign_key('chart_user_id_fkey', 'users',
+ ['user_id'], ['id'])
+
+ known_event_table = sa.Table('known_event',
+ sa.MetaData(),
+ sa.Column('id', sa.Integer(), nullable=False),
+ sa.Column('label', sa.String(length=200), nullable=True),
+ sa.Column('start_date', sa.DateTime(), nullable=True),
+ sa.Column('end_date', sa.DateTime(), nullable=True),
+ sa.Column('user_id', sa.Integer(), nullable=True),
+ sa.Column('known_event_type_id', sa.Integer(), nullable=True),
+ sa.Column('description', sa.Text(), nullable=True),
+ sa.ForeignKeyConstraint(['known_event_type_id'],
+ ['known_event_type.id'], ),
+ sa.PrimaryKeyConstraint('id'))
+ with op.batch_alter_table('chart', copy_from=known_event_table) as batch_op:
+ batch_op.create_foreign_key('known_event_user_id_fkey', 'users',
+ ['user_id'], ['id'])
+
+
+def downgrade():
+ # Downgrade would fail because the broken FK constraint can't be re-created.
+ pass
diff --git a/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py b/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
index 6bc48f1105..633201b03b 100644
--- a/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
+++ b/airflow/migrations/versions/86770d1215c0_add_kubernetes_scheduler_uniqueness.py
@@ -1,16 +1,22 @@
# flake8: noqa
#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
"""add kubernetes scheduler uniqueness
@@ -19,6 +25,8 @@
Create Date: 2018-04-03 15:31:20.814328
"""
+from alembic import op
+import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '86770d1215c0'
@@ -26,20 +34,27 @@
branch_labels = None
depends_on = None
-from alembic import op
-import sqlalchemy as sa
-
-
RESOURCE_TABLE = "kube_worker_uuid"
def upgrade():
+
+ columns_and_constraints = [
+ sa.Column("one_row_id", sa.Boolean, server_default=sa.true(), primary_key=True),
+ sa.Column("worker_uuid", sa.String(255))
+ ]
+
+ conn = op.get_bind()
+
+ # alembic creates an invalid SQL for mssql dialect
+ if conn.dialect.name not in ('mssql'):
+ columns_and_constraints.append(sa.CheckConstraint("one_row_id", name="kube_worker_one_row_id"))
+
table = op.create_table(
RESOURCE_TABLE,
- sa.Column("one_row_id", sa.Boolean, server_default=sa.true(), primary_key=True),
- sa.Column("worker_uuid", sa.String(255)),
- sa.CheckConstraint("one_row_id", name="kube_worker_one_row_id")
+ *columns_and_constraints
)
+
op.bulk_insert(table, [
{"worker_uuid": ""}
])
diff --git a/airflow/migrations/versions/947454bf1dff_add_ti_job_id_index.py b/airflow/migrations/versions/947454bf1dff_add_ti_job_id_index.py
index 4c8bce9b47..6ff41baa28 100644
--- a/airflow/migrations/versions/947454bf1dff_add_ti_job_id_index.py
+++ b/airflow/migrations/versions/947454bf1dff_add_ti_job_id_index.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
(This diff was longer than 20,000 lines, and has been truncated...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services