You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/08/27 09:11:40 UTC
[airflow] tag nightly-master updated (2102122 -> 276a15a)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/airflow.git.
*** WARNING: tag nightly-master was modified! ***
from 2102122 (commit)
to 276a15a (commit)
from 2102122 Handle IntegrityError while creating TIs (#10136)
add cb40517 Merge similar sections on docs/howto/connection/index.rst (#10224)
add 6fe6721 Add missing headinsg on docs/security.rst (#10225)
add c2fa100 Fixed wrong name of workflow in cancel step (#10219)
add 03c48f7 Add MeuVendoo to Airflow Users (#10226)
add d9a499c BoringCyborg Bot: Fix Automated Labels for serialized & secrets (#10228)
add f5b9a0d Add Apache License to .github/workflows/repo-sync.yml (#10229)
add d82a127 Fix typo in .github/ISSUE_TEMPLATE/bug_report.md (#10231)
add 14ee709 Fix typos in docs/howto/secrets-backend (#10233)
add 8a655cf Add airflow connections get command (#10214)
add c295338 Add labels param to Google MLEngine Operators (#10222)
add 0d8bc95 Replace remaining uses of "bail" (#10217)
add 8986e75 Add Playsimple Games to "Who uses Apache Airflow?" (#10253)
add ae9dd56 Fix more typos in docs/ (#10251)
add edc51e3 Remove Unnecessary list literal in Tuple for Kylin Operator (#10252)
add 620400f Add whitespace around operator in docs/build (#10250)
add e7c069c Triggering DAG with Future Date (#10249)
add 53683b8 Fix typo in docs/stable-rest-api/redoc.rst (#10248)
add b989cea Fix link for the Jinja Project in docs/tutorial.rst (#10245)
add 69225d7 Add 'apache.beam' to docs/installation.rst (#10244)
add b7e4d33 Improve documentation in docs/start.rst (#10243)
add 31afc0d Move celery-exclusive feature to CeleryExecutor page (#10242)
add 77f2e22 Add tip about airflow config command on docs/howto/secrets-backend/index.rst (#10239)
add aba29bf Set language on code-block on docs/howto/email-config.rst (#10238)
add 12eed9d Add system tests for CloudSecretManagerBackend (#10235)
add 7930234 Remove redundant "and_" condition when using filter (#10232)
add 637a2c1 Create separate section for Cron Presets (#10247)
add 55021b7 Improve guide about Google Cloud Secret Manager Backend (#10257)
add e2ec5ef Update example on docs/howto/connection/index.rst (#10236)
add e054870 Add Syntax Highlights to code-blocks in docs/best-practices.rst (#10258)
add db8d06a Disable sentry integration by default (#10212)
add 2869b48 Increse number of runs for quarantined tests (#10220)
add d7c4eee Fix redirects URLs (#10259)
add 183cb8d Create "Managing variable" in howto directory (#10241)
add 5503a6a Fix Warning when using a different Sphinx Builder (#10262)
add b43f90a Fix various typos in the repo (#10263)
add 06a1836 Add Missing Apache Providers to docs/installation.rst (#10265)
add ef08831 Added DataprepGetJobsForJobGroupOperator (#10246)
add a3386e5 Fixed GitHub Actions badge (#10268)
add f06fe61 Add Amazon SES hook (#10004)
add 19bc97d Revert "Add Amazon SES hook (#10004)" (#10276)
add 5df2ed1 Add more columns to airflow connections get (#10269)
add dd17c0e Update Gojek in who uses list (#10281)
add 2e3c878 Makes multi-namespace mode optional (#9570)
add a9f7222 Remove duplicate line from 1.10.10 CHANGELOG (#10289)
add 8cd2be9 Fix KubernetesPodOperator reattachment (#10230)
add 82f744b Add type annotations to AwsGlueJobHook, RedshiftHook modules (#10286)
add 422e3f1 Add Authentication for Stable API (#10267)
add 3c374a4 Add reconcile_metadata to reconcile_pods (#10266)
add 0ee4375 Add unittest for WasbTaskHandler (#10284)
add adce6f0 Use Hash of Serialized DAG to determine DAG is changed or not (#10227)
add 1266b29 Run create-user-job as user with specified id (#10291)
add f618cdd Docs: Clarify DAG to image language (#10296)
add 8f8db89 DbApiHook: Support kwargs in get_pandas_df (#9730)
add 4aa05a7 Fixes name of pre-commit cache for multiple branches (#10299)
add ba66ba0 Clarify connection docs (#10294)
add f6734b3 Enable Sphinx spellcheck for doc generation (#10280)
add 7f76b8b Add ClusterPolicyViolation support to airflow local settings (#10282)
add 3b0df4d Use more human readable table heading labels on DAG details (#10305)
add 45e3182 Restrict google-cloud-dataproc to <2.0.0 (#10307)
add a74a7da Limit all google-cloud api to <2.0.0 (#10317)
add 2f0613b Implement Google BigQuery Table Partition Sensor (#10218)
add f91ac9b Unpin pytest (#10314)
add 98f597a Added "sharded" word to spellchecker (#10320)
add 2d4e44c Respect DAG Serialization setting when running sync_perm (#10321)
add 47387a6 Catch Permission Denied exception when getting secret from GCP Secret Manager. (#10326)
add 4454224 Fix clear future recursive when ExternalTaskMarker is used (#9515)
add 5c2bb7b Webserver: Sanitize values passed to origin param (#10334)
add be46d20 Improve idempotency of BigQueryInsertJobOperator (#9590)
add bfa5a8d CI: Fix failing docs-build (#10342)
add 4fc2536 Add ingress to the helm chart (#10064)
add 46ec705 Fixes rat-check pre-commit in case Airflow is added as subrepo (#10347)
add 8e387aa Breeze was slightly too chatty when there was no dirs created (#10346)
add f76936f More informative description of Breeze's --verbose flag. (#10348)
add 88e5c35 Breeze: More fancy environment checking (#10329)
add 6656464 Improve language of a BaseSensorOperator in UPDATING.md (#10332)
add 382c101 Add Bigtable Update Instance Hook/Operator (#10340)
add e195a98 Add type annotations for mlengine_operator_utils (#10297)
add d4e3a42 add more precise type hint for task callbacks (#10355)
add 1ae5bdf Add test for GCSTaskHandler (#9600) (#9861)
add dea345b Fix AwsGlueJobSensor to stop running after the Glue job finished (#9022)
add 558be73 Add redbubble link to Airflow merch (#10359)
add 89b8bdf Docs: Seperate page for each security topic (#10352)
add b51de98 Add to CONTRIBUTING.rst link to backport packages troubleshooting (#10360)
add 01f3796 Add typing coverage to mysql providers package (#10095)
add 439f7dc Use check_output to capture in celery task (#10310)
add 083c3c1 Simplified GCSTaskHandler configuration (#10365)
add 51a1b38 Pylintable list-integrations.py (#10378)
add 98ac27a Simplify cron preset language in docs (#10370)
add 485ecc4 Too much was happening in this pre-commit script (#10345)
add 9228bf2 You can disable spellcheck or documentation when building docs. (#10377)
add b28ea7a Docs: Fix spacing bug in 'Dag Run' (#10372)
add a32e90a Update github flags for Breeze (#10384)
add d6f6d53 Expand JenkinsJobTriggerOperator unit tests (#10353)
add 306a666 Docker images are now consistently labelled and a bit smaller (#10387)
add a3df617 Add guide about custom API authentication (#10312)
add ba00db3 Consistently refer to section names (#10369)
add 8368f49 Correct verb tense for re-running task doc. (#10371)
add 541c47c Add basic auth API auth backend (#10356)
add 49ce908 Moved description of page size limit to security/ (#10392)
add c54d17e Capitalize 'Python' properly in Concepts docs (#10398)
add 77a635e When precommits are run, output is silenced (#10390)
add 3bc3701 Add back 'refresh_all' method in airflow/www/views.py (#10328)
add 2bab38c Update celery.rst (#10400)
add b06a705 Improve headings on docs/executor (#10396)
add 0b3ded7 Correct typo in best-practices.rst (#10401)
add e179853 Kubernetes image is extended rather than customized (#10399)
add db446f2 Replaced aliases for common tools with functions. (#10402)
add 08fe5c4 Constraint CI scripts are now separated out (#10404)
add e1e7f11 Move docker-compose ci.yml to ga.yml as it is GITHUB_* only (#10405)
add 8fcb93b Fixes optimisation where doc only change should build much faster (#10344)
add 882e187 Remove run-ons from scheduler docs. (#10397)
add 3d334fd BugFix: K8s Executor Multinamespace mode is evaluated to true by default (#10410)
add f76938c Make Kubernetes tests pass locally (#10407)
add e195c6a Make KubernetesExecutor recognize kubernetes_labels (#10412)
add 2db8bf3 Group logging & monitoring guides in one section (#10394)
add f1716bc Use sys.exit() instead of exit() (#10414)
add 7fa813f Unnecessary use of list comprehension (#10416)
add 5739ba2 Fix broken breeze script (#10418)
add de7500d CI Images are now pre-build and stored in registry (#10368)
add 2c3ce8e Enable optimisation of image building. (#10422)
add a8e28f1 Fix typo in KubernetesPodOperator (#10419)
add dc27a2a Fix failing breeze (#10424)
add c35a010 Switch to released cancel-workflow-runs action (#10423)
add 30f4617 Add architecture diagram for basic Airflow deployment (#10428)
add 88c7d2e Dataflow operators don't not always create a virtualenv (#10373)
add 27d08b7 Amazon SES Hook (#10391)
add 52dec7b Fix Breeze failure on MacOS (#10440)
add 2f55223 Add AzureBaseHook (#9747)
add 1e37186 Add update endpoint for DAG (#9101) (#9740)
add 95f4768 Building backport packages generates README files (#10445)
add 5bf47e3 Be nice to fork repositories when it comes to scheduled events (#10448)
add fdd68ec Make system test work with 1.10 (#10444)
add 1cf1af6 Do not override in_container scripts when building the image (#10442)
add f0727b3 Change provider configuration keys for OAuth (#9759)
add 3357d8d Fix port number in webserver for kind setup (#10452)
add 7dd6e73 Mount gcloud kubeconifg to breeze (#10439)
add c979de7 Fixes problem with Python image not ready to be pushed (#10430)
add 9dd92a8 Fix typo in "wait_for_done" (#10458)
add 264c466 Fix identation in executor_config example (#10467)
add 4a77211 Remove redudandant checks in test_views.py (#10464)
add 904c1d8 Test exact match of Executor name (#10465)
add 44a36b9 Use assertEqual instead of assertTrue in tests/utils/test_dates.py for proper diff (#10457)
add 7c206a8 Replace assigment with Augmented assignment (#10468)
add 90b9e7e Add Type Annotations & Docstrings to airflow/models/dagrun.py (#10466)
add a8f83ff Fix typos in scripts/perf/scheduler_dag_execution_timing.py (#10463)
add 27339a5 Remove mentions of Airflow Gitter (#10460)
add 515cc72 Fix typo in timed_out (#10459)
add ce9cc1b Stops running workflow_run for scheduled runs in forks (#10473)
add 9774903 Fixes quoting bug introduced in #10473 (#10477)
add 47c6657 Fix broken Kubernetes PodRuntimeInfoEnv (#10478)
add 0c4f7cd Change Support Request template to a link to Slack (#10480)
add c635804 Fixes S3ToRedshift COPY query (#10436)
add 93ba98c Optimise production image building during k8s tests on CI (#10476)
add ee7ca12 Fix broken Markdown refernces in Providers README (#10483)
add 7ee7d7c Move perf_kit to tests.utils (#10470)
add dc3a493 Fix duplicate task_ids in example_http.py (#10485)
add ef8df17 Fix typo in Facebook Ads Provider (#10484)
add c8c3f8b Remove old configuration from BoringCyborg (#10490)
add 82369fa Removed the prerequisite for perf-kit path augmentation (#10492)
add 6eb64ce Make macros.hive pylint compatible (#10495)
add 5c6e4e2 Make Stats pylint compatible (#10496)
add 3a53039 GitHub Registry is now lowercase (#10489)
add b0598b5 Add support for creating multiple replicated clusters in Bigtable hook and operator (#10475)
add 3734876 Implement impersonation in google operators (#10052)
add 946c9de Fixes uploading of doc artifacts. (#10441)
add 4fa7df5 Mounting from sources is disabled for tests (#10472)
add 92858cc Add instructions to verify the release candidate (#10493)
add 8a2c21d Updating Auth0 contributors (#10432)
add be1a67b Make models/crypto.py Pylint-compatible (#10500)
add a128667 Alphabetize committer list (#10512)
add d798257 Add Hurb.com as Airflow User (#10518)
add 08c5135 Cleanup Astronomer contributors list in README (#10520)
add fef73b9 Fix impersonation related bug in bigtable tests (#10521)
add 570f751 Sets default timeout for the job waiting for images (#10517)
add a1eed07 Improve direct impersonation documentation (#10506)
add 8fdcc57 Make www/views.py pylint compatible (#10498)
add f2da6b4 Updated documentation for the CI with mermaid sequence diagrams (#10380)
add 2f2d8db Remove all "noinspection" comments native to IntelliJ (#10525)
add 3696c34 Fix typo in the word "release" (#10528)
add 4f6d53e Make models/taskinstance.py pylint compatible (#10499)
add 1775474 Make configuration.py Pylint compatible (#10494)
add 2dab0e2 Make www/utils.py pylint-compatible (#10497)
add 47265e7 Fix typo in PostgresHook (#10529)
add bdec5fa Improving descriptions in OpenAPI (#10417)
add 74729d6 Fix typo in "Success" (#10537)
add 866701c Fix typo in "Cloud" (#10534)
add 6bed074 Remove unreachable code in test_user_command.py (#10526)
add be77f8e Add a possibility to switch back to building images by secret (#10509)
add 0e0aefb Fix TestAWSDataSyncOperatorUpdate.__init__ method (#10536)
add d1bce91 PyDocStyle: Enable D403: Capitalized first word of docstring (#10530)
add bfefcce Updated REST API call so GET requests pass payload in query string instead of request body (#10462)
add c60e476 Remove mlsd function from hooks/ftp.py (#10538)
add 49cce8b Improve descriptions in OpenAPI Spec file (#10539)
add 133837c Fix typo Sucess to Success (#10540)
add d760265 PyDocStyle: No whitespaces allowed surrounding docstring text (#10533)
add d6ce8c8 Add update mask to patch dag endpoint (#10535)
add 7c0d6ab Enable Black on Connexion API folders (#10545)
add 4c6b759 Fix failing Black test on connexion (#10547)
add 5e82263 Add introduction to Stable RESTT API (#10548)
add 018ae0e The PIP version is not pinned to 19.0.2 any more (#10542)
add c6e6d6d Helm Docker image sources are now included in the Airlfow codebase (#9650)
add fdd9b6f Enable Black on Providers Packages (#10543)
add cd1f794 Bring back some inclusions before we solve cyclic deps problems (#10551)
add cbf3603 Add Airflow 1.10.12 Changelog & Updating guide (#10558)
add a0e7c40 Port isort config from pre-commit to setup.cfg (#10557)
add 417f8de Update configs added in 1.10.12 (#10561)
add 51676b7 Migrate companies list to INTHEWILD.md (#10563)
add d84ec03 Add content to file (forgot to save before committing) (#10565)
add 7c1855f Add License to INTHEWILD.md (#10570)
add 855d495 Move more operators/hooks/sensors to fundamentals (#10567)
add 3a34962 Update CHANGELOG complete class name (#10573)
add 8a7c372 Untangle cyclic deps configuration <> secrets (#10559)
add db378c0 [k8s] Store the raw ti key info to pod annotations (#10568)
add b81bdaa Add OpenSlate to INTHEWILD.md (#10581)
add 35efb1d Add Jobrapido to INTHEWILD.md (#10583)
add 13a3a30 Improve .mailmap (#10582)
add 3b0aa16 updated official docker images in README (#10579)
add 1e5aa44 Spark-on-K8S sensor - add driver logs (#10023)
add 46ac09d Enhanced the Kubernetes Executor doc (#10433)
add 8349061 Improve Docstring for AWS Athena Hook/Operator (#10580)
add 2fc9614 Fix typo in Custom XCom backend (#10588)
add 8624224 Remove Outdated SQLCheckOperator Docstring (#10589)
add 276a15a Add info about update mask to API doc introduction (#10572)
No new revisions were added by this update.
Summary of changes:
.dockerignore | 10 +-
.flake8 | 2 +-
.github/ISSUE_TEMPLATE/bug_report.md | 2 +-
.github/ISSUE_TEMPLATE/config.yml | 5 +
.github/ISSUE_TEMPLATE/support_request.md | 14 -
.github/boring-cyborg.yml | 15 +-
.github/workflows/build-images-workflow-run.yml | 247 ++++
.github/workflows/cancel_other_workflow_runs.yml | 36 -
.github/workflows/ci.yml | 719 ++++++-----
.github/workflows/repo_sync.yml | 19 +-
...{quarantined.yaml => scheduled_quarantined.yml} | 25 +-
.mailmap | 7 +
.pre-commit-config.yaml | 39 +-
.rat-excludes | 5 +
BREEZE.rst | 184 ++-
CHANGELOG.txt | 199 ++-
CI.rst | 662 ++++++++--
CONTRIBUTING.rst | 31 +-
Dockerfile | 59 +-
Dockerfile.ci | 44 +-
IMAGES.rst | 11 +-
INSTALL | 2 +-
README.md => INTHEWILD.md | 294 +----
README.md | 396 +-----
STATIC_CODE_CHECKS.rst | 42 +-
TESTING.rst | 2 +-
UPDATING.md | 79 +-
.../auth/backend/{deny_all.py => basic_auth.py} | 38 +-
airflow/api/auth/backend/deny_all.py | 1 -
airflow/api/auth/backend/kerberos_auth.py | 3 +-
airflow/api/client/json_client.py | 3 +-
airflow/api/common/experimental/delete_dag.py | 3 +-
airflow/api_connexion/endpoints/config_endpoint.py | 14 +-
.../api_connexion/endpoints/connection_endpoint.py | 20 +-
airflow/api_connexion/endpoints/dag_endpoint.py | 42 +-
.../api_connexion/endpoints/dag_run_endpoint.py | 113 +-
.../api_connexion/endpoints/dag_source_endpoint.py | 2 +
.../api_connexion/endpoints/event_log_endpoint.py | 16 +-
.../api_connexion/endpoints/extra_link_endpoint.py | 2 +
airflow/api_connexion/endpoints/health_endpoint.py | 5 +-
.../endpoints/import_error_endpoint.py | 11 +-
airflow/api_connexion/endpoints/log_endpoint.py | 14 +-
airflow/api_connexion/endpoints/pool_endpoint.py | 14 +-
airflow/api_connexion/endpoints/task_endpoint.py | 3 +
.../endpoints/task_instance_endpoint.py | 5 +
.../api_connexion/endpoints/variable_endpoint.py | 15 +-
.../api_connexion/endpoints/version_endpoint.py | 1 +
airflow/api_connexion/endpoints/xcom_endpoint.py | 29 +-
airflow/api_connexion/exceptions.py | 14 +-
airflow/api_connexion/openapi/v1.yaml | 606 ++++++---
airflow/api_connexion/parameters.py | 4 +-
airflow/api_connexion/schemas/common_schema.py | 15 +-
airflow/api_connexion/schemas/config_schema.py | 18 +-
airflow/api_connexion/schemas/connection_schema.py | 9 +-
airflow/api_connexion/schemas/dag_run_schema.py | 10 +-
airflow/api_connexion/schemas/dag_schema.py | 6 +-
airflow/api_connexion/schemas/dag_source_schema.py | 1 +
airflow/api_connexion/schemas/enum_schemas.py | 7 +-
airflow/api_connexion/schemas/error_schema.py | 2 +-
airflow/api_connexion/schemas/event_log_schema.py | 10 +-
airflow/api_connexion/schemas/health_schema.py | 8 +-
airflow/api_connexion/schemas/log_schema.py | 5 +-
airflow/api_connexion/schemas/pool_schema.py | 1 +
airflow/api_connexion/schemas/task_schema.py | 9 +-
airflow/api_connexion/schemas/variable_schema.py | 6 +-
airflow/api_connexion/schemas/version_schema.py | 1 +
airflow/api_connexion/schemas/xcom_schema.py | 9 +-
.../deny_all.py => api_connexion/security.py} | 25 +-
airflow/cli/cli_parser.py | 19 +-
airflow/cli/commands/config_command.py | 4 +-
airflow/cli/commands/connection_command.py | 58 +-
airflow/cli/commands/dag_command.py | 2 +-
airflow/cli/commands/db_command.py | 2 +-
airflow/cli/commands/legacy_commands.py | 2 +-
airflow/cli/commands/sync_perm_command.py | 3 +-
airflow/config_templates/airflow_local_settings.py | 6 +-
airflow/config_templates/config.yml | 26 +-
airflow/config_templates/default_airflow.cfg | 12 +-
.../config_templates/default_webserver_config.py | 8 +-
airflow/configuration.py | 323 +++--
airflow/exceptions.py | 4 +
airflow/executors/base_executor.py | 2 +-
airflow/executors/celery_executor.py | 13 +-
airflow/executors/kubernetes_executor.py | 167 +--
airflow/executors/local_executor.py | 2 -
airflow/hooks/base_hook.py | 5 +-
airflow/hooks/dbapi_hook.py | 6 +-
airflow/jobs/scheduler_job.py | 10 +-
airflow/kubernetes/pod_generator.py | 78 +-
airflow/kubernetes/pod_runtime_info_env.py | 2 +-
airflow/kubernetes/worker_configuration.py | 1 +
airflow/macros/hive.py | 22 +-
...3a5a_add_dag_hash_column_to_serialized_dag_.py} | 38 +-
airflow/models/base.py | 4 +-
airflow/models/baseoperator.py | 46 +-
airflow/models/connection.py | 22 +-
airflow/models/crypto.py | 12 +-
airflow/models/dag.py | 14 +-
airflow/models/dagbag.py | 28 +-
airflow/models/dagcode.py | 12 +-
airflow/models/dagrun.py | 37 +-
airflow/models/pool.py | 2 +-
airflow/models/renderedtifields.py | 10 +-
airflow/models/serialized_dag.py | 14 +-
airflow/models/skipmixin.py | 4 +-
airflow/models/taskinstance.py | 77 +-
airflow/models/variable.py | 22 +-
airflow/operators/sql.py | 6 +-
airflow/plugins_manager.py | 2 -
.../amazon/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/amazon/README.md | 4 +-
.../amazon/aws/example_dags/example_datasync_1.py | 17 +-
.../amazon/aws/example_dags/example_datasync_2.py | 29 +-
.../amazon/aws/example_dags/example_ecs_fargate.py | 7 +-
.../example_emr_job_flow_automatic_steps.py | 14 +-
.../example_emr_job_flow_manual_steps.py | 18 +-
.../example_google_api_to_s3_transfer_advanced.py | 21 +-
.../example_google_api_to_s3_transfer_basic.py | 9 +-
.../example_dags/example_imap_attachment_to_s3.py | 7 +-
.../amazon/aws/example_dags/example_s3_bucket.py | 15 +-
.../aws/example_dags/example_s3_to_redshift.py | 24 +-
airflow/providers/amazon/aws/hooks/athena.py | 63 +-
airflow/providers/amazon/aws/hooks/aws_dynamodb.py | 4 +-
airflow/providers/amazon/aws/hooks/base_aws.py | 20 +-
airflow/providers/amazon/aws/hooks/batch_client.py | 16 +-
.../providers/amazon/aws/hooks/batch_waiters.py | 11 +-
airflow/providers/amazon/aws/hooks/datasync.py | 30 +-
airflow/providers/amazon/aws/hooks/ec2.py | 17 +-
airflow/providers/amazon/aws/hooks/emr.py | 4 +-
airflow/providers/amazon/aws/hooks/glue.py | 94 +-
airflow/providers/amazon/aws/hooks/glue_catalog.py | 12 +-
airflow/providers/amazon/aws/hooks/kinesis.py | 5 +-
.../providers/amazon/aws/hooks/lambda_function.py | 14 +-
airflow/providers/amazon/aws/hooks/logs.py | 14 +-
airflow/providers/amazon/aws/hooks/redshift.py | 36 +-
airflow/providers/amazon/aws/hooks/s3.py | 278 ++--
airflow/providers/amazon/aws/hooks/sagemaker.py | 289 +++--
airflow/providers/amazon/aws/hooks/ses.py | 100 ++
airflow/providers/amazon/aws/hooks/sns.py | 9 +-
airflow/providers/amazon/aws/hooks/sqs.py | 10 +-
.../providers/amazon/aws/hooks/step_function.py | 12 +-
.../amazon/aws/log/cloudwatch_task_handler.py | 16 +-
.../providers/amazon/aws/log/s3_task_handler.py | 8 +-
airflow/providers/amazon/aws/operators/athena.py | 35 +-
airflow/providers/amazon/aws/operators/batch.py | 11 +-
.../amazon/aws/operators/cloud_formation.py | 16 +-
airflow/providers/amazon/aws/operators/datasync.py | 92 +-
.../amazon/aws/operators/ec2_start_instance.py | 24 +-
.../amazon/aws/operators/ec2_stop_instance.py | 24 +-
airflow/providers/amazon/aws/operators/ecs.py | 73 +-
.../amazon/aws/operators/emr_add_steps.py | 22 +-
.../amazon/aws/operators/emr_create_job_flow.py | 24 +-
.../amazon/aws/operators/emr_modify_cluster.py | 13 +-
.../amazon/aws/operators/emr_terminate_job_flow.py | 7 +-
airflow/providers/amazon/aws/operators/glue.py | 59 +-
.../providers/amazon/aws/operators/s3_bucket.py | 27 +-
.../amazon/aws/operators/s3_copy_object.py | 33 +-
.../amazon/aws/operators/s3_delete_objects.py | 9 +-
.../amazon/aws/operators/s3_file_transform.py | 53 +-
airflow/providers/amazon/aws/operators/s3_list.py | 18 +-
.../amazon/aws/operators/sagemaker_base.py | 11 +-
.../amazon/aws/operators/sagemaker_endpoint.py | 39 +-
.../aws/operators/sagemaker_endpoint_config.py | 20 +-
.../amazon/aws/operators/sagemaker_model.py | 13 +-
.../amazon/aws/operators/sagemaker_processing.py | 35 +-
.../amazon/aws/operators/sagemaker_training.py | 29 +-
.../amazon/aws/operators/sagemaker_transform.py | 25 +-
.../amazon/aws/operators/sagemaker_tuning.py | 22 +-
airflow/providers/amazon/aws/operators/sns.py | 17 +-
airflow/providers/amazon/aws/operators/sqs.py | 28 +-
.../step_function_get_execution_output.py | 1 +
.../aws/operators/step_function_start_execution.py | 15 +-
.../amazon/aws/secrets/secrets_manager.py | 13 +-
.../amazon/aws/secrets/systems_manager.py | 9 +-
airflow/providers/amazon/aws/sensors/athena.py | 25 +-
.../amazon/aws/sensors/cloud_formation.py | 16 +-
.../amazon/aws/sensors/ec2_instance_state.py | 24 +-
airflow/providers/amazon/aws/sensors/emr_base.py | 15 +-
.../providers/amazon/aws/sensors/emr_job_flow.py | 17 +-
airflow/providers/amazon/aws/sensors/emr_step.py | 34 +-
airflow/providers/amazon/aws/sensors/glue.py | 14 +-
.../amazon/aws/sensors/glue_catalog_partition.py | 35 +-
airflow/providers/amazon/aws/sensors/redshift.py | 10 +-
airflow/providers/amazon/aws/sensors/s3_key.py | 30 +-
.../amazon/aws/sensors/s3_keys_unchanged.py | 52 +-
airflow/providers/amazon/aws/sensors/s3_prefix.py | 16 +-
.../providers/amazon/aws/sensors/sagemaker_base.py | 9 +-
.../amazon/aws/sensors/sagemaker_endpoint.py | 4 +-
.../amazon/aws/sensors/sagemaker_training.py | 30 +-
.../amazon/aws/sensors/sagemaker_transform.py | 4 +-
.../amazon/aws/sensors/sagemaker_tuning.py | 4 +-
airflow/providers/amazon/aws/sensors/sqs.py | 29 +-
.../amazon/aws/sensors/step_function_execution.py | 9 +-
.../amazon/aws/transfers/dynamodb_to_s3.py | 26 +-
.../providers/amazon/aws/transfers/gcs_to_s3.py | 103 +-
.../amazon/aws/transfers/google_api_to_s3.py | 33 +-
.../amazon/aws/transfers/hive_to_dynamodb.py | 44 +-
.../amazon/aws/transfers/imap_attachment_to_s3.py | 31 +-
.../providers/amazon/aws/transfers/mongo_to_s3.py | 38 +-
.../providers/amazon/aws/transfers/mysql_to_s3.py | 33 +-
.../amazon/aws/transfers/redshift_to_s3.py | 46 +-
.../amazon/aws/transfers/s3_to_redshift.py | 42 +-
.../providers/amazon/aws/transfers/s3_to_sftp.py | 10 +-
.../providers/amazon/aws/transfers/sftp_to_s3.py | 17 +-
.../cassandra/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/apache/cassandra/README.md | 4 +-
.../example_dags/example_cassandra_dag.py | 2 +-
.../providers/apache/cassandra/hooks/cassandra.py | 26 +-
.../providers/apache/cassandra/sensors/record.py | 1 +
.../providers/apache/cassandra/sensors/table.py | 1 +
.../apache/druid/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/apache/druid/README.md | 4 +-
airflow/providers/apache/druid/hooks/druid.py | 35 +-
airflow/providers/apache/druid/operators/druid.py | 18 +-
.../apache/druid/operators/druid_check.py | 6 +-
.../apache/druid/transfers/hive_to_druid.py | 55 +-
.../apache/hdfs/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/apache/hdfs/README.md | 4 +-
airflow/providers/apache/hdfs/hooks/hdfs.py | 45 +-
airflow/providers/apache/hdfs/hooks/webhdfs.py | 23 +-
airflow/providers/apache/hdfs/sensors/hdfs.py | 70 +-
airflow/providers/apache/hdfs/sensors/web_hdfs.py | 8 +-
.../apache/hive/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/apache/hive/README.md | 4 +-
.../hive/example_dags/example_twitter_README.md | 2 +-
.../hive/example_dags/example_twitter_dag.py | 52 +-
airflow/providers/apache/hive/hooks/hive.py | 321 +++--
airflow/providers/apache/hive/operators/hive.py | 68 +-
.../providers/apache/hive/operators/hive_stats.py | 63 +-
.../apache/hive/sensors/hive_partition.py | 37 +-
.../apache/hive/sensors/metastore_partition.py | 20 +-
.../apache/hive/sensors/named_hive_partition.py | 28 +-
.../apache/hive/transfers/hive_to_mysql.py | 37 +-
.../apache/hive/transfers/hive_to_samba.py | 20 +-
.../apache/hive/transfers/mssql_to_hive.py | 28 +-
.../apache/hive/transfers/mysql_to_hive.py | 47 +-
.../providers/apache/hive/transfers/s3_to_hive.py | 105 +-
.../apache/hive/transfers/vertica_to_hive.py | 29 +-
.../apache/kylin/example_dags/example_kylin_dag.py | 10 +-
airflow/providers/apache/kylin/hooks/kylin.py | 31 +-
.../providers/apache/kylin/operators/kylin_cube.py | 75 +-
.../apache/livy/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/apache/livy/README.md | 4 +-
.../apache/livy/example_dags/example_livy.py | 15 +-
airflow/providers/apache/livy/hooks/livy.py | 61 +-
airflow/providers/apache/livy/operators/livy.py | 7 +-
airflow/providers/apache/livy/sensors/livy.py | 5 +-
.../apache/pig/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/apache/pig/README.md | 4 +-
.../apache/pig/example_dags/example_pig.py | 9 +-
airflow/providers/apache/pig/hooks/pig.py | 14 +-
airflow/providers/apache/pig/operators/pig.py | 22 +-
.../apache/pinot/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/apache/pinot/README.md | 4 +-
airflow/providers/apache/pinot/hooks/pinot.py | 111 +-
.../apache/spark/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/apache/spark/README.md | 4 +-
.../apache/spark/example_dags/example_spark_dag.py | 15 +-
airflow/providers/apache/spark/hooks/spark_jdbc.py | 102 +-
.../apache/spark/hooks/spark_jdbc_script.py | 168 +--
airflow/providers/apache/spark/hooks/spark_sql.py | 36 +-
.../providers/apache/spark/hooks/spark_submit.py | 222 ++--
.../providers/apache/spark/operators/spark_jdbc.py | 65 +-
.../providers/apache/spark/operators/spark_sql.py | 64 +-
.../apache/spark/operators/spark_submit.py | 81 +-
.../apache/sqoop/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/apache/sqoop/README.md | 4 +-
airflow/providers/apache/sqoop/hooks/sqoop.py | 186 +--
airflow/providers/apache/sqoop/operators/sqoop.py | 126 +-
.../celery/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/celery/README.md | 4 +-
airflow/providers/celery/sensors/celery_queue.py | 17 +-
.../cloudant/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/cloudant/README.md | 4 +-
airflow/providers/cloudant/hooks/cloudant.py | 5 +-
.../kubernetes/example_dags/example_kubernetes.py | 92 +-
.../example_dags/example_spark_kubernetes.py | 5 +-
.../providers/cncf/kubernetes/hooks/kubernetes.py | 87 +-
.../cncf/kubernetes/operators/kubernetes_pod.py | 51 +-
.../cncf/kubernetes/operators/spark_kubernetes.py | 16 +-
.../cncf/kubernetes/sensors/spark_kubernetes.py | 55 +-
.../databricks/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/databricks/README.md | 4 +-
.../databricks/example_dags/example_databricks.py | 25 +-
airflow/providers/databricks/hooks/databricks.py | 70 +-
.../providers/databricks/operators/databricks.py | 90 +-
.../datadog/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/datadog/README.md | 4 +-
airflow/providers/datadog/hooks/datadog.py | 61 +-
airflow/providers/datadog/sensors/datadog.py | 24 +-
.../dingding/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/dingding/README.md | 4 +-
.../dingding/example_dags/example_dingding.py | 77 +-
airflow/providers/dingding/hooks/dingding.py | 53 +-
airflow/providers/dingding/operators/dingding.py | 24 +-
.../discord/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/discord/README.md | 4 +-
airflow/providers/discord/hooks/discord_webhook.py | 46 +-
.../providers/discord/operators/discord_webhook.py | 28 +-
.../docker/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/docker/README.md | 4 +-
.../docker/example_dags/example_docker.py | 20 +-
.../docker/example_dags/example_docker_swarm.py | 4 +-
airflow/providers/docker/hooks/docker.py | 22 +-
airflow/providers/docker/operators/docker.py | 94 +-
airflow/providers/docker/operators/docker_swarm.py | 15 +-
.../elasticsearch/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/elasticsearch/README.md | 4 +-
.../providers/elasticsearch/hooks/elasticsearch.py | 14 +-
.../providers/elasticsearch/log/es_task_handler.py | 44 +-
.../exasol/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/exasol/README.md | 4 +-
airflow/providers/exasol/hooks/exasol.py | 16 +-
airflow/providers/exasol/operators/exasol.py | 24 +-
.../facebook/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/facebook/README.md | 4 +-
airflow/providers/facebook/ads/hooks/ads.py | 37 +-
.../providers/ftp/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/ftp/README.md | 4 +-
airflow/providers/ftp/hooks/ftp.py | 52 +-
airflow/providers/ftp/sensors/ftp.py | 14 +-
.../google/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/google/README.md | 4 +-
airflow/providers/google/__init__.py | 11 +-
.../google/ads/example_dags/example_ads.py | 4 +-
airflow/providers/google/ads/hooks/ads.py | 11 +-
airflow/providers/google/ads/operators/ads.py | 35 +-
.../providers/google/ads/transfers/ads_to_gcs.py | 41 +-
.../_internal_client/secret_manager_client.py | 21 +-
.../example_automl_nl_text_classification.py | 22 +-
.../example_automl_nl_text_extraction.py | 22 +-
.../example_automl_nl_text_sentiment.py | 22 +-
.../cloud/example_dags/example_automl_tables.py | 43 +-
.../example_dags/example_automl_translation.py | 27 +-
...ple_automl_video_intelligence_classification.py | 18 +-
.../example_automl_video_intelligence_tracking.py | 21 +-
.../example_automl_vision_classification.py | 22 +-
.../example_automl_vision_object_detection.py | 18 +-
.../cloud/example_dags/example_bigquery_dts.py | 18 +-
.../example_dags/example_bigquery_operations.py | 49 +-
.../cloud/example_dags/example_bigquery_queries.py | 38 +-
.../cloud/example_dags/example_bigquery_sensors.py | 103 ++
.../example_dags/example_bigquery_to_bigquery.py | 8 +-
.../cloud/example_dags/example_bigquery_to_gcs.py | 16 +-
.../example_dags/example_bigquery_transfer.py | 16 +-
.../google/cloud/example_dags/example_bigtable.py | 58 +-
.../cloud/example_dags/example_cloud_build.py | 5 +-
.../example_dags/example_cloud_memorystore.py | 15 +-
.../google/cloud/example_dags/example_cloud_sql.py | 185 +--
.../cloud/example_dags/example_cloud_sql_query.py | 215 ++--
.../example_cloud_storage_transfer_service_aws.py | 41 +-
.../example_cloud_storage_transfer_service_gcp.py | 38 +-
.../google/cloud/example_dags/example_compute.py | 37 +-
.../cloud/example_dags/example_compute_igm.py | 75 +-
.../cloud/example_dags/example_datacatalog.py | 32 +-
.../google/cloud/example_dags/example_dataflow.py | 48 +-
.../cloud/example_dags/example_datafusion.py | 54 +-
...mple_postgres_to_gcs.py => example_dataprep.py} | 28 +-
.../google/cloud/example_dags/example_dataproc.py | 30 +-
.../google/cloud/example_dags/example_datastore.py | 41 +-
.../google/cloud/example_dags/example_dlp.py | 12 +-
.../example_dags/example_facebook_ads_to_gcs.py | 32 +-
.../google/cloud/example_dags/example_functions.py | 46 +-
.../google/cloud/example_dags/example_gcs.py | 38 +-
.../cloud/example_dags/example_gcs_to_bigquery.py | 18 +-
.../cloud/example_dags/example_gcs_to_gcs.py | 22 +-
.../example_dags/example_kubernetes_engine.py | 14 +-
.../cloud/example_dags/example_life_sciences.py | 52 +-
.../cloud/example_dags/example_local_to_gcs.py | 7 +-
.../google/cloud/example_dags/example_mlengine.py | 78 +-
.../cloud/example_dags/example_natural_language.py | 18 +-
.../cloud/example_dags/example_postgres_to_gcs.py | 6 +-
.../cloud/example_dags/example_presto_to_gcs.py | 4 +-
.../google/cloud/example_dags/example_pubsub.py | 39 +-
.../cloud/example_dags/example_sftp_to_gcs.py | 4 +-
.../cloud/example_dags/example_sheets_to_gcs.py | 4 +-
.../google/cloud/example_dags/example_spanner.py | 79 +-
.../cloud/example_dags/example_speech_to_text.py | 4 +-
.../cloud/example_dags/example_stackdriver.py | 97 +-
.../google/cloud/example_dags/example_tasks.py | 4 +-
.../google/cloud/example_dags/example_translate.py | 3 +-
.../cloud/example_dags/example_translate_speech.py | 4 +-
.../example_dags/example_video_intelligence.py | 19 +-
.../google/cloud/example_dags/example_vision.py | 30 +-
airflow/providers/google/cloud/hooks/automl.py | 90 +-
airflow/providers/google/cloud/hooks/bigquery.py | 1016 ++++++++-------
.../providers/google/cloud/hooks/bigquery_dts.py | 32 +-
airflow/providers/google/cloud/hooks/bigtable.py | 110 +-
.../providers/google/cloud/hooks/cloud_build.py | 9 +-
.../google/cloud/hooks/cloud_memorystore.py | 4 +-
airflow/providers/google/cloud/hooks/cloud_sql.py | 372 +++---
.../cloud/hooks/cloud_storage_transfer_service.py | 70 +-
airflow/providers/google/cloud/hooks/compute.py | 210 ++-
.../providers/google/cloud/hooks/datacatalog.py | 22 +-
airflow/providers/google/cloud/hooks/dataflow.py | 183 +--
airflow/providers/google/cloud/hooks/datafusion.py | 121 +-
airflow/providers/google/cloud/hooks/dataprep.py | 75 ++
airflow/providers/google/cloud/hooks/dataproc.py | 97 +-
airflow/providers/google/cloud/hooks/datastore.py | 129 +-
airflow/providers/google/cloud/hooks/dlp.py | 91 +-
airflow/providers/google/cloud/hooks/functions.py | 42 +-
airflow/providers/google/cloud/hooks/gcs.py | 197 +--
airflow/providers/google/cloud/hooks/gdm.py | 37 +-
airflow/providers/google/cloud/hooks/kms.py | 14 +-
.../google/cloud/hooks/kubernetes_engine.py | 89 +-
.../providers/google/cloud/hooks/life_sciences.py | 35 +-
airflow/providers/google/cloud/hooks/mlengine.py | 163 +--
.../google/cloud/hooks/natural_language.py | 30 +-
airflow/providers/google/cloud/hooks/pubsub.py | 76 +-
.../providers/google/cloud/hooks/secret_manager.py | 17 +-
airflow/providers/google/cloud/hooks/spanner.py | 127 +-
.../providers/google/cloud/hooks/speech_to_text.py | 6 +-
.../providers/google/cloud/hooks/stackdriver.py | 113 +-
airflow/providers/google/cloud/hooks/tasks.py | 87 +-
.../providers/google/cloud/hooks/text_to_speech.py | 14 +-
airflow/providers/google/cloud/hooks/translate.py | 6 +-
.../google/cloud/hooks/video_intelligence.py | 7 +-
airflow/providers/google/cloud/hooks/vision.py | 80 +-
.../providers/google/cloud/log/gcs_task_handler.py | 143 ++-
.../google/cloud/log/stackdriver_task_handler.py | 29 +-
airflow/providers/google/cloud/operators/automl.py | 320 ++++-
.../providers/google/cloud/operators/bigquery.py | 787 +++++++++---
.../google/cloud/operators/bigquery_dts.py | 71 +-
.../providers/google/cloud/operators/bigtable.py | 379 ++++--
.../google/cloud/operators/cloud_build.py | 43 +-
.../google/cloud/operators/cloud_memorystore.py | 261 +++-
.../providers/google/cloud/operators/cloud_sql.py | 788 ++++++++----
.../operators/cloud_storage_transfer_service.py | 349 ++++-
.../providers/google/cloud/operators/compute.py | 442 +++++--
.../google/cloud/operators/datacatalog.py | 461 ++++++-
.../providers/google/cloud/operators/dataflow.py | 160 ++-
.../providers/google/cloud/operators/datafusion.py | 320 +++--
.../providers/google/cloud/operators/dataprep.py | 54 +
.../providers/google/cloud/operators/dataproc.py | 597 +++++----
.../providers/google/cloud/operators/datastore.py | 383 ++++--
airflow/providers/google/cloud/operators/dlp.py | 713 +++++++++--
.../providers/google/cloud/operators/functions.py | 239 ++--
airflow/providers/google/cloud/operators/gcs.py | 412 ++++--
.../google/cloud/operators/kubernetes_engine.py | 138 +-
.../google/cloud/operators/life_sciences.py | 48 +-
.../providers/google/cloud/operators/mlengine.py | 650 ++++++----
.../google/cloud/operators/natural_language.py | 110 +-
airflow/providers/google/cloud/operators/pubsub.py | 273 +++-
.../providers/google/cloud/operators/spanner.py | 404 ++++--
.../google/cloud/operators/speech_to_text.py | 32 +-
.../google/cloud/operators/stackdriver.py | 370 ++++--
airflow/providers/google/cloud/operators/tasks.py | 306 ++++-
.../google/cloud/operators/text_to_speech.py | 28 +-
.../providers/google/cloud/operators/translate.py | 32 +-
.../google/cloud/operators/translate_speech.py | 49 +-
.../google/cloud/operators/video_intelligence.py | 92 +-
airflow/providers/google/cloud/operators/vision.py | 458 ++++++-
.../google/cloud/secrets/secret_manager.py | 7 +-
airflow/providers/google/cloud/sensors/bigquery.py | 128 +-
.../providers/google/cloud/sensors/bigquery_dts.py | 23 +-
airflow/providers/google/cloud/sensors/bigtable.py | 32 +-
.../sensors/cloud_storage_transfer_service.py | 27 +-
airflow/providers/google/cloud/sensors/gcs.py | 201 ++-
airflow/providers/google/cloud/sensors/pubsub.py | 73 +-
.../google/cloud/transfers/adls_to_gcs.py | 73 +-
.../google/cloud/transfers/bigquery_to_bigquery.py | 75 +-
.../google/cloud/transfers/bigquery_to_gcs.py | 84 +-
.../google/cloud/transfers/bigquery_to_mysql.py | 82 +-
.../google/cloud/transfers/cassandra_to_gcs.py | 74 +-
.../google/cloud/transfers/facebook_ads_to_gcs.py | 33 +-
.../google/cloud/transfers/gcs_to_bigquery.py | 133 +-
.../providers/google/cloud/transfers/gcs_to_gcs.py | 162 ++-
.../google/cloud/transfers/gcs_to_local.py | 71 +-
.../google/cloud/transfers/gcs_to_sftp.py | 72 +-
.../google/cloud/transfers/local_to_gcs.py | 63 +-
.../google/cloud/transfers/mssql_to_gcs.py | 11 +-
.../google/cloud/transfers/mysql_to_gcs.py | 6 +-
.../google/cloud/transfers/postgres_to_gcs.py | 17 +-
.../google/cloud/transfers/presto_to_gcs.py | 6 +-
.../providers/google/cloud/transfers/s3_to_gcs.py | 103 +-
.../google/cloud/transfers/sftp_to_gcs.py | 56 +-
.../google/cloud/transfers/sheets_to_gcs.py | 64 +-
.../providers/google/cloud/transfers/sql_to_gcs.py | 126 +-
.../google/cloud/utils/credentials_provider.py | 68 +-
.../google/cloud/utils/field_sanitizer.py | 15 +-
.../google/cloud/utils/field_validator.py | 185 +--
.../google/cloud/utils/mlengine_operator_utils.py | 67 +-
.../cloud/utils/mlengine_prediction_summary.py | 56 +-
.../providers/google/common/hooks/base_google.py | 94 +-
.../providers/google/common/hooks/discovery_api.py | 11 +-
.../firebase/example_dags/example_firestore.py | 4 +-
.../providers/google/firebase/hooks/firestore.py | 8 +-
.../google/firebase/operators/firestore.py | 28 +-
.../example_dags/example_analytics.py | 17 +-
.../example_dags/example_campaign_manager.py | 33 +-
.../example_dags/example_display_video.py | 47 +-
.../example_dags/example_search_ads.py | 13 +-
.../google/marketing_platform/hooks/analytics.py | 30 +-
.../marketing_platform/hooks/campaign_manager.py | 21 +-
.../marketing_platform/hooks/display_video.py | 18 +-
.../google/marketing_platform/hooks/search_ads.py | 9 +-
.../marketing_platform/operators/analytics.py | 186 ++-
.../operators/campaign_manager.py | 172 ++-
.../marketing_platform/operators/display_video.py | 230 +++-
.../marketing_platform/operators/search_ads.py | 72 +-
.../marketing_platform/sensors/campaign_manager.py | 35 +-
.../marketing_platform/sensors/display_video.py | 53 +-
.../marketing_platform/sensors/search_ads.py | 33 +-
.../suite/example_dags/example_gcs_to_sheets.py | 4 +-
.../google/suite/example_dags/example_sheets.py | 4 +-
airflow/providers/google/suite/hooks/drive.py | 5 +-
airflow/providers/google/suite/hooks/sheets.py | 175 +--
airflow/providers/google/suite/operators/sheets.py | 31 +-
.../google/suite/transfers/gcs_to_gdrive.py | 40 +-
.../google/suite/transfers/gcs_to_sheets.py | 39 +-
.../providers/grpc/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/grpc/README.md | 4 +-
airflow/providers/grpc/hooks/grpc.py | 43 +-
airflow/providers/grpc/operators/grpc.py | 27 +-
.../hashicorp/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/hashicorp/README.md | 4 +-
.../hashicorp/_internal_client/vault_client.py | 121 +-
airflow/providers/hashicorp/hooks/vault.py | 80 +-
airflow/providers/hashicorp/secrets/vault.py | 5 +-
.../providers/http/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/http/README.md | 4 +-
.../providers/http/example_dags/example_http.py | 5 +-
airflow/providers/http/hooks/http.py | 60 +-
airflow/providers/http/operators/http.py | 36 +-
airflow/providers/http/sensors/http.py | 35 +-
.../providers/imap/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/imap/README.md | 4 +-
airflow/providers/imap/hooks/imap.py | 98 +-
airflow/providers/imap/sensors/imap_attachment.py | 20 +-
.../providers/jdbc/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/jdbc/README.md | 4 +-
airflow/providers/jdbc/hooks/jdbc.py | 10 +-
airflow/providers/jdbc/operators/jdbc.py | 15 +-
.../jenkins/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/jenkins/README.md | 4 +-
.../example_dags/example_jenkins_job_trigger.py | 17 +-
.../jenkins/operators/jenkins_job_trigger.py | 78 +-
.../providers/jira/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/jira/README.md | 4 +-
airflow/providers/jira/hooks/jira.py | 34 +-
airflow/providers/jira/operators/jira.py | 20 +-
airflow/providers/jira/sensors/jira.py | 73 +-
.../azure/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/microsoft/azure/README.md | 4 +-
.../example_azure_container_instances.py | 2 +-
.../azure/example_dags/example_azure_cosmosdb.py | 2 +-
airflow/providers/microsoft/azure/hooks/adx.py | 41 +-
.../providers/microsoft/azure/hooks/azure_batch.py | 141 +-
.../azure/hooks/azure_container_instance.py | 54 +-
.../azure/hooks/azure_container_volume.py | 17 +-
.../microsoft/azure/hooks/azure_cosmos.py | 97 +-
.../microsoft/azure/hooks/azure_data_lake.py | 50 +-
.../microsoft/azure/hooks/azure_fileshare.py | 30 +-
.../providers/microsoft/azure/hooks/base_azure.py | 67 +
airflow/providers/microsoft/azure/hooks/wasb.py | 31 +-
.../microsoft/azure/log/wasb_task_handler.py | 19 +-
.../microsoft/azure/operators/adls_list.py | 15 +-
airflow/providers/microsoft/azure/operators/adx.py | 16 +-
.../microsoft/azure/operators/azure_batch.py | 206 +--
.../azure/operators/azure_container_instances.py | 116 +-
.../microsoft/azure/operators/azure_cosmos.py | 16 +-
.../microsoft/azure/operators/wasb_delete_blob.py | 29 +-
.../microsoft/azure/sensors/azure_cosmos.py | 15 +-
airflow/providers/microsoft/azure/sensors/wasb.py | 43 +-
.../microsoft/azure/transfers/file_to_wasb.py | 27 +-
.../azure/transfers/oracle_to_azure_data_lake.py | 52 +-
.../mssql/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/microsoft/mssql/README.md | 4 +-
.../providers/microsoft/mssql/operators/mssql.py | 5 +-
.../winrm/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/microsoft/winrm/README.md | 4 +-
.../microsoft/winrm/example_dags/example_winrm.py | 18 +-
airflow/providers/microsoft/winrm/hooks/winrm.py | 52 +-
.../providers/microsoft/winrm/operators/winrm.py | 35 +-
.../mongo/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/mongo/README.md | 4 +-
airflow/providers/mongo/hooks/mongo.py | 147 ++-
airflow/providers/mongo/sensors/mongo.py | 14 +-
.../mysql/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/mysql/README.md | 4 +-
airflow/providers/mysql/hooks/mysql.py | 66 +-
airflow/providers/mysql/operators/mysql.py | 28 +-
.../providers/mysql/transfers/presto_to_mysql.py | 21 +-
airflow/providers/mysql/transfers/s3_to_mysql.py | 30 +-
.../providers/mysql/transfers/vertica_to_mysql.py | 45 +-
.../providers/odbc/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/odbc/README.md | 4 +-
airflow/providers/odbc/hooks/odbc.py | 10 +-
.../openfaas/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/openfaas/README.md | 4 +-
airflow/providers/openfaas/hooks/openfaas.py | 5 +-
.../opsgenie/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/opsgenie/README.md | 4 +-
airflow/providers/opsgenie/hooks/opsgenie_alert.py | 21 +-
.../providers/opsgenie/operators/opsgenie_alert.py | 53 +-
.../oracle/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/oracle/README.md | 4 +-
airflow/providers/oracle/hooks/oracle.py | 22 +-
airflow/providers/oracle/operators/oracle.py | 19 +-
.../providers/oracle/transfers/oracle_to_oracle.py | 26 +-
.../pagerduty/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/pagerduty/README.md | 4 +-
airflow/providers/pagerduty/hooks/pagerduty.py | 3 +-
.../papermill/example_dags/example_papermill.py | 15 +-
airflow/providers/papermill/operators/papermill.py | 28 +-
.../postgres/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/postgres/README.md | 4 +-
airflow/providers/postgres/hooks/postgres.py | 33 +-
airflow/providers/postgres/operators/postgres.py | 19 +-
.../presto/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/presto/README.md | 4 +-
airflow/providers/presto/hooks/presto.py | 15 +-
.../qubole/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/qubole/README.md | 4 +-
.../qubole/example_dags/example_qubole.py | 85 +-
airflow/providers/qubole/hooks/qubole.py | 41 +-
airflow/providers/qubole/hooks/qubole_check.py | 6 +-
airflow/providers/qubole/operators/qubole.py | 58 +-
airflow/providers/qubole/operators/qubole_check.py | 31 +-
airflow/providers/qubole/sensors/qubole.py | 7 +-
.../redis/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/redis/README.md | 4 +-
airflow/providers/redis/hooks/redis.py | 24 +-
airflow/providers/redis/operators/redis_publish.py | 7 +-
airflow/providers/redis/sensors/redis_key.py | 1 +
airflow/providers/redis/sensors/redis_pub_sub.py | 1 +
.../salesforce/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/salesforce/README.md | 4 +-
.../example_tableau_refresh_workbook.py | 2 +-
airflow/providers/salesforce/hooks/salesforce.py | 38 +-
airflow/providers/salesforce/hooks/tableau.py | 7 +-
.../operators/tableau_refresh_workbook.py | 18 +-
.../salesforce/sensors/tableau_job_status.py | 13 +-
.../samba/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/samba/README.md | 4 +-
airflow/providers/samba/hooks/samba.py | 3 +-
.../segment/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/segment/README.md | 4 +-
airflow/providers/segment/hooks/segment.py | 10 +-
.../segment/operators/segment_track_event.py | 31 +-
airflow/providers/sendgrid/utils/emailer.py | 44 +-
.../providers/sftp/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/sftp/README.md | 4 +-
airflow/providers/sftp/hooks/sftp.py | 24 +-
airflow/providers/sftp/operators/sftp.py | 64 +-
airflow/providers/sftp/sensors/sftp.py | 1 +
.../singularity/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/singularity/README.md | 4 +-
.../example_dags/example_singularity.py | 43 +-
.../providers/singularity/operators/singularity.py | 52 +-
.../slack/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/slack/README.md | 4 +-
airflow/providers/slack/hooks/slack.py | 11 +-
airflow/providers/slack/hooks/slack_webhook.py | 44 +-
airflow/providers/slack/operators/slack.py | 59 +-
airflow/providers/slack/operators/slack_webhook.py | 45 +-
.../snowflake/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/snowflake/README.md | 4 +-
.../snowflake/example_dags/example_snowflake.py | 19 +-
airflow/providers/snowflake/hooks/snowflake.py | 34 +-
airflow/providers/snowflake/operators/snowflake.py | 32 +-
.../snowflake/transfers/s3_to_snowflake.py | 37 +-
.../snowflake/transfers/snowflake_to_slack.py | 20 +-
.../sqlite/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/sqlite/README.md | 4 +-
airflow/providers/sqlite/operators/sqlite.py | 13 +-
.../providers/ssh/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/ssh/README.md | 4 +-
airflow/providers/ssh/hooks/ssh.py | 101 +-
airflow/providers/ssh/operators/ssh.py | 70 +-
.../vertica/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/vertica/README.md | 4 +-
airflow/providers/vertica/hooks/vertica.py | 2 +-
airflow/providers/vertica/operators/vertica.py | 6 +-
.../yandex/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/yandex/README.md | 4 +-
.../example_dags/example_yandexcloud_dataproc.py | 70 +-
airflow/providers/yandex/hooks/yandex.py | 10 +-
.../providers/yandex/hooks/yandexcloud_dataproc.py | 3 +-
.../yandex/operators/yandexcloud_dataproc.py | 187 ++-
.../zendesk/PROVIDERS_CHANGES_2020.06.24.md | 2 +-
airflow/providers/zendesk/README.md | 4 +-
airflow/providers/zendesk/hooks/zendesk.py | 14 +-
airflow/secrets/__init__.py | 95 +-
airflow/sensors/base_sensor_operator.py | 11 +
airflow/sensors/external_task_sensor.py | 3 +-
airflow/sentry.py | 241 ++--
airflow/serialization/serialized_objects.py | 1 -
airflow/settings.py | 14 +-
airflow/stats.py | 102 +-
airflow/ti_deps/deps/trigger_rule_dep.py | 2 +-
airflow/utils/code_utils.py | 13 +
airflow/utils/dag_processing.py | 3 +-
airflow/utils/db.py | 9 +-
airflow/utils/email.py | 76 +-
airflow/utils/file.py | 3 +-
airflow/utils/log/log_reader.py | 2 +-
airflow/www/api/experimental/endpoints.py | 10 +-
airflow/www/extensions/init_manifest_files.py | 3 +-
airflow/www/forms.py | 1 +
airflow/www/templates/airflow/dag_details.html | 104 +-
airflow/www/utils.py | 94 +-
airflow/www/views.py | 673 ++++++----
.../PROVIDERS_README_TEMPLATE.md.jinja2 | 2 +-
backport_packages/import_all_provider_classes.py | 3 +-
backport_packages/refactor_backport_packages.py | 26 +-
backport_packages/setup_backport_packages.py | 24 +-
breeze | 152 ++-
breeze-complete | 20 +-
chart/README.md | 5 +-
.../dockerfiles/README.md | 16 +-
.../dockerfiles/pgbouncer-exporter/README.md | 15 +-
chart/dockerfiles/pgbouncer/Dockerfile | 65 +
.../dockerfiles/pgbouncer/build_and_push.sh | 29 +-
.../dockerfiles/statsd-exporter/Dockerfile | 24 +-
.../dockerfiles/statsd-exporter/build_and_push.sh | 29 +-
chart/dockerfiles/statsd-exporter/mappings.yml | 87 ++
chart/templates/NOTES.txt | 10 +-
chart/templates/create-user-job.yaml | 2 +
chart/templates/flower/flower-ingress.yaml | 51 +
chart/templates/webserver/webserver-ingress.yaml | 63 +
chart/values.yaml | 67 +-
common/_files_for_rebuild_check.sh | 30 -
common/_image_variables.sh | 31 -
dev/README.md | 230 +++-
dev/airflow-jira | 332 -----
dev/airflow-pr | 2 +-
docs/best-practices.rst | 10 +-
docs/{build => build_docs.py} | 305 +++--
docs/concepts.rst | 106 +-
docs/conf.py | 3 +-
docs/configurations-ref.rst | 4 +-
docs/dag-run.rst | 16 +-
docs/executor/celery.rst | 23 +-
docs/executor/index.rst | 5 +-
docs/executor/kubernetes.rst | 18 +-
docs/exts/docroles.py | 3 +-
docs/exts/exampleinclude.py | 3 -
docs/exts/redirects.py | 4 +-
docs/exts/removemarktransform.py | 8 +-
docs/faq.rst | 2 +-
docs/howto/connection/aws.rst | 2 +-
docs/howto/connection/azure.rst | 83 ++
docs/howto/connection/gcp.rst | 105 ++
docs/howto/connection/index.rst | 98 +-
docs/howto/email-config.rst | 6 +-
docs/howto/index.rst | 5 +-
docs/howto/operator/amazon/aws/datasync.rst | 2 +-
docs/howto/operator/amazon/aws/emr.rst | 6 +-
docs/howto/operator/google/cloud/bigquery.rst | 34 +-
docs/howto/operator/google/cloud/bigtable.rst | 23 +
docs/howto/operator/google/cloud/datacatalog.rst | 2 +-
docs/howto/operator/google/cloud/dataprep.rst | 60 +
docs/howto/operator/google/cloud/stackdriver.rst | 4 +-
docs/howto/operator/google/transfer/gcs_to_gcs.rst | 2 +-
.../operator/google/transfer/presto_to_gcs.rst | 2 +-
docs/howto/variable.rst | 72 ++
docs/img/arch-diag-basic.png | Bin 0 -> 16303 bytes
docs/img/arch-diag-kubernetes.png | Bin 0 -> 21804 bytes
docs/img/arch-diag-kubernetes2.png | Bin 0 -> 86384 bytes
docs/index.rst | 8 +-
docs/installation.rst | 10 +
docs/integration.rst | 18 +-
.../{howto => logging-monitoring}/check-health.rst | 2 +-
docs/{ => logging-monitoring}/errors.rst | 6 +-
.../redoc.rst => logging-monitoring/index.rst} | 14 +-
.../logging-tasks.rst} | 26 +-
docs/{ => logging-monitoring}/metrics.rst | 0
.../tracking-user-activity.rst | 2 +-
docs/operators-and-hooks-ref.rst | 62 +-
docs/privacy_notice.rst | 2 +-
docs/project.rst | 42 +-
docs/redirects.txt | 16 +-
docs/rest-api-ref.rst | 2 +-
docs/scheduler.rst | 16 +-
docs/security.rst | 502 --------
docs/security/access-control.rst | 97 ++
docs/security/api.rst | 190 +++
docs/{howto/index.rst => security/flower.rst} | 55 +-
docs/{stable-rest-api => security}/index.rst | 18 +-
docs/security/kerberos.rst | 135 ++
docs/security/secrets/fernet.rst | 62 +
.../redoc.rst => security/secrets/index.rst} | 22 +-
.../aws-secrets-manaager-backend.rst | 0
.../aws-ssm-parameter-store-secrets-backend.rst | 0
.../google-cloud-secret-manager-backend.rst | 71 +-
.../hashicorp-vault-secrets-backend.rst | 0
.../secrets}/secrets-backend/index.rst | 10 +-
.../local-filesystem-secrets-backend.rst | 12 +-
docs/security/webserver.rst | 123 ++
docs/security/workload.rst | 56 +
docs/spelling_wordlist.txt | 1352 ++++++++++++++++++++
.../redoc.rst => stable-rest-api-ref.rst} | 2 +-
docs/start.rst | 37 +-
docs/timezone.rst | 6 +-
docs/tutorial.rst | 2 +-
docs/usage-cli.rst | 104 +-
images/{ => ci}/CI.png | Bin
images/ci/pull_request_ci_flow.md5 | 1 +
images/ci/pull_request_ci_flow.mermaid | 108 ++
images/ci/pull_request_ci_flow.png | Bin 0 -> 195487 bytes
images/ci/push_ci_flow.md5 | 1 +
images/ci/push_ci_flow.mermaid | 117 ++
images/ci/push_ci_flow.png | Bin 0 -> 209013 bytes
images/ci/scheduled_ci_flow.md5 | 1 +
images/ci/scheduled_ci_flow.mermaid | 111 ++
images/ci/scheduled_ci_flow.png | Bin 0 -> 201540 bytes
kubernetes_tests/test_kubernetes_pod_operator.py | 40 +-
pyproject.toml | 4 +
...ci_test_backport_packages_import_all_classes.sh | 4 +-
...ci_test_backport_packages_install_separately.sh | 3 +-
.../ci_branch_constraints.sh} | 23 +-
.../ci_commit_constraints.sh} | 29 +-
scripts/ci/docker-compose/base.yml | 2 +
scripts/ci/docker-compose/{ci.yml => ga.yml} | 0
scripts/ci/docker-compose/local-prod.yml | 2 +-
scripts/ci/docker-compose/local.yml | 3 +-
scripts/ci/docs/ci_docs.sh | 10 +-
scripts/ci/images/ci_build_dockerhub.sh | 4 +-
scripts/ci/images/ci_prepare_ci_image_on_ci.sh | 43 +
scripts/ci/images/ci_prepare_prod_image_on_ci.sh | 33 +-
.../{ci_push_ci_image.sh => ci_push_ci_images.sh} | 2 +-
.../ci_wait_for_all_ci_images.sh} | 48 +-
scripts/ci/images/ci_wait_for_all_prod_images.sh | 49 +
scripts/ci/in_container/run_flake8.sh | 38 -
scripts/ci/in_container/run_mypy.sh | 37 -
.../ci/kubernetes/ci_deploy_app_to_kubernetes.sh | 5 +-
scripts/ci/kubernetes/ci_run_kubernetes_tests.sh | 2 +-
scripts/ci/libraries/_build_images.sh | 277 ++--
scripts/ci/libraries/_docker.env | 1 +
scripts/ci/libraries/_initialization.sh | 562 +++++---
scripts/ci/libraries/_kind.sh | 69 +-
scripts/ci/libraries/_local_mounts.sh | 3 +-
scripts/ci/libraries/_md5sum.sh | 4 +-
scripts/ci/libraries/_permissions.sh | 12 +-
scripts/ci/libraries/_push_pull_remove_images.sh | 222 +++-
scripts/ci/libraries/_runs.sh | 15 +-
scripts/ci/libraries/_verbosity.sh | 89 +-
scripts/ci/openapi/client_codegen_diff.sh | 7 +-
scripts/ci/pre_commit/pre_commit_bat_tests.sh | 6 +-
.../ci/pre_commit/pre_commit_breeze_cmd_line.sh | 5 +-
.../pre_commit_build_providers_dependencies.sh | 3 +-
.../ci/pre_commit/pre_commit_check_integrations.sh | 6 +-
scripts/ci/pre_commit/pre_commit_check_license.sh | 6 +-
scripts/ci/pre_commit/pre_commit_ci_build.sh | 3 +
scripts/ci/pre_commit/pre_commit_flake8.sh | 7 +-
.../ci/pre_commit/pre_commit_lint_dockerfile.sh | 7 +-
.../ci/pre_commit/pre_commit_local_yml_mounts.sh | 4 +-
scripts/ci/pre_commit/pre_commit_mermaid.sh | 97 ++
scripts/ci/pre_commit/pre_commit_mypy.sh | 7 +-
scripts/ci/pre_commit/pre_commit_pylint.sh | 7 +-
scripts/ci/pre_commit/pre_commit_setup_cfg_file.sh | 3 +-
scripts/ci/pylint_todo.txt | 8 +-
.../{ci_bat_tests.sh => bat_tests.sh} | 3 +-
.../{ci_check_license.sh => check_license.sh} | 6 +-
.../ci/static_checks/{ci_flake8.sh => flake8.sh} | 6 +-
.../{ci_lint_dockerfile.sh => lint_dockerfile.sh} | 0
scripts/ci/static_checks/{ci_mypy.sh => mypy.sh} | 3 +-
.../ci/static_checks/{ci_pylint.sh => pylint.sh} | 10 +-
...fresh_pylint_todo.sh => refresh_pylint_todo.sh} | 3 +-
...i_run_static_checks.sh => run_static_checks.sh} | 0
scripts/ci/testing/ci_run_airflow_testing.sh | 4 +-
.../ci/tools/ci_check_if_tests_should_be_run.sh | 2 +-
scripts/ci/tools/ci_clear_tmp.sh | 2 +-
scripts/ci/tools/ci_count_changed_files.sh | 6 +-
scripts/ci/tools/ci_fix_ownership.sh | 2 +-
.../in_container/_in_container_script_init.sh | 2 +-
.../{ci => }/in_container/_in_container_utils.sh | 28 +-
scripts/{ci => }/in_container/airflow_ci.cfg | 0
scripts/{ci => }/in_container/check_environment.sh | 119 +-
.../{ci => }/in_container/configure_environment.sh | 2 +-
scripts/{ci => }/in_container/entrypoint_ci.sh | 22 +-
scripts/{ci => }/in_container/entrypoint_exec.sh | 2 +-
scripts/{ci => }/in_container/krb5/krb5.conf | 0
.../prod}/airflow_scheduler_autorestart.sh | 0
scripts/{ => in_container}/prod/clean-logs.sh | 0
scripts/{ => in_container}/prod/entrypoint_prod.sh | 0
.../in_container/quarantine_issue_header.md | 0
.../{ci => }/in_container/refresh_pylint_todo.sh | 2 +-
scripts/{ci => }/in_container/run_ci_tests.sh | 2 +-
scripts/{ci => }/in_container/run_clear_tmp.sh | 2 +-
scripts/{ci => }/in_container/run_cli_tool.sh | 7 +-
scripts/{ci => }/in_container/run_docs_build.sh | 6 +-
scripts/{ci => }/in_container/run_extract_tests.sh | 2 +-
scripts/{ci => }/in_container/run_fix_ownership.sh | 2 +-
.../run_flake8.sh} | 5 +-
.../in_container/run_generate_constraints.sh | 2 +-
.../run_mypy.sh} | 6 +-
.../in_container/run_prepare_backport_packages.sh | 2 +-
.../in_container/run_prepare_backport_readme.sh | 8 +-
scripts/{ci => }/in_container/run_pylint.sh | 19 +-
scripts/{ci => }/in_container/run_system_tests.sh | 8 +-
.../run_test_package_import_all_classes.sh | 14 +-
.../run_test_package_installation_separately.sh | 10 +-
.../in_container/update_quarantined_test_status.py | 6 +-
scripts/{ => tools}/list-integrations.py | 27 +-
setup.cfg | 3 +
setup.py | 67 +-
.../zendesk/hooks => api/auth/backend}/__init__.py | 0
tests/api/auth/backend/test_basic_auth.py | 135 ++
.../endpoints/test_config_endpoint.py | 62 +-
.../endpoints/test_connection_endpoint.py | 271 ++--
tests/api_connexion/endpoints/test_dag_endpoint.py | 281 ++--
.../endpoints/test_dag_run_endpoint.py | 235 ++--
.../endpoints/test_dag_source_endpoint.py | 76 +-
.../endpoints/test_event_log_endpoint.py | 107 +-
.../endpoints/test_extra_link_endpoint.py | 29 +-
.../endpoints/test_health_endpoint.py | 4 +-
.../endpoints/test_import_error_endpoint.py | 68 +-
tests/api_connexion/endpoints/test_log_endpoint.py | 103 +-
.../api_connexion/endpoints/test_pool_endpoint.py | 144 ++-
.../api_connexion/endpoints/test_task_endpoint.py | 48 +-
.../endpoints/test_task_instance_endpoint.py | 26 +-
.../endpoints/test_variable_endpoint.py | 164 ++-
.../endpoints/test_version_endpoint.py | 8 +-
.../api_connexion/endpoints/test_xcom_endpoint.py | 218 ++--
tests/api_connexion/schemas/test_common_schema.py | 16 +-
tests/api_connexion/schemas/test_config_schema.py | 21 +-
.../schemas/test_connection_schema.py | 67 +-
tests/api_connexion/schemas/test_dag_run_schema.py | 19 +-
tests/api_connexion/schemas/test_dag_schema.py | 7 +-
tests/api_connexion/schemas/test_error_schema.py | 4 +-
.../api_connexion/schemas/test_event_log_schema.py | 48 +-
tests/api_connexion/schemas/test_health_schema.py | 5 +-
tests/api_connexion/schemas/test_task_schema.py | 9 +-
tests/api_connexion/schemas/test_version_schema.py | 8 +-
tests/api_connexion/schemas/test_xcom_schema.py | 28 +-
tests/api_connexion/test_parameters.py | 3 -
tests/bats/bats_utils.bash | 6 +
.../bats/mocks/docker.sh | 3 +-
.../_common_values.sh => tests/bats/mocks/helm.sh | 2 +-
.../_default_branch.sh => tests/bats/mocks/kind.sh | 3 +-
.../bats/mocks/kubectl.sh | 3 +-
tests/build_provider_packages_dependencies.py | 5 +-
tests/cli/commands/test_connection_command.py | 57 +-
tests/cli/commands/test_sync_perm_command.py | 2 +
tests/cli/commands/test_user_command.py | 6 +-
tests/cluster_policies/__init__.py | 59 +
tests/conftest.py | 7 +-
.../dags/test_missing_owner.py | 28 +-
tests/dags/test_task_view_type_check.py | 4 +-
.../dags/test_with_non_default_owner.py | 28 +-
tests/executors/test_celery_executor.py | 11 +-
tests/executors/test_executor_loader.py | 6 +-
tests/executors/test_kubernetes_executor.py | 6 +-
tests/jobs/test_scheduler_job.py | 2 +-
tests/kubernetes/test_pod_generator.py | 65 +-
tests/kubernetes/test_worker_configuration.py | 12 +-
tests/models/test_baseoperator.py | 3 +-
tests/models/test_dagbag.py | 53 +-
tests/models/test_serialized_dag.py | 17 +-
tests/operators/test_sql.py | 26 +-
tests/providers/amazon/aws/hooks/test_athena.py | 72 +-
.../amazon/aws/hooks/test_aws_dynamodb.py | 31 +-
tests/providers/amazon/aws/hooks/test_base_aws.py | 85 +-
.../amazon/aws/hooks/test_batch_client.py | 59 +-
.../amazon/aws/hooks/test_batch_waiters.py | 32 +-
.../amazon/aws/hooks/test_cloud_formation.py | 24 +-
tests/providers/amazon/aws/hooks/test_datasync.py | 45 +-
tests/providers/amazon/aws/hooks/test_ec2.py | 24 +-
tests/providers/amazon/aws/hooks/test_emr.py | 12 +-
tests/providers/amazon/aws/hooks/test_glue.py | 69 +-
.../amazon/aws/hooks/test_glue_catalog.py | 76 +-
tests/providers/amazon/aws/hooks/test_kinesis.py | 24 +-
.../amazon/aws/hooks/test_lambda_function.py | 12 +-
tests/providers/amazon/aws/hooks/test_logs.py | 25 +-
tests/providers/amazon/aws/hooks/test_redshift.py | 11 +-
tests/providers/amazon/aws/hooks/test_s3.py | 72 +-
tests/providers/amazon/aws/hooks/test_sagemaker.py | 468 +++----
tests/providers/amazon/aws/hooks/test_ses.py | 69 +
tests/providers/amazon/aws/hooks/test_sns.py | 17 +-
tests/providers/amazon/aws/hooks/test_sqs.py | 1 -
.../amazon/aws/hooks/test_step_function.py | 13 +-
.../amazon/aws/log/test_cloudwatch_task_handler.py | 104 +-
.../amazon/aws/log/test_s3_task_handler.py | 45 +-
.../providers/amazon/aws/operators/test_athena.py | 94 +-
tests/providers/amazon/aws/operators/test_batch.py | 12 +-
.../amazon/aws/operators/test_cloud_formation.py | 30 +-
.../amazon/aws/operators/test_datasync.py | 133 +-
.../aws/operators/test_ec2_start_instance.py | 16 +-
.../amazon/aws/operators/test_ec2_stop_instance.py | 16 +-
tests/providers/amazon/aws/operators/test_ecs.py | 166 +--
.../amazon/aws/operators/test_ecs_system.py | 15 +-
.../amazon/aws/operators/test_emr_add_steps.py | 120 +-
.../aws/operators/test_emr_create_job_flow.py | 83 +-
.../aws/operators/test_emr_modify_cluster.py | 20 +-
.../amazon/aws/operators/test_emr_system.py | 1 +
.../aws/operators/test_emr_terminate_job_flow.py | 10 +-
.../amazon/aws/operators/test_example_s3_bucket.py | 1 +
tests/providers/amazon/aws/operators/test_glue.py | 27 +-
.../amazon/aws/operators/test_s3_bucket.py | 10 +-
.../amazon/aws/operators/test_s3_copy_object.py | 41 +-
.../amazon/aws/operators/test_s3_delete_objects.py | 49 +-
.../amazon/aws/operators/test_s3_file_transform.py | 24 +-
.../providers/amazon/aws/operators/test_s3_list.py | 6 +-
.../amazon/aws/operators/test_sagemaker_base.py | 41 +-
.../aws/operators/test_sagemaker_endpoint.py | 64 +-
.../operators/test_sagemaker_endpoint_config.py | 18 +-
.../amazon/aws/operators/test_sagemaker_model.py | 20 +-
.../aws/operators/test_sagemaker_processing.py | 110 +-
.../aws/operators/test_sagemaker_training.py | 119 +-
.../aws/operators/test_sagemaker_transform.py | 67 +-
.../amazon/aws/operators/test_sagemaker_tuning.py | 163 +--
tests/providers/amazon/aws/operators/test_sns.py | 1 -
tests/providers/amazon/aws/operators/test_sqs.py | 8 +-
.../test_step_function_get_execution_output.py | 21 +-
.../test_step_function_start_execution.py | 11 +-
.../amazon/aws/secrets/test_secrets_manager.py | 17 +-
.../amazon/aws/secrets/test_systems_manager.py | 45 +-
tests/providers/amazon/aws/sensors/test_athena.py | 13 +-
.../amazon/aws/sensors/test_cloud_formation.py | 13 +-
.../amazon/aws/sensors/test_ec2_instance_state.py | 32 +-
.../providers/amazon/aws/sensors/test_emr_base.py | 35 +-
.../amazon/aws/sensors/test_emr_job_flow.py | 132 +-
.../providers/amazon/aws/sensors/test_emr_step.py | 117 +-
tests/providers/amazon/aws/sensors/test_glue.py | 43 +-
.../aws/sensors/test_glue_catalog_partition.py | 45 +-
.../providers/amazon/aws/sensors/test_redshift.py | 44 +-
tests/providers/amazon/aws/sensors/test_s3_key.py | 32 +-
.../amazon/aws/sensors/test_s3_keys_unchanged.py | 27 +-
.../providers/amazon/aws/sensors/test_s3_prefix.py | 11 +-
.../amazon/aws/sensors/test_sagemaker_base.py | 50 +-
.../amazon/aws/sensors/test_sagemaker_endpoint.py | 34 +-
.../amazon/aws/sensors/test_sagemaker_training.py | 36 +-
.../amazon/aws/sensors/test_sagemaker_transform.py | 34 +-
.../amazon/aws/sensors/test_sagemaker_tuning.py | 34 +-
tests/providers/amazon/aws/sensors/test_sqs.py | 54 +-
.../aws/sensors/test_step_function_execution.py | 39 +-
.../amazon/aws/transfers/test_dynamodb_to_s3.py | 10 +-
.../amazon/aws/transfers/test_gcs_to_s3.py | 110 +-
.../amazon/aws/transfers/test_google_api_to_s3.py | 38 +-
.../aws/transfers/test_google_api_to_s3_system.py | 6 +-
.../amazon/aws/transfers/test_hive_to_dynamodb.py | 63 +-
.../aws/transfers/test_imap_attachment_to_s3.py | 7 +-
.../transfers/test_imap_attachment_to_s3_system.py | 6 +-
.../amazon/aws/transfers/test_mongo_to_s3.py | 24 +-
.../amazon/aws/transfers/test_mysql_to_s3.py | 26 +-
.../amazon/aws/transfers/test_redshift_to_s3.py | 32 +-
.../amazon/aws/transfers/test_s3_to_redshift.py | 22 +-
.../amazon/aws/transfers/test_s3_to_sftp.py | 17 +-
.../amazon/aws/transfers/test_sftp_to_s3.py | 16 +-
.../apache/cassandra/hooks/test_cassandra.py | 114 +-
.../apache/cassandra/sensors/test_table.py | 8 +-
tests/providers/apache/druid/hooks/test_druid.py | 40 +-
.../providers/apache/druid/operators/test_druid.py | 12 +-
.../apache/druid/operators/test_druid_check.py | 7 +-
.../apache/druid/transfers/test_hive_to_druid.py | 50 +-
tests/providers/apache/hdfs/hooks/test_hdfs.py | 33 +-
tests/providers/apache/hdfs/hooks/test_webhdfs.py | 46 +-
tests/providers/apache/hdfs/sensors/test_hdfs.py | 197 +--
.../providers/apache/hdfs/sensors/test_web_hdfs.py | 13 +-
tests/providers/apache/hive/__init__.py | 1 -
tests/providers/apache/hive/hooks/test_hive.py | 665 +++++-----
tests/providers/apache/hive/operators/test_hive.py | 229 ++--
.../apache/hive/operators/test_hive_stats.py | 246 ++--
tests/providers/apache/hive/sensors/test_hdfs.py | 11 +-
.../apache/hive/sensors/test_hive_partition.py | 18 +-
.../hive/sensors/test_metastore_partition.py | 10 +-
.../hive/sensors/test_named_hive_partition.py | 120 +-
.../apache/hive/transfers/test_hive_to_mysql.py | 58 +-
.../apache/hive/transfers/test_hive_to_samba.py | 37 +-
.../apache/hive/transfers/test_mssql_to_hive.py | 30 +-
.../apache/hive/transfers/test_mysql_to_hive.py | 280 ++--
.../apache/hive/transfers/test_s3_to_hive.py | 115 +-
.../apache/hive/transfers/test_vertica_to_hive.py | 27 +-
tests/providers/apache/kylin/hooks/test_kylin.py | 24 +-
.../apache/kylin/operators/test_kylin_cube.py | 89 +-
tests/providers/apache/livy/hooks/test_livy.py | 143 +--
tests/providers/apache/livy/operators/test_livy.py | 47 +-
tests/providers/apache/livy/sensors/test_livy.py | 16 +-
tests/providers/apache/pig/hooks/test_pig.py | 2 +-
tests/providers/apache/pinot/hooks/test_pinot.py | 129 +-
.../apache/spark/hooks/test_spark_jdbc.py | 73 +-
.../apache/spark/hooks/test_spark_jdbc_script.py | 78 +-
.../providers/apache/spark/hooks/test_spark_sql.py | 107 +-
.../apache/spark/hooks/test_spark_submit.py | 595 +++++----
.../apache/spark/operators/test_spark_jdbc.py | 26 +-
.../apache/spark/operators/test_spark_sql.py | 13 +-
.../apache/spark/operators/test_spark_submit.py | 81 +-
tests/providers/apache/sqoop/hooks/test_sqoop.py | 185 +--
.../providers/apache/sqoop/operators/test_sqoop.py | 47 +-
.../providers/celery/sensors/test_celery_queue.py | 38 +-
tests/providers/cloudant/hooks/test_cloudant.py | 13 +-
.../cncf/kubernetes/hooks/test_kubernetes.py | 38 +-
.../kubernetes/operators/test_kubernetes_pod.py | 19 +-
.../kubernetes/operators/test_spark_kubernetes.py | 171 +--
.../operators/test_spark_kubernetes_system.py | 16 +-
.../kubernetes/sensors/test_spark_kubernetes.py | 1088 +++++++++-------
.../providers/databricks/hooks/test_databricks.py | 194 ++-
.../databricks/operators/test_databricks.py | 328 ++---
tests/providers/datadog/hooks/test_datadog.py | 21 +-
tests/providers/datadog/sensors/test_datadog.py | 77 +-
tests/providers/dingding/hooks/test_dingding.py | 127 +-
.../providers/dingding/operators/test_dingding.py | 15 +-
.../discord/hooks/test_discord_webhook.py | 7 +-
.../discord/operators/test_discord_webhook.py | 13 +-
tests/providers/docker/hooks/test_docker.py | 55 +-
tests/providers/docker/operators/test_docker.py | 155 +--
.../docker/operators/test_docker_swarm.py | 39 +-
.../elasticsearch/hooks/test_elasticsearch.py | 12 +-
.../elasticsearch/log/elasticmock/__init__.py | 7 +-
.../log/elasticmock/fake_elasticsearch.py | 223 ++--
.../elasticsearch/log/test_es_task_handler.py | 123 +-
tests/providers/exasol/hooks/test_exasol.py | 8 +-
tests/providers/exasol/operators/test_exasol.py | 35 +-
tests/providers/facebook/ads/hooks/test_ads.py | 36 +-
tests/providers/ftp/hooks/test_ftp.py | 16 +-
tests/providers/ftp/sensors/test_ftp.py | 31 +-
tests/providers/google/ads/hooks/test_ads.py | 8 +-
tests/providers/google/ads/operators/test_ads.py | 8 +-
.../google/ads/transfers/test_ads_to_gcs.py | 18 +-
.../_internal_client/test_secret_manager_client.py | 33 +-
tests/providers/google/cloud/hooks/test_automl.py | 68 +-
.../providers/google/cloud/hooks/test_bigquery.py | 801 +++++-------
.../google/cloud/hooks/test_bigquery_dts.py | 24 +-
.../google/cloud/hooks/test_bigquery_system.py | 7 +-
.../providers/google/cloud/hooks/test_bigtable.py | 239 +++-
.../google/cloud/hooks/test_cloud_build.py | 31 +-
.../google/cloud/hooks/test_cloud_memorystore.py | 98 +-
.../providers/google/cloud/hooks/test_cloud_sql.py | 731 ++++++-----
.../hooks/test_cloud_storage_transfer_service.py | 116 +-
tests/providers/google/cloud/hooks/test_compute.py | 281 ++--
.../google/cloud/hooks/test_datacatalog.py | 271 +---
.../providers/google/cloud/hooks/test_dataflow.py | 624 +++++----
.../google/cloud/hooks/test_datafusion.py | 92 +-
.../providers/google/cloud/hooks/test_dataprep.py | 88 ++
.../providers/google/cloud/hooks/test_dataproc.py | 112 +-
.../providers/google/cloud/hooks/test_datastore.py | 190 +--
tests/providers/google/cloud/hooks/test_dlp.py | 315 ++---
.../providers/google/cloud/hooks/test_functions.py | 142 +-
tests/providers/google/cloud/hooks/test_gcs.py | 376 +++---
tests/providers/google/cloud/hooks/test_gdm.py | 33 +-
tests/providers/google/cloud/hooks/test_kms.py | 14 +-
.../google/cloud/hooks/test_kubernetes_engine.py | 133 +-
.../google/cloud/hooks/test_life_sciences.py | 89 +-
.../providers/google/cloud/hooks/test_mlengine.py | 622 ++++-----
.../google/cloud/hooks/test_natural_language.py | 34 +-
tests/providers/google/cloud/hooks/test_pubsub.py | 146 +--
.../google/cloud/hooks/test_secret_manager.py | 16 +-
tests/providers/google/cloud/hooks/test_spanner.py | 170 +--
.../google/cloud/hooks/test_speech_to_text.py | 5 +-
.../google/cloud/hooks/test_stackdriver.py | 164 +--
tests/providers/google/cloud/hooks/test_tasks.py | 83 +-
.../google/cloud/hooks/test_text_to_speech.py | 5 +-
.../providers/google/cloud/hooks/test_translate.py | 5 +-
.../google/cloud/hooks/test_video_intelligence.py | 5 +-
tests/providers/google/cloud/hooks/test_vision.py | 45 +-
.../google/cloud/log/test_gcs_task_handler.py | 216 ++++
...r_system.py => test_gcs_task_handler_system.py} | 81 +-
.../cloud/log/test_stackdriver_task_handler.py | 122 +-
.../log/test_stackdriver_task_handler_system.py | 35 +-
.../google/cloud/operators/test_automl.py | 63 +-
.../google/cloud/operators/test_bigquery.py | 639 +++++----
.../google/cloud/operators/test_bigquery_dts.py | 3 +-
.../cloud/operators/test_bigquery_dts_system.py | 19 +-
.../google/cloud/operators/test_bigquery_system.py | 4 +
.../google/cloud/operators/test_bigtable.py | 632 ++++++---
.../google/cloud/operators/test_bigtable_system.py | 20 +-
.../google/cloud/operators/test_cloud_build.py | 21 +-
.../cloud/operators/test_cloud_build_system.py | 1 +
.../cloud/operators/test_cloud_memorystore.py | 64 +-
.../google/cloud/operators/test_cloud_sql.py | 682 +++++-----
.../cloud/operators/test_cloud_sql_system.py | 52 +-
.../operators/test_cloud_sql_system_helper.py | 457 ++++---
.../test_cloud_storage_transfer_service.py | 127 +-
.../test_cloud_storage_transfer_service_system.py | 4 +-
.../google/cloud/operators/test_compute.py | 632 ++++-----
.../cloud/operators/test_compute_system_helper.py | 200 ++-
.../google/cloud/operators/test_datacatalog.py | 141 +-
.../google/cloud/operators/test_dataflow.py | 58 +-
.../google/cloud/operators/test_datafusion.py | 47 +-
.../google/cloud/operators/test_dataprep.py | 32 +-
.../google/cloud/operators/test_dataproc.py | 196 ++-
.../google/cloud/operators/test_dataproc_system.py | 8 +-
.../google/cloud/operators/test_datastore.py | 104 +-
.../cloud/operators/test_datastore_system.py | 1 -
tests/providers/google/cloud/operators/test_dlp.py | 261 ++--
.../google/cloud/operators/test_functions.py | 583 +++++----
tests/providers/google/cloud/operators/test_gcs.py | 53 +-
.../cloud/operators/test_gcs_system_helper.py | 6 +-
.../cloud/operators/test_kubernetes_engine.py | 175 +--
.../google/cloud/operators/test_life_sciences.py | 29 +-
.../cloud/operators/test_life_sciences_system.py | 7 +-
.../google/cloud/operators/test_mlengine.py | 361 +++---
.../google/cloud/operators/test_mlengine_system.py | 7 +-
.../google/cloud/operators/test_mlengine_utils.py | 54 +-
.../cloud/operators/test_natural_language.py | 11 +-
.../google/cloud/operators/test_pubsub.py | 84 +-
.../google/cloud/operators/test_spanner.py | 332 ++---
.../google/cloud/operators/test_spanner_system.py | 20 +-
.../google/cloud/operators/test_speech_to_text.py | 12 +-
.../cloud/operators/test_speech_to_text_system.py | 1 -
.../google/cloud/operators/test_stackdriver.py | 144 +--
.../providers/google/cloud/operators/test_tasks.py | 139 +-
.../google/cloud/operators/test_text_to_speech.py | 10 +-
.../cloud/operators/test_text_to_speech_system.py | 1 -
.../google/cloud/operators/test_translate.py | 6 +-
.../cloud/operators/test_translate_speech.py | 38 +-
.../operators/test_translate_speech_system.py | 1 -
.../cloud/operators/test_video_intelligence.py | 37 +-
.../operators/test_video_intelligence_system.py | 8 +-
.../google/cloud/operators/test_vision.py | 133 +-
.../google/cloud/secrets/test_secret_manager.py | 74 +-
.../cloud/secrets/test_secret_manager_system.py | 70 +
.../google/cloud/sensors/test_bigquery.py | 44 +-
.../google/cloud/sensors/test_bigtable.py | 54 +-
tests/providers/google/cloud/sensors/test_gcs.py | 56 +-
.../providers/google/cloud/sensors/test_pubsub.py | 40 +-
.../google/cloud/transfers/test_adls_to_gcs.py | 58 +-
.../cloud/transfers/test_bigquery_to_bigquery.py | 28 +-
.../google/cloud/transfers/test_bigquery_to_gcs.py | 27 +-
.../cloud/transfers/test_bigquery_to_gcs_system.py | 1 -
.../cloud/transfers/test_bigquery_to_mysql.py | 9 +-
.../cloud/transfers/test_cassandra_to_gcs.py | 22 +-
.../cloud/transfers/test_facebook_ads_to_gcs.py | 47 +-
.../transfers/test_facebook_ads_to_gcs_system.py | 26 +-
.../google/cloud/transfers/test_gcs_to_bigquery.py | 41 +-
.../cloud/transfers/test_gcs_to_bigquery_system.py | 1 -
.../google/cloud/transfers/test_gcs_to_gcs.py | 303 +++--
.../cloud/transfers/test_gcs_to_gcs_system.py | 14 +-
.../google/cloud/transfers/test_gcs_to_local.py | 5 +-
.../google/cloud/transfers/test_gcs_to_sftp.py | 27 +-
.../cloud/transfers/test_gcs_to_sftp_system.py | 15 +-
.../google/cloud/transfers/test_local_to_gcs.py | 52 +-
.../cloud/transfers/test_local_to_gcs_system.py | 1 -
.../google/cloud/transfers/test_mssql_to_gcs.py | 33 +-
.../google/cloud/transfers/test_mysql_to_gcs.py | 125 +-
.../google/cloud/transfers/test_postgres_to_gcs.py | 40 +-
.../google/cloud/transfers/test_presto_to_gcs.py | 20 +-
.../cloud/transfers/test_presto_to_gcs_system.py | 15 +-
.../google/cloud/transfers/test_s3_to_gcs.py | 36 +-
.../google/cloud/transfers/test_sftp_to_gcs.py | 16 +-
.../cloud/transfers/test_sftp_to_gcs_system.py | 8 +-
.../google/cloud/transfers/test_sheets_to_gcs.py | 26 +-
.../cloud/transfers/test_sheets_to_gcs_system.py | 1 -
.../google/cloud/transfers/test_sql_to_gcs.py | 102 +-
.../providers/google/cloud/utils/base_gcp_mock.py | 22 +-
.../google/cloud/utils/gcp_authenticator.py | 1 +
.../cloud/utils/test_credentials_provider.py | 206 +--
.../google/cloud/utils/test_field_sanitizer.py | 105 +-
.../google/cloud/utils/test_field_validator.py | 64 +-
.../cloud/utils/test_mlengine_operator_utils.py | 161 +--
.../utils/test_mlengine_prediction_summary.py | 41 +-
.../google/common/hooks/test_base_google.py | 214 ++--
.../google/common/hooks/test_discovery_api.py | 101 +-
.../common/utils/test_id_token_credentials.py | 3 +-
.../google/firebase/hooks/test_firestore.py | 57 +-
.../firebase/operators/test_firestore_system.py | 3 +-
.../marketing_platform/hooks/test_analytics.py | 97 +-
.../hooks/test_campaign_manager.py | 45 +-
.../marketing_platform/hooks/test_display_video.py | 123 +-
.../marketing_platform/hooks/test_search_ads.py | 33 +-
.../marketing_platform/operators/test_analytics.py | 92 +-
.../operators/test_campaign_manager.py | 125 +-
.../operators/test_display_video.py | 189 +--
.../operators/test_display_video_system.py | 3 +-
.../operators/test_search_ads.py | 62 +-
.../operators/test_search_ads_system.py | 3 +-
.../sensors/test_campaign_manager.py | 10 +-
.../sensors/test_display_video.py | 23 +-
.../marketing_platform/sensors/test_search_ads.py | 16 +-
tests/providers/google/suite/hooks/test_drive.py | 4 +-
tests/providers/google/suite/hooks/test_sheets.py | 67 +-
.../google/suite/operators/test_sheets.py | 4 +-
.../google/suite/transfers/test_gcs_to_gdrive.py | 35 +-
.../google/suite/transfers/test_gcs_to_sheets.py | 12 +-
.../suite/transfers/test_gcs_to_sheets_system.py | 1 -
tests/providers/grpc/hooks/test_grpc.py | 109 +-
tests/providers/grpc/operators/test_grpc.py | 18 +-
.../_internal_client/test_vault_client.py | 553 ++++----
tests/providers/hashicorp/hooks/test_vault.py | 370 ++----
tests/providers/hashicorp/secrets/test_vault.py | 96 +-
tests/providers/http/hooks/test_http.py | 137 +-
tests/providers/http/operators/test_http.py | 15 +-
tests/providers/http/operators/test_http_system.py | 5 +-
tests/providers/http/sensors/test_http.py | 43 +-
tests/providers/imap/hooks/test_imap.py | 86 +-
.../providers/imap/sensors/test_imap_attachment.py | 5 +-
tests/providers/jdbc/hooks/test_jdbc.py | 23 +-
tests/providers/jdbc/operators/test_jdbc.py | 10 +-
tests/providers/jenkins/hooks/test_jenkins.py | 29 +-
.../jenkins/operators/test_jenkins_job_trigger.py | 152 ++-
tests/providers/jira/hooks/test_jira.py | 17 +-
tests/providers/jira/operators/test_jira.py | 65 +-
tests/providers/jira/sensors/test_jira.py | 36 +-
tests/providers/microsoft/azure/hooks/test_adx.py | 171 +--
.../microsoft/azure/hooks/test_azure_batch.py | 93 +-
.../azure/hooks/test_azure_container_instance.py | 39 +-
.../azure/hooks/test_azure_container_registry.py | 1 -
.../azure/hooks/test_azure_container_volume.py | 17 +-
.../microsoft/azure/hooks/test_azure_cosmos.py | 54 +-
.../microsoft/azure/hooks/test_azure_data_lake.py | 91 +-
.../microsoft/azure/hooks/test_azure_fileshare.py | 59 +-
.../microsoft/azure/hooks/test_base_azure.py | 79 ++
tests/providers/microsoft/azure/hooks/test_wasb.py | 96 +-
.../hooks => microsoft/azure/log}/__init__.py | 0
.../microsoft/azure/log/test_wasb_task_handler.py | 164 +++
.../microsoft/azure/operators/test_adls_list.py | 17 +-
.../microsoft/azure/operators/test_adx.py | 46 +-
.../microsoft/azure/operators/test_azure_batch.py | 52 +-
.../operators/test_azure_container_instances.py | 115 +-
.../microsoft/azure/operators/test_azure_cosmos.py | 17 +-
.../azure/operators/test_wasb_delete_blob.py | 33 +-
.../providers/microsoft/azure/sensors/test_wasb.py | 56 +-
.../microsoft/azure/transfers/test_file_to_wasb.py | 31 +-
.../transfers/test_oracle_to_azure_data_lake.py | 19 +-
.../microsoft/mssql/operators/test_mssql.py | 4 +-
.../providers/microsoft/winrm/hooks/test_winrm.py | 26 +-
.../microsoft/winrm/operators/test_winrm.py | 10 +-
tests/providers/mongo/hooks/test_mongo.py | 34 +-
tests/providers/mongo/sensors/test_mongo.py | 15 +-
tests/providers/mysql/hooks/test_mysql.py | 85 +-
tests/providers/mysql/operators/test_mysql.py | 37 +-
.../mysql/transfers/test_presto_to_mysql.py | 23 +-
.../providers/mysql/transfers/test_s3_to_mysql.py | 33 +-
.../mysql/transfers/test_vertica_to_mysql.py | 69 +-
tests/providers/odbc/hooks/test_odbc.py | 18 +-
tests/providers/openfaas/hooks/test_openfaas.py | 28 +-
.../opsgenie/hooks/test_opsgenie_alert.py | 39 +-
.../opsgenie/operators/test_opsgenie_alert.py | 25 +-
tests/providers/oracle/hooks/test_oracle.py | 79 +-
tests/providers/oracle/operators/test_oracle.py | 9 +-
.../oracle/transfers/test_oracle_to_oracle.py | 12 +-
tests/providers/pagerduty/hooks/test_pagerduty.py | 48 +-
.../papermill/operators/test_papermill.py | 15 +-
tests/providers/postgres/hooks/test_postgres.py | 72 +-
.../providers/postgres/operators/test_postgres.py | 25 +-
tests/providers/presto/hooks/test_presto.py | 26 +-
tests/providers/qubole/operators/test_qubole.py | 54 +-
.../qubole/operators/test_qubole_check.py | 21 +-
tests/providers/qubole/sensors/test_qubole.py | 14 +-
tests/providers/redis/hooks/test_redis.py | 20 +-
.../redis/operators/test_redis_publish.py | 8 +-
tests/providers/redis/sensors/test_redis_key.py | 11 +-
.../providers/redis/sensors/test_redis_pub_sub.py | 49 +-
.../providers/salesforce/hooks/test_salesforce.py | 7 +-
tests/providers/salesforce/hooks/test_tableau.py | 20 +-
.../operators/test_tableau_refresh_workbook.py | 9 +-
.../salesforce/sensors/test_tableau_job_status.py | 11 +-
tests/providers/samba/hooks/test_samba.py | 25 +-
tests/providers/segment/hooks/test_segment.py | 2 -
.../segment/operators/test_segment_track_event.py | 12 +-
tests/providers/sendgrid/utils/test_emailer.py | 73 +-
tests/providers/sftp/hooks/test_sftp.py | 127 +-
tests/providers/sftp/operators/test_sftp.py | 120 +-
tests/providers/sftp/sensors/test_sftp.py | 43 +-
.../singularity/operators/test_singularity.py | 101 +-
tests/providers/slack/hooks/test_slack.py | 16 +-
tests/providers/slack/hooks/test_slack_webhook.py | 33 +-
tests/providers/slack/operators/test_slack.py | 43 +-
.../slack/operators/test_slack_webhook.py | 30 +-
tests/providers/snowflake/hooks/test_snowflake.py | 89 +-
.../snowflake/operators/test_snowflake.py | 9 +-
.../snowflake/operators/test_snowflake_system.py | 15 +-
.../snowflake/transfers/test_s3_to_snowflake.py | 21 +-
.../snowflake/transfers/test_snowflake_to_slack.py | 20 +-
tests/providers/sqlite/hooks/test_sqlite.py | 2 -
tests/providers/sqlite/operators/test_sqlite.py | 8 +-
tests/providers/ssh/hooks/test_ssh.py | 124 +-
tests/providers/ssh/operators/test_ssh.py | 76 +-
tests/providers/vertica/hooks/test_vertica.py | 18 +-
tests/providers/vertica/operators/test_vertica.py | 8 +-
tests/providers/yandex/hooks/test_yandex.py | 34 +-
.../yandex/hooks/test_yandexcloud_dataproc.py | 36 +-
.../yandex/operators/test_yandexcloud_dataproc.py | 138 +-
tests/providers/zendesk/hooks/test_zendesk.py | 40 +-
tests/secrets/test_secrets.py | 14 +-
tests/sensors/test_external_task_sensor.py | 28 +-
tests/serialization/test_dag_serialization.py | 2 +-
tests/test_project_structure.py | 2 -
tests/test_sentry.py | 9 +-
tests/test_utils/api_connexion_utils.py | 47 +
tests/test_utils/mock_operators.py | 2 +-
.../test_utils/remote_user_api_auth_backend.py | 45 +-
tests/test_utils/system_tests_class.py | 5 +-
.../test_remote_user_api_auth_backend.py | 95 ++
{scripts => tests/utils}/perf/dags/elastic_dag.py | 0
{scripts => tests/utils}/perf/dags/perf_dag_1.py | 0
{scripts => tests/utils}/perf/dags/perf_dag_2.py | 0
{scripts => tests/utils}/perf/dags/sql_perf_dag.py | 0
{scripts => tests/utils}/perf/perf_kit/__init__.py | 29 +-
{scripts => tests/utils}/perf/perf_kit/memory.py | 1 +
{scripts => tests/utils}/perf/perf_kit/python.py | 22 +-
.../utils}/perf/perf_kit/repeat_and_time.py | 26 +-
.../utils}/perf/perf_kit/sqlalchemy.py | 65 +-
.../utils}/perf/scheduler_dag_execution_timing.py | 20 +-
.../utils}/perf/scheduler_ops_metrics.py | 0
{scripts => tests/utils}/perf/sql_queries.py | 0
tests/utils/test_dates.py | 13 +-
tests/utils/test_email.py | 24 +-
tests/www/test_views.py | 65 +-
1391 files changed, 45714 insertions(+), 35725 deletions(-)
create mode 100644 .github/ISSUE_TEMPLATE/config.yml
delete mode 100644 .github/ISSUE_TEMPLATE/support_request.md
create mode 100644 .github/workflows/build-images-workflow-run.yml
delete mode 100644 .github/workflows/cancel_other_workflow_runs.yml
rename .github/workflows/{quarantined.yaml => scheduled_quarantined.yml} (80%)
copy README.md => INTHEWILD.md (70%)
copy airflow/api/auth/backend/{deny_all.py => basic_auth.py} (52%)
copy airflow/{api/auth/backend/deny_all.py => api_connexion/security.py} (66%)
copy airflow/{api_connexion/schemas/health_schema.py => migrations/versions/da3f683c3a5a_add_dag_hash_column_to_serialized_dag_.py} (53%)
create mode 100644 airflow/providers/amazon/aws/hooks/ses.py
create mode 100644 airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py
copy airflow/providers/google/cloud/example_dags/{example_postgres_to_gcs.py => example_dataprep.py} (58%)
create mode 100644 airflow/providers/google/cloud/hooks/dataprep.py
create mode 100644 airflow/providers/google/cloud/operators/dataprep.py
create mode 100644 airflow/providers/microsoft/azure/hooks/base_azure.py
copy scripts/ci/in_container/quarantine_issue_header.md => chart/dockerfiles/README.md (60%)
copy scripts/ci/in_container/quarantine_issue_header.md => chart/dockerfiles/pgbouncer-exporter/README.md (60%)
create mode 100644 chart/dockerfiles/pgbouncer/Dockerfile
copy scripts/prod/clean-logs.sh => chart/dockerfiles/pgbouncer/build_and_push.sh (58%)
copy airflow/api_connexion/schemas/variable_schema.py => chart/dockerfiles/statsd-exporter/Dockerfile (60%)
copy scripts/prod/clean-logs.sh => chart/dockerfiles/statsd-exporter/build_and_push.sh (57%)
create mode 100644 chart/dockerfiles/statsd-exporter/mappings.yml
create mode 100644 chart/templates/flower/flower-ingress.yaml
create mode 100644 chart/templates/webserver/webserver-ingress.yaml
delete mode 100644 common/_files_for_rebuild_check.sh
delete mode 100644 common/_image_variables.sh
delete mode 100755 dev/airflow-jira
rename docs/{build => build_docs.py} (68%)
create mode 100644 docs/howto/connection/azure.rst
create mode 100644 docs/howto/operator/google/cloud/dataprep.rst
create mode 100644 docs/howto/variable.rst
create mode 100644 docs/img/arch-diag-basic.png
create mode 100644 docs/img/arch-diag-kubernetes.png
create mode 100644 docs/img/arch-diag-kubernetes2.png
rename docs/{howto => logging-monitoring}/check-health.rst (97%)
rename docs/{ => logging-monitoring}/errors.rst (91%)
copy docs/{stable-rest-api/redoc.rst => logging-monitoring/index.rst} (80%)
rename docs/{howto/write-logs.rst => logging-monitoring/logging-tasks.rst} (94%)
rename docs/{ => logging-monitoring}/metrics.rst (100%)
rename docs/{howto => logging-monitoring}/tracking-user-activity.rst (98%)
delete mode 100644 docs/security.rst
create mode 100644 docs/security/access-control.rst
create mode 100644 docs/security/api.rst
copy docs/{howto/index.rst => security/flower.rst} (53%)
rename docs/{stable-rest-api => security}/index.rst (60%)
create mode 100644 docs/security/kerberos.rst
create mode 100644 docs/security/secrets/fernet.rst
copy docs/{stable-rest-api/redoc.rst => security/secrets/index.rst} (61%)
rename docs/{howto => security/secrets}/secrets-backend/aws-secrets-manaager-backend.rst (100%)
rename docs/{howto => security/secrets}/secrets-backend/aws-ssm-parameter-store-secrets-backend.rst (100%)
rename docs/{howto => security/secrets}/secrets-backend/google-cloud-secret-manager-backend.rst (74%)
rename docs/{howto => security/secrets}/secrets-backend/hashicorp-vault-secrets-backend.rst (100%)
rename docs/{howto => security/secrets}/secrets-backend/index.rst (89%)
rename docs/{howto => security/secrets}/secrets-backend/local-filesystem-secrets-backend.rst (96%)
create mode 100644 docs/security/webserver.rst
create mode 100644 docs/security/workload.rst
create mode 100644 docs/spelling_wordlist.txt
rename docs/{stable-rest-api/redoc.rst => stable-rest-api-ref.rst} (93%)
copy images/{ => ci}/CI.png (100%)
create mode 100644 images/ci/pull_request_ci_flow.md5
create mode 100644 images/ci/pull_request_ci_flow.mermaid
create mode 100644 images/ci/pull_request_ci_flow.png
create mode 100644 images/ci/push_ci_flow.md5
create mode 100644 images/ci/push_ci_flow.mermaid
create mode 100644 images/ci/push_ci_flow.png
create mode 100644 images/ci/scheduled_ci_flow.md5
create mode 100644 images/ci/scheduled_ci_flow.mermaid
create mode 100644 images/ci/scheduled_ci_flow.png
create mode 100644 pyproject.toml
copy scripts/ci/{docs/ci_docs.sh => constraints/ci_branch_constraints.sh} (75%)
copy scripts/ci/{static_checks/ci_mypy.sh => constraints/ci_commit_constraints.sh} (61%)
rename scripts/ci/docker-compose/{ci.yml => ga.yml} (100%)
rename scripts/ci/images/{ci_push_ci_image.sh => ci_push_ci_images.sh} (98%)
copy scripts/ci/{tools/ci_count_changed_files.sh => images/ci_wait_for_all_ci_images.sh} (50%)
create mode 100755 scripts/ci/images/ci_wait_for_all_prod_images.sh
delete mode 100755 scripts/ci/in_container/run_flake8.sh
delete mode 100755 scripts/ci/in_container/run_mypy.sh
create mode 100755 scripts/ci/pre_commit/pre_commit_mermaid.sh
rename scripts/ci/static_checks/{ci_bat_tests.sh => bat_tests.sh} (97%)
rename scripts/ci/static_checks/{ci_check_license.sh => check_license.sh} (89%)
rename scripts/ci/static_checks/{ci_flake8.sh => flake8.sh} (86%)
rename scripts/ci/static_checks/{ci_lint_dockerfile.sh => lint_dockerfile.sh} (100%)
rename scripts/ci/static_checks/{ci_mypy.sh => mypy.sh} (92%)
rename scripts/ci/static_checks/{ci_pylint.sh => pylint.sh} (84%)
rename scripts/ci/static_checks/{ci_refresh_pylint_todo.sh => refresh_pylint_todo.sh} (91%)
rename scripts/ci/static_checks/{ci_run_static_checks.sh => run_static_checks.sh} (100%)
rename scripts/{ci => }/in_container/_in_container_script_init.sh (94%)
rename scripts/{ci => }/in_container/_in_container_utils.sh (92%)
rename scripts/{ci => }/in_container/airflow_ci.cfg (100%)
rename scripts/{ci => }/in_container/check_environment.sh (50%)
rename scripts/{ci => }/in_container/configure_environment.sh (96%)
rename scripts/{ci => }/in_container/entrypoint_ci.sh (89%)
rename scripts/{ci => }/in_container/entrypoint_exec.sh (92%)
rename scripts/{ci => }/in_container/krb5/krb5.conf (100%)
rename scripts/{ => in_container/prod}/airflow_scheduler_autorestart.sh (100%)
rename scripts/{ => in_container}/prod/clean-logs.sh (100%)
rename scripts/{ => in_container}/prod/entrypoint_prod.sh (100%)
rename scripts/{ci => }/in_container/quarantine_issue_header.md (100%)
rename scripts/{ci => }/in_container/refresh_pylint_todo.sh (92%)
rename scripts/{ci => }/in_container/run_ci_tests.sh (96%)
copy scripts/{ci => }/in_container/run_clear_tmp.sh (92%)
rename scripts/{ci => }/in_container/run_cli_tool.sh (95%)
rename scripts/{ci => }/in_container/run_docs_build.sh (84%)
rename scripts/{ci => }/in_container/run_extract_tests.sh (93%)
copy scripts/{ci => }/in_container/run_fix_ownership.sh (92%)
rename scripts/{ci/in_container/run_clear_tmp.sh => in_container/run_flake8.sh} (89%)
rename scripts/{ci => }/in_container/run_generate_constraints.sh (95%)
rename scripts/{ci/in_container/run_fix_ownership.sh => in_container/run_mypy.sh} (84%)
rename scripts/{ci => }/in_container/run_prepare_backport_packages.sh (98%)
rename scripts/{ci => }/in_container/run_prepare_backport_readme.sh (89%)
rename scripts/{ci => }/in_container/run_pylint.sh (86%)
rename scripts/{ci => }/in_container/run_system_tests.sh (88%)
rename scripts/{ci => }/in_container/run_test_package_import_all_classes.sh (84%)
rename scripts/{ci => }/in_container/run_test_package_installation_separately.sh (91%)
rename scripts/{ci => }/in_container/update_quarantined_test_status.py (98%)
rename scripts/{ => tools}/list-integrations.py (82%)
copy tests/{providers/zendesk/hooks => api/auth/backend}/__init__.py (100%)
create mode 100644 tests/api/auth/backend/test_basic_auth.py
copy common/_default_branch.sh => tests/bats/mocks/docker.sh (90%)
mode change 100644 => 100755
rename common/_common_values.sh => tests/bats/mocks/helm.sh (90%)
mode change 100644 => 100755
copy common/_default_branch.sh => tests/bats/mocks/kind.sh (90%)
mode change 100644 => 100755
rename common/_default_branch.sh => tests/bats/mocks/kubectl.sh (90%)
mode change 100644 => 100755
create mode 100644 tests/cluster_policies/__init__.py
copy airflow/providers/apache/pig/example_dags/example_pig.py => tests/dags/test_missing_owner.py (68%)
copy airflow/providers/apache/pig/example_dags/example_pig.py => tests/dags/test_with_non_default_owner.py (68%)
create mode 100644 tests/providers/amazon/aws/hooks/test_ses.py
create mode 100644 tests/providers/google/cloud/hooks/test_dataprep.py
create mode 100644 tests/providers/google/cloud/log/test_gcs_task_handler.py
copy tests/providers/google/cloud/log/{test_stackdriver_task_handler_system.py => test_gcs_task_handler_system.py} (54%)
copy airflow/providers/apache/pig/example_dags/example_pig.py => tests/providers/google/cloud/operators/test_dataprep.py (55%)
create mode 100644 tests/providers/google/cloud/secrets/test_secret_manager_system.py
create mode 100644 tests/providers/microsoft/azure/hooks/test_base_azure.py
copy tests/providers/{zendesk/hooks => microsoft/azure/log}/__init__.py (100%)
create mode 100644 tests/providers/microsoft/azure/log/test_wasb_task_handler.py
create mode 100644 tests/test_utils/api_connexion_utils.py
copy airflow/api/auth/backend/deny_all.py => tests/test_utils/remote_user_api_auth_backend.py (53%)
create mode 100644 tests/test_utils/test_remote_user_api_auth_backend.py
rename {scripts => tests/utils}/perf/dags/elastic_dag.py (100%)
rename {scripts => tests/utils}/perf/dags/perf_dag_1.py (100%)
rename {scripts => tests/utils}/perf/dags/perf_dag_2.py (100%)
rename {scripts => tests/utils}/perf/dags/sql_perf_dag.py (100%)
rename {scripts => tests/utils}/perf/perf_kit/__init__.py (77%)
rename {scripts => tests/utils}/perf/perf_kit/memory.py (98%)
rename {scripts => tests/utils}/perf/perf_kit/python.py (86%)
rename {scripts => tests/utils}/perf/perf_kit/repeat_and_time.py (88%)
rename {scripts => tests/utils}/perf/perf_kit/sqlalchemy.py (70%)
rename {scripts => tests/utils}/perf/scheduler_dag_execution_timing.py (95%)
rename {scripts => tests/utils}/perf/scheduler_ops_metrics.py (100%)
rename {scripts => tests/utils}/perf/sql_queries.py (100%)