You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by gi...@apache.org on 2021/04/08 01:38:22 UTC
[airflow] tag nightly-master updated (7c17bf0 -> 3e9e954)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/airflow.git.
*** WARNING: tag nightly-master was modified! ***
from 7c17bf0 (commit)
to 3e9e954 (commit)
from 7c17bf0 Removes unused CI feature of printing output on error (#15190)
add e86f5ca Bugfix: Task docs are not shown in the Task Instance Detail View (#15191)
add 266384a Fix string concatenation using `f-strings` (#15200)
add 6706b67 Add CUD REST API endpoints for Roles (#14840)
add 7b577c3 Bugfix: resources in `executor_config` breaks Graph View in UI (#15199)
add 4099108 Update import path and fix typo in `dag-run.rst` (#15201)
add 5606137 Bugfix: Fix overriding `pod_template_file` in KubernetesExecutor (#15197)
add 7cadb63 Import connections from a file (#15177)
add 1087226 Merges quarantined tests into single job (#15153)
add e497228 Updates 3.6 limits for latest versions of a few libraries (#15209)
add 643d158 Add support for modifying celery worker deployment strategy (#15213)
add 6d7a70b Separate Kubernetes pod_launcher from core airflow (#15165)
add c73052f Update .github/CODEOWNERS (#15215)
add 3bd1163 Less docker magic in docs building (#15176)
add e1beefc Fix test bug introduced by #15165 (#15221)
add aebacd7 Constraints are now parallelized and merged in single job (#15211)
add 4ee4429 Fix celery executor bug trying to call len on map (#14883)
add 2d3b63d Build priority packages in separate processes pool (#15214)
add 1bb5587 Fix semantic mistake in ISSUE_TRIAGE_PROCESS.rst (#15224)
add b74f796 Fix deprecated warning hvac auth (#15216)
add 36d9274 Use context manager to manage pools (#15220)
add d89bcad API: Raise `AlreadyExists` exception when the execution_date is same (#15174)
add bc5ced3 Task Instance model: allow pool names >50 chars (#15203)
add eae22ce Adds 'Trino' provider (with lower memory footprint for tests) (#15187)
add 18e2c1d Fixed #14270: Add error message in OOM situations (#15207)
add 12d8e4b Add new Committers to docs (#15235)
add 042be2e Updated documentation for provider packages before April release (#15236)
add 97b7780 Fix missing on_load trigger for folder-based plugins (#15208)
add 8567420 Chart: Update the docs to create Kind cluster (#15237)
add 1806670 Retry pod launching on 409 ApiExceptions (#15137)
add 4e018a8 Refreshed provider's upcoming release with k8s retries (#15239)
add a265fd5 Revert "Fixes failing docs upload on master (#15148)" (#15240)
add e150bbf Clear tasks by task ids in REST API (#14500)
add 6b78394 Fixes pushing constraints (#15243)
add ab47717 Support all terminus task states in Docker Swarm Operator (#14960)
add b62ca0a Docs: Fix name of setting to use internal IP (#15251)
add 7087541 Remove user_id from API schema (#15117)
add 44a6648 Add logs to show last modified in SFTP, FTP and Filesystem sensor (#15134)
add ea0710e Run kubernetes tests in parallel (#15222)
add 019241b Submodules are needed to update constrains (#15242)
add 523fb5c use jquery ready instead of vanilla js (#15258)
add 47cbff9 Refactor/Cleanup Presentation of Graph Task and Path Highlighting (#15257)
add 3e9e954 Display explicit error in case UID has no actual username (#15212)
No new revisions were added by this update.
Summary of changes:
.github/CODEOWNERS | 4 +-
.github/workflows/ci.yml | 154 +++-------
.gitignore | 2 +-
.rat-excludes | 2 +-
BREEZE.rst | 41 +--
CONTRIBUTING.rst | 27 +-
IMAGES.rst | 4 +-
INSTALL | 2 +-
ISSUE_TRIAGE_PROCESS.rst | 2 +-
LICENSE | 1 +
TESTING.rst | 10 +-
UPDATING.md | 6 +
.../api_connexion/endpoints/dag_run_endpoint.py | 16 +-
.../endpoints/role_and_permission_endpoint.py | 89 +++++-
airflow/api_connexion/openapi/v1.yaml | 83 ++++-
.../api_connexion/schemas/task_instance_schema.py | 1 +
airflow/api_connexion/schemas/user_schema.py | 1 -
airflow/cli/cli_parser.py | 11 +
airflow/cli/commands/connection_command.py | 41 +++
airflow/cli/commands/info_command.py | 4 +-
.../example_kubernetes_executor_config.py | 3 +-
airflow/example_dags/tutorial.py | 1 +
airflow/executors/celery_executor.py | 22 +-
airflow/executors/kubernetes_executor.py | 26 +-
airflow/jobs/base_job.py | 4 +-
airflow/kubernetes/pod_launcher.py | 285 +----------------
...{pod_launcher.py => pod_launcher_deprecated.py} | 20 +-
.../basic_template.yaml | 4 +-
...b86_increase_pool_name_size_in_taskinstance.py} | 20 +-
airflow/models/baseoperator.py | 28 +-
airflow/models/dag.py | 5 +
airflow/models/taskinstance.py | 6 +-
airflow/plugins_manager.py | 23 +-
airflow/providers/amazon/CHANGELOG.rst | 20 ++
airflow/providers/amazon/aws/hooks/sns.py | 2 +-
airflow/providers/amazon/provider.yaml | 1 +
airflow/providers/apache/druid/hooks/druid.py | 2 +-
airflow/providers/apache/hive/CHANGELOG.rst | 10 +
airflow/providers/apache/hive/provider.yaml | 1 +
airflow/providers/apache/livy/CHANGELOG.rst | 13 +-
airflow/providers/apache/livy/provider.yaml | 1 +
airflow/providers/cncf/kubernetes/CHANGELOG.rst | 10 +
.../cncf/kubernetes/operators/kubernetes_pod.py | 3 +-
airflow/providers/cncf/kubernetes/provider.yaml | 1 +
.../providers/cncf/kubernetes/utils}/__init__.py | 0
.../cncf/kubernetes/utils}/pod_launcher.py | 14 +-
airflow/providers/dependencies.json | 4 +-
airflow/providers/dingding/hooks/dingding.py | 2 +-
airflow/providers/docker/CHANGELOG.rst | 8 +
airflow/providers/docker/operators/docker_swarm.py | 6 +-
airflow/providers/docker/provider.yaml | 1 +
airflow/providers/facebook/CHANGELOG.rst | 8 +
airflow/providers/facebook/provider.yaml | 1 +
airflow/providers/ftp/sensors/ftp.py | 4 +-
airflow/providers/google/CHANGELOG.rst | 20 ++
...le_presto_to_gcs.py => example_trino_to_gcs.py} | 58 ++--
.../providers/google/cloud/operators/dataflow.py | 6 +-
.../{presto_to_gcs.py => trino_to_gcs.py} | 46 +--
airflow/providers/google/provider.yaml | 5 +
airflow/providers/grpc/CHANGELOG.rst | 8 +
airflow/providers/grpc/provider.yaml | 1 +
airflow/providers/hashicorp/CHANGELOG.rst | 8 +
.../hashicorp/_internal_client/vault_client.py | 4 +-
airflow/providers/hashicorp/provider.yaml | 1 +
.../jenkins/operators/jenkins_job_trigger.py | 4 +-
airflow/providers/microsoft/azure/CHANGELOG.rst | 13 +
airflow/providers/microsoft/azure/provider.yaml | 1 +
airflow/providers/microsoft/winrm/CHANGELOG.rst | 14 +
airflow/providers/microsoft/winrm/hooks/winrm.py | 8 +-
airflow/providers/microsoft/winrm/provider.yaml | 1 +
airflow/providers/mysql/CHANGELOG.rst | 14 +
airflow/providers/mysql/provider.yaml | 6 +-
.../{presto_to_mysql.py => trino_to_mysql.py} | 22 +-
airflow/providers/opsgenie/CHANGELOG.rst | 8 +
airflow/providers/opsgenie/provider.yaml | 1 +
airflow/providers/oracle/CHANGELOG.rst | 8 +
airflow/providers/oracle/provider.yaml | 1 +
airflow/providers/sftp/sensors/sftp.py | 3 +-
airflow/providers/singularity/CHANGELOG.rst | 9 +
airflow/providers/singularity/provider.yaml | 1 +
airflow/providers/snowflake/CHANGELOG.rst | 5 +-
airflow/providers/snowflake/provider.yaml | 1 +
airflow/providers/ssh/CHANGELOG.rst | 8 +
airflow/providers/ssh/hooks/ssh.py | 8 +-
airflow/providers/ssh/provider.yaml | 1 +
airflow/providers/{tableau => trino}/CHANGELOG.rst | 0
.../providers/trino}/__init__.py | 0
.../providers/trino/hooks}/__init__.py | 0
.../hooks/presto.py => trino/hooks/trino.py} | 54 ++--
.../providers/{openfaas => trino}/provider.yaml | 21 +-
airflow/security/permissions.py | 1 +
airflow/sensors/filesystem.py | 4 +
airflow/sensors/sql.py | 1 +
airflow/serialization/schema.json | 7 +-
airflow/task/task_runner/base_task_runner.py | 4 +-
airflow/task/task_runner/cgroup_task_runner.py | 4 +-
airflow/task/task_runner/standard_task_runner.py | 8 +
airflow/utils/cli.py | 5 +-
airflow/utils/db.py | 10 +
airflow/utils/json.py | 2 +-
airflow/utils/platform.py | 23 ++
airflow/www/package.json | 1 +
airflow/www/static/css/graph.css | 42 ++-
airflow/www/static/css/main.css | 10 +-
airflow/www/static/js/task_instance.js | 2 +-
airflow/www/templates/airflow/graph.html | 341 ++++++++++-----------
airflow/www/utils.py | 2 +-
airflow/www/views.py | 2 +-
airflow/www/webpack.config.js | 4 +
airflow/www/yarn.lock | 12 +
breeze | 4 +-
breeze-complete | 2 +-
chart/templates/workers/worker-deployment.yaml | 3 +
chart/tests/test_worker.py | 16 +
chart/values.schema.json | 4 +
chart/values.yaml | 5 +
docs/apache-airflow-providers-amazon/commits.rst | 28 ++
docs/apache-airflow-providers-amazon/index.rst | 25 +-
.../commits.rst | 20 +-
.../apache-airflow-providers-apache-hive/index.rst | 12 +-
.../commits.rst | 23 +-
.../apache-airflow-providers-apache-livy/index.rst | 15 +-
.../commits.rst | 20 +-
.../index.rst | 12 +-
docs/apache-airflow-providers-docker/commits.rst | 16 +-
docs/apache-airflow-providers-docker/index.rst | 10 +-
docs/apache-airflow-providers-facebook/commits.rst | 23 +-
docs/apache-airflow-providers-facebook/index.rst | 10 +-
docs/apache-airflow-providers-google/commits.rst | 29 ++
docs/apache-airflow-providers-google/index.rst | 27 +-
.../operators/cloud/kubernetes_engine.rst | 2 +-
.../{presto_to_gcs.rst => trino_to_gcs.rst} | 40 +--
docs/apache-airflow-providers-grpc/commits.rst | 23 +-
docs/apache-airflow-providers-grpc/index.rst | 10 +-
.../apache-airflow-providers-hashicorp/commits.rst | 23 +-
docs/apache-airflow-providers-hashicorp/index.rst | 10 +-
.../commits.rst | 18 +-
.../index.rst | 16 +-
.../commits.rst | 23 +-
.../index.rst | 16 +-
docs/apache-airflow-providers-mysql/commits.rst | 18 +-
docs/apache-airflow-providers-mysql/index.rst | 17 +-
docs/apache-airflow-providers-opsgenie/commits.rst | 23 +-
docs/apache-airflow-providers-opsgenie/index.rst | 10 +-
docs/apache-airflow-providers-oracle/commits.rst | 23 +-
docs/apache-airflow-providers-oracle/index.rst | 10 +-
.../commits.rst | 3 +-
.../commits.rst | 24 +-
.../apache-airflow-providers-singularity/index.rst | 11 +-
.../apache-airflow-providers-snowflake/commits.rst | 23 +-
docs/apache-airflow-providers-snowflake/index.rst | 12 +-
docs/apache-airflow-providers-ssh/commits.rst | 16 +-
docs/apache-airflow-providers-ssh/index.rst | 10 +-
.../commits.rst | 16 +-
.../index.rst | 27 +-
docs/apache-airflow/concepts.rst | 6 +-
docs/apache-airflow/dag-run.rst | 6 +-
docs/apache-airflow/executor/kubernetes.rst | 2 +-
docs/apache-airflow/extra-packages-ref.rst | 4 +-
docs/apache-airflow/img/graph.png | Bin 225347 -> 110218 bytes
docs/apache-airflow/project.rst | 2 +
docs/build_docs.py | 160 ++++------
docs/conf.py | 2 +-
docs/exts/docs_build/code_utils.py | 16 +-
docs/exts/docs_build/docs_builder.py | 170 +++-------
docs/exts/docs_build/errors.py | 8 +-
docs/exts/docs_build/run_patched_sphinx.py | 105 +++++++
docs/exts/docs_build/spelling_checks.py | 8 +-
docs/exts/provider_init_hack.py | 10 +-
docs/helm-chart/parameters-ref.rst | 3 +
docs/helm-chart/quick-start.rst | 5 +-
docs/integration-logos/trino/trino-og.png | Bin 0 -> 34219 bytes
docs/spelling_wordlist.txt | 7 +
kubernetes_tests/test_kubernetes_executor.py | 10 +-
kubernetes_tests/test_kubernetes_pod_operator.py | 24 +-
.../test_kubernetes_pod_operator_backcompat.py | 24 +-
.../{LICENSE-d3js.txt => LICENSE-d3-shape.txt} | 2 +-
scripts/ci/constraints/ci_commit_constraints.sh | 4 +-
.../ci_generate_all_constraints.sh} | 13 +-
scripts/ci/constraints/ci_generate_constraints.sh | 8 +
scripts/ci/docker-compose/integration-kerberos.yml | 2 +-
scripts/ci/docker-compose/integration-redis.yml | 2 +-
...ntegration-presto.yml => integration-trino.yml} | 20 +-
.../krb5-kdc-server/utils/create_service.sh | 2 +-
.../ci/dockerfiles/{presto => trino}/Dockerfile | 14 +-
.../{presto => trino}/build_and_push.sh | 14 +-
.../ci/dockerfiles/{presto => trino}/entrypoint.sh | 43 ++-
scripts/ci/docs/ci_docs.sh | 14 +-
.../images/ci_wait_for_and_verify_all_ci_images.sh | 5 +-
.../ci_wait_for_and_verify_all_prod_images.sh | 4 +-
.../ci/images/ci_wait_for_and_verify_ci_image.sh | 3 +-
.../ci/images/ci_wait_for_and_verify_prod_image.sh | 17 +-
scripts/ci/kubernetes/ci_run_kubernetes_tests.sh | 10 +-
...tup_cluster_and_deploy_airflow_to_kubernetes.sh | 6 +-
...cluster_and_run_kubernetes_tests_single_job.sh} | 51 ++-
...lusters_and_run_kubernetes_tests_in_parallel.sh | 106 +++++++
scripts/ci/kubernetes/kind-cluster-conf.yaml | 4 +-
scripts/ci/libraries/_all_libs.sh | 2 +
scripts/ci/libraries/_build_images.sh | 23 +-
scripts/ci/libraries/_docker_engine_resources.sh | 10 +-
scripts/ci/libraries/_initialization.sh | 15 +-
scripts/ci/libraries/_kind.sh | 57 ++--
scripts/ci/libraries/_parallel.sh | 59 +++-
scripts/ci/libraries/_testing.sh | 116 +++++++
scripts/ci/libraries/_verbosity.sh | 2 +-
scripts/ci/selective_ci_checks.sh | 2 +
scripts/ci/testing/ci_run_airflow_testing.sh | 140 +--------
scripts/ci/testing/ci_run_quarantined_tests.sh | 90 ++++++
.../ci_run_single_airflow_test_in_docker.sh | 6 +-
scripts/in_container/check_environment.sh | 14 +-
scripts/in_container/entrypoint_ci.sh | 4 +-
.../run_install_and_test_provider_packages.sh | 4 +-
setup.cfg | 8 +-
setup.py | 16 +-
.../endpoints/test_dag_run_endpoint.py | 24 +-
.../endpoints/test_role_and_permission_endpoint.py | 330 +++++++++++++++++++-
.../endpoints/test_task_instance_endpoint.py | 33 +-
.../api_connexion/endpoints/test_user_endpoint.py | 1 -
.../schemas/test_task_instance_schema.py | 15 +-
tests/api_connexion/schemas/test_user_schema.py | 6 +-
tests/cli/commands/test_connection_command.py | 177 +++++++++++
tests/conftest.py | 2 +-
tests/core/test_providers_manager.py | 2 +
.../basic_template.yaml} | 19 +-
tests/executors/test_celery_executor.py | 35 ++-
tests/executors/test_kubernetes_executor.py | 91 +++++-
tests/jobs/test_base_job.py | 2 +-
tests/plugins/test_plugin.py | 7 +
tests/plugins/test_plugins_manager.py | 47 +++
.../kubernetes/operators/test_kubernetes_pod.py | 34 +-
.../hooks => cncf/kubernetes/utils}/__init__.py | 0
.../cncf/kubernetes/utils}/test_pod_launcher.py | 31 +-
.../docker/operators/test_docker_swarm.py | 8 +-
...{test_presto_to_gcs.py => test_trino_to_gcs.py} | 72 ++---
...o_gcs_system.py => test_trino_to_gcs_system.py} | 24 +-
.../_internal_client/test_vault_client.py | 4 +-
tests/providers/hashicorp/hooks/test_vault.py | 6 +-
.../providers/microsoft/winrm/hooks/test_winrm.py | 2 +-
...t_presto_to_mysql.py => test_trino_to_mysql.py} | 36 +--
tests/providers/presto/hooks/test_presto.py | 25 --
.../experimental => providers/trino}/__init__.py | 0
.../trino/hooks}/__init__.py | 0
.../test_presto.py => trino/hooks/test_trino.py} | 54 ++--
tests/serialization/test_dag_serialization.py | 9 +
.../task/task_runner/test_standard_task_runner.py | 63 +++-
tests/www/test_utils.py | 4 +-
246 files changed, 3554 insertions(+), 1735 deletions(-)
copy airflow/kubernetes/{pod_launcher.py => pod_launcher_deprecated.py} (94%)
copy airflow/migrations/versions/{b247b1e3d1ed_add_queued_by_job_id_to_ti.py => 90d1635d7b86_increase_pool_name_size_in_taskinstance.py} (70%)
copy {tests/providers/zendesk/hooks => airflow/providers/cncf/kubernetes/utils}/__init__.py (100%)
copy airflow/{kubernetes => providers/cncf/kubernetes/utils}/pod_launcher.py (96%)
copy airflow/providers/google/cloud/example_dags/{example_presto_to_gcs.py => example_trino_to_gcs.py} (73%)
copy airflow/providers/google/cloud/transfers/{presto_to_gcs.py => trino_to_gcs.py} (82%)
copy airflow/providers/mysql/transfers/{presto_to_mysql.py => trino_to_mysql.py} (81%)
copy airflow/providers/{tableau => trino}/CHANGELOG.rst (100%)
copy {tests/www/api/experimental => airflow/providers/trino}/__init__.py (100%)
copy {tests/www/api/experimental => airflow/providers/trino/hooks}/__init__.py (100%)
copy airflow/providers/{presto/hooks/presto.py => trino/hooks/trino.py} (81%)
copy airflow/providers/{openfaas => trino}/provider.yaml (71%)
copy docs/apache-airflow-providers-google/operators/transfer/{presto_to_gcs.rst => trino_to_gcs.rst} (81%)
copy docs/{apache-airflow-providers-airbyte => apache-airflow-providers-trino}/commits.rst (76%)
copy docs/{apache-airflow-providers-redis => apache-airflow-providers-trino}/index.rst (86%)
create mode 100755 docs/exts/docs_build/run_patched_sphinx.py
create mode 100644 docs/integration-logos/trino/trino-og.png
copy licenses/{LICENSE-d3js.txt => LICENSE-d3-shape.txt} (97%)
copy scripts/ci/{images/ci_wait_for_and_verify_all_ci_images.sh => constraints/ci_generate_all_constraints.sh} (75%)
rename scripts/ci/docker-compose/{integration-presto.yml => integration-trino.yml} (81%)
rename scripts/ci/dockerfiles/{presto => trino}/Dockerfile (78%)
rename scripts/ci/dockerfiles/{presto => trino}/build_and_push.sh (79%)
rename scripts/ci/dockerfiles/{presto => trino}/entrypoint.sh (73%)
copy scripts/ci/{images/ci_wait_for_and_verify_ci_image.sh => kubernetes/ci_setup_cluster_and_run_kubernetes_tests_single_job.sh} (52%)
create mode 100755 scripts/ci/kubernetes/ci_setup_clusters_and_run_kubernetes_tests_in_parallel.sh
create mode 100644 scripts/ci/libraries/_testing.sh
create mode 100755 scripts/ci/testing/ci_run_quarantined_tests.sh
copy tests/{kubernetes/basic_pod.yaml => executors/kubernetes_executor_template_files/basic_template.yaml} (75%)
copy tests/providers/{zendesk/hooks => cncf/kubernetes/utils}/__init__.py (100%)
rename tests/{kubernetes => providers/cncf/kubernetes/utils}/test_pod_launcher.py (86%)
copy tests/providers/google/cloud/transfers/{test_presto_to_gcs.py => test_trino_to_gcs.py} (83%)
copy tests/providers/google/cloud/transfers/{test_presto_to_gcs_system.py => test_trino_to_gcs_system.py} (88%)
copy tests/providers/mysql/transfers/{test_presto_to_mysql.py => test_trino_to_mysql.py} (60%)
copy tests/{www/api/experimental => providers/trino}/__init__.py (100%)
copy tests/{www/api/experimental => providers/trino/hooks}/__init__.py (100%)
copy tests/providers/{presto/hooks/test_presto.py => trino/hooks/test_trino.py} (82%)