You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by gi...@apache.org on 2021/01/03 03:23:44 UTC
[airflow] tag nightly-master updated (6f246b0 -> d6e1d7b)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/airflow.git.
*** WARNING: tag nightly-master was modified! ***
from 6f246b0 (commit)
to d6e1d7b (commit)
from 6f246b0 Fix imagePullPolicy missing in tests (#13316)
add 6d6fe36 Updates IMAGES documentation to reflect Airflow 2.0 changes (#13312)
add 91acdbe Add pre-commit hook limiting hook name length (#13319)
add 8fed541 Rename PIP_VERSION to AIRFLOW_PIP_VERSION (#13320)
add f4745c8 Fix typo in example (#13321)
add c6d66cd Switch to Apache-owned GitHub actions (#13327)
add 98896e4 Also add codecov action to apache airflow repo (#13328)
add d41c6a4 Vastly improves usability of CI logs (#13323)
add 641f63c Prefer newer CLI syntax over legacy in helm chart (#13330)
add 4be27af Fix the behavior for deactivate the authentication option and documenting the process to do it (#13191)
add 09c6549 Add missing sqlite provider for production image (#13332)
add 1f71221 Support google-cloud-tasks>=2.0.0 (#13334)
add e35bdb9 json-merge-patch becomes optional library and has looser restrictions (#13175)
add 3b4290d Re-enables verification of production image (#13329)
add f7d354d Print better error message when tests fail (#13339)
add 04ec45f Add DataprocCreateWorkflowTemplateOperator (#13338)
add 13a9747 Revert "Support google-cloud-tasks>=2.0.0 (#13334)" (#13341)
add d23ac9b Adds missing LDAP "extra" dependencies to ldap provider. (#13308)
add 0d21457 Refactored setup.py to better reflect changes in providers (#13314)
add 82fa048 Production image can also be upgraded to newer dependencies (#13345)
add bafd258 Add Fleek Fashion to the list of Airflow users (#13372)
add 295d66f Fix Grammar in PIP warning (#13380)
add a4a3d3f Minor enhancements to Sensors docs (#13381)
add d5cf993 Fix typo in Open API docs (#13374)
add 1b94346 Bugfix: Sync Access Control defined in DAGs when running sync-perm (#13377)
add 57cbcf6 Fix broken link in PR Welcome message (#13386)
add 98f097e Add integration tests for Apache Pinot (#13195)
add 410ab89 Fix Apache Airflow icon link in Helm Chart (#13387)
add 8212247 Bump version to 2.1.0dev0 (#13382)
add fe45f1b Use 2.0.0 in Airflow docs & Breeze (#13379)
add d079b91 Disable persisting credentials in Github Action's checkout (#13389)
add b52d39f Improves documentation regarding providers and custom connections (#13375)
add dcedb81 Fix mallformed table in production-deployment.rst (#13395)
add 10be375 Simplify CeleryKubernetesExecutor tests (#13307)
add 406181d Add Parquet data type to BaseSQLToGCSOperator (#13359)
add cc9a19d Update celery.rst to fix broken links. (#13400)
add bd74eb0 Allow Tags on AWS Batch Job Submission (#13396)
add 85ac03f Update persists-credentials (#13401)
add a1f4938 Limit old versions of pinotdb to force update on CI (#13402)
add 028d8e8 Remove reference to scheduler run_duration param in docs (#13346)
add 07e7513 Fixed broken aws test_batch_job tests introduced by #13396 (#13406)
add d2964b0 Add example DAG & how-to guide for sqlite (#13196)
add 9de7127 Support google-cloud-bigquery-datatransfer>=3.0.0 (#13337)
add 7e1d28b Warns politely, do not force run a long operation (#13313)
add 6700563 Set minimum SQLite version supported. (#13412)
add d202fd4 Adding documentation explaining "strange" URI required when using AWS… (#13355)
add f6a3c82 Enable interpretation of backslash escapes for colored message (#13418)
add 181d8b6 Developers Quick Guide (#13417)
add 57143d6 Allow ./run_tmux.sh script to run standalone (#13420)
add abf34b8 Kuba openfaas sync call (#13356)
add 43f150b Add last-commit example to static-check --help message. (#13411)
add 27c757d Fix environment checking for Apache Pinot (#13419)
add 09a2413 Improves documentation regarding providers and custom connections 2 (#13410)
add ae625b4 Change timeout s and disables reverse IP lookup for integrations (#13424)
add c674f81 Add verbose flag to ./build_docs.py (#13403)
add e436883 Removes pip download when installing from local packages (#13422)
add f7a1334 Add 'mongo_collection' to template_fields in MongoToS3Operator (#13361)
add 6e1a6ff Fixed failing pylint errors introduced in #13403 (#13429)
add f6518dd Generalize MLEngineStartTrainingJobOperator to custom images (#13318)
add 1016584 Fix pylint issues - broken master (#13427)
add 1fe83a4 Fix selective checks for changes outside of airflow .py files (#13430)
add 0909ddf Adds timeout to all curl commands (#13431)
add 3a73110 Add extras when installing prod image from packages (#13432)
add ac730c3 Improve style of code block on aws-ssm-parameter-store.rst (#13428)
add abcb087 fixup! Adds timeout to all curl commands (#13431) (#13435)
add ad0d3e1 fixup! Fixed failing pylint errors introduced in #13403 (#13429) (#13437)
add c33d2c0 Fix another pylint c-extension-no-member (#13438)
add bc6f5ea Install airflow and providers together from context files (#13441)
add 006107f Fix grammar in API docs (#13444)
add d6e1d7b Replace deprecated decorator (#13443)
No new revisions were added by this update.
Summary of changes:
.github/boring-cyborg.yml | 2 +-
.github/workflows/build-images-workflow-run.yml | 33 +-
.github/workflows/ci.yml | 106 ++-
.github/workflows/codeql-analysis.yml | 2 +
.../workflows/label_when_reviewed_workflow_run.yml | 19 +-
.github/workflows/repo_sync.yml | 4 +-
.github/workflows/scheduled_quarantined.yml | 4 +-
.pre-commit-config.yaml | 28 +-
BREEZE.rst | 53 +-
CI.rst | 2 +-
CONTRIBUTING.rst | 30 +-
CONTRIBUTORS_QUICK_START.rst | 972 +++++++++++++++++++++
Dockerfile | 88 +-
Dockerfile.ci | 59 +-
IMAGES.rst | 132 +--
INSTALL | 20 +-
INTHEWILD.md | 1 +
LOCAL_VIRTUALENV.rst | 4 +-
PULL_REQUEST_WORKFLOW.rst | 20 +-
README.md | 16 +-
STATIC_CODE_CHECKS.rst | 2 +
TESTING.rst | 2 +
UPDATING.md | 2 +-
airflow/api_connexion/openapi/v1.yaml | 8 +-
airflow/cli/commands/sync_perm_command.py | 4 +-
airflow/providers/amazon/README.md | 2 +-
airflow/providers/amazon/aws/hooks/batch_client.py | 4 +
airflow/providers/amazon/aws/operators/batch.py | 7 +
.../providers/amazon/aws/transfers/mongo_to_s3.py | 2 +-
airflow/providers/apache/cassandra/README.md | 2 +-
airflow/providers/apache/druid/README.md | 2 +-
airflow/providers/apache/hdfs/README.md | 2 +-
airflow/providers/apache/hive/README.md | 2 +-
airflow/providers/apache/kylin/README.md | 2 +-
airflow/providers/apache/livy/README.md | 2 +-
airflow/providers/apache/pig/README.md | 2 +-
airflow/providers/apache/pinot/README.md | 2 +-
airflow/providers/apache/spark/README.md | 2 +-
airflow/providers/apache/sqoop/README.md | 2 +-
airflow/providers/celery/README.md | 2 +-
airflow/providers/cloudant/README.md | 2 +-
airflow/providers/cncf/kubernetes/README.md | 2 +-
airflow/providers/databricks/README.md | 2 +-
airflow/providers/datadog/README.md | 2 +-
airflow/providers/dingding/README.md | 2 +-
airflow/providers/discord/README.md | 2 +-
airflow/providers/docker/README.md | 2 +-
airflow/providers/elasticsearch/README.md | 2 +-
airflow/providers/exasol/README.md | 2 +-
airflow/providers/facebook/README.md | 2 +-
airflow/providers/ftp/README.md | 2 +-
airflow/providers/google/ADDITIONAL_INFO.md | 1 +
airflow/providers/google/README.md | 2 +-
.../cloud/example_dags/example_bigquery_dts.py | 20 +-
.../cloud/example_dags/example_bigquery_queries.py | 2 +-
.../google/cloud/example_dags/example_dataproc.py | 32 +-
.../providers/google/cloud/hooks/bigquery_dts.py | 45 +-
.../google/cloud/operators/bigquery_dts.py | 12 +-
.../providers/google/cloud/operators/dataproc.py | 67 +-
.../providers/google/cloud/operators/mlengine.py | 92 +-
.../providers/google/cloud/sensors/bigquery_dts.py | 35 +-
.../google/cloud/transfers/oracle_to_gcs.py | 2 +-
.../providers/google/cloud/transfers/sql_to_gcs.py | 40 +-
airflow/providers/grpc/README.md | 2 +-
airflow/providers/hashicorp/README.md | 2 +-
airflow/providers/http/README.md | 2 +-
airflow/providers/imap/README.md | 2 +-
airflow/providers/jdbc/README.md | 2 +-
airflow/providers/jenkins/README.md | 2 +-
airflow/providers/jira/README.md | 2 +-
airflow/providers/microsoft/azure/README.md | 2 +-
airflow/providers/microsoft/mssql/README.md | 2 +-
airflow/providers/microsoft/winrm/README.md | 2 +-
airflow/providers/mongo/README.md | 2 +-
.../{papermill => mysql}/ADDITIONAL_INFO.md | 7 +-
airflow/providers/mysql/README.md | 2 +-
airflow/providers/odbc/README.md | 2 +-
airflow/providers/openfaas/README.md | 2 +-
airflow/providers/openfaas/hooks/openfaas.py | 18 +-
airflow/providers/opsgenie/README.md | 2 +-
airflow/providers/oracle/README.md | 2 +-
airflow/providers/pagerduty/README.md | 2 +-
airflow/providers/papermill/README.md | 2 +-
airflow/providers/plexus/README.md | 2 +-
airflow/providers/postgres/README.md | 2 +-
airflow/providers/presto/README.md | 2 +-
airflow/providers/qubole/README.md | 2 +-
airflow/providers/redis/README.md | 2 +-
airflow/providers/salesforce/README.md | 2 +-
airflow/providers/samba/README.md | 2 +-
airflow/providers/segment/README.md | 2 +-
airflow/providers/sendgrid/README.md | 2 +-
airflow/providers/sftp/README.md | 2 +-
airflow/providers/singularity/README.md | 2 +-
airflow/providers/slack/README.md | 2 +-
airflow/providers/snowflake/README.md | 2 +-
airflow/providers/sqlite/README.md | 2 +-
.../providers/sqlite/example_dags}/__init__.py | 0
.../sqlite/example_dags/example_sqlite.py | 70 ++
airflow/providers/sqlite/operators/sqlite.py | 12 +-
airflow/providers/sqlite/provider.yaml | 3 +
airflow/providers/ssh/README.md | 2 +-
airflow/providers/telegram/README.md | 2 +-
airflow/providers/vertica/README.md | 2 +-
airflow/providers/yandex/README.md | 2 +-
airflow/providers/zendesk/README.md | 2 +-
airflow/utils/log/logging_mixin.py | 3 +-
...ed.sh => ask_for_recompile_assets_if_needed.sh} | 14 +-
airflow/www/compile_assets.sh | 5 +
airflow/www/security.py | 4 +-
breeze | 29 +-
breeze-complete | 6 +-
chart/Chart.yaml | 2 +-
chart/templates/create-user-job.yaml | 2 +-
chart/templates/flower/flower-deployment.yaml | 2 +-
chart/templates/migrate-database-job.yaml | 2 +-
chart/templates/workers/worker-deployment.yaml | 2 +-
chart/values.yaml | 5 +-
.../PROVIDER_README_TEMPLATE.md.jinja2 | 2 +-
dev/provider_packages/prepare_provider_packages.py | 73 +-
dev/remove_artifacts.sh | 3 +-
.../secrets-backends/aws-ssm-parameter-store.rst | 12 +
.../operators/cloud/dataproc.rst | 24 +
.../connections/sqlite.rst} | 21 +-
docs/apache-airflow-providers-sqlite/index.rst | 13 +
docs/apache-airflow-providers-sqlite/operators.rst | 75 ++
docs/apache-airflow-providers-sqlite/redirects.txt | 1 +
docs/apache-airflow-providers/index.rst | 90 +-
docs/apache-airflow/concepts.rst | 70 +-
docs/apache-airflow/executor/celery.rst | 4 +-
docs/apache-airflow/extra-packages-ref.rst | 409 ++++-----
docs/apache-airflow/howto/connection.rst | 21 +-
docs/apache-airflow/installation.rst | 18 +-
docs/apache-airflow/production-deployment.rst | 127 +--
docs/apache-airflow/security/webserver.rst | 8 +
docs/apache-airflow/start.rst | 2 +-
docs/apache-airflow/upgrade-check.rst | 2 +-
docs/build_docs.py | 18 +-
docs/exts/docs_build/docs_builder.py | 27 +-
images/quick_start/add Interpreter.png | Bin 0 -> 118539 bytes
images/quick_start/add_configuration.png | Bin 0 -> 97805 bytes
images/quick_start/add_env_variable.png | Bin 0 -> 86158 bytes
images/quick_start/airflow_clone.png | Bin 0 -> 175230 bytes
images/quick_start/airflow_fork.png | Bin 0 -> 44861 bytes
images/quick_start/ci_tests.png | Bin 0 -> 74167 bytes
images/quick_start/click_on_clone.png | Bin 0 -> 28944 bytes
images/quick_start/creating_branch_1.png | Bin 0 -> 32659 bytes
images/quick_start/creating_branch_2.png | Bin 0 -> 17802 bytes
images/quick_start/local_airflow.png | Bin 0 -> 21976 bytes
images/quick_start/mysql_connection.png | Bin 0 -> 99933 bytes
images/quick_start/pr1.png | Bin 0 -> 59941 bytes
images/quick_start/pr2.png | Bin 0 -> 57712 bytes
images/quick_start/pr3.png | Bin 0 -> 163389 bytes
images/quick_start/pycharm_clone.png | Bin 0 -> 25074 bytes
images/quick_start/start_airflow_tmux.png | Bin 0 -> 212341 bytes
.../ci/build_airflow/ci_build_airflow_package.sh | 5 +-
scripts/ci/constraints/ci_generate_constraints.sh | 2 +-
scripts/ci/docker-compose/_docker.env | 1 +
...integration-mongo.yml => integration-pinot.yml} | 15 +-
scripts/ci/docs/ci_docs.sh | 2 +-
scripts/ci/images/ci_build_dockerhub.sh | 5 +-
scripts/ci/images/ci_prepare_ci_image_on_ci.sh | 4 +-
scripts/ci/images/ci_prepare_prod_image_on_ci.sh | 5 +-
scripts/ci/images/ci_verify_ci_image.sh | 13 +-
scripts/ci/images/ci_verify_prod_image.sh | 154 +++-
scripts/ci/images/ci_wait_for_ci_image.sh | 6 +-
scripts/ci/images/ci_wait_for_prod_image.sh | 7 +-
scripts/ci/kubernetes/ci_run_kubernetes_tests.sh | 121 +--
...tup_cluster_and_deploy_airflow_to_kubernetes.sh | 2 +-
scripts/ci/libraries/_build_airflow_packages.sh | 4 +-
scripts/ci/libraries/_build_images.sh | 68 +-
scripts/ci/libraries/_initialization.sh | 20 +-
scripts/ci/libraries/_kind.sh | 51 +-
scripts/ci/libraries/_md5sum.sh | 5 +
scripts/ci/libraries/_permissions.sh | 10 +-
scripts/ci/libraries/_push_pull_remove_images.sh | 9 +-
scripts/ci/libraries/_runs.sh | 11 +-
scripts/ci/libraries/_sanity_checks.sh | 6 +-
scripts/ci/libraries/_script_init.sh | 14 +
scripts/ci/libraries/_start_end.sh | 27 +
.../pre_commit_check_extras_have_providers.py | 69 +-
.../ci/pre_commit/pre_commit_check_order_setup.py | 129 ++-
.../pre_commit_check_pre_commit_hook_names.py | 61 ++
.../pre_commit_check_setup_extra_packages_ref.py | 273 ++++--
scripts/ci/pre_commit/pre_commit_mermaid.sh | 2 +-
.../ci_install_and_test_provider_packages.sh | 3 +-
.../ci_prepare_provider_packages.sh | 4 +-
.../ci_prepare_provider_readmes.sh | 4 +-
scripts/ci/selective_ci_checks.sh | 88 +-
scripts/ci/testing/ci_run_airflow_testing.sh | 90 +-
scripts/in_container/_in_container_utils.sh | 36 +-
scripts/in_container/check_environment.sh | 50 +-
scripts/in_container/entrypoint_ci.sh | 14 +-
scripts/in_container/prod/entrypoint_prod.sh | 20 +-
scripts/in_container/run_generate_constraints.sh | 2 +-
.../run_install_and_test_provider_packages.sh | 164 ++--
.../in_container/run_prepare_provider_packages.sh | 255 +++---
.../in_container/run_prepare_provider_readme.sh | 5 +-
scripts/in_container/run_tmux.sh | 68 +-
scripts/in_container/run_tmux_welcome.sh | 2 +-
setup.cfg | 10 +-
setup.py | 554 +++++-------
tests/always/test_project_structure.py | 1 -
tests/build_provider_packages_dependencies.py | 4 +-
tests/cli/commands/test_sync_perm_command.py | 25 +-
tests/executors/test_celery_kubernetes_executor.py | 265 +++---
tests/providers/amazon/aws/operators/test_batch.py | 4 +
.../amazon/aws/transfers/test_mongo_to_s3.py | 2 +-
tests/providers/apache/pinot/hooks/test_pinot.py | 12 +
.../google/cloud/hooks/test_bigquery_dts.py | 39 +-
.../google/cloud/operators/test_bigquery_dts.py | 37 +-
.../google/cloud/operators/test_dataproc.py | 39 +
.../google/cloud/operators/test_mlengine.py | 48 +-
.../google/cloud/sensors/test_bigquery_dts.py | 39 +-
.../google/cloud/transfers/test_oracle_to_gcs.py | 2 +-
.../google/cloud/transfers/test_sql_to_gcs.py | 110 ++-
tests/providers/openfaas/hooks/test_openfaas.py | 28 +
tests/www/test_security.py | 49 ++
218 files changed, 4440 insertions(+), 2036 deletions(-)
create mode 100644 CONTRIBUTORS_QUICK_START.rst
copy airflow/providers/{papermill => mysql}/ADDITIONAL_INFO.md (76%)
copy {tests/www/api/experimental => airflow/providers/sqlite/example_dags}/__init__.py (100%)
create mode 100644 airflow/providers/sqlite/example_dags/example_sqlite.py
rename airflow/www/{compile_assets_if_needed.sh => ask_for_recompile_assets_if_needed.sh} (79%)
copy docs/{apache-airflow-providers-amazon/logging/index.rst => apache-airflow-providers-sqlite/connections/sqlite.rst} (68%)
create mode 100644 docs/apache-airflow-providers-sqlite/operators.rst
create mode 100644 docs/apache-airflow-providers-sqlite/redirects.txt
create mode 100644 images/quick_start/add Interpreter.png
create mode 100644 images/quick_start/add_configuration.png
create mode 100644 images/quick_start/add_env_variable.png
create mode 100644 images/quick_start/airflow_clone.png
create mode 100644 images/quick_start/airflow_fork.png
create mode 100644 images/quick_start/ci_tests.png
create mode 100644 images/quick_start/click_on_clone.png
create mode 100644 images/quick_start/creating_branch_1.png
create mode 100644 images/quick_start/creating_branch_2.png
create mode 100644 images/quick_start/local_airflow.png
create mode 100644 images/quick_start/mysql_connection.png
create mode 100644 images/quick_start/pr1.png
create mode 100644 images/quick_start/pr2.png
create mode 100644 images/quick_start/pr3.png
create mode 100644 images/quick_start/pycharm_clone.png
create mode 100644 images/quick_start/start_airflow_tmux.png
copy scripts/ci/docker-compose/{integration-mongo.yml => integration-pinot.yml} (85%)
create mode 100755 scripts/ci/pre_commit/pre_commit_check_pre_commit_hook_names.py