You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ur...@apache.org on 2021/08/16 08:03:16 UTC
[airflow] branch aip-39-docs updated (69d4725 -> cfcd786)
This is an automated email from the ASF dual-hosted git repository.
uranusjr pushed a change to branch aip-39-docs
in repository https://gitbox.apache.org/repos/asf/airflow.git.
discard 69d4725 WIP
add 719709b Skip PV's lost+found dir when cleaning logs (k8s) (#17547)
add 8bd748d Fix spelling in ``CeleryExecutor`` docs (#17553)
add b06d528 Don't cache Google Secret Manager client (#17539)
add ec5b4fe AIP-40: Add Deferrable "Async" Tasks (#15389)
add 3773224 Chart: Allow podTemplate to be templated (#17560)
add a1834ce Use dag_maker fixture in test_processor.py (#17506)
add 944dc32 Fix Invalid log order in ElasticsearchTaskHandler (#17551)
add 951006e Path correction in docs for airflow core (#17567)
add 4556828 AIP-39: DagRun.data_interval_start|end (#16352)
add 4598eb0 Quarantine test_mark_success_no_kill test (#17580)
add 9875757 Add Mongo projections to hook and transfer (#17379)
add b9109f8 Chart: Support ``extraContainers`` in k8s workers (#17562)
add e7eeaa6 Doc: Fix docstrings for ``MongoToS3Operator`` (#17588)
add 2c26b15 Make `pandas` an optional core dependency (#17575)
add 8b03d9d Fixed long delays in the Temporal trigger (#17564)
add 113bfdf Document overriding ``XCom.clear`` for data lifecycle management (#17589)
add e69c6a7 fix: filter condition of TaskInstance does not work #17535 (#17548)
add 60ddcd1 Rescue if a DagRun's DAG was removed from db (#17544)
add 7b4ce7b Run more than just basic checks on ``.pre-commit-config.yaml`` change (#17590)
add 2092988 Simplify 404 page (#17501)
add 57d60d2 Bump `mysql-connector-python` to latest version (#17596)
add 6b327c1 Remove redundant ``numpy`` dependency (#17594)
add f2a8a73 Remove redundant note on HTTP Provider (#17595)
add ba3d5b3 Add more ``project_urls`` for PyPI (#17598)
add d11d3e6 Use built-in ``cached_property`` on Python 3.8 for Asana provider (#17597)
add 7db43f7 Remove upper-limit on ``tenacity`` (#17593)
add 833e109 Improve validation of Group id (#17578)
add 88e3379 Fix MySQL database character set instruction (#17603)
add 4eff7a4 Simplify bug report template (#17561)
add 55b9b70 Fix ``triggerer`` query where limit is not supported in some MySQL version (#17601)
add bb3b319 Chart: Fix elasticsearch-secret template port default function (#17428)
add b7e5a24 Add cherry-pick notes to the release README (#17610)
add 721d4e7 Update documentation regarding Python 3.9 support (#17611)
add 1632c9f Add new LocalFilesystemToS3Operator under Amazon provider (#17168) (#17382)
add 6df3ee7 Fix redacting secrets in context exceptions. (#17618)
add bd4f0c3 Use gunicorn to serve logs generated by worker (#17591)
add 9b3ed1f Move worker_log_server_port option to the logging section (#17621)
add a40fe81 Switch Alibaba OSS tests to us-east-1 (#17616)
add efd4eb3 Fix spelling of Airflow (#17624)
add cff5f18 Improve cross-links to operators and hooks references (#17622)
add c96b152 Add Triggerer warning banner in UI (#17565)
add b5f7e9e Force emulation on ARM platforms for Docker (#17627)
add 934b76b Enhanced configure_environment.sh declared readonly varaible (#17619)
add cfcd786 WIP
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (69d4725)
\
N -- N -- N refs/heads/aip-39-docs (cfcd786)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/ISSUE_TEMPLATE/bug_report.md | 80 ++--
.pre-commit-config.yaml | 3 +-
BREEZE.rst | 12 +-
CONTRIBUTING.rst | 4 +-
Dockerfile | 2 +-
INSTALL | 4 +-
README.md | 2 +-
UPDATING.md | 17 +
airflow/api/common/experimental/mark_tasks.py | 4 +-
.../api_connexion/endpoints/dag_run_endpoint.py | 19 +-
airflow/api_connexion/openapi/v1.yaml | 2 +
airflow/cli/cli_parser.py | 20 +
airflow/cli/commands/standalone_command.py | 10 +-
.../{kerberos_command.py => triggerer_command.py} | 28 +-
.../hooks/test_sqs.py => airflow/compat/asyncio.py | 22 +-
airflow/config_templates/config.yml | 39 +-
airflow/config_templates/default_airflow.cfg | 21 +-
airflow/config_templates/default_test.cfg | 2 +-
airflow/configuration.py | 1 +
airflow/exceptions.py | 34 +-
airflow/executors/celery_executor.py | 6 +-
airflow/hooks/dbapi.py | 5 +-
airflow/jobs/backfill_job.py | 47 ++-
airflow/jobs/scheduler_job.py | 43 +-
airflow/jobs/triggerer_job.py | 393 ++++++++++++++++++
...a_interval_start_end_to_dagmodel_and_dagrun.py} | 59 +--
...54bebd308c5f_add_trigger_table_and_task_info.py | 69 ++++
airflow/models/__init__.py | 1 +
airflow/models/baseoperator.py | 27 +-
airflow/models/dag.py | 168 ++++++--
airflow/models/dagrun.py | 23 +-
airflow/models/taskinstance.py | 455 ++++++++++++++++-----
airflow/models/trigger.py | 193 +++++++++
airflow/operators/python.py | 7 +-
airflow/operators/subdag.py | 8 +-
...xample_s3_to_sftp.py => example_local_to_s3.py} | 27 +-
.../s3_copy_object.py => transfers/local_to_s3.py} | 81 ++--
.../providers/amazon/aws/transfers/mongo_to_s3.py | 7 +
airflow/providers/asana/hooks/asana.py | 6 +-
.../providers/elasticsearch/log/es_task_handler.py | 8 +-
.../google/cloud/secrets/secret_manager.py | 9 +-
airflow/providers/mongo/hooks/mongo.py | 29 +-
airflow/sensors/date_time.py | 20 +
airflow/sensors/time_delta.py | 21 +
airflow/sensors/time_sensor.py | 22 +
airflow/settings.py | 1 +
airflow/timetables/base.py | 56 ++-
airflow/timetables/interval.py | 9 +-
airflow/timetables/simple.py | 8 +-
.../zendesk/hooks => airflow/triggers}/__init__.py | 0
airflow/triggers/base.py | 98 +++++
airflow/triggers/temporal.py | 76 ++++
airflow/triggers/testing.py | 52 +++
airflow/utils/file.py | 2 +
airflow/utils/helpers.py | 23 +-
airflow/utils/json.py | 12 +-
airflow/utils/log/file_task_handler.py | 2 +-
airflow/utils/log/secrets_masker.py | 10 +-
airflow/utils/serve_logs.py | 43 +-
airflow/utils/sqlalchemy.py | 47 ++-
airflow/utils/state.py | 8 +-
airflow/utils/task_group.py | 14 +-
airflow/www/extensions/init_views.py | 4 +-
airflow/www/static/css/graph.css | 4 +
airflow/www/static/css/main.css | 4 +
airflow/www/static/css/tree.css | 4 +
airflow/www/static/js/circles.js | 130 ------
airflow/www/static/js/dag.js | 2 +-
airflow/www/static/js/graph.js | 1 +
airflow/www/templates/airflow/main.html | 15 +
.../airflow/{circles.html => not_found.html} | 28 +-
airflow/www/utils.py | 25 +-
airflow/www/views.py | 19 +-
airflow/www/webpack.config.js | 1 -
chart/files/pod-template-file.kubernetes-helm-yaml | 3 +
chart/templates/configmaps/configmap.yaml | 2 +-
chart/templates/secrets/elasticsearch-secret.yaml | 2 +-
chart/tests/test_configmap.py | 19 +
chart/tests/test_elasticsearch_secret.py | 17 +
chart/tests/test_pod_template_file.py | 18 +
chart/values.schema.json | 6 +-
chart/values.yaml | 8 +-
dev/README_RELEASE_AIRFLOW.md | 20 +
dev/provider_packages/SETUP_TEMPLATE.py.jinja2 | 3 +
.../operators-and-hooks-ref/index.rst | 2 +
docs/apache-airflow/concepts/dags.rst | 4 -
docs/apache-airflow/concepts/deferring.rst | 163 ++++++++
docs/apache-airflow/concepts/index.rst | 1 +
docs/apache-airflow/concepts/operators.rst | 11 +-
docs/apache-airflow/concepts/smart-sensors.rst | 6 +
docs/apache-airflow/concepts/tasks.rst | 1 +
docs/apache-airflow/concepts/xcoms.rst | 2 +
docs/apache-airflow/dag-run.rst | 2 +-
docs/apache-airflow/executor/celery.rst | 2 +-
docs/apache-airflow/extra-packages-ref.rst | 2 +
.../howto/customize-state-colors-ui.rst | 1 +
docs/apache-airflow/howto/set-up-database.rst | 2 +-
docs/apache-airflow/howto/timetable.rst | 4 +-
docs/apache-airflow/installation.rst | 7 +-
.../logging-monitoring/logging-tasks.rst | 13 +
docs/apache-airflow/migrations-ref.rst | 6 +-
docs/apache-airflow/start/docker-compose.yaml | 4 +-
docs/apache-airflow/templates-ref.rst | 108 +++--
docs/helm-chart/using-additional-containers.rst | 9 +-
docs/spelling_wordlist.txt | 8 +
scripts/ci/libraries/_initialization.sh | 4 +
scripts/ci/selective_ci_checks.sh | 1 +
scripts/in_container/configure_environment.sh | 4 +-
scripts/in_container/prod/clean-logs.sh | 2 +-
setup.cfg | 11 +-
setup.py | 21 +-
.../endpoints/test_dag_run_endpoint.py | 5 +-
tests/cli/commands/test_dag_command.py | 28 +-
...ersion_command.py => test_triggerer_command.py} | 27 +-
tests/conftest.py | 82 ++++
tests/core/test_config_templates.py | 1 +
tests/core/test_core.py | 45 +-
tests/dag_processing/test_processor.py | 98 ++---
tests/decorators/test_python.py | 2 +
tests/jobs/test_backfill_job.py | 30 +-
tests/jobs/test_local_task_job.py | 1 +
tests/jobs/test_scheduler_job.py | 79 +++-
tests/jobs/test_triggerer_job.py | 341 +++++++++++++++
tests/models/test_dag.py | 110 +++--
tests/models/test_taskinstance.py | 192 ++++-----
tests/models/test_trigger.py | 123 ++++++
tests/operators/test_python.py | 2 +
tests/operators/test_subdag_operator.py | 50 ++-
tests/providers/alibaba/cloud/hooks/test_oss.py | 2 +-
.../providers/alibaba/cloud/operators/test_oss.py | 2 +-
.../alibaba/cloud/sensors/test_oss_key.py | 2 +-
.../amazon/aws/transfers/test_local_to_s3.py | 93 +++++
.../amazon/aws/transfers/test_mongo_to_s3.py | 4 +-
.../elasticsearch/log/test_es_task_handler.py | 45 +-
tests/providers/mongo/hooks/test_mongo.py | 20 +-
tests/sensors/test_base.py | 5 +-
tests/sensors/test_python.py | 2 +
tests/test_utils/db.py | 4 +
.../perf/scheduler_dag_execution_timing.py | 2 +-
tests/timetables/test_time_table_iter_ranges.py | 38 --
tests/triggers/test_temporal.py | 84 ++++
tests/utils/log/test_secrets_masker.py | 76 ++++
tests/utils/test_helpers.py | 76 +++-
tests/utils/test_serve_logs.py | 4 +-
tests/www/views/test_views.py | 17 +
tests/www/views/test_views_base.py | 5 +-
tests/www/views/test_views_blocked.py | 84 ++++
tests/www/views/test_views_trigger_dag.py | 9 +-
148 files changed, 4004 insertions(+), 1043 deletions(-)
copy airflow/cli/commands/{kerberos_command.py => triggerer_command.py} (64%)
copy tests/providers/amazon/aws/hooks/test_sqs.py => airflow/compat/asyncio.py (68%)
create mode 100644 airflow/jobs/triggerer_job.py
copy airflow/migrations/versions/{e9304a3141f0_make_xcom_pkey_columns_non_nullable.py => 142555e44c17_add_data_interval_start_end_to_dagmodel_and_dagrun.py} (50%)
create mode 100644 airflow/migrations/versions/54bebd308c5f_add_trigger_table_and_task_info.py
create mode 100644 airflow/models/trigger.py
copy airflow/providers/amazon/aws/example_dags/{example_s3_to_sftp.py => example_local_to_s3.py} (64%)
copy airflow/providers/amazon/aws/{operators/s3_copy_object.py => transfers/local_to_s3.py} (54%)
copy {tests/providers/zendesk/hooks => airflow/triggers}/__init__.py (100%)
create mode 100644 airflow/triggers/base.py
create mode 100644 airflow/triggers/temporal.py
create mode 100644 airflow/triggers/testing.py
delete mode 100644 airflow/www/static/js/circles.js
rename airflow/www/templates/airflow/{circles.html => not_found.html} (61%)
create mode 100644 docs/apache-airflow/concepts/deferring.rst
copy tests/cli/commands/{test_version_command.py => test_triggerer_command.py} (60%)
create mode 100644 tests/jobs/test_triggerer_job.py
create mode 100644 tests/models/test_trigger.py
create mode 100644 tests/providers/amazon/aws/transfers/test_local_to_s3.py
delete mode 100644 tests/timetables/test_time_table_iter_ranges.py
create mode 100644 tests/triggers/test_temporal.py
create mode 100644 tests/www/views/test_views_blocked.py