You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2023/07/07 16:30:03 UTC
[airflow] branch v2-6-stable updated (d2f0d100da -> eb24742d53)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a change to branch v2-6-stable
in repository https://gitbox.apache.org/repos/asf/airflow.git
from d2f0d100da Update RELEASE_NOTES.rst
add 38968eafcf Improve workflow to release images (#31916)
add 8b53f06499 Fix print message in release_candidate_command.py (#31910)
add 8f4eae1c7a Update 2.6.2 release date (#31982)
add f949a1ed87 Add ™ to Airflow in prominent places (#31977)
add f0a6f9651c Add pkgconf to system dependencies for the CI/prod image (#32075)
add 2779be2cda Fix copy artifacts to svn command (#31975)
add 06e3640069 Add release management command that generates SHOM information (#31835)
add 55ab6a5586 Bump: kind 0.20.0 (#31922)
add d4a1212ade Add command to generate requirements that are needed by a provider (#31943)
add cc5c4ffe59 Remove casting of forward_credentials for shell and start-airflow breeze (#32087)
add 4b3af9d32c Exit asset compilation fork in start-airflow after it completed (#32114)
add d042f010d3 Avoid permission error when asset compilation exists before killing it (#32116)
add b19721907d Add release management command to retreive provider version metadata (#32119)
add bdda263423 Add airflow to docs packages if only docs/apache-airflow/* files changed (#32149)
add 5c38fa8b54 add mobuchowski to committers breeze variable (#32228)
add 2c8c00795c Add phanikumv to committers (#32229)
add 1e80b2e941 Add vincbeck as committer (#32240)
add b08f9c9d93 Add apache-airflow-providers to filter in selective checks if needed (#32284)
add 75cc3ab3e7 Update git-sync description in Helm Chart documentation (#32181)
add 1cc4c091f5 Separate out advanced logging configuration (#32131)
add 6bb0b7e74c Update references to restructured documentation from airflow core (#32282)
add 06a88536db Update security model of Airflow (#32098)
add 4e1c0bb638 Fix Pydantic 2 pickiness about model definition (#32307)
add 1bc6c99935 Limit Pydantic to < 2.0.0 until we solve 2.0.0 incompatibilities (#32312)
add 43afd2c9c3 Fix SemVer link in security documentation (#32320)
add 4d8886dcf1 Fix airflow module version check when using ExternalPythonOperator and debug logging level (#30367)
add 8561950887 fix issue: DAG's on_failure_callback is not invoked when task failed during testing dag. (#30965)
add 6f266d857a This fixes an issue were a database session got stuck (#31128)
add 187e79f136 fix connection uri parsing when the host includes a scheme (#31465)
add d21a599b9a Documentation update on Plugin updates. (#31781)
add ae3408c6eb Increase readability for prerequisites.rst (#31794)
add 309f2f9495 Add comments explaining need for special "trigger end" log message (#31812)
add 54f2d74d50 Add a check for trailing slash in webserver base_url (#31833)
add 347ab00f16 added difference between Deferrable and Non-Deferrable Operators (#31840)
add af5f7fe32b adding config changes to docker docs (#31842)
add 52a2fd3bf1 Decode old-style nested Xcom value (#31866)
add ad7f474eec Fix `operator_extra_links` property serialization in mapped tasks (#31904)
add 737f2114b7 Increase readability in start.rst (#31906)
add 2d400d497b Fix typos in ISSUE_TRIAGE_PROCESS.rst (#31919)
add ee6a78334a Fix number param html type in trigger template (#31946)
add cd85a8614b Fix task.sensor annotation in type stub (#31954)
add 497509e9b7 Fix masking nested variable fields (#31964)
add 352cfa8df1 Skip python version check in some pre-commit when loading setup.py (#31965)
add e991f6689d Fix number param html type in trigger template (#31980)
add c000015083 Fix ExternalTaskSensor when there is not task group TIs for the current execution date (#32009)
add ed2d3acedd Flash an error msg instead of failure in `rendered-templates` when map index is not found (#32011)
add c78e16588e Check for DAG ID in query param from url as well as kwargs (#32014)
add 23d117049a Return None if an XComArg fails to resolve in a multiple_outputs Task (#32027)
add 2813c41c1b Fix hashing of dag_dependencies in serialized dag (#32037)
add e0e8d71b0e Render list items in rendered fields view (#32042)
add fd7f63f196 Use re2 for matching untrusted regex (#32060)
add 3e8a2174cd Update comment for editable installs wrt setuptools (#32090)
add d6cc9e4bb1 Ensure that main triggerer thread exits if the async thread fails (#32092)
add a7a676eb08 Adding missing hyperlink to the tutorial documentation (#32105)
add 1f64d3c426 docs(authoring-and-scheduling/deferring): add a note that we'll need to restart triggerer to reflect any trigger change (#32140)
add 050b3856dc Update callbacks.rst (#32148)
add 028af0cafc Use mapIndex to display extra links per mapped task. (#32154)
add 3ed537c934 Documentation Changes (#32185)
add 600a9254b4 Add libeos_dev on ARM runtime PROD image (#32200)
add 7d0d67198e fix bugged singleton implementation (#32218)
add 14480751dd Remove executor_class from Job - fixing backfil for custom executors (#32219)
add 1a24e08d1f Add comment to warn off a potential wrong fix (#32230)
add 9c7e2897b7 Add links to DAGRun / DAG / Task in templates-ref.rst (#32245)
add 38df98bac8 Fix behaviour of LazyDictWithCache wheni import fails (#32248)
add 555d73c0a1 Add information for users who ask for requirements (#32262)
add 3e4c5e840a Clarify Listener API behavior (#32269)
add 288fb12e83 Update python.rst with multi-Scheduler LocalExecutor info (#32310)
add 9d7a7e7558 Update index.rst doc to correct grammar (#32315)
add bc7516b565 Retry transactions on occasional deadlocks for rendered fields (#32341)
add fb30cbdb94 Fix try_number shown in the `task?task_id=&dag_id=&execution_date=` (#32361)
add cd210181ee Update pipeline.rst (#32369)
add 3b51035b5b feat(jobs/triggerer_job_runner): add triggerer canceled log (#31757)
add 8ff7dfbd9e Sanitize `DagRun.run_id` and allow flexibility (#32293)
add 191daab7a4 Hide sensitive values from extra in connection edit form (#32309)
add 04a3a0ab27 Fix ts/eslint warnings (#32321)
add 43720fb18a Add information on ARM MySQL support in Docker images (#32401)
add f10204b59f Catch the exception that triggerer initialization failed (#31999)
add c92574f471 Less precise check for error message in trigger initialization (#32035)
add cf31c6f136 Fix triggerers alive check and add a new conf for triggerer heartbeat rate (#32123)
add fc61875ca3 Fix failing main due to new release of jsonschema (#32405)
add b329587988 Use linear time regular expressions (#32303)
add aa4cda365a Limit click until we fix mypy issues (#32413)
add fc4934e1c7 Use WebEncoder to encode DagRun.conf in DagRun's list view (#32385)
add 62859f0f23 Update trigger url to fix test
add d4afc4d801 fixup! feat(jobs/triggerer_job_runner): add triggerer canceled log (#31757)
add 00af083419 Add explanation about tag creation and cleanup (#32406)
add b550d650dd Update version to 2.6.3
add eb24742d53 Update RELEASE_NOTES.rst
No new revisions were added by this update.
Summary of changes:
.github/{SECURITY.rst => SECURITY.md} | 115 +-
.github/boring-cyborg.yml | 2 +-
.github/workflows/release_dockerhub_image.yml | 5 +
.pre-commit-config.yaml | 10 +-
BREEZE.rst | 52 +
Dockerfile | 4 +-
Dockerfile.ci | 4 +-
ISSUE_TRIAGE_PROCESS.rst | 4 +-
README.md | 30 +-
RELEASE_NOTES.rst | 66 +
STATIC_CODE_CHECKS.rst | 2 +
airflow/__init__.py | 5 +-
.../api_connexion/endpoints/provider_endpoint.py | 4 +-
airflow/api_connexion/openapi/v1.yaml | 2 +-
airflow/cli/commands/provider_command.py | 4 +-
airflow/cli/commands/triggerer_command.py | 3 +-
airflow/cli/commands/user_command.py | 4 +-
airflow/config_templates/config.yml | 16 +
airflow/config_templates/default_airflow.cfg | 8 +
airflow/config_templates/default_celery.py | 7 +-
airflow/configuration.py | 23 +-
airflow/decorators/__init__.pyi | 2 +-
airflow/decorators/base.py | 8 +-
airflow/executors/kubernetes_executor.py | 9 +-
airflow/jobs/backfill_job_runner.py | 5 +-
airflow/jobs/job.py | 3 -
airflow/jobs/triggerer_job_runner.py | 53 +-
airflow/kubernetes/pod_generator.py | 4 +-
airflow/kubernetes/pod_generator_deprecated.py | 5 +-
airflow/lineage/__init__.py | 17 +-
airflow/models/baseoperator.py | 8 +-
airflow/models/connection.py | 35 +-
airflow/models/dag.py | 88 +-
airflow/models/dagrun.py | 16 +-
airflow/models/renderedtifields.py | 1 +
airflow/models/taskinstance.py | 18 +-
airflow/models/trigger.py | 13 +-
airflow/models/variable.py | 2 +-
airflow/models/xcom_arg.py | 7 +
airflow/operators/python.py | 5 +-
airflow/providers_manager.py | 20 +-
airflow/security/utils.py | 5 +-
airflow/sensors/external_task.py | 13 +-
airflow/serialization/pydantic/dataset.py | 8 +-
airflow/serialization/serde.py | 15 +-
airflow/serialization/serialized_objects.py | 10 +-
airflow/utils/cli.py | 4 +-
airflow/utils/db.py | 1 +
airflow/utils/db_cleanup.py | 5 +-
airflow/utils/email.py | 5 +-
airflow/utils/file.py | 13 +-
airflow/utils/log/colored_log.py | 4 +-
airflow/utils/log/logging_mixin.py | 5 +-
airflow/utils/log/secrets_masker.py | 10 +-
.../{compat/functools.pyi => utils/singleton.py} | 17 +-
airflow/utils/task_group.py | 9 +-
airflow/www/alias-rest-types.js | 1 +
airflow/www/auth.py | 3 +-
airflow/www/babel.config.js | 2 +-
airflow/www/extensions/init_wsgi_middlewares.py | 8 +-
airflow/www/fab_security/manager.py | 8 +-
airflow/www/jest-setup.js | 2 +
airflow/www/package.json | 2 +-
airflow/www/static/js/api/useDatasets.ts | 2 +-
airflow/www/static/js/api/useExtraLinks.ts | 63 +-
airflow/www/static/js/callModal.js | 3 +
airflow/www/static/js/components/Table/Cells.tsx | 2 +
.../www/static/js/components/Table/Table.test.tsx | 2 +-
airflow/www/static/js/components/Table/index.tsx | 6 +-
airflow/www/static/js/components/Tooltip.tsx | 3 +-
airflow/www/static/js/connection_form.js | 4 +
airflow/www/static/js/dag/details/graph/utils.ts | 2 +-
.../js/dag/details/taskInstance/ExtraLinks.tsx | 10 +-
.../static/js/dag/details/taskInstance/index.tsx | 9 +
.../www/static/js/dag/grid/dagRuns/index.test.tsx | 1 +
airflow/www/static/js/dag/grid/index.test.tsx | 1 +
airflow/www/static/js/dag_dependencies.js | 2 +
airflow/www/static/js/dags.js | 1 +
airflow/www/static/js/datasets/List.test.tsx | 12 +-
airflow/www/static/js/datasets/List.tsx | 2 +
airflow/www/static/js/datetime_utils.js | 1 +
airflow/www/static/js/gantt.js | 13 +-
airflow/www/static/js/graph.js | 1 +
airflow/www/static/js/main.js | 19 +-
airflow/www/static/js/task.js | 2 +
airflow/www/static/js/task_instances.js | 2 +
airflow/www/static/js/ti_log.js | 9 +-
.../www/static/js/types/react-table-config.d.ts | 4 +-
.../www/static/js/utils/URLSearchParamWrapper.ts | 3 +-
airflow/www/static/js/utils/graph.ts | 14 +-
airflow/www/templates/airflow/trigger.html | 4 +-
airflow/www/utils.py | 35 +-
airflow/www/views.py | 127 +-
dev/README_RELEASE_AIRFLOW.md | 11 +-
dev/breeze/src/airflow_breeze/breeze.py | 2 +
.../src/airflow_breeze/commands/ci_commands.py | 3 +-
.../airflow_breeze/commands/developer_commands.py | 4 +-
.../commands/developer_commands_config.py | 2 +-
.../commands/release_candidate_command.py | 2 +-
.../src/airflow_breeze/commands/release_command.py | 20 +-
.../commands/release_management_commands.py | 78 +-
.../commands/release_management_commands_config.py | 3 +
.../src/airflow_breeze/commands/sbom_commands.py | 272 ++
.../commands/sbom_commands_config.py | 61 +
.../src/airflow_breeze/commands/setup_commands.py | 6 +-
.../src/airflow_breeze/configure_rich_click.py | 24 +-
dev/breeze/src/airflow_breeze/global_constants.py | 20 +-
dev/breeze/src/airflow_breeze/pre_commit_ids.py | 1 +
dev/breeze/src/airflow_breeze/utils/cdxgen.py | 282 ++
.../src/airflow_breeze/utils/common_options.py | 8 +
.../airflow_breeze/utils/docker_command_utils.py | 3 +
dev/breeze/src/airflow_breeze/utils/github.py | 156 +
.../src/airflow_breeze/utils/github_actions.py | 30 -
dev/breeze/src/airflow_breeze/utils/parallel.py | 6 +
dev/breeze/src/airflow_breeze/utils/path_utils.py | 6 +-
.../airflow_breeze/utils/provider_dependencies.py | 32 +-
dev/breeze/src/airflow_breeze/utils/run_utils.py | 15 +-
.../src/airflow_breeze/utils/selective_checks.py | 9 +-
dev/breeze/tests/test_selective_checks.py | 30 +
dev/images/release_prod_image.png | Bin 76309 -> 309833 bytes
dev/images/release_prod_image_rc.png | Bin 65816 -> 298821 bytes
.../logging/stackdriver.rst | 3 +-
.../core-extensions/auth-backends.rst | 2 +-
.../core-extensions/secrets-backends.rst | 2 +-
.../administration-and-deployment/index.rst | 1 -
.../administration-and-deployment/kubernetes.rst | 4 +-
.../administration-and-deployment/listeners.rst | 2 +-
.../advanced-logging-configuration.rst | 91 +
.../logging-monitoring/callbacks.rst | 2 +-
.../logging-monitoring/index.rst | 1 +
.../logging-monitoring/logging-architecture.rst | 10 +-
.../logging-monitoring/logging-tasks.rst | 53 +-
.../administration-and-deployment/scheduler.rst | 1 +
.../security/index.rst | 28 -
.../authoring-and-scheduling/deferring.rst | 23 +
.../authoring-and-scheduling/plugins.rst | 2 +-
.../core-concepts/executor/local.rst | 8 +
docs/apache-airflow/core-concepts/index.rst | 2 +-
docs/apache-airflow/howto/add-dag-tags.rst | 4 +
docs/apache-airflow/howto/connection.rst | 6 +-
docs/apache-airflow/howto/variable.rst | 2 +-
docs/apache-airflow/index.rst | 33 +-
docs/apache-airflow/installation/index.rst | 78 +-
.../installation/installing-from-pypi.rst | 17 +-
.../installation/installing-from-sources.rst | 4 +-
docs/apache-airflow/installation/prerequisites.rst | 10 +-
.../installation/setting-up-the-database.rst | 2 +-
.../installation/supported-versions.rst | 4 +-
docs/apache-airflow/installation/upgrading.rst | 4 +-
docs/apache-airflow/integration.rst | 8 +-
docs/apache-airflow/project.rst | 3 +
docs/apache-airflow/public-airflow-interface.rst | 2 +-
docs/apache-airflow/redirects.txt | 35 +-
.../security/access-control.rst | 0
.../security/api.rst | 2 +-
.../security/audit_logs.rst | 0
.../security/flower.rst | 0
docs/apache-airflow/security/index.rst | 164 +
.../security/kerberos.rst | 0
.../security/secrets/fernet.rst | 0
.../security/secrets/index.rst | 0
.../security/secrets/mask-sensitive-values.rst | 0
.../security/secrets/secrets-backend/index.rst | 0
.../local-filesystem-secrets-backend.rst | 0
.../security/webserver.rst | 2 +-
.../security/workload.rst | 0
docs/apache-airflow/start.rst | 2 +-
docs/apache-airflow/templates-ref.rst | 24 +-
docs/apache-airflow/tutorial/fundamentals.rst | 2 +-
docs/apache-airflow/tutorial/pipeline.rst | 5 +-
docs/docker-stack/README.md | 10 +-
docs/docker-stack/build.rst | 23 +
docs/docker-stack/changelog.rst | 8 +-
.../Dockerfile | 7 +-
.../extending/add-apt-packages/Dockerfile | 2 +-
.../add-build-essential-extend/Dockerfile | 2 +-
.../extending/add-providers/Dockerfile | 2 +-
.../extending/add-pypi-packages/Dockerfile | 2 +-
.../extending/add-requirement-packages/Dockerfile | 2 +-
.../extending/custom-providers/Dockerfile | 2 +-
.../extending/embedding-dags/Dockerfile | 2 +-
.../extending/writable-directory/Dockerfile | 2 +-
docs/docker-stack/entrypoint.rst | 14 +-
docs/docker-stack/index.rst | 25 +-
docs/helm-chart/manage-dags-files.rst | 80 +-
docs/spelling_wordlist.txt | 4 +-
generated/README.md | 5 +
generated/provider_metadata.json | 5156 ++++++++++++++++++++
images/breeze/output-commands-hash.txt | 38 +-
images/breeze/output-commands.svg | 54 +-
images/breeze/output_build-docs.svg | 118 +-
images/breeze/output_ci-image.svg | 24 +-
images/breeze/output_ci-image_build.svg | 144 +-
images/breeze/output_ci-image_pull.svg | 66 +-
images/breeze/output_ci-image_verify.svg | 40 +-
images/breeze/output_k8s.svg | 58 +-
images/breeze/output_k8s_build-k8s-image.svg | 52 +-
images/breeze/output_k8s_configure-cluster.svg | 60 +-
images/breeze/output_k8s_create-cluster.svg | 62 +-
images/breeze/output_k8s_delete-cluster.svg | 34 +-
images/breeze/output_k8s_deploy-airflow.svg | 70 +-
images/breeze/output_k8s_k9s.svg | 32 +-
images/breeze/output_k8s_logs.svg | 36 +-
images/breeze/output_k8s_run-complete-tests.svg | 90 +-
images/breeze/output_k8s_shell.svg | 40 +-
images/breeze/output_k8s_status.svg | 36 +-
images/breeze/output_k8s_tests.svg | 68 +-
images/breeze/output_k8s_upload-k8s-image.svg | 58 +-
images/breeze/output_prod-image.svg | 24 +-
images/breeze/output_prod-image_build.svg | 182 +-
images/breeze/output_prod-image_pull.svg | 64 +-
images/breeze/output_prod-image_verify.svg | 40 +-
images/breeze/output_release-management.svg | 50 +-
...put_release-management_generate-constraints.svg | 62 +-
...management_generate-issue-content-providers.svg | 32 +-
...ease-management_generate-providers-metadata.svg | 112 +
...elease-management_install-provider-packages.svg | 74 +-
...e-management_prepare-provider-documentation.svg | 38 +-
...elease-management_prepare-provider-packages.svg | 30 +-
...tput_release-management_release-prod-images.svg | 46 +-
...release-management_verify-provider-packages.svg | 60 +-
images/breeze/output_sbom.svg | 107 +
.../output_sbom_generate-provider-requirements.svg | 174 +
.../breeze/output_sbom_update-sbom-information.svg | 186 +
images/breeze/output_setup.svg | 30 +-
.../output_setup_check-all-params-in-groups.svg | 62 +-
images/breeze/output_setup_config.svg | 40 +-
.../output_setup_regenerate-command-images.svg | 72 +-
images/breeze/output_static-checks.svg | 32 +-
images/breeze/output_testing.svg | 24 +-
images/breeze/output_testing_integration-tests.svg | 68 +-
images/breeze/output_testing_tests.svg | 108 +-
kubernetes_tests/test_base.py | 6 +-
pyproject.toml | 12 +-
.../pre_commit_check_setup_extra_packages_ref.py | 2 +
scripts/ci/pre_commit/pre_commit_insert_extras.py | 3 +
.../ci/pre_commit/pre_commit_supported_versions.py | 2 +-
scripts/docker/install_os_dependencies.sh | 4 +-
scripts/in_container/verify_providers.py | 4 +
setup.cfg | 5 +-
setup.py | 10 +-
tests/always/test_providers_manager.py | 36 +-
tests/decorators/test_python.py | 47 +-
tests/jobs/test_base_job.py | 1 -
tests/jobs/test_triggerer_job.py | 99 +-
tests/models/test_connection.py | 188 +
tests/models/test_dag.py | 42 +
tests/models/test_dagrun.py | 32 +
tests/models/test_param.py | 4 +-
tests/models/test_serialized_dag.py | 46 +-
tests/models/test_trigger.py | 64 +-
tests/models/test_variable.py | 35 +
tests/sensors/test_external_task_sensor.py | 20 +
tests/serialization/test_dag_serialization.py | 40 +-
tests/serialization/test_serde.py | 19 +-
tests/utils/log/test_secrets_masker.py | 2 +-
tests/utils/test_singleton.py | 65 +
tests/www/test_app.py | 51 +-
tests/www/test_utils.py | 24 +-
tests/www/views/test_views_connection.py | 71 +-
tests/www/views/test_views_rendered.py | 78 +-
tests/www/views/test_views_tasks.py | 56 +
tests/www/views/test_views_trigger_dag.py | 30 +
263 files changed, 10459 insertions(+), 1816 deletions(-)
rename .github/{SECURITY.rst => SECURITY.md} (53%)
copy airflow/{compat/functools.pyi => utils/singleton.py} (68%)
create mode 100644 dev/breeze/src/airflow_breeze/commands/sbom_commands.py
create mode 100644 dev/breeze/src/airflow_breeze/commands/sbom_commands_config.py
create mode 100644 dev/breeze/src/airflow_breeze/utils/cdxgen.py
create mode 100644 dev/breeze/src/airflow_breeze/utils/github.py
delete mode 100644 dev/breeze/src/airflow_breeze/utils/github_actions.py
create mode 100644 docs/apache-airflow/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst
delete mode 100644 docs/apache-airflow/administration-and-deployment/security/index.rst
rename docs/apache-airflow/{administration-and-deployment => }/security/access-control.rst (100%)
rename docs/apache-airflow/{administration-and-deployment => }/security/api.rst (98%)
rename docs/apache-airflow/{administration-and-deployment => }/security/audit_logs.rst (100%)
rename docs/apache-airflow/{administration-and-deployment => }/security/flower.rst (100%)
create mode 100644 docs/apache-airflow/security/index.rst
rename docs/apache-airflow/{administration-and-deployment => }/security/kerberos.rst (100%)
rename docs/apache-airflow/{administration-and-deployment => }/security/secrets/fernet.rst (100%)
rename docs/apache-airflow/{administration-and-deployment => }/security/secrets/index.rst (100%)
rename docs/apache-airflow/{administration-and-deployment => }/security/secrets/mask-sensitive-values.rst (100%)
rename docs/apache-airflow/{administration-and-deployment => }/security/secrets/secrets-backend/index.rst (100%)
rename docs/apache-airflow/{administration-and-deployment => }/security/secrets/secrets-backend/local-filesystem-secrets-backend.rst (100%)
rename docs/apache-airflow/{administration-and-deployment => }/security/webserver.rst (99%)
rename docs/apache-airflow/{administration-and-deployment => }/security/workload.rst (100%)
copy docs/docker-stack/docker-examples/extending/{embedding-dags => add-airflow-configuration}/Dockerfile (88%)
create mode 100644 generated/provider_metadata.json
create mode 100644 images/breeze/output_release-management_generate-providers-metadata.svg
create mode 100644 images/breeze/output_sbom.svg
create mode 100644 images/breeze/output_sbom_generate-provider-requirements.svg
create mode 100644 images/breeze/output_sbom_update-sbom-information.svg
create mode 100644 tests/models/test_connection.py
create mode 100644 tests/utils/test_singleton.py