You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "baha-bouali (via GitHub)" <gi...@apache.org> on 2026/09/08 22:36:25 UTC
[PR] Deprecate legacy remote logging branches in airflow_local_settings [airflow]
baha-bouali opened a new pull request, #72752:
URL: https://github.com/apache/airflow/pull/72752
All nine remote logging backends now ship RemoteLogIO.from_config and register their scheme in provider yaml, so ProvidersManager dispatch can resolve them and the per-scheme chain no longer needs to. Operators need a concrete signal before Airflow 4 removes it, naming both the provider version that supersedes their branch and the configuration change some backends still require.
The warning has to be conditional rather than unconditional: the default logging_config_class imports this module for its DEFAULT_LOGGING_CONFIG dict on every deployment, so the chain still executes even once provider dispatch has built the real handler. Warning on execution alone would reach every operator, including those with nothing left to migrate.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate legacy remote logging branches in airflow_local_settings [airflow]
Posted by "baha-bouali (via GitHub)" <gi...@apache.org>.
baha-bouali commented on PR #72752:
URL: https://github.com/apache/airflow/pull/72752#issuecomment-5598945482
cc @jason810496. This is the first part of the follow-up. Take a look whenever you have a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate legacy remote logging branches in airflow_local_settings [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #72752:
URL: https://github.com/apache/airflow/pull/72752#issuecomment-6511715251
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate legacy remote logging branches in airflow_local_settings [airflow]
Posted by "baha-bouali (via GitHub)" <gi...@apache.org>.
baha-bouali commented on code in PR #72752:
URL: https://github.com/apache/airflow/pull/72752#discussion_r4000332077
##########
airflow-core/newsfragments/72752.significant.rst:
##########
@@ -0,0 +1,47 @@
+Deprecate the per-scheme ``if/elif`` chain in ``airflow_local_settings.py``
Review Comment:
Will be removed in my next commit.
Just to make it right next time: is it because `significant` the wrong type? or is that because the deprecation was already announced for #67056?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate legacy remote logging branches in airflow_local_settings [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #72752:
URL: https://github.com/apache/airflow/pull/72752#discussion_r4001877444
##########
airflow-core/newsfragments/72752.significant.rst:
##########
@@ -0,0 +1,47 @@
+Deprecate the per-scheme ``if/elif`` chain in ``airflow_local_settings.py``
Review Comment:
The change we made is compatible for users so we don't need the significant file. Only if there's breaking change for users, then we need to add the significant file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate legacy remote logging branches in airflow_local_settings [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #72752:
URL: https://github.com/apache/airflow/pull/72752#discussion_r3996153315
##########
airflow-core/newsfragments/72752.significant.rst:
##########
@@ -0,0 +1,47 @@
+Deprecate the per-scheme ``if/elif`` chain in ``airflow_local_settings.py``
Review Comment:
Please remove the entire `airflow-core/newsfragments/72752.significant.rst`.
##########
airflow-core/src/airflow/config_templates/airflow_local_settings.py:
##########
@@ -152,6 +153,67 @@ def _default_conn_name_from(mod_path, hook_name):
return None
+# First provider distribution version whose ``RemoteLogIO`` exposes ``from_config`` *and*
+# registers the scheme in its provider.yaml ``remote-logging:`` block. Named in the deprecation
+# message so a Deployment Manager knows exactly which upgrade retires the legacy branch.
+_PROVIDER_DISPATCH_MIN_VERSIONS: dict[str, tuple[str, str]] = {
+ "s3": ("apache-airflow-providers-amazon", "9.33.0"),
+ "cloudwatch": ("apache-airflow-providers-amazon", "9.33.0"),
+ "gs": ("apache-airflow-providers-google", "22.3.0"),
+ "stackdriver": ("apache-airflow-providers-google", "22.3.0"),
+ "wasb": ("apache-airflow-providers-microsoft-azure", "14.1.0"),
+ "oss": ("apache-airflow-providers-alibaba", "3.4.0"),
+ "hdfs": ("apache-airflow-providers-apache-hdfs", "4.13.0"),
+ "elasticsearch": ("apache-airflow-providers-elasticsearch", "6.9.0"),
+ "opensearch": ("apache-airflow-providers-opensearch", "1.12.0"),
+}
+
+# Scheme of ``[logging] remote_base_log_folder``; the key ProvidersManager dispatches on.
+# Set below when remote logging is enabled.
+_configured_scheme: str = ""
+
+
+def _warn_legacy_remote_logging(remote_log_io: type, scheme: str) -> None:
+ """
+ Warn when this legacy branch, rather than provider dispatch, is what configures remote logging.
+
+ ``airflow.logging_config._get_logging_config`` imports this module for its
+ ``DEFAULT_LOGGING_CONFIG`` dict on every stock deployment, so the chain below still runs
+ even when ProvidersManager scheme dispatch has already built the real handler. Warning
+ unconditionally would therefore fire for every operator, including those with nothing left
+ to migrate, so a branch warns only when dispatch cannot supersede it:
+
+ * the installed provider predates ``from_config``, and so registers no scheme; or
+ * ``[logging] remote_base_log_folder`` carries no scheme to dispatch on -- a bare
+ ``wasb-logs`` path, or Elasticsearch/OpenSearch selected through their ``host`` option.
+ """
Review Comment:
Please make the docstring more concise, the purpose of the function is clear enough.
##########
airflow-core/src/airflow/config_templates/airflow_local_settings.py:
##########
@@ -152,6 +153,67 @@ def _default_conn_name_from(mod_path, hook_name):
return None
+# First provider distribution version whose ``RemoteLogIO`` exposes ``from_config`` *and*
+# registers the scheme in its provider.yaml ``remote-logging:`` block. Named in the deprecation
+# message so a Deployment Manager knows exactly which upgrade retires the legacy branch.
+_PROVIDER_DISPATCH_MIN_VERSIONS: dict[str, tuple[str, str]] = {
+ "s3": ("apache-airflow-providers-amazon", "9.33.0"),
+ "cloudwatch": ("apache-airflow-providers-amazon", "9.33.0"),
+ "gs": ("apache-airflow-providers-google", "22.3.0"),
+ "stackdriver": ("apache-airflow-providers-google", "22.3.0"),
+ "wasb": ("apache-airflow-providers-microsoft-azure", "14.1.0"),
+ "oss": ("apache-airflow-providers-alibaba", "3.4.0"),
+ "hdfs": ("apache-airflow-providers-apache-hdfs", "4.13.0"),
+ "elasticsearch": ("apache-airflow-providers-elasticsearch", "6.9.0"),
+ "opensearch": ("apache-airflow-providers-opensearch", "1.12.0"),
+}
+
+# Scheme of ``[logging] remote_base_log_folder``; the key ProvidersManager dispatches on.
+# Set below when remote logging is enabled.
+_configured_scheme: str = ""
+
+
+def _warn_legacy_remote_logging(remote_log_io: type, scheme: str) -> None:
+ """
+ Warn when this legacy branch, rather than provider dispatch, is what configures remote logging.
+
+ ``airflow.logging_config._get_logging_config`` imports this module for its
+ ``DEFAULT_LOGGING_CONFIG`` dict on every stock deployment, so the chain below still runs
+ even when ProvidersManager scheme dispatch has already built the real handler. Warning
+ unconditionally would therefore fire for every operator, including those with nothing left
+ to migrate, so a branch warns only when dispatch cannot supersede it:
+
+ * the installed provider predates ``from_config``, and so registers no scheme; or
+ * ``[logging] remote_base_log_folder`` carries no scheme to dispatch on -- a bare
+ ``wasb-logs`` path, or Elasticsearch/OpenSearch selected through their ``host`` option.
+ """
+ distribution, min_version = _PROVIDER_DISPATCH_MIN_VERSIONS[scheme]
+ provider_supports_dispatch = hasattr(remote_log_io, "from_config")
+
+ if provider_supports_dispatch and _configured_scheme == scheme:
+ return
+
+ if not provider_supports_dispatch:
+ remedy = (
+ f"Upgrade {distribution} to {min_version} or newer, which registers the {scheme!r} "
+ f"scheme and builds this handler from {remote_log_io.__name__}.from_config()."
+ )
+ else:
+ remedy = (
+ f"{distribution} {min_version} or newer already registers the {scheme!r} scheme; set "
+ f'[logging] remote_base_log_folder to a "{scheme}://" URL so it is dispatched on. '
+ f"Keep the backend options you already set, as from_config() still reads them."
+ )
+
+ warnings.warn(
+ f"Remote logging for {scheme!r} is being configured by the if/elif chain in "
+ f"airflow_local_settings.py. That chain is deprecated and will be removed in Airflow 4, "
+ f"after which remote logging is resolved only through provider registration. {remedy}",
+ RemovedInAirflow4Warning,
+ stacklevel=2,
+ )
Review Comment:
Please shorten the warning message and all the hint. Additionally, we shouldn't mention that much internal details to public users, just something more straightforward e.g. Recommend to update ... to ... for the provider based discovery as Airflow 4 will remove the the hardcoded discovery in core. (perhaps need more refinement as well)
##########
airflow-core/src/airflow/config_templates/airflow_local_settings.py:
##########
@@ -152,6 +153,67 @@ def _default_conn_name_from(mod_path, hook_name):
return None
+# First provider distribution version whose ``RemoteLogIO`` exposes ``from_config`` *and*
+# registers the scheme in its provider.yaml ``remote-logging:`` block. Named in the deprecation
+# message so a Deployment Manager knows exactly which upgrade retires the legacy branch.
+_PROVIDER_DISPATCH_MIN_VERSIONS: dict[str, tuple[str, str]] = {
+ "s3": ("apache-airflow-providers-amazon", "9.33.0"),
+ "cloudwatch": ("apache-airflow-providers-amazon", "9.33.0"),
+ "gs": ("apache-airflow-providers-google", "22.3.0"),
+ "stackdriver": ("apache-airflow-providers-google", "22.3.0"),
+ "wasb": ("apache-airflow-providers-microsoft-azure", "14.1.0"),
+ "oss": ("apache-airflow-providers-alibaba", "3.4.0"),
+ "hdfs": ("apache-airflow-providers-apache-hdfs", "4.13.0"),
+ "elasticsearch": ("apache-airflow-providers-elasticsearch", "6.9.0"),
+ "opensearch": ("apache-airflow-providers-opensearch", "1.12.0"),
+}
+
+# Scheme of ``[logging] remote_base_log_folder``; the key ProvidersManager dispatches on.
+# Set below when remote logging is enabled.
+_configured_scheme: str = ""
Review Comment:
No need to add a new global variable, we can reference the existing `remote_base_log_folder` in the function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate legacy remote logging branches in airflow_local_settings [airflow]
Posted by "baha-bouali (via GitHub)" <gi...@apache.org>.
baha-bouali commented on code in PR #72752:
URL: https://github.com/apache/airflow/pull/72752#discussion_r4009549608
##########
airflow-core/src/airflow/config_templates/airflow_local_settings.py:
##########
@@ -152,6 +153,67 @@ def _default_conn_name_from(mod_path, hook_name):
return None
+# First provider distribution version whose ``RemoteLogIO`` exposes ``from_config`` *and*
+# registers the scheme in its provider.yaml ``remote-logging:`` block. Named in the deprecation
+# message so a Deployment Manager knows exactly which upgrade retires the legacy branch.
+_PROVIDER_DISPATCH_MIN_VERSIONS: dict[str, tuple[str, str]] = {
+ "s3": ("apache-airflow-providers-amazon", "9.33.0"),
+ "cloudwatch": ("apache-airflow-providers-amazon", "9.33.0"),
+ "gs": ("apache-airflow-providers-google", "22.3.0"),
+ "stackdriver": ("apache-airflow-providers-google", "22.3.0"),
+ "wasb": ("apache-airflow-providers-microsoft-azure", "14.1.0"),
+ "oss": ("apache-airflow-providers-alibaba", "3.4.0"),
+ "hdfs": ("apache-airflow-providers-apache-hdfs", "4.13.0"),
+ "elasticsearch": ("apache-airflow-providers-elasticsearch", "6.9.0"),
+ "opensearch": ("apache-airflow-providers-opensearch", "1.12.0"),
+}
+
+# Scheme of ``[logging] remote_base_log_folder``; the key ProvidersManager dispatches on.
+# Set below when remote logging is enabled.
+_configured_scheme: str = ""
+
+
+def _warn_legacy_remote_logging(remote_log_io: type, scheme: str) -> None:
+ """
+ Warn when this legacy branch, rather than provider dispatch, is what configures remote logging.
+
+ ``airflow.logging_config._get_logging_config`` imports this module for its
+ ``DEFAULT_LOGGING_CONFIG`` dict on every stock deployment, so the chain below still runs
+ even when ProvidersManager scheme dispatch has already built the real handler. Warning
+ unconditionally would therefore fire for every operator, including those with nothing left
+ to migrate, so a branch warns only when dispatch cannot supersede it:
+
+ * the installed provider predates ``from_config``, and so registers no scheme; or
+ * ``[logging] remote_base_log_folder`` carries no scheme to dispatch on -- a bare
+ ``wasb-logs`` path, or Elasticsearch/OpenSearch selected through their ``host`` option.
+ """
Review Comment:
Trimmed it to just one-liner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate legacy remote logging branches in airflow_local_settings [airflow]
Posted by "baha-bouali (via GitHub)" <gi...@apache.org>.
baha-bouali commented on code in PR #72752:
URL: https://github.com/apache/airflow/pull/72752#discussion_r4009818880
##########
airflow-core/src/airflow/config_templates/airflow_local_settings.py:
##########
@@ -152,6 +153,67 @@ def _default_conn_name_from(mod_path, hook_name):
return None
+# First provider distribution version whose ``RemoteLogIO`` exposes ``from_config`` *and*
+# registers the scheme in its provider.yaml ``remote-logging:`` block. Named in the deprecation
+# message so a Deployment Manager knows exactly which upgrade retires the legacy branch.
+_PROVIDER_DISPATCH_MIN_VERSIONS: dict[str, tuple[str, str]] = {
+ "s3": ("apache-airflow-providers-amazon", "9.33.0"),
+ "cloudwatch": ("apache-airflow-providers-amazon", "9.33.0"),
+ "gs": ("apache-airflow-providers-google", "22.3.0"),
+ "stackdriver": ("apache-airflow-providers-google", "22.3.0"),
+ "wasb": ("apache-airflow-providers-microsoft-azure", "14.1.0"),
+ "oss": ("apache-airflow-providers-alibaba", "3.4.0"),
+ "hdfs": ("apache-airflow-providers-apache-hdfs", "4.13.0"),
+ "elasticsearch": ("apache-airflow-providers-elasticsearch", "6.9.0"),
+ "opensearch": ("apache-airflow-providers-opensearch", "1.12.0"),
+}
+
+# Scheme of ``[logging] remote_base_log_folder``; the key ProvidersManager dispatches on.
+# Set below when remote logging is enabled.
+_configured_scheme: str = ""
+
+
+def _warn_legacy_remote_logging(remote_log_io: type, scheme: str) -> None:
+ """
+ Warn when this legacy branch, rather than provider dispatch, is what configures remote logging.
+
+ ``airflow.logging_config._get_logging_config`` imports this module for its
+ ``DEFAULT_LOGGING_CONFIG`` dict on every stock deployment, so the chain below still runs
+ even when ProvidersManager scheme dispatch has already built the real handler. Warning
+ unconditionally would therefore fire for every operator, including those with nothing left
+ to migrate, so a branch warns only when dispatch cannot supersede it:
+
+ * the installed provider predates ``from_config``, and so registers no scheme; or
+ * ``[logging] remote_base_log_folder`` carries no scheme to dispatch on -- a bare
+ ``wasb-logs`` path, or Elasticsearch/OpenSearch selected through their ``host`` option.
+ """
+ distribution, min_version = _PROVIDER_DISPATCH_MIN_VERSIONS[scheme]
+ provider_supports_dispatch = hasattr(remote_log_io, "from_config")
+
+ if provider_supports_dispatch and _configured_scheme == scheme:
+ return
+
+ if not provider_supports_dispatch:
+ remedy = (
+ f"Upgrade {distribution} to {min_version} or newer, which registers the {scheme!r} "
+ f"scheme and builds this handler from {remote_log_io.__name__}.from_config()."
+ )
+ else:
+ remedy = (
+ f"{distribution} {min_version} or newer already registers the {scheme!r} scheme; set "
+ f'[logging] remote_base_log_folder to a "{scheme}://" URL so it is dispatched on. '
+ f"Keep the backend options you already set, as from_config() still reads them."
+ )
+
+ warnings.warn(
+ f"Remote logging for {scheme!r} is being configured by the if/elif chain in "
+ f"airflow_local_settings.py. That chain is deprecated and will be removed in Airflow 4, "
+ f"after which remote logging is resolved only through provider registration. {remedy}",
+ RemovedInAirflow4Warning,
+ stacklevel=2,
+ )
Review Comment:
One suggestion: ES/OS migration is additive. In fact, we added `remote_base_log_folder=elasticsearch://...` and keep the `[elasticsearch]` section, since `from_config()` still reads `target_index`, `host_field`, etc... from it.
It's where `remote_base_log_folder` is a router rather than a destination.
I'd suggest to cover that. Whether here (inside the warning) or in the `ES/OS` logging docs. (I'd go with the second so the warning message stays short).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org