You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "sunank200 (via GitHub)" <gi...@apache.org> on 2025/03/05 06:26:01 UTC

[PR] Handle default values from Airflow 2 gracefully [airflow]

sunank200 opened a new pull request, #47371:
URL: https://github.com/apache/airflow/pull/47371

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   - Handle default values from Airflow 2 gracefully.
   
   closes: [#46972](https://github.com/apache/airflow/issues/46972)
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1993100164


##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -546,6 +563,12 @@ def lint_config(args) -> None:
         ):
             lint_issues.append(configuration.message)
 
+        if not skip_default_checks:

Review Comment:
   changed it



##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -531,6 +545,9 @@ def lint_config(args) -> None:
     ignore_sections = args.ignore_section or []
     ignore_options = args.ignore_option or []
 
+    skip_default_checks = args.skip_default_checks or False

Review Comment:
   changed it



##########
airflow/configuration.py:
##########
@@ -403,6 +403,64 @@ def inversed_deprecated_sections(self):
     upgraded_values: dict[tuple[str, str], str]
     """Mapping of (section,option) to the old value that was upgraded"""
 
+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(
+                r"^(?:" + re.escape("{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log") + r"|"
+                r"dag_id=\{\s*ti\.dag_id\s*\}/run_id=\{\s*ti\.run_id\s*\}/task_id=\{\s*ti\.task_id\s*\}/"
+                r"\{%%\s*if\s+ti\.map_index\s*>=\s*0\s*%%\}map_index=\{\s*ti\.map_index\s*\}/\{%%\s*endif\s*%%\}"
+                r"attempt=\{\s*try_number\s*\}\.log" + r")$"
+            ),
+            "3.0",
+            "The default value for 'log_filename_template' from Airflow 2 may break task logs in the new UI.",
+        ),
+        ("core", "dag_ignore_file_syntax"): (
+            re.compile(r"^regexp$"),
+            "3.0",
+            "The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",
+        ),
+        ("api", "auth_backends"): (
+            re.compile(r"^airflow\.api\.auth\.backend\.session$"),
+            "3.0",
+            "The default auth backend has changed from 'airflow.api.auth.backend.session' to "
+            "'airflow.providers.fab.auth_manager.api.auth.backend.session' in Airflow 3.0.",
+        ),
+        ("scheduler", "task_instance_heartbeat_timeout"): (
+            re.compile(r"^300$"),
+            "3.0",
+            "Airflow 2.x used 'scheduler_zombie_task_threshold' with a default of 300. In 3.0 this setting "
+            "has been renamed to 'task_instance_heartbeat_timeout' (with the same numeric default), which may "
+            "cause legacy configurations to be ignored.",
+        ),
+    }
+
+    def handle_incompatible_airflow2_defaults(self, upgrade: bool | None = True) -> None:
+        """
+        Check and optionally upgrade default configuration values from Airflow 2 that are incompatible with Airflow 3.
+
+        :param upgrade: If True, auto-upgrade legacy defaults. If False, only warn about the legacy defaults without making any changes.
+        """
+        for (section, key), (old_pattern, _version, message) in self.legacy_incompatible_defaults.items():
+            default_val = self.get_default_value(section, key, raw=True)
+            current_value = self.get(section, key, fallback="")
+            if self._using_old_value(old_pattern, current_value):
+                if upgrade:
+                    self.upgraded_values[(section, key)] = default_val
+                    updated_val = old_pattern.sub(default_val, current_value)
+                    self._update_env_var(section, key, updated_val)
+                    warnings.warn(
+                        f"[{section}] {key}: {message} Auto-upgraded from '{current_value}' to '{updated_val}'.",
+                        FutureWarning,
+                        stacklevel=1,
+                    )
+                else:
+                    warnings.warn(
+                        f"[{section}] {key}: {message} Current value: '{current_value}'. "
+                        f"Consider updating to '{default_val}'.",
+                        FutureWarning,
+                        stacklevel=1,
+                    )

Review Comment:
   changed it



##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -546,6 +563,12 @@ def lint_config(args) -> None:
         ):
             lint_issues.append(configuration.message)
 
+        if not skip_default_checks:
+            if upgrade_defaults:
+                conf.handle_incompatible_airflow2_defaults(upgrade=True)
+            else:
+                conf.handle_incompatible_airflow2_defaults(upgrade=False)

Review Comment:
   changed it



##########
airflow/configuration.py:
##########
@@ -403,6 +403,64 @@ def inversed_deprecated_sections(self):
     upgraded_values: dict[tuple[str, str], str]
     """Mapping of (section,option) to the old value that was upgraded"""
 
+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(
+                r"^(?:" + re.escape("{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log") + r"|"
+                r"dag_id=\{\s*ti\.dag_id\s*\}/run_id=\{\s*ti\.run_id\s*\}/task_id=\{\s*ti\.task_id\s*\}/"
+                r"\{%%\s*if\s+ti\.map_index\s*>=\s*0\s*%%\}map_index=\{\s*ti\.map_index\s*\}/\{%%\s*endif\s*%%\}"
+                r"attempt=\{\s*try_number\s*\}\.log" + r")$"
+            ),
+            "3.0",
+            "The default value for 'log_filename_template' from Airflow 2 may break task logs in the new UI.",
+        ),
+        ("core", "dag_ignore_file_syntax"): (
+            re.compile(r"^regexp$"),
+            "3.0",
+            "The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",
+        ),
+        ("api", "auth_backends"): (
+            re.compile(r"^airflow\.api\.auth\.backend\.session$"),
+            "3.0",
+            "The default auth backend has changed from 'airflow.api.auth.backend.session' to "
+            "'airflow.providers.fab.auth_manager.api.auth.backend.session' in Airflow 3.0.",
+        ),
+        ("scheduler", "task_instance_heartbeat_timeout"): (
+            re.compile(r"^300$"),
+            "3.0",
+            "Airflow 2.x used 'scheduler_zombie_task_threshold' with a default of 300. In 3.0 this setting "
+            "has been renamed to 'task_instance_heartbeat_timeout' (with the same numeric default), which may "
+            "cause legacy configurations to be ignored.",
+        ),

Review Comment:
   removed it



##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -531,6 +545,9 @@ def lint_config(args) -> None:
     ignore_sections = args.ignore_section or []
     ignore_options = args.ignore_option or []
 
+    skip_default_checks = args.skip_default_checks or False
+    upgrade_defaults = args.upgrade_defaults or True

Review Comment:
   changed it



##########
airflow/configuration.py:
##########
@@ -403,6 +403,64 @@ def inversed_deprecated_sections(self):
     upgraded_values: dict[tuple[str, str], str]
     """Mapping of (section,option) to the old value that was upgraded"""
 
+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(
+                r"^(?:" + re.escape("{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log") + r"|"
+                r"dag_id=\{\s*ti\.dag_id\s*\}/run_id=\{\s*ti\.run_id\s*\}/task_id=\{\s*ti\.task_id\s*\}/"
+                r"\{%%\s*if\s+ti\.map_index\s*>=\s*0\s*%%\}map_index=\{\s*ti\.map_index\s*\}/\{%%\s*endif\s*%%\}"
+                r"attempt=\{\s*try_number\s*\}\.log" + r")$"
+            ),
+            "3.0",
+            "The default value for 'log_filename_template' from Airflow 2 may break task logs in the new UI.",
+        ),
+        ("core", "dag_ignore_file_syntax"): (
+            re.compile(r"^regexp$"),
+            "3.0",
+            "The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",
+        ),
+        ("api", "auth_backends"): (
+            re.compile(r"^airflow\.api\.auth\.backend\.session$"),
+            "3.0",
+            "The default auth backend has changed from 'airflow.api.auth.backend.session' to "
+            "'airflow.providers.fab.auth_manager.api.auth.backend.session' in Airflow 3.0.",
+        ),
+        ("scheduler", "task_instance_heartbeat_timeout"): (
+            re.compile(r"^300$"),
+            "3.0",
+            "Airflow 2.x used 'scheduler_zombie_task_threshold' with a default of 300. In 3.0 this setting "
+            "has been renamed to 'task_instance_heartbeat_timeout' (with the same numeric default), which may "
+            "cause legacy configurations to be ignored.",
+        ),
+    }
+
+    def handle_incompatible_airflow2_defaults(self, upgrade: bool | None = True) -> None:

Review Comment:
   changed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1994448649


##########
airflow/cli/cli_config.py:
##########
@@ -791,6 +791,16 @@ def string_lower_type(val):
     ("--section",),
     help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),

Review Comment:
   Not sure. Maybe `--patch-problematic-defaults` instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1994880338


##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -546,6 +560,9 @@ def lint_config(args) -> None:
         ):
             lint_issues.append(configuration.message)
 
+        if not args.skip_problematic_default_checks:
+            conf.handle_incompatible_airflow2_defaults(upgrade=args.upgrade_problematic_defaults)
+

Review Comment:
   It is unfortunate that this is what you get:
   
   ```
   $ airflow config lint
   /Users/jedc/github/airflow/airflow/configuration.py:440 FutureWarning: [core] dag_ignore_file_syntax: The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0. Auto-upgraded from 'regexp' to 'glob'.
   No issues found in your airflow.cfg. It is ready for Airflow 3!
   ```
   
   The warning is from `configuration.py`, where it got auto-upgraded.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on PR #47371:
URL: https://github.com/apache/airflow/pull/47371#issuecomment-2709588569

   @vatsrahul1001 can we progress on this, as discussed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1994873513


##########
tests/cli/commands/remote_commands/test_config_command.py:
##########
@@ -447,3 +445,30 @@ def test_lint_detects_configs_with_env_vars(self, env_var, config_change, expect
 
         assert expected_message in normalized_output
         assert config_change.suggestion in normalized_output
+
+    def test_lint_detects_default_value_change_upgrade(self):

Review Comment:
   Also means this test should be refactored and moved into [test_configuration.py](https://github.com/apache/airflow/blob/main/tests/core/test_configuration.py).



##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -546,6 +560,9 @@ def lint_config(args) -> None:
         ):
             lint_issues.append(configuration.message)
 
+        if not args.skip_problematic_default_checks:
+            conf.handle_incompatible_airflow2_defaults(upgrade=args.upgrade_problematic_defaults)
+

Review Comment:
   Which means, since this has already run and upgraded, you can get rid of this completely - and the args, etc. All of it.



##########
airflow/configuration.py:
##########
@@ -403,6 +403,52 @@ def inversed_deprecated_sections(self):
     upgraded_values: dict[tuple[str, str], str]
     """Mapping of (section,option) to the old value that was upgraded"""
 
+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(
+                r"^(?:" + re.escape("{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log") + r"|"
+                r"dag_id=\{\s*ti\.dag_id\s*\}/run_id=\{\s*ti\.run_id\s*\}/task_id=\{\s*ti\.task_id\s*\}/"
+                r"\{%%\s*if\s+ti\.map_index\s*>=\s*0\s*%%\}map_index=\{\s*ti\.map_index\s*\}/\{%%\s*endif\s*%%\}"
+                r"attempt=\{\s*try_number\s*\}\.log" + r")$"
+            ),
+            "3.0",
+            "The default value for 'log_filename_template' from Airflow 2 may break task logs in the new UI.",
+        ),
+        ("core", "dag_ignore_file_syntax"): (
+            re.compile(r"^regexp$"),
+            "3.0",
+            "The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",
+        ),

Review Comment:
   ```suggestion
   ```
   
   Looking at this, automatically changing folks away from the default may break things, not using an old default. e.g. if their airflowignore was regex, but we automatically swap to glob, things won't be happy.
   
   We should leave this one out I believe.
   



##########
airflow/cli/cli_config.py:
##########
@@ -791,6 +791,16 @@ def string_lower_type(val):
     ("--section",),
     help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),

Review Comment:
   So, now that you bring it up, not sure this flag is even worth having. Upgrading it or not in the config lint command process doesn't seem that helpful?
   
   But wait! In `configuration.py` you are already upgrading to the new default, meaning by the time `config lint` runs, well, it's already run and updated stuff. So having this run in `config lint` at all isn't doing anything really.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1994885498


##########
airflow/configuration.py:
##########
@@ -403,6 +403,52 @@ def inversed_deprecated_sections(self):
     upgraded_values: dict[tuple[str, str], str]
     """Mapping of (section,option) to the old value that was upgraded"""
 
+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(

Review Comment:
   fwiw, I took the default out of a fresh 2.10.5 config file, and this regex did _not_ match against it. The regex may have been cut off at the end of the task_id line (that's naively what printed, at least).
   
   Give that a test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1994894731


##########
airflow/cli/cli_config.py:
##########
@@ -791,6 +791,16 @@ def string_lower_type(val):
     ("--section",),
     help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),

Review Comment:
   I can maybe have `upgrade=False` in configuration.py, which wouldn't upgrade the confs by default. Is this what we intend to do? Or do we always want to update in any case?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1992881525


##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -546,6 +563,12 @@ def lint_config(args) -> None:
         ):
             lint_issues.append(configuration.message)
 
+        if not skip_default_checks:

Review Comment:
   ```suggestion
           if not args.skip_problematic_default_checks:
   ```



##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -531,6 +545,9 @@ def lint_config(args) -> None:
     ignore_sections = args.ignore_section or []
     ignore_options = args.ignore_option or []
 
+    skip_default_checks = args.skip_default_checks or False
+    upgrade_defaults = args.upgrade_defaults or True

Review Comment:
   ```suggestion
   ```
   
   With `or True`, won't this always be true 🙃 



##########
airflow/cli/cli_config.py:
##########
@@ -791,6 +791,16 @@ def string_lower_type(val):
     ("--section",),
     help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),

Review Comment:
   ```suggestion
       ("--upgrade-problematic-defaults",),
   ```
   
   It's probably worth renaming these flags to include `problematic` - without it, it implies it'll upgrade all of the old defaults (which isn't a bad idea!).
   
   Probably also means the Arg variable name should change too.



##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -546,6 +563,12 @@ def lint_config(args) -> None:
         ):
             lint_issues.append(configuration.message)
 
+        if not skip_default_checks:
+            if upgrade_defaults:
+                conf.handle_incompatible_airflow2_defaults(upgrade=True)
+            else:
+                conf.handle_incompatible_airflow2_defaults(upgrade=False)

Review Comment:
   ```suggestion
               conf.handle_incompatible_airflow2_defaults(upgrade=args.upgrade_problematic_defaults)
   ```
   
   (looks funny, but the whitespace is right I believe)
   
   No need to do a conditional against a bool, to then just hard code a bool.



##########
airflow/cli/cli_config.py:
##########
@@ -791,6 +791,16 @@ def string_lower_type(val):
     ("--section",),
     help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),
+    help="Automatically upgrade problematic default values from Airflow 2 to the new recommended values.",
+    action="store_true",
+)
+ARG_LINT_CONFIG_SKIP_DEFAULT_CHECKS = Arg(
+    ("--skip-default-checks",),

Review Comment:
   ```suggestion
       ("--skip-problematic-default-checks",),
   ```



##########
airflow/configuration.py:
##########
@@ -403,6 +403,64 @@ def inversed_deprecated_sections(self):
     upgraded_values: dict[tuple[str, str], str]
     """Mapping of (section,option) to the old value that was upgraded"""
 
+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(
+                r"^(?:" + re.escape("{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log") + r"|"
+                r"dag_id=\{\s*ti\.dag_id\s*\}/run_id=\{\s*ti\.run_id\s*\}/task_id=\{\s*ti\.task_id\s*\}/"
+                r"\{%%\s*if\s+ti\.map_index\s*>=\s*0\s*%%\}map_index=\{\s*ti\.map_index\s*\}/\{%%\s*endif\s*%%\}"
+                r"attempt=\{\s*try_number\s*\}\.log" + r")$"
+            ),
+            "3.0",
+            "The default value for 'log_filename_template' from Airflow 2 may break task logs in the new UI.",
+        ),
+        ("core", "dag_ignore_file_syntax"): (
+            re.compile(r"^regexp$"),
+            "3.0",
+            "The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",
+        ),
+        ("api", "auth_backends"): (
+            re.compile(r"^airflow\.api\.auth\.backend\.session$"),
+            "3.0",
+            "The default auth backend has changed from 'airflow.api.auth.backend.session' to "
+            "'airflow.providers.fab.auth_manager.api.auth.backend.session' in Airflow 3.0.",
+        ),

Review Comment:
   ```suggestion
   ```
   
   This has been deleted in #47399.



##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -531,6 +545,9 @@ def lint_config(args) -> None:
     ignore_sections = args.ignore_section or []
     ignore_options = args.ignore_option or []
 
+    skip_default_checks = args.skip_default_checks or False

Review Comment:
   ```suggestion
   ```
   
   This should just work as-is. Probably not worth assigning to a separate variable. (1 of 2)



##########
airflow/configuration.py:
##########
@@ -403,6 +403,64 @@ def inversed_deprecated_sections(self):
     upgraded_values: dict[tuple[str, str], str]
     """Mapping of (section,option) to the old value that was upgraded"""
 
+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(
+                r"^(?:" + re.escape("{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log") + r"|"
+                r"dag_id=\{\s*ti\.dag_id\s*\}/run_id=\{\s*ti\.run_id\s*\}/task_id=\{\s*ti\.task_id\s*\}/"
+                r"\{%%\s*if\s+ti\.map_index\s*>=\s*0\s*%%\}map_index=\{\s*ti\.map_index\s*\}/\{%%\s*endif\s*%%\}"
+                r"attempt=\{\s*try_number\s*\}\.log" + r")$"
+            ),
+            "3.0",
+            "The default value for 'log_filename_template' from Airflow 2 may break task logs in the new UI.",
+        ),
+        ("core", "dag_ignore_file_syntax"): (
+            re.compile(r"^regexp$"),
+            "3.0",
+            "The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",
+        ),
+        ("api", "auth_backends"): (
+            re.compile(r"^airflow\.api\.auth\.backend\.session$"),
+            "3.0",
+            "The default auth backend has changed from 'airflow.api.auth.backend.session' to "
+            "'airflow.providers.fab.auth_manager.api.auth.backend.session' in Airflow 3.0.",
+        ),
+        ("scheduler", "task_instance_heartbeat_timeout"): (
+            re.compile(r"^300$"),
+            "3.0",
+            "Airflow 2.x used 'scheduler_zombie_task_threshold' with a default of 300. In 3.0 this setting "
+            "has been renamed to 'task_instance_heartbeat_timeout' (with the same numeric default), which may "
+            "cause legacy configurations to be ignored.",
+        ),
+    }
+
+    def handle_incompatible_airflow2_defaults(self, upgrade: bool | None = True) -> None:

Review Comment:
   ```suggestion
       def handle_incompatible_airflow2_defaults(self, upgrade: bool) -> None:
   ```
   
   Should always be a bool.



##########
airflow/configuration.py:
##########
@@ -403,6 +403,64 @@ def inversed_deprecated_sections(self):
     upgraded_values: dict[tuple[str, str], str]
     """Mapping of (section,option) to the old value that was upgraded"""
 
+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(
+                r"^(?:" + re.escape("{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log") + r"|"
+                r"dag_id=\{\s*ti\.dag_id\s*\}/run_id=\{\s*ti\.run_id\s*\}/task_id=\{\s*ti\.task_id\s*\}/"
+                r"\{%%\s*if\s+ti\.map_index\s*>=\s*0\s*%%\}map_index=\{\s*ti\.map_index\s*\}/\{%%\s*endif\s*%%\}"
+                r"attempt=\{\s*try_number\s*\}\.log" + r")$"
+            ),
+            "3.0",
+            "The default value for 'log_filename_template' from Airflow 2 may break task logs in the new UI.",
+        ),
+        ("core", "dag_ignore_file_syntax"): (
+            re.compile(r"^regexp$"),
+            "3.0",
+            "The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",
+        ),
+        ("api", "auth_backends"): (
+            re.compile(r"^airflow\.api\.auth\.backend\.session$"),
+            "3.0",
+            "The default auth backend has changed from 'airflow.api.auth.backend.session' to "
+            "'airflow.providers.fab.auth_manager.api.auth.backend.session' in Airflow 3.0.",
+        ),
+        ("scheduler", "task_instance_heartbeat_timeout"): (
+            re.compile(r"^300$"),
+            "3.0",
+            "Airflow 2.x used 'scheduler_zombie_task_threshold' with a default of 300. In 3.0 this setting "
+            "has been renamed to 'task_instance_heartbeat_timeout' (with the same numeric default), which may "
+            "cause legacy configurations to be ignored.",
+        ),
+    }
+
+    def handle_incompatible_airflow2_defaults(self, upgrade: bool | None = True) -> None:
+        """
+        Check and optionally upgrade default configuration values from Airflow 2 that are incompatible with Airflow 3.
+
+        :param upgrade: If True, auto-upgrade legacy defaults. If False, only warn about the legacy defaults without making any changes.
+        """
+        for (section, key), (old_pattern, _version, message) in self.legacy_incompatible_defaults.items():
+            default_val = self.get_default_value(section, key, raw=True)
+            current_value = self.get(section, key, fallback="")
+            if self._using_old_value(old_pattern, current_value):
+                if upgrade:
+                    self.upgraded_values[(section, key)] = default_val
+                    updated_val = old_pattern.sub(default_val, current_value)
+                    self._update_env_var(section, key, updated_val)
+                    warnings.warn(
+                        f"[{section}] {key}: {message} Auto-upgraded from '{current_value}' to '{updated_val}'.",
+                        FutureWarning,
+                        stacklevel=1,
+                    )
+                else:
+                    warnings.warn(
+                        f"[{section}] {key}: {message} Current value: '{current_value}'. "
+                        f"Consider updating to '{default_val}'.",
+                        FutureWarning,
+                        stacklevel=1,
+                    )

Review Comment:
   ```suggestion
               current_value = self.get(section, key, fallback="")
               if not self._using_old_value(old_pattern, current_value):
                   continue
                   
               default_val = self.get_default_value(section, key, raw=True)
               if upgrade:
                   self.upgraded_values[(section, key)] = default_val
                   updated_val = old_pattern.sub(default_val, current_value)
                   self._update_env_var(section, key, updated_val)
                   warnings.warn(
                       f"[{section}] {key}: {message} Auto-upgraded from '{current_value}' to '{updated_val}'.",
                       FutureWarning,
                       stacklevel=1,
                   )
               else:
                   warnings.warn(
                       f"[{section}] {key}: {message} Current value: '{current_value}'. "
                       f"Consider updating to '{default_val}'.",
                       FutureWarning,
                       stacklevel=1,
                   )
   ```
   
   Less nesting, also only gets the default value if we will need it.



##########
airflow/configuration.py:
##########
@@ -403,6 +403,64 @@ def inversed_deprecated_sections(self):
     upgraded_values: dict[tuple[str, str], str]
     """Mapping of (section,option) to the old value that was upgraded"""
 
+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(
+                r"^(?:" + re.escape("{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log") + r"|"
+                r"dag_id=\{\s*ti\.dag_id\s*\}/run_id=\{\s*ti\.run_id\s*\}/task_id=\{\s*ti\.task_id\s*\}/"
+                r"\{%%\s*if\s+ti\.map_index\s*>=\s*0\s*%%\}map_index=\{\s*ti\.map_index\s*\}/\{%%\s*endif\s*%%\}"
+                r"attempt=\{\s*try_number\s*\}\.log" + r")$"
+            ),
+            "3.0",
+            "The default value for 'log_filename_template' from Airflow 2 may break task logs in the new UI.",
+        ),
+        ("core", "dag_ignore_file_syntax"): (
+            re.compile(r"^regexp$"),
+            "3.0",
+            "The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",
+        ),
+        ("api", "auth_backends"): (
+            re.compile(r"^airflow\.api\.auth\.backend\.session$"),
+            "3.0",
+            "The default auth backend has changed from 'airflow.api.auth.backend.session' to "
+            "'airflow.providers.fab.auth_manager.api.auth.backend.session' in Airflow 3.0.",
+        ),
+        ("scheduler", "task_instance_heartbeat_timeout"): (
+            re.compile(r"^300$"),
+            "3.0",
+            "Airflow 2.x used 'scheduler_zombie_task_threshold' with a default of 300. In 3.0 this setting "
+            "has been renamed to 'task_instance_heartbeat_timeout' (with the same numeric default), which may "
+            "cause legacy configurations to be ignored.",
+        ),

Review Comment:
   ```suggestion
   ```
   
   Shouldn't this just go to the normal [deprecated_options list](https://github.com/apache/airflow/blob/46759a355d603167535591745c70037d691c866a/airflow/configuration.py#L327) instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1981746747


##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -227,6 +227,21 @@ def message(self) -> str:
         suggestion="Remove TaskContextLogger: Replaced by the Log table for better handling of task log "
         "messages outside the execution context.",
     ),
+    ConfigChange(
+        config=ConfigParameter("logging", "log_filename_template"),

Review Comment:
   Will this hit regardless of the value? Not sure that's what we want.



##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -227,6 +227,21 @@ def message(self) -> str:
         suggestion="Remove TaskContextLogger: Replaced by the Log table for better handling of task log "
         "messages outside the execution context.",
     ),
+    ConfigChange(
+        config=ConfigParameter("logging", "log_filename_template"),
+        suggestion=(
+            "The default value in Airflow 2 (e.g., '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log') "
+            "breaks task logs in the new UI. Please update to the new recommended format as per "
+            "https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#log-filename-template."
+        ),
+        was_deprecated=False,
+    ),
+    ConfigChange(
+        config=ConfigParameter("logging", "log_processor_filename_template"),

Review Comment:
   Looks down a few lines, this config is already removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1982048433


##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -227,6 +227,21 @@ def message(self) -> str:
         suggestion="Remove TaskContextLogger: Replaced by the Log table for better handling of task log "
         "messages outside the execution context.",
     ),
+    ConfigChange(
+        config=ConfigParameter("logging", "log_filename_template"),

Review Comment:
   This is right. I have added option to check default values now.



##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -227,6 +227,21 @@ def message(self) -> str:
         suggestion="Remove TaskContextLogger: Replaced by the Log table for better handling of task log "
         "messages outside the execution context.",
     ),
+    ConfigChange(
+        config=ConfigParameter("logging", "log_filename_template"),
+        suggestion=(
+            "The default value in Airflow 2 (e.g., '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log') "
+            "breaks task logs in the new UI. Please update to the new recommended format as per "
+            "https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#log-filename-template."
+        ),
+        was_deprecated=False,
+    ),
+    ConfigChange(
+        config=ConfigParameter("logging", "log_processor_filename_template"),

Review Comment:
   removed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1994958554


##########
airflow/cli/cli_config.py:
##########
@@ -791,6 +791,16 @@ def string_lower_type(val):
     ("--section",),
     help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),

Review Comment:
   We do want upgrading by default, these are known bad values. I was just pointing out that doing thigns in config lint would never find/warn/upgrade anything - it's already too late.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1993236205


##########
airflow/cli/cli_config.py:
##########
@@ -791,6 +791,16 @@ def string_lower_type(val):
     ("--section",),
     help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),

Review Comment:
   @uranusjr any suggestion on prefix i could add to the name?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on PR #47371:
URL: https://github.com/apache/airflow/pull/47371#issuecomment-2723792484

   Looking closer, this duplicates the functionality of [deprecated_values](https://github.com/apache/airflow/blob/cd1340bba5d801ae23619453016eec68019ba5d2/airflow/configuration.py#L355). We can just update that.
   
   I've opened #47761 to do that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 closed pull request #47371: Handle default values from Airflow 2 gracefully
URL: https://github.com/apache/airflow/pull/47371


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1994448649


##########
airflow/cli/cli_config.py:
##########
@@ -791,6 +791,16 @@ def string_lower_type(val):
     ("--section",),
     help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),

Review Comment:
   Not sure. Maybe “patch”?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on PR #47371:
URL: https://github.com/apache/airflow/pull/47371#issuecomment-2723836360

   Closing this as discussed with @jedcunningham as per https://github.com/apache/airflow/pull/47371#issuecomment-2723792484


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1992999284


##########
airflow/cli/cli_config.py:
##########
@@ -791,6 +791,16 @@ def string_lower_type(val):
     ("--section",),
     help="The section name",
 )
+ARG_LINT_CONFIG_UPGRADE_DEFAULTS = Arg(
+    ("--upgrade-defaults",),

Review Comment:
   This only “upgrades” them in the current process, right? As far as I can tell the new values are not persisted. I wonder if there is a better verb to describe this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1993087490


##########
airflow/configuration.py:
##########
@@ -403,6 +403,64 @@ def inversed_deprecated_sections(self):
     upgraded_values: dict[tuple[str, str], str]
     """Mapping of (section,option) to the old value that was upgraded"""
 
+    legacy_incompatible_defaults: dict[tuple[str, str], tuple[Pattern, str, str]] = {
+        ("logging", "log_filename_template"): (
+            re.compile(
+                r"^(?:" + re.escape("{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log") + r"|"
+                r"dag_id=\{\s*ti\.dag_id\s*\}/run_id=\{\s*ti\.run_id\s*\}/task_id=\{\s*ti\.task_id\s*\}/"
+                r"\{%%\s*if\s+ti\.map_index\s*>=\s*0\s*%%\}map_index=\{\s*ti\.map_index\s*\}/\{%%\s*endif\s*%%\}"
+                r"attempt=\{\s*try_number\s*\}\.log" + r")$"
+            ),
+            "3.0",
+            "The default value for 'log_filename_template' from Airflow 2 may break task logs in the new UI.",
+        ),
+        ("core", "dag_ignore_file_syntax"): (
+            re.compile(r"^regexp$"),
+            "3.0",
+            "The default value changed from 'regexp' in Airflow 2.x to 'glob' in Airflow 3.0.",
+        ),
+        ("api", "auth_backends"): (
+            re.compile(r"^airflow\.api\.auth\.backend\.session$"),
+            "3.0",
+            "The default auth backend has changed from 'airflow.api.auth.backend.session' to "
+            "'airflow.providers.fab.auth_manager.api.auth.backend.session' in Airflow 3.0.",
+        ),

Review Comment:
   removed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Handle default values from Airflow 2 gracefully [airflow]

Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #47371:
URL: https://github.com/apache/airflow/pull/47371#discussion_r1994897415


##########
airflow/cli/commands/remote_commands/config_command.py:
##########
@@ -546,6 +560,9 @@ def lint_config(args) -> None:
         ):
             lint_issues.append(configuration.message)
 
+        if not args.skip_problematic_default_checks:
+            conf.handle_incompatible_airflow2_defaults(upgrade=args.upgrade_problematic_defaults)
+

Review Comment:
   This was a behaviour that I intentionally added where it automatically upgraded by default. If we want to change this to upgrade only via `airflow config lint`, I can pass `upgrade=False` in configuration.py method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org