You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "zach-overflow (via GitHub)" <gi...@apache.org> on 2025/10/09 20:57:23 UTC

[PR] Fix .airflowignore order precedence [airflow]

zach-overflow opened a new pull request, #56509:
URL: https://github.com/apache/airflow/pull/56509

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   
   Closes: #56299
   
   ## What?
   * Fixes a bug where the order of precedence is opposite of what is expected for gitignore-like glob patterns for `.airflowignore` rules.
   * Add more unit test cases for include/exclude behavior (`/**/*` patterns, individual explicit path includes / excludes).
   * Cleans up the unit tests:
       * Ensures the test ignore/include files names match their expected behavior
       * Add human-readable diff output for test failures
   
   ## Testing
   * Ran `uv run pytest airflow-core/tests/unit/utils --skip-db-tests -n auto` without failure
   * Ran `prek` without failure
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #56509:
URL: https://github.com/apache/airflow/pull/56509#discussion_r2426973200


##########
airflow-core/src/airflow/utils/file.py:
##########
@@ -102,24 +102,21 @@ def compile(pattern: str, base_dir: Path, definition_file: Path) -> _IgnoreRule
             relative_to = definition_file.parent
 
         ignore_pattern = GitWildMatchPattern(pattern)
-        return _GlobIgnoreRule(ignore_pattern, relative_to)
+        return _GlobIgnoreRule(wild_match_pattern=ignore_pattern, relative_to=relative_to)
 
     @staticmethod
     def match(path: Path, rules: list[_IgnoreRule]) -> bool:
-        """Match a list of ignore rules against the supplied path."""
+        """Match a list of ignore rules against the supplied path, accounting for exclusion rules and ordering."""
         matched = False
-        for r in rules:
-            if not isinstance(r, _GlobIgnoreRule):
-                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(r)}")
-            rule: _GlobIgnoreRule = r  # explicit typing to make mypy play nicely
+        for rule in rules:
+            if not isinstance(rule, _GlobIgnoreRule):
+                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(rule)}")
             rel_path = str(path.relative_to(rule.relative_to) if rule.relative_to else path.name)
-            matched = rule.wild_match_pattern.match_file(rel_path) is not None
-
-            if matched:
-                if not rule.wild_match_pattern.include:
-                    return False
-
-                return matched
+            if (
+                rule.wild_match_pattern.include is not None
+                and rule.wild_match_pattern.match_file(rel_path) is not None
+            ):
+                matched = rule.wild_match_pattern.include
 
         return matched

Review Comment:
   Also might be worth checking if https://github.com/apache/airflow/issues/56126 is also fixed (maybe adding a test)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "Desdroid (via GitHub)" <gi...@apache.org>.
Desdroid commented on PR #56509:
URL: https://github.com/apache/airflow/pull/56509#issuecomment-3415614018

   @potiuk pointed me to this PR from #56766 but it unfortunately won't solve the problem I described there.
   I try to ignore all files except those in a specific directory with a glob like `!subdir1/dags/*.py` 
   The glob is working on online validators e.g. [here](https://www.digitalocean.com/community/tools/glob?comments=true&glob=%21subdir1%2F%2A.py&matches=false&tests=test_notload.py&tests=subdir1%2Ftest_load_sub1.py&tests=test_notload_sub.py&tests=subdir2%2Ftest_shouldignore.py&tests=subdir1%2Futils%2Ftest_notload_utils.py): 
   I also provided a test case in the issue - I'll add it here for reference, but I don't know if it belongs to this PR or if this shouldn't be a separate issue - it has been closed as dupe though:
   Content of `airflow-core/tests/unit/plugins/test_plugin_ignore.py`
   ```python
   #
   # Licensed to the Apache Software Foundation (ASF) under one
   # or more contributor license agreements.  See the NOTICE file
   # distributed with this work for additional information
   # regarding copyright ownership.  The ASF licenses this file
   # to you under the Apache License, Version 2.0 (the
   # "License"); you may not use this file except in compliance
   # with the License.  You may obtain a copy of the License at
   #
   #   http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing,
   # software distributed under the License is distributed on an
   # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   # KIND, either express or implied.  See the License for the
   # specific language governing permissions and limitations
   # under the License.
   from __future__ import annotations
   
   from pathlib import Path
   from unittest.mock import patch
   
   from airflow import settings
   from airflow.utils.file import find_path_from_directory
   
   
   def populate_dir(root_path):
       """
       Create all directories and files that should be ignored. And set base path.
       """
       # Temp dir name includes an ignored token "not", but it shouldn't matter since it's in the base path.
       test_dir = root_path / "onotole"
       plugin_folder_path = test_dir / "test_ignore"
       plugin_folder_path.mkdir(parents=True)
       for name in ("subdir1", "subdir2", "subdir3"):
           (plugin_folder_path / name).mkdir()
       files_content = [
           ["test_load.py", "#Should not be ignored file"],
           ["test_notload.py", 'raise Exception("This file should have been ignored!")'],
           [".airflowignore", "#ignore test\nnot\nsubdir2"],
           [".airflowignore_glob", "#ignore test\n**/*not*\nsubdir2/"],
           ["subdir1/.airflowignore", "#ignore test\nnone\n_glob"],
           ["subdir1/.airflowignore_glob", "#ignore test\n*none*"],
           ["subdir1/test_load_sub1.py", "#Should not be ignored file"],
           ["test_notload_sub.py", 'raise Exception("This file should have been ignored!")'],
           ["subdir1/test_noneload_sub1.py", 'raise Exception("This file should have been ignored!")'],
           ["subdir2/test_shouldignore.py", 'raise Exception("This file should have been ignored!")'],
           ["subdir3/test_notload_sub3.py", 'raise Exception("This file should have been ignored!")'],
       ]
       for file_path, content in files_content:
           (plugin_folder_path / file_path).write_text(content)
       patch.object(settings, "PLUGINS_FOLDER", return_value=plugin_folder_path)
       return plugin_folder_path
   
   
   def populate_dir_for_negation(root_path):
       """
       Create all directories and files that should be ignored. And set base path.
       """
       # Temp dir name includes an ignored token "not", but it shouldn't matter since it's in the base path.
       test_dir = root_path / "negationtest"
       plugin_folder_path = test_dir / "test_negated_ignore"
       plugin_folder_path.mkdir(parents=True)
       for name in ("subdir1", "subdir2", "subdir1/utils"):
           (plugin_folder_path / name).mkdir(parents=True)
       files_content = [
           ["test_notload.py", 'raise Exception("This file should have been ignored!")'],
           [".airflowignore_glob", "#ignore test\n!subdir1/*.py"],
           ["subdir1/test_load_sub1.py", "#Should not be ignored file"],
           ["subdir2/test_shouldignore.py", 'raise Exception("This file should have been ignored!")'],
           ["subdir1/utils/test_notload_utils.py", 'raise Exception("This file should have been ignored!")'],
       ]
       for file_path, content in files_content:
           (plugin_folder_path / file_path).write_text(content)
       patch.object(settings, "PLUGINS_FOLDER", return_value=plugin_folder_path)
       print(plugin_folder_path)
       return plugin_folder_path
   
   
   class TestIgnorePluginFile:
       """
       Test that the .airflowignore work and whether the file is properly ignored.
       """
   
       def test_find_not_should_ignore_path_regexp(self, tmp_path):
           """
           Test that the .airflowignore regexp works and whether the files are properly ignored.
           """
           plugin_folder_path = populate_dir(tmp_path)
   
           detected_files = set()
           should_ignore_files = {
               "test_notload.py",
               "test_notload_sub.py",
               "test_noneload_sub1.py",
               "test_shouldignore.py",
               ".airflowignore_glob",
           }
           should_not_ignore_files = {
               "test_load.py",
               "test_load_sub1.py",
           }
           ignore_list_file = ".airflowignore"
           for file_path in find_path_from_directory(plugin_folder_path, ignore_list_file, "regexp"):
               file_path = Path(file_path)
               if file_path.is_file() and file_path.suffix == ".py":
                   detected_files.add(file_path.name)
           assert detected_files == should_not_ignore_files
           assert detected_files.isdisjoint(should_ignore_files)
   
       def test_find_not_should_ignore_path_glob(self, tmp_path):
           """
           Test that the .airflowignore glob syntax works and whether the files are properly ignored.
           """
           plugin_folder_path = populate_dir(tmp_path)
   
           detected_files = set()
           should_ignore_files = {
               "test_notload.py",
               "test_notload_sub.py",
               "test_noneload_sub1.py",
           }
           should_not_ignore_files = {
               "test_load.py",
               "test_load_sub1.py",
               "test_shouldignore.py",  # moved to here because it should not ignore, as we do not ignore all
               # things from subdir 2
           }
           ignore_list_file = ".airflowignore_glob"
           print("-" * 20)
           for file_path in find_path_from_directory(plugin_folder_path, ignore_list_file, "glob"):
               file_path = Path(file_path)
               if file_path.is_file() and file_path.suffix == ".py":
                   detected_files.add(file_path.name)
                   print(file_path)
   
           print("-" * 20)
           print(detected_files)
           assert detected_files == should_not_ignore_files
           assert detected_files.isdisjoint(should_ignore_files)
   
       def test_find_not_should_ignore_path_glob_with_negation(self, tmp_path):
           """
           Test that the .airflowignore glob syntax with negation works and whether the files are properly ignored.
           """
           plugin_folder_path = populate_dir_for_negation(tmp_path)
   
           detected_files = set()
           should_ignore_files = {
               "test_notload_utils.py",
               "test_shouldignore.py",
               "test_notload.py",
               "test_notload_sub.py"
           }
           should_not_ignore_files = {
               "test_load_sub1.py",
           }
           ignore_list_file = ".airflowignore_glob"
           print("-" * 20)
           for file_path in find_path_from_directory(plugin_folder_path, ignore_list_file, "glob"):
               file_path = Path(file_path)
               if file_path.is_file() and file_path.suffix == ".py":
                   detected_files.add(file_path.name)
                   print(file_path)
   
           print("-" * 20)
           print(detected_files)
           assert detected_files == should_not_ignore_files
           assert detected_files.isdisjoint(should_ignore_files)
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #56509:
URL: https://github.com/apache/airflow/pull/56509


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on code in PR #56509:
URL: https://github.com/apache/airflow/pull/56509#discussion_r2422636503


##########
airflow-core/src/airflow/utils/file.py:
##########
@@ -102,24 +102,21 @@ def compile(pattern: str, base_dir: Path, definition_file: Path) -> _IgnoreRule
             relative_to = definition_file.parent
 
         ignore_pattern = GitWildMatchPattern(pattern)
-        return _GlobIgnoreRule(ignore_pattern, relative_to)
+        return _GlobIgnoreRule(wild_match_pattern=ignore_pattern, relative_to=relative_to)
 
     @staticmethod
     def match(path: Path, rules: list[_IgnoreRule]) -> bool:
-        """Match a list of ignore rules against the supplied path."""
+        """Match a list of ignore rules against the supplied path, accounting for exclusion rules and ordering."""
         matched = False
-        for r in rules:
-            if not isinstance(r, _GlobIgnoreRule):
-                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(r)}")
-            rule: _GlobIgnoreRule = r  # explicit typing to make mypy play nicely
+        for rule in rules:
+            if not isinstance(rule, _GlobIgnoreRule):
+                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(rule)}")
             rel_path = str(path.relative_to(rule.relative_to) if rule.relative_to else path.name)
-            matched = rule.wild_match_pattern.match_file(rel_path) is not None
-
-            if matched:
-                if not rule.wild_match_pattern.include:
-                    return False
-
-                return matched
+            if (
+                rule.wild_match_pattern.include is not None
+                and rule.wild_match_pattern.match_file(rel_path) is not None
+            ):
+                matched = rule.wild_match_pattern.include
 
         return matched

Review Comment:
   TBH, I don't know what was the behaviour of our ignore file, but I know that it should have the same behavior as the other ignore files (git, docker, ...).
   
   Let's take [.gitignore](https://git-scm.com/docs/gitignore) as reference:
   > Each line in a gitignore file specifies a pattern. When deciding whether to ignore a path, Git normally checks gitignore patterns from multiple sources, with the following order of precedence, from highest to lowest (within one level of precedence, the last matching pattern decides the outcome)
   
   So since the last rule that match the file decides if it should be included or excluded, and since it's useless to process all the rules if there is a match in one of last rules in the file, why we don't process them in reverse order and return the outcome of the first match:
   
   ```python
       @staticmethod
       def match(path: Path, rules: list[_IgnoreRule]) -> bool:
           """Match a list of ignore rules against the supplied path."""
           for r in reversed(rules):
               if not isinstance(r, _GlobIgnoreRule):
                   raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(r)}")
   
               rule: _GlobIgnoreRule = r  # explicit typing to make mypy play nicely
               rel_path = str(path.relative_to(rule.relative_to) if rule.relative_to else path.name)
   
               if rule.wild_match_pattern.match_file(rel_path) is not None:  # matched
                   return bool(rule.wild_match_pattern.include)  # included
   
           return False
   ```
   
   For your question about `rule.wild_match_pattern.include`, if I remember correctly, some patterns are only a matcher, without supporting inclusion/exclusion logic, and `include` is represented as a `None` in that case. But these patterns are not `GitWildMatchPattern` so we should be safe.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56509:
URL: https://github.com/apache/airflow/pull/56509#issuecomment-3419046209

   ### Backport successfully created: v3-1-test
   
   <table>
                   <tr>
                       <th>Status</th>
                       <th>Branch</th>
                       <th>Result</th>
                   </tr>
                   <tr>
                       <td>✅</td>
                       <td>v3-1-test</td>
                       <td><a href="https://github.com/apache/airflow/pull/56832"><img src="https://img.shields.io/badge/PR-56832-blue" alt="PR Link"></a></td>
                   </tr>
               </table>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #56509:
URL: https://github.com/apache/airflow/pull/56509#issuecomment-3415993762

   Can we resolve remainign problems and merge it in ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "Nataneljpwd (via GitHub)" <gi...@apache.org>.
Nataneljpwd commented on code in PR #56509:
URL: https://github.com/apache/airflow/pull/56509#discussion_r2423340246


##########
airflow-core/src/airflow/utils/file.py:
##########
@@ -102,24 +102,21 @@ def compile(pattern: str, base_dir: Path, definition_file: Path) -> _IgnoreRule
             relative_to = definition_file.parent
 
         ignore_pattern = GitWildMatchPattern(pattern)
-        return _GlobIgnoreRule(ignore_pattern, relative_to)
+        return _GlobIgnoreRule(wild_match_pattern=ignore_pattern, relative_to=relative_to)
 
     @staticmethod
     def match(path: Path, rules: list[_IgnoreRule]) -> bool:
-        """Match a list of ignore rules against the supplied path."""
+        """Match a list of ignore rules against the supplied path, accounting for exclusion rules and ordering."""
         matched = False
-        for r in rules:
-            if not isinstance(r, _GlobIgnoreRule):
-                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(r)}")
-            rule: _GlobIgnoreRule = r  # explicit typing to make mypy play nicely
+        for rule in rules:
+            if not isinstance(rule, _GlobIgnoreRule):
+                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(rule)}")
             rel_path = str(path.relative_to(rule.relative_to) if rule.relative_to else path.name)
-            matched = rule.wild_match_pattern.match_file(rel_path) is not None
-
-            if matched:
-                if not rule.wild_match_pattern.include:
-                    return False
-
-                return matched
+            if (
+                rule.wild_match_pattern.include is not None
+                and rule.wild_match_pattern.match_file(rel_path) is not None
+            ):
+                matched = rule.wild_match_pattern.include
 
         return matched

Review Comment:
   I am not sure that the reversed would work, as per the pathspec:
   
   > An optional prefix "!" which negates the pattern; **any matching file excluded by a previous pattern will become included again. It is not possible to re-include a file if a parent directory of that file is excluded**. Git doesn’t list excluded directories for performance reasons, so any patterns on contained files have no effect, no matter where they are defined. Put a backslash ("\") in front of the first "!" for patterns that begin with a literal "!", for example, "\!important!.txt".
   
   That means that if a directory was ignored first, but will be included, I am pretty sure it works by git pathspec now, I will take a closer look and examine it again, the behaviour is complex but I looked at git's implantation and it seems to match



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #56509:
URL: https://github.com/apache/airflow/pull/56509#discussion_r2435436787


##########
airflow-core/src/airflow/utils/file.py:
##########
@@ -102,24 +102,21 @@ def compile(pattern: str, base_dir: Path, definition_file: Path) -> _IgnoreRule
             relative_to = definition_file.parent
 
         ignore_pattern = GitWildMatchPattern(pattern)
-        return _GlobIgnoreRule(ignore_pattern, relative_to)
+        return _GlobIgnoreRule(wild_match_pattern=ignore_pattern, relative_to=relative_to)
 
     @staticmethod
     def match(path: Path, rules: list[_IgnoreRule]) -> bool:
-        """Match a list of ignore rules against the supplied path."""
+        """Match a list of ignore rules against the supplied path, accounting for exclusion rules and ordering."""
         matched = False
-        for r in rules:
-            if not isinstance(r, _GlobIgnoreRule):
-                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(r)}")
-            rule: _GlobIgnoreRule = r  # explicit typing to make mypy play nicely
+        for rule in rules:
+            if not isinstance(rule, _GlobIgnoreRule):
+                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(rule)}")
             rel_path = str(path.relative_to(rule.relative_to) if rule.relative_to else path.name)
-            matched = rule.wild_match_pattern.match_file(rel_path) is not None
-
-            if matched:
-                if not rule.wild_match_pattern.include:
-                    return False
-
-                return matched
+            if (
+                rule.wild_match_pattern.include is not None
+                and rule.wild_match_pattern.match_file(rel_path) is not None
+            ):
+                matched = rule.wild_match_pattern.include
 
         return matched

Review Comment:
   Is it possible to double-check if the #56126  is fixed @zach-overflow  ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "Nataneljpwd (via GitHub)" <gi...@apache.org>.
Nataneljpwd commented on code in PR #56509:
URL: https://github.com/apache/airflow/pull/56509#discussion_r2423340338


##########
airflow-core/src/airflow/utils/file.py:
##########
@@ -102,24 +102,21 @@ def compile(pattern: str, base_dir: Path, definition_file: Path) -> _IgnoreRule
             relative_to = definition_file.parent
 
         ignore_pattern = GitWildMatchPattern(pattern)
-        return _GlobIgnoreRule(ignore_pattern, relative_to)
+        return _GlobIgnoreRule(wild_match_pattern=ignore_pattern, relative_to=relative_to)
 
     @staticmethod
     def match(path: Path, rules: list[_IgnoreRule]) -> bool:
-        """Match a list of ignore rules against the supplied path."""
+        """Match a list of ignore rules against the supplied path, accounting for exclusion rules and ordering."""
         matched = False
-        for r in rules:
-            if not isinstance(r, _GlobIgnoreRule):
-                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(r)}")
-            rule: _GlobIgnoreRule = r  # explicit typing to make mypy play nicely
+        for rule in rules:
+            if not isinstance(rule, _GlobIgnoreRule):
+                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(rule)}")
             rel_path = str(path.relative_to(rule.relative_to) if rule.relative_to else path.name)
-            matched = rule.wild_match_pattern.match_file(rel_path) is not None
-
-            if matched:
-                if not rule.wild_match_pattern.include:
-                    return False
-
-                return matched
+            if (
+                rule.wild_match_pattern.include is not None
+                and rule.wild_match_pattern.match_file(rel_path) is not None
+            ):
+                matched = rule.wild_match_pattern.include
 
         return matched

Review Comment:
   I am not sure that the reversed would work, as per the pathspec:
   
   > An optional prefix "!" which negates the pattern; **any matching file excluded by a previous pattern will become included again. It is not possible to re-include a file if a parent directory of that file is excluded**. Git doesn’t list excluded directories for performance reasons, so any patterns on contained files have no effect, no matter where they are defined. Put a backslash ("\") in front of the first "!" for patterns that begin with a literal "!", for example, "\!important!.txt".
   
   That means that if a directory was ignored first, but will be included, I am pretty sure it works by git pathspec now, I will take a closer look and examine it again, the behaviour is complex but I looked at git's implantation and it seems to match



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "Nataneljpwd (via GitHub)" <gi...@apache.org>.
Nataneljpwd commented on code in PR #56509:
URL: https://github.com/apache/airflow/pull/56509#discussion_r2423329273


##########
airflow-core/src/airflow/utils/file.py:
##########
@@ -102,24 +102,21 @@ def compile(pattern: str, base_dir: Path, definition_file: Path) -> _IgnoreRule
             relative_to = definition_file.parent
 
         ignore_pattern = GitWildMatchPattern(pattern)
-        return _GlobIgnoreRule(ignore_pattern, relative_to)
+        return _GlobIgnoreRule(wild_match_pattern=ignore_pattern, relative_to=relative_to)
 
     @staticmethod
     def match(path: Path, rules: list[_IgnoreRule]) -> bool:
-        """Match a list of ignore rules against the supplied path."""
+        """Match a list of ignore rules against the supplied path, accounting for exclusion rules and ordering."""
         matched = False
-        for r in rules:
-            if not isinstance(r, _GlobIgnoreRule):
-                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(r)}")
-            rule: _GlobIgnoreRule = r  # explicit typing to make mypy play nicely
+        for rule in rules:
+            if not isinstance(rule, _GlobIgnoreRule):
+                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(rule)}")
             rel_path = str(path.relative_to(rule.relative_to) if rule.relative_to else path.name)
-            matched = rule.wild_match_pattern.match_file(rel_path) is not None
-
-            if matched:
-                if not rule.wild_match_pattern.include:
-                    return False
-
-                return matched
+            if (
+                rule.wild_match_pattern.include is not None

Review Comment:
   Hello, I have worked on it before, the include is whether we should leave the file if it is true or ignore it ( as if not parsing it ), as per pathspec here
   
   https://github.com/cpburnz/python-pathspec/blob/e4977fc41a36d6cba9529a377ff48bf67d6371af/pathspec/pattern.py#L39
   
   Basically it is set to true if there is an exclamation mark in the first char



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "zach-overflow (via GitHub)" <gi...@apache.org>.
zach-overflow commented on PR #56509:
URL: https://github.com/apache/airflow/pull/56509#issuecomment-3423073944

   > Can we resolve remaining problems and merge it in ?
   
   My bad for the slow followup here. Happy to create a followup PR if desired. I was also not sure regarding the point that @Nataneljpwd raised [here](https://github.com/apache/airflow/pull/56509#discussion_r2423340246). My understanding is that the reverse order suggestion would not handle the quoted gitignore behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "Nataneljpwd (via GitHub)" <gi...@apache.org>.
Nataneljpwd commented on code in PR #56509:
URL: https://github.com/apache/airflow/pull/56509#discussion_r2423340338


##########
airflow-core/src/airflow/utils/file.py:
##########
@@ -102,24 +102,21 @@ def compile(pattern: str, base_dir: Path, definition_file: Path) -> _IgnoreRule
             relative_to = definition_file.parent
 
         ignore_pattern = GitWildMatchPattern(pattern)
-        return _GlobIgnoreRule(ignore_pattern, relative_to)
+        return _GlobIgnoreRule(wild_match_pattern=ignore_pattern, relative_to=relative_to)
 
     @staticmethod
     def match(path: Path, rules: list[_IgnoreRule]) -> bool:
-        """Match a list of ignore rules against the supplied path."""
+        """Match a list of ignore rules against the supplied path, accounting for exclusion rules and ordering."""
         matched = False
-        for r in rules:
-            if not isinstance(r, _GlobIgnoreRule):
-                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(r)}")
-            rule: _GlobIgnoreRule = r  # explicit typing to make mypy play nicely
+        for rule in rules:
+            if not isinstance(rule, _GlobIgnoreRule):
+                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(rule)}")
             rel_path = str(path.relative_to(rule.relative_to) if rule.relative_to else path.name)
-            matched = rule.wild_match_pattern.match_file(rel_path) is not None
-
-            if matched:
-                if not rule.wild_match_pattern.include:
-                    return False
-
-                return matched
+            if (
+                rule.wild_match_pattern.include is not None
+                and rule.wild_match_pattern.match_file(rel_path) is not None
+            ):
+                matched = rule.wild_match_pattern.include
 
         return matched

Review Comment:
   I am not sure that the reversed would work, as per the pathspec:
   
   > An optional prefix "!" which negates the pattern; **any matching file excluded by a previous pattern will become included again. It is not possible to re-include a file if a parent directory of that file is excluded**. Git doesn’t list excluded directories for performance reasons, so any patterns on contained files have no effect, no matter where they are defined. Put a backslash ("\") in front of the first "!" for patterns that begin with a literal "!", for example, "\!important!.txt".
   
   That means that if a directory was ignored first, but will be included, I am pretty sure it works by git pathspec now, I will take a closer look and examine it again, the behaviour is complex but I looked at git's implantation and it seems to match



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix .airflowignore order precedence [airflow]

Posted by "zach-overflow (via GitHub)" <gi...@apache.org>.
zach-overflow commented on code in PR #56509:
URL: https://github.com/apache/airflow/pull/56509#discussion_r2417953463


##########
airflow-core/src/airflow/utils/file.py:
##########
@@ -102,24 +102,21 @@ def compile(pattern: str, base_dir: Path, definition_file: Path) -> _IgnoreRule
             relative_to = definition_file.parent
 
         ignore_pattern = GitWildMatchPattern(pattern)
-        return _GlobIgnoreRule(ignore_pattern, relative_to)
+        return _GlobIgnoreRule(wild_match_pattern=ignore_pattern, relative_to=relative_to)
 
     @staticmethod
     def match(path: Path, rules: list[_IgnoreRule]) -> bool:
-        """Match a list of ignore rules against the supplied path."""
+        """Match a list of ignore rules against the supplied path, accounting for exclusion rules and ordering."""
         matched = False
-        for r in rules:
-            if not isinstance(r, _GlobIgnoreRule):
-                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(r)}")
-            rule: _GlobIgnoreRule = r  # explicit typing to make mypy play nicely
+        for rule in rules:
+            if not isinstance(rule, _GlobIgnoreRule):
+                raise ValueError(f"_GlobIgnoreRule cannot match rules of type: {type(rule)}")
             rel_path = str(path.relative_to(rule.relative_to) if rule.relative_to else path.name)
-            matched = rule.wild_match_pattern.match_file(rel_path) is not None
-
-            if matched:
-                if not rule.wild_match_pattern.include:
-                    return False
-
-                return matched
+            if (
+                rule.wild_match_pattern.include is not None

Review Comment:
   I'd appreciate any input on this condition, as it is a bit unclear what the expected behavior is for the situation when `rule.wild_match_pattern.include` is `None`. The `pathspec.pattern.Pattern` docstring leaves this a bit vague IMO:
   
   > *include* (:class:`bool` or :data:`None`) is whether the matched files
   		should be included (:data:`True`), excluded (:data:`False`), or is a
   		null-operation (:data:`None`)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org