You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Desdroid (via GitHub)" <gi...@apache.org> on 2025/10/17 11:44:35 UTC

[I] Airflowignore glob pattern with negation not correctly applied [airflow]

Desdroid opened a new issue, #56766:
URL: https://github.com/apache/airflow/issues/56766

   ### Apache Airflow version
   
   3.1.0
   
   ### If "Other Airflow 2/3 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   We have an .airflowignore file with a glob pattern that uses negation. 
   e.g. 
   ```
   !subdir1/*.py
   ```
   This .airflowignore should only parse dags directly in the `subdir1` and ignore all other files.
   
   The glob pattern is not applied as expected, whereas online tools show that the pattern should work as expected: 
   
   https://www.digitalocean.com/community/tools/glob?comments=true&glob=%21subdir1%2F%2A.py&matches=false&tests=test_notload.py&tests=subdir1%2Ftest_load_sub1.py&tests=test_notload_sub.py&tests=subdir1%2Futils%2Ftest_notload_utils.py
   
   
   
   
   ### What you think should happen instead?
   
   Airflowignore should respect the negated glob pattern.
   
   ### How to reproduce
   
   This is an adapted unit test for `airflow-core/tests/unit/plugins/test_plugin_ignore.py` that shows the error
   
   ```python
   from __future__ import annotations
   
   from pathlib import Path
   from unittest.mock import patch
   
   from airflow import settings
   from airflow.utils.file import find_path_from_directory
   
   def populate_dir_for_negation(root_path):
       """
       Create all directories and files that should be ignored. And set base path.
       """
       # Temp dir name includes an ignored token "not", but it shouldn't matter since it's in the base path.
       test_dir = root_path / "negationtest"
       plugin_folder_path = test_dir / "test_negated_ignore"
       plugin_folder_path.mkdir(parents=True)
       for name in ("subdir1", "subdir2", "subdir1/utils"):
           (plugin_folder_path / name).mkdir(parents=True)
       files_content = [
           ["test_notload.py", 'raise Exception("This file should have been ignored!")'],
           [".airflowignore_glob", "#ignore test\n!subdir1/*.py"],
           ["subdir1/test_load_sub1.py", "#Should not be ignored file"],
           ["test_notload_sub.py", 'raise Exception("This file should have been ignored!")'],
           ["subdir2/test_shouldignore.py", 'raise Exception("This file should have been ignored!")'],
           ["subdir1/utils/test_notload_utils.py", 'raise Exception("This file should have been ignored!")'],
       ]
       for file_path, content in files_content:
           (plugin_folder_path / file_path).write_text(content)
       patch.object(settings, "PLUGINS_FOLDER", return_value=plugin_folder_path)
       return plugin_folder_path
   
   
   class TestIgnorePluginFile:
       """
       Test that the .airflowignore work and whether the file is properly ignored.
       """
   
       def test_find_not_should_ignore_path_glob_with_negation(self, tmp_path):
           """
           Test that the .airflowignore glob syntax with negation works and whether the files are properly ignored.
           """
           plugin_folder_path = populate_dir_for_negation(tmp_path)
   
           detected_files = set()
           should_ignore_files = {
               "test_notload_utils.py",
               "test_shouldignore.py",
               "test_notload.py",
               "test_notload_sub.py"
           }
           should_not_ignore_files = {
               "test_load_sub1.py",
           }
           ignore_list_file = ".airflowignore_glob"
           print("-" * 20)
           for file_path in find_path_from_directory(plugin_folder_path, ignore_list_file, "glob"):
               file_path = Path(file_path)
               if file_path.is_file() and file_path.suffix == ".py":
                   detected_files.add(file_path.name)
                   print(file_path)
   
           print("-" * 20)
           print(detected_files)
           assert detected_files == should_not_ignore_files
           assert detected_files.isdisjoint(should_ignore_files)
   ```
   
   ### Operating System
   
   debian
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflowignore glob pattern with negation not correctly applied [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #56766: Airflowignore glob pattern with negation not correctly applied
URL: https://github.com/apache/airflow/issues/56766


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflowignore glob pattern with negation not correctly applied [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #56766:
URL: https://github.com/apache/airflow/issues/56766#issuecomment-3415991424

   cool !
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflowignore glob pattern with negation not correctly applied [airflow]

Posted by "Desdroid (via GitHub)" <gi...@apache.org>.
Desdroid commented on issue #56766:
URL: https://github.com/apache/airflow/issues/56766#issuecomment-3415727977

   Screw that in the test I forgot the general ignore pattern. It's solved by #56509 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflowignore glob pattern with negation not correctly applied [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #56766:
URL: https://github.com/apache/airflow/issues/56766#issuecomment-3415509409

   Duploicate of https://github.com/apache/airflow/issues/56299 . Can you please look at the  PR linked in the issue @Desdroid and apply it locally and check if it solves your issue - see https://github.com/apache/airflow/pull/56509 for details - simply patching airflow installation locally and checking if your problem is solved ?  That would be highly appreciated as the author of the PR is looking for confirmation that their solution is solving the problem? This would be great way of giving back to the community for the free software you have - by providing your feedback


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflowignore glob pattern with negation not correctly applied [airflow]

Posted by "Desdroid (via GitHub)" <gi...@apache.org>.
Desdroid closed issue #56766: Airflowignore glob pattern with negation not correctly applied
URL: https://github.com/apache/airflow/issues/56766


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org