You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/06 13:57:34 UTC
[GitHub] [airflow] NickYadance opened a new issue, #23532: Airflow .airflowignore not handling soft link properly.
NickYadance opened a new issue, #23532:
URL: https://github.com/apache/airflow/issues/23532
### Apache Airflow version
2.3.0 (latest released)
### What happened
Soft link and folder under same root folder will be handled as the same relative path. Say i have dags folder which looks like this:
```
-dags:
-- .airflowignore
-- folder
-- soft-links-to-folder -> folder
```
and .airflowignore:
```
folder/
```
both folder and soft-links-to-folder will be ignored.
### What you think should happen instead
only the folder shoulld be ignored. This is the expected behavior in airflow 2.0.4, before i upgraded. The root cause is that both _RegexpIgnoreRule and _GlobIgnoreRule is calling `relative_to` method to get search path.
```
str(test_path.relative_to(rule.base_dir))
```
`relative_to` won't return soft link's real name but folder's name.
```
class _RegexpIgnoreRule(NamedTuple):
"""Typed namedtuple with utility functions for regexp ignore rules"""
pattern: Pattern
base_dir: Path
@staticmethod
def compile(pattern: str, base_dir: Path, definition_file: Path) -> Optional[_IgnoreRule]:
"""Build an ignore rule from the supplied regexp pattern and log a useful warning if it is invalid"""
try:
return _RegexpIgnoreRule(re.compile(pattern), base_dir.resolve())
except re.error as e:
log.warning("Ignoring invalid regex '%s' from %s: %s", pattern, definition_file, e)
return None
@staticmethod
def match(path: Path, rules: List[_IgnoreRule]) -> bool:
"""Match a list of ignore rules against the supplied path"""
test_path: Path = path.resolve()
for rule in rules:
if not isinstance(rule, _RegexpIgnoreRule):
raise ValueError(f"_RegexpIgnoreRule cannot match rules of type: {type(rule)}")
if rule.pattern.search(str(test_path.relative_to(rule.base_dir))) is not None:
return True
return False
```
### How to reproduce
easy to reproduce steps, simply construct folder like below:
```
-dags:
-- .airflowignore
-- folder
-- soft-links-to-folder -> folder
```
ignore file:
```
folder/
```
### Operating System
ubuntu
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #23532: Airflow .airflowignore not handling soft link properly.
URL: https://github.com/apache/airflow/issues/23532
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] NickYadance commented on issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
NickYadance commented on issue #23532:
URL: https://github.com/apache/airflow/issues/23532#issuecomment-1119713350
> Here is a sample test case of the report. I guess this is more with the fact `resolve` is called before match that will resolve symlink to the original folder here. As per report "soft-links-to-folder" will resolve to "folder" and get ignored
>
> https://github.com/apache/airflow/blob/6cc41abf6912fd2705b9ef7cf368c888c43c8af8/airflow/utils/file.py#L68
>
> Test case passes before changes in #22051 . cc : @ianbuss
>
> ```python
> # tests/plugins/test_plugin_ignore.py
> def test_symlink_not_ignored(self):
> shutil.rmtree(self.test_dir)
> self.test_dir = tempfile.mkdtemp(prefix="onotole")
> source = os.path.join(self.test_dir, "folder")
> target = os.path.join(self.test_dir, "symlink")
> py_file = os.path.join(source, "hello_world.py")
> ignore_file = os.path.join(self.test_dir, ".airflowignore")
> os.mkdir(source)
> os.symlink(source, target)
>
> with open(ignore_file, 'w') as f:
> f.write("folder")
>
> with open(py_file, 'w') as f:
> f.write("print('hello world')")
>
>
> ignore_list_file = ".airflowignore"
> found = []
>
> for path in find_path_from_directory(self.test_dir, ignore_list_file):
> found.append(path)
>
> assert os.path.join(self.test_dir, "symlink", "hello_world.py") in found
> ```
Correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ianbuss commented on issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
ianbuss commented on issue #23532:
URL: https://github.com/apache/airflow/issues/23532#issuecomment-1119800719
Have prepped a simple initial PR which should hopefully restore the original behaviour (and includes the test case provided by @tirkarthi - thanks!) but would be good to get some additional eyes on it. If we want to make larger changes to the symlink handling that should perhaps be a future PR with further thought? Depends on the timeline of 2.3.1 I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #23532:
URL: https://github.com/apache/airflow/issues/23532#issuecomment-1119651925
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] NickYadance commented on issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
NickYadance commented on issue #23532:
URL: https://github.com/apache/airflow/issues/23532#issuecomment-1119755802
@ianbuss sure , check this:
<img width="906" alt="image" src="https://user-images.githubusercontent.com/10060849/167166752-061b91af-ee7f-4a7f-a883-0f20b2d27e84.png">
Can't find a simple way to work around this. Maybe try to delete soft link manually after git-sync is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tirkarthi commented on issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
tirkarthi commented on issue #23532:
URL: https://github.com/apache/airflow/issues/23532#issuecomment-1119710494
Here is a sample test case of the report. I guess this is more with the fact `resolve` is called before match that will resolve symlink to the original folder here. As per report "soft-links-to-folder" will resolve to "folder" and get ignored
https://github.com/apache/airflow/blob/6cc41abf6912fd2705b9ef7cf368c888c43c8af8/airflow/utils/file.py#L68
Test case passes before changes in https://github.com/apache/airflow/pull/22051 . cc : @ianbuss
```python
# tests/plugins/test_plugin_ignore.py
def test_symlink_not_ignored(self):
shutil.rmtree(self.test_dir)
self.test_dir = tempfile.mkdtemp(prefix="onotole")
source = os.path.join(self.test_dir, "folder")
target = os.path.join(self.test_dir, "symlink")
py_file = os.path.join(source, "hello_world.py")
ignore_file = os.path.join(self.test_dir, ".airflowignore")
os.mkdir(source)
os.symlink(source, target)
with open(ignore_file, 'w') as f:
f.write("folder")
with open(py_file, 'w') as f:
f.write("print('hello world')")
ignore_list_file = ".airflowignore"
found = []
for path in find_path_from_directory(self.test_dir, ignore_list_file):
found.append(path)
assert os.path.join(self.test_dir, "symlink", "hello_world.py") in found
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ianbuss commented on issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
ianbuss commented on issue #23532:
URL: https://github.com/apache/airflow/issues/23532#issuecomment-1119770040
OK, and you have a `.airflowignore` that has `repo.git/` or similar?
I guess there's a wider question here about whether Airflow should be following symlinks at all as it walks the directory tree. I think in some circumstances it makes sense to do that, where it refers to targets out of the dag directory tree, like for example:
```
dags/
team1 -> /mnt/share/team1
team2 -> /git/team2
```
Longer term, I'm less sure of the utility of following "local" symlinks to targets within the same directory structure as it will result in the same files being found more than once and you have to craft the ignore files to ignore either the symlinked path or the real path. We could update the logic to ignore symlinks that are "in tree". Curious what you think @ashb , @kaxil , @potiuk ?
But for now we should try to restore the behaviour to pre-2.3.0 as it is a regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23532:
URL: https://github.com/apache/airflow/issues/23532#issuecomment-1119781028
Agree soft-links in tree are bad idea that will lead to duplication which is never good in this case. But soft-links are also dangerous for another reason - bacause of potential recursion problem which are difficult to diagnose (unless we protect against them). But also they are useful to "cut the cake" differently so to speak (i.e. have different hierarchy in DAGs and in your original repo where you keep them.
I'd say we should restore the symlink behaviour, allow the symlinks but protect against in-tree and recursion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] NickYadance commented on issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
NickYadance commented on issue #23532:
URL: https://github.com/apache/airflow/issues/23532#issuecomment-1119706435
The background is that i use kubernetes/git-sync to sync git into the dag folder which creates the soft link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ianbuss commented on issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
ianbuss commented on issue #23532:
URL: https://github.com/apache/airflow/issues/23532#issuecomment-1119721967
Yep, you're right @tirkarthi this looks like something we need to handle better and expand the unit tests to cover expected cases. @NickYadance can you give a sample directory outline (with file, dirs and symlinks) when git-sync is in the picture?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ianbuss commented on issue #23532: Airflow .airflowignore not handling soft link properly.
Posted by GitBox <gi...@apache.org>.
ianbuss commented on issue #23532:
URL: https://github.com/apache/airflow/issues/23532#issuecomment-1119736848
Hopefully this can be relatively easily fixed with a change to `absolute()` rather than `resolve()` on the path but we should set up a realistic test case for the git-sync use case as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org