You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Taragolis (via GitHub)" <gi...@apache.org> on 2024/02/27 21:37:44 UTC
[PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Taragolis opened a new pull request, #37757:
URL: https://github.com/apache/airflow/pull/37757
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
This is attempt to prevent use `functools.lru_cache` or `functools.cache` on methods which lead to memory leak, e.g. object can't be garbage collected if it use methods decorated by this decorators.
Simple reproducible example
```python
import functools
class A:
def __init__(self):
self.a = {"a": 1, "b": 2}
@functools.lru_cache(maxsize=None)
def get_val(self, key: str):
return self.a[key]
for _ in range(100):
xxx = A()
xxx.get_val(key="a")
xxx.get_val("a")
xxx.get_val("b")
del xxx
import gc
gc.collect()
a_instances = [o for o in gc.get_objects() if isinstance(o, A)]
assert len(a_instances) == 0, len(a_instances)
# AssertionError: 100
```
Instead of that better to use use `functools.cached_property` (if it could be a property), ruff [`B018` rule](https://docs.astral.sh/ruff/rules/cached-instance-method/#example) suggestion or use some third-party library, e.g. [`methodtools`](https://github.com/youknowone/methodtools/)
```python
import methodtools
class A:
def __init__(self):
self.a = {"a": 1, "b": 2}
@methodtools.lru_cache(maxsize=None)
def get_val(self, key: str):
return self.a[key]
for _ in range(100):
xxx = A()
xxx.get_val(key="a")
xxx.get_val(key="a")
xxx.get_val("b")
del xxx
import gc
gc.collect()
a_instances = [o for o in gc.get_objects() if isinstance(o, A)]
assert len(a_instances) == 0, len(a_instances) # Do not raise assertion error
```
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1505123675
##########
.pre-commit-config.yaml:
##########
@@ -855,7 +855,7 @@ repos:
files: ^dev/breeze/.*$
pass_filenames: false
require_serial: true
- additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml']
+ additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml', 'methodtools']
Review Comment:
Well it was to soon to say that this is a solution, I've create a test for check hashing and got and error in all case
There is different reason for that: __slots__, weakref do not work well with this types, recursion error.
```python
...
def test_hashing(self):
o = ObjectStoragePath(f"file:///tmp/{str(uuid.uuid4())}")
s = set()
for _ in range(10):
s.add(o)
assert len(s) == 1
```
`@functools.lru_cache`
---
```console
def test_hashing(self):
o = ObjectStoragePath(f"file:///tmp/{str(uuid.uuid4())}")
s = set()
for _ in range(10):
> s.add(o)
E RecursionError: maximum recursion depth exceeded while calling a Python object
```
`@methodtools.lru_cache(max_size=None)`
---
```cosnsole
def test_hashing(self):
o = ObjectStoragePath(f"file:///tmp/{str(uuid.uuid4())}")
s = set()
for _ in range(10):
> s.add(o)
E TypeError: unhashable type: 'ObjectStoragePath'
```
Use cached property with name defined in `__slots__`
---
```console
tests/io/test_path.py:None (tests/io/test_path.py)
test_path.py:31: in <module>
from airflow.io.path import ObjectStoragePath
../../airflow/io/path.py:72: in <module>
class ObjectStoragePath(CloudPath):
E ValueError: '_hash' in __slots__ conflicts with class variable
```
Use cached property with name not defined in `__slots__`
---
```console
> raise TypeError(msg) from None
E TypeError: No '__dict__' attribute on 'ObjectStoragePath' instance to cache '_awesome_hash' property.
```
@bolkedebruin I have a couple questions:
1. Do we actually need to make ObjectStoragePath hashable?
2. Do we need to make hash cachable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1547076493
##########
.pre-commit-config.yaml:
##########
@@ -985,6 +985,13 @@ repos:
files: \.py$
exclude: ^.*/.*_vendor/
additional_dependencies: ['rich>=12.4.4']
+ - id: check-compat-cache-on-methods
+ name: Check that compat cache do not use on class methods
+ entry: ./scripts/ci/pre_commit/pre_commit_compat_cache_on_methods.py
Review Comment:
I have a PR to rename the pre-commit scripts so this might need to change later. Not a big problem though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1505062170
##########
.pre-commit-config.yaml:
##########
@@ -855,7 +855,7 @@ repos:
files: ^dev/breeze/.*$
pass_filenames: false
require_serial: true
- additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml']
+ additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml', 'methodtools']
Review Comment:
Indeed. Really bad idea to use lru_cache on class methods.
But... Do we need to add methodtools?
Why don't we replace all those usages with @cached_property actually ?
The only problematic one is I think __hash__ in `io` but it could be rather easily fixed without adding methodtools:
```python
@cached_property
def _hash(self):
return hash(str(self))
def __hash__(self):
return self._hash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37757:
URL: https://github.com/apache/airflow/pull/37757#issuecomment-2032674265
> The problem here with tests that Prod images build do not reflect changes into requirements in `hatch_build.py` and it doesn't install `methodtools`
That's already fixed in #38678 -> I rebased your PR @Taragolis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1547305445
##########
.pre-commit-config.yaml:
##########
@@ -985,6 +985,13 @@ repos:
files: \.py$
exclude: ^.*/.*_vendor/
additional_dependencies: ['rich>=12.4.4']
+ - id: check-compat-cache-on-methods
+ name: Check that compat cache do not use on class methods
+ entry: ./scripts/ci/pre_commit/pre_commit_compat_cache_on_methods.py
Review Comment:
You want to remove `pre_commit_` prefix? Because it is my main desire when I open this folder 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37757:
URL: https://github.com/apache/airflow/pull/37757#issuecomment-2032677469
(and even better - extraction of the common code added then is in #38682 - ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1505078924
##########
.pre-commit-config.yaml:
##########
@@ -855,7 +855,7 @@ repos:
files: ^dev/breeze/.*$
pass_filenames: false
require_serial: true
- additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml']
+ additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml', 'methodtools']
Review Comment:
> But... Do we need to add methodtools?
If we could resolve all 4 cases without it, why not? One case with google provider could be also resolved by cached_property, but I was too lazy for rewrite tests 🤣
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1505077750
##########
.pre-commit-config.yaml:
##########
@@ -855,7 +855,7 @@ repos:
files: ^dev/breeze/.*$
pass_filenames: false
require_serial: true
- additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml']
+ additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml', 'methodtools']
Review Comment:
In case of `__hash__` I do not even know how it work before
```python
import functools
class A:
@functools.lru_cache
def __hash__(self) -> int:
return hash(str(self))
hash(A()) # RecursionError: maximum recursion depth exceeded while calling a Python object
```
With cached property I can't see any problem here, so it could be used here instead of `methodtools`
```python
import functools
class A:
@functools.cached_property
def _hash(self):
return hash(str(self))
def __hash__(self) -> int:
return self._hash
s = set()
a = A()
for _ in range(100):
s.add(a)
assert len(s) == 1
del a
del s
import gc
gc.collect()
a_instances = [o for o in gc.get_objects() if isinstance(o, A)]
assert len(a_instances) == 0, len(a_instances)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #37757:
URL: https://github.com/apache/airflow/pull/37757#issuecomment-2032778319
But this failure also show that docker compose tests can't show useful information in case if API failed because it just fail before show logs from the container. I will add workaround for that later on in the separate PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis merged PR #37757:
URL: https://github.com/apache/airflow/pull/37757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1505128250
##########
.pre-commit-config.yaml:
##########
@@ -855,7 +855,7 @@ repos:
files: ^dev/breeze/.*$
pass_filenames: false
require_serial: true
- additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml']
+ additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml', 'methodtools']
Review Comment:
Old school caching by use class attribute https://github.com/apache/airflow/pull/37757/commits/35a38d8a7154a6dac8cae3569fc20fb8773ab5bf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1505037795
##########
.pre-commit-config.yaml:
##########
@@ -855,7 +855,7 @@ repos:
files: ^dev/breeze/.*$
pass_filenames: false
require_serial: true
- additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml']
+ additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml', 'methodtools']
Review Comment:
I hope this one is acceptable, because I have no idea how to decorate something without import it first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #37757:
URL: https://github.com/apache/airflow/pull/37757#issuecomment-2031281694
The problem here with tests that Prod images build do not reflect changes into requirements in `hatch_build.py` and it doesn't install `methodtools`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1547344094
##########
airflow/models/abstractoperator.py:
##########
@@ -492,7 +492,7 @@ def get_extra_links(self, ti: TaskInstance, link_name: str) -> str | None:
return link.get_link(self.unmap(None), ti.dag_run.logical_date) # type: ignore[misc]
return link.get_link(self.unmap(None), ti_key=ti.key)
- @cache
+ @methodtools.lru_cache(maxsize=None)
Review Comment:
It is an issue exists (by creator) https://github.com/youknowone/methodtools/issues/19
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1547330500
##########
.pre-commit-config.yaml:
##########
@@ -985,6 +985,13 @@ repos:
files: \.py$
exclude: ^.*/.*_vendor/
additional_dependencies: ['rich>=12.4.4']
+ - id: check-compat-cache-on-methods
+ name: Check that compat cache do not use on class methods
+ entry: ./scripts/ci/pre_commit/pre_commit_compat_cache_on_methods.py
Review Comment:
Oh, yeah... I see https://github.com/apache/airflow/pull/38667 🥳
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #37757:
URL: https://github.com/apache/airflow/pull/37757#issuecomment-2032763101
> That's already fixed in https://github.com/apache/airflow/pull/38678 -> I rebased your PR @Taragolis
Hehe, I've also rebased it. DOUBLE REBASE 🤣
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #37757:
URL: https://github.com/apache/airflow/pull/37757#issuecomment-2027785730
WeightRule is enum, and members of enum are singletons, so it is pretty safe to use it there, but in the other cases it still looks like a problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1547343239
##########
.pre-commit-config.yaml:
##########
@@ -985,6 +985,13 @@ repos:
files: \.py$
exclude: ^.*/.*_vendor/
additional_dependencies: ['rich>=12.4.4']
+ - id: check-compat-cache-on-methods
+ name: Check that compat cache do not use on class methods
+ entry: ./scripts/ci/pre_commit/pre_commit_compat_cache_on_methods.py
Review Comment:
Also remove that prefix from script file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1505064063
##########
.pre-commit-config.yaml:
##########
@@ -855,7 +855,7 @@ repos:
files: ^dev/breeze/.*$
pass_filenames: false
require_serial: true
- additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml']
+ additional_dependencies: ['click', 'rich>=12.4.4', 'pyyaml', 'methodtools']
Review Comment:
Or am I missing something :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #37757:
URL: https://github.com/apache/airflow/pull/37757#issuecomment-2027769895
@potiuk I've also found that `ruff` do not detect compat decorator `airflow.compat.functools.cache` and we use it in some methods places into the core, which might cause memory leakage
https://github.com/apache/airflow/blob/8c44bcb50803cad1595a8586058db67d5cc0ce2f/airflow/models/abstractoperator.py#L494-L497
https://github.com/apache/airflow/blob/8c44bcb50803cad1595a8586058db67d5cc0ce2f/airflow/models/mappedoperator.py#L336-L341
https://github.com/apache/airflow/blob/fce3a583348162f655282d032eca654dcb67b497/airflow/models/taskinstance.py#L641-L643
https://github.com/apache/airflow/blob/fce3a583348162f655282d032eca654dcb67b497/airflow/models/taskinstance.py#L674-L678
https://github.com/apache/airflow/blob/fce3a583348162f655282d032eca654dcb67b497/airflow/models/taskinstance.py#L681-L685
https://github.com/apache/airflow/blob/fce3a583348162f655282d032eca654dcb67b497/airflow/models/taskinstance.py#L715-L718
https://github.com/apache/airflow/blob/fce3a583348162f655282d032eca654dcb67b497/airflow/models/taskinstance.py#L727-L730
https://github.com/apache/airflow/blob/0a74928894fb57b0160208262ccacad12da23fc7/airflow/utils/task_group.py#L588-L592
https://github.com/apache/airflow/blob/f60d458dc08a5d5fbe5903fffca8f7b03009f49a/airflow/utils/weight_rule.py#L37-L41
https://github.com/apache/airflow/blob/8c44bcb50803cad1595a8586058db67d5cc0ce2f/airflow/www/views.py#L336-L339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #37757:
URL: https://github.com/apache/airflow/pull/37757#discussion_r1547076978
##########
airflow/models/abstractoperator.py:
##########
@@ -492,7 +492,7 @@ def get_extra_links(self, ti: TaskInstance, link_name: str) -> str | None:
return link.get_link(self.unmap(None), ti.dag_run.logical_date) # type: ignore[misc]
return link.get_link(self.unmap(None), ti_key=ti.key)
- @cache
+ @methodtools.lru_cache(maxsize=None)
Review Comment:
I wonder if methodtools should also have a cache shorthand… that’s a separate issue I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37757:
URL: https://github.com/apache/airflow/pull/37757#issuecomment-2032822547
> But this failure also show that docker compose tests can't show useful information in case if API failed because it just fail before show logs from the container. I will add workaround for that later on in the separate PR
Yep we could dump and upload logs as artifacts from all conainers as we do in case of failure in k8s/regular tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed pull request #37757: Use `methodtools.lru_cache` instead of `functools.lru_cache` in class methods
URL: https://github.com/apache/airflow/pull/37757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org