You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/15 21:48:55 UTC
[GitHub] [airflow] ashb opened a new pull request, #24486: WIP: Lazy import and do less at import time
ashb opened a new pull request, #24486:
URL: https://github.com/apache/airflow/pull/24486
This may be "overkill" for the benefit it actually gives.
Goal here is to make the healthchecks for triggerer/scheduler quicker by having to do less
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
kaxil commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1213375112
> Shall I resume this and get the SQLA model loading working do you think?
Yup, lets try it out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r963592465
##########
airflow/jobs/__init__.py:
##########
@@ -16,8 +16,3 @@
# specific language governing permissions and limitations
# under the License.
#
-import airflow.jobs.backfill_job
Review Comment:
Found a good solution to this that I think is foolproof https://github.com/apache/airflow/pull/24486/commits/685498fa0800c3ca0e51ac41cbd3e863bdab4da6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #24486: Lazily import many modules to improve import speed
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r964901587
##########
airflow/models/__init__.py:
##########
@@ -78,3 +50,94 @@
"XCom",
"clear_task_instances",
]
+
+
+def import_all_models():
+ for name in __lazy_imports:
+ __getattr__(name)
+
+ import airflow.jobs.backfill_job
+ import airflow.jobs.base_job
+ import airflow.jobs.local_task_job
+ import airflow.jobs.scheduler_job
+ import airflow.jobs.triggerer_job
+ import airflow.models.dagwarning
+ import airflow.models.dataset
+ import airflow.models.serialized_dag
+ import airflow.models.tasklog
+
+
+def __getattr__(name):
+ # PEP-562: Lazy loaded attributes on python modules
+ path = __lazy_imports.get(name)
+ if not path:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ from airflow.utils.module_loading import import_string
+
+ val = import_string(f'{path}.{name}')
+ # Store for next time
+ globals()[name] = val
+ return val
+
+
+__lazy_imports = {
+ 'DAG': 'airflow.models.dag',
+ 'ID_LEN': 'airflow.models.base',
+ 'XCOM_RETURN_KEY': 'airflow.models.xcom',
+ 'Base': 'airflow.models.base',
+ 'BaseOperator': 'airflow.models.baseoperator',
+ 'BaseOperatorLink': 'airflow.models.baseoperator',
+ 'Connection': 'airflow.models.connection',
+ 'DagBag': 'airflow.models.dagbag',
+ 'DagModel': 'airflow.models.dag',
+ 'DagPickle': 'airflow.models.dagpickle',
+ 'DagRun': 'airflow.models.dagrun',
+ 'DagTag': 'airflow.models.dag',
+ 'DbCallbackRequest': 'airflow.models.db_callback_request',
+ 'ImportError': 'airflow.models.errors',
+ 'Log': 'airflow.models.log',
+ 'MappedOperator': 'airflow.models.mappedoperator',
+ 'Operator': 'airflow.models.operator',
+ 'Param': 'airflow.models.param',
+ 'Pool': 'airflow.models.pool',
+ 'RenderedTaskInstanceFields': 'airflow.models.renderedtifields',
+ 'SkipMixin': 'airflow.models.skipmixin',
+ 'SlaMiss': 'airflow.models.slamiss',
+ 'TaskFail': 'airflow.models.taskfail',
+ 'TaskInstance': 'airflow.models.taskinstance',
+ 'TaskReschedule': 'airflow.models.taskreschedule',
+ 'Trigger': 'airflow.models.trigger',
+ 'Variable': 'airflow.models.variable',
+ 'XCom': 'airflow.models.xcom',
+ 'clear_task_instances': 'airflow.models.taskinstance',
+}
+
+STATICA_HACK = True
+globals()['kcah_acitats'[::-1].upper()] = False
+if STATICA_HACK: # pragma: no cover
Review Comment:
Yup this works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1156980481
/cc @ianbuss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #24486: Lazily import many modules to improve import speed
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r963995828
##########
airflow/utils/cli.py:
##########
@@ -156,17 +153,6 @@ def _build_metrics(func_name, namespace):
metrics['execution_date'] = tmp_dic.get('execution_date')
metrics['host_name'] = socket.gethostname()
- extra = json.dumps({k: metrics[k] for k in ('host_name', 'full_command')})
- log = Log(
- event=f'cli_{func_name}',
- task_instance=None,
- owner=metrics['user'],
- extra=extra,
- task_id=metrics.get('task_id'),
- dag_id=metrics.get('dag_id'),
- execution_date=metrics.get('execution_date'),
- )
- metrics['log'] = log
Review Comment:
So this metric is lost forever…? (although I don’t really understand how this works)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a diff in pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
mik-laj commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r899685949
##########
airflow/jobs/__init__.py:
##########
@@ -16,8 +16,3 @@
# specific language governing permissions and limitations
# under the License.
#
-import airflow.jobs.backfill_job
Review Comment:
It is a bit risky. See: https://github.com/apache/airflow/pull/19185
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1158635171
> I really hope one day https://peps.python.org/pep-0690/ will be in place so that we don't have to do all those any more.
Yeah, that would be nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #24486: Lazily import many modules to improve import speed
Posted by GitBox <gi...@apache.org>.
ashb merged PR #24486:
URL: https://github.com/apache/airflow/pull/24486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1201862865
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1202280944
Shall I resume this and get the SQLA model loading working do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #24486: Lazily import many modules to improve import speed
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r964001199
##########
airflow/utils/cli.py:
##########
@@ -156,17 +153,6 @@ def _build_metrics(func_name, namespace):
metrics['execution_date'] = tmp_dic.get('execution_date')
metrics['host_name'] = socket.gethostname()
- extra = json.dumps({k: metrics[k] for k in ('host_name', 'full_command')})
- log = Log(
- event=f'cli_{func_name}',
- task_instance=None,
- owner=metrics['user'],
- extra=extra,
- task_id=metrics.get('task_id'),
- dag_id=metrics.get('dag_id'),
- execution_date=metrics.get('execution_date'),
- )
- metrics['log'] = log
Review Comment:
It was only used to write the Log object elsewhere. So I just created it there instead.
("metric" was an odd name for this variable in the first place)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r963454872
##########
airflow/serialization/json_schema.py:
##########
@@ -19,14 +19,15 @@
"""jsonschema for validating serialized DAG and operator."""
import pkgutil
-from typing import Iterable
-
-import jsonschema
+from typing import TYPE_CHECKING, Iterable
from airflow.exceptions import AirflowException
from airflow.settings import json
from airflow.typing_compat import Protocol
+if TYPE_CHECKING:
+ import jsonschema
Review Comment:
A schema validation library being slow… I guess that’s how third-party libraries work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r963447734
##########
airflow/serialization/json_schema.py:
##########
@@ -19,14 +19,15 @@
"""jsonschema for validating serialized DAG and operator."""
import pkgutil
-from typing import Iterable
-
-import jsonschema
+from typing import TYPE_CHECKING, Iterable
from airflow.exceptions import AirflowException
from airflow.settings import json
from airflow.typing_compat import Protocol
+if TYPE_CHECKING:
+ import jsonschema
Review Comment:
It was high up in the list of slow imports!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1238140077
I tried adding a `airflow/models/__init__.pyi` containing this:
```python
# MyPy doesn't/can't recoginze the PEP 562 style lazy imports, so we have to
# tell it about the imports
from .dag import DAG
from .base import ID_LEN
from .xcom import XCOM_RETURN_KEY
from .base import Base
from .baseoperator import BaseOperator
from .baseoperator import BaseOperatorLink
from .connection import Connection
from .dagbag import DagBag
from .dag import DagModel
from .dagpickle import DagPickle
from .dagrun import DagRun
from .dag import DagTag
from .db_callback_request import DbCallbackRequest
from .errors import ImportError
from .log import Log
from .mappedoperator import MappedOperator
from .operator import Operator
from .param import Param
from .pool import Pool
from .renderedtifields import RenderedTaskInstanceFields
from .skipmixin import SkipMixin
from .slamiss import SlaMiss
from .taskfail import TaskFail
from .taskinstance import TaskInstance
from .taskreschedule import TaskReschedule
from .trigger import Trigger
from .variable import Variable
from .xcom import XCom
from .taskinstance import clear_task_instances
def import_all_models(): ...
```
But I ended up with loads of errors along the lines of:
```
airflow/www/views.py:109: error: Module "airflow.models" has no attribute "Connection" [attr-defined]
```
I wonder what is going on ... Any ideas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r899897530
##########
airflow/jobs/__init__.py:
##########
@@ -16,8 +16,3 @@
# specific language governing permissions and limitations
# under the License.
#
-import airflow.jobs.backfill_job
Review Comment:
@mik-laj Yeah I had a feeling something like that might happen - hence why this is in draft and I've got the "load_all_models" fn in airflow.models
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1201986536
I recently listened to some of the folks behind PEP 690 discuss lazy imports and came away with the impression that even after it lands Airflow is highly likely not be able to use it. Lazy imports has a lot of implications and PEP 690 quite heavy-handedly make lazy imports the default, and Airflow has so many dependencies I can easily it might _never_ be able to be fully compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #24486: Lazily import many modules to improve import speed
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r964413402
##########
airflow/models/__init__.py:
##########
@@ -78,3 +50,94 @@
"XCom",
"clear_task_instances",
]
+
+
+def import_all_models():
+ for name in __lazy_imports:
+ __getattr__(name)
+
+ import airflow.jobs.backfill_job
+ import airflow.jobs.base_job
+ import airflow.jobs.local_task_job
+ import airflow.jobs.scheduler_job
+ import airflow.jobs.triggerer_job
+ import airflow.models.dagwarning
+ import airflow.models.dataset
+ import airflow.models.serialized_dag
+ import airflow.models.tasklog
+
+
+def __getattr__(name):
+ # PEP-562: Lazy loaded attributes on python modules
+ path = __lazy_imports.get(name)
+ if not path:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ from airflow.utils.module_loading import import_string
+
+ val = import_string(f'{path}.{name}')
+ # Store for next time
+ globals()[name] = val
+ return val
+
+
+__lazy_imports = {
+ 'DAG': 'airflow.models.dag',
+ 'ID_LEN': 'airflow.models.base',
+ 'XCOM_RETURN_KEY': 'airflow.models.xcom',
+ 'Base': 'airflow.models.base',
+ 'BaseOperator': 'airflow.models.baseoperator',
+ 'BaseOperatorLink': 'airflow.models.baseoperator',
+ 'Connection': 'airflow.models.connection',
+ 'DagBag': 'airflow.models.dagbag',
+ 'DagModel': 'airflow.models.dag',
+ 'DagPickle': 'airflow.models.dagpickle',
+ 'DagRun': 'airflow.models.dagrun',
+ 'DagTag': 'airflow.models.dag',
+ 'DbCallbackRequest': 'airflow.models.db_callback_request',
+ 'ImportError': 'airflow.models.errors',
+ 'Log': 'airflow.models.log',
+ 'MappedOperator': 'airflow.models.mappedoperator',
+ 'Operator': 'airflow.models.operator',
+ 'Param': 'airflow.models.param',
+ 'Pool': 'airflow.models.pool',
+ 'RenderedTaskInstanceFields': 'airflow.models.renderedtifields',
+ 'SkipMixin': 'airflow.models.skipmixin',
+ 'SlaMiss': 'airflow.models.slamiss',
+ 'TaskFail': 'airflow.models.taskfail',
+ 'TaskInstance': 'airflow.models.taskinstance',
+ 'TaskReschedule': 'airflow.models.taskreschedule',
+ 'Trigger': 'airflow.models.trigger',
+ 'Variable': 'airflow.models.variable',
+ 'XCom': 'airflow.models.xcom',
+ 'clear_task_instances': 'airflow.models.taskinstance',
+}
+
+STATICA_HACK = True
+globals()['kcah_acitats'[::-1].upper()] = False
+if STATICA_HACK: # pragma: no cover
Review Comment:
Would `TYPE_CHECKING` not work here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #24486: Lazily import many modules to improve import speed
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r964889813
##########
airflow/models/__init__.py:
##########
@@ -78,3 +50,94 @@
"XCom",
"clear_task_instances",
]
+
+
+def import_all_models():
+ for name in __lazy_imports:
+ __getattr__(name)
+
+ import airflow.jobs.backfill_job
+ import airflow.jobs.base_job
+ import airflow.jobs.local_task_job
+ import airflow.jobs.scheduler_job
+ import airflow.jobs.triggerer_job
+ import airflow.models.dagwarning
+ import airflow.models.dataset
+ import airflow.models.serialized_dag
+ import airflow.models.tasklog
+
+
+def __getattr__(name):
+ # PEP-562: Lazy loaded attributes on python modules
+ path = __lazy_imports.get(name)
+ if not path:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ from airflow.utils.module_loading import import_string
+
+ val = import_string(f'{path}.{name}')
+ # Store for next time
+ globals()[name] = val
+ return val
+
+
+__lazy_imports = {
+ 'DAG': 'airflow.models.dag',
+ 'ID_LEN': 'airflow.models.base',
+ 'XCOM_RETURN_KEY': 'airflow.models.xcom',
+ 'Base': 'airflow.models.base',
+ 'BaseOperator': 'airflow.models.baseoperator',
+ 'BaseOperatorLink': 'airflow.models.baseoperator',
+ 'Connection': 'airflow.models.connection',
+ 'DagBag': 'airflow.models.dagbag',
+ 'DagModel': 'airflow.models.dag',
+ 'DagPickle': 'airflow.models.dagpickle',
+ 'DagRun': 'airflow.models.dagrun',
+ 'DagTag': 'airflow.models.dag',
+ 'DbCallbackRequest': 'airflow.models.db_callback_request',
+ 'ImportError': 'airflow.models.errors',
+ 'Log': 'airflow.models.log',
+ 'MappedOperator': 'airflow.models.mappedoperator',
+ 'Operator': 'airflow.models.operator',
+ 'Param': 'airflow.models.param',
+ 'Pool': 'airflow.models.pool',
+ 'RenderedTaskInstanceFields': 'airflow.models.renderedtifields',
+ 'SkipMixin': 'airflow.models.skipmixin',
+ 'SlaMiss': 'airflow.models.slamiss',
+ 'TaskFail': 'airflow.models.taskfail',
+ 'TaskInstance': 'airflow.models.taskinstance',
+ 'TaskReschedule': 'airflow.models.taskreschedule',
+ 'Trigger': 'airflow.models.trigger',
+ 'Variable': 'airflow.models.variable',
+ 'XCom': 'airflow.models.xcom',
+ 'clear_task_instances': 'airflow.models.taskinstance',
+}
+
+STATICA_HACK = True
+globals()['kcah_acitats'[::-1].upper()] = False
+if STATICA_HACK: # pragma: no cover
Review Comment:
oh yes, probably! Testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1238023509
I wonder why this change causes these mypy errors to appear:
```
tests/system/providers/dbt/cloud/example_dbt_cloud.py:80: error: Unsupported
operand types for >> ("List[object]" and "EmptyOperator") [operator]
[get_run_results_artifact, job_run_sensor] >> end
^
tests/system/providers/microsoft/azure/example_adf_run_pipeline.py:76: error:
Unsupported operand types for >> ("List[object]" and "EmptyOperator")
[operator]
[run_pipeline1, pipeline_run_sensor] >> end
^
Found 2 errors in 2 files (checked 2258 source files)
Run mypy for providers.................................................................Failed
- hook id: run-mypy
- exit code: 1
airflow/providers/atlassian/jira/operators/jira.py:70: error: Extra argument
"conf" from **args for "execute" of "JiraOperator" [misc]
resource = self.get_jira_resource_method.execute(*...
^
airflow/providers/qubole/operators/qubole.py:244: error: Cannot determine type
of "on_failure_callback" [has-type]
if self.on_failure_callback is None:
^
airflow/providers/qubole/operators/qubole.py:247: error: Cannot determine type
of "on_retry_callback" [has-type]
if self.on_retry_callback is None:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1237120423
Wow mypy is confusing/wrong sometimes:
```
airflow/example_dags/example_latest_only_with_trigger.py:39: error: Unexpected
keyword argument "task_id" for "LatestOnlyOperator" [call-arg]
latest_only = LatestOnlyOperator(task_id='latest_only')
^
airflow/utils/log/logging_mixin.py: note: "LatestOnlyOperator" defined here
airflow/example_dags/example_latest_only.py:34: error: Unexpected keyword
argument "task_id" for "LatestOnlyOperator" [call-arg]
latest_only = LatestOnlyOperator(task_id='latest_only')
^
airflow/utils/log/logging_mixin.py: note: "LatestOnlyOperator" defined here
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1157406094
I really hope one day https://peps.python.org/pep-0690/ will be in place so that we don't have to do all those any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1202358879
I think it might be worth it. Would be nice to see how much we gain though in those casese
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1202265600
> I recently listened to some of the folks behind PEP 690 discuss lazy imports and came away with the impression that even after it lands Airflow is highly likely not be able to use it. Lazy imports has a lot of implications and PEP 690 quite heavy-handedly make lazy imports the default, and Airflow has so many dependencies I can easily it might _never_ be able to be fully compatible.
Yeah. We've probably listened to the same Talk Python episode :). I think it's not even likely to have PEP 690 around in 3.10 or maybe 3.11. Unless we roll our own sleeves up and help. I think it would be great to actually HELP in PEP 690 by making Airlfow works with lazy imports (maybe by improving PEP 690 implementation as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #24486:
URL: https://github.com/apache/airflow/pull/24486#discussion_r963423076
##########
airflow/serialization/json_schema.py:
##########
@@ -19,14 +19,15 @@
"""jsonschema for validating serialized DAG and operator."""
import pkgutil
-from typing import Iterable
-
-import jsonschema
+from typing import TYPE_CHECKING, Iterable
from airflow.exceptions import AirflowException
from airflow.settings import json
from airflow.typing_compat import Protocol
+if TYPE_CHECKING:
+ import jsonschema
Review Comment:
Does this change much?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1237975398
I've found an easy way to make sure all models are loaded when needed:
```
event.listen(sqlalchemy.orm.mapper, "before_configured", import_all_models, once=True)
```
This will ensure that on the first time any model class is queried or instantiated all other models will be loaded, which ensures we don't get the "class not found in registry" error message, but still means imports are "quick".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #24486: Lazily import many modules to improve import speed
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1239707877
Rebasing to check still good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #24486: WIP: Lazy import and do less at import time
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #24486:
URL: https://github.com/apache/airflow/pull/24486#issuecomment-1238040040
I was testing this with `python -X importtime -c 'import airflow.jobs.scheduler_job' 2>import-before-schedjob.log` and [tuna](https://github.com/nschloe/tuna) for display:
Before:
After:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org