You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2025/12/20 10:39:44 UTC
[PR] Make example DAG import timeout configurable [airflow]
potiuk opened a new pull request, #59664:
URL: https://github.com/apache/airflow/pull/59664
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make example DAG import timeout configurable [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #59664:
URL: https://github.com/apache/airflow/pull/59664#issuecomment-3677830955
Looking at the code I'd propose a lazy import in the hook... am busy atm but can propose an improvement in the afternoon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make example DAG import timeout configurable [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #59664:
URL: https://github.com/apache/airflow/pull/59664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make example DAG import timeout configurable [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #59664:
URL: https://github.com/apache/airflow/pull/59664#issuecomment-3677842458
I see it in the same way too. But 30s in a bag of tests is too less and this makes configuring that possible which is nice. It could also be a quick workaround to identify actual issues vs timeouts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make example DAG import timeout configurable [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #59664:
URL: https://github.com/apache/airflow/pull/59664#issuecomment-3677879166
>
I see it in the same way too. But 30s in a bag of tests is too less and this makes configuring that possible which is nice. It could also be a quick workaround to identify actual issues vs timeouts
Yes - that's also true - with our tests running in parallel and "maxing out" the workers - some operations might take 2x / 3x slower than usual - and that's also a good thing to have because we can identify and mark certain examples as "slow" - and fix them later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make example DAG import timeout configurable [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #59664:
URL: https://github.com/apache/airflow/pull/59664#issuecomment-3677738497
workaround long import time for gen_ai example. The flamegraph from importing it shows that 43% of time is **just** importing experiment_service. This line:
```python
from airflow.providers.google.cloud.operators.vertex_ai.experiment_service import (
CreateExperimentOperator,
DeleteExperimentOperator,
DeleteExperimentRunOperator,
)
```
And vertex_ai is known from taking long to import.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make example DAG import timeout configurable [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #59664:
URL: https://github.com/apache/airflow/pull/59664#issuecomment-3678133521
Okay, I attempted for a while to lazy import in https://github.com/apache/airflow/compare/main...jscheffl:airflow:bugfix/optimize-import-overhead-in-google-aiplatform?expand=1 and finally needed to give-up in the area where @amoghrajesh already extracted to `_get_metrics()` whichh is not as lazy imported as it was attempted, finally there the metrics seem to need to be Python objects and are loaded at time of Operator initialization... and then the full vertex_ai stack is loaded :-( So not so easy to optimize w/o changing/breaking operator interfaces or copying massive boilerplate from Python packaged examples into the Dag itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make example DAG import timeout configurable [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #59664:
URL: https://github.com/apache/airflow/pull/59664#issuecomment-3677841125
> Looking at the code I'd propose a lazy import in the hook... am busy atm but can propose an improvement in the afternoon.
Yep. I did not want to go too much in detail of those - but yes. It's possible :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org