You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "KoviAnusha (via GitHub)" <gi...@apache.org> on 2025/10/29 05:48:28 UTC
[PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
KoviAnusha opened a new pull request, #57466:
URL: https://github.com/apache/airflow/pull/57466
related: #56735
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476274016
##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -66,6 +66,8 @@
if TYPE_CHECKING:
from sqlalchemy.orm.attributes import InstrumentedAttribute
from sqlalchemy.sql import ColumnElement, Select
+else:
+ from sqlalchemy.orm.attributes import InstrumentedAttribute
Review Comment:
InstrumentedAttribute is needed because in SQLAlchemy 2.0 the typing rules got stricter. We have to clearly differentiate normal SQL expressions (like ColumnElement) from ORM model attributes. ORM attributes (InstrumentedAttribute) come with extra ORM-related stuff like relationship tracking and helper methods that regular columns don’t have.
##########
airflow-core/src/airflow/api_fastapi/core_api/services/public/dag_run.py:
##########
@@ -57,7 +57,7 @@ def _serialize_xcoms(self) -> dict[str, Any]:
task_ids=self.result_task_ids,
dag_ids=self.dag_id,
)
- xcom_query = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()
+ xcom_results = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()
Review Comment:
@uranusjr, may I know why you want to remove .all()?
My thought process is we cannot remove .all() because itertools.groupby() on line 70 requires a concrete sequence, not a lazy iterator. Without .all(), session.scalars() returns a ScalarResult (lazy iterator). itertools.groupby() needs to iterate through the data multiple times. The lazy iterator might not preserve the sorted order between iterations
With .all(), we get a materialized Sequence[XComModel]. itertools.groupby() can reliably group the sorted data. The .all() is functionally necessary, not just for type checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476344998
##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
from airflow.models.dagrun import DagRun
dagruns_select_with_state_count = (
- select(
+ select( # type: ignore[call-overload]
Review Comment:
Just a comment above this line would be good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476327024
##########
airflow-core/src/airflow/api_fastapi/core_api/services/public/dag_run.py:
##########
@@ -57,7 +57,7 @@ def _serialize_xcoms(self) -> dict[str, Any]:
task_ids=self.result_task_ids,
dag_ids=self.dag_id,
)
- xcom_query = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()
+ xcom_results = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()
Review Comment:
It is documented as only needing an iterable. https://docs.python.org/3/library/itertools.html#itertools.groupby
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2471830348
##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -66,6 +66,8 @@
if TYPE_CHECKING:
from sqlalchemy.orm.attributes import InstrumentedAttribute
from sqlalchemy.sql import ColumnElement, Select
+else:
+ from sqlalchemy.orm.attributes import InstrumentedAttribute
Review Comment:
I don’t think we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476283322
##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
from airflow.models.dagrun import DagRun
dagruns_select_with_state_count = (
- select(
+ select( # type: ignore[call-overload]
Review Comment:
Yes, this is a limitation in SQLAlchemy 2.0's type declarations. It only defines overloads for up to 10 specific parameter combinations. It can't handle the complex mix of InstrumentedAttribute, func.count(), and .label() expressions we're using, even though it's perfectly valid SQLAlchemy code that works at runtime.
I tried using the below (unpacking approach)
dagruns_select_with_state_count = (
**select(*[**
DagRun.dag_id,
DagRun.state,
DagModel.dag_display_name,
func.count(DagRun.state).label("count"),
])
.join(DagModel, DagRun.dag_id == DagModel.dag_id)
.group_by(DagRun.dag_id, DagRun.state, DagModel.dag_display_name)
.order_by(DagRun.dag_id)
)
This is not working since the unpacking approach doesn't work either because MyPy sees list[object] instead of the individual types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2471831282
##########
airflow-core/src/airflow/api_fastapi/core_api/services/public/dag_run.py:
##########
@@ -57,7 +57,7 @@ def _serialize_xcoms(self) -> dict[str, Any]:
task_ids=self.result_task_ids,
dag_ids=self.dag_id,
)
- xcom_query = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()
+ xcom_results = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()
Review Comment:
```suggestion
xcom_results = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2471829684
##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
from airflow.models.dagrun import DagRun
dagruns_select_with_state_count = (
- select(
+ select( # type: ignore[call-overload]
Review Comment:
Why do we need to add this ignore (since we don’t use anything out of ordinary; everything is just standard). Is this a bug in SQLAlchemy’s type declarations? Can it be avoided?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476326125
##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -66,6 +66,8 @@
if TYPE_CHECKING:
from sqlalchemy.orm.attributes import InstrumentedAttribute
from sqlalchemy.sql import ColumnElement, Select
+else:
+ from sqlalchemy.orm.attributes import InstrumentedAttribute
Review Comment:
But InstrumentedAttribute is already imported for typing (like 67). This makes it available at runtime too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476335464
##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
from airflow.models.dagrun import DagRun
dagruns_select_with_state_count = (
- select(
+ select( # type: ignore[call-overload]
Review Comment:
That’s unfortunate. Would it be viable if we use `cast` on the `func.count` value to fix this instead? Either way, let’s also add a comment describing why the ignore/cast is needed so it could potentially be revisited if SQLAlchemy improves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476430078
##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
from airflow.models.dagrun import DagRun
dagruns_select_with_state_count = (
- select(
+ select( # type: ignore[call-overload]
Review Comment:
@uranusjr this worked. Thanks for the suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476345755
##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -66,6 +66,8 @@
if TYPE_CHECKING:
from sqlalchemy.orm.attributes import InstrumentedAttribute
from sqlalchemy.sql import ColumnElement, Select
+else:
+ from sqlalchemy.orm.attributes import InstrumentedAttribute
Review Comment:
Now I get what you are saying. Yes, its duplicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #57466:
URL: https://github.com/apache/airflow/pull/57466
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476395180
##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
from airflow.models.dagrun import DagRun
dagruns_select_with_state_count = (
- select(
+ select( # type: ignore[call-overload]
Review Comment:
Sounds good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476342924
##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
from airflow.models.dagrun import DagRun
dagruns_select_with_state_count = (
- select(
+ select( # type: ignore[call-overload]
Review Comment:
I can check that. If that also fails, could you please guide me on where I should document it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2471830842
##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -410,7 +412,11 @@ def filter_param_factory(
def depends_filter(value: T | None = query) -> FilterParam[T | None]:
if transform_callable:
value = transform_callable(value)
- return FilterParam(attribute, value, filter_option, skip_none)
+ # Cast to InstrumentedAttribute for type compatibility
+ from typing import cast
Review Comment:
Please import at the top.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org