You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "KoviAnusha (via GitHub)" <gi...@apache.org> on 2025/10/29 05:48:28 UTC

[PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

KoviAnusha opened a new pull request, #57466:
URL: https://github.com/apache/airflow/pull/57466

   
   related: #56735
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476274016


##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -66,6 +66,8 @@
 if TYPE_CHECKING:
     from sqlalchemy.orm.attributes import InstrumentedAttribute
     from sqlalchemy.sql import ColumnElement, Select
+else:
+    from sqlalchemy.orm.attributes import InstrumentedAttribute

Review Comment:
   InstrumentedAttribute is needed because in SQLAlchemy 2.0 the typing rules got stricter. We have to clearly differentiate normal SQL expressions (like ColumnElement) from ORM model attributes. ORM attributes (InstrumentedAttribute) come with extra ORM-related stuff like relationship tracking and helper methods that regular columns don’t have.
   



##########
airflow-core/src/airflow/api_fastapi/core_api/services/public/dag_run.py:
##########
@@ -57,7 +57,7 @@ def _serialize_xcoms(self) -> dict[str, Any]:
             task_ids=self.result_task_ids,
             dag_ids=self.dag_id,
         )
-        xcom_query = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()
+        xcom_results = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()

Review Comment:
   @uranusjr, may I know why you want to remove .all()? 
   
   My thought process is we cannot remove .all() because itertools.groupby() on line 70 requires a concrete sequence, not a lazy iterator. Without .all(), session.scalars() returns a ScalarResult (lazy iterator). itertools.groupby() needs to iterate through the data multiple times. The lazy iterator might not preserve the sorted order between iterations
   
   With .all(), we get a materialized Sequence[XComModel]. itertools.groupby() can reliably group the sorted data. The .all() is functionally necessary, not just for type checking.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476344998


##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
 from airflow.models.dagrun import DagRun
 
 dagruns_select_with_state_count = (
-    select(
+    select(  # type: ignore[call-overload]

Review Comment:
   Just a comment above this line would be good



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476327024


##########
airflow-core/src/airflow/api_fastapi/core_api/services/public/dag_run.py:
##########
@@ -57,7 +57,7 @@ def _serialize_xcoms(self) -> dict[str, Any]:
             task_ids=self.result_task_ids,
             dag_ids=self.dag_id,
         )
-        xcom_query = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()
+        xcom_results = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()

Review Comment:
   It is documented as only needing an iterable. https://docs.python.org/3/library/itertools.html#itertools.groupby



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2471830348


##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -66,6 +66,8 @@
 if TYPE_CHECKING:
     from sqlalchemy.orm.attributes import InstrumentedAttribute
     from sqlalchemy.sql import ColumnElement, Select
+else:
+    from sqlalchemy.orm.attributes import InstrumentedAttribute

Review Comment:
   I don’t think we need this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476283322


##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
 from airflow.models.dagrun import DagRun
 
 dagruns_select_with_state_count = (
-    select(
+    select(  # type: ignore[call-overload]

Review Comment:
   Yes, this is a limitation in SQLAlchemy 2.0's type declarations. It only defines overloads for up to 10 specific parameter combinations. It can't handle the complex mix of InstrumentedAttribute, func.count(), and .label() expressions we're using, even though it's perfectly valid SQLAlchemy code that works at runtime.
   
   I tried using the below (unpacking approach)
   
   dagruns_select_with_state_count = (
       **select(*[**
           DagRun.dag_id,
           DagRun.state,
           DagModel.dag_display_name,
           func.count(DagRun.state).label("count"),
       ])
       .join(DagModel, DagRun.dag_id == DagModel.dag_id)
       .group_by(DagRun.dag_id, DagRun.state, DagModel.dag_display_name)
       .order_by(DagRun.dag_id)
   )
   
   This is not working since the unpacking approach doesn't work either because MyPy sees list[object] instead of the individual types.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2471831282


##########
airflow-core/src/airflow/api_fastapi/core_api/services/public/dag_run.py:
##########
@@ -57,7 +57,7 @@ def _serialize_xcoms(self) -> dict[str, Any]:
             task_ids=self.result_task_ids,
             dag_ids=self.dag_id,
         )
-        xcom_query = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()
+        xcom_results = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index)).all()

Review Comment:
   ```suggestion
           xcom_results = self.session.scalars(xcom_query.order_by(XComModel.task_id, XComModel.map_index))
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2471829684


##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
 from airflow.models.dagrun import DagRun
 
 dagruns_select_with_state_count = (
-    select(
+    select(  # type: ignore[call-overload]

Review Comment:
   Why do we need to add this ignore (since we don’t use anything out of ordinary; everything is just standard). Is this a bug in SQLAlchemy’s type declarations? Can it be avoided?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476326125


##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -66,6 +66,8 @@
 if TYPE_CHECKING:
     from sqlalchemy.orm.attributes import InstrumentedAttribute
     from sqlalchemy.sql import ColumnElement, Select
+else:
+    from sqlalchemy.orm.attributes import InstrumentedAttribute

Review Comment:
   But InstrumentedAttribute is already imported for typing (like 67). This makes it available at runtime too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476335464


##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
 from airflow.models.dagrun import DagRun
 
 dagruns_select_with_state_count = (
-    select(
+    select(  # type: ignore[call-overload]

Review Comment:
   That’s unfortunate. Would it be viable if we use `cast` on the `func.count` value to fix this instead? Either way, let’s also add a comment describing why the ignore/cast is needed so it could potentially be revisited if SQLAlchemy improves.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476430078


##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
 from airflow.models.dagrun import DagRun
 
 dagruns_select_with_state_count = (
-    select(
+    select(  # type: ignore[call-overload]

Review Comment:
   @uranusjr this worked. Thanks for the suggestion!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476345755


##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -66,6 +66,8 @@
 if TYPE_CHECKING:
     from sqlalchemy.orm.attributes import InstrumentedAttribute
     from sqlalchemy.sql import ColumnElement, Select
+else:
+    from sqlalchemy.orm.attributes import InstrumentedAttribute

Review Comment:
   Now I get what you are saying. Yes, its duplicate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #57466:
URL: https://github.com/apache/airflow/pull/57466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476395180


##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
 from airflow.models.dagrun import DagRun
 
 dagruns_select_with_state_count = (
-    select(
+    select(  # type: ignore[call-overload]

Review Comment:
   Sounds good!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2476342924


##########
airflow-core/src/airflow/api_fastapi/common/db/dag_runs.py:
##########
@@ -23,11 +23,11 @@
 from airflow.models.dagrun import DagRun
 
 dagruns_select_with_state_count = (
-    select(
+    select(  # type: ignore[call-overload]

Review Comment:
   I can check that. If that also fails, could you please guide me on where I should document it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix MyPy type errors in api_fastapi remaining files for Sqlalchemy 2 migration [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #57466:
URL: https://github.com/apache/airflow/pull/57466#discussion_r2471830842


##########
airflow-core/src/airflow/api_fastapi/common/parameters.py:
##########
@@ -410,7 +412,11 @@ def filter_param_factory(
     def depends_filter(value: T | None = query) -> FilterParam[T | None]:
         if transform_callable:
             value = transform_callable(value)
-        return FilterParam(attribute, value, filter_option, skip_none)
+        # Cast to InstrumentedAttribute for type compatibility
+        from typing import cast

Review Comment:
   Please import at the top.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org