You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2025/02/05 17:47:50 UTC

[PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

jason810496 opened a new pull request, #46484:
URL: https://github.com/apache/airflow/pull/46484

   
   related: #45995 
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1966492515


##########
airflow/models/dagrun.py:
##########
@@ -314,7 +314,11 @@ def validate_run_id(self, key: str, run_id: str) -> str | None:
     @property
     def dag_versions(self) -> list[DagVersion]:
         """Return the DAG versions associated with the TIs of this DagRun."""
-        dag_versions = list(dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions)))
+        dag_versions = [
+            dv
+            for dv in dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions))
+            if dv is not None
+        ]
         sorted_ = sorted(dag_versions, key=lambda dv: dv.id)

Review Comment:
   I have tried fix by the following, but result in the same error.
   ```python
   if not dag_versions:
       return []
   ```
   > It’s because some TI or TIH do not find the related version. That might not be a real use case, just a test thing. TI has no dag version initialized in the test.
   
   Agree, maybe refactor the `setup_dag_runs` in `TestRecentDagRuns` with `dag_maker.create_dagrun` would resolve the "test thing" case.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2678067762

   I have identified the problem, I will fix it in another PR that's not directly related to this one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1946772748


##########
airflow/api_fastapi/core_api/routes/public/dag_run.py:
##########
@@ -63,15 +65,85 @@
 from airflow.api_fastapi.logging.decorators import action_logging
 from airflow.exceptions import ParamValidationError
 from airflow.listeners.listener import get_listener_manager
-from airflow.models import DAG, DagModel, DagRun
+from airflow.models import DAG, DagModel, DagRun, TaskInstance
 from airflow.models.dag_version import DagVersion
+from airflow.models.taskinstancehistory import TaskInstanceHistory
 from airflow.timetables.base import DataInterval
 from airflow.utils.state import DagRunState
 from airflow.utils.types import DagRunTriggeredByType, DagRunType
 
 dag_run_router = AirflowRouter(tags=["DagRun"], prefix="/dags/{dag_id}/dagRuns")
 
 
+def get_dag_run_with_task_ids(dag_id: str, run_id: str, session: SessionDep) -> DagRun:
+    """Get the DagRun with the given task_ids."""
+    return session.scalar(
+        select(DagRun)
+        .options(joinedload(DagRun.task_instances).load_only("task_id"))
+        .filter_by(dag_id=dag_id, run_id=run_id)
+    )
+
+
+def get_dag_version_ids_among_dag_run(dag_id: str, dag_run_id: str, task_ids: list[str], session: SessionDep):
+    """Get the DagVersions from the TaskInstances and TaskInstanceHistories of the DagRun."""
+    task_instance_dag_version_ids = select(TaskInstance.dag_version_id).where(
+        TaskInstance.dag_id == dag_id, TaskInstance.run_id == dag_run_id, TaskInstance.task_id.in_(task_ids)
+    )
+    task_instance_history_dag_version_ids = select(TaskInstanceHistory.dag_version_id).where(
+        TaskInstanceHistory.dag_id == dag_id,
+        TaskInstanceHistory.run_id == dag_run_id,
+        TaskInstanceHistory.task_id.in_(task_ids),
+    )
+    return (
+        session.execute(
+            task_instance_dag_version_ids.distinct().union(task_instance_history_dag_version_ids.distinct())
+        )
+        .scalars()
+        .all()
+    )

Review Comment:
   Hello @jason810496,
   
   To not make this issue too big / hard, @ephraimbuddy suggested to help on the ORM / database side (migration, ORM update etc...). I created another issue for that https://github.com/apache/airflow/issues/46565. Once Ephraim is done working on that, you can continue integrating the change into the RestAPI.
   
   Does that work for you ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1966148982


##########
airflow/models/dagrun.py:
##########
@@ -314,7 +314,11 @@ def validate_run_id(self, key: str, run_id: str) -> str | None:
     @property
     def dag_versions(self) -> list[DagVersion]:
         """Return the DAG versions associated with the TIs of this DagRun."""
-        dag_versions = list(dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions)))
+        dag_versions = [
+            dv
+            for dv in dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions))
+            if dv is not None
+        ]
         sorted_ = sorted(dag_versions, key=lambda dv: dv.id)

Review Comment:
   It’s because some TI or TIH do not find the related version. That might not be a real use case, just a test thing. TI has no dag version initialized in the test.
   
   can a TIH have None for Version when joining to populate the relationship <=> Can versions be removed I guess would be the use case.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1966511236


##########
tests/api_fastapi/conftest.py:
##########
@@ -77,24 +68,36 @@ def make_dag_with_multiple_versions(dag_maker, session):
             ): '[{ "name": "dag_maker", "classpath": "airflow.dag_processing.bundles.git.GitDagBundle", "kwargs": {"subdir": "dags", "tracking_ref": "main", "refresh_interval": 0}}, { "name": "another_bundle_name", "classpath": "airflow.dag_processing.bundles.git.GitDagBundle", "kwargs": {"subdir": "dags", "tracking_ref": "main", "refresh_interval": 0}}]'
         }
     ):
-        dag_id = "dag_with_multiple_versions"
-
-        for version_number in range(1, 4):
-            with dag_maker(dag_id) as dag:
-                for task_number in range(version_number):
-                    EmptyOperator(task_id=f"task{task_number + 1}")
-            dag.sync_to_db()
-            SerializedDagModel.write_dag(
-                dag, bundle_name="dag_maker", bundle_version=f"some_commit_hash{version_number}"
-            )
-            dag_maker.create_dagrun(
-                run_id=f"run{version_number}",
-                logical_date=datetime.datetime(2020, 1, version_number, tzinfo=datetime.timezone.utc),
-                dag_version=DagVersion.get_version(dag_id=dag_id, version_number=version_number),
-            )
         yield
 
-        clear_db_connections(False)
+    clear_db_connections(False)

Review Comment:
   Not the right index?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1966541837


##########
tests/api_fastapi/conftest.py:
##########
@@ -77,24 +68,36 @@ def make_dag_with_multiple_versions(dag_maker, session):
             ): '[{ "name": "dag_maker", "classpath": "airflow.dag_processing.bundles.git.GitDagBundle", "kwargs": {"subdir": "dags", "tracking_ref": "main", "refresh_interval": 0}}, { "name": "another_bundle_name", "classpath": "airflow.dag_processing.bundles.git.GitDagBundle", "kwargs": {"subdir": "dags", "tracking_ref": "main", "refresh_interval": 0}}]'
         }
     ):
-        dag_id = "dag_with_multiple_versions"
-
-        for version_number in range(1, 4):
-            with dag_maker(dag_id) as dag:
-                for task_number in range(version_number):
-                    EmptyOperator(task_id=f"task{task_number + 1}")
-            dag.sync_to_db()
-            SerializedDagModel.write_dag(
-                dag, bundle_name="dag_maker", bundle_version=f"some_commit_hash{version_number}"
-            )
-            dag_maker.create_dagrun(
-                run_id=f"run{version_number}",
-                logical_date=datetime.datetime(2020, 1, version_number, tzinfo=datetime.timezone.utc),
-                dag_version=DagVersion.get_version(dag_id=dag_id, version_number=version_number),
-            )
         yield
 
-        clear_db_connections(False)
+    clear_db_connections(False)

Review Comment:
   It's intended to be at that indentation level, ensuring that only `yield` is executed within the `with conf_vars` context.
   
   https://github.com/jason810496/airflow/blob/feature/AIP-65/Add-dag-versions-to-DagRunResponse/tests/api_fastapi/conftest.py#L50-L74



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1959492466


##########
airflow/api_fastapi/core_api/datamodels/dag_run.py:
##########
@@ -69,6 +70,7 @@ class DAGRunResponse(BaseModel):
     triggered_by: DagRunTriggeredByType
     conf: dict
     note: str | None
+    dag_versions: list[UUID]

Review Comment:
   Yes I think so



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2675181533

   > Nice. Code looks good. Just a few minor comments, non blocking
   > 
   > 
   > 
   > We just need to rebase and rerun the CI I think there was some unrelated failure.
   
   For the test CLI that fail is a not flaky test. It's caused by DagBundleManager logging a new line, and break the json.loads ( I  will push the fix tomorrow ) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1946286668


##########
airflow/api_fastapi/core_api/datamodels/dag_run.py:
##########
@@ -69,6 +70,7 @@ class DAGRunResponse(BaseModel):
     triggered_by: DagRunTriggeredByType
     conf: dict
     note: str | None
+    dag_versions: list[UUID]

Review Comment:
   > Add `dag_versions` as an array of uuids to the DagRunResponse in the public REST API
   
   Just to make double check, as @bbovenzi describe above in the issue, so we need `dag_version_numbers` instead of list of uuids ?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2678075696

   PR here https://github.com/apache/airflow/pull/47016
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2677852033

   > The CI failure is due to sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked. We may need to retrigger the SQLite test.
   
   Ok, I just restarted the failed step.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1965747681


##########
airflow/models/dagrun.py:
##########
@@ -314,7 +314,11 @@ def validate_run_id(self, key: str, run_id: str) -> str | None:
     @property
     def dag_versions(self) -> list[DagVersion]:
         """Return the DAG versions associated with the TIs of this DagRun."""
-        dag_versions = list(dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions)))
+        dag_versions = [
+            dv
+            for dv in dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions))
+            if dv is not None
+        ]
         sorted_ = sorted(dag_versions, key=lambda dv: dv.id)

Review Comment:
   Can you find out why it errored like that? if it's because the dag_versions is None, then we should just check it and return empty list:
   ```python
   if not dag_versions:
       return []
   ```
   And adjust the return type



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1946755759


##########
airflow/api_fastapi/core_api/datamodels/dag_run.py:
##########
@@ -69,6 +70,7 @@ class DAGRunResponse(BaseModel):
     triggered_by: DagRunTriggeredByType
     conf: dict
     note: str | None
+    dag_versions: list[UUID]

Review Comment:
   As an array of integer representing a `DagVersion.version_number` 



##########
airflow/api_fastapi/core_api/datamodels/dag_run.py:
##########
@@ -69,6 +70,7 @@ class DAGRunResponse(BaseModel):
     triggered_by: DagRunTriggeredByType
     conf: dict
     note: str | None
+    dag_versions: list[UUID]

Review Comment:
   As an array of integer representing a `DagVersion.version_number` I think



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1959291004


##########
airflow/api_fastapi/core_api/datamodels/dag_run.py:
##########
@@ -69,6 +70,7 @@ class DAGRunResponse(BaseModel):
     triggered_by: DagRunTriggeredByType
     conf: dict
     note: str | None
+    dag_versions: list[UUID]

Review Comment:
   Hi @pierrejeambrun , do we need to change `dag_versions` from `dag_versions: list[int]` to `dag_versions: listDagVersionResponse]` ? 
   Since after [AIP-66 Refactor DagRun to DagVersion association #46626](https://github.com/apache/airflow/pull/46626) be merged, `dag_run. dag_versions` could directly retrieve `DagVersion` models.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2669099520

   > Yes bundles need to be configured.
   > 
   > You can take a look at the `make_dag_with_multiple_versions` which does that. (You need a configuration + git connection for it to work in your test)
   
   Big thanks to @pierrejeambrun! You saved my day. I'm still fixing tests locally, then the PR will be ready.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1946286668


##########
airflow/api_fastapi/core_api/datamodels/dag_run.py:
##########
@@ -69,6 +70,7 @@ class DAGRunResponse(BaseModel):
     triggered_by: DagRunTriggeredByType
     conf: dict
     note: str | None
+    dag_versions: list[UUID]

Review Comment:
   > Add `dag_versions` as an array of uuids to the DagRunResponse in the public REST API
   
   Just to make double check, as @bbovenzi describe above in the issue, so we need `dag_version_numbers` instead of `dag_versions` as an array of uuids ?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1966148982


##########
airflow/models/dagrun.py:
##########
@@ -314,7 +314,11 @@ def validate_run_id(self, key: str, run_id: str) -> str | None:
     @property
     def dag_versions(self) -> list[DagVersion]:
         """Return the DAG versions associated with the TIs of this DagRun."""
-        dag_versions = list(dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions)))
+        dag_versions = [
+            dv
+            for dv in dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions))
+            if dv is not None
+        ]
         sorted_ = sorted(dag_versions, key=lambda dv: dv.id)

Review Comment:
   It’s because some TI or TIH do not find the related version. That might not be a real use case, just a test thing.
   
   can a TIH have None on versions when joining to populate the relationship <=> Can versions be removed I guess would be the use case.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1964911576


##########
airflow/models/dagrun.py:
##########
@@ -314,7 +314,11 @@ def validate_run_id(self, key: str, run_id: str) -> str | None:
     @property
     def dag_versions(self) -> list[DagVersion]:
         """Return the DAG versions associated with the TIs of this DagRun."""
-        dag_versions = list(dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions)))
+        dag_versions = [
+            dv
+            for dv in dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions))
+            if dv is not None
+        ]
         sorted_ = sorted(dag_versions, key=lambda dv: dv.id)

Review Comment:
   Encountered the following error, so I filtered out `None` in `dag_versions`.
   ```
   airflow/api_fastapi/core_api/routes/ui/dags.py:145: in recent_dag_runs
       dag_versions=dag_run.dag_versions,
   airflow/models/dagrun.py:321: in dag_versions
       sorted_ = sorted(dag_versions, key=lambda dv: dv.id)
   airflow/models/dagrun.py:321: in <lambda>
       sorted_ = sorted(dag_versions, key=lambda dv: dv.id)
   E   AttributeError: 'NoneType' object has no attribute 'id'
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun merged PR #46484:
URL: https://github.com/apache/airflow/pull/46484


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1946280413


##########
airflow/api_fastapi/core_api/routes/public/dag_run.py:
##########
@@ -63,15 +65,85 @@
 from airflow.api_fastapi.logging.decorators import action_logging
 from airflow.exceptions import ParamValidationError
 from airflow.listeners.listener import get_listener_manager
-from airflow.models import DAG, DagModel, DagRun
+from airflow.models import DAG, DagModel, DagRun, TaskInstance
 from airflow.models.dag_version import DagVersion
+from airflow.models.taskinstancehistory import TaskInstanceHistory
 from airflow.timetables.base import DataInterval
 from airflow.utils.state import DagRunState
 from airflow.utils.types import DagRunTriggeredByType, DagRunType
 
 dag_run_router = AirflowRouter(tags=["DagRun"], prefix="/dags/{dag_id}/dagRuns")
 
 
+def get_dag_run_with_task_ids(dag_id: str, run_id: str, session: SessionDep) -> DagRun:
+    """Get the DagRun with the given task_ids."""
+    return session.scalar(
+        select(DagRun)
+        .options(joinedload(DagRun.task_instances).load_only("task_id"))
+        .filter_by(dag_id=dag_id, run_id=run_id)
+    )
+
+
+def get_dag_version_ids_among_dag_run(dag_id: str, dag_run_id: str, task_ids: list[str], session: SessionDep):
+    """Get the DagVersions from the TaskInstances and TaskInstanceHistories of the DagRun."""
+    task_instance_dag_version_ids = select(TaskInstance.dag_version_id).where(
+        TaskInstance.dag_id == dag_id, TaskInstance.run_id == dag_run_id, TaskInstance.task_id.in_(task_ids)
+    )
+    task_instance_history_dag_version_ids = select(TaskInstanceHistory.dag_version_id).where(
+        TaskInstanceHistory.dag_id == dag_id,
+        TaskInstanceHistory.run_id == dag_run_id,
+        TaskInstanceHistory.task_id.in_(task_ids),
+    )
+    return (
+        session.execute(
+            task_instance_dag_version_ids.distinct().union(task_instance_history_dag_version_ids.distinct())
+        )
+        .scalars()
+        .all()
+    )

Review Comment:
   Ideally we would prefer to have a proxy object on the DagRun ORM object that would allow us to retrieve all that when doing `DagRun.dag_versions` (that would be a list of ORM DagVersion objects). Or simply a list of int representing the dag run version numbers.
   
   
   Importantly we need the ability to eagerly load that when we query DagRuns so we have no extra lazy queries emitted later on when accessing this property.
   
   You can take a look at relationships or associations proxies. Basically setting up two relationships (history_versions, and current_versions) and maybe adding a property that returns the concatenation of both. 
   
   Or even better 1 relationship that aggregates the version from related TaskInstance and TaskInstanceHistoryResponse (but that might be harder to implement.)



##########
airflow/api_fastapi/core_api/routes/public/dag_run.py:
##########
@@ -63,15 +65,85 @@
 from airflow.api_fastapi.logging.decorators import action_logging
 from airflow.exceptions import ParamValidationError
 from airflow.listeners.listener import get_listener_manager
-from airflow.models import DAG, DagModel, DagRun
+from airflow.models import DAG, DagModel, DagRun, TaskInstance
 from airflow.models.dag_version import DagVersion
+from airflow.models.taskinstancehistory import TaskInstanceHistory
 from airflow.timetables.base import DataInterval
 from airflow.utils.state import DagRunState
 from airflow.utils.types import DagRunTriggeredByType, DagRunType
 
 dag_run_router = AirflowRouter(tags=["DagRun"], prefix="/dags/{dag_id}/dagRuns")
 
 
+def get_dag_run_with_task_ids(dag_id: str, run_id: str, session: SessionDep) -> DagRun:
+    """Get the DagRun with the given task_ids."""
+    return session.scalar(
+        select(DagRun)
+        .options(joinedload(DagRun.task_instances).load_only("task_id"))
+        .filter_by(dag_id=dag_id, run_id=run_id)
+    )
+
+
+def get_dag_version_ids_among_dag_run(dag_id: str, dag_run_id: str, task_ids: list[str], session: SessionDep):
+    """Get the DagVersions from the TaskInstances and TaskInstanceHistories of the DagRun."""
+    task_instance_dag_version_ids = select(TaskInstance.dag_version_id).where(
+        TaskInstance.dag_id == dag_id, TaskInstance.run_id == dag_run_id, TaskInstance.task_id.in_(task_ids)
+    )
+    task_instance_history_dag_version_ids = select(TaskInstanceHistory.dag_version_id).where(
+        TaskInstanceHistory.dag_id == dag_id,
+        TaskInstanceHistory.run_id == dag_run_id,
+        TaskInstanceHistory.task_id.in_(task_ids),
+    )
+    return (
+        session.execute(
+            task_instance_dag_version_ids.distinct().union(task_instance_history_dag_version_ids.distinct())
+        )
+        .scalars()
+        .all()
+    )

Review Comment:
   This is a fallback solution, because we need to emit an extra custom query to fetch those `version_ids` and implement a lot of application code logic to get the expected behavior.
   
   And this code should not live in the `routes` folder.



##########
airflow/api_fastapi/core_api/datamodels/dag_run.py:
##########
@@ -69,6 +70,7 @@ class DAGRunResponse(BaseModel):
     triggered_by: DagRunTriggeredByType
     conf: dict
     note: str | None
+    dag_versions: list[UUID]

Review Comment:
   We use the `version_number` as a key on other endpoints. I think we should return `dag_version_numbers` which makes also more sense from a use point of view. (UUID don't bring much information)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1946772748


##########
airflow/api_fastapi/core_api/routes/public/dag_run.py:
##########
@@ -63,15 +65,85 @@
 from airflow.api_fastapi.logging.decorators import action_logging
 from airflow.exceptions import ParamValidationError
 from airflow.listeners.listener import get_listener_manager
-from airflow.models import DAG, DagModel, DagRun
+from airflow.models import DAG, DagModel, DagRun, TaskInstance
 from airflow.models.dag_version import DagVersion
+from airflow.models.taskinstancehistory import TaskInstanceHistory
 from airflow.timetables.base import DataInterval
 from airflow.utils.state import DagRunState
 from airflow.utils.types import DagRunTriggeredByType, DagRunType
 
 dag_run_router = AirflowRouter(tags=["DagRun"], prefix="/dags/{dag_id}/dagRuns")
 
 
+def get_dag_run_with_task_ids(dag_id: str, run_id: str, session: SessionDep) -> DagRun:
+    """Get the DagRun with the given task_ids."""
+    return session.scalar(
+        select(DagRun)
+        .options(joinedload(DagRun.task_instances).load_only("task_id"))
+        .filter_by(dag_id=dag_id, run_id=run_id)
+    )
+
+
+def get_dag_version_ids_among_dag_run(dag_id: str, dag_run_id: str, task_ids: list[str], session: SessionDep):
+    """Get the DagVersions from the TaskInstances and TaskInstanceHistories of the DagRun."""
+    task_instance_dag_version_ids = select(TaskInstance.dag_version_id).where(
+        TaskInstance.dag_id == dag_id, TaskInstance.run_id == dag_run_id, TaskInstance.task_id.in_(task_ids)
+    )
+    task_instance_history_dag_version_ids = select(TaskInstanceHistory.dag_version_id).where(
+        TaskInstanceHistory.dag_id == dag_id,
+        TaskInstanceHistory.run_id == dag_run_id,
+        TaskInstanceHistory.task_id.in_(task_ids),
+    )
+    return (
+        session.execute(
+            task_instance_dag_version_ids.distinct().union(task_instance_history_dag_version_ids.distinct())
+        )
+        .scalars()
+        .all()
+    )

Review Comment:
   Hello @jason810496,
   
   To not make this issue too big / hard, @ephraimbuddy suggested to help on the ORM / database side (migration, ORM update etc...). I created another issue for that https://github.com/apache/airflow/issues/46565. Once Ephraim is done working on it, you can continue integrating the change into the RestAPI.
   
   Does that work for you ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2678086551

   This should fix your CI issue https://github.com/apache/airflow/pull/47016


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1966491458


##########
tests/api_fastapi/core_api/routes/public/test_dag_run.py:
##########
@@ -146,14 +148,41 @@ def setup(request, dag_maker, session=None):
     )
 
     dag_maker.sync_dagbag_to_db()
-    dag_maker.dag_model
     dag_maker.dag_model.has_task_concurrency_limits = True
     session.merge(ti1)
     session.merge(ti2)
     session.merge(dag_maker.dag_model)
     session.commit()
 
 
+def get_dag_versions_dict(dag_version: DagVersion):
+    return DagVersionResponse.model_validate(dag_version, from_attributes=True).model_dump(mode="json")
+
+
+def get_dag_run_dict(run: DagRun):
+    return {
+        "dag_run_id": run.run_id,
+        "dag_id": run.dag_id,
+        "logical_date": from_datetime_to_zulu_without_ms(run.logical_date),
+        "queued_at": from_datetime_to_zulu(run.queued_at) if run.queued_at else None,
+        "run_after": from_datetime_to_zulu_without_ms(run.run_after),
+        "start_date": from_datetime_to_zulu_without_ms(run.start_date),
+        "end_date": from_datetime_to_zulu(run.end_date),
+        "data_interval_start": from_datetime_to_zulu_without_ms(run.data_interval_start),
+        "data_interval_end": from_datetime_to_zulu_without_ms(run.data_interval_end),
+        "last_scheduling_decision": (
+            from_datetime_to_zulu(run.last_scheduling_decision) if run.last_scheduling_decision else None
+        ),
+        "run_type": run.run_type,
+        "state": run.state,
+        "external_trigger": run.external_trigger,
+        "triggered_by": run.triggered_by.value,
+        "conf": run.conf,
+        "note": run.note,
+        "dag_versions": [get_dag_versions_dict(dag_version) for dag_version in run.dag_versions],

Review Comment:
   Sure, I just refactored the `get_dag_versions_dict`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2677995091

   Failing again, that's not flaky
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1965717886


##########
tests/api_fastapi/core_api/routes/public/test_dag_run.py:
##########
@@ -146,14 +148,41 @@ def setup(request, dag_maker, session=None):
     )
 
     dag_maker.sync_dagbag_to_db()
-    dag_maker.dag_model
     dag_maker.dag_model.has_task_concurrency_limits = True
     session.merge(ti1)
     session.merge(ti2)
     session.merge(dag_maker.dag_model)
     session.commit()
 
 
+def get_dag_versions_dict(dag_version: DagVersion):
+    return DagVersionResponse.model_validate(dag_version, from_attributes=True).model_dump(mode="json")
+
+
+def get_dag_run_dict(run: DagRun):
+    return {
+        "dag_run_id": run.run_id,
+        "dag_id": run.dag_id,
+        "logical_date": from_datetime_to_zulu_without_ms(run.logical_date),
+        "queued_at": from_datetime_to_zulu(run.queued_at) if run.queued_at else None,
+        "run_after": from_datetime_to_zulu_without_ms(run.run_after),
+        "start_date": from_datetime_to_zulu_without_ms(run.start_date),
+        "end_date": from_datetime_to_zulu(run.end_date),
+        "data_interval_start": from_datetime_to_zulu_without_ms(run.data_interval_start),
+        "data_interval_end": from_datetime_to_zulu_without_ms(run.data_interval_end),
+        "last_scheduling_decision": (
+            from_datetime_to_zulu(run.last_scheduling_decision) if run.last_scheduling_decision else None
+        ),
+        "run_type": run.run_type,
+        "state": run.state,
+        "external_trigger": run.external_trigger,
+        "triggered_by": run.triggered_by.value,
+        "conf": run.conf,
+        "note": run.note,
+        "dag_versions": [get_dag_versions_dict(dag_version) for dag_version in run.dag_versions],

Review Comment:
   ```suggestion
           "dag_versions": [get_dag_version_dict(dag_version) for dag_version in run.dag_versions],
   ```
   
   or
   
   ```suggestion
           "dag_versions": get_dag_versions_dict(run.dag_versions),
   ```



##########
airflow/models/dagrun.py:
##########
@@ -314,7 +314,11 @@ def validate_run_id(self, key: str, run_id: str) -> str | None:
     @property
     def dag_versions(self) -> list[DagVersion]:
         """Return the DAG versions associated with the TIs of this DagRun."""
-        dag_versions = list(dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions)))
+        dag_versions = [
+            dv
+            for dv in dict.fromkeys(list(self._tih_dag_versions) + list(self._ti_dag_versions))
+            if dv is not None
+        ]

Review Comment:
   Looks good to me.



##########
tests/api_fastapi/conftest.py:
##########
@@ -49,15 +49,7 @@ def create_test_client(apps="all"):
 
 
 @pytest.fixture
-def make_dag_with_multiple_versions(dag_maker, session):
-    """
-    Create DAG with multiple versions
-
-    Version 1 will have 1 task, version 2 will have 2 tasks, and version 3 will have 3 tasks.
-
-    Configure the associated dag_bundles.
-    """
-
+def configure_git_connection_for_dag_bundle(session):

Review Comment:
   Nice



##########
tests/api_fastapi/core_api/routes/public/test_dag_run.py:
##########
@@ -146,14 +148,41 @@ def setup(request, dag_maker, session=None):
     )
 
     dag_maker.sync_dagbag_to_db()
-    dag_maker.dag_model
     dag_maker.dag_model.has_task_concurrency_limits = True
     session.merge(ti1)
     session.merge(ti2)
     session.merge(dag_maker.dag_model)
     session.commit()
 
 
+def get_dag_versions_dict(dag_version: DagVersion):
+    return DagVersionResponse.model_validate(dag_version, from_attributes=True).model_dump(mode="json")

Review Comment:
   I think we can omit it
   ```suggestion
       return DagVersionResponse.model_validate(dag_version, from_attributes=True).model_dump()
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1966511236


##########
tests/api_fastapi/conftest.py:
##########
@@ -77,24 +68,36 @@ def make_dag_with_multiple_versions(dag_maker, session):
             ): '[{ "name": "dag_maker", "classpath": "airflow.dag_processing.bundles.git.GitDagBundle", "kwargs": {"subdir": "dags", "tracking_ref": "main", "refresh_interval": 0}}, { "name": "another_bundle_name", "classpath": "airflow.dag_processing.bundles.git.GitDagBundle", "kwargs": {"subdir": "dags", "tracking_ref": "main", "refresh_interval": 0}}]'
         }
     ):
-        dag_id = "dag_with_multiple_versions"
-
-        for version_number in range(1, 4):
-            with dag_maker(dag_id) as dag:
-                for task_number in range(version_number):
-                    EmptyOperator(task_id=f"task{task_number + 1}")
-            dag.sync_to_db()
-            SerializedDagModel.write_dag(
-                dag, bundle_name="dag_maker", bundle_version=f"some_commit_hash{version_number}"
-            )
-            dag_maker.create_dagrun(
-                run_id=f"run{version_number}",
-                logical_date=datetime.datetime(2020, 1, version_number, tzinfo=datetime.timezone.utc),
-                dag_version=DagVersion.get_version(dag_id=dag_id, version_number=version_number),
-            )
         yield
 
-        clear_db_connections(False)
+    clear_db_connections(False)

Review Comment:
   Not the right indent?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2637637753

   Hi @pierrejeambrun,  
   
   Could you help me confirm whether the current implementation of `airflow/api_fastapi/core_api/routes/public/dag_run.py` aligns with the issue? I'm not entirely sure if this is the best approach to retrieve DAG versions for each `dag_run`.  
   
   If there are no concerns with the current implementation (my main concern is performance), I’ll move some common queries to the `api_fastapi.common.db` module and complete the `/recent_dag_runs` UI endpoint. (`/recent_dag_runs` also depends on `DAGRunResponse`.)  
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2668783952

   Yes bundles need to be configured.
   
   You can take a look at the `make_dag_with_multiple_versions` which does that. (You need a configuration + git connection for it to work in your test)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2667919187

   Hi @pierrejeambrun,  
   
   Do I need to configure `DagBundlesManager` explicitly to set `("dag_processor", "dag_bundle_config_list")`?  
   I'm encountering the following error when running the `test_dag_run` test:  
   
   ```
      |   File "/opt/airflow/airflow/api_fastapi/core_api/datamodels/dag_versions.py", line 43, in bundle_url
       |     return DagBundlesManager().view_url(self.bundle_name, self.bundle_version)
       |   File "/opt/airflow/airflow/dag_processing/bundles/manager.py", line 144, in view_url
       |     bundle = self.get_bundle(name, version)
       |   File "/opt/airflow/airflow/dag_processing/bundles/manager.py", line 130, in get_bundle
       |     raise ValueError(f"Requested bundle '{name}' is not configured.")
       | ValueError: Requested bundle 'dag_maker' is not configured.
   ```
   
   I checked `test_dag_versions.py` but didn’t find any additional configurations for `DagBundlesManager`. However, in `test_dag_versions`, `DagBundlesManager._bundle_config` does contain the `dag_maker` key, whereas in `test_dag_run`, it does not.  
   
   Comparing `conf.getjson("dag_processor", "dag_bundle_config_list")` in both tests:  
   
   **`test_versions` output:**  
   ```
   bundle_name: another_bundle_name, bundle_version: some_commit_hash
   backends
   [{'classpath': 'airflow.dag_processing.bundles.git.GitDagBundle',
     'kwargs': {'refresh_interval': 0, 'subdir': 'dags', 'tracking_ref': 'main'},
     'name': 'dag_maker'},
    {'classpath': 'airflow.dag_processing.bundles.git.GitDagBundle',
     'kwargs': {'refresh_interval': 0, 'subdir': 'dags', 'tracking_ref': 'main'},
     'name': 'another_bundle_name'}]
   ```
   
   **`test_dag_run` output:**  
   ```
   bundle_name: dag_maker, bundle_version: None
   backends
   [{'classpath': 'airflow.dag_processing.bundles.local.LocalDagBundle',
     'kwargs': {},
     'name': 'dags-folder'}]
   [2025-02-19T08:25:53.289+0000] {manager.py:101} INFO - DAG bundles loaded: dags-folder, example_dags
   _bundle_config
   {'dags-folder': (<class 'airflow.dag_processing.bundles.local.LocalDagBundle'>,
                    {}),
    'example_dags': (<class 'airflow.dag_processing.bundles.local.LocalDagBundle'>,
                     {'path': '/opt/airflow/airflow/example_dags'})}
   FAILED
   ```
   
   I even tried explicitly setting `("dag_processor", "dag_bundle_config_list")` using `conf_vars`, but the configuration still does not include `dag_maker` for `bundle_name`.  
   
   Would appreciate any guidance on this—thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1959291004


##########
airflow/api_fastapi/core_api/datamodels/dag_run.py:
##########
@@ -69,6 +70,7 @@ class DAGRunResponse(BaseModel):
     triggered_by: DagRunTriggeredByType
     conf: dict
     note: str | None
+    dag_versions: list[UUID]

Review Comment:
   Hi @pierrejeambrun , do we need to change `dag_versions` to `dag_versions: listDagVersionResponse]` ? 
   Since after [AIP-66 Refactor DagRun to DagVersion association #46626](https://github.com/apache/airflow/pull/46626) be merged, `dag_run. dag_versions` could directly retrieve `DagVersion` models.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1946779881


##########
airflow/api_fastapi/core_api/routes/public/dag_run.py:
##########
@@ -63,15 +65,85 @@
 from airflow.api_fastapi.logging.decorators import action_logging
 from airflow.exceptions import ParamValidationError
 from airflow.listeners.listener import get_listener_manager
-from airflow.models import DAG, DagModel, DagRun
+from airflow.models import DAG, DagModel, DagRun, TaskInstance
 from airflow.models.dag_version import DagVersion
+from airflow.models.taskinstancehistory import TaskInstanceHistory
 from airflow.timetables.base import DataInterval
 from airflow.utils.state import DagRunState
 from airflow.utils.types import DagRunTriggeredByType, DagRunType
 
 dag_run_router = AirflowRouter(tags=["DagRun"], prefix="/dags/{dag_id}/dagRuns")
 
 
+def get_dag_run_with_task_ids(dag_id: str, run_id: str, session: SessionDep) -> DagRun:
+    """Get the DagRun with the given task_ids."""
+    return session.scalar(
+        select(DagRun)
+        .options(joinedload(DagRun.task_instances).load_only("task_id"))
+        .filter_by(dag_id=dag_id, run_id=run_id)
+    )
+
+
+def get_dag_version_ids_among_dag_run(dag_id: str, dag_run_id: str, task_ids: list[str], session: SessionDep):
+    """Get the DagVersions from the TaskInstances and TaskInstanceHistories of the DagRun."""
+    task_instance_dag_version_ids = select(TaskInstance.dag_version_id).where(
+        TaskInstance.dag_id == dag_id, TaskInstance.run_id == dag_run_id, TaskInstance.task_id.in_(task_ids)
+    )
+    task_instance_history_dag_version_ids = select(TaskInstanceHistory.dag_version_id).where(
+        TaskInstanceHistory.dag_id == dag_id,
+        TaskInstanceHistory.run_id == dag_run_id,
+        TaskInstanceHistory.task_id.in_(task_ids),
+    )
+    return (
+        session.execute(
+            task_instance_dag_version_ids.distinct().union(task_instance_history_dag_version_ids.distinct())
+        )
+        .scalars()
+        .all()
+    )

Review Comment:
   I just added the `dag_versions_from_task_instance` and `dag_versions_from_task_instance_history` association proxies to `DagRun` and also added the `dag_versions` property. However, I'm okay with waiting for #46565.  
   
   If it's not finished by the weekend, I can take care of #46565 then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2682310601

   Rebased, conflict solved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #46484:
URL: https://github.com/apache/airflow/pull/46484#discussion_r1966491257


##########
tests/api_fastapi/core_api/routes/public/test_dag_run.py:
##########
@@ -146,14 +148,41 @@ def setup(request, dag_maker, session=None):
     )
 
     dag_maker.sync_dagbag_to_db()
-    dag_maker.dag_model
     dag_maker.dag_model.has_task_concurrency_limits = True
     session.merge(ti1)
     session.merge(ti2)
     session.merge(dag_maker.dag_model)
     session.commit()
 
 
+def get_dag_versions_dict(dag_version: DagVersion):
+    return DagVersionResponse.model_validate(dag_version, from_attributes=True).model_dump(mode="json")

Review Comment:
   We must set `mode="json"` (the default is `mode="python"`) or `created_at` and `id` will be Python `datetime` and `UUID` objects instead of strings.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-65 | Add dag versions to DAGRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #46484:
URL: https://github.com/apache/airflow/pull/46484#issuecomment-2677848372

   > Looks good, some tests need fixing.
   
   The CI failure is due to `sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked`. We may need to retrigger the SQLite test ~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org