You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "bbovenzi (via GitHub)" <gi...@apache.org> on 2025/01/23 23:30:52 UTC

[I] AIP-65 | Add dag versions to DagRunResponse [airflow]

bbovenzi opened a new issue, #45995:
URL: https://github.com/apache/airflow/issues/45995

   ### Body
   
   Add `dag_versions` as an array of uuids to the DagRunResponse in the public REST API. Most of the time this will be an array of length 1, but some dag runs can have multiple versions.
   
   ### Committer
   
   - [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2631236399

   Ideally I think you are right. Version could have changed in between tries and we need to explore the TaskInstanceHistory to get that information.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2634087976

   > Ideally I think you are right. Version could have changed in between tries and we need to explore the TaskInstanceHistory to get that information.
   
   Thanks for confirming !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2623142607

   I can open a PR to remove that direct link between DagRun and DagVersion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2613838154

   Hi @bbovenzi , I can work on this issue, could you assign to me ? Thanks !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2622050665

   @jason810496 Assigned!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2622538375

   
   Hi @bbovenzi, may I ask when the case of "but some DAG runs can have multiple versions" would occur?  
   
   Since there is a `dag_version_id` in the `DagRun` model, shouldn't there be only one `dag_version` per `DagRun`?  
   https://github.com/apache/airflow/blob/main/airflow/models/dagrun.py#L169-L170
   
   Or is making `dag_versions` a list mainly for UI purposes?
   For example, `latest_dag_runs: list[DAGRunResponse]` is used in `DAGWithLatestDagRunsResponse`. However, even in this case, each item in `latest_dag_runs` still corresponds to a `DagRun` model, which should have only one `dag_version` per `DagRun`.  
   
   Or is this related to `DagModel` instead of `DagRun`? Since `DagModel` has a `dag_versions` relationship, there can be multiple `dag_versions` for a `DagModel`.  
   
   Thanks!  
   cc @pierrejeambrun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2628996472

   After reading through AIP-63, 64, and 65 and reviewing the related codebase today, this endpoint should also consider `TaskInstanceHistory`, right?  
   It should join `DagRun`, `TaskInstance`, `TaskInstanceHistory`, and `DagVersion`, somewhat like `get_task_instance_tries`, but with granularity not down to the `task_id` level.
   
   https://github.com/apache/airflow/blob/260a988292855569e55777bcd1c7a085be3db0c1/airflow/api_fastapi/core_api/routes/public/task_instances.py#L260-L293


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2627098266

   Hi @pierrejeambrun,  
   
   After some POC work and deeper investigation, I found that even when **"Sometimes a dag version can change in the middle of a dag run. "** happens, all `TaskInstance`s of the same `DagRun` will still have the same `dag_version`. This means that the idea **"Retrieve DagRun versions as an aggregate of the DagRun TIs versions (i.e., all different versions used for all TIs are the version of the DagRun)"** is unlikely to happen.  
   
   Here’s my breakdown:  
   
   1. `dag_version` is passed as an argument to `dag.create_dagrun` and then passed down to `_create_orm_dagrun`.  
   2. The `DagRun` instance is created as `run` in `_create_orm_dagrun`, which then calls `run.verify_integrity`.  
   3. The `task_creator` factory, obtained from `self._get_task_creator`, creates instances of the `TaskInstance` model (let’s simplify these as `TIs`). Importantly, the **`dag_version_id` of all `TIs` is always taken from `self.dag_version_id`**, meaning all `dag_versions` of `TIs` will be the same as that of the `DagRun`.  
   4. All `TIs` created by `task_creator` are persisted using `self._create_task_instances`.  
   
   In summary, I believe we don’t need to remove the **direct link between `DagRun` and `DagVersion`**, and the return type of `dag_version` in this API will always be a single `UUID`.  
   
   Am I correct in this assumption?  
   Or is there actually a scenario where **different `dag_versions` can be found among `TIs` in the same `DagRun`** that I haven't considered?  
   
   Thanks!  
   
   cc @ephraimbuddy  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2627399959

   You are spot on how it behaves now, but we need to refactor that so it _will_ be possible at some point in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2623140389

   There is a FK from version to TI and from dagrun to versions.
   
   Dagrun to version needs to be deleted. And we need to retrieve dagrun versions as an aggregate of the dagrun TIs versions. (All different versions used for all TIs are the version of the dagrun)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on issue #45995:
URL: https://github.com/apache/airflow/issues/45995#issuecomment-2622638757

   Oh yeah this one is quite complicated. Sometimes a dag version can change in the middle of a dag run. We may need to wait until that schema is updated to allow a many to many relationship cc: @jedcunningham @ephraimbuddy 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] AIP-65 | Add dag versions to DagRunResponse [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun closed issue #45995: AIP-65 | Add dag versions to DagRunResponse
URL: https://github.com/apache/airflow/issues/45995


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org