You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Raul824 (via GitHub)" <gi...@apache.org> on 2025/09/11 01:11:13 UTC

[I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Raul824 opened a new issue, #55489:
URL: https://github.com/apache/airflow/issues/55489

   ### Description
   
   We migrated control M to airflow around 2023. It was a mass migration and we tried to replicate the logic and use Airflow in whatever we could do best.
   
   Our most dags were dependent on each other on date basis instead of time so we used execution_date_fn to achieve the dependency functionality.
   
   It was working all good till 2.x versions. But now am facing a peculiar issue with asset aware schedules.
   
   We grouped multiple batch jobs into a single dag using taskgroups according to the area of those batch jobs.
   These taskgroups have external task sensor as well depending on other taskgroup in external dags.
   
   This all works fine in cases if a Dag is Scheduled or being triggered by TriggerDagrunOperator.
   But now due to changes in AssetAwareScheduling in Airflow 3.x the migration which I have to do is stuck as the dags being triggered by AssetAwareScheduling are not having data_interval_start and data_interval_end.
   
   
   How do I set the dependencies now as my external dags are having logical_date value  2025-09-01 13:59:00+00:00
   But the asset aware ones have logical_date=None data_interval_start=None data_interval_end=None
   
   So if there is a delay and due to which the DAG A with logical date value 2025-09-01 13:59:00+00:00 triggered the DAG B with asset aware scheduling, how do I check whether DAG C for 2025-09-01 13:59:00+00:00 has completed or not.
   
   In Airflow 2.x versions I was able to check all external dags using data_interval_start. But now I am lost.
   
   Airflow 2.x
   
   <img width="1143" height="967" alt="Image" src="https://github.com/user-attachments/assets/67cd7c9c-ddee-4d06-888e-d8297c720865" />
   
   
   Airflow 3.x
   
   <img width="1027" height="1094" alt="Image" src="https://github.com/user-attachments/assets/d487d5b2-0d5e-4296-b749-15299180a4ae" />
   
   
   Any suggestions would be helpful as I have to migrate 728 dags which consist of 1612 task groups.
   
   
   
   ### Use case/motivation
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "karjaljo (via GitHub)" <gi...@apache.org>.
karjaljo commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3990889130

   Sounds great, thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3281820873

   @Lee-W - is it possible to get information about the DAG run that outlet to an Asset in the downstream DAG? I know that we can do this, but I'm curious if there is more information that's available.
   
   https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/asset-scheduling.html#fetching-information-from-a-triggering-asset-event


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3281787172

   @Raul824, you should be able to click into the DAG Run that created a "triggering" Asset Event. Would that satisfy the functionality that you're looking for?
   
   <img width="1351" height="664" alt="Image" src="https://github.com/user-attachments/assets/3b7c7504-2adf-4d47-9540-94c5e33acfd9" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3490091049

   As mentioned above, you can just pass the information in the event extra if you want. We explcitly removed data interval (and logical date) from the downstream dag run in Airflow 3 since it is impossible to create a logic that generates a data interval for asset-triggered dags that’s meaningful for everyone, especially since a dag can depend on more than one asset, potentially declared on dags with different schedules.
   
   We are working on a new, more generic concept called asset partitions that would allow you to declare how you want to derive intervals from upstream schedules. This will also be available in time-scheduled dag runs (and eventually replace data intervals), so time and asset-triggered dag scheduling can use the same interface to retrieve information.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "vikrantkumar-max (via GitHub)" <gi...@apache.org>.
vikrantkumar-max commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3278529690

   I would like to work on this issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "Raul824 (via GitHub)" <gi...@apache.org>.
Raul824 commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3490181986

   @uranusjr Just let me know how can I set an external task sensor on Asset Aware triggered dag.
   
   Removing logical date from asset aware dag has broken external task sensors on Asset Aware dags.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "Raul824 (via GitHub)" <gi...@apache.org>.
Raul824 commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3278567961

   @vikrantkumar-max do you know if  there is a way to pass logical date to Asset Triggered dag as it is even harder to set dependency on the Asset Aware Triggered Dag as they just get generated with run date of the Asset Completion.
   
   As below is an example of Asset Triggered now there is no value on which I can create a logical date value to set DAG_TEST external task sensor dependency on below.
   
   <DagRun UDPDVLP_9_SSOT_DAY1_B5 @ None: asset_triggered__2025-09-11T00:07:08.968068+00:00_xddTOcGu, state:failed, queued_at: 2025-09-11 00:59:12.408188+00:00. run_type: asset_triggered>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3281372354

   This seems to be intentional as per #46460 and the linked issues to wiki.
   
   cc: @Lee-W 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3287166689

   I just took a quick look. Do you know thing `context.run_after` is something you're looking for instead?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr closed issue #55489: Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag.
URL: https://github.com/apache/airflow/issues/55489


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3281724864

   yes, this is part of AIP-83


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "vikramkoka (via GitHub)" <gi...@apache.org>.
vikramkoka commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3474979073

   cc: @cmarteepants @uranusjr 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "Raul824 (via GitHub)" <gi...@apache.org>.
Raul824 commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3287688199

   @Lee-W no I want a date accessible from external dags as well. Can you access DAG_A's run_after from DAG_B. 
   Could you please read the use cases I have put "run_after" isn't available when you are trying to poke a dag for external task dependency.
   I can't even understand how the external task dependency works now for Asset Triggered dags as Logical Date itself is None now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "Raul824 (via GitHub)" <gi...@apache.org>.
Raul824 commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3277097081

   My main issue is there should be at-least one common date/timestamp between all triggering strategies so that they can be linked using external task sensors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3281849573

   Append that information from `context` to the asset event through `yield Metadata`.... should work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "Raul824 (via GitHub)" <gi...@apache.org>.
Raul824 commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3283404264

   Well my main concern is external dag dependency logic creation across different strategies.
   as @Lee-W @jroachgolf84 you guys suggest getting source run id and yield metadata is possible from Asset Triggered dag. 
   
   But my main point is A consistent date/timestamp which is common across the different triggering strategies let me try to elaborate.
   
   Let's take three daily scheduled dags DAG_SCHED_A, DAG_SCHED_B and DAG_SCHED_X.
   DAG_SCHED_A is triggering a dag let's name it DAG_TRIG_C.
   DAG_TRIG_C is triggering a dag using asset aware scheduling DAG_ASSET_D.
   
   DAG_ASSET_D has to run after DAG_SCHED_X 
   and DAG_SCHED_B has to run after DAG_ASSET_D.
   
   Due to different logical date generation it becomes very hard to set these dependencies.
   
   DAG_ASSET_D external Dependency on DAG_SCHED_X --> It can be achieved by yield metadata or extracting source run id and checking the dependency based on source run logical date.
   
   DAG_SCHED_B external Dependency on DAG_ASSET_D --> This becomes tricky as there is no logical date in DAG_ASSET_D, and getting source run logical date of DAG_ASSET_D from DAG_SCHED_B would be too complex.
   
   If a logical date can be passed to Asset Triggered dag that would actually solve the problem of bridging the gap between dependencies.
   
   **From your first scheduled dag itself you can pass a logical date to your triggered dag (which is already possible in trigger dag operator) then from your triggered dag if you can pass a logical date to you Asset Triggered dag it would bridge the missing link to align your dependencies based on one logical date of first Upstream Dag.**
   
   I know I can re-write most of these to use triggered dag instead of Asset Aware but, **I think logical date as None has opened this possibility to open an option of passing a logical date to Asset Triggered dag, it can be optional but it would help align your dags for better external dag dependencies.**


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3990698268

   In 3.2, we'll be able to use `dag_run.partition_key` for a similar purpose.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Asset Aware Scheduling - Getting the logical date of the asset which triggered the dag. [airflow]

Posted by "karjaljo (via GitHub)" <gi...@apache.org>.
karjaljo commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3990641833

   Was not sure if it's better to open a new thread so I posted here first even though this is closed.
   
   My use case is as follows. I have a set of files coming in at random times within 5min intervals. Each of these files have the timestamp of the 5min end time for instance data_file_202603031205.h5. The first dag makes sure that enough data has arrived and then creates a dataset event. Next DAG is triggered and it does some additional processing for the files with the same timestamp. Now this worked perfectly in airflow 2.11, but now that I try to upgrade to 3.1 this whole processing chain fails due to change in the logical_date value missing from the dataset/asset triggered dagrun. Is there any way to get around this?
   
   How do I run this kind of task in airflow 3.1 in a dag that is triggered by an asset event ?
   poll_for_network_status = FileSensor(
         task_id="wait_for_network_status",
         fs_conn_id="fs_tutka",
         filepath=f"/path/to/file/{{{{ logical_date.strftime('%Y%m%d%H%M') }}}}_data_file.h”5,
         deferrable=True,
         poke_interval=10,
         timeout=60 * 5,
         mode="reschedule",
     )
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org