[GitHub] [airflow] jedcunningham commented on issue #26256: "triggered runs" dataset counter doesn't update until *next* run and never goes above 1

[GitBox <gi...@apache.org>]

jedcunningham commented on issue #26256:
URL: https://github.com/apache/airflow/issues/26256#issuecomment-1242223978

   So there are a couple of things going on here.
   
   The first is that the "triggered dagruns" column is empty sometimes. This is being fixed in #26256.
   
   Next is that you expect greater than 1 with the DAGs you are testing with. You have many updates to a single dataset, and a single downstream consumer. With this setup, by definition each dataset event can only trigger a single downstream run - even though there are many events, each specific event can only trigger the one downstream DAG once. In contrast, the follow setup where there are multiple consumers of the dataset will properly show more than 1 triggered dagrun per event.
   
   ```python
   from datetime import datetime
   
   from airflow import DAG, Dataset
   from airflow.operators.empty import EmptyOperator
   
   dataset = Dataset("hello")
   
   with DAG(
       dag_id="dataset_many_upstream",
       start_date=datetime(2022, 8, 1),
       schedule="@daily",
   ):
       EmptyOperator(task_id="upstream_task", outlets=[dataset])
   
   with DAG(
       dag_id="dataset_many_downstream1",
       start_date=datetime(2022, 8, 1),
       schedule=[dataset],
   ):
       EmptyOperator(task_id="downstream")
   
   with DAG(
       dag_id="dataset_many_downstream2",
       start_date=datetime(2022, 8, 1),
       schedule=[dataset],
   ):
       EmptyOperator(task_id="downstream")
   ```
   
   ![Screen Shot 2022-09-09 at 10 29 21 AM](https://user-images.githubusercontent.com/66968678/189401731-b9c69a8d-789a-4d1f-8717-2f70949848c0.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org