[D] Dataset and data intervals [airflow]

["akomisarek (via GitHub)" <gi...@apache.org>]

GitHub user akomisarek created a discussion: Dataset and data intervals

What's the intention behind data interval for Dataset? The documentation doesn't mentioned it at all, code implements something (but due to unreliable behaviour for Datasets during backfill/catch-up it doesn't seem to be something to rely on). 

Astronomer documentation claims it is even not supported:
https://www.astronomer.io/docs/learn/airflow-datasets/#dataset-schedules
```
DAGs that are triggered by datasets do not have the concept of a data interval. If you need information about the triggering event in your downstream DAG, you can use the parameter triggering_dataset_events from the context. This parameter provides a list of all the triggering dataset events with the parameters [timestamp, source_dag_id, source_task_id, source_run_id, source_map_index ]. See [Retrieving information in a downstream task](https://www.astronomer.io/docs/learn/airflow-datasets#retrieving-extra-information-in-a-downstream-task) for an example.
```

Is using the data interval information in template dangerous (it seems provided)? But alternatively how to access WHAT exactly was modified in the Dataset? Any tips here how people are reading the relevant data/partition/files/rows? 



GitHub link: https://github.com/apache/airflow/discussions/46479

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org