You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jialerchew (via GitHub)" <gi...@apache.org> on 2023/05/02 11:20:22 UTC

[GitHub] [airflow] jialerchew opened a new issue, #31013: Datasets: Allow expiration time for datasets

jialerchew opened a new issue, #31013:
URL: https://github.com/apache/airflow/issues/31013

   ### Description
   
   **The current state:** Example, if DAG is defined with 3 Datasets. Once 3 datasets are updated at least once, Airflow will schedule the DAG. This is pretty good for most use cases.
   
   However, in the event if one of the Dataset Producers stopped working and missed/failed some of its supposed runs, the downstream DAG shouldn't be executed anymore, and the most recent Dataset should be made invalid.
   
   An quick way to implement this would be a concept of "freshness", where the Dataset will only be valid for a period of time.
   
   Using the example below, if `example_dataset_2` is already 24 hours old, and there is a "freshness" threshold of **12 hours**, I dont want `example_dataset_2` to be counted as “updated” anymore. Hence, when `example_dataset_3` updates, the DAG will still not be triggered, because `example_dataset_2` has already passed the 12-hour-freshness threshold.
   
   ![image](https://user-images.githubusercontent.com/8098670/235651695-6bbfa7e3-2030-449b-82bd-b3ef58e5dbdc.png)
   
   
   ### Use case/motivation
   
   My team is trying to move towards "reactive" DAGs, where we don't want to schedule downstreams DAGs and use sensors. This is because we are trying to reduce redundant DAG executions, and it's easier for the team to manually retrigger failed DAG runs. (Just trigger upstream, and it will automatically run downstream DAGs)
   
   Datasets is the perfect use case for us; however we are not completely comfortable to switch towards Datasets because it doesn't protect us from outdated Datasets. Regular scheduling + sensors combo do not face this issue because that method always refers to the exact task_id defined from `execution_delta`.
   
   We don't need such precision, just a way to measure "freshness" of Datasets would be good enough.
   
   Inspired from this [thread](https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1682791720916639) on Airflow Slack.
   
   ### Related issues
   
   Found this [other issue](https://github.com/apache/airflow/issues/30974), which is the complete opposite of this feature request.
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #31013: Datasets: Allow expiration time for datasets

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #31013:
URL: https://github.com/apache/airflow/issues/31013#issuecomment-1531297010

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #31013: Datasets: Allow expiration time for datasets

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31013:
URL: https://github.com/apache/airflow/issues/31013#issuecomment-1531395463

   Similar to #30974:
   
   Just a reminder on that one (I personally have no opinion enough to chime in) but - since this is a change How airflow architecture is designed around datasets and their dependencies, whatever is discussed here shoudl be brought to the devlist as proposal: https://airflow.apache.org/community/ -> people might not follow this discussion / feature request here and only a devlist discussion will make it into "implementable" change. I guess it's fine to disucss it here to get the idea and concepts hashed out, but whoever would likel to implement, it should be summarised there, and the actual decision making should happen on the devlist - possibly in the form of AIP (Airflow Improvement Proposal) if the scope of it will turn to be big. 
   
   For me this is at the border of feature/discussion. Usually feature is something that adds a small thing that does not change the internals, in this case we are talking about changing the ways how dependencies are resolved by scheduler, when the scheduler considers  them fulfilled, and when they are "eligible" for running, so something rather critical in terms of "core of airflow". I could see that one converted to discsussion, but I think those two issue could be discussed together as set of things to improve to datasets and brought as an AIP for discussion there (it's ok to discuss them here - but only when they are brought to devlist the might have a chance to have some decisions on those).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] da-tubi commented on issue #31013: Datasets: Allow expiration time for datasets

Posted by "da-tubi (via GitHub)" <gi...@apache.org>.
da-tubi commented on issue #31013:
URL: https://github.com/apache/airflow/issues/31013#issuecomment-1643568519

   > My team is trying to move towards "reactive" DAGs, where we don't want to schedule downstreams DAGs and use sensors. This is because we are trying to reduce redundant DAG executions, and it's easier for the team to manually retrigger failed DAG runs. (Just trigger upstream, and it will automatically run downstream DAGs)
   
   We need "reactive" DAGs too.
   
   In my option, we need a Dataset with a logical start date.
   
   For example:
   
   | dataset | logical date |
   |-------------|------------------|
   | example dataset 1 | 2023/01/01 01:00:00 | 
   | example dataset 2 | 2023/01/01 02:00:00 | 
   | example dataset 3 | 2023/01/01 03:00:00 | 
   
   The above three dataset events should trigger the dag_run.
   
   But if there are not in the same logical date, it will not trigger the dag_run.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] da-tubi commented on issue #31013: Datasets: Allow expiration time for datasets

Posted by "da-tubi (via GitHub)" <gi...@apache.org>.
da-tubi commented on issue #31013:
URL: https://github.com/apache/airflow/issues/31013#issuecomment-1643650422

   ![image](https://github.com/apache/airflow/assets/66561265/bc6fe3d5-5fed-4740-a68c-73b215f3c5e2)
   
   I made a graph to visualize the problem.
   
   There are 5 dataset events for the three datasets. But only 4 downstream (the red line) event triggerred. And the green line which should be triggerred but it is missing.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #31013: Datasets: Allow expiration time for datasets

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31013:
URL: https://github.com/apache/airflow/issues/31013#issuecomment-1656940982

   Let me repeat again. All those ideas are great to discuss them in "devlist" not here. Discussions are a great place to have a quick discussion on some subject, but in order to propose and add significant change to airflow, one has to follow the way where proposals are made, discussions are done and decisions are made by voting. There is already discussion started in the devlist  about dataset:
   
   https://lists.apache.org/thread/l9shrbv35yssp3gy41r2f1381k697bqn
   
   and clearly an "Airflow Improvement Proposal" - https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals  is going to written soon describing the features that Dataset direction is going to be implemented (look at the discussion). If you want to take part in shaping the future of Dataset - taking part in THAT discussion and in creating and writing the Airflow Improvement Proposal is the right approach. Discussing it here is nothing than "back-chatter" which might be simply missed by those who will be discussing the proposal. 
   
   I will close this one as an issue and suggest anyone who wants to dicuss a change to dataset discuss it in the devlist discussion and later when AIP is proposed to take part in shaping it up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed issue #31013: Datasets: Allow expiration time for datasets

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #31013: Datasets: Allow expiration time for datasets
URL: https://github.com/apache/airflow/issues/31013


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org