[GitHub] [airflow] chuxiangfeng commented on issue #30974: Datasets: Allow time based force dependency when dataset is not ready.

["chuxiangfeng (via GitHub)" <gi...@apache.org>]

chuxiangfeng commented on issue #30974:
URL: https://github.com/apache/airflow/issues/30974#issuecomment-1534409209

   > Hi, just to provide a bit more info on one of the use cases that inspired this issue:
   > 
   > * We have two datasets, say A and B, both of which are provided by third parties (and therefore fail occasionally)
   > * A is updated daily, B is updated weekly
   > * Dataset C is effectively a join of A and B
   >   
   >   * Whenever A is updated, C needs to be updated
   >   * We take the newly available A, and the _latest available_ version of B to compute C
   > 
   > Our current plan to handle this is to mark only A as an inlet to C, and handle the dependency on B within the DAG itself. Disadvantage would be that B doesn't show up in the Datasets view at all, which we're not too happy about - A and B are both crucial to the system, it's just that B has a higher TTL.
   
   Yes, we will have this dilemma and expect to enhance it in later versions with algorithms like Dagster freshness scheduling.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org