Re: [I] Datasets: Allow time based force dependency when dataset is not ready. [airflow]

["cmarteepants (via GitHub)" <gi...@apache.org>]

cmarteepants commented on issue #30974:
URL: https://github.com/apache/airflow/issues/30974#issuecomment-1921943496

   > Hi, just to provide a bit more info on one of the use cases that inspired this issue:
   > 
   > * We have two datasets, say A and B, both of which are provided by third parties (and therefore fail occasionally)
   > * A is updated daily, B is updated weekly
   > * Dataset C is effectively a join of A and B
   >   
   >   * Whenever A is updated, C needs to be updated
   >   * We take the newly available A, and the _latest available_ version of B to compute C
   > 
   > Our current plan to handle this is to mark only A as an inlet to C, and handle the dependency on B within the DAG itself. Disadvantage would be that B doesn't show up in the Datasets view at all, which we're not too happy about - A and B are both crucial to the system, it's just that B has a higher TTL.
   
   @dakshin-k I know it's been awhile, but I am curious re your example:
   
   Let's say you had the ability to compute C whenever A or B is updated, allowing you to set A and B as inlets to C - would that work for your use-case? It would allow for C to be computed whenever A is updated using the most recent version of B, but it also would mean that C would be computed an extra time - whenever B is updated. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org