You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/01 15:30:15 UTC

[GitHub] [airflow] ashb commented on a diff in pull request #24969: WIP - Move dataset dagrun creation to scheduler main

ashb commented on code in PR #24969:
URL: https://github.com/apache/airflow/pull/24969#discussion_r934589371


##########
airflow/models/dag.py:
##########
@@ -2991,14 +2991,38 @@ def deactivate_deleted_dags(cls, alive_dag_filelocs: List[str], session=NEW_SESS
                 continue
 
     @classmethod
-    def dags_needing_dagruns(cls, session: Session):
+    def dags_needing_dagruns(cls, session: Session) -> Tuple[Query, Dict]:
         """
         Return (and lock) a list of Dag objects that are due to create a new DagRun.
 
         This will return a resultset of rows  that is row-level-locked with a "SELECT ... FOR UPDATE" query,
         you should ensure that any scheduling decisions are made in a single transaction -- as soon as the
         transaction is committed it will be unlocked.
         """
+        from airflow.models.dataset import DatasetDagRef, DatasetDagRunQueue as DDRQ
+
+        # these dag ids are triggered by datasets, and they are ready to go.
+        dataset_triggered_dag_info_list = {
+            x.dag_id: (x.first_event_time, x.last_event_time)
+            for x in session.query(
+                DatasetDagRef.dag_id,
+                func.max(DDRQ.created_at).label('last_event_time'),
+                func.max(DDRQ.created_at).label('first_event_time'),
+            )
+            .join(
+                DDRQ,
+                and_(
+                    DDRQ.dataset_id == DatasetDagRef.dataset_id,
+                    DDRQ.target_dag_id == DatasetDagRef.dag_id,
+                ),
+                isouter=True,
+            )

Review Comment:
   We should define the relationships on the DDRQ table (that way we only have to have it once, not in every place we might want to use this rel) then this can become something like
   
   
   ```suggestion
               .join(DatassetDagRef._something_)
   ```



##########
airflow/jobs/scheduler_job.py:
##########
@@ -997,8 +998,17 @@ def _get_next_dagruns_to_examine(self, state: DagRunState, session: Session):
     @retry_db_transaction
     def _create_dagruns_for_dags(self, guard, session):
         """Find Dag Models needing DagRuns and Create Dag Runs with retries in case of OperationalError"""
-        query = DagModel.dags_needing_dagruns(session)
-        self._create_dag_runs(query.all(), session)
+        query, dataset_triggered_dag_info = DagModel.dags_needing_dagruns(session)
+        all_dags_needing_dag_runs = set(query.all())
+        dataset_triggered_dags = [
+            x for x in all_dags_needing_dag_runs if x.dag_id in dataset_triggered_dag_info.keys()

Review Comment:
   ```suggestion
               dag for dag in all_dags_needing_dag_runs if dag.dag_id in dataset_triggered_dag_info
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org