[GitHub] [airflow] uranusjr commented on a diff in pull request #26260: Implement fast insert for mysql and sqlite

[GitBox <gi...@apache.org>]

uranusjr commented on code in PR #26260:
URL: https://github.com/apache/airflow/pull/26260#discussion_r972730269


##########
airflow/datasets/manager.py:
##########
@@ -94,14 +95,26 @@ def _slow_path_queue_dagruns(self, dataset: DatasetModel, session: Session) -> N
             except exc.IntegrityError:
                 self.log.debug("Skipping record %s", item, exc_info=True)
 
-    def _postgres_queue_dagruns(self, dataset: DatasetModel, session: Session) -> None:
-        from sqlalchemy.dialects.postgresql import insert
-
-        stmt = insert(DatasetDagRunQueue).values(dataset_id=dataset.id).on_conflict_do_nothing()
-        session.execute(
-            stmt,
-            [{'target_dag_id': target_dag.dag_id} for target_dag in dataset.consuming_dags],
-        )
+    def _queue_dagruns_fast_path(self, dataset: DatasetModel, session: Session) -> None:
+        prefix = None
+        if session.bind.dialect.name == "postgresql":
+            from sqlalchemy.dialects.postgresql import insert
+        elif session.bind.dialect.name == "sqlite":
+            from sqlalchemy.dialects.sqlite import insert
+        elif session.bind.dialect.name == "mysql":
+            from sqlalchemy.dialects.mysql import insert
+
+            prefix = 'IGNORE'
+        else:
+            raise ValueError("Only sqlite, postgres, and mysql supported with this method.")
+        params = [{'target_dag_id': target_dag.dag_id} for target_dag in dataset.consuming_dags]
+        if params:

Review Comment:
   It would if `consuming_dags` is empty (not sure if it’s possible in the context).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org