[GitHub] [airflow] dstandish commented on a diff in pull request #26260: Implement fast insert for mysql and sqlite

[GitBox <gi...@apache.org>]

dstandish commented on code in PR #26260:
URL: https://github.com/apache/airflow/pull/26260#discussion_r973203953


##########
airflow/datasets/manager.py:
##########
@@ -94,14 +95,26 @@ def _slow_path_queue_dagruns(self, dataset: DatasetModel, session: Session) -> N
             except exc.IntegrityError:
                 self.log.debug("Skipping record %s", item, exc_info=True)
 
-    def _postgres_queue_dagruns(self, dataset: DatasetModel, session: Session) -> None:
-        from sqlalchemy.dialects.postgresql import insert
-
-        stmt = insert(DatasetDagRunQueue).values(dataset_id=dataset.id).on_conflict_do_nothing()
-        session.execute(
-            stmt,
-            [{'target_dag_id': target_dag.dag_id} for target_dag in dataset.consuming_dags],
-        )
+    def _queue_dagruns_fast_path(self, dataset: DatasetModel, session: Session) -> None:
+        prefix = None
+        if session.bind.dialect.name == "postgresql":
+            from sqlalchemy.dialects.postgresql import insert
+        elif session.bind.dialect.name == "sqlite":
+            from sqlalchemy.dialects.sqlite import insert
+        elif session.bind.dialect.name == "mysql":
+            from sqlalchemy.dialects.mysql import insert
+
+            prefix = 'IGNORE'
+        else:
+            raise ValueError("Only sqlite, postgres, and mysql supported with this method.")
+        params = [{'target_dag_id': target_dag.dag_id} for target_dag in dataset.consuming_dags]
+        if params:

Review Comment:
   yeah... this is vestigial... a remnant from refactor.... 
   it doesn't hurt to be defensive in this way, and handle calling with empty set... but better readability would be to exit based on empty dag_ids list!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org