[GitHub] [shardingsphere] sandynz commented on a diff in pull request #24570: Optimize cdc client insert performance, use batch mode.

["sandynz (via GitHub)" <gi...@apache.org>]

sandynz commented on code in PR #24570:
URL: https://github.com/apache/shardingsphere/pull/24570#discussion_r1133428075


##########
kernel/data-pipeline/cdc/client/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/client/importer/DataSourceImporter.java:
##########
@@ -57,7 +60,47 @@ public DataSourceImporter(final String databaseType, final ImportDataSourceParam
     }
     
     @Override
-    public void write(final Record record) throws Exception {
+    public void write(final List<Record> records) throws Exception {
+        List<Record> insertRecords = new LinkedList<>();
+        List<Record> otherRecords = new LinkedList<>();
+        for (Record each : records) {
+            if (DataChangeType.INSERT == each.getDataChangeType()) {
+                insertRecords.add(each);
+                continue;
+            }
+            otherRecords.add(each);

Review Comment:
   Could we support batch delete?



##########
kernel/data-pipeline/cdc/client/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/client/importer/DataSourceImporter.java:
##########
@@ -57,7 +60,47 @@ public DataSourceImporter(final String databaseType, final ImportDataSourceParam
     }
     
     @Override
-    public void write(final Record record) throws Exception {
+    public void write(final List<Record> records) throws Exception {

Review Comment:
   Does `records` belong to the same table? If not, then need to group by table firstly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org