You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "azexcy (via GitHub)" <gi...@apache.org> on 2023/03/13 02:11:51 UTC

[GitHub] [shardingsphere] azexcy opened a new pull request, #24570: Optimize cdc client insert performance, use batch mode.

azexcy opened a new pull request, #24570:
URL: https://github.com/apache/shardingsphere/pull/24570

   Relate #22500 
   
   Changes proposed in this pull request:
     - Optimize cdc client insert performance, use batch mode.
   
   ---
   
   Before committing this PR, I'm sure that I have checked the following options:
   - [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
   - [ ] I have self-reviewed the commit code.
   - [ ] I have (or in comment I request) added corresponding labels for the pull request.
   - [ ] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
   - [ ] I have made corresponding changes to the documentation.
   - [ ] I have added corresponding unit tests for my changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] codecov-commenter commented on pull request #24570: Optimize cdc client insert performance, use batch mode.

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24570:
URL: https://github.com/apache/shardingsphere/pull/24570#issuecomment-1465429035

   ## [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/24570?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#24570](https://codecov.io/gh/apache/shardingsphere/pull/24570?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8e97d8d) into [master](https://codecov.io/gh/apache/shardingsphere/commit/ab67665e6c8d6048250ee4921985642d9aa6729b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ab67665) will **decrease** coverage by `0.03%`.
   > The diff coverage is `0.00%`.
   
   > :exclamation: Current head 8e97d8d differs from pull request most recent head e533647. Consider uploading reports for the commit e533647 to get more accurate results
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #24570      +/-   ##
   ============================================
   - Coverage     50.01%   49.98%   -0.03%     
     Complexity     1595     1595              
   ============================================
     Files          3310     3310              
     Lines         54053    54082      +29     
     Branches       9941     9949       +8     
   ============================================
     Hits          27034    27034              
   - Misses        24603    24632      +29     
     Partials       2416     2416              
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/24570?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...pipeline/cdc/client/handler/CDCRequestHandler.java](https://codecov.io/gh/apache/shardingsphere/pull/24570?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jZGMvY2xpZW50L2hhbmRsZXIvQ0RDUmVxdWVzdEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...peline/cdc/client/importer/DataSourceImporter.java](https://codecov.io/gh/apache/shardingsphere/pull/24570?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jZGMvY2xpZW50L2ltcG9ydGVyL0RhdGFTb3VyY2VJbXBvcnRlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] azexcy closed pull request #24570: Optimize cdc client insert performance, use batch mode.

Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy closed pull request #24570: Optimize cdc client insert performance, use batch mode.
URL: https://github.com/apache/shardingsphere/pull/24570


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] azexcy commented on pull request #24570: Optimize cdc client insert performance, use batch mode.

Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on PR #24570:
URL: https://github.com/apache/shardingsphere/pull/24570#issuecomment-1471500968

   After the CDCprotocol refactor, the code is out of date, so closed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] sandynz commented on a diff in pull request #24570: Optimize cdc client insert performance, use batch mode.

Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #24570:
URL: https://github.com/apache/shardingsphere/pull/24570#discussion_r1133428075


##########
kernel/data-pipeline/cdc/client/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/client/importer/DataSourceImporter.java:
##########
@@ -57,7 +60,47 @@ public DataSourceImporter(final String databaseType, final ImportDataSourceParam
     }
     
     @Override
-    public void write(final Record record) throws Exception {
+    public void write(final List<Record> records) throws Exception {
+        List<Record> insertRecords = new LinkedList<>();
+        List<Record> otherRecords = new LinkedList<>();
+        for (Record each : records) {
+            if (DataChangeType.INSERT == each.getDataChangeType()) {
+                insertRecords.add(each);
+                continue;
+            }
+            otherRecords.add(each);

Review Comment:
   Could we support batch delete?



##########
kernel/data-pipeline/cdc/client/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/client/importer/DataSourceImporter.java:
##########
@@ -57,7 +60,47 @@ public DataSourceImporter(final String databaseType, final ImportDataSourceParam
     }
     
     @Override
-    public void write(final Record record) throws Exception {
+    public void write(final List<Record> records) throws Exception {

Review Comment:
   Does `records` belong to the same table? If not, then need to group by table firstly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org