You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "azexcy (via GitHub)" <gi...@apache.org> on 2023/03/13 02:11:51 UTC
[GitHub] [shardingsphere] azexcy opened a new pull request, #24570: Optimize cdc client insert performance, use batch mode.
azexcy opened a new pull request, #24570:
URL: https://github.com/apache/shardingsphere/pull/24570
Relate #22500
Changes proposed in this pull request:
- Optimize cdc client insert performance, use batch mode.
---
Before committing this PR, I'm sure that I have checked the following options:
- [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
- [ ] I have self-reviewed the commit code.
- [ ] I have (or in comment I request) added corresponding labels for the pull request.
- [ ] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
- [ ] I have made corresponding changes to the documentation.
- [ ] I have added corresponding unit tests for my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter commented on pull request #24570: Optimize cdc client insert performance, use batch mode.
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24570:
URL: https://github.com/apache/shardingsphere/pull/24570#issuecomment-1465429035
## [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/24570?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#24570](https://codecov.io/gh/apache/shardingsphere/pull/24570?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8e97d8d) into [master](https://codecov.io/gh/apache/shardingsphere/commit/ab67665e6c8d6048250ee4921985642d9aa6729b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ab67665) will **decrease** coverage by `0.03%`.
> The diff coverage is `0.00%`.
> :exclamation: Current head 8e97d8d differs from pull request most recent head e533647. Consider uploading reports for the commit e533647 to get more accurate results
```diff
@@ Coverage Diff @@
## master #24570 +/- ##
============================================
- Coverage 50.01% 49.98% -0.03%
Complexity 1595 1595
============================================
Files 3310 3310
Lines 54053 54082 +29
Branches 9941 9949 +8
============================================
Hits 27034 27034
- Misses 24603 24632 +29
Partials 2416 2416
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/24570?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pipeline/cdc/client/handler/CDCRequestHandler.java](https://codecov.io/gh/apache/shardingsphere/pull/24570?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jZGMvY2xpZW50L2hhbmRsZXIvQ0RDUmVxdWVzdEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...peline/cdc/client/importer/DataSourceImporter.java](https://codecov.io/gh/apache/shardingsphere/pull/24570?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jZGMvY2xpZW50L2ltcG9ydGVyL0RhdGFTb3VyY2VJbXBvcnRlci5qYXZh) | `0.00% <0.00%> (ø)` | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy closed pull request #24570: Optimize cdc client insert performance, use batch mode.
Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy closed pull request #24570: Optimize cdc client insert performance, use batch mode.
URL: https://github.com/apache/shardingsphere/pull/24570
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on pull request #24570: Optimize cdc client insert performance, use batch mode.
Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on PR #24570:
URL: https://github.com/apache/shardingsphere/pull/24570#issuecomment-1471500968
After the CDCprotocol refactor, the code is out of date, so closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #24570: Optimize cdc client insert performance, use batch mode.
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #24570:
URL: https://github.com/apache/shardingsphere/pull/24570#discussion_r1133428075
##########
kernel/data-pipeline/cdc/client/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/client/importer/DataSourceImporter.java:
##########
@@ -57,7 +60,47 @@ public DataSourceImporter(final String databaseType, final ImportDataSourceParam
}
@Override
- public void write(final Record record) throws Exception {
+ public void write(final List<Record> records) throws Exception {
+ List<Record> insertRecords = new LinkedList<>();
+ List<Record> otherRecords = new LinkedList<>();
+ for (Record each : records) {
+ if (DataChangeType.INSERT == each.getDataChangeType()) {
+ insertRecords.add(each);
+ continue;
+ }
+ otherRecords.add(each);
Review Comment:
Could we support batch delete?
##########
kernel/data-pipeline/cdc/client/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/client/importer/DataSourceImporter.java:
##########
@@ -57,7 +60,47 @@ public DataSourceImporter(final String databaseType, final ImportDataSourceParam
}
@Override
- public void write(final Record record) throws Exception {
+ public void write(final List<Record> records) throws Exception {
Review Comment:
Does `records` belong to the same table? If not, then need to group by table firstly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org