You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "azexcy (via GitHub)" <gi...@apache.org> on 2023/03/03 08:56:25 UTC
[GitHub] [shardingsphere] azexcy opened a new pull request, #24440: Rename database to databaseName at CDC
azexcy opened a new pull request, #24440:
URL: https://github.com/apache/shardingsphere/pull/24440
Related #22500
Changes proposed in this pull request:
- Rename database to databaseName at CDC
- Adjust convert Date value position
---
Before committing this PR, I'm sure that I have checked the following options:
- [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
- [ ] I have self-reviewed the commit code.
- [ ] I have (or in comment I request) added corresponding labels for the pull request.
- [ ] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
- [ ] I have made corresponding changes to the documentation.
- [ ] I have added corresponding unit tests for my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz merged pull request #24440: Improve CDC protocol
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz merged PR #24440:
URL: https://github.com/apache/shardingsphere/pull/24440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter commented on pull request #24440: Adjust convert Date value position
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24440:
URL: https://github.com/apache/shardingsphere/pull/24440#issuecomment-1453247046
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#24440](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (85d779a) into [master](https://codecov.io/gh/apache/shardingsphere/commit/2f0a6ce11f683bffbe3d010b4fa1531bfdf374ed?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2f0a6ce) will **decrease** coverage by `0.02%`.
> The diff coverage is `48.38%`.
> :exclamation: Current head 85d779a differs from pull request most recent head e95a19a. Consider uploading reports for the commit e95a19a to get more accurate results
```diff
@@ Coverage Diff @@
## master #24440 +/- ##
============================================
- Coverage 49.93% 49.92% -0.02%
- Complexity 1583 1584 +1
============================================
Files 3256 3257 +1
Lines 53562 53581 +19
Branches 9860 9870 +10
============================================
+ Hits 26748 26750 +2
- Misses 24422 24432 +10
- Partials 2392 2399 +7
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...r/segment/strategy/KeyGenerateStrategySegment.java](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmVhdHVyZXMvc2hhcmRpbmcvZGlzdHNxbC9zdGF0ZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3NoYXJkaW5nL2Rpc3RzcWwvcGFyc2VyL3NlZ21lbnQvc3RyYXRlZ3kvS2V5R2VuZXJhdGVTdHJhdGVneVNlZ21lbnQuamF2YQ==) | `0.00% <ø> (ø)` | |
| [...data/pipeline/cdc/client/util/AnyValueConvert.java](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jZGMvY2xpZW50L3V0aWwvQW55VmFsdWVDb252ZXJ0LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...pipeline/cdc/util/DataRecordResultConvertUtil.java](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvY2RjL3V0aWwvRGF0YVJlY29yZFJlc3VsdENvbnZlcnRVdGlsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ere/data/pipeline/core/util/PipelineJdbcUtils.java](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jb3JlL3V0aWwvUGlwZWxpbmVKZGJjVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [.../pipeline/mysql/ingest/MySQLIncrementalDumper.java](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvZGlhbGVjdC9teXNxbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9teXNxbC9pbmdlc3QvTXlTUUxJbmNyZW1lbnRhbER1bXBlci5qYXZh) | `64.04% <0.00%> (-3.02%)` | :arrow_down: |
| [.../jdbc/core/driver/spi/ApolloDriverURLProvider.java](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-amRiYy9jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9kcml2ZXIvamRiYy9jb3JlL2RyaXZlci9zcGkvQXBvbGxvRHJpdmVyVVJMUHJvdmlkZXIuamF2YQ==) | `57.14% <57.14%> (ø)` | |
| [...data/pipeline/cdc/util/ColumnValueConvertUtil.java](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvY2RjL3V0aWwvQ29sdW1uVmFsdWVDb252ZXJ0VXRpbC5qYXZh) | `63.79% <71.42%> (-1.67%)` | :arrow_down: |
| [.../value/string/MySQLVarcharBinlogProtocolValue.java](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGItcHJvdG9jb2wvbXlzcWwvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RiL3Byb3RvY29sL215c3FsL3BhY2tldC9iaW5sb2cvcm93L2NvbHVtbi92YWx1ZS9zdHJpbmcvTXlTUUxWYXJjaGFyQmlubG9nUHJvdG9jb2xWYWx1ZS5qYXZh) | `100.00% <100.00%> (ø)` | |
| [...ere/data/pipeline/cdc/api/job/type/CDCJobType.java](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvY2RjL2FwaS9qb2IvdHlwZS9DRENKb2JUeXBlLmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...rdingsphere/transaction/utils/AutoCommitUtils.java](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL3RyYW5zYWN0aW9uL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3RyYW5zYWN0aW9uL3V0aWxzL0F1dG9Db21taXRVdGlscy5qYXZh) | `75.00% <100.00%> (-8.34%)` | :arrow_down: |
| ... and [3 more](https://codecov.io/gh/apache/shardingsphere/pull/24440?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #24440: Improve CDC protocol
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #24440:
URL: https://github.com/apache/shardingsphere/pull/24440#discussion_r1136928007
##########
kernel/data-pipeline/cdc/client/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/client/importer/DataSourceImporter.java:
##########
@@ -98,19 +128,63 @@ public void write(final Record record) throws Exception {
}
}
- private Optional<String> buildSQL(final Record record) {
+ private String buildKey(final String schema, final String tableName) {
+ return schema.isEmpty() ? tableName : String.join(".", schema, tableName);
+ }
+
+ private StandardTableMetaData loadTableMetaData(final String schema, final String tableName) {
+ StandardTableMetaData result = tableMetaDataMap.get(buildKey(schema, tableName));
+ if (null != result) {
+ return result;
+ }
+ result = loader.getTableMetaData(Strings.emptyToNull(schema), tableName);
+ tableMetaDataMap.put(buildKey(schema, tableName), result);
+ return result;
+ }
+
+ private Optional<String> buildSQL(final Record record, final List<String> uniqueKeyNames) {
switch (record.getDataChangeType()) {
case INSERT:
- return Optional.ofNullable(sqlBuilder.buildInsertSQL(record));
+ return Optional.ofNullable(sqlBuilder.buildInsertSQL(record, uniqueKeyNames));
case UPDATE:
- return Optional.ofNullable(sqlBuilder.buildUpdateSQL(record));
+ return Optional.ofNullable(sqlBuilder.buildUpdateSQL(record, uniqueKeyNames));
case DELETE:
- return Optional.ofNullable(sqlBuilder.buildDeleteSQL(record));
+ return Optional.ofNullable(sqlBuilder.buildDeleteSQL(record, uniqueKeyNames));
default:
return Optional.empty();
}
}
+ private Object convertValueFromAny(final StandardTableMetaData tableMetaData, final TableColumn tableColumn) {
+ StandardColumnMetaData columnMetaData = tableMetaData.getColumnMetaData(tableColumn.getName());
+ Object object;
+ try {
+ object = AnyValueConvert.convertToObject(tableColumn.getValue());
Review Comment:
`AnyValueConvert` could be `ProtobufAnyValueConverter`
##########
kernel/data-pipeline/cdc/client/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/client/importer/DataSourceImporter.java:
##########
@@ -98,19 +128,63 @@ public void write(final Record record) throws Exception {
}
}
- private Optional<String> buildSQL(final Record record) {
+ private String buildKey(final String schema, final String tableName) {
+ return schema.isEmpty() ? tableName : String.join(".", schema, tableName);
+ }
+
+ private StandardTableMetaData loadTableMetaData(final String schema, final String tableName) {
+ StandardTableMetaData result = tableMetaDataMap.get(buildKey(schema, tableName));
+ if (null != result) {
+ return result;
+ }
+ result = loader.getTableMetaData(Strings.emptyToNull(schema), tableName);
+ tableMetaDataMap.put(buildKey(schema, tableName), result);
+ return result;
+ }
+
+ private Optional<String> buildSQL(final Record record, final List<String> uniqueKeyNames) {
switch (record.getDataChangeType()) {
case INSERT:
- return Optional.ofNullable(sqlBuilder.buildInsertSQL(record));
+ return Optional.ofNullable(sqlBuilder.buildInsertSQL(record, uniqueKeyNames));
case UPDATE:
- return Optional.ofNullable(sqlBuilder.buildUpdateSQL(record));
+ return Optional.ofNullable(sqlBuilder.buildUpdateSQL(record, uniqueKeyNames));
case DELETE:
- return Optional.ofNullable(sqlBuilder.buildDeleteSQL(record));
+ return Optional.ofNullable(sqlBuilder.buildDeleteSQL(record, uniqueKeyNames));
default:
return Optional.empty();
}
}
+ private Object convertValueFromAny(final StandardTableMetaData tableMetaData, final TableColumn tableColumn) {
+ StandardColumnMetaData columnMetaData = tableMetaData.getColumnMetaData(tableColumn.getName());
+ Object object;
Review Comment:
`object` could be `result`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #24440: Improve CDC protocol
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #24440:
URL: https://github.com/apache/shardingsphere/pull/24440#discussion_r1137139927
##########
kernel/data-pipeline/cdc/client/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/client/handler/CDCRequestHandler.java:
##########
@@ -92,21 +90,18 @@ private void sendStartStreamingDataRequest(final ChannelHandlerContext ctx, fina
private void processDataRecords(final ChannelHandlerContext ctx, final DataRecordResult result) {
List<Record> recordsList = result.getRecordsList();
- for (Record each : recordsList) {
Review Comment:
`getRecordsList` could be `getRecordList` (`repeated Record record` in protobuf)
##########
kernel/data-pipeline/cdc/client/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/client/handler/CDCRequestHandler.java:
##########
@@ -49,11 +47,11 @@ public final class CDCRequestHandler extends ChannelInboundHandlerAdapter {
private final StartCDCClientParameter parameter;
- private final Importer importer;
+ private final RecordConsumer consumer;
Review Comment:
Could `RecordConsumer` be `java.util.function.Consumer<List<Record>>`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org