You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "azexcy (via GitHub)" <gi...@apache.org> on 2023/02/09 11:29:22 UTC
[GitHub] [shardingsphere] azexcy opened a new pull request, #24080: CDC support incremental mode
azexcy opened a new pull request, #24080:
URL: https://github.com/apache/shardingsphere/pull/24080
Changes proposed in this pull request:
- CDC support incremental mode
- Fix mismatch comment
- Improve CDC prepare stage
---
Before committing this PR, I'm sure that I have checked the following options:
- [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
- [ ] I have self-reviewed the commit code.
- [ ] I have (or in comment I request) added corresponding labels for the pull request.
- [ ] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
- [ ] I have made corresponding changes to the documentation.
- [ ] I have added corresponding unit tests for my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz merged pull request #24080: Improve CDC pure incremental mode
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz merged PR #24080:
URL: https://github.com/apache/shardingsphere/pull/24080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter commented on pull request #24080: CDC support incremental mode
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24080:
URL: https://github.com/apache/shardingsphere/pull/24080#issuecomment-1424075383
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#24080](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e4dea53) into [master](https://codecov.io/gh/apache/shardingsphere/commit/7b367f689f8aeaa2767eaa73c83a5d64adf2d48b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7b367f6) will **decrease** coverage by `0.10%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #24080 +/- ##
============================================
- Coverage 50.13% 50.04% -0.10%
- Complexity 1567 1579 +12
============================================
Files 3245 3255 +10
Lines 53214 53436 +222
Branches 9788 9826 +38
============================================
+ Hits 26681 26742 +61
- Misses 24190 24337 +147
- Partials 2343 2357 +14
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ngsphere/data/pipeline/cdc/api/impl/CDCJobAPI.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvY2RjL2FwaS9pbXBsL0NEQ0pvYkFQSS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ta/pipeline/cdc/context/job/CDCJobItemContext.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvY2RjL2NvbnRleHQvam9iL0NEQ0pvYkl0ZW1Db250ZXh0LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...data/pipeline/cdc/core/prepare/CDCJobPreparer.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvY2RjL2NvcmUvcHJlcGFyZS9DRENKb2JQcmVwYXJlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...i/impl/AbstractInventoryIncrementalJobAPIImpl.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jb3JlL2FwaS9pbXBsL0Fic3RyYWN0SW52ZW50b3J5SW5jcmVtZW50YWxKb2JBUElJbXBsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...handler/distsql/ral/hint/enums/HintSourceType.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9oYW5kbGVyL2Rpc3RzcWwvcmFsL2hpbnQvZW51bXMvSGludFNvdXJjZVR5cGUuamF2YQ==) | `0.00% <0.00%> (-42.86%)` | :arrow_down: |
| [...istsql/ral/queryable/ShowDistVariableExecutor.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9oYW5kbGVyL2Rpc3RzcWwvcmFsL3F1ZXJ5YWJsZS9TaG93RGlzdFZhcmlhYmxlRXhlY3V0b3IuamF2YQ==) | `64.28% <0.00%> (-29.84%)` | :arrow_down: |
| [...ngsphere/infra/context/kernel/KernelProcessor.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvY29udGV4dC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvaW5mcmEvY29udGV4dC9rZXJuZWwvS2VybmVsUHJvY2Vzc29yLmphdmE=) | `65.00% <0.00%> (-27.31%)` | :arrow_down: |
| [...stsql/ral/queryable/ShowDistVariablesExecutor.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9oYW5kbGVyL2Rpc3RzcWwvcmFsL3F1ZXJ5YWJsZS9TaG93RGlzdFZhcmlhYmxlc0V4ZWN1dG9yLmphdmE=) | `77.27% <0.00%> (-22.73%)` | :arrow_down: |
| [.../distsql/ral/updatable/SetDistVariableHandler.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9oYW5kbGVyL2Rpc3RzcWwvcmFsL3VwZGF0YWJsZS9TZXREaXN0VmFyaWFibGVIYW5kbGVyLmphdmE=) | `72.91% <0.00%> (-18.26%)` | :arrow_down: |
| [...proxy/backend/config/ProxyConfigurationLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9jb25maWcvUHJveHlDb25maWd1cmF0aW9uTG9hZGVyLmphdmE=) | `54.54% <0.00%> (-2.06%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #24080: Improve CDC pure incremental mode
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #24080:
URL: https://github.com/apache/shardingsphere/pull/24080#discussion_r1101436707
##########
kernel/data-pipeline/cdc/core/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/api/impl/CDCJobAPI.java:
##########
@@ -127,9 +135,43 @@ public boolean createJob(final CreateSubscriptionJobParameter event) {
JobConfigurationPOJO jobConfigPOJO = convertJobConfiguration(jobConfig);
jobConfigPOJO.setDisabled(true);
repositoryAPI.persist(jobConfigKey, YamlEngine.marshal(jobConfigPOJO));
+ if (SubscriptionMode.INCREMENTAL.name().equals(param.getSubscriptionMode())) {
+ initIncrementalPosition(jobConfig);
+ }
return true;
}
+ private void initIncrementalPosition(final CDCJobConfiguration jobConfig) {
+ if (SubscriptionMode.FULL.name().equals(jobConfig.getSubscriptionMode())) {
+ return;
+ }
+ PipelineDataSourceManager dataSourceManager = new DefaultPipelineDataSourceManager();
+ String jobId = jobConfig.getJobId();
+ try {
+ for (int i = 0; i < jobConfig.getJobShardingCount(); i++) {
+ if (getJobItemProgress(jobId, i).isPresent()) {
+ continue;
+ }
+ TableNameSchemaNameMapping tableNameSchemaNameMapping = getTableNameSchemaNameMapping(jobConfig.getTableNames());
+ DumperConfiguration dumperConfig = buildDumperConfiguration(jobConfig, i, tableNameSchemaNameMapping);
+ InventoryIncrementalJobItemProgress jobItemProgress = new InventoryIncrementalJobItemProgress();
+ jobItemProgress.setSourceDatabaseType(jobConfig.getSourceDatabaseType());
+ jobItemProgress.setDataSourceName(dumperConfig.getDataSourceName());
+ IncrementalTaskProgress incrementalTaskProgress = new IncrementalTaskProgress();
+ incrementalTaskProgress.setPosition(PipelineJobPreparerUtils.getIncrementalPosition(null, dumperConfig, dataSourceManager));
+ jobItemProgress.setIncremental(new JobItemIncrementalTasksProgress(incrementalTaskProgress));
+ jobItemProgress.setStatus(JobStatus.PREPARE_SUCCESS);
+ PipelineAPIFactory.getGovernanceRepositoryAPI().persistJobItemProgress(jobId, i, YamlEngine.marshal(getJobItemProgressSwapper().swapToYamlConfiguration(jobItemProgress)));
+ }
+ } catch (final SQLException ex) {
+ log.error("Get incremental position failed", ex);
+ //
+ throw new RuntimeException(String.format("Get %s incremental position failed", jobConfig.getDatabase()));
Review Comment:
It's better to use dedicated exception (sub-class of PipelineSQLException), but not RuntimeException
##########
kernel/data-pipeline/cdc/core/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/core/prepare/CDCJobPreparer.java:
##########
@@ -53,26 +55,25 @@ public final class CDCJobPreparer {
* @param jobItemContext job item context
*/
public void prepare(final CDCJobItemContext jobItemContext) {
- if (!jobAPI.getJobItemProgress(jobItemContext.getJobId(), jobItemContext.getShardingItem()).isPresent()) {
+ Optional<InventoryIncrementalJobItemProgress> jobItemProgress = jobAPI.getJobItemProgress(jobItemContext.getJobId(), jobItemContext.getShardingItem());
+ if (!jobItemProgress.isPresent()) {
jobAPI.persistJobItemProgress(jobItemContext);
}
if (jobItemContext.isStopping()) {
PipelineJobCenter.stop(jobItemContext.getJobId());
return;
}
- updateJobItemStatus(JobStatus.PREPARING, jobItemContext);
+ boolean needUpdateJobStatus = !jobItemProgress.isPresent() || JobStatus.PREPARING.equals(jobItemContext.getStatus()) || JobStatus.RUNNING.equals(jobItemContext.getStatus())
+ || JobStatus.PREPARING_FAILURE.equals(jobItemContext.getStatus());
+ if (needUpdateJobStatus) {
+ jobItemContext.setStatus(JobStatus.PREPARE_SUCCESS);
+ }
initIncrementalTasks(jobItemContext);
CDCJobConfiguration jobConfig = jobItemContext.getJobConfig();
if (SubscriptionMode.FULL.name().equals(jobConfig.getSubscriptionMode())) {
initInventoryTasks(jobItemContext);
}
jobAPI.persistJobItemProgress(jobItemContext);
Review Comment:
Could we keep `updateJobItemStatus(JobStatus.PREPARING, jobItemContext);` and `updateJobItemStatus(JobStatus.PREPARE_SUCCESS, jobItemContext);`;
And remove `jobAPI.persistJobItemProgress(jobItemContext);`?
##########
kernel/data-pipeline/cdc/core/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/api/impl/CDCJobAPI.java:
##########
@@ -195,12 +232,16 @@ private TableNameSchemaNameMapping getTableNameSchemaNameMapping(final List<Stri
return new TableNameSchemaNameMapping(tableNameSchemaMap);
}
- private static DumperConfiguration buildDumperConfiguration(final CDCJobConfiguration jobConfig, final String dataSourceName, final PipelineDataSourceConfiguration sourceDataSourceConfig,
- final Map<ActualTableName, LogicTableName> tableNameMap, final TableNameSchemaNameMapping tableNameSchemaNameMapping) {
+ private static DumperConfiguration buildDumperConfiguration(final CDCJobConfiguration jobConfig, final int jobShardingItem, final TableNameSchemaNameMapping tableNameSchemaNameMapping) {
+ JobDataNodeLine dataNodeLine = jobConfig.getJobShardingDataNodes().get(jobShardingItem);
+ Map<ActualTableName, LogicTableName> tableNameMap = new LinkedHashMap<>();
+ dataNodeLine.getEntries().forEach(each -> each.getDataNodes().forEach(node -> tableNameMap.put(new ActualTableName(node.getTableName()), new LogicTableName(each.getLogicTableName()))));
+ String dataSourceName = dataNodeLine.getEntries().iterator().next().getDataNodes().iterator().next().getDataSourceName();
+ StandardPipelineDataSourceConfiguration actualDataSourceConfiguration = jobConfig.getDataSourceConfig().getActualDataSourceConfiguration(dataSourceName);
Review Comment:
`actualDataSourceConfiguration` could be `actualDataSourceConfig`
##########
kernel/data-pipeline/cdc/core/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/api/impl/CDCJobAPI.java:
##########
@@ -127,9 +135,43 @@ public boolean createJob(final CreateSubscriptionJobParameter event) {
JobConfigurationPOJO jobConfigPOJO = convertJobConfiguration(jobConfig);
jobConfigPOJO.setDisabled(true);
repositoryAPI.persist(jobConfigKey, YamlEngine.marshal(jobConfigPOJO));
+ if (SubscriptionMode.INCREMENTAL.name().equals(param.getSubscriptionMode())) {
+ initIncrementalPosition(jobConfig);
+ }
return true;
}
+ private void initIncrementalPosition(final CDCJobConfiguration jobConfig) {
+ if (SubscriptionMode.FULL.name().equals(jobConfig.getSubscriptionMode())) {
+ return;
+ }
Review Comment:
Looks this block code is not required. Since 1) There's already if condition before invocation, 2) The FULL check doesn't filter other options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org