You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "azexcy (via GitHub)" <gi...@apache.org> on 2023/02/09 11:29:22 UTC

[GitHub] [shardingsphere] azexcy opened a new pull request, #24080: CDC support incremental mode

azexcy opened a new pull request, #24080:
URL: https://github.com/apache/shardingsphere/pull/24080

   
   
   Changes proposed in this pull request:
     - CDC support incremental mode
     - Fix mismatch comment
     - Improve CDC prepare stage 
   
   ---
   
   Before committing this PR, I'm sure that I have checked the following options:
   - [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
   - [ ] I have self-reviewed the commit code.
   - [ ] I have (or in comment I request) added corresponding labels for the pull request.
   - [ ] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
   - [ ] I have made corresponding changes to the documentation.
   - [ ] I have added corresponding unit tests for my changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] sandynz merged pull request #24080: Improve CDC pure incremental mode

Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz merged PR #24080:
URL: https://github.com/apache/shardingsphere/pull/24080


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] codecov-commenter commented on pull request #24080: CDC support incremental mode

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24080:
URL: https://github.com/apache/shardingsphere/pull/24080#issuecomment-1424075383

   # [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#24080](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e4dea53) into [master](https://codecov.io/gh/apache/shardingsphere/commit/7b367f689f8aeaa2767eaa73c83a5d64adf2d48b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7b367f6) will **decrease** coverage by `0.10%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #24080      +/-   ##
   ============================================
   - Coverage     50.13%   50.04%   -0.10%     
   - Complexity     1567     1579      +12     
   ============================================
     Files          3245     3255      +10     
     Lines         53214    53436     +222     
     Branches       9788     9826      +38     
   ============================================
   + Hits          26681    26742      +61     
   - Misses        24190    24337     +147     
   - Partials       2343     2357      +14     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ngsphere/data/pipeline/cdc/api/impl/CDCJobAPI.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvY2RjL2FwaS9pbXBsL0NEQ0pvYkFQSS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ta/pipeline/cdc/context/job/CDCJobItemContext.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvY2RjL2NvbnRleHQvam9iL0NEQ0pvYkl0ZW1Db250ZXh0LmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...data/pipeline/cdc/core/prepare/CDCJobPreparer.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY2RjL2NvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvY2RjL2NvcmUvcHJlcGFyZS9DRENKb2JQcmVwYXJlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...i/impl/AbstractInventoryIncrementalJobAPIImpl.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jb3JlL2FwaS9pbXBsL0Fic3RyYWN0SW52ZW50b3J5SW5jcmVtZW50YWxKb2JBUElJbXBsLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...handler/distsql/ral/hint/enums/HintSourceType.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9oYW5kbGVyL2Rpc3RzcWwvcmFsL2hpbnQvZW51bXMvSGludFNvdXJjZVR5cGUuamF2YQ==) | `0.00% <0.00%> (-42.86%)` | :arrow_down: |
   | [...istsql/ral/queryable/ShowDistVariableExecutor.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9oYW5kbGVyL2Rpc3RzcWwvcmFsL3F1ZXJ5YWJsZS9TaG93RGlzdFZhcmlhYmxlRXhlY3V0b3IuamF2YQ==) | `64.28% <0.00%> (-29.84%)` | :arrow_down: |
   | [...ngsphere/infra/context/kernel/KernelProcessor.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvY29udGV4dC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvaW5mcmEvY29udGV4dC9rZXJuZWwvS2VybmVsUHJvY2Vzc29yLmphdmE=) | `65.00% <0.00%> (-27.31%)` | :arrow_down: |
   | [...stsql/ral/queryable/ShowDistVariablesExecutor.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9oYW5kbGVyL2Rpc3RzcWwvcmFsL3F1ZXJ5YWJsZS9TaG93RGlzdFZhcmlhYmxlc0V4ZWN1dG9yLmphdmE=) | `77.27% <0.00%> (-22.73%)` | :arrow_down: |
   | [.../distsql/ral/updatable/SetDistVariableHandler.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9oYW5kbGVyL2Rpc3RzcWwvcmFsL3VwZGF0YWJsZS9TZXREaXN0VmFyaWFibGVIYW5kbGVyLmphdmE=) | `72.91% <0.00%> (-18.26%)` | :arrow_down: |
   | [...proxy/backend/config/ProxyConfigurationLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHJveHkvYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9jb25maWcvUHJveHlDb25maWd1cmF0aW9uTG9hZGVyLmphdmE=) | `54.54% <0.00%> (-2.06%)` | :arrow_down: |
   | ... and [19 more](https://codecov.io/gh/apache/shardingsphere/pull/24080?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] sandynz commented on a diff in pull request #24080: Improve CDC pure incremental mode

Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #24080:
URL: https://github.com/apache/shardingsphere/pull/24080#discussion_r1101436707


##########
kernel/data-pipeline/cdc/core/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/api/impl/CDCJobAPI.java:
##########
@@ -127,9 +135,43 @@ public boolean createJob(final CreateSubscriptionJobParameter event) {
         JobConfigurationPOJO jobConfigPOJO = convertJobConfiguration(jobConfig);
         jobConfigPOJO.setDisabled(true);
         repositoryAPI.persist(jobConfigKey, YamlEngine.marshal(jobConfigPOJO));
+        if (SubscriptionMode.INCREMENTAL.name().equals(param.getSubscriptionMode())) {
+            initIncrementalPosition(jobConfig);
+        }
         return true;
     }
     
+    private void initIncrementalPosition(final CDCJobConfiguration jobConfig) {
+        if (SubscriptionMode.FULL.name().equals(jobConfig.getSubscriptionMode())) {
+            return;
+        }
+        PipelineDataSourceManager dataSourceManager = new DefaultPipelineDataSourceManager();
+        String jobId = jobConfig.getJobId();
+        try {
+            for (int i = 0; i < jobConfig.getJobShardingCount(); i++) {
+                if (getJobItemProgress(jobId, i).isPresent()) {
+                    continue;
+                }
+                TableNameSchemaNameMapping tableNameSchemaNameMapping = getTableNameSchemaNameMapping(jobConfig.getTableNames());
+                DumperConfiguration dumperConfig = buildDumperConfiguration(jobConfig, i, tableNameSchemaNameMapping);
+                InventoryIncrementalJobItemProgress jobItemProgress = new InventoryIncrementalJobItemProgress();
+                jobItemProgress.setSourceDatabaseType(jobConfig.getSourceDatabaseType());
+                jobItemProgress.setDataSourceName(dumperConfig.getDataSourceName());
+                IncrementalTaskProgress incrementalTaskProgress = new IncrementalTaskProgress();
+                incrementalTaskProgress.setPosition(PipelineJobPreparerUtils.getIncrementalPosition(null, dumperConfig, dataSourceManager));
+                jobItemProgress.setIncremental(new JobItemIncrementalTasksProgress(incrementalTaskProgress));
+                jobItemProgress.setStatus(JobStatus.PREPARE_SUCCESS);
+                PipelineAPIFactory.getGovernanceRepositoryAPI().persistJobItemProgress(jobId, i, YamlEngine.marshal(getJobItemProgressSwapper().swapToYamlConfiguration(jobItemProgress)));
+            }
+        } catch (final SQLException ex) {
+            log.error("Get incremental position failed", ex);
+            //
+            throw new RuntimeException(String.format("Get %s incremental position failed", jobConfig.getDatabase()));

Review Comment:
   It's better to use dedicated exception (sub-class of PipelineSQLException), but not RuntimeException



##########
kernel/data-pipeline/cdc/core/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/core/prepare/CDCJobPreparer.java:
##########
@@ -53,26 +55,25 @@ public final class CDCJobPreparer {
      * @param jobItemContext job item context
      */
     public void prepare(final CDCJobItemContext jobItemContext) {
-        if (!jobAPI.getJobItemProgress(jobItemContext.getJobId(), jobItemContext.getShardingItem()).isPresent()) {
+        Optional<InventoryIncrementalJobItemProgress> jobItemProgress = jobAPI.getJobItemProgress(jobItemContext.getJobId(), jobItemContext.getShardingItem());
+        if (!jobItemProgress.isPresent()) {
             jobAPI.persistJobItemProgress(jobItemContext);
         }
         if (jobItemContext.isStopping()) {
             PipelineJobCenter.stop(jobItemContext.getJobId());
             return;
         }
-        updateJobItemStatus(JobStatus.PREPARING, jobItemContext);
+        boolean needUpdateJobStatus = !jobItemProgress.isPresent() || JobStatus.PREPARING.equals(jobItemContext.getStatus()) || JobStatus.RUNNING.equals(jobItemContext.getStatus())
+                || JobStatus.PREPARING_FAILURE.equals(jobItemContext.getStatus());
+        if (needUpdateJobStatus) {
+            jobItemContext.setStatus(JobStatus.PREPARE_SUCCESS);
+        }
         initIncrementalTasks(jobItemContext);
         CDCJobConfiguration jobConfig = jobItemContext.getJobConfig();
         if (SubscriptionMode.FULL.name().equals(jobConfig.getSubscriptionMode())) {
             initInventoryTasks(jobItemContext);
         }
         jobAPI.persistJobItemProgress(jobItemContext);

Review Comment:
   Could we keep `updateJobItemStatus(JobStatus.PREPARING, jobItemContext);` and `updateJobItemStatus(JobStatus.PREPARE_SUCCESS, jobItemContext);`;
   And remove `jobAPI.persistJobItemProgress(jobItemContext);`?



##########
kernel/data-pipeline/cdc/core/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/api/impl/CDCJobAPI.java:
##########
@@ -195,12 +232,16 @@ private TableNameSchemaNameMapping getTableNameSchemaNameMapping(final List<Stri
         return new TableNameSchemaNameMapping(tableNameSchemaMap);
     }
     
-    private static DumperConfiguration buildDumperConfiguration(final CDCJobConfiguration jobConfig, final String dataSourceName, final PipelineDataSourceConfiguration sourceDataSourceConfig,
-                                                                final Map<ActualTableName, LogicTableName> tableNameMap, final TableNameSchemaNameMapping tableNameSchemaNameMapping) {
+    private static DumperConfiguration buildDumperConfiguration(final CDCJobConfiguration jobConfig, final int jobShardingItem, final TableNameSchemaNameMapping tableNameSchemaNameMapping) {
+        JobDataNodeLine dataNodeLine = jobConfig.getJobShardingDataNodes().get(jobShardingItem);
+        Map<ActualTableName, LogicTableName> tableNameMap = new LinkedHashMap<>();
+        dataNodeLine.getEntries().forEach(each -> each.getDataNodes().forEach(node -> tableNameMap.put(new ActualTableName(node.getTableName()), new LogicTableName(each.getLogicTableName()))));
+        String dataSourceName = dataNodeLine.getEntries().iterator().next().getDataNodes().iterator().next().getDataSourceName();
+        StandardPipelineDataSourceConfiguration actualDataSourceConfiguration = jobConfig.getDataSourceConfig().getActualDataSourceConfiguration(dataSourceName);

Review Comment:
   `actualDataSourceConfiguration` could be `actualDataSourceConfig`



##########
kernel/data-pipeline/cdc/core/src/main/java/org/apache/shardingsphere/data/pipeline/cdc/api/impl/CDCJobAPI.java:
##########
@@ -127,9 +135,43 @@ public boolean createJob(final CreateSubscriptionJobParameter event) {
         JobConfigurationPOJO jobConfigPOJO = convertJobConfiguration(jobConfig);
         jobConfigPOJO.setDisabled(true);
         repositoryAPI.persist(jobConfigKey, YamlEngine.marshal(jobConfigPOJO));
+        if (SubscriptionMode.INCREMENTAL.name().equals(param.getSubscriptionMode())) {
+            initIncrementalPosition(jobConfig);
+        }
         return true;
     }
     
+    private void initIncrementalPosition(final CDCJobConfiguration jobConfig) {
+        if (SubscriptionMode.FULL.name().equals(jobConfig.getSubscriptionMode())) {
+            return;
+        }

Review Comment:
   Looks this block code is not required. Since 1) There's already if condition before invocation, 2) The FULL check doesn't filter other options.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org