You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2022/09/13 01:57:40 UTC

[GitHub] [shardingsphere] azexcy opened a new pull request, #20944: Support unique key table check migration

azexcy opened a new pull request, #20944:
URL: https://github.com/apache/shardingsphere/pull/20944

   Support unique key table check migration
   
   Changes proposed in this pull request:
     - Support unique key table check migration
     - Add migration IT
   
   ---
   
   Before committing this PR, I'm sure that I have checked the following options:
   - [x] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
   - [x] I have self-reviewed the commit code.
   - [x] I have triggered maven check: `mvn clean install -B -T2C -DskipTests -Dmaven.javadoc.skip=true -e`.
   - [x] I have made corresponding changes to the documentation.
   - [x] I have added corresponding unit tests for my changes.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] sandynz commented on a diff in pull request #20944: Support unique key table check migration

Posted by GitBox <gi...@apache.org>.
sandynz commented on code in PR #20944:
URL: https://github.com/apache/shardingsphere/pull/20944#discussion_r969073932


##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-core/src/main/java/org/apache/shardingsphere/data/pipeline/scenario/migration/MigrationDataConsistencyChecker.java:
##########
@@ -179,7 +180,14 @@ private Map<String, DataConsistencyContentCheckResult> checkData(final DataConsi
                     throw new PipelineDataConsistencyCheckFailedException("Can not get metadata for table " + each);
                 }
                 Collection<String> columnNames = tableMetaData.getColumnNames();
-                PipelineColumnMetaData uniqueKey = tableMetaData.getColumnMetaData(tableMetaData.getPrimaryKeyColumns().get(0));
+                PipelineColumnMetaData uniqueKey;
+                if (!tableMetaData.getPrimaryKeyColumns().isEmpty()) {
+                    uniqueKey = tableMetaData.getColumnMetaData(tableMetaData.getPrimaryKeyColumns().get(0));
+                } else {
+                    PipelineIndexMetaData pipelineIndexMetaData = tableMetaData.getUniqueIndexes().stream().filter(index -> index.getColumns().size() == 1).findFirst()
+                            .orElseThrow(() -> new PipelineDataConsistencyCheckFailedException("No support, neither the primary key and the unique key exist."));
+                    uniqueKey = tableMetaData.getColumnMetaData(pipelineIndexMetaData.getColumns().get(0).getOrdinalPosition());
+                }

Review Comment:
   1, Could we put these code in unified method? Seems there's similar code in InventoryTaskSplitter.
   
   2, Is it possible to persist unique key in job configuration?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] sandynz commented on a diff in pull request #20944: Support unique key table check migration

Posted by GitBox <gi...@apache.org>.
sandynz commented on code in PR #20944:
URL: https://github.com/apache/shardingsphere/pull/20944#discussion_r969168061


##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-api/src/main/java/org/apache/shardingsphere/data/pipeline/api/metadata/yaml/YamlPipelineColumnMetaData.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.shardingsphere.data.pipeline.api.metadata.yaml;
+
+import lombok.Data;
+import org.apache.shardingsphere.infra.util.yaml.YamlConfiguration;
+
+@Data
+public final class YamlPipelineColumnMetaData implements YamlConfiguration {

Review Comment:
   Class javadoc is required.
   
   And also YamlPipelineColumnMetaDataSwapper



##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-api/src/main/java/org/apache/shardingsphere/data/pipeline/api/config/job/PipelineJobConfiguration.java:
##########
@@ -42,4 +44,11 @@ public interface PipelineJobConfiguration {
      * @return source database type
      */
     String getSourceDatabaseType();
+    
+    /**
+     * Get unique key column.
+     *
+     * @return unique key column
+     */
+    PipelineColumnMetaData getUniqueKeyColumn();

Review Comment:
   It's not common enough to put here



##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-core/src/main/java/org/apache/shardingsphere/data/pipeline/core/util/PipelineTableMetaDataUtil.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.shardingsphere.data.pipeline.core.util;
+
+import lombok.SneakyThrows;
+import org.apache.shardingsphere.data.pipeline.api.datasource.PipelineDataSourceWrapper;
+import org.apache.shardingsphere.data.pipeline.api.datasource.config.impl.StandardPipelineDataSourceConfiguration;
+import org.apache.shardingsphere.data.pipeline.api.metadata.model.PipelineColumnMetaData;
+import org.apache.shardingsphere.data.pipeline.api.metadata.model.PipelineIndexMetaData;
+import org.apache.shardingsphere.data.pipeline.api.metadata.model.PipelineTableMetaData;
+import org.apache.shardingsphere.data.pipeline.core.datasource.PipelineDataSourceFactory;
+import org.apache.shardingsphere.data.pipeline.core.exception.job.PipelineJobCreationException;
+import org.apache.shardingsphere.data.pipeline.core.metadata.loader.StandardPipelineTableMetaDataLoader;
+
+import java.sql.SQLException;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * Pipeline table meta data util.
+ */
+public final class PipelineTableMetaDataUtil {
+    
+    /**
+     * Get pipeline table meta data.
+     *
+     * @param schemaName schema name
+     * @param tableName table name
+     * @param dataSourceConfiguration source configuration
+     * @return pipeline table meta data
+     */
+    @SneakyThrows(SQLException.class)
+    public static PipelineTableMetaData getPipelineTableMetaData(final String schemaName, final String tableName, final StandardPipelineDataSourceConfiguration dataSourceConfiguration) {
+        try (PipelineDataSourceWrapper dataSource = PipelineDataSourceFactory.newInstance(dataSourceConfiguration)) {
+            return new StandardPipelineTableMetaDataLoader(dataSource).getTableMetaData(schemaName, tableName);

Review Comment:
   It's better to share `StandardPipelineTableMetaDataLoader` instance to reduce meta data loading time.
   
   And also another `getPipelineTableMetaData` method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] sandynz commented on a diff in pull request #20944: Support unique key table check migration

Posted by GitBox <gi...@apache.org>.
sandynz commented on code in PR #20944:
URL: https://github.com/apache/shardingsphere/pull/20944#discussion_r969503529


##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-core/src/main/java/org/apache/shardingsphere/data/pipeline/scenario/migration/MigrationJobPreparer.java:
##########
@@ -143,8 +145,12 @@ private void prepareTarget(final MigrationJobItemContext jobItemContext) throws
     }
     
     private void initInventoryTasks(final MigrationJobItemContext jobItemContext) {
+        InventoryDumperConfiguration inventoryDumperConfiguration = new InventoryDumperConfiguration(jobItemContext.getTaskConfig().getDumperConfig());

Review Comment:
   `inventoryDumperConfiguration` could be `inventoryDumperConfig` or `dumperConfig`



##########
shardingsphere-test/shardingsphere-pipeline-test/src/test/java/org/apache/shardingsphere/data/pipeline/core/api/impl/MigrationJobAPIImplTest.java:
##########
@@ -78,7 +81,7 @@ public static void beforeClass() {
         jobAPI = MigrationJobAPIFactory.getInstance();
         Map<String, Object> props = new HashMap<>();
         // TODO if resource availability is checked, then it should not work
-        props.put("jdbcUrl", "jdbc:mysql://localhost:3306/test");
+        props.put("jdbcUrl", "jdbc:h2:mem:test_ds_0;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL");

Review Comment:
   TODO could be removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] sandynz merged pull request #20944: Support unique key table check migration

Posted by GitBox <gi...@apache.org>.
sandynz merged PR #20944:
URL: https://github.com/apache/shardingsphere/pull/20944


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] codecov-commenter commented on pull request #20944: Support unique key table check migration

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #20944:
URL: https://github.com/apache/shardingsphere/pull/20944#issuecomment-1244827378

   # [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/20944?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#20944](https://codecov.io/gh/apache/shardingsphere/pull/20944?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ad87d70) into [master](https://codecov.io/gh/apache/shardingsphere/commit/b7770e72a2bc072dca13daddb2966e9adfce4590?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b7770e7) will **increase** coverage by `0.13%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #20944      +/-   ##
   ============================================
   + Coverage     61.64%   61.77%   +0.13%     
   - Complexity     2462     2464       +2     
   ============================================
     Files          4022     4035      +13     
     Lines         55560    55468      -92     
     Branches       9422     9403      -19     
   ============================================
   + Hits          34249    34268      +19     
   + Misses        18429    18331      -98     
   + Partials       2882     2869      -13     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/20944?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...rio/migration/MigrationDataConsistencyChecker.java](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUta2VybmVsL3NoYXJkaW5nc3BoZXJlLWRhdGEtcGlwZWxpbmUvc2hhcmRpbmdzcGhlcmUtZGF0YS1waXBlbGluZS1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9kYXRhL3BpcGVsaW5lL3NjZW5hcmlvL21pZ3JhdGlvbi9NaWdyYXRpb25EYXRhQ29uc2lzdGVuY3lDaGVja2VyLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ta/pipeline/postgresql/ingest/wal/WalPosition.java](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUta2VybmVsL3NoYXJkaW5nc3BoZXJlLWRhdGEtcGlwZWxpbmUvc2hhcmRpbmdzcGhlcmUtZGF0YS1waXBlbGluZS1kaWFsZWN0L3NoYXJkaW5nc3BoZXJlLWRhdGEtcGlwZWxpbmUtcG9zdGdyZXNxbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9wb3N0Z3Jlc3FsL2luZ2VzdC93YWwvV2FsUG9zaXRpb24uamF2YQ==) | `66.66% <0.00%> (-13.34%)` | :arrow_down: |
   | [...a/pipeline/mysql/ingest/binlog/BinlogPosition.java](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUta2VybmVsL3NoYXJkaW5nc3BoZXJlLWRhdGEtcGlwZWxpbmUvc2hhcmRpbmdzcGhlcmUtZGF0YS1waXBlbGluZS1kaWFsZWN0L3NoYXJkaW5nc3BoZXJlLWRhdGEtcGlwZWxpbmUtbXlzcWwvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvbXlzcWwvaW5nZXN0L2JpbmxvZy9CaW5sb2dQb3NpdGlvbi5qYXZh) | `75.00% <0.00%> (-8.34%)` | :arrow_down: |
   | [...or/ddl/SingleTableDropSchemaMetadataValidator.java](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUta2VybmVsL3NoYXJkaW5nc3BoZXJlLXNpbmdsZS10YWJsZS9zaGFyZGluZ3NwaGVyZS1zaW5nbGUtdGFibGUtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc2luZ2xldGFibGUvcm91dGUvdmFsaWRhdG9yL2RkbC9TaW5nbGVUYWJsZURyb3BTY2hlbWFNZXRhZGF0YVZhbGlkYXRvci5qYXZh) | `88.88% <0.00%> (-2.03%)` | :arrow_down: |
   | [...ql/parser/core/EncryptDistSQLStatementVisitor.java](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtZW5jcnlwdC9zaGFyZGluZ3NwaGVyZS1lbmNyeXB0LWRpc3RzcWwvc2hhcmRpbmdzcGhlcmUtZW5jcnlwdC1kaXN0c3FsLXBhcnNlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZW5jcnlwdC9kaXN0c3FsL3BhcnNlci9jb3JlL0VuY3J5cHREaXN0U1FMU3RhdGVtZW50VmlzaXRvci5qYXZh) | `70.58% <0.00%> (-1.64%)` | :arrow_down: |
   | [...dingsphere/authority/checker/AuthorityChecker.java](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUta2VybmVsL3NoYXJkaW5nc3BoZXJlLWF1dGhvcml0eS9zaGFyZGluZ3NwaGVyZS1hdXRob3JpdHktY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvYXV0aG9yaXR5L2NoZWNrZXIvQXV0aG9yaXR5Q2hlY2tlci5qYXZh) | `21.56% <0.00%> (-1.08%)` | :arrow_down: |
   | [...e/sql/parser/sql/common/util/WhereExtractUtil.java](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtc3FsLXBhcnNlci9zaGFyZGluZ3NwaGVyZS1zcWwtcGFyc2VyLXN0YXRlbWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc3FsL3BhcnNlci9zcWwvY29tbW9uL3V0aWwvV2hlcmVFeHRyYWN0VXRpbC5qYXZh) | `93.75% <0.00%> (-0.70%)` | :arrow_down: |
   | [...er/query/ReadwriteSplittingRuleQueryResultSet.java](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtcmVhZHdyaXRlLXNwbGl0dGluZy9zaGFyZGluZ3NwaGVyZS1yZWFkd3JpdGUtc3BsaXR0aW5nLWRpc3RzcWwvc2hhcmRpbmdzcGhlcmUtcmVhZHdyaXRlLXNwbGl0dGluZy1kaXN0c3FsLWhhbmRsZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3JlYWR3cml0ZXNwbGl0dGluZy9kaXN0c3FsL2hhbmRsZXIvcXVlcnkvUmVhZHdyaXRlU3BsaXR0aW5nUnVsZVF1ZXJ5UmVzdWx0U2V0LmphdmE=) | `95.23% <0.00%> (-0.60%)` | :arrow_down: |
   | [...ource/metadata/ShardingSphereDatabaseMetaData.java](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtamRiYy9zaGFyZGluZ3NwaGVyZS1qZGJjLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RyaXZlci9qZGJjL2NvcmUvZGF0YXNvdXJjZS9tZXRhZGF0YS9TaGFyZGluZ1NwaGVyZURhdGFiYXNlTWV0YURhdGEuamF2YQ==) | `91.22% <0.00%> (-0.44%)` | :arrow_down: |
   | [...ore/ReadwriteSplittingDistSQLStatementVisitor.java](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtcmVhZHdyaXRlLXNwbGl0dGluZy9zaGFyZGluZ3NwaGVyZS1yZWFkd3JpdGUtc3BsaXR0aW5nLWRpc3RzcWwvc2hhcmRpbmdzcGhlcmUtcmVhZHdyaXRlLXNwbGl0dGluZy1kaXN0c3FsLXBhcnNlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcmVhZHdyaXRlc3BsaXR0aW5nL2Rpc3RzcWwvcGFyc2VyL2NvcmUvUmVhZHdyaXRlU3BsaXR0aW5nRGlzdFNRTFN0YXRlbWVudFZpc2l0b3IuamF2YQ==) | `44.44% <0.00%> (-0.30%)` | :arrow_down: |
   | ... and [266 more](https://codecov.io/gh/apache/shardingsphere/pull/20944/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org