You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2020/09/08 08:35:41 UTC

[GitHub] [shardingsphere] xbkaishui opened a new pull request #7328: Optimize startup time, ignore load unConfigured table's column and index

xbkaishui opened a new pull request #7328:
URL: https://github.com/apache/shardingsphere/pull/7328


   Fixes #6212 .
   
   Changes proposed in this pull request:
   - Remove the load procedure for unconfigured tables. load only table names for unconfigured tables.
   - Remove the refresh procedure for unconfigured tables
   - Remove the multi-thread logic for unconfigured tables. (Configured tables still use multi-thread)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [shardingsphere] tristaZero merged pull request #7328: Optimize startup time, ignore load unConfigured table's column and index

Posted by GitBox <gi...@apache.org>.
tristaZero merged pull request #7328:
URL: https://github.com/apache/shardingsphere/pull/7328


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [shardingsphere] xbkaishui commented on pull request #7328: Optimize startup time, ignore load unConfigured table's column and index

Posted by GitBox <gi...@apache.org>.
xbkaishui commented on pull request #7328:
URL: https://github.com/apache/shardingsphere/pull/7328#issuecomment-688742420


   @tristaZero  Please help to review, Thanks 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [shardingsphere] coveralls commented on pull request #7328: Optimize startup time, ignore load unConfigured table's column and index

Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #7328:
URL: https://github.com/apache/shardingsphere/pull/7328#issuecomment-688761210


   ## Pull Request Test Coverage Report for [Build 14548](https://coveralls.io/builds/33297948)
   
   * **14** of **17**   **(82.35%)**  changed or added relevant lines in **6** files are covered.
   * **1** unchanged line in **1** file lost coverage.
   * Overall coverage increased (+**0.004%**) to **34.132%**
   
   ---
   
   |  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
   | :-----|--------------|--------|---: |
   | [shardingsphere-sql-parser/shardingsphere-sql-parser-binder/src/main/java/org/apache/shardingsphere/sql/parser/binder/metadata/table/TableMetaDataLoader.java](https://coveralls.io/builds/33297948/source?filename=shardingsphere-sql-parser%2Fshardingsphere-sql-parser-binder%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fshardingsphere%2Fsql%2Fparser%2Fbinder%2Fmetadata%2Ftable%2FTableMetaDataLoader.java#L70) | 4 | 5 | 80.0%
   | [shardingsphere-sql-parser/shardingsphere-sql-parser-binder/src/main/java/org/apache/shardingsphere/sql/parser/binder/metadata/schema/SchemaMetaDataLoader.java](https://coveralls.io/builds/33297948/source?filename=shardingsphere-sql-parser%2Fshardingsphere-sql-parser-binder%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fshardingsphere%2Fsql%2Fparser%2Fbinder%2Fmetadata%2Fschema%2FSchemaMetaDataLoader.java#L81) | 1 | 3 | 33.33%
   <!-- | **Total:** | **14** | **17** | **82.35%** | -->
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-----|--------------|--: |
   | [shardingsphere-sql-parser/shardingsphere-sql-parser-binder/src/main/java/org/apache/shardingsphere/sql/parser/binder/metadata/schema/SchemaMetaDataLoader.java](https://coveralls.io/builds/33297948/source?filename=shardingsphere-sql-parser%2Fshardingsphere-sql-parser-binder%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fshardingsphere%2Fsql%2Fparser%2Fbinder%2Fmetadata%2Fschema%2FSchemaMetaDataLoader.java#L83) | 1 | 63.64% |
   <!-- | **Total:** | **1** |  | -->
   
   |  Totals | [![Coverage Status](https://coveralls.io/builds/33297948/badge)](https://coveralls.io/builds/33297948) |
   | :-- | --: |
   | Change from base [Build 14544](https://coveralls.io/builds/33296409): |  0.004% |
   | Covered Lines: | 35447 |
   | Relevant Lines: | 103854 |
   
   ---
   ##### 💛  - [Coveralls](https://coveralls.io)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [shardingsphere] codecov-commenter commented on pull request #7328: Optimize startup time, ignore load unConfigured table's column and index

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7328:
URL: https://github.com/apache/shardingsphere/pull/7328#issuecomment-688752680


   # [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/7328?src=pr&el=h1) Report
   > Merging [#7328](https://codecov.io/gh/apache/shardingsphere/pull/7328?src=pr&el=desc) into [master](https://codecov.io/gh/apache/shardingsphere/commit/bc8a93d8fe16b5ddac13083a60cda32d3ca44fbb?el=desc) will **increase** coverage by `0.03%`.
   > The diff coverage is `64.70%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/shardingsphere/pull/7328/graphs/tree.svg?width=650&height=150&src=pr&token=ZvlXpWa7so)](https://codecov.io/gh/apache/shardingsphere/pull/7328?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7328      +/-   ##
   ============================================
   + Coverage     75.61%   75.64%   +0.03%     
   - Complexity      466      468       +2     
   ============================================
     Files          1256     1259       +3     
     Lines         21789    21773      -16     
     Branches       3943     3940       -3     
   ============================================
   - Hits          16475    16471       -4     
   + Misses         4282     4270      -12     
     Partials       1032     1032              
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/7328?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...r/binder/metadata/schema/SchemaMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtc3FsLXBhcnNlci9zaGFyZGluZ3NwaGVyZS1zcWwtcGFyc2VyLWJpbmRlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc3FsL3BhcnNlci9iaW5kZXIvbWV0YWRhdGEvc2NoZW1hL1NjaGVtYU1ldGFEYXRhTG9hZGVyLmphdmE=) | `45.45% <33.33%> (+23.23%)` | `0.00 <0.00> (ø)` | |
   | [...ser/binder/metadata/table/TableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtc3FsLXBhcnNlci9zaGFyZGluZ3NwaGVyZS1zcWwtcGFyc2VyLWJpbmRlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc3FsL3BhcnNlci9iaW5kZXIvbWV0YWRhdGEvdGFibGUvVGFibGVNZXRhRGF0YUxvYWRlci5qYXZh) | `61.53% <40.00%> (-13.47%)` | `0.00 <0.00> (ø)` | |
   | [...pl/AlterTableStatementMetaDataRefreshStrategy.java](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9yZWZyZXNoL2ltcGwvQWx0ZXJUYWJsZVN0YXRlbWVudE1ldGFEYXRhUmVmcmVzaFN0cmF0ZWd5LmphdmE=) | `83.33% <66.66%> (-16.67%)` | `0.00 <0.00> (ø)` | |
   | [...l/CreateTableStatementMetaDataRefreshStrategy.java](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9yZWZyZXNoL2ltcGwvQ3JlYXRlVGFibGVTdGF0ZW1lbnRNZXRhRGF0YVJlZnJlc2hTdHJhdGVneS5qYXZh) | `59.09% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | [...nfra/metadata/schema/RuleSchemaMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvUnVsZVNjaGVtYU1ldGFEYXRhTG9hZGVyLmphdmE=) | `97.77% <100.00%> (+7.45%)` | `0.00 <0.00> (ø)` | |
   | [...ingsphere/infra/context/SchemaContextsBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29udGV4dC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvaW5mcmEvY29udGV4dC9TY2hlbWFDb250ZXh0c0J1aWxkZXIuamF2YQ==) | `95.45% <100.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | [...here/proxy/backend/schema/ProxySchemaContexts.java](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtcHJveHkvc2hhcmRpbmdzcGhlcmUtcHJveHktYmFja2VuZC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvcHJveHkvYmFja2VuZC9zY2hlbWEvUHJveHlTY2hlbWFDb250ZXh0cy5qYXZh) | `61.11% <0.00%> (-13.31%)` | `0.00% <0.00%> (ø%)` | |
   | [...rdingsphere/driver/executor/StatementExecutor.java](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtamRiYy9zaGFyZGluZ3NwaGVyZS1qZGJjLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RyaXZlci9leGVjdXRvci9TdGF0ZW1lbnRFeGVjdXRvci5qYXZh) | `75.00% <0.00%> (-5.44%)` | `0.00% <0.00%> (ø%)` | |
   | [...ere/driver/executor/PreparedStatementExecutor.java](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtamRiYy9zaGFyZGluZ3NwaGVyZS1qZGJjLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RyaXZlci9leGVjdXRvci9QcmVwYXJlZFN0YXRlbWVudEV4ZWN1dG9yLmphdmE=) | `70.83% <0.00%> (-5.36%)` | `0.00% <0.00%> (ø%)` | |
   | [.../query/text/query/MySQLComQueryPacketExecutor.java](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtcHJveHkvc2hhcmRpbmdzcGhlcmUtcHJveHktZnJvbnRlbmQvc2hhcmRpbmdzcGhlcmUtcHJveHktZnJvbnRlbmQtbXlzcWwvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3Byb3h5L2Zyb250ZW5kL215c3FsL2NvbW1hbmQvcXVlcnkvdGV4dC9xdWVyeS9NeVNRTENvbVF1ZXJ5UGFja2V0RXhlY3V0b3IuamF2YQ==) | `65.85% <0.00%> (-1.59%)` | `0.00% <0.00%> (ø%)` | |
   | ... and [15 more](https://codecov.io/gh/apache/shardingsphere/pull/7328/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/7328?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/7328?src=pr&el=footer). Last update [bc8a93d...c9bac92](https://codecov.io/gh/apache/shardingsphere/pull/7328?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [shardingsphere] tristaZero commented on a change in pull request #7328: Optimize startup time, ignore load unConfigured table's column and index

Posted by GitBox <gi...@apache.org>.
tristaZero commented on a change in pull request #7328:
URL: https://github.com/apache/shardingsphere/pull/7328#discussion_r484818640



##########
File path: shardingsphere-integration-test/shardingsphere-test-suite/src/test/java/org/apache/shardingsphere/dbtest/engine/dql/BaseDQLIT.java
##########
@@ -123,6 +123,9 @@ private void assertMetaData(final ResultSetMetaData actualMetaData, final List<D
         if ("shadow".equals(getRuleType())) {
             return;
         }
+        if (0 == actualMetaData.getColumnCount()) {

Review comment:
       // TODO comment is required.

##########
File path: shardingsphere-sql-parser/shardingsphere-sql-parser-binder/src/main/java/org/apache/shardingsphere/sql/parser/binder/metadata/table/TableMetaDataLoader.java
##########
@@ -54,6 +55,24 @@
         }
     }
     
+    /**
+     * Load table without column and index meta data, this is for unconfigured table.
+     *
+     * @param dataSource data source
+     * @param table table name
+     * @param databaseType database type
+     * @return table meta data
+     * @throws SQLException SQL exception
+     */
+    public static Optional<TableMetaData> loadWithoutColumnMetaData(final DataSource dataSource, final String table, final String databaseType) throws SQLException {
+        try (MetaDataConnection connection = new MetaDataConnection(dataSource.getConnection())) {
+            if (!isTableExist(connection, table, databaseType)) {
+                return Optional.empty();
+            }
+            return Optional.of(new TableMetaData(Collections.emptyList(), Collections.emptyList()));

Review comment:
       // TODO comment is required.

##########
File path: shardingsphere-sql-parser/shardingsphere-sql-parser-binder/src/main/java/org/apache/shardingsphere/sql/parser/binder/metadata/schema/SchemaMetaDataLoader.java
##########
@@ -87,22 +78,11 @@ public static SchemaMetaData load(final DataSource dataSource, final int maxConn
         if (tableNames.isEmpty()) {
             return new SchemaMetaData(Collections.emptyMap());
         }
-        List<List<String>> tableGroups = Lists.partition(tableNames, Math.max(tableNames.size() / maxConnectionCount, 1));
-        Map<String, TableMetaData> tableMetaDataMap = 1 == tableGroups.size()
-                ? load(dataSource.getConnection(), tableGroups.get(0), databaseType) : asyncLoad(dataSource, maxConnectionCount, tableNames, tableGroups, databaseType);
+        Map<String, TableMetaData> tableMetaDataMap = new HashMap<>(tableNames.size(), 1);
+        tableNames.forEach(tableName -> tableMetaDataMap.put(tableName, new TableMetaData(Collections.emptyList(), Collections.emptyList())));

Review comment:
       // TODO comment is required.

##########
File path: shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/refresh/impl/CreateTableStatementMetaDataRefreshStrategy.java
##########
@@ -53,7 +53,7 @@ public void refreshMetaData(final ShardingSphereMetaData metaData, final Databas
     private void refreshUnconfiguredMetaData(final ShardingSphereMetaData metaData, 
                                              final DatabaseType databaseType, final Map<String, DataSource> dataSourceMap, final String tableName) throws SQLException {
         for (Entry<String, DataSource> entry : dataSourceMap.entrySet()) {
-            Optional<TableMetaData> tableMetaData = TableMetaDataLoader.load(entry.getValue(), tableName, databaseType.getName());
+            Optional<TableMetaData> tableMetaData = TableMetaDataLoader.loadWithoutColumnMetaData(entry.getValue(), tableName, databaseType.getName());

Review comment:
       // TODO comment is required.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org