You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2021/08/11 09:48:41 UTC
[GitHub] [shardingsphere] tuichenchuxin opened a new pull request #11767: improve metadata load performance
tuichenchuxin opened a new pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767
Fixes #11718 .
Changes proposed in this pull request:
- SchemaBuilder#build improve performance
- add test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on a change in pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on a change in pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#discussion_r689086804
##########
File path: shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/builder/TableMetaDataBuilder.java
##########
@@ -79,7 +83,36 @@
}
return Optional.empty();
}
-
+
+ /**
+ * Load logic table metadata.
+ *
+ * @param materials schema builder materials
+ * @param executorService executorService
+ * @return table meta data collection
+ * @throws SQLException SQL exception
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public static Optional<Collection<TableMetaData>> loadLogicTables(final SchemaBuilderMaterials materials, final ExecutorService executorService) throws SQLException {
Review comment:
@tuichenchuxin Why add Optional for Collection result?
##########
File path: shardingsphere-features/shardingsphere-encrypt/shardingsphere-encrypt-core/src/main/java/org/apache/shardingsphere/encrypt/metadata/EncryptTableMetaDataBuilder.java
##########
@@ -49,6 +58,71 @@
? TableMetaDataLoader.load(dataSourceMap.get(dataSourceName), tableName, databaseType) : Optional.empty();
}
+ @Override
+ public Optional<Map<String, TableMetaData>> load(final Collection<String> tableNames, final DatabaseType databaseType, final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
+ final EncryptRule rule, final ConfigurationProperties props, final ExecutorService executorService) throws SQLException {
+ Optional<DialectTableMetaDataLoader> loader = findDialectTableMetaDataLoader(databaseType);
+ return loader.isPresent() ? loadByDialect(loader.get(), tableNames, dataSourceMap, dataNodes, rule, executorService)
Review comment:
@tuichenchuxin Can we remove the method of default?
##########
File path: shardingsphere-features/shardingsphere-encrypt/shardingsphere-encrypt-core/src/main/java/org/apache/shardingsphere/encrypt/metadata/EncryptTableMetaDataBuilder.java
##########
@@ -49,6 +58,71 @@
? TableMetaDataLoader.load(dataSourceMap.get(dataSourceName), tableName, databaseType) : Optional.empty();
}
+ @Override
+ public Optional<Map<String, TableMetaData>> load(final Collection<String> tableNames, final DatabaseType databaseType, final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
+ final EncryptRule rule, final ConfigurationProperties props, final ExecutorService executorService) throws SQLException {
+ Optional<DialectTableMetaDataLoader> loader = findDialectTableMetaDataLoader(databaseType);
+ return loader.isPresent() ? loadByDialect(loader.get(), tableNames, dataSourceMap, dataNodes, rule, executorService)
+ : loadByDefault(tableNames, dataSourceMap, dataNodes, rule, databaseType);
+ }
+
+ private Optional<Map<String, TableMetaData>> loadByDialect(final DialectTableMetaDataLoader dialectTableMetaDataLoader, final Collection<String> tableNames,
+ final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes, final EncryptRule rule,
+ final ExecutorService executorService) throws SQLException {
+ if (!tableNames.stream().allMatch(tableName -> rule.findEncryptTable(tableName).isPresent())) {
Review comment:
@tuichenchuxin Is it better to load the table where the encryption rule exists?
##########
File path: shardingsphere-features/shardingsphere-encrypt/shardingsphere-encrypt-core/src/main/java/org/apache/shardingsphere/encrypt/metadata/EncryptTableMetaDataBuilder.java
##########
@@ -49,6 +58,71 @@
? TableMetaDataLoader.load(dataSourceMap.get(dataSourceName), tableName, databaseType) : Optional.empty();
}
+ @Override
+ public Optional<Map<String, TableMetaData>> load(final Collection<String> tableNames, final DatabaseType databaseType, final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
+ final EncryptRule rule, final ConfigurationProperties props, final ExecutorService executorService) throws SQLException {
+ Optional<DialectTableMetaDataLoader> loader = findDialectTableMetaDataLoader(databaseType);
+ return loader.isPresent() ? loadByDialect(loader.get(), tableNames, dataSourceMap, dataNodes, rule, executorService)
+ : loadByDefault(tableNames, dataSourceMap, dataNodes, rule, databaseType);
+ }
+
+ private Optional<Map<String, TableMetaData>> loadByDialect(final DialectTableMetaDataLoader dialectTableMetaDataLoader, final Collection<String> tableNames,
+ final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes, final EncryptRule rule,
+ final ExecutorService executorService) throws SQLException {
+ if (!tableNames.stream().allMatch(tableName -> rule.findEncryptTable(tableName).isPresent())) {
+ return Optional.empty();
+ }
+ Map<String, TableMetaData> result = new LinkedHashMap<>();
+ Map<String, List<String>> dataSourceTablesMap = new LinkedHashMap<>();
+ for (String tableName : tableNames) {
Review comment:
@tuichenchuxin It may be better to extract as a method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter commented on pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#issuecomment-898150963
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#11767](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (79f18e7) into [master](https://codecov.io/gh/apache/shardingsphere/commit/798ca3a9b99817aa0a8bc8fe9fcc24a69c133522?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (798ca3a) will **decrease** coverage by `0.17%`.
> The diff coverage is `48.38%`.
[](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #11767 +/- ##
============================================
- Coverage 63.71% 63.54% -0.18%
- Complexity 1193 1215 +22
============================================
Files 2289 2306 +17
Lines 34801 35006 +205
Branches 6029 6072 +43
============================================
+ Hits 22172 22243 +71
- Misses 10857 10975 +118
- Partials 1772 1788 +16
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...harding/metadata/ShardingTableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmcvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmctY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc2hhcmRpbmcvbWV0YWRhdGEvU2hhcmRpbmdUYWJsZU1ldGFEYXRhQnVpbGRlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../builder/loader/dialect/H2TableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9IMlRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `80.68% <62.50%> (-6.38%)` | :arrow_down: |
| [...ilder/loader/dialect/MySQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9NeVNRTFRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `83.63% <66.66%> (-3.41%)` | :arrow_down: |
| [...r/loader/dialect/SQLServerTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9TUUxTZXJ2ZXJUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `83.01% <66.66%> (-3.52%)` | :arrow_down: |
| [...lder/loader/dialect/OracleTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9PcmFjbGVUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `81.08% <70.00%> (-5.04%)` | :arrow_down: |
| [.../encrypt/metadata/EncryptTableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtZW5jcnlwdC9zaGFyZGluZ3NwaGVyZS1lbmNyeXB0LWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2VuY3J5cHQvbWV0YWRhdGEvRW5jcnlwdFRhYmxlTWV0YURhdGFCdWlsZGVyLmphdmE=) | `81.25% <73.80%> (-14.21%)` | :arrow_down: |
| [.../loader/dialect/PostgreSQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9Qb3N0Z3JlU1FMVGFibGVNZXRhRGF0YUxvYWRlci5qYXZh) | `82.81% <75.00%> (-2.91%)` | :arrow_down: |
| [.../metadata/schema/builder/TableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9UYWJsZU1ldGFEYXRhQnVpbGRlci5qYXZh) | `85.71% <78.57%> (-4.77%)` | :arrow_down: |
| [...e/infra/metadata/schema/builder/SchemaBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9TY2hlbWFCdWlsZGVyLmphdmE=) | `72.00% <83.33%> (+2.76%)` | :arrow_up: |
| [...sphere/scaling/core/job/schedule/JobScheduler.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtc2NhbGluZy9zaGFyZGluZ3NwaGVyZS1zY2FsaW5nLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3NjYWxpbmcvY29yZS9qb2Ivc2NoZWR1bGUvSm9iU2NoZWR1bGVyLmphdmE=) | `22.22% <0.00%> (-29.63%)` | :arrow_down: |
| ... and [74 more](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [798ca3a...79f18e7](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on a change in pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on a change in pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#discussion_r690050286
##########
File path: shardingsphere-features/shardingsphere-encrypt/shardingsphere-encrypt-core/src/main/java/org/apache/shardingsphere/encrypt/metadata/EncryptTableMetaDataBuilder.java
##########
@@ -49,6 +57,31 @@
? TableMetaDataLoader.load(dataSourceMap.get(dataSourceName), tableName, databaseType) : Optional.empty();
}
+ @Override
+ public Map<String, TableMetaData> load(final Collection<String> tableNames, final EncryptRule rule, final SchemaBuilderMaterials materials,
+ final ExecutorService executorService) throws SQLException {
+ Optional<DialectTableMetaDataLoader> loader = TableMetaDataLoader.findDialectTableMetaDataLoader(materials.getDatabaseType());
+ Collection<String> loadTableNames = tableNames.stream().filter(each -> rule.findEncryptTable(each).isPresent()).collect(Collectors.toList());
+ if (loadTableNames.isEmpty()) {
+ return Collections.emptyMap();
+ }
+ Map<String, Collection<String>> dataSourceTables = getTableGroup(loadTableNames, materials);
+ return loader.isPresent() ? TableMetaDataLoader.load(loader.get(), dataSourceTables, materials.getDataSourceMap(), executorService)
+ : TableMetaDataLoader.load(dataSourceTables, materials.getDatabaseType(), materials.getDataSourceMap());
+ }
+
+ private Map<String, Collection<String>> getTableGroup(final Collection<String> tableNames, final SchemaBuilderMaterials materials) {
+ Map<String, Collection<String>> result = new LinkedHashMap<>();
+ DataNodes dataNodes = new DataNodes(materials.getRules());
+ for (String tableName : tableNames) {
Review comment:
@tuichenchuxin modify `tableName` to `each`.
##########
File path: shardingsphere-features/shardingsphere-sharding/shardingsphere-sharding-core/src/main/java/org/apache/shardingsphere/sharding/metadata/ShardingTableMetaDataBuilder.java
##########
@@ -82,6 +87,63 @@
return Optional.of(actualTableMetaDataMap.values().iterator().next());
}
+ @Override
+ public Map<String, TableMetaData> load(final Collection<String> tableNames, final ShardingRule rule, final SchemaBuilderMaterials materials,
+ final ExecutorService executorService) throws SQLException {
+ Collection<String> loadTableNames = tableNames.stream().filter(each -> rule.findTableRule(each).isPresent()).collect(Collectors.toList());
+ if (loadTableNames.isEmpty()) {
+ return Collections.emptyMap();
+ }
+ boolean isCheckingMetaData = materials.getProps().getValue(ConfigurationPropertyKey.CHECK_TABLE_METADATA_ENABLED);
+ return isCheckingMetaData ? loadWithCheck(loadTableNames, rule, materials) : loadWithOutCheck(loadTableNames, rule, materials, executorService);
+ }
+
+ private Map<String, TableMetaData> loadWithCheck(final Collection<String> tableNames, final ShardingRule rule, final SchemaBuilderMaterials materials) {
+ int maxConnectionsSizePerQuery = materials.getProps().getValue(ConfigurationPropertyKey.MAX_CONNECTIONS_SIZE_PER_QUERY);
+ Map<String, TableMetaData> result = new HashMap<>();
+ for (String tableName : tableNames) {
Review comment:
@tuichenchuxin Same problem.
##########
File path: shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/builder/loader/TableMetaDataLoader.java
##########
@@ -59,6 +69,26 @@
}
}
+ /**
+ * Load table meta data.
+ *
+ * @param dataSourceTable data source table name map
+ * @param databaseType database type
+ * @param dataSourceMap data source map
+ * @return table meta data map
+ * @throws SQLException SQL exception
+ */
+ public static Map<String, TableMetaData> load(final Map<String, Collection<String>> dataSourceTable, final DatabaseType databaseType,
+ final Map<String, DataSource> dataSourceMap) throws SQLException {
+ Map<String, TableMetaData> result = new LinkedHashMap<>();
+ for (Entry<String, Collection<String>> each : dataSourceTable.entrySet()) {
Review comment:
@tuichenchuxin `entry` is better.
##########
File path: shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/builder/loader/TableMetaDataLoader.java
##########
@@ -59,6 +69,26 @@
}
}
+ /**
+ * Load table meta data.
+ *
+ * @param dataSourceTable data source table name map
+ * @param databaseType database type
+ * @param dataSourceMap data source map
+ * @return table meta data map
+ * @throws SQLException SQL exception
+ */
+ public static Map<String, TableMetaData> load(final Map<String, Collection<String>> dataSourceTable, final DatabaseType databaseType,
+ final Map<String, DataSource> dataSourceMap) throws SQLException {
+ Map<String, TableMetaData> result = new LinkedHashMap<>();
+ for (Entry<String, Collection<String>> each : dataSourceTable.entrySet()) {
+ for (String tableName : each.getValue()) {
Review comment:
@tuichenchuxin `each` is better.
##########
File path: shardingsphere-features/shardingsphere-sharding/shardingsphere-sharding-core/src/main/java/org/apache/shardingsphere/sharding/metadata/ShardingTableMetaDataBuilder.java
##########
@@ -82,6 +87,63 @@
return Optional.of(actualTableMetaDataMap.values().iterator().next());
}
+ @Override
+ public Map<String, TableMetaData> load(final Collection<String> tableNames, final ShardingRule rule, final SchemaBuilderMaterials materials,
+ final ExecutorService executorService) throws SQLException {
+ Collection<String> loadTableNames = tableNames.stream().filter(each -> rule.findTableRule(each).isPresent()).collect(Collectors.toList());
+ if (loadTableNames.isEmpty()) {
+ return Collections.emptyMap();
+ }
+ boolean isCheckingMetaData = materials.getProps().getValue(ConfigurationPropertyKey.CHECK_TABLE_METADATA_ENABLED);
+ return isCheckingMetaData ? loadWithCheck(loadTableNames, rule, materials) : loadWithOutCheck(loadTableNames, rule, materials, executorService);
+ }
+
+ private Map<String, TableMetaData> loadWithCheck(final Collection<String> tableNames, final ShardingRule rule, final SchemaBuilderMaterials materials) {
+ int maxConnectionsSizePerQuery = materials.getProps().getValue(ConfigurationPropertyKey.MAX_CONNECTIONS_SIZE_PER_QUERY);
+ Map<String, TableMetaData> result = new HashMap<>();
+ for (String tableName : tableNames) {
+ TableRule tableRule = rule.getTableRule(tableName);
+ Map<String, TableMetaData> actualTableMetaDataMap = parallelLoadTables(materials.getDatabaseType(), materials.getDataSourceMap(),
+ new DataNodes(materials.getRules()), tableName, maxConnectionsSizePerQuery);
+ if (actualTableMetaDataMap.isEmpty()) {
+ continue;
+ }
+ checkUniformed(tableRule.getLogicTable(), actualTableMetaDataMap, rule);
+ result.put(tableRule.getLogicTable(), actualTableMetaDataMap.values().iterator().next());
+ }
+ return result;
+ }
+
+ private Map<String, TableMetaData> loadWithOutCheck(final Collection<String> tableNames, final ShardingRule rule,
+ final SchemaBuilderMaterials materials, final ExecutorService executorService) throws SQLException {
+ Optional<DialectTableMetaDataLoader> loader = TableMetaDataLoader.findDialectTableMetaDataLoader(materials.getDatabaseType());
+ Map<String, Collection<String>> dataSourceTables = getTableGroup(tableNames, materials);
+ return loader.isPresent() ? new LinkedHashMap<>(getLogicTableName(TableMetaDataLoader.load(loader.get(), dataSourceTables, materials.getDataSourceMap(), executorService).values(), rule))
Review comment:
@tuichenchuxin Is it better to get the results first and then decorate them?
##########
File path: shardingsphere-features/shardingsphere-sharding/shardingsphere-sharding-core/src/main/java/org/apache/shardingsphere/sharding/metadata/ShardingTableMetaDataBuilder.java
##########
@@ -82,6 +87,63 @@
return Optional.of(actualTableMetaDataMap.values().iterator().next());
}
+ @Override
+ public Map<String, TableMetaData> load(final Collection<String> tableNames, final ShardingRule rule, final SchemaBuilderMaterials materials,
+ final ExecutorService executorService) throws SQLException {
+ Collection<String> loadTableNames = tableNames.stream().filter(each -> rule.findTableRule(each).isPresent()).collect(Collectors.toList());
+ if (loadTableNames.isEmpty()) {
+ return Collections.emptyMap();
+ }
+ boolean isCheckingMetaData = materials.getProps().getValue(ConfigurationPropertyKey.CHECK_TABLE_METADATA_ENABLED);
+ return isCheckingMetaData ? loadWithCheck(loadTableNames, rule, materials) : loadWithOutCheck(loadTableNames, rule, materials, executorService);
+ }
+
+ private Map<String, TableMetaData> loadWithCheck(final Collection<String> tableNames, final ShardingRule rule, final SchemaBuilderMaterials materials) {
+ int maxConnectionsSizePerQuery = materials.getProps().getValue(ConfigurationPropertyKey.MAX_CONNECTIONS_SIZE_PER_QUERY);
+ Map<String, TableMetaData> result = new HashMap<>();
+ for (String tableName : tableNames) {
+ TableRule tableRule = rule.getTableRule(tableName);
+ Map<String, TableMetaData> actualTableMetaDataMap = parallelLoadTables(materials.getDatabaseType(), materials.getDataSourceMap(),
+ new DataNodes(materials.getRules()), tableName, maxConnectionsSizePerQuery);
+ if (actualTableMetaDataMap.isEmpty()) {
+ continue;
+ }
+ checkUniformed(tableRule.getLogicTable(), actualTableMetaDataMap, rule);
+ result.put(tableRule.getLogicTable(), actualTableMetaDataMap.values().iterator().next());
+ }
+ return result;
+ }
+
+ private Map<String, TableMetaData> loadWithOutCheck(final Collection<String> tableNames, final ShardingRule rule,
+ final SchemaBuilderMaterials materials, final ExecutorService executorService) throws SQLException {
+ Optional<DialectTableMetaDataLoader> loader = TableMetaDataLoader.findDialectTableMetaDataLoader(materials.getDatabaseType());
+ Map<String, Collection<String>> dataSourceTables = getTableGroup(tableNames, materials);
+ return loader.isPresent() ? new LinkedHashMap<>(getLogicTableName(TableMetaDataLoader.load(loader.get(), dataSourceTables, materials.getDataSourceMap(), executorService).values(), rule))
+ : new LinkedHashMap<>(getLogicTableName(TableMetaDataLoader.load(dataSourceTables, materials.getDatabaseType(), materials.getDataSourceMap()).values(), rule));
+ }
+
+ private Map<String, Collection<String>> getTableGroup(final Collection<String> tableNames, final SchemaBuilderMaterials materials) {
+ DataNodes dataNodes = new DataNodes(materials.getRules());
+ Map<String, Collection<String>> result = new LinkedHashMap<>();
+ for (String tableName : tableNames) {
+ DataNode dataNode = dataNodes.getDataNodes(tableName).iterator().next();
+ Collection<String> tables = result.getOrDefault(dataNode.getDataSourceName(), new LinkedList<>());
+ tables.add(dataNode.getTableName());
+ result.putIfAbsent(dataNode.getDataSourceName(), tables);
+ }
+ return result;
+ }
+
+ private Map<String, TableMetaData> getLogicTableName(final Collection<TableMetaData> tableMetaDatas, final ShardingRule rule) {
+ Map<String, TableMetaData> result = new LinkedHashMap<>();
+ for (TableMetaData each : tableMetaDatas) {
+ if (rule.findLogicTableByActualTable(each.getName()).isPresent()) {
Review comment:
@tuichenchuxin Please avoid multiple calls `findLogicTableByActualTable` method, using `ifPresent`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter edited a comment on pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#issuecomment-898150963
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#11767](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (99d3c82) into [master](https://codecov.io/gh/apache/shardingsphere/commit/291dcd46bb25dc3ef5757e480a4f7d68b15bdb5f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (291dcd4) will **decrease** coverage by `0.15%`.
> The diff coverage is `75.62%`.
[](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #11767 +/- ##
============================================
- Coverage 63.70% 63.55% -0.16%
- Complexity 1196 1221 +25
============================================
Files 2306 2313 +7
Lines 34894 35085 +191
Branches 6051 6104 +53
============================================
+ Hits 22230 22299 +69
- Misses 10889 10979 +90
- Partials 1775 1807 +32
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ingsphere/infra/executor/check/SQLCheckEngine.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtZXhlY3V0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2luZnJhL2V4ZWN1dG9yL2NoZWNrL1NRTENoZWNrRW5naW5lLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...or/sql/prepare/AbstractExecutionPrepareEngine.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtZXhlY3V0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2luZnJhL2V4ZWN1dG9yL3NxbC9wcmVwYXJlL0Fic3RyYWN0RXhlY3V0aW9uUHJlcGFyZUVuZ2luZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...initializer/impl/AbstractBootstrapInitializer.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtcHJveHkvc2hhcmRpbmdzcGhlcmUtcHJveHktYm9vdHN0cmFwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9wcm94eS9pbml0aWFsaXplci9pbXBsL0Fic3RyYWN0Qm9vdHN0cmFwSW5pdGlhbGl6ZXIuamF2YQ==) | `0.00% <ø> (-5.62%)` | :arrow_down: |
| [...ilder/loader/dialect/MySQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9NeVNRTFRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `81.81% <50.00%> (-5.22%)` | :arrow_down: |
| [.../builder/loader/dialect/H2TableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9IMlRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `79.54% <55.55%> (-7.52%)` | :arrow_down: |
| [...lder/loader/dialect/OracleTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9PcmFjbGVUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `79.72% <58.33%> (-6.39%)` | :arrow_down: |
| [...infra/rule/builder/ShardingSphereRulesBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9ydWxlL2J1aWxkZXIvU2hhcmRpbmdTcGhlcmVSdWxlc0J1aWxkZXIuamF2YQ==) | `61.22% <60.00%> (ø)` | |
| [...r/loader/dialect/SQLServerTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9TUUxTZXJ2ZXJUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `81.13% <62.50%> (-5.41%)` | :arrow_down: |
| [.../loader/dialect/PostgreSQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9Qb3N0Z3JlU1FMVGFibGVNZXRhRGF0YUxvYWRlci5qYXZh) | `81.25% <66.66%> (-4.47%)` | :arrow_down: |
| [...harding/metadata/ShardingTableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmcvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmctY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc2hhcmRpbmcvbWV0YWRhdGEvU2hhcmRpbmdUYWJsZU1ldGFEYXRhQnVpbGRlci5qYXZh) | `61.36% <82.45%> (+61.36%)` | :arrow_up: |
| ... and [137 more](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2083867...99d3c82](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on a change in pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on a change in pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#discussion_r689261091
##########
File path: shardingsphere-features/shardingsphere-encrypt/shardingsphere-encrypt-core/src/main/java/org/apache/shardingsphere/encrypt/metadata/EncryptTableMetaDataBuilder.java
##########
@@ -49,6 +60,73 @@
? TableMetaDataLoader.load(dataSourceMap.get(dataSourceName), tableName, databaseType) : Optional.empty();
}
+ @Override
+ public Map<String, TableMetaData> load(final Collection<String> tableNames, final DatabaseType databaseType, final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
+ final EncryptRule rule, final ConfigurationProperties props, final ExecutorService executorService) throws SQLException {
+ Optional<DialectTableMetaDataLoader> loader = findDialectTableMetaDataLoader(databaseType);
+ Collection<String> loadTableNames = tableNames.stream().filter(tableName -> rule.findEncryptTable(tableName).isPresent()).collect(Collectors.toList());
+ if (loadTableNames.isEmpty()) {
+ return Collections.emptyMap();
+ }
+ return loader.isPresent() ? loadByDialect(loader.get(), loadTableNames, dataSourceMap, dataNodes, executorService)
+ : loadByDefault(loadTableNames, dataSourceMap, dataNodes, rule, databaseType);
+ }
+
+ private Map<String, TableMetaData> loadByDialect(final DialectTableMetaDataLoader dialectTableMetaDataLoader, final Collection<String> tableNames,
+ final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
+ final ExecutorService executorService) throws SQLException {
+ Map<String, TableMetaData> result = new LinkedHashMap<>();
+ Map<String, Collection<String>> dataSourceTablesMap = getDataSourceTablesGroup(tableNames, dataSourceMap, dataNodes);
+ Collection<Future<Map<String, TableMetaData>>> futures = new LinkedList<>();
+ for (Map.Entry<String, Collection<String>> each : dataSourceTablesMap.entrySet()) {
+ futures.add(executorService.submit(() -> dialectTableMetaDataLoader
+ .load(dataSourceMap.get(each.getKey()), each.getValue(), false)));
+ }
+ try {
+ for (Future<Map<String, TableMetaData>> each : futures) {
+ result.putAll(each.get());
+ }
+ } catch (final InterruptedException | ExecutionException ex) {
+ if (ex.getCause() instanceof SQLException) {
+ throw (SQLException) ex.getCause();
+ }
+ throw new ShardingSphereException(ex);
+ }
+ return result;
+ }
+
+ private Map<String, Collection<String>> getDataSourceTablesGroup(final Collection<String> tableNames, final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes) {
+ Map<String, Collection<String>> result = new LinkedHashMap<>();
+ for (String tableName : tableNames) {
+ String dataSourceName = dataNodes.getDataNodes(tableName).stream().map(DataNode::getDataSourceName).findFirst().orElseGet(() -> dataSourceMap.keySet().iterator().next());
+ Collection<String> collection = result.getOrDefault(dataSourceName, new LinkedList<>());
+ collection.add(tableName);
+ result.put(dataSourceName, collection);
Review comment:
@tuichenchuxin Use `putIfAbsent` is better.
##########
File path: shardingsphere-features/shardingsphere-encrypt/shardingsphere-encrypt-core/src/main/java/org/apache/shardingsphere/encrypt/metadata/EncryptTableMetaDataBuilder.java
##########
@@ -49,6 +60,73 @@
? TableMetaDataLoader.load(dataSourceMap.get(dataSourceName), tableName, databaseType) : Optional.empty();
}
+ @Override
+ public Map<String, TableMetaData> load(final Collection<String> tableNames, final DatabaseType databaseType, final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
+ final EncryptRule rule, final ConfigurationProperties props, final ExecutorService executorService) throws SQLException {
+ Optional<DialectTableMetaDataLoader> loader = findDialectTableMetaDataLoader(databaseType);
+ Collection<String> loadTableNames = tableNames.stream().filter(tableName -> rule.findEncryptTable(tableName).isPresent()).collect(Collectors.toList());
+ if (loadTableNames.isEmpty()) {
+ return Collections.emptyMap();
+ }
+ return loader.isPresent() ? loadByDialect(loader.get(), loadTableNames, dataSourceMap, dataNodes, executorService)
+ : loadByDefault(loadTableNames, dataSourceMap, dataNodes, rule, databaseType);
+ }
+
+ private Map<String, TableMetaData> loadByDialect(final DialectTableMetaDataLoader dialectTableMetaDataLoader, final Collection<String> tableNames,
+ final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
+ final ExecutorService executorService) throws SQLException {
+ Map<String, TableMetaData> result = new LinkedHashMap<>();
+ Map<String, Collection<String>> dataSourceTablesMap = getDataSourceTablesGroup(tableNames, dataSourceMap, dataNodes);
+ Collection<Future<Map<String, TableMetaData>>> futures = new LinkedList<>();
+ for (Map.Entry<String, Collection<String>> each : dataSourceTablesMap.entrySet()) {
Review comment:
@tuichenchuxin Please use static import for `Map.Entry`.
##########
File path: shardingsphere-features/shardingsphere-encrypt/shardingsphere-encrypt-core/src/main/java/org/apache/shardingsphere/encrypt/metadata/EncryptTableMetaDataBuilder.java
##########
@@ -49,6 +60,73 @@
? TableMetaDataLoader.load(dataSourceMap.get(dataSourceName), tableName, databaseType) : Optional.empty();
}
+ @Override
+ public Map<String, TableMetaData> load(final Collection<String> tableNames, final DatabaseType databaseType, final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
+ final EncryptRule rule, final ConfigurationProperties props, final ExecutorService executorService) throws SQLException {
+ Optional<DialectTableMetaDataLoader> loader = findDialectTableMetaDataLoader(databaseType);
+ Collection<String> loadTableNames = tableNames.stream().filter(tableName -> rule.findEncryptTable(tableName).isPresent()).collect(Collectors.toList());
+ if (loadTableNames.isEmpty()) {
+ return Collections.emptyMap();
+ }
+ return loader.isPresent() ? loadByDialect(loader.get(), loadTableNames, dataSourceMap, dataNodes, executorService)
+ : loadByDefault(loadTableNames, dataSourceMap, dataNodes, rule, databaseType);
+ }
+
+ private Map<String, TableMetaData> loadByDialect(final DialectTableMetaDataLoader dialectTableMetaDataLoader, final Collection<String> tableNames,
+ final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
+ final ExecutorService executorService) throws SQLException {
+ Map<String, TableMetaData> result = new LinkedHashMap<>();
+ Map<String, Collection<String>> dataSourceTablesMap = getDataSourceTablesGroup(tableNames, dataSourceMap, dataNodes);
+ Collection<Future<Map<String, TableMetaData>>> futures = new LinkedList<>();
+ for (Map.Entry<String, Collection<String>> each : dataSourceTablesMap.entrySet()) {
+ futures.add(executorService.submit(() -> dialectTableMetaDataLoader
+ .load(dataSourceMap.get(each.getKey()), each.getValue(), false)));
+ }
+ try {
+ for (Future<Map<String, TableMetaData>> each : futures) {
+ result.putAll(each.get());
+ }
+ } catch (final InterruptedException | ExecutionException ex) {
+ if (ex.getCause() instanceof SQLException) {
+ throw (SQLException) ex.getCause();
+ }
+ throw new ShardingSphereException(ex);
+ }
+ return result;
+ }
+
+ private Map<String, Collection<String>> getDataSourceTablesGroup(final Collection<String> tableNames, final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes) {
+ Map<String, Collection<String>> result = new LinkedHashMap<>();
+ for (String tableName : tableNames) {
+ String dataSourceName = dataNodes.getDataNodes(tableName).stream().map(DataNode::getDataSourceName).findFirst().orElseGet(() -> dataSourceMap.keySet().iterator().next());
+ Collection<String> collection = result.getOrDefault(dataSourceName, new LinkedList<>());
Review comment:
@tuichenchuxin `tableNames` is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] taojintianxia commented on a change in pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
taojintianxia commented on a change in pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#discussion_r689316844
##########
File path: shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/builder/loader/dialect/H2TableMetaDataLoader.java
##########
@@ -164,14 +178,15 @@ private String getPrimaryKeyMetaDataSQL(final Collection<String> existedTables)
return result;
}
- private String getGeneratedInfoSQL(final Collection<String> existedTables) {
- return existedTables.isEmpty() ? GENERATED_INFO_SQL
- : String.format(GENERATED_INFO_SQL_WITH_EXISTED_TABLES, existedTables.stream().map(each -> String.format("'%s'", each)).collect(Collectors.joining(",")));
+ private String getGeneratedInfoSQL(final Collection<String> tables, final boolean isExclude) {
Review comment:
conditional-operator appear two times in one clause ? I believe this is easy to make people confused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter edited a comment on pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#issuecomment-898150963
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#11767](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8d80d2d) into [master](https://codecov.io/gh/apache/shardingsphere/commit/291dcd46bb25dc3ef5757e480a4f7d68b15bdb5f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (291dcd4) will **decrease** coverage by `0.19%`.
> The diff coverage is `71.06%`.
[](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #11767 +/- ##
============================================
- Coverage 63.70% 63.51% -0.20%
- Complexity 1196 1216 +20
============================================
Files 2306 2309 +3
Lines 34894 35033 +139
Branches 6051 6099 +48
============================================
+ Hits 22230 22251 +21
- Misses 10889 10978 +89
- Partials 1775 1804 +29
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ingsphere/infra/executor/check/SQLCheckEngine.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtZXhlY3V0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2luZnJhL2V4ZWN1dG9yL2NoZWNrL1NRTENoZWNrRW5naW5lLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...or/sql/prepare/AbstractExecutionPrepareEngine.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtZXhlY3V0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2luZnJhL2V4ZWN1dG9yL3NxbC9wcmVwYXJlL0Fic3RyYWN0RXhlY3V0aW9uUHJlcGFyZUVuZ2luZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...initializer/impl/AbstractBootstrapInitializer.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtcHJveHkvc2hhcmRpbmdzcGhlcmUtcHJveHktYm9vdHN0cmFwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9wcm94eS9pbml0aWFsaXplci9pbXBsL0Fic3RyYWN0Qm9vdHN0cmFwSW5pdGlhbGl6ZXIuamF2YQ==) | `4.93% <ø> (-0.68%)` | :arrow_down: |
| [...ilder/loader/dialect/MySQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9NeVNRTFRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `81.81% <50.00%> (-5.22%)` | :arrow_down: |
| [.../builder/loader/dialect/H2TableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9IMlRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `79.54% <55.55%> (-7.52%)` | :arrow_down: |
| [...lder/loader/dialect/OracleTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9PcmFjbGVUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `79.72% <58.33%> (-6.39%)` | :arrow_down: |
| [...infra/rule/builder/ShardingSphereRulesBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9ydWxlL2J1aWxkZXIvU2hhcmRpbmdTcGhlcmVSdWxlc0J1aWxkZXIuamF2YQ==) | `61.22% <60.00%> (ø)` | |
| [...r/loader/dialect/SQLServerTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9TUUxTZXJ2ZXJUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `81.13% <62.50%> (-5.41%)` | :arrow_down: |
| [.../loader/dialect/PostgreSQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9Qb3N0Z3JlU1FMVGFibGVNZXRhRGF0YUxvYWRlci5qYXZh) | `81.25% <66.66%> (-4.47%)` | :arrow_down: |
| [.../encrypt/metadata/EncryptTableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtZW5jcnlwdC9zaGFyZGluZ3NwaGVyZS1lbmNyeXB0LWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2VuY3J5cHQvbWV0YWRhdGEvRW5jcnlwdFRhYmxlTWV0YURhdGFCdWlsZGVyLmphdmE=) | `81.25% <73.80%> (-14.21%)` | :arrow_down: |
| ... and [63 more](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2083867...8d80d2d](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter edited a comment on pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#issuecomment-898150963
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#11767](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b59e383) into [master](https://codecov.io/gh/apache/shardingsphere/commit/291dcd46bb25dc3ef5757e480a4f7d68b15bdb5f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (291dcd4) will **increase** coverage by `0.08%`.
> The diff coverage is `73.40%`.
[](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #11767 +/- ##
============================================
+ Coverage 63.70% 63.79% +0.08%
- Complexity 1196 1218 +22
============================================
Files 2306 2313 +7
Lines 34894 35060 +166
Branches 6051 6084 +33
============================================
+ Hits 22230 22366 +136
- Misses 10889 10892 +3
- Partials 1775 1802 +27
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ingsphere/infra/executor/check/SQLCheckEngine.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtZXhlY3V0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2luZnJhL2V4ZWN1dG9yL2NoZWNrL1NRTENoZWNrRW5naW5lLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...or/sql/prepare/AbstractExecutionPrepareEngine.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtZXhlY3V0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2luZnJhL2V4ZWN1dG9yL3NxbC9wcmVwYXJlL0Fic3RyYWN0RXhlY3V0aW9uUHJlcGFyZUVuZ2luZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...initializer/impl/AbstractBootstrapInitializer.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtcHJveHkvc2hhcmRpbmdzcGhlcmUtcHJveHktYm9vdHN0cmFwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9wcm94eS9pbml0aWFsaXplci9pbXBsL0Fic3RyYWN0Qm9vdHN0cmFwSW5pdGlhbGl6ZXIuamF2YQ==) | `5.61% <0.00%> (ø)` | |
| [...itializer/impl/GovernanceBootstrapInitializer.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtcHJveHkvc2hhcmRpbmdzcGhlcmUtcHJveHktYm9vdHN0cmFwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9wcm94eS9pbml0aWFsaXplci9pbXBsL0dvdmVybmFuY2VCb290c3RyYXBJbml0aWFsaXplci5qYXZh) | `26.31% <ø> (+1.31%)` | :arrow_up: |
| [...initializer/impl/StandardBootstrapInitializer.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtcHJveHkvc2hhcmRpbmdzcGhlcmUtcHJveHktYm9vdHN0cmFwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9wcm94eS9pbml0aWFsaXplci9pbXBsL1N0YW5kYXJkQm9vdHN0cmFwSW5pdGlhbGl6ZXIuamF2YQ==) | `50.00% <ø> (+7.14%)` | :arrow_up: |
| [...infra/rule/builder/ShardingSphereRulesBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9ydWxlL2J1aWxkZXIvU2hhcmRpbmdTcGhlcmVSdWxlc0J1aWxkZXIuamF2YQ==) | `61.22% <60.00%> (ø)` | |
| [.../builder/loader/dialect/H2TableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9IMlRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `80.68% <62.50%> (-6.38%)` | :arrow_down: |
| [...ilder/loader/dialect/MySQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9NeVNRTFRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `83.63% <66.66%> (-3.41%)` | :arrow_down: |
| [...r/loader/dialect/SQLServerTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9TUUxTZXJ2ZXJUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `83.01% <66.66%> (-3.52%)` | :arrow_down: |
| [...lder/loader/dialect/OracleTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9PcmFjbGVUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `81.08% <70.00%> (-5.04%)` | :arrow_down: |
| ... and [30 more](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2083867...b59e383](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter edited a comment on pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#issuecomment-898150963
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#11767](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8cdcbf1) into [master](https://codecov.io/gh/apache/shardingsphere/commit/798ca3a9b99817aa0a8bc8fe9fcc24a69c133522?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (798ca3a) will **decrease** coverage by `0.11%`.
> The diff coverage is `46.87%`.
[](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #11767 +/- ##
============================================
- Coverage 63.71% 63.59% -0.12%
- Complexity 1193 1215 +22
============================================
Files 2289 2306 +17
Lines 34801 35011 +210
Branches 6029 6080 +51
============================================
+ Hits 22172 22265 +93
- Misses 10857 10957 +100
- Partials 1772 1789 +17
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...harding/metadata/ShardingTableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmcvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmctY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc2hhcmRpbmcvbWV0YWRhdGEvU2hhcmRpbmdUYWJsZU1ldGFEYXRhQnVpbGRlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../builder/loader/dialect/H2TableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9IMlRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `80.68% <62.50%> (-6.38%)` | :arrow_down: |
| [...ilder/loader/dialect/MySQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9NeVNRTFRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `83.63% <66.66%> (-3.41%)` | :arrow_down: |
| [...r/loader/dialect/SQLServerTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9TUUxTZXJ2ZXJUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `83.01% <66.66%> (-3.52%)` | :arrow_down: |
| [...lder/loader/dialect/OracleTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9PcmFjbGVUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `81.08% <70.00%> (-5.04%)` | :arrow_down: |
| [.../encrypt/metadata/EncryptTableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtZW5jcnlwdC9zaGFyZGluZ3NwaGVyZS1lbmNyeXB0LWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2VuY3J5cHQvbWV0YWRhdGEvRW5jcnlwdFRhYmxlTWV0YURhdGFCdWlsZGVyLmphdmE=) | `81.25% <73.80%> (-14.21%)` | :arrow_down: |
| [.../loader/dialect/PostgreSQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9Qb3N0Z3JlU1FMVGFibGVNZXRhRGF0YUxvYWRlci5qYXZh) | `82.81% <75.00%> (-2.91%)` | :arrow_down: |
| [.../metadata/schema/builder/TableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9UYWJsZU1ldGFEYXRhQnVpbGRlci5qYXZh) | `85.71% <78.57%> (-4.77%)` | :arrow_down: |
| [...e/infra/metadata/schema/builder/SchemaBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9TY2hlbWFCdWlsZGVyLmphdmE=) | `72.00% <83.33%> (+2.76%)` | :arrow_up: |
| [...e/shardingsphere/shadow/route/ShadowSQLRouter.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtc2hhZG93L3NoYXJkaW5nc3BoZXJlLXNoYWRvdy1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9zaGFkb3cvcm91dGUvU2hhZG93U1FMUm91dGVyLmphdmE=) | `88.88% <0.00%> (-8.62%)` | :arrow_down: |
| ... and [70 more](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [798ca3a...8cdcbf1](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on a change in pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on a change in pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#discussion_r690069926
##########
File path: shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/builder/TableMetaDataBuilder.java
##########
@@ -79,7 +83,40 @@
}
return Optional.empty();
}
-
+
+ /**
+ * Load logic table metadata.
+ *
+ * @param materials schema builder materials
+ * @param executorService executorService
+ * @return table meta data collection
+ * @throws SQLException SQL exception
+ */
+ @SuppressWarnings("rawtypes")
+ public static Collection<TableMetaData> loadLogicTables(final SchemaBuilderMaterials materials, final ExecutorService executorService) throws SQLException {
+ Collection<TableMetaData> result = new LinkedList<>();
+ for (Entry<ShardingSphereRule, RuleBasedTableMetaDataBuilder> entry : OrderedSPIRegistry.getRegisteredServices(RuleBasedTableMetaDataBuilder.class, materials.getRules()).entrySet()) {
+ if (entry.getKey() instanceof TableContainedRule) {
+ loadTableContainedRuleTables(materials, executorService, result, entry);
+ }
+ }
+ return result;
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static void loadTableContainedRuleTables(final SchemaBuilderMaterials materials, final ExecutorService executorService, final Collection<TableMetaData> result,
+ final Entry<ShardingSphereRule, RuleBasedTableMetaDataBuilder> ruleBuilderEntry) throws SQLException {
+ TableContainedRule rule = (TableContainedRule) ruleBuilderEntry.getKey();
+ RuleBasedTableMetaDataBuilder loader = ruleBuilderEntry.getValue();
+ Collection<String> needLoadTables = rule.getTables().stream()
+ .filter(each -> !result.stream().map(TableMetaData::getName).collect(Collectors.toList()).contains(each)).collect(Collectors.toList());
Review comment:
@tuichenchuxin Please extract this logic to improve performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tuichenchuxin commented on a change in pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
tuichenchuxin commented on a change in pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#discussion_r689328012
##########
File path: shardingsphere-features/shardingsphere-encrypt/shardingsphere-encrypt-core/src/main/java/org/apache/shardingsphere/encrypt/metadata/EncryptTableMetaDataBuilder.java
##########
@@ -49,6 +61,72 @@
? TableMetaDataLoader.load(dataSourceMap.get(dataSourceName), tableName, databaseType) : Optional.empty();
}
+ @Override
+ public Map<String, TableMetaData> load(final Collection<String> tableNames, final DatabaseType databaseType, final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
Review comment:
ok，I'll try to decrease parameters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter edited a comment on pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#issuecomment-898150963
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#11767](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9a7b4af) into [master](https://codecov.io/gh/apache/shardingsphere/commit/291dcd46bb25dc3ef5757e480a4f7d68b15bdb5f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (291dcd4) will **increase** coverage by `0.11%`.
> The diff coverage is `73.91%`.
> :exclamation: Current head 9a7b4af differs from pull request most recent head 5984d86. Consider uploading reports for the commit 5984d86 to get more accurate results
[](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #11767 +/- ##
============================================
+ Coverage 63.70% 63.82% +0.11%
- Complexity 1196 1216 +20
============================================
Files 2306 2306
Lines 34894 35012 +118
Branches 6051 6080 +29
============================================
+ Hits 22230 22345 +115
+ Misses 10889 10868 -21
- Partials 1775 1799 +24
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../builder/loader/dialect/H2TableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9IMlRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `80.68% <62.50%> (-6.38%)` | :arrow_down: |
| [...ilder/loader/dialect/MySQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9NeVNRTFRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `83.63% <66.66%> (-3.41%)` | :arrow_down: |
| [...r/loader/dialect/SQLServerTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9TUUxTZXJ2ZXJUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `83.01% <66.66%> (-3.52%)` | :arrow_down: |
| [...lder/loader/dialect/OracleTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9PcmFjbGVUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `81.08% <70.00%> (-5.04%)` | :arrow_down: |
| [.../encrypt/metadata/EncryptTableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtZW5jcnlwdC9zaGFyZGluZ3NwaGVyZS1lbmNyeXB0LWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2VuY3J5cHQvbWV0YWRhdGEvRW5jcnlwdFRhYmxlTWV0YURhdGFCdWlsZGVyLmphdmE=) | `81.25% <73.80%> (-14.21%)` | :arrow_down: |
| [.../loader/dialect/PostgreSQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9Qb3N0Z3JlU1FMVGFibGVNZXRhRGF0YUxvYWRlci5qYXZh) | `82.81% <75.00%> (-2.91%)` | :arrow_down: |
| [...harding/metadata/ShardingTableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmcvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmctY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc2hhcmRpbmcvbWV0YWRhdGEvU2hhcmRpbmdUYWJsZU1ldGFEYXRhQnVpbGRlci5qYXZh) | `58.77% <76.78%> (+58.77%)` | :arrow_up: |
| [.../metadata/schema/builder/TableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9UYWJsZU1ldGFEYXRhQnVpbGRlci5qYXZh) | `86.11% <80.00%> (-4.37%)` | :arrow_down: |
| [...e/infra/metadata/schema/builder/SchemaBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9TY2hlbWFCdWlsZGVyLmphdmE=) | `72.00% <83.33%> (+2.76%)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2083867...5984d86](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter edited a comment on pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#issuecomment-898150963
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#11767](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (391db7f) into [master](https://codecov.io/gh/apache/shardingsphere/commit/798ca3a9b99817aa0a8bc8fe9fcc24a69c133522?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (798ca3a) will **decrease** coverage by `0.10%`.
> The diff coverage is `48.38%`.
[](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #11767 +/- ##
============================================
- Coverage 63.71% 63.60% -0.11%
- Complexity 1193 1215 +22
============================================
Files 2289 2306 +17
Lines 34801 35006 +205
Branches 6029 6072 +43
============================================
+ Hits 22172 22265 +93
- Misses 10857 10952 +95
- Partials 1772 1789 +17
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...harding/metadata/ShardingTableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmcvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmctY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc2hhcmRpbmcvbWV0YWRhdGEvU2hhcmRpbmdUYWJsZU1ldGFEYXRhQnVpbGRlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../builder/loader/dialect/H2TableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9IMlRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `80.68% <62.50%> (-6.38%)` | :arrow_down: |
| [...ilder/loader/dialect/MySQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9NeVNRTFRhYmxlTWV0YURhdGFMb2FkZXIuamF2YQ==) | `83.63% <66.66%> (-3.41%)` | :arrow_down: |
| [...r/loader/dialect/SQLServerTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9TUUxTZXJ2ZXJUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `83.01% <66.66%> (-3.52%)` | :arrow_down: |
| [...lder/loader/dialect/OracleTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9PcmFjbGVUYWJsZU1ldGFEYXRhTG9hZGVyLmphdmE=) | `81.08% <70.00%> (-5.04%)` | :arrow_down: |
| [.../encrypt/metadata/EncryptTableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtZW5jcnlwdC9zaGFyZGluZ3NwaGVyZS1lbmNyeXB0LWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2VuY3J5cHQvbWV0YWRhdGEvRW5jcnlwdFRhYmxlTWV0YURhdGFCdWlsZGVyLmphdmE=) | `81.25% <73.80%> (-14.21%)` | :arrow_down: |
| [.../loader/dialect/PostgreSQLTableMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9sb2FkZXIvZGlhbGVjdC9Qb3N0Z3JlU1FMVGFibGVNZXRhRGF0YUxvYWRlci5qYXZh) | `82.81% <75.00%> (-2.91%)` | :arrow_down: |
| [.../metadata/schema/builder/TableMetaDataBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9UYWJsZU1ldGFEYXRhQnVpbGRlci5qYXZh) | `85.71% <78.57%> (-4.77%)` | :arrow_down: |
| [...e/infra/metadata/schema/builder/SchemaBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvYnVpbGRlci9TY2hlbWFCdWlsZGVyLmphdmE=) | `72.00% <83.33%> (+2.76%)` | :arrow_up: |
| [...e/shardingsphere/shadow/route/ShadowSQLRouter.java](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtc2hhZG93L3NoYXJkaW5nc3BoZXJlLXNoYWRvdy1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9zaGFkb3cvcm91dGUvU2hhZG93U1FMUm91dGVyLmphdmE=) | `88.88% <0.00%> (-8.62%)` | :arrow_down: |
| ... and [70 more](https://codecov.io/gh/apache/shardingsphere/pull/11767/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [798ca3a...391db7f](https://codecov.io/gh/apache/shardingsphere/pull/11767?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] taojintianxia commented on a change in pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
taojintianxia commented on a change in pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767#discussion_r689312147
##########
File path: shardingsphere-features/shardingsphere-encrypt/shardingsphere-encrypt-core/src/main/java/org/apache/shardingsphere/encrypt/metadata/EncryptTableMetaDataBuilder.java
##########
@@ -49,6 +61,72 @@
? TableMetaDataLoader.load(dataSourceMap.get(dataSourceName), tableName, databaseType) : Optional.empty();
}
+ @Override
+ public Map<String, TableMetaData> load(final Collection<String> tableNames, final DatabaseType databaseType, final Map<String, DataSource> dataSourceMap, final DataNodes dataNodes,
Review comment:
too much parameters, if possible, why not encapsulate as a bean ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tristaZero merged pull request #11767: improve metadata load performance
Posted by GitBox <gi...@apache.org>.
tristaZero merged pull request #11767:
URL: https://github.com/apache/shardingsphere/pull/11767
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org