You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2020/01/15 10:10:35 UTC
[GitHub] [incubator-shardingsphere] beijing-penguin opened a new pull
request #3988: Using multithreading make loadDefaultTables faster and
remove repeat
beijing-penguin opened a new pull request #3988: Using multithreading make loadDefaultTables faster and remove repeat
URL: https://github.com/apache/incubator-shardingsphere/pull/3988
Fixes #3979.
Changes proposed in this pull request:
- Using multithreading make loadDefaultTables faster and remove repeat
load tableMetaData
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero commented on issue #3988:
Using multithreading make loadDefaultTables faster and remove repeat load
tableMetaData
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#issuecomment-575515321
/ci
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] beijing-penguin commented on issue
#3988: Using multithreading make loadDefaultTables faster and remove repeat
load tableMetaData
Posted by GitBox <gi...@apache.org>.
beijing-penguin commented on issue #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#issuecomment-574592977
/ci
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero merged pull request #3988:
Using multithreading make loadDefaultTables faster
Posted by GitBox <gi...@apache.org>.
tristaZero merged pull request #3988: Using multithreading make loadDefaultTables faster
URL: https://github.com/apache/incubator-shardingsphere/pull/3988
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] beijing-penguin commented on issue
#3988: Using multithreading make loadDefaultTables faster and remove repeat
load tableMetaData
Posted by GitBox <gi...@apache.org>.
beijing-penguin commented on issue #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#issuecomment-575439077
/ci
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] SteNicholas commented on a change in
pull request #3988: Using multithreading make loadDefaultTables faster and
remove repeat load tableMetaData
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on a change in pull request #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#discussion_r367220920
##########
File path: sharding-core/sharding-core-execute/src/main/java/org/apache/shardingsphere/sharding/execute/metadata/loader/ShardingTableMetaDataLoader.java
##########
@@ -206,19 +207,46 @@ public TableMetas loadAll(final ShardingRule shardingRule) throws SQLException {
return result;
}
- private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule) throws SQLException {
+ private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule, final Map<String, TableMetaData> result) throws SQLException {
Optional<String> actualDefaultDataSourceName = shardingRule.findActualDefaultDataSourceName();
if (!actualDefaultDataSourceName.isPresent()) {
return Collections.emptyMap();
}
Collection<String> tableNames = loadAllTableNames(actualDefaultDataSourceName.get());
- Map<String, TableMetaData> result = new HashMap<>(tableNames.size(), 1);
- for (String each : tableNames) {
- result.put(each, load(each, shardingRule));
+ removeRepeatTable(tableNames, result);
+ List<TableMetaData> metaList = executorEngine.execute(getTableNamesInput(tableNames), new GroupedCallback<String, TableMetaData>() {
+ @Override
+ public Collection<TableMetaData> execute(final Collection<String> inputs, final boolean isTrunkThread, final Map<String, Object> dataMap) throws SQLException {
+ String logicTableName = inputs.iterator().next();
+ Collection<TableMetaData> result = new LinkedList<>();
+ result.add(load(logicTableName, shardingRule));
+ return result;
+ }
+ });
+ Object[] tableNameArr = tableNames.toArray();
+ for (int i = 0, size = tableNames.size(); i < size; i++) {
Review comment:
Why not `for (int i = 0; i < tableNames.size(); i++) `?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] beijing-penguin commented on a change in
pull request #3988: Using multithreading make loadDefaultTables faster and
remove repeat load tableMetaData
Posted by GitBox <gi...@apache.org>.
beijing-penguin commented on a change in pull request #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#discussion_r367245267
##########
File path: sharding-core/sharding-core-execute/src/main/java/org/apache/shardingsphere/sharding/execute/metadata/loader/ShardingTableMetaDataLoader.java
##########
@@ -206,19 +207,46 @@ public TableMetas loadAll(final ShardingRule shardingRule) throws SQLException {
return result;
}
- private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule) throws SQLException {
+ private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule, final Map<String, TableMetaData> result) throws SQLException {
Optional<String> actualDefaultDataSourceName = shardingRule.findActualDefaultDataSourceName();
if (!actualDefaultDataSourceName.isPresent()) {
return Collections.emptyMap();
}
Collection<String> tableNames = loadAllTableNames(actualDefaultDataSourceName.get());
- Map<String, TableMetaData> result = new HashMap<>(tableNames.size(), 1);
- for (String each : tableNames) {
- result.put(each, load(each, shardingRule));
+ removeRepeatTable(tableNames, result);
Review comment:
> Why did you remove repeat table? For performance or anything else?
For performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] beijing-penguin commented on issue
#3988: Using multithreading make loadDefaultTables faster and remove repeat
load tableMetaData
Posted by GitBox <gi...@apache.org>.
beijing-penguin commented on issue #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#issuecomment-575500743
/ci
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] beijing-penguin commented on a change in
pull request #3988: Using multithreading make loadDefaultTables faster and
remove repeat load tableMetaData
Posted by GitBox <gi...@apache.org>.
beijing-penguin commented on a change in pull request #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#discussion_r367245631
##########
File path: sharding-core/sharding-core-execute/src/main/java/org/apache/shardingsphere/sharding/execute/metadata/loader/ShardingTableMetaDataLoader.java
##########
@@ -206,19 +207,46 @@ public TableMetas loadAll(final ShardingRule shardingRule) throws SQLException {
return result;
}
- private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule) throws SQLException {
+ private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule, final Map<String, TableMetaData> result) throws SQLException {
Optional<String> actualDefaultDataSourceName = shardingRule.findActualDefaultDataSourceName();
if (!actualDefaultDataSourceName.isPresent()) {
return Collections.emptyMap();
}
Collection<String> tableNames = loadAllTableNames(actualDefaultDataSourceName.get());
- Map<String, TableMetaData> result = new HashMap<>(tableNames.size(), 1);
- for (String each : tableNames) {
- result.put(each, load(each, shardingRule));
+ removeRepeatTable(tableNames, result);
+ List<TableMetaData> metaList = executorEngine.execute(getTableNamesInput(tableNames), new GroupedCallback<String, TableMetaData>() {
+ @Override
+ public Collection<TableMetaData> execute(final Collection<String> inputs, final boolean isTrunkThread, final Map<String, Object> dataMap) throws SQLException {
+ String logicTableName = inputs.iterator().next();
+ Collection<TableMetaData> result = new LinkedList<>();
+ result.add(load(logicTableName, shardingRule));
+ return result;
+ }
+ });
+ Object[] tableNameArr = tableNames.toArray();
+ for (int i = 0, size = tableNames.size(); i < size; i++) {
Review comment:
> Why not `for (int i = 0; i < tableNames.size(); i++) `?
Avoid size() method multiple computations,,,for performance
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero commented on a change in pull
request #3988: Using multithreading make loadDefaultTables faster and
remove repeat load tableMetaData
Posted by GitBox <gi...@apache.org>.
tristaZero commented on a change in pull request #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#discussion_r366844620
##########
File path: sharding-core/sharding-core-execute/src/main/java/org/apache/shardingsphere/sharding/execute/metadata/loader/ShardingTableMetaDataLoader.java
##########
@@ -206,19 +207,46 @@ public TableMetas loadAll(final ShardingRule shardingRule) throws SQLException {
return result;
}
- private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule) throws SQLException {
+ private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule, final Map<String, TableMetaData> result) throws SQLException {
Optional<String> actualDefaultDataSourceName = shardingRule.findActualDefaultDataSourceName();
if (!actualDefaultDataSourceName.isPresent()) {
return Collections.emptyMap();
}
Collection<String> tableNames = loadAllTableNames(actualDefaultDataSourceName.get());
- Map<String, TableMetaData> result = new HashMap<>(tableNames.size(), 1);
- for (String each : tableNames) {
- result.put(each, load(each, shardingRule));
+ removeRepeatTable(tableNames, result);
Review comment:
If there are two tables with the same name `t1` and different structures in ds0 and default ds seperately, do you think removeRepeatTable() could return correct result list?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] coveralls commented on issue #3988:
Using multithreading make loadDefaultTables faster and remove repeat load
tableMetaData
Posted by GitBox <gi...@apache.org>.
coveralls commented on issue #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#issuecomment-574639874
## Pull Request Test Coverage Report for [Build 1381](https://coveralls.io/builds/28123225)
* **0** of **20** **(0.0%)** changed or added relevant lines in **1** file are covered.
* No unchanged relevant lines lost coverage.
* Overall coverage decreased (**-0.06%**) to **65.893%**
---
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [sharding-core/sharding-core-execute/src/main/java/org/apache/shardingsphere/sharding/execute/metadata/loader/ShardingTableMetaDataLoader.java](https://coveralls.io/builds/28123225/source?filename=sharding-core%2Fsharding-core-execute%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fshardingsphere%2Fsharding%2Fexecute%2Fmetadata%2Floader%2FShardingTableMetaDataLoader.java#L198) | 0 | 20 | 0.0%
<!-- | **Total:** | **0** | **20** | **0.0%** | -->
| Totals | [](https://coveralls.io/builds/28123225) |
| :-- | --: |
| Change from base [Build 689](https://coveralls.io/builds/28117346): | -0.06% |
| Covered Lines: | 10815 |
| Relevant Lines: | 16413 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] beijing-penguin commented on a change in
pull request #3988: Using multithreading make loadDefaultTables faster and
remove repeat load tableMetaData
Posted by GitBox <gi...@apache.org>.
beijing-penguin commented on a change in pull request #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#discussion_r367245008
##########
File path: sharding-core/sharding-core-execute/src/main/java/org/apache/shardingsphere/sharding/execute/metadata/loader/ShardingTableMetaDataLoader.java
##########
@@ -206,19 +207,46 @@ public TableMetas loadAll(final ShardingRule shardingRule) throws SQLException {
return result;
}
- private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule) throws SQLException {
+ private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule, final Map<String, TableMetaData> result) throws SQLException {
Optional<String> actualDefaultDataSourceName = shardingRule.findActualDefaultDataSourceName();
if (!actualDefaultDataSourceName.isPresent()) {
return Collections.emptyMap();
}
Collection<String> tableNames = loadAllTableNames(actualDefaultDataSourceName.get());
- Map<String, TableMetaData> result = new HashMap<>(tableNames.size(), 1);
- for (String each : tableNames) {
- result.put(each, load(each, shardingRule));
+ removeRepeatTable(tableNames, result);
Review comment:
> If there are two tables with the same name `t1` and different structures in ds0 and default ds seperately, do you think removeRepeatTable() could return correct result list?
yes, load() methed will be get out the same as tableMetaData by table name...loadShardingTables or loadDefaultTables execute load() method and only first parameter table name is different.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero commented on issue #3988:
Using multithreading make loadDefaultTables faster and remove repeat load
tableMetaData
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#issuecomment-574626422
/ci
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] coveralls edited a comment on issue
#3988: Using multithreading make loadDefaultTables faster
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on issue #3988: Using multithreading make loadDefaultTables faster
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#issuecomment-574639874
## Pull Request Test Coverage Report for [Build 1402](https://coveralls.io/builds/28168639)
* **0** of **13** **(0.0%)** changed or added relevant lines in **1** file are covered.
* **45** unchanged lines in **1** file lost coverage.
* Overall coverage decreased (**-0.04%**) to **65.911%**
---
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [sharding-core/sharding-core-execute/src/main/java/org/apache/shardingsphere/sharding/execute/metadata/loader/ShardingTableMetaDataLoader.java](https://coveralls.io/builds/28168639/source?filename=sharding-core%2Fsharding-core-execute%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fshardingsphere%2Fsharding%2Fexecute%2Fmetadata%2Floader%2FShardingTableMetaDataLoader.java#L217) | 0 | 13 | 0.0%
<!-- | **Total:** | **0** | **13** | **0.0%** | -->
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [sharding-core/sharding-core-execute/src/main/java/org/apache/shardingsphere/sharding/execute/metadata/loader/ShardingTableMetaDataLoader.java](https://coveralls.io/builds/28168639/source?filename=sharding-core%2Fsharding-core-execute%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fshardingsphere%2Fsharding%2Fexecute%2Fmetadata%2Floader%2FShardingTableMetaDataLoader.java#L62) | 45 | 0.0% |
<!-- | **Total:** | **45** | | -->
| Totals | [](https://coveralls.io/builds/28168639) |
| :-- | --: |
| Change from base [Build 693](https://coveralls.io/builds/28167793): | -0.04% |
| Covered Lines: | 10816 |
| Relevant Lines: | 16410 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero commented on issue #3988:
Using multithreading make loadDefaultTables faster and remove repeat load
tableMetaData
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#issuecomment-575439676
/ci
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] SteNicholas commented on a change in
pull request #3988: Using multithreading make loadDefaultTables faster and
remove repeat load tableMetaData
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on a change in pull request #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#discussion_r367220680
##########
File path: sharding-core/sharding-core-execute/src/main/java/org/apache/shardingsphere/sharding/execute/metadata/loader/ShardingTableMetaDataLoader.java
##########
@@ -206,19 +207,46 @@ public TableMetas loadAll(final ShardingRule shardingRule) throws SQLException {
return result;
}
- private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule) throws SQLException {
+ private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule, final Map<String, TableMetaData> result) throws SQLException {
Optional<String> actualDefaultDataSourceName = shardingRule.findActualDefaultDataSourceName();
if (!actualDefaultDataSourceName.isPresent()) {
return Collections.emptyMap();
}
Collection<String> tableNames = loadAllTableNames(actualDefaultDataSourceName.get());
- Map<String, TableMetaData> result = new HashMap<>(tableNames.size(), 1);
- for (String each : tableNames) {
- result.put(each, load(each, shardingRule));
+ removeRepeatTable(tableNames, result);
Review comment:
Why did you remove repeat table? For performance or anything else?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] beijing-penguin commented on issue
#3988: Using multithreading make loadDefaultTables faster and remove repeat
load tableMetaData
Posted by GitBox <gi...@apache.org>.
beijing-penguin commented on issue #3988: Using multithreading make loadDefaultTables faster and remove repeat load tableMetaData
URL: https://github.com/apache/incubator-shardingsphere/pull/3988#issuecomment-575514607
/run ci
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services