You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2020/03/05 03:42:26 UTC
[GitHub] [incubator-shardingsphere] dalong316 opened a new issue #4604: Too
many tables loaded slowly
dalong316 opened a new issue #4604: Too many tables loaded slowly
URL: https://github.com/apache/incubator-shardingsphere/issues/4604
At present, I have more than 3000 sub tables. It takes too long to load sharding at the start of the project. I have set the consistency check metadata to "false" and it takes effect, but the time is still very long. I have located the code part that takes a long time
private Map<String, TableMetaData> loadDefaultTables(final ShardingRule shardingRule) throws SQLException {
Map<String, TableMetaData> result = new HashMap<>(shardingRule.getTableRules().size(), 1);
Optional<String> actualDefaultDataSourceName = shardingRule.findActualDefaultDataSourceName();
if (actualDefaultDataSourceName.isPresent()) {
for (String each : getAllTableNames(actualDefaultDataSourceName.get())) {
result.put(each, tableMetaDataLoader.load(each, shardingRule));
}
}
return result;
}
tableMetaDataLoader.load(each, shardingRule)
*This part is the main reason
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero edited a comment on issue
#4604: Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-595590905
@dalong316 Hi which version is your adoption?
What you mentioned has optimized in #4091, could you compile our master branch to give a test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-596077706
> How many threads are open to use multithreading
> @tristaZero
`max.connections.size.per.query` decides how many connections SS can use to execute SQL on one DB which represents the number of multiple threads.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] dalong316 commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
dalong316 commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-595680511
I've compiled it several times and reported an error. When can I release the official version when I fix this problem
@tristaZero
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] dalong316 commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
dalong316 commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-596051019
@kimmking
Can you give me your email address or WeChat?
Because there are a lot of things, I ca n’t write in it
This is my WeChat: 18811452212
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] codingIsCrazy commented on issue #4604:
Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
codingIsCrazy commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-597997196
4.0.1 is ok?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] dalong316 commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
dalong316 commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-597494990
Improve the loading efficiency by modifying the configuration of max.connections.size.per.query,
I now have 1100+ watches, 10 threads, about 120 seconds
100 threads about 60 seconds
But need to consider the situation of different connection pools
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-596351574
@dalong316
How about your test report?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-595590905
@dalong316 Hi which version is your adoption?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [shardingsphere] tristaZero commented on issue #4604: Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4604:
URL: https://github.com/apache/shardingsphere/issues/4604#issuecomment-748779630
Hi,
> Cause: java.lang.ClassCastException: org.apache.shardingsphere.sharding.api.config.strategy.sharding.HintShardingStrategyConfiguration cannot be cast to org.apache.shardingsphere.sharding.api.config.strategy.sharding.StandardShardingStrategyConfiguration"
The error log shows obviously. Plus, [Examples](https://github.com/apache/shardingsphere/tree/5.0.0-alpha/examples) is a great guide.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-shardingsphere] tristaZero commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-596077331
> I've compiled it several times and reported an error. When can I release the official version when I fix this problem
> @tristaZero
What's the exception?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] kimmking commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
kimmking commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-596039877
@dalong316 can you send me a 3000 table-ddl and sharding-rule, my wx is kimmking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [shardingsphere] pototallxg commented on issue #4604: Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
pototallxg commented on issue #4604:
URL: https://github.com/apache/shardingsphere/issues/4604#issuecomment-747824272
public DataSource shardingDataSource(Map<String, DataSource> dataSourceMap) throws SQLException {
ShardingRuleConfiguration shardingRuleConfig = new ShardingRuleConfiguration();
Collection<ShardingTableRuleConfiguration> tables = new ArrayList<>();
tables.add(new ShardingTableRuleConfiguration("HSale_Key", "cunfang-slave0.HSale_Key"));
tables.add(new ShardingTableRuleConfiguration("HSale_ShiKan", "cunfang-slave0.HSale_ShiKan"));
ShardingStrategyConfiguration dbShardingConfig = new HintShardingStrategyConfiguration("dbShardingAlgorithm");
Properties dbShardingAlgorithmrProps = new Properties();
dbShardingAlgorithmrProps.setProperty("algorithm-class-name", "com.cunfang.house.dbengine.shardingconfig.dbsharding.DataSourceRoutingAlgorithm");
shardingRuleConfig.getShardingAlgorithms().put("dbShardingAlgorithm", new ShardingSphereAlgorithmConfiguration("INLINE", dbShardingAlgorithmrProps));
ShardingStrategyConfiguration tableShardingConfig = new HintShardingStrategyConfiguration("tableShardingAlgorithm");
Properties tableShardingAlgorithmrProps = new Properties();
tableShardingAlgorithmrProps.setProperty("algorithm-class-name", "com.cunfang.house.dbengine.shardingconfig.tablesharding.TableKeyHintShardingAlgorithm");
shardingRuleConfig.getShardingAlgorithms().put("dbShardingAlgorithm", new ShardingSphereAlgorithmConfiguration("INLINE", tableShardingAlgorithmrProps));
shardingRuleConfig.setDefaultDatabaseShardingStrategy(dbShardingConfig);
shardingRuleConfig.setDefaultTableShardingStrategy(tableShardingConfig);
shardingRuleConfig.setTables(tables);
return ShardingSphereDataSourceFactory.createDataSource(dataSourceMap, Collections.singleton(shardingRuleConfig), new Properties());
}
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-shardingsphere] tristaZero edited a comment on issue
#4604: Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-595590905
@dalong316 Hi which version is your adoption?
What you mentioned has optimized in #4091, could you compile our dev branch to give a test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-597655642
@dalong316 Thanks for your report.
Here is the summary to help speed up the metadata loading when service startups.
1. Update to 4.0.1 above, which helps speed up the process of loading table metadata from `the default dataSource`
2. Configure `max.connections.size.per.query`(Default value is 1) higher referring to connection pool you adopt.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] dalong316 commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
dalong316 commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-595621406
How many threads are open to use multithreading
@tristaZero
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [shardingsphere] YX-GitH commented on issue #4604: Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
YX-GitH commented on issue #4604:
URL: https://github.com/apache/shardingsphere/issues/4604#issuecomment-738725996
hi , I use 4.1.1 but it do not work， it also slow to load meta data when start up，Is there any other solution？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-shardingsphere] tristaZero commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-595583151
@dalong316
Hi much feedback from community concerning this issue make us give priority to this one.
After it finishes, could you give us a hand to test it better or not?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] kimmking commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
kimmking commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-596356649
@dalong316 aliyun maven mirror is good choice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] dalong316 commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
dalong316 commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-596178032
I compiled the master with ./mvnw clean install -Prelease, and then introduced the jar package into the project. As a result, the project could not find the corresponding class.
Later, I introduced the source code to my project and found that the situation is the same, and there are many external jar packages that cannot be referenced, such as some jars from Google, and many similar cases, basically the jar packages cannot be referenced.
@tristaZero
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [shardingsphere] pototallxg commented on issue #4604: Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
pototallxg commented on issue #4604:
URL: https://github.com/apache/shardingsphere/issues/4604#issuecomment-750711949
i find the prop 'check-table-metadata-enabled =false'，not work in lowwer DB version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tristaZero commented on issue #4604: Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4604:
URL: https://github.com/apache/shardingsphere/issues/4604#issuecomment-739721400
Hi @YX-GitH
If possible, could you try the latest version 5.0.0-alpha, which carries a significant improvement on metadata loading?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-shardingsphere] dalong316 commented on issue #4604: Too
slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
dalong316 commented on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-595620973
Okay, I'll test it now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] tristaZero closed issue #4604: Too slow
to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero closed issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [shardingsphere] pototallxg commented on issue #4604: Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
pototallxg commented on issue #4604:
URL: https://github.com/apache/shardingsphere/issues/4604#issuecomment-747824067
i have the same problem,switch 5.0, error:
nested exception is org.apache.ibatis.exceptions.PersistenceException: \r\n### Error updating database. Cause: java.lang.ClassCastException: org.apache.shardingsphere.sharding.api.config.strategy.sharding.HintShardingStrategyConfiguration cannot be cast to org.apache.shardingsphere.sharding.api.config.strategy.sharding.StandardShardingStrategyConfiguration\r\n### The error may exist in com/cunfang/house/mapper/maintenance/HSaleShiKanMapper.java (best guess)\r\n### The error may involve com.cunfang.house.mapper.maintenance.HSaleShiKanMapper.insert-Inline\r\n### The error occurred while setting parameters\r\n### SQL: INSERT INTO HSale_ShiKan ( shiKanTime, titleUrl, vrId, vrTitleUrl, city, houseId, insertBid, insertUserName, insertTime, status ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )\r\n### Cause: java.lang.ClassCastException: org.apache.shardingsphere.sharding.api.config.strategy.sharding.HintShardingStrategyConfiguration cannot be cast to org.apache.shardingsphere.sharding
.api.config.strategy.sharding.StandardShardingStrategyConfiguration",
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tristaZero commented on issue #4604: Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4604:
URL: https://github.com/apache/shardingsphere/issues/4604#issuecomment-750726278
> 'check-table-metadata-enabled =false' not work in lowwer DB version.
This property is in the range of `ShardingSphere`, so it definitely takes no effect on Databases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-shardingsphere] tristaZero edited a comment on issue
#4604: Too slow to load meta data when start up
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4604: Too slow to load meta data when start up
URL: https://github.com/apache/incubator-shardingsphere/issues/4604#issuecomment-597655642
@dalong316 Thanks for your report.
Here is the summary to help speed up the metadata loading when service startups.
### How to speed up the metadata loading when service starts up?
1. Update to 4.0.1 above, which helps speed up the process of loading table metadata from `the default dataSource`
2. Configure `max.connections.size.per.query`(Default value is 1) higher referring to connection pool you adopt.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services