You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2021/06/23 08:06:01 UTC
[GitHub] [shardingsphere] childewuque opened a new issue #10938: about init metadata of tables
childewuque opened a new issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938
shardingsphere jdbc 5
springboot2.3
I have a project it uses only one logic table，but there are 10000 other tables in the same db for other projects
When my project starting，shardingsphere jdbc will init all tables' metadata,then the process seem like "sleep"
Can I exclude other tables in my sharding project?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tuichenchuxin commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
tuichenchuxin commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-989582434
@childewuque Thank you for your feedback, I'll check it later. And are you willing to submit a pr to optimize this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque edited a comment on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque edited a comment on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-867276151
> The version 5.0.0-beta has already resolved this problem.
Thanks
I try the version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tuichenchuxin commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
tuichenchuxin commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-988633812
@childewuque
> 3.The slowest point is the line in SingleTableMetaDataBuilder.load() --line 51：
Collection tableMetaDataLoaderMaterials = TableMetaDataUtil.getTableMetaDataLoadMaterial(needLoadTables, materials, false);
This line is all memory operations, It shouldn't be so slow. I'll check it.
>Now my solution is that all tables are configed at actualDataNodes for exclude ,but it is toooooo long.
This way may not work. Is it worked?
>lazy load may be a good idea, we'll consider it, but for now, may not skip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-988531945
@tuichenchuxin @strongduanmu
I now use v5.0.0, and I find that the range of excluded tables is smaller than v5.0.0-beta, v5.0.0 only excludes getAllActualTables
Therefore, 5.0.0 initialized more tables metadata and the project started more slowly
In a database with many tables, only one table is used for sharding, and the other tables are not used by shardingsphere at all. How can the project be started more quickly?
Thanks.
SingleTableRule class:
private Collection<String> getExcludedTables(final Collection<ShardingSphereRule> rules) {
return rules.stream().filter(each -> each instanceof DataNodeContainedRule).flatMap(each -> ((DataNodeContainedRule) each).getAllTables().stream()).collect(Collectors.toSet());
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] terrymanu commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-866748295
What is the actual version, 5 is never released yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-927422077
Performance optimization has been completed, welcome to use the master branch test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-866786003
> What is the actual version? 5 is never released yet
Master，downloaded on May 20
I find the method in ShardingRule.java:
private Collection<String> getExcludedTables() {
Collection<String> result = new HashSet<>(getTables());
result.addAll(getAllActualTables());
result.addAll(broadcastTables);
return result;
}
So I add all tables at actual-data-nodes for excluding other tables.
sample code:
tables:
sharding_product:
actual-data-nodes: sdb_$->{1..4}.fads_product_$->{['0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f']}$->{['0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f']},sdb_$->{1..4}.shop_product_$->{['0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f']}$->{['0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f']}
............
very long...
Do you have more better suggestions for the case:
Only one table to shard in my project but there are too many other tables in the same db.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-867317240
I test and debug the project by configging one logic table at actual-data-nodes now,the 5.0.0-beta version problem still exists.
FederateSchemaMetadata.java
private void initTables(ShardingSphereMetaData metaData) throws SQLException {
Collection<DataNodeContainedRule> dataNodeRules = this.getDataNodeContainedRules(metaData);
Map<String, Collection<DataNode>> tableDataNodes = this.getTableDataNodes(dataNodeRules);
Map<String, Collection<String>> dataSourceRules = this.getDataSourceRules(metaData);
Iterator var5 = tableDataNodes.entrySet().iterator(); // blocked here
more ideas?
And may Shardingsphere-jdbc only init logic tables and broadcast?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-867276151
> The version 5.0.0-beta has already resolved this problem.
Thanks
I try the version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tuichenchuxin commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
tuichenchuxin commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-992052782
@childewuque we have optimized multi tables load.
14000 singleTables load from 140s To 17s.
May be you can try again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque edited a comment on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque edited a comment on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-989574848
>
It seems there are some loops on tableNames/singleTableDataNodes , and tableNames or singleTableDataNodes has 24k elements, so it's slow.
loop points,E.g :
org.apache.shardingsphere.infra.metadata.schema.builder.util.TableMetaDataUtil line 51
org.apache.shardingsphere.infra.datanode.DataNodes line 55
I exclude all tables at actualDataNodes then the project starts normally and works fine.
I don't know why shardingsphere will exclude the tables at actualDataNodes, but my project only use one logic table,so all tabls of the db are configured at actualDataNodes, it seems no problem in theory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-988615412
> @childewuque Can you help to position the code which is slow?
Thanks for the quick response!
1.There are about 6000 tables in every DB. But only one logic table named prod_score is sharded in my project.
2.actualDataNodes: dataSource$->{1..4}.prod_score_$->{(Integer.toHexString(0)..Integer.toHexString(255)).collect{t ->t.padLeft(2,'0')}}
3.The slowest point is the line in SingleTableMetaDataBuilder.load() --line 51：
Collection<TableMetaDataLoaderMaterial> tableMetaDataLoaderMaterials = TableMetaDataUtil.getTableMetaDataLoadMaterial(needLoadTables, materials, false);
Because there are about 24000 tables in Collection<String> needLoadTables.
Now my solution is that all tables are configed at actualDataNodes for exclude ,but it is toooooo long.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] terrymanu commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-866802051
The version 5.0.0-beta has already resolve this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque edited a comment on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque edited a comment on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-989574848
>
It seems there are some loops on tableNames/singleTableDataNodes , and tableNames or singleTableDataNodes has 24k elements, so it's slow.
loop points,E.g :
org.apache.shardingsphere.infra.metadata.schema.builder.util.TableMetaDataUtil line 51
org.apache.shardingsphere.infra.datanode.DataNodes line 55
I exclude all tables at actualDataNodes then the project starts normally and works fine.
I don't know why shardingsphere will exclude initializing metadata of the tables at actualDataNodes, but my project only use one logic table,so all tables of the db are configured at actualDataNodes, it seems no problem in theory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque edited a comment on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque edited a comment on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-989574848
>
It seems there are some loops on tableNames/singleTableDataNodes , and tableNames or singleTableDataNodes has 24k elements, so it's slow.
loop point,E.g :
org.apache.shardingsphere.infra.metadata.schema.builder.util.TableMetaDataUtil line 51
org.apache.shardingsphere.infra.datanode.DataNodes line 55
I exclude all tables at actualDataNodes then the project starts normally and works fine.
I don't know why shardingsphere will exclude the tables at actualDataNodes, but my project only use one logic table,so all tabls of the db are configured at actualDataNodes, it seems no problem in theory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-989654451
Thanks!
I want but I don't know the global purpose of all tables metadata.
So just for my case,if I can ,Maybe I will add a configuration item to decide initializing tables metadata.
haha
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-868457595
Another project has same problem,it uses 20 db ,every db has about 5000 tables.
Only 1 logic tables (256*20 actual tables) are used in the project.
Sometimes the project can start in 5~10 minutes,sometimes it is blocked.
max-connections-size-per-query=1000 now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque edited a comment on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque edited a comment on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-868457595
Another project has same problem,it uses 20 db ,every db has about 5000 tables.
Only 1 logic table (256*20 actual tables) is used in the project.
Sometimes the project can start in 5~10 minutes,sometimes it is blocked.
max-connections-size-per-query=1000 now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-989574848
>
It seems there are some loops on tableNames/singleTableDataNodes , and tableNames or singleTableDataNodes has 24k elements, so it's slow.
E.g :
org.apache.shardingsphere.infra.metadata.schema.builder.util.TableMetaDataUtil line 51
org.apache.shardingsphere.infra.datanode.DataNodes line 55
I exclude all tables at actualDataNodes then the project starts normally and works fine.
I don't know why shardingsphere will exclude the tables at actualDataNodes, but my project only use one logic table,so all tabls of the db are configured at actualDataNodes, it seems no problem in theory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] qiancheng8866 commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
qiancheng8866 commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-871113402
same question,up+1!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu closed issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
strongduanmu closed issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-988618703
@tuichenchuxin I don't know if it is possible to initialize only the sharding tables ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque edited a comment on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque edited a comment on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-988618703
@tuichenchuxin I don't know if it is possible to initialize only the tables that must be initialized or let tables be initialized lazy.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tuichenchuxin commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
tuichenchuxin commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-927309287
@childewuque @qiancheng8866 I had improved the performance about loading table meta data on master branch. Welcome to try to test master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] terrymanu edited a comment on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
terrymanu edited a comment on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-866802051
The version 5.0.0-beta has already resolved this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] terrymanu edited a comment on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
terrymanu edited a comment on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-866748295
What is the actual version? 5 is never released yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque edited a comment on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque edited a comment on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-989654451
Thanks!
I want but I don't know the global purpose of all tables metadata.
So just for my case,if I can ,Maybe I will add a configuration item to decide whether to initializing tables metadata.
haha
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tuichenchuxin commented on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
tuichenchuxin commented on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-988534442
@childewuque Can you help to position the code which is slow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] childewuque edited a comment on issue #10938: about init metadata of tables
Posted by GitBox <gi...@apache.org>.
childewuque edited a comment on issue #10938:
URL: https://github.com/apache/shardingsphere/issues/10938#issuecomment-989574848
>
It seems there are some loops on tableNames/singleTableDataNodes , and tableNames or singleTableDataNodes has 24k elements, so it's slow.
loop points,E.g :
org.apache.shardingsphere.infra.metadata.schema.builder.util.TableMetaDataUtil line 51
org.apache.shardingsphere.infra.datanode.DataNodes line 55
I exclude all tables at actualDataNodes then the project starts normally and works fine.
I don't know why shardingsphere will exclude initializing the tables metadata at actualDataNodes, but my project only use one logic table,so all tabls of the db are configured at actualDataNodes, it seems no problem in theory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org