You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2022/05/01 08:52:03 UTC
[GitHub] [shardingsphere] cheese8 opened a new pull request, #17251: OracleSchemaMetaDataLoader.load tables batchly
cheese8 opened a new pull request, #17251:
URL: https://github.com/apache/shardingsphere/pull/17251
Fixes #17250.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on a diff in pull request #17251: OracleSchemaMetaDataLoader.load tables batchly
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on code in PR #17251:
URL: https://github.com/apache/shardingsphere/pull/17251#discussion_r862542187
##########
shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/loader/dialect/OracleSchemaMetaDataLoader.java:
##########
@@ -66,17 +68,32 @@ public final class OracleSchemaMetaDataLoader implements DialectSchemaMetaDataLo
private static final int IDENTITY_COLUMN_START_MINOR_VERSION = 1;
+ private static final int BATCH_SIZE = 1000;
Review Comment:
@cheese8 Can we run some tests to determine which batch size perform best for metadata load?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter commented on pull request #17251: OracleSchemaMetaDataLoader.load tables batchly
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #17251:
URL: https://github.com/apache/shardingsphere/pull/17251#issuecomment-1114210427
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/17251?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#17251](https://codecov.io/gh/apache/shardingsphere/pull/17251?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fc21a6c) into [master](https://codecov.io/gh/apache/shardingsphere/commit/01a9c9e069db5ab7a82d30b2cb06f2ace6053c90?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01a9c9e) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #17251 +/- ##
=========================================
Coverage 60.00% 60.00%
- Complexity 2116 2117 +1
=========================================
Files 3527 3528 +1
Lines 52375 52387 +12
Branches 8898 8900 +2
=========================================
+ Hits 31425 31433 +8
- Misses 18320 18324 +4
Partials 2630 2630
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/17251?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ema/loader/dialect/OracleSchemaMetaDataLoader.java](https://codecov.io/gh/apache/shardingsphere/pull/17251/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9zY2hlbWEvbG9hZGVyL2RpYWxlY3QvT3JhY2xlU2NoZW1hTWV0YURhdGFMb2FkZXIuamF2YQ==) | `89.41% <100.00%> (+1.10%)` | :arrow_up: |
| [...ry/cache/subscriber/ScalingRegistrySubscriber.java](https://codecov.io/gh/apache/shardingsphere/pull/17251/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtbW9kZS9zaGFyZGluZ3NwaGVyZS1tb2RlLXR5cGUvc2hhcmRpbmdzcGhlcmUtY2x1c3Rlci1tb2RlL3NoYXJkaW5nc3BoZXJlLWNsdXN0ZXItbW9kZS1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9tb2RlL21hbmFnZXIvY2x1c3Rlci9jb29yZGluYXRvci9yZWdpc3RyeS9jYWNoZS9zdWJzY3JpYmVyL1NjYWxpbmdSZWdpc3RyeVN1YnNjcmliZXIuamF2YQ==) | `24.00% <0.00%> (-1.00%)` | :arrow_down: |
| [...ate/CreateShardingScalingRuleStatementUpdater.java](https://codecov.io/gh/apache/shardingsphere/pull/17251/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmcvc2hhcmRpbmdzcGhlcmUtc2hhcmRpbmctZGlzdHNxbC9zaGFyZGluZ3NwaGVyZS1zaGFyZGluZy1kaXN0c3FsLWhhbmRsZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3NjYWxpbmcvZGlzdHNxbC9oYW5kbGVyL3VwZGF0ZS9DcmVhdGVTaGFyZGluZ1NjYWxpbmdSdWxlU3RhdGVtZW50VXBkYXRlci5qYXZh) | `91.80% <0.00%> (-0.14%)` | :arrow_down: |
| [...apache/shardingsphere/infra/datanode/DataNode.java](https://codecov.io/gh/apache/shardingsphere/pull/17251/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9kYXRhbm9kZS9EYXRhTm9kZS5qYXZh) | `69.56% <0.00%> (ø)` | |
| [...eline/scenario/rulealtered/RuleAlteredContext.java](https://codecov.io/gh/apache/shardingsphere/pull/17251/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUta2VybmVsL3NoYXJkaW5nc3BoZXJlLWRhdGEtcGlwZWxpbmUvc2hhcmRpbmdzcGhlcmUtZGF0YS1waXBlbGluZS1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9kYXRhL3BpcGVsaW5lL3NjZW5hcmlvL3J1bGVhbHRlcmVkL1J1bGVBbHRlcmVkQ29udGV4dC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ne/spi/ratelimit/JobRateLimitAlgorithmFactory.java](https://codecov.io/gh/apache/shardingsphere/pull/17251/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUta2VybmVsL3NoYXJkaW5nc3BoZXJlLWRhdGEtcGlwZWxpbmUvc2hhcmRpbmdzcGhlcmUtZGF0YS1waXBlbGluZS1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RhdGEvcGlwZWxpbmUvc3BpL3JhdGVsaW1pdC9Kb2JSYXRlTGltaXRBbGdvcml0aG1GYWN0b3J5LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...jdbc/core/connection/ShardingSphereConnection.java](https://codecov.io/gh/apache/shardingsphere/pull/17251/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtamRiYy9zaGFyZGluZ3NwaGVyZS1qZGJjLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RyaXZlci9qZGJjL2NvcmUvY29ubmVjdGlvbi9TaGFyZGluZ1NwaGVyZUNvbm5lY3Rpb24uamF2YQ==) | `71.95% <0.00%> (+0.34%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/17251?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/17251?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [01a9c9e...fc21a6c](https://codecov.io/gh/apache/shardingsphere/pull/17251?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] cheese8 commented on a diff in pull request #17251: Fix ORA-01795 while OracleSchemaMetaDataLoader.load tables count greater than 1000
Posted by GitBox <gi...@apache.org>.
cheese8 commented on code in PR #17251:
URL: https://github.com/apache/shardingsphere/pull/17251#discussion_r863309163
##########
shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/loader/dialect/OracleSchemaMetaDataLoader.java:
##########
@@ -66,20 +69,28 @@ public final class OracleSchemaMetaDataLoader implements DialectSchemaMetaDataLo
private static final int IDENTITY_COLUMN_START_MINOR_VERSION = 1;
+ private static final int BATCH_SIZE = 1000;
+
@Override
public Collection<SchemaMetaData> load(final DataSource dataSource, final Collection<String> tables, final String defaultSchemaName) throws SQLException {
Map<String, TableMetaData> tableMetaDataMap = new LinkedHashMap<>();
- Map<String, Collection<ColumnMetaData>> columnMetaDataMap = loadColumnMetaDataMap(dataSource, tables);
+ Map<String, Collection<ColumnMetaData>> columnMetaDataMap = new HashMap<>(tables.size(), 0.75f);
Review Comment:
Not thinking of it, default 0.75 used. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] cheese8 commented on a diff in pull request #17251: OracleSchemaMetaDataLoader.load tables batchly
Posted by GitBox <gi...@apache.org>.
cheese8 commented on code in PR #17251:
URL: https://github.com/apache/shardingsphere/pull/17251#discussion_r862558224
##########
shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/loader/dialect/OracleSchemaMetaDataLoader.java:
##########
@@ -66,17 +68,32 @@ public final class OracleSchemaMetaDataLoader implements DialectSchemaMetaDataLo
private static final int IDENTITY_COLUMN_START_MINOR_VERSION = 1;
+ private static final int BATCH_SIZE = 1000;
Review Comment:
The max what oracle supported is 1000 which ORA-01795 shows, and I think the less database queried the best performance get.
And `performance optimized for get connection` PRed to reuse the connection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu merged pull request #17251: Fix ORA-01795 while OracleSchemaMetaDataLoader.load tables count greater than 1000
Posted by GitBox <gi...@apache.org>.
strongduanmu merged PR #17251:
URL: https://github.com/apache/shardingsphere/pull/17251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on a diff in pull request #17251: Fix ORA-01795 while OracleSchemaMetaDataLoader.load tables count greater than 1000
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on code in PR #17251:
URL: https://github.com/apache/shardingsphere/pull/17251#discussion_r862910546
##########
shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/loader/dialect/OracleSchemaMetaDataLoader.java:
##########
@@ -66,20 +68,37 @@ public final class OracleSchemaMetaDataLoader implements DialectSchemaMetaDataLo
private static final int IDENTITY_COLUMN_START_MINOR_VERSION = 1;
+ private static final int BATCH_SIZE = 1000;
+
@Override
public Collection<SchemaMetaData> load(final DataSource dataSource, final Collection<String> tables, final String defaultSchemaName) throws SQLException {
Map<String, TableMetaData> tableMetaDataMap = new LinkedHashMap<>();
- Map<String, Collection<ColumnMetaData>> columnMetaDataMap = loadColumnMetaDataMap(dataSource, tables);
+ Map<String, Collection<ColumnMetaData>> columnMetaDataMap = new HashMap<>(tables.size());
+ Collection[] splitTables = splitTables(new ArrayList(tables), BATCH_SIZE);
+ try (Connection connection = dataSource.getConnection()) {
+ for (Collection<String> subTables : splitTables) {
Review Comment:
Please rename subTables to each.
##########
shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/loader/dialect/OracleSchemaMetaDataLoader.java:
##########
@@ -66,20 +68,37 @@ public final class OracleSchemaMetaDataLoader implements DialectSchemaMetaDataLo
private static final int IDENTITY_COLUMN_START_MINOR_VERSION = 1;
+ private static final int BATCH_SIZE = 1000;
+
@Override
public Collection<SchemaMetaData> load(final DataSource dataSource, final Collection<String> tables, final String defaultSchemaName) throws SQLException {
Map<String, TableMetaData> tableMetaDataMap = new LinkedHashMap<>();
- Map<String, Collection<ColumnMetaData>> columnMetaDataMap = loadColumnMetaDataMap(dataSource, tables);
+ Map<String, Collection<ColumnMetaData>> columnMetaDataMap = new HashMap<>(tables.size());
+ Collection[] splitTables = splitTables(new ArrayList(tables), BATCH_SIZE);
Review Comment:
@cheese8 Can we use guava `Lists.partition` to replace this method?
##########
shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/loader/dialect/OracleSchemaMetaDataLoader.java:
##########
@@ -66,20 +68,37 @@ public final class OracleSchemaMetaDataLoader implements DialectSchemaMetaDataLo
private static final int IDENTITY_COLUMN_START_MINOR_VERSION = 1;
+ private static final int BATCH_SIZE = 1000;
+
@Override
public Collection<SchemaMetaData> load(final DataSource dataSource, final Collection<String> tables, final String defaultSchemaName) throws SQLException {
Map<String, TableMetaData> tableMetaDataMap = new LinkedHashMap<>();
- Map<String, Collection<ColumnMetaData>> columnMetaDataMap = loadColumnMetaDataMap(dataSource, tables);
+ Map<String, Collection<ColumnMetaData>> columnMetaDataMap = new HashMap<>(tables.size());
Review Comment:
@cheese8 Please add load factor when using new HashMap().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on a diff in pull request #17251: Fix ORA-01795 while OracleSchemaMetaDataLoader.load tables count greater than 1000
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on code in PR #17251:
URL: https://github.com/apache/shardingsphere/pull/17251#discussion_r863271601
##########
shardingsphere-infra/shardingsphere-infra-common/src/main/java/org/apache/shardingsphere/infra/metadata/schema/loader/dialect/OracleSchemaMetaDataLoader.java:
##########
@@ -66,20 +69,28 @@ public final class OracleSchemaMetaDataLoader implements DialectSchemaMetaDataLo
private static final int IDENTITY_COLUMN_START_MINOR_VERSION = 1;
+ private static final int BATCH_SIZE = 1000;
+
@Override
public Collection<SchemaMetaData> load(final DataSource dataSource, final Collection<String> tables, final String defaultSchemaName) throws SQLException {
Map<String, TableMetaData> tableMetaDataMap = new LinkedHashMap<>();
- Map<String, Collection<ColumnMetaData>> columnMetaDataMap = loadColumnMetaDataMap(dataSource, tables);
+ Map<String, Collection<ColumnMetaData>> columnMetaDataMap = new HashMap<>(tables.size(), 0.75f);
Review Comment:
Why set load factor to 0.75? Not 1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org