You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2022/07/19 11:40:16 UTC
[GitHub] [shardingsphere] azexcy opened a new pull request, #19361: Fixed openGauss scaling IT problems
azexcy opened a new pull request, #19361:
URL: https://github.com/apache/shardingsphere/pull/19361
Changes proposed in this pull request:
- Fix the problems at openGauss scaling IT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on a diff in pull request #19361: Fixed openGauss scaling IT problems
Posted by GitBox <gi...@apache.org>.
azexcy commented on code in PR #19361:
URL: https://github.com/apache/shardingsphere/pull/19361#discussion_r925340895
##########
shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-scaling/src/test/java/org/apache/shardingsphere/integration/data/pipeline/framework/container/proxy/ShardingSphereProxyDockerContainer.java:
##########
@@ -44,9 +44,9 @@ public ShardingSphereProxyDockerContainer(final DatabaseType databaseType) {
protected void configure() {
withExposedPorts(3307);
mapConfigurationFiles();
- if (DatabaseTypeUtil.isPostgreSQL(databaseType) || DatabaseTypeUtil.isOpenGauss(databaseType)) {
+ if (DatabaseTypeUtil.isPostgreSQL(databaseType)) {
setWaitStrategy(new JDBCConnectionWaitStrategy(() -> DriverManager.getConnection(DataSourceEnvironment.getURL(databaseType, getHost(), getMappedPort(3307), "postgres"), "root", "root")));
Review Comment:
Previously it was to ensure that the proxy was ready to be connected, but now I test find Neither is needed anymore， so I remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on a diff in pull request #19361: Fixed openGauss scaling IT problems
Posted by GitBox <gi...@apache.org>.
azexcy commented on code in PR #19361:
URL: https://github.com/apache/shardingsphere/pull/19361#discussion_r925153216
##########
shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-scaling/src/test/java/org/apache/shardingsphere/integration/data/pipeline/framework/container/proxy/ShardingSphereProxyDockerContainer.java:
##########
@@ -44,9 +44,9 @@ public ShardingSphereProxyDockerContainer(final DatabaseType databaseType) {
protected void configure() {
withExposedPorts(3307);
mapConfigurationFiles();
- if (DatabaseTypeUtil.isPostgreSQL(databaseType) || DatabaseTypeUtil.isOpenGauss(databaseType)) {
+ if (DatabaseTypeUtil.isPostgreSQL(databaseType)) {
setWaitStrategy(new JDBCConnectionWaitStrategy(() -> DriverManager.getConnection(DataSourceEnvironment.getURL(databaseType, getHost(), getMappedPort(3307), "postgres"), "root", "root")));
Review Comment:
Yes, is not config setWaitStrategy, will use default wait strategy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter commented on pull request #19361: Fixed openGauss scaling IT problems
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #19361:
URL: https://github.com/apache/shardingsphere/pull/19361#issuecomment-1189779353
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/19361?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#19361](https://codecov.io/gh/apache/shardingsphere/pull/19361?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (27e46af) into [master](https://codecov.io/gh/apache/shardingsphere/commit/3620a24aa52afff115eae765020fb51be556a46e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3620a24) will **increase** coverage by `0.03%`.
> The diff coverage is `16.66%`.
```diff
@@ Coverage Diff @@
## master #19361 +/- ##
============================================
+ Coverage 59.63% 59.66% +0.03%
Complexity 2373 2373
============================================
Files 3818 3818
Lines 54367 54372 +5
Branches 7628 7630 +2
============================================
+ Hits 32420 32441 +21
+ Misses 19147 19128 -19
- Partials 2800 2803 +3
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/19361?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../core/metadata/generator/PipelineDDLGenerator.java](https://codecov.io/gh/apache/shardingsphere/pull/19361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUta2VybmVsL3NoYXJkaW5nc3BoZXJlLWRhdGEtcGlwZWxpbmUvc2hhcmRpbmdzcGhlcmUtZGF0YS1waXBlbGluZS1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9kYXRhL3BpcGVsaW5lL2NvcmUvbWV0YWRhdGEvZ2VuZXJhdG9yL1BpcGVsaW5lRERMR2VuZXJhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...cryptForUseDefaultInsertColumnsTokenGenerator.java](https://codecov.io/gh/apache/shardingsphere/pull/19361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtZmVhdHVyZXMvc2hhcmRpbmdzcGhlcmUtZW5jcnlwdC9zaGFyZGluZ3NwaGVyZS1lbmNyeXB0LWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2VuY3J5cHQvcmV3cml0ZS90b2tlbi9nZW5lcmF0b3IvRW5jcnlwdEZvclVzZURlZmF1bHRJbnNlcnRDb2x1bW5zVG9rZW5HZW5lcmF0b3IuamF2YQ==) | `61.36% <50.00%> (+58.98%)` | :arrow_up: |
| [...re/driver/api/ShardingSphereDataSourceFactory.java](https://codecov.io/gh/apache/shardingsphere/pull/19361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtamRiYy9zaGFyZGluZ3NwaGVyZS1qZGJjLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL2RyaXZlci9hcGkvU2hhcmRpbmdTcGhlcmVEYXRhU291cmNlRmFjdG9yeS5qYXZh) | `54.54% <0.00%> (-45.46%)` | :arrow_down: |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz merged pull request #19361: Fixed openGauss scaling IT problems
Posted by GitBox <gi...@apache.org>.
sandynz merged PR #19361:
URL: https://github.com/apache/shardingsphere/pull/19361
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on a diff in pull request #19361: Fixed openGauss scaling IT problems
Posted by GitBox <gi...@apache.org>.
azexcy commented on code in PR #19361:
URL: https://github.com/apache/shardingsphere/pull/19361#discussion_r925340895
##########
shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-scaling/src/test/java/org/apache/shardingsphere/integration/data/pipeline/framework/container/proxy/ShardingSphereProxyDockerContainer.java:
##########
@@ -44,9 +44,9 @@ public ShardingSphereProxyDockerContainer(final DatabaseType databaseType) {
protected void configure() {
withExposedPorts(3307);
mapConfigurationFiles();
- if (DatabaseTypeUtil.isPostgreSQL(databaseType) || DatabaseTypeUtil.isOpenGauss(databaseType)) {
+ if (DatabaseTypeUtil.isPostgreSQL(databaseType)) {
setWaitStrategy(new JDBCConnectionWaitStrategy(() -> DriverManager.getConnection(DataSourceEnvironment.getURL(databaseType, getHost(), getMappedPort(3307), "postgres"), "root", "root")));
Review Comment:
Previously it was to ensure that the proxy was ready to be connected, but now I test find it's not required， and opengauss can't use it now, so I remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #19361: Fixed openGauss scaling IT problems
Posted by GitBox <gi...@apache.org>.
sandynz commented on code in PR #19361:
URL: https://github.com/apache/shardingsphere/pull/19361#discussion_r925208903
##########
shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-scaling/src/test/java/org/apache/shardingsphere/integration/data/pipeline/framework/container/proxy/ShardingSphereProxyDockerContainer.java:
##########
@@ -44,9 +44,9 @@ public ShardingSphereProxyDockerContainer(final DatabaseType databaseType) {
protected void configure() {
withExposedPorts(3307);
mapConfigurationFiles();
- if (DatabaseTypeUtil.isPostgreSQL(databaseType) || DatabaseTypeUtil.isOpenGauss(databaseType)) {
+ if (DatabaseTypeUtil.isPostgreSQL(databaseType)) {
setWaitStrategy(new JDBCConnectionWaitStrategy(() -> DriverManager.getConnection(DataSourceEnvironment.getURL(databaseType, getHost(), getMappedPort(3307), "postgres"), "root", "root")));
Review Comment:
OK. And why PostgreSQL and MySQL need a customized one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on a diff in pull request #19361: Fixed openGauss scaling IT problems
Posted by GitBox <gi...@apache.org>.
azexcy commented on code in PR #19361:
URL: https://github.com/apache/shardingsphere/pull/19361#discussion_r925334268
##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-core/src/main/java/org/apache/shardingsphere/data/pipeline/core/metadata/generator/PipelineDDLGenerator.java:
##########
@@ -225,6 +225,10 @@ private Optional<String> decorateOpenGauss(final DatabaseType databaseType, fina
if (logicSQL.toLowerCase().startsWith(SET_SEARCH_PATH_PREFIX)) {
return Optional.empty();
}
- return Optional.of(replaceTableNameWithPrefix(logicSQL, schemaName + ".", databaseType, databaseName));
+ String result = replaceTableNameWithPrefix(logicSQL, schemaName + ".", databaseType, databaseName);
+ if (result.toUpperCase().startsWith("CREATE TABLE")) {
+ result = result.replaceFirst("CREATE TABLE", "CREATE TABLE IF NOT EXISTS");
+ }
+ return Optional.of(result);
Review Comment:
Only use for openGauss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on a diff in pull request #19361: Fixed openGauss scaling IT problems
Posted by GitBox <gi...@apache.org>.
TeslaCN commented on code in PR #19361:
URL: https://github.com/apache/shardingsphere/pull/19361#discussion_r925396690
##########
shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-scaling/src/test/resources/env/opengauss/initdb.sql:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+CREATE DATABASE scaling_it_0;
+CREATE DATABASE scaling_it_1;
+CREATE DATABASE scaling_it_2;
+CREATE DATABASE scaling_it_3;
+CREATE DATABASE scaling_it_4;
+CREATE USER scaling WITH ENCRYPTED PASSWORD 'Root@123';
+CREATE DATABASE scaling;
+ALTER ROLE scaling CREATEDB REPLICATION;
+GRANT CREATE, CONNECT ON DATABASE scaling_it_0 TO scaling;
+GRANT CREATE, CONNECT ON DATABASE scaling_it_1 TO scaling;
+GRANT CREATE, CONNECT ON DATABASE scaling_it_2 TO scaling;
+GRANT CREATE, CONNECT ON DATABASE scaling_it_3 TO scaling;
+GRANT CREATE, CONNECT ON DATABASE scaling_it_4 TO scaling;
Review Comment:
Add line break to the end of file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #19361: Fixed openGauss scaling IT problems
Posted by GitBox <gi...@apache.org>.
sandynz commented on code in PR #19361:
URL: https://github.com/apache/shardingsphere/pull/19361#discussion_r925144493
##########
shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-scaling/src/test/resources/env/opengauss/initdb.sql:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+CREATE DATABASE scaling_it_0;
+CREATE DATABASE scaling_it_1;
+CREATE DATABASE scaling_it_2;
+CREATE DATABASE scaling_it_3;
+CREATE DATABASE scaling_it_4;
+ALTER ROLE gaussdb CREATEDB REPLICATION;
Review Comment:
Could we create role/user and then grant privileges for it? Like PostgreSQL scaling IT
##########
shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-scaling/src/test/java/org/apache/shardingsphere/integration/data/pipeline/framework/container/proxy/ShardingSphereProxyDockerContainer.java:
##########
@@ -44,9 +44,9 @@ public ShardingSphereProxyDockerContainer(final DatabaseType databaseType) {
protected void configure() {
withExposedPorts(3307);
mapConfigurationFiles();
- if (DatabaseTypeUtil.isPostgreSQL(databaseType) || DatabaseTypeUtil.isOpenGauss(databaseType)) {
+ if (DatabaseTypeUtil.isPostgreSQL(databaseType)) {
setWaitStrategy(new JDBCConnectionWaitStrategy(() -> DriverManager.getConnection(DataSourceEnvironment.getURL(databaseType, getHost(), getMappedPort(3307), "postgres"), "root", "root")));
Review Comment:
Does it means openGauss case doesn't need to wait proxy ready?
##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-core/src/main/java/org/apache/shardingsphere/data/pipeline/core/metadata/generator/PipelineDDLGenerator.java:
##########
@@ -225,6 +225,10 @@ private Optional<String> decorateOpenGauss(final DatabaseType databaseType, fina
if (logicSQL.toLowerCase().startsWith(SET_SEARCH_PATH_PREFIX)) {
return Optional.empty();
}
- return Optional.of(replaceTableNameWithPrefix(logicSQL, schemaName + ".", databaseType, databaseName));
+ String result = replaceTableNameWithPrefix(logicSQL, schemaName + ".", databaseType, databaseName);
+ if (result.toUpperCase().startsWith("CREATE TABLE")) {
+ result = result.replaceFirst("CREATE TABLE", "CREATE TABLE IF NOT EXISTS");
+ }
+ return Optional.of(result);
Review Comment:
1, `result.toUpperCase().startsWith` and `result.replaceFirst` might have different String value, it's not consistent;
2, Will `IF NOT EXISTS` be added in underlying SQL generator? If true, then result might include duplicated `IF NOT EXISTS`
##########
shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-scaling/src/test/java/org/apache/shardingsphere/integration/data/pipeline/cases/base/BaseITCase.java:
##########
@@ -269,7 +270,15 @@ protected void createScalingRule() {
}
protected void createSchema(final String schemaName) {
- executeWithLog(String.format("CREATE SCHEMA %s", schemaName));
+ try {
+ executeWithLog(String.format("CREATE SCHEMA %s", schemaName));
+ } catch (final BadSqlGrammarException ex) {
+ if (ex.getMessage().contains("schema \"test\" already exists")) {
+ log.info("schema {} already exists.", schemaName);
+ } else {
Review Comment:
It's not good enough to check hard-coded error message.
Is there another way to check whether schema is existing or not? e.g. from database metadata?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org