You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2022/08/07 15:08:53 UTC
[GitHub] [shardingsphere] huangdx0726 opened a new pull request, #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
huangdx0726 opened a new pull request, #19949:
URL: https://github.com/apache/shardingsphere/pull/19949
Fixes #19938 .
Changes proposed in this pull request:
-
-
-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] huangdx0726 commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1209096530
```java
protected void addTargetResource() {
String addTargetResource = commonSQLCommand.getTargetAddResourceTemplate().replace("${user}", username)
.replace("${password}", password)
.replace("${ds2}", getActualJdbcUrlTemplate(DS_2))
.replace("${ds3}", getActualJdbcUrlTemplate(DS_3))
.replace("${ds4}", getActualJdbcUrlTemplate(DS_4));
executeWithLog(addTargetResource);
List<Map<String, Object>> resources = queryForListWithLog("SHOW DATABASE RESOURCES from sharding_db");
assertThat(resources.size(), is(5)); // faild line ，actul is 3
assertBeforeApplyScalingMetadataCorrectly();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1209242405
I pulled the code of this PR and ran TextPrimaryKeyScalingIT, it's reproduced:
```
[Thread-18] o.a.s.d.p.s.r.RuleAlteredJob - job prepare failed, 0130317c30317c3054317c7368617264696e675f6462-1
org.apache.shardingsphere.infra.util.spi.exception.ServiceProviderNotFoundException: SPI-00001: No implementation class load from SPI `org.apache.shardingsphere.scaling.core.job.importer.ImporterCreator` with type `MySQL`.
at org.apache.shardingsphere.infra.util.spi.type.typed.TypedSPIRegistry.getRegisteredService(TypedSPIRegistry.java:100)
at org.apache.shardingsphere.scaling.core.job.importer.ImporterCreatorFactory.getInstance(ImporterCreatorFactory.java:41)
at org.apache.shardingsphere.data.pipeline.core.task.IncrementalTask.createImporters(IncrementalTask.java:100)
at org.apache.shardingsphere.data.pipeline.core.task.IncrementalTask.<init>(IncrementalTask.java:79)
at org.apache.shardingsphere.data.pipeline.scenario.rulealtered.RuleAlteredJobPreparer.initIncrementalTasks(RuleAlteredJobPreparer.java:200)
at org.apache.shardingsphere.data.pipeline.scenario.rulealtered.RuleAlteredJobPreparer.prepare(RuleAlteredJobPreparer.java:92)
at org.apache.shardingsphere.data.pipeline.scenario.rulealtered.RuleAlteredJob.prepare(RuleAlteredJob.java:101)
at org.apache.shardingsphere.data.pipeline.scenario.rulealtered.RuleAlteredJob.lambda$execute$0(RuleAlteredJob.java:71)
at java.lang.Thread.run(Thread.java:748)
```
This is the root cause. You could follow this one.
job item progress:
```
sourceDatabaseType: MySQL
status: PREPARING_FAILURE
```
It caused another issue in ShowScalingJobStatusQueryResultSet. I'll fix this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1207782459
Hi @huangdx0726 , the Continuous Integration (unit test) always cost more than 60 minutes (I've re-ran it).
From the CI log, seems it's blocked on shardingsphere-pipeline-test module.
Could you run shardingsphere-pipeline-test module unit test on local and check what's wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1208846439
@huangdx0726 , Scaling Integration Test failed, you could try to debug on local.
- Refer to [Scaling IT]( https://shardingsphere.apache.org/document/current/cn/test-manual/scaling-integration-test/ ), set `scaling.it.env.type=NATIVE` in it-env.properties;
- Refer to [Conf]( https://github.com/apache/shardingsphere/tree/master/shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-scaling/src/test/resources/env/mysql ) for MySQL conf (my.cnf) and Proxy conf (server-5.yaml or server-8.yaml)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz merged pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz merged PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1207946077
@huangdx0726 , OK, since DefaultImporter is created now but not FixtureImporter. It need incrementalTask stop, but it might need to change much unit test classes.
In order to make it compatible with current unit tests, and we need to remove ScalingEntry finally, we could take another way.
Possible changes:
- Add ImporterCreator interface, it extends TypedSPI, marked as `@SingletonSPI`
- ImporterCreator has 2 sub-classes, FixtureImporterCreator and DefaultImporterCreator; FixtureImporterCreator's type is H2, DefaultImporterCreator's type is MySQL and typeAliases are PostgreSQL & openGauss
- Refactor ImporterFactory to ImporterCreatorFactory
Examples:
- PositionInitializer and PositionInitializerFactory, #15180
- PipelineChannelCreator and PipelineChannelCreatorFactory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1209193348
mode type must be `Cluster` for now to test scaling. You could copy the content of `shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-scaling/src/test/resources/env/mysql/server-5.yaml` into proxy `server.yaml`.
And `config-sharding.yaml` is not necessary to configure. Just run the failed IT class in IDE, e.g. MySQLGeneralScalingIT / TextPrimaryKeyScalingIT after proxy start successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1210121927
@huangdx0726 , thanks for your contribution.
There're several similar refactoring, I'll create issues later, welcome for more contributions.
Notice:
You could import [code style idea]( https://github.com/apache/shardingsphere/blob/master/src/resources/code-style-idea.xml ) into IDEA, so java importing and blanks between methods could be formatted correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] huangdx0726 commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1209237280
what 's the zk version? @sandynz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on code in PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#discussion_r941938952
##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-core/src/test/resources/META-INF/services/org.apache.shardingsphere.scaling.core.job.importer.ImporterCreator:
##########
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.shardingsphere.scaling.core.job.importer.DefaultImporterCreator
Review Comment:
This file should be moved from `test/resources/META-INF/services` to `main/resources/META-INF/services`
##########
shardingsphere-test/shardingsphere-pipeline-test/src/test/resources/META-INF/services/org.apache.shardingsphere.scaling.core.job.importer.ImporterCreator:
##########
@@ -0,0 +1,19 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.shardingsphere.data.pipeline.core.fixture.FixtureImporterCreator
+org.apache.shardingsphere.scaling.core.job.importer.DefaultImporterCreator
Review Comment:
`org.apache.shardingsphere.scaling.core.job.importer.DefaultImporterCreator` could be removed since it's already registered in pipeline-core module, just keep `org.apache.shardingsphere.data.pipeline.core.fixture.FixtureImporterCreator` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1209286596
@huangdx0726 , looks you've put `org.apache.shardingsphere.scaling.core.job.importer.DefaultImporterCreator` in `test/resources`, you could put it in `main/resources`.
And you'd better add `ImporterCreatorFactoryTest` in `shardingsphere-pipeline-test` module, and test whether `ImporterCreator` could be got by "MySQL", "PostgreSQL", "openGauss" or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] huangdx0726 commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1209167425
I try to change `assertThat(resources.size(), is(3))`, then the following error is occurred.
```java
[ERROR] 2022-08-09 16:56:37.661 [main] o.a.s.i.d.p.cases.base.BaseITCase - data access error
org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [SHOW SCALING LIST]; SQL state [C1997]; error code [1997]; Runtime exception: [Mode configuration is required.]; nested exception is java.sql.SQLException: Runtime exception: [Mode configuration is required.]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:90)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:418)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:477)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:487)
at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:513)
at org.apache.shardingsphere.integration.data.pipeline.cases.base.BaseITCase.queryForListWithLog(BaseITCase.java:308)
at org.apache.shardingsphere.integration.data.pipeline.cases.base.BaseITCase.getScalingJobId(BaseITCase.java:347)
at org.apache.shardingsphere.integration.data.pipeline.cases.general.MySQLGeneralScalingIT.assertManualScalingSuccess(MySQLGeneralScalingIT.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
Caused by: java.sql.SQLException: Runtime exception: [Mode configuration is required.]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2491)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2449)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1381)
at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:111)
at com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:460)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
... 40 common frames omitted
java.lang.RuntimeException: can't get result from proxy
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] huangdx0726 commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1209242237
```java
[ERROR] 2022-08-09 19:08:39.402 [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl - Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:981)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:952)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:342)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] huangdx0726 commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1209171268
My local proxy conf is used: `server.yaml`
```yaml
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
######################################################################################################
#
# If you want to configure governance, authorization and proxy properties, please refer to this file.
#
######################################################################################################
mode:
type: Standalone
# repository:
# type: ZooKeeper
# props:
# namespace: governance_ds
# server-lists: localhost:2181
# retryIntervalMilliseconds: 500
# timeToLiveSeconds: 60
# maxRetries: 3
# operationTimeoutMilliseconds: 500
# overwrite: false
#
rules:
- !AUTHORITY
users:
- root@:Root@123
provider:
type: ALL_PERMITTED
props:
max-connections-size-per-query: 1
kernel-executor-size: 16 # Infinite by default.
proxy-frontend-flush-threshold: 128 # The default value is 128.
proxy-hint-enabled: true
sql-show: true
sql-federation-enabled: true
proxy-mysql-default-version: 5.7.22
proxy-default-port: 3308 # Proxy default port.
# proxy-netty-backlog: 1024 # Proxy netty backlog.
```
`config-sharding.yaml`
```yaml
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
######################################################################################################
#
# Here you can configure the rules for the proxy.
# This example is configuration of sharding rule.
#
######################################################################################################
#
databaseName: sharding_db
#
#dataSources:
# ds_0:
# url: jdbc:postgresql://127.0.0.1:5432/demo_ds_0
# username: postgres
# password: postgres
# connectionTimeoutMilliseconds: 30000
# idleTimeoutMilliseconds: 60000
# maxLifetimeMilliseconds: 1800000
# maxPoolSize: 50
# minPoolSize: 1
# ds_1:
# url: jdbc:postgresql://127.0.0.1:5432/demo_ds_1
# username: postgres
# password: postgres
# connectionTimeoutMilliseconds: 30000
# idleTimeoutMilliseconds: 60000
# maxLifetimeMilliseconds: 1800000
# maxPoolSize: 50
# minPoolSize: 1
#
#rules:
#- !SHARDING
# tables:
# t_order:
# actualDataNodes: ds_${0..1}.t_order_${0..1}
# tableStrategy:
# standard:
# shardingColumn: order_id
# shardingAlgorithmName: t_order_inline
# keyGenerateStrategy:
# column: order_id
# keyGeneratorName: snowflake
# auditStrategy:
# auditorNames:
# - sharding_key_required_auditor
# allowHintDisable: true
# t_order_item:
# actualDataNodes: ds_${0..1}.t_order_item_${0..1}
# tableStrategy:
# standard:
# shardingColumn: order_id
# shardingAlgorithmName: t_order_item_inline
# keyGenerateStrategy:
# column: order_item_id
# keyGeneratorName: snowflake
# bindingTables:
# - t_order,t_order_item
# defaultDatabaseStrategy:
# standard:
# shardingColumn: user_id
# shardingAlgorithmName: database_inline
# defaultTableStrategy:
# none:
# defaultAuditStrategy:
# auditorNames:
# - sharding_key_required_auditor
# allowHintDisable: true
#
# shardingAlgorithms:
# database_inline:
# type: INLINE
# props:
# algorithm-expression: ds_${user_id % 2}
# t_order_inline:
# type: INLINE
# props:
# algorithm-expression: t_order_${order_id % 2}
# t_order_item_inline:
# type: INLINE
# props:
# algorithm-expression: t_order_item_${order_id % 2}
#
# keyGenerators:
# snowflake:
# type: SNOWFLAKE
#
# auditors:
# sharding_key_required_auditor:
# type: DML_SHARDING_CONDITIONS
#
# scalingName: default_scaling
# scaling:
# default_scaling:
# input:
# workerThread: 40
# batchSize: 1000
# output:
# workerThread: 40
# batchSize: 1000
# streamChannel:
# type: MEMORY
# props:
# block-queue-size: 10000
# completionDetector:
# type: IDLE
# props:
# incremental-task-idle-seconds-threshold: 1800
# dataConsistencyChecker:
# type: DATA_MATCH
# props:
# chunk-size: 1000
######################################################################################################
#
# If you want to connect to MySQL, you should manually copy MySQL driver to lib directory.
#
######################################################################################################
#databaseName: sharding_db
#
dataSources:
ds_0:
url: jdbc:mysql://127.0.0.1:3307/demo_ds_0?serverTimezone=UTC&useSSL=false
username: root
password: root
# connectionTimeoutMilliseconds: 30000
# idleTimeoutMilliseconds: 60000
# maxLifetimeMilliseconds: 1800000
# maxPoolSize: 50
# minPoolSize: 1
ds_1:
url: jdbc:mysql://127.0.0.1:3307/demo_ds_1?serverTimezone=UTC&useSSL=false
username: root
password: root
# connectionTimeoutMilliseconds: 30000
# idleTimeoutMilliseconds: 60000
# maxLifetimeMilliseconds: 1800000
# maxPoolSize: 50
# minPoolSize: 1
#
rules:
- !SHARDING
tables:
t_order:
actualDataNodes: ds_${0..1}.t_order_${0..1}
tableStrategy:
standard:
shardingColumn: order_id
shardingAlgorithmName: t_order_inline
keyGenerateStrategy:
column: order_id
keyGeneratorName: snowflake
auditStrategy:
auditorNames:
- sharding_key_required_auditor
allowHintDisable: true
t_order_item:
actualDataNodes: ds_${0..1}.t_order_item_${0..1}
tableStrategy:
standard:
shardingColumn: order_id
shardingAlgorithmName: t_order_item_inline
keyGenerateStrategy:
column: order_item_id
keyGeneratorName: snowflake
bindingTables:
- t_order,t_order_item
defaultDatabaseStrategy:
standard:
shardingColumn: user_id
shardingAlgorithmName: database_inline
defaultTableStrategy:
none:
defaultAuditStrategy:
auditorNames:
- sharding_key_required_auditor
allowHintDisable: true
shardingAlgorithms:
database_inline:
type: INLINE
props:
algorithm-expression: ds_${user_id % 2}
t_order_inline:
type: INLINE
props:
algorithm-expression: t_order_${order_id % 2}
t_order_item_inline:
type: INLINE
props:
algorithm-expression: t_order_item_${order_id % 2}
keyGenerators:
snowflake:
type: SNOWFLAKE
auditors:
sharding_key_required_auditor:
type: DML_SHARDING_CONDITIONS
scalingName: default_scaling
scaling:
default_scaling:
input:
workerThread: 40
batchSize: 1000
output:
workerThread: 40
batchSize: 1000
streamChannel:
type: MEMORY
props:
block-queue-size: 10000
completionDetector:
type: IDLE
props:
incremental-task-idle-seconds-threshold: 1800
dataConsistencyChecker:
type: DATA_MATCH
props:
chunk-size: 1000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] huangdx0726 commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1207814196
the method `assertStart` of `IncrementalTaskTest` is blocked . it's blocked line in `IncrementalTask`
```java
private void waitForResult(final Future<?> future) {
try {
future.get(); //blocked line
} catch (final InterruptedException ignored) {
} catch (final ExecutionException ex) {
throw new PipelineJobExecutionException(String.format("Task %s execute failed ", taskId), ex.getCause());
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] huangdx0726 commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1210123940
ok， Thank you for your help .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] huangdx0726 commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1207949787
OK, i will try to fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on code in PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#discussion_r940207935
##########
shardingsphere-test/shardingsphere-pipeline-test/src/test/resources/META-INF/services/org.apache.shardingsphere.scaling.core.job.importer.ImporterCreator:
##########
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.shardingsphere.data.pipeline.core.fixture.FixtureImporterCreator
Review Comment:
It should be ended with new line
##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-core/src/main/java/org/apache/shardingsphere/scaling/core/job/importer/DefaultImporterCreator.java:
##########
@@ -27,22 +27,27 @@
import org.apache.shardingsphere.data.pipeline.spi.importer.Importer;
/**
- * Importer factory.
+ * Default importer creator.
*/
-@NoArgsConstructor(access = AccessLevel.PRIVATE)
-public final class ImporterFactory {
+public class DefaultImporterCreator implements ImporterCreator {
- /**
- * Create importer.
- *
- * @param importerConfig importer configuration
- * @param dataSourceManager data source manager
- * @param channel channel
- * @param jobProgressListener job progress listener
- * @return importer
- */
- public static Importer createImporter(final ImporterConfiguration importerConfig, final PipelineDataSourceManager dataSourceManager, final PipelineChannel channel,
- final PipelineJobProgressListener jobProgressListener) {
+ @Override
+ public String getType() {
+ return "MySQL";
+ }
+
+ @Override
+ public Collection<String> getTypeAliases() {
+ Collection<String> aliases = new LinkedList<>();
+ aliases.add("PostgreSQL");
+ aliases.add("openGauss");
+ return aliases;
+ }
Review Comment:
It's better to put getType() and getTypeAliases() at the bottom of class, conventionally
##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-core/src/test/resources/META-INF/services/org.apache.shardingsphere.scaling.core.job.importer.ImporterCreator:
##########
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.shardingsphere.scaling.core.job.importer.DefaultImporterCreator
Review Comment:
It should be ended with new line
##########
shardingsphere-kernel/shardingsphere-data-pipeline/shardingsphere-data-pipeline-core/src/main/java/org/apache/shardingsphere/scaling/core/job/importer/DefaultImporterCreator.java:
##########
@@ -27,22 +27,27 @@
import org.apache.shardingsphere.data.pipeline.spi.importer.Importer;
/**
- * Importer factory.
+ * Default importer creator.
*/
-@NoArgsConstructor(access = AccessLevel.PRIVATE)
-public final class ImporterFactory {
+public class DefaultImporterCreator implements ImporterCreator {
Review Comment:
`final` could be added for class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1209147407
From GitHub scaling-it-test log, I didn't find `assertThat(resources.size(), is(5));` error of `addTargetResource`. But there's exception:
```
[INFO ] 2022-08-08 15:02:26.094 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: java.lang.NullPointerException: null
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at org.apache.shardingsphere.scaling.distsql.handler.query.ShowScalingJobStatusQueryResultSet.lambda$init$0(ShowScalingJobStatusQueryResultSet.java:51)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.util.Iterator.forEachRemaining(Iterator.java:116)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at org.apache.shardingsphere.scaling.distsql.handler.query.ShowScalingJobStatusQueryResultSet.init(ShowScalingJobStatusQueryResultSet.java:65)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at org.apache.shardingsphere.proxy.backend.handler.distsql.ral.scaling.query.QueryableScalingRALBackendHandler.execute(QueryableScalingRALBackendHandler.java:50)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at org.apache.shardingsphere.proxy.backend.handler.distsql.ral.scaling.query.QueryableScalingRALBackendHandler.execute(QueryableScalingRALBackendHandler.java:39)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at org.apache.shardingsphere.proxy.backend.handler.DatabaseRequiredBackendHandler.execute(DatabaseRequiredBackendHandler.java:51)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at org.apache.shardingsphere.proxy.frontend.mysql.command.query.text.query.MySQLComQueryPacketExecutor.execute(MySQLComQueryPacketExecutor.java:94)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at org.apache.shardingsphere.proxy.frontend.command.CommandExecutorTask.executeCommand(CommandExecutorTask.java:112)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at org.apache.shardingsphere.proxy.frontend.command.CommandExecutorTask.run(CommandExecutorTask.java:78)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[INFO ] 2022-08-08 15:02:26.095 [docker-java-stream-512339527] :Scaling-Proxy - STDOUT: at java.lang.Thread.run(Thread.java:748)
Error: 2022-08-08 15:02:27.437 [main] o.a.s.i.d.p.cases.base.BaseITCase - data access error
org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [SHOW SCALING STATUS 0130317c30317c3054317c7368617264696e675f6462]; SQL state [C1997]; error code [1997]; Runtime exception: [null]; nested exception is java.sql.SQLException: Runtime exception: [null]
```
Did you reproduce it on local?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #19949: Remove AbstractImporter sub-classes and simplify ImporterFactory
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #19949:
URL: https://github.com/apache/shardingsphere/pull/19949#issuecomment-1209244414
> what 's the zk version? @sandynz
3.6.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org