You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2021/11/16 02:27:27 UTC
[GitHub] [shardingsphere] marker-wu opened a new issue #13632: Does sharding-scaling only support Distsql?
marker-wu opened a new issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632
Does sharding-scaling only support Distsql? Can other methods be supported? For example: the way to configure in yaml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] marker-wu edited a comment on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
marker-wu edited a comment on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-970156626
> Some verification:
>
> 1. Are you running version `5.0.0`?
yes
> 2. Could you paste your current schema rules and rule cache? Register center (e.g. ZooKeeper) path example: `/metadata/sharding_db/rules`, `/metadata/sharding_db/rules/cache/xxx`.
/metadata/sharding_db/rules：
```
- !SHARDING
bindingTables:
- t_order,t_order_item
defaultDatabaseStrategy:
standard:
shardingAlgorithmName: database_inline
shardingColumn: user_id
defaultTableStrategy:
none: ''
keyGenerators:
snowflake:
props:
worker-id: 123
type: SNOWFLAKE
shardingAlgorithms:
database_inline:
props:
algorithm-expression: ds_${user_id % 2}
type: INLINE
t_order_inline:
props:
algorithm-expression: t_order_${order_id % 2}
type: INLINE
t_order_item_inline:
props:
algorithm-expression: t_order_item_${order_id % 2}
type: INLINE
tables:
t_order:
actualDataNodes: ds_${0..1}.t_order_${0..1}
keyGenerateStrategy:
column: order_id
keyGeneratorName: snowflake
logicTable: t_order
tableStrategy:
standard:
shardingAlgorithmName: t_order_inline
shardingColumn: order_id
t_order_item:
actualDataNodes: ds_${0..1}.t_order_item_${0..1}
keyGenerateStrategy:
column: order_item_id
keyGeneratorName: snowflake
logicTable: t_order_item
tableStrategy:
standard:
shardingAlgorithmName: t_order_item_inline
shardingColumn: order_id
```
/metadata/sharding_db/rules/cache/0a81aa01235c42738b80eda6ad3dd0ec：
```
- !SHARDING
bindingTables:
- t_order,t_order_item
defaultDatabaseStrategy:
standard:
shardingAlgorithmName: database_inline
shardingColumn: user_id
defaultTableStrategy:
none: ''
keyGenerators:
snowflake:
props:
worker-id: 123
type: SNOWFLAKE
t_order_snowflake:
props:
worker-id: '123'
type: snowflake
t_order_item_snowflake:
props:
worker-id: '124'
type: snowflake
shardingAlgorithms:
database_inline:
props:
algorithm-expression: ds_${user_id % 2}
type: INLINE
t_order_inline:
props:
algorithm-expression: t_order_${order_id % 2}
type: INLINE
t_order_item_inline:
props:
algorithm-expression: t_order_item_${order_id % 2}
type: INLINE
scaling_inline:
props:
algorithm-expression: t_order_${order_id % 3} +2
type: inline
tables:
t_order:
actualDataNodes: ds_${2..4}.t_order${0..1}
databaseStrategy:
standard:
shardingAlgorithmName: scaling_inline
shardingColumn: user_id
keyGenerateStrategy:
column: order_id
keyGeneratorName: t_order_snowflake
logicTable: t_order
tableStrategy:
standard:
shardingAlgorithmName: t_order_inline
shardingColumn: order_id
t_order_item:
actualDataNodes: ds_${2..4}.t_order_item_${0..1}
databaseStrategy:
standard:
shardingAlgorithmName: scaling_inline
shardingColumn: user_id
keyGenerateStrategy:
column: order_item_id
keyGeneratorName: t_order_item_snowflake
logicTable: t_order_item
tableStrategy:
standard:
shardingAlgorithmName: t_order_item_inline
shardingColumn: order_id
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
sandynz commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-983373928
I reproduced this issue on commit 3f1ca7f190a04723b9929729c953271667286879.
### Reproduce steps
1. Cleanup `/metadata/sharding_db`
2. in `config-sharding.yaml`
```
dataSources:
ds_0:
url: jdbc:mysql://127.0.0.1:3306/scaling_ds_0?serverTimezone=UTC&useSSL=false
username: root
password: root
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
ds_1:
url: jdbc:mysql://127.0.0.1:3306/scaling_ds_1?serverTimezone=UTC&useSSL=false
username: root
password: root
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
```
3. in `server.yaml`
```
mode:
type: Cluster
repository:
type: ZooKeeper
props:
namespace: governance_ds
server-lists: localhost:2181
retryIntervalMilliseconds: 500
timeToLiveSeconds: 60
maxRetries: 3
operationTimeoutMilliseconds: 500
overwrite: true
```
4. start proxy
5. `/metadata/sharding_db/dataSources`
```
ds_0:
schema: null
transactionIsolation: null
initializationFailTimeout: 1
validationTimeout: 5000
catalog: null
maxLifetime: 1800000
leakDetectionThreshold: 0
connectionInitSql: null
minimumIdle: 1
password: root
dataSourceJNDI: null
connectionTestQuery: null
idleTimeout: 60000
driverClassName: null
jdbcUrl: jdbc:mysql://127.0.0.1:3306/scaling_ds_0?serverTimezone=UTC&useSSL=false
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
maximumPoolSize: 50
connectionTimeout: 30000
username: root
poolName: HikariPool-1
ds_1:
schema: null
transactionIsolation: null
initializationFailTimeout: 1
validationTimeout: 5000
catalog: null
maxLifetime: 1800000
leakDetectionThreshold: 0
connectionInitSql: null
minimumIdle: 1
password: root
dataSourceJNDI: null
connectionTestQuery: null
idleTimeout: 60000
driverClassName: null
jdbcUrl: jdbc:mysql://127.0.0.1:3306/scaling_ds_1?serverTimezone=UTC&useSSL=false
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
maximumPoolSize: 50
connectionTimeout: 30000
username: root
poolName: HikariPool-2
```
`driverClassName: null` and also much other keys with value null, these keys should not be persisted in registry center.
### Reason analyze
I'm not sure which change cause this issue, I use cached metadata generated several months before, it should be caused by latest months PR.
Possible solution:
`AbstractDataSourceCreator.findAllGetterProperties` get all property name from target by reflection, and put them into result map, no matter property value is null or not. We could filter these keys here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] menghaoranss closed issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
menghaoranss closed issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
sandynz commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-970143820
Some verification:
1. Are you running version `5.0.0`?
2. Could you paste your current schema rules and rule cache? Register center (e.g. ZooKeeper) path example: `/metadata/sharding_db/rules`, `/metadata/sharding_db/rules/cache/xxx`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] marker-wu commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
marker-wu commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-971321514
I did not configure the schema and driveClassName attributes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] marker-wu commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
marker-wu commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-970156626
> Some verification:
>
> 1. Are you running version `5.0.0`?
yes
> 2. Could you paste your current schema rules and rule cache? Register center (e.g. ZooKeeper) path example: `/metadata/sharding_db/rules`, `/metadata/sharding_db/rules/cache/xxx`.
/metadata/sharding_db/rules：
- !SHARDING
bindingTables:
- t_order,t_order_item
defaultDatabaseStrategy:
standard:
shardingAlgorithmName: database_inline
shardingColumn: user_id
defaultTableStrategy:
none: ''
keyGenerators:
snowflake:
props:
worker-id: 123
type: SNOWFLAKE
shardingAlgorithms:
database_inline:
props:
algorithm-expression: ds_${user_id % 2}
type: INLINE
t_order_inline:
props:
algorithm-expression: t_order_${order_id % 2}
type: INLINE
t_order_item_inline:
props:
algorithm-expression: t_order_item_${order_id % 2}
type: INLINE
tables:
t_order:
actualDataNodes: ds_${0..1}.t_order_${0..1}
keyGenerateStrategy:
column: order_id
keyGeneratorName: snowflake
logicTable: t_order
tableStrategy:
standard:
shardingAlgorithmName: t_order_inline
shardingColumn: order_id
t_order_item:
actualDataNodes: ds_${0..1}.t_order_item_${0..1}
keyGenerateStrategy:
column: order_item_id
keyGeneratorName: snowflake
logicTable: t_order_item
tableStrategy:
standard:
shardingAlgorithmName: t_order_item_inline
shardingColumn: order_id
/metadata/sharding_db/rules/cache/0a81aa01235c42738b80eda6ad3dd0ec：
- !SHARDING
bindingTables:
- t_order,t_order_item
defaultDatabaseStrategy:
standard:
shardingAlgorithmName: database_inline
shardingColumn: user_id
defaultTableStrategy:
none: ''
keyGenerators:
snowflake:
props:
worker-id: 123
type: SNOWFLAKE
t_order_snowflake:
props:
worker-id: '123'
type: snowflake
t_order_item_snowflake:
props:
worker-id: '124'
type: snowflake
shardingAlgorithms:
database_inline:
props:
algorithm-expression: ds_${user_id % 2}
type: INLINE
t_order_inline:
props:
algorithm-expression: t_order_${order_id % 2}
type: INLINE
t_order_item_inline:
props:
algorithm-expression: t_order_item_${order_id % 2}
type: INLINE
scaling_inline:
props:
algorithm-expression: t_order_${order_id % 3} +2
type: inline
tables:
t_order:
actualDataNodes: ds_${2..4}.t_order${0..1}
databaseStrategy:
standard:
shardingAlgorithmName: scaling_inline
shardingColumn: user_id
keyGenerateStrategy:
column: order_id
keyGeneratorName: t_order_snowflake
logicTable: t_order
tableStrategy:
standard:
shardingAlgorithmName: t_order_inline
shardingColumn: order_id
t_order_item:
actualDataNodes: ds_${2..4}.t_order_item_${0..1}
databaseStrategy:
standard:
shardingAlgorithmName: scaling_inline
shardingColumn: user_id
keyGenerateStrategy:
column: order_item_id
keyGeneratorName: t_order_item_snowflake
logicTable: t_order_item
tableStrategy:
standard:
shardingAlgorithmName: t_order_item_inline
shardingColumn: order_id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
sandynz commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-971463700
> How long will the whole process end?
Based on your configuration, if you enabled `clusterAutoSwitchAlgorithm` and set `incremental-task-idle-minute-threshold`, it's `30` minutes as following configuration:
```
scaling:
blockQueueSize: 10000
workerThread: 40
clusterAutoSwitchAlgorithm:
type: IDLE
props:
incremental-task-idle-minute-threshold: 30
dataConsistencyCheckAlgorithm:
type: DEFAULT
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] marker-wu edited a comment on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
marker-wu edited a comment on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-970079053
I created the scaling task according to the guide document (https://shardingsphere.apache.org/document/5.0.0/cn/user-manual/shardingsphere-scaling/usage/), but it did not take effect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] marker-wu commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
marker-wu commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-970079053
I created the scaling task according to the guide document (https://shardingsphere.apache.org/document/5.0.0/cn/user-manual/shardingsphere-scaling/usage/), but it did not take effect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
sandynz commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-971359511
> I did not configure the schema and driveClassName attributes，But after the proxy is started, there will be these attributes in ZK
@menghaoranss Could you help to have a look at this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
sandynz commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-970152414
There're some not supported scenerios for now, you could try master branch or next release, refer to https://shardingsphere.apache.org/document/current/cn/user-manual/shardingsphere-scaling/usage/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] marker-wu edited a comment on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
marker-wu edited a comment on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-971321514
I did not configure the schema and driveClassName attributes，But after the proxy is started, there will be these attributes in ZK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
sandynz commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-971358788
I reproduced this exception in `StandardJDBCDataSourceConfigurationTest`, new lines is added in `config_standard_jdbc_target.yaml`:
```
schema: null
driverClassName: null
```
Caused exception:
```
Caused by: java.lang.NullPointerException
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at com.zaxxer.hikari.HikariConfig.attemptFromContextLoader(HikariConfig.java:899)
at com.zaxxer.hikari.HikariConfig.setDriverClassName(HikariConfig.java:474)
... 38 more
```
It's caused by `driverClassName` null value.
But I didn't find this case before and don't know why there's `driverClassName: null` in ZK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] marker-wu commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
marker-wu commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-971366132
> I reproduced this exception in `StandardJDBCDataSourceConfigurationTest`, new lines is added in `config_standard_jdbc_target.yaml`:
>
> ```
> schema: null
> driverClassName: null
> ```
>
> Caused exception:
>
> ```
> Caused by: java.lang.NullPointerException
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at com.zaxxer.hikari.HikariConfig.attemptFromContextLoader(HikariConfig.java:899)
> at com.zaxxer.hikari.HikariConfig.setDriverClassName(HikariConfig.java:474)
> ... 38 more
> ```
>
> It's caused by `driverClassName` null value.
>
> But I didn't find this case before and don't know why there's `driverClassName: null` in ZK.
I added the handling of the driveClassName attribute, and now it runs normally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
sandynz commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-969755178
Currently, it just support management by DistSQL (and also commercial admin console).
Configure by YAML is not planned for now.
Do you have some trouble on DistSQL?
How do you like to start scaling job by YAML?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] marker-wu commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
marker-wu commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-971375451
How long will the whole process end?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
sandynz commented on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-971220209
@marker-wu `tables` rule should be not supported in version `5.0.0`, I don't know what's wrong for now.
Seems scaling job config is wrong, there's unknown property `driverClassName`.
Could you paste `/metadata/sharding_db/dataSources` (erase sensitive db ip and password) and `/scaling/6474xxx` value in register center? And also full exception stack trace in log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] marker-wu edited a comment on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
marker-wu edited a comment on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-971321514
I did not configure the schema and driveClassName attributes，But proxy startup only has these attributes in ZK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] marker-wu edited a comment on issue #13632: Does sharding-scaling only support Distsql?
Posted by GitBox <gi...@apache.org>.
marker-wu edited a comment on issue #13632:
URL: https://github.com/apache/shardingsphere/issues/13632#issuecomment-971321514
I did not configure the schema and driveClassName attributes，But after the proxy is started, there will be these attributes in ZK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org