You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2021/02/01 08:02:09 UTC
[GitHub] [shardingsphere] lmhmhl opened a new pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
lmhmhl opened a new pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249
Ref #9152
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] lmhmhl edited a comment on pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
Posted by GitBox <gi...@apache.org>.
lmhmhl edited a comment on pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249#issuecomment-773826359
@tristaZero
Thanks for you comment, which help me make it clear and boost my confidence, I raise a new PR, please review it, I have to say you help me a lot. Thx.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] lmhmhl commented on pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
Posted by GitBox <gi...@apache.org>.
lmhmhl commented on pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249#issuecomment-773729643
@tristaZero
I have to say to there do exist some problems in my test, the datasource is created by `YamlShardingSphereDataSourceFactory` using yml file, in order to create two connections to different database instances to build different table structures, I separately get connections from `dataSourceMap` using database name. But the yml configuration are not work on these two sub connection, so I can only construct the sharding data all by myself, and the encrypt or replica query are not easy to test. Basically, I have to figure out why yml configuration are not work on sub connection fetched by `dataSourceMap`. Should I create two yml file or two datasources, and I have tried a lot but all failed, so this PR is just a compromise, I am in confused now, I would be very glad if u can give me some advices.
```java
@Before
public void initTable() {
try {
ShardingSphereConnection conn = dataSource.getConnection();
Map<String, DataSource> dataSourceMap = conn.getDataSourceMap();
Connection database0 = dataSourceMap.get("calcite_jdbc_0").getConnection();
Connection database1 = dataSourceMap.get("calcite_jdbc_1").getConnection();
RunScript.execute(database0, new InputStreamReader(Objects.requireNonNull(AbstractSQLTest.class.getClassLoader().getResourceAsStream("sql/calcite_data_0.sql"))));
RunScript.execute(database1, new InputStreamReader(Objects.requireNonNull(AbstractSQLTest.class.getClassLoader().getResourceAsStream("sql/calcite_data_1.sql"))));
conn.close();
} catch (final SQLException ex) {
throw new RuntimeException(ex);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tristaZero commented on a change in pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
Posted by GitBox <gi...@apache.org>.
tristaZero commented on a change in pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249#discussion_r568295546
##########
File path: shardingsphere-jdbc/shardingsphere-jdbc-core/src/test/resources/config/config-calcite.yaml
##########
@@ -23,10 +23,54 @@ rules:
tableStrategy:
standard:
shardingColumn: item_id
- shardingAlgorithmName: table_inline
+ shardingAlgorithmName: table_inline_item_id
+ t_user_encrypt_calcite_sharding:
+ actualDataNodes: calcite_jdbc_1.t_user_encrypt_calcite_sharding_${0..1}
Review comment:
> calcite_jdbc_1.t_user_encrypt_calcite_sharding_${0..1}
It is supposed to be `calcite_ds.t_user_encrypt_calcite_sharding_${0..1}` to leverage `replicaQuery` power.
##########
File path: shardingsphere-jdbc/shardingsphere-jdbc-core/src/test/resources/config/config-calcite.yaml
##########
@@ -23,10 +23,54 @@ rules:
tableStrategy:
standard:
shardingColumn: item_id
- shardingAlgorithmName: table_inline
+ shardingAlgorithmName: table_inline_item_id
+ t_user_encrypt_calcite_sharding:
+ actualDataNodes: calcite_jdbc_1.t_user_encrypt_calcite_sharding_${0..1}
+ tableStrategy:
+ standard:
+ shardingColumn: user_id
+ shardingAlgorithmName: table_inline_user_id
shardingAlgorithms:
- table_inline:
+ table_inline_item_id:
type: INLINE
props:
algorithm-expression: t_order_item_calcite_sharding_${item_id % 2}
+ table_inline_user_id:
+ type: INLINE
+ props:
+ algorithm-expression: t_user_encrypt_calcite_sharding_${user_id % 2}
+ - !ENCRYPT
+ encryptors:
+ encryptor_aes:
+ type: aes
+ props:
+ aes-key-value: 123456abc
+ tables:
+ t_user_encrypt_calcite:
+ columns:
+ pwd:
+ plainColumn: plain_pwd
+ cipherColumn: cipher_pwd
+ encryptorName: encryptor_aes
+ t_user_encrypt_calcite_sharding:
+ columns:
+ pwd:
+ plainColumn: plain_pwd
+ cipherColumn: cipher_pwd
+ encryptorName: encryptor_aes
+ - !REPLICA_QUERY
+ dataSources:
+ calcite_ds:
+ name: calcite_ds
+ primaryDataSourceName:
+ - calcite_jdbc_1
+ replicaDataSourceNames:
+ - calcite_jdbc_2
+ loadBalancerName: roundRobin
+ loadBalancers:
+ roundRobin:
+ type: ROUND_ROBIN
+props:
+ sql-show: true
+ query-with-cipher-column: false
Review comment:
Could we set `query-with-cipher-column` = `true` to indeed examine whether this SQL federation feature can work well with the encryption rule?
##########
File path: shardingsphere-jdbc/shardingsphere-jdbc-core/src/test/resources/config/config-calcite.yaml
##########
@@ -23,10 +23,54 @@ rules:
tableStrategy:
standard:
shardingColumn: item_id
- shardingAlgorithmName: table_inline
+ shardingAlgorithmName: table_inline_item_id
+ t_user_encrypt_calcite_sharding:
+ actualDataNodes: calcite_jdbc_1.t_user_encrypt_calcite_sharding_${0..1}
+ tableStrategy:
+ standard:
+ shardingColumn: user_id
+ shardingAlgorithmName: table_inline_user_id
shardingAlgorithms:
- table_inline:
+ table_inline_item_id:
type: INLINE
props:
algorithm-expression: t_order_item_calcite_sharding_${item_id % 2}
+ table_inline_user_id:
+ type: INLINE
+ props:
+ algorithm-expression: t_user_encrypt_calcite_sharding_${user_id % 2}
+ - !ENCRYPT
+ encryptors:
+ encryptor_aes:
+ type: aes
+ props:
+ aes-key-value: 123456abc
+ tables:
+ t_user_encrypt_calcite:
+ columns:
+ pwd:
+ plainColumn: plain_pwd
+ cipherColumn: cipher_pwd
+ encryptorName: encryptor_aes
+ t_user_encrypt_calcite_sharding:
+ columns:
+ pwd:
+ plainColumn: plain_pwd
+ cipherColumn: cipher_pwd
+ encryptorName: encryptor_aes
+ - !REPLICA_QUERY
+ dataSources:
+ calcite_ds:
+ name: calcite_ds
+ primaryDataSourceName:
+ - calcite_jdbc_1
Review comment:
If there is replication relationship between `calcite_jdbc_1` and `calcite_jdbc_2`, maybe `calcite_data_2.sql` is still needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tristaZero commented on pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
Posted by GitBox <gi...@apache.org>.
tristaZero commented on pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249#issuecomment-773772327
Hi @lmhmhl ,
> I have tried a lot but all failed, so this PR is just a compromise, I am in confused now, I would be very glad if u can give me some advices.
I know this task is a little tricky, but that's why I want to have you taking charge of it. : ) Basically, I believe you can do better than other newbies.
Secondly, I suppose we can get more things while doing community tasks. So please take it easy. If any confusion, we can discuss it and figure it out.
I am unsure whether I got your point wholly. However, the primary blocker you met looks like the function of `ShardingSphereDataSource` (Virtual DB) and the `Map<String, DataSource>` (Actual DBs) from `conn.getDataSourceMap()`.
### Map<String, DataSource> for setting up test env
We use all the actual data sources to prepare initial data for databases. There is unnecessary to consider `ShardingSphereDataSource`, therefore I think `initTable()` has no critical issues. Just use them to execute the corresponding initialization scrip.
### ShardingSphereDataSource for test
After having the expected test databases, we will use `ShardingSphereDataSource` to test `logic SQLs` in UT.
This phase's key point is to write the configuration file, which means configure logic DB, table, columns with these actual ones. Definitely, this part of the work mainly needs your deep understanding of different configurations. If you have any questions about this one, please contact me.
Could you figure out which one is your concern?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] lmhmhl commented on pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
Posted by GitBox <gi...@apache.org>.
lmhmhl commented on pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249#issuecomment-773729643
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tristaZero commented on pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
Posted by GitBox <gi...@apache.org>.
tristaZero commented on pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249#issuecomment-773772327
Hi @lmhmhl ,
> I have tried a lot but all failed, so this PR is just a compromise, I am in confused now, I would be very glad if u can give me some advices.
I know this task is a little tricky, but that's why I want to have you taking charge of it. : ) Basically, I believe you can do better than other newbies.
Secondly, I suppose we can get more things while doing community tasks. So please take it easy. If any confusion, we can discuss it and figure it out.
I am unsure whether I got your point wholly. However, the primary blocker you met looks like the function of `ShardingSphereDataSource` (Virtual DB) and the `Map<String, DataSource>` (Actual DBs) from `conn.getDataSourceMap()`.
### Map<String, DataSource> for setting up test env
We use all the actual data sources to prepare initial data for databases. There is unnecessary to consider `ShardingSphereDataSource`, therefore I think `initTable()` has no critical issues. Just use them to execute the corresponding initialization scrip.
### ShardingSphereDataSource for test
After having the expected test databases, we will use `ShardingSphereDataSource` to test `logic SQLs` in UT.
This phase's key point is to write the configuration file, which means configure logic DB, table, columns with these actual ones. Definitely, this part of the work mainly needs your deep understanding of different configurations. If you have any questions about this one, please contact me.
Could you figure out which one is your concern?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tristaZero merged pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
Posted by GitBox <gi...@apache.org>.
tristaZero merged pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] lmhmhl commented on pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
Posted by GitBox <gi...@apache.org>.
lmhmhl commented on pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249#issuecomment-773826359
> Hi @lmhmhl ,
>
> > I have tried a lot but all failed, so this PR is just a compromise, I am in confused now, I would be very glad if u can give me some advices.
>
> I know this task is a little tricky, but that's why I want to have you taking charge of it. : ) Basically, I believe you can do better than other newbies.
> Secondly, I suppose we can get more things while doing community tasks. So please take it easy. If any confusion, we can discuss it and figure it out.
>
> I am unsure whether I got your point wholly. However, the primary blocker you met looks like the function of `ShardingSphereDataSource` (Virtual DB) and the `Map<String, DataSource>` (Actual DBs) from `conn.getDataSourceMap()`.
>
> ### Map<String, DataSource> for setting up test env
> We use all the actual data sources to prepare initial data for databases. There is unnecessary to consider `ShardingSphereDataSource`, therefore I think `initTable()` has no critical issues. Just use them to execute the corresponding initialization scrip.
>
> ### ShardingSphereDataSource for test
> After having the expected test databases, we will use `ShardingSphereDataSource` to test `logic SQLs` in UT.
> This phase's key point is to write the configuration file, which means configure logic DB, table, columns with these actual ones. Definitely, this part of the work mainly needs your deep understanding of different configurations. If you have any questions about this one, please contact me.
> Could you figure out which one is your concern?
@tristaZero
Thanks for you comment, which help me make it clear and boost my confidence, I raise a new PR, please review it, I have to say you help me a lot. Thx.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tristaZero commented on a change in pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
Posted by GitBox <gi...@apache.org>.
tristaZero commented on a change in pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249#discussion_r568295546
##########
File path: shardingsphere-jdbc/shardingsphere-jdbc-core/src/test/resources/config/config-calcite.yaml
##########
@@ -23,10 +23,54 @@ rules:
tableStrategy:
standard:
shardingColumn: item_id
- shardingAlgorithmName: table_inline
+ shardingAlgorithmName: table_inline_item_id
+ t_user_encrypt_calcite_sharding:
+ actualDataNodes: calcite_jdbc_1.t_user_encrypt_calcite_sharding_${0..1}
Review comment:
> calcite_jdbc_1.t_user_encrypt_calcite_sharding_${0..1}
It is supposed to be `calcite_ds.t_user_encrypt_calcite_sharding_${0..1}` to leverage `replicaQuery` power.
##########
File path: shardingsphere-jdbc/shardingsphere-jdbc-core/src/test/resources/config/config-calcite.yaml
##########
@@ -23,10 +23,54 @@ rules:
tableStrategy:
standard:
shardingColumn: item_id
- shardingAlgorithmName: table_inline
+ shardingAlgorithmName: table_inline_item_id
+ t_user_encrypt_calcite_sharding:
+ actualDataNodes: calcite_jdbc_1.t_user_encrypt_calcite_sharding_${0..1}
+ tableStrategy:
+ standard:
+ shardingColumn: user_id
+ shardingAlgorithmName: table_inline_user_id
shardingAlgorithms:
- table_inline:
+ table_inline_item_id:
type: INLINE
props:
algorithm-expression: t_order_item_calcite_sharding_${item_id % 2}
+ table_inline_user_id:
+ type: INLINE
+ props:
+ algorithm-expression: t_user_encrypt_calcite_sharding_${user_id % 2}
+ - !ENCRYPT
+ encryptors:
+ encryptor_aes:
+ type: aes
+ props:
+ aes-key-value: 123456abc
+ tables:
+ t_user_encrypt_calcite:
+ columns:
+ pwd:
+ plainColumn: plain_pwd
+ cipherColumn: cipher_pwd
+ encryptorName: encryptor_aes
+ t_user_encrypt_calcite_sharding:
+ columns:
+ pwd:
+ plainColumn: plain_pwd
+ cipherColumn: cipher_pwd
+ encryptorName: encryptor_aes
+ - !REPLICA_QUERY
+ dataSources:
+ calcite_ds:
+ name: calcite_ds
+ primaryDataSourceName:
+ - calcite_jdbc_1
+ replicaDataSourceNames:
+ - calcite_jdbc_2
+ loadBalancerName: roundRobin
+ loadBalancers:
+ roundRobin:
+ type: ROUND_ROBIN
+props:
+ sql-show: true
+ query-with-cipher-column: false
Review comment:
Could we set `query-with-cipher-column` = `true` to indeed examine whether this SQL federation feature can work well with the encryption rule?
##########
File path: shardingsphere-jdbc/shardingsphere-jdbc-core/src/test/resources/config/config-calcite.yaml
##########
@@ -23,10 +23,54 @@ rules:
tableStrategy:
standard:
shardingColumn: item_id
- shardingAlgorithmName: table_inline
+ shardingAlgorithmName: table_inline_item_id
+ t_user_encrypt_calcite_sharding:
+ actualDataNodes: calcite_jdbc_1.t_user_encrypt_calcite_sharding_${0..1}
+ tableStrategy:
+ standard:
+ shardingColumn: user_id
+ shardingAlgorithmName: table_inline_user_id
shardingAlgorithms:
- table_inline:
+ table_inline_item_id:
type: INLINE
props:
algorithm-expression: t_order_item_calcite_sharding_${item_id % 2}
+ table_inline_user_id:
+ type: INLINE
+ props:
+ algorithm-expression: t_user_encrypt_calcite_sharding_${user_id % 2}
+ - !ENCRYPT
+ encryptors:
+ encryptor_aes:
+ type: aes
+ props:
+ aes-key-value: 123456abc
+ tables:
+ t_user_encrypt_calcite:
+ columns:
+ pwd:
+ plainColumn: plain_pwd
+ cipherColumn: cipher_pwd
+ encryptorName: encryptor_aes
+ t_user_encrypt_calcite_sharding:
+ columns:
+ pwd:
+ plainColumn: plain_pwd
+ cipherColumn: cipher_pwd
+ encryptorName: encryptor_aes
+ - !REPLICA_QUERY
+ dataSources:
+ calcite_ds:
+ name: calcite_ds
+ primaryDataSourceName:
+ - calcite_jdbc_1
Review comment:
If there is replication relationship between `calcite_jdbc_1` and `calcite_jdbc_2`, maybe `calcite_data_2.sql` is still needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] lmhmhl edited a comment on pull request #9249: Test encrypt rule, replicaQuery rule and sharding rule all exist when do query in calcite.
Posted by GitBox <gi...@apache.org>.
lmhmhl edited a comment on pull request #9249:
URL: https://github.com/apache/shardingsphere/pull/9249#issuecomment-773826359
@tristaZero
Thanks for you comment, which help me make it clear and boost my confidence, I raise a new PR, please review it, I have to say you help me a lot. Thx.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org