You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "liuxiaocs7 (via GitHub)" <gi...@apache.org> on 2023/03/28 03:58:28 UTC
[GitHub] [shardingsphere] liuxiaocs7 opened a new issue, #24874: CDC client Usage
liuxiaocs7 opened a new issue, #24874:
URL: https://github.com/apache/shardingsphere/issues/24874
## Question
Hello, community, I try to use the new cdc feature, but it cannot be used normally, the steps are as follows:
First i use `apache-shardingsphere-5.3.1-shardingsphere-proxy-bin`, conf:
```yaml
mode:
type: Cluster
repository:
type: ZooKeeper
props:
namespace: governance_ds
server-lists: localhost:2181
retryIntervalMilliseconds: 500
timeToLiveSeconds: 60
maxRetries: 3
operationTimeoutMilliseconds: 500
authority:
users:
- user: root@%
password: root
privilege:
type: ALL_PERMITTED
cdc:
enabled: true
port: 33071
```
sharding conf:
```yaml
databaseName: sharding_db
dataSources:
ds_0:
url: jdbc:mysql://127.0.0.1:3306/sparktest?serverTimezone=UTC&useSSL=false
username: root
password: lx123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
```
ShardingSphere Proxy runs successfully
```shell
- special exclusive bonus offers not available to others
- everything you need to get the most out of using Atomikos!
[INFO ] 2023-03-28 11:43:09.579 [main] o.apache.curator.utils.Compatibility - Using org.apache.zookeeper.server.quorum.MultipleAddresses
[INFO ] 2023-03-28 11:43:09.591 [main] o.a.s.d.p.c.e.PipelineJobWorker - start worker initialization
[INFO ] 2023-03-28 11:43:09.598 [main] o.a.s.d.p.c.e.PipelineJobWorker - worker initialization done
[INFO ] 2023-03-28 11:43:09.642 [main] o.a.s.p.v.ShardingSphereProxyVersion - Database name is `MySQL`, version is `8.0.26`, database name is `sharding_db`
Thanks for using Atomikos! This installation is not registered yet.
REGISTER FOR FREE at http://www.atomikos.com/Main/RegisterYourDownload and receive:
- tips & advice
- working demos
- access to the full documentation
- special exclusive bonus offers not available to others
- everything you need to get the most out of using Atomikos!
[INFO ] 2023-03-28 11:43:10.497 [main] o.a.s.p.frontend.ShardingSphereProxy - ShardingSphere-Proxy Cluster mode started successfully
```
But when i run Bootstrap under `pipeline/cdc/example`
```java
public static void main(final String[] args) {
StartCDCClientParameter parameter = new StartCDCClientParameter(records -> log.info("records: {}", records));
parameter.setAddress("127.0.0.1");
parameter.setPort(33071);
parameter.setUsername("root");
parameter.setPassword("root");
parameter.setDatabase("sharding_db");
parameter.setFull(true);
parameter.setSchemaTables(Collections.singletonList(SchemaTable.newBuilder().setTable("t_user").build()));
CDCClient cdcClient = new CDCClient(parameter);
cdcClient.start();
```
it got errors as follow:
```shell
11:49:21.877 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.handler.LoginRequestHandler - Server greeting result, server version: 5.3.1, protocol version: 1
11:49:21.948 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.handler.LoginRequestHandler - Login success, username root
11:49:21.960 [nioEventLoopGroup-2-1] ERROR org.apache.shardingsphere.data.pipeline.cdc.client.handler.CDCRequestHandler - received error response request_id: "e172b6e6-2c2a-4696-a3bd-29440fe77d0c"
status: FAILED
error_code: "2"
error_message: "Illegal create subscription request parameter"
```
---
I also tried the latest version on the master branch, `apache-shardingsphere-5.3.2-SNAPSHOT-shardingsphere-proxy-bin`
server.yaml:
```yaml
mode:
type: Cluster
repository:
type: ZooKeeper
props:
namespace: governance_ds
server-lists: localhost:2181
retryIntervalMilliseconds: 500
timeToLiveSeconds: 60
maxRetries: 3
operationTimeoutMilliseconds: 500
authority:
users:
- user: root@%
password: root
privilege:
type: ALL_PERMITTED
props:
cdc-server-port: 33071 # CDC server port
```
sharding conf:
```yaml
databaseName: sharding_db
dataSources:
ds_0:
url: jdbc:mysql://127.0.0.1:3306/sparktest?serverTimezone=UTC&useSSL=false
username: root
password: lx123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
```
Once restarted directly, we got:
```
Exception in thread "main" Can't construct a java object for !CDC; exception=Invalid tag: !CDC
in 'string', line 7, column 3:
- !CDC
^
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:364)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObjectNoCheck(BaseConstructor.java:270)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:253)
```
Re-run after deleting zk's `governance_ds` node, .ShardingSphereProxy runs successfully.
```shell
[INFO ] 2023-03-28 11:54:41.028 [main] o.a.s.d.p.c.e.PipelineJobWorker - start worker initialization
[INFO ] 2023-03-28 11:54:41.033 [main] o.a.s.d.p.c.e.PipelineJobWorker - worker initialization done
[INFO ] 2023-03-28 11:54:41.075 [main] o.a.s.p.v.ShardingSphereProxyVersion - Database name is `MySQL`, version is `8.0.26`, database name is `sharding_db`
Thanks for using Atomikos! This installation is not registered yet.
REGISTER FOR FREE at http://www.atomikos.com/Main/RegisterYourDownload and receive:
- tips & advice
- working demos
- access to the full documentation
- special exclusive bonus offers not available to others
- everything you need to get the most out of using Atomikos!
[INFO ] 2023-03-28 11:54:41.640 [main] o.a.s.p.frontend.ShardingSphereProxy - ShardingSphere-Proxy Cluster mode started successfully
```
But when i start `Bootstrap`, i got another error:
```
11:55:39.913 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.handler.LoginRequestHandler - Server greeting result, server version: 5.3.2-SNAPSHOT, protocol version: 1
11:55:39.958 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.handler.LoginRequestHandler - Login success, username root
11:55:39.971 [nioEventLoopGroup-2-1] ERROR org.apache.shardingsphere.data.pipeline.cdc.client.handler.CDCRequestHandler - received error response status: FAILED
error_code: "HY000"
error_message: "Rule `ShardingRule` should have and only have one instance."
```
I am wondering where my configuration is wrong, thank you!
---
My Environments:
Win10
shardingsphere-proxy run on windows by `start.bat`
```
java version "17.0.6" 2023-01-17 LTS
Java(TM) SE Runtime Environment (build 17.0.6+9-LTS-190)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.6+9-LTS-190, mixed mode, sharing)
```
```
Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)
Maven home: D:\Development\Tools\Maven\apache-maven-3.8.1\bin\..
Java version: 17.0.6, vendor: Oracle Corporation, runtime: D:\Development\Tools\Java\jdk-17
Default locale: zh_CN, platform encoding: GBK
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
```
Zk deployed by Docker.
**For English only**, other languages will not accept.
Before asking a question, make sure you have:
- Googled your question.
- Searched open and closed [GitHub issues](https://github.com/apache/shardingsphere/issues).
- Read documentation: [ShardingSphere Doc](https://shardingsphere.apache.org/document/current/en/overview).
Please pay attention on issues you submitted, because we maybe need more details.
If no response anymore and we cannot reproduce it on current information, we will **close it**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on issue #24874: CDC client Usage
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on issue #24874:
URL: https://github.com/apache/shardingsphere/issues/24874#issuecomment-1486565230
Hi @liuxiaocs7 , nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] liuxiaocs7 commented on issue #24874: CDC client Usage
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on issue #24874:
URL: https://github.com/apache/shardingsphere/issues/24874#issuecomment-1486190511
> ```
> 11:55:39.913 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.handler.LoginRequestHandler - Server greeting result, server version: 5.3.2-SNAPSHOT, protocol version: 1
> 11:55:39.958 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.handler.LoginRequestHandler - Login success, username root
> 11:55:39.971 [nioEventLoopGroup-2-1] ERROR org.apache.shardingsphere.data.pipeline.cdc.client.handler.CDCRequestHandler - received error response status: FAILED
> error_code: "HY000"
> error_message: "Rule `ShardingRule` should have and only have one instance."
> ```
>
> Could you check you sharding rule config? also you can debug at `org.apache.shardingsphere.proxy.backend.handler.cdc.CDCBackendHandler#streamData`, and tell us why sharding rule isn't 1
I would like to ask whether the sharding rule config is in `config-sharding.yaml` and this code blocks:
```
#rules:
#- !SHARDING
# tables:
# t_order:
# actualDataNodes: ds_${0..1}.t_order_${0..1}
# tableStrategy:
# standard:
# shardingColumn: order_id
# shardingAlgorithmName: t_order_inline
# keyGenerateStrategy:
# column: order_id
# keyGeneratorName: snowflake
# auditStrategy:
# auditorNames:
# - sharding_key_required_auditor
# allowHintDisable: true
# t_order_item:
# actualDataNodes: ds_${0..1}.t_order_item_${0..1}
# tableStrategy:
# standard:
# shardingColumn: order_id
# shardingAlgorithmName: t_order_item_inline
# keyGenerateStrategy:
# column: order_item_id
# keyGeneratorName: snowflake
# bindingTables:
# - t_order,t_order_item
# defaultDatabaseStrategy:
# standard:
# shardingColumn: user_id
# shardingAlgorithmName: database_inline
# defaultTableStrategy:
# none:
# defaultAuditStrategy:
# auditorNames:
# - sharding_key_required_auditor
# allowHintDisable: true
#
# shardingAlgorithms:
# database_inline:
# type: INLINE
# props:
# algorithm-expression: ds_${user_id % 2}
# t_order_inline:
# type: INLINE
# props:
# algorithm-expression: t_order_${order_id % 2}
# t_order_item_inline:
# type: INLINE
# props:
# algorithm-expression: t_order_item_${order_id % 2}
#
# keyGenerators:
# snowflake:
# type: SNOWFLAKE
#
# auditors:
# sharding_key_required_auditor:
# type: DML_SHARDING_CONDITIONS
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] liuxiaocs7 commented on issue #24874: CDC client Usage
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on issue #24874:
URL: https://github.com/apache/shardingsphere/issues/24874#issuecomment-1486506701
Hi, @azexcy and @sandynz, thanks for your help!
I found my mistake, i modified `config-sharding.yaml` under `apache-shardingsphere-5.3.1-shardingsphere-proxy-bin` rather than `apache-shardingsphere-5.3.2-SNAPSHOT-shardingsphere-proxy-bin`.
After add these conf to `config-sharding.yaml` under `apache-shardingsphere-5.3.2-SNAPSHOT-shardingsphere-proxy-bin`
```yaml
rules:
- !SHARDING
tables:
t_user:
actualDataNodes: ds_0.t_user
```
I can see output in console:
```shell
16:38:33.509 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.handler.LoginRequestHandler - Server greeting result, server version: 5.3.2-SNAPSHOT, protocol version: 1
16:38:33.557 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.handler.LoginRequestHandler - Login success, username root
16:38:43.986 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.example.Bootstrap - records: [before {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
before {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
after {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Int64Value"
value: "\b\002"
}
}
after {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.StringValue"
value: "\n\003bbb"
}
}
meta_data {
database: "sharding_db"
schema: ""
table: "t_user"
}
data_change_type: INSERT
, before {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
before {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
after {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Int64Value"
value: "\b\004"
}
}
after {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.StringValue"
value: "\n\004dddd"
}
}
meta_data {
database: "sharding_db"
schema: ""
table: "t_user"
}
data_change_type: INSERT
, before {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
before {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
after {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Int64Value"
value: "\b\003"
}
}
after {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.StringValue"
value: "\n\003ccc"
}
}
meta_data {
database: "sharding_db"
schema: ""
table: "t_user"
}
data_change_type: INSERT
, before {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
before {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
after {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Int64Value"
value: "\b\001"
}
}
after {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.StringValue"
value: "\n\003aaa"
}
}
meta_data {
database: "sharding_db"
schema: ""
table: "t_user"
}
data_change_type: INSERT
, before {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
before {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
after {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Int64Value"
value: "\b\005"
}
}
after {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.StringValue"
value: "\n\003eee"
}
}
meta_data {
database: "sharding_db"
schema: ""
table: "t_user"
}
data_change_type: INSERT
]
```
This matches the database initial information, and i try to add a new record:
```
mysql> insert into t_user(id, name) values(6, "ffffff");
Query OK, 1 row affected (0.01 sec)
```
```
16:43:07.107 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.example.Bootstrap - records: [before {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
before {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.Empty"
}
}
after {
name: "id"
value {
type_url: "type.googleapis.com/google.protobuf.Int32Value"
value: "\b\006"
}
}
after {
name: "name"
value {
type_url: "type.googleapis.com/google.protobuf.BytesValue"
value: "\n\006ffffff"
}
}
meta_data {
database: "sharding_db"
schema: ""
table: "t_user"
}
data_change_type: INSERT
]
```
Incremental changes can be seen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on issue #24874: CDC client Usage
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on issue #24874:
URL: https://github.com/apache/shardingsphere/issues/24874#issuecomment-1486192787
Hi @liuxiaocs7 , yes, `rules` should be configured in `config-sharding.yaml`. Or else use DistSQL totally.
And you need to clean namespace in ZooKeeper first, if you continue edit `config-sharding.yaml`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on issue #24874: CDC client Usage
Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on issue #24874:
URL: https://github.com/apache/shardingsphere/issues/24874#issuecomment-1486188244
```
11:55:39.913 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.handler.LoginRequestHandler - Server greeting result, server version: 5.3.2-SNAPSHOT, protocol version: 1
11:55:39.958 [nioEventLoopGroup-2-1] INFO org.apache.shardingsphere.data.pipeline.cdc.client.handler.LoginRequestHandler - Login success, username root
11:55:39.971 [nioEventLoopGroup-2-1] ERROR org.apache.shardingsphere.data.pipeline.cdc.client.handler.CDCRequestHandler - received error response status: FAILED
error_code: "HY000"
error_message: "Rule `ShardingRule` should have and only have one instance."
```
Could you check you sharding rule config? also you can debug at `org.apache.shardingsphere.proxy.backend.handler.cdc.CDCBackendHandler#streamData`, and tell us why sharding rule isn't 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] liuxiaocs7 commented on issue #24874: CDC client Usage
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on issue #24874:
URL: https://github.com/apache/shardingsphere/issues/24874#issuecomment-1486174426
Hi, @sandynz and @azexcy, could you help take a look in your free time, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] liuxiaocs7 closed issue #24874: CDC client Usage
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 closed issue #24874: CDC client Usage
URL: https://github.com/apache/shardingsphere/issues/24874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org