You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by zh...@apache.org on 2022/04/26 14:08:54 UTC
[shardingsphere] branch master updated: Update scaling doc (#17131)
This is an automated email from the ASF dual-hosted git repository.
zhangliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/shardingsphere.git
The following commit(s) were added to refs/heads/master by this push:
new fcb300501e1 Update scaling doc (#17131)
fcb300501e1 is described below
commit fcb300501e13a46773d43fbea2598d669d702622
Author: Hongsheng Zhong <sa...@126.com>
AuthorDate: Tue Apr 26 22:08:48 2022 +0800
Update scaling doc (#17131)
---
.../user-manual/shardingsphere-scaling/build.cn.md | 57 +++++++++++++++++++---
.../user-manual/shardingsphere-scaling/build.en.md | 57 +++++++++++++++++++---
.../user-manual/shardingsphere-scaling/usage.cn.md | 14 +++---
.../user-manual/shardingsphere-scaling/usage.en.md | 10 ++--
4 files changed, 110 insertions(+), 28 deletions(-)
diff --git a/docs/document/content/user-manual/shardingsphere-scaling/build.cn.md b/docs/document/content/user-manual/shardingsphere-scaling/build.cn.md
index 6a7b7056294..51b685095eb 100644
--- a/docs/document/content/user-manual/shardingsphere-scaling/build.cn.md
+++ b/docs/document/content/user-manual/shardingsphere-scaling/build.cn.md
@@ -81,15 +81,19 @@ rules:
chunk-size: # 一次查询操作返回的最大记录数
```
-配置示例：
+`dataConsistencyChecker` 的 `type` 可以通过执行 DistSQL `SHOW SCALING CHECK ALGORITHMS` 查询到。简单对比：
+- `DATA_MATCH`：支持所有数据库，但是性能不是最好的。
+- `CRC32_MATCH`：只支持 `MySQL`，但是性能更好。
+
+自动模式配置示例：
```yaml
rules:
- !SHARDING
# 忽略的配置
- scalingName: default_scaling
+ scalingName: scaling_auto
scaling:
- default_scaling:
+ scaling_auto:
input:
workerThread: 40
batchSize: 1000
@@ -110,13 +114,53 @@ rules:
chunk-size: 1000
```
-以上的 `completionDetector`，`dataConsistencyChecker` 都可以通过实现 SPI 自定义。可以参考现有实现，详情请参见[开发者手册#弹性伸缩](/cn/dev-manual/scaling/)。
+手动模式配置示例：
+```yaml
+rules:
+- !SHARDING
+ # 忽略的配置
+
+ scalingName: scaling_manual
+ scaling:
+ scaling_manual:
+ input:
+ workerThread: 40
+ batchSize: 1000
+ output:
+ workerThread: 40
+ batchSize: 1000
+ streamChannel:
+ type: MEMORY
+ props:
+ block-queue-size: 10000
+ dataConsistencyChecker:
+ type: DATA_MATCH
+ props:
+ chunk-size: 1000
+```
方法2：通过 DistSQL 配置 scaling
-创建 scaling 配置示例：
+自动模式配置示例：
+```sql
+CREATE SHARDING SCALING RULE scaling_auto (
+INPUT(
+ WORKER_THREAD=40,
+ BATCH_SIZE=1000
+),
+OUTPUT(
+ WORKER_THREAD=40,
+ BATCH_SIZE=1000
+),
+STREAM_CHANNEL(TYPE(NAME=MEMORY, PROPERTIES("block-queue-size"=10000))),
+COMPLETION_DETECTOR(TYPE(NAME=IDLE, PROPERTIES("incremental-task-idle-minute-threshold"=30))),
+DATA_CONSISTENCY_CHECKER(TYPE(NAME=DATA_MATCH, PROPERTIES("chunk-size"=1000)))
+);
+```
+
+手动模式配置示例：
```sql
-CREATE SHARDING SCALING RULE default_scaling (
+CREATE SHARDING SCALING RULE scaling_manual (
INPUT(
WORKER_THREAD=40,
BATCH_SIZE=1000
@@ -126,7 +170,6 @@ OUTPUT(
BATCH_SIZE=1000
),
STREAM_CHANNEL(TYPE(NAME=MEMORY, PROPERTIES("block-queue-size"=10000))),
-COMPLETION_DETECTOR(TYPE(NAME=IDLE, PROPERTIES("incremental-task-idle-minute-threshold"=3))),
DATA_CONSISTENCY_CHECKER(TYPE(NAME=DATA_MATCH, PROPERTIES("chunk-size"=1000)))
);
```
diff --git a/docs/document/content/user-manual/shardingsphere-scaling/build.en.md b/docs/document/content/user-manual/shardingsphere-scaling/build.en.md
index 2bea5017fc8..6e902c4fc99 100644
--- a/docs/document/content/user-manual/shardingsphere-scaling/build.en.md
+++ b/docs/document/content/user-manual/shardingsphere-scaling/build.en.md
@@ -80,15 +80,19 @@ rules:
chunk-size: # Maximum records count of a query operation for check
```
-Configuration Example:
+`type` of `dataConsistencyChecker` could be got by executing DistSQL `SHOW SCALING CHECK ALGORITHMS`. Simple comparison:
+- `DATA_MATCH` : Support all types of databases, but it's not the best performant one.
+- `CRC32_MATCH` : Support `MySQL`, performance is better than `DATA_MATCH`.
+
+Auto Mode Configuration Example:
```yaml
rules:
- !SHARDING
# ignored configuration
- scalingName: default_scaling
+ scalingName: scaling_auto
scaling:
- default_scaling:
+ scaling_auto:
input:
workerThread: 40
batchSize: 1000
@@ -109,13 +113,53 @@ rules:
chunk-size: 1000
```
-You could customize `completionDetector`, `dataConsistencyChecker` algorithm by implementing SPI. Current implementation could be referenced, please refer to [Dev Manual#Scaling](/en/dev-manual/scaling/) for more details.
+Manual Mode Configuration Example:
+```yaml
+rules:
+- !SHARDING
+ # ignored configuration
+
+ scalingName: scaling_manual
+ scaling:
+ scaling_manual:
+ input:
+ workerThread: 40
+ batchSize: 1000
+ output:
+ workerThread: 40
+ batchSize: 1000
+ streamChannel:
+ type: MEMORY
+ props:
+ block-queue-size: 10000
+ dataConsistencyChecker:
+ type: DATA_MATCH
+ props:
+ chunk-size: 1000
+```
Way 2: Configure scaling by DistSQL
-Create scaling configuration example:
+Auto Mode Configuration Example:
+```sql
+CREATE SHARDING SCALING RULE scaling_auto (
+INPUT(
+ WORKER_THREAD=40,
+ BATCH_SIZE=1000
+),
+OUTPUT(
+ WORKER_THREAD=40,
+ BATCH_SIZE=1000
+),
+STREAM_CHANNEL(TYPE(NAME=MEMORY, PROPERTIES("block-queue-size"=10000))),
+COMPLETION_DETECTOR(TYPE(NAME=IDLE, PROPERTIES("incremental-task-idle-minute-threshold"=30))),
+DATA_CONSISTENCY_CHECKER(TYPE(NAME=DATA_MATCH, PROPERTIES("chunk-size"=1000)))
+);
+```
+
+Manual Mode Configuration Example:
```sql
-CREATE SHARDING SCALING RULE default_scaling (
+CREATE SHARDING SCALING RULE scaling_manual (
INPUT(
WORKER_THREAD=40,
BATCH_SIZE=1000
@@ -125,7 +169,6 @@ OUTPUT(
BATCH_SIZE=1000
),
STREAM_CHANNEL(TYPE(NAME=MEMORY, PROPERTIES("block-queue-size"=10000))),
-COMPLETION_DETECTOR(TYPE(NAME=IDLE, PROPERTIES("incremental-task-idle-minute-threshold"=3))),
DATA_CONSISTENCY_CHECKER(TYPE(NAME=DATA_MATCH, PROPERTIES("chunk-size"=1000)))
);
```
diff --git a/docs/document/content/user-manual/shardingsphere-scaling/usage.cn.md b/docs/document/content/user-manual/shardingsphere-scaling/usage.cn.md
index dc7b2efba54..61a7c253ced 100644
--- a/docs/document/content/user-manual/shardingsphere-scaling/usage.cn.md
+++ b/docs/document/content/user-manual/shardingsphere-scaling/usage.cn.md
@@ -28,13 +28,13 @@ weight = 2
功能支持情况：
-| 功能 | MySQL | PostgreSQL | openGauss |
-| --------------------- | ------------- | ------------- | ------------- |
+| 功能 | MySQL | PostgreSQL | openGauss |
+| --------------------- | ------------- |--------------| ------------- |
| 全量迁移 | 支持 | 支持 | 支持 |
| 增量迁移 | 支持 | 支持 | 支持 |
-| 自动建表 | 支持 | 不支持 | 支持 |
+| 自动建表 | 支持 | 支持 | 支持 |
| DATA_MATCH一致性校验 | 支持 | 支持 | 支持 |
-| CRC32_MATCH一致性校验 | 支持 | 不支持 | 不支持 |
+| CRC32_MATCH一致性校验 | 支持 | 不支持 | 不支持 |
**注意**：
@@ -281,12 +281,10 @@ mysql> preview SELECT COUNT(1) FROM t_order;
#### 其他 DistSQL
详情请参见 [RAL #弹性伸缩](/cn/user-manual/shardingsphere-proxy/distsql/syntax/ral/#%E5%BC%B9%E6%80%A7%E4%BC%B8%E7%BC%A9)。
-### DistSQL 手动模式接口
-
-数据校验、切换配置等操作可以手动执行。详情请参见：[RAL #弹性伸缩](/cn/user-manual/shardingsphere-proxy/distsql/syntax/ral/#%E5%BC%B9%E6%80%A7%E4%BC%B8%E7%BC%A9)。
-
### DistSQL 手动模式完整流程示例
+手动模式下，数据校验、切换配置等操作可以手动执行。详情请参见：[RAL #弹性伸缩](/cn/user-manual/shardingsphere-proxy/distsql/syntax/ral/#%E5%BC%B9%E6%80%A7%E4%BC%B8%E7%BC%A9)。
+
本示例演示从已有 MySQL 数据库迁移到 proxy。
除了明确说明在 MySQL 执行的 SQL，其他都是在 proxy 执行。
diff --git a/docs/document/content/user-manual/shardingsphere-scaling/usage.en.md b/docs/document/content/user-manual/shardingsphere-scaling/usage.en.md
index 8ee16a65fee..a19dd25da36 100644
--- a/docs/document/content/user-manual/shardingsphere-scaling/usage.en.md
+++ b/docs/document/content/user-manual/shardingsphere-scaling/usage.en.md
@@ -29,10 +29,10 @@ If the backend database is in following table, please download JDBC driver jar a
Supported features:
| Feature | MySQL | PostgreSQL | openGauss |
-| ---------------------------------------- | ------------- | ------------- | ------------- |
+| ---------------------------------------- | ------------- |---------------| ------------- |
| Inventory migration | Supported | Supported | Supported |
| Incremental migration | Supported | Supported | Supported |
-| Create table automatically | Supported | Unsupported | Supported |
+| Create table automatically | Supported | Supported | Supported |
| DATA_MATCH data consistency check | Supported | Supported | Supported |
| CRC32_MATCH data consistency check | Supported | Unsupported | Unsupported |
@@ -283,12 +283,10 @@ mysql> PREVIEW SELECT COUNT(1) FROM t_order;
#### Other DistSQL
Please refer to [RAL#Scaling](/en/user-manual/shardingsphere-proxy/distsql/syntax/ral/#scaling) for more details.
-### DistSQL API for manual mode
-
-Data consistency check and switch configuration could be emitted manually. Please refer to [RAL#Scaling](/en/user-manual/shardingsphere-proxy/distsql/syntax/ral/#scaling) for more details.
-
### DistSQL manual mode whole process example
+On manual mode, data consistency check and switch configuration could be emitted manually. Please refer to [RAL#Scaling](/en/user-manual/shardingsphere-proxy/distsql/syntax/ral/#scaling) for more details.
+
This example show how to migrate data from MySQL to proxy.
Most SQLs should be executed in proxy, except few ones mentioned for MySQL.