You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by az...@apache.org on 2023/04/24 08:13:41 UTC
[shardingsphere] branch master updated: Update Pipeline E2E Test doc (#25303)
This is an automated email from the ASF dual-hosted git repository.
azexin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/shardingsphere.git
The following commit(s) were added to refs/heads/master by this push:
new 198d82e3816 Update Pipeline E2E Test doc (#25303)
198d82e3816 is described below
commit 198d82e381669c5f3e4f6ff2dfe4e696ecfacdda
Author: Hongsheng Zhong <zh...@apache.org>
AuthorDate: Mon Apr 24 16:13:34 2023 +0800
Update Pipeline E2E Test doc (#25303)
* Rename scaling-integration-test/ to pipeline-e2e-test/
* Update Pipeline E2E Test doc
* Revert "Rename scaling-integration-test/ to pipeline-e2e-test/"
This reverts commit 643dafab3a6d6ad30e5d71d9d9145c1f9473c7bb.
---
.../scaling-integration-test/_index.cn.md | 71 +++++++++++----------
.../scaling-integration-test/_index.en.md | 74 +++++++++++-----------
2 files changed, 72 insertions(+), 73 deletions(-)
diff --git a/docs/document/content/test-manual/scaling-integration-test/_index.cn.md b/docs/document/content/test-manual/scaling-integration-test/_index.cn.md
index 1fa7ceaca48..90a422bf7c6 100644
--- a/docs/document/content/test-manual/scaling-integration-test/_index.cn.md
+++ b/docs/document/content/test-manual/scaling-integration-test/_index.cn.md
@@ -1,58 +1,57 @@
+++
pre = "<b>6.4. </b>"
-title = "Pipeline 集成测试"
+title = "Pipeline E2E 测试"
weight = 4
+++
## 测试目的
-验证数据迁移以及依赖模块功能的正确性。
+验证 pipeline 各个场景的功能正确性。
-## 测试环境
+## 测试环境类型
-目前支持 Native 和 Docker 两种环境。
+目前支持 Native 和 Docker。
-1. Native 环境直接运行在开发者提供的测试环境中，需要用户自己启动 ShardingSphere-Proxy 和对应的数据库实例，适于调试场景；
-1. Docker 环境由 Maven 运行，适用于云编译环境和测试 ShardingSphere-Proxy 的场景，如：GitHub Action。
+1. Native：运行在开发者本机环境。需要在本机启动 ShardingSphere-Proxy 实例（通过 proxy 安装包或者在 IDE 运行 `org.apache.shardingsphere.proxy.Bootstrap`）和数据库实例。一般用于本机调试。
+1. Docker：运行在 Maven 插件拉起的 Docker 环境。一般用于 GitHub Action，也可以在本机运行。
-目前支持的数据库类型：MySQL、PostgreSQL、openGauss。
+支持的数据库：MySQL、PostgreSQL、openGauss。
## 使用指南
-模块路径 `shardingsphere-test/shardingsphere-test-e2e/shardingsphere-test-e2e-pipeline` 。
+模块路径 `test/e2e/operation/pipeline`。
### 环境配置
-`${DOCKER-IMAGE}` 表示 docker 镜像名称，如 `mysql:8` 。 `${DATABASE-TYPE}` 表示数据库类型。
-目录：`src/test/resources/env`
-- `it-env.properties`：集成测试启动参数。
-- `${DATABASE-TYPE}/server.yaml`：数据库对应的 ShardingSphere-Proxy 配置文件。
-- `${DATABASE-TYPE}/initdb.sql`：数据库初始化 SQL。
-- `${DATABASE-TYPE}/*.cnf,*.conf`：以 cnf 或者 conf 结尾的文件，是数据库的配置文件，用于 Docker 挂载。
-- `common/command.xml`：测试中用到的DistSQL。
-- `scenario/`：存放测试场景中的 SQL。
+
+`${DOCKER-IMAGE}` 表示 docker 镜像名称，如 `mysql:5.7` 。 `${DATABASE-TYPE}` 表示数据库类型。
+
+目录：`src/test/resources/env/`
+- `it-env.properties`：环境配置文件。
+- `${DATABASE-TYPE}/server.yaml`：ShardingSphere-Proxy 配置文件。
+- `${DATABASE-TYPE}/initdb.sql`：数据库初始化 SQL 文件。
+- `${DATABASE-TYPE}/*.cnf,*.conf`：数据库配置文件。
+- `common/*.xml`：测试用到的 DistSQL 文件。
+- `scenario/`：各个测试场景的 SQL 文件。
### 测试用例
-目前所有的测试用例，都直接继承自 `BaseExtraSQLITCase`，间接继承了 `BaseITCase`。
-- `BaseITCase`：提供了通用方法给子类
-- `BaseExtraSQLITCase`：提供了建表、CRUD 语句执行方法
-用例示例：MySQLGeneralPipelineE2EIT。
+用例示例：MySQLMigrationGeneralE2EIT。
覆盖的功能点如下：
- 库级别迁移（所有表）
- 表级别迁移（任意多个表）
- 迁移数据一致性校验
-- 数据迁移过程中支持停写
- 数据迁移过程中支持重启
- 数据迁移支持整型主键
- 数据迁移支持字符串主键
- 使用非管理员账号进行数据迁移
### 运行测试用例
-`it-env.properties` 所有属性值都可以通过 Maven 命令行 `-D` 的方式传入，优先级高于配置文件。
+
+`it-env.properties` 所有属性都可以通过 Maven 命令行 `-D` 的方式传入，优先级高于配置文件。
#### Native 环境启动
-使用者在本地提前启动 ShardingSphere-Proxy 以及依赖的配置中心（如 ZooKeeper）和数据库。
+开发者在本地提前启动 ShardingSphere-Proxy、注册中心（如 ZooKeeper）和数据库。
要求 ShardingSphere-Proxy 的端口是 3307。
以 MySQL 为例，`it-env.properties` 可以配置如下：
```
@@ -63,31 +62,33 @@ pipeline.it.native.mysql.password=root
pipeline.it.native.mysql.port=3306
```
-找到对应的用例，在 IDE 下使用 Junit 的方式启动即可。
+找到对应的测试类，在 IDE 启动运行。
-#### Docker环境启动
+#### Docker 环境启动
+
+参考 `.github/workflows/e2e-pipeline.yml`。
第一步：打包镜像
```
-./mvnw -B clean install -am -pl shardingsphere-test/shardingsphere-test-e2e/shardingsphere-test-e2e-pipeline -Pit.env.docker -DskipTests
+./mvnw -B clean install -am -pl test/e2e/operation/pipeline -Pit.env.docker -DskipTests
```
-运行以上命令会构建出一个用于集成测试的 Docker 镜像 apache/shardingsphere-proxy-test:latest，该镜像设置了远程调试的端口，默认是3308。 如果仅修改了测试代码，可以复用已有的测试镜像，无须重新构建。
+运行以上命令会构建出一个用于 E2E 测试的 Docker 镜像 `apache/shardingsphere-proxy-test:latest`。
-Docker 模式下，如果需要对 Docker 镜像启动参数进行调整，可以对修改 ShardingSphereProxyDockerContainer 文件中的相关配置。
+该镜像设置了远程调试的端口，默认是 `3308`。
-ShardingSphere-Proxy 输出的日志带有 :Pipeline-Proxy 前缀。
+如果仅修改了测试代码，可以复用已有的测试镜像。
-使用 Maven 的方式运行用例。以 MySQL 为例：
+第二步：修改 `it-env.properties` 配置
```
-./mvnw -nsu -B install -f shardingsphere-test/shardingsphere-test-e2e/shardingsphere-test-e2e-pipeline/pom.xml -Dpipeline.it.env.type=DOCKER -Dpipeline.it.docker.mysql.version=${image-name}
+pipeline.it.env.type=DOCKER
+pipeline.it.docker.mysql.version=mysql:5.7
```
-也可以使用 IDE 的方式运行用例。`it-env.properties` 可以配置如下：
+第三步：通过 Maven 运行测试用例。以 MySQL 为例：
```
-pipeline.it.env.type=DOCKER
-pipeline.it.docker.mysql.version=mysql:5.7
-```
\ No newline at end of file
+./mvnw -nsu -B install -f test/e2e/operation/pipeline/pom.xml -Dpipeline.it.env.type=docker -Dpipeline.it.docker.mysql.version=mysql:5.7
+```
diff --git a/docs/document/content/test-manual/scaling-integration-test/_index.en.md b/docs/document/content/test-manual/scaling-integration-test/_index.en.md
index c3b5894a0d9..01a7ec548ca 100644
--- a/docs/document/content/test-manual/scaling-integration-test/_index.en.md
+++ b/docs/document/content/test-manual/scaling-integration-test/_index.en.md
@@ -1,48 +1,44 @@
+++
pre = "<b>6.4. </b>"
-title = "Pipeline Integration Test"
+title = "Pipeline E2E Test"
weight = 4
+++
## Objectives
-Verify the functional correctness of data migration and dependency modules.
+Verify the functional correctness of pipeline scenarios.
-## Test environment
+## Test environment type
-Currently, Native and Docker environments are supported.
-1. The Native environment runs directly in the test environment provided by the developer, and users need to start ShardingSphere-Proxy and the corresponding database instance by themselves, which is suitable for debugging scenarios.
-2. The Docker environment is run by Maven, which is suitable for cloud compilation environment and ShardingSphere-Proxy testing scenarios, such as GitHub Action.
+Currently, Native and Docker are available.
+1. Native : Run on developer local machine. Need to start ShardingSphere-Proxy instance (run proxy installation package or run `org.apache.shardingsphere.proxy.Bootstrap` in IDE) and database instance by developer. It could be used for local debugging.
+2. Docker : Run on Docker started by Maven plugin. It could be used for GitHub Actions, and it could be used for local debugging too.
-Currently, you can use MySQL, PostgreSQL and openGuass databases.
+Supported databases: MySQL, PostgreSQL and openGuass.
## User guide
-Module path: `shardingsphere-test/shardingsphere-test-e2e/shardingsphere-test-e2e-pipeline`.
+Module path: `test/e2e/operation/pipeline`.
### Environment setup
-`${DOCKER-IMAGE}` refers to the name of a Docker mirror, such as `mysql:8`. `${DATABASE-TYPE}` refers to database types.
-Directory: `src/test/resources/env`
-- `it-env.properties`: the startup parameters of integration testing.
-- `${DATABASE-TYPE}/server.yaml`: ShardingSphere-Proxy configuration file corresponding to the database.
-- `${DATABASE-TYPE}/initdb.sql`: The database initializes SQL.
-- `${DATABASE-TYPE}/*.cnf,*.conf`: Files ending with cnf or conf are database configuration files for Docker mount.
-- `common/command.xml`: The DistSQL used in the test.
-- `scenario/`: Store SQL in the test scenarios.
+`${DOCKER-IMAGE}` refers to the name of a Docker mirror, such as `mysql:5.7`. `${DATABASE-TYPE}` refers to database types.
-### Test case
+Directory: `src/test/resources/env/`
+- `it-env.properties`: Environment setup configuration file.
+- `${DATABASE-TYPE}/server.yaml`: ShardingSphere-Proxy configuration fi;e.
+- `${DATABASE-TYPE}/initdb.sql`: Database initialization SQL file.
+- `${DATABASE-TYPE}/*.cnf,*.conf`: Database configuration files.
+- `common/*.xml`: DistSQL files.
+- `scenario/`: SQL files for different scenarios.
-Currently, all the test cases are directly inherited from `BaseExtraSQLITCase` and indirectly inherited from `BaseITCase`.
-- `BaseITCase`: Provide generic methods for sub-class.
-- `BaseExtraSQLITCase`: Provide table creation and CRUD statement execution methods.
+### Test case
-Test case example: MySQLGeneralPipelineE2EIT.
+Test case example: MySQLMigrationGeneralE2EIT.
Functions included:
- Database-level migration (all tables).
- Table-level migration (any number).
- Verify migration data consistency.
-- Stop writing is supported during data migration.
- Support restart during data migration.
- Support integer primary keys during data migration.
- Support string primary keys during data migration.
@@ -50,13 +46,13 @@ Functions included:
### Running the test case
-All property values of `it-env.properties` can be introduced by the Maven command line `-D`, and its priority is higher than that of the configuration file.
+Any property of `it-env.properties` could be defined by Maven command line parameter `-D`, and its priority is higher than configuration file.
#### Native environment setup
-The user starts ShardingSphere-Proxy locally in advance, along with dependent configuration centers (such as ZooKeeper) and databases.
-The port required for ShardingSphere-Proxy is 3307.
-Take MySQL as an example, `it-env.properties` can be configured as follows:
+Start ShardingSphere-Proxy, registry center (e.g. ZooKeeper) and database.
+Suppose ShardingSphere-Proxy port is 3307.
+Take MySQL as an example, `it-env.properties` could be configured as follows:
```
pipeline.it.env.type=NATIVE
pipeline.it.native.database=mysql
@@ -65,32 +61,34 @@ pipeline.it.native.mysql.password=root
pipeline.it.native.mysql.port=3306
```
-Find the appropriate test case and start it with Junit under the IDE.
+Find test class and start it on IDE.
#### Docker environment setup
-Step 1: Package mirror.
+Refer to `.github/workflows/e2e-pipeline.yml` for more details.
+
+Step 1: Build docker image.
```
-./mvnw -B clean install -am -pl shardingsphere-test/shardingsphere-test-e2e/shardingsphere-test-e2e-pipeline -Pit.env.docker -DskipTests
+./mvnw -B clean install -am -pl test/e2e/operation/pipeline -Pit.env.docker -DskipTests
```
-Running the above command will build a Docker mirror apache/shardingsphere-proxy-test:latest used for integration testing.
-The mirror sets the port for remote debugging and the default port is 3308. If only the test code is modified, you can reuse the existing test mirror without rebuilding it.
+Running the above command will build a Docker image `apache/shardingsphere-proxy-test:latest`.
-If you need to adjust Docker mirror startup parameters, you can modify the configuration of the ShardingSphereProxyDockerContainer file.
+The Docker image has port `3308` for remote debugging.
-The output log of ShardingSphere-Proxy has the prefix Pipeline-Proxy.
+If only test code is modified, you could reuse existing docker image.
-Use Maven to run the test cases. Take MySQL as an example:
+Step 2: Configure `it-env.properties`.
```
-./mvnw -nsu -B install -f shardingsphere-test/shardingsphere-test-e2e/shardingsphere-test-e2e-pipeline/pom.xml -Dpipeline.it.env.type=DOCKER -Dpipeline.it.docker.mysql.version=${image-name}
+pipeline.it.env.type=DOCKER
+pipeline.it.docker.mysql.version=mysql:5.7
```
-You can also use IDE to run test cases. `it-env.properties` can be configured as follows:
+Step 3: Run test cases.
+Take MySQL as an example:
```
-pipeline.it.env.type=DOCKER
-pipeline.it.docker.mysql.version=mysql:5.7
+./mvnw -nsu -B install -f test/e2e/operation/pipeline/pom.xml -Dpipeline.it.env.type=docker -Dpipeline.it.docker.mysql.version=mysql:5.7
```