You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2020/02/24 08:16:04 UTC
[GitHub] [incubator-shardingsphere] menghaoranss opened a new issue #4438:
Orchestration unit test development guide for new contributors
menghaoranss opened a new issue #4438: Orchestration unit test development guide for new contributors
URL: https://github.com/apache/incubator-shardingsphere/issues/4438
## New contributors guide for unit test task of orchestration
Welcome to Apache ShardingSphere!
Here are some tips for new contributors.
### Prepare the environment
Apache ShardingSphere is based on the Java platform, so you need related environment tools:
-JDK8 version,
-Java development tool IDE, IDEA or Eclipse is recommended, install lombok plugin
-Git, or git integrated in the IDE
-Maven3.5
If running Sharding-Proxy locally, you also need:
-MySQL 5.6 or 5.7 server. If you want to verify PostGreSQL or other database related functions, you need to install the corresponding database software.
-Zookeeper 3.5.6 or other version. If you want to verify the configuration center of Apollo or Nacos, you need to install the corresponding software.
If you want to debug and run UI related projects, you need to install:
-Node 8.11.1 or higher and corresponding NPM 5.6.0 or higher.
### Project instruction
#### Download source code
Project address: https://github.com/apache/incubator-shardingsphere
#### Source code description
The source code includes the following sections:
-Main project: SQL Engine, Sharding-jdbc, Sharding-proxy, Orchestration, etc.
-ShardingSphere-UI: frontend (NodeJS / VUE), backend (Java)
-doc project: Documentation, which is also the official website. Need to use Go-based Hugo tools to compile to static content.
-examples project: use examples.
#### Branch description
We advocate modifying and committing on a dedicated branch, and finally request a merge to the trunk through a pull request (PR).
Currently, the backbone of apache / incubator-shardingsphere is master and the version is 5.0.0-RC [N] -SNAPSHOT.
There is also a 4.0.1 branch that is under maintenance 4.x. The latest version currently released is 4.0.0 (note that it is not 4.0.0-RC3). The entire official website and documentation are based on this version.
A new branch of doc5.x was newly pulled in order to rewrite 5.x documentation and generate a new version of the website.
> Improve the unit test work, you can directly PR to the master.
##### Clone project
First, you need to add a star to the project by clicking star on the top right corner on github.
Then, click fork, which will generate a new copy of your id path.
For example, mine is:
> https://github.com/kimmking/incubator-shardingsphere
##### Download source
At present, if you git clone the entire project from github, it is about 240M, (because you want to download all the commit records).
You can use the following methods to reduce the first download data to 20M:
> git clone https://github.com/{your id} / incubator-shardingsphere --depth 1
This will only keep the last 2 commits and the current source code.
At this time, a new folder incubator-shardingsphere will be generated in your current directory, which is the entire project source code.
You can see the current branch with the following command:
git branch -vv
The asterisk is the current branch.
##### Adding remote branches
Then we can use the command
> git remote add apache https://github.com/apache/incubator-shardingsphere
Add the project under apache as one of our remote projects.
Can be used later
git fetch apache master: amaster
Branch apache / master remotely and pull down locally to a new branch called amaster.
Can also be used
> git checkout -b bmaster
Pull a new branch called bmaster from the current branch.
Then we can modify and debug on this branch.
After the modification is completed, you can pass
git add.
>git commit -a
To submit changes.
Finally we can pass
git push
Or follow the prompts and use set-upstream to push the new local branch to the remote branch with the same name in our own space.
##### Submit PR
At this point we can submit a PR on github.
Move our branch to the apache / master branch.
#### Compilation Instructions
We generally use
> mvn clean package install
To compile the project.
Can also be used
> mvn clean package install -Prelease
To package the project.
##### Code style
In order to be compatible with the previous version, the entire project compilation level has been Java 1.7, so the classes and features of Java 8 cannot be used at present, such as LocalDatetime or Stream API.
The entire project needs to execute checkstyle at compile time, and the code specifications are stipulated in the checkstyle rules. E.g:
-Cannot introduce unused imports
-Javadoc must be added to public methods and classes, with the first sentence beginning with a capital and ending with an English period
-Indentation uses 4 spaces, so we can turn on the option to show spaces in the IDE
-Space is required after the comma
-If statements need to have spaces after
-Leave a blank line at the end of the file
-Blank lines are not allowed within methods
Some are subject to the same requirements as unit test code.
##### Compile time
At present, the entire project compilation takes about 22 minutes.
There are a few tricks to reduce this time.
##### Dependency Download
Our project uses maven to download dependencies. If your machine is slow to access the maven central repository, you can consider using a domestic aliyun image. Specific methods can be searched online.
If the dependency is downloaded for the first time on the day, the original download can be used in the future. You need to add `-o` to the mvn parameter to indicate offline compilation.
For UI projects, you can first install the NodeJS environment locally, and then use cnpm instead of the original npm command. Can reduce 2/3 compilation time.
##### Task branch
It is recommended that for each project, a new branch is pulled from apache / master, and the name is the name of the project.
Because the trunk has been changing, other partners are constantly maintaining and committing. If your task branch is not submitted on the day of the new pull branch, it is recommended to pull the apache / master and synchronize it to Branch of your own tasks, early detection of conflicts, and ensure that your branch can be merged with the trunk at any time. Refer to the following command:
git checkout amaster
>git pull
>
>git checkout issue3402
>
>git merge amaster
After the task is completed (that is, PR and merged to apache / master), you can directly delete the branch:
Git branch -D issue3402
#### Continuous Integration
Currently, two continuous integrations are used in the project, one is apache jenkins and the other is travis-ci, which are triggered automatically when there is a PR submission in apache / incubator-shardingsphere.
Because of the environment, if one of them reports an error, don't panic, and a success is usually enough, otherwise, check the detailed log for troubleshooting.
### Unit test DEMO
#### Example 1
The development tool uses IDEA as an example to write unit tests for the following classes:
`org.apache.shardingsphere.orchestration.center.yaml.swapper.OrchestrationConfigurationYamlSwapper`
+ Use the shortcut Ctrl + Shift + T in IDEA to add a unit test class for the current class. The default test class name is `OrchestrationConfigurationYamlSwapperTest`
+ Override methods written in the class, as there are 2 methods in the above class
```java
/**
* Swap from InstanceConfiguration to YamlInstanceConfiguration.
*
* @param data data to be swapped
* @return YAML instance configuration
*/
@Override
public YamlOrchestrationConfiguration swap(final OrchestrationConfiguration data) {
Map<String, YamlInstanceConfiguration> yamlInstanceConfigurationMap = new HashMap();
Map<String, InstanceConfiguration> instanceConfigurationMap = data.getInstanceConfigurationMap();
for (Entry<String, InstanceConfiguration> each : instanceConfigurationMap.entrySet()) {
InstanceConfigurationYamlSwapper swapper = new InstanceConfigurationYamlSwapper();
yamlInstanceConfigurationMap.put(each.getKey(), swapper.swap(each.getValue()));
}
YamlOrchestrationConfiguration result = new YamlOrchestrationConfiguration();
result.setInstanceConfigurationMap(yamlInstanceConfigurationMap);
return result;
}
/**
* Swap from YamlInstanceConfiguration to InstanceConfiguration.
*
* @param yamlConfiguration YAML instance configuration
* @return swapped object
*/
@Override
public OrchestrationConfiguration swap(final YamlOrchestrationConfiguration yamlConfiguration) {
Map<String, InstanceConfiguration> instanceConfigurationMap = new HashMap();
Map<String, YamlInstanceConfiguration> yamlInstanceConfigurationMap = yamlConfiguration.getInstanceConfigurationMap();
for (Entry<String, YamlInstanceConfiguration> each : yamlInstanceConfigurationMap.entrySet()) {
InstanceConfigurationYamlSwapper swapper = new InstanceConfigurationYamlSwapper();
instanceConfigurationMap.put(each.getKey(), swapper.swap(each.getValue()));
}
OrchestrationConfiguration result = new OrchestrationConfiguration(instanceConfigurationMap);
return result;
}
```
+ The unit test method name starts with assert. Try to describe the content of this method as clearly as possible. The test code must follow the code specifications and ensure coverage. The above unit test methods are as follows：
```java
@Test
public void assertSwapToYamlOrchestrationConfiguration() {
OrchestrationConfiguration data = getOrchestrationConfiguration();
YamlOrchestrationConfiguration result = new OrchestrationConfigurationYamlSwapper().swap(data);
for (String each : result.getInstanceConfigurationMap().keySet()) {
assertNotNull(result.getInstanceConfigurationMap().get(each));
assertThat(result.getInstanceConfigurationMap().get(each).getOrchestrationType(),
is(data.getInstanceConfigurationMap().get(each).getOrchestrationType()));
assertThat(result.getInstanceConfigurationMap().get(each).getInstanceType(),
is(data.getInstanceConfigurationMap().get(each).getType()));
assertThat(result.getInstanceConfigurationMap().get(each).getNamespace(),
is(data.getInstanceConfigurationMap().get(each).getNamespace()));
assertThat(result.getInstanceConfigurationMap().get(each).getServerLists(),
is(data.getInstanceConfigurationMap().get(each).getServerLists()));
assertThat(result.getInstanceConfigurationMap().get(each).getProps(),
is(data.getInstanceConfigurationMap().get(each).getProperties()));
}
}
@Test
public void assertSwapToOrchestrationConfiguration() {
YamlOrchestrationConfiguration data = getYamlOrchestrationConfiguration();
OrchestrationConfiguration result = new OrchestrationConfigurationYamlSwapper().swap(data);
for (String each : result.getInstanceConfigurationMap().keySet()) {
assertNotNull(result.getInstanceConfigurationMap().get(each));
assertThat(result.getInstanceConfigurationMap().get(each).getOrchestrationType(),
is(data.getInstanceConfigurationMap().get(each).getOrchestrationType()));
assertThat(result.getInstanceConfigurationMap().get(each).getType(),
is(data.getInstanceConfigurationMap().get(each).getInstanceType()));
assertThat(result.getInstanceConfigurationMap().get(each).getNamespace(),
is(data.getInstanceConfigurationMap().get(each).getNamespace()));
assertThat(result.getInstanceConfigurationMap().get(each).getServerLists(),
is(data.getInstanceConfigurationMap().get(each).getServerLists()));
assertThat(result.getInstanceConfigurationMap().get(each).getProperties(),
is(data.getInstanceConfigurationMap().get(each).getProps()));
}
}
```
+ For detailed development specifications, please refer to [ShardingShphere Code of Conduct](https://shardingsphere.apache.org/community/cn/contribute/code-conduct/)
### Other
Everyone is welcome to contribute code and documentation. If you have any questions during the period, you can always communicate on the issue board or group.
Finally, we wish you all a pleasant trip to ShardingSphere.
：）
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] menghaoranss closed issue #4438:
Orchestration unit test development guide for new contributors
Posted by GitBox <gi...@apache.org>.
menghaoranss closed issue #4438: Orchestration unit test development guide for new contributors
URL: https://github.com/apache/incubator-shardingsphere/issues/4438
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] menghaoranss commented on issue #4438:
Orchestration unit test development guide for new contributors
Posted by GitBox <gi...@apache.org>.
menghaoranss commented on issue #4438: Orchestration unit test development guide for new contributors
URL: https://github.com/apache/incubator-shardingsphere/issues/4438#issuecomment-590284904
以下是对于新贡献者的一些提示信息。
### 环境准备
Apache ShardingSphere 是基于Java平台的，所以你需要相关的环境工具：
- JDK8版本，
- Java开发工具IDE，推荐IDEA或Eclipse，安装lombok插件
- Git，或者IDE里集成的git
- Maven3.5
如果本地运行Sharding-Proxy，还需要：
- MySQL 5.6 或 5.7 server，如果要验证PostGreSQL或其他数据库相关功能，需要安装对应的数据库软件。
- Zookeeper 3.5.6 或其他版本，如果要验证Apollo或Nacos的配置中心，也需要安装对应的软件。
如果要调试和运行UI相关项目，需要安装：
- Node 8.11.1以上版本以及对应的NPM 5.6.0以上版本。
### 项目说明
#### 下载源码
项目地址：https://github.com/apache/incubator-shardingsphere
#### 源码说明
源码包括以下几个部分：
- 主项目：SQL Engine，Sharding-jdbc，Sharding-proxy，Orchestration等等
- ShardingSphere-UI：frontend(NodeJS/VUE)，backend(Java)
- doc项目：文档，同时也是官方网站。需要使用基于go的Hugo工具编译成静态内容。
- examples项目：使用示例。
#### 分支说明
我们提倡在专用分支上进行修改和提交，最后通过PR（Pull Request）方式要求合并到主干。
目前apache/incubator-shardingsphere的主干是master，版本为5.0.0-RC[N]-SNAPSHOT。
还有一个4.0.1的分支是在维护中的4.x版本，目前发布的最新版本是4.0.0，（注意，不是4.0.0-RC3）。整个官网和文档，都是基于这个版本的。
新拉了一个doc5.x的分支，为了重新编写5.x的文档，以及生成新版本的网站。
> 完善单元测试的工作，可以直接PR到master上。
##### 克隆项目
首先，你需要在github上，点击右上角的star，为项目增加一个星。
然后，点击fork，这样就会在你的id路径下，产生一个新的复制版本。
例如，我的就是：
> https://github.com/kimmking/incubator-shardingsphere
##### 下载源码
目前如果从github整个项目 git clone下来，大概240M，（因为要下载所有的提交记录）。
可以使用以下方式将首次下载数据降低到20M：
> git clone https://github.com/{你的id}/incubator-shardingsphere --depth 1
这样只会保留最后2次提交记录，和当前的源码。
这时你的当前目录会生成一个新的文件夹incubator-shardingsphere，这里面就是整个项目源码。
你可以通过以下命令看当前的分支：
> git branch -vv
带星号的就是当前分支。
##### 添加远程分支
然后我们可以通过命令
> git remote add apache https://github.com/apache/incubator-shardingsphere
将apache下的项目添加为我们的一个远程项目。
后面可以使用
> git fetch apache master:amaster
将apache/master远程分支，拉倒本地成为一个叫amaster的新分支。
也可以使用
> git checkout -b bmaster
从当前分支拉一个新的叫bmaster的分支。
然后我们可以在这个分支上进行修改和调试。
修改完成以后，可以通过
> git add .
>
> git commit -a
来提交修改。
最后我们可以通过
> git push
或者按照提示，使用set-upstream的方式，将本地新分支push到我们自己空间的远程同名分支。
##### 提交PR
此时我们就可以在github上提交PR了。
将我们的分支，PR到apache/master分支上。
#### 编译说明
我们一般用
> mvn clean package install
来编译项目。
也可以使用
> mvn clean package install -Prelease
来打包项目。
##### 代码风格
由于目前为了兼容前面的版本，整个项目编译级别一直为Java1.7，所以Java8的类和特性，目前不能使用，例如LocalDatetime或者Stream API等。
整个项目需要编译时执行checkstyle，代码规范在checkstyle规则里约定。例如：
- 不能引入未使用的import
- 在公开的方法和类上必须添加javadoc，且第一句大写开头，英文句号结尾
- 缩进使用4个空格，所以我们可以把IDE里显示空格选项打开
- 逗号后需要有空格
- if等语句以后需要有空格
- 文件最后保留一个空行
- 方法内不允许空行
有些是需要单元测试代码一样遵守的。
##### 编译时间
目前整个项目编译大概需要22分钟所有。
可以通过一些技巧，减少这个时间。
##### 依赖下载
我们的项目使用maven下载依赖，如果你的机器访问maven中央库较慢，可以考虑配置使用国内的aliyun镜像。具体方法可以网上搜索。
如果当天第一次下载了依赖性，后续可以使用原有的下载，需要在mvn参数里加`- o`表示离线编译。
对于UI项目，可以先在本地安装NodeJS环境，然后使用cnpm代替原来的npm命令。能减少2/3的编译时间。
##### 任务分支
建议每次为一个项目，从apache/master上新拉一个专用分支，名称为这个项目的名称，比如我常用的`issue3402` 或者 `config-path-resolver` 之类的。
因为主干一直在变化，有其他伙伴在不断的维护和提交，如果自己的任务分支没有在新拉分支的当天提交，建议第二天开始修改之前，先拉一边apache/master，并将其同步到自己的任务分支，及早发现冲突问题，保证自己的分支是随时可以跟主干合并的。参考如下命令：
> git checkout amaster
>
> git pull
>
> git checkout issue3402
>
> git merge amaster
任务完成后（即PR以及被merge到apache/master），可以直接删除掉分支：
> git branch -D issue3402
#### 持续集成
目前项目里使用了两个持续集成，一个是apache jenkins，一个是travis-ci，都是在apache/incubator-shardingsphere有PR提交时自动触发的。
因为环境的关系，如果其中有一个报错不用慌张，有一个成功一般就可以了，否则看详细日志进行排查。
### 单元测试DEMO
#### 示例一
开发工具以IDEA为例，为以下类编写单元测试：
`org.apache.shardingsphere.orchestration.center.yaml.swapper.OrchestrationConfigurationYamlSwapper`
+ 在IDEA中使用快捷键 Ctrl+Shift+T 为当前类添加单元测试类，测试类名默认如 `OrchestrationConfigurationYamlSwapperTest`
+ 覆盖类中自己编写的方法，如以上类中有2个方法
```java
/**
* Swap from InstanceConfiguration to YamlInstanceConfiguration.
*
* @param data data to be swapped
* @return YAML instance configuration
*/
@Override
public YamlOrchestrationConfiguration swap(final OrchestrationConfiguration data) {
Map<String, YamlInstanceConfiguration> yamlInstanceConfigurationMap = new HashMap();
Map<String, InstanceConfiguration> instanceConfigurationMap = data.getInstanceConfigurationMap();
for (Entry<String, InstanceConfiguration> each : instanceConfigurationMap.entrySet()) {
InstanceConfigurationYamlSwapper swapper = new InstanceConfigurationYamlSwapper();
yamlInstanceConfigurationMap.put(each.getKey(), swapper.swap(each.getValue()));
}
YamlOrchestrationConfiguration result = new YamlOrchestrationConfiguration();
result.setInstanceConfigurationMap(yamlInstanceConfigurationMap);
return result;
}
/**
* Swap from YamlInstanceConfiguration to InstanceConfiguration.
*
* @param yamlConfiguration YAML instance configuration
* @return swapped object
*/
@Override
public OrchestrationConfiguration swap(final YamlOrchestrationConfiguration yamlConfiguration) {
Map<String, InstanceConfiguration> instanceConfigurationMap = new HashMap();
Map<String, YamlInstanceConfiguration> yamlInstanceConfigurationMap = yamlConfiguration.getInstanceConfigurationMap();
for (Entry<String, YamlInstanceConfiguration> each : yamlInstanceConfigurationMap.entrySet()) {
InstanceConfigurationYamlSwapper swapper = new InstanceConfigurationYamlSwapper();
instanceConfigurationMap.put(each.getKey(), swapper.swap(each.getValue()));
}
OrchestrationConfiguration result = new OrchestrationConfiguration(instanceConfigurationMap);
return result;
}
```
+ 单元测试方法名以assert开头，尽量能清楚描述此方法测试的内容，测试代码需遵循代码规范，并尽量保证覆盖，以上类单元测试方法如下：
```java
@Test
public void assertSwapToYamlOrchestrationConfiguration() {
OrchestrationConfiguration data = getOrchestrationConfiguration();
YamlOrchestrationConfiguration result = new OrchestrationConfigurationYamlSwapper().swap(data);
for (String each : result.getInstanceConfigurationMap().keySet()) {
assertNotNull(result.getInstanceConfigurationMap().get(each));
assertThat(result.getInstanceConfigurationMap().get(each).getOrchestrationType(),
is(data.getInstanceConfigurationMap().get(each).getOrchestrationType()));
assertThat(result.getInstanceConfigurationMap().get(each).getInstanceType(),
is(data.getInstanceConfigurationMap().get(each).getType()));
assertThat(result.getInstanceConfigurationMap().get(each).getNamespace(),
is(data.getInstanceConfigurationMap().get(each).getNamespace()));
assertThat(result.getInstanceConfigurationMap().get(each).getServerLists(),
is(data.getInstanceConfigurationMap().get(each).getServerLists()));
assertThat(result.getInstanceConfigurationMap().get(each).getProps(),
is(data.getInstanceConfigurationMap().get(each).getProperties()));
}
}
@Test
public void assertSwapToOrchestrationConfiguration() {
YamlOrchestrationConfiguration data = getYamlOrchestrationConfiguration();
OrchestrationConfiguration result = new OrchestrationConfigurationYamlSwapper().swap(data);
for (String each : result.getInstanceConfigurationMap().keySet()) {
assertNotNull(result.getInstanceConfigurationMap().get(each));
assertThat(result.getInstanceConfigurationMap().get(each).getOrchestrationType(),
is(data.getInstanceConfigurationMap().get(each).getOrchestrationType()));
assertThat(result.getInstanceConfigurationMap().get(each).getType(),
is(data.getInstanceConfigurationMap().get(each).getInstanceType()));
assertThat(result.getInstanceConfigurationMap().get(each).getNamespace(),
is(data.getInstanceConfigurationMap().get(each).getNamespace()));
assertThat(result.getInstanceConfigurationMap().get(each).getServerLists(),
is(data.getInstanceConfigurationMap().get(each).getServerLists()));
assertThat(result.getInstanceConfigurationMap().get(each).getProperties(),
is(data.getInstanceConfigurationMap().get(each).getProps()));
}
}
```
+ 了解详细开发规范请参考[ShardingShphere开发规范](https://shardingsphere.apache.org/community/cn/contribute/code-conduct/)
### 其他
欢迎大家贡献代码和文档，期间有任何问题，都可以随时在issue board或群里交流。
最后，祝大家有一个愉快的ShardingSphere之旅。
：）
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-shardingsphere] kimmking commented on issue #4438:
Orchestration unit test development guide for new contributors
Posted by GitBox <gi...@apache.org>.
kimmking commented on issue #4438: Orchestration unit test development guide for new contributors
URL: https://github.com/apache/incubator-shardingsphere/issues/4438#issuecomment-590278217
@menghaoranss Chinese version comment here, it can help new contributors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services