You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/12/17 09:41:46 UTC
[GitHub] [skywalking-website] kezhenxu94 opened a new pull request #172: blog: add the design doc of NGE2E
kezhenxu94 opened a new pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172
This is opened to get quick feedback about the design, technical details should be shared later.
Appreciate if @hanahmily can take a look at the KinD part and @mrproliu can take a look at the locally debugging part (you complained about that mostly 😅)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545574805
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
Review comment:
OK with the new example snippet, it totally makes sense to me, let me update this part later
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] lujiajing1126 commented on pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
lujiajing1126 commented on pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#issuecomment-747841533
I am also curious about the cleanup section as Wu Sheng mentioned in the review.
IMHO, the `cleanup` action should always appear together with the `setup` action since if it cleanup should be done if the corresponding setup succeeds regardless of other actions and setup. The whole procedure is similar to a stack.
For example, if you has prepared a cluster, you always want to shutdown the cluster at the end.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545647130
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > The `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+```shell
+swctl e2e cleanup --env=compose --file=docker-compose.yaml
+swctl e2e cleanup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml
+swctl e2e cleanup # If configuration file e2e.yaml is present
+```
+
+This step requires the same options in the setup step so that it can clean up all things necessarily.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
+│ ├── Dockerfile
+│ ├── pom.xml
+│ └── src
+├── docker-compose.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+or
+
+```text
+case-name
+├── kind.yaml
+├── bookinfo
+│ ├── bookinfo.yaml
+│ └── bookinfo-gateway.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+a sample of `e2e.yaml` may be
+
+```yaml
+setup:
+ env: kind
+ file: kind.yaml
+ resources: bookinfo.yaml
+ command: | # it can be a shell script or anything executable
+ istioctl install --profile=demo -y
+ kubectl label namespace default istio-injection=enabled
+
+ # OR
+ # env: compose
+ # file: docker-compose.yaml
+
+ wait:
+ for: service-health
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] wu-sheng commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545063196
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > the `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+This step is automatically done by the framework according to the setup step.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
Review comment:
How to have multiple projects? Is the CLI going to iterate all folders in the `case-name`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#issuecomment-747859972
@lujiajing1126
> IMHO, the `cleanup` action should always appear together with the `setup` action since `cleanup` should be done if the corresponding setup succeeds regardless of other actions and setup. The whole procedure is similar to a stack.
Definitely `cleanup` should ALWAYS appear together with `setup`, please read this in the doc:
"this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case"
Let me explain, when you run the wrapper command, it ensures the `cleanup` step to be executed no mater what other commands exit codes, but when you run the steps one by one manually (when debugging), you don't want the cluster to be destroyed, right? You want to keep the scene, hence the `cleanup` step needs the same options as the `setup` because at this time, there is not context what had been set up if you run `cleanup` separately, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 merged pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 merged pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] wu-sheng commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545075236
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
Review comment:
I mean, is `cleanup` an swctl command too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] lujiajing1126 commented on pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
lujiajing1126 commented on pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#issuecomment-747880434
> Let me explain, when you run the wrapper command, it ensures the `cleanup` step to be executed no mater what other commands exit codes, but when you run the steps one by one manually (when debugging), you don't want the cluster to be destroyed after tests, right? You want to keep the scene to check what's wrong inside the cluster, after you find out the reason, you will destroy the cluster via `cleanup` command, hence the `cleanup` step needs the same options as the `setup` because at this time, there is not context what had been set up if you run `cleanup` separately, right?
I see. Thanks for the explanation. I appreciate this design which makes every step debuggable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] lujiajing1126 edited a comment on pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
lujiajing1126 edited a comment on pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#issuecomment-747841533
I am also curious about the cleanup section as Wu Sheng mentioned in the review.
IMHO, the `cleanup` action should always appear together with the `setup` action since `cleanup` should be done if the corresponding setup succeeds regardless of other actions and setup. The whole procedure is similar to a stack.
For example, if you has prepared a cluster, you always want to shutdown the cluster at the end even not a single test passes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] CrocoRyan commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
CrocoRyan commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545700980
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > the `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+This step is automatically done by the framework according to the setup step.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
+│ ├── Dockerfile
+│ ├── pom.xml
+│ └── src
+├── docker-compose.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+or
+
+```text
+case-name
+├── kind.yaml
+├── bookinfo
+│ ├── bookinfo.yaml
+│ └── bookinfo-gateway.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+a sample of `e2e.yaml` may be
+
+```yaml
+setup:
+ env: kind
+ file: kind.yaml
+ resources: bookinfo.yaml
+ command: | # it can be a shell script or anything executable
+ istioctl install --profile=demo -y
+ kubectl label namespace default istio-injection=enabled
+
+ # OR
+ # env: compose
+ # file: docker-compose.yaml
+
+ wait:
+ for: service-health
Review comment:
[#172 (comment)](https://github.com/apache/skywalking-website/pull/172#discussion_r545571540) is a great integrated solution and it fulfills the step-by-step logic, and that solve my question. thanks for pointing out to me.
As for the naming, until is definitely a much clearer option in my view
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545068483
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > the `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+This step is automatically done by the framework according to the setup step.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
Review comment:
You commented on line `agent-service`, the E2E doesn't care about this/these services, these services are used on demand of the test case itself.
Consider this, we might have `provider` service (Java) and `consumer` service (Python) to test context propagation between different languages, and they are ONLY used in the `docker-compose`, when the `setup` step is executed, it pulls up the `docker-compose` and the 2 services will be built (if necessary) and run, the CLI is not aware of these 2 services at all
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] wu-sheng commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545539933
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
Review comment:
> How to sync images from the build process? Would we sync them after kind boot successfully?
What do you mean sync images?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] wu-sheng commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545061080
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
Review comment:
Is this going to be waiting until terminating this command? Because `Cleanup` seems not related to any command.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#issuecomment-748044656
I think all the comments are addressed, if I missed any, please ping me. Thanks for everyone who participated the discussion, 🙇
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] lujiajing1126 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
lujiajing1126 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545592250
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
Review comment:
Some questions for the `wait`,
1. Should it be `--wait-for=`? `--wait=for=` seems strange to me.
2. If multiple(either -number or -kinds) resources are defined, e.g. multiple containers, pods, services, or even mixed, how to define the semantics of `service-health`, `pod-ready`?
3. The keywords of `wait` like `pod-ready`, `service-health` seem to be context-sensitive, i.e. related to the `--env` and `--resources`. Would it be a problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] hanahmily commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
hanahmily commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545537708
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
Review comment:
Some questions:
1. How to sync images from the build process? Would we sync them after kind boot successfully?
2. Is one resource applied depends on its previous is ready?
3. We need a `label selector` to select the workloads we care about.
4. Could we wait for deployment/statefulset/daemonset `available` status instead of the pod ready. `Available` means the readiness prod is successful, and pods are working continually for at least several seconds that are specificated by `MinReadySeconds`
5. Pls replace `resources` with `manifests` to follow k8s' convention.
6. I failed to find any steps to setup skywalking components in this example. In that case, my preference is to depend on `swck` to provision them since they are all go based project.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] lujiajing1126 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
lujiajing1126 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545544815
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
Review comment:
Consider also `initialDelay`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545563202
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
Review comment:
> Some questions:
>
> 1. How to sync images from the build process? Would we sync them after kind boot successfully?
This can be done by the `--command` option, see line 145.
> 2. Is one resource applied depends on its previous is ready?
I don't think so, according to Kubernetes eventual consistency, is that necessary?
> 3. We need a `label selector` to select the workloads we care about.
Can you please elaborate more why we need this in a test case (Do the workloads matter)?
> 4. Could we wait for deployment/statefulset/daemonset `available` status instead of the pod ready. `Available` means the readiness prod is successful, and pods are working continually for at least several seconds that are specificated by `MinReadySeconds`
The `--wait=for=pod-ready` is only an example, there can be `--wait=for=service/ready`, `--wait=for=deployments/available`, `--wait=for=daemonset/available`, etc.
> 5. Pls replace `resources` with `manifests` to follow k8s' convention.
Will do.
> 6. I failed to find any steps to setup skywalking components in this example. In that case, my preference is to depend on `swck` to provision them since they are all go based project.
SkyWalking components should be set up via `--resource/manifests=` or `--command` IMO because how to set them up varies from case to case, you may want to set up via `swck`, but for sub-project like http://github.com/apache/skywalking-kubernetes/ , we may want to set them up via Helm (because the setup step itself is a test case!), or in `--env=docker-compose` case, they're set up via `docker-compose`. Does it make sense?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] wu-sheng commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545156359
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > the `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+This step is automatically done by the framework according to the setup step.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
Review comment:
Got it, make good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545627176
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
Review comment:
1. Sounds good
2. https://github.com/apache/skywalking-website/pull/172#discussion_r545571540 this should address your concern
3. The one who writes the `wait` should be aware of what kind of `env` the case uses, otherwise we can simply fail the case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] wu-sheng commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545080430
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > the `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+This step is automatically done by the framework according to the setup step.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
Review comment:
I know the e2e framework doesn't care about this at the runtime. But, the CI has a step to package all these as images, right? Who will drive this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] wu-sheng commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545055720
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
Review comment:
`k`->`K`. Same as following, many lower cases as the first letter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] hanahmily commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
hanahmily commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545544971
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
Review comment:
Could we add some functions to support async scenarios, like gomega 's http://onsi.github.io/gomega/#making-asynchronous-assertions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] CrocoRyan commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
CrocoRyan commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545700980
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > the `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+This step is automatically done by the framework according to the setup step.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
+│ ├── Dockerfile
+│ ├── pom.xml
+│ └── src
+├── docker-compose.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+or
+
+```text
+case-name
+├── kind.yaml
+├── bookinfo
+│ ├── bookinfo.yaml
+│ └── bookinfo-gateway.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+a sample of `e2e.yaml` may be
+
+```yaml
+setup:
+ env: kind
+ file: kind.yaml
+ resources: bookinfo.yaml
+ command: | # it can be a shell script or anything executable
+ istioctl install --profile=demo -y
+ kubectl label namespace default istio-injection=enabled
+
+ # OR
+ # env: compose
+ # file: docker-compose.yaml
+
+ wait:
+ for: service-health
Review comment:
[#172 (comment)](https://github.com/apache/skywalking-website/pull/172#discussion_r545571540) is a great integrated solution and it fulfills the step-by-step logic, and that solve my question. Thanks for pointing out to me. What I was worrying previously is that your command set may involved in various conditions to wait before executing. Every command do requires no more than 1 prerequisite to execute, but when you list them under the "command" key, there will be multiple conditions in total.
As for the naming, until is definitely a much clearer option in my view
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] lujiajing1126 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
lujiajing1126 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545592250
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
Review comment:
Some question for the `wait`,
1. Should it be `--wait-for=`? `--wait=for=` seems strange for me.
2. If multiple(either -number or -kinds) resources are defined, e.g. multiple containers, pods, services, or even mixed, how to define the semantics of `service-health`, `pod-ready`?
3. The keywords of `wait` like `pod-ready`, `service-health` seem to be context-sensitive, i.e. related to the `--env` and `--resources`. Would it be a problem?
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
Review comment:
Some questions for the `wait`,
1. Should it be `--wait-for=`? `--wait=for=` seems strange for me.
2. If multiple(either -number or -kinds) resources are defined, e.g. multiple containers, pods, services, or even mixed, how to define the semantics of `service-health`, `pod-ready`?
3. The keywords of `wait` like `pod-ready`, `service-health` seem to be context-sensitive, i.e. related to the `--env` and `--resources`. Would it be a problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545063462
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
Review comment:
> Is this going to be waiting until terminating this command? Because `Cleanup` seems not related to any command.
This will only wait until all services/pods are healthy/ready and exit, the `cleanup` will clean things up according to what is configured in the setup step
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] hanahmily commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
hanahmily commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545571540
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
Review comment:
> Can you please elaborate more why we need this in a test case (Do the workloads matter)?
NO. 2 and NO. 3 are from the same idea, that we should run a test case step by step. Deploy a manifest then wait or make sure all of the components are ready, then move to the next step. The benefit of this pattern is to split the focus, which makes contributors debug more easily.
And you mention `command` option, which brings some flexibility but makes more confuses me indeed. In general, we need the deployment-wait workflow in command either, for example, after deploying SW, we have to wait for deployment and pod available. I tweak the example a bit to explain what I need :
```
setup:
env: kind
file: kind.yaml
steps:
- command: kubectl apply -f skywalking.yml
wait:
namespace: istio-system
label-selector: app=skywalking
for: deployment-avaiable
- command: istioctl install --profile=demo -y
wait:
namespace: istio-system
label-selector: app=istio
for: deployment-ready
- command: kubectl label namespace default istio-injection=enabled
- command: kubectl apply -f bookinfo.yaml
wait:
namespace: default
for: deployment-ready
```
All of the steps are sync or async. If it's a sync step, we could check the return code to verify the result, while in the async step we should leverage the async function `wait` to check the result is what I expect eventually.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] lujiajing1126 edited a comment on pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
lujiajing1126 edited a comment on pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#issuecomment-747841533
I am also curious about the cleanup section as Wu Sheng mentioned in the review.
IMHO, the `cleanup` action should always appear together with the `setup` action since `cleanup` should be done if the corresponding setup succeeds regardless of other actions and setup. The whole procedure is similar to a stack.
For example, if you has prepared a cluster, you always want to shutdown the cluster at the end.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] CrocoRyan commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
CrocoRyan commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545633863
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > the `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+This step is automatically done by the framework according to the setup step.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
+│ ├── Dockerfile
+│ ├── pom.xml
+│ └── src
+├── docker-compose.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+or
+
+```text
+case-name
+├── kind.yaml
+├── bookinfo
+│ ├── bookinfo.yaml
+│ └── bookinfo-gateway.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+a sample of `e2e.yaml` may be
+
+```yaml
+setup:
+ env: kind
+ file: kind.yaml
+ resources: bookinfo.yaml
+ command: | # it can be a shell script or anything executable
+ istioctl install --profile=demo -y
+ kubectl label namespace default istio-injection=enabled
+
+ # OR
+ # env: compose
+ # file: docker-compose.yaml
+
+ wait:
+ for: service-health
Review comment:
I have questions about the "wait" segment:
Are we able to merge those wait step into one independent block , if there're multiple prerequisites to start the test?
Plus, the "for" syntax seems to be a little bit ambiguous since many of users initially recognize this as some sort of iteration thing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545086483
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > the `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+This step is automatically done by the framework according to the setup step.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
Review comment:
The `build` phase of `docker-compose` should drive this, `docker-compose` can build an image according to the Dockerfile if the image is absent.
<img width="713" alt="image" src="https://user-images.githubusercontent.com/15965696/102493363-03f7f100-40ae-11eb-9464-5b78b7f785d2.png">
> If the services (for example, `provider`, `consumer`) are shared in many cases, we might pre-build them in an earlier process, that's another enhancement the framework doesn't care about again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 edited a comment on pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 edited a comment on pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#issuecomment-747859972
@lujiajing1126
> IMHO, the `cleanup` action should always appear together with the `setup` action since `cleanup` should be done if the corresponding setup succeeds regardless of other actions and setup. The whole procedure is similar to a stack.
Definitely `cleanup` should ALWAYS appear together with `setup`, please read this in the doc:
"this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case"
Let me explain, when you run the wrapper command, it ensures the `cleanup` step to be executed no mater what other commands exit codes, but when you run the steps one by one manually (when debugging), you don't want the cluster to be destroyed after tests, right? You want to keep the scene to check what's wrong inside the cluster, after you find out the reason, you will destroy the cluster via `cleanup` command, hence the `cleanup` step needs the same options as the `setup` because at this time, there is not context what had been set up if you run `cleanup` separately, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] CrocoRyan commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
CrocoRyan commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545700980
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > the `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+This step is automatically done by the framework according to the setup step.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
+│ ├── Dockerfile
+│ ├── pom.xml
+│ └── src
+├── docker-compose.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+or
+
+```text
+case-name
+├── kind.yaml
+├── bookinfo
+│ ├── bookinfo.yaml
+│ └── bookinfo-gateway.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+a sample of `e2e.yaml` may be
+
+```yaml
+setup:
+ env: kind
+ file: kind.yaml
+ resources: bookinfo.yaml
+ command: | # it can be a shell script or anything executable
+ istioctl install --profile=demo -y
+ kubectl label namespace default istio-injection=enabled
+
+ # OR
+ # env: compose
+ # file: docker-compose.yaml
+
+ wait:
+ for: service-health
Review comment:
[#172 (comment)](https://github.com/apache/skywalking-website/pull/172#discussion_r545571540) is a great integrated solution and it fulfills the step-by-step logic, and that solve my question. Thanks for pointing out to me. What I was worrying previously is that your command set may involved in various conditions to wait before executing. Every command do requires no more than 1 prerequisite to execute, but when you list them under the "command" key, there will be numerous conditions in total.
As for the naming, until is definitely a much clearer option in my view
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#issuecomment-748550908
> I think all the comments are addressed, if I missed any, please ping me. Thanks for everyone who participated the discussion, 🙇
I'm merging this for now, @fgksgf and @Humbertzhang will start to work on it,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] hanahmily commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
hanahmily commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545538683
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > The `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+```shell
+swctl e2e cleanup --env=compose --file=docker-compose.yaml
+swctl e2e cleanup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml
+swctl e2e cleanup # If configuration file e2e.yaml is present
+```
+
+This step requires the same options in the setup step so that it can clean up all things necessarily.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
+│ ├── Dockerfile
+│ ├── pom.xml
+│ └── src
+├── docker-compose.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+or
+
+```text
+case-name
+├── kind.yaml
+├── bookinfo
+│ ├── bookinfo.yaml
+│ └── bookinfo-gateway.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+a sample of `e2e.yaml` may be
+
+```yaml
+setup:
+ env: kind
+ file: kind.yaml
+ resources: bookinfo.yaml
+ command: | # it can be a shell script or anything executable
+ istioctl install --profile=demo -y
+ kubectl label namespace default istio-injection=enabled
+
+ # OR
+ # env: compose
+ # file: docker-compose.yaml
+
+ wait:
+ for: service-health
Review comment:
As I mentioned before, I prefer to match `resource/command` and `wait` into a group in order to ensure every step is expected before moving forward.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545638322
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
+swctl e2e setup # If configuration file e2e.yaml is present
+```
+
+- `--env`: the environment, may be `compose` or `kind`, represents docker-compose and KinD respectively;
+- `--file`: the `docker-compose.yaml` or `kind.yaml` file that declares how to set up the environment;
+- `--resources`: for KinD, the resources files/directories to apply (using `kubectl apply -f`);
+- `--command`: a command to run after the environment is started, this may be useful when users need to install some extra tools or apply resources from command line, like `istioctl install --profile=demo`;
+- `--wait`: wait until the given condition is met; the most frequently-used strategy should be `for=service-health`, that makes the `e2e setup` command to wait for all services to be `healthy`; other possible strategies may be something like `for="log:Started Successfully"`, `for="http:localhost:8080/healthcheck"`, etc. if really needed;
+
+
+### Trigger Inputs
+
+```shell
+swctl e2e trigger --interval=3s --times=0 --action=http --url="localhost:8080/users"
+swctl e2e trigger --interval=3s --times=0 --action=cmd --cmd="curl localhost:8080/users"
+swctl e2e trigger # If configuration file e2e.yaml is present
+```
+
+- `--interval=3s`: trigger the action every 3 seconds;
+- `--times=0`: how many times to trigger the action, `0=infinite`;
+- `--action=http`: the action of the trigger, i.e. "perform an http request as an input";
+- `--action=cmd`: the action of the trigger, i.e. "execute the `cmd` as an input";
+
+
+### Query Output
+
+```shell
+swctl service ls
+```
+
+this does exactly the same as what `swctl` is doing at present;
+
+
+### Verify
+
+```shell
+swctl e2e verify --actual=actual.data.yaml --expected=expected.data.yaml
+swctl e2e verify --query="service ls" --expected=expected.data.yaml
+swctl e2e verify # If configuration file e2e.yaml is present
+```
+
+- `--actual`: the actual data file;
+- `--expected`: the expected data file;
+- `--query`: the query to get the actual data, will be executed in command `swctl`;
+ > the `--query` option will get the output into a temporary file and use the `--actual` under the hood;
+
+
+### Cleanup
+
+This step is automatically done by the framework according to the setup step.
+
+### Summarize
+
+To summarize, the directory structure of a test case might be
+
+```text
+case-name
+├── agent-service # optional, an arbitrary project that is used in the docker-compose.yaml if needed
+│ ├── Dockerfile
+│ ├── pom.xml
+│ └── src
+├── docker-compose.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+or
+
+```text
+case-name
+├── kind.yaml
+├── bookinfo
+│ ├── bookinfo.yaml
+│ └── bookinfo-gateway.yaml
+├── e2e.yaml # see a sample below
+└── testdata
+ ├── expected.endpoints.service1.yaml
+ ├── expected.endpoints.service2.yaml
+ └── expected.services.yaml
+```
+
+a sample of `e2e.yaml` may be
+
+```yaml
+setup:
+ env: kind
+ file: kind.yaml
+ resources: bookinfo.yaml
+ command: | # it can be a shell script or anything executable
+ istioctl install --profile=demo -y
+ kubectl label namespace default istio-injection=enabled
+
+ # OR
+ # env: compose
+ # file: docker-compose.yaml
+
+ wait:
+ for: service-health
Review comment:
> Are we able to merge those wait step into one independent block , if there're multiple prerequisites to start the test?
https://github.com/apache/skywalking-website/pull/172#discussion_r545571540 this suggestion should address your concern, if we have corresponding `wait` for each resource, one should configure a `wait` for each resource and after all of the resources are `wait`ed, things should be ready, do you have scenarios where one resource needs 1+ prerequisites?
> Plus, the "for" syntax seems to be a little bit ambiguous since many of users initially recognize this as some sort of iteration thing
suggestion? `until`? "wait for" is a grammar I think,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545082669
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,276 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- this framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- this project doesn't take the plugin tests into account, at least for now;
+- this project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- this documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
Review comment:
Yes. Huh I see, `cleanup` needs the same options (`--env`, `--file`) too, I'll update this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] hanahmily commented on a change in pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
hanahmily commented on a change in pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#discussion_r545541294
##########
File path: content/blog/e2e-design/index.md
##########
@@ -0,0 +1,282 @@
+---
+title: "[Design] NGE2E - Next Generation End-to-End Testing Framework"
+date: 2020-12-14
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework"
+---
+
+NGE2E is the next generation End-to-End Testing framework that aims to help developers to set up, debug, and verify E2E tests with ease. It's built based on the lessons learnt from tens of hundreds of test cases in the SkyWalking main repo.
+
+# Goal
+
+- Keep the feature parity with the existing E2E framework in SkyWalking main repo;
+- Support both [docker-compose](https://docs.docker.com/compose/) and [KinD](https://kind.sigs.k8s.io) to orchestrate the tested services under different environments;
+- Get rid of the heavy `Java/Maven` stack, which exists in the current E2E; be language independent as much as possible, users only need to configure YAMLs and run commands, without writing codes;
+
+# Non-Goal
+
+- This framework is not involved with the build process, i.e. it won't do something like `mvn package` or `docker build`, the artifacts (`.tar`, docker images) should be ready in an earlier process before this;
+- This project doesn't take the plugin tests into account, at least for now;
+- This project doesn't mean to add/remove any new/existing test case to/from the main repo;
+- This documentation won't cover too much technical details of how to implement the framework, that should go into an individual documentation;
+
+# Design
+
+Before diving into the design details, let's take a quick look at how the end user might use NGE2E.
+
+> All the following commands are mock, and are open to debate.
+
+To run a test case in a directory `/path/to/the/case/directory`
+
+```shell
+swctl e2e run /path/to/the/case/directory
+
+# or
+
+cd /path/to/the/case/directory && swctl e2e run
+```
+
+This will run the test case in the specified directory, this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case:
+
+**NOTE**: because all the options can be loaded from a configuration file, so as long as a configuration file (say `e2e.yaml`) is given in the directory, every command should be able to run in bare mode (without any option explicitly specified in the command line);
+
+### Set Up Services
+
+```shell
+swctl e2e setup --env=compose --file=docker-compose.yaml --wait=for=service-health
+swctl e2e setup --env=kind --file=kind.yaml --resources=bookinfo.yaml,gateway.yaml --wait=for=pod-ready
Review comment:
The host(github action host or local machine) image repo is isolated with a kind cluster. read https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster for more info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-website] kezhenxu94 edited a comment on pull request #172: blog: add the design doc of NGE2E
Posted by GitBox <gi...@apache.org>.
kezhenxu94 edited a comment on pull request #172:
URL: https://github.com/apache/skywalking-website/pull/172#issuecomment-747859972
@lujiajing1126
> IMHO, the `cleanup` action should always appear together with the `setup` action since `cleanup` should be done if the corresponding setup succeeds regardless of other actions and setup. The whole procedure is similar to a stack.
Definitely `cleanup` should ALWAYS appear together with `setup`, please read this in the doc:
"this command is a wrapper that glues all the following commands, which can be executed separately, for example, to debug the case"
Let me explain, when you run the wrapper command, it ensures the `cleanup` step to be executed no mater what other commands exit codes, but when you run the steps one by one manually (when debugging), you don't want the cluster to be destroyed after tests, right? You want to keep the scene to check what's wrong inside the cluster, after you find out the reason, you will destroy the cluster via `cleanup` command, hence the `cleanup` step needs the same options as the `setup` because at this time, there is not context what had been set up if you run `cleanup` separately, right?
I'll update the part to indicate that `cleanup` will always be executed when running `e2e run` as a whole
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org