[GitHub] [skywalking-website] kezhenxu94 commented on a change in pull request #206: Add design doc of NGE2E Verifier

[GitBox <gi...@apache.org>]

kezhenxu94 commented on a change in pull request #206:
URL: https://github.com/apache/skywalking-website/pull/206#discussion_r567633189



##########
File path: content/blog/2021-02-01-e2e-verifier-design/index.md
##########
@@ -0,0 +1,182 @@
+---
+title: "[Design] The Verifier of NGE2E"
+date: 2021-02-01
+author: "[Zhenxu Ke](https://github.com/kezhenxu94), Tetrate.io; [Huaxi Jiang](http://github.com/fgksgf), Committer; [Ke Zhang](http://github.com/HumbertZhang), Committer"
+description: "The design of Next Generation End-to-End Testing Framework  Verifier"
+
+tags:
+- Testing
+---
+
+## Background
+
+The verifier is an important part of the next generation End-to-End Testing framework (NGE2E), which is responsible for verifying whether the actual output satisfies the expected template.
+
+## Design Thinking
+
+We will implement the verifier with [Go template](https://golang.org/pkg/text/template/), plus some enhancements. Firstly, users need to write a Go template file with provided functions and actions to describe how the expected data looks like. Then the verifer renders the template with the actual data object. Finally, the verifier compares the rendered output with the actual data. If the rendered output is not the same with the actual output, it means the actual data is inconsist with the expected data. Otherwise, it means the actual data match the expected data. On failure, the verifier will also print out what are different between expected and actual data.
+
+## Branches / Actions
+
+The verifier inherits all the actions from the standard Go template, such as `if`, `with`, `range`, etc. In addition, we also provide some custom actions to satisfy our own needs.
+
+### At Least Once of List Elements Match
+
+`atLeastOnce` checks if there is at least one element of list matches the given template.
+
+Examples:
+
+```yaml
+metrics:
+{{- atLeastOnce .metrics }}
+  name: {{ notEmpty .name }}
+  id: {{ notEmpty .id }}
+  value: {{ gt .value .0 }}
+{{- end }}
+```
+
+Note that between the `{{- atLeastOnce .metrics }}` and `{{- end }}` it is a single object, instead of a list.
+
+## Functions
+
+Users can use these provided functions in the template to describe the expected data.
+
+### Not Empty
+
+`notEmpty` checks if the string `s` is empty.
+
+Example:
+
+```yaml
+id: {{ notEmpty .id }}
+```
+
+### Regexp match
+
+`regexp` checks if string `s` matches the regular expression pattern.
+
+Examples:
+
+```yaml
+label: {{ regexp .label "ratings.*" }}
+```
+
+### Base64
+
+`b64enc s` returns the Base64 encoded string of s.
+
+Examples:
+
+```yaml
+id: {{ b64enc "User" }}.static-suffix # this evalutes the base64 encoded string of "User", concatenated with a static suffix ".static-suffix"
+```
+
+Result:
+
+```yaml
+id: VXNlcg==.static-suffix
+```
+
+## Full Example
+
+Here is an example of expected data:
+
+```yaml
+# expected.data.yaml
+nodes:
+  - id: {{ b64enc "User" }}.0
+    name: User
+    type: USER
+    isReal: false
+  - id: {{ b64enc "Your_ApplicationName" }}.1
+    name: Your_ApplicationName
+    type: Tomcat
+    isReal: true
+  - id: {{ $h2ID := (index .nodes 2).id }}{{ notEmpty $h2ID }} # We assert that nodes[2].id is not empty and save it to variable `h2ID` for later use
+    name: localhost:-1
+    type: H2
+    isReal: false
+calls:
+  - id: {{ notEmpty (index .calls 0).id }}
+    source: {{ b64enc "Your_ApplicationName" }}.1
+    target: {{ $h2ID }} # We use the previously assigned variable `h2Id` to asert that the `target` is equal to the `id` of the nodes[2]
+    detectPoints:
+      - CLIENT
+  - id: {{ b64enc "User" }}.0-{{ b64enc "Your_ApplicationName" }}.1
+    source: {{ b64enc "User" }}.0
+    target: {{ b64enc "Your_ApplicationName" }}.1
+    detectPoints:
+      - SERVER
+```
+
+will validate this data:

Review comment:
       > I want to be clear one more thing, does this verification will compare the size of the list? The logic behind the question also includes, do we check every rule has one and only one matched data. Or just at least having one matched.
   
   Rules inside `{{ atLeastOnce }}` blocks will do an "at least once" verification, otherwise they are compared / verified literally (include list size of course).
   
   This reminds me that we need another action `{{ ignoreOrders }}` that can verify 2 lists ignoring their elements' orders @fgksgf , please add this to the doc.
   
   I think this covers the general use cases of lists comparison.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org