You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@skywalking.apache.org by "kezhenxu94@apache" <ke...@apache.org> on 2020/11/15 09:27:06 UTC
[DISCUSSION] Build the next generation of E2E test framework
Hi the community, I want to raise a discussion to build the next generation of E2E test framework for Apache SkyWalking, which may not be a high priority but a necessity.
As we already know, tests in SkyWalking play a very important role in the contribution process, and the current E2E tests just work well to verify every single pull request before merging, so why bother to build the next generation of E2E test framework?
1. In the SkyWalking's ecosystem, there are many sub-projects that vary in terms of language, almost all of them need E2E tests to help us verify the pull requests, but we can see that they reimplement the E2E test framework in their languages, (or even worse, some of them don’t have E2E tests at all). Reimplementing the E2E test framework is unnecesarry and introduces more duplicated work when adding a new test case. The ultimate goal is to reuse all the test cases when needed.
2. The current E2E framework is built with Java, running Java-based program is not an easy thing for non-Java developers (maintainers of other sub-projects such as PHP, C++, LUA, etc.), they need to set up the environment correctly and then run the tests, neither is it an easy skill to debug the tests.
3. It’s a good opportunity to optimize the E2E tests because we added many cases and many duplicated codes that need to be removed.
Therefore we're planning to build an unified, easy-to-use, and fast E2E test framework to benifit all the sub-projects.
Here are some rough ideas about this framework:
0. (Unchanged) We will continue to use docker-compose.yaml to orchestrate the services that are to be tested, it’s proved to be friendly and debuggable because we can start it directly even if we are not writing E2E tests.
1. This framework will take full advantages of the CLI project to query the traces/metrics/logs, we don’t want to maintain two versions of query codes anymore, (now we have a version in the test codes and one in the CLI).
2. We will build a CLI tool to compare the expected data and the actual data, we now have a custom schema of the expected data yaml file, the verification codes should be packaged into an executable command so that it can be executed standalone without Java and maven knowledge. I hope we can leverage existing standards to write the YAML files and do verifications.
3. Build an orchestrating tool to start the `docker-compose.yaml`, health check, query actual data, and verify the result.
4. (Nice to have) wrap all the tools as a GitHub Actions to share between sub-projects.
If you have any other better idea or want to complement to this proposal, please reply here. I will create an issue to track these tasks later.
We have two students (who just finished the Summer Code 2020 Projects) to lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is a not high priority, you have adequate time to get yourselves familiar with how the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work as well as the ecosystem. Thanks for your interests and willingness to help.
[1] https://github.com/Humbertzhang
[2] https://github.com/fgksgf/
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Sheng Wu <wu...@gmail.com>.
It makes sense to make e2e driver and maybe validators in the INFRA repo.
Sheng Wu 吴晟
Twitter, wusheng1108
kezhenxu94@apache <ke...@apache.org> 于2020年12月26日周六 上午8:53写道：
> Because now we have skywalking-eyes repository to host the infra tools, we
> can put the new E2E framework there so that it doesn’t mess up the CLI
> repo. Here [1] I created a new directory to host the E2E codes.
>
> [1] https://github.com/apache/skywalking-eyes/pull/11
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94
>
> > On Dec 17, 2020, at 17:45, kezhenxu94@apache <ke...@apache.org>
> wrote:
> >
> > Hey, I’ve just drafted a design doc here [1], anyone interested in this
> please feel free to leave your comments, thanks!
> >
> > [1] https://github.com/apache/skywalking-website/pull/172
> >
> >> On Nov 26, 2020, at 14:41, Sheng Wu <wu...@gmail.com> wrote:
> >>
> >> You could learn from the Satellite project design,
> >> 1. Discuss here, https://github.com/apache/skywalking/issues/5871
> >> 2. Final design posted as a blog,
> >>
> http://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/
> >>
> >> People could learn from these things when you join the community in the
> >> future.
> >>
> >> Sheng Wu 吴晟
> >> Twitter, wusheng1108
> >>
> >>
> >> Hoshea Jiang <fg...@gmail.com> 于2020年11月26日周四 下午1:38写道：
> >>
> >>> I agree with the conclusion, and I hope to have some diagrams such as
> >>> flowcharts or data flow diagrams, which help provide an overview of
> the E2E
> >>> test before we start to do this.
> >>>
> >>> Ke Zhang <hu...@gmail.com> 于2020年11月26日周四 下午12:34写道：
> >>>
> >>>> I think supporting both docker-compose and KIND in the E2E test is
> >>>> reasonable and necessary, and I also agree with other conclusions.😀
> >>>>
> >>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月26日周四 上午11:58写道：
> >>>>
> >>>>>
> >>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
> >>>>>> 2. Totally replace the Java and Maven based driving system.
> >>>>>> 3. Enhance CLI to validate the GraphQL
> >>>>>> 4. Keep the agent test tool unchanged for now as it is already a
> >>>> separate
> >>>>>> system from the e2e.
> >>>>>
> >>>>> This conclusion is good for me.
> >>>>>
> >>>>> What are others opinions?
> >>>>>
> >>>>> —————————
> >>>>> Zhenxu Ke (柯振旭)
> >>>>> GitHub @kezhenxu94
> >>>>>
> >>>>>> On Nov 17, 2020, at 2:42 PM, Sheng Wu <wu...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>> Using k8s and docker-compose as 2 options in the test process is
> >>>>>> reasonable, and should be supported.
> >>>>>>
> >>>>>> Let's keep the conclusion as simple as possible. In the next
> >>> generation
> >>>>> e2e
> >>>>>> framework
> >>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
> >>>>>> 2. Totally replace the Java and Maven based driving system.
> >>>>>> 3. Enhance CLI to validate the GraphQL
> >>>>>> 4. Keep the agent test tool unchanged for now as it is already a
> >>>> separate
> >>>>>> system from the e2e.
> >>>>>>
> >>>>>> Are these good enough for everyone?
> >>>>>>
> >>>>>> Sheng Wu 吴晟
> >>>>>> Twitter, wusheng1108
> >>>>>>
> >>>>>>
> >>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
> >>>>>>
> >>>>>>>>
> >>>>>>>> We can see from above, your discussion actually is only related to
> >>> 3.
> >>>>>>>
> >>>>>>>
> >>>>>>> That's the only I intend to discuss here if you notice my first
> >>>>>>> response('May SWCK help about provision and demo") in this thread.
> >>> For
> >>>>>>> other parts, I don't have any tips to share, though.
> >>>>>>> Let me explain it more clearly. I want kubernetes to be seen as an
> >>>>>>> alternative to docker-compose to play a running-engine role in the
> >>>> test
> >>>>>>> framework. If kubernetes can't replace
> >>>>>>> the latter one Is it able to participate in it to solve the
> >>> dedicated
> >>>> or
> >>>>>>> special issues.
> >>>>>>>
> >>>>>>> Our e2e test and plugin test frameworks are already top-level
> >>>> complicity
> >>>>>>>> thing in the worldwide open source field. Please don't over-design
> >>>> it,
> >>>>>>> and
> >>>>>>>> too aggressive, the world will stay in the hybrid env for a very
> >>> long
> >>>>>>> time,
> >>>>>>>> same as the developer.
> >>>>>>>
> >>>>>>>
> >>>>>>> Reusing current elaborate output might be reasonable. But in this
> >>>>> thread,
> >>>>>>> we're talking about the potential solution to build a
> >>>>>>> next-generation framework, which might mean that the current
> >>> framework
> >>>>>>> is too complicated to maintain; we need a more tiny way to support
> >>>> more
> >>>>> and
> >>>>>>> more cases.
> >>>>>>>
> >>>>>>> And finally, SWCK is gonna and has to build a similar test
> >>> framework.
> >>>> If
> >>>>>>> the test framework opts for the kubernetes solution, we could share
> >>>> test
> >>>>>>> cases and the underlying framework. It's a more efficient
> >>>>>>> and consistent path for the entire system.
> >>>>>>>
> >>>>>>> regards, Hongtao.
> >>>>>>>
> >>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
> >>>>>>>
> >>>>>>>> Hi
> >>>>>>>>
> >>>>>>>> I don't think we have to make the decision on k8s or out-of-k8s.
> >>>>>>>> e2e and all integration tests have their scenarios like SkyWalking
> >>>> have
> >>>>>>>> multiple languages based and mesh-based, even Prometheus adoption.
> >>>>>>>> The same thing happens on the test field, k8s, and out of k8s.
> >>>>>>>>
> >>>>>>>> I would like to agree with both of you, k8s is needed, no matter
> in
> >>>>> mesh
> >>>>>>>> solution(ALS, telemetry v2), but also not anyone needs that, such
> >>> as
> >>>>>>> agent
> >>>>>>>> developer. We wouldn't deny the fact, the docker-compose is more
> >>>>>>>> lightweight and easier to learn.
> >>>>>>>> So, let me get this straight, for the test you need
> >>>>>>>> 1. Build the source from different repos
> >>>>>>>> 2. Build the images
> >>>>>>>> 3. Use some platform(docker-compose or k8s) to run the images in
> >>> one
> >>>>>>> piece.
> >>>>>>>> 4. Give some inputs to the env
> >>>>>>>> 5. Use some tools/ways to check what should happen according to
> >>> (4)'s
> >>>>>>> input
> >>>>>>>> 6. Checked or timeout failure.
> >>>>>>>>
> >>>>>>>> We can see from above, your discussion actually is only related to
> >>> 3.
> >>>>>>> Let's
> >>>>>>>> not see this as a battle, we need both. Many developers/users use
> >>> OAP
> >>>>> and
> >>>>>>>> SkyWalking out of k8s, that is a simple fact.
> >>>>>>>> Our e2e test and plugin test frameworks are already top-level
> >>>>> complicity
> >>>>>>>> thing in the worldwide open source field. Please don't over-design
> >>>> it,
> >>>>>>> and
> >>>>>>>> too aggressive, the world will stay in the hybrid env for a very
> >>> long
> >>>>>>> time,
> >>>>>>>> same as the developer.
> >>>>>>>>
> >>>>>>>> Also, let's keep in mind, k8s is popular becomes it doesn't
> require
> >>>> the
> >>>>>>>> developers to understand as the VM did. We as an open-source
> >>>> community,
> >>>>>>> are
> >>>>>>>> focusing on the developer's capabilities.
> >>>>>>>>
> >>>>>>>> Sheng Wu 吴晟
> >>>>>>>> Twitter, wusheng1108
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
> >>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> but I have been thinking whether or not it is overkill for
> >>> testing
> >>>>>>>>>> purposes to introduce Kubernetes things.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> we all know the fact docker-compose is more portable than
> >>>> Kubernetes,
> >>>>>>>>> friendly to local running. But there're also some benefits to
> >>>> replace
> >>>>>>> it
> >>>>>>>>> with kubernetes system:
> >>>>>>>>>
> >>>>>>>>> 1. SWCK has to test based on a real Kubernetes environment if the
> >>>> main
> >>>>>>>>> repo test framework doesn't support it(following the
> >>> docker-compose
> >>>>>>>> stack),
> >>>>>>>>> That means skywalking ecosystem MUST introduce kubernetes which
> is
> >>>> not
> >>>>>>> an
> >>>>>>>>> optional component.
> >>>>>>>>>
> >>>>>>>>> 2. Kubernetes support to scale applications at runtime, that help
> >>>>>>>> improve
> >>>>>>>>> the process of e2e. For instance, we could merge a single
> instance
> >>>> and
> >>>>>>>>> cluster test, by scaling up replicas. This strategy will reduce
> >>> the
> >>>>>>> total
> >>>>>>>>> time sharply.
> >>>>>>>>>
> >>>>>>>>> If we are going to use KIND, I hope we can give a script or
> >>>> something
> >>>>>>>>>> similar to install all the needed components and create cluster
> >>> for
> >>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That's SWCK's capacity to provide a standard and simple way to
> >>>> create
> >>>>> a
> >>>>>>>>> cluster in every environment,
> >>>>>>>>>
> >>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一
> >>> 下午7:00写道：
> >>>>>>>>>
> >>>>>>>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting
> >>>>>>>> Kubernetes-based
> >>>>>>>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND
> too,
> >>>>>>> but I
> >>>>>>>>>> have been thinking whether or not it is overkill for testing
> >>>> purposes
> >>>>>>>> to
> >>>>>>>>>> introduce Kubernetes things.
> >>>>>>>>>>
> >>>>>>>>>> Since you mentioned this, we are open to discuss this much more
> >>>>>>> deeply:
> >>>>>>>>>>
> >>>>>>>>>> It would be definitely a benefit to involve more “end”s in the
> >>>>>>> so-call
> >>>>>>>>>> “end to end” tests, I’m +1 to leverage SWCK as well.
> >>>>>>>>>>
> >>>>>>>>>> If we are going to use KIND, I hope we can give a script or
> >>>> something
> >>>>>>>>>> similar to install all the needed components and create cluster
> >>> for
> >>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> —————————
> >>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>
> >>>>>>>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <hanahmily@apache.org
> >
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> May SWCK help about provision and demo
> >>>>>>>>>>>
> >>>>>>>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone
> >>>>>>>> instance
> >>>>>>>>>> or
> >>>>>>>>>>> cluster for e2e. We could compose indicated CR yamls for
> >>> different
> >>>>>>>>>>> scenarios.
> >>>>>>>>>>> 2. Different demo for different cases is mandatory for e2e and
> >>>>>>> swck,
> >>>>>>>> we
> >>>>>>>>>>> could build a group of demo projects for them. With
> Kubernetes's
> >>>>>>>>> support,
> >>>>>>>>>>> we could
> >>>>>>>>>>> get free of verbose scripts that are written for installation,
> >>>>>>>>>> checking
> >>>>>>>>>>> the state of running applications, collecting logs.
> >>>>>>>>>>>
> >>>>>>>>>>> Ppl might have concerts about the minkube we currently have,
> >>>>>>> which's
> >>>>>>>>> hard
> >>>>>>>>>>> to debug and runs slowly.
> >>>>>>>>>>>
> >>>>>>>>>>> I prefer to kind[1] instead of minkube. From my experience,
> kind
> >>>>>>>> works
> >>>>>>>>>> fine
> >>>>>>>>>>> in an e2e scenario(limited cpu, memory resources), and it is
> run
> >>>>>>>>> million
> >>>>>>>>>>> times
> >>>>>>>>>>> on my team's e2e environment, which proves the stability and
> >>>>>>>>>>> maintainability of it.
> >>>>>>>>>>>
> >>>>>>>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
> >>>>>>>>>>>
> >>>>>>>>>>> regards, Hongtao
> >>>>>>>>>>>
> >>>>>>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
> >>>>>>>>>>>
> >>>>>>>>>>>> Inline
> >>>>>>>>>>>>
> >>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> >>>>>>> 下午11:16写道：
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Jiajing,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for sharing your context and joining the discussion.
> >>> It’s
> >>>>>>>> true
> >>>>>>>>>>>> that
> >>>>>>>>>>>>> we’re actually tackling the same problems in this domain.
> Some
> >>>> of
> >>>>>>>> the
> >>>>>>>>>>>>> issues you mentioned are solved in the current E2E framework
> >>>>>>> (like
> >>>>>>>>>>>> `retry`,
> >>>>>>>>>>>>> `timeout`, etc.), while some others are still there without
> >>>> ideal
> >>>>>>>>>>>> solution.
> >>>>>>>>>>>>> Other discussions, please see my comments inline.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> - the first issue is useful in health check steps, just like
> >>>> the
> >>>>>>>>> ones
> >>>>>>>>>>>>>> defined in readiness probe and liveness probe in kubelet.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes, the reason why we decided to pick docker-compose is
> that,
> >>>> it
> >>>>>>>> has
> >>>>>>>>>> the
> >>>>>>>>>>>>> same functionality with kubelet in terms of dependent
> services
> >>>>>>>>> startup
> >>>>>>>>>>>>> order and healthiness checking, while it’s more lite-weight
> >>> and
> >>>>>>>> easy
> >>>>>>>>>> for
> >>>>>>>>>>>>> both developers and GitHub Actions environment. Right?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> - the second issue is critical for highly-customized
> >>>> assertions,
> >>>>>>>> we
> >>>>>>>>>>>>>> can leverage the `text/template` module to provide highly
> >>>>>>>> extensive
> >>>>>>>>>>>>>> assertion functions
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> `text/template` is also the first thought that came to my
> mind
> >>>>>>> when
> >>>>>>>>>>>>> considering customized assertors, customized assertors are
> for
> >>>>>>> sure
> >>>>>>>>>>>>> critical in SkyWalking E2E tests because we have many kinds
> of
> >>>>>>>>>>>> assertions,
> >>>>>>>>>>>>> some of which are not foreseen until we actually need them,
> >>>>>>>>>>>>> `AtLeastOneGreaterThanZero` is an example.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> - the last but not the least, the lifecycle hooks can be
> used
> >>>>>>> for
> >>>>>>>>>>>>>> accurate control of test setup and teardown, such as
> >>>>>>>> `docker-compose
> >>>>>>>>>>>>>> build, up -d, down`
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The lifecycle hooks are what I didn’t take into consideration
> >>>>>>>>> because I
> >>>>>>>>>>>>> thought it is quite natural for test framework to provide
> this
> >>>>>>>>>> features,
> >>>>>>>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Together with the aforementioned ideas, we will have a
> >>>>>>> go-written,
> >>>>>>>>>>>>>> yaml-data driven e2e testing framework which can overcome
> the
> >>>>>>>>> current
> >>>>>>>>>>>>>> problems that occur,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I agree.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> in which the most annoying issue is the fragmentation of
> >>>>>>> different
> >>>>>>>>>>>>>> scripts in different submodules, IMO.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> That’s why I list the last item in the previous email, (Nice
> >>> to
> >>>>>>>> have,
> >>>>>>>>>>>> wrap
> >>>>>>>>>>>>> them into a GitHub Action), with that GitHub Action, we can
> >>>>>>> simply
> >>>>>>>>>>>>> configure a workflow/job in every submodule, and the tests
> are
> >>>>>>>>> executed
> >>>>>>>>>>>>> with the shared scripts, no copy-pasting.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Note, GitHub Action will require an Apache release every time
> >>>>>>>> because
> >>>>>>>>> we
> >>>>>>>>>>>> are distributing the binary.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sheng Wu 吴晟
> >>>>>>>>>>>> Twitter, wusheng1108
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>> Jiajing
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
> >>>>>>>>>>>> kezhenxu94@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> So, we would provide a script set to drive this?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> My question is more about,
> >>>>>>>>>>>>>>>> how to work with docker-compose
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It’s the same as what we’re doing now in the current
> version
> >>>> of
> >>>>>>>> E2E
> >>>>>>>>>>>>> tests, now we have many `docker-compose.yaml` files, and all
> >>> the
> >>>>>>>>>>>>> `docker-compose.yaml` files are reusable in the new
> framework,
> >>>> we
> >>>>>>>>> will
> >>>>>>>>>>>> keep
> >>>>>>>>>>>>> this mechanism as it is proved to be convenient and easy to
> >>> use
> >>>>>>>>>> (remember
> >>>>>>>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage
> >>>> E2E
> >>>>>>>>> test,
> >>>>>>>>>>>> not
> >>>>>>>>>>>>> yet merged though, it’s fast/easy to add a new case).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> —————————
> >>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
> >>>>>>>> wu.sheng.841108@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> >>>>>>>>> 下午9:16写道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I want to ask, how the docker-compose lands in your
> >>> mind? I
> >>>>>>>>> think
> >>>>>>>>>>>> CLI
> >>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>> control this part.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> docker-compose just fits the requirements perfectly that
> >>> we
> >>>>>>>> need,
> >>>>>>>>>> we
> >>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>> been using it in the current E2E and turns out it has
> >>>>>>>> advantages
> >>>>>>>>> in
> >>>>>>>>>>>>>>>>> starting many services correctly with health check and
> >>>>>>>> dependency
> >>>>>>>>>>>>>>>>> declaration. (One may say Kubernetes can do the same
> >>> thing,
> >>>>>>> but
> >>>>>>>>>> it’s
> >>>>>>>>>>>>> kind
> >>>>>>>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
> >>>>>>>> related
> >>>>>>>>>>>>> things,
> >>>>>>>>>>>>>>>>> it just provides query ability (the same as what it is
> >>> doing
> >>>>>>>> now,
> >>>>>>>>>>>>> just need
> >>>>>>>>>>>>>>>>> to check whether it covers all the needed query
> interfaces
> >>>> or
> >>>>>>>>> not),
> >>>>>>>>>>>>> we will
> >>>>>>>>>>>>>>>>> have a dedicated control program to
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no
> >>>> new
> >>>>>>>>>>>>> knowledge);
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> So, we would provide a script set to drive this? My
> >>> question
> >>>>>>> is
> >>>>>>>>> more
> >>>>>>>>>>>>> about,
> >>>>>>>>>>>>>>>> how to work with docker-compose, rather than challanging
> >>> the
> >>>>>>>>> choice.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Sheng Wu 吴晟
> >>>>>>>>>>>>>>>> Twitter, wusheng1108
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing
> >>>> repository
> >>>>>>>> and
> >>>>>>>>>>>>> codes);
> >>>>>>>>>>>>>>>>> (3) verify the actual data (by verification tool, a new
> >>>> tool,
> >>>>>>>>>> TODO);
> >>>>>>>>>>>>>>>>> (4) and determine the test result (success or failed);
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> —————————
> >>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Sheng Wu 吴晟
> >>>>>>>>>>>>>>>>>> Twitter, wusheng1108
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org>
> 于2020年11月15日周日
> >>>>>>>>>> 下午5:27写道：
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build
> >>>> the
> >>>>>>>>> next
> >>>>>>>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking,
> >>>>>>> which
> >>>>>>>>> may
> >>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>> be a
> >>>>>>>>>>>>>>>>>>> high priority but a necessity.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very
> >>>>>>> important
> >>>>>>>>>> role
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> contribution process, and the current E2E tests just
> >>> work
> >>>>>>>> well
> >>>>>>>>> to
> >>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>> every single pull request before merging, so why bother
> >>> to
> >>>>>>>>> build
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> next
> >>>>>>>>>>>>>>>>>>> generation of E2E test framework?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
> >>>>>>> sub-projects
> >>>>>>>>>> that
> >>>>>>>>>>>>> vary
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to
> >>>>>>> help
> >>>>>>>> us
> >>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> pull requests, but we can see that they reimplement the
> >>>> E2E
> >>>>>>>>> test
> >>>>>>>>>>>>>>>>> framework
> >>>>>>>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t
> >>>> have
> >>>>>>>> E2E
> >>>>>>>>>>>>> tests at
> >>>>>>>>>>>>>>>>>>> all). Reimplementing the E2E test framework is
> >>> unnecesarry
> >>>>>>>> and
> >>>>>>>>>>>>>>>>> introduces
> >>>>>>>>>>>>>>>>>>> more duplicated work when adding a new test case. The
> >>>>>>>> ultimate
> >>>>>>>>>>>> goal
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> reuse all the test cases when needed.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2. The current E2E framework is built with Java,
> running
> >>>>>>>>>>>> Java-based
> >>>>>>>>>>>>>>>>>>> program is not an easy thing for non-Java developers
> >>>>>>>>> (maintainers
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to
> >>>> set
> >>>>>>>> up
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> environment correctly and then run the tests, neither
> is
> >>>> it
> >>>>>>>> an
> >>>>>>>>>>>> easy
> >>>>>>>>>>>>>>>>> skill
> >>>>>>>>>>>>>>>>>>> to debug the tests.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
> >>>>>>> because
> >>>>>>>> we
> >>>>>>>>>>>>> added
> >>>>>>>>>>>>>>>>> many
> >>>>>>>>>>>>>>>>>>> cases and many duplicated codes that need to be
> removed.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Therefore we're planning to build an unified,
> >>> easy-to-use,
> >>>>>>>> and
> >>>>>>>>>>>> fast
> >>>>>>>>>>>>> E2E
> >>>>>>>>>>>>>>>>>>> test framework to benifit all the sub-projects.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Here are some rough ideas about this framework:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 0. (Unchanged) We will continue to use
> >>> docker-compose.yaml
> >>>>>>> to
> >>>>>>>>>>>>>>>>> orchestrate
> >>>>>>>>>>>>>>>>>>> the services that are to be tested, it’s proved to be
> >>>>>>>> friendly
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> debuggable because we can start it directly even if we
> >>> are
> >>>>>>>> not
> >>>>>>>>>>>>> writing
> >>>>>>>>>>>>>>>>> E2E
> >>>>>>>>>>>>>>>>>>> tests.
> >>>>>>>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI
> >>>>>>>> project
> >>>>>>>>> to
> >>>>>>>>>>>>> query
> >>>>>>>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
> >>>>>>>> versions
> >>>>>>>>>> of
> >>>>>>>>>>>>> query
> >>>>>>>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes
> >>>> and
> >>>>>>>> one
> >>>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> CLI).
> >>>>>>>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected
> data
> >>>>>>> and
> >>>>>>>>> the
> >>>>>>>>>>>>> actual
> >>>>>>>>>>>>>>>>>>> data, we now have a custom schema of the expected data
> >>>> yaml
> >>>>>>>>> file,
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> verification codes should be packaged into an
> executable
> >>>>>>>>> command
> >>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>> that it
> >>>>>>>>>>>>>>>>>>> can be executed standalone without Java and maven
> >>>>>>> knowledge.
> >>>>>>>> I
> >>>>>>>>>>>> hope
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> leverage existing standards to write the YAML files and
> >>> do
> >>>>>>>>>>>>>>>>> verifications.
> >>>>>>>>>>>>>>>>>>> 3. Build an orchestrating tool to start the
> >>>>>>>>>> `docker-compose.yaml`,
> >>>>>>>>>>>>>>>>> health
> >>>>>>>>>>>>>>>>>>> check, query actual data, and verify the result.
> >>>>>>>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub
> Actions
> >>>> to
> >>>>>>>>> share
> >>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>> sub-projects.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> If you have any other better idea or want to complement
> >>> to
> >>>>>>>> this
> >>>>>>>>>>>>>>>>> proposal,
> >>>>>>>>>>>>>>>>>>> please reply here. I will create an issue to track
> these
> >>>>>>>> tasks
> >>>>>>>>>>>>> later.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> We have two students (who just finished the Summer Code
> >>>>>>> 2020
> >>>>>>>>>>>>> Projects)
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I
> >>>> said
> >>>>>>>>> this
> >>>>>>>>>>>> is
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>> high priority, you have adequate time to get yourselves
> >>>>>>>>> familiar
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
> >>>>>>>> SkyWalking
> >>>>>>>>>>>> work
> >>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> well
> >>>>>>>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and
> >>>> willingness
> >>>>>>>> to
> >>>>>>>>>>>> help.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> [1] https://github.com/Humbertzhang
> >>>>>>>>>>>>>>>>>>> [2] https://github.com/fgksgf/
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> —————————
> >>>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> —————————
> >>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> —————————
> >>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >
> >
> >
> > —————————
> > Zhenxu Ke (柯振旭)
> > GitHub @kezhenxu94
>
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by "kezhenxu94@apache" <ke...@apache.org>.
Because now we have skywalking-eyes repository to host the infra tools, we can put the new E2E framework there so that it doesn’t mess up the CLI repo. Here [1] I created a new directory to host the E2E codes.
[1] https://github.com/apache/skywalking-eyes/pull/11
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
> On Dec 17, 2020, at 17:45, kezhenxu94@apache <ke...@apache.org> wrote:
>
> Hey, I’ve just drafted a design doc here [1], anyone interested in this please feel free to leave your comments, thanks!
>
> [1] https://github.com/apache/skywalking-website/pull/172
>
>> On Nov 26, 2020, at 14:41, Sheng Wu <wu...@gmail.com> wrote:
>>
>> You could learn from the Satellite project design,
>> 1. Discuss here, https://github.com/apache/skywalking/issues/5871
>> 2. Final design posted as a blog,
>> http://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/
>>
>> People could learn from these things when you join the community in the
>> future.
>>
>> Sheng Wu 吴晟
>> Twitter, wusheng1108
>>
>>
>> Hoshea Jiang <fg...@gmail.com> 于2020年11月26日周四 下午1:38写道：
>>
>>> I agree with the conclusion, and I hope to have some diagrams such as
>>> flowcharts or data flow diagrams, which help provide an overview of the E2E
>>> test before we start to do this.
>>>
>>> Ke Zhang <hu...@gmail.com> 于2020年11月26日周四 下午12:34写道：
>>>
>>>> I think supporting both docker-compose and KIND in the E2E test is
>>>> reasonable and necessary, and I also agree with other conclusions.😀
>>>>
>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月26日周四 上午11:58写道：
>>>>
>>>>>
>>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
>>>>>> 2. Totally replace the Java and Maven based driving system.
>>>>>> 3. Enhance CLI to validate the GraphQL
>>>>>> 4. Keep the agent test tool unchanged for now as it is already a
>>>> separate
>>>>>> system from the e2e.
>>>>>
>>>>> This conclusion is good for me.
>>>>>
>>>>> What are others opinions?
>>>>>
>>>>> —————————
>>>>> Zhenxu Ke (柯振旭)
>>>>> GitHub @kezhenxu94
>>>>>
>>>>>> On Nov 17, 2020, at 2:42 PM, Sheng Wu <wu...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Using k8s and docker-compose as 2 options in the test process is
>>>>>> reasonable, and should be supported.
>>>>>>
>>>>>> Let's keep the conclusion as simple as possible. In the next
>>> generation
>>>>> e2e
>>>>>> framework
>>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
>>>>>> 2. Totally replace the Java and Maven based driving system.
>>>>>> 3. Enhance CLI to validate the GraphQL
>>>>>> 4. Keep the agent test tool unchanged for now as it is already a
>>>> separate
>>>>>> system from the e2e.
>>>>>>
>>>>>> Are these good enough for everyone?
>>>>>>
>>>>>> Sheng Wu 吴晟
>>>>>> Twitter, wusheng1108
>>>>>>
>>>>>>
>>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
>>>>>>
>>>>>>>>
>>>>>>>> We can see from above, your discussion actually is only related to
>>> 3.
>>>>>>>
>>>>>>>
>>>>>>> That's the only I intend to discuss here if you notice my first
>>>>>>> response('May SWCK help about provision and demo") in this thread.
>>> For
>>>>>>> other parts, I don't have any tips to share, though.
>>>>>>> Let me explain it more clearly. I want kubernetes to be seen as an
>>>>>>> alternative to docker-compose to play a running-engine role in the
>>>> test
>>>>>>> framework. If kubernetes can't replace
>>>>>>> the latter one Is it able to participate in it to solve the
>>> dedicated
>>>> or
>>>>>>> special issues.
>>>>>>>
>>>>>>> Our e2e test and plugin test frameworks are already top-level
>>>> complicity
>>>>>>>> thing in the worldwide open source field. Please don't over-design
>>>> it,
>>>>>>> and
>>>>>>>> too aggressive, the world will stay in the hybrid env for a very
>>> long
>>>>>>> time,
>>>>>>>> same as the developer.
>>>>>>>
>>>>>>>
>>>>>>> Reusing current elaborate output might be reasonable. But in this
>>>>> thread,
>>>>>>> we're talking about the potential solution to build a
>>>>>>> next-generation framework, which might mean that the current
>>> framework
>>>>>>> is too complicated to maintain; we need a more tiny way to support
>>>> more
>>>>> and
>>>>>>> more cases.
>>>>>>>
>>>>>>> And finally, SWCK is gonna and has to build a similar test
>>> framework.
>>>> If
>>>>>>> the test framework opts for the kubernetes solution, we could share
>>>> test
>>>>>>> cases and the underlying framework. It's a more efficient
>>>>>>> and consistent path for the entire system.
>>>>>>>
>>>>>>> regards, Hongtao.
>>>>>>>
>>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I don't think we have to make the decision on k8s or out-of-k8s.
>>>>>>>> e2e and all integration tests have their scenarios like SkyWalking
>>>> have
>>>>>>>> multiple languages based and mesh-based, even Prometheus adoption.
>>>>>>>> The same thing happens on the test field, k8s, and out of k8s.
>>>>>>>>
>>>>>>>> I would like to agree with both of you, k8s is needed, no matter in
>>>>> mesh
>>>>>>>> solution(ALS, telemetry v2), but also not anyone needs that, such
>>> as
>>>>>>> agent
>>>>>>>> developer. We wouldn't deny the fact, the docker-compose is more
>>>>>>>> lightweight and easier to learn.
>>>>>>>> So, let me get this straight, for the test you need
>>>>>>>> 1. Build the source from different repos
>>>>>>>> 2. Build the images
>>>>>>>> 3. Use some platform(docker-compose or k8s) to run the images in
>>> one
>>>>>>> piece.
>>>>>>>> 4. Give some inputs to the env
>>>>>>>> 5. Use some tools/ways to check what should happen according to
>>> (4)'s
>>>>>>> input
>>>>>>>> 6. Checked or timeout failure.
>>>>>>>>
>>>>>>>> We can see from above, your discussion actually is only related to
>>> 3.
>>>>>>> Let's
>>>>>>>> not see this as a battle, we need both. Many developers/users use
>>> OAP
>>>>> and
>>>>>>>> SkyWalking out of k8s, that is a simple fact.
>>>>>>>> Our e2e test and plugin test frameworks are already top-level
>>>>> complicity
>>>>>>>> thing in the worldwide open source field. Please don't over-design
>>>> it,
>>>>>>> and
>>>>>>>> too aggressive, the world will stay in the hybrid env for a very
>>> long
>>>>>>> time,
>>>>>>>> same as the developer.
>>>>>>>>
>>>>>>>> Also, let's keep in mind, k8s is popular becomes it doesn't require
>>>> the
>>>>>>>> developers to understand as the VM did. We as an open-source
>>>> community,
>>>>>>> are
>>>>>>>> focusing on the developer's capabilities.
>>>>>>>>
>>>>>>>> Sheng Wu 吴晟
>>>>>>>> Twitter, wusheng1108
>>>>>>>>
>>>>>>>>
>>>>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> but I have been thinking whether or not it is overkill for
>>> testing
>>>>>>>>>> purposes to introduce Kubernetes things.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> we all know the fact docker-compose is more portable than
>>>> Kubernetes,
>>>>>>>>> friendly to local running. But there're also some benefits to
>>>> replace
>>>>>>> it
>>>>>>>>> with kubernetes system:
>>>>>>>>>
>>>>>>>>> 1. SWCK has to test based on a real Kubernetes environment if the
>>>> main
>>>>>>>>> repo test framework doesn't support it(following the
>>> docker-compose
>>>>>>>> stack),
>>>>>>>>> That means skywalking ecosystem MUST introduce kubernetes which is
>>>> not
>>>>>>> an
>>>>>>>>> optional component.
>>>>>>>>>
>>>>>>>>> 2. Kubernetes support to scale applications at runtime, that help
>>>>>>>> improve
>>>>>>>>> the process of e2e. For instance, we could merge a single instance
>>>> and
>>>>>>>>> cluster test, by scaling up replicas. This strategy will reduce
>>> the
>>>>>>> total
>>>>>>>>> time sharply.
>>>>>>>>>
>>>>>>>>> If we are going to use KIND, I hope we can give a script or
>>>> something
>>>>>>>>>> similar to install all the needed components and create cluster
>>> for
>>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That's SWCK's capacity to provide a standard and simple way to
>>>> create
>>>>> a
>>>>>>>>> cluster in every environment,
>>>>>>>>>
>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一
>>> 下午7:00写道：
>>>>>>>>>
>>>>>>>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting
>>>>>>>> Kubernetes-based
>>>>>>>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too,
>>>>>>> but I
>>>>>>>>>> have been thinking whether or not it is overkill for testing
>>>> purposes
>>>>>>>> to
>>>>>>>>>> introduce Kubernetes things.
>>>>>>>>>>
>>>>>>>>>> Since you mentioned this, we are open to discuss this much more
>>>>>>> deeply:
>>>>>>>>>>
>>>>>>>>>> It would be definitely a benefit to involve more “end”s in the
>>>>>>> so-call
>>>>>>>>>> “end to end” tests, I’m +1 to leverage SWCK as well.
>>>>>>>>>>
>>>>>>>>>> If we are going to use KIND, I hope we can give a script or
>>>> something
>>>>>>>>>> similar to install all the needed components and create cluster
>>> for
>>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> —————————
>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>
>>>>>>>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> May SWCK help about provision and demo
>>>>>>>>>>>
>>>>>>>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone
>>>>>>>> instance
>>>>>>>>>> or
>>>>>>>>>>> cluster for e2e. We could compose indicated CR yamls for
>>> different
>>>>>>>>>>> scenarios.
>>>>>>>>>>> 2. Different demo for different cases is mandatory for e2e and
>>>>>>> swck,
>>>>>>>> we
>>>>>>>>>>> could build a group of demo projects for them. With Kubernetes's
>>>>>>>>> support,
>>>>>>>>>>> we could
>>>>>>>>>>> get free of verbose scripts that are written for installation,
>>>>>>>>>> checking
>>>>>>>>>>> the state of running applications, collecting logs.
>>>>>>>>>>>
>>>>>>>>>>> Ppl might have concerts about the minkube we currently have,
>>>>>>> which's
>>>>>>>>> hard
>>>>>>>>>>> to debug and runs slowly.
>>>>>>>>>>>
>>>>>>>>>>> I prefer to kind[1] instead of minkube. From my experience, kind
>>>>>>>> works
>>>>>>>>>> fine
>>>>>>>>>>> in an e2e scenario(limited cpu, memory resources), and it is run
>>>>>>>>> million
>>>>>>>>>>> times
>>>>>>>>>>> on my team's e2e environment, which proves the stability and
>>>>>>>>>>> maintainability of it.
>>>>>>>>>>>
>>>>>>>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
>>>>>>>>>>>
>>>>>>>>>>> regards, Hongtao
>>>>>>>>>>>
>>>>>>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
>>>>>>>>>>>
>>>>>>>>>>>> Inline
>>>>>>>>>>>>
>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>>> 下午11:16写道：
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Jiajing,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for sharing your context and joining the discussion.
>>> It’s
>>>>>>>> true
>>>>>>>>>>>> that
>>>>>>>>>>>>> we’re actually tackling the same problems in this domain. Some
>>>> of
>>>>>>>> the
>>>>>>>>>>>>> issues you mentioned are solved in the current E2E framework
>>>>>>> (like
>>>>>>>>>>>> `retry`,
>>>>>>>>>>>>> `timeout`, etc.), while some others are still there without
>>>> ideal
>>>>>>>>>>>> solution.
>>>>>>>>>>>>> Other discussions, please see my comments inline.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> - the first issue is useful in health check steps, just like
>>>> the
>>>>>>>>> ones
>>>>>>>>>>>>>> defined in readiness probe and liveness probe in kubelet.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, the reason why we decided to pick docker-compose is that,
>>>> it
>>>>>>>> has
>>>>>>>>>> the
>>>>>>>>>>>>> same functionality with kubelet in terms of dependent services
>>>>>>>>> startup
>>>>>>>>>>>>> order and healthiness checking, while it’s more lite-weight
>>> and
>>>>>>>> easy
>>>>>>>>>> for
>>>>>>>>>>>>> both developers and GitHub Actions environment. Right?
>>>>>>>>>>>>>
>>>>>>>>>>>>>> - the second issue is critical for highly-customized
>>>> assertions,
>>>>>>>> we
>>>>>>>>>>>>>> can leverage the `text/template` module to provide highly
>>>>>>>> extensive
>>>>>>>>>>>>>> assertion functions
>>>>>>>>>>>>>
>>>>>>>>>>>>> `text/template` is also the first thought that came to my mind
>>>>>>> when
>>>>>>>>>>>>> considering customized assertors, customized assertors are for
>>>>>>> sure
>>>>>>>>>>>>> critical in SkyWalking E2E tests because we have many kinds of
>>>>>>>>>>>> assertions,
>>>>>>>>>>>>> some of which are not foreseen until we actually need them,
>>>>>>>>>>>>> `AtLeastOneGreaterThanZero` is an example.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> - the last but not the least, the lifecycle hooks can be used
>>>>>>> for
>>>>>>>>>>>>>> accurate control of test setup and teardown, such as
>>>>>>>> `docker-compose
>>>>>>>>>>>>>> build, up -d, down`
>>>>>>>>>>>>>
>>>>>>>>>>>>> The lifecycle hooks are what I didn’t take into consideration
>>>>>>>>> because I
>>>>>>>>>>>>> thought it is quite natural for test framework to provide this
>>>>>>>>>> features,
>>>>>>>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Together with the aforementioned ideas, we will have a
>>>>>>> go-written,
>>>>>>>>>>>>>> yaml-data driven e2e testing framework which can overcome the
>>>>>>>>> current
>>>>>>>>>>>>>> problems that occur,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> in which the most annoying issue is the fragmentation of
>>>>>>> different
>>>>>>>>>>>>>> scripts in different submodules, IMO.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That’s why I list the last item in the previous email, (Nice
>>> to
>>>>>>>> have,
>>>>>>>>>>>> wrap
>>>>>>>>>>>>> them into a GitHub Action), with that GitHub Action, we can
>>>>>>> simply
>>>>>>>>>>>>> configure a workflow/job in every submodule, and the tests are
>>>>>>>>> executed
>>>>>>>>>>>>> with the shared scripts, no copy-pasting.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Note, GitHub Action will require an Apache release every time
>>>>>>>> because
>>>>>>>>> we
>>>>>>>>>>>> are distributing the binary.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Jiajing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
>>>>>>>>>>>> kezhenxu94@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So, we would provide a script set to drive this?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My question is more about,
>>>>>>>>>>>>>>>> how to work with docker-compose
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It’s the same as what we’re doing now in the current version
>>>> of
>>>>>>>> E2E
>>>>>>>>>>>>> tests, now we have many `docker-compose.yaml` files, and all
>>> the
>>>>>>>>>>>>> `docker-compose.yaml` files are reusable in the new framework,
>>>> we
>>>>>>>>> will
>>>>>>>>>>>> keep
>>>>>>>>>>>>> this mechanism as it is proved to be convenient and easy to
>>> use
>>>>>>>>>> (remember
>>>>>>>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage
>>>> E2E
>>>>>>>>> test,
>>>>>>>>>>>> not
>>>>>>>>>>>>> yet merged though, it’s fast/easy to add a new case).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
>>>>>>>> wu.sheng.841108@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>>>>> 下午9:16写道：
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I want to ask, how the docker-compose lands in your
>>> mind? I
>>>>>>>>> think
>>>>>>>>>>>> CLI
>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>> control this part.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> docker-compose just fits the requirements perfectly that
>>> we
>>>>>>>> need,
>>>>>>>>>> we
>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> been using it in the current E2E and turns out it has
>>>>>>>> advantages
>>>>>>>>> in
>>>>>>>>>>>>>>>>> starting many services correctly with health check and
>>>>>>>> dependency
>>>>>>>>>>>>>>>>> declaration. (One may say Kubernetes can do the same
>>> thing,
>>>>>>> but
>>>>>>>>>> it’s
>>>>>>>>>>>>> kind
>>>>>>>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
>>>>>>>> related
>>>>>>>>>>>>> things,
>>>>>>>>>>>>>>>>> it just provides query ability (the same as what it is
>>> doing
>>>>>>>> now,
>>>>>>>>>>>>> just need
>>>>>>>>>>>>>>>>> to check whether it covers all the needed query interfaces
>>>> or
>>>>>>>>> not),
>>>>>>>>>>>>> we will
>>>>>>>>>>>>>>>>> have a dedicated control program to
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no
>>>> new
>>>>>>>>>>>>> knowledge);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So, we would provide a script set to drive this? My
>>> question
>>>>>>> is
>>>>>>>>> more
>>>>>>>>>>>>> about,
>>>>>>>>>>>>>>>> how to work with docker-compose, rather than challanging
>>> the
>>>>>>>>> choice.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing
>>>> repository
>>>>>>>> and
>>>>>>>>>>>>> codes);
>>>>>>>>>>>>>>>>> (3) verify the actual data (by verification tool, a new
>>>> tool,
>>>>>>>>>> TODO);
>>>>>>>>>>>>>>>>> (4) and determine the test result (success or failed);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>>>>>> 下午5:27写道：
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build
>>>> the
>>>>>>>>> next
>>>>>>>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking,
>>>>>>> which
>>>>>>>>> may
>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> be a
>>>>>>>>>>>>>>>>>>> high priority but a necessity.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very
>>>>>>> important
>>>>>>>>>> role
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> contribution process, and the current E2E tests just
>>> work
>>>>>>>> well
>>>>>>>>> to
>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>> every single pull request before merging, so why bother
>>> to
>>>>>>>>> build
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> next
>>>>>>>>>>>>>>>>>>> generation of E2E test framework?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
>>>>>>> sub-projects
>>>>>>>>>> that
>>>>>>>>>>>>> vary
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to
>>>>>>> help
>>>>>>>> us
>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> pull requests, but we can see that they reimplement the
>>>> E2E
>>>>>>>>> test
>>>>>>>>>>>>>>>>> framework
>>>>>>>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t
>>>> have
>>>>>>>> E2E
>>>>>>>>>>>>> tests at
>>>>>>>>>>>>>>>>>>> all). Reimplementing the E2E test framework is
>>> unnecesarry
>>>>>>>> and
>>>>>>>>>>>>>>>>> introduces
>>>>>>>>>>>>>>>>>>> more duplicated work when adding a new test case. The
>>>>>>>> ultimate
>>>>>>>>>>>> goal
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> reuse all the test cases when needed.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running
>>>>>>>>>>>> Java-based
>>>>>>>>>>>>>>>>>>> program is not an easy thing for non-Java developers
>>>>>>>>> (maintainers
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to
>>>> set
>>>>>>>> up
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> environment correctly and then run the tests, neither is
>>>> it
>>>>>>>> an
>>>>>>>>>>>> easy
>>>>>>>>>>>>>>>>> skill
>>>>>>>>>>>>>>>>>>> to debug the tests.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
>>>>>>> because
>>>>>>>> we
>>>>>>>>>>>>> added
>>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>> cases and many duplicated codes that need to be removed.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Therefore we're planning to build an unified,
>>> easy-to-use,
>>>>>>>> and
>>>>>>>>>>>> fast
>>>>>>>>>>>>> E2E
>>>>>>>>>>>>>>>>>>> test framework to benifit all the sub-projects.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here are some rough ideas about this framework:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0. (Unchanged) We will continue to use
>>> docker-compose.yaml
>>>>>>> to
>>>>>>>>>>>>>>>>> orchestrate
>>>>>>>>>>>>>>>>>>> the services that are to be tested, it’s proved to be
>>>>>>>> friendly
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> debuggable because we can start it directly even if we
>>> are
>>>>>>>> not
>>>>>>>>>>>>> writing
>>>>>>>>>>>>>>>>> E2E
>>>>>>>>>>>>>>>>>>> tests.
>>>>>>>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI
>>>>>>>> project
>>>>>>>>> to
>>>>>>>>>>>>> query
>>>>>>>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
>>>>>>>> versions
>>>>>>>>>> of
>>>>>>>>>>>>> query
>>>>>>>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes
>>>> and
>>>>>>>> one
>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> CLI).
>>>>>>>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data
>>>>>>> and
>>>>>>>>> the
>>>>>>>>>>>>> actual
>>>>>>>>>>>>>>>>>>> data, we now have a custom schema of the expected data
>>>> yaml
>>>>>>>>> file,
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> verification codes should be packaged into an executable
>>>>>>>>> command
>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> that it
>>>>>>>>>>>>>>>>>>> can be executed standalone without Java and maven
>>>>>>> knowledge.
>>>>>>>> I
>>>>>>>>>>>> hope
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> leverage existing standards to write the YAML files and
>>> do
>>>>>>>>>>>>>>>>> verifications.
>>>>>>>>>>>>>>>>>>> 3. Build an orchestrating tool to start the
>>>>>>>>>> `docker-compose.yaml`,
>>>>>>>>>>>>>>>>> health
>>>>>>>>>>>>>>>>>>> check, query actual data, and verify the result.
>>>>>>>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions
>>>> to
>>>>>>>>> share
>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>> sub-projects.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If you have any other better idea or want to complement
>>> to
>>>>>>>> this
>>>>>>>>>>>>>>>>> proposal,
>>>>>>>>>>>>>>>>>>> please reply here. I will create an issue to track these
>>>>>>>> tasks
>>>>>>>>>>>>> later.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We have two students (who just finished the Summer Code
>>>>>>> 2020
>>>>>>>>>>>>> Projects)
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I
>>>> said
>>>>>>>>> this
>>>>>>>>>>>> is
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> high priority, you have adequate time to get yourselves
>>>>>>>>> familiar
>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
>>>>>>>> SkyWalking
>>>>>>>>>>>> work
>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> well
>>>>>>>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and
>>>> willingness
>>>>>>>> to
>>>>>>>>>>>> help.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1] https://github.com/Humbertzhang
>>>>>>>>>>>>>>>>>>> [2] https://github.com/fgksgf/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> —————————
>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>
>
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by "kezhenxu94@apache" <ke...@apache.org>.
Hey, I’ve just drafted a design doc here [1], anyone interested in this please feel free to leave your comments, thanks!
[1] https://github.com/apache/skywalking-website/pull/172
> On Nov 26, 2020, at 14:41, Sheng Wu <wu...@gmail.com> wrote:
>
> You could learn from the Satellite project design,
> 1. Discuss here, https://github.com/apache/skywalking/issues/5871
> 2. Final design posted as a blog,
> http://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/
>
> People could learn from these things when you join the community in the
> future.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Hoshea Jiang <fg...@gmail.com> 于2020年11月26日周四 下午1:38写道：
>
>> I agree with the conclusion, and I hope to have some diagrams such as
>> flowcharts or data flow diagrams, which help provide an overview of the E2E
>> test before we start to do this.
>>
>> Ke Zhang <hu...@gmail.com> 于2020年11月26日周四 下午12:34写道：
>>
>>> I think supporting both docker-compose and KIND in the E2E test is
>>> reasonable and necessary, and I also agree with other conclusions.😀
>>>
>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月26日周四 上午11:58写道：
>>>
>>>>
>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
>>>>> 2. Totally replace the Java and Maven based driving system.
>>>>> 3. Enhance CLI to validate the GraphQL
>>>>> 4. Keep the agent test tool unchanged for now as it is already a
>>> separate
>>>>> system from the e2e.
>>>>
>>>> This conclusion is good for me.
>>>>
>>>> What are others opinions?
>>>>
>>>> —————————
>>>> Zhenxu Ke (柯振旭)
>>>> GitHub @kezhenxu94
>>>>
>>>>> On Nov 17, 2020, at 2:42 PM, Sheng Wu <wu...@gmail.com>
>>> wrote:
>>>>>
>>>>> Using k8s and docker-compose as 2 options in the test process is
>>>>> reasonable, and should be supported.
>>>>>
>>>>> Let's keep the conclusion as simple as possible. In the next
>> generation
>>>> e2e
>>>>> framework
>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
>>>>> 2. Totally replace the Java and Maven based driving system.
>>>>> 3. Enhance CLI to validate the GraphQL
>>>>> 4. Keep the agent test tool unchanged for now as it is already a
>>> separate
>>>>> system from the e2e.
>>>>>
>>>>> Are these good enough for everyone?
>>>>>
>>>>> Sheng Wu 吴晟
>>>>> Twitter, wusheng1108
>>>>>
>>>>>
>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
>>>>>
>>>>>>>
>>>>>>> We can see from above, your discussion actually is only related to
>> 3.
>>>>>>
>>>>>>
>>>>>> That's the only I intend to discuss here if you notice my first
>>>>>> response('May SWCK help about provision and demo") in this thread.
>> For
>>>>>> other parts, I don't have any tips to share, though.
>>>>>> Let me explain it more clearly. I want kubernetes to be seen as an
>>>>>> alternative to docker-compose to play a running-engine role in the
>>> test
>>>>>> framework. If kubernetes can't replace
>>>>>> the latter one Is it able to participate in it to solve the
>> dedicated
>>> or
>>>>>> special issues.
>>>>>>
>>>>>> Our e2e test and plugin test frameworks are already top-level
>>> complicity
>>>>>>> thing in the worldwide open source field. Please don't over-design
>>> it,
>>>>>> and
>>>>>>> too aggressive, the world will stay in the hybrid env for a very
>> long
>>>>>> time,
>>>>>>> same as the developer.
>>>>>>
>>>>>>
>>>>>> Reusing current elaborate output might be reasonable. But in this
>>>> thread,
>>>>>> we're talking about the potential solution to build a
>>>>>> next-generation framework, which might mean that the current
>> framework
>>>>>> is too complicated to maintain; we need a more tiny way to support
>>> more
>>>> and
>>>>>> more cases.
>>>>>>
>>>>>> And finally, SWCK is gonna and has to build a similar test
>> framework.
>>> If
>>>>>> the test framework opts for the kubernetes solution, we could share
>>> test
>>>>>> cases and the underlying framework. It's a more efficient
>>>>>> and consistent path for the entire system.
>>>>>>
>>>>>> regards, Hongtao.
>>>>>>
>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I don't think we have to make the decision on k8s or out-of-k8s.
>>>>>>> e2e and all integration tests have their scenarios like SkyWalking
>>> have
>>>>>>> multiple languages based and mesh-based, even Prometheus adoption.
>>>>>>> The same thing happens on the test field, k8s, and out of k8s.
>>>>>>>
>>>>>>> I would like to agree with both of you, k8s is needed, no matter in
>>>> mesh
>>>>>>> solution(ALS, telemetry v2), but also not anyone needs that, such
>> as
>>>>>> agent
>>>>>>> developer. We wouldn't deny the fact, the docker-compose is more
>>>>>>> lightweight and easier to learn.
>>>>>>> So, let me get this straight, for the test you need
>>>>>>> 1. Build the source from different repos
>>>>>>> 2. Build the images
>>>>>>> 3. Use some platform(docker-compose or k8s) to run the images in
>> one
>>>>>> piece.
>>>>>>> 4. Give some inputs to the env
>>>>>>> 5. Use some tools/ways to check what should happen according to
>> (4)'s
>>>>>> input
>>>>>>> 6. Checked or timeout failure.
>>>>>>>
>>>>>>> We can see from above, your discussion actually is only related to
>> 3.
>>>>>> Let's
>>>>>>> not see this as a battle, we need both. Many developers/users use
>> OAP
>>>> and
>>>>>>> SkyWalking out of k8s, that is a simple fact.
>>>>>>> Our e2e test and plugin test frameworks are already top-level
>>>> complicity
>>>>>>> thing in the worldwide open source field. Please don't over-design
>>> it,
>>>>>> and
>>>>>>> too aggressive, the world will stay in the hybrid env for a very
>> long
>>>>>> time,
>>>>>>> same as the developer.
>>>>>>>
>>>>>>> Also, let's keep in mind, k8s is popular becomes it doesn't require
>>> the
>>>>>>> developers to understand as the VM did. We as an open-source
>>> community,
>>>>>> are
>>>>>>> focusing on the developer's capabilities.
>>>>>>>
>>>>>>> Sheng Wu 吴晟
>>>>>>> Twitter, wusheng1108
>>>>>>>
>>>>>>>
>>>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
>>>>>>>
>>>>>>>>>
>>>>>>>>> but I have been thinking whether or not it is overkill for
>> testing
>>>>>>>>> purposes to introduce Kubernetes things.
>>>>>>>>
>>>>>>>>
>>>>>>>> we all know the fact docker-compose is more portable than
>>> Kubernetes,
>>>>>>>> friendly to local running. But there're also some benefits to
>>> replace
>>>>>> it
>>>>>>>> with kubernetes system:
>>>>>>>>
>>>>>>>> 1. SWCK has to test based on a real Kubernetes environment if the
>>> main
>>>>>>>> repo test framework doesn't support it(following the
>> docker-compose
>>>>>>> stack),
>>>>>>>> That means skywalking ecosystem MUST introduce kubernetes which is
>>> not
>>>>>> an
>>>>>>>> optional component.
>>>>>>>>
>>>>>>>> 2. Kubernetes support to scale applications at runtime, that help
>>>>>>> improve
>>>>>>>> the process of e2e. For instance, we could merge a single instance
>>> and
>>>>>>>> cluster test, by scaling up replicas. This strategy will reduce
>> the
>>>>>> total
>>>>>>>> time sharply.
>>>>>>>>
>>>>>>>> If we are going to use KIND, I hope we can give a script or
>>> something
>>>>>>>>> similar to install all the needed components and create cluster
>> for
>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
>>>>>>>>
>>>>>>>>
>>>>>>>> That's SWCK's capacity to provide a standard and simple way to
>>> create
>>>> a
>>>>>>>> cluster in every environment,
>>>>>>>>
>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一
>> 下午7:00写道：
>>>>>>>>
>>>>>>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting
>>>>>>> Kubernetes-based
>>>>>>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too,
>>>>>> but I
>>>>>>>>> have been thinking whether or not it is overkill for testing
>>> purposes
>>>>>>> to
>>>>>>>>> introduce Kubernetes things.
>>>>>>>>>
>>>>>>>>> Since you mentioned this, we are open to discuss this much more
>>>>>> deeply:
>>>>>>>>>
>>>>>>>>> It would be definitely a benefit to involve more “end”s in the
>>>>>> so-call
>>>>>>>>> “end to end” tests, I’m +1 to leverage SWCK as well.
>>>>>>>>>
>>>>>>>>> If we are going to use KIND, I hope we can give a script or
>>> something
>>>>>>>>> similar to install all the needed components and create cluster
>> for
>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> —————————
>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>
>>>>>>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> May SWCK help about provision and demo
>>>>>>>>>>
>>>>>>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone
>>>>>>> instance
>>>>>>>>> or
>>>>>>>>>> cluster for e2e. We could compose indicated CR yamls for
>> different
>>>>>>>>>> scenarios.
>>>>>>>>>> 2. Different demo for different cases is mandatory for e2e and
>>>>>> swck,
>>>>>>> we
>>>>>>>>>> could build a group of demo projects for them. With Kubernetes's
>>>>>>>> support,
>>>>>>>>>> we could
>>>>>>>>>> get free of verbose scripts that are written for installation,
>>>>>>>>> checking
>>>>>>>>>> the state of running applications, collecting logs.
>>>>>>>>>>
>>>>>>>>>> Ppl might have concerts about the minkube we currently have,
>>>>>> which's
>>>>>>>> hard
>>>>>>>>>> to debug and runs slowly.
>>>>>>>>>>
>>>>>>>>>> I prefer to kind[1] instead of minkube. From my experience, kind
>>>>>>> works
>>>>>>>>> fine
>>>>>>>>>> in an e2e scenario(limited cpu, memory resources), and it is run
>>>>>>>> million
>>>>>>>>>> times
>>>>>>>>>> on my team's e2e environment, which proves the stability and
>>>>>>>>>> maintainability of it.
>>>>>>>>>>
>>>>>>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
>>>>>>>>>>
>>>>>>>>>> regards, Hongtao
>>>>>>>>>>
>>>>>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
>>>>>>>>>>
>>>>>>>>>>> Inline
>>>>>>>>>>>
>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>> 下午11:16写道：
>>>>>>>>>>>
>>>>>>>>>>>> Hi Jiajing,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for sharing your context and joining the discussion.
>> It’s
>>>>>>> true
>>>>>>>>>>> that
>>>>>>>>>>>> we’re actually tackling the same problems in this domain. Some
>>> of
>>>>>>> the
>>>>>>>>>>>> issues you mentioned are solved in the current E2E framework
>>>>>> (like
>>>>>>>>>>> `retry`,
>>>>>>>>>>>> `timeout`, etc.), while some others are still there without
>>> ideal
>>>>>>>>>>> solution.
>>>>>>>>>>>> Other discussions, please see my comments inline.
>>>>>>>>>>>>
>>>>>>>>>>>>> - the first issue is useful in health check steps, just like
>>> the
>>>>>>>> ones
>>>>>>>>>>>>> defined in readiness probe and liveness probe in kubelet.
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, the reason why we decided to pick docker-compose is that,
>>> it
>>>>>>> has
>>>>>>>>> the
>>>>>>>>>>>> same functionality with kubelet in terms of dependent services
>>>>>>>> startup
>>>>>>>>>>>> order and healthiness checking, while it’s more lite-weight
>> and
>>>>>>> easy
>>>>>>>>> for
>>>>>>>>>>>> both developers and GitHub Actions environment. Right?
>>>>>>>>>>>>
>>>>>>>>>>>>> - the second issue is critical for highly-customized
>>> assertions,
>>>>>>> we
>>>>>>>>>>>>> can leverage the `text/template` module to provide highly
>>>>>>> extensive
>>>>>>>>>>>>> assertion functions
>>>>>>>>>>>>
>>>>>>>>>>>> `text/template` is also the first thought that came to my mind
>>>>>> when
>>>>>>>>>>>> considering customized assertors, customized assertors are for
>>>>>> sure
>>>>>>>>>>>> critical in SkyWalking E2E tests because we have many kinds of
>>>>>>>>>>> assertions,
>>>>>>>>>>>> some of which are not foreseen until we actually need them,
>>>>>>>>>>>> `AtLeastOneGreaterThanZero` is an example.
>>>>>>>>>>>>
>>>>>>>>>>>>> - the last but not the least, the lifecycle hooks can be used
>>>>>> for
>>>>>>>>>>>>> accurate control of test setup and teardown, such as
>>>>>>> `docker-compose
>>>>>>>>>>>>> build, up -d, down`
>>>>>>>>>>>>
>>>>>>>>>>>> The lifecycle hooks are what I didn’t take into consideration
>>>>>>>> because I
>>>>>>>>>>>> thought it is quite natural for test framework to provide this
>>>>>>>>> features,
>>>>>>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part.
>>>>>>>>>>>>
>>>>>>>>>>>>> Together with the aforementioned ideas, we will have a
>>>>>> go-written,
>>>>>>>>>>>>> yaml-data driven e2e testing framework which can overcome the
>>>>>>>> current
>>>>>>>>>>>>> problems that occur,
>>>>>>>>>>>>
>>>>>>>>>>>> I agree.
>>>>>>>>>>>>
>>>>>>>>>>>>> in which the most annoying issue is the fragmentation of
>>>>>> different
>>>>>>>>>>>>> scripts in different submodules, IMO.
>>>>>>>>>>>>
>>>>>>>>>>>> That’s why I list the last item in the previous email, (Nice
>> to
>>>>>>> have,
>>>>>>>>>>> wrap
>>>>>>>>>>>> them into a GitHub Action), with that GitHub Action, we can
>>>>>> simply
>>>>>>>>>>>> configure a workflow/job in every submodule, and the tests are
>>>>>>>> executed
>>>>>>>>>>>> with the shared scripts, no copy-pasting.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Note, GitHub Action will require an Apache release every time
>>>>>>> because
>>>>>>>> we
>>>>>>>>>>> are distributing the binary.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Jiajing
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
>>>>>>>>>>> kezhenxu94@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, we would provide a script set to drive this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My question is more about,
>>>>>>>>>>>>>>> how to work with docker-compose
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It’s the same as what we’re doing now in the current version
>>> of
>>>>>>> E2E
>>>>>>>>>>>> tests, now we have many `docker-compose.yaml` files, and all
>> the
>>>>>>>>>>>> `docker-compose.yaml` files are reusable in the new framework,
>>> we
>>>>>>>> will
>>>>>>>>>>> keep
>>>>>>>>>>>> this mechanism as it is proved to be convenient and easy to
>> use
>>>>>>>>> (remember
>>>>>>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage
>>> E2E
>>>>>>>> test,
>>>>>>>>>>> not
>>>>>>>>>>>> yet merged though, it’s fast/easy to add a new case).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
>>>>>>> wu.sheng.841108@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>>>> 下午9:16写道：
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I want to ask, how the docker-compose lands in your
>> mind? I
>>>>>>>> think
>>>>>>>>>>> CLI
>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>> control this part.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> docker-compose just fits the requirements perfectly that
>> we
>>>>>>> need,
>>>>>>>>> we
>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> been using it in the current E2E and turns out it has
>>>>>>> advantages
>>>>>>>> in
>>>>>>>>>>>>>>>> starting many services correctly with health check and
>>>>>>> dependency
>>>>>>>>>>>>>>>> declaration. (One may say Kubernetes can do the same
>> thing,
>>>>>> but
>>>>>>>>> it’s
>>>>>>>>>>>> kind
>>>>>>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
>>>>>>> related
>>>>>>>>>>>> things,
>>>>>>>>>>>>>>>> it just provides query ability (the same as what it is
>> doing
>>>>>>> now,
>>>>>>>>>>>> just need
>>>>>>>>>>>>>>>> to check whether it covers all the needed query interfaces
>>> or
>>>>>>>> not),
>>>>>>>>>>>> we will
>>>>>>>>>>>>>>>> have a dedicated control program to
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no
>>> new
>>>>>>>>>>>> knowledge);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, we would provide a script set to drive this? My
>> question
>>>>>> is
>>>>>>>> more
>>>>>>>>>>>> about,
>>>>>>>>>>>>>>> how to work with docker-compose, rather than challanging
>> the
>>>>>>>> choice.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing
>>> repository
>>>>>>> and
>>>>>>>>>>>> codes);
>>>>>>>>>>>>>>>> (3) verify the actual data (by verification tool, a new
>>> tool,
>>>>>>>>> TODO);
>>>>>>>>>>>>>>>> (4) and determine the test result (success or failed);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>>>>> 下午5:27写道：
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build
>>> the
>>>>>>>> next
>>>>>>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking,
>>>>>> which
>>>>>>>> may
>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> be a
>>>>>>>>>>>>>>>>>> high priority but a necessity.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very
>>>>>> important
>>>>>>>>> role
>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> contribution process, and the current E2E tests just
>> work
>>>>>>> well
>>>>>>>> to
>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>> every single pull request before merging, so why bother
>> to
>>>>>>>> build
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> next
>>>>>>>>>>>>>>>>>> generation of E2E test framework?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
>>>>>> sub-projects
>>>>>>>>> that
>>>>>>>>>>>> vary
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to
>>>>>> help
>>>>>>> us
>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> pull requests, but we can see that they reimplement the
>>> E2E
>>>>>>>> test
>>>>>>>>>>>>>>>> framework
>>>>>>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t
>>> have
>>>>>>> E2E
>>>>>>>>>>>> tests at
>>>>>>>>>>>>>>>>>> all). Reimplementing the E2E test framework is
>> unnecesarry
>>>>>>> and
>>>>>>>>>>>>>>>> introduces
>>>>>>>>>>>>>>>>>> more duplicated work when adding a new test case. The
>>>>>>> ultimate
>>>>>>>>>>> goal
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> reuse all the test cases when needed.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running
>>>>>>>>>>> Java-based
>>>>>>>>>>>>>>>>>> program is not an easy thing for non-Java developers
>>>>>>>> (maintainers
>>>>>>>>>>> of
>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to
>>> set
>>>>>>> up
>>>>>>>>> the
>>>>>>>>>>>>>>>>>> environment correctly and then run the tests, neither is
>>> it
>>>>>>> an
>>>>>>>>>>> easy
>>>>>>>>>>>>>>>> skill
>>>>>>>>>>>>>>>>>> to debug the tests.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
>>>>>> because
>>>>>>> we
>>>>>>>>>>>> added
>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>> cases and many duplicated codes that need to be removed.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Therefore we're planning to build an unified,
>> easy-to-use,
>>>>>>> and
>>>>>>>>>>> fast
>>>>>>>>>>>> E2E
>>>>>>>>>>>>>>>>>> test framework to benifit all the sub-projects.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Here are some rough ideas about this framework:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 0. (Unchanged) We will continue to use
>> docker-compose.yaml
>>>>>> to
>>>>>>>>>>>>>>>> orchestrate
>>>>>>>>>>>>>>>>>> the services that are to be tested, it’s proved to be
>>>>>>> friendly
>>>>>>>>> and
>>>>>>>>>>>>>>>>>> debuggable because we can start it directly even if we
>> are
>>>>>>> not
>>>>>>>>>>>> writing
>>>>>>>>>>>>>>>> E2E
>>>>>>>>>>>>>>>>>> tests.
>>>>>>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI
>>>>>>> project
>>>>>>>> to
>>>>>>>>>>>> query
>>>>>>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
>>>>>>> versions
>>>>>>>>> of
>>>>>>>>>>>> query
>>>>>>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes
>>> and
>>>>>>> one
>>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> CLI).
>>>>>>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data
>>>>>> and
>>>>>>>> the
>>>>>>>>>>>> actual
>>>>>>>>>>>>>>>>>> data, we now have a custom schema of the expected data
>>> yaml
>>>>>>>> file,
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> verification codes should be packaged into an executable
>>>>>>>> command
>>>>>>>>>>> so
>>>>>>>>>>>>>>>> that it
>>>>>>>>>>>>>>>>>> can be executed standalone without Java and maven
>>>>>> knowledge.
>>>>>>> I
>>>>>>>>>>> hope
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> leverage existing standards to write the YAML files and
>> do
>>>>>>>>>>>>>>>> verifications.
>>>>>>>>>>>>>>>>>> 3. Build an orchestrating tool to start the
>>>>>>>>> `docker-compose.yaml`,
>>>>>>>>>>>>>>>> health
>>>>>>>>>>>>>>>>>> check, query actual data, and verify the result.
>>>>>>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions
>>> to
>>>>>>>> share
>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>> sub-projects.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If you have any other better idea or want to complement
>> to
>>>>>>> this
>>>>>>>>>>>>>>>> proposal,
>>>>>>>>>>>>>>>>>> please reply here. I will create an issue to track these
>>>>>>> tasks
>>>>>>>>>>>> later.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We have two students (who just finished the Summer Code
>>>>>> 2020
>>>>>>>>>>>> Projects)
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I
>>> said
>>>>>>>> this
>>>>>>>>>>> is
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> high priority, you have adequate time to get yourselves
>>>>>>>> familiar
>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
>>>>>>> SkyWalking
>>>>>>>>>>> work
>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> well
>>>>>>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and
>>> willingness
>>>>>>> to
>>>>>>>>>>> help.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1] https://github.com/Humbertzhang
>>>>>>>>>>>>>>>>>> [2] https://github.com/fgksgf/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> —————————
>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Sheng Wu <wu...@gmail.com>.
You could learn from the Satellite project design,
1. Discuss here, https://github.com/apache/skywalking/issues/5871
2. Final design posted as a blog,
http://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/
People could learn from these things when you join the community in the
future.
Sheng Wu 吴晟
Twitter, wusheng1108
Hoshea Jiang <fg...@gmail.com> 于2020年11月26日周四 下午1:38写道：
> I agree with the conclusion, and I hope to have some diagrams such as
> flowcharts or data flow diagrams, which help provide an overview of the E2E
> test before we start to do this.
>
> Ke Zhang <hu...@gmail.com> 于2020年11月26日周四 下午12:34写道：
>
> > I think supporting both docker-compose and KIND in the E2E test is
> > reasonable and necessary, and I also agree with other conclusions.😀
> >
> > kezhenxu94@apache <ke...@apache.org> 于2020年11月26日周四 上午11:58写道：
> >
> > >
> > > > 1. Docker-compose and kind are 2 options to set up the environments.
> > > > 2. Totally replace the Java and Maven based driving system.
> > > > 3. Enhance CLI to validate the GraphQL
> > > > 4. Keep the agent test tool unchanged for now as it is already a
> > separate
> > > > system from the e2e.
> > >
> > > This conclusion is good for me.
> > >
> > > What are others opinions?
> > >
> > > —————————
> > > Zhenxu Ke (柯振旭)
> > > GitHub @kezhenxu94
> > >
> > > > On Nov 17, 2020, at 2:42 PM, Sheng Wu <wu...@gmail.com>
> > wrote:
> > > >
> > > > Using k8s and docker-compose as 2 options in the test process is
> > > > reasonable, and should be supported.
> > > >
> > > > Let's keep the conclusion as simple as possible. In the next
> generation
> > > e2e
> > > > framework
> > > > 1. Docker-compose and kind are 2 options to set up the environments.
> > > > 2. Totally replace the Java and Maven based driving system.
> > > > 3. Enhance CLI to validate the GraphQL
> > > > 4. Keep the agent test tool unchanged for now as it is already a
> > separate
> > > > system from the e2e.
> > > >
> > > > Are these good enough for everyone?
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
> > > >
> > > >>>
> > > >>> We can see from above, your discussion actually is only related to
> 3.
> > > >>
> > > >>
> > > >> That's the only I intend to discuss here if you notice my first
> > > >> response('May SWCK help about provision and demo") in this thread.
> For
> > > >> other parts, I don't have any tips to share, though.
> > > >> Let me explain it more clearly. I want kubernetes to be seen as an
> > > >> alternative to docker-compose to play a running-engine role in the
> > test
> > > >> framework. If kubernetes can't replace
> > > >> the latter one Is it able to participate in it to solve the
> dedicated
> > or
> > > >> special issues.
> > > >>
> > > >> Our e2e test and plugin test frameworks are already top-level
> > complicity
> > > >>> thing in the worldwide open source field. Please don't over-design
> > it,
> > > >> and
> > > >>> too aggressive, the world will stay in the hybrid env for a very
> long
> > > >> time,
> > > >>> same as the developer.
> > > >>
> > > >>
> > > >> Reusing current elaborate output might be reasonable. But in this
> > > thread,
> > > >> we're talking about the potential solution to build a
> > > >> next-generation framework, which might mean that the current
> framework
> > > >> is too complicated to maintain; we need a more tiny way to support
> > more
> > > and
> > > >> more cases.
> > > >>
> > > >> And finally, SWCK is gonna and has to build a similar test
> framework.
> > If
> > > >> the test framework opts for the kubernetes solution, we could share
> > test
> > > >> cases and the underlying framework. It's a more efficient
> > > >> and consistent path for the entire system.
> > > >>
> > > >> regards, Hongtao.
> > > >>
> > > >> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
> > > >>
> > > >>> Hi
> > > >>>
> > > >>> I don't think we have to make the decision on k8s or out-of-k8s.
> > > >>> e2e and all integration tests have their scenarios like SkyWalking
> > have
> > > >>> multiple languages based and mesh-based, even Prometheus adoption.
> > > >>> The same thing happens on the test field, k8s, and out of k8s.
> > > >>>
> > > >>> I would like to agree with both of you, k8s is needed, no matter in
> > > mesh
> > > >>> solution(ALS, telemetry v2), but also not anyone needs that, such
> as
> > > >> agent
> > > >>> developer. We wouldn't deny the fact, the docker-compose is more
> > > >>> lightweight and easier to learn.
> > > >>> So, let me get this straight, for the test you need
> > > >>> 1. Build the source from different repos
> > > >>> 2. Build the images
> > > >>> 3. Use some platform(docker-compose or k8s) to run the images in
> one
> > > >> piece.
> > > >>> 4. Give some inputs to the env
> > > >>> 5. Use some tools/ways to check what should happen according to
> (4)'s
> > > >> input
> > > >>> 6. Checked or timeout failure.
> > > >>>
> > > >>> We can see from above, your discussion actually is only related to
> 3.
> > > >> Let's
> > > >>> not see this as a battle, we need both. Many developers/users use
> OAP
> > > and
> > > >>> SkyWalking out of k8s, that is a simple fact.
> > > >>> Our e2e test and plugin test frameworks are already top-level
> > > complicity
> > > >>> thing in the worldwide open source field. Please don't over-design
> > it,
> > > >> and
> > > >>> too aggressive, the world will stay in the hybrid env for a very
> long
> > > >> time,
> > > >>> same as the developer.
> > > >>>
> > > >>> Also, let's keep in mind, k8s is popular becomes it doesn't require
> > the
> > > >>> developers to understand as the VM did. We as an open-source
> > community,
> > > >> are
> > > >>> focusing on the developer's capabilities.
> > > >>>
> > > >>> Sheng Wu 吴晟
> > > >>> Twitter, wusheng1108
> > > >>>
> > > >>>
> > > >>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
> > > >>>
> > > >>>>>
> > > >>>>> but I have been thinking whether or not it is overkill for
> testing
> > > >>>>> purposes to introduce Kubernetes things.
> > > >>>>
> > > >>>>
> > > >>>> we all know the fact docker-compose is more portable than
> > Kubernetes,
> > > >>>> friendly to local running. But there're also some benefits to
> > replace
> > > >> it
> > > >>>> with kubernetes system:
> > > >>>>
> > > >>>> 1. SWCK has to test based on a real Kubernetes environment if the
> > main
> > > >>>> repo test framework doesn't support it(following the
> docker-compose
> > > >>> stack),
> > > >>>> That means skywalking ecosystem MUST introduce kubernetes which is
> > not
> > > >> an
> > > >>>> optional component.
> > > >>>>
> > > >>>> 2. Kubernetes support to scale applications at runtime, that help
> > > >>> improve
> > > >>>> the process of e2e. For instance, we could merge a single instance
> > and
> > > >>>> cluster test, by scaling up replicas. This strategy will reduce
> the
> > > >> total
> > > >>>> time sharply.
> > > >>>>
> > > >>>> If we are going to use KIND, I hope we can give a script or
> > something
> > > >>>>> similar to install all the needed components and create cluster
> for
> > > >>>>> convenient testing locally, Ke Zhang and Huaxi.
> > > >>>>
> > > >>>>
> > > >>>> That's SWCK's capacity to provide a standard and simple way to
> > create
> > > a
> > > >>>> cluster in every environment,
> > > >>>>
> > > >>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一
> 下午7:00写道：
> > > >>>>
> > > >>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting
> > > >>> Kubernetes-based
> > > >>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too,
> > > >> but I
> > > >>>>> have been thinking whether or not it is overkill for testing
> > purposes
> > > >>> to
> > > >>>>> introduce Kubernetes things.
> > > >>>>>
> > > >>>>> Since you mentioned this, we are open to discuss this much more
> > > >> deeply:
> > > >>>>>
> > > >>>>> It would be definitely a benefit to involve more “end”s in the
> > > >> so-call
> > > >>>>> “end to end” tests, I’m +1 to leverage SWCK as well.
> > > >>>>>
> > > >>>>> If we are going to use KIND, I hope we can give a script or
> > something
> > > >>>>> similar to install all the needed components and create cluster
> for
> > > >>>>> convenient testing locally, Ke Zhang and Huaxi.
> > > >>>>>
> > > >>>>>
> > > >>>>> —————————
> > > >>>>> Zhenxu Ke (柯振旭)
> > > >>>>> GitHub @kezhenxu94
> > > >>>>>
> > > >>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org>
> > > >>> wrote:
> > > >>>>>>
> > > >>>>>> May SWCK help about provision and demo
> > > >>>>>>
> > > >>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone
> > > >>> instance
> > > >>>>> or
> > > >>>>>> cluster for e2e. We could compose indicated CR yamls for
> different
> > > >>>>>> scenarios.
> > > >>>>>> 2. Different demo for different cases is mandatory for e2e and
> > > >> swck,
> > > >>> we
> > > >>>>>> could build a group of demo projects for them. With Kubernetes's
> > > >>>> support,
> > > >>>>>> we could
> > > >>>>>> get free of verbose scripts that are written for installation,
> > > >>>>> checking
> > > >>>>>> the state of running applications, collecting logs.
> > > >>>>>>
> > > >>>>>> Ppl might have concerts about the minkube we currently have,
> > > >> which's
> > > >>>> hard
> > > >>>>>> to debug and runs slowly.
> > > >>>>>>
> > > >>>>>> I prefer to kind[1] instead of minkube. From my experience, kind
> > > >>> works
> > > >>>>> fine
> > > >>>>>> in an e2e scenario(limited cpu, memory resources), and it is run
> > > >>>> million
> > > >>>>>> times
> > > >>>>>> on my team's e2e environment, which proves the stability and
> > > >>>>>> maintainability of it.
> > > >>>>>>
> > > >>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
> > > >>>>>>
> > > >>>>>> regards, Hongtao
> > > >>>>>>
> > > >>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
> > > >>>>>>
> > > >>>>>>> Inline
> > > >>>>>>>
> > > >>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > > >> 下午11:16写道：
> > > >>>>>>>
> > > >>>>>>>> Hi Jiajing,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for sharing your context and joining the discussion.
> It’s
> > > >>> true
> > > >>>>>>> that
> > > >>>>>>>> we’re actually tackling the same problems in this domain. Some
> > of
> > > >>> the
> > > >>>>>>>> issues you mentioned are solved in the current E2E framework
> > > >> (like
> > > >>>>>>> `retry`,
> > > >>>>>>>> `timeout`, etc.), while some others are still there without
> > ideal
> > > >>>>>>> solution.
> > > >>>>>>>> Other discussions, please see my comments inline.
> > > >>>>>>>>
> > > >>>>>>>>> - the first issue is useful in health check steps, just like
> > the
> > > >>>> ones
> > > >>>>>>>>> defined in readiness probe and liveness probe in kubelet.
> > > >>>>>>>>
> > > >>>>>>>> Yes, the reason why we decided to pick docker-compose is that,
> > it
> > > >>> has
> > > >>>>> the
> > > >>>>>>>> same functionality with kubelet in terms of dependent services
> > > >>>> startup
> > > >>>>>>>> order and healthiness checking, while it’s more lite-weight
> and
> > > >>> easy
> > > >>>>> for
> > > >>>>>>>> both developers and GitHub Actions environment. Right?
> > > >>>>>>>>
> > > >>>>>>>>> - the second issue is critical for highly-customized
> > assertions,
> > > >>> we
> > > >>>>>>>>> can leverage the `text/template` module to provide highly
> > > >>> extensive
> > > >>>>>>>>> assertion functions
> > > >>>>>>>>
> > > >>>>>>>> `text/template` is also the first thought that came to my mind
> > > >> when
> > > >>>>>>>> considering customized assertors, customized assertors are for
> > > >> sure
> > > >>>>>>>> critical in SkyWalking E2E tests because we have many kinds of
> > > >>>>>>> assertions,
> > > >>>>>>>> some of which are not foreseen until we actually need them,
> > > >>>>>>>> `AtLeastOneGreaterThanZero` is an example.
> > > >>>>>>>>
> > > >>>>>>>>> - the last but not the least, the lifecycle hooks can be used
> > > >> for
> > > >>>>>>>>> accurate control of test setup and teardown, such as
> > > >>> `docker-compose
> > > >>>>>>>>> build, up -d, down`
> > > >>>>>>>>
> > > >>>>>>>> The lifecycle hooks are what I didn’t take into consideration
> > > >>>> because I
> > > >>>>>>>> thought it is quite natural for test framework to provide this
> > > >>>>> features,
> > > >>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part.
> > > >>>>>>>>
> > > >>>>>>>>> Together with the aforementioned ideas, we will have a
> > > >> go-written,
> > > >>>>>>>>> yaml-data driven e2e testing framework which can overcome the
> > > >>>> current
> > > >>>>>>>>> problems that occur,
> > > >>>>>>>>
> > > >>>>>>>> I agree.
> > > >>>>>>>>
> > > >>>>>>>>> in which the most annoying issue is the fragmentation of
> > > >> different
> > > >>>>>>>>> scripts in different submodules, IMO.
> > > >>>>>>>>
> > > >>>>>>>> That’s why I list the last item in the previous email, (Nice
> to
> > > >>> have,
> > > >>>>>>> wrap
> > > >>>>>>>> them into a GitHub Action), with that GitHub Action, we can
> > > >> simply
> > > >>>>>>>> configure a workflow/job in every submodule, and the tests are
> > > >>>> executed
> > > >>>>>>>> with the shared scripts, no copy-pasting.
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>> Note, GitHub Action will require an Apache release every time
> > > >>> because
> > > >>>> we
> > > >>>>>>> are distributing the binary.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Sheng Wu 吴晟
> > > >>>>>>> Twitter, wusheng1108
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Regards,
> > > >>>>>>>>> Jiajing
> > > >>>>>>>>>
> > > >>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
> > > >>>>>>> kezhenxu94@apache.org>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> So, we would provide a script set to drive this?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends.
> > > >>>>>>>>>>
> > > >>>>>>>>>>> My question is more about,
> > > >>>>>>>>>>> how to work with docker-compose
> > > >>>>>>>>>>
> > > >>>>>>>>>> It’s the same as what we’re doing now in the current version
> > of
> > > >>> E2E
> > > >>>>>>>> tests, now we have many `docker-compose.yaml` files, and all
> the
> > > >>>>>>>> `docker-compose.yaml` files are reusable in the new framework,
> > we
> > > >>>> will
> > > >>>>>>> keep
> > > >>>>>>>> this mechanism as it is proved to be convenient and easy to
> use
> > > >>>>> (remember
> > > >>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage
> > E2E
> > > >>>> test,
> > > >>>>>>> not
> > > >>>>>>>> yet merged though, it’s fast/easy to add a new case).
> > > >>>>>>>>>>
> > > >>>>>>>>>> —————————
> > > >>>>>>>>>> Zhenxu Ke (柯振旭)
> > > >>>>>>>>>> GitHub @kezhenxu94
> > > >>>>>>>>>>
> > > >>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
> > > >>> wu.sheng.841108@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > > >>>> 下午9:16写道：
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> I want to ask, how the docker-compose lands in your
> mind? I
> > > >>>> think
> > > >>>>>>> CLI
> > > >>>>>>>>>>>> can't
> > > >>>>>>>>>>>>> control this part.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> docker-compose just fits the requirements perfectly that
> we
> > > >>> need,
> > > >>>>> we
> > > >>>>>>>> have
> > > >>>>>>>>>>>> been using it in the current E2E and turns out it has
> > > >>> advantages
> > > >>>> in
> > > >>>>>>>>>>>> starting many services correctly with health check and
> > > >>> dependency
> > > >>>>>>>>>>>> declaration. (One may say Kubernetes can do the same
> thing,
> > > >> but
> > > >>>>> it’s
> > > >>>>>>>> kind
> > > >>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex).
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
> > > >>> related
> > > >>>>>>>> things,
> > > >>>>>>>>>>>> it just provides query ability (the same as what it is
> doing
> > > >>> now,
> > > >>>>>>>> just need
> > > >>>>>>>>>>>> to check whether it covers all the needed query interfaces
> > or
> > > >>>> not),
> > > >>>>>>>> we will
> > > >>>>>>>>>>>> have a dedicated control program to
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no
> > new
> > > >>>>>>>> knowledge);
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> So, we would provide a script set to drive this? My
> question
> > > >> is
> > > >>>> more
> > > >>>>>>>> about,
> > > >>>>>>>>>>> how to work with docker-compose, rather than challanging
> the
> > > >>>> choice.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Sheng Wu 吴晟
> > > >>>>>>>>>>> Twitter, wusheng1108
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing
> > repository
> > > >>> and
> > > >>>>>>>> codes);
> > > >>>>>>>>>>>> (3) verify the actual data (by verification tool, a new
> > tool,
> > > >>>>> TODO);
> > > >>>>>>>>>>>> (4) and determine the test result (success or failed);
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> —————————
> > > >>>>>>>>>>>> Zhenxu Ke (柯振旭)
> > > >>>>>>>>>>>> GitHub @kezhenxu94
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Sheng Wu 吴晟
> > > >>>>>>>>>>>>> Twitter, wusheng1108
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > > >>>>> 下午5:27写道：
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build
> > the
> > > >>>> next
> > > >>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking,
> > > >> which
> > > >>>> may
> > > >>>>>>>> not
> > > >>>>>>>>>>>> be a
> > > >>>>>>>>>>>>>> high priority but a necessity.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very
> > > >> important
> > > >>>>> role
> > > >>>>>>>> in
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> contribution process, and the current E2E tests just
> work
> > > >>> well
> > > >>>> to
> > > >>>>>>>> verify
> > > >>>>>>>>>>>>>> every single pull request before merging, so why bother
> to
> > > >>>> build
> > > >>>>>>> the
> > > >>>>>>>>>>>> next
> > > >>>>>>>>>>>>>> generation of E2E test framework?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
> > > >> sub-projects
> > > >>>>> that
> > > >>>>>>>> vary
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to
> > > >> help
> > > >>> us
> > > >>>>>>>> verify
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> pull requests, but we can see that they reimplement the
> > E2E
> > > >>>> test
> > > >>>>>>>>>>>> framework
> > > >>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t
> > have
> > > >>> E2E
> > > >>>>>>>> tests at
> > > >>>>>>>>>>>>>> all). Reimplementing the E2E test framework is
> unnecesarry
> > > >>> and
> > > >>>>>>>>>>>> introduces
> > > >>>>>>>>>>>>>> more duplicated work when adding a new test case. The
> > > >>> ultimate
> > > >>>>>>> goal
> > > >>>>>>>> is
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> reuse all the test cases when needed.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running
> > > >>>>>>> Java-based
> > > >>>>>>>>>>>>>> program is not an easy thing for non-Java developers
> > > >>>> (maintainers
> > > >>>>>>> of
> > > >>>>>>>>>>>> other
> > > >>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to
> > set
> > > >>> up
> > > >>>>> the
> > > >>>>>>>>>>>>>> environment correctly and then run the tests, neither is
> > it
> > > >>> an
> > > >>>>>>> easy
> > > >>>>>>>>>>>> skill
> > > >>>>>>>>>>>>>> to debug the tests.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
> > > >> because
> > > >>> we
> > > >>>>>>>> added
> > > >>>>>>>>>>>> many
> > > >>>>>>>>>>>>>> cases and many duplicated codes that need to be removed.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Therefore we're planning to build an unified,
> easy-to-use,
> > > >>> and
> > > >>>>>>> fast
> > > >>>>>>>> E2E
> > > >>>>>>>>>>>>>> test framework to benifit all the sub-projects.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Here are some rough ideas about this framework:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> 0. (Unchanged) We will continue to use
> docker-compose.yaml
> > > >> to
> > > >>>>>>>>>>>> orchestrate
> > > >>>>>>>>>>>>>> the services that are to be tested, it’s proved to be
> > > >>> friendly
> > > >>>>> and
> > > >>>>>>>>>>>>>> debuggable because we can start it directly even if we
> are
> > > >>> not
> > > >>>>>>>> writing
> > > >>>>>>>>>>>> E2E
> > > >>>>>>>>>>>>>> tests.
> > > >>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI
> > > >>> project
> > > >>>> to
> > > >>>>>>>> query
> > > >>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
> > > >>> versions
> > > >>>>> of
> > > >>>>>>>> query
> > > >>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes
> > and
> > > >>> one
> > > >>>>> in
> > > >>>>>>>> the
> > > >>>>>>>>>>>> CLI).
> > > >>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data
> > > >> and
> > > >>>> the
> > > >>>>>>>> actual
> > > >>>>>>>>>>>>>> data, we now have a custom schema of the expected data
> > yaml
> > > >>>> file,
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>> verification codes should be packaged into an executable
> > > >>>> command
> > > >>>>>>> so
> > > >>>>>>>>>>>> that it
> > > >>>>>>>>>>>>>> can be executed standalone without Java and maven
> > > >> knowledge.
> > > >>> I
> > > >>>>>>> hope
> > > >>>>>>>> we
> > > >>>>>>>>>>>> can
> > > >>>>>>>>>>>>>> leverage existing standards to write the YAML files and
> do
> > > >>>>>>>>>>>> verifications.
> > > >>>>>>>>>>>>>> 3. Build an orchestrating tool to start the
> > > >>>>> `docker-compose.yaml`,
> > > >>>>>>>>>>>> health
> > > >>>>>>>>>>>>>> check, query actual data, and verify the result.
> > > >>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions
> > to
> > > >>>> share
> > > >>>>>>>>>>>> between
> > > >>>>>>>>>>>>>> sub-projects.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> If you have any other better idea or want to complement
> to
> > > >>> this
> > > >>>>>>>>>>>> proposal,
> > > >>>>>>>>>>>>>> please reply here. I will create an issue to track these
> > > >>> tasks
> > > >>>>>>>> later.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> We have two students (who just finished the Summer Code
> > > >> 2020
> > > >>>>>>>> Projects)
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I
> > said
> > > >>>> this
> > > >>>>>>> is
> > > >>>>>>>> a
> > > >>>>>>>>>>>> not
> > > >>>>>>>>>>>>>> high priority, you have adequate time to get yourselves
> > > >>>> familiar
> > > >>>>>>>> with
> > > >>>>>>>>>>>> how
> > > >>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
> > > >>> SkyWalking
> > > >>>>>>> work
> > > >>>>>>>> as
> > > >>>>>>>>>>>> well
> > > >>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and
> > willingness
> > > >>> to
> > > >>>>>>> help.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> [1] https://github.com/Humbertzhang
> > > >>>>>>>>>>>>>> [2] https://github.com/fgksgf/
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> —————————
> > > >>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> > > >>>>>>>>>>>>>> GitHub @kezhenxu94
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> —————————
> > > >>>>>>>>>>>> Zhenxu Ke (柯振旭)
> > > >>>>>>>>>>>> GitHub @kezhenxu94
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> —————————
> > > >>>>>>>> Zhenxu Ke (柯振旭)
> > > >>>>>>>> GitHub @kezhenxu94
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Hoshea Jiang <fg...@gmail.com>.
I agree with the conclusion, and I hope to have some diagrams such as
flowcharts or data flow diagrams, which help provide an overview of the E2E
test before we start to do this.
Ke Zhang <hu...@gmail.com> 于2020年11月26日周四 下午12:34写道：
> I think supporting both docker-compose and KIND in the E2E test is
> reasonable and necessary, and I also agree with other conclusions.😀
>
> kezhenxu94@apache <ke...@apache.org> 于2020年11月26日周四 上午11:58写道：
>
> >
> > > 1. Docker-compose and kind are 2 options to set up the environments.
> > > 2. Totally replace the Java and Maven based driving system.
> > > 3. Enhance CLI to validate the GraphQL
> > > 4. Keep the agent test tool unchanged for now as it is already a
> separate
> > > system from the e2e.
> >
> > This conclusion is good for me.
> >
> > What are others opinions?
> >
> > —————————
> > Zhenxu Ke (柯振旭)
> > GitHub @kezhenxu94
> >
> > > On Nov 17, 2020, at 2:42 PM, Sheng Wu <wu...@gmail.com>
> wrote:
> > >
> > > Using k8s and docker-compose as 2 options in the test process is
> > > reasonable, and should be supported.
> > >
> > > Let's keep the conclusion as simple as possible. In the next generation
> > e2e
> > > framework
> > > 1. Docker-compose and kind are 2 options to set up the environments.
> > > 2. Totally replace the Java and Maven based driving system.
> > > 3. Enhance CLI to validate the GraphQL
> > > 4. Keep the agent test tool unchanged for now as it is already a
> separate
> > > system from the e2e.
> > >
> > > Are these good enough for everyone?
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
> > >
> > >>>
> > >>> We can see from above, your discussion actually is only related to 3.
> > >>
> > >>
> > >> That's the only I intend to discuss here if you notice my first
> > >> response('May SWCK help about provision and demo") in this thread. For
> > >> other parts, I don't have any tips to share, though.
> > >> Let me explain it more clearly. I want kubernetes to be seen as an
> > >> alternative to docker-compose to play a running-engine role in the
> test
> > >> framework. If kubernetes can't replace
> > >> the latter one Is it able to participate in it to solve the dedicated
> or
> > >> special issues.
> > >>
> > >> Our e2e test and plugin test frameworks are already top-level
> complicity
> > >>> thing in the worldwide open source field. Please don't over-design
> it,
> > >> and
> > >>> too aggressive, the world will stay in the hybrid env for a very long
> > >> time,
> > >>> same as the developer.
> > >>
> > >>
> > >> Reusing current elaborate output might be reasonable. But in this
> > thread,
> > >> we're talking about the potential solution to build a
> > >> next-generation framework, which might mean that the current framework
> > >> is too complicated to maintain; we need a more tiny way to support
> more
> > and
> > >> more cases.
> > >>
> > >> And finally, SWCK is gonna and has to build a similar test framework.
> If
> > >> the test framework opts for the kubernetes solution, we could share
> test
> > >> cases and the underlying framework. It's a more efficient
> > >> and consistent path for the entire system.
> > >>
> > >> regards, Hongtao.
> > >>
> > >> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
> > >>
> > >>> Hi
> > >>>
> > >>> I don't think we have to make the decision on k8s or out-of-k8s.
> > >>> e2e and all integration tests have their scenarios like SkyWalking
> have
> > >>> multiple languages based and mesh-based, even Prometheus adoption.
> > >>> The same thing happens on the test field, k8s, and out of k8s.
> > >>>
> > >>> I would like to agree with both of you, k8s is needed, no matter in
> > mesh
> > >>> solution(ALS, telemetry v2), but also not anyone needs that, such as
> > >> agent
> > >>> developer. We wouldn't deny the fact, the docker-compose is more
> > >>> lightweight and easier to learn.
> > >>> So, let me get this straight, for the test you need
> > >>> 1. Build the source from different repos
> > >>> 2. Build the images
> > >>> 3. Use some platform(docker-compose or k8s) to run the images in one
> > >> piece.
> > >>> 4. Give some inputs to the env
> > >>> 5. Use some tools/ways to check what should happen according to (4)'s
> > >> input
> > >>> 6. Checked or timeout failure.
> > >>>
> > >>> We can see from above, your discussion actually is only related to 3.
> > >> Let's
> > >>> not see this as a battle, we need both. Many developers/users use OAP
> > and
> > >>> SkyWalking out of k8s, that is a simple fact.
> > >>> Our e2e test and plugin test frameworks are already top-level
> > complicity
> > >>> thing in the worldwide open source field. Please don't over-design
> it,
> > >> and
> > >>> too aggressive, the world will stay in the hybrid env for a very long
> > >> time,
> > >>> same as the developer.
> > >>>
> > >>> Also, let's keep in mind, k8s is popular becomes it doesn't require
> the
> > >>> developers to understand as the VM did. We as an open-source
> community,
> > >> are
> > >>> focusing on the developer's capabilities.
> > >>>
> > >>> Sheng Wu 吴晟
> > >>> Twitter, wusheng1108
> > >>>
> > >>>
> > >>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
> > >>>
> > >>>>>
> > >>>>> but I have been thinking whether or not it is overkill for testing
> > >>>>> purposes to introduce Kubernetes things.
> > >>>>
> > >>>>
> > >>>> we all know the fact docker-compose is more portable than
> Kubernetes,
> > >>>> friendly to local running. But there're also some benefits to
> replace
> > >> it
> > >>>> with kubernetes system:
> > >>>>
> > >>>> 1. SWCK has to test based on a real Kubernetes environment if the
> main
> > >>>> repo test framework doesn't support it(following the docker-compose
> > >>> stack),
> > >>>> That means skywalking ecosystem MUST introduce kubernetes which is
> not
> > >> an
> > >>>> optional component.
> > >>>>
> > >>>> 2. Kubernetes support to scale applications at runtime, that help
> > >>> improve
> > >>>> the process of e2e. For instance, we could merge a single instance
> and
> > >>>> cluster test, by scaling up replicas. This strategy will reduce the
> > >> total
> > >>>> time sharply.
> > >>>>
> > >>>> If we are going to use KIND, I hope we can give a script or
> something
> > >>>>> similar to install all the needed components and create cluster for
> > >>>>> convenient testing locally, Ke Zhang and Huaxi.
> > >>>>
> > >>>>
> > >>>> That's SWCK's capacity to provide a standard and simple way to
> create
> > a
> > >>>> cluster in every environment,
> > >>>>
> > >>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一 下午7:00写道：
> > >>>>
> > >>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting
> > >>> Kubernetes-based
> > >>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too,
> > >> but I
> > >>>>> have been thinking whether or not it is overkill for testing
> purposes
> > >>> to
> > >>>>> introduce Kubernetes things.
> > >>>>>
> > >>>>> Since you mentioned this, we are open to discuss this much more
> > >> deeply:
> > >>>>>
> > >>>>> It would be definitely a benefit to involve more “end”s in the
> > >> so-call
> > >>>>> “end to end” tests, I’m +1 to leverage SWCK as well.
> > >>>>>
> > >>>>> If we are going to use KIND, I hope we can give a script or
> something
> > >>>>> similar to install all the needed components and create cluster for
> > >>>>> convenient testing locally, Ke Zhang and Huaxi.
> > >>>>>
> > >>>>>
> > >>>>> —————————
> > >>>>> Zhenxu Ke (柯振旭)
> > >>>>> GitHub @kezhenxu94
> > >>>>>
> > >>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org>
> > >>> wrote:
> > >>>>>>
> > >>>>>> May SWCK help about provision and demo
> > >>>>>>
> > >>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone
> > >>> instance
> > >>>>> or
> > >>>>>> cluster for e2e. We could compose indicated CR yamls for different
> > >>>>>> scenarios.
> > >>>>>> 2. Different demo for different cases is mandatory for e2e and
> > >> swck,
> > >>> we
> > >>>>>> could build a group of demo projects for them. With Kubernetes's
> > >>>> support,
> > >>>>>> we could
> > >>>>>> get free of verbose scripts that are written for installation,
> > >>>>> checking
> > >>>>>> the state of running applications, collecting logs.
> > >>>>>>
> > >>>>>> Ppl might have concerts about the minkube we currently have,
> > >> which's
> > >>>> hard
> > >>>>>> to debug and runs slowly.
> > >>>>>>
> > >>>>>> I prefer to kind[1] instead of minkube. From my experience, kind
> > >>> works
> > >>>>> fine
> > >>>>>> in an e2e scenario(limited cpu, memory resources), and it is run
> > >>>> million
> > >>>>>> times
> > >>>>>> on my team's e2e environment, which proves the stability and
> > >>>>>> maintainability of it.
> > >>>>>>
> > >>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
> > >>>>>>
> > >>>>>> regards, Hongtao
> > >>>>>>
> > >>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
> > >>>>>>
> > >>>>>>> Inline
> > >>>>>>>
> > >>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > >> 下午11:16写道：
> > >>>>>>>
> > >>>>>>>> Hi Jiajing,
> > >>>>>>>>
> > >>>>>>>> Thanks for sharing your context and joining the discussion. It’s
> > >>> true
> > >>>>>>> that
> > >>>>>>>> we’re actually tackling the same problems in this domain. Some
> of
> > >>> the
> > >>>>>>>> issues you mentioned are solved in the current E2E framework
> > >> (like
> > >>>>>>> `retry`,
> > >>>>>>>> `timeout`, etc.), while some others are still there without
> ideal
> > >>>>>>> solution.
> > >>>>>>>> Other discussions, please see my comments inline.
> > >>>>>>>>
> > >>>>>>>>> - the first issue is useful in health check steps, just like
> the
> > >>>> ones
> > >>>>>>>>> defined in readiness probe and liveness probe in kubelet.
> > >>>>>>>>
> > >>>>>>>> Yes, the reason why we decided to pick docker-compose is that,
> it
> > >>> has
> > >>>>> the
> > >>>>>>>> same functionality with kubelet in terms of dependent services
> > >>>> startup
> > >>>>>>>> order and healthiness checking, while it’s more lite-weight and
> > >>> easy
> > >>>>> for
> > >>>>>>>> both developers and GitHub Actions environment. Right?
> > >>>>>>>>
> > >>>>>>>>> - the second issue is critical for highly-customized
> assertions,
> > >>> we
> > >>>>>>>>> can leverage the `text/template` module to provide highly
> > >>> extensive
> > >>>>>>>>> assertion functions
> > >>>>>>>>
> > >>>>>>>> `text/template` is also the first thought that came to my mind
> > >> when
> > >>>>>>>> considering customized assertors, customized assertors are for
> > >> sure
> > >>>>>>>> critical in SkyWalking E2E tests because we have many kinds of
> > >>>>>>> assertions,
> > >>>>>>>> some of which are not foreseen until we actually need them,
> > >>>>>>>> `AtLeastOneGreaterThanZero` is an example.
> > >>>>>>>>
> > >>>>>>>>> - the last but not the least, the lifecycle hooks can be used
> > >> for
> > >>>>>>>>> accurate control of test setup and teardown, such as
> > >>> `docker-compose
> > >>>>>>>>> build, up -d, down`
> > >>>>>>>>
> > >>>>>>>> The lifecycle hooks are what I didn’t take into consideration
> > >>>> because I
> > >>>>>>>> thought it is quite natural for test framework to provide this
> > >>>>> features,
> > >>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part.
> > >>>>>>>>
> > >>>>>>>>> Together with the aforementioned ideas, we will have a
> > >> go-written,
> > >>>>>>>>> yaml-data driven e2e testing framework which can overcome the
> > >>>> current
> > >>>>>>>>> problems that occur,
> > >>>>>>>>
> > >>>>>>>> I agree.
> > >>>>>>>>
> > >>>>>>>>> in which the most annoying issue is the fragmentation of
> > >> different
> > >>>>>>>>> scripts in different submodules, IMO.
> > >>>>>>>>
> > >>>>>>>> That’s why I list the last item in the previous email, (Nice to
> > >>> have,
> > >>>>>>> wrap
> > >>>>>>>> them into a GitHub Action), with that GitHub Action, we can
> > >> simply
> > >>>>>>>> configure a workflow/job in every submodule, and the tests are
> > >>>> executed
> > >>>>>>>> with the shared scripts, no copy-pasting.
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> Note, GitHub Action will require an Apache release every time
> > >>> because
> > >>>> we
> > >>>>>>> are distributing the binary.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Sheng Wu 吴晟
> > >>>>>>> Twitter, wusheng1108
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Regards,
> > >>>>>>>>> Jiajing
> > >>>>>>>>>
> > >>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
> > >>>>>>> kezhenxu94@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> So, we would provide a script set to drive this?
> > >>>>>>>>>>
> > >>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends.
> > >>>>>>>>>>
> > >>>>>>>>>>> My question is more about,
> > >>>>>>>>>>> how to work with docker-compose
> > >>>>>>>>>>
> > >>>>>>>>>> It’s the same as what we’re doing now in the current version
> of
> > >>> E2E
> > >>>>>>>> tests, now we have many `docker-compose.yaml` files, and all the
> > >>>>>>>> `docker-compose.yaml` files are reusable in the new framework,
> we
> > >>>> will
> > >>>>>>> keep
> > >>>>>>>> this mechanism as it is proved to be convenient and easy to use
> > >>>>> (remember
> > >>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage
> E2E
> > >>>> test,
> > >>>>>>> not
> > >>>>>>>> yet merged though, it’s fast/easy to add a new case).
> > >>>>>>>>>>
> > >>>>>>>>>> —————————
> > >>>>>>>>>> Zhenxu Ke (柯振旭)
> > >>>>>>>>>> GitHub @kezhenxu94
> > >>>>>>>>>>
> > >>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
> > >>> wu.sheng.841108@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > >>>> 下午9:16写道：
> > >>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> I want to ask, how the docker-compose lands in your mind? I
> > >>>> think
> > >>>>>>> CLI
> > >>>>>>>>>>>> can't
> > >>>>>>>>>>>>> control this part.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> docker-compose just fits the requirements perfectly that we
> > >>> need,
> > >>>>> we
> > >>>>>>>> have
> > >>>>>>>>>>>> been using it in the current E2E and turns out it has
> > >>> advantages
> > >>>> in
> > >>>>>>>>>>>> starting many services correctly with health check and
> > >>> dependency
> > >>>>>>>>>>>> declaration. (One may say Kubernetes can do the same thing,
> > >> but
> > >>>>> it’s
> > >>>>>>>> kind
> > >>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
> > >>> related
> > >>>>>>>> things,
> > >>>>>>>>>>>> it just provides query ability (the same as what it is doing
> > >>> now,
> > >>>>>>>> just need
> > >>>>>>>>>>>> to check whether it covers all the needed query interfaces
> or
> > >>>> not),
> > >>>>>>>> we will
> > >>>>>>>>>>>> have a dedicated control program to
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no
> new
> > >>>>>>>> knowledge);
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> So, we would provide a script set to drive this? My question
> > >> is
> > >>>> more
> > >>>>>>>> about,
> > >>>>>>>>>>> how to work with docker-compose, rather than challanging the
> > >>>> choice.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Sheng Wu 吴晟
> > >>>>>>>>>>> Twitter, wusheng1108
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing
> repository
> > >>> and
> > >>>>>>>> codes);
> > >>>>>>>>>>>> (3) verify the actual data (by verification tool, a new
> tool,
> > >>>>> TODO);
> > >>>>>>>>>>>> (4) and determine the test result (success or failed);
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> —————————
> > >>>>>>>>>>>> Zhenxu Ke (柯振旭)
> > >>>>>>>>>>>> GitHub @kezhenxu94
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Sheng Wu 吴晟
> > >>>>>>>>>>>>> Twitter, wusheng1108
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > >>>>> 下午5:27写道：
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build
> the
> > >>>> next
> > >>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking,
> > >> which
> > >>>> may
> > >>>>>>>> not
> > >>>>>>>>>>>> be a
> > >>>>>>>>>>>>>> high priority but a necessity.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very
> > >> important
> > >>>>> role
> > >>>>>>>> in
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> contribution process, and the current E2E tests just work
> > >>> well
> > >>>> to
> > >>>>>>>> verify
> > >>>>>>>>>>>>>> every single pull request before merging, so why bother to
> > >>>> build
> > >>>>>>> the
> > >>>>>>>>>>>> next
> > >>>>>>>>>>>>>> generation of E2E test framework?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
> > >> sub-projects
> > >>>>> that
> > >>>>>>>> vary
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to
> > >> help
> > >>> us
> > >>>>>>>> verify
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> pull requests, but we can see that they reimplement the
> E2E
> > >>>> test
> > >>>>>>>>>>>> framework
> > >>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t
> have
> > >>> E2E
> > >>>>>>>> tests at
> > >>>>>>>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry
> > >>> and
> > >>>>>>>>>>>> introduces
> > >>>>>>>>>>>>>> more duplicated work when adding a new test case. The
> > >>> ultimate
> > >>>>>>> goal
> > >>>>>>>> is
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> reuse all the test cases when needed.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running
> > >>>>>>> Java-based
> > >>>>>>>>>>>>>> program is not an easy thing for non-Java developers
> > >>>> (maintainers
> > >>>>>>> of
> > >>>>>>>>>>>> other
> > >>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to
> set
> > >>> up
> > >>>>> the
> > >>>>>>>>>>>>>> environment correctly and then run the tests, neither is
> it
> > >>> an
> > >>>>>>> easy
> > >>>>>>>>>>>> skill
> > >>>>>>>>>>>>>> to debug the tests.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
> > >> because
> > >>> we
> > >>>>>>>> added
> > >>>>>>>>>>>> many
> > >>>>>>>>>>>>>> cases and many duplicated codes that need to be removed.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Therefore we're planning to build an unified, easy-to-use,
> > >>> and
> > >>>>>>> fast
> > >>>>>>>> E2E
> > >>>>>>>>>>>>>> test framework to benifit all the sub-projects.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Here are some rough ideas about this framework:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml
> > >> to
> > >>>>>>>>>>>> orchestrate
> > >>>>>>>>>>>>>> the services that are to be tested, it’s proved to be
> > >>> friendly
> > >>>>> and
> > >>>>>>>>>>>>>> debuggable because we can start it directly even if we are
> > >>> not
> > >>>>>>>> writing
> > >>>>>>>>>>>> E2E
> > >>>>>>>>>>>>>> tests.
> > >>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI
> > >>> project
> > >>>> to
> > >>>>>>>> query
> > >>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
> > >>> versions
> > >>>>> of
> > >>>>>>>> query
> > >>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes
> and
> > >>> one
> > >>>>> in
> > >>>>>>>> the
> > >>>>>>>>>>>> CLI).
> > >>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data
> > >> and
> > >>>> the
> > >>>>>>>> actual
> > >>>>>>>>>>>>>> data, we now have a custom schema of the expected data
> yaml
> > >>>> file,
> > >>>>>>>> the
> > >>>>>>>>>>>>>> verification codes should be packaged into an executable
> > >>>> command
> > >>>>>>> so
> > >>>>>>>>>>>> that it
> > >>>>>>>>>>>>>> can be executed standalone without Java and maven
> > >> knowledge.
> > >>> I
> > >>>>>>> hope
> > >>>>>>>> we
> > >>>>>>>>>>>> can
> > >>>>>>>>>>>>>> leverage existing standards to write the YAML files and do
> > >>>>>>>>>>>> verifications.
> > >>>>>>>>>>>>>> 3. Build an orchestrating tool to start the
> > >>>>> `docker-compose.yaml`,
> > >>>>>>>>>>>> health
> > >>>>>>>>>>>>>> check, query actual data, and verify the result.
> > >>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions
> to
> > >>>> share
> > >>>>>>>>>>>> between
> > >>>>>>>>>>>>>> sub-projects.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> If you have any other better idea or want to complement to
> > >>> this
> > >>>>>>>>>>>> proposal,
> > >>>>>>>>>>>>>> please reply here. I will create an issue to track these
> > >>> tasks
> > >>>>>>>> later.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> We have two students (who just finished the Summer Code
> > >> 2020
> > >>>>>>>> Projects)
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I
> said
> > >>>> this
> > >>>>>>> is
> > >>>>>>>> a
> > >>>>>>>>>>>> not
> > >>>>>>>>>>>>>> high priority, you have adequate time to get yourselves
> > >>>> familiar
> > >>>>>>>> with
> > >>>>>>>>>>>> how
> > >>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
> > >>> SkyWalking
> > >>>>>>> work
> > >>>>>>>> as
> > >>>>>>>>>>>> well
> > >>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and
> willingness
> > >>> to
> > >>>>>>> help.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> [1] https://github.com/Humbertzhang
> > >>>>>>>>>>>>>> [2] https://github.com/fgksgf/
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> —————————
> > >>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> > >>>>>>>>>>>>>> GitHub @kezhenxu94
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> —————————
> > >>>>>>>>>>>> Zhenxu Ke (柯振旭)
> > >>>>>>>>>>>> GitHub @kezhenxu94
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> —————————
> > >>>>>>>> Zhenxu Ke (柯振旭)
> > >>>>>>>> GitHub @kezhenxu94
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Ke Zhang <hu...@gmail.com>.
I think supporting both docker-compose and KIND in the E2E test is
reasonable and necessary, and I also agree with other conclusions.😀
kezhenxu94@apache <ke...@apache.org> 于2020年11月26日周四 上午11:58写道：
>
> > 1. Docker-compose and kind are 2 options to set up the environments.
> > 2. Totally replace the Java and Maven based driving system.
> > 3. Enhance CLI to validate the GraphQL
> > 4. Keep the agent test tool unchanged for now as it is already a separate
> > system from the e2e.
>
> This conclusion is good for me.
>
> What are others opinions?
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94
>
> > On Nov 17, 2020, at 2:42 PM, Sheng Wu <wu...@gmail.com> wrote:
> >
> > Using k8s and docker-compose as 2 options in the test process is
> > reasonable, and should be supported.
> >
> > Let's keep the conclusion as simple as possible. In the next generation
> e2e
> > framework
> > 1. Docker-compose and kind are 2 options to set up the environments.
> > 2. Totally replace the Java and Maven based driving system.
> > 3. Enhance CLI to validate the GraphQL
> > 4. Keep the agent test tool unchanged for now as it is already a separate
> > system from the e2e.
> >
> > Are these good enough for everyone?
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
> >
> >>>
> >>> We can see from above, your discussion actually is only related to 3.
> >>
> >>
> >> That's the only I intend to discuss here if you notice my first
> >> response('May SWCK help about provision and demo") in this thread. For
> >> other parts, I don't have any tips to share, though.
> >> Let me explain it more clearly. I want kubernetes to be seen as an
> >> alternative to docker-compose to play a running-engine role in the test
> >> framework. If kubernetes can't replace
> >> the latter one Is it able to participate in it to solve the dedicated or
> >> special issues.
> >>
> >> Our e2e test and plugin test frameworks are already top-level complicity
> >>> thing in the worldwide open source field. Please don't over-design it,
> >> and
> >>> too aggressive, the world will stay in the hybrid env for a very long
> >> time,
> >>> same as the developer.
> >>
> >>
> >> Reusing current elaborate output might be reasonable. But in this
> thread,
> >> we're talking about the potential solution to build a
> >> next-generation framework, which might mean that the current framework
> >> is too complicated to maintain; we need a more tiny way to support more
> and
> >> more cases.
> >>
> >> And finally, SWCK is gonna and has to build a similar test framework. If
> >> the test framework opts for the kubernetes solution, we could share test
> >> cases and the underlying framework. It's a more efficient
> >> and consistent path for the entire system.
> >>
> >> regards, Hongtao.
> >>
> >> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
> >>
> >>> Hi
> >>>
> >>> I don't think we have to make the decision on k8s or out-of-k8s.
> >>> e2e and all integration tests have their scenarios like SkyWalking have
> >>> multiple languages based and mesh-based, even Prometheus adoption.
> >>> The same thing happens on the test field, k8s, and out of k8s.
> >>>
> >>> I would like to agree with both of you, k8s is needed, no matter in
> mesh
> >>> solution(ALS, telemetry v2), but also not anyone needs that, such as
> >> agent
> >>> developer. We wouldn't deny the fact, the docker-compose is more
> >>> lightweight and easier to learn.
> >>> So, let me get this straight, for the test you need
> >>> 1. Build the source from different repos
> >>> 2. Build the images
> >>> 3. Use some platform(docker-compose or k8s) to run the images in one
> >> piece.
> >>> 4. Give some inputs to the env
> >>> 5. Use some tools/ways to check what should happen according to (4)'s
> >> input
> >>> 6. Checked or timeout failure.
> >>>
> >>> We can see from above, your discussion actually is only related to 3.
> >> Let's
> >>> not see this as a battle, we need both. Many developers/users use OAP
> and
> >>> SkyWalking out of k8s, that is a simple fact.
> >>> Our e2e test and plugin test frameworks are already top-level
> complicity
> >>> thing in the worldwide open source field. Please don't over-design it,
> >> and
> >>> too aggressive, the world will stay in the hybrid env for a very long
> >> time,
> >>> same as the developer.
> >>>
> >>> Also, let's keep in mind, k8s is popular becomes it doesn't require the
> >>> developers to understand as the VM did. We as an open-source community,
> >> are
> >>> focusing on the developer's capabilities.
> >>>
> >>> Sheng Wu 吴晟
> >>> Twitter, wusheng1108
> >>>
> >>>
> >>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
> >>>
> >>>>>
> >>>>> but I have been thinking whether or not it is overkill for testing
> >>>>> purposes to introduce Kubernetes things.
> >>>>
> >>>>
> >>>> we all know the fact docker-compose is more portable than Kubernetes,
> >>>> friendly to local running. But there're also some benefits to replace
> >> it
> >>>> with kubernetes system:
> >>>>
> >>>> 1. SWCK has to test based on a real Kubernetes environment if the main
> >>>> repo test framework doesn't support it(following the docker-compose
> >>> stack),
> >>>> That means skywalking ecosystem MUST introduce kubernetes which is not
> >> an
> >>>> optional component.
> >>>>
> >>>> 2. Kubernetes support to scale applications at runtime, that help
> >>> improve
> >>>> the process of e2e. For instance, we could merge a single instance and
> >>>> cluster test, by scaling up replicas. This strategy will reduce the
> >> total
> >>>> time sharply.
> >>>>
> >>>> If we are going to use KIND, I hope we can give a script or something
> >>>>> similar to install all the needed components and create cluster for
> >>>>> convenient testing locally, Ke Zhang and Huaxi.
> >>>>
> >>>>
> >>>> That's SWCK's capacity to provide a standard and simple way to create
> a
> >>>> cluster in every environment,
> >>>>
> >>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一 下午7:00写道：
> >>>>
> >>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting
> >>> Kubernetes-based
> >>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too,
> >> but I
> >>>>> have been thinking whether or not it is overkill for testing purposes
> >>> to
> >>>>> introduce Kubernetes things.
> >>>>>
> >>>>> Since you mentioned this, we are open to discuss this much more
> >> deeply:
> >>>>>
> >>>>> It would be definitely a benefit to involve more “end”s in the
> >> so-call
> >>>>> “end to end” tests, I’m +1 to leverage SWCK as well.
> >>>>>
> >>>>> If we are going to use KIND, I hope we can give a script or something
> >>>>> similar to install all the needed components and create cluster for
> >>>>> convenient testing locally, Ke Zhang and Huaxi.
> >>>>>
> >>>>>
> >>>>> —————————
> >>>>> Zhenxu Ke (柯振旭)
> >>>>> GitHub @kezhenxu94
> >>>>>
> >>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org>
> >>> wrote:
> >>>>>>
> >>>>>> May SWCK help about provision and demo
> >>>>>>
> >>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone
> >>> instance
> >>>>> or
> >>>>>> cluster for e2e. We could compose indicated CR yamls for different
> >>>>>> scenarios.
> >>>>>> 2. Different demo for different cases is mandatory for e2e and
> >> swck,
> >>> we
> >>>>>> could build a group of demo projects for them. With Kubernetes's
> >>>> support,
> >>>>>> we could
> >>>>>> get free of verbose scripts that are written for installation,
> >>>>> checking
> >>>>>> the state of running applications, collecting logs.
> >>>>>>
> >>>>>> Ppl might have concerts about the minkube we currently have,
> >> which's
> >>>> hard
> >>>>>> to debug and runs slowly.
> >>>>>>
> >>>>>> I prefer to kind[1] instead of minkube. From my experience, kind
> >>> works
> >>>>> fine
> >>>>>> in an e2e scenario(limited cpu, memory resources), and it is run
> >>>> million
> >>>>>> times
> >>>>>> on my team's e2e environment, which proves the stability and
> >>>>>> maintainability of it.
> >>>>>>
> >>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
> >>>>>>
> >>>>>> regards, Hongtao
> >>>>>>
> >>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
> >>>>>>
> >>>>>>> Inline
> >>>>>>>
> >>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> >> 下午11:16写道：
> >>>>>>>
> >>>>>>>> Hi Jiajing,
> >>>>>>>>
> >>>>>>>> Thanks for sharing your context and joining the discussion. It’s
> >>> true
> >>>>>>> that
> >>>>>>>> we’re actually tackling the same problems in this domain. Some of
> >>> the
> >>>>>>>> issues you mentioned are solved in the current E2E framework
> >> (like
> >>>>>>> `retry`,
> >>>>>>>> `timeout`, etc.), while some others are still there without ideal
> >>>>>>> solution.
> >>>>>>>> Other discussions, please see my comments inline.
> >>>>>>>>
> >>>>>>>>> - the first issue is useful in health check steps, just like the
> >>>> ones
> >>>>>>>>> defined in readiness probe and liveness probe in kubelet.
> >>>>>>>>
> >>>>>>>> Yes, the reason why we decided to pick docker-compose is that, it
> >>> has
> >>>>> the
> >>>>>>>> same functionality with kubelet in terms of dependent services
> >>>> startup
> >>>>>>>> order and healthiness checking, while it’s more lite-weight and
> >>> easy
> >>>>> for
> >>>>>>>> both developers and GitHub Actions environment. Right?
> >>>>>>>>
> >>>>>>>>> - the second issue is critical for highly-customized assertions,
> >>> we
> >>>>>>>>> can leverage the `text/template` module to provide highly
> >>> extensive
> >>>>>>>>> assertion functions
> >>>>>>>>
> >>>>>>>> `text/template` is also the first thought that came to my mind
> >> when
> >>>>>>>> considering customized assertors, customized assertors are for
> >> sure
> >>>>>>>> critical in SkyWalking E2E tests because we have many kinds of
> >>>>>>> assertions,
> >>>>>>>> some of which are not foreseen until we actually need them,
> >>>>>>>> `AtLeastOneGreaterThanZero` is an example.
> >>>>>>>>
> >>>>>>>>> - the last but not the least, the lifecycle hooks can be used
> >> for
> >>>>>>>>> accurate control of test setup and teardown, such as
> >>> `docker-compose
> >>>>>>>>> build, up -d, down`
> >>>>>>>>
> >>>>>>>> The lifecycle hooks are what I didn’t take into consideration
> >>>> because I
> >>>>>>>> thought it is quite natural for test framework to provide this
> >>>>> features,
> >>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part.
> >>>>>>>>
> >>>>>>>>> Together with the aforementioned ideas, we will have a
> >> go-written,
> >>>>>>>>> yaml-data driven e2e testing framework which can overcome the
> >>>> current
> >>>>>>>>> problems that occur,
> >>>>>>>>
> >>>>>>>> I agree.
> >>>>>>>>
> >>>>>>>>> in which the most annoying issue is the fragmentation of
> >> different
> >>>>>>>>> scripts in different submodules, IMO.
> >>>>>>>>
> >>>>>>>> That’s why I list the last item in the previous email, (Nice to
> >>> have,
> >>>>>>> wrap
> >>>>>>>> them into a GitHub Action), with that GitHub Action, we can
> >> simply
> >>>>>>>> configure a workflow/job in every submodule, and the tests are
> >>>> executed
> >>>>>>>> with the shared scripts, no copy-pasting.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Note, GitHub Action will require an Apache release every time
> >>> because
> >>>> we
> >>>>>>> are distributing the binary.
> >>>>>>>
> >>>>>>>
> >>>>>>> Sheng Wu 吴晟
> >>>>>>> Twitter, wusheng1108
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Jiajing
> >>>>>>>>>
> >>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
> >>>>>>> kezhenxu94@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> So, we would provide a script set to drive this?
> >>>>>>>>>>
> >>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends.
> >>>>>>>>>>
> >>>>>>>>>>> My question is more about,
> >>>>>>>>>>> how to work with docker-compose
> >>>>>>>>>>
> >>>>>>>>>> It’s the same as what we’re doing now in the current version of
> >>> E2E
> >>>>>>>> tests, now we have many `docker-compose.yaml` files, and all the
> >>>>>>>> `docker-compose.yaml` files are reusable in the new framework, we
> >>>> will
> >>>>>>> keep
> >>>>>>>> this mechanism as it is proved to be convenient and easy to use
> >>>>> (remember
> >>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E
> >>>> test,
> >>>>>>> not
> >>>>>>>> yet merged though, it’s fast/easy to add a new case).
> >>>>>>>>>>
> >>>>>>>>>> —————————
> >>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>
> >>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
> >>> wu.sheng.841108@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> >>>> 下午9:16写道：
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I want to ask, how the docker-compose lands in your mind? I
> >>>> think
> >>>>>>> CLI
> >>>>>>>>>>>> can't
> >>>>>>>>>>>>> control this part.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> docker-compose just fits the requirements perfectly that we
> >>> need,
> >>>>> we
> >>>>>>>> have
> >>>>>>>>>>>> been using it in the current E2E and turns out it has
> >>> advantages
> >>>> in
> >>>>>>>>>>>> starting many services correctly with health check and
> >>> dependency
> >>>>>>>>>>>> declaration. (One may say Kubernetes can do the same thing,
> >> but
> >>>>> it’s
> >>>>>>>> kind
> >>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex).
> >>>>>>>>>>>>
> >>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
> >>> related
> >>>>>>>> things,
> >>>>>>>>>>>> it just provides query ability (the same as what it is doing
> >>> now,
> >>>>>>>> just need
> >>>>>>>>>>>> to check whether it covers all the needed query interfaces or
> >>>> not),
> >>>>>>>> we will
> >>>>>>>>>>>> have a dedicated control program to
> >>>>>>>>>>>>
> >>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no new
> >>>>>>>> knowledge);
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> So, we would provide a script set to drive this? My question
> >> is
> >>>> more
> >>>>>>>> about,
> >>>>>>>>>>> how to work with docker-compose, rather than challanging the
> >>>> choice.
> >>>>>>>>>>>
> >>>>>>>>>>> Sheng Wu 吴晟
> >>>>>>>>>>> Twitter, wusheng1108
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository
> >>> and
> >>>>>>>> codes);
> >>>>>>>>>>>> (3) verify the actual data (by verification tool, a new tool,
> >>>>> TODO);
> >>>>>>>>>>>> (4) and determine the test result (success or failed);
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> —————————
> >>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Sheng Wu 吴晟
> >>>>>>>>>>>>> Twitter, wusheng1108
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> >>>>> 下午5:27写道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build the
> >>>> next
> >>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking,
> >> which
> >>>> may
> >>>>>>>> not
> >>>>>>>>>>>> be a
> >>>>>>>>>>>>>> high priority but a necessity.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very
> >> important
> >>>>> role
> >>>>>>>> in
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> contribution process, and the current E2E tests just work
> >>> well
> >>>> to
> >>>>>>>> verify
> >>>>>>>>>>>>>> every single pull request before merging, so why bother to
> >>>> build
> >>>>>>> the
> >>>>>>>>>>>> next
> >>>>>>>>>>>>>> generation of E2E test framework?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
> >> sub-projects
> >>>>> that
> >>>>>>>> vary
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to
> >> help
> >>> us
> >>>>>>>> verify
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> pull requests, but we can see that they reimplement the E2E
> >>>> test
> >>>>>>>>>>>> framework
> >>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t have
> >>> E2E
> >>>>>>>> tests at
> >>>>>>>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry
> >>> and
> >>>>>>>>>>>> introduces
> >>>>>>>>>>>>>> more duplicated work when adding a new test case. The
> >>> ultimate
> >>>>>>> goal
> >>>>>>>> is
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> reuse all the test cases when needed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running
> >>>>>>> Java-based
> >>>>>>>>>>>>>> program is not an easy thing for non-Java developers
> >>>> (maintainers
> >>>>>>> of
> >>>>>>>>>>>> other
> >>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set
> >>> up
> >>>>> the
> >>>>>>>>>>>>>> environment correctly and then run the tests, neither is it
> >>> an
> >>>>>>> easy
> >>>>>>>>>>>> skill
> >>>>>>>>>>>>>> to debug the tests.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
> >> because
> >>> we
> >>>>>>>> added
> >>>>>>>>>>>> many
> >>>>>>>>>>>>>> cases and many duplicated codes that need to be removed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Therefore we're planning to build an unified, easy-to-use,
> >>> and
> >>>>>>> fast
> >>>>>>>> E2E
> >>>>>>>>>>>>>> test framework to benifit all the sub-projects.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Here are some rough ideas about this framework:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml
> >> to
> >>>>>>>>>>>> orchestrate
> >>>>>>>>>>>>>> the services that are to be tested, it’s proved to be
> >>> friendly
> >>>>> and
> >>>>>>>>>>>>>> debuggable because we can start it directly even if we are
> >>> not
> >>>>>>>> writing
> >>>>>>>>>>>> E2E
> >>>>>>>>>>>>>> tests.
> >>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI
> >>> project
> >>>> to
> >>>>>>>> query
> >>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
> >>> versions
> >>>>> of
> >>>>>>>> query
> >>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes and
> >>> one
> >>>>> in
> >>>>>>>> the
> >>>>>>>>>>>> CLI).
> >>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data
> >> and
> >>>> the
> >>>>>>>> actual
> >>>>>>>>>>>>>> data, we now have a custom schema of the expected data yaml
> >>>> file,
> >>>>>>>> the
> >>>>>>>>>>>>>> verification codes should be packaged into an executable
> >>>> command
> >>>>>>> so
> >>>>>>>>>>>> that it
> >>>>>>>>>>>>>> can be executed standalone without Java and maven
> >> knowledge.
> >>> I
> >>>>>>> hope
> >>>>>>>> we
> >>>>>>>>>>>> can
> >>>>>>>>>>>>>> leverage existing standards to write the YAML files and do
> >>>>>>>>>>>> verifications.
> >>>>>>>>>>>>>> 3. Build an orchestrating tool to start the
> >>>>> `docker-compose.yaml`,
> >>>>>>>>>>>> health
> >>>>>>>>>>>>>> check, query actual data, and verify the result.
> >>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to
> >>>> share
> >>>>>>>>>>>> between
> >>>>>>>>>>>>>> sub-projects.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If you have any other better idea or want to complement to
> >>> this
> >>>>>>>>>>>> proposal,
> >>>>>>>>>>>>>> please reply here. I will create an issue to track these
> >>> tasks
> >>>>>>>> later.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We have two students (who just finished the Summer Code
> >> 2020
> >>>>>>>> Projects)
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said
> >>>> this
> >>>>>>> is
> >>>>>>>> a
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>> high priority, you have adequate time to get yourselves
> >>>> familiar
> >>>>>>>> with
> >>>>>>>>>>>> how
> >>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
> >>> SkyWalking
> >>>>>>> work
> >>>>>>>> as
> >>>>>>>>>>>> well
> >>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and willingness
> >>> to
> >>>>>>> help.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [1] https://github.com/Humbertzhang
> >>>>>>>>>>>>>> [2] https://github.com/fgksgf/
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> —————————
> >>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> —————————
> >>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> —————————
> >>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>> GitHub @kezhenxu94
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by "kezhenxu94@apache" <ke...@apache.org>.
> 1. Docker-compose and kind are 2 options to set up the environments.
> 2. Totally replace the Java and Maven based driving system.
> 3. Enhance CLI to validate the GraphQL
> 4. Keep the agent test tool unchanged for now as it is already a separate
> system from the e2e.
This conclusion is good for me.
What are others opinions?
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
> On Nov 17, 2020, at 2:42 PM, Sheng Wu <wu...@gmail.com> wrote:
>
> Using k8s and docker-compose as 2 options in the test process is
> reasonable, and should be supported.
>
> Let's keep the conclusion as simple as possible. In the next generation e2e
> framework
> 1. Docker-compose and kind are 2 options to set up the environments.
> 2. Totally replace the Java and Maven based driving system.
> 3. Enhance CLI to validate the GraphQL
> 4. Keep the agent test tool unchanged for now as it is already a separate
> system from the e2e.
>
> Are these good enough for everyone?
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
>
>>>
>>> We can see from above, your discussion actually is only related to 3.
>>
>>
>> That's the only I intend to discuss here if you notice my first
>> response('May SWCK help about provision and demo") in this thread. For
>> other parts, I don't have any tips to share, though.
>> Let me explain it more clearly. I want kubernetes to be seen as an
>> alternative to docker-compose to play a running-engine role in the test
>> framework. If kubernetes can't replace
>> the latter one Is it able to participate in it to solve the dedicated or
>> special issues.
>>
>> Our e2e test and plugin test frameworks are already top-level complicity
>>> thing in the worldwide open source field. Please don't over-design it,
>> and
>>> too aggressive, the world will stay in the hybrid env for a very long
>> time,
>>> same as the developer.
>>
>>
>> Reusing current elaborate output might be reasonable. But in this thread,
>> we're talking about the potential solution to build a
>> next-generation framework, which might mean that the current framework
>> is too complicated to maintain; we need a more tiny way to support more and
>> more cases.
>>
>> And finally, SWCK is gonna and has to build a similar test framework. If
>> the test framework opts for the kubernetes solution, we could share test
>> cases and the underlying framework. It's a more efficient
>> and consistent path for the entire system.
>>
>> regards, Hongtao.
>>
>> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
>>
>>> Hi
>>>
>>> I don't think we have to make the decision on k8s or out-of-k8s.
>>> e2e and all integration tests have their scenarios like SkyWalking have
>>> multiple languages based and mesh-based, even Prometheus adoption.
>>> The same thing happens on the test field, k8s, and out of k8s.
>>>
>>> I would like to agree with both of you, k8s is needed, no matter in mesh
>>> solution(ALS, telemetry v2), but also not anyone needs that, such as
>> agent
>>> developer. We wouldn't deny the fact, the docker-compose is more
>>> lightweight and easier to learn.
>>> So, let me get this straight, for the test you need
>>> 1. Build the source from different repos
>>> 2. Build the images
>>> 3. Use some platform(docker-compose or k8s) to run the images in one
>> piece.
>>> 4. Give some inputs to the env
>>> 5. Use some tools/ways to check what should happen according to (4)'s
>> input
>>> 6. Checked or timeout failure.
>>>
>>> We can see from above, your discussion actually is only related to 3.
>> Let's
>>> not see this as a battle, we need both. Many developers/users use OAP and
>>> SkyWalking out of k8s, that is a simple fact.
>>> Our e2e test and plugin test frameworks are already top-level complicity
>>> thing in the worldwide open source field. Please don't over-design it,
>> and
>>> too aggressive, the world will stay in the hybrid env for a very long
>> time,
>>> same as the developer.
>>>
>>> Also, let's keep in mind, k8s is popular becomes it doesn't require the
>>> developers to understand as the VM did. We as an open-source community,
>> are
>>> focusing on the developer's capabilities.
>>>
>>> Sheng Wu 吴晟
>>> Twitter, wusheng1108
>>>
>>>
>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
>>>
>>>>>
>>>>> but I have been thinking whether or not it is overkill for testing
>>>>> purposes to introduce Kubernetes things.
>>>>
>>>>
>>>> we all know the fact docker-compose is more portable than Kubernetes,
>>>> friendly to local running. But there're also some benefits to replace
>> it
>>>> with kubernetes system:
>>>>
>>>> 1. SWCK has to test based on a real Kubernetes environment if the main
>>>> repo test framework doesn't support it(following the docker-compose
>>> stack),
>>>> That means skywalking ecosystem MUST introduce kubernetes which is not
>> an
>>>> optional component.
>>>>
>>>> 2. Kubernetes support to scale applications at runtime, that help
>>> improve
>>>> the process of e2e. For instance, we could merge a single instance and
>>>> cluster test, by scaling up replicas. This strategy will reduce the
>> total
>>>> time sharply.
>>>>
>>>> If we are going to use KIND, I hope we can give a script or something
>>>>> similar to install all the needed components and create cluster for
>>>>> convenient testing locally, Ke Zhang and Huaxi.
>>>>
>>>>
>>>> That's SWCK's capacity to provide a standard and simple way to create a
>>>> cluster in every environment,
>>>>
>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一 下午7:00写道：
>>>>
>>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting
>>> Kubernetes-based
>>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too,
>> but I
>>>>> have been thinking whether or not it is overkill for testing purposes
>>> to
>>>>> introduce Kubernetes things.
>>>>>
>>>>> Since you mentioned this, we are open to discuss this much more
>> deeply:
>>>>>
>>>>> It would be definitely a benefit to involve more “end”s in the
>> so-call
>>>>> “end to end” tests, I’m +1 to leverage SWCK as well.
>>>>>
>>>>> If we are going to use KIND, I hope we can give a script or something
>>>>> similar to install all the needed components and create cluster for
>>>>> convenient testing locally, Ke Zhang and Huaxi.
>>>>>
>>>>>
>>>>> —————————
>>>>> Zhenxu Ke (柯振旭)
>>>>> GitHub @kezhenxu94
>>>>>
>>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org>
>>> wrote:
>>>>>>
>>>>>> May SWCK help about provision and demo
>>>>>>
>>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone
>>> instance
>>>>> or
>>>>>> cluster for e2e. We could compose indicated CR yamls for different
>>>>>> scenarios.
>>>>>> 2. Different demo for different cases is mandatory for e2e and
>> swck,
>>> we
>>>>>> could build a group of demo projects for them. With Kubernetes's
>>>> support,
>>>>>> we could
>>>>>> get free of verbose scripts that are written for installation,
>>>>> checking
>>>>>> the state of running applications, collecting logs.
>>>>>>
>>>>>> Ppl might have concerts about the minkube we currently have,
>> which's
>>>> hard
>>>>>> to debug and runs slowly.
>>>>>>
>>>>>> I prefer to kind[1] instead of minkube. From my experience, kind
>>> works
>>>>> fine
>>>>>> in an e2e scenario(limited cpu, memory resources), and it is run
>>>> million
>>>>>> times
>>>>>> on my team's e2e environment, which proves the stability and
>>>>>> maintainability of it.
>>>>>>
>>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
>>>>>>
>>>>>> regards, Hongtao
>>>>>>
>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
>>>>>>
>>>>>>> Inline
>>>>>>>
>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>> 下午11:16写道：
>>>>>>>
>>>>>>>> Hi Jiajing,
>>>>>>>>
>>>>>>>> Thanks for sharing your context and joining the discussion. It’s
>>> true
>>>>>>> that
>>>>>>>> we’re actually tackling the same problems in this domain. Some of
>>> the
>>>>>>>> issues you mentioned are solved in the current E2E framework
>> (like
>>>>>>> `retry`,
>>>>>>>> `timeout`, etc.), while some others are still there without ideal
>>>>>>> solution.
>>>>>>>> Other discussions, please see my comments inline.
>>>>>>>>
>>>>>>>>> - the first issue is useful in health check steps, just like the
>>>> ones
>>>>>>>>> defined in readiness probe and liveness probe in kubelet.
>>>>>>>>
>>>>>>>> Yes, the reason why we decided to pick docker-compose is that, it
>>> has
>>>>> the
>>>>>>>> same functionality with kubelet in terms of dependent services
>>>> startup
>>>>>>>> order and healthiness checking, while it’s more lite-weight and
>>> easy
>>>>> for
>>>>>>>> both developers and GitHub Actions environment. Right?
>>>>>>>>
>>>>>>>>> - the second issue is critical for highly-customized assertions,
>>> we
>>>>>>>>> can leverage the `text/template` module to provide highly
>>> extensive
>>>>>>>>> assertion functions
>>>>>>>>
>>>>>>>> `text/template` is also the first thought that came to my mind
>> when
>>>>>>>> considering customized assertors, customized assertors are for
>> sure
>>>>>>>> critical in SkyWalking E2E tests because we have many kinds of
>>>>>>> assertions,
>>>>>>>> some of which are not foreseen until we actually need them,
>>>>>>>> `AtLeastOneGreaterThanZero` is an example.
>>>>>>>>
>>>>>>>>> - the last but not the least, the lifecycle hooks can be used
>> for
>>>>>>>>> accurate control of test setup and teardown, such as
>>> `docker-compose
>>>>>>>>> build, up -d, down`
>>>>>>>>
>>>>>>>> The lifecycle hooks are what I didn’t take into consideration
>>>> because I
>>>>>>>> thought it is quite natural for test framework to provide this
>>>>> features,
>>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part.
>>>>>>>>
>>>>>>>>> Together with the aforementioned ideas, we will have a
>> go-written,
>>>>>>>>> yaml-data driven e2e testing framework which can overcome the
>>>> current
>>>>>>>>> problems that occur,
>>>>>>>>
>>>>>>>> I agree.
>>>>>>>>
>>>>>>>>> in which the most annoying issue is the fragmentation of
>> different
>>>>>>>>> scripts in different submodules, IMO.
>>>>>>>>
>>>>>>>> That’s why I list the last item in the previous email, (Nice to
>>> have,
>>>>>>> wrap
>>>>>>>> them into a GitHub Action), with that GitHub Action, we can
>> simply
>>>>>>>> configure a workflow/job in every submodule, and the tests are
>>>> executed
>>>>>>>> with the shared scripts, no copy-pasting.
>>>>>>>>
>>>>>>>
>>>>>>> Note, GitHub Action will require an Apache release every time
>>> because
>>>> we
>>>>>>> are distributing the binary.
>>>>>>>
>>>>>>>
>>>>>>> Sheng Wu 吴晟
>>>>>>> Twitter, wusheng1108
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Jiajing
>>>>>>>>>
>>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
>>>>>>> kezhenxu94@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> So, we would provide a script set to drive this?
>>>>>>>>>>
>>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends.
>>>>>>>>>>
>>>>>>>>>>> My question is more about,
>>>>>>>>>>> how to work with docker-compose
>>>>>>>>>>
>>>>>>>>>> It’s the same as what we’re doing now in the current version of
>>> E2E
>>>>>>>> tests, now we have many `docker-compose.yaml` files, and all the
>>>>>>>> `docker-compose.yaml` files are reusable in the new framework, we
>>>> will
>>>>>>> keep
>>>>>>>> this mechanism as it is proved to be convenient and easy to use
>>>>> (remember
>>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E
>>>> test,
>>>>>>> not
>>>>>>>> yet merged though, it’s fast/easy to add a new case).
>>>>>>>>>>
>>>>>>>>>> —————————
>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>
>>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
>>> wu.sheng.841108@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>> 下午9:16写道：
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I want to ask, how the docker-compose lands in your mind? I
>>>> think
>>>>>>> CLI
>>>>>>>>>>>> can't
>>>>>>>>>>>>> control this part.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> docker-compose just fits the requirements perfectly that we
>>> need,
>>>>> we
>>>>>>>> have
>>>>>>>>>>>> been using it in the current E2E and turns out it has
>>> advantages
>>>> in
>>>>>>>>>>>> starting many services correctly with health check and
>>> dependency
>>>>>>>>>>>> declaration. (One may say Kubernetes can do the same thing,
>> but
>>>>> it’s
>>>>>>>> kind
>>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex).
>>>>>>>>>>>>
>>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
>>> related
>>>>>>>> things,
>>>>>>>>>>>> it just provides query ability (the same as what it is doing
>>> now,
>>>>>>>> just need
>>>>>>>>>>>> to check whether it covers all the needed query interfaces or
>>>> not),
>>>>>>>> we will
>>>>>>>>>>>> have a dedicated control program to
>>>>>>>>>>>>
>>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no new
>>>>>>>> knowledge);
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So, we would provide a script set to drive this? My question
>> is
>>>> more
>>>>>>>> about,
>>>>>>>>>>> how to work with docker-compose, rather than challanging the
>>>> choice.
>>>>>>>>>>>
>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository
>>> and
>>>>>>>> codes);
>>>>>>>>>>>> (3) verify the actual data (by verification tool, a new tool,
>>>>> TODO);
>>>>>>>>>>>> (4) and determine the test result (success or failed);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> —————————
>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>> 下午5:27写道：
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build the
>>>> next
>>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking,
>> which
>>>> may
>>>>>>>> not
>>>>>>>>>>>> be a
>>>>>>>>>>>>>> high priority but a necessity.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very
>> important
>>>>> role
>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>>>> contribution process, and the current E2E tests just work
>>> well
>>>> to
>>>>>>>> verify
>>>>>>>>>>>>>> every single pull request before merging, so why bother to
>>>> build
>>>>>>> the
>>>>>>>>>>>> next
>>>>>>>>>>>>>> generation of E2E test framework?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
>> sub-projects
>>>>> that
>>>>>>>> vary
>>>>>>>>>>>> in
>>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to
>> help
>>> us
>>>>>>>> verify
>>>>>>>>>>>> the
>>>>>>>>>>>>>> pull requests, but we can see that they reimplement the E2E
>>>> test
>>>>>>>>>>>> framework
>>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t have
>>> E2E
>>>>>>>> tests at
>>>>>>>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry
>>> and
>>>>>>>>>>>> introduces
>>>>>>>>>>>>>> more duplicated work when adding a new test case. The
>>> ultimate
>>>>>>> goal
>>>>>>>> is
>>>>>>>>>>>> to
>>>>>>>>>>>>>> reuse all the test cases when needed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running
>>>>>>> Java-based
>>>>>>>>>>>>>> program is not an easy thing for non-Java developers
>>>> (maintainers
>>>>>>> of
>>>>>>>>>>>> other
>>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set
>>> up
>>>>> the
>>>>>>>>>>>>>> environment correctly and then run the tests, neither is it
>>> an
>>>>>>> easy
>>>>>>>>>>>> skill
>>>>>>>>>>>>>> to debug the tests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
>> because
>>> we
>>>>>>>> added
>>>>>>>>>>>> many
>>>>>>>>>>>>>> cases and many duplicated codes that need to be removed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Therefore we're planning to build an unified, easy-to-use,
>>> and
>>>>>>> fast
>>>>>>>> E2E
>>>>>>>>>>>>>> test framework to benifit all the sub-projects.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here are some rough ideas about this framework:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml
>> to
>>>>>>>>>>>> orchestrate
>>>>>>>>>>>>>> the services that are to be tested, it’s proved to be
>>> friendly
>>>>> and
>>>>>>>>>>>>>> debuggable because we can start it directly even if we are
>>> not
>>>>>>>> writing
>>>>>>>>>>>> E2E
>>>>>>>>>>>>>> tests.
>>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI
>>> project
>>>> to
>>>>>>>> query
>>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
>>> versions
>>>>> of
>>>>>>>> query
>>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes and
>>> one
>>>>> in
>>>>>>>> the
>>>>>>>>>>>> CLI).
>>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data
>> and
>>>> the
>>>>>>>> actual
>>>>>>>>>>>>>> data, we now have a custom schema of the expected data yaml
>>>> file,
>>>>>>>> the
>>>>>>>>>>>>>> verification codes should be packaged into an executable
>>>> command
>>>>>>> so
>>>>>>>>>>>> that it
>>>>>>>>>>>>>> can be executed standalone without Java and maven
>> knowledge.
>>> I
>>>>>>> hope
>>>>>>>> we
>>>>>>>>>>>> can
>>>>>>>>>>>>>> leverage existing standards to write the YAML files and do
>>>>>>>>>>>> verifications.
>>>>>>>>>>>>>> 3. Build an orchestrating tool to start the
>>>>> `docker-compose.yaml`,
>>>>>>>>>>>> health
>>>>>>>>>>>>>> check, query actual data, and verify the result.
>>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to
>>>> share
>>>>>>>>>>>> between
>>>>>>>>>>>>>> sub-projects.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you have any other better idea or want to complement to
>>> this
>>>>>>>>>>>> proposal,
>>>>>>>>>>>>>> please reply here. I will create an issue to track these
>>> tasks
>>>>>>>> later.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have two students (who just finished the Summer Code
>> 2020
>>>>>>>> Projects)
>>>>>>>>>>>> to
>>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said
>>>> this
>>>>>>> is
>>>>>>>> a
>>>>>>>>>>>> not
>>>>>>>>>>>>>> high priority, you have adequate time to get yourselves
>>>> familiar
>>>>>>>> with
>>>>>>>>>>>> how
>>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
>>> SkyWalking
>>>>>>> work
>>>>>>>> as
>>>>>>>>>>>> well
>>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and willingness
>>> to
>>>>>>> help.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] https://github.com/Humbertzhang
>>>>>>>>>>>>>> [2] https://github.com/fgksgf/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> —————————
>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> —————————
>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>> GitHub @kezhenxu94
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Sheng Wu <wu...@gmail.com>.
Using k8s and docker-compose as 2 options in the test process is
reasonable, and should be supported.
Let's keep the conclusion as simple as possible. In the next generation e2e
framework
1. Docker-compose and kind are 2 options to set up the environments.
2. Totally replace the Java and Maven based driving system.
3. Enhance CLI to validate the GraphQL
4. Keep the agent test tool unchanged for now as it is already a separate
system from the e2e.
Are these good enough for everyone?
Sheng Wu 吴晟
Twitter, wusheng1108
Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
> >
> > We can see from above, your discussion actually is only related to 3.
>
>
> That's the only I intend to discuss here if you notice my first
> response('May SWCK help about provision and demo") in this thread. For
> other parts, I don't have any tips to share, though.
> Let me explain it more clearly. I want kubernetes to be seen as an
> alternative to docker-compose to play a running-engine role in the test
> framework. If kubernetes can't replace
> the latter one Is it able to participate in it to solve the dedicated or
> special issues.
>
> Our e2e test and plugin test frameworks are already top-level complicity
> > thing in the worldwide open source field. Please don't over-design it,
> and
> > too aggressive, the world will stay in the hybrid env for a very long
> time,
> > same as the developer.
>
>
> Reusing current elaborate output might be reasonable. But in this thread,
> we're talking about the potential solution to build a
> next-generation framework, which might mean that the current framework
> is too complicated to maintain; we need a more tiny way to support more and
> more cases.
>
> And finally, SWCK is gonna and has to build a similar test framework. If
> the test framework opts for the kubernetes solution, we could share test
> cases and the underlying framework. It's a more efficient
> and consistent path for the entire system.
>
> regards, Hongtao.
>
> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
>
> > Hi
> >
> > I don't think we have to make the decision on k8s or out-of-k8s.
> > e2e and all integration tests have their scenarios like SkyWalking have
> > multiple languages based and mesh-based, even Prometheus adoption.
> > The same thing happens on the test field, k8s, and out of k8s.
> >
> > I would like to agree with both of you, k8s is needed, no matter in mesh
> > solution(ALS, telemetry v2), but also not anyone needs that, such as
> agent
> > developer. We wouldn't deny the fact, the docker-compose is more
> > lightweight and easier to learn.
> > So, let me get this straight, for the test you need
> > 1. Build the source from different repos
> > 2. Build the images
> > 3. Use some platform(docker-compose or k8s) to run the images in one
> piece.
> > 4. Give some inputs to the env
> > 5. Use some tools/ways to check what should happen according to (4)'s
> input
> > 6. Checked or timeout failure.
> >
> > We can see from above, your discussion actually is only related to 3.
> Let's
> > not see this as a battle, we need both. Many developers/users use OAP and
> > SkyWalking out of k8s, that is a simple fact.
> > Our e2e test and plugin test frameworks are already top-level complicity
> > thing in the worldwide open source field. Please don't over-design it,
> and
> > too aggressive, the world will stay in the hybrid env for a very long
> time,
> > same as the developer.
> >
> > Also, let's keep in mind, k8s is popular becomes it doesn't require the
> > developers to understand as the VM did. We as an open-source community,
> are
> > focusing on the developer's capabilities.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
> >
> > > >
> > > > but I have been thinking whether or not it is overkill for testing
> > > > purposes to introduce Kubernetes things.
> > >
> > >
> > > we all know the fact docker-compose is more portable than Kubernetes,
> > > friendly to local running. But there're also some benefits to replace
> it
> > > with kubernetes system:
> > >
> > > 1. SWCK has to test based on a real Kubernetes environment if the main
> > > repo test framework doesn't support it(following the docker-compose
> > stack),
> > > That means skywalking ecosystem MUST introduce kubernetes which is not
> an
> > > optional component.
> > >
> > > 2. Kubernetes support to scale applications at runtime, that help
> > improve
> > > the process of e2e. For instance, we could merge a single instance and
> > > cluster test, by scaling up replicas. This strategy will reduce the
> total
> > > time sharply.
> > >
> > > If we are going to use KIND, I hope we can give a script or something
> > > > similar to install all the needed components and create cluster for
> > > > convenient testing locally, Ke Zhang and Huaxi.
> > >
> > >
> > > That's SWCK's capacity to provide a standard and simple way to create a
> > > cluster in every environment,
> > >
> > > kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一 下午7:00写道：
> > >
> > > > Thanks Hongtao’s suggestion, I’ve thought about adopting
> > Kubernetes-based
> > > > tech stack (e.g. minikube or KIND), I personally prefer KIND too,
> but I
> > > > have been thinking whether or not it is overkill for testing purposes
> > to
> > > > introduce Kubernetes things.
> > > >
> > > > Since you mentioned this, we are open to discuss this much more
> deeply:
> > > >
> > > > It would be definitely a benefit to involve more “end”s in the
> so-call
> > > > “end to end” tests, I’m +1 to leverage SWCK as well.
> > > >
> > > > If we are going to use KIND, I hope we can give a script or something
> > > > similar to install all the needed components and create cluster for
> > > > convenient testing locally, Ke Zhang and Huaxi.
> > > >
> > > >
> > > > —————————
> > > > Zhenxu Ke (柯振旭)
> > > > GitHub @kezhenxu94
> > > >
> > > > > On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org>
> > wrote:
> > > > >
> > > > > May SWCK help about provision and demo
> > > > >
> > > > > 1. SWCK is able to provide a stable and healthy OAP standalone
> > instance
> > > > or
> > > > > cluster for e2e. We could compose indicated CR yamls for different
> > > > > scenarios.
> > > > > 2. Different demo for different cases is mandatory for e2e and
> swck,
> > we
> > > > > could build a group of demo projects for them. With Kubernetes's
> > > support,
> > > > > we could
> > > > > get free of verbose scripts that are written for installation,
> > > > checking
> > > > > the state of running applications, collecting logs.
> > > > >
> > > > > Ppl might have concerts about the minkube we currently have,
> which's
> > > hard
> > > > > to debug and runs slowly.
> > > > >
> > > > > I prefer to kind[1] instead of minkube. From my experience, kind
> > works
> > > > fine
> > > > > in an e2e scenario(limited cpu, memory resources), and it is run
> > > million
> > > > > times
> > > > > on my team's e2e environment, which proves the stability and
> > > > > maintainability of it.
> > > > >
> > > > > * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
> > > > >
> > > > > regards, Hongtao
> > > > >
> > > > > Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
> > > > >
> > > > >> Inline
> > > > >>
> > > > >> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> 下午11:16写道：
> > > > >>
> > > > >>> Hi Jiajing,
> > > > >>>
> > > > >>> Thanks for sharing your context and joining the discussion. It’s
> > true
> > > > >> that
> > > > >>> we’re actually tackling the same problems in this domain. Some of
> > the
> > > > >>> issues you mentioned are solved in the current E2E framework
> (like
> > > > >> `retry`,
> > > > >>> `timeout`, etc.), while some others are still there without ideal
> > > > >> solution.
> > > > >>> Other discussions, please see my comments inline.
> > > > >>>
> > > > >>>> - the first issue is useful in health check steps, just like the
> > > ones
> > > > >>>> defined in readiness probe and liveness probe in kubelet.
> > > > >>>
> > > > >>> Yes, the reason why we decided to pick docker-compose is that, it
> > has
> > > > the
> > > > >>> same functionality with kubelet in terms of dependent services
> > > startup
> > > > >>> order and healthiness checking, while it’s more lite-weight and
> > easy
> > > > for
> > > > >>> both developers and GitHub Actions environment. Right?
> > > > >>>
> > > > >>>> - the second issue is critical for highly-customized assertions,
> > we
> > > > >>>> can leverage the `text/template` module to provide highly
> > extensive
> > > > >>>> assertion functions
> > > > >>>
> > > > >>> `text/template` is also the first thought that came to my mind
> when
> > > > >>> considering customized assertors, customized assertors are for
> sure
> > > > >>> critical in SkyWalking E2E tests because we have many kinds of
> > > > >> assertions,
> > > > >>> some of which are not foreseen until we actually need them,
> > > > >>> `AtLeastOneGreaterThanZero` is an example.
> > > > >>>
> > > > >>>> - the last but not the least, the lifecycle hooks can be used
> for
> > > > >>>> accurate control of test setup and teardown, such as
> > `docker-compose
> > > > >>>> build, up -d, down`
> > > > >>>
> > > > >>> The lifecycle hooks are what I didn’t take into consideration
> > > because I
> > > > >>> thought it is quite natural for test framework to provide this
> > > > features,
> > > > >>> maybe Ke Zhang and Huaxi can do some research on this part.
> > > > >>>
> > > > >>>> Together with the aforementioned ideas, we will have a
> go-written,
> > > > >>>> yaml-data driven e2e testing framework which can overcome the
> > > current
> > > > >>>> problems that occur,
> > > > >>>
> > > > >>> I agree.
> > > > >>>
> > > > >>>> in which the most annoying issue is the fragmentation of
> different
> > > > >>>> scripts in different submodules, IMO.
> > > > >>>
> > > > >>> That’s why I list the last item in the previous email, (Nice to
> > have,
> > > > >> wrap
> > > > >>> them into a GitHub Action), with that GitHub Action, we can
> simply
> > > > >>> configure a workflow/job in every submodule, and the tests are
> > > executed
> > > > >>> with the shared scripts, no copy-pasting.
> > > > >>>
> > > > >>
> > > > >> Note, GitHub Action will require an Apache release every time
> > because
> > > we
> > > > >> are distributing the binary.
> > > > >>
> > > > >>
> > > > >> Sheng Wu 吴晟
> > > > >> Twitter, wusheng1108
> > > > >>
> > > > >>
> > > > >>>
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>> Jiajing
> > > > >>>>
> > > > >>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
> > > > >> kezhenxu94@apache.org>
> > > > >>> wrote:
> > > > >>>>>
> > > > >>>>>> So, we would provide a script set to drive this?
> > > > >>>>>
> > > > >>>>> Yes, a script or a GoLang-written CLI, it depends.
> > > > >>>>>
> > > > >>>>>> My question is more about,
> > > > >>>>>> how to work with docker-compose
> > > > >>>>>
> > > > >>>>> It’s the same as what we’re doing now in the current version of
> > E2E
> > > > >>> tests, now we have many `docker-compose.yaml` files, and all the
> > > > >>> `docker-compose.yaml` files are reusable in the new framework, we
> > > will
> > > > >> keep
> > > > >>> this mechanism as it is proved to be convenient and easy to use
> > > > (remember
> > > > >>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E
> > > test,
> > > > >> not
> > > > >>> yet merged though, it’s fast/easy to add a new case).
> > > > >>>>>
> > > > >>>>> —————————
> > > > >>>>> Zhenxu Ke (柯振旭)
> > > > >>>>> GitHub @kezhenxu94
> > > > >>>>>
> > > > >>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
> > wu.sheng.841108@gmail.com>
> > > > >>> wrote:
> > > > >>>>>>
> > > > >>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > > 下午9:16写道：
> > > > >>>>>>
> > > > >>>>>>>
> > > > >>>>>>>> I want to ask, how the docker-compose lands in your mind? I
> > > think
> > > > >> CLI
> > > > >>>>>>> can't
> > > > >>>>>>>> control this part.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> docker-compose just fits the requirements perfectly that we
> > need,
> > > > we
> > > > >>> have
> > > > >>>>>>> been using it in the current E2E and turns out it has
> > advantages
> > > in
> > > > >>>>>>> starting many services correctly with health check and
> > dependency
> > > > >>>>>>> declaration. (One may say Kubernetes can do the same thing,
> but
> > > > it’s
> > > > >>> kind
> > > > >>>>>>> of heavy in E2E and GitHub Actions and complex).
> > > > >>>>>>>
> > > > >>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
> > related
> > > > >>> things,
> > > > >>>>>>> it just provides query ability (the same as what it is doing
> > now,
> > > > >>> just need
> > > > >>>>>>> to check whether it covers all the needed query interfaces or
> > > not),
> > > > >>> we will
> > > > >>>>>>> have a dedicated control program to
> > > > >>>>>>>
> > > > >>>>>>> (1) start docker-compose (existing standard mechanism, no new
> > > > >>> knowledge);
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> So, we would provide a script set to drive this? My question
> is
> > > more
> > > > >>> about,
> > > > >>>>>> how to work with docker-compose, rather than challanging the
> > > choice.
> > > > >>>>>>
> > > > >>>>>> Sheng Wu 吴晟
> > > > >>>>>> Twitter, wusheng1108
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository
> > and
> > > > >>> codes);
> > > > >>>>>>> (3) verify the actual data (by verification tool, a new tool,
> > > > TODO);
> > > > >>>>>>> (4) and determine the test result (success or failed);
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> —————————
> > > > >>>>>>> Zhenxu Ke (柯振旭)
> > > > >>>>>>> GitHub @kezhenxu94
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>> Sheng Wu 吴晟
> > > > >>>>>>>> Twitter, wusheng1108
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > > > 下午5:27写道：
> > > > >>>>>>>>
> > > > >>>>>>>>> Hi the community, I want to raise a discussion to build the
> > > next
> > > > >>>>>>>>> generation of E2E test framework for Apache SkyWalking,
> which
> > > may
> > > > >>> not
> > > > >>>>>>> be a
> > > > >>>>>>>>> high priority but a necessity.
> > > > >>>>>>>>>
> > > > >>>>>>>>> As we already know, tests in SkyWalking play a very
> important
> > > > role
> > > > >>> in
> > > > >>>>>>> the
> > > > >>>>>>>>> contribution process, and the current E2E tests just work
> > well
> > > to
> > > > >>> verify
> > > > >>>>>>>>> every single pull request before merging, so why bother to
> > > build
> > > > >> the
> > > > >>>>>>> next
> > > > >>>>>>>>> generation of E2E test framework?
> > > > >>>>>>>>>
> > > > >>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
> sub-projects
> > > > that
> > > > >>> vary
> > > > >>>>>>> in
> > > > >>>>>>>>> terms of language, almost all of them need E2E tests to
> help
> > us
> > > > >>> verify
> > > > >>>>>>> the
> > > > >>>>>>>>> pull requests, but we can see that they reimplement the E2E
> > > test
> > > > >>>>>>> framework
> > > > >>>>>>>>> in their languages, (or even worse, some of them don’t have
> > E2E
> > > > >>> tests at
> > > > >>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry
> > and
> > > > >>>>>>> introduces
> > > > >>>>>>>>> more duplicated work when adding a new test case. The
> > ultimate
> > > > >> goal
> > > > >>> is
> > > > >>>>>>> to
> > > > >>>>>>>>> reuse all the test cases when needed.
> > > > >>>>>>>>>
> > > > >>>>>>>>> 2. The current E2E framework is built with Java, running
> > > > >> Java-based
> > > > >>>>>>>>> program is not an easy thing for non-Java developers
> > > (maintainers
> > > > >> of
> > > > >>>>>>> other
> > > > >>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set
> > up
> > > > the
> > > > >>>>>>>>> environment correctly and then run the tests, neither is it
> > an
> > > > >> easy
> > > > >>>>>>> skill
> > > > >>>>>>>>> to debug the tests.
> > > > >>>>>>>>>
> > > > >>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
> because
> > we
> > > > >>> added
> > > > >>>>>>> many
> > > > >>>>>>>>> cases and many duplicated codes that need to be removed.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Therefore we're planning to build an unified, easy-to-use,
> > and
> > > > >> fast
> > > > >>> E2E
> > > > >>>>>>>>> test framework to benifit all the sub-projects.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Here are some rough ideas about this framework:
> > > > >>>>>>>>>
> > > > >>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml
> to
> > > > >>>>>>> orchestrate
> > > > >>>>>>>>> the services that are to be tested, it’s proved to be
> > friendly
> > > > and
> > > > >>>>>>>>> debuggable because we can start it directly even if we are
> > not
> > > > >>> writing
> > > > >>>>>>> E2E
> > > > >>>>>>>>> tests.
> > > > >>>>>>>>> 1. This framework will take full advantages of the CLI
> > project
> > > to
> > > > >>> query
> > > > >>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
> > versions
> > > > of
> > > > >>> query
> > > > >>>>>>>>> codes anymore, (now we have a version in the test codes and
> > one
> > > > in
> > > > >>> the
> > > > >>>>>>> CLI).
> > > > >>>>>>>>> 2. We will build a CLI tool to compare the expected data
> and
> > > the
> > > > >>> actual
> > > > >>>>>>>>> data, we now have a custom schema of the expected data yaml
> > > file,
> > > > >>> the
> > > > >>>>>>>>> verification codes should be packaged into an executable
> > > command
> > > > >> so
> > > > >>>>>>> that it
> > > > >>>>>>>>> can be executed standalone without Java and maven
> knowledge.
> > I
> > > > >> hope
> > > > >>> we
> > > > >>>>>>> can
> > > > >>>>>>>>> leverage existing standards to write the YAML files and do
> > > > >>>>>>> verifications.
> > > > >>>>>>>>> 3. Build an orchestrating tool to start the
> > > > `docker-compose.yaml`,
> > > > >>>>>>> health
> > > > >>>>>>>>> check, query actual data, and verify the result.
> > > > >>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to
> > > share
> > > > >>>>>>> between
> > > > >>>>>>>>> sub-projects.
> > > > >>>>>>>>>
> > > > >>>>>>>>> If you have any other better idea or want to complement to
> > this
> > > > >>>>>>> proposal,
> > > > >>>>>>>>> please reply here. I will create an issue to track these
> > tasks
> > > > >>> later.
> > > > >>>>>>>>>
> > > > >>>>>>>>> We have two students (who just finished the Summer Code
> 2020
> > > > >>> Projects)
> > > > >>>>>>> to
> > > > >>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said
> > > this
> > > > >> is
> > > > >>> a
> > > > >>>>>>> not
> > > > >>>>>>>>> high priority, you have adequate time to get yourselves
> > > familiar
> > > > >>> with
> > > > >>>>>>> how
> > > > >>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
> > SkyWalking
> > > > >> work
> > > > >>> as
> > > > >>>>>>> well
> > > > >>>>>>>>> as the ecosystem. Thanks for your interests and willingness
> > to
> > > > >> help.
> > > > >>>>>>>>>
> > > > >>>>>>>>> [1] https://github.com/Humbertzhang
> > > > >>>>>>>>> [2] https://github.com/fgksgf/
> > > > >>>>>>>>>
> > > > >>>>>>>>> —————————
> > > > >>>>>>>>> Zhenxu Ke (柯振旭)
> > > > >>>>>>>>> GitHub @kezhenxu94
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> —————————
> > > > >>>>>>> Zhenxu Ke (柯振旭)
> > > > >>>>>>> GitHub @kezhenxu94
> > > > >>>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> —————————
> > > > >>> Zhenxu Ke (柯振旭)
> > > > >>> GitHub @kezhenxu94
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Hongtao Gao <ha...@apache.org>.
>
> We can see from above, your discussion actually is only related to 3.
That's the only I intend to discuss here if you notice my first
response('May SWCK help about provision and demo") in this thread. For
other parts, I don't have any tips to share, though.
Let me explain it more clearly. I want kubernetes to be seen as an
alternative to docker-compose to play a running-engine role in the test
framework. If kubernetes can't replace
the latter one Is it able to participate in it to solve the dedicated or
special issues.
Our e2e test and plugin test frameworks are already top-level complicity
> thing in the worldwide open source field. Please don't over-design it, and
> too aggressive, the world will stay in the hybrid env for a very long time,
> same as the developer.
Reusing current elaborate output might be reasonable. But in this thread,
we're talking about the potential solution to build a
next-generation framework, which might mean that the current framework
is too complicated to maintain; we need a more tiny way to support more and
more cases.
And finally, SWCK is gonna and has to build a similar test framework. If
the test framework opts for the kubernetes solution, we could share test
cases and the underlying framework. It's a more efficient
and consistent path for the entire system.
regards, Hongtao.
Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
> Hi
>
> I don't think we have to make the decision on k8s or out-of-k8s.
> e2e and all integration tests have their scenarios like SkyWalking have
> multiple languages based and mesh-based, even Prometheus adoption.
> The same thing happens on the test field, k8s, and out of k8s.
>
> I would like to agree with both of you, k8s is needed, no matter in mesh
> solution(ALS, telemetry v2), but also not anyone needs that, such as agent
> developer. We wouldn't deny the fact, the docker-compose is more
> lightweight and easier to learn.
> So, let me get this straight, for the test you need
> 1. Build the source from different repos
> 2. Build the images
> 3. Use some platform(docker-compose or k8s) to run the images in one piece.
> 4. Give some inputs to the env
> 5. Use some tools/ways to check what should happen according to (4)'s input
> 6. Checked or timeout failure.
>
> We can see from above, your discussion actually is only related to 3. Let's
> not see this as a battle, we need both. Many developers/users use OAP and
> SkyWalking out of k8s, that is a simple fact.
> Our e2e test and plugin test frameworks are already top-level complicity
> thing in the worldwide open source field. Please don't over-design it, and
> too aggressive, the world will stay in the hybrid env for a very long time,
> same as the developer.
>
> Also, let's keep in mind, k8s is popular becomes it doesn't require the
> developers to understand as the VM did. We as an open-source community, are
> focusing on the developer's capabilities.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
>
> > >
> > > but I have been thinking whether or not it is overkill for testing
> > > purposes to introduce Kubernetes things.
> >
> >
> > we all know the fact docker-compose is more portable than Kubernetes,
> > friendly to local running. But there're also some benefits to replace it
> > with kubernetes system:
> >
> > 1. SWCK has to test based on a real Kubernetes environment if the main
> > repo test framework doesn't support it(following the docker-compose
> stack),
> > That means skywalking ecosystem MUST introduce kubernetes which is not an
> > optional component.
> >
> > 2. Kubernetes support to scale applications at runtime, that help
> improve
> > the process of e2e. For instance, we could merge a single instance and
> > cluster test, by scaling up replicas. This strategy will reduce the total
> > time sharply.
> >
> > If we are going to use KIND, I hope we can give a script or something
> > > similar to install all the needed components and create cluster for
> > > convenient testing locally, Ke Zhang and Huaxi.
> >
> >
> > That's SWCK's capacity to provide a standard and simple way to create a
> > cluster in every environment,
> >
> > kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一 下午7:00写道：
> >
> > > Thanks Hongtao’s suggestion, I’ve thought about adopting
> Kubernetes-based
> > > tech stack (e.g. minikube or KIND), I personally prefer KIND too, but I
> > > have been thinking whether or not it is overkill for testing purposes
> to
> > > introduce Kubernetes things.
> > >
> > > Since you mentioned this, we are open to discuss this much more deeply:
> > >
> > > It would be definitely a benefit to involve more “end”s in the so-call
> > > “end to end” tests, I’m +1 to leverage SWCK as well.
> > >
> > > If we are going to use KIND, I hope we can give a script or something
> > > similar to install all the needed components and create cluster for
> > > convenient testing locally, Ke Zhang and Huaxi.
> > >
> > >
> > > —————————
> > > Zhenxu Ke (柯振旭)
> > > GitHub @kezhenxu94
> > >
> > > > On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org>
> wrote:
> > > >
> > > > May SWCK help about provision and demo
> > > >
> > > > 1. SWCK is able to provide a stable and healthy OAP standalone
> instance
> > > or
> > > > cluster for e2e. We could compose indicated CR yamls for different
> > > > scenarios.
> > > > 2. Different demo for different cases is mandatory for e2e and swck,
> we
> > > > could build a group of demo projects for them. With Kubernetes's
> > support,
> > > > we could
> > > > get free of verbose scripts that are written for installation,
> > > checking
> > > > the state of running applications, collecting logs.
> > > >
> > > > Ppl might have concerts about the minkube we currently have, which's
> > hard
> > > > to debug and runs slowly.
> > > >
> > > > I prefer to kind[1] instead of minkube. From my experience, kind
> works
> > > fine
> > > > in an e2e scenario(limited cpu, memory resources), and it is run
> > million
> > > > times
> > > > on my team's e2e environment, which proves the stability and
> > > > maintainability of it.
> > > >
> > > > * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
> > > >
> > > > regards, Hongtao
> > > >
> > > > Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
> > > >
> > > >> Inline
> > > >>
> > > >> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午11:16写道：
> > > >>
> > > >>> Hi Jiajing,
> > > >>>
> > > >>> Thanks for sharing your context and joining the discussion. It’s
> true
> > > >> that
> > > >>> we’re actually tackling the same problems in this domain. Some of
> the
> > > >>> issues you mentioned are solved in the current E2E framework (like
> > > >> `retry`,
> > > >>> `timeout`, etc.), while some others are still there without ideal
> > > >> solution.
> > > >>> Other discussions, please see my comments inline.
> > > >>>
> > > >>>> - the first issue is useful in health check steps, just like the
> > ones
> > > >>>> defined in readiness probe and liveness probe in kubelet.
> > > >>>
> > > >>> Yes, the reason why we decided to pick docker-compose is that, it
> has
> > > the
> > > >>> same functionality with kubelet in terms of dependent services
> > startup
> > > >>> order and healthiness checking, while it’s more lite-weight and
> easy
> > > for
> > > >>> both developers and GitHub Actions environment. Right?
> > > >>>
> > > >>>> - the second issue is critical for highly-customized assertions,
> we
> > > >>>> can leverage the `text/template` module to provide highly
> extensive
> > > >>>> assertion functions
> > > >>>
> > > >>> `text/template` is also the first thought that came to my mind when
> > > >>> considering customized assertors, customized assertors are for sure
> > > >>> critical in SkyWalking E2E tests because we have many kinds of
> > > >> assertions,
> > > >>> some of which are not foreseen until we actually need them,
> > > >>> `AtLeastOneGreaterThanZero` is an example.
> > > >>>
> > > >>>> - the last but not the least, the lifecycle hooks can be used for
> > > >>>> accurate control of test setup and teardown, such as
> `docker-compose
> > > >>>> build, up -d, down`
> > > >>>
> > > >>> The lifecycle hooks are what I didn’t take into consideration
> > because I
> > > >>> thought it is quite natural for test framework to provide this
> > > features,
> > > >>> maybe Ke Zhang and Huaxi can do some research on this part.
> > > >>>
> > > >>>> Together with the aforementioned ideas, we will have a go-written,
> > > >>>> yaml-data driven e2e testing framework which can overcome the
> > current
> > > >>>> problems that occur,
> > > >>>
> > > >>> I agree.
> > > >>>
> > > >>>> in which the most annoying issue is the fragmentation of different
> > > >>>> scripts in different submodules, IMO.
> > > >>>
> > > >>> That’s why I list the last item in the previous email, (Nice to
> have,
> > > >> wrap
> > > >>> them into a GitHub Action), with that GitHub Action, we can simply
> > > >>> configure a workflow/job in every submodule, and the tests are
> > executed
> > > >>> with the shared scripts, no copy-pasting.
> > > >>>
> > > >>
> > > >> Note, GitHub Action will require an Apache release every time
> because
> > we
> > > >> are distributing the binary.
> > > >>
> > > >>
> > > >> Sheng Wu 吴晟
> > > >> Twitter, wusheng1108
> > > >>
> > > >>
> > > >>>
> > > >>>>
> > > >>>> Regards,
> > > >>>> Jiajing
> > > >>>>
> > > >>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
> > > >> kezhenxu94@apache.org>
> > > >>> wrote:
> > > >>>>>
> > > >>>>>> So, we would provide a script set to drive this?
> > > >>>>>
> > > >>>>> Yes, a script or a GoLang-written CLI, it depends.
> > > >>>>>
> > > >>>>>> My question is more about,
> > > >>>>>> how to work with docker-compose
> > > >>>>>
> > > >>>>> It’s the same as what we’re doing now in the current version of
> E2E
> > > >>> tests, now we have many `docker-compose.yaml` files, and all the
> > > >>> `docker-compose.yaml` files are reusable in the new framework, we
> > will
> > > >> keep
> > > >>> this mechanism as it is proved to be convenient and easy to use
> > > (remember
> > > >>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E
> > test,
> > > >> not
> > > >>> yet merged though, it’s fast/easy to add a new case).
> > > >>>>>
> > > >>>>> —————————
> > > >>>>> Zhenxu Ke (柯振旭)
> > > >>>>> GitHub @kezhenxu94
> > > >>>>>
> > > >>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
> wu.sheng.841108@gmail.com>
> > > >>> wrote:
> > > >>>>>>
> > > >>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > 下午9:16写道：
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>>> I want to ask, how the docker-compose lands in your mind? I
> > think
> > > >> CLI
> > > >>>>>>> can't
> > > >>>>>>>> control this part.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> docker-compose just fits the requirements perfectly that we
> need,
> > > we
> > > >>> have
> > > >>>>>>> been using it in the current E2E and turns out it has
> advantages
> > in
> > > >>>>>>> starting many services correctly with health check and
> dependency
> > > >>>>>>> declaration. (One may say Kubernetes can do the same thing, but
> > > it’s
> > > >>> kind
> > > >>>>>>> of heavy in E2E and GitHub Actions and complex).
> > > >>>>>>>
> > > >>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
> related
> > > >>> things,
> > > >>>>>>> it just provides query ability (the same as what it is doing
> now,
> > > >>> just need
> > > >>>>>>> to check whether it covers all the needed query interfaces or
> > not),
> > > >>> we will
> > > >>>>>>> have a dedicated control program to
> > > >>>>>>>
> > > >>>>>>> (1) start docker-compose (existing standard mechanism, no new
> > > >>> knowledge);
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> So, we would provide a script set to drive this? My question is
> > more
> > > >>> about,
> > > >>>>>> how to work with docker-compose, rather than challanging the
> > choice.
> > > >>>>>>
> > > >>>>>> Sheng Wu 吴晟
> > > >>>>>> Twitter, wusheng1108
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository
> and
> > > >>> codes);
> > > >>>>>>> (3) verify the actual data (by verification tool, a new tool,
> > > TODO);
> > > >>>>>>> (4) and determine the test result (success or failed);
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> —————————
> > > >>>>>>> Zhenxu Ke (柯振旭)
> > > >>>>>>> GitHub @kezhenxu94
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>> Sheng Wu 吴晟
> > > >>>>>>>> Twitter, wusheng1108
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > > 下午5:27写道：
> > > >>>>>>>>
> > > >>>>>>>>> Hi the community, I want to raise a discussion to build the
> > next
> > > >>>>>>>>> generation of E2E test framework for Apache SkyWalking, which
> > may
> > > >>> not
> > > >>>>>>> be a
> > > >>>>>>>>> high priority but a necessity.
> > > >>>>>>>>>
> > > >>>>>>>>> As we already know, tests in SkyWalking play a very important
> > > role
> > > >>> in
> > > >>>>>>> the
> > > >>>>>>>>> contribution process, and the current E2E tests just work
> well
> > to
> > > >>> verify
> > > >>>>>>>>> every single pull request before merging, so why bother to
> > build
> > > >> the
> > > >>>>>>> next
> > > >>>>>>>>> generation of E2E test framework?
> > > >>>>>>>>>
> > > >>>>>>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects
> > > that
> > > >>> vary
> > > >>>>>>> in
> > > >>>>>>>>> terms of language, almost all of them need E2E tests to help
> us
> > > >>> verify
> > > >>>>>>> the
> > > >>>>>>>>> pull requests, but we can see that they reimplement the E2E
> > test
> > > >>>>>>> framework
> > > >>>>>>>>> in their languages, (or even worse, some of them don’t have
> E2E
> > > >>> tests at
> > > >>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry
> and
> > > >>>>>>> introduces
> > > >>>>>>>>> more duplicated work when adding a new test case. The
> ultimate
> > > >> goal
> > > >>> is
> > > >>>>>>> to
> > > >>>>>>>>> reuse all the test cases when needed.
> > > >>>>>>>>>
> > > >>>>>>>>> 2. The current E2E framework is built with Java, running
> > > >> Java-based
> > > >>>>>>>>> program is not an easy thing for non-Java developers
> > (maintainers
> > > >> of
> > > >>>>>>> other
> > > >>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set
> up
> > > the
> > > >>>>>>>>> environment correctly and then run the tests, neither is it
> an
> > > >> easy
> > > >>>>>>> skill
> > > >>>>>>>>> to debug the tests.
> > > >>>>>>>>>
> > > >>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests because
> we
> > > >>> added
> > > >>>>>>> many
> > > >>>>>>>>> cases and many duplicated codes that need to be removed.
> > > >>>>>>>>>
> > > >>>>>>>>> Therefore we're planning to build an unified, easy-to-use,
> and
> > > >> fast
> > > >>> E2E
> > > >>>>>>>>> test framework to benifit all the sub-projects.
> > > >>>>>>>>>
> > > >>>>>>>>> Here are some rough ideas about this framework:
> > > >>>>>>>>>
> > > >>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to
> > > >>>>>>> orchestrate
> > > >>>>>>>>> the services that are to be tested, it’s proved to be
> friendly
> > > and
> > > >>>>>>>>> debuggable because we can start it directly even if we are
> not
> > > >>> writing
> > > >>>>>>> E2E
> > > >>>>>>>>> tests.
> > > >>>>>>>>> 1. This framework will take full advantages of the CLI
> project
> > to
> > > >>> query
> > > >>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
> versions
> > > of
> > > >>> query
> > > >>>>>>>>> codes anymore, (now we have a version in the test codes and
> one
> > > in
> > > >>> the
> > > >>>>>>> CLI).
> > > >>>>>>>>> 2. We will build a CLI tool to compare the expected data and
> > the
> > > >>> actual
> > > >>>>>>>>> data, we now have a custom schema of the expected data yaml
> > file,
> > > >>> the
> > > >>>>>>>>> verification codes should be packaged into an executable
> > command
> > > >> so
> > > >>>>>>> that it
> > > >>>>>>>>> can be executed standalone without Java and maven knowledge.
> I
> > > >> hope
> > > >>> we
> > > >>>>>>> can
> > > >>>>>>>>> leverage existing standards to write the YAML files and do
> > > >>>>>>> verifications.
> > > >>>>>>>>> 3. Build an orchestrating tool to start the
> > > `docker-compose.yaml`,
> > > >>>>>>> health
> > > >>>>>>>>> check, query actual data, and verify the result.
> > > >>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to
> > share
> > > >>>>>>> between
> > > >>>>>>>>> sub-projects.
> > > >>>>>>>>>
> > > >>>>>>>>> If you have any other better idea or want to complement to
> this
> > > >>>>>>> proposal,
> > > >>>>>>>>> please reply here. I will create an issue to track these
> tasks
> > > >>> later.
> > > >>>>>>>>>
> > > >>>>>>>>> We have two students (who just finished the Summer Code 2020
> > > >>> Projects)
> > > >>>>>>> to
> > > >>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said
> > this
> > > >> is
> > > >>> a
> > > >>>>>>> not
> > > >>>>>>>>> high priority, you have adequate time to get yourselves
> > familiar
> > > >>> with
> > > >>>>>>> how
> > > >>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
> SkyWalking
> > > >> work
> > > >>> as
> > > >>>>>>> well
> > > >>>>>>>>> as the ecosystem. Thanks for your interests and willingness
> to
> > > >> help.
> > > >>>>>>>>>
> > > >>>>>>>>> [1] https://github.com/Humbertzhang
> > > >>>>>>>>> [2] https://github.com/fgksgf/
> > > >>>>>>>>>
> > > >>>>>>>>> —————————
> > > >>>>>>>>> Zhenxu Ke (柯振旭)
> > > >>>>>>>>> GitHub @kezhenxu94
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>> —————————
> > > >>>>>>> Zhenxu Ke (柯振旭)
> > > >>>>>>> GitHub @kezhenxu94
> > > >>>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> —————————
> > > >>> Zhenxu Ke (柯振旭)
> > > >>> GitHub @kezhenxu94
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Sheng Wu <wu...@gmail.com>.
Hi
I don't think we have to make the decision on k8s or out-of-k8s.
e2e and all integration tests have their scenarios like SkyWalking have
multiple languages based and mesh-based, even Prometheus adoption.
The same thing happens on the test field, k8s, and out of k8s.
I would like to agree with both of you, k8s is needed, no matter in mesh
solution(ALS, telemetry v2), but also not anyone needs that, such as agent
developer. We wouldn't deny the fact, the docker-compose is more
lightweight and easier to learn.
So, let me get this straight, for the test you need
1. Build the source from different repos
2. Build the images
3. Use some platform(docker-compose or k8s) to run the images in one piece.
4. Give some inputs to the env
5. Use some tools/ways to check what should happen according to (4)'s input
6. Checked or timeout failure.
We can see from above, your discussion actually is only related to 3. Let's
not see this as a battle, we need both. Many developers/users use OAP and
SkyWalking out of k8s, that is a simple fact.
Our e2e test and plugin test frameworks are already top-level complicity
thing in the worldwide open source field. Please don't over-design it, and
too aggressive, the world will stay in the hybrid env for a very long time,
same as the developer.
Also, let's keep in mind, k8s is popular becomes it doesn't require the
developers to understand as the VM did. We as an open-source community, are
focusing on the developer's capabilities.
Sheng Wu 吴晟
Twitter, wusheng1108
Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
> >
> > but I have been thinking whether or not it is overkill for testing
> > purposes to introduce Kubernetes things.
>
>
> we all know the fact docker-compose is more portable than Kubernetes,
> friendly to local running. But there're also some benefits to replace it
> with kubernetes system:
>
> 1. SWCK has to test based on a real Kubernetes environment if the main
> repo test framework doesn't support it(following the docker-compose stack),
> That means skywalking ecosystem MUST introduce kubernetes which is not an
> optional component.
>
> 2. Kubernetes support to scale applications at runtime, that help improve
> the process of e2e. For instance, we could merge a single instance and
> cluster test, by scaling up replicas. This strategy will reduce the total
> time sharply.
>
> If we are going to use KIND, I hope we can give a script or something
> > similar to install all the needed components and create cluster for
> > convenient testing locally, Ke Zhang and Huaxi.
>
>
> That's SWCK's capacity to provide a standard and simple way to create a
> cluster in every environment,
>
> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一 下午7:00写道：
>
> > Thanks Hongtao’s suggestion, I’ve thought about adopting Kubernetes-based
> > tech stack (e.g. minikube or KIND), I personally prefer KIND too, but I
> > have been thinking whether or not it is overkill for testing purposes to
> > introduce Kubernetes things.
> >
> > Since you mentioned this, we are open to discuss this much more deeply:
> >
> > It would be definitely a benefit to involve more “end”s in the so-call
> > “end to end” tests, I’m +1 to leverage SWCK as well.
> >
> > If we are going to use KIND, I hope we can give a script or something
> > similar to install all the needed components and create cluster for
> > convenient testing locally, Ke Zhang and Huaxi.
> >
> >
> > —————————
> > Zhenxu Ke (柯振旭)
> > GitHub @kezhenxu94
> >
> > > On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org> wrote:
> > >
> > > May SWCK help about provision and demo
> > >
> > > 1. SWCK is able to provide a stable and healthy OAP standalone instance
> > or
> > > cluster for e2e. We could compose indicated CR yamls for different
> > > scenarios.
> > > 2. Different demo for different cases is mandatory for e2e and swck, we
> > > could build a group of demo projects for them. With Kubernetes's
> support,
> > > we could
> > > get free of verbose scripts that are written for installation,
> > checking
> > > the state of running applications, collecting logs.
> > >
> > > Ppl might have concerts about the minkube we currently have, which's
> hard
> > > to debug and runs slowly.
> > >
> > > I prefer to kind[1] instead of minkube. From my experience, kind works
> > fine
> > > in an e2e scenario(limited cpu, memory resources), and it is run
> million
> > > times
> > > on my team's e2e environment, which proves the stability and
> > > maintainability of it.
> > >
> > > * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
> > >
> > > regards, Hongtao
> > >
> > > Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
> > >
> > >> Inline
> > >>
> > >> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午11:16写道：
> > >>
> > >>> Hi Jiajing,
> > >>>
> > >>> Thanks for sharing your context and joining the discussion. It’s true
> > >> that
> > >>> we’re actually tackling the same problems in this domain. Some of the
> > >>> issues you mentioned are solved in the current E2E framework (like
> > >> `retry`,
> > >>> `timeout`, etc.), while some others are still there without ideal
> > >> solution.
> > >>> Other discussions, please see my comments inline.
> > >>>
> > >>>> - the first issue is useful in health check steps, just like the
> ones
> > >>>> defined in readiness probe and liveness probe in kubelet.
> > >>>
> > >>> Yes, the reason why we decided to pick docker-compose is that, it has
> > the
> > >>> same functionality with kubelet in terms of dependent services
> startup
> > >>> order and healthiness checking, while it’s more lite-weight and easy
> > for
> > >>> both developers and GitHub Actions environment. Right?
> > >>>
> > >>>> - the second issue is critical for highly-customized assertions, we
> > >>>> can leverage the `text/template` module to provide highly extensive
> > >>>> assertion functions
> > >>>
> > >>> `text/template` is also the first thought that came to my mind when
> > >>> considering customized assertors, customized assertors are for sure
> > >>> critical in SkyWalking E2E tests because we have many kinds of
> > >> assertions,
> > >>> some of which are not foreseen until we actually need them,
> > >>> `AtLeastOneGreaterThanZero` is an example.
> > >>>
> > >>>> - the last but not the least, the lifecycle hooks can be used for
> > >>>> accurate control of test setup and teardown, such as `docker-compose
> > >>>> build, up -d, down`
> > >>>
> > >>> The lifecycle hooks are what I didn’t take into consideration
> because I
> > >>> thought it is quite natural for test framework to provide this
> > features,
> > >>> maybe Ke Zhang and Huaxi can do some research on this part.
> > >>>
> > >>>> Together with the aforementioned ideas, we will have a go-written,
> > >>>> yaml-data driven e2e testing framework which can overcome the
> current
> > >>>> problems that occur,
> > >>>
> > >>> I agree.
> > >>>
> > >>>> in which the most annoying issue is the fragmentation of different
> > >>>> scripts in different submodules, IMO.
> > >>>
> > >>> That’s why I list the last item in the previous email, (Nice to have,
> > >> wrap
> > >>> them into a GitHub Action), with that GitHub Action, we can simply
> > >>> configure a workflow/job in every submodule, and the tests are
> executed
> > >>> with the shared scripts, no copy-pasting.
> > >>>
> > >>
> > >> Note, GitHub Action will require an Apache release every time because
> we
> > >> are distributing the binary.
> > >>
> > >>
> > >> Sheng Wu 吴晟
> > >> Twitter, wusheng1108
> > >>
> > >>
> > >>>
> > >>>>
> > >>>> Regards,
> > >>>> Jiajing
> > >>>>
> > >>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
> > >> kezhenxu94@apache.org>
> > >>> wrote:
> > >>>>>
> > >>>>>> So, we would provide a script set to drive this?
> > >>>>>
> > >>>>> Yes, a script or a GoLang-written CLI, it depends.
> > >>>>>
> > >>>>>> My question is more about,
> > >>>>>> how to work with docker-compose
> > >>>>>
> > >>>>> It’s the same as what we’re doing now in the current version of E2E
> > >>> tests, now we have many `docker-compose.yaml` files, and all the
> > >>> `docker-compose.yaml` files are reusable in the new framework, we
> will
> > >> keep
> > >>> this mechanism as it is proved to be convenient and easy to use
> > (remember
> > >>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E
> test,
> > >> not
> > >>> yet merged though, it’s fast/easy to add a new case).
> > >>>>>
> > >>>>> —————————
> > >>>>> Zhenxu Ke (柯振旭)
> > >>>>> GitHub @kezhenxu94
> > >>>>>
> > >>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <wu...@gmail.com>
> > >>> wrote:
> > >>>>>>
> > >>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> 下午9:16写道：
> > >>>>>>
> > >>>>>>>
> > >>>>>>>> I want to ask, how the docker-compose lands in your mind? I
> think
> > >> CLI
> > >>>>>>> can't
> > >>>>>>>> control this part.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> docker-compose just fits the requirements perfectly that we need,
> > we
> > >>> have
> > >>>>>>> been using it in the current E2E and turns out it has advantages
> in
> > >>>>>>> starting many services correctly with health check and dependency
> > >>>>>>> declaration. (One may say Kubernetes can do the same thing, but
> > it’s
> > >>> kind
> > >>>>>>> of heavy in E2E and GitHub Actions and complex).
> > >>>>>>>
> > >>>>>>> The SkyWalking-CLI won’t get involved with docker-compose related
> > >>> things,
> > >>>>>>> it just provides query ability (the same as what it is doing now,
> > >>> just need
> > >>>>>>> to check whether it covers all the needed query interfaces or
> not),
> > >>> we will
> > >>>>>>> have a dedicated control program to
> > >>>>>>>
> > >>>>>>> (1) start docker-compose (existing standard mechanism, no new
> > >>> knowledge);
> > >>>>>>>
> > >>>>>>
> > >>>>>> So, we would provide a script set to drive this? My question is
> more
> > >>> about,
> > >>>>>> how to work with docker-compose, rather than challanging the
> choice.
> > >>>>>>
> > >>>>>> Sheng Wu 吴晟
> > >>>>>> Twitter, wusheng1108
> > >>>>>>
> > >>>>>>
> > >>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository and
> > >>> codes);
> > >>>>>>> (3) verify the actual data (by verification tool, a new tool,
> > TODO);
> > >>>>>>> (4) and determine the test result (success or failed);
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> —————————
> > >>>>>>> Zhenxu Ke (柯振旭)
> > >>>>>>> GitHub @kezhenxu94
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Sheng Wu 吴晟
> > >>>>>>>> Twitter, wusheng1108
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> > 下午5:27写道：
> > >>>>>>>>
> > >>>>>>>>> Hi the community, I want to raise a discussion to build the
> next
> > >>>>>>>>> generation of E2E test framework for Apache SkyWalking, which
> may
> > >>> not
> > >>>>>>> be a
> > >>>>>>>>> high priority but a necessity.
> > >>>>>>>>>
> > >>>>>>>>> As we already know, tests in SkyWalking play a very important
> > role
> > >>> in
> > >>>>>>> the
> > >>>>>>>>> contribution process, and the current E2E tests just work well
> to
> > >>> verify
> > >>>>>>>>> every single pull request before merging, so why bother to
> build
> > >> the
> > >>>>>>> next
> > >>>>>>>>> generation of E2E test framework?
> > >>>>>>>>>
> > >>>>>>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects
> > that
> > >>> vary
> > >>>>>>> in
> > >>>>>>>>> terms of language, almost all of them need E2E tests to help us
> > >>> verify
> > >>>>>>> the
> > >>>>>>>>> pull requests, but we can see that they reimplement the E2E
> test
> > >>>>>>> framework
> > >>>>>>>>> in their languages, (or even worse, some of them don’t have E2E
> > >>> tests at
> > >>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry and
> > >>>>>>> introduces
> > >>>>>>>>> more duplicated work when adding a new test case. The ultimate
> > >> goal
> > >>> is
> > >>>>>>> to
> > >>>>>>>>> reuse all the test cases when needed.
> > >>>>>>>>>
> > >>>>>>>>> 2. The current E2E framework is built with Java, running
> > >> Java-based
> > >>>>>>>>> program is not an easy thing for non-Java developers
> (maintainers
> > >> of
> > >>>>>>> other
> > >>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up
> > the
> > >>>>>>>>> environment correctly and then run the tests, neither is it an
> > >> easy
> > >>>>>>> skill
> > >>>>>>>>> to debug the tests.
> > >>>>>>>>>
> > >>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests because we
> > >>> added
> > >>>>>>> many
> > >>>>>>>>> cases and many duplicated codes that need to be removed.
> > >>>>>>>>>
> > >>>>>>>>> Therefore we're planning to build an unified, easy-to-use, and
> > >> fast
> > >>> E2E
> > >>>>>>>>> test framework to benifit all the sub-projects.
> > >>>>>>>>>
> > >>>>>>>>> Here are some rough ideas about this framework:
> > >>>>>>>>>
> > >>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to
> > >>>>>>> orchestrate
> > >>>>>>>>> the services that are to be tested, it’s proved to be friendly
> > and
> > >>>>>>>>> debuggable because we can start it directly even if we are not
> > >>> writing
> > >>>>>>> E2E
> > >>>>>>>>> tests.
> > >>>>>>>>> 1. This framework will take full advantages of the CLI project
> to
> > >>> query
> > >>>>>>>>> the traces/metrics/logs, we don’t want to maintain two versions
> > of
> > >>> query
> > >>>>>>>>> codes anymore, (now we have a version in the test codes and one
> > in
> > >>> the
> > >>>>>>> CLI).
> > >>>>>>>>> 2. We will build a CLI tool to compare the expected data and
> the
> > >>> actual
> > >>>>>>>>> data, we now have a custom schema of the expected data yaml
> file,
> > >>> the
> > >>>>>>>>> verification codes should be packaged into an executable
> command
> > >> so
> > >>>>>>> that it
> > >>>>>>>>> can be executed standalone without Java and maven knowledge. I
> > >> hope
> > >>> we
> > >>>>>>> can
> > >>>>>>>>> leverage existing standards to write the YAML files and do
> > >>>>>>> verifications.
> > >>>>>>>>> 3. Build an orchestrating tool to start the
> > `docker-compose.yaml`,
> > >>>>>>> health
> > >>>>>>>>> check, query actual data, and verify the result.
> > >>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to
> share
> > >>>>>>> between
> > >>>>>>>>> sub-projects.
> > >>>>>>>>>
> > >>>>>>>>> If you have any other better idea or want to complement to this
> > >>>>>>> proposal,
> > >>>>>>>>> please reply here. I will create an issue to track these tasks
> > >>> later.
> > >>>>>>>>>
> > >>>>>>>>> We have two students (who just finished the Summer Code 2020
> > >>> Projects)
> > >>>>>>> to
> > >>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said
> this
> > >> is
> > >>> a
> > >>>>>>> not
> > >>>>>>>>> high priority, you have adequate time to get yourselves
> familiar
> > >>> with
> > >>>>>>> how
> > >>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking
> > >> work
> > >>> as
> > >>>>>>> well
> > >>>>>>>>> as the ecosystem. Thanks for your interests and willingness to
> > >> help.
> > >>>>>>>>>
> > >>>>>>>>> [1] https://github.com/Humbertzhang
> > >>>>>>>>> [2] https://github.com/fgksgf/
> > >>>>>>>>>
> > >>>>>>>>> —————————
> > >>>>>>>>> Zhenxu Ke (柯振旭)
> > >>>>>>>>> GitHub @kezhenxu94
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>> —————————
> > >>>>>>> Zhenxu Ke (柯振旭)
> > >>>>>>> GitHub @kezhenxu94
> > >>>>>
> > >>>
> > >>>
> > >>>
> > >>> —————————
> > >>> Zhenxu Ke (柯振旭)
> > >>> GitHub @kezhenxu94
> > >>>
> > >>>
> > >>
> >
> >
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Hongtao Gao <ha...@apache.org>.
>
> but I have been thinking whether or not it is overkill for testing
> purposes to introduce Kubernetes things.
we all know the fact docker-compose is more portable than Kubernetes,
friendly to local running. But there're also some benefits to replace it
with kubernetes system:
1. SWCK has to test based on a real Kubernetes environment if the main
repo test framework doesn't support it(following the docker-compose stack),
That means skywalking ecosystem MUST introduce kubernetes which is not an
optional component.
2. Kubernetes support to scale applications at runtime, that help improve
the process of e2e. For instance, we could merge a single instance and
cluster test, by scaling up replicas. This strategy will reduce the total
time sharply.
If we are going to use KIND, I hope we can give a script or something
> similar to install all the needed components and create cluster for
> convenient testing locally, Ke Zhang and Huaxi.
That's SWCK's capacity to provide a standard and simple way to create a
cluster in every environment,
kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一 下午7:00写道：
> Thanks Hongtao’s suggestion, I’ve thought about adopting Kubernetes-based
> tech stack (e.g. minikube or KIND), I personally prefer KIND too, but I
> have been thinking whether or not it is overkill for testing purposes to
> introduce Kubernetes things.
>
> Since you mentioned this, we are open to discuss this much more deeply:
>
> It would be definitely a benefit to involve more “end”s in the so-call
> “end to end” tests, I’m +1 to leverage SWCK as well.
>
> If we are going to use KIND, I hope we can give a script or something
> similar to install all the needed components and create cluster for
> convenient testing locally, Ke Zhang and Huaxi.
>
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94
>
> > On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org> wrote:
> >
> > May SWCK help about provision and demo
> >
> > 1. SWCK is able to provide a stable and healthy OAP standalone instance
> or
> > cluster for e2e. We could compose indicated CR yamls for different
> > scenarios.
> > 2. Different demo for different cases is mandatory for e2e and swck, we
> > could build a group of demo projects for them. With Kubernetes's support,
> > we could
> > get free of verbose scripts that are written for installation,
> checking
> > the state of running applications, collecting logs.
> >
> > Ppl might have concerts about the minkube we currently have, which's hard
> > to debug and runs slowly.
> >
> > I prefer to kind[1] instead of minkube. From my experience, kind works
> fine
> > in an e2e scenario(limited cpu, memory resources), and it is run million
> > times
> > on my team's e2e environment, which proves the stability and
> > maintainability of it.
> >
> > * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
> >
> > regards, Hongtao
> >
> > Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
> >
> >> Inline
> >>
> >> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午11:16写道：
> >>
> >>> Hi Jiajing,
> >>>
> >>> Thanks for sharing your context and joining the discussion. It’s true
> >> that
> >>> we’re actually tackling the same problems in this domain. Some of the
> >>> issues you mentioned are solved in the current E2E framework (like
> >> `retry`,
> >>> `timeout`, etc.), while some others are still there without ideal
> >> solution.
> >>> Other discussions, please see my comments inline.
> >>>
> >>>> - the first issue is useful in health check steps, just like the ones
> >>>> defined in readiness probe and liveness probe in kubelet.
> >>>
> >>> Yes, the reason why we decided to pick docker-compose is that, it has
> the
> >>> same functionality with kubelet in terms of dependent services startup
> >>> order and healthiness checking, while it’s more lite-weight and easy
> for
> >>> both developers and GitHub Actions environment. Right?
> >>>
> >>>> - the second issue is critical for highly-customized assertions, we
> >>>> can leverage the `text/template` module to provide highly extensive
> >>>> assertion functions
> >>>
> >>> `text/template` is also the first thought that came to my mind when
> >>> considering customized assertors, customized assertors are for sure
> >>> critical in SkyWalking E2E tests because we have many kinds of
> >> assertions,
> >>> some of which are not foreseen until we actually need them,
> >>> `AtLeastOneGreaterThanZero` is an example.
> >>>
> >>>> - the last but not the least, the lifecycle hooks can be used for
> >>>> accurate control of test setup and teardown, such as `docker-compose
> >>>> build, up -d, down`
> >>>
> >>> The lifecycle hooks are what I didn’t take into consideration because I
> >>> thought it is quite natural for test framework to provide this
> features,
> >>> maybe Ke Zhang and Huaxi can do some research on this part.
> >>>
> >>>> Together with the aforementioned ideas, we will have a go-written,
> >>>> yaml-data driven e2e testing framework which can overcome the current
> >>>> problems that occur,
> >>>
> >>> I agree.
> >>>
> >>>> in which the most annoying issue is the fragmentation of different
> >>>> scripts in different submodules, IMO.
> >>>
> >>> That’s why I list the last item in the previous email, (Nice to have,
> >> wrap
> >>> them into a GitHub Action), with that GitHub Action, we can simply
> >>> configure a workflow/job in every submodule, and the tests are executed
> >>> with the shared scripts, no copy-pasting.
> >>>
> >>
> >> Note, GitHub Action will require an Apache release every time because we
> >> are distributing the binary.
> >>
> >>
> >> Sheng Wu 吴晟
> >> Twitter, wusheng1108
> >>
> >>
> >>>
> >>>>
> >>>> Regards,
> >>>> Jiajing
> >>>>
> >>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
> >> kezhenxu94@apache.org>
> >>> wrote:
> >>>>>
> >>>>>> So, we would provide a script set to drive this?
> >>>>>
> >>>>> Yes, a script or a GoLang-written CLI, it depends.
> >>>>>
> >>>>>> My question is more about,
> >>>>>> how to work with docker-compose
> >>>>>
> >>>>> It’s the same as what we’re doing now in the current version of E2E
> >>> tests, now we have many `docker-compose.yaml` files, and all the
> >>> `docker-compose.yaml` files are reusable in the new framework, we will
> >> keep
> >>> this mechanism as it is proved to be convenient and easy to use
> (remember
> >>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test,
> >> not
> >>> yet merged though, it’s fast/easy to add a new case).
> >>>>>
> >>>>> —————————
> >>>>> Zhenxu Ke (柯振旭)
> >>>>> GitHub @kezhenxu94
> >>>>>
> >>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <wu...@gmail.com>
> >>> wrote:
> >>>>>>
> >>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午9:16写道：
> >>>>>>
> >>>>>>>
> >>>>>>>> I want to ask, how the docker-compose lands in your mind? I think
> >> CLI
> >>>>>>> can't
> >>>>>>>> control this part.
> >>>>>>>
> >>>>>>>
> >>>>>>> docker-compose just fits the requirements perfectly that we need,
> we
> >>> have
> >>>>>>> been using it in the current E2E and turns out it has advantages in
> >>>>>>> starting many services correctly with health check and dependency
> >>>>>>> declaration. (One may say Kubernetes can do the same thing, but
> it’s
> >>> kind
> >>>>>>> of heavy in E2E and GitHub Actions and complex).
> >>>>>>>
> >>>>>>> The SkyWalking-CLI won’t get involved with docker-compose related
> >>> things,
> >>>>>>> it just provides query ability (the same as what it is doing now,
> >>> just need
> >>>>>>> to check whether it covers all the needed query interfaces or not),
> >>> we will
> >>>>>>> have a dedicated control program to
> >>>>>>>
> >>>>>>> (1) start docker-compose (existing standard mechanism, no new
> >>> knowledge);
> >>>>>>>
> >>>>>>
> >>>>>> So, we would provide a script set to drive this? My question is more
> >>> about,
> >>>>>> how to work with docker-compose, rather than challanging the choice.
> >>>>>>
> >>>>>> Sheng Wu 吴晟
> >>>>>> Twitter, wusheng1108
> >>>>>>
> >>>>>>
> >>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository and
> >>> codes);
> >>>>>>> (3) verify the actual data (by verification tool, a new tool,
> TODO);
> >>>>>>> (4) and determine the test result (success or failed);
> >>>>>>>
> >>>>>>>
> >>>>>>> —————————
> >>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>> GitHub @kezhenxu94
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Sheng Wu 吴晟
> >>>>>>>> Twitter, wusheng1108
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> 下午5:27写道：
> >>>>>>>>
> >>>>>>>>> Hi the community, I want to raise a discussion to build the next
> >>>>>>>>> generation of E2E test framework for Apache SkyWalking, which may
> >>> not
> >>>>>>> be a
> >>>>>>>>> high priority but a necessity.
> >>>>>>>>>
> >>>>>>>>> As we already know, tests in SkyWalking play a very important
> role
> >>> in
> >>>>>>> the
> >>>>>>>>> contribution process, and the current E2E tests just work well to
> >>> verify
> >>>>>>>>> every single pull request before merging, so why bother to build
> >> the
> >>>>>>> next
> >>>>>>>>> generation of E2E test framework?
> >>>>>>>>>
> >>>>>>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects
> that
> >>> vary
> >>>>>>> in
> >>>>>>>>> terms of language, almost all of them need E2E tests to help us
> >>> verify
> >>>>>>> the
> >>>>>>>>> pull requests, but we can see that they reimplement the E2E test
> >>>>>>> framework
> >>>>>>>>> in their languages, (or even worse, some of them don’t have E2E
> >>> tests at
> >>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry and
> >>>>>>> introduces
> >>>>>>>>> more duplicated work when adding a new test case. The ultimate
> >> goal
> >>> is
> >>>>>>> to
> >>>>>>>>> reuse all the test cases when needed.
> >>>>>>>>>
> >>>>>>>>> 2. The current E2E framework is built with Java, running
> >> Java-based
> >>>>>>>>> program is not an easy thing for non-Java developers (maintainers
> >> of
> >>>>>>> other
> >>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up
> the
> >>>>>>>>> environment correctly and then run the tests, neither is it an
> >> easy
> >>>>>>> skill
> >>>>>>>>> to debug the tests.
> >>>>>>>>>
> >>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests because we
> >>> added
> >>>>>>> many
> >>>>>>>>> cases and many duplicated codes that need to be removed.
> >>>>>>>>>
> >>>>>>>>> Therefore we're planning to build an unified, easy-to-use, and
> >> fast
> >>> E2E
> >>>>>>>>> test framework to benifit all the sub-projects.
> >>>>>>>>>
> >>>>>>>>> Here are some rough ideas about this framework:
> >>>>>>>>>
> >>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to
> >>>>>>> orchestrate
> >>>>>>>>> the services that are to be tested, it’s proved to be friendly
> and
> >>>>>>>>> debuggable because we can start it directly even if we are not
> >>> writing
> >>>>>>> E2E
> >>>>>>>>> tests.
> >>>>>>>>> 1. This framework will take full advantages of the CLI project to
> >>> query
> >>>>>>>>> the traces/metrics/logs, we don’t want to maintain two versions
> of
> >>> query
> >>>>>>>>> codes anymore, (now we have a version in the test codes and one
> in
> >>> the
> >>>>>>> CLI).
> >>>>>>>>> 2. We will build a CLI tool to compare the expected data and the
> >>> actual
> >>>>>>>>> data, we now have a custom schema of the expected data yaml file,
> >>> the
> >>>>>>>>> verification codes should be packaged into an executable command
> >> so
> >>>>>>> that it
> >>>>>>>>> can be executed standalone without Java and maven knowledge. I
> >> hope
> >>> we
> >>>>>>> can
> >>>>>>>>> leverage existing standards to write the YAML files and do
> >>>>>>> verifications.
> >>>>>>>>> 3. Build an orchestrating tool to start the
> `docker-compose.yaml`,
> >>>>>>> health
> >>>>>>>>> check, query actual data, and verify the result.
> >>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share
> >>>>>>> between
> >>>>>>>>> sub-projects.
> >>>>>>>>>
> >>>>>>>>> If you have any other better idea or want to complement to this
> >>>>>>> proposal,
> >>>>>>>>> please reply here. I will create an issue to track these tasks
> >>> later.
> >>>>>>>>>
> >>>>>>>>> We have two students (who just finished the Summer Code 2020
> >>> Projects)
> >>>>>>> to
> >>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this
> >> is
> >>> a
> >>>>>>> not
> >>>>>>>>> high priority, you have adequate time to get yourselves familiar
> >>> with
> >>>>>>> how
> >>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking
> >> work
> >>> as
> >>>>>>> well
> >>>>>>>>> as the ecosystem. Thanks for your interests and willingness to
> >> help.
> >>>>>>>>>
> >>>>>>>>> [1] https://github.com/Humbertzhang
> >>>>>>>>> [2] https://github.com/fgksgf/
> >>>>>>>>>
> >>>>>>>>> —————————
> >>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>> —————————
> >>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>> GitHub @kezhenxu94
> >>>>>
> >>>
> >>>
> >>>
> >>> —————————
> >>> Zhenxu Ke (柯振旭)
> >>> GitHub @kezhenxu94
> >>>
> >>>
> >>
>
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by "kezhenxu94@apache" <ke...@apache.org>.
Thanks Hongtao’s suggestion, I’ve thought about adopting Kubernetes-based tech stack (e.g. minikube or KIND), I personally prefer KIND too, but I have been thinking whether or not it is overkill for testing purposes to introduce Kubernetes things.
Since you mentioned this, we are open to discuss this much more deeply:
It would be definitely a benefit to involve more “end”s in the so-call “end to end” tests, I’m +1 to leverage SWCK as well.
If we are going to use KIND, I hope we can give a script or something similar to install all the needed components and create cluster for convenient testing locally, Ke Zhang and Huaxi.
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org> wrote:
>
> May SWCK help about provision and demo
>
> 1. SWCK is able to provide a stable and healthy OAP standalone instance or
> cluster for e2e. We could compose indicated CR yamls for different
> scenarios.
> 2. Different demo for different cases is mandatory for e2e and swck, we
> could build a group of demo projects for them. With Kubernetes's support,
> we could
> get free of verbose scripts that are written for installation, checking
> the state of running applications, collecting logs.
>
> Ppl might have concerts about the minkube we currently have, which's hard
> to debug and runs slowly.
>
> I prefer to kind[1] instead of minkube. From my experience, kind works fine
> in an e2e scenario(limited cpu, memory resources), and it is run million
> times
> on my team's e2e environment, which proves the stability and
> maintainability of it.
>
> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
>
> regards, Hongtao
>
> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
>
>> Inline
>>
>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午11:16写道：
>>
>>> Hi Jiajing,
>>>
>>> Thanks for sharing your context and joining the discussion. It’s true
>> that
>>> we’re actually tackling the same problems in this domain. Some of the
>>> issues you mentioned are solved in the current E2E framework (like
>> `retry`,
>>> `timeout`, etc.), while some others are still there without ideal
>> solution.
>>> Other discussions, please see my comments inline.
>>>
>>>> - the first issue is useful in health check steps, just like the ones
>>>> defined in readiness probe and liveness probe in kubelet.
>>>
>>> Yes, the reason why we decided to pick docker-compose is that, it has the
>>> same functionality with kubelet in terms of dependent services startup
>>> order and healthiness checking, while it’s more lite-weight and easy for
>>> both developers and GitHub Actions environment. Right?
>>>
>>>> - the second issue is critical for highly-customized assertions, we
>>>> can leverage the `text/template` module to provide highly extensive
>>>> assertion functions
>>>
>>> `text/template` is also the first thought that came to my mind when
>>> considering customized assertors, customized assertors are for sure
>>> critical in SkyWalking E2E tests because we have many kinds of
>> assertions,
>>> some of which are not foreseen until we actually need them,
>>> `AtLeastOneGreaterThanZero` is an example.
>>>
>>>> - the last but not the least, the lifecycle hooks can be used for
>>>> accurate control of test setup and teardown, such as `docker-compose
>>>> build, up -d, down`
>>>
>>> The lifecycle hooks are what I didn’t take into consideration because I
>>> thought it is quite natural for test framework to provide this features,
>>> maybe Ke Zhang and Huaxi can do some research on this part.
>>>
>>>> Together with the aforementioned ideas, we will have a go-written,
>>>> yaml-data driven e2e testing framework which can overcome the current
>>>> problems that occur,
>>>
>>> I agree.
>>>
>>>> in which the most annoying issue is the fragmentation of different
>>>> scripts in different submodules, IMO.
>>>
>>> That’s why I list the last item in the previous email, (Nice to have,
>> wrap
>>> them into a GitHub Action), with that GitHub Action, we can simply
>>> configure a workflow/job in every submodule, and the tests are executed
>>> with the shared scripts, no copy-pasting.
>>>
>>
>> Note, GitHub Action will require an Apache release every time because we
>> are distributing the binary.
>>
>>
>> Sheng Wu 吴晟
>> Twitter, wusheng1108
>>
>>
>>>
>>>>
>>>> Regards,
>>>> Jiajing
>>>>
>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
>> kezhenxu94@apache.org>
>>> wrote:
>>>>>
>>>>>> So, we would provide a script set to drive this?
>>>>>
>>>>> Yes, a script or a GoLang-written CLI, it depends.
>>>>>
>>>>>> My question is more about,
>>>>>> how to work with docker-compose
>>>>>
>>>>> It’s the same as what we’re doing now in the current version of E2E
>>> tests, now we have many `docker-compose.yaml` files, and all the
>>> `docker-compose.yaml` files are reusable in the new framework, we will
>> keep
>>> this mechanism as it is proved to be convenient and easy to use (remember
>>> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test,
>> not
>>> yet merged though, it’s fast/easy to add a new case).
>>>>>
>>>>> —————————
>>>>> Zhenxu Ke (柯振旭)
>>>>> GitHub @kezhenxu94
>>>>>
>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <wu...@gmail.com>
>>> wrote:
>>>>>>
>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午9:16写道：
>>>>>>
>>>>>>>
>>>>>>>> I want to ask, how the docker-compose lands in your mind? I think
>> CLI
>>>>>>> can't
>>>>>>>> control this part.
>>>>>>>
>>>>>>>
>>>>>>> docker-compose just fits the requirements perfectly that we need, we
>>> have
>>>>>>> been using it in the current E2E and turns out it has advantages in
>>>>>>> starting many services correctly with health check and dependency
>>>>>>> declaration. (One may say Kubernetes can do the same thing, but it’s
>>> kind
>>>>>>> of heavy in E2E and GitHub Actions and complex).
>>>>>>>
>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose related
>>> things,
>>>>>>> it just provides query ability (the same as what it is doing now,
>>> just need
>>>>>>> to check whether it covers all the needed query interfaces or not),
>>> we will
>>>>>>> have a dedicated control program to
>>>>>>>
>>>>>>> (1) start docker-compose (existing standard mechanism, no new
>>> knowledge);
>>>>>>>
>>>>>>
>>>>>> So, we would provide a script set to drive this? My question is more
>>> about,
>>>>>> how to work with docker-compose, rather than challanging the choice.
>>>>>>
>>>>>> Sheng Wu 吴晟
>>>>>> Twitter, wusheng1108
>>>>>>
>>>>>>
>>>>>>> (2) query actual data (by SkyWalking-CLI, existing repository and
>>> codes);
>>>>>>> (3) verify the actual data (by verification tool, a new tool, TODO);
>>>>>>> (4) and determine the test result (success or failed);
>>>>>>>
>>>>>>>
>>>>>>> —————————
>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>> GitHub @kezhenxu94
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Sheng Wu 吴晟
>>>>>>>> Twitter, wusheng1108
>>>>>>>>
>>>>>>>>
>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午5:27写道：
>>>>>>>>
>>>>>>>>> Hi the community, I want to raise a discussion to build the next
>>>>>>>>> generation of E2E test framework for Apache SkyWalking, which may
>>> not
>>>>>>> be a
>>>>>>>>> high priority but a necessity.
>>>>>>>>>
>>>>>>>>> As we already know, tests in SkyWalking play a very important role
>>> in
>>>>>>> the
>>>>>>>>> contribution process, and the current E2E tests just work well to
>>> verify
>>>>>>>>> every single pull request before merging, so why bother to build
>> the
>>>>>>> next
>>>>>>>>> generation of E2E test framework?
>>>>>>>>>
>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects that
>>> vary
>>>>>>> in
>>>>>>>>> terms of language, almost all of them need E2E tests to help us
>>> verify
>>>>>>> the
>>>>>>>>> pull requests, but we can see that they reimplement the E2E test
>>>>>>> framework
>>>>>>>>> in their languages, (or even worse, some of them don’t have E2E
>>> tests at
>>>>>>>>> all). Reimplementing the E2E test framework is unnecesarry and
>>>>>>> introduces
>>>>>>>>> more duplicated work when adding a new test case. The ultimate
>> goal
>>> is
>>>>>>> to
>>>>>>>>> reuse all the test cases when needed.
>>>>>>>>>
>>>>>>>>> 2. The current E2E framework is built with Java, running
>> Java-based
>>>>>>>>> program is not an easy thing for non-Java developers (maintainers
>> of
>>>>>>> other
>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the
>>>>>>>>> environment correctly and then run the tests, neither is it an
>> easy
>>>>>>> skill
>>>>>>>>> to debug the tests.
>>>>>>>>>
>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests because we
>>> added
>>>>>>> many
>>>>>>>>> cases and many duplicated codes that need to be removed.
>>>>>>>>>
>>>>>>>>> Therefore we're planning to build an unified, easy-to-use, and
>> fast
>>> E2E
>>>>>>>>> test framework to benifit all the sub-projects.
>>>>>>>>>
>>>>>>>>> Here are some rough ideas about this framework:
>>>>>>>>>
>>>>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to
>>>>>>> orchestrate
>>>>>>>>> the services that are to be tested, it’s proved to be friendly and
>>>>>>>>> debuggable because we can start it directly even if we are not
>>> writing
>>>>>>> E2E
>>>>>>>>> tests.
>>>>>>>>> 1. This framework will take full advantages of the CLI project to
>>> query
>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two versions of
>>> query
>>>>>>>>> codes anymore, (now we have a version in the test codes and one in
>>> the
>>>>>>> CLI).
>>>>>>>>> 2. We will build a CLI tool to compare the expected data and the
>>> actual
>>>>>>>>> data, we now have a custom schema of the expected data yaml file,
>>> the
>>>>>>>>> verification codes should be packaged into an executable command
>> so
>>>>>>> that it
>>>>>>>>> can be executed standalone without Java and maven knowledge. I
>> hope
>>> we
>>>>>>> can
>>>>>>>>> leverage existing standards to write the YAML files and do
>>>>>>> verifications.
>>>>>>>>> 3. Build an orchestrating tool to start the `docker-compose.yaml`,
>>>>>>> health
>>>>>>>>> check, query actual data, and verify the result.
>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share
>>>>>>> between
>>>>>>>>> sub-projects.
>>>>>>>>>
>>>>>>>>> If you have any other better idea or want to complement to this
>>>>>>> proposal,
>>>>>>>>> please reply here. I will create an issue to track these tasks
>>> later.
>>>>>>>>>
>>>>>>>>> We have two students (who just finished the Summer Code 2020
>>> Projects)
>>>>>>> to
>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this
>> is
>>> a
>>>>>>> not
>>>>>>>>> high priority, you have adequate time to get yourselves familiar
>>> with
>>>>>>> how
>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking
>> work
>>> as
>>>>>>> well
>>>>>>>>> as the ecosystem. Thanks for your interests and willingness to
>> help.
>>>>>>>>>
>>>>>>>>> [1] https://github.com/Humbertzhang
>>>>>>>>> [2] https://github.com/fgksgf/
>>>>>>>>>
>>>>>>>>> —————————
>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> —————————
>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>> GitHub @kezhenxu94
>>>>>
>>>
>>>
>>>
>>> —————————
>>> Zhenxu Ke (柯振旭)
>>> GitHub @kezhenxu94
>>>
>>>
>>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Hongtao Gao <ha...@apache.org>.
May SWCK help about provision and demo
1. SWCK is able to provide a stable and healthy OAP standalone instance or
cluster for e2e. We could compose indicated CR yamls for different
scenarios.
2. Different demo for different cases is mandatory for e2e and swck, we
could build a group of demo projects for them. With Kubernetes's support,
we could
get free of verbose scripts that are written for installation, checking
the state of running applications, collecting logs.
Ppl might have concerts about the minkube we currently have, which's hard
to debug and runs slowly.
I prefer to kind[1] instead of minkube. From my experience, kind works fine
in an e2e scenario(limited cpu, memory resources), and it is run million
times
on my team's e2e environment, which proves the stability and
maintainability of it.
* 1.https://kind.sigs.k8s.io/docs/user/quick-start/
regards, Hongtao
Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
> Inline
>
> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午11:16写道：
>
> > Hi Jiajing,
> >
> > Thanks for sharing your context and joining the discussion. It’s true
> that
> > we’re actually tackling the same problems in this domain. Some of the
> > issues you mentioned are solved in the current E2E framework (like
> `retry`,
> > `timeout`, etc.), while some others are still there without ideal
> solution.
> > Other discussions, please see my comments inline.
> >
> > > - the first issue is useful in health check steps, just like the ones
> > > defined in readiness probe and liveness probe in kubelet.
> >
> > Yes, the reason why we decided to pick docker-compose is that, it has the
> > same functionality with kubelet in terms of dependent services startup
> > order and healthiness checking, while it’s more lite-weight and easy for
> > both developers and GitHub Actions environment. Right?
> >
> > > - the second issue is critical for highly-customized assertions, we
> > > can leverage the `text/template` module to provide highly extensive
> > > assertion functions
> >
> > `text/template` is also the first thought that came to my mind when
> > considering customized assertors, customized assertors are for sure
> > critical in SkyWalking E2E tests because we have many kinds of
> assertions,
> > some of which are not foreseen until we actually need them,
> > `AtLeastOneGreaterThanZero` is an example.
> >
> > > - the last but not the least, the lifecycle hooks can be used for
> > > accurate control of test setup and teardown, such as `docker-compose
> > > build, up -d, down`
> >
> > The lifecycle hooks are what I didn’t take into consideration because I
> > thought it is quite natural for test framework to provide this features,
> > maybe Ke Zhang and Huaxi can do some research on this part.
> >
> > > Together with the aforementioned ideas, we will have a go-written,
> > > yaml-data driven e2e testing framework which can overcome the current
> > > problems that occur,
> >
> > I agree.
> >
> > > in which the most annoying issue is the fragmentation of different
> > > scripts in different submodules, IMO.
> >
> > That’s why I list the last item in the previous email, (Nice to have,
> wrap
> > them into a GitHub Action), with that GitHub Action, we can simply
> > configure a workflow/job in every submodule, and the tests are executed
> > with the shared scripts, no copy-pasting.
> >
>
> Note, GitHub Action will require an Apache release every time because we
> are distributing the binary.
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> >
> > >
> > > Regards,
> > > Jiajing
> > >
> > > On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
> kezhenxu94@apache.org>
> > wrote:
> > >>
> > >>> So, we would provide a script set to drive this?
> > >>
> > >> Yes, a script or a GoLang-written CLI, it depends.
> > >>
> > >>> My question is more about,
> > >>> how to work with docker-compose
> > >>
> > >> It’s the same as what we’re doing now in the current version of E2E
> > tests, now we have many `docker-compose.yaml` files, and all the
> > `docker-compose.yaml` files are reusable in the new framework, we will
> keep
> > this mechanism as it is proved to be convenient and easy to use (remember
> > when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test,
> not
> > yet merged though, it’s fast/easy to add a new case).
> > >>
> > >> —————————
> > >> Zhenxu Ke (柯振旭)
> > >> GitHub @kezhenxu94
> > >>
> > >>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <wu...@gmail.com>
> > wrote:
> > >>>
> > >>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午9:16写道：
> > >>>
> > >>>>
> > >>>>> I want to ask, how the docker-compose lands in your mind? I think
> CLI
> > >>>> can't
> > >>>>> control this part.
> > >>>>
> > >>>>
> > >>>> docker-compose just fits the requirements perfectly that we need, we
> > have
> > >>>> been using it in the current E2E and turns out it has advantages in
> > >>>> starting many services correctly with health check and dependency
> > >>>> declaration. (One may say Kubernetes can do the same thing, but it’s
> > kind
> > >>>> of heavy in E2E and GitHub Actions and complex).
> > >>>>
> > >>>> The SkyWalking-CLI won’t get involved with docker-compose related
> > things,
> > >>>> it just provides query ability (the same as what it is doing now,
> > just need
> > >>>> to check whether it covers all the needed query interfaces or not),
> > we will
> > >>>> have a dedicated control program to
> > >>>>
> > >>>> (1) start docker-compose (existing standard mechanism, no new
> > knowledge);
> > >>>>
> > >>>
> > >>> So, we would provide a script set to drive this? My question is more
> > about,
> > >>> how to work with docker-compose, rather than challanging the choice.
> > >>>
> > >>> Sheng Wu 吴晟
> > >>> Twitter, wusheng1108
> > >>>
> > >>>
> > >>>> (2) query actual data (by SkyWalking-CLI, existing repository and
> > codes);
> > >>>> (3) verify the actual data (by verification tool, a new tool, TODO);
> > >>>> (4) and determine the test result (success or failed);
> > >>>>
> > >>>>
> > >>>> —————————
> > >>>> Zhenxu Ke (柯振旭)
> > >>>> GitHub @kezhenxu94
> > >>>>
> > >>>>
> > >>>>
> > >>>>> Sheng Wu 吴晟
> > >>>>> Twitter, wusheng1108
> > >>>>>
> > >>>>>
> > >>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午5:27写道：
> > >>>>>
> > >>>>>> Hi the community, I want to raise a discussion to build the next
> > >>>>>> generation of E2E test framework for Apache SkyWalking, which may
> > not
> > >>>> be a
> > >>>>>> high priority but a necessity.
> > >>>>>>
> > >>>>>> As we already know, tests in SkyWalking play a very important role
> > in
> > >>>> the
> > >>>>>> contribution process, and the current E2E tests just work well to
> > verify
> > >>>>>> every single pull request before merging, so why bother to build
> the
> > >>>> next
> > >>>>>> generation of E2E test framework?
> > >>>>>>
> > >>>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects that
> > vary
> > >>>> in
> > >>>>>> terms of language, almost all of them need E2E tests to help us
> > verify
> > >>>> the
> > >>>>>> pull requests, but we can see that they reimplement the E2E test
> > >>>> framework
> > >>>>>> in their languages, (or even worse, some of them don’t have E2E
> > tests at
> > >>>>>> all). Reimplementing the E2E test framework is unnecesarry and
> > >>>> introduces
> > >>>>>> more duplicated work when adding a new test case. The ultimate
> goal
> > is
> > >>>> to
> > >>>>>> reuse all the test cases when needed.
> > >>>>>>
> > >>>>>> 2. The current E2E framework is built with Java, running
> Java-based
> > >>>>>> program is not an easy thing for non-Java developers (maintainers
> of
> > >>>> other
> > >>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the
> > >>>>>> environment correctly and then run the tests, neither is it an
> easy
> > >>>> skill
> > >>>>>> to debug the tests.
> > >>>>>>
> > >>>>>> 3. It’s a good opportunity to optimize the E2E tests because we
> > added
> > >>>> many
> > >>>>>> cases and many duplicated codes that need to be removed.
> > >>>>>>
> > >>>>>> Therefore we're planning to build an unified, easy-to-use, and
> fast
> > E2E
> > >>>>>> test framework to benifit all the sub-projects.
> > >>>>>>
> > >>>>>> Here are some rough ideas about this framework:
> > >>>>>>
> > >>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to
> > >>>> orchestrate
> > >>>>>> the services that are to be tested, it’s proved to be friendly and
> > >>>>>> debuggable because we can start it directly even if we are not
> > writing
> > >>>> E2E
> > >>>>>> tests.
> > >>>>>> 1. This framework will take full advantages of the CLI project to
> > query
> > >>>>>> the traces/metrics/logs, we don’t want to maintain two versions of
> > query
> > >>>>>> codes anymore, (now we have a version in the test codes and one in
> > the
> > >>>> CLI).
> > >>>>>> 2. We will build a CLI tool to compare the expected data and the
> > actual
> > >>>>>> data, we now have a custom schema of the expected data yaml file,
> > the
> > >>>>>> verification codes should be packaged into an executable command
> so
> > >>>> that it
> > >>>>>> can be executed standalone without Java and maven knowledge. I
> hope
> > we
> > >>>> can
> > >>>>>> leverage existing standards to write the YAML files and do
> > >>>> verifications.
> > >>>>>> 3. Build an orchestrating tool to start the `docker-compose.yaml`,
> > >>>> health
> > >>>>>> check, query actual data, and verify the result.
> > >>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share
> > >>>> between
> > >>>>>> sub-projects.
> > >>>>>>
> > >>>>>> If you have any other better idea or want to complement to this
> > >>>> proposal,
> > >>>>>> please reply here. I will create an issue to track these tasks
> > later.
> > >>>>>>
> > >>>>>> We have two students (who just finished the Summer Code 2020
> > Projects)
> > >>>> to
> > >>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this
> is
> > a
> > >>>> not
> > >>>>>> high priority, you have adequate time to get yourselves familiar
> > with
> > >>>> how
> > >>>>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking
> work
> > as
> > >>>> well
> > >>>>>> as the ecosystem. Thanks for your interests and willingness to
> help.
> > >>>>>>
> > >>>>>> [1] https://github.com/Humbertzhang
> > >>>>>> [2] https://github.com/fgksgf/
> > >>>>>>
> > >>>>>> —————————
> > >>>>>> Zhenxu Ke (柯振旭)
> > >>>>>> GitHub @kezhenxu94
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>> —————————
> > >>>> Zhenxu Ke (柯振旭)
> > >>>> GitHub @kezhenxu94
> > >>
> >
> >
> >
> > —————————
> > Zhenxu Ke (柯振旭)
> > GitHub @kezhenxu94
> >
> >
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Sheng Wu <wu...@gmail.com>.
Inline
kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午11:16写道：
> Hi Jiajing,
>
> Thanks for sharing your context and joining the discussion. It’s true that
> we’re actually tackling the same problems in this domain. Some of the
> issues you mentioned are solved in the current E2E framework (like `retry`,
> `timeout`, etc.), while some others are still there without ideal solution.
> Other discussions, please see my comments inline.
>
> > - the first issue is useful in health check steps, just like the ones
> > defined in readiness probe and liveness probe in kubelet.
>
> Yes, the reason why we decided to pick docker-compose is that, it has the
> same functionality with kubelet in terms of dependent services startup
> order and healthiness checking, while it’s more lite-weight and easy for
> both developers and GitHub Actions environment. Right?
>
> > - the second issue is critical for highly-customized assertions, we
> > can leverage the `text/template` module to provide highly extensive
> > assertion functions
>
> `text/template` is also the first thought that came to my mind when
> considering customized assertors, customized assertors are for sure
> critical in SkyWalking E2E tests because we have many kinds of assertions,
> some of which are not foreseen until we actually need them,
> `AtLeastOneGreaterThanZero` is an example.
>
> > - the last but not the least, the lifecycle hooks can be used for
> > accurate control of test setup and teardown, such as `docker-compose
> > build, up -d, down`
>
> The lifecycle hooks are what I didn’t take into consideration because I
> thought it is quite natural for test framework to provide this features,
> maybe Ke Zhang and Huaxi can do some research on this part.
>
> > Together with the aforementioned ideas, we will have a go-written,
> > yaml-data driven e2e testing framework which can overcome the current
> > problems that occur,
>
> I agree.
>
> > in which the most annoying issue is the fragmentation of different
> > scripts in different submodules, IMO.
>
> That’s why I list the last item in the previous email, (Nice to have, wrap
> them into a GitHub Action), with that GitHub Action, we can simply
> configure a workflow/job in every submodule, and the tests are executed
> with the shared scripts, no copy-pasting.
>
Note, GitHub Action will require an Apache release every time because we
are distributing the binary.
Sheng Wu 吴晟
Twitter, wusheng1108
>
> >
> > Regards,
> > Jiajing
> >
> > On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <ke...@apache.org>
> wrote:
> >>
> >>> So, we would provide a script set to drive this?
> >>
> >> Yes, a script or a GoLang-written CLI, it depends.
> >>
> >>> My question is more about,
> >>> how to work with docker-compose
> >>
> >> It’s the same as what we’re doing now in the current version of E2E
> tests, now we have many `docker-compose.yaml` files, and all the
> `docker-compose.yaml` files are reusable in the new framework, we will keep
> this mechanism as it is proved to be convenient and easy to use (remember
> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test, not
> yet merged though, it’s fast/easy to add a new case).
> >>
> >> —————————
> >> Zhenxu Ke (柯振旭)
> >> GitHub @kezhenxu94
> >>
> >>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <wu...@gmail.com>
> wrote:
> >>>
> >>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午9:16写道：
> >>>
> >>>>
> >>>>> I want to ask, how the docker-compose lands in your mind? I think CLI
> >>>> can't
> >>>>> control this part.
> >>>>
> >>>>
> >>>> docker-compose just fits the requirements perfectly that we need, we
> have
> >>>> been using it in the current E2E and turns out it has advantages in
> >>>> starting many services correctly with health check and dependency
> >>>> declaration. (One may say Kubernetes can do the same thing, but it’s
> kind
> >>>> of heavy in E2E and GitHub Actions and complex).
> >>>>
> >>>> The SkyWalking-CLI won’t get involved with docker-compose related
> things,
> >>>> it just provides query ability (the same as what it is doing now,
> just need
> >>>> to check whether it covers all the needed query interfaces or not),
> we will
> >>>> have a dedicated control program to
> >>>>
> >>>> (1) start docker-compose (existing standard mechanism, no new
> knowledge);
> >>>>
> >>>
> >>> So, we would provide a script set to drive this? My question is more
> about,
> >>> how to work with docker-compose, rather than challanging the choice.
> >>>
> >>> Sheng Wu 吴晟
> >>> Twitter, wusheng1108
> >>>
> >>>
> >>>> (2) query actual data (by SkyWalking-CLI, existing repository and
> codes);
> >>>> (3) verify the actual data (by verification tool, a new tool, TODO);
> >>>> (4) and determine the test result (success or failed);
> >>>>
> >>>>
> >>>> —————————
> >>>> Zhenxu Ke (柯振旭)
> >>>> GitHub @kezhenxu94
> >>>>
> >>>>
> >>>>
> >>>>> Sheng Wu 吴晟
> >>>>> Twitter, wusheng1108
> >>>>>
> >>>>>
> >>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午5:27写道：
> >>>>>
> >>>>>> Hi the community, I want to raise a discussion to build the next
> >>>>>> generation of E2E test framework for Apache SkyWalking, which may
> not
> >>>> be a
> >>>>>> high priority but a necessity.
> >>>>>>
> >>>>>> As we already know, tests in SkyWalking play a very important role
> in
> >>>> the
> >>>>>> contribution process, and the current E2E tests just work well to
> verify
> >>>>>> every single pull request before merging, so why bother to build the
> >>>> next
> >>>>>> generation of E2E test framework?
> >>>>>>
> >>>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects that
> vary
> >>>> in
> >>>>>> terms of language, almost all of them need E2E tests to help us
> verify
> >>>> the
> >>>>>> pull requests, but we can see that they reimplement the E2E test
> >>>> framework
> >>>>>> in their languages, (or even worse, some of them don’t have E2E
> tests at
> >>>>>> all). Reimplementing the E2E test framework is unnecesarry and
> >>>> introduces
> >>>>>> more duplicated work when adding a new test case. The ultimate goal
> is
> >>>> to
> >>>>>> reuse all the test cases when needed.
> >>>>>>
> >>>>>> 2. The current E2E framework is built with Java, running Java-based
> >>>>>> program is not an easy thing for non-Java developers (maintainers of
> >>>> other
> >>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the
> >>>>>> environment correctly and then run the tests, neither is it an easy
> >>>> skill
> >>>>>> to debug the tests.
> >>>>>>
> >>>>>> 3. It’s a good opportunity to optimize the E2E tests because we
> added
> >>>> many
> >>>>>> cases and many duplicated codes that need to be removed.
> >>>>>>
> >>>>>> Therefore we're planning to build an unified, easy-to-use, and fast
> E2E
> >>>>>> test framework to benifit all the sub-projects.
> >>>>>>
> >>>>>> Here are some rough ideas about this framework:
> >>>>>>
> >>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to
> >>>> orchestrate
> >>>>>> the services that are to be tested, it’s proved to be friendly and
> >>>>>> debuggable because we can start it directly even if we are not
> writing
> >>>> E2E
> >>>>>> tests.
> >>>>>> 1. This framework will take full advantages of the CLI project to
> query
> >>>>>> the traces/metrics/logs, we don’t want to maintain two versions of
> query
> >>>>>> codes anymore, (now we have a version in the test codes and one in
> the
> >>>> CLI).
> >>>>>> 2. We will build a CLI tool to compare the expected data and the
> actual
> >>>>>> data, we now have a custom schema of the expected data yaml file,
> the
> >>>>>> verification codes should be packaged into an executable command so
> >>>> that it
> >>>>>> can be executed standalone without Java and maven knowledge. I hope
> we
> >>>> can
> >>>>>> leverage existing standards to write the YAML files and do
> >>>> verifications.
> >>>>>> 3. Build an orchestrating tool to start the `docker-compose.yaml`,
> >>>> health
> >>>>>> check, query actual data, and verify the result.
> >>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share
> >>>> between
> >>>>>> sub-projects.
> >>>>>>
> >>>>>> If you have any other better idea or want to complement to this
> >>>> proposal,
> >>>>>> please reply here. I will create an issue to track these tasks
> later.
> >>>>>>
> >>>>>> We have two students (who just finished the Summer Code 2020
> Projects)
> >>>> to
> >>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is
> a
> >>>> not
> >>>>>> high priority, you have adequate time to get yourselves familiar
> with
> >>>> how
> >>>>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work
> as
> >>>> well
> >>>>>> as the ecosystem. Thanks for your interests and willingness to help.
> >>>>>>
> >>>>>> [1] https://github.com/Humbertzhang
> >>>>>> [2] https://github.com/fgksgf/
> >>>>>>
> >>>>>> —————————
> >>>>>> Zhenxu Ke (柯振旭)
> >>>>>> GitHub @kezhenxu94
> >>>>>>
> >>>>>>
> >>>>
> >>>> —————————
> >>>> Zhenxu Ke (柯振旭)
> >>>> GitHub @kezhenxu94
> >>
>
>
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94
>
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by "kezhenxu94@apache" <ke...@apache.org>.
Hi Jiajing,
Thanks for sharing your context and joining the discussion. It’s true that we’re actually tackling the same problems in this domain. Some of the issues you mentioned are solved in the current E2E framework (like `retry`, `timeout`, etc.), while some others are still there without ideal solution. Other discussions, please see my comments inline.
> - the first issue is useful in health check steps, just like the ones
> defined in readiness probe and liveness probe in kubelet.
Yes, the reason why we decided to pick docker-compose is that, it has the same functionality with kubelet in terms of dependent services startup order and healthiness checking, while it’s more lite-weight and easy for both developers and GitHub Actions environment. Right?
> - the second issue is critical for highly-customized assertions, we
> can leverage the `text/template` module to provide highly extensive
> assertion functions
`text/template` is also the first thought that came to my mind when considering customized assertors, customized assertors are for sure critical in SkyWalking E2E tests because we have many kinds of assertions, some of which are not foreseen until we actually need them, `AtLeastOneGreaterThanZero` is an example.
> - the last but not the least, the lifecycle hooks can be used for
> accurate control of test setup and teardown, such as `docker-compose
> build, up -d, down`
The lifecycle hooks are what I didn’t take into consideration because I thought it is quite natural for test framework to provide this features, maybe Ke Zhang and Huaxi can do some research on this part.
> Together with the aforementioned ideas, we will have a go-written,
> yaml-data driven e2e testing framework which can overcome the current
> problems that occur,
I agree.
> in which the most annoying issue is the fragmentation of different
> scripts in different submodules, IMO.
That’s why I list the last item in the previous email, (Nice to have, wrap them into a GitHub Action), with that GitHub Action, we can simply configure a workflow/job in every submodule, and the tests are executed with the shared scripts, no copy-pasting.
>
> Regards,
> Jiajing
>
> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <ke...@apache.org> wrote:
>>
>>> So, we would provide a script set to drive this?
>>
>> Yes, a script or a GoLang-written CLI, it depends.
>>
>>> My question is more about,
>>> how to work with docker-compose
>>
>> It’s the same as what we’re doing now in the current version of E2E tests, now we have many `docker-compose.yaml` files, and all the `docker-compose.yaml` files are reusable in the new framework, we will keep this mechanism as it is proved to be convenient and easy to use (remember when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test, not yet merged though, it’s fast/easy to add a new case).
>>
>> —————————
>> Zhenxu Ke (柯振旭)
>> GitHub @kezhenxu94
>>
>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <wu...@gmail.com> wrote:
>>>
>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午9:16写道：
>>>
>>>>
>>>>> I want to ask, how the docker-compose lands in your mind? I think CLI
>>>> can't
>>>>> control this part.
>>>>
>>>>
>>>> docker-compose just fits the requirements perfectly that we need, we have
>>>> been using it in the current E2E and turns out it has advantages in
>>>> starting many services correctly with health check and dependency
>>>> declaration. (One may say Kubernetes can do the same thing, but it’s kind
>>>> of heavy in E2E and GitHub Actions and complex).
>>>>
>>>> The SkyWalking-CLI won’t get involved with docker-compose related things,
>>>> it just provides query ability (the same as what it is doing now, just need
>>>> to check whether it covers all the needed query interfaces or not), we will
>>>> have a dedicated control program to
>>>>
>>>> (1) start docker-compose (existing standard mechanism, no new knowledge);
>>>>
>>>
>>> So, we would provide a script set to drive this? My question is more about,
>>> how to work with docker-compose, rather than challanging the choice.
>>>
>>> Sheng Wu 吴晟
>>> Twitter, wusheng1108
>>>
>>>
>>>> (2) query actual data (by SkyWalking-CLI, existing repository and codes);
>>>> (3) verify the actual data (by verification tool, a new tool, TODO);
>>>> (4) and determine the test result (success or failed);
>>>>
>>>>
>>>> —————————
>>>> Zhenxu Ke (柯振旭)
>>>> GitHub @kezhenxu94
>>>>
>>>>
>>>>
>>>>> Sheng Wu 吴晟
>>>>> Twitter, wusheng1108
>>>>>
>>>>>
>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午5:27写道：
>>>>>
>>>>>> Hi the community, I want to raise a discussion to build the next
>>>>>> generation of E2E test framework for Apache SkyWalking, which may not
>>>> be a
>>>>>> high priority but a necessity.
>>>>>>
>>>>>> As we already know, tests in SkyWalking play a very important role in
>>>> the
>>>>>> contribution process, and the current E2E tests just work well to verify
>>>>>> every single pull request before merging, so why bother to build the
>>>> next
>>>>>> generation of E2E test framework?
>>>>>>
>>>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects that vary
>>>> in
>>>>>> terms of language, almost all of them need E2E tests to help us verify
>>>> the
>>>>>> pull requests, but we can see that they reimplement the E2E test
>>>> framework
>>>>>> in their languages, (or even worse, some of them don’t have E2E tests at
>>>>>> all). Reimplementing the E2E test framework is unnecesarry and
>>>> introduces
>>>>>> more duplicated work when adding a new test case. The ultimate goal is
>>>> to
>>>>>> reuse all the test cases when needed.
>>>>>>
>>>>>> 2. The current E2E framework is built with Java, running Java-based
>>>>>> program is not an easy thing for non-Java developers (maintainers of
>>>> other
>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the
>>>>>> environment correctly and then run the tests, neither is it an easy
>>>> skill
>>>>>> to debug the tests.
>>>>>>
>>>>>> 3. It’s a good opportunity to optimize the E2E tests because we added
>>>> many
>>>>>> cases and many duplicated codes that need to be removed.
>>>>>>
>>>>>> Therefore we're planning to build an unified, easy-to-use, and fast E2E
>>>>>> test framework to benifit all the sub-projects.
>>>>>>
>>>>>> Here are some rough ideas about this framework:
>>>>>>
>>>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to
>>>> orchestrate
>>>>>> the services that are to be tested, it’s proved to be friendly and
>>>>>> debuggable because we can start it directly even if we are not writing
>>>> E2E
>>>>>> tests.
>>>>>> 1. This framework will take full advantages of the CLI project to query
>>>>>> the traces/metrics/logs, we don’t want to maintain two versions of query
>>>>>> codes anymore, (now we have a version in the test codes and one in the
>>>> CLI).
>>>>>> 2. We will build a CLI tool to compare the expected data and the actual
>>>>>> data, we now have a custom schema of the expected data yaml file, the
>>>>>> verification codes should be packaged into an executable command so
>>>> that it
>>>>>> can be executed standalone without Java and maven knowledge. I hope we
>>>> can
>>>>>> leverage existing standards to write the YAML files and do
>>>> verifications.
>>>>>> 3. Build an orchestrating tool to start the `docker-compose.yaml`,
>>>> health
>>>>>> check, query actual data, and verify the result.
>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share
>>>> between
>>>>>> sub-projects.
>>>>>>
>>>>>> If you have any other better idea or want to complement to this
>>>> proposal,
>>>>>> please reply here. I will create an issue to track these tasks later.
>>>>>>
>>>>>> We have two students (who just finished the Summer Code 2020 Projects)
>>>> to
>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is a
>>>> not
>>>>>> high priority, you have adequate time to get yourselves familiar with
>>>> how
>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work as
>>>> well
>>>>>> as the ecosystem. Thanks for your interests and willingness to help.
>>>>>>
>>>>>> [1] https://github.com/Humbertzhang
>>>>>> [2] https://github.com/fgksgf/
>>>>>>
>>>>>> —————————
>>>>>> Zhenxu Ke (柯振旭)
>>>>>> GitHub @kezhenxu94
>>>>>>
>>>>>>
>>>>
>>>> —————————
>>>> Zhenxu Ke (柯振旭)
>>>> GitHub @kezhenxu94
>>
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Sheng Wu <wu...@gmail.com>.
Hi Jiajing
Thanks for joining the discussion.
SkyWalking has the e2e test framework, like Zhenxu Said, but it is
Java/Maven-stack based.
I did a quick review about `ovh/venom` project, but it seems to target the
request and validation, but SkyWalking CLI should take that responsibility,
which it has most, including GraphQL query and health check. Also, the same
as the language you preferred, it is go-written too.
We are not building something from 0, CLI is already a part of the
SkyWalking ecosystem, https://github.com/apache/skywalking-cli.
Sheng Wu 吴晟
Twitter, wusheng1108
陆家靖 <lu...@gmail.com> 于2020年11月15日周日 下午10:41写道：
> Dear all,
>
> In our team, we are also working on an e2e framework recently for the
> java agent and the tracing system.
>
> After some investigation in the open-source community, we find that
> https://github.com/ovh/venom is some how fit the usage at the very
> beginning,
>
> - It has some abstraction of test cases and one can define various
> test steps for a test case.
> - It provides many kinds of executors which allow us to run commands,
> manipulate with json data, send http requests, send messages to MQ and
> etc.
>
> But after some deep review, I found the following issues which concern us
> a lot,
>
> - It does not provide semantics such as `retry`, `failureThreshold`
> and `timeout`.
> - The power of assertions grammar is limited with space separated syntax.
> - Lifecycle (i.e. before and after hooks) is not supported.
>
> In our context, which I suppose is similar for Skywalking project,
>
> - the first issue is useful in health check steps, just like the ones
> defined in readiness probe and liveness probe in kubelet.
> - the second issue is critical for highly-customized assertions, we
> can leverage the `text/template` module to provide highly extensive
> assertion functions
> - the last but not the least, the lifecycle hooks can be used for
> accurate control of test setup and teardown, such as `docker-compose
> build, up -d, down`
>
> For the current query/verification code mentioned in the previous mail
> by Zhenxu, we can make it as an internal executor plugin.
>
> Together with the aforementioned ideas, we will have a go-written,
> yaml-data driven e2e testing framework which can overcome the current
> problems that occur,
> in which the most annoying issue is the fragmentation of different
> scripts in different submodules, IMO.
>
> Regards,
> Jiajing
>
> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <ke...@apache.org>
> wrote:
> >
> > > So, we would provide a script set to drive this?
> >
> > Yes, a script or a GoLang-written CLI, it depends.
> >
> > > My question is more about,
> > > how to work with docker-compose
> >
> > It’s the same as what we’re doing now in the current version of E2E
> tests, now we have many `docker-compose.yaml` files, and all the
> `docker-compose.yaml` files are reusable in the new framework, we will keep
> this mechanism as it is proved to be convenient and easy to use (remember
> when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test, not
> yet merged though, it’s fast/easy to add a new case).
> >
> > —————————
> > Zhenxu Ke (柯振旭)
> > GitHub @kezhenxu94
> >
> > > On Nov 15, 2020, at 9:29 PM, Sheng Wu <wu...@gmail.com>
> wrote:
> > >
> > > kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午9:16写道：
> > >
> > >>
> > >>> I want to ask, how the docker-compose lands in your mind? I think CLI
> > >> can't
> > >>> control this part.
> > >>
> > >>
> > >> docker-compose just fits the requirements perfectly that we need, we
> have
> > >> been using it in the current E2E and turns out it has advantages in
> > >> starting many services correctly with health check and dependency
> > >> declaration. (One may say Kubernetes can do the same thing, but it’s
> kind
> > >> of heavy in E2E and GitHub Actions and complex).
> > >>
> > >> The SkyWalking-CLI won’t get involved with docker-compose related
> things,
> > >> it just provides query ability (the same as what it is doing now,
> just need
> > >> to check whether it covers all the needed query interfaces or not),
> we will
> > >> have a dedicated control program to
> > >>
> > >> (1) start docker-compose (existing standard mechanism, no new
> knowledge);
> > >>
> > >
> > > So, we would provide a script set to drive this? My question is more
> about,
> > > how to work with docker-compose, rather than challanging the choice.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > >> (2) query actual data (by SkyWalking-CLI, existing repository and
> codes);
> > >> (3) verify the actual data (by verification tool, a new tool, TODO);
> > >> (4) and determine the test result (success or failed);
> > >>
> > >>
> > >> —————————
> > >> Zhenxu Ke (柯振旭)
> > >> GitHub @kezhenxu94
> > >>
> > >>
> > >>
> > >>> Sheng Wu 吴晟
> > >>> Twitter, wusheng1108
> > >>>
> > >>>
> > >>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午5:27写道：
> > >>>
> > >>>> Hi the community, I want to raise a discussion to build the next
> > >>>> generation of E2E test framework for Apache SkyWalking, which may
> not
> > >> be a
> > >>>> high priority but a necessity.
> > >>>>
> > >>>> As we already know, tests in SkyWalking play a very important role
> in
> > >> the
> > >>>> contribution process, and the current E2E tests just work well to
> verify
> > >>>> every single pull request before merging, so why bother to build the
> > >> next
> > >>>> generation of E2E test framework?
> > >>>>
> > >>>> 1. In the SkyWalking's ecosystem, there are many sub-projects that
> vary
> > >> in
> > >>>> terms of language, almost all of them need E2E tests to help us
> verify
> > >> the
> > >>>> pull requests, but we can see that they reimplement the E2E test
> > >> framework
> > >>>> in their languages, (or even worse, some of them don’t have E2E
> tests at
> > >>>> all). Reimplementing the E2E test framework is unnecesarry and
> > >> introduces
> > >>>> more duplicated work when adding a new test case. The ultimate goal
> is
> > >> to
> > >>>> reuse all the test cases when needed.
> > >>>>
> > >>>> 2. The current E2E framework is built with Java, running Java-based
> > >>>> program is not an easy thing for non-Java developers (maintainers of
> > >> other
> > >>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the
> > >>>> environment correctly and then run the tests, neither is it an easy
> > >> skill
> > >>>> to debug the tests.
> > >>>>
> > >>>> 3. It’s a good opportunity to optimize the E2E tests because we
> added
> > >> many
> > >>>> cases and many duplicated codes that need to be removed.
> > >>>>
> > >>>> Therefore we're planning to build an unified, easy-to-use, and fast
> E2E
> > >>>> test framework to benifit all the sub-projects.
> > >>>>
> > >>>> Here are some rough ideas about this framework:
> > >>>>
> > >>>> 0. (Unchanged) We will continue to use docker-compose.yaml to
> > >> orchestrate
> > >>>> the services that are to be tested, it’s proved to be friendly and
> > >>>> debuggable because we can start it directly even if we are not
> writing
> > >> E2E
> > >>>> tests.
> > >>>> 1. This framework will take full advantages of the CLI project to
> query
> > >>>> the traces/metrics/logs, we don’t want to maintain two versions of
> query
> > >>>> codes anymore, (now we have a version in the test codes and one in
> the
> > >> CLI).
> > >>>> 2. We will build a CLI tool to compare the expected data and the
> actual
> > >>>> data, we now have a custom schema of the expected data yaml file,
> the
> > >>>> verification codes should be packaged into an executable command so
> > >> that it
> > >>>> can be executed standalone without Java and maven knowledge. I hope
> we
> > >> can
> > >>>> leverage existing standards to write the YAML files and do
> > >> verifications.
> > >>>> 3. Build an orchestrating tool to start the `docker-compose.yaml`,
> > >> health
> > >>>> check, query actual data, and verify the result.
> > >>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share
> > >> between
> > >>>> sub-projects.
> > >>>>
> > >>>> If you have any other better idea or want to complement to this
> > >> proposal,
> > >>>> please reply here. I will create an issue to track these tasks
> later.
> > >>>>
> > >>>> We have two students (who just finished the Summer Code 2020
> Projects)
> > >> to
> > >>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is
> a
> > >> not
> > >>>> high priority, you have adequate time to get yourselves familiar
> with
> > >> how
> > >>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work
> as
> > >> well
> > >>>> as the ecosystem. Thanks for your interests and willingness to help.
> > >>>>
> > >>>> [1] https://github.com/Humbertzhang
> > >>>> [2] https://github.com/fgksgf/
> > >>>>
> > >>>> —————————
> > >>>> Zhenxu Ke (柯振旭)
> > >>>> GitHub @kezhenxu94
> > >>>>
> > >>>>
> > >>
> > >> —————————
> > >> Zhenxu Ke (柯振旭)
> > >> GitHub @kezhenxu94
> >
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by 陆家靖 <lu...@gmail.com>.
Dear all,
In our team, we are also working on an e2e framework recently for the
java agent and the tracing system.
After some investigation in the open-source community, we find that
https://github.com/ovh/venom is some how fit the usage at the very
beginning,
- It has some abstraction of test cases and one can define various
test steps for a test case.
- It provides many kinds of executors which allow us to run commands,
manipulate with json data, send http requests, send messages to MQ and
etc.
But after some deep review, I found the following issues which concern us a lot,
- It does not provide semantics such as `retry`, `failureThreshold`
and `timeout`.
- The power of assertions grammar is limited with space separated syntax.
- Lifecycle (i.e. before and after hooks) is not supported.
In our context, which I suppose is similar for Skywalking project,
- the first issue is useful in health check steps, just like the ones
defined in readiness probe and liveness probe in kubelet.
- the second issue is critical for highly-customized assertions, we
can leverage the `text/template` module to provide highly extensive
assertion functions
- the last but not the least, the lifecycle hooks can be used for
accurate control of test setup and teardown, such as `docker-compose
build, up -d, down`
For the current query/verification code mentioned in the previous mail
by Zhenxu, we can make it as an internal executor plugin.
Together with the aforementioned ideas, we will have a go-written,
yaml-data driven e2e testing framework which can overcome the current
problems that occur,
in which the most annoying issue is the fragmentation of different
scripts in different submodules, IMO.
Regards,
Jiajing
On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <ke...@apache.org> wrote:
>
> > So, we would provide a script set to drive this?
>
> Yes, a script or a GoLang-written CLI, it depends.
>
> > My question is more about,
> > how to work with docker-compose
>
> It’s the same as what we’re doing now in the current version of E2E tests, now we have many `docker-compose.yaml` files, and all the `docker-compose.yaml` files are reusable in the new framework, we will keep this mechanism as it is proved to be convenient and easy to use (remember when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test, not yet merged though, it’s fast/easy to add a new case).
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94
>
> > On Nov 15, 2020, at 9:29 PM, Sheng Wu <wu...@gmail.com> wrote:
> >
> > kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午9:16写道：
> >
> >>
> >>> I want to ask, how the docker-compose lands in your mind? I think CLI
> >> can't
> >>> control this part.
> >>
> >>
> >> docker-compose just fits the requirements perfectly that we need, we have
> >> been using it in the current E2E and turns out it has advantages in
> >> starting many services correctly with health check and dependency
> >> declaration. (One may say Kubernetes can do the same thing, but it’s kind
> >> of heavy in E2E and GitHub Actions and complex).
> >>
> >> The SkyWalking-CLI won’t get involved with docker-compose related things,
> >> it just provides query ability (the same as what it is doing now, just need
> >> to check whether it covers all the needed query interfaces or not), we will
> >> have a dedicated control program to
> >>
> >> (1) start docker-compose (existing standard mechanism, no new knowledge);
> >>
> >
> > So, we would provide a script set to drive this? My question is more about,
> > how to work with docker-compose, rather than challanging the choice.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> >> (2) query actual data (by SkyWalking-CLI, existing repository and codes);
> >> (3) verify the actual data (by verification tool, a new tool, TODO);
> >> (4) and determine the test result (success or failed);
> >>
> >>
> >> —————————
> >> Zhenxu Ke (柯振旭)
> >> GitHub @kezhenxu94
> >>
> >>
> >>
> >>> Sheng Wu 吴晟
> >>> Twitter, wusheng1108
> >>>
> >>>
> >>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午5:27写道：
> >>>
> >>>> Hi the community, I want to raise a discussion to build the next
> >>>> generation of E2E test framework for Apache SkyWalking, which may not
> >> be a
> >>>> high priority but a necessity.
> >>>>
> >>>> As we already know, tests in SkyWalking play a very important role in
> >> the
> >>>> contribution process, and the current E2E tests just work well to verify
> >>>> every single pull request before merging, so why bother to build the
> >> next
> >>>> generation of E2E test framework?
> >>>>
> >>>> 1. In the SkyWalking's ecosystem, there are many sub-projects that vary
> >> in
> >>>> terms of language, almost all of them need E2E tests to help us verify
> >> the
> >>>> pull requests, but we can see that they reimplement the E2E test
> >> framework
> >>>> in their languages, (or even worse, some of them don’t have E2E tests at
> >>>> all). Reimplementing the E2E test framework is unnecesarry and
> >> introduces
> >>>> more duplicated work when adding a new test case. The ultimate goal is
> >> to
> >>>> reuse all the test cases when needed.
> >>>>
> >>>> 2. The current E2E framework is built with Java, running Java-based
> >>>> program is not an easy thing for non-Java developers (maintainers of
> >> other
> >>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the
> >>>> environment correctly and then run the tests, neither is it an easy
> >> skill
> >>>> to debug the tests.
> >>>>
> >>>> 3. It’s a good opportunity to optimize the E2E tests because we added
> >> many
> >>>> cases and many duplicated codes that need to be removed.
> >>>>
> >>>> Therefore we're planning to build an unified, easy-to-use, and fast E2E
> >>>> test framework to benifit all the sub-projects.
> >>>>
> >>>> Here are some rough ideas about this framework:
> >>>>
> >>>> 0. (Unchanged) We will continue to use docker-compose.yaml to
> >> orchestrate
> >>>> the services that are to be tested, it’s proved to be friendly and
> >>>> debuggable because we can start it directly even if we are not writing
> >> E2E
> >>>> tests.
> >>>> 1. This framework will take full advantages of the CLI project to query
> >>>> the traces/metrics/logs, we don’t want to maintain two versions of query
> >>>> codes anymore, (now we have a version in the test codes and one in the
> >> CLI).
> >>>> 2. We will build a CLI tool to compare the expected data and the actual
> >>>> data, we now have a custom schema of the expected data yaml file, the
> >>>> verification codes should be packaged into an executable command so
> >> that it
> >>>> can be executed standalone without Java and maven knowledge. I hope we
> >> can
> >>>> leverage existing standards to write the YAML files and do
> >> verifications.
> >>>> 3. Build an orchestrating tool to start the `docker-compose.yaml`,
> >> health
> >>>> check, query actual data, and verify the result.
> >>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share
> >> between
> >>>> sub-projects.
> >>>>
> >>>> If you have any other better idea or want to complement to this
> >> proposal,
> >>>> please reply here. I will create an issue to track these tasks later.
> >>>>
> >>>> We have two students (who just finished the Summer Code 2020 Projects)
> >> to
> >>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is a
> >> not
> >>>> high priority, you have adequate time to get yourselves familiar with
> >> how
> >>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work as
> >> well
> >>>> as the ecosystem. Thanks for your interests and willingness to help.
> >>>>
> >>>> [1] https://github.com/Humbertzhang
> >>>> [2] https://github.com/fgksgf/
> >>>>
> >>>> —————————
> >>>> Zhenxu Ke (柯振旭)
> >>>> GitHub @kezhenxu94
> >>>>
> >>>>
> >>
> >> —————————
> >> Zhenxu Ke (柯振旭)
> >> GitHub @kezhenxu94
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by "kezhenxu94@apache" <ke...@apache.org>.
> So, we would provide a script set to drive this?
Yes, a script or a GoLang-written CLI, it depends.
> My question is more about,
> how to work with docker-compose
It’s the same as what we’re doing now in the current version of E2E tests, now we have many `docker-compose.yaml` files, and all the `docker-compose.yaml` files are reusable in the new framework, we will keep this mechanism as it is proved to be convenient and easy to use (remember when we added an ElasticSearch 7.9 case, and the TiDB storage E2E test, not yet merged though, it’s fast/easy to add a new case).
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
> On Nov 15, 2020, at 9:29 PM, Sheng Wu <wu...@gmail.com> wrote:
>
> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午9:16写道：
>
>>
>>> I want to ask, how the docker-compose lands in your mind? I think CLI
>> can't
>>> control this part.
>>
>>
>> docker-compose just fits the requirements perfectly that we need, we have
>> been using it in the current E2E and turns out it has advantages in
>> starting many services correctly with health check and dependency
>> declaration. (One may say Kubernetes can do the same thing, but it’s kind
>> of heavy in E2E and GitHub Actions and complex).
>>
>> The SkyWalking-CLI won’t get involved with docker-compose related things,
>> it just provides query ability (the same as what it is doing now, just need
>> to check whether it covers all the needed query interfaces or not), we will
>> have a dedicated control program to
>>
>> (1) start docker-compose (existing standard mechanism, no new knowledge);
>>
>
> So, we would provide a script set to drive this? My question is more about,
> how to work with docker-compose, rather than challanging the choice.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
>> (2) query actual data (by SkyWalking-CLI, existing repository and codes);
>> (3) verify the actual data (by verification tool, a new tool, TODO);
>> (4) and determine the test result (success or failed);
>>
>>
>> —————————
>> Zhenxu Ke (柯振旭)
>> GitHub @kezhenxu94
>>
>>
>>
>>> Sheng Wu 吴晟
>>> Twitter, wusheng1108
>>>
>>>
>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午5:27写道：
>>>
>>>> Hi the community, I want to raise a discussion to build the next
>>>> generation of E2E test framework for Apache SkyWalking, which may not
>> be a
>>>> high priority but a necessity.
>>>>
>>>> As we already know, tests in SkyWalking play a very important role in
>> the
>>>> contribution process, and the current E2E tests just work well to verify
>>>> every single pull request before merging, so why bother to build the
>> next
>>>> generation of E2E test framework?
>>>>
>>>> 1. In the SkyWalking's ecosystem, there are many sub-projects that vary
>> in
>>>> terms of language, almost all of them need E2E tests to help us verify
>> the
>>>> pull requests, but we can see that they reimplement the E2E test
>> framework
>>>> in their languages, (or even worse, some of them don’t have E2E tests at
>>>> all). Reimplementing the E2E test framework is unnecesarry and
>> introduces
>>>> more duplicated work when adding a new test case. The ultimate goal is
>> to
>>>> reuse all the test cases when needed.
>>>>
>>>> 2. The current E2E framework is built with Java, running Java-based
>>>> program is not an easy thing for non-Java developers (maintainers of
>> other
>>>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the
>>>> environment correctly and then run the tests, neither is it an easy
>> skill
>>>> to debug the tests.
>>>>
>>>> 3. It’s a good opportunity to optimize the E2E tests because we added
>> many
>>>> cases and many duplicated codes that need to be removed.
>>>>
>>>> Therefore we're planning to build an unified, easy-to-use, and fast E2E
>>>> test framework to benifit all the sub-projects.
>>>>
>>>> Here are some rough ideas about this framework:
>>>>
>>>> 0. (Unchanged) We will continue to use docker-compose.yaml to
>> orchestrate
>>>> the services that are to be tested, it’s proved to be friendly and
>>>> debuggable because we can start it directly even if we are not writing
>> E2E
>>>> tests.
>>>> 1. This framework will take full advantages of the CLI project to query
>>>> the traces/metrics/logs, we don’t want to maintain two versions of query
>>>> codes anymore, (now we have a version in the test codes and one in the
>> CLI).
>>>> 2. We will build a CLI tool to compare the expected data and the actual
>>>> data, we now have a custom schema of the expected data yaml file, the
>>>> verification codes should be packaged into an executable command so
>> that it
>>>> can be executed standalone without Java and maven knowledge. I hope we
>> can
>>>> leverage existing standards to write the YAML files and do
>> verifications.
>>>> 3. Build an orchestrating tool to start the `docker-compose.yaml`,
>> health
>>>> check, query actual data, and verify the result.
>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share
>> between
>>>> sub-projects.
>>>>
>>>> If you have any other better idea or want to complement to this
>> proposal,
>>>> please reply here. I will create an issue to track these tasks later.
>>>>
>>>> We have two students (who just finished the Summer Code 2020 Projects)
>> to
>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is a
>> not
>>>> high priority, you have adequate time to get yourselves familiar with
>> how
>>>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work as
>> well
>>>> as the ecosystem. Thanks for your interests and willingness to help.
>>>>
>>>> [1] https://github.com/Humbertzhang
>>>> [2] https://github.com/fgksgf/
>>>>
>>>> —————————
>>>> Zhenxu Ke (柯振旭)
>>>> GitHub @kezhenxu94
>>>>
>>>>
>>
>> —————————
>> Zhenxu Ke (柯振旭)
>> GitHub @kezhenxu94
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Sheng Wu <wu...@gmail.com>.
kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午9:16写道：
>
> > I want to ask, how the docker-compose lands in your mind? I think CLI
> can't
> > control this part.
>
>
> docker-compose just fits the requirements perfectly that we need, we have
> been using it in the current E2E and turns out it has advantages in
> starting many services correctly with health check and dependency
> declaration. (One may say Kubernetes can do the same thing, but it’s kind
> of heavy in E2E and GitHub Actions and complex).
>
> The SkyWalking-CLI won’t get involved with docker-compose related things,
> it just provides query ability (the same as what it is doing now, just need
> to check whether it covers all the needed query interfaces or not), we will
> have a dedicated control program to
>
> (1) start docker-compose (existing standard mechanism, no new knowledge);
>
So, we would provide a script set to drive this? My question is more about,
how to work with docker-compose, rather than challanging the choice.
Sheng Wu 吴晟
Twitter, wusheng1108
> (2) query actual data (by SkyWalking-CLI, existing repository and codes);
> (3) verify the actual data (by verification tool, a new tool, TODO);
> (4) and determine the test result (success or failed);
>
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94
>
>
>
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午5:27写道：
> >
> >> Hi the community, I want to raise a discussion to build the next
> >> generation of E2E test framework for Apache SkyWalking, which may not
> be a
> >> high priority but a necessity.
> >>
> >> As we already know, tests in SkyWalking play a very important role in
> the
> >> contribution process, and the current E2E tests just work well to verify
> >> every single pull request before merging, so why bother to build the
> next
> >> generation of E2E test framework?
> >>
> >> 1. In the SkyWalking's ecosystem, there are many sub-projects that vary
> in
> >> terms of language, almost all of them need E2E tests to help us verify
> the
> >> pull requests, but we can see that they reimplement the E2E test
> framework
> >> in their languages, (or even worse, some of them don’t have E2E tests at
> >> all). Reimplementing the E2E test framework is unnecesarry and
> introduces
> >> more duplicated work when adding a new test case. The ultimate goal is
> to
> >> reuse all the test cases when needed.
> >>
> >> 2. The current E2E framework is built with Java, running Java-based
> >> program is not an easy thing for non-Java developers (maintainers of
> other
> >> sub-projects such as PHP, C++, LUA, etc.), they need to set up the
> >> environment correctly and then run the tests, neither is it an easy
> skill
> >> to debug the tests.
> >>
> >> 3. It’s a good opportunity to optimize the E2E tests because we added
> many
> >> cases and many duplicated codes that need to be removed.
> >>
> >> Therefore we're planning to build an unified, easy-to-use, and fast E2E
> >> test framework to benifit all the sub-projects.
> >>
> >> Here are some rough ideas about this framework:
> >>
> >> 0. (Unchanged) We will continue to use docker-compose.yaml to
> orchestrate
> >> the services that are to be tested, it’s proved to be friendly and
> >> debuggable because we can start it directly even if we are not writing
> E2E
> >> tests.
> >> 1. This framework will take full advantages of the CLI project to query
> >> the traces/metrics/logs, we don’t want to maintain two versions of query
> >> codes anymore, (now we have a version in the test codes and one in the
> CLI).
> >> 2. We will build a CLI tool to compare the expected data and the actual
> >> data, we now have a custom schema of the expected data yaml file, the
> >> verification codes should be packaged into an executable command so
> that it
> >> can be executed standalone without Java and maven knowledge. I hope we
> can
> >> leverage existing standards to write the YAML files and do
> verifications.
> >> 3. Build an orchestrating tool to start the `docker-compose.yaml`,
> health
> >> check, query actual data, and verify the result.
> >> 4. (Nice to have) wrap all the tools as a GitHub Actions to share
> between
> >> sub-projects.
> >>
> >> If you have any other better idea or want to complement to this
> proposal,
> >> please reply here. I will create an issue to track these tasks later.
> >>
> >> We have two students (who just finished the Summer Code 2020 Projects)
> to
> >> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is a
> not
> >> high priority, you have adequate time to get yourselves familiar with
> how
> >> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work as
> well
> >> as the ecosystem. Thanks for your interests and willingness to help.
> >>
> >> [1] https://github.com/Humbertzhang
> >> [2] https://github.com/fgksgf/
> >>
> >> —————————
> >> Zhenxu Ke (柯振旭)
> >> GitHub @kezhenxu94
> >>
> >>
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94
>
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by "kezhenxu94@apache" <ke...@apache.org>.
> I want to ask, how the docker-compose lands in your mind? I think CLI can't
> control this part.
docker-compose just fits the requirements perfectly that we need, we have been using it in the current E2E and turns out it has advantages in starting many services correctly with health check and dependency declaration. (One may say Kubernetes can do the same thing, but it’s kind of heavy in E2E and GitHub Actions and complex).
The SkyWalking-CLI won’t get involved with docker-compose related things, it just provides query ability (the same as what it is doing now, just need to check whether it covers all the needed query interfaces or not), we will have a dedicated control program to
(1) start docker-compose (existing standard mechanism, no new knowledge);
(2) query actual data (by SkyWalking-CLI, existing repository and codes);
(3) verify the actual data (by verification tool, a new tool, TODO);
(4) and determine the test result (success or failed);
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午5:27写道：
>
>> Hi the community, I want to raise a discussion to build the next
>> generation of E2E test framework for Apache SkyWalking, which may not be a
>> high priority but a necessity.
>>
>> As we already know, tests in SkyWalking play a very important role in the
>> contribution process, and the current E2E tests just work well to verify
>> every single pull request before merging, so why bother to build the next
>> generation of E2E test framework?
>>
>> 1. In the SkyWalking's ecosystem, there are many sub-projects that vary in
>> terms of language, almost all of them need E2E tests to help us verify the
>> pull requests, but we can see that they reimplement the E2E test framework
>> in their languages, (or even worse, some of them don’t have E2E tests at
>> all). Reimplementing the E2E test framework is unnecesarry and introduces
>> more duplicated work when adding a new test case. The ultimate goal is to
>> reuse all the test cases when needed.
>>
>> 2. The current E2E framework is built with Java, running Java-based
>> program is not an easy thing for non-Java developers (maintainers of other
>> sub-projects such as PHP, C++, LUA, etc.), they need to set up the
>> environment correctly and then run the tests, neither is it an easy skill
>> to debug the tests.
>>
>> 3. It’s a good opportunity to optimize the E2E tests because we added many
>> cases and many duplicated codes that need to be removed.
>>
>> Therefore we're planning to build an unified, easy-to-use, and fast E2E
>> test framework to benifit all the sub-projects.
>>
>> Here are some rough ideas about this framework:
>>
>> 0. (Unchanged) We will continue to use docker-compose.yaml to orchestrate
>> the services that are to be tested, it’s proved to be friendly and
>> debuggable because we can start it directly even if we are not writing E2E
>> tests.
>> 1. This framework will take full advantages of the CLI project to query
>> the traces/metrics/logs, we don’t want to maintain two versions of query
>> codes anymore, (now we have a version in the test codes and one in the CLI).
>> 2. We will build a CLI tool to compare the expected data and the actual
>> data, we now have a custom schema of the expected data yaml file, the
>> verification codes should be packaged into an executable command so that it
>> can be executed standalone without Java and maven knowledge. I hope we can
>> leverage existing standards to write the YAML files and do verifications.
>> 3. Build an orchestrating tool to start the `docker-compose.yaml`, health
>> check, query actual data, and verify the result.
>> 4. (Nice to have) wrap all the tools as a GitHub Actions to share between
>> sub-projects.
>>
>> If you have any other better idea or want to complement to this proposal,
>> please reply here. I will create an issue to track these tasks later.
>>
>> We have two students (who just finished the Summer Code 2020 Projects) to
>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is a not
>> high priority, you have adequate time to get yourselves familiar with how
>> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work as well
>> as the ecosystem. Thanks for your interests and willingness to help.
>>
>> [1] https://github.com/Humbertzhang
>> [2] https://github.com/fgksgf/
>>
>> —————————
>> Zhenxu Ke (柯振旭)
>> GitHub @kezhenxu94
>>
>>
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Sheng Wu <wu...@gmail.com>.
Supre, when we first discussed this weeks ago, I didn't realize we will
have the students lead this!!
Glad to see this.
CLI verification tool seems to become a very important part of the e2e
part, which is great.
I want to ask, how the docker-compose lands in your mind? I think CLI can't
control this part.
Sheng Wu 吴晟
Twitter, wusheng1108
kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日 下午5:27写道：
> Hi the community, I want to raise a discussion to build the next
> generation of E2E test framework for Apache SkyWalking, which may not be a
> high priority but a necessity.
>
> As we already know, tests in SkyWalking play a very important role in the
> contribution process, and the current E2E tests just work well to verify
> every single pull request before merging, so why bother to build the next
> generation of E2E test framework?
>
> 1. In the SkyWalking's ecosystem, there are many sub-projects that vary in
> terms of language, almost all of them need E2E tests to help us verify the
> pull requests, but we can see that they reimplement the E2E test framework
> in their languages, (or even worse, some of them don’t have E2E tests at
> all). Reimplementing the E2E test framework is unnecesarry and introduces
> more duplicated work when adding a new test case. The ultimate goal is to
> reuse all the test cases when needed.
>
> 2. The current E2E framework is built with Java, running Java-based
> program is not an easy thing for non-Java developers (maintainers of other
> sub-projects such as PHP, C++, LUA, etc.), they need to set up the
> environment correctly and then run the tests, neither is it an easy skill
> to debug the tests.
>
> 3. It’s a good opportunity to optimize the E2E tests because we added many
> cases and many duplicated codes that need to be removed.
>
> Therefore we're planning to build an unified, easy-to-use, and fast E2E
> test framework to benifit all the sub-projects.
>
> Here are some rough ideas about this framework:
>
> 0. (Unchanged) We will continue to use docker-compose.yaml to orchestrate
> the services that are to be tested, it’s proved to be friendly and
> debuggable because we can start it directly even if we are not writing E2E
> tests.
> 1. This framework will take full advantages of the CLI project to query
> the traces/metrics/logs, we don’t want to maintain two versions of query
> codes anymore, (now we have a version in the test codes and one in the CLI).
> 2. We will build a CLI tool to compare the expected data and the actual
> data, we now have a custom schema of the expected data yaml file, the
> verification codes should be packaged into an executable command so that it
> can be executed standalone without Java and maven knowledge. I hope we can
> leverage existing standards to write the YAML files and do verifications.
> 3. Build an orchestrating tool to start the `docker-compose.yaml`, health
> check, query actual data, and verify the result.
> 4. (Nice to have) wrap all the tools as a GitHub Actions to share between
> sub-projects.
>
> If you have any other better idea or want to complement to this proposal,
> please reply here. I will create an issue to track these tasks later.
>
> We have two students (who just finished the Summer Code 2020 Projects) to
> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I said this is a not
> high priority, you have adequate time to get yourselves familiar with how
> the components (OAP, WebUI, CLI, storages, etc.) of SkyWalking work as well
> as the ecosystem. Thanks for your interests and willingness to help.
>
> [1] https://github.com/Humbertzhang
> [2] https://github.com/fgksgf/
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94
>
>