You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@skywalking.apache.org by "kezhenxu94@apache" <ke...@apache.org> on 2020/12/17 09:45:00 UTC
Re: [DISCUSSION] Build the next generation of E2E test framework
Hey, I’ve just drafted a design doc here [1], anyone interested in this please feel free to leave your comments, thanks!
[1] https://github.com/apache/skywalking-website/pull/172
> On Nov 26, 2020, at 14:41, Sheng Wu <wu...@gmail.com> wrote:
>
> You could learn from the Satellite project design,
> 1. Discuss here, https://github.com/apache/skywalking/issues/5871
> 2. Final design posted as a blog,
> http://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/
>
> People could learn from these things when you join the community in the
> future.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Hoshea Jiang <fg...@gmail.com> 于2020年11月26日周四 下午1:38写道：
>
>> I agree with the conclusion, and I hope to have some diagrams such as
>> flowcharts or data flow diagrams, which help provide an overview of the E2E
>> test before we start to do this.
>>
>> Ke Zhang <hu...@gmail.com> 于2020年11月26日周四 下午12:34写道：
>>
>>> I think supporting both docker-compose and KIND in the E2E test is
>>> reasonable and necessary, and I also agree with other conclusions.😀
>>>
>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月26日周四 上午11:58写道：
>>>
>>>>
>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
>>>>> 2. Totally replace the Java and Maven based driving system.
>>>>> 3. Enhance CLI to validate the GraphQL
>>>>> 4. Keep the agent test tool unchanged for now as it is already a
>>> separate
>>>>> system from the e2e.
>>>>
>>>> This conclusion is good for me.
>>>>
>>>> What are others opinions?
>>>>
>>>> —————————
>>>> Zhenxu Ke (柯振旭)
>>>> GitHub @kezhenxu94
>>>>
>>>>> On Nov 17, 2020, at 2:42 PM, Sheng Wu <wu...@gmail.com>
>>> wrote:
>>>>>
>>>>> Using k8s and docker-compose as 2 options in the test process is
>>>>> reasonable, and should be supported.
>>>>>
>>>>> Let's keep the conclusion as simple as possible. In the next
>> generation
>>>> e2e
>>>>> framework
>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
>>>>> 2. Totally replace the Java and Maven based driving system.
>>>>> 3. Enhance CLI to validate the GraphQL
>>>>> 4. Keep the agent test tool unchanged for now as it is already a
>>> separate
>>>>> system from the e2e.
>>>>>
>>>>> Are these good enough for everyone?
>>>>>
>>>>> Sheng Wu 吴晟
>>>>> Twitter, wusheng1108
>>>>>
>>>>>
>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
>>>>>
>>>>>>>
>>>>>>> We can see from above, your discussion actually is only related to
>> 3.
>>>>>>
>>>>>>
>>>>>> That's the only I intend to discuss here if you notice my first
>>>>>> response('May SWCK help about provision and demo") in this thread.
>> For
>>>>>> other parts, I don't have any tips to share, though.
>>>>>> Let me explain it more clearly. I want kubernetes to be seen as an
>>>>>> alternative to docker-compose to play a running-engine role in the
>>> test
>>>>>> framework. If kubernetes can't replace
>>>>>> the latter one Is it able to participate in it to solve the
>> dedicated
>>> or
>>>>>> special issues.
>>>>>>
>>>>>> Our e2e test and plugin test frameworks are already top-level
>>> complicity
>>>>>>> thing in the worldwide open source field. Please don't over-design
>>> it,
>>>>>> and
>>>>>>> too aggressive, the world will stay in the hybrid env for a very
>> long
>>>>>> time,
>>>>>>> same as the developer.
>>>>>>
>>>>>>
>>>>>> Reusing current elaborate output might be reasonable. But in this
>>>> thread,
>>>>>> we're talking about the potential solution to build a
>>>>>> next-generation framework, which might mean that the current
>> framework
>>>>>> is too complicated to maintain; we need a more tiny way to support
>>> more
>>>> and
>>>>>> more cases.
>>>>>>
>>>>>> And finally, SWCK is gonna and has to build a similar test
>> framework.
>>> If
>>>>>> the test framework opts for the kubernetes solution, we could share
>>> test
>>>>>> cases and the underlying framework. It's a more efficient
>>>>>> and consistent path for the entire system.
>>>>>>
>>>>>> regards, Hongtao.
>>>>>>
>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I don't think we have to make the decision on k8s or out-of-k8s.
>>>>>>> e2e and all integration tests have their scenarios like SkyWalking
>>> have
>>>>>>> multiple languages based and mesh-based, even Prometheus adoption.
>>>>>>> The same thing happens on the test field, k8s, and out of k8s.
>>>>>>>
>>>>>>> I would like to agree with both of you, k8s is needed, no matter in
>>>> mesh
>>>>>>> solution(ALS, telemetry v2), but also not anyone needs that, such
>> as
>>>>>> agent
>>>>>>> developer. We wouldn't deny the fact, the docker-compose is more
>>>>>>> lightweight and easier to learn.
>>>>>>> So, let me get this straight, for the test you need
>>>>>>> 1. Build the source from different repos
>>>>>>> 2. Build the images
>>>>>>> 3. Use some platform(docker-compose or k8s) to run the images in
>> one
>>>>>> piece.
>>>>>>> 4. Give some inputs to the env
>>>>>>> 5. Use some tools/ways to check what should happen according to
>> (4)'s
>>>>>> input
>>>>>>> 6. Checked or timeout failure.
>>>>>>>
>>>>>>> We can see from above, your discussion actually is only related to
>> 3.
>>>>>> Let's
>>>>>>> not see this as a battle, we need both. Many developers/users use
>> OAP
>>>> and
>>>>>>> SkyWalking out of k8s, that is a simple fact.
>>>>>>> Our e2e test and plugin test frameworks are already top-level
>>>> complicity
>>>>>>> thing in the worldwide open source field. Please don't over-design
>>> it,
>>>>>> and
>>>>>>> too aggressive, the world will stay in the hybrid env for a very
>> long
>>>>>> time,
>>>>>>> same as the developer.
>>>>>>>
>>>>>>> Also, let's keep in mind, k8s is popular becomes it doesn't require
>>> the
>>>>>>> developers to understand as the VM did. We as an open-source
>>> community,
>>>>>> are
>>>>>>> focusing on the developer's capabilities.
>>>>>>>
>>>>>>> Sheng Wu 吴晟
>>>>>>> Twitter, wusheng1108
>>>>>>>
>>>>>>>
>>>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
>>>>>>>
>>>>>>>>>
>>>>>>>>> but I have been thinking whether or not it is overkill for
>> testing
>>>>>>>>> purposes to introduce Kubernetes things.
>>>>>>>>
>>>>>>>>
>>>>>>>> we all know the fact docker-compose is more portable than
>>> Kubernetes,
>>>>>>>> friendly to local running. But there're also some benefits to
>>> replace
>>>>>> it
>>>>>>>> with kubernetes system:
>>>>>>>>
>>>>>>>> 1. SWCK has to test based on a real Kubernetes environment if the
>>> main
>>>>>>>> repo test framework doesn't support it(following the
>> docker-compose
>>>>>>> stack),
>>>>>>>> That means skywalking ecosystem MUST introduce kubernetes which is
>>> not
>>>>>> an
>>>>>>>> optional component.
>>>>>>>>
>>>>>>>> 2. Kubernetes support to scale applications at runtime, that help
>>>>>>> improve
>>>>>>>> the process of e2e. For instance, we could merge a single instance
>>> and
>>>>>>>> cluster test, by scaling up replicas. This strategy will reduce
>> the
>>>>>> total
>>>>>>>> time sharply.
>>>>>>>>
>>>>>>>> If we are going to use KIND, I hope we can give a script or
>>> something
>>>>>>>>> similar to install all the needed components and create cluster
>> for
>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
>>>>>>>>
>>>>>>>>
>>>>>>>> That's SWCK's capacity to provide a standard and simple way to
>>> create
>>>> a
>>>>>>>> cluster in every environment,
>>>>>>>>
>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一
>> 下午7:00写道：
>>>>>>>>
>>>>>>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting
>>>>>>> Kubernetes-based
>>>>>>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too,
>>>>>> but I
>>>>>>>>> have been thinking whether or not it is overkill for testing
>>> purposes
>>>>>>> to
>>>>>>>>> introduce Kubernetes things.
>>>>>>>>>
>>>>>>>>> Since you mentioned this, we are open to discuss this much more
>>>>>> deeply:
>>>>>>>>>
>>>>>>>>> It would be definitely a benefit to involve more “end”s in the
>>>>>> so-call
>>>>>>>>> “end to end” tests, I’m +1 to leverage SWCK as well.
>>>>>>>>>
>>>>>>>>> If we are going to use KIND, I hope we can give a script or
>>> something
>>>>>>>>> similar to install all the needed components and create cluster
>> for
>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> —————————
>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>
>>>>>>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> May SWCK help about provision and demo
>>>>>>>>>>
>>>>>>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone
>>>>>>> instance
>>>>>>>>> or
>>>>>>>>>> cluster for e2e. We could compose indicated CR yamls for
>> different
>>>>>>>>>> scenarios.
>>>>>>>>>> 2. Different demo for different cases is mandatory for e2e and
>>>>>> swck,
>>>>>>> we
>>>>>>>>>> could build a group of demo projects for them. With Kubernetes's
>>>>>>>> support,
>>>>>>>>>> we could
>>>>>>>>>> get free of verbose scripts that are written for installation,
>>>>>>>>> checking
>>>>>>>>>> the state of running applications, collecting logs.
>>>>>>>>>>
>>>>>>>>>> Ppl might have concerts about the minkube we currently have,
>>>>>> which's
>>>>>>>> hard
>>>>>>>>>> to debug and runs slowly.
>>>>>>>>>>
>>>>>>>>>> I prefer to kind[1] instead of minkube. From my experience, kind
>>>>>>> works
>>>>>>>>> fine
>>>>>>>>>> in an e2e scenario(limited cpu, memory resources), and it is run
>>>>>>>> million
>>>>>>>>>> times
>>>>>>>>>> on my team's e2e environment, which proves the stability and
>>>>>>>>>> maintainability of it.
>>>>>>>>>>
>>>>>>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
>>>>>>>>>>
>>>>>>>>>> regards, Hongtao
>>>>>>>>>>
>>>>>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
>>>>>>>>>>
>>>>>>>>>>> Inline
>>>>>>>>>>>
>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>> 下午11:16写道：
>>>>>>>>>>>
>>>>>>>>>>>> Hi Jiajing,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for sharing your context and joining the discussion.
>> It’s
>>>>>>> true
>>>>>>>>>>> that
>>>>>>>>>>>> we’re actually tackling the same problems in this domain. Some
>>> of
>>>>>>> the
>>>>>>>>>>>> issues you mentioned are solved in the current E2E framework
>>>>>> (like
>>>>>>>>>>> `retry`,
>>>>>>>>>>>> `timeout`, etc.), while some others are still there without
>>> ideal
>>>>>>>>>>> solution.
>>>>>>>>>>>> Other discussions, please see my comments inline.
>>>>>>>>>>>>
>>>>>>>>>>>>> - the first issue is useful in health check steps, just like
>>> the
>>>>>>>> ones
>>>>>>>>>>>>> defined in readiness probe and liveness probe in kubelet.
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, the reason why we decided to pick docker-compose is that,
>>> it
>>>>>>> has
>>>>>>>>> the
>>>>>>>>>>>> same functionality with kubelet in terms of dependent services
>>>>>>>> startup
>>>>>>>>>>>> order and healthiness checking, while it’s more lite-weight
>> and
>>>>>>> easy
>>>>>>>>> for
>>>>>>>>>>>> both developers and GitHub Actions environment. Right?
>>>>>>>>>>>>
>>>>>>>>>>>>> - the second issue is critical for highly-customized
>>> assertions,
>>>>>>> we
>>>>>>>>>>>>> can leverage the `text/template` module to provide highly
>>>>>>> extensive
>>>>>>>>>>>>> assertion functions
>>>>>>>>>>>>
>>>>>>>>>>>> `text/template` is also the first thought that came to my mind
>>>>>> when
>>>>>>>>>>>> considering customized assertors, customized assertors are for
>>>>>> sure
>>>>>>>>>>>> critical in SkyWalking E2E tests because we have many kinds of
>>>>>>>>>>> assertions,
>>>>>>>>>>>> some of which are not foreseen until we actually need them,
>>>>>>>>>>>> `AtLeastOneGreaterThanZero` is an example.
>>>>>>>>>>>>
>>>>>>>>>>>>> - the last but not the least, the lifecycle hooks can be used
>>>>>> for
>>>>>>>>>>>>> accurate control of test setup and teardown, such as
>>>>>>> `docker-compose
>>>>>>>>>>>>> build, up -d, down`
>>>>>>>>>>>>
>>>>>>>>>>>> The lifecycle hooks are what I didn’t take into consideration
>>>>>>>> because I
>>>>>>>>>>>> thought it is quite natural for test framework to provide this
>>>>>>>>> features,
>>>>>>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part.
>>>>>>>>>>>>
>>>>>>>>>>>>> Together with the aforementioned ideas, we will have a
>>>>>> go-written,
>>>>>>>>>>>>> yaml-data driven e2e testing framework which can overcome the
>>>>>>>> current
>>>>>>>>>>>>> problems that occur,
>>>>>>>>>>>>
>>>>>>>>>>>> I agree.
>>>>>>>>>>>>
>>>>>>>>>>>>> in which the most annoying issue is the fragmentation of
>>>>>> different
>>>>>>>>>>>>> scripts in different submodules, IMO.
>>>>>>>>>>>>
>>>>>>>>>>>> That’s why I list the last item in the previous email, (Nice
>> to
>>>>>>> have,
>>>>>>>>>>> wrap
>>>>>>>>>>>> them into a GitHub Action), with that GitHub Action, we can
>>>>>> simply
>>>>>>>>>>>> configure a workflow/job in every submodule, and the tests are
>>>>>>>> executed
>>>>>>>>>>>> with the shared scripts, no copy-pasting.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Note, GitHub Action will require an Apache release every time
>>>>>>> because
>>>>>>>> we
>>>>>>>>>>> are distributing the binary.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Jiajing
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
>>>>>>>>>>> kezhenxu94@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, we would provide a script set to drive this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My question is more about,
>>>>>>>>>>>>>>> how to work with docker-compose
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It’s the same as what we’re doing now in the current version
>>> of
>>>>>>> E2E
>>>>>>>>>>>> tests, now we have many `docker-compose.yaml` files, and all
>> the
>>>>>>>>>>>> `docker-compose.yaml` files are reusable in the new framework,
>>> we
>>>>>>>> will
>>>>>>>>>>> keep
>>>>>>>>>>>> this mechanism as it is proved to be convenient and easy to
>> use
>>>>>>>>> (remember
>>>>>>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage
>>> E2E
>>>>>>>> test,
>>>>>>>>>>> not
>>>>>>>>>>>> yet merged though, it’s fast/easy to add a new case).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
>>>>>>> wu.sheng.841108@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>>>> 下午9:16写道：
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I want to ask, how the docker-compose lands in your
>> mind? I
>>>>>>>> think
>>>>>>>>>>> CLI
>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>> control this part.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> docker-compose just fits the requirements perfectly that
>> we
>>>>>>> need,
>>>>>>>>> we
>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> been using it in the current E2E and turns out it has
>>>>>>> advantages
>>>>>>>> in
>>>>>>>>>>>>>>>> starting many services correctly with health check and
>>>>>>> dependency
>>>>>>>>>>>>>>>> declaration. (One may say Kubernetes can do the same
>> thing,
>>>>>> but
>>>>>>>>> it’s
>>>>>>>>>>>> kind
>>>>>>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
>>>>>>> related
>>>>>>>>>>>> things,
>>>>>>>>>>>>>>>> it just provides query ability (the same as what it is
>> doing
>>>>>>> now,
>>>>>>>>>>>> just need
>>>>>>>>>>>>>>>> to check whether it covers all the needed query interfaces
>>> or
>>>>>>>> not),
>>>>>>>>>>>> we will
>>>>>>>>>>>>>>>> have a dedicated control program to
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no
>>> new
>>>>>>>>>>>> knowledge);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, we would provide a script set to drive this? My
>> question
>>>>>> is
>>>>>>>> more
>>>>>>>>>>>> about,
>>>>>>>>>>>>>>> how to work with docker-compose, rather than challanging
>> the
>>>>>>>> choice.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing
>>> repository
>>>>>>> and
>>>>>>>>>>>> codes);
>>>>>>>>>>>>>>>> (3) verify the actual data (by verification tool, a new
>>> tool,
>>>>>>>>> TODO);
>>>>>>>>>>>>>>>> (4) and determine the test result (success or failed);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>>>>> 下午5:27写道：
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build
>>> the
>>>>>>>> next
>>>>>>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking,
>>>>>> which
>>>>>>>> may
>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> be a
>>>>>>>>>>>>>>>>>> high priority but a necessity.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very
>>>>>> important
>>>>>>>>> role
>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> contribution process, and the current E2E tests just
>> work
>>>>>>> well
>>>>>>>> to
>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>> every single pull request before merging, so why bother
>> to
>>>>>>>> build
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> next
>>>>>>>>>>>>>>>>>> generation of E2E test framework?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
>>>>>> sub-projects
>>>>>>>>> that
>>>>>>>>>>>> vary
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to
>>>>>> help
>>>>>>> us
>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> pull requests, but we can see that they reimplement the
>>> E2E
>>>>>>>> test
>>>>>>>>>>>>>>>> framework
>>>>>>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t
>>> have
>>>>>>> E2E
>>>>>>>>>>>> tests at
>>>>>>>>>>>>>>>>>> all). Reimplementing the E2E test framework is
>> unnecesarry
>>>>>>> and
>>>>>>>>>>>>>>>> introduces
>>>>>>>>>>>>>>>>>> more duplicated work when adding a new test case. The
>>>>>>> ultimate
>>>>>>>>>>> goal
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> reuse all the test cases when needed.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running
>>>>>>>>>>> Java-based
>>>>>>>>>>>>>>>>>> program is not an easy thing for non-Java developers
>>>>>>>> (maintainers
>>>>>>>>>>> of
>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to
>>> set
>>>>>>> up
>>>>>>>>> the
>>>>>>>>>>>>>>>>>> environment correctly and then run the tests, neither is
>>> it
>>>>>>> an
>>>>>>>>>>> easy
>>>>>>>>>>>>>>>> skill
>>>>>>>>>>>>>>>>>> to debug the tests.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
>>>>>> because
>>>>>>> we
>>>>>>>>>>>> added
>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>> cases and many duplicated codes that need to be removed.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Therefore we're planning to build an unified,
>> easy-to-use,
>>>>>>> and
>>>>>>>>>>> fast
>>>>>>>>>>>> E2E
>>>>>>>>>>>>>>>>>> test framework to benifit all the sub-projects.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Here are some rough ideas about this framework:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 0. (Unchanged) We will continue to use
>> docker-compose.yaml
>>>>>> to
>>>>>>>>>>>>>>>> orchestrate
>>>>>>>>>>>>>>>>>> the services that are to be tested, it’s proved to be
>>>>>>> friendly
>>>>>>>>> and
>>>>>>>>>>>>>>>>>> debuggable because we can start it directly even if we
>> are
>>>>>>> not
>>>>>>>>>>>> writing
>>>>>>>>>>>>>>>> E2E
>>>>>>>>>>>>>>>>>> tests.
>>>>>>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI
>>>>>>> project
>>>>>>>> to
>>>>>>>>>>>> query
>>>>>>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
>>>>>>> versions
>>>>>>>>> of
>>>>>>>>>>>> query
>>>>>>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes
>>> and
>>>>>>> one
>>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> CLI).
>>>>>>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data
>>>>>> and
>>>>>>>> the
>>>>>>>>>>>> actual
>>>>>>>>>>>>>>>>>> data, we now have a custom schema of the expected data
>>> yaml
>>>>>>>> file,
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> verification codes should be packaged into an executable
>>>>>>>> command
>>>>>>>>>>> so
>>>>>>>>>>>>>>>> that it
>>>>>>>>>>>>>>>>>> can be executed standalone without Java and maven
>>>>>> knowledge.
>>>>>>> I
>>>>>>>>>>> hope
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> leverage existing standards to write the YAML files and
>> do
>>>>>>>>>>>>>>>> verifications.
>>>>>>>>>>>>>>>>>> 3. Build an orchestrating tool to start the
>>>>>>>>> `docker-compose.yaml`,
>>>>>>>>>>>>>>>> health
>>>>>>>>>>>>>>>>>> check, query actual data, and verify the result.
>>>>>>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions
>>> to
>>>>>>>> share
>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>> sub-projects.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If you have any other better idea or want to complement
>> to
>>>>>>> this
>>>>>>>>>>>>>>>> proposal,
>>>>>>>>>>>>>>>>>> please reply here. I will create an issue to track these
>>>>>>> tasks
>>>>>>>>>>>> later.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We have two students (who just finished the Summer Code
>>>>>> 2020
>>>>>>>>>>>> Projects)
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I
>>> said
>>>>>>>> this
>>>>>>>>>>> is
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> high priority, you have adequate time to get yourselves
>>>>>>>> familiar
>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
>>>>>>> SkyWalking
>>>>>>>>>>> work
>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> well
>>>>>>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and
>>> willingness
>>>>>>> to
>>>>>>>>>>> help.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1] https://github.com/Humbertzhang
>>>>>>>>>>>>>>>>>> [2] https://github.com/fgksgf/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> —————————
>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by Sheng Wu <wu...@gmail.com>.
It makes sense to make e2e driver and maybe validators in the INFRA repo.
Sheng Wu 吴晟
Twitter, wusheng1108
kezhenxu94@apache <ke...@apache.org> 于2020年12月26日周六 上午8:53写道：
> Because now we have skywalking-eyes repository to host the infra tools, we
> can put the new E2E framework there so that it doesn’t mess up the CLI
> repo. Here [1] I created a new directory to host the E2E codes.
>
> [1] https://github.com/apache/skywalking-eyes/pull/11
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94
>
> > On Dec 17, 2020, at 17:45, kezhenxu94@apache <ke...@apache.org>
> wrote:
> >
> > Hey, I’ve just drafted a design doc here [1], anyone interested in this
> please feel free to leave your comments, thanks!
> >
> > [1] https://github.com/apache/skywalking-website/pull/172
> >
> >> On Nov 26, 2020, at 14:41, Sheng Wu <wu...@gmail.com> wrote:
> >>
> >> You could learn from the Satellite project design,
> >> 1. Discuss here, https://github.com/apache/skywalking/issues/5871
> >> 2. Final design posted as a blog,
> >>
> http://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/
> >>
> >> People could learn from these things when you join the community in the
> >> future.
> >>
> >> Sheng Wu 吴晟
> >> Twitter, wusheng1108
> >>
> >>
> >> Hoshea Jiang <fg...@gmail.com> 于2020年11月26日周四 下午1:38写道：
> >>
> >>> I agree with the conclusion, and I hope to have some diagrams such as
> >>> flowcharts or data flow diagrams, which help provide an overview of
> the E2E
> >>> test before we start to do this.
> >>>
> >>> Ke Zhang <hu...@gmail.com> 于2020年11月26日周四 下午12:34写道：
> >>>
> >>>> I think supporting both docker-compose and KIND in the E2E test is
> >>>> reasonable and necessary, and I also agree with other conclusions.😀
> >>>>
> >>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月26日周四 上午11:58写道：
> >>>>
> >>>>>
> >>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
> >>>>>> 2. Totally replace the Java and Maven based driving system.
> >>>>>> 3. Enhance CLI to validate the GraphQL
> >>>>>> 4. Keep the agent test tool unchanged for now as it is already a
> >>>> separate
> >>>>>> system from the e2e.
> >>>>>
> >>>>> This conclusion is good for me.
> >>>>>
> >>>>> What are others opinions?
> >>>>>
> >>>>> —————————
> >>>>> Zhenxu Ke (柯振旭)
> >>>>> GitHub @kezhenxu94
> >>>>>
> >>>>>> On Nov 17, 2020, at 2:42 PM, Sheng Wu <wu...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>> Using k8s and docker-compose as 2 options in the test process is
> >>>>>> reasonable, and should be supported.
> >>>>>>
> >>>>>> Let's keep the conclusion as simple as possible. In the next
> >>> generation
> >>>>> e2e
> >>>>>> framework
> >>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
> >>>>>> 2. Totally replace the Java and Maven based driving system.
> >>>>>> 3. Enhance CLI to validate the GraphQL
> >>>>>> 4. Keep the agent test tool unchanged for now as it is already a
> >>>> separate
> >>>>>> system from the e2e.
> >>>>>>
> >>>>>> Are these good enough for everyone?
> >>>>>>
> >>>>>> Sheng Wu 吴晟
> >>>>>> Twitter, wusheng1108
> >>>>>>
> >>>>>>
> >>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
> >>>>>>
> >>>>>>>>
> >>>>>>>> We can see from above, your discussion actually is only related to
> >>> 3.
> >>>>>>>
> >>>>>>>
> >>>>>>> That's the only I intend to discuss here if you notice my first
> >>>>>>> response('May SWCK help about provision and demo") in this thread.
> >>> For
> >>>>>>> other parts, I don't have any tips to share, though.
> >>>>>>> Let me explain it more clearly. I want kubernetes to be seen as an
> >>>>>>> alternative to docker-compose to play a running-engine role in the
> >>>> test
> >>>>>>> framework. If kubernetes can't replace
> >>>>>>> the latter one Is it able to participate in it to solve the
> >>> dedicated
> >>>> or
> >>>>>>> special issues.
> >>>>>>>
> >>>>>>> Our e2e test and plugin test frameworks are already top-level
> >>>> complicity
> >>>>>>>> thing in the worldwide open source field. Please don't over-design
> >>>> it,
> >>>>>>> and
> >>>>>>>> too aggressive, the world will stay in the hybrid env for a very
> >>> long
> >>>>>>> time,
> >>>>>>>> same as the developer.
> >>>>>>>
> >>>>>>>
> >>>>>>> Reusing current elaborate output might be reasonable. But in this
> >>>>> thread,
> >>>>>>> we're talking about the potential solution to build a
> >>>>>>> next-generation framework, which might mean that the current
> >>> framework
> >>>>>>> is too complicated to maintain; we need a more tiny way to support
> >>>> more
> >>>>> and
> >>>>>>> more cases.
> >>>>>>>
> >>>>>>> And finally, SWCK is gonna and has to build a similar test
> >>> framework.
> >>>> If
> >>>>>>> the test framework opts for the kubernetes solution, we could share
> >>>> test
> >>>>>>> cases and the underlying framework. It's a more efficient
> >>>>>>> and consistent path for the entire system.
> >>>>>>>
> >>>>>>> regards, Hongtao.
> >>>>>>>
> >>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
> >>>>>>>
> >>>>>>>> Hi
> >>>>>>>>
> >>>>>>>> I don't think we have to make the decision on k8s or out-of-k8s.
> >>>>>>>> e2e and all integration tests have their scenarios like SkyWalking
> >>>> have
> >>>>>>>> multiple languages based and mesh-based, even Prometheus adoption.
> >>>>>>>> The same thing happens on the test field, k8s, and out of k8s.
> >>>>>>>>
> >>>>>>>> I would like to agree with both of you, k8s is needed, no matter
> in
> >>>>> mesh
> >>>>>>>> solution(ALS, telemetry v2), but also not anyone needs that, such
> >>> as
> >>>>>>> agent
> >>>>>>>> developer. We wouldn't deny the fact, the docker-compose is more
> >>>>>>>> lightweight and easier to learn.
> >>>>>>>> So, let me get this straight, for the test you need
> >>>>>>>> 1. Build the source from different repos
> >>>>>>>> 2. Build the images
> >>>>>>>> 3. Use some platform(docker-compose or k8s) to run the images in
> >>> one
> >>>>>>> piece.
> >>>>>>>> 4. Give some inputs to the env
> >>>>>>>> 5. Use some tools/ways to check what should happen according to
> >>> (4)'s
> >>>>>>> input
> >>>>>>>> 6. Checked or timeout failure.
> >>>>>>>>
> >>>>>>>> We can see from above, your discussion actually is only related to
> >>> 3.
> >>>>>>> Let's
> >>>>>>>> not see this as a battle, we need both. Many developers/users use
> >>> OAP
> >>>>> and
> >>>>>>>> SkyWalking out of k8s, that is a simple fact.
> >>>>>>>> Our e2e test and plugin test frameworks are already top-level
> >>>>> complicity
> >>>>>>>> thing in the worldwide open source field. Please don't over-design
> >>>> it,
> >>>>>>> and
> >>>>>>>> too aggressive, the world will stay in the hybrid env for a very
> >>> long
> >>>>>>> time,
> >>>>>>>> same as the developer.
> >>>>>>>>
> >>>>>>>> Also, let's keep in mind, k8s is popular becomes it doesn't
> require
> >>>> the
> >>>>>>>> developers to understand as the VM did. We as an open-source
> >>>> community,
> >>>>>>> are
> >>>>>>>> focusing on the developer's capabilities.
> >>>>>>>>
> >>>>>>>> Sheng Wu 吴晟
> >>>>>>>> Twitter, wusheng1108
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
> >>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> but I have been thinking whether or not it is overkill for
> >>> testing
> >>>>>>>>>> purposes to introduce Kubernetes things.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> we all know the fact docker-compose is more portable than
> >>>> Kubernetes,
> >>>>>>>>> friendly to local running. But there're also some benefits to
> >>>> replace
> >>>>>>> it
> >>>>>>>>> with kubernetes system:
> >>>>>>>>>
> >>>>>>>>> 1. SWCK has to test based on a real Kubernetes environment if the
> >>>> main
> >>>>>>>>> repo test framework doesn't support it(following the
> >>> docker-compose
> >>>>>>>> stack),
> >>>>>>>>> That means skywalking ecosystem MUST introduce kubernetes which
> is
> >>>> not
> >>>>>>> an
> >>>>>>>>> optional component.
> >>>>>>>>>
> >>>>>>>>> 2. Kubernetes support to scale applications at runtime, that help
> >>>>>>>> improve
> >>>>>>>>> the process of e2e. For instance, we could merge a single
> instance
> >>>> and
> >>>>>>>>> cluster test, by scaling up replicas. This strategy will reduce
> >>> the
> >>>>>>> total
> >>>>>>>>> time sharply.
> >>>>>>>>>
> >>>>>>>>> If we are going to use KIND, I hope we can give a script or
> >>>> something
> >>>>>>>>>> similar to install all the needed components and create cluster
> >>> for
> >>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That's SWCK's capacity to provide a standard and simple way to
> >>>> create
> >>>>> a
> >>>>>>>>> cluster in every environment,
> >>>>>>>>>
> >>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一
> >>> 下午7:00写道：
> >>>>>>>>>
> >>>>>>>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting
> >>>>>>>> Kubernetes-based
> >>>>>>>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND
> too,
> >>>>>>> but I
> >>>>>>>>>> have been thinking whether or not it is overkill for testing
> >>>> purposes
> >>>>>>>> to
> >>>>>>>>>> introduce Kubernetes things.
> >>>>>>>>>>
> >>>>>>>>>> Since you mentioned this, we are open to discuss this much more
> >>>>>>> deeply:
> >>>>>>>>>>
> >>>>>>>>>> It would be definitely a benefit to involve more “end”s in the
> >>>>>>> so-call
> >>>>>>>>>> “end to end” tests, I’m +1 to leverage SWCK as well.
> >>>>>>>>>>
> >>>>>>>>>> If we are going to use KIND, I hope we can give a script or
> >>>> something
> >>>>>>>>>> similar to install all the needed components and create cluster
> >>> for
> >>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> —————————
> >>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>
> >>>>>>>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <hanahmily@apache.org
> >
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> May SWCK help about provision and demo
> >>>>>>>>>>>
> >>>>>>>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone
> >>>>>>>> instance
> >>>>>>>>>> or
> >>>>>>>>>>> cluster for e2e. We could compose indicated CR yamls for
> >>> different
> >>>>>>>>>>> scenarios.
> >>>>>>>>>>> 2. Different demo for different cases is mandatory for e2e and
> >>>>>>> swck,
> >>>>>>>> we
> >>>>>>>>>>> could build a group of demo projects for them. With
> Kubernetes's
> >>>>>>>>> support,
> >>>>>>>>>>> we could
> >>>>>>>>>>> get free of verbose scripts that are written for installation,
> >>>>>>>>>> checking
> >>>>>>>>>>> the state of running applications, collecting logs.
> >>>>>>>>>>>
> >>>>>>>>>>> Ppl might have concerts about the minkube we currently have,
> >>>>>>> which's
> >>>>>>>>> hard
> >>>>>>>>>>> to debug and runs slowly.
> >>>>>>>>>>>
> >>>>>>>>>>> I prefer to kind[1] instead of minkube. From my experience,
> kind
> >>>>>>>> works
> >>>>>>>>>> fine
> >>>>>>>>>>> in an e2e scenario(limited cpu, memory resources), and it is
> run
> >>>>>>>>> million
> >>>>>>>>>>> times
> >>>>>>>>>>> on my team's e2e environment, which proves the stability and
> >>>>>>>>>>> maintainability of it.
> >>>>>>>>>>>
> >>>>>>>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
> >>>>>>>>>>>
> >>>>>>>>>>> regards, Hongtao
> >>>>>>>>>>>
> >>>>>>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
> >>>>>>>>>>>
> >>>>>>>>>>>> Inline
> >>>>>>>>>>>>
> >>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> >>>>>>> 下午11:16写道：
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Jiajing,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for sharing your context and joining the discussion.
> >>> It’s
> >>>>>>>> true
> >>>>>>>>>>>> that
> >>>>>>>>>>>>> we’re actually tackling the same problems in this domain.
> Some
> >>>> of
> >>>>>>>> the
> >>>>>>>>>>>>> issues you mentioned are solved in the current E2E framework
> >>>>>>> (like
> >>>>>>>>>>>> `retry`,
> >>>>>>>>>>>>> `timeout`, etc.), while some others are still there without
> >>>> ideal
> >>>>>>>>>>>> solution.
> >>>>>>>>>>>>> Other discussions, please see my comments inline.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> - the first issue is useful in health check steps, just like
> >>>> the
> >>>>>>>>> ones
> >>>>>>>>>>>>>> defined in readiness probe and liveness probe in kubelet.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes, the reason why we decided to pick docker-compose is
> that,
> >>>> it
> >>>>>>>> has
> >>>>>>>>>> the
> >>>>>>>>>>>>> same functionality with kubelet in terms of dependent
> services
> >>>>>>>>> startup
> >>>>>>>>>>>>> order and healthiness checking, while it’s more lite-weight
> >>> and
> >>>>>>>> easy
> >>>>>>>>>> for
> >>>>>>>>>>>>> both developers and GitHub Actions environment. Right?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> - the second issue is critical for highly-customized
> >>>> assertions,
> >>>>>>>> we
> >>>>>>>>>>>>>> can leverage the `text/template` module to provide highly
> >>>>>>>> extensive
> >>>>>>>>>>>>>> assertion functions
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> `text/template` is also the first thought that came to my
> mind
> >>>>>>> when
> >>>>>>>>>>>>> considering customized assertors, customized assertors are
> for
> >>>>>>> sure
> >>>>>>>>>>>>> critical in SkyWalking E2E tests because we have many kinds
> of
> >>>>>>>>>>>> assertions,
> >>>>>>>>>>>>> some of which are not foreseen until we actually need them,
> >>>>>>>>>>>>> `AtLeastOneGreaterThanZero` is an example.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> - the last but not the least, the lifecycle hooks can be
> used
> >>>>>>> for
> >>>>>>>>>>>>>> accurate control of test setup and teardown, such as
> >>>>>>>> `docker-compose
> >>>>>>>>>>>>>> build, up -d, down`
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The lifecycle hooks are what I didn’t take into consideration
> >>>>>>>>> because I
> >>>>>>>>>>>>> thought it is quite natural for test framework to provide
> this
> >>>>>>>>>> features,
> >>>>>>>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Together with the aforementioned ideas, we will have a
> >>>>>>> go-written,
> >>>>>>>>>>>>>> yaml-data driven e2e testing framework which can overcome
> the
> >>>>>>>>> current
> >>>>>>>>>>>>>> problems that occur,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I agree.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> in which the most annoying issue is the fragmentation of
> >>>>>>> different
> >>>>>>>>>>>>>> scripts in different submodules, IMO.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> That’s why I list the last item in the previous email, (Nice
> >>> to
> >>>>>>>> have,
> >>>>>>>>>>>> wrap
> >>>>>>>>>>>>> them into a GitHub Action), with that GitHub Action, we can
> >>>>>>> simply
> >>>>>>>>>>>>> configure a workflow/job in every submodule, and the tests
> are
> >>>>>>>>> executed
> >>>>>>>>>>>>> with the shared scripts, no copy-pasting.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Note, GitHub Action will require an Apache release every time
> >>>>>>>> because
> >>>>>>>>> we
> >>>>>>>>>>>> are distributing the binary.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sheng Wu 吴晟
> >>>>>>>>>>>> Twitter, wusheng1108
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>> Jiajing
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
> >>>>>>>>>>>> kezhenxu94@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> So, we would provide a script set to drive this?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> My question is more about,
> >>>>>>>>>>>>>>>> how to work with docker-compose
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It’s the same as what we’re doing now in the current
> version
> >>>> of
> >>>>>>>> E2E
> >>>>>>>>>>>>> tests, now we have many `docker-compose.yaml` files, and all
> >>> the
> >>>>>>>>>>>>> `docker-compose.yaml` files are reusable in the new
> framework,
> >>>> we
> >>>>>>>>> will
> >>>>>>>>>>>> keep
> >>>>>>>>>>>>> this mechanism as it is proved to be convenient and easy to
> >>> use
> >>>>>>>>>> (remember
> >>>>>>>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage
> >>>> E2E
> >>>>>>>>> test,
> >>>>>>>>>>>> not
> >>>>>>>>>>>>> yet merged though, it’s fast/easy to add a new case).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> —————————
> >>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
> >>>>>>>> wu.sheng.841108@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
> >>>>>>>>> 下午9:16写道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I want to ask, how the docker-compose lands in your
> >>> mind? I
> >>>>>>>>> think
> >>>>>>>>>>>> CLI
> >>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>> control this part.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> docker-compose just fits the requirements perfectly that
> >>> we
> >>>>>>>> need,
> >>>>>>>>>> we
> >>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>> been using it in the current E2E and turns out it has
> >>>>>>>> advantages
> >>>>>>>>> in
> >>>>>>>>>>>>>>>>> starting many services correctly with health check and
> >>>>>>>> dependency
> >>>>>>>>>>>>>>>>> declaration. (One may say Kubernetes can do the same
> >>> thing,
> >>>>>>> but
> >>>>>>>>>> it’s
> >>>>>>>>>>>>> kind
> >>>>>>>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
> >>>>>>>> related
> >>>>>>>>>>>>> things,
> >>>>>>>>>>>>>>>>> it just provides query ability (the same as what it is
> >>> doing
> >>>>>>>> now,
> >>>>>>>>>>>>> just need
> >>>>>>>>>>>>>>>>> to check whether it covers all the needed query
> interfaces
> >>>> or
> >>>>>>>>> not),
> >>>>>>>>>>>>> we will
> >>>>>>>>>>>>>>>>> have a dedicated control program to
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no
> >>>> new
> >>>>>>>>>>>>> knowledge);
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> So, we would provide a script set to drive this? My
> >>> question
> >>>>>>> is
> >>>>>>>>> more
> >>>>>>>>>>>>> about,
> >>>>>>>>>>>>>>>> how to work with docker-compose, rather than challanging
> >>> the
> >>>>>>>>> choice.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Sheng Wu 吴晟
> >>>>>>>>>>>>>>>> Twitter, wusheng1108
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing
> >>>> repository
> >>>>>>>> and
> >>>>>>>>>>>>> codes);
> >>>>>>>>>>>>>>>>> (3) verify the actual data (by verification tool, a new
> >>>> tool,
> >>>>>>>>>> TODO);
> >>>>>>>>>>>>>>>>> (4) and determine the test result (success or failed);
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> —————————
> >>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Sheng Wu 吴晟
> >>>>>>>>>>>>>>>>>> Twitter, wusheng1108
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org>
> 于2020年11月15日周日
> >>>>>>>>>> 下午5:27写道：
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build
> >>>> the
> >>>>>>>>> next
> >>>>>>>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking,
> >>>>>>> which
> >>>>>>>>> may
> >>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>> be a
> >>>>>>>>>>>>>>>>>>> high priority but a necessity.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very
> >>>>>>> important
> >>>>>>>>>> role
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> contribution process, and the current E2E tests just
> >>> work
> >>>>>>>> well
> >>>>>>>>> to
> >>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>> every single pull request before merging, so why bother
> >>> to
> >>>>>>>>> build
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> next
> >>>>>>>>>>>>>>>>>>> generation of E2E test framework?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
> >>>>>>> sub-projects
> >>>>>>>>>> that
> >>>>>>>>>>>>> vary
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to
> >>>>>>> help
> >>>>>>>> us
> >>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> pull requests, but we can see that they reimplement the
> >>>> E2E
> >>>>>>>>> test
> >>>>>>>>>>>>>>>>> framework
> >>>>>>>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t
> >>>> have
> >>>>>>>> E2E
> >>>>>>>>>>>>> tests at
> >>>>>>>>>>>>>>>>>>> all). Reimplementing the E2E test framework is
> >>> unnecesarry
> >>>>>>>> and
> >>>>>>>>>>>>>>>>> introduces
> >>>>>>>>>>>>>>>>>>> more duplicated work when adding a new test case. The
> >>>>>>>> ultimate
> >>>>>>>>>>>> goal
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> reuse all the test cases when needed.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2. The current E2E framework is built with Java,
> running
> >>>>>>>>>>>> Java-based
> >>>>>>>>>>>>>>>>>>> program is not an easy thing for non-Java developers
> >>>>>>>>> (maintainers
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to
> >>>> set
> >>>>>>>> up
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> environment correctly and then run the tests, neither
> is
> >>>> it
> >>>>>>>> an
> >>>>>>>>>>>> easy
> >>>>>>>>>>>>>>>>> skill
> >>>>>>>>>>>>>>>>>>> to debug the tests.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
> >>>>>>> because
> >>>>>>>> we
> >>>>>>>>>>>>> added
> >>>>>>>>>>>>>>>>> many
> >>>>>>>>>>>>>>>>>>> cases and many duplicated codes that need to be
> removed.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Therefore we're planning to build an unified,
> >>> easy-to-use,
> >>>>>>>> and
> >>>>>>>>>>>> fast
> >>>>>>>>>>>>> E2E
> >>>>>>>>>>>>>>>>>>> test framework to benifit all the sub-projects.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Here are some rough ideas about this framework:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 0. (Unchanged) We will continue to use
> >>> docker-compose.yaml
> >>>>>>> to
> >>>>>>>>>>>>>>>>> orchestrate
> >>>>>>>>>>>>>>>>>>> the services that are to be tested, it’s proved to be
> >>>>>>>> friendly
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> debuggable because we can start it directly even if we
> >>> are
> >>>>>>>> not
> >>>>>>>>>>>>> writing
> >>>>>>>>>>>>>>>>> E2E
> >>>>>>>>>>>>>>>>>>> tests.
> >>>>>>>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI
> >>>>>>>> project
> >>>>>>>>> to
> >>>>>>>>>>>>> query
> >>>>>>>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
> >>>>>>>> versions
> >>>>>>>>>> of
> >>>>>>>>>>>>> query
> >>>>>>>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes
> >>>> and
> >>>>>>>> one
> >>>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> CLI).
> >>>>>>>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected
> data
> >>>>>>> and
> >>>>>>>>> the
> >>>>>>>>>>>>> actual
> >>>>>>>>>>>>>>>>>>> data, we now have a custom schema of the expected data
> >>>> yaml
> >>>>>>>>> file,
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> verification codes should be packaged into an
> executable
> >>>>>>>>> command
> >>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>> that it
> >>>>>>>>>>>>>>>>>>> can be executed standalone without Java and maven
> >>>>>>> knowledge.
> >>>>>>>> I
> >>>>>>>>>>>> hope
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> leverage existing standards to write the YAML files and
> >>> do
> >>>>>>>>>>>>>>>>> verifications.
> >>>>>>>>>>>>>>>>>>> 3. Build an orchestrating tool to start the
> >>>>>>>>>> `docker-compose.yaml`,
> >>>>>>>>>>>>>>>>> health
> >>>>>>>>>>>>>>>>>>> check, query actual data, and verify the result.
> >>>>>>>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub
> Actions
> >>>> to
> >>>>>>>>> share
> >>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>> sub-projects.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> If you have any other better idea or want to complement
> >>> to
> >>>>>>>> this
> >>>>>>>>>>>>>>>>> proposal,
> >>>>>>>>>>>>>>>>>>> please reply here. I will create an issue to track
> these
> >>>>>>>> tasks
> >>>>>>>>>>>>> later.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> We have two students (who just finished the Summer Code
> >>>>>>> 2020
> >>>>>>>>>>>>> Projects)
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I
> >>>> said
> >>>>>>>>> this
> >>>>>>>>>>>> is
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>> high priority, you have adequate time to get yourselves
> >>>>>>>>> familiar
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
> >>>>>>>> SkyWalking
> >>>>>>>>>>>> work
> >>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> well
> >>>>>>>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and
> >>>> willingness
> >>>>>>>> to
> >>>>>>>>>>>> help.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> [1] https://github.com/Humbertzhang
> >>>>>>>>>>>>>>>>>>> [2] https://github.com/fgksgf/
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> —————————
> >>>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> —————————
> >>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> —————————
> >>>>>>>>>>>>> Zhenxu Ke (柯振旭)
> >>>>>>>>>>>>> GitHub @kezhenxu94
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >
> >
> >
> > —————————
> > Zhenxu Ke (柯振旭)
> > GitHub @kezhenxu94
>
>
Re: [DISCUSSION] Build the next generation of E2E test framework
Posted by "kezhenxu94@apache" <ke...@apache.org>.
Because now we have skywalking-eyes repository to host the infra tools, we can put the new E2E framework there so that it doesn’t mess up the CLI repo. Here [1] I created a new directory to host the E2E codes.
[1] https://github.com/apache/skywalking-eyes/pull/11
—————————
Zhenxu Ke (柯振旭)
GitHub @kezhenxu94
> On Dec 17, 2020, at 17:45, kezhenxu94@apache <ke...@apache.org> wrote:
>
> Hey, I’ve just drafted a design doc here [1], anyone interested in this please feel free to leave your comments, thanks!
>
> [1] https://github.com/apache/skywalking-website/pull/172
>
>> On Nov 26, 2020, at 14:41, Sheng Wu <wu...@gmail.com> wrote:
>>
>> You could learn from the Satellite project design,
>> 1. Discuss here, https://github.com/apache/skywalking/issues/5871
>> 2. Final design posted as a blog,
>> http://skywalking.apache.org/blog/2020-11-25-skywalking-satellite-0.1.0-design/
>>
>> People could learn from these things when you join the community in the
>> future.
>>
>> Sheng Wu 吴晟
>> Twitter, wusheng1108
>>
>>
>> Hoshea Jiang <fg...@gmail.com> 于2020年11月26日周四 下午1:38写道：
>>
>>> I agree with the conclusion, and I hope to have some diagrams such as
>>> flowcharts or data flow diagrams, which help provide an overview of the E2E
>>> test before we start to do this.
>>>
>>> Ke Zhang <hu...@gmail.com> 于2020年11月26日周四 下午12:34写道：
>>>
>>>> I think supporting both docker-compose and KIND in the E2E test is
>>>> reasonable and necessary, and I also agree with other conclusions.😀
>>>>
>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月26日周四 上午11:58写道：
>>>>
>>>>>
>>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
>>>>>> 2. Totally replace the Java and Maven based driving system.
>>>>>> 3. Enhance CLI to validate the GraphQL
>>>>>> 4. Keep the agent test tool unchanged for now as it is already a
>>>> separate
>>>>>> system from the e2e.
>>>>>
>>>>> This conclusion is good for me.
>>>>>
>>>>> What are others opinions?
>>>>>
>>>>> —————————
>>>>> Zhenxu Ke (柯振旭)
>>>>> GitHub @kezhenxu94
>>>>>
>>>>>> On Nov 17, 2020, at 2:42 PM, Sheng Wu <wu...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Using k8s and docker-compose as 2 options in the test process is
>>>>>> reasonable, and should be supported.
>>>>>>
>>>>>> Let's keep the conclusion as simple as possible. In the next
>>> generation
>>>>> e2e
>>>>>> framework
>>>>>> 1. Docker-compose and kind are 2 options to set up the environments.
>>>>>> 2. Totally replace the Java and Maven based driving system.
>>>>>> 3. Enhance CLI to validate the GraphQL
>>>>>> 4. Keep the agent test tool unchanged for now as it is already a
>>>> separate
>>>>>> system from the e2e.
>>>>>>
>>>>>> Are these good enough for everyone?
>>>>>>
>>>>>> Sheng Wu 吴晟
>>>>>> Twitter, wusheng1108
>>>>>>
>>>>>>
>>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 下午12:41写道：
>>>>>>
>>>>>>>>
>>>>>>>> We can see from above, your discussion actually is only related to
>>> 3.
>>>>>>>
>>>>>>>
>>>>>>> That's the only I intend to discuss here if you notice my first
>>>>>>> response('May SWCK help about provision and demo") in this thread.
>>> For
>>>>>>> other parts, I don't have any tips to share, though.
>>>>>>> Let me explain it more clearly. I want kubernetes to be seen as an
>>>>>>> alternative to docker-compose to play a running-engine role in the
>>>> test
>>>>>>> framework. If kubernetes can't replace
>>>>>>> the latter one Is it able to participate in it to solve the
>>> dedicated
>>>> or
>>>>>>> special issues.
>>>>>>>
>>>>>>> Our e2e test and plugin test frameworks are already top-level
>>>> complicity
>>>>>>>> thing in the worldwide open source field. Please don't over-design
>>>> it,
>>>>>>> and
>>>>>>>> too aggressive, the world will stay in the hybrid env for a very
>>> long
>>>>>>> time,
>>>>>>>> same as the developer.
>>>>>>>
>>>>>>>
>>>>>>> Reusing current elaborate output might be reasonable. But in this
>>>>> thread,
>>>>>>> we're talking about the potential solution to build a
>>>>>>> next-generation framework, which might mean that the current
>>> framework
>>>>>>> is too complicated to maintain; we need a more tiny way to support
>>>> more
>>>>> and
>>>>>>> more cases.
>>>>>>>
>>>>>>> And finally, SWCK is gonna and has to build a similar test
>>> framework.
>>>> If
>>>>>>> the test framework opts for the kubernetes solution, we could share
>>>> test
>>>>>>> cases and the underlying framework. It's a more efficient
>>>>>>> and consistent path for the entire system.
>>>>>>>
>>>>>>> regards, Hongtao.
>>>>>>>
>>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月17日周二 上午10:33写道：
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I don't think we have to make the decision on k8s or out-of-k8s.
>>>>>>>> e2e and all integration tests have their scenarios like SkyWalking
>>>> have
>>>>>>>> multiple languages based and mesh-based, even Prometheus adoption.
>>>>>>>> The same thing happens on the test field, k8s, and out of k8s.
>>>>>>>>
>>>>>>>> I would like to agree with both of you, k8s is needed, no matter in
>>>>> mesh
>>>>>>>> solution(ALS, telemetry v2), but also not anyone needs that, such
>>> as
>>>>>>> agent
>>>>>>>> developer. We wouldn't deny the fact, the docker-compose is more
>>>>>>>> lightweight and easier to learn.
>>>>>>>> So, let me get this straight, for the test you need
>>>>>>>> 1. Build the source from different repos
>>>>>>>> 2. Build the images
>>>>>>>> 3. Use some platform(docker-compose or k8s) to run the images in
>>> one
>>>>>>> piece.
>>>>>>>> 4. Give some inputs to the env
>>>>>>>> 5. Use some tools/ways to check what should happen according to
>>> (4)'s
>>>>>>> input
>>>>>>>> 6. Checked or timeout failure.
>>>>>>>>
>>>>>>>> We can see from above, your discussion actually is only related to
>>> 3.
>>>>>>> Let's
>>>>>>>> not see this as a battle, we need both. Many developers/users use
>>> OAP
>>>>> and
>>>>>>>> SkyWalking out of k8s, that is a simple fact.
>>>>>>>> Our e2e test and plugin test frameworks are already top-level
>>>>> complicity
>>>>>>>> thing in the worldwide open source field. Please don't over-design
>>>> it,
>>>>>>> and
>>>>>>>> too aggressive, the world will stay in the hybrid env for a very
>>> long
>>>>>>> time,
>>>>>>>> same as the developer.
>>>>>>>>
>>>>>>>> Also, let's keep in mind, k8s is popular becomes it doesn't require
>>>> the
>>>>>>>> developers to understand as the VM did. We as an open-source
>>>> community,
>>>>>>> are
>>>>>>>> focusing on the developer's capabilities.
>>>>>>>>
>>>>>>>> Sheng Wu 吴晟
>>>>>>>> Twitter, wusheng1108
>>>>>>>>
>>>>>>>>
>>>>>>>> Hongtao Gao <ha...@apache.org> 于2020年11月17日周二 上午12:07写道：
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> but I have been thinking whether or not it is overkill for
>>> testing
>>>>>>>>>> purposes to introduce Kubernetes things.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> we all know the fact docker-compose is more portable than
>>>> Kubernetes,
>>>>>>>>> friendly to local running. But there're also some benefits to
>>>> replace
>>>>>>> it
>>>>>>>>> with kubernetes system:
>>>>>>>>>
>>>>>>>>> 1. SWCK has to test based on a real Kubernetes environment if the
>>>> main
>>>>>>>>> repo test framework doesn't support it(following the
>>> docker-compose
>>>>>>>> stack),
>>>>>>>>> That means skywalking ecosystem MUST introduce kubernetes which is
>>>> not
>>>>>>> an
>>>>>>>>> optional component.
>>>>>>>>>
>>>>>>>>> 2. Kubernetes support to scale applications at runtime, that help
>>>>>>>> improve
>>>>>>>>> the process of e2e. For instance, we could merge a single instance
>>>> and
>>>>>>>>> cluster test, by scaling up replicas. This strategy will reduce
>>> the
>>>>>>> total
>>>>>>>>> time sharply.
>>>>>>>>>
>>>>>>>>> If we are going to use KIND, I hope we can give a script or
>>>> something
>>>>>>>>>> similar to install all the needed components and create cluster
>>> for
>>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That's SWCK's capacity to provide a standard and simple way to
>>>> create
>>>>> a
>>>>>>>>> cluster in every environment,
>>>>>>>>>
>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月16日周一
>>> 下午7:00写道：
>>>>>>>>>
>>>>>>>>>> Thanks Hongtao’s suggestion, I’ve thought about adopting
>>>>>>>> Kubernetes-based
>>>>>>>>>> tech stack (e.g. minikube or KIND), I personally prefer KIND too,
>>>>>>> but I
>>>>>>>>>> have been thinking whether or not it is overkill for testing
>>>> purposes
>>>>>>>> to
>>>>>>>>>> introduce Kubernetes things.
>>>>>>>>>>
>>>>>>>>>> Since you mentioned this, we are open to discuss this much more
>>>>>>> deeply:
>>>>>>>>>>
>>>>>>>>>> It would be definitely a benefit to involve more “end”s in the
>>>>>>> so-call
>>>>>>>>>> “end to end” tests, I’m +1 to leverage SWCK as well.
>>>>>>>>>>
>>>>>>>>>> If we are going to use KIND, I hope we can give a script or
>>>> something
>>>>>>>>>> similar to install all the needed components and create cluster
>>> for
>>>>>>>>>> convenient testing locally, Ke Zhang and Huaxi.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> —————————
>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>
>>>>>>>>>>> On Nov 16, 2020, at 4:03 PM, Hongtao Gao <ha...@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> May SWCK help about provision and demo
>>>>>>>>>>>
>>>>>>>>>>> 1. SWCK is able to provide a stable and healthy OAP standalone
>>>>>>>> instance
>>>>>>>>>> or
>>>>>>>>>>> cluster for e2e. We could compose indicated CR yamls for
>>> different
>>>>>>>>>>> scenarios.
>>>>>>>>>>> 2. Different demo for different cases is mandatory for e2e and
>>>>>>> swck,
>>>>>>>> we
>>>>>>>>>>> could build a group of demo projects for them. With Kubernetes's
>>>>>>>>> support,
>>>>>>>>>>> we could
>>>>>>>>>>> get free of verbose scripts that are written for installation,
>>>>>>>>>> checking
>>>>>>>>>>> the state of running applications, collecting logs.
>>>>>>>>>>>
>>>>>>>>>>> Ppl might have concerts about the minkube we currently have,
>>>>>>> which's
>>>>>>>>> hard
>>>>>>>>>>> to debug and runs slowly.
>>>>>>>>>>>
>>>>>>>>>>> I prefer to kind[1] instead of minkube. From my experience, kind
>>>>>>>> works
>>>>>>>>>> fine
>>>>>>>>>>> in an e2e scenario(limited cpu, memory resources), and it is run
>>>>>>>>> million
>>>>>>>>>>> times
>>>>>>>>>>> on my team's e2e environment, which proves the stability and
>>>>>>>>>>> maintainability of it.
>>>>>>>>>>>
>>>>>>>>>>> * 1.https://kind.sigs.k8s.io/docs/user/quick-start/
>>>>>>>>>>>
>>>>>>>>>>> regards, Hongtao
>>>>>>>>>>>
>>>>>>>>>>> Sheng Wu <wu...@gmail.com> 于2020年11月16日周一 上午8:30写道：
>>>>>>>>>>>
>>>>>>>>>>>> Inline
>>>>>>>>>>>>
>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>>> 下午11:16写道：
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Jiajing,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for sharing your context and joining the discussion.
>>> It’s
>>>>>>>> true
>>>>>>>>>>>> that
>>>>>>>>>>>>> we’re actually tackling the same problems in this domain. Some
>>>> of
>>>>>>>> the
>>>>>>>>>>>>> issues you mentioned are solved in the current E2E framework
>>>>>>> (like
>>>>>>>>>>>> `retry`,
>>>>>>>>>>>>> `timeout`, etc.), while some others are still there without
>>>> ideal
>>>>>>>>>>>> solution.
>>>>>>>>>>>>> Other discussions, please see my comments inline.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> - the first issue is useful in health check steps, just like
>>>> the
>>>>>>>>> ones
>>>>>>>>>>>>>> defined in readiness probe and liveness probe in kubelet.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, the reason why we decided to pick docker-compose is that,
>>>> it
>>>>>>>> has
>>>>>>>>>> the
>>>>>>>>>>>>> same functionality with kubelet in terms of dependent services
>>>>>>>>> startup
>>>>>>>>>>>>> order and healthiness checking, while it’s more lite-weight
>>> and
>>>>>>>> easy
>>>>>>>>>> for
>>>>>>>>>>>>> both developers and GitHub Actions environment. Right?
>>>>>>>>>>>>>
>>>>>>>>>>>>>> - the second issue is critical for highly-customized
>>>> assertions,
>>>>>>>> we
>>>>>>>>>>>>>> can leverage the `text/template` module to provide highly
>>>>>>>> extensive
>>>>>>>>>>>>>> assertion functions
>>>>>>>>>>>>>
>>>>>>>>>>>>> `text/template` is also the first thought that came to my mind
>>>>>>> when
>>>>>>>>>>>>> considering customized assertors, customized assertors are for
>>>>>>> sure
>>>>>>>>>>>>> critical in SkyWalking E2E tests because we have many kinds of
>>>>>>>>>>>> assertions,
>>>>>>>>>>>>> some of which are not foreseen until we actually need them,
>>>>>>>>>>>>> `AtLeastOneGreaterThanZero` is an example.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> - the last but not the least, the lifecycle hooks can be used
>>>>>>> for
>>>>>>>>>>>>>> accurate control of test setup and teardown, such as
>>>>>>>> `docker-compose
>>>>>>>>>>>>>> build, up -d, down`
>>>>>>>>>>>>>
>>>>>>>>>>>>> The lifecycle hooks are what I didn’t take into consideration
>>>>>>>>> because I
>>>>>>>>>>>>> thought it is quite natural for test framework to provide this
>>>>>>>>>> features,
>>>>>>>>>>>>> maybe Ke Zhang and Huaxi can do some research on this part.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Together with the aforementioned ideas, we will have a
>>>>>>> go-written,
>>>>>>>>>>>>>> yaml-data driven e2e testing framework which can overcome the
>>>>>>>>> current
>>>>>>>>>>>>>> problems that occur,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> in which the most annoying issue is the fragmentation of
>>>>>>> different
>>>>>>>>>>>>>> scripts in different submodules, IMO.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That’s why I list the last item in the previous email, (Nice
>>> to
>>>>>>>> have,
>>>>>>>>>>>> wrap
>>>>>>>>>>>>> them into a GitHub Action), with that GitHub Action, we can
>>>>>>> simply
>>>>>>>>>>>>> configure a workflow/job in every submodule, and the tests are
>>>>>>>>> executed
>>>>>>>>>>>>> with the shared scripts, no copy-pasting.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Note, GitHub Action will require an Apache release every time
>>>>>>>> because
>>>>>>>>> we
>>>>>>>>>>>> are distributing the binary.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Jiajing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Nov 15, 2020 at 9:43 PM kezhenxu94@apache <
>>>>>>>>>>>> kezhenxu94@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So, we would provide a script set to drive this?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, a script or a GoLang-written CLI, it depends.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My question is more about,
>>>>>>>>>>>>>>>> how to work with docker-compose
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It’s the same as what we’re doing now in the current version
>>>> of
>>>>>>>> E2E
>>>>>>>>>>>>> tests, now we have many `docker-compose.yaml` files, and all
>>> the
>>>>>>>>>>>>> `docker-compose.yaml` files are reusable in the new framework,
>>>> we
>>>>>>>>> will
>>>>>>>>>>>> keep
>>>>>>>>>>>>> this mechanism as it is proved to be convenient and easy to
>>> use
>>>>>>>>>> (remember
>>>>>>>>>>>>> when we added an ElasticSearch 7.9 case, and the TiDB storage
>>>> E2E
>>>>>>>>> test,
>>>>>>>>>>>> not
>>>>>>>>>>>>> yet merged though, it’s fast/easy to add a new case).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Nov 15, 2020, at 9:29 PM, Sheng Wu <
>>>>>>>> wu.sheng.841108@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>>>>> 下午9:16写道：
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I want to ask, how the docker-compose lands in your
>>> mind? I
>>>>>>>>> think
>>>>>>>>>>>> CLI
>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>> control this part.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> docker-compose just fits the requirements perfectly that
>>> we
>>>>>>>> need,
>>>>>>>>>> we
>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> been using it in the current E2E and turns out it has
>>>>>>>> advantages
>>>>>>>>> in
>>>>>>>>>>>>>>>>> starting many services correctly with health check and
>>>>>>>> dependency
>>>>>>>>>>>>>>>>> declaration. (One may say Kubernetes can do the same
>>> thing,
>>>>>>> but
>>>>>>>>>> it’s
>>>>>>>>>>>>> kind
>>>>>>>>>>>>>>>>> of heavy in E2E and GitHub Actions and complex).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The SkyWalking-CLI won’t get involved with docker-compose
>>>>>>>> related
>>>>>>>>>>>>> things,
>>>>>>>>>>>>>>>>> it just provides query ability (the same as what it is
>>> doing
>>>>>>>> now,
>>>>>>>>>>>>> just need
>>>>>>>>>>>>>>>>> to check whether it covers all the needed query interfaces
>>>> or
>>>>>>>>> not),
>>>>>>>>>>>>> we will
>>>>>>>>>>>>>>>>> have a dedicated control program to
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (1) start docker-compose (existing standard mechanism, no
>>>> new
>>>>>>>>>>>>> knowledge);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So, we would provide a script set to drive this? My
>>> question
>>>>>>> is
>>>>>>>>> more
>>>>>>>>>>>>> about,
>>>>>>>>>>>>>>>> how to work with docker-compose, rather than challanging
>>> the
>>>>>>>>> choice.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (2) query actual data (by SkyWalking-CLI, existing
>>>> repository
>>>>>>>> and
>>>>>>>>>>>>> codes);
>>>>>>>>>>>>>>>>> (3) verify the actual data (by verification tool, a new
>>>> tool,
>>>>>>>>>> TODO);
>>>>>>>>>>>>>>>>> (4) and determine the test result (success or failed);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sheng Wu 吴晟
>>>>>>>>>>>>>>>>>> Twitter, wusheng1108
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> kezhenxu94@apache <ke...@apache.org> 于2020年11月15日周日
>>>>>>>>>> 下午5:27写道：
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi the community, I want to raise a discussion to build
>>>> the
>>>>>>>>> next
>>>>>>>>>>>>>>>>>>> generation of E2E test framework for Apache SkyWalking,
>>>>>>> which
>>>>>>>>> may
>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> be a
>>>>>>>>>>>>>>>>>>> high priority but a necessity.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> As we already know, tests in SkyWalking play a very
>>>>>>> important
>>>>>>>>>> role
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> contribution process, and the current E2E tests just
>>> work
>>>>>>>> well
>>>>>>>>> to
>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>>>> every single pull request before merging, so why bother
>>> to
>>>>>>>>> build
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> next
>>>>>>>>>>>>>>>>>>> generation of E2E test framework?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1. In the SkyWalking's ecosystem, there are many
>>>>>>> sub-projects
>>>>>>>>>> that
>>>>>>>>>>>>> vary
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> terms of language, almost all of them need E2E tests to
>>>>>>> help
>>>>>>>> us
>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> pull requests, but we can see that they reimplement the
>>>> E2E
>>>>>>>>> test
>>>>>>>>>>>>>>>>> framework
>>>>>>>>>>>>>>>>>>> in their languages, (or even worse, some of them don’t
>>>> have
>>>>>>>> E2E
>>>>>>>>>>>>> tests at
>>>>>>>>>>>>>>>>>>> all). Reimplementing the E2E test framework is
>>> unnecesarry
>>>>>>>> and
>>>>>>>>>>>>>>>>> introduces
>>>>>>>>>>>>>>>>>>> more duplicated work when adding a new test case. The
>>>>>>>> ultimate
>>>>>>>>>>>> goal
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> reuse all the test cases when needed.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2. The current E2E framework is built with Java, running
>>>>>>>>>>>> Java-based
>>>>>>>>>>>>>>>>>>> program is not an easy thing for non-Java developers
>>>>>>>>> (maintainers
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>> sub-projects such as PHP, C++, LUA, etc.), they need to
>>>> set
>>>>>>>> up
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> environment correctly and then run the tests, neither is
>>>> it
>>>>>>>> an
>>>>>>>>>>>> easy
>>>>>>>>>>>>>>>>> skill
>>>>>>>>>>>>>>>>>>> to debug the tests.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 3. It’s a good opportunity to optimize the E2E tests
>>>>>>> because
>>>>>>>> we
>>>>>>>>>>>>> added
>>>>>>>>>>>>>>>>> many
>>>>>>>>>>>>>>>>>>> cases and many duplicated codes that need to be removed.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Therefore we're planning to build an unified,
>>> easy-to-use,
>>>>>>>> and
>>>>>>>>>>>> fast
>>>>>>>>>>>>> E2E
>>>>>>>>>>>>>>>>>>> test framework to benifit all the sub-projects.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here are some rough ideas about this framework:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0. (Unchanged) We will continue to use
>>> docker-compose.yaml
>>>>>>> to
>>>>>>>>>>>>>>>>> orchestrate
>>>>>>>>>>>>>>>>>>> the services that are to be tested, it’s proved to be
>>>>>>>> friendly
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> debuggable because we can start it directly even if we
>>> are
>>>>>>>> not
>>>>>>>>>>>>> writing
>>>>>>>>>>>>>>>>> E2E
>>>>>>>>>>>>>>>>>>> tests.
>>>>>>>>>>>>>>>>>>> 1. This framework will take full advantages of the CLI
>>>>>>>> project
>>>>>>>>> to
>>>>>>>>>>>>> query
>>>>>>>>>>>>>>>>>>> the traces/metrics/logs, we don’t want to maintain two
>>>>>>>> versions
>>>>>>>>>> of
>>>>>>>>>>>>> query
>>>>>>>>>>>>>>>>>>> codes anymore, (now we have a version in the test codes
>>>> and
>>>>>>>> one
>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> CLI).
>>>>>>>>>>>>>>>>>>> 2. We will build a CLI tool to compare the expected data
>>>>>>> and
>>>>>>>>> the
>>>>>>>>>>>>> actual
>>>>>>>>>>>>>>>>>>> data, we now have a custom schema of the expected data
>>>> yaml
>>>>>>>>> file,
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> verification codes should be packaged into an executable
>>>>>>>>> command
>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> that it
>>>>>>>>>>>>>>>>>>> can be executed standalone without Java and maven
>>>>>>> knowledge.
>>>>>>>> I
>>>>>>>>>>>> hope
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> leverage existing standards to write the YAML files and
>>> do
>>>>>>>>>>>>>>>>> verifications.
>>>>>>>>>>>>>>>>>>> 3. Build an orchestrating tool to start the
>>>>>>>>>> `docker-compose.yaml`,
>>>>>>>>>>>>>>>>> health
>>>>>>>>>>>>>>>>>>> check, query actual data, and verify the result.
>>>>>>>>>>>>>>>>>>> 4. (Nice to have) wrap all the tools as a GitHub Actions
>>>> to
>>>>>>>>> share
>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>> sub-projects.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If you have any other better idea or want to complement
>>> to
>>>>>>>> this
>>>>>>>>>>>>>>>>> proposal,
>>>>>>>>>>>>>>>>>>> please reply here. I will create an issue to track these
>>>>>>>> tasks
>>>>>>>>>>>>> later.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We have two students (who just finished the Summer Code
>>>>>>> 2020
>>>>>>>>>>>>> Projects)
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> lead this part, Ke Zhang [1] and Huaxi Jiang [2], as I
>>>> said
>>>>>>>>> this
>>>>>>>>>>>> is
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> high priority, you have adequate time to get yourselves
>>>>>>>>> familiar
>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>>> the components (OAP, WebUI, CLI, storages, etc.) of
>>>>>>>> SkyWalking
>>>>>>>>>>>> work
>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> well
>>>>>>>>>>>>>>>>>>> as the ecosystem. Thanks for your interests and
>>>> willingness
>>>>>>>> to
>>>>>>>>>>>> help.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1] https://github.com/Humbertzhang
>>>>>>>>>>>>>>>>>>> [2] https://github.com/fgksgf/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> —————————
>>>>>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> —————————
>>>>>>>>>>>>> Zhenxu Ke (柯振旭)
>>>>>>>>>>>>> GitHub @kezhenxu94
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>
>
>
> —————————
> Zhenxu Ke (柯振旭)
> GitHub @kezhenxu94