You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yunikorn.apache.org by Tao Yang <yt...@gmail.com> on 2020/04/17 07:49:13 UTC
[Discussion] Add integration testing framework and cases
Hello everyone
We are planning to add integration testing framework and initial test cases
in https://issues.apache.org/jira/browse/YUNIKORN-29, general thoughts are
as follows.
Basic testing framework includes:
1. AppInfo struct: define basic information, requests and status of an
application
2. AppManager interface and its implementations like DeploymentAppManager:
support useful operations like create/delete/refresh(status)/wait(for apps
to be satisfied or cleaned up) for testing.
3. CommonConfig struct: keep several common configurations for testing like
schedulerName, outputRootPath, namespace, queue etc.
4. Output tools like chart painter and file writer with specific format
(like csv).
Initial test cases:
1. Throughput : Request a certain number of pods via different schedulers
and then record the distributions of scheduled pods, draw results of
different schedulers on the same chart or write results into a file.
2. Node Fairness: Request a certain number of pods via different schedulers
and then record the distributions of node usage, draw results on charts
separately for different schedulers and write results into a file.
3. Queue Fairness (Only for YuniKorn since K8s scheduler doesn't support it
yet): Prepare queues with different capacities, request pods with different
number or resource for these queues, then record the usage of queues, draw
results on the same chart and write results into a file.
Implementation:
1. Package structure: add test/integration package in yunikorn-k8shim
project, the source files with structures or tools of testing framework
will be directly putted in this package, and test cases will be seperately
managed by case package in sub-directories: cases/<special-case> such as
cases/throughput, cases/node-fairness, cases/queue-fairness
2. Configurations: cases keep their specific configurations themselves and
all configurations can be maintained in a single yaml file together with
common configurations, the config structure like this:
```
common:
schedulerName: yunikorn
maxWaitSeconds: 60
queue: root.default
namespace: default
outputRootPath: /tmp
cases:
throughput:
schedulerNames:
- yunikorn
- default-scheduelr
appName: throughput-test
numPods: 10000
...
node-fairness:
...
queue-fairness:
...
```
3. Executions: all cases have main function themselves and need one
command-line argument: configPath, so that we can directly run specified
case and easily perform integration testing on different scenarios.
Any comments and suggestions are welcome!
Thanks,
Tao
Re: [Discussion] Add integration testing framework and cases
Posted by Tao Yang <yt...@gmail.com>.
Thanks Ayub for the suggestions, I think they are all very useful for
setting up a testing framework.
Looking forward to join the meeting to have a further discussion, @Weiwei
Yang <ww...@apache.org>, could you please help to organize it?
Thanks,
Tao
Ayub Pathan <ay...@apache.org> 于2020年4月21日周二 上午8:28写道：
> Thanks Tao for initiating this thread, I am super excited to be part of
> this discussion.
>
> Here are my two cents..
>
> Framework:
>
> 1. Consider adding something like kubeconfigManager, as this allows
> extensibility to work with any cluster(not just with the one running
> locally).
> 2. If I have understood correctly, the nature of the framework is
> behavior driven. If you agree, consider something which is Ginkgo
> <https://onsi.github.io/ginkgo/>. It provides many
> inbuilt extensions and third party integrations.
> 3. Consider using Viper <https://github.com/spf13/viper> for
> configuration management, as this helps setting defaults,
> reading/writing
> config files, fileformat conversions, environment variables management
> etc.
>
> I have some more thoughts around test cases as well. Let us set up a
> meeting to discuss these points in detail, as Weiwei suggested.
>
> Thanks
> Ayub Khan
>
> On Sun, Apr 19, 2020 at 9:08 PM Tao Yang <yt...@gmail.com> wrote:
>
> > Thanks Weiwei for the comments.
> >
> > Of course we should have some verification results in cases, my initial
> > thought is to execute cases separately but your comments made me realized
> > that maybe it's better to manage them together and provide functions as
> > following:
> > (1) We can keep loosely coupled relationships between framework and
> cases,
> > for example, a new case can be registered into the framework and with
> > specified type (such as benchmark, smoke, or chaos-monkey) and tags (like
> > 'constraints', 'performance', 'fairness', 'function' ...), and provide an
> > common entrance with which users can decide which cases to be included:
> > all, specified type or tags, or specified one.
> > (2) Each cases can return verification results (such as "All requirements
> > of app1 are satisfied":succeed) to the framework and output useful
> > information (which may helps to show execution process or locate the
> cause
> > of a failure) into log file. Chaos-monkey tests can be managed in one
> case
> > with type 'chaos-monkey' which has its own structure and can reuse tools
> in
> > the framework.
> > (3) After all desired test cases are done, a testing report with details
> of
> > these cases can be generated on demand, an example can be:
> > ```
> > Case: allocation
> > Tags: function
> > Status: Failed
> > Verifications:
> > Allocate pods with node constraint: Succeed
> > Allocate pods with affinity constraint: Succeed
> > Allocate pods with anti-affinity constraint: Failed
> > ```
> >
> > Above is my rough idea for that, hope to hear your thoughts.
> >
> > Thanks,
> > Tao
> >
> > Weiwei Yang <ww...@apache.org> 于2020年4月18日周六 上午5:44写道：
> >
> > > Hi Tao
> > >
> > > This is great, we are super interested.
> > > Let's track this effort with
> > > https://issues.apache.org/jira/browse/YUNIKORN-33.
> > > A couple of comments
> > >
> > > 1. There might be one thing missing is, how we can make this
> > > sustainable as "tests". which means we can 1) run this easily, 2)
> > > generate
> > > a result, and then 3) verify the result. Do we have #3 now? If not,
> we
> > > need
> > > to think about how to design a proper algorithm to verify if each
> > > scenario
> > > passes.
> > > 2. Another thing to add is the chaos monkey tests. This can be
> phase-2
> > > but it might be good we can consider this now, to make sure our
> > > structure
> > > can support that.
> > >
> > > We can set up some meetings next week to discuss this in more detail.
> > > Thanks
> > >
> > > On Fri, Apr 17, 2020 at 12:49 AM Tao Yang <yt...@gmail.com>
> > wrote:
> > >
> > > > Hello everyone
> > > >
> > > > We are planning to add integration testing framework and initial test
> > > cases
> > > > in https://issues.apache.org/jira/browse/YUNIKORN-29, general
> thoughts
> > > are
> > > > as follows.
> > > >
> > > > Basic testing framework includes:
> > > > 1. AppInfo struct: define basic information, requests and status of
> an
> > > > application
> > > > 2. AppManager interface and its implementations like
> > > DeploymentAppManager:
> > > > support useful operations like create/delete/refresh(status)/wait(for
> > > apps
> > > > to be satisfied or cleaned up) for testing.
> > > > 3. CommonConfig struct: keep several common configurations for
> testing
> > > like
> > > > schedulerName, outputRootPath, namespace, queue etc.
> > > > 4. Output tools like chart painter and file writer with specific
> format
> > > > (like csv).
> > > >
> > > > Initial test cases:
> > > > 1. Throughput : Request a certain number of pods via different
> > schedulers
> > > > and then record the distributions of scheduled pods, draw results of
> > > > different schedulers on the same chart or write results into a file.
> > > > 2. Node Fairness: Request a certain number of pods via different
> > > schedulers
> > > > and then record the distributions of node usage, draw results on
> charts
> > > > separately for different schedulers and write results into a file.
> > > > 3. Queue Fairness (Only for YuniKorn since K8s scheduler doesn't
> > support
> > > it
> > > > yet): Prepare queues with different capacities, request pods with
> > > different
> > > > number or resource for these queues, then record the usage of queues,
> > > draw
> > > > results on the same chart and write results into a file.
> > > >
> > > > Implementation:
> > > > 1. Package structure: add test/integration package in yunikorn-k8shim
> > > > project, the source files with structures or tools of testing
> framework
> > > > will be directly putted in this package, and test cases will be
> > > seperately
> > > > managed by case package in sub-directories: cases/<special-case> such
> > as
> > > > cases/throughput, cases/node-fairness, cases/queue-fairness
> > > > 2. Configurations: cases keep their specific configurations
> themselves
> > > and
> > > > all configurations can be maintained in a single yaml file together
> > with
> > > > common configurations, the config structure like this:
> > > > ```
> > > > common:
> > > > schedulerName: yunikorn
> > > > maxWaitSeconds: 60
> > > > queue: root.default
> > > > namespace: default
> > > > outputRootPath: /tmp
> > > >
> > > > cases:
> > > > throughput:
> > > > schedulerNames:
> > > > - yunikorn
> > > > - default-scheduelr
> > > > appName: throughput-test
> > > > numPods: 10000
> > > > ...
> > > > node-fairness:
> > > > ...
> > > > queue-fairness:
> > > > ...
> > > > ```
> > > > 3. Executions: all cases have main function themselves and need one
> > > > command-line argument: configPath, so that we can directly run
> > specified
> > > > case and easily perform integration testing on different scenarios.
> > > >
> > > > Any comments and suggestions are welcome!
> > > >
> > > > Thanks,
> > > > Tao
> > > >
> > >
> >
>
Re: [Discussion] Add integration testing framework and cases
Posted by Ayub Pathan <ay...@apache.org>.
Thanks Tao for initiating this thread, I am super excited to be part of
this discussion.
Here are my two cents..
Framework:
1. Consider adding something like kubeconfigManager, as this allows
extensibility to work with any cluster(not just with the one running
locally).
2. If I have understood correctly, the nature of the framework is
behavior driven. If you agree, consider something which is Ginkgo
<https://onsi.github.io/ginkgo/>. It provides many
inbuilt extensions and third party integrations.
3. Consider using Viper <https://github.com/spf13/viper> for
configuration management, as this helps setting defaults, reading/writing
config files, fileformat conversions, environment variables management etc.
I have some more thoughts around test cases as well. Let us set up a
meeting to discuss these points in detail, as Weiwei suggested.
Thanks
Ayub Khan
On Sun, Apr 19, 2020 at 9:08 PM Tao Yang <yt...@gmail.com> wrote:
> Thanks Weiwei for the comments.
>
> Of course we should have some verification results in cases, my initial
> thought is to execute cases separately but your comments made me realized
> that maybe it's better to manage them together and provide functions as
> following:
> (1) We can keep loosely coupled relationships between framework and cases,
> for example, a new case can be registered into the framework and with
> specified type (such as benchmark, smoke, or chaos-monkey) and tags (like
> 'constraints', 'performance', 'fairness', 'function' ...), and provide an
> common entrance with which users can decide which cases to be included:
> all, specified type or tags, or specified one.
> (2) Each cases can return verification results (such as "All requirements
> of app1 are satisfied":succeed) to the framework and output useful
> information (which may helps to show execution process or locate the cause
> of a failure) into log file. Chaos-monkey tests can be managed in one case
> with type 'chaos-monkey' which has its own structure and can reuse tools in
> the framework.
> (3) After all desired test cases are done, a testing report with details of
> these cases can be generated on demand, an example can be:
> ```
> Case: allocation
> Tags: function
> Status: Failed
> Verifications:
> Allocate pods with node constraint: Succeed
> Allocate pods with affinity constraint: Succeed
> Allocate pods with anti-affinity constraint: Failed
> ```
>
> Above is my rough idea for that, hope to hear your thoughts.
>
> Thanks,
> Tao
>
> Weiwei Yang <ww...@apache.org> 于2020年4月18日周六 上午5:44写道：
>
> > Hi Tao
> >
> > This is great, we are super interested.
> > Let's track this effort with
> > https://issues.apache.org/jira/browse/YUNIKORN-33.
> > A couple of comments
> >
> > 1. There might be one thing missing is, how we can make this
> > sustainable as "tests". which means we can 1) run this easily, 2)
> > generate
> > a result, and then 3) verify the result. Do we have #3 now? If not, we
> > need
> > to think about how to design a proper algorithm to verify if each
> > scenario
> > passes.
> > 2. Another thing to add is the chaos monkey tests. This can be phase-2
> > but it might be good we can consider this now, to make sure our
> > structure
> > can support that.
> >
> > We can set up some meetings next week to discuss this in more detail.
> > Thanks
> >
> > On Fri, Apr 17, 2020 at 12:49 AM Tao Yang <yt...@gmail.com>
> wrote:
> >
> > > Hello everyone
> > >
> > > We are planning to add integration testing framework and initial test
> > cases
> > > in https://issues.apache.org/jira/browse/YUNIKORN-29, general thoughts
> > are
> > > as follows.
> > >
> > > Basic testing framework includes:
> > > 1. AppInfo struct: define basic information, requests and status of an
> > > application
> > > 2. AppManager interface and its implementations like
> > DeploymentAppManager:
> > > support useful operations like create/delete/refresh(status)/wait(for
> > apps
> > > to be satisfied or cleaned up) for testing.
> > > 3. CommonConfig struct: keep several common configurations for testing
> > like
> > > schedulerName, outputRootPath, namespace, queue etc.
> > > 4. Output tools like chart painter and file writer with specific format
> > > (like csv).
> > >
> > > Initial test cases:
> > > 1. Throughput : Request a certain number of pods via different
> schedulers
> > > and then record the distributions of scheduled pods, draw results of
> > > different schedulers on the same chart or write results into a file.
> > > 2. Node Fairness: Request a certain number of pods via different
> > schedulers
> > > and then record the distributions of node usage, draw results on charts
> > > separately for different schedulers and write results into a file.
> > > 3. Queue Fairness (Only for YuniKorn since K8s scheduler doesn't
> support
> > it
> > > yet): Prepare queues with different capacities, request pods with
> > different
> > > number or resource for these queues, then record the usage of queues,
> > draw
> > > results on the same chart and write results into a file.
> > >
> > > Implementation:
> > > 1. Package structure: add test/integration package in yunikorn-k8shim
> > > project, the source files with structures or tools of testing framework
> > > will be directly putted in this package, and test cases will be
> > seperately
> > > managed by case package in sub-directories: cases/<special-case> such
> as
> > > cases/throughput, cases/node-fairness, cases/queue-fairness
> > > 2. Configurations: cases keep their specific configurations themselves
> > and
> > > all configurations can be maintained in a single yaml file together
> with
> > > common configurations, the config structure like this:
> > > ```
> > > common:
> > > schedulerName: yunikorn
> > > maxWaitSeconds: 60
> > > queue: root.default
> > > namespace: default
> > > outputRootPath: /tmp
> > >
> > > cases:
> > > throughput:
> > > schedulerNames:
> > > - yunikorn
> > > - default-scheduelr
> > > appName: throughput-test
> > > numPods: 10000
> > > ...
> > > node-fairness:
> > > ...
> > > queue-fairness:
> > > ...
> > > ```
> > > 3. Executions: all cases have main function themselves and need one
> > > command-line argument: configPath, so that we can directly run
> specified
> > > case and easily perform integration testing on different scenarios.
> > >
> > > Any comments and suggestions are welcome!
> > >
> > > Thanks,
> > > Tao
> > >
> >
>
Re: [Discussion] Add integration testing framework and cases
Posted by Tao Yang <yt...@gmail.com>.
Thanks Weiwei for the comments.
Of course we should have some verification results in cases, my initial
thought is to execute cases separately but your comments made me realized
that maybe it's better to manage them together and provide functions as
following:
(1) We can keep loosely coupled relationships between framework and cases,
for example, a new case can be registered into the framework and with
specified type (such as benchmark, smoke, or chaos-monkey) and tags (like
'constraints', 'performance', 'fairness', 'function' ...), and provide an
common entrance with which users can decide which cases to be included:
all, specified type or tags, or specified one.
(2) Each cases can return verification results (such as "All requirements
of app1 are satisfied":succeed) to the framework and output useful
information (which may helps to show execution process or locate the cause
of a failure) into log file. Chaos-monkey tests can be managed in one case
with type 'chaos-monkey' which has its own structure and can reuse tools in
the framework.
(3) After all desired test cases are done, a testing report with details of
these cases can be generated on demand, an example can be:
```
Case: allocation
Tags: function
Status: Failed
Verifications:
Allocate pods with node constraint: Succeed
Allocate pods with affinity constraint: Succeed
Allocate pods with anti-affinity constraint: Failed
```
Above is my rough idea for that, hope to hear your thoughts.
Thanks,
Tao
Weiwei Yang <ww...@apache.org> 于2020年4月18日周六 上午5:44写道：
> Hi Tao
>
> This is great, we are super interested.
> Let's track this effort with
> https://issues.apache.org/jira/browse/YUNIKORN-33.
> A couple of comments
>
> 1. There might be one thing missing is, how we can make this
> sustainable as "tests". which means we can 1) run this easily, 2)
> generate
> a result, and then 3) verify the result. Do we have #3 now? If not, we
> need
> to think about how to design a proper algorithm to verify if each
> scenario
> passes.
> 2. Another thing to add is the chaos monkey tests. This can be phase-2
> but it might be good we can consider this now, to make sure our
> structure
> can support that.
>
> We can set up some meetings next week to discuss this in more detail.
> Thanks
>
> On Fri, Apr 17, 2020 at 12:49 AM Tao Yang <yt...@gmail.com> wrote:
>
> > Hello everyone
> >
> > We are planning to add integration testing framework and initial test
> cases
> > in https://issues.apache.org/jira/browse/YUNIKORN-29, general thoughts
> are
> > as follows.
> >
> > Basic testing framework includes:
> > 1. AppInfo struct: define basic information, requests and status of an
> > application
> > 2. AppManager interface and its implementations like
> DeploymentAppManager:
> > support useful operations like create/delete/refresh(status)/wait(for
> apps
> > to be satisfied or cleaned up) for testing.
> > 3. CommonConfig struct: keep several common configurations for testing
> like
> > schedulerName, outputRootPath, namespace, queue etc.
> > 4. Output tools like chart painter and file writer with specific format
> > (like csv).
> >
> > Initial test cases:
> > 1. Throughput : Request a certain number of pods via different schedulers
> > and then record the distributions of scheduled pods, draw results of
> > different schedulers on the same chart or write results into a file.
> > 2. Node Fairness: Request a certain number of pods via different
> schedulers
> > and then record the distributions of node usage, draw results on charts
> > separately for different schedulers and write results into a file.
> > 3. Queue Fairness (Only for YuniKorn since K8s scheduler doesn't support
> it
> > yet): Prepare queues with different capacities, request pods with
> different
> > number or resource for these queues, then record the usage of queues,
> draw
> > results on the same chart and write results into a file.
> >
> > Implementation:
> > 1. Package structure: add test/integration package in yunikorn-k8shim
> > project, the source files with structures or tools of testing framework
> > will be directly putted in this package, and test cases will be
> seperately
> > managed by case package in sub-directories: cases/<special-case> such as
> > cases/throughput, cases/node-fairness, cases/queue-fairness
> > 2. Configurations: cases keep their specific configurations themselves
> and
> > all configurations can be maintained in a single yaml file together with
> > common configurations, the config structure like this:
> > ```
> > common:
> > schedulerName: yunikorn
> > maxWaitSeconds: 60
> > queue: root.default
> > namespace: default
> > outputRootPath: /tmp
> >
> > cases:
> > throughput:
> > schedulerNames:
> > - yunikorn
> > - default-scheduelr
> > appName: throughput-test
> > numPods: 10000
> > ...
> > node-fairness:
> > ...
> > queue-fairness:
> > ...
> > ```
> > 3. Executions: all cases have main function themselves and need one
> > command-line argument: configPath, so that we can directly run specified
> > case and easily perform integration testing on different scenarios.
> >
> > Any comments and suggestions are welcome!
> >
> > Thanks,
> > Tao
> >
>
Re: [Discussion] Add integration testing framework and cases
Posted by Weiwei Yang <ww...@apache.org>.
Hi Tao
This is great, we are super interested.
Let's track this effort with
https://issues.apache.org/jira/browse/YUNIKORN-33.
A couple of comments
1. There might be one thing missing is, how we can make this
sustainable as "tests". which means we can 1) run this easily, 2) generate
a result, and then 3) verify the result. Do we have #3 now? If not, we need
to think about how to design a proper algorithm to verify if each scenario
passes.
2. Another thing to add is the chaos monkey tests. This can be phase-2
but it might be good we can consider this now, to make sure our structure
can support that.
We can set up some meetings next week to discuss this in more detail.
Thanks
On Fri, Apr 17, 2020 at 12:49 AM Tao Yang <yt...@gmail.com> wrote:
> Hello everyone
>
> We are planning to add integration testing framework and initial test cases
> in https://issues.apache.org/jira/browse/YUNIKORN-29, general thoughts are
> as follows.
>
> Basic testing framework includes:
> 1. AppInfo struct: define basic information, requests and status of an
> application
> 2. AppManager interface and its implementations like DeploymentAppManager:
> support useful operations like create/delete/refresh(status)/wait(for apps
> to be satisfied or cleaned up) for testing.
> 3. CommonConfig struct: keep several common configurations for testing like
> schedulerName, outputRootPath, namespace, queue etc.
> 4. Output tools like chart painter and file writer with specific format
> (like csv).
>
> Initial test cases:
> 1. Throughput : Request a certain number of pods via different schedulers
> and then record the distributions of scheduled pods, draw results of
> different schedulers on the same chart or write results into a file.
> 2. Node Fairness: Request a certain number of pods via different schedulers
> and then record the distributions of node usage, draw results on charts
> separately for different schedulers and write results into a file.
> 3. Queue Fairness (Only for YuniKorn since K8s scheduler doesn't support it
> yet): Prepare queues with different capacities, request pods with different
> number or resource for these queues, then record the usage of queues, draw
> results on the same chart and write results into a file.
>
> Implementation:
> 1. Package structure: add test/integration package in yunikorn-k8shim
> project, the source files with structures or tools of testing framework
> will be directly putted in this package, and test cases will be seperately
> managed by case package in sub-directories: cases/<special-case> such as
> cases/throughput, cases/node-fairness, cases/queue-fairness
> 2. Configurations: cases keep their specific configurations themselves and
> all configurations can be maintained in a single yaml file together with
> common configurations, the config structure like this:
> ```
> common:
> schedulerName: yunikorn
> maxWaitSeconds: 60
> queue: root.default
> namespace: default
> outputRootPath: /tmp
>
> cases:
> throughput:
> schedulerNames:
> - yunikorn
> - default-scheduelr
> appName: throughput-test
> numPods: 10000
> ...
> node-fairness:
> ...
> queue-fairness:
> ...
> ```
> 3. Executions: all cases have main function themselves and need one
> command-line argument: configPath, so that we can directly run specified
> case and easily perform integration testing on different scenarios.
>
> Any comments and suggestions are welcome!
>
> Thanks,
> Tao
>