You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicecomb.apache.org by wjm wjm <zz...@gmail.com> on 2018/07/10 01:41:10 UTC
[DISCUSS] about integration test
currently we embed integration test with PR CI
but:
1.we will add more and more integration test case
2.test time will be more and more longer
3.integration test should include work with multiple SC versions
4.......
so, maybe it's better that make integration test to be a independent
project, and run it once a day at least
some other idea of integration test:
https://issues.apache.org/jira/browse/SCB-719
what's your suggestion?
Re: [DISCUSS] about integration test
Posted by wjm wjm <zz...@gmail.com>.
we can try to move CI to huawei cloud, that must faster than Travis
if fast enough, then no need to make integration test independent
2018-07-11 6:59 GMT+08:00 Willem Jiang <wi...@gmail.com>:
> I think we may change the travis script a little bit, to find out if we can
> run the integration test once per day.
>
>
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Wed, Jul 11, 2018 at 12:45 AM, wjm wjm <zz...@gmail.com> wrote:
>
> > multiple SC is no problem, infact, not all test case should be run for
> > multiple SC
> >
> > the problem is Travis is so slow
> > yes, we can not ask too much about that, so my suggestion is to make
> > integration test independent
> > that will not block our PR
> > and integration test run once a day at least, can make sure there are not
> > too many mistakes.
> >
> > for contributors, still must provide unit test case, and better to
> provide
> > integration test, that not too much different than current
> >
> > 2018-07-10 21:17 GMT+08:00 Willem Jiang <wi...@gmail.com>:
> >
> > > As Travis is a free service for Open Source projects, we cannot ask too
> > > much about that.
> > > Current ASF provides Jenkins Service here[1], there are some
> shortcoming
> > > if we use it, such as lots of projects use it, and it doesn't
> integrate
> > > with github.
> > >
> > > For running the test case with different SC, I think we could leverage
> > the
> > > env properties[2] to run the test separately.
> > > Current we run the saga build with Spring Boot 1.x and Spring Boot 2.x
> > [3],
> > > I think it quite easy to list all the SC versions as the env properties
> > > those we need to support.
> > >
> > > [1]https://builds.apache.org/
> > > [2]
> > > https://github.com/apache/incubator-servicecomb-saga/
> > > blob/master/.travis.yml#L28
> > > [3]https://travis-ci.org/apache/incubator-servicecomb-saga
> > >
> > >
> > > Willem Jiang
> > >
> > > Twitter: willemjiang
> > > Weibo: 姜宁willem
> > >
> > > On Tue, Jul 10, 2018 at 8:56 PM, wjm wjm <zz...@gmail.com> wrote:
> > >
> > > > yes
> > > > everything no problem. but:
> > > >
> > > > 1.Travis is so slow, not our integration test too slow
> > > > when run in our development environment, it's very fast.
> > > > 2.when run all test case multi times with different SC, will produce
> > too
> > > > many log, that will treat Travis log size limit.
> > > >
> > > > 2018-07-10 19:23 GMT+08:00 Sure <su...@qq.com>:
> > > >
> > > > > Agree!
> > > > >
> > > > >
> > > > > 1. Just like willem.jiang said, we can push multiple SC versions to
> > > > docker
> > > > > hub
> > > > > 2. Chassis integration test can run with different SC version in
> > every
> > > > time
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ------------------ 原始邮件 ------------------
> > > > > 发件人: "willem.jiang"<wi...@gmail.com>;
> > > > > 发送时间: 2018年7月10日(星期二) 下午5:16
> > > > > 收件人: "dev"<de...@servicecomb.apache.org>;
> > > > >
> > > > > 主题: Re: [DISCUSS] about integration test
> > > > >
> > > > >
> > > > >
> > > > > Current ServiceComb CI just leverage service of Travis[1] which
> has a
> > > > very
> > > > > good integration with Github.
> > > > >
> > > > > Current we don't test the multiple SC at same time, but I think
> with
> > > the
> > > > > help of docker, it could easy for us to run the examples against
> > > multiple
> > > > > version of SC.
> > > > > From the proposal, we need multiple node to handle the test, it
> could
> > > be
> > > > a
> > > > > challenge for us if we still use Travis.
> > > > >
> > > > > If we put the integration test in a separated project we may need
> to
> > > > find a
> > > > > way to let the integration test looks up the latest snap short
> > artifact
> > > > for
> > > > > running the test.
> > > > >
> > > > > Any thought?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Willem Jiang
> > > > >
> > > > > Twitter: willemjiang
> > > > > Weibo: 姜宁willem
> > > > >
> > > > > On Tue, Jul 10, 2018 at 9:41 AM, wjm wjm <zz...@gmail.com> wrote:
> > > > >
> > > > > > currently we embed integration test with PR CI
> > > > > >
> > > > > > but:
> > > > > > 1.we will add more and more integration test case
> > > > > > 2.test time will be more and more longer
> > > > > > 3.integration test should include work with multiple SC versions
> > > > > > 4.......
> > > > > >
> > > > > > so, maybe it's better that make integration test to be a
> > independent
> > > > > > project, and run it once a day at least
> > > > > >
> > > > > > some other idea of integration test:
> > > > > > https://issues.apache.org/jira/browse/SCB-719
> > > > > >
> > > > > > what's your suggestion?
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] about integration test
Posted by Willem Jiang <wi...@gmail.com>.
I think we may change the travis script a little bit, to find out if we can
run the integration test once per day.
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Wed, Jul 11, 2018 at 12:45 AM, wjm wjm <zz...@gmail.com> wrote:
> multiple SC is no problem, infact, not all test case should be run for
> multiple SC
>
> the problem is Travis is so slow
> yes, we can not ask too much about that, so my suggestion is to make
> integration test independent
> that will not block our PR
> and integration test run once a day at least, can make sure there are not
> too many mistakes.
>
> for contributors, still must provide unit test case, and better to provide
> integration test, that not too much different than current
>
> 2018-07-10 21:17 GMT+08:00 Willem Jiang <wi...@gmail.com>:
>
> > As Travis is a free service for Open Source projects, we cannot ask too
> > much about that.
> > Current ASF provides Jenkins Service here[1], there are some shortcoming
> > if we use it, such as lots of projects use it, and it doesn't integrate
> > with github.
> >
> > For running the test case with different SC, I think we could leverage
> the
> > env properties[2] to run the test separately.
> > Current we run the saga build with Spring Boot 1.x and Spring Boot 2.x
> [3],
> > I think it quite easy to list all the SC versions as the env properties
> > those we need to support.
> >
> > [1]https://builds.apache.org/
> > [2]
> > https://github.com/apache/incubator-servicecomb-saga/
> > blob/master/.travis.yml#L28
> > [3]https://travis-ci.org/apache/incubator-servicecomb-saga
> >
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Tue, Jul 10, 2018 at 8:56 PM, wjm wjm <zz...@gmail.com> wrote:
> >
> > > yes
> > > everything no problem. but:
> > >
> > > 1.Travis is so slow, not our integration test too slow
> > > when run in our development environment, it's very fast.
> > > 2.when run all test case multi times with different SC, will produce
> too
> > > many log, that will treat Travis log size limit.
> > >
> > > 2018-07-10 19:23 GMT+08:00 Sure <su...@qq.com>:
> > >
> > > > Agree!
> > > >
> > > >
> > > > 1. Just like willem.jiang said, we can push multiple SC versions to
> > > docker
> > > > hub
> > > > 2. Chassis integration test can run with different SC version in
> every
> > > time
> > > >
> > > >
> > > >
> > > >
> > > > ------------------ 原始邮件 ------------------
> > > > 发件人: "willem.jiang"<wi...@gmail.com>;
> > > > 发送时间: 2018年7月10日(星期二) 下午5:16
> > > > 收件人: "dev"<de...@servicecomb.apache.org>;
> > > >
> > > > 主题: Re: [DISCUSS] about integration test
> > > >
> > > >
> > > >
> > > > Current ServiceComb CI just leverage service of Travis[1] which has a
> > > very
> > > > good integration with Github.
> > > >
> > > > Current we don't test the multiple SC at same time, but I think with
> > the
> > > > help of docker, it could easy for us to run the examples against
> > multiple
> > > > version of SC.
> > > > From the proposal, we need multiple node to handle the test, it could
> > be
> > > a
> > > > challenge for us if we still use Travis.
> > > >
> > > > If we put the integration test in a separated project we may need to
> > > find a
> > > > way to let the integration test looks up the latest snap short
> artifact
> > > for
> > > > running the test.
> > > >
> > > > Any thought?
> > > >
> > > >
> > > >
> > > >
> > > > Willem Jiang
> > > >
> > > > Twitter: willemjiang
> > > > Weibo: 姜宁willem
> > > >
> > > > On Tue, Jul 10, 2018 at 9:41 AM, wjm wjm <zz...@gmail.com> wrote:
> > > >
> > > > > currently we embed integration test with PR CI
> > > > >
> > > > > but:
> > > > > 1.we will add more and more integration test case
> > > > > 2.test time will be more and more longer
> > > > > 3.integration test should include work with multiple SC versions
> > > > > 4.......
> > > > >
> > > > > so, maybe it's better that make integration test to be a
> independent
> > > > > project, and run it once a day at least
> > > > >
> > > > > some other idea of integration test:
> > > > > https://issues.apache.org/jira/browse/SCB-719
> > > > >
> > > > > what's your suggestion?
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] about integration test
Posted by wjm wjm <zz...@gmail.com>.
multiple SC is no problem, infact, not all test case should be run for
multiple SC
the problem is Travis is so slow
yes, we can not ask too much about that, so my suggestion is to make
integration test independent
that will not block our PR
and integration test run once a day at least, can make sure there are not
too many mistakes.
for contributors, still must provide unit test case, and better to provide
integration test, that not too much different than current
2018-07-10 21:17 GMT+08:00 Willem Jiang <wi...@gmail.com>:
> As Travis is a free service for Open Source projects, we cannot ask too
> much about that.
> Current ASF provides Jenkins Service here[1], there are some shortcoming
> if we use it, such as lots of projects use it, and it doesn't integrate
> with github.
>
> For running the test case with different SC, I think we could leverage the
> env properties[2] to run the test separately.
> Current we run the saga build with Spring Boot 1.x and Spring Boot 2.x [3],
> I think it quite easy to list all the SC versions as the env properties
> those we need to support.
>
> [1]https://builds.apache.org/
> [2]
> https://github.com/apache/incubator-servicecomb-saga/
> blob/master/.travis.yml#L28
> [3]https://travis-ci.org/apache/incubator-servicecomb-saga
>
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Tue, Jul 10, 2018 at 8:56 PM, wjm wjm <zz...@gmail.com> wrote:
>
> > yes
> > everything no problem. but:
> >
> > 1.Travis is so slow, not our integration test too slow
> > when run in our development environment, it's very fast.
> > 2.when run all test case multi times with different SC, will produce too
> > many log, that will treat Travis log size limit.
> >
> > 2018-07-10 19:23 GMT+08:00 Sure <su...@qq.com>:
> >
> > > Agree!
> > >
> > >
> > > 1. Just like willem.jiang said, we can push multiple SC versions to
> > docker
> > > hub
> > > 2. Chassis integration test can run with different SC version in every
> > time
> > >
> > >
> > >
> > >
> > > ------------------ 原始邮件 ------------------
> > > 发件人: "willem.jiang"<wi...@gmail.com>;
> > > 发送时间: 2018年7月10日(星期二) 下午5:16
> > > 收件人: "dev"<de...@servicecomb.apache.org>;
> > >
> > > 主题: Re: [DISCUSS] about integration test
> > >
> > >
> > >
> > > Current ServiceComb CI just leverage service of Travis[1] which has a
> > very
> > > good integration with Github.
> > >
> > > Current we don't test the multiple SC at same time, but I think with
> the
> > > help of docker, it could easy for us to run the examples against
> multiple
> > > version of SC.
> > > From the proposal, we need multiple node to handle the test, it could
> be
> > a
> > > challenge for us if we still use Travis.
> > >
> > > If we put the integration test in a separated project we may need to
> > find a
> > > way to let the integration test looks up the latest snap short artifact
> > for
> > > running the test.
> > >
> > > Any thought?
> > >
> > >
> > >
> > >
> > > Willem Jiang
> > >
> > > Twitter: willemjiang
> > > Weibo: 姜宁willem
> > >
> > > On Tue, Jul 10, 2018 at 9:41 AM, wjm wjm <zz...@gmail.com> wrote:
> > >
> > > > currently we embed integration test with PR CI
> > > >
> > > > but:
> > > > 1.we will add more and more integration test case
> > > > 2.test time will be more and more longer
> > > > 3.integration test should include work with multiple SC versions
> > > > 4.......
> > > >
> > > > so, maybe it's better that make integration test to be a independent
> > > > project, and run it once a day at least
> > > >
> > > > some other idea of integration test:
> > > > https://issues.apache.org/jira/browse/SCB-719
> > > >
> > > > what's your suggestion?
> > > >
> > >
> >
>
Re: [DISCUSS] about integration test
Posted by Willem Jiang <wi...@gmail.com>.
As Travis is a free service for Open Source projects, we cannot ask too
much about that.
Current ASF provides Jenkins Service here[1], there are some shortcoming
if we use it, such as lots of projects use it, and it doesn't integrate
with github.
For running the test case with different SC, I think we could leverage the
env properties[2] to run the test separately.
Current we run the saga build with Spring Boot 1.x and Spring Boot 2.x [3],
I think it quite easy to list all the SC versions as the env properties
those we need to support.
[1]https://builds.apache.org/
[2]
https://github.com/apache/incubator-servicecomb-saga/blob/master/.travis.yml#L28
[3]https://travis-ci.org/apache/incubator-servicecomb-saga
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Tue, Jul 10, 2018 at 8:56 PM, wjm wjm <zz...@gmail.com> wrote:
> yes
> everything no problem. but:
>
> 1.Travis is so slow, not our integration test too slow
> when run in our development environment, it's very fast.
> 2.when run all test case multi times with different SC, will produce too
> many log, that will treat Travis log size limit.
>
> 2018-07-10 19:23 GMT+08:00 Sure <su...@qq.com>:
>
> > Agree!
> >
> >
> > 1. Just like willem.jiang said, we can push multiple SC versions to
> docker
> > hub
> > 2. Chassis integration test can run with different SC version in every
> time
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "willem.jiang"<wi...@gmail.com>;
> > 发送时间: 2018年7月10日(星期二) 下午5:16
> > 收件人: "dev"<de...@servicecomb.apache.org>;
> >
> > 主题: Re: [DISCUSS] about integration test
> >
> >
> >
> > Current ServiceComb CI just leverage service of Travis[1] which has a
> very
> > good integration with Github.
> >
> > Current we don't test the multiple SC at same time, but I think with the
> > help of docker, it could easy for us to run the examples against multiple
> > version of SC.
> > From the proposal, we need multiple node to handle the test, it could be
> a
> > challenge for us if we still use Travis.
> >
> > If we put the integration test in a separated project we may need to
> find a
> > way to let the integration test looks up the latest snap short artifact
> for
> > running the test.
> >
> > Any thought?
> >
> >
> >
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Tue, Jul 10, 2018 at 9:41 AM, wjm wjm <zz...@gmail.com> wrote:
> >
> > > currently we embed integration test with PR CI
> > >
> > > but:
> > > 1.we will add more and more integration test case
> > > 2.test time will be more and more longer
> > > 3.integration test should include work with multiple SC versions
> > > 4.......
> > >
> > > so, maybe it's better that make integration test to be a independent
> > > project, and run it once a day at least
> > >
> > > some other idea of integration test:
> > > https://issues.apache.org/jira/browse/SCB-719
> > >
> > > what's your suggestion?
> > >
> >
>
Re: [DISCUSS] about integration test
Posted by wjm wjm <zz...@gmail.com>.
yes
everything no problem. but:
1.Travis is so slow, not our integration test too slow
when run in our development environment, it's very fast.
2.when run all test case multi times with different SC, will produce too
many log, that will treat Travis log size limit.
2018-07-10 19:23 GMT+08:00 Sure <su...@qq.com>:
> Agree!
>
>
> 1. Just like willem.jiang said, we can push multiple SC versions to docker
> hub
> 2. Chassis integration test can run with different SC version in every time
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "willem.jiang"<wi...@gmail.com>;
> 发送时间: 2018年7月10日(星期二) 下午5:16
> 收件人: "dev"<de...@servicecomb.apache.org>;
>
> 主题: Re: [DISCUSS] about integration test
>
>
>
> Current ServiceComb CI just leverage service of Travis[1] which has a very
> good integration with Github.
>
> Current we don't test the multiple SC at same time, but I think with the
> help of docker, it could easy for us to run the examples against multiple
> version of SC.
> From the proposal, we need multiple node to handle the test, it could be a
> challenge for us if we still use Travis.
>
> If we put the integration test in a separated project we may need to find a
> way to let the integration test looks up the latest snap short artifact for
> running the test.
>
> Any thought?
>
>
>
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Tue, Jul 10, 2018 at 9:41 AM, wjm wjm <zz...@gmail.com> wrote:
>
> > currently we embed integration test with PR CI
> >
> > but:
> > 1.we will add more and more integration test case
> > 2.test time will be more and more longer
> > 3.integration test should include work with multiple SC versions
> > 4.......
> >
> > so, maybe it's better that make integration test to be a independent
> > project, and run it once a day at least
> >
> > some other idea of integration test:
> > https://issues.apache.org/jira/browse/SCB-719
> >
> > what's your suggestion?
> >
>
回复： [DISCUSS] about integration test
Posted by Sure <su...@qq.com>.
Agree!
1. Just like willem.jiang said, we can push multiple SC versions to docker hub
2. Chassis integration test can run with different SC version in every time
------------------ 原始邮件 ------------------
发件人: "willem.jiang"<wi...@gmail.com>;
发送时间: 2018年7月10日(星期二) 下午5:16
收件人: "dev"<de...@servicecomb.apache.org>;
主题: Re: [DISCUSS] about integration test
Current ServiceComb CI just leverage service of Travis[1] which has a very
good integration with Github.
Current we don't test the multiple SC at same time, but I think with the
help of docker, it could easy for us to run the examples against multiple
version of SC.
From the proposal, we need multiple node to handle the test, it could be a
challenge for us if we still use Travis.
If we put the integration test in a separated project we may need to find a
way to let the integration test looks up the latest snap short artifact for
running the test.
Any thought?
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Tue, Jul 10, 2018 at 9:41 AM, wjm wjm <zz...@gmail.com> wrote:
> currently we embed integration test with PR CI
>
> but:
> 1.we will add more and more integration test case
> 2.test time will be more and more longer
> 3.integration test should include work with multiple SC versions
> 4.......
>
> so, maybe it's better that make integration test to be a independent
> project, and run it once a day at least
>
> some other idea of integration test:
> https://issues.apache.org/jira/browse/SCB-719
>
> what's your suggestion?
>
Re: [DISCUSS] about integration test
Posted by Willem Jiang <wi...@gmail.com>.
Current ServiceComb CI just leverage service of Travis[1] which has a very
good integration with Github.
Current we don't test the multiple SC at same time, but I think with the
help of docker, it could easy for us to run the examples against multiple
version of SC.
From the proposal, we need multiple node to handle the test, it could be a
challenge for us if we still use Travis.
If we put the integration test in a separated project we may need to find a
way to let the integration test looks up the latest snap short artifact for
running the test.
Any thought?
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Tue, Jul 10, 2018 at 9:41 AM, wjm wjm <zz...@gmail.com> wrote:
> currently we embed integration test with PR CI
>
> but:
> 1.we will add more and more integration test case
> 2.test time will be more and more longer
> 3.integration test should include work with multiple SC versions
> 4.......
>
> so, maybe it's better that make integration test to be a independent
> project, and run it once a day at least
>
> some other idea of integration test:
> https://issues.apache.org/jira/browse/SCB-719
>
> what's your suggestion?
>
Re: [DISCUSS] about integration test
Posted by Zen Lin <ze...@gmail.com>.
ServiceComb is an open source project that has been adopted for a variety
of commercial scenarios. As wrote in the report, at least more than 10
users have used ServiceComb for commercial use.
Therefore, our integration testing will be more and more comprehensive and
huge, it is a good idea to resolve this problem, but directly separating
the inheritance test to the independent project will cause some problems:
1. This is contrary to the ordinary practice of open source projects,
causing the unconsistent experience for our developers.
2. When the developers commmit, they will need to commit to both projects,
which will affect the uniform traceability of the test code and function
code.
3. The results of the integration test will lag, which will have an impact
on the community's timely feedback when submitting the function code.
So my suggestion is whether it can be compromised:
Select out the time-consuming parts of the integration test and then reduce
the run cycle, but don't strip it into separate branches.
Any other thoughts?
Best Regards,
---
Zen Lin
zenlintechnofreak@gmail.com
Focused on Micro Service and Apache ServiceComb
wjm wjm <zz...@gmail.com> 于2018年7月10日周二 上午9:41写道：
> currently we embed integration test with PR CI
>
> but:
> 1.we will add more and more integration test case
> 2.test time will be more and more longer
> 3.integration test should include work with multiple SC versions
> 4.......
>
> so, maybe it's better that make integration test to be a independent
> project, and run it once a day at least
>
> some other idea of integration test:
> https://issues.apache.org/jira/browse/SCB-719
>
> what's your suggestion?
>