You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2018/02/12 09:53:16 UTC
Jenkins for Apache Kylin Integration test
Hello Kylin developers,
As you know, Kylin has a full regression test (also called "integration
test"), which covers UT, cube build (batch & streaming), SQL queries and
others. With this regression test, we can know whether a change is safe or
not (to some extent, not 100%). Many issues can be found with this test.
Since this test need be executed in a Hadoop environment, at least be a
Hadoop Sandbox with 10GB memory and 40 GB disk. Apache Foundation doesn't
have the resource for this purpose. So in the past, we can only run it on
the private server; The status and result are invisible for Kylin
community, that impacts the communication and efficiency.
Now, we received a donation to run such a dedicated CI server
(T2.xlarge instance) on AWS for at least 6 months. It will run every 8 hrs
(pull from master branch), each round takes about 1 hr; Once the test is
failed, it will automatically send email to the one who breaks the CI.
Later we plan to automate it with JIRA/Github workflow.
http://34.226.50.254:8081/
Anonymous user can only view the test result and log. The Hadoop YARN RM
and history server are open for checking job status, you can access it at
8088 and 19888 port.
The server information is also updated in the "CI and Code Analysis"
section on https://kylin.apache.org/development/
Let's use this CI to check the status and communicate on the specific
problem. If you have any suggestion, please raise it in the dev mailing
list.
--
Best regards,
Shaofeng Shi 史少锋
Re: Jenkins for Apache Kylin Integration test
Posted by ShaoFeng Shi <sh...@apache.org>.
Hello,
Just FYI: recently the CI in AWS is very unstable, the job somehow failed
with OOM error, and then yarn resource manager won't start. With top, I see
some yarn process occupies more than 300% CPU. I tried to restore from an
old image, but still no luck. While in my local CI, it is stable. I guess
we may have a noisy neighborhood but no time to investigate. This public CI
is offline now, please use local environment for integration testing. The
image is there so one day we need it, we can quickly restore.
2018-02-22 11:57 GMT+08:00 Li Yang <li...@apache.org>:
> Cool ~~
>
> On Mon, Feb 12, 2018 at 6:04 PM, Luke Han <lu...@gmail.com> wrote:
>
> > This is great progress of our community, hope this could help
> > a little bit to speed up everyone's productivity.
> >
> > Thanks Shaofeng for all the efforts.
> >
> > Luke
> >
> > ShaoFeng Shi <sh...@apache.org>于2018年2月12日周一 下午5:54写道：
> >
> > > Hello Kylin developers,
> > >
> > > As you know, Kylin has a full regression test (also called "integration
> > > test"), which covers UT, cube build (batch & streaming), SQL queries
> and
> > > others. With this regression test, we can know whether a change is safe
> > or
> > > not (to some extent, not 100%). Many issues can be found with this
> test.
> > >
> > > Since this test need be executed in a Hadoop environment, at least be a
> > > Hadoop Sandbox with 10GB memory and 40 GB disk. Apache Foundation
> doesn't
> > > have the resource for this purpose. So in the past, we can only run it
> on
> > > the private server; The status and result are invisible for Kylin
> > > community, that impacts the communication and efficiency.
> > >
> > > Now, we received a donation to run such a dedicated CI server
> > > (T2.xlarge instance) on AWS for at least 6 months. It will run every 8
> > hrs
> > > (pull from master branch), each round takes about 1 hr; Once the test
> is
> > > failed, it will automatically send email to the one who breaks the CI.
> > > Later we plan to automate it with JIRA/Github workflow.
> > >
> > > http://34.226.50.254:8081/
> > >
> > > Anonymous user can only view the test result and log. The Hadoop YARN
> RM
> > > and history server are open for checking job status, you can access it
> at
> > > 8088 and 19888 port.
> > >
> > > The server information is also updated in the "CI and Code Analysis"
> > > section on https://kylin.apache.org/development/
> > >
> > > Let's use this CI to check the status and communicate on the specific
> > > problem. If you have any suggestion, please raise it in the dev mailing
> > > list.
> > >
> > > --
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > >
> >
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Jenkins for Apache Kylin Integration test
Posted by Li Yang <li...@apache.org>.
Cool ~~
On Mon, Feb 12, 2018 at 6:04 PM, Luke Han <lu...@gmail.com> wrote:
> This is great progress of our community, hope this could help
> a little bit to speed up everyone's productivity.
>
> Thanks Shaofeng for all the efforts.
>
> Luke
>
> ShaoFeng Shi <sh...@apache.org>于2018年2月12日周一 下午5:54写道：
>
> > Hello Kylin developers,
> >
> > As you know, Kylin has a full regression test (also called "integration
> > test"), which covers UT, cube build (batch & streaming), SQL queries and
> > others. With this regression test, we can know whether a change is safe
> or
> > not (to some extent, not 100%). Many issues can be found with this test.
> >
> > Since this test need be executed in a Hadoop environment, at least be a
> > Hadoop Sandbox with 10GB memory and 40 GB disk. Apache Foundation doesn't
> > have the resource for this purpose. So in the past, we can only run it on
> > the private server; The status and result are invisible for Kylin
> > community, that impacts the communication and efficiency.
> >
> > Now, we received a donation to run such a dedicated CI server
> > (T2.xlarge instance) on AWS for at least 6 months. It will run every 8
> hrs
> > (pull from master branch), each round takes about 1 hr; Once the test is
> > failed, it will automatically send email to the one who breaks the CI.
> > Later we plan to automate it with JIRA/Github workflow.
> >
> > http://34.226.50.254:8081/
> >
> > Anonymous user can only view the test result and log. The Hadoop YARN RM
> > and history server are open for checking job status, you can access it at
> > 8088 and 19888 port.
> >
> > The server information is also updated in the "CI and Code Analysis"
> > section on https://kylin.apache.org/development/
> >
> > Let's use this CI to check the status and communicate on the specific
> > problem. If you have any suggestion, please raise it in the dev mailing
> > list.
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>
Re: Jenkins for Apache Kylin Integration test
Posted by Luke Han <lu...@gmail.com>.
This is great progress of our community, hope this could help
a little bit to speed up everyone's productivity.
Thanks Shaofeng for all the efforts.
Luke
ShaoFeng Shi <sh...@apache.org>于2018年2月12日周一 下午5:54写道：
> Hello Kylin developers,
>
> As you know, Kylin has a full regression test (also called "integration
> test"), which covers UT, cube build (batch & streaming), SQL queries and
> others. With this regression test, we can know whether a change is safe or
> not (to some extent, not 100%). Many issues can be found with this test.
>
> Since this test need be executed in a Hadoop environment, at least be a
> Hadoop Sandbox with 10GB memory and 40 GB disk. Apache Foundation doesn't
> have the resource for this purpose. So in the past, we can only run it on
> the private server; The status and result are invisible for Kylin
> community, that impacts the communication and efficiency.
>
> Now, we received a donation to run such a dedicated CI server
> (T2.xlarge instance) on AWS for at least 6 months. It will run every 8 hrs
> (pull from master branch), each round takes about 1 hr; Once the test is
> failed, it will automatically send email to the one who breaks the CI.
> Later we plan to automate it with JIRA/Github workflow.
>
> http://34.226.50.254:8081/
>
> Anonymous user can only view the test result and log. The Hadoop YARN RM
> and history server are open for checking job status, you can access it at
> 8088 and 19888 port.
>
> The server information is also updated in the "CI and Code Analysis"
> section on https://kylin.apache.org/development/
>
> Let's use this CI to check the status and communicate on the specific
> problem. If you have any suggestion, please raise it in the dev mailing
> list.
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>