You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by Hyung Sung Shim <hs...@nflabs.com> on 2015/12/21 03:56:17 UTC
[DISCUSS] CI system for zeppelin
Dear devs.
I'd like to propose new CI system for the Zeppelin project which is testing
in NFLabs.
I knew in some cases current build system (travis) can not help developping
Zeppelin.
1. It can't clsasloading test at the spark cluster because of now CI
environment is not the actual environment.
(zeppelin and Spark is running in the same host.)
2. It can't test on the various Operating System and JDK version
environments. (only build in the travis host OS and single version of JDK)
3. It can't test on the various spark cluster environments. (now only build
with the spark standalone.)
4. It can't parallel test because of build time. (now spark standalone test
takes 40 minutes.)
*. The proposed CI system can
1. clsasloading test at the spark cluster in actual environment by testing
Zeppelin in the docker container completely separated with Spark cluster.
(Spark cluster also runs on docker.)
2. test on the various Operating System and JDK version environments.
3. test on the various spark cluster environments.
- now we build zeppelin with the spark standalone and yarn and mesos
backends.
4. run parallel build
- It takes less than 20 minutes when we build the zeppelin with the spark
standalone and yarn and mesos backends concurrently.
*. Build system introduction.
Build system is simply composed of the Jenkins and Docker and some scripts.
You can get the informations (how to run and how to add new build features
etc..) here(https://github.com/NFLabs/zeppelin-ci).
*. Further discussion
1. How this proposed CI system can integrated to current Zeppelin github
repository (hook).
2. To run this CI system, we'll need infrastructrure.
I hope it helps both zeppelin developers and users.
Please share your opinion.
Thanks in advance!
Re: [DISCUSS] CI system for zeppelin
Posted by Alexander Bezzubov <bz...@apache.org>.
Thank you for a proposal - to have a containerised CI solution will allow
us to test with more version of interpreters and real clusters, as well as
reduce build times, which is awesome.
I wish we could also keep the simplicity of setup using `.travis.yaml`
rather then job configuration details in CI system, but that is minor.
Are there any plans to donate this work to ASF? If so, we could keep it in
i.e ./dev/ci/ after you think it's ready
So please keep us posted on the progress - indeed it would be great to see
it working on some repo.
As for infrastructure, it might be quite challenging to setup
builds.apache.org that way (I can make a research after the New Year), so
we might need to think more, what are the options.
On Mon, Dec 21, 2015 at 4:54 PM, Hyung Sung Shim <hs...@nflabs.com> wrote:
> Thank you for your feedback Dear Jongyoul Lee.
>
> 2015-12-20 23:47 GMT-08:00 Jongyoul Lee <jo...@gmail.com>:
>
> > Looks good!! This will reduce Zeppelin's testing issue.
> >
> > On Mon, Dec 21, 2015 at 4:42 PM, Hyung Sung Shim <hs...@nflabs.com>
> > wrote:
> >
> > > Thank you for your reply. moon. :)
> > >
> > > Now we are testing the CI in private.
> > > So CI test environments for anybody is going to be prepared.
> > > I'll let you know when CI test environments is ready.
> > >
> > > Thanks.
> > >
> > > 2015-12-20 22:16 GMT-08:00 moon soo Lee <mo...@apache.org>:
> > >
> > > > Thanks for proposing a new CI system.
> > > > It will definitely improves Zeppelin development.
> > > >
> > > > Is there any github repository that configured with this proposed CI
> > > > system? so anyone can make a pullrequest there and test it. That
> would
> > be
> > > > really helpful.
> > > >
> > > > There're jenkins that managed by ASF. https://ci.apache.org/#jenkins
> .
> > > > But not sure this CI can be run on ASF jenkins, while it requires
> > docker.
> > > > If someone can clarify, that would be appreciated.
> > > >
> > > > Thanks,
> > > > moon
> > > >
> > > > On Mon, Dec 21, 2015 at 11:59 AM Amos B. Elberg <
> amos.elberg@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > I think I know a PR you could use to test it out. :)
> > > > >
> > > > > From: Hyung Sung Shim <hs...@nflabs.com>
> > > > > Reply: dev@zeppelin.incubator.apache.org <
> > > > > dev@zeppelin.incubator.apache.org>
> > > > > Date: December 20, 2015 at 9:56:31 PM
> > > > > To: dev@zeppelin.incubator.apache.org <
> > > dev@zeppelin.incubator.apache.org
> > > > >
> > > > > Subject: [DISCUSS] CI system for zeppelin
> > > > >
> > > > > Dear devs.
> > > > >
> > > > > I'd like to propose new CI system for the Zeppelin project which is
> > > > testing
> > > > > in NFLabs.
> > > > >
> > > > > I knew in some cases current build system (travis) can not help
> > > > developping
> > > > > Zeppelin.
> > > > > 1. It can't clsasloading test at the spark cluster because of now
> CI
> > > > > environment is not the actual environment.
> > > > > (zeppelin and Spark is running in the same host.)
> > > > > 2. It can't test on the various Operating System and JDK version
> > > > > environments. (only build in the travis host OS and single version
> of
> > > > JDK)
> > > > > 3. It can't test on the various spark cluster environments. (now
> only
> > > > build
> > > > > with the spark standalone.)
> > > > > 4. It can't parallel test because of build time. (now spark
> > standalone
> > > > test
> > > > > takes 40 minutes.)
> > > > >
> > > > > *. The proposed CI system can
> > > > > 1. clsasloading test at the spark cluster in actual environment by
> > > > testing
> > > > > Zeppelin in the docker container completely separated with Spark
> > > cluster.
> > > > > (Spark cluster also runs on docker.)
> > > > > 2. test on the various Operating System and JDK version
> environments.
> > > > > 3. test on the various spark cluster environments.
> > > > > - now we build zeppelin with the spark standalone and yarn and
> mesos
> > > > > backends.
> > > > > 4. run parallel build
> > > > > - It takes less than 20 minutes when we build the zeppelin with the
> > > spark
> > > > > standalone and yarn and mesos backends concurrently.
> > > > >
> > > > > *. Build system introduction.
> > > > > Build system is simply composed of the Jenkins and Docker and some
> > > > scripts.
> > > > > You can get the informations (how to run and how to add new build
> > > > features
> > > > > etc..) here(https://github.com/NFLabs/zeppelin-ci).
> > > > >
> > > > > *. Further discussion
> > > > > 1. How this proposed CI system can integrated to current Zeppelin
> > > github
> > > > > repository (hook).
> > > > > 2. To run this CI system, we'll need infrastructrure.
> > > > >
> > > > > I hope it helps both zeppelin developers and users.
> > > > >
> > > > > Please share your opinion.
> > > > > Thanks in advance!
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > 이종열, Jongyoul Lee, 李宗烈
> > http://madeng.net
> >
>
Re: [DISCUSS] CI system for zeppelin
Posted by Hyung Sung Shim <hs...@nflabs.com>.
Thank you for your feedback Dear Jongyoul Lee.
2015-12-20 23:47 GMT-08:00 Jongyoul Lee <jo...@gmail.com>:
> Looks good!! This will reduce Zeppelin's testing issue.
>
> On Mon, Dec 21, 2015 at 4:42 PM, Hyung Sung Shim <hs...@nflabs.com>
> wrote:
>
> > Thank you for your reply. moon. :)
> >
> > Now we are testing the CI in private.
> > So CI test environments for anybody is going to be prepared.
> > I'll let you know when CI test environments is ready.
> >
> > Thanks.
> >
> > 2015-12-20 22:16 GMT-08:00 moon soo Lee <mo...@apache.org>:
> >
> > > Thanks for proposing a new CI system.
> > > It will definitely improves Zeppelin development.
> > >
> > > Is there any github repository that configured with this proposed CI
> > > system? so anyone can make a pullrequest there and test it. That would
> be
> > > really helpful.
> > >
> > > There're jenkins that managed by ASF. https://ci.apache.org/#jenkins.
> > > But not sure this CI can be run on ASF jenkins, while it requires
> docker.
> > > If someone can clarify, that would be appreciated.
> > >
> > > Thanks,
> > > moon
> > >
> > > On Mon, Dec 21, 2015 at 11:59 AM Amos B. Elberg <amos.elberg@gmail.com
> >
> > > wrote:
> > >
> > > > I think I know a PR you could use to test it out. :)
> > > >
> > > > From: Hyung Sung Shim <hs...@nflabs.com>
> > > > Reply: dev@zeppelin.incubator.apache.org <
> > > > dev@zeppelin.incubator.apache.org>
> > > > Date: December 20, 2015 at 9:56:31 PM
> > > > To: dev@zeppelin.incubator.apache.org <
> > dev@zeppelin.incubator.apache.org
> > > >
> > > > Subject: [DISCUSS] CI system for zeppelin
> > > >
> > > > Dear devs.
> > > >
> > > > I'd like to propose new CI system for the Zeppelin project which is
> > > testing
> > > > in NFLabs.
> > > >
> > > > I knew in some cases current build system (travis) can not help
> > > developping
> > > > Zeppelin.
> > > > 1. It can't clsasloading test at the spark cluster because of now CI
> > > > environment is not the actual environment.
> > > > (zeppelin and Spark is running in the same host.)
> > > > 2. It can't test on the various Operating System and JDK version
> > > > environments. (only build in the travis host OS and single version of
> > > JDK)
> > > > 3. It can't test on the various spark cluster environments. (now only
> > > build
> > > > with the spark standalone.)
> > > > 4. It can't parallel test because of build time. (now spark
> standalone
> > > test
> > > > takes 40 minutes.)
> > > >
> > > > *. The proposed CI system can
> > > > 1. clsasloading test at the spark cluster in actual environment by
> > > testing
> > > > Zeppelin in the docker container completely separated with Spark
> > cluster.
> > > > (Spark cluster also runs on docker.)
> > > > 2. test on the various Operating System and JDK version environments.
> > > > 3. test on the various spark cluster environments.
> > > > - now we build zeppelin with the spark standalone and yarn and mesos
> > > > backends.
> > > > 4. run parallel build
> > > > - It takes less than 20 minutes when we build the zeppelin with the
> > spark
> > > > standalone and yarn and mesos backends concurrently.
> > > >
> > > > *. Build system introduction.
> > > > Build system is simply composed of the Jenkins and Docker and some
> > > scripts.
> > > > You can get the informations (how to run and how to add new build
> > > features
> > > > etc..) here(https://github.com/NFLabs/zeppelin-ci).
> > > >
> > > > *. Further discussion
> > > > 1. How this proposed CI system can integrated to current Zeppelin
> > github
> > > > repository (hook).
> > > > 2. To run this CI system, we'll need infrastructrure.
> > > >
> > > > I hope it helps both zeppelin developers and users.
> > > >
> > > > Please share your opinion.
> > > > Thanks in advance!
> > > >
> > >
> >
>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
Re: [DISCUSS] CI system for zeppelin
Posted by Jongyoul Lee <jo...@gmail.com>.
Looks good!! This will reduce Zeppelin's testing issue.
On Mon, Dec 21, 2015 at 4:42 PM, Hyung Sung Shim <hs...@nflabs.com> wrote:
> Thank you for your reply. moon. :)
>
> Now we are testing the CI in private.
> So CI test environments for anybody is going to be prepared.
> I'll let you know when CI test environments is ready.
>
> Thanks.
>
> 2015-12-20 22:16 GMT-08:00 moon soo Lee <mo...@apache.org>:
>
> > Thanks for proposing a new CI system.
> > It will definitely improves Zeppelin development.
> >
> > Is there any github repository that configured with this proposed CI
> > system? so anyone can make a pullrequest there and test it. That would be
> > really helpful.
> >
> > There're jenkins that managed by ASF. https://ci.apache.org/#jenkins.
> > But not sure this CI can be run on ASF jenkins, while it requires docker.
> > If someone can clarify, that would be appreciated.
> >
> > Thanks,
> > moon
> >
> > On Mon, Dec 21, 2015 at 11:59 AM Amos B. Elberg <am...@gmail.com>
> > wrote:
> >
> > > I think I know a PR you could use to test it out. :)
> > >
> > > From: Hyung Sung Shim <hs...@nflabs.com>
> > > Reply: dev@zeppelin.incubator.apache.org <
> > > dev@zeppelin.incubator.apache.org>
> > > Date: December 20, 2015 at 9:56:31 PM
> > > To: dev@zeppelin.incubator.apache.org <
> dev@zeppelin.incubator.apache.org
> > >
> > > Subject: [DISCUSS] CI system for zeppelin
> > >
> > > Dear devs.
> > >
> > > I'd like to propose new CI system for the Zeppelin project which is
> > testing
> > > in NFLabs.
> > >
> > > I knew in some cases current build system (travis) can not help
> > developping
> > > Zeppelin.
> > > 1. It can't clsasloading test at the spark cluster because of now CI
> > > environment is not the actual environment.
> > > (zeppelin and Spark is running in the same host.)
> > > 2. It can't test on the various Operating System and JDK version
> > > environments. (only build in the travis host OS and single version of
> > JDK)
> > > 3. It can't test on the various spark cluster environments. (now only
> > build
> > > with the spark standalone.)
> > > 4. It can't parallel test because of build time. (now spark standalone
> > test
> > > takes 40 minutes.)
> > >
> > > *. The proposed CI system can
> > > 1. clsasloading test at the spark cluster in actual environment by
> > testing
> > > Zeppelin in the docker container completely separated with Spark
> cluster.
> > > (Spark cluster also runs on docker.)
> > > 2. test on the various Operating System and JDK version environments.
> > > 3. test on the various spark cluster environments.
> > > - now we build zeppelin with the spark standalone and yarn and mesos
> > > backends.
> > > 4. run parallel build
> > > - It takes less than 20 minutes when we build the zeppelin with the
> spark
> > > standalone and yarn and mesos backends concurrently.
> > >
> > > *. Build system introduction.
> > > Build system is simply composed of the Jenkins and Docker and some
> > scripts.
> > > You can get the informations (how to run and how to add new build
> > features
> > > etc..) here(https://github.com/NFLabs/zeppelin-ci).
> > >
> > > *. Further discussion
> > > 1. How this proposed CI system can integrated to current Zeppelin
> github
> > > repository (hook).
> > > 2. To run this CI system, we'll need infrastructrure.
> > >
> > > I hope it helps both zeppelin developers and users.
> > >
> > > Please share your opinion.
> > > Thanks in advance!
> > >
> >
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: [DISCUSS] CI system for zeppelin
Posted by Hyung Sung Shim <hs...@nflabs.com>.
Thank you for your reply. moon. :)
Now we are testing the CI in private.
So CI test environments for anybody is going to be prepared.
I'll let you know when CI test environments is ready.
Thanks.
2015-12-20 22:16 GMT-08:00 moon soo Lee <mo...@apache.org>:
> Thanks for proposing a new CI system.
> It will definitely improves Zeppelin development.
>
> Is there any github repository that configured with this proposed CI
> system? so anyone can make a pullrequest there and test it. That would be
> really helpful.
>
> There're jenkins that managed by ASF. https://ci.apache.org/#jenkins.
> But not sure this CI can be run on ASF jenkins, while it requires docker.
> If someone can clarify, that would be appreciated.
>
> Thanks,
> moon
>
> On Mon, Dec 21, 2015 at 11:59 AM Amos B. Elberg <am...@gmail.com>
> wrote:
>
> > I think I know a PR you could use to test it out. :)
> >
> > From: Hyung Sung Shim <hs...@nflabs.com>
> > Reply: dev@zeppelin.incubator.apache.org <
> > dev@zeppelin.incubator.apache.org>
> > Date: December 20, 2015 at 9:56:31 PM
> > To: dev@zeppelin.incubator.apache.org <dev@zeppelin.incubator.apache.org
> >
> > Subject: [DISCUSS] CI system for zeppelin
> >
> > Dear devs.
> >
> > I'd like to propose new CI system for the Zeppelin project which is
> testing
> > in NFLabs.
> >
> > I knew in some cases current build system (travis) can not help
> developping
> > Zeppelin.
> > 1. It can't clsasloading test at the spark cluster because of now CI
> > environment is not the actual environment.
> > (zeppelin and Spark is running in the same host.)
> > 2. It can't test on the various Operating System and JDK version
> > environments. (only build in the travis host OS and single version of
> JDK)
> > 3. It can't test on the various spark cluster environments. (now only
> build
> > with the spark standalone.)
> > 4. It can't parallel test because of build time. (now spark standalone
> test
> > takes 40 minutes.)
> >
> > *. The proposed CI system can
> > 1. clsasloading test at the spark cluster in actual environment by
> testing
> > Zeppelin in the docker container completely separated with Spark cluster.
> > (Spark cluster also runs on docker.)
> > 2. test on the various Operating System and JDK version environments.
> > 3. test on the various spark cluster environments.
> > - now we build zeppelin with the spark standalone and yarn and mesos
> > backends.
> > 4. run parallel build
> > - It takes less than 20 minutes when we build the zeppelin with the spark
> > standalone and yarn and mesos backends concurrently.
> >
> > *. Build system introduction.
> > Build system is simply composed of the Jenkins and Docker and some
> scripts.
> > You can get the informations (how to run and how to add new build
> features
> > etc..) here(https://github.com/NFLabs/zeppelin-ci).
> >
> > *. Further discussion
> > 1. How this proposed CI system can integrated to current Zeppelin github
> > repository (hook).
> > 2. To run this CI system, we'll need infrastructrure.
> >
> > I hope it helps both zeppelin developers and users.
> >
> > Please share your opinion.
> > Thanks in advance!
> >
>
Re: [DISCUSS] CI system for zeppelin
Posted by moon soo Lee <mo...@apache.org>.
Thanks for proposing a new CI system.
It will definitely improves Zeppelin development.
Is there any github repository that configured with this proposed CI
system? so anyone can make a pullrequest there and test it. That would be
really helpful.
There're jenkins that managed by ASF. https://ci.apache.org/#jenkins.
But not sure this CI can be run on ASF jenkins, while it requires docker.
If someone can clarify, that would be appreciated.
Thanks,
moon
On Mon, Dec 21, 2015 at 11:59 AM Amos B. Elberg <am...@gmail.com>
wrote:
> I think I know a PR you could use to test it out. :)
>
> From: Hyung Sung Shim <hs...@nflabs.com>
> Reply: dev@zeppelin.incubator.apache.org <
> dev@zeppelin.incubator.apache.org>
> Date: December 20, 2015 at 9:56:31 PM
> To: dev@zeppelin.incubator.apache.org <de...@zeppelin.incubator.apache.org>
> Subject: [DISCUSS] CI system for zeppelin
>
> Dear devs.
>
> I'd like to propose new CI system for the Zeppelin project which is testing
> in NFLabs.
>
> I knew in some cases current build system (travis) can not help developping
> Zeppelin.
> 1. It can't clsasloading test at the spark cluster because of now CI
> environment is not the actual environment.
> (zeppelin and Spark is running in the same host.)
> 2. It can't test on the various Operating System and JDK version
> environments. (only build in the travis host OS and single version of JDK)
> 3. It can't test on the various spark cluster environments. (now only build
> with the spark standalone.)
> 4. It can't parallel test because of build time. (now spark standalone test
> takes 40 minutes.)
>
> *. The proposed CI system can
> 1. clsasloading test at the spark cluster in actual environment by testing
> Zeppelin in the docker container completely separated with Spark cluster.
> (Spark cluster also runs on docker.)
> 2. test on the various Operating System and JDK version environments.
> 3. test on the various spark cluster environments.
> - now we build zeppelin with the spark standalone and yarn and mesos
> backends.
> 4. run parallel build
> - It takes less than 20 minutes when we build the zeppelin with the spark
> standalone and yarn and mesos backends concurrently.
>
> *. Build system introduction.
> Build system is simply composed of the Jenkins and Docker and some scripts.
> You can get the informations (how to run and how to add new build features
> etc..) here(https://github.com/NFLabs/zeppelin-ci).
>
> *. Further discussion
> 1. How this proposed CI system can integrated to current Zeppelin github
> repository (hook).
> 2. To run this CI system, we'll need infrastructrure.
>
> I hope it helps both zeppelin developers and users.
>
> Please share your opinion.
> Thanks in advance!
>
Re: [DISCUSS] CI system for zeppelin
Posted by "Amos B. Elberg" <am...@gmail.com>.
I think I know a PR you could use to test it out. :)
From: Hyung Sung Shim <hs...@nflabs.com>
Reply: dev@zeppelin.incubator.apache.org <de...@zeppelin.incubator.apache.org>
Date: December 20, 2015 at 9:56:31 PM
To: dev@zeppelin.incubator.apache.org <de...@zeppelin.incubator.apache.org>
Subject: [DISCUSS] CI system for zeppelin
Dear devs.
I'd like to propose new CI system for the Zeppelin project which is testing
in NFLabs.
I knew in some cases current build system (travis) can not help developping
Zeppelin.
1. It can't clsasloading test at the spark cluster because of now CI
environment is not the actual environment.
(zeppelin and Spark is running in the same host.)
2. It can't test on the various Operating System and JDK version
environments. (only build in the travis host OS and single version of JDK)
3. It can't test on the various spark cluster environments. (now only build
with the spark standalone.)
4. It can't parallel test because of build time. (now spark standalone test
takes 40 minutes.)
*. The proposed CI system can
1. clsasloading test at the spark cluster in actual environment by testing
Zeppelin in the docker container completely separated with Spark cluster.
(Spark cluster also runs on docker.)
2. test on the various Operating System and JDK version environments.
3. test on the various spark cluster environments.
- now we build zeppelin with the spark standalone and yarn and mesos
backends.
4. run parallel build
- It takes less than 20 minutes when we build the zeppelin with the spark
standalone and yarn and mesos backends concurrently.
*. Build system introduction.
Build system is simply composed of the Jenkins and Docker and some scripts.
You can get the informations (how to run and how to add new build features
etc..) here(https://github.com/NFLabs/zeppelin-ci).
*. Further discussion
1. How this proposed CI system can integrated to current Zeppelin github
repository (hook).
2. To run this CI system, we'll need infrastructrure.
I hope it helps both zeppelin developers and users.
Please share your opinion.
Thanks in advance!