You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Neelesh Salian <ns...@cloudera.com> on 2016/10/22 19:56:12 UTC
A CI Server for Beam?
Hello,
I wanted to check to see if there is continuous integration service used to
build and test Beam similar to Travis.
Would certainly help speed up adding code to the project.
Thank you in advance.
--
Neelesh Srinivas Salian
Engineer
Re: A CI Server for Beam?
Posted by Neelesh Salian <ns...@cloudera.com>.
Found the yml file.
All good.
On Sat, Oct 22, 2016 at 12:56 PM, Neelesh Salian <ns...@cloudera.com>
wrote:
> Hello,
>
> I wanted to check to see if there is continuous integration service used
> to build and test Beam similar to Travis.
> Would certainly help speed up adding code to the project.
>
> Thank you in advance.
>
> --
> Neelesh Srinivas Salian
> Engineer
>
>
>
>
--
Neelesh Srinivas Salian
Customer Operations Engineer
Re: A CI Server for Beam?
Posted by Stephen Sisk <si...@google.com.INVALID>.
Hi Sergio -
as I mentioned in my email about integration testing sources[1], we'll also
need a cluster hosting mechanism for that and I'm checking out some options.
In addition to mesos+marathon, I'm checking out kubernetes and docker
swarm. I started out excited about mesos, but so far mesos has been more
complicated to set up on GCE (24 detailed steps you have to get right) than
kubernetes (1 step), but I'm curious to see if mesos+marathon is easier to
use with flink/spark etc..
[1]
https://lists.apache.org/thread.html/518d78478ae9b6a56d6a690033071aa6e3b817546499c4f0f18d247d@%3Cdev.beam.apache.org%3E
S
On Sun, Oct 23, 2016 at 11:35 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Yes. I also have a mesos/marathon to bootstrap a cluster a run test
> pipelines on it. We are discussing with Jason to use this with our Jenkins.
>
> Regards
> JB
>
>
>
> On Oct 24, 2016, 08:33, at 08:33, "Sergio Fernández" <wi...@apache.org>
> wrote:
> >On Mon, Oct 24, 2016 at 8:21 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> >wrote:
> >
> >> Yes. I also blogged about this.
> >>
> >
> >This article
> >
> http://blog.nanthrax.net/2016/08/apache-beam-in-action-same-code-several-execution-engines/
> >right?
> >
> >
> >> Basically I used one Maven profile per runner. Same code, different
> >> runners.
> >>
> >
> >Exactly that was the idea :-)
> >
> >Have you tried more complex setups? For instance, deploy to clusters
> >with
> >the different runners.
> >
> >Very interesting stuff, JB!
> >
> >Cheers,
> >
> >
> >
> >On Oct 24, 2016, 08:12, at 08:12, "Sergio Fernández"
> ><wi...@apache.org>
> >> wrote:
> >> >Related, but a bit off-topic: has anybody setup a Bean Pipeline CI
> >> >against
> >> >the different runners?
> >> >
> >> >I normally do this against a custom Docker-based setup, but just
> >> >against
> >> >Spark. Probably it'd be very interesting to run a pipeline against
> >> >different runners to get broader conclusions.
> >> >
> >> >If anybody has worked on a similar idea, I'd appreciate to discuss
> >it.
> >> >
> >> >Thanks in advance.
> >> >
> >> >On Mon, Oct 24, 2016 at 2:37 AM, Frances Perry
> ><fj...@google.com.invalid>
> >> >wrote:
> >> >
> >> >> Jason recently started documenting the project's testing setup.
> >See
> >> >> http://beam.incubator.apache.org/contribute/testing/
> >> >>
> >> >> On Sat, Oct 22, 2016 at 11:18 PM, Jean-Baptiste Onofré
> >> ><jb...@nanthrax.net>
> >> >> wrote:
> >> >>
> >> >> > Hi
> >> >> >
> >> >> > We use both Jenkins and Travis.
> >> >> >
> >> >> > Jenkins is on build.apache.org and you can find Travis yml in
> >> >source.
> >> >> >
> >> >> > Regards
> >> >> > JB
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Oct 22, 2016, 21:56, at 21:56, Neelesh Salian
> >> ><ns...@cloudera.com>
> >> >> > wrote:
> >> >> > >Hello,
> >> >> > >
> >> >> > >I wanted to check to see if there is continuous integration
> >> >service
> >> >> > >used to
> >> >> > >build and test Beam similar to Travis.
> >> >> > >Would certainly help speed up adding code to the project.
> >> >> > >
> >> >> > >Thank you in advance.
> >> >> > >
> >> >> > >--
> >> >> > >Neelesh Srinivas Salian
> >> >> > >Engineer
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> >--
> >> >Sergio Fernández
> >> >Partner Technology Manager
> >> >Redlink GmbH
> >> >m: +43 6602747925 <+43%20660%202747925>
> >> >e: sergio.fernandez@redlink.co
> >> >w: http://redlink.co
> >>
> >
> >
> >
> >--
> >Sergio Fernández
> >Partner Technology Manager
> >Redlink GmbH
> >m: +43 6602747925 <+43%20660%202747925>
> >e: sergio.fernandez@redlink.co
> >w: http://redlink.co
>
Re: A CI Server for Beam?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Yes. I also have a mesos/marathon to bootstrap a cluster a run test pipelines on it. We are discussing with Jason to use this with our Jenkins.
Regards
JB
\u2063\u200b
On Oct 24, 2016, 08:33, at 08:33, "Sergio Fern�ndez" <wi...@apache.org> wrote:
>On Mon, Oct 24, 2016 at 8:21 AM, Jean-Baptiste Onofr� <jb...@nanthrax.net>
>wrote:
>
>> Yes. I also blogged about this.
>>
>
>This article
>http://blog.nanthrax.net/2016/08/apache-beam-in-action-same-code-several-execution-engines/
>right?
>
>
>> Basically I used one Maven profile per runner. Same code, different
>> runners.
>>
>
>Exactly that was the idea :-)
>
>Have you tried more complex setups? For instance, deploy to clusters
>with
>the different runners.
>
>Very interesting stuff, JB!
>
>Cheers,
>
>
>
>On Oct 24, 2016, 08:12, at 08:12, "Sergio Fern�ndez"
><wi...@apache.org>
>> wrote:
>> >Related, but a bit off-topic: has anybody setup a Bean Pipeline CI
>> >against
>> >the different runners?
>> >
>> >I normally do this against a custom Docker-based setup, but just
>> >against
>> >Spark. Probably it'd be very interesting to run a pipeline against
>> >different runners to get broader conclusions.
>> >
>> >If anybody has worked on a similar idea, I'd appreciate to discuss
>it.
>> >
>> >Thanks in advance.
>> >
>> >On Mon, Oct 24, 2016 at 2:37 AM, Frances Perry
><fj...@google.com.invalid>
>> >wrote:
>> >
>> >> Jason recently started documenting the project's testing setup.
>See
>> >> http://beam.incubator.apache.org/contribute/testing/
>> >>
>> >> On Sat, Oct 22, 2016 at 11:18 PM, Jean-Baptiste Onofr�
>> ><jb...@nanthrax.net>
>> >> wrote:
>> >>
>> >> > Hi
>> >> >
>> >> > We use both Jenkins and Travis.
>> >> >
>> >> > Jenkins is on build.apache.org and you can find Travis yml in
>> >source.
>> >> >
>> >> > Regards
>> >> > JB
>> >> >
>> >> > \u2063\u200b
>> >> >
>> >> > On Oct 22, 2016, 21:56, at 21:56, Neelesh Salian
>> ><ns...@cloudera.com>
>> >> > wrote:
>> >> > >Hello,
>> >> > >
>> >> > >I wanted to check to see if there is continuous integration
>> >service
>> >> > >used to
>> >> > >build and test Beam similar to Travis.
>> >> > >Would certainly help speed up adding code to the project.
>> >> > >
>> >> > >Thank you in advance.
>> >> > >
>> >> > >--
>> >> > >Neelesh Srinivas Salian
>> >> > >Engineer
>> >> >
>> >>
>> >
>> >
>> >
>> >--
>> >Sergio Fern�ndez
>> >Partner Technology Manager
>> >Redlink GmbH
>> >m: +43 6602747925
>> >e: sergio.fernandez@redlink.co
>> >w: http://redlink.co
>>
>
>
>
>--
>Sergio Fern�ndez
>Partner Technology Manager
>Redlink GmbH
>m: +43 6602747925
>e: sergio.fernandez@redlink.co
>w: http://redlink.co
Re: A CI Server for Beam?
Posted by Sergio Fernández <wi...@apache.org>.
On Mon, Oct 24, 2016 at 8:21 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Yes. I also blogged about this.
>
This article
http://blog.nanthrax.net/2016/08/apache-beam-in-action-same-code-several-execution-engines/
right?
> Basically I used one Maven profile per runner. Same code, different
> runners.
>
Exactly that was the idea :-)
Have you tried more complex setups? For instance, deploy to clusters with
the different runners.
Very interesting stuff, JB!
Cheers,
On Oct 24, 2016, 08:12, at 08:12, "Sergio Fernández" <wi...@apache.org>
> wrote:
> >Related, but a bit off-topic: has anybody setup a Bean Pipeline CI
> >against
> >the different runners?
> >
> >I normally do this against a custom Docker-based setup, but just
> >against
> >Spark. Probably it'd be very interesting to run a pipeline against
> >different runners to get broader conclusions.
> >
> >If anybody has worked on a similar idea, I'd appreciate to discuss it.
> >
> >Thanks in advance.
> >
> >On Mon, Oct 24, 2016 at 2:37 AM, Frances Perry <fj...@google.com.invalid>
> >wrote:
> >
> >> Jason recently started documenting the project's testing setup. See
> >> http://beam.incubator.apache.org/contribute/testing/
> >>
> >> On Sat, Oct 22, 2016 at 11:18 PM, Jean-Baptiste Onofré
> ><jb...@nanthrax.net>
> >> wrote:
> >>
> >> > Hi
> >> >
> >> > We use both Jenkins and Travis.
> >> >
> >> > Jenkins is on build.apache.org and you can find Travis yml in
> >source.
> >> >
> >> > Regards
> >> > JB
> >> >
> >> >
> >> >
> >> > On Oct 22, 2016, 21:56, at 21:56, Neelesh Salian
> ><ns...@cloudera.com>
> >> > wrote:
> >> > >Hello,
> >> > >
> >> > >I wanted to check to see if there is continuous integration
> >service
> >> > >used to
> >> > >build and test Beam similar to Travis.
> >> > >Would certainly help speed up adding code to the project.
> >> > >
> >> > >Thank you in advance.
> >> > >
> >> > >--
> >> > >Neelesh Srinivas Salian
> >> > >Engineer
> >> >
> >>
> >
> >
> >
> >--
> >Sergio Fernández
> >Partner Technology Manager
> >Redlink GmbH
> >m: +43 6602747925
> >e: sergio.fernandez@redlink.co
> >w: http://redlink.co
>
--
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernandez@redlink.co
w: http://redlink.co
Re: A CI Server for Beam?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Yes. I also blogged about this.
Basically I used one Maven profile per runner. Same code, different runners.
Regards
JB
\u2063\u200b
On Oct 24, 2016, 08:12, at 08:12, "Sergio Fern�ndez" <wi...@apache.org> wrote:
>Related, but a bit off-topic: has anybody setup a Bean Pipeline CI
>against
>the different runners?
>
>I normally do this against a custom Docker-based setup, but just
>against
>Spark. Probably it'd be very interesting to run a pipeline against
>different runners to get broader conclusions.
>
>If anybody has worked on a similar idea, I'd appreciate to discuss it.
>
>Thanks in advance.
>
>On Mon, Oct 24, 2016 at 2:37 AM, Frances Perry <fj...@google.com.invalid>
>wrote:
>
>> Jason recently started documenting the project's testing setup. See
>> http://beam.incubator.apache.org/contribute/testing/
>>
>> On Sat, Oct 22, 2016 at 11:18 PM, Jean-Baptiste Onofr�
><jb...@nanthrax.net>
>> wrote:
>>
>> > Hi
>> >
>> > We use both Jenkins and Travis.
>> >
>> > Jenkins is on build.apache.org and you can find Travis yml in
>source.
>> >
>> > Regards
>> > JB
>> >
>> > \u2063\u200b
>> >
>> > On Oct 22, 2016, 21:56, at 21:56, Neelesh Salian
><ns...@cloudera.com>
>> > wrote:
>> > >Hello,
>> > >
>> > >I wanted to check to see if there is continuous integration
>service
>> > >used to
>> > >build and test Beam similar to Travis.
>> > >Would certainly help speed up adding code to the project.
>> > >
>> > >Thank you in advance.
>> > >
>> > >--
>> > >Neelesh Srinivas Salian
>> > >Engineer
>> >
>>
>
>
>
>--
>Sergio Fern�ndez
>Partner Technology Manager
>Redlink GmbH
>m: +43 6602747925
>e: sergio.fernandez@redlink.co
>w: http://redlink.co
Re: A CI Server for Beam?
Posted by Sergio Fernández <wi...@apache.org>.
Related, but a bit off-topic: has anybody setup a Bean Pipeline CI against
the different runners?
I normally do this against a custom Docker-based setup, but just against
Spark. Probably it'd be very interesting to run a pipeline against
different runners to get broader conclusions.
If anybody has worked on a similar idea, I'd appreciate to discuss it.
Thanks in advance.
On Mon, Oct 24, 2016 at 2:37 AM, Frances Perry <fj...@google.com.invalid>
wrote:
> Jason recently started documenting the project's testing setup. See
> http://beam.incubator.apache.org/contribute/testing/
>
> On Sat, Oct 22, 2016 at 11:18 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi
> >
> > We use both Jenkins and Travis.
> >
> > Jenkins is on build.apache.org and you can find Travis yml in source.
> >
> > Regards
> > JB
> >
> >
> >
> > On Oct 22, 2016, 21:56, at 21:56, Neelesh Salian <ns...@cloudera.com>
> > wrote:
> > >Hello,
> > >
> > >I wanted to check to see if there is continuous integration service
> > >used to
> > >build and test Beam similar to Travis.
> > >Would certainly help speed up adding code to the project.
> > >
> > >Thank you in advance.
> > >
> > >--
> > >Neelesh Srinivas Salian
> > >Engineer
> >
>
--
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernandez@redlink.co
w: http://redlink.co
Re: A CI Server for Beam?
Posted by Frances Perry <fj...@google.com.INVALID>.
Jason recently started documenting the project's testing setup. See
http://beam.incubator.apache.org/contribute/testing/
On Sat, Oct 22, 2016 at 11:18 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi
>
> We use both Jenkins and Travis.
>
> Jenkins is on build.apache.org and you can find Travis yml in source.
>
> Regards
> JB
>
>
>
> On Oct 22, 2016, 21:56, at 21:56, Neelesh Salian <ns...@cloudera.com>
> wrote:
> >Hello,
> >
> >I wanted to check to see if there is continuous integration service
> >used to
> >build and test Beam similar to Travis.
> >Would certainly help speed up adding code to the project.
> >
> >Thank you in advance.
> >
> >--
> >Neelesh Srinivas Salian
> >Engineer
>
Re: A CI Server for Beam?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi
We use both Jenkins and Travis.
Jenkins is on build.apache.org and you can find Travis yml in source.
Regards
JB
\u2063\u200b
On Oct 22, 2016, 21:56, at 21:56, Neelesh Salian <ns...@cloudera.com> wrote:
>Hello,
>
>I wanted to check to see if there is continuous integration service
>used to
>build and test Beam similar to Travis.
>Would certainly help speed up adding code to the project.
>
>Thank you in advance.
>
>--
>Neelesh Srinivas Salian
>Engineer