You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Stadin, Benjamin" <Be...@heidelberg-mobil.com> on 2016/11/21 18:37:38 UTC

Beam on Kubernetes

Hi,

Are there any examples for running Beam with Kubernetes?

~Ben

Re: Beam on Kubernetes

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Amit,

correct, Karaf/Cellar (DOSGi/RSA) runner is very similar (however 
powered by Hazelcast).

Regards
JB

On 11/22/2016 11:06 AM, Amit Sela wrote:
> @JB: you started working on DOSGi/Karaf runner, no ? couldn't
> Kuberenetes-based services be similar ?
>
> @Ben: JB is right about the difference between the execution engine
> (Spark, Flink, Apex) and the resource manager (YARN/Mesos), but you
> might be able to use Beam directly against services on top of
> Kuberenetes, though in a very limited way (executing DoFns basically).
>
> I would be interested to hear more though, I'm not too familiar with
> Kubernetes (yet..) but as it becomes popular I'm wondering if/how it
> could converge with Beam.
>
> Amit.
>
> On Mon, Nov 21, 2016 at 9:29 PM Jean-Baptiste Onofr� <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
>     Hi Ben,
>
>     I did examples with Mesos and Marathon, but not yet with Kubernetes.
>     Basically, it depends of the execution engines: Beam runner targets an
>     execution engine (like Spark or Flink).
>
>     So, for instance, as Spark and Flink support Kubernetes, it should not
>     be a problem (for Spark, regular spark-submit can work with Beam
>     pipelines with Spark runner).
>
>     Regards
>     JB
>
>     On 11/21/2016 07:37 PM, Stadin, Benjamin wrote:
>     > Hi,
>     >
>     > Are there any examples for running Beam with Kubernetes?
>     >
>     > ~Ben
>
>     --
>     Jean-Baptiste Onofr�
>     jbonofre@apache.org <ma...@apache.org>
>     http://blog.nanthrax.net
>     Talend - http://www.talend.com
>

-- 
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Beam on Kubernetes

Posted by Amit Sela <am...@gmail.com>.
@JB: you started working on DOSGi/Karaf runner, no ? couldn't
Kuberenetes-based services be similar ?

@Ben: JB is right about the difference between the execution engine (Spark,
Flink, Apex) and the resource manager (YARN/Mesos), but you might be able
to use Beam directly against services on top of Kuberenetes, though in a
very limited way (executing DoFns basically).

I would be interested to hear more though, I'm not too familiar with
Kubernetes (yet..) but as it becomes popular I'm wondering if/how it could
converge with Beam.

Amit.

On Mon, Nov 21, 2016 at 9:29 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi Ben,
>
> I did examples with Mesos and Marathon, but not yet with Kubernetes.
> Basically, it depends of the execution engines: Beam runner targets an
> execution engine (like Spark or Flink).
>
> So, for instance, as Spark and Flink support Kubernetes, it should not
> be a problem (for Spark, regular spark-submit can work with Beam
> pipelines with Spark runner).
>
> Regards
> JB
>
> On 11/21/2016 07:37 PM, Stadin, Benjamin wrote:
> > Hi,
> >
> > Are there any examples for running Beam with Kubernetes?
> >
> > ~Ben
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Beam on Kubernetes

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Ben,

I did examples with Mesos and Marathon, but not yet with Kubernetes. 
Basically, it depends of the execution engines: Beam runner targets an 
execution engine (like Spark or Flink).

So, for instance, as Spark and Flink support Kubernetes, it should not 
be a problem (for Spark, regular spark-submit can work with Beam 
pipelines with Spark runner).

Regards
JB

On 11/21/2016 07:37 PM, Stadin, Benjamin wrote:
> Hi,
>
> Are there any examples for running Beam with Kubernetes?
>
> ~Ben

-- 
Jean-Baptiste Onofr
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com