You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Stadin, Benjamin" <Be...@heidelberg-mobil.com> on 2016/11/21 18:37:38 UTC
Beam on Kubernetes
Hi,
Are there any examples for running Beam with Kubernetes?
~Ben
Re: Beam on Kubernetes
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Amit,
correct, Karaf/Cellar (DOSGi/RSA) runner is very similar (however
powered by Hazelcast).
Regards
JB
On 11/22/2016 11:06 AM, Amit Sela wrote:
> @JB: you started working on DOSGi/Karaf runner, no ? couldn't
> Kuberenetes-based services be similar ?
>
> @Ben: JB is right about the difference between the execution engine
> (Spark, Flink, Apex) and the resource manager (YARN/Mesos), but you
> might be able to use Beam directly against services on top of
> Kuberenetes, though in a very limited way (executing DoFns basically).
>
> I would be interested to hear more though, I'm not too familiar with
> Kubernetes (yet..) but as it becomes popular I'm wondering if/how it
> could converge with Beam.
>
> Amit.
>
> On Mon, Nov 21, 2016 at 9:29 PM Jean-Baptiste Onofr� <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> Hi Ben,
>
> I did examples with Mesos and Marathon, but not yet with Kubernetes.
> Basically, it depends of the execution engines: Beam runner targets an
> execution engine (like Spark or Flink).
>
> So, for instance, as Spark and Flink support Kubernetes, it should not
> be a problem (for Spark, regular spark-submit can work with Beam
> pipelines with Spark runner).
>
> Regards
> JB
>
> On 11/21/2016 07:37 PM, Stadin, Benjamin wrote:
> > Hi,
> >
> > Are there any examples for running Beam with Kubernetes?
> >
> > ~Ben
>
> --
> Jean-Baptiste Onofr�
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Beam on Kubernetes
Posted by Amit Sela <am...@gmail.com>.
@JB: you started working on DOSGi/Karaf runner, no ? couldn't
Kuberenetes-based services be similar ?
@Ben: JB is right about the difference between the execution engine (Spark,
Flink, Apex) and the resource manager (YARN/Mesos), but you might be able
to use Beam directly against services on top of Kuberenetes, though in a
very limited way (executing DoFns basically).
I would be interested to hear more though, I'm not too familiar with
Kubernetes (yet..) but as it becomes popular I'm wondering if/how it could
converge with Beam.
Amit.
On Mon, Nov 21, 2016 at 9:29 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Ben,
>
> I did examples with Mesos and Marathon, but not yet with Kubernetes.
> Basically, it depends of the execution engines: Beam runner targets an
> execution engine (like Spark or Flink).
>
> So, for instance, as Spark and Flink support Kubernetes, it should not
> be a problem (for Spark, regular spark-submit can work with Beam
> pipelines with Spark runner).
>
> Regards
> JB
>
> On 11/21/2016 07:37 PM, Stadin, Benjamin wrote:
> > Hi,
> >
> > Are there any examples for running Beam with Kubernetes?
> >
> > ~Ben
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Beam on Kubernetes
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Ben,
I did examples with Mesos and Marathon, but not yet with Kubernetes.
Basically, it depends of the execution engines: Beam runner targets an
execution engine (like Spark or Flink).
So, for instance, as Spark and Flink support Kubernetes, it should not
be a problem (for Spark, regular spark-submit can work with Beam
pipelines with Spark runner).
Regards
JB
On 11/21/2016 07:37 PM, Stadin, Benjamin wrote:
> Hi,
>
> Are there any examples for running Beam with Kubernetes?
>
> ~Ben
--
Jean-Baptiste Onofr
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com