You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Andrea Medeghini <An...@10XBANKING.COM> on 2019/09/10 11:38:30 UTC
How do I run Beam Python pipelines using Flink deployed on
Kubernetes?
Hello,
Can I run Beam Python pipelines with Flink deployed on Kubernetes?
I have successfully managed to run a Beam Python pipeline using the Portable runner and the job service pointing to a local Flink server running as Docker containers.
I was able to achieve that mounting the Docker socket in my Flink container and running Flink as root, so the class DockerEnvironmentFactory can create the Python harness container.
Unfortunately, I can't use the same solution when Flink is running in Kubernetes. Moreover, I don't want to create the Python harness container using the Docker command from my pods.
I noticed there is an implementation called ExternalEnvironmentFactory, but I am not sure how to use it.
Is there a way to deploy a side container and use a different factory to run the Python harness process?
What is the correct approach?
Thanks,
Andrea
[10x]<https://www.10xbanking.com/>
10x Banking
Please consider the environment before printing this e-mail
**********************************************************************************************************************************************************
This e-mail, including any attachments, is confidential. If you are not the intended recipient, you may not copy, forward, disclose or use any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete this message and any copies and attachments.
10x Future Technologies Services Limited is registered in England & Wales
Registered Number: 11262659
Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG
Re: How do I run Beam Python pipelines using Flink deployed on
Kubernetes?
Posted by Maximilian Michels <mx...@apache.org>.
Hi Andrea,
You could use the new worker_pool_main.py which was developed for the
Kubernetes use case. It works together with the external environment
factory.
Cheers,
Max
On 11.09.19 18:51, Lukasz Cwik wrote:
> Yes.
>
> On Wed, Sep 11, 2019 at 3:12 AM Andrea Medeghini
> <Andrea.Medeghini@10xbanking.com
> <ma...@10xbanking.com>> wrote:
>
> Hi Lukasz,____
>
> Thanks for your answer. ____
>
> I had a look at the ProcessEnvironmentFactory, and I understand that
> the factory is expecting to run the boot process which is in the
> python container (source file boot.go). Is that right?____
>
> __ __
>
> Best,____
>
> Andrea____
>
> __ __
>
> *From: *Lukasz Cwik <lcwik@google.com <ma...@google.com>>
> *Reply-To: *"user@beam.apache.org <ma...@beam.apache.org>"
> <user@beam.apache.org <ma...@beam.apache.org>>
> *Date: *Tuesday, 10 September 2019 at 23:19
> *To: *user <user@beam.apache.org <ma...@beam.apache.org>>
> *Subject: *Re: How do I run Beam Python pipelines using Flink
> deployed on Kubernetes?____
>
> __ __
>
> [CAUTION - EXTERNAL EMAIL]____
>
> External environments are mainly used for testing since they
> represent environments that are already running. There are other
> clever uses of it as well but unlikely what your looking for. ____
>
> __ __
>
> The docker way will be nice once there is integration between the
> FlinkRunner and its supported cluster managers to automatically
> manage the SDK container instances. Until that exists you could look
> at using the process environment (it launches the SDK as a child
> process beside the Flink workers).____
>
> __ __
>
> On Tue, Sep 10, 2019 at 4:38 AM Andrea Medeghini
> <Andrea.Medeghini@10xbanking.com
> <ma...@10xbanking.com>> wrote:____
>
> Hello,____
>
> Can I run Beam Python pipelines with Flink deployed on
> Kubernetes?____
>
> ____
>
> I have successfully managed to run a Beam Python pipeline using
> the Portable runner and the job service pointing to a local
> Flink server running as Docker containers.____
>
> I was able to achieve that mounting the Docker socket in my
> Flink container and running Flink as root, so the class
> DockerEnvironmentFactory can create the Python harness
> container. ____
>
> Unfortunately, I can't use the same solution when Flink is
> running in Kubernetes. Moreover, I don't want to create the
> Python harness container using the Docker command from my pods.____
>
> I noticed there is an implementation called
> ExternalEnvironmentFactory, but I am not sure how to use it.____
>
> Is there a way to deploy a side container and use a different
> factory to run the Python harness process?____
>
> What is the correct approach?____
>
> ____
>
> Thanks,____
>
> Andrea____
>
> ____
>
> ____
>
> __ __
>
> Image removed by sender. 10x <https://www.10xbanking.com/>____
>
>
> 10x Banking
>
> Please consider the environment before printing this e-mail
> **********************************************************************************************************************************************************
> This e-mail, including any attachments, is confidential. If you
> are not the intended recipient, you may not copy, forward,
> disclose or use any part of it. If you have received this e-mail
> in error, please notify the sender immediately by return e-mail
> and delete this message and any copies and attachments.
>
> 10x Future Technologies Services Limited is registered in
> England & Wales
> Registered Number: 11262659
> Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG ____
>
>
>
> 10x <https://www.10xbanking.com/>
>
> 10x Banking
>
> Please consider the environment before printing this e-mail
> **********************************************************************************************************************************************************
>
> This e-mail, including any attachments, is confidential. If you are
> not the intended recipient, you may not copy, forward, disclose or
> use any part of it. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete
> this message and any copies and attachments.
>
> 10x Future Technologies Services Limited is registered in England &
> Wales
> Registered Number: 11262659
> Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG
>
Re: How do I run Beam Python pipelines using Flink deployed on Kubernetes?
Posted by Lukasz Cwik <lc...@google.com>.
Yes.
On Wed, Sep 11, 2019 at 3:12 AM Andrea Medeghini <
Andrea.Medeghini@10xbanking.com> wrote:
> Hi Lukasz,
>
> Thanks for your answer.
>
> I had a look at the ProcessEnvironmentFactory, and I understand that the
> factory is expecting to run the boot process which is in the python
> container (source file boot.go). Is that right?
>
>
>
> Best,
>
> Andrea
>
>
>
> *From: *Lukasz Cwik <lc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Tuesday, 10 September 2019 at 23:19
> *To: *user <us...@beam.apache.org>
> *Subject: *Re: How do I run Beam Python pipelines using Flink deployed on
> Kubernetes?
>
>
>
> [CAUTION - EXTERNAL EMAIL]
>
> External environments are mainly used for testing since they represent
> environments that are already running. There are other clever uses of it as
> well but unlikely what your looking for.
>
>
>
> The docker way will be nice once there is integration between the
> FlinkRunner and its supported cluster managers to automatically manage the
> SDK container instances. Until that exists you could look at using the
> process environment (it launches the SDK as a child process beside the
> Flink workers).
>
>
>
> On Tue, Sep 10, 2019 at 4:38 AM Andrea Medeghini <
> Andrea.Medeghini@10xbanking.com> wrote:
>
> Hello,
>
> Can I run Beam Python pipelines with Flink deployed on Kubernetes?
>
>
>
> I have successfully managed to run a Beam Python pipeline using the
> Portable runner and the job service pointing to a local Flink server
> running as Docker containers.
>
> I was able to achieve that mounting the Docker socket in my Flink
> container and running Flink as root, so the class DockerEnvironmentFactory
> can create the Python harness container.
>
> Unfortunately, I can't use the same solution when Flink is running in
> Kubernetes. Moreover, I don't want to create the Python harness container
> using the Docker command from my pods.
>
> I noticed there is an implementation called ExternalEnvironmentFactory,
> but I am not sure how to use it.
>
> Is there a way to deploy a side container and use a different factory to
> run the Python harness process?
>
> What is the correct approach?
>
>
>
> Thanks,
>
> Andrea
>
>
>
>
>
>
>
> [image: Image removed by sender. 10x] <https://www.10xbanking.com/>
>
>
> 10x Banking
>
> Please consider the environment before printing this e-mail
>
> **********************************************************************************************************************************************************
> This e-mail, including any attachments, is confidential. If you are not
> the intended recipient, you may not copy, forward, disclose or use any part
> of it. If you have received this e-mail in error, please notify the sender
> immediately by return e-mail and delete this message and any copies and
> attachments.
>
> 10x Future Technologies Services Limited is registered in England & Wales
> Registered Number: 11262659
> Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG
>
>
>
> [image: 10x] <https://www.10xbanking.com/>
>
> 10x Banking
>
> Please consider the environment before printing this e-mail
>
> **********************************************************************************************************************************************************
> This e-mail, including any attachments, is confidential. If you are not
> the intended recipient, you may not copy, forward, disclose or use any part
> of it. If you have received this e-mail in error, please notify the sender
> immediately by return e-mail and delete this message and any copies and
> attachments.
>
> 10x Future Technologies Services Limited is registered in England & Wales
> Registered Number: 11262659
> Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG
>
Re: How do I run Beam Python pipelines using Flink deployed on
Kubernetes?
Posted by Andrea Medeghini <An...@10XBANKING.COM>.
Hi Lukasz,
Thanks for your answer.
I had a look at the ProcessEnvironmentFactory, and I understand that the factory is expecting to run the boot process which is in the python container (source file boot.go). Is that right?
Best,
Andrea
From: Lukasz Cwik <lc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Tuesday, 10 September 2019 at 23:19
To: user <us...@beam.apache.org>
Subject: Re: How do I run Beam Python pipelines using Flink deployed on Kubernetes?
[CAUTION - EXTERNAL EMAIL]
External environments are mainly used for testing since they represent environments that are already running. There are other clever uses of it as well but unlikely what your looking for.
The docker way will be nice once there is integration between the FlinkRunner and its supported cluster managers to automatically manage the SDK container instances. Until that exists you could look at using the process environment (it launches the SDK as a child process beside the Flink workers).
On Tue, Sep 10, 2019 at 4:38 AM Andrea Medeghini <An...@10xbanking.com>> wrote:
Hello,
Can I run Beam Python pipelines with Flink deployed on Kubernetes?
I have successfully managed to run a Beam Python pipeline using the Portable runner and the job service pointing to a local Flink server running as Docker containers.
I was able to achieve that mounting the Docker socket in my Flink container and running Flink as root, so the class DockerEnvironmentFactory can create the Python harness container.
Unfortunately, I can't use the same solution when Flink is running in Kubernetes. Moreover, I don't want to create the Python harness container using the Docker command from my pods.
I noticed there is an implementation called ExternalEnvironmentFactory, but I am not sure how to use it.
Is there a way to deploy a side container and use a different factory to run the Python harness process?
What is the correct approach?
Thanks,
Andrea
[Image removed by sender. 10x]<https://www.10xbanking.com/>
10x Banking
Please consider the environment before printing this e-mail
**********************************************************************************************************************************************************
This e-mail, including any attachments, is confidential. If you are not the intended recipient, you may not copy, forward, disclose or use any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete this message and any copies and attachments.
10x Future Technologies Services Limited is registered in England & Wales
Registered Number: 11262659
Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG
[10x]<https://www.10xbanking.com/>
10x Banking
Please consider the environment before printing this e-mail
**********************************************************************************************************************************************************
This e-mail, including any attachments, is confidential. If you are not the intended recipient, you may not copy, forward, disclose or use any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete this message and any copies and attachments.
10x Future Technologies Services Limited is registered in England & Wales
Registered Number: 11262659
Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG
Re: How do I run Beam Python pipelines using Flink deployed on Kubernetes?
Posted by Lukasz Cwik <lc...@google.com>.
External environments are mainly used for testing since they represent
environments that are already running. There are other clever uses of it as
well but unlikely what your looking for.
The docker way will be nice once there is integration between the
FlinkRunner and its supported cluster managers to automatically manage the
SDK container instances. Until that exists you could look at using the
process environment (it launches the SDK as a child process beside the
Flink workers).
On Tue, Sep 10, 2019 at 4:38 AM Andrea Medeghini <
Andrea.Medeghini@10xbanking.com> wrote:
> Hello,
>
> Can I run Beam Python pipelines with Flink deployed on Kubernetes?
>
>
>
> I have successfully managed to run a Beam Python pipeline using the
> Portable runner and the job service pointing to a local Flink server
> running as Docker containers.
>
> I was able to achieve that mounting the Docker socket in my Flink
> container and running Flink as root, so the class DockerEnvironmentFactory
> can create the Python harness container.
>
> Unfortunately, I can't use the same solution when Flink is running in
> Kubernetes. Moreover, I don't want to create the Python harness container
> using the Docker command from my pods.
>
> I noticed there is an implementation called ExternalEnvironmentFactory,
> but I am not sure how to use it.
>
> Is there a way to deploy a side container and use a different factory to
> run the Python harness process?
>
> What is the correct approach?
>
>
>
> Thanks,
>
> Andrea
>
>
>
>
>
>
> [image: 10x] <https://www.10xbanking.com/>
>
> 10x Banking
>
> Please consider the environment before printing this e-mail
>
> **********************************************************************************************************************************************************
> This e-mail, including any attachments, is confidential. If you are not
> the intended recipient, you may not copy, forward, disclose or use any part
> of it. If you have received this e-mail in error, please notify the sender
> immediately by return e-mail and delete this message and any copies and
> attachments.
>
> 10x Future Technologies Services Limited is registered in England & Wales
> Registered Number: 11262659
> Registered Address: 5 Howick Place, Victoria, London, SW1P 1WG
>