You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Chakravarthy varaga <ch...@gmail.com> on 2017/03/23 16:20:10 UTC
deploying flink cluster in AWS - Containerized
Hi Team,
We are doing a PoC to deploy Flink cluster on AWS. All runtime
components will be dockerized.
I have few questions in relation to discover & security:
1. How does Job Manager discover task managers? Do they talk to over
TCP ?
2. If the runtime components TM, JM are containerized how are the IPs
resolved dynamically? Basically do I have to configure the JM with the
hostnames of the TMs. If so, if the TMs are on ephemeral IPs and on restart
of TM how does the job manager know the TM's (IP/Host). Before I go into
DNS and subnets, I'd like to understand how they disvoer & talk to each
other !
3. I went through some Flink materials on the web on security
precisely on kerebros. However how do I ensure that user level
authentication is applied on job management. For ex., only certain users
are allowed to start/stop jobs ? This question is in relation to if flink
is deployed as standalone-cluster
Thanks & Regards
CVP
Re: deploying flink cluster in AWS - Containerized
Posted by Till Rohrmann <tr...@apache.org>.
A small addition for the component discovery: Flink works so that the
TaskManagers register at the JobManager. This means that the TaskManager
have to somehow retrieve the JobManager's address. Either you do it as
Philippe described it or you use the HA mode with ZooKeeper. In the latter
case, the leading JobManager will write its address to ZooKeeper from where
the TaskManager's can retrieve it.
Cheers,
Till
On Fri, Mar 24, 2017 at 11:34 AM, Philippe Caparroy <
philippe.caparroy@orange.fr> wrote:
> Weave allows encryption of the vpn, and your Flink containers can be
> secured using kerberos https://ci.apache.org/projects/flink/flink-docs-
> release-1.2/setup/config.html#kerberos-based-security.
>
> Le 24 mars 2017 à 11:16, Chakravarthy varaga <ch...@gmail.com> a
> écrit :
>
> Hi,
>
> Thanks for your inputs. It kind of makes sense to use a container
> orchestrator to plough through networking under the hood.
> How do you tackle security?
>
> I don't see a way to authorize users for job management. I understand
> few orchestrators provide name space isolation and security policies on
> these. How do this work if the flink cluster is standalone on AWS ?
>
>
> Best Regards
> CVP
>
> On Fri, Mar 24, 2017 at 8:49 AM, Philippe Caparroy <
> philippe.caparroy@orange.fr> wrote:
>
>> Hi,
>>
>> If I can give my 2 cents.
>>
>> One simple solution to your problem is using weave (
>> https://www.weave.works/) a Docker network plugin.
>>
>> We’ve been working for more then year with dockerized
>> (Flink+zookeeper+Yarn+spark+Kafka+hadoop+elasticsearch ) cluster using
>> weave.
>>
>> Design your docker container so that you can set the cluster size on
>> startup (number of task manager stand job managers should be a docker arg).
>>
>> Weave will act as a switch with dns server embedded. Your containers will
>> only have to be configured with hosts names such as :
>> flink.taskmanager-1.weave.local, link.taskmanager-2.weave.local,
>> flink.jobmanager-1.weave.local, and so on …
>>
>> with flink Yarn it’s even simpler, but you have to dockerize a Yarn
>> cluster.
>>
>> It works perfectly on bare metal machines and in the cloud
>> (digital-ocean, aws,…).
>>
>>
>>
>> Le 24 mars 2017 à 08:50, Chakravarthy varaga <ch...@gmail.com>
>> a écrit :
>>
>> Hi,
>>
>> I request someone to help here.
>>
>> Best Regards
>> CVP
>>
>> On Thu, Mar 23, 2017 at 10:13 PM, Chakravarthy varaga <
>> chakravarthyvp@gmail.com> wrote:
>>
>>> I'm looking forward to hearing some updates on this...
>>>
>>> Any help here is highly appreciated !!
>>>
>>> On Thu, Mar 23, 2017 at 4:20 PM, Chakravarthy varaga <
>>> chakravarthyvp@gmail.com> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> We are doing a PoC to deploy Flink cluster on AWS. All runtime
>>>> components will be dockerized.
>>>>
>>>> I have few questions in relation to discover & security:
>>>>
>>>> 1. How does Job Manager discover task managers? Do they talk to
>>>> over TCP ?
>>>>
>>>> 2. If the runtime components TM, JM are containerized how are the
>>>> IPs resolved dynamically? Basically do I have to configure the JM with the
>>>> hostnames of the TMs. If so, if the TMs are on ephemeral IPs and on restart
>>>> of TM how does the job manager know the TM's (IP/Host). Before I go into
>>>> DNS and subnets, I'd like to understand how they disvoer & talk to each
>>>> other !
>>>>
>>>> 3. I went through some Flink materials on the web on security
>>>> precisely on kerebros. However how do I ensure that user level
>>>> authentication is applied on job management. For ex., only certain users
>>>> are allowed to start/stop jobs ? This question is in relation to if flink
>>>> is deployed as standalone-cluster
>>>>
>>>> Thanks & Regards
>>>> CVP
>>>>
>>>
>>>
>>
>>
>
>
Re: deploying flink cluster in AWS - Containerized
Posted by Philippe Caparroy <ph...@orange.fr>.
Weave allows encryption of the vpn, and your Flink containers can be secured using kerberos https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/config.html#kerberos-based-security.
> Le 24 mars 2017 à 11:16, Chakravarthy varaga <ch...@gmail.com> a écrit :
>
> Hi,
>
> Thanks for your inputs. It kind of makes sense to use a container orchestrator to plough through networking under the hood.
> How do you tackle security?
>
> I don't see a way to authorize users for job management. I understand few orchestrators provide name space isolation and security policies on these. How do this work if the flink cluster is standalone on AWS ?
>
>
> Best Regards
> CVP
>
> On Fri, Mar 24, 2017 at 8:49 AM, Philippe Caparroy <philippe.caparroy@orange.fr <ma...@orange.fr>> wrote:
> Hi,
>
> If I can give my 2 cents.
>
> One simple solution to your problem is using weave (https://www.weave.works/ <https://www.weave.works/>) a Docker network plugin.
>
> We’ve been working for more then year with dockerized (Flink+zookeeper+Yarn+spark+Kafka+hadoop+elasticsearch ) cluster using weave.
>
> Design your docker container so that you can set the cluster size on startup (number of task manager stand job managers should be a docker arg).
>
> Weave will act as a switch with dns server embedded. Your containers will only have to be configured with hosts names such as : flink.taskmanager-1.weave.local, link.taskmanager-2.weave.local, flink.jobmanager-1.weave.local, and so on …
>
> with flink Yarn it’s even simpler, but you have to dockerize a Yarn cluster.
>
> It works perfectly on bare metal machines and in the cloud (digital-ocean, aws,…).
>
>
>
>> Le 24 mars 2017 à 08:50, Chakravarthy varaga <chakravarthyvp@gmail.com <ma...@gmail.com>> a écrit :
>>
>> Hi,
>>
>> I request someone to help here.
>>
>> Best Regards
>> CVP
>>
>> On Thu, Mar 23, 2017 at 10:13 PM, Chakravarthy varaga <chakravarthyvp@gmail.com <ma...@gmail.com>> wrote:
>> I'm looking forward to hearing some updates on this...
>>
>> Any help here is highly appreciated !!
>>
>> On Thu, Mar 23, 2017 at 4:20 PM, Chakravarthy varaga <chakravarthyvp@gmail.com <ma...@gmail.com>> wrote:
>> Hi Team,
>>
>> We are doing a PoC to deploy Flink cluster on AWS. All runtime components will be dockerized.
>>
>> I have few questions in relation to discover & security:
>>
>> 1. How does Job Manager discover task managers? Do they talk to over TCP ?
>>
>> 2. If the runtime components TM, JM are containerized how are the IPs resolved dynamically? Basically do I have to configure the JM with the hostnames of the TMs. If so, if the TMs are on ephemeral IPs and on restart of TM how does the job manager know the TM's (IP/Host). Before I go into DNS and subnets, I'd like to understand how they disvoer & talk to each other !
>>
>> 3. I went through some Flink materials on the web on security precisely on kerebros. However how do I ensure that user level authentication is applied on job management. For ex., only certain users are allowed to start/stop jobs ? This question is in relation to if flink is deployed as standalone-cluster
>>
>> Thanks & Regards
>> CVP
>>
>>
>
>
Re: deploying flink cluster in AWS - Containerized
Posted by Chakravarthy varaga <ch...@gmail.com>.
Hi,
Thanks for your inputs. It kind of makes sense to use a container
orchestrator to plough through networking under the hood.
How do you tackle security?
I don't see a way to authorize users for job management. I understand
few orchestrators provide name space isolation and security policies on
these. How do this work if the flink cluster is standalone on AWS ?
Best Regards
CVP
On Fri, Mar 24, 2017 at 8:49 AM, Philippe Caparroy <
philippe.caparroy@orange.fr> wrote:
> Hi,
>
> If I can give my 2 cents.
>
> One simple solution to your problem is using weave (
> https://www.weave.works/) a Docker network plugin.
>
> We’ve been working for more then year with dockerized
> (Flink+zookeeper+Yarn+spark+Kafka+hadoop+elasticsearch ) cluster using
> weave.
>
> Design your docker container so that you can set the cluster size on
> startup (number of task manager stand job managers should be a docker arg).
>
> Weave will act as a switch with dns server embedded. Your containers will
> only have to be configured with hosts names such as :
> flink.taskmanager-1.weave.local, link.taskmanager-2.weave.local,
> flink.jobmanager-1.weave.local, and so on …
>
> with flink Yarn it’s even simpler, but you have to dockerize a Yarn
> cluster.
>
> It works perfectly on bare metal machines and in the cloud (digital-ocean,
> aws,…).
>
>
>
> Le 24 mars 2017 à 08:50, Chakravarthy varaga <ch...@gmail.com> a
> écrit :
>
> Hi,
>
> I request someone to help here.
>
> Best Regards
> CVP
>
> On Thu, Mar 23, 2017 at 10:13 PM, Chakravarthy varaga <
> chakravarthyvp@gmail.com> wrote:
>
>> I'm looking forward to hearing some updates on this...
>>
>> Any help here is highly appreciated !!
>>
>> On Thu, Mar 23, 2017 at 4:20 PM, Chakravarthy varaga <
>> chakravarthyvp@gmail.com> wrote:
>>
>>> Hi Team,
>>>
>>> We are doing a PoC to deploy Flink cluster on AWS. All runtime
>>> components will be dockerized.
>>>
>>> I have few questions in relation to discover & security:
>>>
>>> 1. How does Job Manager discover task managers? Do they talk to
>>> over TCP ?
>>>
>>> 2. If the runtime components TM, JM are containerized how are the
>>> IPs resolved dynamically? Basically do I have to configure the JM with the
>>> hostnames of the TMs. If so, if the TMs are on ephemeral IPs and on restart
>>> of TM how does the job manager know the TM's (IP/Host). Before I go into
>>> DNS and subnets, I'd like to understand how they disvoer & talk to each
>>> other !
>>>
>>> 3. I went through some Flink materials on the web on security
>>> precisely on kerebros. However how do I ensure that user level
>>> authentication is applied on job management. For ex., only certain users
>>> are allowed to start/stop jobs ? This question is in relation to if flink
>>> is deployed as standalone-cluster
>>>
>>> Thanks & Regards
>>> CVP
>>>
>>
>>
>
>
Re: deploying flink cluster in AWS - Containerized
Posted by Philippe Caparroy <ph...@orange.fr>.
Hi,
If I can give my 2 cents.
One simple solution to your problem is using weave (https://www.weave.works/) a Docker network plugin.
We’ve been working for more then year with dockerized (Flink+zookeeper+Yarn+spark+Kafka+hadoop+elasticsearch ) cluster using weave.
Design your docker container so that you can set the cluster size on startup (number of task manager stand job managers should be a docker arg).
Weave will act as a switch with dns server embedded. Your containers will only have to be configured with hosts names such as : flink.taskmanager-1.weave.local, link.taskmanager-2.weave.local, flink.jobmanager-1.weave.local, and so on …
with flink Yarn it’s even simpler, but you have to dockerize a Yarn cluster.
It works perfectly on bare metal machines and in the cloud (digital-ocean, aws,…).
> Le 24 mars 2017 à 08:50, Chakravarthy varaga <ch...@gmail.com> a écrit :
>
> Hi,
>
> I request someone to help here.
>
> Best Regards
> CVP
>
> On Thu, Mar 23, 2017 at 10:13 PM, Chakravarthy varaga <chakravarthyvp@gmail.com <ma...@gmail.com>> wrote:
> I'm looking forward to hearing some updates on this...
>
> Any help here is highly appreciated !!
>
> On Thu, Mar 23, 2017 at 4:20 PM, Chakravarthy varaga <chakravarthyvp@gmail.com <ma...@gmail.com>> wrote:
> Hi Team,
>
> We are doing a PoC to deploy Flink cluster on AWS. All runtime components will be dockerized.
>
> I have few questions in relation to discover & security:
>
> 1. How does Job Manager discover task managers? Do they talk to over TCP ?
>
> 2. If the runtime components TM, JM are containerized how are the IPs resolved dynamically? Basically do I have to configure the JM with the hostnames of the TMs. If so, if the TMs are on ephemeral IPs and on restart of TM how does the job manager know the TM's (IP/Host). Before I go into DNS and subnets, I'd like to understand how they disvoer & talk to each other !
>
> 3. I went through some Flink materials on the web on security precisely on kerebros. However how do I ensure that user level authentication is applied on job management. For ex., only certain users are allowed to start/stop jobs ? This question is in relation to if flink is deployed as standalone-cluster
>
> Thanks & Regards
> CVP
>
>
Re: deploying flink cluster in AWS - Containerized
Posted by Chakravarthy varaga <ch...@gmail.com>.
Hi,
I request someone to help here.
Best Regards
CVP
On Thu, Mar 23, 2017 at 10:13 PM, Chakravarthy varaga <
chakravarthyvp@gmail.com> wrote:
> I'm looking forward to hearing some updates on this...
>
> Any help here is highly appreciated !!
>
> On Thu, Mar 23, 2017 at 4:20 PM, Chakravarthy varaga <
> chakravarthyvp@gmail.com> wrote:
>
>> Hi Team,
>>
>> We are doing a PoC to deploy Flink cluster on AWS. All runtime
>> components will be dockerized.
>>
>> I have few questions in relation to discover & security:
>>
>> 1. How does Job Manager discover task managers? Do they talk to
>> over TCP ?
>>
>> 2. If the runtime components TM, JM are containerized how are the
>> IPs resolved dynamically? Basically do I have to configure the JM with the
>> hostnames of the TMs. If so, if the TMs are on ephemeral IPs and on restart
>> of TM how does the job manager know the TM's (IP/Host). Before I go into
>> DNS and subnets, I'd like to understand how they disvoer & talk to each
>> other !
>>
>> 3. I went through some Flink materials on the web on security
>> precisely on kerebros. However how do I ensure that user level
>> authentication is applied on job management. For ex., only certain users
>> are allowed to start/stop jobs ? This question is in relation to if flink
>> is deployed as standalone-cluster
>>
>> Thanks & Regards
>> CVP
>>
>
>
Re: deploying flink cluster in AWS - Containerized
Posted by Chakravarthy varaga <ch...@gmail.com>.
I'm looking forward to hearing some updates on this...
Any help here is highly appreciated !!
On Thu, Mar 23, 2017 at 4:20 PM, Chakravarthy varaga <
chakravarthyvp@gmail.com> wrote:
> Hi Team,
>
> We are doing a PoC to deploy Flink cluster on AWS. All runtime
> components will be dockerized.
>
> I have few questions in relation to discover & security:
>
> 1. How does Job Manager discover task managers? Do they talk to over
> TCP ?
>
> 2. If the runtime components TM, JM are containerized how are the
> IPs resolved dynamically? Basically do I have to configure the JM with the
> hostnames of the TMs. If so, if the TMs are on ephemeral IPs and on restart
> of TM how does the job manager know the TM's (IP/Host). Before I go into
> DNS and subnets, I'd like to understand how they disvoer & talk to each
> other !
>
> 3. I went through some Flink materials on the web on security
> precisely on kerebros. However how do I ensure that user level
> authentication is applied on job management. For ex., only certain users
> are allowed to start/stop jobs ? This question is in relation to if flink
> is deployed as standalone-cluster
>
> Thanks & Regards
> CVP
>