You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Rahul Raj <ra...@gmail.com> on 2017/09/26 07:21:21 UTC
Flink Application Jar file on Docker container
Currently I have a Flink Application Jar file running on Mesos cluster. The
flink application simply reads data from Kafka and put it to HDFS.
Now we are planning to create a docker image to run this application jar
file inside docker containers on Mesos cluster via Marathon.
Below are the questions that I am looking answers for:
1. While building the docker image, how do I include flink-1.3.2 set up
and my mesos config in flink?
2. How shall I run my existing flink application jar?
3. Will running my flink application jar on docker containers will run it
on mesos slaves on different docker containers? How docker , Flink, mesos ,
Marathon will work together in my case?
Rahul Raj
Re: Flink Application Jar file on Docker container
Posted by Eron Wright <er...@gmail.com>.
There was also a talk about containerization at Flink Forward that touches
on some of your questions.
https://www.youtube.com/watch?v=w721NI-mtAA&t=2s&list=PLDX4T_cnKjD0JeULl1X6iTn7VIkDeYX_X&index=33
Eron
On Wed, Sep 27, 2017 at 9:12 AM, Stefan Richter <s.richter@data-artisans.com
> wrote:
> Hi,
>
> from the top of my head, I cannot see why this should not be possible,
> task managers just need to be able to connect to their job manager.
> Unfortunately, I cannot give a real guarantee here because I am not that
> deeply involved in this aspect of Flink.
>
> Best,
> Stefan
>
> Am 27.09.2017 um 12:39 schrieb Rahul Raj <ra...@gmail.com>:
>
> Hi Stefan,
>
> I have a question in my mid out of curiosity.... Is it possible to run
> flink application within docker container by using flink cluster set up on
> host?
>
> Rahul Raj
>
> On 26 September 2017 at 17:29, Stefan Richter <s.richter@data-artisans.com
> > wrote:
>
>> Hi,
>>
>> if I correctly understood the approach outlined on github, you can start
>> a standalone job manager and the task manager get the JM information either
>> through the provided configuration or through Zookeeper. Take a look at the
>> „running section“, e.g.:
>>
>> 1) „Via Mesos/Marathon: Start a standalone JobManager (you need to
>> replace the flink_recovery_zookeeper_quorum variable with a valid
>> setting for your cluster) [...]“
>> 2) „Via standalone Docker: Start a standalone JobManager (with host
>> networking, binding on 127.0.0.1) […]“
>>
>> Best,
>> Stefan
>>
>>
>> Am 26.09.2017 um 12:43 schrieb Rahul Raj <ra...@gmail.com>:
>>
>> Hi Stefan,
>>
>> Thanks a lot for your answer and sharing the link
>> https://github.com/mesoshq/flink. I went through this and saw its
>> spawning Jobmanager and taskmanager. Now I think, this should be happening.
>> First JobManager will be started on flink cluster on one node, then task
>> manager will be started on another node and both should be running in
>> docker containers on different nodes. Now, my question is how flink's
>> JobManager will get to know about the taskManagers as they are in in
>> different docker containers on different nodes? Will it happen via Mesos?
>>
>> Can we use mesos-appmaster.sh instead which is already built in flink for
>> deployment on mesos?
>>
>> Rahul Raj
>>
>> On 26 September 2017 at 15:32, Stefan Richter <
>> s.richter@data-artisans.com> wrote:
>>
>>> Hi,
>>>
>>> as in my answer to your previous mail, I suggest to take a look at
>>> https://github.com/mesoshq/flink . Unfortunately, there is not yet a
>>> lot documentation about the internals of how this works, so I am also
>>> looping in Till who might know more about specific questions about how
>>> things work together exactly.
>>>
>>> Best,
>>> Stefan
>>>
>>>
>>> Am 26.09.2017 um 09:21 schrieb Rahul Raj <ra...@gmail.com>:
>>>
>>> Currently I have a Flink Application Jar file running on Mesos cluster.
>>> The flink application simply reads data from Kafka and put it to HDFS.
>>>
>>> Now we are planning to create a docker image to run this application
>>> jar file inside docker containers on Mesos cluster via Marathon.
>>>
>>> Below are the questions that I am looking answers for:
>>>
>>> 1. While building the docker image, how do I include flink-1.3.2 set up
>>> and my mesos config in flink?
>>>
>>> 2. How shall I run my existing flink application jar?
>>>
>>> 3. Will running my flink application jar on docker containers will run
>>> it on mesos slaves on different docker containers? How docker , Flink,
>>> mesos , Marathon will work together in my case?
>>>
>>> Rahul Raj
>>>
>>>
>>>
>>
>>
>
>
Re: Flink Application Jar file on Docker container
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
from the top of my head, I cannot see why this should not be possible, task managers just need to be able to connect to their job manager. Unfortunately, I cannot give a real guarantee here because I am not that deeply involved in this aspect of Flink.
Best,
Stefan
> Am 27.09.2017 um 12:39 schrieb Rahul Raj <ra...@gmail.com>:
>
> Hi Stefan,
>
> I have a question in my mid out of curiosity.... Is it possible to run flink application within docker container by using flink cluster set up on host?
>
> Rahul Raj
>
> On 26 September 2017 at 17:29, Stefan Richter <s.richter@data-artisans.com <ma...@data-artisans.com>> wrote:
> Hi,
>
> if I correctly understood the approach outlined on github, you can start a standalone job manager and the task manager get the JM information either through the provided configuration or through Zookeeper. Take a look at the „running section“, e.g.:
>
> 1) „Via Mesos/Marathon: Start a standalone JobManager (you need to replace the flink_recovery_zookeeper_quorum variable with a valid setting for your cluster) [...]“
> 2) „Via standalone Docker: Start a standalone JobManager (with host networking, binding on 127.0.0.1) […]“
>
> Best,
> Stefan
>
>
>> Am 26.09.2017 um 12:43 schrieb Rahul Raj <rahulrajmsrit@gmail.com <ma...@gmail.com>>:
>>
>> Hi Stefan,
>>
>> Thanks a lot for your answer and sharing the link https://github.com/mesoshq/flink <https://github.com/mesoshq/flink>. I went through this and saw its spawning Jobmanager and taskmanager. Now I think, this should be happening. First JobManager will be started on flink cluster on one node, then task manager will be started on another node and both should be running in docker containers on different nodes. Now, my question is how flink's JobManager will get to know about the taskManagers as they are in in different docker containers on different nodes? Will it happen via Mesos?
>>
>> Can we use mesos-appmaster.sh instead which is already built in flink for deployment on mesos?
>>
>> Rahul Raj
>>
>> On 26 September 2017 at 15:32, Stefan Richter <s.richter@data-artisans.com <ma...@data-artisans.com>> wrote:
>> Hi,
>>
>> as in my answer to your previous mail, I suggest to take a look at https://github.com/mesoshq/flink <https://github.com/mesoshq/flink> . Unfortunately, there is not yet a lot documentation about the internals of how this works, so I am also looping in Till who might know more about specific questions about how things work together exactly.
>>
>> Best,
>> Stefan
>>
>>> Am 26.09.2017 um 09:21 schrieb Rahul Raj <rahulrajmsrit@gmail.com <ma...@gmail.com>>:
>>>
>>> Currently I have a Flink Application Jar file running on Mesos cluster. The flink application simply reads data from Kafka and put it to HDFS.
>>>
>>> Now we are planning to create a docker image to run this application jar file inside docker containers on Mesos cluster via Marathon.
>>>
>>> Below are the questions that I am looking answers for:
>>>
>>> 1. While building the docker image, how do I include flink-1.3.2 set up and my mesos config in flink?
>>>
>>> 2. How shall I run my existing flink application jar?
>>>
>>> 3. Will running my flink application jar on docker containers will run it on mesos slaves on different docker containers? How docker , Flink, mesos , Marathon will work together in my case?
>>>
>>> Rahul Raj
>>
>>
>
>
Re: Flink Application Jar file on Docker container
Posted by Rahul Raj <ra...@gmail.com>.
Hi Stefan,
I have a question in my mid out of curiosity.... Is it possible to run
flink application within docker container by using flink cluster set up on
host?
Rahul Raj
On 26 September 2017 at 17:29, Stefan Richter <s....@data-artisans.com>
wrote:
> Hi,
>
> if I correctly understood the approach outlined on github, you can start a
> standalone job manager and the task manager get the JM information either
> through the provided configuration or through Zookeeper. Take a look at the
> „running section“, e.g.:
>
> 1) „Via Mesos/Marathon: Start a standalone JobManager (you need to replace
> the flink_recovery_zookeeper_quorum variable with a valid setting for
> your cluster) [...]“
> 2) „Via standalone Docker: Start a standalone JobManager (with host
> networking, binding on 127.0.0.1) […]“
>
> Best,
> Stefan
>
>
> Am 26.09.2017 um 12:43 schrieb Rahul Raj <ra...@gmail.com>:
>
> Hi Stefan,
>
> Thanks a lot for your answer and sharing the link
> https://github.com/mesoshq/flink. I went through this and saw its
> spawning Jobmanager and taskmanager. Now I think, this should be happening.
> First JobManager will be started on flink cluster on one node, then task
> manager will be started on another node and both should be running in
> docker containers on different nodes. Now, my question is how flink's
> JobManager will get to know about the taskManagers as they are in in
> different docker containers on different nodes? Will it happen via Mesos?
>
> Can we use mesos-appmaster.sh instead which is already built in flink for
> deployment on mesos?
>
> Rahul Raj
>
> On 26 September 2017 at 15:32, Stefan Richter <s.richter@data-artisans.com
> > wrote:
>
>> Hi,
>>
>> as in my answer to your previous mail, I suggest to take a look at
>> https://github.com/mesoshq/flink . Unfortunately, there is not yet a lot
>> documentation about the internals of how this works, so I am also looping
>> in Till who might know more about specific questions about how things work
>> together exactly.
>>
>> Best,
>> Stefan
>>
>>
>> Am 26.09.2017 um 09:21 schrieb Rahul Raj <ra...@gmail.com>:
>>
>> Currently I have a Flink Application Jar file running on Mesos cluster.
>> The flink application simply reads data from Kafka and put it to HDFS.
>>
>> Now we are planning to create a docker image to run this application jar
>> file inside docker containers on Mesos cluster via Marathon.
>>
>> Below are the questions that I am looking answers for:
>>
>> 1. While building the docker image, how do I include flink-1.3.2 set up
>> and my mesos config in flink?
>>
>> 2. How shall I run my existing flink application jar?
>>
>> 3. Will running my flink application jar on docker containers will run it
>> on mesos slaves on different docker containers? How docker , Flink, mesos ,
>> Marathon will work together in my case?
>>
>> Rahul Raj
>>
>>
>>
>
>
Re: Flink Application Jar file on Docker container
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
if I correctly understood the approach outlined on github, you can start a standalone job manager and the task manager get the JM information either through the provided configuration or through Zookeeper. Take a look at the „running section“, e.g.:
1) „Via Mesos/Marathon: Start a standalone JobManager (you need to replace the flink_recovery_zookeeper_quorum variable with a valid setting for your cluster) [...]“
2) „Via standalone Docker: Start a standalone JobManager (with host networking, binding on 127.0.0.1) […]“
Best,
Stefan
> Am 26.09.2017 um 12:43 schrieb Rahul Raj <ra...@gmail.com>:
>
> Hi Stefan,
>
> Thanks a lot for your answer and sharing the link https://github.com/mesoshq/flink <https://github.com/mesoshq/flink>. I went through this and saw its spawning Jobmanager and taskmanager. Now I think, this should be happening. First JobManager will be started on flink cluster on one node, then task manager will be started on another node and both should be running in docker containers on different nodes. Now, my question is how flink's JobManager will get to know about the taskManagers as they are in in different docker containers on different nodes? Will it happen via Mesos?
>
> Can we use mesos-appmaster.sh instead which is already built in flink for deployment on mesos?
>
> Rahul Raj
>
> On 26 September 2017 at 15:32, Stefan Richter <s.richter@data-artisans.com <ma...@data-artisans.com>> wrote:
> Hi,
>
> as in my answer to your previous mail, I suggest to take a look at https://github.com/mesoshq/flink <https://github.com/mesoshq/flink> . Unfortunately, there is not yet a lot documentation about the internals of how this works, so I am also looping in Till who might know more about specific questions about how things work together exactly.
>
> Best,
> Stefan
>
>> Am 26.09.2017 um 09:21 schrieb Rahul Raj <rahulrajmsrit@gmail.com <ma...@gmail.com>>:
>>
>> Currently I have a Flink Application Jar file running on Mesos cluster. The flink application simply reads data from Kafka and put it to HDFS.
>>
>> Now we are planning to create a docker image to run this application jar file inside docker containers on Mesos cluster via Marathon.
>>
>> Below are the questions that I am looking answers for:
>>
>> 1. While building the docker image, how do I include flink-1.3.2 set up and my mesos config in flink?
>>
>> 2. How shall I run my existing flink application jar?
>>
>> 3. Will running my flink application jar on docker containers will run it on mesos slaves on different docker containers? How docker , Flink, mesos , Marathon will work together in my case?
>>
>> Rahul Raj
>
>
Re: Flink Application Jar file on Docker container
Posted by Rahul Raj <ra...@gmail.com>.
Hi Stefan,
Thanks a lot for your answer and sharing the link
https://github.com/mesoshq/flink. I went through this and saw its spawning
Jobmanager and taskmanager. Now I think, this should be happening. First
JobManager will be started on flink cluster on one node, then task manager
will be started on another node and both should be running in docker
containers on different nodes. Now, my question is how flink's JobManager
will get to know about the taskManagers as they are in in different docker
containers on different nodes? Will it happen via Mesos?
Can we use mesos-appmaster.sh instead which is already built in flink for
deployment on mesos?
Rahul Raj
On 26 September 2017 at 15:32, Stefan Richter <s....@data-artisans.com>
wrote:
> Hi,
>
> as in my answer to your previous mail, I suggest to take a look at
> https://github.com/mesoshq/flink . Unfortunately, there is not yet a lot
> documentation about the internals of how this works, so I am also looping
> in Till who might know more about specific questions about how things work
> together exactly.
>
> Best,
> Stefan
>
>
> Am 26.09.2017 um 09:21 schrieb Rahul Raj <ra...@gmail.com>:
>
> Currently I have a Flink Application Jar file running on Mesos cluster.
> The flink application simply reads data from Kafka and put it to HDFS.
>
> Now we are planning to create a docker image to run this application jar
> file inside docker containers on Mesos cluster via Marathon.
>
> Below are the questions that I am looking answers for:
>
> 1. While building the docker image, how do I include flink-1.3.2 set up
> and my mesos config in flink?
>
> 2. How shall I run my existing flink application jar?
>
> 3. Will running my flink application jar on docker containers will run it
> on mesos slaves on different docker containers? How docker , Flink, mesos ,
> Marathon will work together in my case?
>
> Rahul Raj
>
>
>
Re: Flink Application Jar file on Docker container
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
as in my answer to your previous mail, I suggest to take a look at https://github.com/mesoshq/flink <https://github.com/mesoshq/flink> . Unfortunately, there is not yet a lot documentation about the internals of how this works, so I am also looping in Till who might know more about specific questions about how things work together exactly.
Best,
Stefan
> Am 26.09.2017 um 09:21 schrieb Rahul Raj <ra...@gmail.com>:
>
> Currently I have a Flink Application Jar file running on Mesos cluster. The flink application simply reads data from Kafka and put it to HDFS.
>
> Now we are planning to create a docker image to run this application jar file inside docker containers on Mesos cluster via Marathon.
>
> Below are the questions that I am looking answers for:
>
> 1. While building the docker image, how do I include flink-1.3.2 set up and my mesos config in flink?
>
> 2. How shall I run my existing flink application jar?
>
> 3. Will running my flink application jar on docker containers will run it on mesos slaves on different docker containers? How docker , Flink, mesos , Marathon will work together in my case?
>
> Rahul Raj