FLIP-6 Docker / Kubernetes

[anaray <an...@gmail.com>]

Hi,

I was reading *Docker/Kubernetes* section of FLIP-6
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077).
I have question regarding point 1 ie "A job-specific image with the job
libraries, configured to launch the job (with a JobManager)"

How does job submission happens here ? Do we need write a script to submit
the job?





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: FLIP-6 Docker / Kubernetes

[Gary Yao <ga...@data-artisans.com>]

Hi,

Unfortunately, there won't be native support for Kubernetes in Flink 1.5 yet
[1]. That is, you will still have to deploy a standalone session cluster and
submit a job via the REST API.

Can you open an issue in JIRA regarding the Blob Cache corruption?
FLINK-5908 is
closed already.

Best,
Gary

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Migration-to-Flip6-Kubernetes-tp18937p19132.html


On Mon, May 7, 2018 at 7:48 PM, anaray <an...@gmail.com> wrote:

> Thank You Fabian. We are looking forward for this feature, especially a way
> to bundle the job jar along with the container (even in taskmanagers).
>
> In production we deploy flink (1 job=1 cluster) in Docker Swarm. One of the
> main issue we face is related to blob download when the taskmanager fails
> and restarts . Occasionally we run into blob corruption issue, and job
> fails
> ( FLINK-5908 <https://issues.apache.org/jira/browse/FLINK-5908>  ). This
> prompted us to monitor job and resubmit the job again using automated
> scripts! Since, we go with 1 cluster = 1 job, we can bundle the job jar,
> and
> totally avoid the blob downloads, and automate the deployment & management
> of job life-cycle. And we are planning to move to Kubernetes shortly
>
>
>
> Best,
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/
>

Re: FLIP-6 Docker / Kubernetes

[anaray <an...@gmail.com>]

Thank You Fabian. We are looking forward for this feature, especially a way
to bundle the job jar along with the container (even in taskmanagers). 

In production we deploy flink (1 job=1 cluster) in Docker Swarm. One of the
main issue we face is related to blob download when the taskmanager fails
and restarts . Occasionally we run into blob corruption issue, and job fails
( FLINK-5908 <https://issues.apache.org/jira/browse/FLINK-5908>  ). This
prompted us to monitor job and resubmit the job again using automated
scripts! Since, we go with 1 cluster = 1 job, we can bundle the job jar, and
totally avoid the blob downloads, and automate the deployment & management
of job life-cycle. And we are planning to move to Kubernetes shortly



Best,



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: FLIP-6 Docker / Kubernetes

[Fabian Hueske <fh...@gmail.com>]

Hi,

Most, but not all, of the FLIP-6 features will be released with Flink 1.5.0.
I'm not sure if this deployment mode will be fully supported in 1.5.0. Gary
(in CC) might know details here.

Anyway, the deployment would work by starting the image using regular
Docker/Kubernetes tools.
The image would automatically bring up the JobManager and wait for
TaskManagers to connect to run the embedded application.

Best, Fabian

2018-05-02 1:29 GMT+02:00 anaray <an...@gmail.com>:

> Hi,
>
> I was reading *Docker/Kubernetes* section of FLIP-6
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077
> ).
> I have question regarding point 1 ie "A job-specific image with the job
> libraries, configured to launch the job (with a JobManager)"
>
> How does job submission happens here ? Do we need write a script to submit
> the job?
>
>
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.
> nabble.com/
>