[jira] [Commented] (FLINK-4114) Need a way to manage multiple named, long-lived jobs on a single YARN cluster in an automated manner

["Chris Hogue (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353046#comment-15353046 ] 

Chris Hogue commented on FLINK-4114:
------------------------------------

I should add that since filing this I realized I missed a parameter on the yarn integration that we can use to accomplish this. The -ynm param allows us to name the yarn application. Then along with the yarn application -list command we can find the application ID associated with a named job. It's possibly slightly brittle as we end up grep'ing and awk'ing out the application ID from the list command, but it should be functional.

If you want to continue with this JIRA to provide a more direct integration that's great, just wanted to mention that we do seem to have a way to do this for now. It'd be fine if you want to lower the priority of this IMO.

> Need a way to manage multiple named, long-lived jobs on a single YARN cluster in an automated manner
> ----------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-4114
>                 URL: https://issues.apache.org/jira/browse/FLINK-4114
>             Project: Flink
>          Issue Type: Improvement
>          Components: YARN Client
>            Reporter: Chris Hogue
>
> We are running several Flink jobs on a single YARN cluster. Currently each Flink job is run in its own YARN session (and thus its own YARN application ID). The difficulty comes in that we want to manage each of these jobs individually by name. For example we want to start, stop, update one job without affecting others. The primary access to these jobs is via the YARN application ID, which is not meaningful to discern which flink job it is running.
> It would be nice if we had tools that would allow us to manage the flink jobs by name and have it do the right thing with the YARN session. Today we can use 'flink run' and have it start a YARN session for that job, but from that point forward we have only the YARN application ID to work with.
> As a concrete example suppose we have 2 jobs with names JobA and JobB. We'd want a way to so something like: 
> flink run <JobA jar>; flink run <JobB jar>
> We'd then want to be able to call:
> flink cancel JobA
> The cancel command would spin down the YARN session for JobA in addition to the flink job, leaving JobB running as normal. I've simplified the commands leaving out other options for illustrative purposes. And we'll want to be able to use savepoints through these steps as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)