[jira] [Created] (FLINK-4114) Need a way to manage multiple named, long-lived jobs on a single YARN cluster in an automated manner

["Chris Hogue (JIRA)" <ji...@apache.org>]

Chris Hogue created FLINK-4114:
----------------------------------

             Summary: Need a way to manage multiple named, long-lived jobs on a single YARN cluster in an automated manner
                 Key: FLINK-4114
                 URL: https://issues.apache.org/jira/browse/FLINK-4114
             Project: Flink
          Issue Type: Bug
            Reporter: Chris Hogue


We are running several Flink jobs on a single YARN cluster. Currently each Flink job is run in its own YARN session (and thus its own YARN application ID). The difficulty comes in that we want to manage each of these jobs individually by name. For example we want to start, stop, update one job without affecting others. The primary access to these jobs is via the YARN application ID, which is not meaningful to discern which flink job it is running.

It would be nice if we had tools that would allow us to manage the flink jobs by name and have it do the right thing with the YARN session. Today we can use 'flink run' and have it start a YARN session for that job, but from that point forward we have only the YARN application ID to work with.

As a concrete example suppose we have 2 jobs with names JobA and JobB. We'd want a way to so something like: 
flink run <JobA jar>; flink run <JobB jar>

We'd then want to be able to call:

flink cancel JobA

The cancel command would spin down the YARN session for JobA in addition to the flink job, leaving JobB running as normal. I've simplified the commands leaving out other options for illustrative purposes. And we'll want to be able to use savepoints through these steps as well.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)