You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/07/07 14:03:00 UTC
[jira] [Commented] (FLINK-7080) Deploy Yarn cluster with job
[ https://issues.apache.org/jira/browse/FLINK-7080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078120#comment-16078120 ]
ASF GitHub Bot commented on FLINK-7080:
---------------------------------------
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/4284
[FLINK-7080] [yarn] Add Yarn per job mode deployment
This PR is based on #4280, #4270, #4271, #4259, #4260, #4261, #4272 and #4281.
This PR adds the functionality to deploy a Yarn per-job cluster via the `YarnClusterDescriptorV2.deployJob(ClusterSpecification, JobGraph)`. The `JobGraph` has to contain the user code jar which is required to run the job.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink addPerJobYarnDeployment
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4284.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4284
----
commit b415fdba6334aa8eeed9fecb3834dbce5ceecc1e
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-07T11:40:29Z
[FLINK-7125] [yarn] Remove Configuration loading from AbstractYarnClusterDescriptor
Instead the AbstractYarnClusterDescriptor is passed in a Configuration instance which
is sent to the started application master.
Pass in configuration directory manually
Remove configurationDirectory resolution from AbstractYarnClusterDescriptor
commit 8de588bf3b2bc984ecc25603e474df443f5cc265
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-06T07:33:18Z
[FLINK-7110] [client] Add per-job cluster deployment interface
commit c68942eb7e154e78bef16930fa8f421f0123483e
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-06T12:00:21Z
[FLINK-7113] Make ClusterDescriptor independent of cluster size
The deploySession method now is given a ClusterSpecification which specifies the
size of the cluster which it is supposed to deploy.
commit 7003cae3c7530b6560dd8ebd365b1b1313a109e0
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-04T21:15:52Z
[FLINK-7103] [dispatcher] Add skeletal structure of Dispatcher component
The Dispatcher is responsible for receiving job submissions, persisting the JobGraphs,
spawning JobManager to execute the jobs and recovering the jobs in case of a master
failure. This commit adds the basic skeleton including the RPC call for job submission.
Add cleanup logic for finished jobs
Pass BlobService to JobManagerRunner
commit 24fa27535d7064cd2018c67969f9d2c2ef0238f3
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-05T11:38:35Z
[FLINK-7105] Make ActorSystems non daemonic
In order to not having to explicitly wait on the termination of an ActorSystem
in the main thread, we now create the ActorSystems in non-daemonic mode. That
way the process won't terminate if there is still an active actor.
commit 60294e77b5d9cb91b9e31c0a80d3514d32c80c88
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-05T15:16:52Z
[FLINK-7108] [yarn] Add YARN entry points based on the generic entry point
Add the YarnSesssionClusterEntrypoint and the YarnJobClusterEntrypoint which extend
SessionClusterEntrypoint and JobClusterEntrypoint, respectively.
Add new Yarn session and per-job cluster entry points
Remove old Flip-6 Yarn per job entry point
commit dee79e55de3575cacdc9ce0d95a4c43974c2d0f6
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-06T15:16:18Z
[FLINK-7082] Add generic entry point for session and per-job clusters
commit a940663e7e2894dc1ddd365ca6ae539b52a85c65
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-06T15:16:54Z
[FLINK-7086] Add Flip-6 standalone session cluster entry point
commit fce951241e3a4723a6c32b6c08b1aea7513ec397
Author: Till Rohrmann <tr...@apache.org>
Date: 2017-07-06T16:26:41Z
[FLINK-7080] [yarn] Add Yarn per job mode deployment
Upload user code jar from JobGraph
----
> Deploy Yarn cluster with job
> ----------------------------
>
> Key: FLINK-7080
> URL: https://issues.apache.org/jira/browse/FLINK-7080
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management, YARN
> Reporter: Till Rohrmann
> Assignee: Till Rohrmann
> Labels: flip-6
>
> In order to start a yarn per-job cluster, we have to start a Yarn application running Flink which includes the job to be executed. One way to do it is to upload the serialized form of the {{JobGraph}} as a file. The Yarn entry point can then read the {{JobGraph}} from this file and start the {{JobManager}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)