You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Yesha Vora (JIRA)" <ji...@apache.org> on 2018/06/13 23:27:00 UTC
[jira] [Updated] (YARN-8283) [Umbrella] MaWo - A Master Worker
framework on top of YARN Services
[ https://issues.apache.org/jira/browse/YARN-8283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yesha Vora updated YARN-8283:
-----------------------------
Attachment: [Design Doc] [YARN-8283] MaWo - A Master Worker framework on top of YARN Services.pdf
> [Umbrella] MaWo - A Master Worker framework on top of YARN Services
> -------------------------------------------------------------------
>
> Key: YARN-8283
> URL: https://issues.apache.org/jira/browse/YARN-8283
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Yesha Vora
> Assignee: Yesha Vora
> Priority: Major
> Attachments: [Design Doc] [YARN-8283] MaWo - A Master Worker framework on top of YARN Services.pdf
>
>
> There is a need for an application / framework to handle Master-Worker scenarios. There are existing frameworks on YARN which can be used to run a job in distributed manner such as Mapreduce, Tez, Spark etc. But master-worker use-cases usually are force-fed into one of these existing frameworks which have been designed primarily around data-parallelism instead of generic Master Worker type of computations.
> In this JIRA, we’d like to contribute MaWo - a YARN Service based framework that achieves this goal. The overall goal is to create an app that can take an input job specification with tasks, their durations and have a Master dish the tasks off to a predetermined set of workers. The components will be responsible for making sure that the tasks and the overall job finish in specific time durations.
> We have been using a version of the MaWo framework for running unit tests of Hadoop in a parallel manner on an existing Hadoop YARN cluster. What typically takes 10 hours to run all of Hadoop project’s unit-tests can finish under 20 minutes on a MaWo app of about 50 containers!
> YARN-3307 was an original attempt at this but through a first-class YARN app. In this JIRA, we instead use YARN Service for orchestration so that our code can focus on the core Master Worker paradigm.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org