You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Arun Suresh (JIRA)" <ji...@apache.org> on 2016/08/03 11:40:20 UTC
[jira] [Commented] (YARN-4602) Simple and Scalable Message Service
for YARN application
[ https://issues.apache.org/jira/browse/YARN-4602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405770#comment-15405770 ]
Arun Suresh commented on YARN-4602:
-----------------------------------
[~djp], wondering if you've taken a look at apache REEF. It uses an event driven framework called https://reef.apache.org/wake.html that also supports messaging. It uses Netty under the hood.
> Simple and Scalable Message Service for YARN application
> --------------------------------------------------------
>
> Key: YARN-4602
> URL: https://issues.apache.org/jira/browse/YARN-4602
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications, resourcemanager
> Reporter: Junping Du
> Assignee: Junping Du
>
> We are proposing to support MR AM restart with work preserving in MAPREDUCE-6608 (https://issues.apache.org/jira/browse/MAPREDUCE-6608) that when AM get failed for some reason, the inflight tasks will keep running/pending until new AM attempt comes back to continue. One of prerequisite is tasks should know where the new AM attempt get launched so TaskUmbilicalProtocol can get retry between clients and new server.
> There could be the same requirement for other applications running on YARN too. Some application decide to handle message delivery itself, e.g. Long running services can leverage Slider agent to notify messages back and forth. However, vanilla applications on YARN is hard to achieve this because Hadoop RPC mechanism essentially is a single way of communication. Although two directions mechanism like heartbeats (between NM-RM or AM-RM) can get built on top of it, it make less sense to build the same mechanism between AM and its application containers - or it need to handle massive of client connections in AM which could be the new bottleneck for scalability and very complicated in state maintaining. Instead, we need a new message mechanism that is simple and scalable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org