You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Oleksandr Shevchenko (JIRA)" <ji...@apache.org> on 2018/11/27 10:17:00 UTC
[jira] [Comment Edited] (YARN-9023) Pluggable YARN Auxiliary
Services
[ https://issues.apache.org/jira/browse/YARN-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700168#comment-16700168 ]
Oleksandr Shevchenko edited comment on YARN-9023 at 11/27/18 10:16 AM:
-----------------------------------------------------------------------
Thanks a lot, [~eyang] for the detailed explanation of problems which exist in current auxiliary service design.
1. I agree that much better to run auxiliary services in YARN containers (something like we do for YARN Service support [https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html]) instead of as a parasitic process of YARN demons. In addition to your points, containerization of auxiliary services resolves the problem with HA and do not fail YARN demon when auxiliary service is down. Perhaps, we can combine YARN Services and Auxiliary Services on top-level API since both services are long-running services and have a similar aim, differing only start time.
What do you think?
2. Also, there is the problem if we want to start Auxiliary Service before YARN demon start. As I mentioned in the description, for example, when we need to create/setup some storage for YARN demon data (for example for RM state store or AM intermediate data) before demon started. As the result, we can't start container with Auxiliary Service for NM on the same node to prepare some services/storages/configurations before NM start.
How we can achieve this goal?
Let me investigate deeper how we can implement this. I'll try to prepare design doc for this.
Thanks!
was (Author: oshevchenko):
Thanks a lot, [~eyang] for the detailed explanation of problems which exist in current auxiliary service design. I agree that much better to run auxiliary services in YARN containers (something like we do for YARN Service support https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html) instead of as a parasitic process of YARN demons. In addition to your points, containerization of auxiliary services resolves the problem with HA and do not fail YARN demon when auxiliary service is down. Perhaps, we can combine YARN Services and Auxiliary Services on top-level API since both services are long-running services and have a similar aim, differing only start time. What do you think?
Also, there is the problem if we want to start Auxiliary Service before YARN demon start. As I mentioned in the description, for example, when we need to create/setup some storage for YARN demon data (for example for RM state store or AM intermediate data) before demon started. As the result, we can't start container with Auxiliary Service for NM on the same node to prepare some services/storages/configurations before NM start.
How we can achieve this goal?
Let me investigate deeper how we can implement this. I'll try to prepare design doc for this.
Thanks!
> Pluggable YARN Auxiliary Services
> ---------------------------------
>
> Key: YARN-9023
> URL: https://issues.apache.org/jira/browse/YARN-9023
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Oleksandr Shevchenko
> Assignee: Oleksandr Shevchenko
> Priority: Major
> Attachments: Pluggable YARN Auxiliary Services Doc v1.pdf
>
>
> The propose is to create customizable auxiliary services infrastructure for all YARN services to provide abilities to extends or change functionality for all YARN demons.
> Currently, YARN has the ability to add auxiliary services only for NodeManager.
> There are some cases when we need to add auxiliary services for some other YARN demons. For example, in YARN-2267 proposed to add Auxiliary Service support in RM for implementation monitor plugins or some alarm services. We faced with one more case, we need to create/setup some storage for YARN demon data (for example for RM state store or AM intermediate data) before demon started. Also, will be good if we can use some auxiliary service in different YARN demons.
> The end users can use YARN Auxiliary Services (YAS) to customize or extend the functionality of all YARN demons. We can provide an ability to create aux service both for one and for all demons. This will make easy creating of aux service for shuffle or just to up some service within another YARN daemon. We will get a unified and more extensible architecture for all aux services which may need to create in the future.
> Please look at the attached design doc for the details. [^Pluggable YARN Auxiliary Services Doc v1.pdf]
> Also, you can add your comments to this doc: https://docs.google.com/document/d/1Pxs8eeWZwnlwryPj7gyN8Ej1O-FRBSX_C0ispV9ORT8/edit?usp=sharing
> Thanks a lot for any comments and attention to this feature!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org