You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2017/10/19 22:30:00 UTC
[jira] [Comment Edited] (YARN-7217) Improve API service usability
for updating service spec and state
[ https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211832#comment-16211832 ]
Eric Yang edited comment on YARN-7217 at 10/19/17 10:29 PM:
------------------------------------------------------------
Hi [~billie.rinaldi], thank you for the review.
# Does YARN resource manager have privileges to all users's home directory? In non-secure cluster, the answer is yes. However, this might not be true in secure cluster. This is one of the reasons that I did not implement getServicesList on fs. If we want getServiceList on fs implemented for non-secure cluster, it can be included.
# I will fix actionBuild.
# I think it's best to add configuration to YarnCommon, there are unit tests that valid the configuration addition, and reduce chance of using duplicated names.
# I will fix Solr version to hadoop-project/pom.xml
# YARN-7193 includes definition for global application definition in your last point. It provides a set of API to register Yarnfile with additional metadata like organization, description, icon, and download into Solr. I couldn't get anyone to review that JIRA, hence, I break down the functionality into small bits that can be consumed. The first step is to offload storing of Yarnfile from HDFS to SOLR as improvement for this issue. Application catalog can be built on top of what is proposed here. Once application catalog is built, then we can start to think about how the money and software exchange take place for appstore. Appstore idea might not get traction in Apache because its a non-profit organization. We can think about that idea later.
was (Author: eyang):
Hi [~billie.rinaldi], thank you for the review.
# Does YARN resource manager have privileges to all users's home directory? In non-secure cluster, the answer is yes. However, this might not be true in secure cluster. This is one of the reasons that I did not implement getServicesList on fs. If we want getServiceList on fs implemented for non-secure cluster, it can be included.
# actionBuild is validating and persist application on HDFS. There is no external caller to actionBuild from REST API. Perhaps, make actionBuild a private method? actionCreate is the API that external callers uses.
# I think it's best to add configuration to YarnCommon, there are unit tests that valid the configuration addition, and reduce chance of using duplicated names.
# I will fix Solr version to hadoop-project/pom.xml
# YARN-7193 includes definition for global application definition in your last point. It provides a set of API to register Yarnfile with additional metadata like organization, description, icon, and download into Solr. I couldn't get anyone to review that JIRA, hence, I break down the functionality into small bits that can be consumed. The first step is to offload storing of Yarnfile from HDFS to SOLR as improvement for this issue. Application catalog can be built on top of what is proposed here. Once application catalog is built, then we can start to think about how the money and software exchange take place for appstore. Appstore idea might not get traction in Apache because its a non-profit organization. We can think about that idea later.
> Improve API service usability for updating service spec and state
> -----------------------------------------------------------------
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch, YARN-7217.yarn-native-services.002.patch, YARN-7217.yarn-native-services.003.patch, YARN-7217.yarn-native-services.004.patch
>
>
> API service for deploy, and manage YARN services have several limitations.
> {{updateService}} API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers. (This was removed in YARN-7323).
> The overloading is buggy depending on how the configuration should be applied.
> h4. Scenario 1
> A user retrieves Service object from getService call, and the Service object contains state: STARTED. The user would like to increase number of containers for the deployed service. The JSON has been updated to increase container count. The PUT method does not actually increase container count.
> h4. Scenario 2
> A user retrieves Service object from getService call, and the Service object contains state: STOPPED. The user would like to make a environment configuration change. The configuration does not get updated after PUT method.
> This is possible to address by rearranging the logic of START/STOP after configuration update. However, there are other potential combinations that can break PUT method. For example, user like to make configuration changes, but not yet restart the service until a later time.
> h4. Scenario 3
> There is no API to list all deployed applications by the same user.
> h4. Scenario 4
> Desired state (spec) and current state are represented by the same Service object. There is no easy way to identify "state" is desired state to reach or, the current state of the service. It would be nice to have ability to retrieve both desired state, and current state with separated entry points. By implementing /spec and /state, it can resolve this problem.
> h4. Scenario 5
> List all services deploy by the same user can trigger a directory listing operation on namenode if hdfs is used as storage for metadata. When hundred of users use Service UI to view or deploy applications, this will trigger denial of services attack on namenode. The sparse small metadata files also reduce efficiency of Namenode memory usage. Hence, a cache layer for storing service metadata can reduce namenode stress.
> h3. Proposed change
> ApiService can separate the PUT method into two PUT methods for configuration changes vs operation changes. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/spec
> Request Data:
> {
> "name": "amp",
> "components": [
> {
> "name": "mysql",
> "number_of_containers": 2,
> "artifact": {
> "id": "centos/mysql-57-centos7:latest",
> "type": "DOCKER"
> },
> "run_privileged_container": false,
> "launch_command": "",
> "resource": {
> "cpus": 1,
> "memory": "2048"
> },
> "configuration": {
> "env": {
> "MYSQL_USER":"${USER}",
> "MYSQL_PASSWORD":"password"
> }
> }
> }
> ],
> "quicklinks": {
> "Apache Document Root": "http://httpd.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/",
> "PHP MyAdmin": "http://phpmyadmin.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/"
> }
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "amp",
> "components": [
> {
> "name": "mysql",
> "state": "STOPPED"
> }
> ]
> }
> {code}
> SOLR can be used to cache Yarnfile to improve lookup performance and reduce stress of namenode small file problems and high frequency lookup. SOLR is chosen for caching metadata because its indexing feature can be used to build full text search for application catalog as well.
> For service that requires configuration to increase or decrease node count. The calling sequence is:
> {code}
> # GET /ws/v1/services/{service_name}/spec
> # Change number_of_containers to desired number.
> # PUT /ws/v1/services/{service_name}/spec to update the spec.
> # PUT /ws/v1/services/{service_name}/state to stop existing service.
> # PUT /ws/v1/services/{service_name}/state to start service.
> {code}
> For components that can increase node count without rewrite configuration:
> {code}
> # GET /ws/v1/services/{service_name}/spec
> # Change number_of_containers to desired number.
> # PUT /ws/v1/services/{service_name}/spec to update the spec.
> # PUT /ws/v1/services/{service_name}/component/{component_name} to change node count.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org