You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by sm...@apache.org on 2014/07/08 09:08:36 UTC
svn commit: r1608675 - in /incubator/slider/site/trunk/content:
developing/building.md docs/getting_started.md images/image_0.png
images/image_0_app_master_ui_lrg.png images/image_1.png
images/image_1_apps_rm_ui-lrg.png index.md
Author: smohanty
Date: Tue Jul 8 07:08:35 2014
New Revision: 1608675
URL: http://svn.apache.org/r1608675
Log:
SLIDER-121 manually generate slider site from CMS content for now (preparation for next release)
Added:
incubator/slider/site/trunk/content/images/image_0_app_master_ui_lrg.png (with props)
incubator/slider/site/trunk/content/images/image_1_apps_rm_ui-lrg.png (with props)
Removed:
incubator/slider/site/trunk/content/images/image_0.png
incubator/slider/site/trunk/content/images/image_1.png
Modified:
incubator/slider/site/trunk/content/developing/building.md
incubator/slider/site/trunk/content/docs/getting_started.md
incubator/slider/site/trunk/content/index.md
Modified: incubator/slider/site/trunk/content/developing/building.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/building.md?rev=1608675&r1=1608674&r2=1608675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/developing/building.md (original)
+++ incubator/slider/site/trunk/content/developing/building.md Tue Jul 8 07:08:35 2014
@@ -31,6 +31,10 @@ of the local host working. Tests will fa
Slider is built on Java 7 -please have a JDK for Java 7 or 8 set up
+### Python
+
+Slider requires python 2.7
+
### Maven
You will need a version of Maven 3.0+, set up with enough memory
Modified: incubator/slider/site/trunk/content/docs/getting_started.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/getting_started.md?rev=1608675&r1=1608674&r2=1608675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/getting_started.md (original)
+++ incubator/slider/site/trunk/content/docs/getting_started.md Tue Jul 8 07:08:35 2014
@@ -102,6 +102,8 @@ Expand the tar file: `tar -xvf slider-0
Browse to the Slider directory: `cd slider-0.30.0/bin`
+*Add java to the PATH.*
+
export PATH=$PATH:/usr/jdk64/jdk1.7.0_45/bin
(or the path to the JDK bin directory)
@@ -165,91 +167,19 @@ Perform the following steps to create th
hdfs dfs -mkdir /slider/agent
- hdfs dfs -mkdir /slider/agent/conf
-
- hdfs dfs -copyFromLocal ${slider-install-dir}/slider-0.30.0/agent/slider-agent-0.30.0.tar.gz /slider/agent
-
-### Create and deploy Slider Agent configuration
-
-Create an agent config file (agent.ini) based on the sample available at:
-
- ${slider-install-dir}/slider-0.30.0/agent/conf/agent.ini
-
-The sample agent.ini file can be used as is (see below). Some of the parameters of interest are:
-
-# `log_level` = INFO or DEBUG, to control the verbosity of log
-# `app_log_dir` = the relative location of the application log file
-# `log_dir` = the relative location of the agent and command log file
-
- [server]
- hostname=localhost
- port=8440
- secured_port=8441
- check_path=/ws/v1/slider/agents/
- register_path=/ws/v1/slider/agents/{name}/register
- heartbeat_path=/ws/v1/slider/agents/{name}/heartbeat
-
- [agent]
- app_pkg_dir=app/definition
- app_install_dir=app/install
- app_run_dir=app/run
- app_task_dir=app/command-log
- app_log_dir=app/log
- app_tmp_dir=app/tmp
- log_dir=infra/log
- run_dir=infra/run
- version_file=infra/version
- log_level=INFO
-
- [python]
-
- [command]
- max_retries=2
- sleep_between_retries=1
-
- [security]
-
- [heartbeat]
- state_interval=6
- log_lines_count=300
-
-
-Once created, deploy the agent.ini file to HDFS:
-
- su yarn
-
- hdfs dfs -copyFromLocal agent.ini /slider/agent/conf
+ hdfs dfs -copyFromLocal ${slider-install-dir}/slider-0.30.0/agent/slider-agent.tar.gz /slider/agent
-## <a name="downsample"></a>Download Sample Application Packages
-There are three sample application packages available for download to use with Slider:
+## <a name="downsample"></a>Create Application Packages
-<table>
- <tr>
- <td>Application</td>
- <td>Version</td>
- <td>URL</td>
- </tr>
- <tr>
- <td>Apache HBase</td>
- <td>0.96.0</td>
- <td>http://public-repo-1.hortonworks.com/slider/hbase_v096.tar</td>
- </tr>
- <tr>
- <td>Apache Storm</td>
- <td>0.9.1</td>
- <td>http://public-repo-1.hortonworks.com/slider/storm_v091.tar</td>
- </tr>
- <tr>
- <td>Apache Accumulo</td>
- <td>1.5.1</td>
- <td>http://public-repo-1.hortonworks.com/slider/accumulo_v151.tar</td>
- </tr>
-</table>
+Slider "master" branch has three sample application packages available use with Slider:
+* app-packages/hbase: The README.txt file describes how to create a Slider Application Package for HBase. By default it will create a package for HBase 0.98.3 but you can create the same for other versions.
+* app-packages/accumulo: The README.txt file describes how to create a Slider Application Package for Accumulo.
+* app-packages/storm: TBD
-Download the packages and deploy one of these sample applications to YARN via Slider using the steps below.
-
+Create one or more Slider application packages and follow the steps below to install them.
+
## <a name="installapp"></a>Install, Configure, Start and Verify Sample Application
* [Load Sample Application Package](#load)
@@ -264,17 +194,7 @@ Download the packages and deploy one of
### <a name="load"></a>Load Sample Application Package
- hdfs dfs -copyFromLocal *sample-application-package/slider
-
-If necessary, create HDFS folders needed by the application. For example, HBase requires the following HDFS-based setup:
-
- su hdfs
-
- hdfs dfs -mkdir /apps
-
- hdfs dfs -mkdir /apps/hbase
-
- hdfs dfs -chown yarn:hdfs /apps/hbase
+ hdfs dfs -copyFromLocal *sample-application-package* /slider
### <a name="create"></a>Create Application Specifications
@@ -282,8 +202,8 @@ Configuring a Slider application consist
and the *[Application Configuration](#appconfig). Below are guidelines for creating these files.
*Note: There are sample Resource Specifications (**resources.json**) and Application Configuration
-(**appConfig.json**) files in the *[Appendix](#appendixa)* and also in the root directory of the
-Sample Applications packages (e.g. /**hbase-v096/resources.json** and /**hbase-v096/appConfig.json**).*
+(**appConfig.json**) files in the root directory of the
+Sample Applications packages (e.g. /**resources.json** and /**appConfig.json**). The Slider git repo also contains sample appConfig.json and resources.json in the application packages directory (e.g. app-packages/hbase/appConfig.json)*
#### <a name="resspec"></a>Resource Specification
@@ -293,12 +213,6 @@ As Slider creates each instance of a com
All this information goes into the **Resources Specification** file ("Resource Spec") named `resources.json`. The Resource Spec tells Slider how many instances of each component in the application (such as an HBase RegionServer) to deploy and the parameters for YARN.
-Sample Resource Spec files are available in the Appendix:
-
-* [Appendix A: Storm Sample Resource Specification](#heading=h.1hj8hn5xne7c)
-
-* [Appendix B: HBase Sample Resource Specification](#heading=h.l7z5mvhvxmzv)
-
Store the Resource Spec file on your local disk (e.g. `/tmp/resources.json`).
#### <a name="appconfig"></a>Application Configuration
@@ -307,20 +221,6 @@ Alongside the Resource Spec there is the
For example, the heap sizes of the JVMs, The App Config defines the configuration details **specific to the application and component** instances. For HBase, this includes any values for the *to-be-generated *hbase-site.xml file, as well as options for individual components, such as their heap size.
-Sample App Configs are available in the Appendix:
-
-* [Appendix A: Storm Sample Application Configuration](#heading=h.2qai3c6w260l)
-
-* [Appendix B: HBase Sample Application Configuration](#heading=h.hetv1wn44c5x)
-
-Store the appConfig.json file on your local disc and a copy in HDFS:
-
- su yarn
-
- hdfs dfs -mkdir /slider/appconf
-
- hdfs dfs -copyFromLocal appConf.json /slider/appconf
-
### <a name="start"></a>Start the Application
Once the steps above are completed, the application can be started through the **Slider Command Line Interface (CLI)**.
@@ -331,17 +231,52 @@ Change directory to the "bin" directory
Execute the following command:
- ./slider create cl1 --manager yourResourceManagerHost:8050 --image hdfs://yourNameNodeHost:8020/slider/agent/slider-agent-0.30.0.tar.gz --template appConfig.json --resources resources.json
+ ./slider create cl1 --image hdfs://yourNameNodeHost:8020/slider/agent/slider-agent.tar.gz --template appConfig.json --resources resources.json
### <a name="verify"></a>Verify the Application
The successful launch of the application can be verified via the YARN Resource Manager Web UI. In most instances, this UI is accessible via a web browser at port 8088 of the Resource Manager Host:
-
+
The specific information for the running application is accessible via the "ApplicationMaster" link that can be seen in the far right column of the row associated with the running application (probably the top row):
-
+
+
+### <a name="client-config"></a>Obtain client config
+
+An application publishes several useful details that can be used to manage and communicate with the application instance.
+
+**All published data**
+
+All published data are available at the `/ws/v1/slider/publisher` end point of the AppMaster (e.g. http://c6403.ambari.apache.org:34701/ws/v1/slider/publisher). Its obtainable through `slider-client status <app name>` where field `info.am.web.url` specifies the base address. The URL is also advertised in the AppMaster tracking UI.
+
+**Client configuration**
+Client configurations are at `/ws/v1/slider/publisher/slider\<config name>` where *config name* can be
+
+* site config file name without extension, such as `hbase-site` (e.g. http://c6403.ambari.apache.org:34701/ws/v1/slider/publisher/slider/hbase-site) where the output is json formatted name-value pairs
+* or, with extension such as `hbase-site.xml` (http://c6403.ambari.apache.org:34701/ws/v1/slider/publisher/slider/hbase-site.xml) to get an XML formatted output that can be consumed by the clients
+
+**Log locations**
+
+The log locations for various containers in the application instance are at `ws/v1/slider/publisher/slider/logfolders`
+
+ {
+
+ "description": "LogFolders",
+ "entries": {
+ "c6403.ambari.apache.org-container_1404783501621_0002_01_000002-AGENT_LOG_ROOT": "/hadoop/yarn/log/application_1404783501621_0002/container_1404783501621_0002_01_000002",
+ "c6403.ambari.apache.org-container_1404783501621_0002_01_000003-AGENT_LOG_ROOT": "/hadoop/yarn/log/application_1404783501621_0002/container_1404783501621_0002_01_000003",
+ "c6403.ambari.apache.org-container_1404783501621_0002_01_000003-AGENT_WORK_ROOT": "/hadoop/yarn/local/usercache/yarn/appcache/application_1404783501621_0002/container_1404783501621_0002_01_000003",
+ "c6403.ambari.apache.org-container_1404783501621_0002_01_000002-AGENT_WORK_ROOT": "/hadoop/yarn/local/usercache/yarn/appcache/application_1404783501621_0002/container_1404783501621_0002_01_000002"
+ },
+ "updated": 0,
+ "empty": false
+ }
+
+**Quick links**
+
+An application may publish some quick links for the monitoring UI, JMX endpoint, etc. as the listening ports are dynamically allocated and thus need to be published. These information is available at `/ws/v1/slider/publisher/slider/quicklinks`.
### <a name="manage"></a>Manage the Application Lifecycle
@@ -349,161 +284,83 @@ Once started, applications can be frozen
#### Frozen:
- ./slider freeze cl1 --manager yourResourceManagerHost:8050 --filesystem hdfs://yourNameNodeHost:8020
+ ./slider freeze cl1
#### Thawed:
- ./slider thaw cl1 --manager yourResourceManagerHost:8050 --filesystem hdfs://yourNameNodeHost:8020
+ ./slider thaw cl1
#### Destroyed:
- ./slider destroy cl1 --manager yourResourceManagerHost:8050 --filesystem hdfs://yourNameNodeHost:8020
+ ./slider destroy
#### Flexed:
- ./slider flex cl1 --component worker 5 --manager yourResourceManagerHost:8050 --filesystem hdfs://yourNameNodeHost:8020
+ ./slider flex cl1 --component worker 5
-# <a name="appendixa"></a>Appendix A: Apache Storm Sample Application Specifications
-## Storm Resource Specification Sample
+# <a name="appendixa"></a>Appendix A: Debugging Slider-Agent
- {
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- },
- "components" : {
- "slider-appmaster" : {
- },
- "NIMBUS" : {
- "yarn.role.priority" : "1",
- "yarn.component.instances" : "1"
- },
- "STORM_REST_API" : {
- "yarn.role.priority" : "2",
- "yarn.component.instances" : "1"
- },
- "STORM_UI_SERVER" : {
- "yarn.role.priority" : "3",
- "yarn.component.instances" : "1"
- },
- "DRPC_SERVER" : {
- "yarn.role.priority" : "4",
- "yarn.component.instances" : "1"
- },
- "SUPERVISOR" : {
- "yarn.role.priority" : "5",
- "yarn.component.instances" : "1"
- }
- }
- }
+### Create and deploy Slider Agent configuration
+Create an agent config file (agent.ini) based on the sample available at:
-## Storm Application Configuration Sample
+ ${slider-install-dir}/slider-0.30.0/agent/conf/agent.ini
- {
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- "A site property for type XYZ with name AA": "its value",
- "site.XYZ.AA": "Value",
- "site.hbase-site.hbase.regionserver.port": "0",
- "site.core-site.fs.defaultFS": "${NN_URI}",
- "Using a well known keyword": "Such as NN_HOST for name node host",
- "site.hdfs-site.dfs.namenode.http-address": "${NN_HOST}:50070",
- "a global property used by app scripts": "not affiliated with any site-xml",
- "site.global.app_user": "yarn",
- "Another example of available keywords": "Such as AGENT_LOG_ROOT",
- "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
- "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
- }
- }
+The sample agent.ini file can be used as is (see below). Some of the parameters of interest are:
+`log_level` = INFO or DEBUG, to control the verbosity of log
-# <a name="appendixb"></a>Appendix B: Apache HBase Sample Application Specifications
+`app_log_dir` = the relative location of the application log file
-## HBase Resource Specification Sample
+`log_dir` = the relative location of the agent and command log file
- {
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- },
- "components" : {
- "HBASE_MASTER" : {
- "yarn.role.priority" : "1",
- "yarn.component.instances" : "1"
- },
- "slider-appmaster" : {
- },
- "HBASE_REGIONSERVER" : {
- "yarn.role.priority" : "2",
- "yarn.component.instances" : "1"
- }
- }
- }
+ [server]
+ hostname=localhost
+ port=8440
+ secured_port=8441
+ check_path=/ws/v1/slider/agents/
+ register_path=/ws/v1/slider/agents/{name}/register
+ heartbeat_path=/ws/v1/slider/agents/{name}/heartbeat
+ [agent]
+ app_pkg_dir=app/definition
+ app_install_dir=app/install
+ app_run_dir=app/run
+ app_dbg_cmd=
+ debug_mode_enabled=true
-## HBase Application Configuration Sample
+ app_task_dir=app/command-log
+ app_log_dir=app/log
+ app_tmp_dir=app/tmp
- {
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- "agent.conf": "/slider/agent/conf/agent.ini",
- "agent.version": "/slider/agent/version",
- "application.def": "/slider/hbase_v096.tar",
- "config_types": "core-site,hdfs-site,hbase-site",
- "java_home": "/usr/jdk64/jdk1.7.0_45",
- "package_list": "files/hbase-0.96.1-hadoop2-bin.tar",
- "site.global.app_user": "yarn",
- "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
- "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
- "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/hbase-0.96.1-hadoop2",
- "site.global.app_install_dir": "${AGENT_WORK_ROOT}/app/install",
- "site.global.hbase_master_heapsize": "1024m",
- "site.global.hbase_regionserver_heapsize": "1024m",
- "site.global.user_group": "hadoop",
- "site.global.security_enabled": "false",
- "site.hbase-site.hbase.hstore.flush.retries.number": "120",
- "site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
- "site.hbase-site.hbase.hstore.compactionThreshold": "3",
- "site.hbase-site.hbase.rootdir": "${NN_URI}/apps/hbase/data",
- "site.hbase-site.hbase.stagingdir": "${NN_URI}/apps/hbase/staging",
- "site.hbase-site.hbase.regionserver.handler.count": "60",
- "site.hbase-site.hbase.regionserver.global.memstore.lowerLimit": "0.38",
- "site.hbase-site.hbase.hregion.memstore.block.multiplier": "2",
- "site.hbase-site.hbase.hregion.memstore.flush.size": "134217728",
- "site.hbase-site.hbase.superuser": "yarn",
- "site.hbase-site.hbase.zookeeper.property.clientPort": "2181",
- "site.hbase-site.hbase.regionserver.global.memstore.upperLimit": "0.4",
- "site.hbase-site.zookeeper.session.timeout": "30000",
- "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
- "site.hbase-site.hbase.local.dir": "${hbase.tmp.dir}/local",
- "site.hbase-site.hbase.hregion.max.filesize": "10737418240",
- "site.hbase-site.hfile.block.cache.size": "0.40",
- "site.hbase-site.hbase.security.authentication": "simple",
- "site.hbase-site.hbase.defaults.for.version.skip": "true",
- "site.hbase-site.hbase.zookeeper.quorum": "${ZK_HOST}",
- "site.hbase-site.zookeeper.znode.parent": "/hbase-unsecure",
- "site.hbase-site.hbase.hstore.blockingStoreFiles": "10",
- "site.hbase-site.hbase.hregion.majorcompaction": "86400000",
- "site.hbase-site.hbase.security.authorization": "false",
- "site.hbase-site.hbase.cluster.distributed": "true",
- "site.hbase-site.hbase.hregion.memstore.mslab.enabled": "true",
- "site.hbase-site.hbase.client.scanner.caching": "100",
- "site.hbase-site.hbase.zookeeper.useMulti": "true",
- "site.hbase-site.hbase.regionserver.info.port": "0",
- "site.hbase-site.hbase.master.info.port": "60010",
- "site.hbase-site.hbase.regionserver.port": "0",
- "site.core-site.fs.defaultFS": "${NN_URI}",
- "site.hdfs-site.dfs.namenode.https-address": "${NN_HOST}:50470",
- "site.hdfs-site.dfs.namenode.http-address": "${NN_HOST}:50070"
- }
- }
+ log_dir=infra/log
+ run_dir=infra/run
+ version_file=infra/version
+
+ log_level=INFO
+
+ [python]
+
+ [command]
+ max_retries=2
+ sleep_between_retries=1
+
+ [security]
+
+ [heartbeat]
+ state_interval=6
+ log_lines_count=300
+
+
+Once created, deploy the agent.ini file to HDFS (it can be deployed in any location accessible to application instance):
+
+ su yarn
+
+ hdfs dfs -copyFromLocal agent.ini /slider/agent/conf
+
+Modify the --template json file (appConfig.json) and add the location of the agent.ini file.
+
+ "agent.conf": "/slider/agent/conf/agent.ini"
Added: incubator/slider/site/trunk/content/images/image_0_app_master_ui_lrg.png
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/images/image_0_app_master_ui_lrg.png?rev=1608675&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/slider/site/trunk/content/images/image_0_app_master_ui_lrg.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/slider/site/trunk/content/images/image_1_apps_rm_ui-lrg.png
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/images/image_1_apps_rm_ui-lrg.png?rev=1608675&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/slider/site/trunk/content/images/image_1_apps_rm_ui-lrg.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: incubator/slider/site/trunk/content/index.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/index.md?rev=1608675&r1=1608674&r2=1608675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/index.md (original)
+++ incubator/slider/site/trunk/content/index.md Tue Jul 8 07:08:35 2014
@@ -78,10 +78,21 @@ code changes and as of this writing, it
## Developing
-* [Architecture](/design/index.html)
-* [Developing](/developing/index.html)
+* [Building](/developing/building.html)
+* [Debugging](/docs/debugging.html)
+* [Testing](/developing/testing.html)
+* [Functional Testing](/developing/functional_tests.html)
+* [Manual Testing](/developing/manual_testing.html)
+* [Releasing](/developing/releasing.html)
+
+
+## Design
+* [Overview](/design/architecture.html)
* [Application Needs](/docs/slider_specs/application_needs.html)
+* [Expected Behavior](/design/specification/index.html)
* [Service Registry](/design/registry/index.html)
+* [Role History](/design/rolehistory.html)
+
## Disclaimer