You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by st...@apache.org on 2014/11/21 13:28:23 UTC
svn commit: r1640906 - in /incubator/slider/site/trunk/content:
developing/debugging.md developing/demo.md developing/index.md
docs/debugging.md docs/getting_started.md docs/getting_started_40.md index.md
Author: stevel
Date: Fri Nov 21 12:28:22 2014
New Revision: 1640906
URL: http://svn.apache.org/r1640906
Log:
SLIDER-527 add slider demo script
Added:
incubator/slider/site/trunk/content/developing/debugging.md
- copied unchanged from r1637989, incubator/slider/site/trunk/content/docs/debugging.md
incubator/slider/site/trunk/content/developing/demo.md
Removed:
incubator/slider/site/trunk/content/docs/debugging.md
incubator/slider/site/trunk/content/docs/getting_started_40.md
Modified:
incubator/slider/site/trunk/content/developing/index.md
incubator/slider/site/trunk/content/docs/getting_started.md
incubator/slider/site/trunk/content/index.md
Added: incubator/slider/site/trunk/content/developing/demo.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/demo.md?rev=1640906&view=auto
==============================================================================
--- incubator/slider/site/trunk/content/developing/demo.md (added)
+++ incubator/slider/site/trunk/content/developing/demo.md Fri Nov 21 12:28:22 2014
@@ -0,0 +1,137 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Simple Slider Demo
+
+These instructions cover how to give a simple demo of slider, showing
+* creating an application instance
+* starting and stopping an instance
+* publishing information in the registry
+* flexing an instance
+* recovery from component failure
+* recovery from AM failure
+
+It does not cover
+* Configuring the application instance via `app_conf.json`
+* Functional use of the deployed application other than viewing its web pages
+* Security
+
+## Preparation
+
+Before starting the demo you need
+
+1. A Hadoop 2.6+ cluster, one node or greater
+1. A local installation of the slider client (unzipped into a directory is sufficient)
+1. A `slider-client.xml` file containing the binding information for the Hadoop cluster
+
+### Hadoop cluster setup
+
+The cluster must be set up with no checks on memory use, and no minimum size on container allocations. This ensures that all requested VMs will be created, irrespective of how much memory there is available on the VM.
+
+
+ <property>
+ <name>yarn.scheduler.minimum-allocation-mb</name>
+ <value>64</value>
+ </property>
+
+ <property>
+ <name>yarn.log-aggregation-enable</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>yarn.nodemanager.pmem-check-enabled</name>
+ <value>false</value>
+ </property>
+
+ <property>
+ <name>yarn.nodemanager.vmem-check-enabled</name>
+ <value>false</value>
+ </property>
+
+
+### Local environment and path setup
+
+
+#### Bash
+
+ export HADOOP_CONF_DIR=~/clusters/demo/hadoop-conf
+
+ export SLIDER_CONF_DIR=~/clusters/demo/slider;
+
+ export PATH=~/slider/slider-assembly/target/slider-0.60.0-incubating-all/slider-0.60.0-incubating/bin
+
+#### Fish
+
+
+ set -gx HADOOP_CONF_DIR ~/clusters/demo/hadoop-conf
+
+ set -gx SLIDER_CONF_DIR ~/clusters/demo/slider;
+
+ set -gx PATH $PATH ~/slider/slider-assembly/target/slider-0.60.0-incubating-all/slider-0.60.0-incubating/bin
+
+
+#### Tests
+
+
+ slider version
+ slider list
+
+### Creating the HDFS paths
+
+HDFS needs to have the user's path created, with the username of the user giving the demo, here `demo`. This username *MUST* be that of the local user.
+
+
+ hdfs dfs -chmod 777 /
+ hdfs dfs -mkdir -p /user/demo
+ hdfs dfs -chown demo /user/demo
+ hdfs dfs -chmod 777 /user
+
+The YARN user account needs to be present too (in a reformatted VM this may be lost)
+
+ hdfs dfs -mkdir -p /user/yarn
+ hdfs dfs -chown yarn /user/yarn
+
+### Building and installing the application
+
+This demo uses HBase, so must have the slider hbase application package installed in the cluster.
+
+1. Download the hbase bin package, such as `hbase-0.99.1-bin.tar.gz`
+1. rename it `hbase-0.99.1.tar.gz`, so that the name of the file matches the path to hbase inside it
+1. Untar it somewhere too (e.g. `/Users/demo/hbase-0.99.1`)
+1. Build the HBase application package
+
+ mvn clean package -Phbase-app-package -Dpkg.version=0.99.1 \
+ -Dpkg.name=hbase-0.99.1.tar.gz \
+ -Dpkg.src=/Users/demo/hbase-0.99.1
+
+1. Upload it
+
+ slider install-package --name HBASE --package target/slider-hbase-app-package-0.99.1.zip
+
+### Preparing the resource definitions
+
+
+Take the template app's `resources-default.json` and `appConf-default.json` *FROM the target/ subdirectory*. They must be the ones with package versions fully expanded.
+
+In `appConfig.json`
+
+1. cut the heap size down to `"256m"` in all components
+1. make sure any needed path to the JDK base directory is valid.
+
+In `resources.json`
+
+1. set `"yarn.memory": "128"` everywhere,
+1. Turn set the number of instances the THRIFT and REST components to 0.
+
Modified: incubator/slider/site/trunk/content/developing/index.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/index.md?rev=1640906&r1=1640905&r2=1640906&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/developing/index.md (original)
+++ incubator/slider/site/trunk/content/developing/index.md Fri Nov 21 12:28:22 2014
@@ -23,15 +23,16 @@ strongly encourage people to do so.
Here are documents covering how to go about building, testing and releasing
Slider
-* [Building](/developing/building.html)
-* [Debugging](/docs/debugging.html)
-* [Testing](/developing/testing.html)
-* [Functional Testing](/developing/functional_tests.html)
-* [Manual Testing](/developing/manual_testing.html)
-* [Releasing](/developing/releasing.html)
-* [Releasing](/developing/releasing.html)
-* [Style Guide](/developing/style.html)
-* [Windows Development and Testing](/developing/windows.html)
+* [Building](building.html)
+* [Debugging](debugging.html)
+* [Testing](testing.html)
+* [Functional Testing](functional_tests.html)
+* [Manual Testing](manual_testing.html)
+* [Releasing](releasing.html)
+* [Releasing](releasing.html)
+* [Style Guide](style.html)
+* [Windows Development and Testing](windows.html)
+* [Demo Script](demo.html)
Modified: incubator/slider/site/trunk/content/docs/getting_started.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/getting_started.md?rev=1640906&r1=1640905&r2=1640906&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/getting_started.md (original)
+++ incubator/slider/site/trunk/content/docs/getting_started.md Fri Nov 21 12:28:22 2014
@@ -16,9 +16,9 @@
-->
# Apache Slider: Getting Started
+
This page is updated to reflect the latest code in "develop".
-For the version applicable to the last release go to [Slider-40](/docs/getting_started_40.html).
## Introduction
@@ -91,7 +91,7 @@ There are other options detailed in the
## <a name="build"></a>Build Slider
* From the top level directory, execute `mvn clean site:site site:stage package -DskipTests`
-* Use the generated compressed tar file in slider-assembly/target directory (e.g. slider-0.50.0-all.tar.gz or slider-0.50.0-all.zip) for the subsequent steps
+* Use the generated compressed tar file in slider-assembly/target directory (e.g. slider-0.60.0-all.tar.gz or slider-0.60.0-all.zip) for the subsequent steps
* If you are cloning the Slider git repo, go to `master` branch for the latest released or `develop` for the latest under development
## <a name="install"></a>Install Slider
@@ -107,11 +107,11 @@ You can run Slider app as any user. Only
Login as the "yarn" user (assuming this is a host associated with the installed cluster). E.g., `su yarn`
*This assumes that all apps are being run as âyarnâ user*
-Expand the tar file: `tar -xvf slider-0.50.0-incubating-all.tar.gz` or `unzip slider-0.50.0-incubating-all.zip`
+Expand the tar file: `tar -xvf slider-0.60.0-incubating-all.tar.gz` or `unzip slider-0.60.0-incubating-all.zip`
## Configure Slider
-Browse to the Slider directory: `cd slider-0.50.0/conf`
+Browse to the Slider directory: `cd slider-0.60.0/conf`
Edit slider-env.sh and specify correct values.
@@ -120,7 +120,7 @@ Edit slider-env.sh and specify correct v
If you are on a node that does not have the hadoop conf folder then you can add the relevant configurations into slider-client.xml.
-For example, if you are targeting a non-secure cluster with no HA for NameNode or ResourceManager, modify Slider configuration file `${slider-install-dir}/slider-0.50.0/conf/slider-client.xml` to add the following properties:
+For example, if you are targeting a non-secure cluster with no HA for NameNode or ResourceManager, modify Slider configuration file `${slider-install-dir}/slider-0.60.0/conf/slider-client.xml` to add the following properties:
<property>
<name>yarn.application.classpath</name>
@@ -154,9 +154,9 @@ If the target cluster is on Windows then
Execute:
- ${slider-install-dir}/slider-0.50.0/bin/slider version
+ ${slider-install-dir}/slider-0.60.0/bin/slider version
OR
- python %slider-install-dir%/slider-0.50.0/bin/slider.py version
+ python %slider-install-dir%/slider-0.60.0/bin/slider.py version
Ensure there are no errors and you can see "Compiled against Hadoop 2.6.0". **This ensures that Slider is installed correctly.**
@@ -178,13 +178,13 @@ Perform the following steps to create th
Slider "master" branch has several sample application packages available for use with Slider:
-* app-packages/memcached-win or memcached: The README.txt describes how to create the Slider Application package for memcached for linux or windows. **memcached** app packages are a good place to [start](slider_specs/hello_world_slider_app.html)
+* `app-packages/memcached-win` or memcached: The README.txt describes how to create the Slider Application package for memcached for linux or windows. **memcached** app packages are a good place to [start](slider_specs/hello_world_slider_app.html)
-* app-packages/hbase or hbase-win: The README.txt file describes how to create a Slider Application Package for HBase. By default it will create a package for HBase 0.98.3 but you can create the same for other versions.
+* `app-packages/hbase` or hbase-win: The README.txt file describes how to create a Slider Application Package for HBase. By default it will create a package for HBase 0.98.3 but you can create the same for other versions.
-* app-packages/accumulo: The README.txt file describes how to create a Slider Application Package for Accumulo.
+* `app-packages/accumulo`: The README.txt file describes how to create a Slider Application Package for Accumulo.
-* app-packages/storm or storm-win: The README.txt describes how to create the Slider Application package for Storm.
+* `app-packages/storm` or `storm-win`: The README.txt describes how to create the Slider Application package for Storm.
Create one or more Slider application packages and follow the steps below to install them.
@@ -250,7 +250,7 @@ Once the steps above are completed, the
Change directory to the "bin" directory under the slider installation
- cd ${slider-install-dir}/slider-0.50.0/bin
+ cd ${slider-install-dir}/slider-0.60.0/bin
Execute the following command:
@@ -330,7 +330,7 @@ Once started, applications can be frozen
Create an agent config file (agent.ini) based on the sample available at:
- ${slider-install-dir}/slider-0.50.0/agent/conf/agent.ini
+ ${slider-install-dir}/slider-0.60.0/agent/conf/agent.ini
The sample agent.ini file can be used as is (see below). Some of the parameters of interest are:
Modified: incubator/slider/site/trunk/content/index.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/index.md?rev=1640906&r1=1640905&r2=1640906&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/index.md (original)
+++ incubator/slider/site/trunk/content/index.md Fri Nov 21 12:28:22 2014
@@ -74,7 +74,6 @@ code changes and as of this writing, it
* [How to define a new slider-packaged application](/docs/slider_specs/index.html)
* [Application configuration model](/docs/configuration/index.html)
* [High Availability](/docs/high_availability.html)
-* [Debugging](/docs/debugging.html)
* [Troubleshooting](/docs/troubleshooting.html)