You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by st...@apache.org on 2014/11/25 20:03:56 UTC
svn commit: r1641675 - in /incubator/slider/site/trunk/content: ./ design/
design/registry/ design/specification/ developing/ docs/
docs/configuration/ docs/slider_specs/ downloads/
Author: stevel
Date: Tue Nov 25 19:03:55 2014
New Revision: 1641675
URL: http://svn.apache.org/r1641675
Log:
SLIDER-193 Improve site documentation
Modified:
incubator/slider/site/trunk/content/design/architecture.md
incubator/slider/site/trunk/content/design/registry/index.md
incubator/slider/site/trunk/content/design/rolehistory.md
incubator/slider/site/trunk/content/design/specification/cli-actions.md
incubator/slider/site/trunk/content/developing/building.md
incubator/slider/site/trunk/content/developing/functional_tests.md
incubator/slider/site/trunk/content/developing/manual_testing.md
incubator/slider/site/trunk/content/developing/releasing.md
incubator/slider/site/trunk/content/developing/windows.md
incubator/slider/site/trunk/content/docs/client-configuration.md
incubator/slider/site/trunk/content/docs/configuration/specification.md
incubator/slider/site/trunk/content/docs/examples.md
incubator/slider/site/trunk/content/docs/getting_started.md
incubator/slider/site/trunk/content/docs/manpage.md
incubator/slider/site/trunk/content/docs/security.md
incubator/slider/site/trunk/content/docs/slider_specs/canonical_scenarios.md
incubator/slider/site/trunk/content/docs/slider_specs/specifying_exports.md
incubator/slider/site/trunk/content/downloads/index.md
incubator/slider/site/trunk/content/index.md
Modified: incubator/slider/site/trunk/content/design/architecture.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/design/architecture.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/design/architecture.md (original)
+++ incubator/slider/site/trunk/content/design/architecture.md Tue Nov 25 19:03:55 2014
@@ -140,6 +140,7 @@ To enable access to the database files a
AM will pass this same identity down to the created containers. This information *identifies* the user -but does not *authenticate* them: they are trusted to be who they claim to be.
### Agent to Application Master Secure Communication
-By default, one-way SSL is leveraged to secure the communication between Slider agents and the Application Master. However, two-way SSL can be enabled. A more detailed discussion of the SSL implementation in Slider can be found [here](/design/ssl_implementation.html).
+
+By default, one-way SSL is leveraged to secure the communication between Slider agents and the Application Master. However, two-way SSL can be enabled. A more detailed discussion of the SSL implementation in Slider can be found [in the SSL documentation](/design/ssl_implementation.html).
Modified: incubator/slider/site/trunk/content/design/registry/index.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/design/registry/index.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/design/registry/index.md (original)
+++ incubator/slider/site/trunk/content/design/registry/index.md Tue Nov 25 19:03:55 2014
@@ -17,6 +17,20 @@
# Apache Slider Service Registry
+
+## Important: This document is obsolete
+
+This secion has been supplanted by the YARN Service Registry; a cross-application
+service registry built into Apache Hadoop, with special integration with
+the YARN Resource Manager, security in a secure cluster, and other features.
+
+This registry , the "YARN-913" was implemented and tested by the Slider team,
+so we are confident that it works for Slider as well as other applications.
+Indeed, for the Hadoop 2.6 release, Slider was the functional test suite for
+the registry.
+
+## Introduction
+
The service registry is a core part of the Slider Architecture -it is how
dynamically generated configurations are published for clients to pick up.
Modified: incubator/slider/site/trunk/content/design/rolehistory.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/design/rolehistory.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/design/rolehistory.md (original)
+++ incubator/slider/site/trunk/content/design/rolehistory.md Tue Nov 25 19:03:55 2014
@@ -122,10 +122,12 @@ used in the past.
* This history is used when selecting a node for a role.
* This history remembers when nodes were allocated. These are re-requested
-when thawing a cluster.
+when starting an application instance again, and when re-requesting
+components after any node failure.
* It must also remember when nodes were released -these are re-requested
-when returning the cluster size to a previous size during flex operations.
+when expanding a set of instances of a specific component
+to a previous size during flex operations.
* It has to track nodes for which Slider has an outstanding container request
with YARN. This ensures that the same node is not requested more than once
@@ -239,7 +241,7 @@ let YARN choose.
* Size of the data structure is `O(nodes * role-instances`). This
could be mitigated by regular cleansing of the structure. For example, at
-thaw time (or intermittently) all unused nodes > 2 weeks old could be dropped.
+startup time (or intermittently) all unused nodes > 2 weeks old could be dropped.
* Locating a free node could take `O(nodes)` lookups -and if the criteria of "newest"
is included, will take exactly `O(nodes)` lookups. As an optimization, a list
of recently explicitly released nodes can be maintained.
@@ -300,7 +302,7 @@ Records the details about all of a roles
NodeEntry.available(): boolean = active - releasing == 0 && requested == 0
The two fields `releasing` and `requested` are used to track the ongoing
-state of YARN requests; they do not need to be persisted across freeze/thaw
+state of YARN requests; they do not need to be persisted across stop/start
cycles. They may be relevant across AM restart, but without other data
structures in the AM, not enough to track what the AM was up to before
it was restarted. The strategy will be to ignore unexpected allocation
@@ -402,7 +404,7 @@ list of all Nodes which are available fo
using a comparator that places the most recently released node ahead of older
nodes.
-This list is not persisted -when a Slider Cluster is frozen it is moot, and when
+This list is not persisted -when a Slider Cluster is stopped it is moot, and when
an AM is restarted this structure will be rebuilt.
1. When a node is needed for a new request, this list is consulted first.
@@ -445,12 +447,14 @@ then re-requesting containers on it will
### Bootstrap
-1. Persistent Role History file not found; empty data structures created.
+1Persistent Role History file not found; empty data structures created.
-### Thaw
+### Restart
-When thawing, the Role History should be loaded -if it is missing Slider
-must revert to the bootstrap actions.
+When starting a cluster, the Role History should be loaded.
+
+If the history is missing *or cannot be loaded for any reason*,
+Slider must revert to the bootstrap actions.
If found, the Role History will contain Slider's view of the Slider Cluster's
state at the time the history was saved, explicitly recording the last-used
@@ -497,7 +501,7 @@ and the available node list contains a s
### AM Restart
-1: Create the initial data structures as the thaw operation
+1: Create the initial data structures as a normal start operation
2: update the structure with the list of live nodes, removing those nodes
from the list of available nodes
@@ -601,15 +605,15 @@ nodemap is required (unless there is som
up), or a sorted version of the nodemap is maintained, with a node placed
at the front of this list whenever its is updated.
-**Thaw-time problems**
+**Startup-time problems**
-There is also the risk that while thawing, the `rolehistory.saved`
+There is also the risk that while starting an applicatin instance, the `rolehistory.saved`
flag may be updated while the cluster flex is in progress, so making the saved
nodes appear out of date. Perhaps the list of recently released nodes could
-be rebuilt at thaw time.
+be rebuilt at startup time.
The proposed `recentlyReleasedList` addresses this, though it creates
-another data structure to maintain and rebuild at cluster thaw time
+another data structure to maintain and rebuild at startup time
from the last-used fields in the node entries.
### AM Callback : onContainersAllocated
@@ -949,8 +953,7 @@ Another entry is saved -presumably the s
{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":1,"active":true,"last_used":0}}}
{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":2,"active":true,"last_used":0}}}
-At this point the cluster was frozen and thawed. Slider does not save the cluster state
-at freeze time, but does as it is rebuilt.
+At this point the cluster was stopped and started.
When the cluster is restarted, every node that was active for a role at the time the file was saved `1384183476028`
is given a last_used timestamp of that time.
@@ -975,12 +978,12 @@ And here, another region server has star
{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":1,"active":true,"last_used":1384183476028}}}
{"entry":{"org.apache.hoya.avro.NodeEntryRecord":{"host":"192.168.1.85","role":2,"active":true,"last_used":1384183476028}}}
-The `last_used` timestamps will not be changed until the cluster is shrunk or thawed, as the `active` flag being set
+The `last_used` timestamps will not be changed until the cluster is shrunk or restarted, as the `active` flag being set
implies that the server is running both roles at the save time of `1384183512217`.
## Resolved issues
-> How best to distinguish at thaw time from nodes used just before thawing
+> How best to distinguish at startup time from nodes used recently
from nodes used some period before? Should the RoleHistory simply forget
about nodes which are older than some threshold when reading in the history?
@@ -996,7 +999,7 @@ that node creates the container and then
to add blacklisting too? Or actually monitor the container start time, and
if a container hasn't been there for very long, don't pick it.
-Startup failures drop the node from the ready-to-use list; the node is no longer
+Container Startup failures drop the node from the ready-to-use list; the node is no longer
trusted. We don't blacklist it (yet)
Modified: incubator/slider/site/trunk/content/design/specification/cli-actions.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/design/specification/cli-actions.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/design/specification/cli-actions.md (original)
+++ incubator/slider/site/trunk/content/design/specification/cli-actions.md Tue Nov 25 19:03:55 2014
@@ -209,12 +209,11 @@ All files that were in the configuration
isfile(HDFS', dest + [filename(c)])
-## Action: Thaw
-**Note: thaw has been renamed to start in the develop branch**
+## Action: Start
- thaw <instancename> [--wait <timeout>]
+ start <instancename> [--wait <timeout>]
-Thaw takes an application instance with configuration and (possibly) data on disk, and
+Takes an application instance with configuration and (possibly) data on disk, and
attempts to create a live application with the specified number of nodes
#### Preconditions
@@ -238,7 +237,7 @@ The cluster specification must exist, be
### Postconditions
-After the thaw has been performed, there is now a queued request in YARN
+After the start operation has been performed, there is now a queued request in YARN
for the chosen (how?) queue
YARN'.Queues'[amqueue] = YARN.Queues[amqueue] + [launch("slider", instancename, requirements, context)]
@@ -415,19 +414,18 @@ to run/exiting on or nearly immediately.
## Action: Create
-Create is simply `build` + `thaw` in sequence - the postconditions from the first
+Create is simply `build` + `start` in sequence - the postconditions from the first
action are intended to match the preconditions of the second.
-## Action: Freeze
-**Note: freeze has been renamed to stop in the develop branch**
+## Action: Stop
- freeze instancename [--wait time] [--message message]
+ stop instancename [--wait time] [--message message]
-The *freeze* action "freezes" the cluster: all its nodes running in the YARN
+The *stop* action "stop" the cluster: all its nodes running in the YARN
cluster are stopped, leaving all the persistent state.
The operation is intended to be idempotent: it is not an error if
-freeze is invoked on an already frozen cluster
+`stop` is invoked on an already stopped cluster
#### Preconditions
@@ -446,7 +444,7 @@ If the `--wait` argument specified a wai
until the cluster has finished or the wait time was exceeded.
If the `--message` argument specified a message -it must appear in the
-YARN logs as the reason the cluster was frozen.
+YARN logs as the reason the cluster was stopped.
The outcome should be the same:
@@ -509,7 +507,7 @@ satisfied.
## Action: Destroy
-Idempotent operation to destroy a frozen cluster -it succeeds if the
+Idempotent operation to destroy a stopped cluster -it succeeds if the
cluster has already been destroyed/is unknown, but not if it is
actually running.
Modified: incubator/slider/site/trunk/content/developing/building.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/building.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/developing/building.md (original)
+++ incubator/slider/site/trunk/content/developing/building.md Tue Nov 25 19:03:55 2014
@@ -30,7 +30,7 @@ work, remote network access is often a p
### Java
-Slider is built on Java 6 or later
+Slider is built on Java 6 or later.
### Python
@@ -43,7 +43,7 @@ You will need a version of Maven 3.0+, s
MAVEN_OPTS=-Xms256m -Xmx512m -Djava.awt.headless=true
-*Important*: As of October 6, 2013, Maven 3.1 is not supported due to
+*Important*: As of December 2014, Maven 3.1 is not supported due to
[version issues](https://cwiki.apache.org/confluence/display/MAVEN/AetherClassNotFound).
### Protoc
@@ -491,10 +491,14 @@ a Java-only production codebase.
Here are some handy aliases to make maven easier
- alias mci='mvn clean install -DskipTests'
- alias mi='mvn install -DskipTests'
- alias mvct='mvn clean test'
- alias mvnsite='mvn site:site -Dmaven.javadoc.skip=true'
- alias mvt='mvn test'
-
+ alias mi="mvn install -DskipTests"
+ alias mvi="mvn install -DskipTests"
+ alias mci="mvn clean install -DskipTests"
+ alias mvt="mvn test"
+ alias mvct="mvn clean test"
+ alias mvp="mvn package -DskipTests"
+ alias mvcp="mvn clean package -DskipTests"
+ alias mvnsite="mvn site:site -Dmaven.javadoc.skip=true"
+ alias mvs="mvn site:site -Dmaven.javadoc.skip=true -DskipTests"
+ alias mvndep="mvn dependency:tree -Dverbose"
Modified: incubator/slider/site/trunk/content/developing/functional_tests.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/functional_tests.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/developing/functional_tests.md (original)
+++ incubator/slider/site/trunk/content/developing/functional_tests.md Tue Nov 25 19:03:55 2014
@@ -160,16 +160,6 @@ need to be changed
<description>Size in MB to ask for containers</description>
<value>192</value>
</property>
-
- <property>
- <name>slider.test.launch.python</name>
- <description>
- Flag to mandate use of the slider.py script.
- Ignored on windows âwhich always uses the python script
- </description>
- <value>false</value>
- </property>
-
Note that while the same properties need to be set in
`slider-core/src/test/resources/slider-client.xml`, those tests take a file in the local
Modified: incubator/slider/site/trunk/content/developing/manual_testing.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/manual_testing.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/developing/manual_testing.md (original)
+++ incubator/slider/site/trunk/content/developing/manual_testing.md Tue Nov 25 19:03:55 2014
@@ -18,7 +18,7 @@
# Manually Testing Apache Slider
Manual testing invloves using Slider package and an AppPackage to perform basic
- cluster functionalities such as create/destroy, flex up/down, and freeze/thaw.
+ cluster functionalities such as create/destroy, flex up/down, and start/stop.
A python helper script is provided that can be used to automatically test and app package.
## `SliderTester.py`
Modified: incubator/slider/site/trunk/content/developing/releasing.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/releasing.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/developing/releasing.md (original)
+++ incubator/slider/site/trunk/content/developing/releasing.md Tue Nov 25 19:03:55 2014
@@ -399,7 +399,12 @@ Publish the release as below:
svn co https://dist.apache.org/repos/dist/release/incubator/slider slider
cd slider
- Edit the KEYS file as needed. The instructions are at the top.
+
+Make sure your PGP keys is in the KEYS file. The instructions to add a key
+are at the top of the file
+
+Then...
+
mkdir $SLIDER_RELEASE $SLIDER_RELEASE/src
cd $SLIDER_RELEASE/src
scp ~/public_html/$SLIDER_RELEASE-rc0/* .
@@ -500,4 +505,32 @@ the actual development version.
git commit -a -m "$SLIDER_RELEASE_JIRA reverting development POMs to $SLIDER_DEV"
- 1. Continue the release process from step 12.
\ No newline at end of file
+ 1. Continue the release process from step 12.
+
+## Publishing artifacts
+
+ Once the vote has (successfully) completed on both slider-dev and incubator-general, the artifacts may be published in two ways
+
+1. tarballs for download via the apache distribution system
+1. JAR files for download via the maven repository
+
+### Publishing files for apache release
+
+See the [ASF release guidelines](http://www.apache.org/dev/release.html)
+
+### Publishing the maven artifacts
+
+* See [ASF guide to publishing maven artifacts](http://www.apache.org/dev/publishing-maven-artifacts.html)
+* See [Apache Parent Pom](http://repo.maven.apache.org/maven2/org/apache/apache/10/apache-10.pom) for a template POM containing the release plugins. This is not used in the slider release
+âit was used as a source of some of the actions; the remainder being from the hadoop project.
+
+
+ Build the artifacts off the SVN commit used for the release.
+
+ mvn -Papache-release
+
+#### Important: App package versions
+
+The (slider 0.60) app package versions mean the artifacts are not suitable for uploading; they generate files such as `slider-storm-app-package-${pkg.version}.pom` which don't contain a valid version number.
+
+We plan to fix this by moving the app packages out of the slider project itself, so drive them off other version information.
\ No newline at end of file
Modified: incubator/slider/site/trunk/content/developing/windows.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/windows.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/developing/windows.md (original)
+++ incubator/slider/site/trunk/content/developing/windows.md Tue Nov 25 19:03:55 2014
@@ -19,7 +19,7 @@ This documents what you need to build an
1. Windows server 2012
1. Create an admin account for yourself
-1. take snapshots as you go along
+1. On a virtual machine: take snapshots as you go along
# Core installs
@@ -27,7 +27,6 @@ This documents what you need to build an
into a dir with no spaces, `C:\bin\openssl`. Add its bin dir to your path.
(this will also need the Visual C++ 2008 Redistributables (DLL libraries)
installed.
-*
## Virtual Box
@@ -61,9 +60,7 @@ is "private", then edit the firewall to
1. Everything Hadoop's `BUILDING.TXT` needs. This includes the cygwin tools, protoc,
python and more. If you cannot build Hadoop from the `Windows SDK Command Prompt`
then don't proceed.
-1. PFE32 is still one of the best plain text editors for development around,
-even just for viewing logs & test results.
-1. ...JEdit is fairly nimble too, though not as lightweight.
+1. JEdit is a good text editor for developing and editing simple files.
1. Chrome can be used a browser instead of IE if you want âbut you should still
view all web pages in IE to verify they look OK.
1. [Sysinternals](http://technet.microsoft.com/en-us/sysinternals/bb545021.aspx).
Modified: incubator/slider/site/trunk/content/docs/client-configuration.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/client-configuration.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/client-configuration.md (original)
+++ incubator/slider/site/trunk/content/docs/client-configuration.md Tue Nov 25 19:03:55 2014
@@ -43,12 +43,14 @@ identity, the *principal* of both the HD
also need some JVM options set in order for Java's Kerberos module to
correctly identify itself to these services.
-It cannot rely on local `HADOOP_PREFIX/conf/hadoop-site.xml` and
-`$YARN_PREFIX/conf/yarn-site.xml` files -because it is designed to
+It cannot rely on local `"$HADOOP_CONF_DIR/hadoop-site.xml` and
+`$HADOOP_CONF_DIR/yarn-site.xml` files -because it is designed to
work on client machines that may not have Hadoop and YARN installed.
-Instead all client-side (non-JVM) options can be predefined in the
-configuration file `slider-client.xml`.
+If these files are present, and the `HADOOP_CONF_DIR` environment variable is
+set to the directory that contains them âslider will pick up the settings in
+these files. If not, all settings must be placed in the file `slider-client.xml`
+
## Setting Slider JVM options
@@ -175,10 +177,6 @@ as the source of the `log4j.properties`
As well as standard YARN and Hadoop configuration options, Slider supports
a limited number of slider-specific configuration parameters.
- <property>
- <name>slider.zookeeper.quorum</name>
- <value>localhost:2181,zookeeper2:4545</value>
- </property>
<property>
<name>slider.yarn.queue</name>
@@ -186,11 +184,6 @@ a limited number of slider-specific conf
</property>
<property>
- <name>slider.yarn.queue</name>
- <value>default</value>
- </property>
-
- <property>
<name>slider.yarn.queue.priority</name>
<value>1</value>
</property>
@@ -246,8 +239,7 @@ priority
Default value: `1`.
- bin/slider thaw cl1 -D slider.yarn.queue.priority=5
-
+ bin/slider start cl1 -D slider.yarn.queue.priority=5
#### `slider.cluster.directory.permissions`
@@ -278,7 +270,14 @@ Slider uses the (evolving) YARN service
instances, to locate them, and as a means to publish the configuration information
needed to connect to the slider-deployed applications.
-### `slider.zookeeper.quorum` - the zookeeper quorum.
+These values should be set in the `hadoop-site.xml` or `yarn-site.xml` files
+for a cluster, for use by all applications. In particular, if the YARN
+Resource Manager is set to automatically create the paths in the registry for
+users âa required feature in secure clustersâ then it must share the same settings.
+
+Consult the YARN registry documentation for more information.
+
+### `hadoop.registry.zk.quorum` - the zookeeper quorum.
This defines the zookeeper quorum for this YARN cluster.
@@ -289,11 +288,25 @@ It is also used as the default zookeeper
uses zookeeper in its configuration -the value set when the application is
defined will be copied into the instance definition file.
-### `"slider.registry.path"` - the zookeeper path for the service registry
+Example:
+
+ <property>
+ <name>hadoop.registry.zk.quorum</name>
+ <value>zk1:2181,zookeeper2:4545</value>
+ </property>
+
+
+### `hadoop.registry.zk.root` - the zookeeper path for the service registry
This declares the the zookeeper path for the service registry.
+ <property>
+ <name>hadoop.registry.zk.root</name>
+ <value>/registry</value>
+ </property>
+
+
## Debugging configuration issues
If the slider packages are set to log at debug level in the log4j configuration
@@ -302,11 +315,17 @@ file, details on properties will be part
## How client options are passed down to created clusters.
-Apart from the filesystem bindings, Client configuration options are
+Most cient configuration options are
not passed down to the XML site specification of the created cluster.
-The sole options passed down are the HDFS bindings: `fs.defaultFS`,
+
+Exceptions:
+
+1. the HDFS bindings: `fs.defaultFS`,
which is passed down both as that property and as `fs.default.name`,
-and, in a secure cluster, the security flag (`slider.security.enabled`)
+and,
+
+2. in a secure cluster, the security flag (`slider.security.enabled`)
and the HDFS Kerberos principal.
+3. The yarn registry options.
Modified: incubator/slider/site/trunk/content/docs/configuration/specification.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/configuration/specification.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/configuration/specification.md (original)
+++ incubator/slider/site/trunk/content/docs/configuration/specification.md Tue Nov 25 19:03:55 2014
@@ -55,7 +55,7 @@ aspects of the Slider application master
This information defines the *desired state* of a cluster.
Users may edit these sections, either via the CLI, or by directly editing the `cluster.json` file of
- a frozen cluster.
+ a stopped cluster.
1. Status information provided by a running cluster. These include:
information about the cluster, statistics, information about reach role in
Modified: incubator/slider/site/trunk/content/docs/examples.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/examples.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/examples.md (original)
+++ incubator/slider/site/trunk/content/docs/examples.md Tue Nov 25 19:03:55 2014
@@ -125,12 +125,12 @@ This is for demos only, otherwise you lo
- # freeze the cluster (*freeze has been renamed to stop in develop branch*)
- slider freeze cl1 \
+ # stop a cluster
+ slider stop cl1 \
--manager master:8032 --filesystem hdfs://master:9090
- # thaw a cluster (*thaw has been renamed to start in develop branch*)
- slider thaw cl1 \
+ # start a cluster
+ slider start cl1 \
--manager master:8032 --filesystem hdfs://master:9090
# destroy the cluster
Modified: incubator/slider/site/trunk/content/docs/getting_started.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/getting_started.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/getting_started.md (original)
+++ incubator/slider/site/trunk/content/docs/getting_started.md Tue Nov 25 19:03:55 2014
@@ -113,21 +113,39 @@ Expand the tar file: `tar -xvf slider-0
Browse to the Slider directory: `cd slider-0.60.0/conf`
-Edit slider-env.sh and specify correct values.
+Edit `slider-env.sh` and specify correct values.
export JAVA_HOME=/usr/jdk64/jdk1.7.0_67
export HADOOP_CONF_DIR=/etc/hadoop/conf
+
+(You only need to set `JAVA_HOME` if it is not already set)
-If you are on a node that does not have the hadoop conf folder then you can add the relevant configurations into slider-client.xml.
+If you are on a node that does not have the hadoop conf folder then you can add the relevant configurations into `slider-client.xml`.
+
+You can also simply configure `slider-client.xml` with the path to the Hadoop configuration
+directory.
+
+This can be absolute
+
+ <property>
+ <name>HADOOP_CONF_DIR</name>
+ <value>/etc/hadoop/conf</value>
+ </property>
+
+or it can be relative to the property `SLIDER_CONF_DIR`, which is set to the
+directory containing the `slider-client.xml` file.
+
+ <property>
+ <name>HADOOP_CONF_DIR</name>
+ <value>${SLIDER_CONF_DIR}/../hadoop-conf</value>
+ </property>
+
+If the hadoop configuration dir is configured this way, the
For example, if you are targeting a non-secure cluster with no HA for NameNode or ResourceManager, modify Slider configuration file `${slider-install-dir}/slider-0.60.0/conf/slider-client.xml` to add the following properties:
<property>
- <name>yarn.application.classpath</name>
- <value>/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*</value>
- </property>
- <property>
- <name>slider.zookeeper.quorum</name>
+ <name>hadoop.registry.zk.quorum</name>
<value>yourZooKeeperHost:port</value>
</property>
<property>
@@ -143,19 +161,13 @@ For example, if you are targeting a non-
<value>hdfs://yourNameNodeHost:8020</value>
</property>
-If the target cluster is on Windows then make sure that the `yarn.application.classpath` is set accordingly.
-
- <property>
- <name>yarn.application.classpath</name>
- <value>
- %HADOOP_CONF_DIR%,%HADOOP_COMMON_HOME%/share/hadoop/common/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/*,...
- </value>
- </property>
Execute:
${slider-install-dir}/slider-0.60.0/bin/slider version
- OR
+
+OR
+
python %slider-install-dir%/slider-0.60.0/bin/slider.py version
Ensure there are no errors and you can see "Compiled against Hadoop 2.6.0". **This ensures that Slider is installed correctly.**
@@ -305,7 +317,7 @@ An application may publish some quick li
### <a name="manage"></a>Manage the Application Lifecycle
-Once started, applications can be frozen/stopped, thawed/restarted, and destroyed/removed as follows:
+Once started, applications can be stopped, (re)started, and destroyed as follows:
#### Stopped:
Modified: incubator/slider/site/trunk/content/docs/manpage.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/manpage.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/manpage.md (original)
+++ incubator/slider/site/trunk/content/docs/manpage.md Tue Nov 25 19:03:55 2014
@@ -64,7 +64,8 @@ and existing but not running "stopped" a
1. the count of each component must initially be specified when an application instance is created.
1. Users can flex an application instance: adding or removing components dynamically.
-If the application instance is live, the changes will have immediate effect. If not, the changes will be picked up when the instance is next thawed.
+If the application instance is live, the changes will have immediate effect.
+If not, the changes will be picked up when the instance is next started.
<!--- ======================================================================= -->
@@ -116,7 +117,7 @@ COMMANDS
Build an instance of the given name, with the specific options.
-It is not started; this can be done later with a `thaw` command.
+It is not started; this can be done later with a `start` command.
### `slider create <name>`
@@ -593,7 +594,6 @@ This simplifies path creation, testing a
### `start <name> [--wait milliseconds] [--out <filename>]`
-(**start** used to be **thaw** in develop branch)
Start a stopped application instance, recreating it from its previously
saved state.
@@ -625,7 +625,6 @@ Examples:
### `stop <name> [--force] [--wait time] [--message text]`
-(**stop** used to be **freeze** in earlier releases)
stop the application instance. The running application is stopped. Its settings are retained in HDFS.
Modified: incubator/slider/site/trunk/content/docs/security.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/security.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/security.md (original)
+++ incubator/slider/site/trunk/content/docs/security.md Tue Nov 25 19:03:55 2014
@@ -210,7 +210,7 @@ wishes to talk with the SliderAM -a toke
the caller authenticates itself as the user that has access rights
to the cluster
-To allow the client to freeze a Slider application instance while they are unable to acquire
+To allow the client to stop a Slider application instance while they are unable to acquire
a token to authenticate with the AM, use the `--force` option.
### How to enable a secure Slider client
Modified: incubator/slider/site/trunk/content/docs/slider_specs/canonical_scenarios.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/canonical_scenarios.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/canonical_scenarios.md (original)
+++ incubator/slider/site/trunk/content/docs/slider_specs/canonical_scenarios.md Tue Nov 25 19:03:55 2014
@@ -143,7 +143,7 @@ The following administrative functions a
* Install the application - the installation command will take care of the population of the application resources into the pre-determined application resource directory in HDFS. The required configuration and binary directories will also be created.
-* start/thaw the application - Slider/Yarn runtime negotiates and instantiates the number of component containers designated by the cluster description and the components are started.
+* start the application - Slider/Yarn runtime negotiates and instantiates the number of component containers designated by the cluster description and the components are started.
* stop/freeze the application - similar to stopping, applicaiton (or a subset of their components) can be stopped.
Modified: incubator/slider/site/trunk/content/docs/slider_specs/specifying_exports.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/docs/slider_specs/specifying_exports.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/docs/slider_specs/specifying_exports.md (original)
+++ incubator/slider/site/trunk/content/docs/slider_specs/specifying_exports.md Tue Nov 25 19:03:55 2014
@@ -36,7 +36,7 @@ By default all configurations are export
By default an arbitrary master is chosen as the master responsible for exporting the config. *What this means is that when this master is STARTED the applied config known at that time is exported*. Otherwise, you can specify which master component type should export configuration by specifying `<publishConfig>true</publishConfig>` under `<component>`.
## Exporting formatted data at application instance level
-At application level one or more exportGroups can be defined where each group can contain one or more exported name-value pairs. The sample export group shown below exports two data items "app.monitor" and "nimbus.host_port" for a Storm application instance.
+At application level one or more exportGroups can be defined where each group can contain one or more exported name-value pairs. The sample export group shown below exports two data items `app.monitor` and `nimbus.host_port` for a Storm application instance.
<exportGroups>
<exportGroup>
@@ -159,4 +159,6 @@ Sample:
]
},
"empty": false
- }
\ No newline at end of file
+ }
+
+
\ No newline at end of file
Modified: incubator/slider/site/trunk/content/downloads/index.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/downloads/index.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/downloads/index.md (original)
+++ incubator/slider/site/trunk/content/downloads/index.md Tue Nov 25 19:03:55 2014
@@ -21,36 +21,14 @@ Be sure to verify your downloads by thes
## Current Release
-### 0.60.0-incubating
-The most recent Apache Slider™ release is version 0.60.0-incubating. See the [release notes][REL_NOTES_60].
+The latest release of Apache Slider™ can be
+[downloaded from the ASF](http://www.apache.org/dyn/closer.cgi/incubator/slider/)
-<table class="table">
-<tr>
-<th>Release Artifact</th><th>Signature & Hashes</th></tr>
-<tr>
-<td><a href="https://www.apache.org/dyn/closer.cgi/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.tar.gz">apache-slider-0.60.0-incubating-source-release.tar.gz</a></td>
-<td>
-<a href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.tar.gz.asc">ASC</a>
-<a href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.tar.gz.md5">MD5</a>
-<a href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.tar.gz.sha">SHA</a>
-</td>
-</tr>
-<tr>
-<td><a href="https://www.apache.org/dyn/closer.cgi/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.zip">apache-slider-0.60.0-incubating-source-release.zip</a></td>
-<td>
-<a href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.zip.asc">ASC</a>
-<a href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.zip.md5">MD5</a>
-<a href="http://www.apache.org/dist/incubator/slider/0.60.0-incubating/src/apache-slider-0.60.0-incubating-source-release.zip.sha">SHA</a>
-</td>
-</tr>
-</table>
-Older releases may be found in the [archives][ARCHIVES].
+Older releases may be found in the [archives](https://archive.apache.org/dist/incubator/slider)
[VERIFY_PROCEDURES]: https://www.apache.org/info/verification.html "Verify"
[GPG_KEYS]: https://www.apache.org/dist/incubator/slider/KEYS "KEYS"
[ARCHIVES]: https://archive.apache.org/dist/incubator/slider
-
-[REL_NOTES_60]: /release_notes/release-0.60.0.html "0.60.0-incubating Release Notes"
Modified: incubator/slider/site/trunk/content/index.md
URL: http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/index.md?rev=1641675&r1=1641674&r2=1641675&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/index.md (original)
+++ incubator/slider/site/trunk/content/index.md Tue Nov 25 19:03:55 2014
@@ -24,9 +24,9 @@ Apache Slider is a YARN application to d
monitor them and make them larger or smaller as desired -even while
the application is running.
-Applications can be stopped, "frozen" and restarted, "thawed" later; the distribution
-of the deployed application across the YARN cluster is persisted -enabling
-a best-effort placement close to the previous locations on a cluster thaw.
+Applications can be stopped then started; the distribution
+of the deployed application across the YARN cluster is persisted âenabling
+a best-effort placement close to the previous locations.
Applications which remember the previous placement of data (such as HBase)
can exhibit fast start-up times from this feature.
@@ -55,8 +55,8 @@ The Slider tool is a Java command line a
The tool persists the information as JSON documents in HDFS.
Once the cluster has been started, the cluster can be made to grow or shrink
-using the Slider commands. The cluster can also be stopped, *frozen*
-and later resumed, *thawed*.
+using the Slider commands. The cluster can also be stopped
+and later restarted.
Slider implements all its functionality through YARN APIs and the existing
application shell scripts. The goal of the application was to have minimal