You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by st...@apache.org on 2014/06/30 17:37:12 UTC
[23/50] [abbrv] SLIDER-121 removed site documentation from git source
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/core.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/core.md b/src/site/markdown/configuration/core.md
deleted file mode 100644
index 319bf77..0000000
--- a/src/site/markdown/configuration/core.md
+++ /dev/null
@@ -1,407 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider Core Configuration Specification
-
-
-## Terminology
-
-
-*Application* A single application, such as an HBase cluster. An application
-is distribed across the YARN cluster.
-
-*Component* A single executable part of the larger application. An application
-may have multiple components, and multiple instances of each component.
-
-*YARN* Yet Another Resource Negotiator
-
-*YARN Resource Requirements* The requirements for a YARN resource request.
-Currently this consists of RAM and CPU requirements.
-
-*YARN Container*. An allocation portion of a servers resources granted
-to satisfy the requested YARN resource requirements. A process can be deployed
-to a container.
-
-
-*`resources.json`*: A file that describes the
-size of the application in terms of its component requirements: how many,
-and what their resource requirements are.
-
-*`application.json`*: A file that describes the
-size of the application in terms of its component requirements: how many,
-and what their resource requirements are.
-
-## Structure
-
-Configurations are stored in well-formed JSON files.
-1. Text MUST be saved in the UTF-8 format.
-1. Duplicate entries MUST NOT occur in any section.
-1. The ordering of elements is NOT significant.
-
-The JSON specification files all have a similar structure
-
-1. A `schema` string indicating version. Currently this is temporarily set to
-
- "http://example.org/specification/v2.0.0"
-
-
-1. A global section, `/global` containing string properties
-1. A component section, `/components`.
-1. 0 or more sections under `/components` for each component, identified by component name,
- containing string properties.
-1. 0 or 1 section `/metadata` containing arbitrary metadata (such as a description,
-author, or any other information that is not parsed or processed directly).
-
-
-The simplest valid specification file is
-
- {
- "schema": "http://example.org/specification/v2.0.0",
-
- "global": {
- },
- "components": {
- }
- }
-
-
-## Property inheritance model and *resolution*
-
-
-There is a simple global to component inheritance model.
-
-1. Properties defined in `/global` define parameters across the entire application.
-1. Properties defined a section under `/components` define parameters for
-a specific component in the application.
-1. All global properties are propagated to each component.
-1. A component section may override any global property.
-1. The final set of configuration properties for a component is the global
-properties extended and overridden by the global set.
-1. The process of expanding the properties is termed *resolution*; the *resolved*
-specification is the outcome.
-1. There is NO form of explicitly cross-referencing another attribute. This
-MAY be added in future.
-1. There is NO sharing of information from the different `.json` files in a
-an application configuration.
-
-### Example
-
-Here is an example configuration
-
- {
- "schema": "http://example.org/specification/v2.0.0",
-
- "global": {
- "g1": "a",
- "g2": "b"
- },
- "components": {
- "simple": {
- },
- "master": {
- "name": "m",
- "g1": "overridden"
-
- },
- "worker": {
- "name": "w",
- "g1": "overridden-by-worker",
- "timeout": "1000"
- }
- }
- }
-
-The `/global` section defines two properties
-
- g1="a"
- g2="b"
-
-These are the values visible to any part of the application which is
-not itself one of the components.
-
-
-There are three components defined, `simple`, `master` and `worker`.
-
-
-#### component `simple`:
-
- g1="a"
- g2="b"
-
-
-No settings have been defined specifically for the component; the global
-settings are applied.
-
-#### component `master`:
-
- name="m",
- g1="overridden"
- g2="b"
-
-A new attribute, `name`, has been defined with the value `"m"`, and the
-global property `g1` has been overridden with the new value, `"overridden"`.
-The global property `g2` is passed down unchanged.
-
-
-#### component `worker`:
-
- name="w",
- g1="overridden-by-worker"
- g2="b"
- timeout: "1000"
-
-A new attribute, `name`, has been defined with the value `"w"`, and another,
-`timeout`, value "1000".
-
-The global property `g1` has been overridden with the new value, `"overridden-by-worker"`.
-
-The global property `g2` is passed down unchanged.
-
-This example shows some key points about the design
-
-* each component gets its own map of properties, which is independent from
- that of other components.
-* all global properties are either present or overridden by a new value.
- They can not be "undefined"
-* new properties defined in a component are not visible to any other component.
-
-The final *resolved* model is as follows
-
- {
- "schema": "http://example.org/specification/v2.0.0",
-
- "global": {
- "g1": "a",
- "g2": "b"
- },
- "components": {
- "simple": {
- "g1": "a",
- "g2": "b"
- },
- "master": {
- "name": "m",
- "g1": "overridden",
- "g2": "b"
- },
- "worker": {
- "name": "m",
- "g1": "overridden-by-worker",
- "g2": "b",
- "timeout": "1000"
- }
- }
- }
-
-This the specification JSON that would have generate exactly the same result as
-in the example, without any propagation of data from the global section
-to individual components.
-
-Note that a resolved specification can still have the resolution operation applied
-to it -it just does not have any effect.
-
-## Metadata
-
-The metadata section can contain arbitrary string values for use in diagnostics
-and by other applications.
-
-To avoid conflict with other applications, please use a unique name in strings,
-such as java-style package names.
-
-# Resource Requirements: `resources.json`
-
-This file declares the resource requirements for YARN for the components
-of an application.
-
-`instances`: the number of instances of a role desired.
-`yarn.vcores`: number of "virtual" required by a component.
-`yarn.memory`: the number of megabytes required by a component.
-
-
- {
- "schema": "http://example.org/specification/v2.0.0",
-
- "metadata": {
- "description": "example of a resources file"
- },
-
- "global": {
- "yarn.vcores": "1",
- "yarn.memory": "512"
- },
-
- "components": {
- "master": {
- "instances": "1",
- "yarn.memory": "1024"
- },
- "worker": {
- "instances":"5"
- }
- }
- }
-
-The resolved file would be
-
- {
- "schema": "http://example.org/specification/v2.0.0",
-
- "metadata": {
- "description": "example of a resources file"
- },
-
- "global": {
- "yarn.vcores": "1",
- "yarn.memory": "512"
- },
-
- "components": {
- "master": {
- "instances": "1",
- "yarn.vcores": "1",
- "yarn.memory": "1024"
- },
- "worker": {
- "instances":"5",
- "yarn.vcores": "1",
- "yarn.memory": "512"
- }
- }
- }
-
-This declares this deployment of the application to consist of one instance of
-the master component, using 1 vcore and 1024MB of RAM, and five worker components
-each using one vcore and 512 MB of RAM.
-
-
-## Internal information, `internal.json`
-
-This contains internal data related to the deployment -it is not
-intended for manual editing.
-
-There MAY be a component, `diagnostics`. If defined, its content contains
-diagnostic information for support calls, and MUST NOT be interpreted
-during application deployment, (though it may be included in the generation
-of diagnostics reports)
-
-
- {
- "schema": "http://example.org/specification/v2.0.0",
-
- "metadata": {
- "description": "Internal configuration DO NOT EDIT"
- },
- "global": {
- "name": "small_cluster",
- "application": "hdfs://cluster:8020/apps/hbase/v/1.0.0/application.tar"
- },
- "components": {
-
- "diagnostics": {
- "create.hadoop.deployed.info": "(release-2.3.0) @dfe463",
- "create.hadoop.build.info": "2.3.0",
- "create.time.millis": "1393512091276",
- "create.time": "27 Feb 2014 14:41:31 GMT"
- }
- }
- }
-
-
-## Deployment specification: `app_configuration.json`
-
-
-This defines parameters that are to be used when creating the instance of the
-application, and instances of the individual components.
-
- {
- "schema": "http://example.org/specification/v2.0.0",
-
- "global": {
-
- "zookeeper.port": "2181",
- "zookeeper.path": "/yarnapps_small_cluster",
- "zookeeper.hosts": "zoo1,zoo2,zoo3",
- "env.MALLOC_ARENA_MAX": "4",
- "site.hbase.master.startup.retainassign": "true",
- "site.fs.defaultFS": "hdfs://cluster:8020",
- "site.fs.default.name": "hdfs://cluster:8020",
- "site.hbase.master.info.port": "0",
- "site.hbase.regionserver.info.port": "0"
- },
- "components": {
-
- "worker": {
- "jvm.heapsize": "512M"
- },
- "master": {
- "jvm.heapsize": "512M"
- }
- }
- }
-
-The resolved specification defines the values that are passed to the
-different components.
-
- {
- "schema": "http://example.org/specification/v2.0.0",
-
- "global": {
- "zookeeper.port": "2181",
- "zookeeper.path": "/yarnapps_small_cluster",
- "zookeeper.hosts": "zoo1,zoo2,zoo3",
- "env.MALLOC_ARENA_MAX": "4",
- "site.hbase.master.startup.retainassign": "true",
- "site.fs.defaultFS": "hdfs://cluster:8020",
- "site.fs.default.name": "hdfs://cluster:8020",
- "site.hbase.master.info.port": "0",
- "site.hbase.regionserver.info.port": "0"
- },
- "components": {
-
- "worker": {
- "zookeeper.port": "2181",
- "zookeeper.path": "/yarnapps_small_cluster",
- "zookeeper.hosts": "zoo1,zoo2,zoo3",
- "env.MALLOC_ARENA_MAX": "4",
- "site.hbase.master.startup.retainassign": "true",
- "site.fs.defaultFS": "hdfs://cluster:8020",
- "site.fs.default.name": "hdfs://cluster:8020",
- "site.hbase.master.info.port": "0",
- "site.hbase.regionserver.info.port": "0",
- "jvm.heapsize": "512M"
- },
- "master": {
- "zookeeper.port": "2181",
- "zookeeper.path": "/yarnapps_small_cluster",
- "zookeeper.hosts": "zoo1,zoo2,zoo3",
- "env.MALLOC_ARENA_MAX": "4",
- "site.hbase.master.startup.retainassign": "true",
- "site.fs.defaultFS": "hdfs://cluster:8020",
- "site.fs.default.name": "hdfs://cluster:8020",
- "site.hbase.master.info.port": "0",
- "site.hbase.regionserver.info.port": "0",
- "jvm.heapsize": "512M"
- }
- }
- }
-
-The `site.` properties have been passed down to each component, components
-whose templates may generate local site configurations. The override model
-does not prevent any component from overriding global configuration so as
-to create local configurations incompatible with the global state. (i.e.,
-there is no way to declare an attribute as final). It is the responsibility
-of the author of the configuration file (and their tools) to detect such issues.
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/example-app_configuration-resolved.json
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/example-app_configuration-resolved.json b/src/site/markdown/configuration/example-app_configuration-resolved.json
deleted file mode 100644
index 5b90ba9..0000000
--- a/src/site/markdown/configuration/example-app_configuration-resolved.json
+++ /dev/null
@@ -1,42 +0,0 @@
-{
- "schema": "http://example.org/specification/v2.0.0",
-
- "global": {
- "zookeeper.port": "2181",
- "zookeeper.path": "/yarnapps_small_cluster",
- "zookeeper.hosts": "zoo1,zoo2,zoo3",
- "env.MALLOC_ARENA_MAX": "4",
- "site.hbase.master.startup.retainassign": "true",
- "site.fs.defaultFS": "hdfs://cluster:8020",
- "site.fs.default.name": "hdfs://cluster:8020",
- "site.hbase.master.info.port": "0",
- "site.hbase.regionserver.info.port": "0"
- },
- "components": {
-
- "worker": {
- "zookeeper.port": "2181",
- "zookeeper.path": "/yarnapps_small_cluster",
- "zookeeper.hosts": "zoo1,zoo2,zoo3",
- "env.MALLOC_ARENA_MAX": "4",
- "site.hbase.master.startup.retainassign": "true",
- "site.fs.defaultFS": "hdfs://cluster:8020",
- "site.fs.default.name": "hdfs://cluster:8020",
- "site.hbase.master.info.port": "0",
- "site.hbase.regionserver.info.port": "0",
- "jvm.heapsize": "512M"
- },
- "master": {
- "zookeeper.port": "2181",
- "zookeeper.path": "/yarnapps_small_cluster",
- "zookeeper.hosts": "zoo1,zoo2,zoo3",
- "env.MALLOC_ARENA_MAX": "4",
- "site.hbase.master.startup.retainassign": "true",
- "site.fs.defaultFS": "hdfs://cluster:8020",
- "site.fs.default.name": "hdfs://cluster:8020",
- "site.hbase.master.info.port": "0",
- "site.hbase.regionserver.info.port": "0",
- "jvm.heapsize": "512M"
- }
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/example-app_configuration.json
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/example-app_configuration.json b/src/site/markdown/configuration/example-app_configuration.json
deleted file mode 100644
index 489acda..0000000
--- a/src/site/markdown/configuration/example-app_configuration.json
+++ /dev/null
@@ -1,25 +0,0 @@
-{
- "schema": "http://example.org/specification/v2.0.0",
-
- "global": {
-
- "zookeeper.port": "2181",
- "zookeeper.path": "/yarnapps_small_cluster",
- "zookeeper.hosts": "zoo1,zoo2,zoo3",
- "env.MALLOC_ARENA_MAX": "4",
- "site.hbase.master.startup.retainassign": "true",
- "site.fs.defaultFS": "hdfs://cluster:8020",
- "site.fs.default.name": "hdfs://cluster:8020",
- "site.hbase.master.info.port": "0",
- "site.hbase.regionserver.info.port": "0"
- },
- "components": {
-
- "worker": {
- "jvm.heapsize": "512M"
- },
- "master": {
- "jvm.heapsize": "512M"
- }
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/example-empty.json
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/example-empty.json b/src/site/markdown/configuration/example-empty.json
deleted file mode 100644
index 5c05163..0000000
--- a/src/site/markdown/configuration/example-empty.json
+++ /dev/null
@@ -1,8 +0,0 @@
-{
- "schema": "http://example.org/specification/v2.0.0",
-
- "global": {
- },
- "components": {
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/example-internal.json
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/example-internal.json b/src/site/markdown/configuration/example-internal.json
deleted file mode 100644
index 8617d1f..0000000
--- a/src/site/markdown/configuration/example-internal.json
+++ /dev/null
@@ -1,21 +0,0 @@
-{
- "schema": "http://example.org/specification/v2.0.0",
-
- "metadata": {
- "description": "Internal configuration DO NOT EDIT"
- },
- "global": {
- "application.name": "small_cluster",
- "application.type": "hbase",
- "application": "hdfs://cluster:8020/apps/hbase/v/1.0.0/application.tar"
- },
- "components": {
-
- "diagnostics": {
- "create.hadoop.deployed.info": "(release-2.3.0) @dfe463",
- "create.hadoop.build.info": "2.3.0",
- "create.time.millis": "1393512091276",
- "create.time": "27 Feb 2014 14:41:31 GMT"
- }
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/example-overridden-resolved.json
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/example-overridden-resolved.json b/src/site/markdown/configuration/example-overridden-resolved.json
deleted file mode 100644
index 2b810b5..0000000
--- a/src/site/markdown/configuration/example-overridden-resolved.json
+++ /dev/null
@@ -1,25 +0,0 @@
-{
- "schema": "http://example.org/specification/v2.0.0",
-
- "global": {
- "g1": "a",
- "g2": "b"
- },
- "components": {
- "simple": {
- "g1": "a",
- "g2": "b"
- },
- "master": {
- "name": "m",
- "g1": "overridden",
- "g2": "b"
- },
- "worker": {
- "name": "m",
- "g1": "overridden-by-worker",
- "g2": "b",
- "timeout": "1000"
- }
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/example-overridden.json
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/example-overridden.json b/src/site/markdown/configuration/example-overridden.json
deleted file mode 100644
index 9a74143..0000000
--- a/src/site/markdown/configuration/example-overridden.json
+++ /dev/null
@@ -1,23 +0,0 @@
-{
- "schema": "http://example.org/specification/v2.0.0",
-
- "global": {
- "g1": "a",
- "g2": "b"
- },
- "components": {
- "simple": {
- },
- "master": {
- "name": "m",
- "g1": "overridden"
-
- },
- "worker": {
- "name": "m",
- "g1": "overridden-by-worker",
- "timeout": "1000"
-
- }
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/example-resources.json
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/example-resources.json b/src/site/markdown/configuration/example-resources.json
deleted file mode 100644
index 06c3b54..0000000
--- a/src/site/markdown/configuration/example-resources.json
+++ /dev/null
@@ -1,25 +0,0 @@
-{
- "schema": "http://example.org/specification/v2.0.0",
-
- "metadata": {
- "description": "example of a resources file"
- },
-
- "global": {
- "yarn.vcores": "1",
- "yarn.memory": "512"
- },
-
- "components": {
- "master": {
- "instances": "1",
- "yarn.vcores": "1",
- "yarn.memory": "1024"
- },
- "worker": {
- "instances":"5",
- "yarn.vcores": "1",
- "yarn.memory": "512"
- }
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/index-markdown.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/index-markdown.md b/src/site/markdown/configuration/index-markdown.md
deleted file mode 100644
index b5a2186..0000000
--- a/src/site/markdown/configuration/index-markdown.md
+++ /dev/null
@@ -1,31 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider: Specification of an application instance, revision 2.0
-
-The specification of an applicaton, comprises
-1. The persistent description of an application's configuration
-1. The persistent description of the desired topology and YARN resource
-requirements.
-1. The dynamic description of the running application, including information
-on the location of components and aggregated statistics.
-
-
-1. [Redesign](redesign.md)
-1. [Specification](specification.md)
-1. [Example: current](original-hbase.json)
-1. [Example: proposed](proposed-hbase.json)
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/index.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/index.md b/src/site/markdown/configuration/index.md
deleted file mode 100644
index 160f84d..0000000
--- a/src/site/markdown/configuration/index.md
+++ /dev/null
@@ -1,38 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider: Specification of an application instance, revision 2.0
-
-The specification of an applicaton, comprises
-1. The persistent description of an application's configuration
-1. The persistent description of the desired topology and YARN resource
-requirements.
-1. The dynamic description of the running application, including information
-on the location of components and aggregated statistics.
-
-The specifics of this are covered in the [Core Configuration Specification](core.md)
-
-
-## Historical References
-
-1. [Specification](specification.html)
-1. [Redesign](redesign.html)
-
-
-1. [Example: current](original-hbase.json)
-1. [Example: proposed](proposed-hbase.json)
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/original-hbase.json
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/original-hbase.json b/src/site/markdown/configuration/original-hbase.json
deleted file mode 100644
index 257caeb..0000000
--- a/src/site/markdown/configuration/original-hbase.json
+++ /dev/null
@@ -1,139 +0,0 @@
-{
- "version": "1.0",
- "name": "test_cluster_lifecycle",
- "type": "hbase",
- "state": 3,
- "createTime": 1393512091276,
- "updateTime": 1393512117286,
- "originConfigurationPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/snapshot",
- "generatedConfigurationPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/generated",
- "dataPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/database",
- "options": {
- "zookeeper.port": "2181",
- "site.hbase.master.startup.retainassign": "true",
- "slider.cluster.application.image.path": "hdfs://sandbox:8020/hbase.tar.gz",
- "site.fs.defaultFS": "hdfs://sandbox:8020",
- "slider.container.failure.threshold": "5",
- "site.fs.default.name": "hdfs://sandbox:8020",
- "slider.cluster.directory.permissions": "0770",
- "slider.am.monitoring.enabled": "false",
- "zookeeper.path": "/yarnapps_slider_stevel_test_cluster_lifecycle",
- "slider.tmp.dir": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/tmp/am",
- "slider.data.directory.permissions": "0770",
- "zookeeper.hosts": "sandbox",
- "slider.container.failure.shortlife": "60"
- },
- "info": {
- "create.hadoop.deployed.info": "(detached from release-2.3.0) @dfe46336fbc6a044bc124392ec06b85",
- "create.application.build.info": "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
- "create.hadoop.build.info": "2.3.0",
- "create.time.millis": "1393512091276",
- "create.time": "27 Feb 2014 14:41:31 GMT",
- "slider.am.restart.supported": "false",
- "live.time": "27 Feb 2014 14:41:56 GMT",
- "live.time.millis": "1393512116881",
- "status.time": "27 Feb 2014 14:42:08 GMT",
- "status.time.millis": "1393512128726",
- "yarn.vcores": "32",
- "yarn.memory": "2048",
- "status.application.build.info": "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
- "status.hadoop.build.info": "2.3.0",
- "status.hadoop.deployed.info": "bigwheel-m16-2.2.0 @704f1e463ebc4fb89353011407e965"
- },
- "statistics": {
- "worker": {
- "containers.start.started": 0,
- "containers.live": 0,
- "containers.start.failed": 0,
- "containers.active.requests": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.desired": 0,
- "containers.requested": 0
- },
- "slider": {
- "containers.unknown.completed": 0,
- "containers.start.started": 0,
- "containers.live": 1,
- "containers.start.failed": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.surplus": 0
- },
- "master": {
- "containers.start.started": 0,
- "containers.live": 0,
- "containers.start.failed": 0,
- "containers.active.requests": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.desired": 0,
- "containers.requested": 0
- }
- },
- "status": {
- },
- "instances": {
- "slider": [ "container_1393511571284_0002_01_000001" ]
- },
- "roles": {
- "worker": {
- "yarn.memory": "768",
- "env.MALLOC_ARENA_MAX": "4",
- "role.instances": "0",
- "role.requested.instances": "0",
- "role.name": "worker",
- "role.failed.starting.instances": "0",
- "role.actual.instances": "0",
- "jvm.heapsize": "512M",
- "yarn.vcores": "1",
- "role.releasing.instances": "0",
- "role.failed.instances": "0",
- "app.infoport": "0"
- },
- "slider": {
- "yarn.memory": "256",
- "env.MALLOC_ARENA_MAX": "4",
- "role.instances": "1",
- "role.requested.instances": "0",
- "role.name": "slider",
- "role.failed.starting.instances": "0",
- "role.actual.instances": "1",
- "jvm.heapsize": "256M",
- "yarn.vcores": "1",
- "role.releasing.instances": "0",
- "role.failed.instances": "0"
- },
- "master": {
- "yarn.memory": "1024",
- "env.MALLOC_ARENA_MAX": "4",
- "role.instances": "0",
- "role.requested.instances": "0",
- "role.name": "master",
- "role.failed.starting.instances": "0",
- "role.actual.instances": "0",
- "jvm.heapsize": "512M",
- "yarn.vcores": "1",
- "role.releasing.instances": "0",
- "role.failed.instances": "0",
- "app.infoport": "0"
- }
- },
- "clientProperties": {
- "fs.defaultFS": "hdfs://sandbox:8020",
- "hbase.cluster.distributed": "true",
- "hbase.master.info.port": "0",
- "hbase.master.port": "0",
- "hbase.master.startup.retainassign": "true",
- "hbase.regionserver.hlog.tolerable.lowreplication": "1",
- "hbase.regionserver.info.port": "0",
- "hbase.regionserver.port": "0",
- "hbase.rootdir": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/database",
- "hbase.tmp.dir": "./hbase-tmp",
- "hbase.zookeeper.property.clientPort": "2181",
- "hbase.zookeeper.quorum": "sandbox",
- "slider.template.origin": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/snapshot/hbase-site.xml",
- "slider.unused.option": "1",
- "zookeeper.znode.parent": "/yarnapps_slider_stevel_test_cluster_lifecycle"
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/proposed-hbase.json
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/proposed-hbase.json b/src/site/markdown/configuration/proposed-hbase.json
deleted file mode 100644
index c4f637f..0000000
--- a/src/site/markdown/configuration/proposed-hbase.json
+++ /dev/null
@@ -1,273 +0,0 @@
-{
- "version": "2.0.0",
- "name": "test_cluster_lifecycle",
- "valid`": true,
-
- "slider-internal":{
- "type": "hbase",
- "createTime": 1393512091276,
- "updateTime": 1393512117286,
- "originConfigurationPath": "hdfs://sandbox:8020/user/slider/.slider/cluster/test_cluster_lifecycle/snapshot",
- "generatedConfigurationPath": "hdfs://sandbox:8020/user/slider/.slider/cluster/test_cluster_lifecycle/generated",
- "dataPath": "hdfs://sandbox:8020/user/slider/.slider/cluster/test_cluster_lifecycle/database",
- "slider.tmp.dir": "hdfs://sandbox:8020/user/slider/.slider/cluster/test_cluster_lifecycle/tmp/am",
- "slider.cluster.directory.permissions": "0770",
- "slider.data.directory.permissions": "0770"
- },
-
- "options": {
- "slider.am.monitoring.enabled": "false",
- "slider.cluster.application.image.path": "hdfs://sandbox:8020/hbase.tar.gz",
- "slider.container.failure.threshold": "5",
- "slider.container.failure.shortlife": "60",
- "zookeeper.port": "2181",
- "zookeeper.path": "/yarnapps_slider_slider_test_cluster_lifecycle",
- "zookeeper.hosts": "sandbox",
- "site.hbase.master.startup.retainassign": "true",
- "site.fs.defaultFS": "hdfs://sandbox:8020",
- "site.fs.default.name": "hdfs://sandbox:8020",
- "env.MALLOC_ARENA_MAX": "4",
- "site.hbase.master.info.port": "0",
- "site.hbase.regionserver.info.port": "0"
- },
-
- "diagnostics": {
- "create.hadoop.deployed.info": "(detached from release-2.3.0) @dfe46336fbc6a044bc124392ec06b85",
- "create.application.build.info": "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by slider",
- "create.hadoop.build.info": "2.3.0",
- "create.time.millis": "1393512091276",
- "create.time": "27 Feb 2014 14:41:31 GMT"
- },
-
- "info": {
- "slider.am.restart.supported": "false",
- "live.time": "27 Feb 2014 14:41:56 GMT",
- "live.time.millis": "1393512116881",
- "status.time": "27 Feb 2014 14:42:08 GMT",
- "status.time.millis": "1393512128726",
- "yarn.vcores": "32",
- "yarn.memory": "2048",
- "status.application.build.info": "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by slider",
- "status.hadoop.build.info": "2.3.0",
- "status.hadoop.deployed.info": "bigwheel-m16-2.2.0 @704f1e463ebc4fb89353011407e965"
- },
-
- "statistics": {
-
- "cluster": {
- "containers.unknown.completed": 0,
- "containers.start.completed": 3,
- "containers.live": 1,
- "containers.start.failed": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.surplus": 0
-
- },
- "roles": {
- "worker": {
- "containers.start.completed": 0,
- "containers.live": 2,
- "containers.start.failed": 0,
- "containers.active.requests": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.desired": 2,
- "containers.requested": 0
- },
- "master": {
- "containers.start.completed": 0,
- "containers.live": 1,
- "containers.start.failed": 0,
- "containers.active.requests": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.desired": 1,
- "containers.requested": 0
- }
- }
- },
-
- "instances": {
- "slider": [ "container_1393511571284_0002_01_000001" ],
- "master": [ "container_1393511571284_0002_01_000003" ],
- "worker": [
- "container_1393511571284_0002_01_000002",
- "container_1393511571284_0002_01_000004"
- ]
- },
-
- "roles": {
- "worker": {
- "yarn.memory": "768",
- "role.instances": "0",
- "role.name": "worker",
- "jvm.heapsize": "512M",
- "yarn.vcores": "1"
- },
- "slider": {
- "yarn.memory": "256",
- "role.instances": "1",
- "role.name": "slider",
- "jvm.heapsize": "256M",
- "yarn.vcores": "1"
- },
- "master": {
- "yarn.memory": "1024",
- "role.instances": "0",
- "role.name": "master",
- "jvm.heapsize": "512M",
- "yarn.vcores": "1"
- }
- },
-
-
- "clientProperties": {
- "fs.defaultFS": "hdfs://sandbox:8020",
- "hbase.cluster.distributed": "true",
- "hbase.master.info.port": "0",
- "hbase.master.port": "0",
- "hbase.master.startup.retainassign": "true",
- "hbase.regionserver.hlog.tolerable.lowreplication": "1",
- "hbase.regionserver.info.port": "0",
- "hbase.regionserver.port": "0",
- "hbase.rootdir": "hdfs://sandbox:8020/user/slider/.slider/cluster/test_cluster_lifecycle/database",
- "hbase.tmp.dir": "./hbase-tmp",
- "hbase.zookeeper.property.clientPort": "2181",
- "hbase.zookeeper.quorum": "sandbox",
- "zookeeper.znode.parent": "/yarnapps_slider_slider_test_cluster_lifecycle"
- },
-
-
- "clientfiles": {
- "hbase-site.xml": "site information for HBase",
- "log4.properties": "log4.property file"
- },
-
- "provider":{
- "load":0.4,
- "urls": {
- "master": ["http://node4:28209"],
- "worker": ["http://node4:28717", "http://node6:31268"]
- }
- },
-
- "status": {
- "live": {
- "worker": {
- "container_1394032374441_0001_01_000003": {
- "name": "container_1394032374441_0001_01_000003",
- "role": "worker",
- "roleId": 1,
- "createTime": 1394032384451,
- "startTime": 1394032384503,
- "released": false,
- "host": "192.168.1.88",
- "state": 3,
- "exitCode": 0,
- "command": "hbase-0.98.0/bin/hbase --config $PROPAGATED_CONFDIR regionserver start 1><LOG_DIR>/region-server.txt 2>&1 ; ",
- "diagnostics": "",
- "environment": [
- "HADOOP_USER_NAME=\"slider\"",
- "HBASE_LOG_DIR=\"/tmp/slider-slider\"",
- "HBASE_HEAPSIZE=\"256\"",
- "MALLOC_ARENA_MAX=\"4\"",
- "PROPAGATED_CONFDIR=\"$PWD/propagatedconf\""
- ]
- },
- "container_1394032374441_0001_01_000002": {
- "name": "container_1394032374441_0001_01_000002",
- "role": "worker",
- "roleId": 1,
- "createTime": 1394032384451,
- "startTime": 1394032384552,
- "released": false,
- "host": "192.168.1.86",
- "state": 3,
- "exitCode": 0,
- "command": "hbase-0.98.0/bin/hbase --config $PROPAGATED_CONFDIR regionserver start 1><LOG_DIR>/region-server.txt 2>&1 ; ",
- "diagnostics": "",
- "environment": [
- "HADOOP_USER_NAME=\"slider\"",
- "HBASE_LOG_DIR=\"/tmp/slider-slider\"",
- "HBASE_HEAPSIZE=\"256\"",
- "MALLOC_ARENA_MAX=\"4\"",
- "PROPAGATED_CONFDIR=\"$PWD/propagatedconf\""
- ]
- }
- },
- "slider": {
- "container_1394032374441_0001_01_000001": {
- "name": "container_1394032374441_0001_01_000001",
- "role": "slider",
- "roleId": 0,
- "createTime": 0,
- "startTime": 0,
- "released": false,
- "host": "slider-8.local",
- "state": 3,
- "exitCode": 0,
- "command": "",
- "diagnostics": ""
- }
- },
- "master": {
- "container_1394032374441_0001_01_000004": {
- "name": "container_1394032374441_0001_01_000004",
- "role": "master",
- "roleId": 2,
- "createTime": 1394032384451,
- "startTime": 1394032384573,
- "released": false,
- "host": "192.168.1.86",
- "state": 3,
- "exitCode": 0,
- "command": "hbase-0.98.0/bin/hbase --config $PROPAGATED_CONFDIR master start 1><LOG_DIR>/master.txt 2>&1 ; ",
- "diagnostics": "",
- "environment": [
- "HADOOP_USER_NAME=\"slider\"",
- "HBASE_LOG_DIR=\"/tmp/slider-slider\"",
- "HBASE_HEAPSIZE=\"256\"",
- "MALLOC_ARENA_MAX=\"4\"",
- "PROPAGATED_CONFDIR=\"$PWD/propagatedconf\""
- ]
- }
- }
- },
- "failed": {
-
- },
-
- "rolestatus": {
- "worker": {
- "role.instances": "2",
- "role.requested.instances": "0",
- "role.failed.starting.instances": "0",
- "role.actual.instances": "2",
- "role.releasing.instances": "0",
- "role.failed.instances": "1"
- },
- "slider": {
- "role.instances": "1",
- "role.requested.instances": "0",
- "role.name": "slider",
- "role.actual.instances": "1",
- "role.releasing.instances": "0",
- "role.failed.instances": "0"
- },
- "master": {
- "role.instances": "1",
- "role.requested.instances": "1",
- "role.name": "master",
- "role.failed.starting.instances": "0",
- "role.actual.instances": "0",
- "role.releasing.instances": "0",
- "role.failed.instances": "0"
- }
- }
- }
-
-
-
-
-}
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/redesign.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/redesign.md b/src/site/markdown/configuration/redesign.md
deleted file mode 100644
index 0006662..0000000
--- a/src/site/markdown/configuration/redesign.md
+++ /dev/null
@@ -1,478 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider Cluster Specification
-
-### Notation:
-
-In this document, a full path to a value is represented as a path
-`options/zookeeper.port` ; an assigment as `options/zookeeper.port=2181`.
-
-A wildcard indicates all entries matching a path: `options/zookeeper.*`
-or `/roles/*/yarn.memory`
-
-
-## History
-
-The Slider cluster specification was implicitly defined in the file
-`org.apache.slider.api.ClusterDescription`. It had a number of roles
-
-1. Persistent representaton of cluster state
-1. Internal model of desired cluster state within the Application Master.
-1. Dynamic representation of current cluster state when the AM
-was queried, marshalled over the network as JSON.
-1. Description of updated state when reconfiguring a running cluster.
-
-Initially the dynamic status included a complete history of all containers
--this soon highlit some restrictions on the maximum size of a JSON-formatted
-string in Hadoop's "classic" RPC: 32K, after which the string was silently
-truncated. Accordingly, this history was dropped.
-
-Having moved to Protocol Buffers as the IPC wire format, with a web view
-alongside, this history could be reconsidered.
-
-The initial design place most values into the root entry, and relied
-on Jaxon introspection to set and retrieve the values -it was a
-Java-first specification, with no external specificatin or regression tests.
-
-As the number of entries in the root increased, the design switched to storing
-more attributes into specific sections *under* the root path:
-
-* `info`: read-only information about the cluster.
-* `statistics`: Numeric statistics about the cluster
-
-# Sections
-
-## Root
-
-Contains various string and integer values
-
- "version": "1.0",
- "name": "test_cluster_lifecycle",
- "type": "hbase",
- "state": 3,
- "createTime": 1393512091276,
- "updateTime": 1393512117286,
- "originConfigurationPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/snapshot",
- "generatedConfigurationPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/generated",
- "dataPath": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/database",
-
-
-* `version`: version of the JSON file. Not currently used
-to validate version compatibility; at this point in time
-releases may not be able to read existing .json files.
-
-* `name`: cluster name
-* `type`: reference to the provider type -this triggers a Hadoop configuration
-property lookup to find the implementation classes.
-* `state`: an enumeration value of the cluster state.
-
- int STATE_INCOMPLETE = 0;
- int STATE_SUBMITTED = 1;
- int STATE_CREATED = 2;
- int STATE_LIVE = 3;
- int STATE_STOPPED = 4;
- int STATE_DESTROYED = 5;
-
- Only two states are persisted, "incomplete" and "created", though more
- are used internally.
- The `incomplete` state is used during cluster create/build,
- allowing an incomplete JSON file to be written
- -so minimising the window for race conditions on cluster construction.
-
-* `createTime` and `updateTime`: timestamps, informative only.
- The `createTime` value is duplicated in `/info/createTimeMillis`
-* `originConfigurationPath`, `generatedConfigurationPath`, `dataPath` paths
-used internally -if changed the cluster may not start.
-
-*Proposed*:
-1. Move all state bar `name` and cluster state
-into a section `/slider-internal`.
-1. The cluster state is moved from an enum to a simple
- boolean, `valid`, set to true when the cluster JSON
- has been fully constructed.
-
-## `/info`
-
-Read-only list of information about the application. Generally this is
-intended to be used for debugging and testing.
-
-### Persisted values: static information about the file history
-
- "info" : {
- "create.hadoop.deployed.info" : "(detached from release-2.3.0) @dfe46336fbc6a044bc124392ec06b85",
- "create.application.build.info" : "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
- "create.hadoop.build.info" : "2.3.0",
- "create.time.millis" : "1393512091276",
- },
-
-*Proposed*: move persisted info K-V pairs to a section `/diagnostics`.
-
-### Dynamic values:
-
-
- whether the AM supports service restart without killing all the containers hosting
- the role instances:
-
- "slider.am.restart.supported" : "false",
-
- timestamps of the cluster going live, and when the status query was made
-
- "live.time" : "27 Feb 2014 14:41:56 GMT",
- "live.time.millis" : "1393512116881",
- "status.time" : "27 Feb 2014 14:42:08 GMT",
- "status.time.millis" : "1393512128726",
-
- yarn data provided to the AM
-
- "yarn.vcores" : "32",
- "yarn.memory" : "2048",
-
- information about the application and hadoop versions in use. Here
- the application was built using Hadoop 2.3.0, but is running against the version
- of Hadoop built for HDP-2.
-
- "status.application.build.info" : "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
- "status.hadoop.build.info" : "2.3.0",
- "status.hadoop.deployed.info" : "bigwheel-m16-2.2.0 @704f1e463ebc4fb89353011407e965"
-
-
- ## `instances`
-
- Information about the live containers in a cluster
-
- "instances": {
- "slider": [ "container_1393511571284_0002_01_000001" ],
- "master": [ "container_1393511571284_0002_01_000003" ],
- "worker": [
- "container_1393511571284_0002_01_000002",
- "container_1393511571284_0002_01_000004"
- ]
- },
-
-There's no information about location, nor is there any history about containers
-that are no longer part of the cluster (i.e. failed & released containers).
-
-It could be possible to include a list of previous containers,
-though Slider would need to be selective about how many to store
-(or how much detail to retain) on those previous containers.
-
-Perhaps the list could be allowed to grow without limit, but detail
-only preserved on the last 100. If more containers fail than that,
-there is likely to be a problem which the most recent containers
-will also display.
-
-*Proposed*
-
-1. Return to the full serialization of container state -but only for running containers.
-1. Have a list of failed containers, but only include last 8; make it a rolling
-buffer. This avoids a significantly failing role to overload the status document.
-
-
- ## `statistics`
-
- Statistics on each role.
-
- They can be divided into counters that only increase
-
- "containers.start.completed": 0,
- "containers.start.failed": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.requested": 0
-
-and those that vary depending upon the current state
-
- "containers.live": 0,
- "containers.active.requests": 0,
- "containers.desired": 0,
-
-
-* Propose: move these values out of statistics into some other section, as they
-are state, not statistics*
-
-
- "statistics": {
- "worker": {
- "containers.start.completed": 0,
- "containers.live": 2,
- "containers.start.failed": 0,
- "containers.active.requests": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.desired": 2,
- "containers.requested": 0
- },
- "slider": {
- "containers.unknown.completed": 0,
- "containers.start.completed": 3,
- "containers.live": 1,
- "containers.start.failed": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.surplus": 0
- },
- "master": {
- "containers.start.completed": 0,
- "containers.live": 1,
- "containers.start.failed": 0,
- "containers.active.requests": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.desired": 1,
- "containers.requested": 0
- }
- },
-
-The `/statistics/slider` section is unusual in that it provides the aggregate statistics
-of the cluster -this is not obvious. A different name could be used -but
-again, there's a risk of clash with or confusion with a role.
-
-Better to have a specific `/statistics/cluster` element,
-and to move the roles' statistics under `/statistics/roles`:
-
- "statistics": {
- "cluster": {
- "containers.unknown.completed": 0,
- "containers.start.completed": 3,
- "containers.live": 1,
- "containers.start.failed": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.surplus": 0
-
- },
- "roles": {
- "worker": {
- "containers.start.completed": 0,
- "containers.live": 2,
- "containers.start.failed": 0,
- "containers.active.requests": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.desired": 2,
- "containers.requested": 0
- },
- "master": {
- "containers.start.completed": 0,
- "containers.live": 1,
- "containers.start.failed": 0,
- "containers.active.requests": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.desired": 1,
- "containers.requested": 0
- }
- }
- },
-
-This approach allows extra statistics sections to be added (perhaps
-by providers), without any changes to the toplevel section.
-
-## Options
-
-A list of options used by Slider and its providers to build up the AM
-and the configurations of the deployed service components
-
-
- "options": {
- "zookeeper.port": "2181",
- "site.hbase.master.startup.retainassign": "true",
- "slider.cluster.application.image.path": "hdfs://sandbox:8020/hbase.tar.gz",
- "site.fs.defaultFS": "hdfs://sandbox:8020",
- "slider.container.failure.threshold": "5",
- "site.fs.default.name": "hdfs://sandbox:8020",
- "slider.cluster.directory.permissions": "0770",
- "slider.am.monitoring.enabled": "false",
- "zookeeper.path": "/yarnapps_slider_stevel_test_cluster_lifecycle",
- "slider.tmp.dir": "hdfs://sandbox:8020/user/stevel/.slider/cluster/test_cluster_lifecycle/tmp/am",
- "slider.data.directory.permissions": "0770",
- "zookeeper.hosts": "sandbox",
- "slider.container.failure.shortlife": "60"
- },
-
-Some for these options have been created by slider itself ("slider.tmp.dir")
-for internal use -and are cluster specific. If/when the ability to use
-an existing json file as a template for a new cluster is added, having these
-options in the configuration will create problems
-
-
-# Proposed Changes
-
-
-## Move Slider internal state to `/slider-internal`
-
-Move all slider "private" data to an internal section,`/slider-internal`
-including those in the toplevel directory and in `/options`
-
-## Allow `/options` and `roles/*/` options entries to take the value "null".
-
-This would be a definition that the value must be defined before the cluster
-can start. Provider templates could declare this.
-
-## Make client configuration retrieval hierarchical -and maybe move out of the
-status
-
-The current design assumes that it is a -site.xml file being served up. This
-does not work for alternate file formats generated by the Provider.
-
-## Role Options
-
-The `/roles/$ROLENAME/` clauses each provide options for a
-specific role.
-
-This includes
-1. `role.instances`: defines the number of instances of a role to create
-1. `env.` environment variables for launching the container
-1. `yarn.` properties to configure YARN requests.
-1. `jvm.heapsize`: an option supported by some providers to
-fix the heap size of a component.
-1. `app.infoport`: an option supported by some providers (e.g. HBase)
-to fix the port to which a role (master or worker) binds its web UI.
-
-
-
- "worker": {
- "yarn.memory": "768",
- "env.MALLOC_ARENA_MAX": "4",
- "role.instances": "0",
- "role.name": "worker",
- "jvm.heapsize": "512M",
- "yarn.vcores": "1",
- "app.infoport": "0"
- },
-
-In a live cluster, the role information also includes status information
-about the cluster.
-
- "master": {
- "yarn.memory": "1024",
- "env.MALLOC_ARENA_MAX": "4",
- "role.instances": "0",
- "role.requested.instances": "0",
- "role.name": "master",
- "role.failed.starting.instances": "0",
- "role.actual.instances": "0",
- "jvm.heapsize": "512M",
- "yarn.vcores": "1",
- "role.releasing.instances": "0",
- "role.failed.instances": "0",
- "app.infoport": "0"
- }
-
-The role `slider` represents the Slider Application Master itself.
-
-
- "slider": {
- "yarn.memory": "256",
- "env.MALLOC_ARENA_MAX": "4",
- "role.instances": "1",
- "role.name": "slider",
- "jvm.heapsize": "256M",
- "yarn.vcores": "1",
- },
-
-### Proposed:
-1. move all dynamic role status to its own clauses.
-1. use a simple inheritance model from `/options`
-1. don't allow role entries to alter the cluster state.
-
-### Proposed: `/clientProperties` continues return Key-val pairs
-
-The `/clientProperties` section will remain, with key-val pairs of type
-string, the expectation being this is where providers can insert specific
-single attributes for client applications.
-
-These values can be converted to application-specific files on the client,
-in code -as done today in the Slider CLI-, or via template expansion (beyond
-the scope of this document.
-
-
-
-### Proposed: alongside `/clientProperties` comes `/clientfiles`
-
-This section will list all files that an application instance can generate
-for clients, along with with a description.
-
- "/clientfiles/hbase-site.xml": "site information for HBase"
- "/clientfiles/log4.properties": "log4.property file"
-
-A new CLI command would be added to retrieve a client file.
-1. The specific file must be named.
-1. If it is not present, an error must be raised.
-1. If it is present, it is downloaded and output to the console/to a named
-destination file/directory `--outfile <file>` and `--outdir <dir>`
-1. If the `--list` argument is provided, the list of available files is
-returned (e.g.)
-
- hbase-site.xml: site information for HBase
- log4.properties: log4.property file
-
-*No attempt to parse/process the body of the messages will be returned.*
-
-In a REST implementation of the client API, /clientconf would be a path
-to the list of options; each file a path underneath.
-
-Client configuration file retrieval outside the status completely;
-the status just lists the possible values; a separate call returns them.
-
-This will permit binary content to be retrieved, and avoid any marshalling
-problems and inefficiencies.
-
-With this change, there will now be two ways to generate client configuration
-files
-
-* Client-side: as today
-* Server-side: via the provider
-
-Client side is more extensible as it allows for arbitrary clients; server-side
-is restricted to those files which the application provider is capable of
-generating. The advantage of the server-side option is that for those files
-about which the provider is aware of, they will be visible through the
-REST and Web UIs, so trivially retrieved.
-
-### Stop intermixing role specification with role current state
-
-Create a new section, `rolestatus`, which lists the current status
-of the roles: how many are running vs requested, how many are being
-released.
-
-There's some overlap here with the `/statistics` field, so we should
-either merge them or clearly separate the two. Only the `role.failed`
-properties match entries in the statistics -perhaps they should be cut.
-
-#### provider-specific status
-
-Allow providers to publish information to the status, in their
-own section.
-
-There already is support for providers updating the cluster status
-in Slider 12.1 and earlier, but it has flaws
-
-A key one is that it is done sychronously on a `getStatus()` call;
-as providers may perform a live query of their status (example, the HBase
-provider looks up the Web UI ports published by HBase to zookeeper),
-there's overhead, and if the operation blocks (example: when HBase hasn't
-ever been deployed and the zookeeper path is empty), then the status
-call blocks.
-
-*Proposed:*
-
-1. There is a specific `/provider` section
-1. There's no restriction on what JSON is permitted in this section.
-1. Providers may make their own updates to the application state to read and
-write this block -operations that are asynchronous to any status queries.
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/resolved-resources.json
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/resolved-resources.json b/src/site/markdown/configuration/resolved-resources.json
deleted file mode 100644
index 5299897..0000000
--- a/src/site/markdown/configuration/resolved-resources.json
+++ /dev/null
@@ -1,22 +0,0 @@
-{
- "schema": "http://example.org/specification/v2.0.0",
-
- "metadata": {
- "description": "example of a resources file"
- },
-
- "global": {
- "yarn.vcores": "1",
- "yarn.memory": "512"
- },
-
- "components": {
- "master": {
- "instances": "1",
- "yarn.memory": "1024"
- },
- "worker": {
- "instances":"5"
- }
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/configuration/specification.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/configuration/specification.md b/src/site/markdown/configuration/specification.md
deleted file mode 100644
index 2ee5c50..0000000
--- a/src/site/markdown/configuration/specification.md
+++ /dev/null
@@ -1,512 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider Specification of the "Cluster Description"
-
-* This is partially obsolete. Slider still returns the Slider Cluster Description
-as changing it will break most of the unit tests -once these are updated
-this document will be completely obsolete and replaced with a new one.
-
-
-### Notation:
-
-In this document, a full path to a value is represented as a path
-`options/zookeeper.port` ; an assigment as `options/zookeeper.port=2181`.
-
-A wildcard indicates all entries matching a path: `options/zookeeper.*`
-or `/roles/*/yarn.memory`
-
-
-## Core Concepts
-
-The specificaton of an application instance is defined in an application instance
-directory, `${user.home}/.slidera/clusters/${clustername}/cluster.json`)
-
-
-## Sections for specifying and describing cluster state
-
-The cluster desciption is hierarchal, with standardized sections.
-
-Different sections have one of three roles.
-
-1. Storage and specification of internal properties used to define a cluster -properties
-that should not be modified by users -doing so is likely to render the
-cluster undeployable.
-
-1. Storage and specification of the components deployed by Slider.
-These sections define options for the deployed application, the size of
-the deployed application, attributes of the deployed roles, and customizable
-aspects of the Slider application master.
-
- This information defines the *desired state* of a cluster.
-
- Users may edit these sections, either via the CLI, or by directly editing the `cluster.json` file of
- a frozen cluster.
-
-1. Status information provided by a running cluster. These include:
- information about the cluster, statistics, information about reach role in
- the cluster -as well as other aspects of the deployment.
-
- This information describes the *actual state* of a cluster.
-
-Using a common format for both the specification and description of a cluster
-may be confusing, but it is designed to unify the logic needed to parse
-and process cluster descriptions. There is only one JSON file to parse
--merely different sections of relevance at different times.
-
-## Role-by-role subsections
-
-A slider-deployed application consists of the single Slider application master,
-and one or more roles -specific components in the actual application.
-
-The `/roles` section contains a listing for each role,
-declaring the number of instances of each role desired,
-possibly along with some details defining the actual execution of the application.
-
-The `/statistics/roles/` section returns statistics on each role,
-while `/instances` has a per-role entry listing the YARN
-containers hosting instances.
-
-
-## Cluster information for applications
-
-The AM/application provider may generate information for use by client applications.
-
-There are three ways to provide this
-
-1. A section in which simple key-value pairs are provided for interpretation
-by client applications -usually to generate configuration documents
-2. A listing of files that may be provided directly to a client. The API to provide these files is not covered by this document.
-3. A provider-specific section in which arbitrary values and structures may be defined. This allows greater flexibility in the information that a provider can publish -though it does imply custom code to process this data on the client.
-
-
-# Persistent Specification Sections
-
-## "/" : root
-
-The root contains a limited number of key-value pairs,
-
-* `version`: string; required.
-The version of the JSON file, as an `x.y.z` version string.
- 1. Applications MUST verify that they can process a specific version.
- 1. The version number SHOULD be incremented in the minor "z" value
- after enhancements that are considered backwards compatible.
- Incompatible updates MUST be updated with a new "y" value.
- The final, "x" number, is to be reserved for major reworkings
- of the cluster specification itself (this document or its
- successors).
-
-* `name`: string; required. Cluster name;
-* `type`: string; required.
-Reference to the provider type -this triggers a Hadoop configuration
-property lookup to find the implementation classes.
-* `valid`: boolean; required.
-Flag to indicate whether or not a specification is considered valid.
-If false, the rest of the document is in an unknown state.
-
-## `/slider-internal`: internal confiugration
-
-Stores internal configuration options. These parameters
-are not defined in this document.
-
-## `/diagnostics`: diagnostics sections
-
-Persisted list of information about Slider.
-
-Static information about the file history
-
- "diagnostics" : {
- "create.hadoop.deployed.info" :
- "(detached from release-2.3.0) @dfe46336fbc6a044bc124392ec06b85",
- "create.application.build.info" :
- "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
- "create.hadoop.build.info" : "2.3.0",
- "create.time.millis" : "1393512091276",
- },
-
-This information is not intended to provide anything other
-than diagnostics to an application; the values and their meaning
-are not defined. All applications MUST be able to process
-an empty or absent `/diagnostics` section.
-
-## Options: cluster options
-
-A persisted list of options used by Slider and its providers to build up the AM
-and the configurations of the deployed service components
-
-
- "options": {
- "slider.am.monitoring.enabled": "false",
- "slider.cluster.application.image.path": "hdfs://sandbox:8020/hbase.tar.gz",
- "slider.container.failure.threshold": "5",
- "slider.container.failure.shortlife": "60",
- "zookeeper.port": "2181",
- "zookeeper.path": "/yarnapps_slider_stevel_test_cluster_lifecycle",
- "zookeeper.hosts": "sandbox",
- "site.hbase.master.startup.retainassign": "true",
- "site.fs.defaultFS": "hdfs://sandbox:8020",
- "site.fs.default.name": "hdfs://sandbox:8020",
- "env.MALLOC_ARENA_MAX": "4",
- "site.hbase.master.info.port": "0",
- "site.hbase.regionserver.info.port": "0"
- },
-
-Many of the properties are automatically set by Slider when a cluster is constructed.
-They may be edited afterwards.
-
-
-### Standard Option types
-
-All option values MUST be strings.
-
-#### `slider.`
-All options that begin with `slider.` are intended for use by slider and
-providers to configure the Slider application master itself, and the
-application. For example, `slider.container.failure.threshold` defines
-the number of times a container must fail before the role (and hence the cluster)
-is considered to have failed. As another example, the zookeeper bindings
-such as `zookeeper.hosts` are read by the HBase and Ambari providers, and
-used to modify the applications' site configurations with application-specific
-properties.
-
-#### `site.`
-
-These are properties that are expected to be propagated to an application's
- `site` configuration -if such a configuration is created. For HBase, the
- site file is `hbase-site.xml`; for Accumulo it is `accumulo-site.xml`
-
-1. The destination property is taken by removing the prefix `site.`, and
-setting the shortened key with the defined value.
-1. Not all applications have the notion of a site file; These applications MAY
-ignore the settings.
-1. Providers MAY validate site settings to recognise invalid values. This
-aids identifying and diagnosing startup problems.
-
-#### `env.`
-
-These are options to configure environment variables in the roles. When
-a container is started, all `env.` options have the prefix removed, and
-are then set as environment variables in the target context.
-
-1. The Slider AM uses these values to configure itself, after following the
-option/role merge process.
-1. Application providers SHOULD follow the same process.
-
-
-## '/roles': role declarations
-
-The `/roles/$ROLENAME/` clauses each provide options for a
-specific role.
-
-This includes
-1. `role.instances`: defines the number of instances of a role to create
-1. `env.` environment variables for launching the container
-1. `yarn.` properties to configure YARN requests.
-1. `jvm.heapsize`: an option supported by some providers to
-fix the heap size of a component.
-
-
- "worker": {
- "yarn.memory": "768",
- "env.MALLOC_ARENA_MAX": "4",
- "role.instances": "0",
- "role.name": "worker",
- "role.failed.starting.instances": "0",
- "jvm.heapsize": "512M",
- "yarn.vcores": "1",
- },
-
-
-The role `slider` represents the Slider Application Master itself.
-
-
- "slider": {
- "yarn.memory": "256",
- "env.MALLOC_ARENA_MAX": "4",
- "role.instances": "1",
- "role.name": "slider",
- "jvm.heapsize": "256M",
- "yarn.vcores": "1",
- },
-
-Providers may support a fixed number of roles -or they may support a dynamic
-number of roles defined at run-time, potentially from other data sources.
-
-## How `/options` and role options are merged.
-
-The options declared for a specific role are merged with the cluster-wide options
-to define the final options for a role. This is implemented in a simple
-override model: role-specific options can override any site-wide options.
-
-1. The options defined in `/options` are used to create the initial option
-map for each role.
-1. The role's options are then applied to the map -this may overwrite definitions
-from the `/options` section.
-1. There is no way to "undefine" a cluster option, merely overwrite it.
-1. The merged map is then used by the provider to create the component.
-1. The special `slider` role is used in the CLI to define the attributes of the AM.
-
-Options set on a role do not affect any site-wide options: they
-are specific to the invidual role being created.
-
-As such, overwriting a `site.` option may have no effect -or it it may
-change the value of a site configuration document *in that specific role instance*.
-
-### Standard role options
-
-* `role.instances` : number; required.
- The number of instances of that role desired in the application.
-* `yarn.vcores` : number.
- The number of YARN "virtual cores" to request for each role instance.
- The larger the number, the more CPU allocation -and potentially the longer
- time to satisfy the request and so instantiate the node.
- If the value '"-1"` is used -for any role but `slider`-the maximum value
- available to the application is requested.
-* `yarn.memory` : number.
- The number in Megabytes of RAM to request for each role instance.
- The larger the number, the more memory allocation -and potentially the longer
- time to satisfy the request and so instantiate the node.
- If the value '"-1"` is used -for any role but `slider`-the maximum value
- available to the application is requested.
-
-* `env.` environment variables.
-String environment variables to use when setting up the container
-
-### Provider-specific role options
-
-* `jvm.heapsize` -the amount of memory for a provider to allocate for
- a processes JVM. Example "512M". This option MAY be implemented by a provider.
-
-
-
-
-
-# Dynamic Information Sections
-
-These are the parts of the document that provide dynamic run-time
-information about an application. They are provided by the
-Slider Application Master when a request for the cluster status is issued.
-
-## `/info`
-
-Dynamic set of string key-value pairs containing
-information about the running application -as provided by th
-
-The values in this section are not normatively defined.
-
-Here are some standard values
-
-* `slider.am.restart.supported"` whether the AM supports service restart without killing all the containers hosting
- the role instances:
-
- "slider.am.restart.supported" : "false",
-
-* timestamps of the cluster going live, and when the status query was made
-
- "live.time" : "27 Feb 2014 14:41:56 GMT",
- "live.time.millis" : "1393512116881",
- "status.time" : "27 Feb 2014 14:42:08 GMT",
- "status.time.millis" : "1393512128726",
-
-* yarn data provided to the AM
-
- "yarn.vcores" : "32",
- "yarn.memory" : "2048",
-
-* information about the application and hadoop versions in use. Here
- the application was built using Hadoop 2.3.0, but is running against the version
- of Hadoop built for HDP-2.
-
- "status.application.build.info" : "Slider Core-0.13.0-SNAPSHOT Built against commit# 1a94ee4aa1 on Java 1.7.0_45 by stevel",
- "status.hadoop.build.info" : "2.3.0",
- "status.hadoop.deployed.info" : "bigwheel-m16-2.2.0 @704f1e463ebc4fb89353011407e965"
-
-
-As with the `/diagnostics` section, this area is primarily intended
-for debugging.
-
- ## `/instances`: instance list
-
- Information about the live containers in a cluster
-
- "instances": {
- "slider": [ "container_1393511571284_0002_01_000001" ],
- "master": [ "container_1393511571284_0002_01_000003" ],
- "worker": [
- "container_1393511571284_0002_01_000002",
- "container_1393511571284_0002_01_000004"
- ]
- },
-
-
-## `/status`: detailed dynamic state
-
-This provides more detail on the application including live and failed instances
-
-### `/status/live`: live role instances by container
-
- "cluster": {
- "live": {
- "worker": {
- "container_1394032374441_0001_01_000003": {
- "name": "container_1394032374441_0001_01_000003",
- "role": "worker",
- "roleId": 1,
- "createTime": 1394032384451,
- "startTime": 1394032384503,
- "released": false,
- "host": "192.168.1.88",
- "state": 3,
- "exitCode": 0,
- "command": "hbase-0.98.0/bin/hbase --config $PROPAGATED_CONFDIR regionserver start 1><LOG_DIR>/region-server.txt 2>&1 ; ",
- "diagnostics": "",
- "environment": [
- "HADOOP_USER_NAME=\"slider\"",
- "HBASE_LOG_DIR=\"/tmp/slider-slider\"",
- "HBASE_HEAPSIZE=\"256\"",
- "MALLOC_ARENA_MAX=\"4\"",
- "PROPAGATED_CONFDIR=\"$PWD/propagatedconf\""
- ]
- }
- }
- failed : {}
- }
-
-All live instances MUST be described in `/status/live`
-
-Failed clusters MAY be listed in the `/status/failed` section, specifically,
-a limited set of recently failed clusters SHOULD be provided.
-
-Future versions of this document may introduce more sections under `/status`.
-
-### `/status/rolestatus`: role status information
-
-This lists the current status of the roles:
-How many are running vs requested, how many are being
-released.
-
-
- "rolestatus": {
- "worker": {
- "role.instances": "2",
- "role.requested.instances": "0",
- "role.failed.starting.instances": "0",
- "role.actual.instances": "2",
- "role.releasing.instances": "0",
- "role.failed.instances": "1"
- },
- "slider": {
- "role.instances": "1",
- "role.requested.instances": "0",
- "role.name": "slider",
- "role.actual.instances": "1",
- "role.releasing.instances": "0",
- "role.failed.instances": "0"
- },
- "master": {
- "role.instances": "1",
- "role.requested.instances": "1",
- "role.name": "master",
- "role.failed.starting.instances": "0",
- "role.actual.instances": "0",
- "role.releasing.instances": "0",
- "role.failed.instances": "0"
- }
- }
-
-
-### `/status/provider`: provider-specific information
-
-Providers MAY publish information to the `/status/provider` section.
-
-1. There's no restriction on what JSON is permitted in this section.
-1. Providers may make their own updates to the application state to read and
-write this block -operations that are asynchronous to any status queries.
-
-
-
-## `/statistics`: aggregate statistics
-
-Statistics on the cluster and each role in the cluster
-
-Better to have a specific `/statistics/cluster` element,
-and to move the roles' statistics under `/statistics/roles`:
-
- "statistics": {
- "cluster": {
- "containers.unknown.completed": 0,
- "containers.start.completed": 3,
- "containers.live": 1,
- "containers.start.failed": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.surplus": 0
- },
- "roles": {
- "worker": {
- "containers.start.completed": 0,
- "containers.live": 2,
- "containers.start.failed": 0,
- "containers.active.requests": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.desired": 2,
- "containers.requested": 0
- },
- "master": {
- "containers.start.completed": 0,
- "containers.live": 1,
- "containers.start.failed": 0,
- "containers.active.requests": 0,
- "containers.failed": 0,
- "containers.completed": 0,
- "containers.desired": 1,
- "containers.requested": 0
- }
- }
- },
-
-`/statistics/cluster` provides aggregate statistics for the entire cluster.
-
-Under `/statistics/roles` MUST come an entry for each role in the cluster.
-
-All simple values in statistics section are integers.
-
-
-### `/clientProperties`
-
-The `/clientProperties` section contains key-val pairs of type
-string, the expectation being this is where providers can insert specific
-single attributes for client applications.
-
-These values can be converted to application-specific files on the client,
-in code -as done today in the Slider CLI-, or via template expansion (beyond
-the scope of this document.
-
-
-### `/clientfiles`
-
-This section list all files that an application instance MAY generate
-for clients, along with with a description.
-
- "/clientfiles/hbase-site.xml": "site information for HBase"
- "/clientfiles/log4.properties": "log4.property file"
-
-Client configuration file retrieval is by other means; this
-status operation merely lists files that are available;
-
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/debugging.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/debugging.md b/src/site/markdown/debugging.md
deleted file mode 100644
index fdc2d94..0000000
--- a/src/site/markdown/debugging.md
+++ /dev/null
@@ -1,92 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Debugging Apache Slider
-There are a number of options available to you for debugging Slider applications. They include:
-
-* Using Slider logging
-* IDE-based remote debugging of the Application Master
-
-## Using Slider logging
-There are a number of options for viewing the generated log files:
-
-1. Using a web browser
-2. Accessing the host machine
-
-### Using a web browser
-
-The log files are accessible via the Yarn Resource Manager UI. From the main page (e.g. `http://${YARN_RESOURCE_MGR_HOST}:8088`),
-click on the link for the application instance of interest, and then click on the `logs` link.
-This will present you with a page with links to the `slider-err.txt` file and the `slider-out.txt` file.
-The former is the file you should select -it is where the applicaton logs go
-Once the log page is presented, click on the link at the top of the page ("Click here for full log") to view the entire file.
-
-If the file `slider-out.txt` is empty, then examine `slider-err.txt` -an empty
-output log usually means that the java process failed to start -this should be
-logged in the error file.
-
-
-### Accessing the host machine
-
-If access to other log files is required, there is the option of logging in
- to the host machine on which the application component is running
- -provided you have the correct permissions.
-
-The root directory for all YARN associated files is the value of `yarn.nodemanager.log-dirs` in `yarn-site.xml` - e.g. `/hadoop/yarn/log`.
-Below the root directory you will find an application and container sub-directory (e.g. `/application_1398372047522_0009/container_1398372047522_0009_01_000001/`).
-Below the container directory you will find any log files associated with the processes running in the given Yarn container.
-
-Within a container log the following files are useful while debugging the application.
-
-**agent.log**
-
-E.g. `application_1398098639743_0024/container_1398098639743_0024_01_000003/infra/log/agent.log`
-This file contains the logs from the Slider-Agent.
-
-**application component log**
-
-E.g. `./log/application_1398098639743_0024/container_1398098639743_0024_01_000003/app/log/hbase-yarn-regionserver-c6403.ambari.apache.org.log`
-
-The location of the application log is defined by the application. "${AGENT_LOG_ROOT}" is a symbol available to the app developers to use as a root folder for logging.
-
-**agent operations log**
-
-E.g. ./log/application_1398098639743_0024/container_1398098639743_0024_01_000003/app/command-log/
-
-The command logs produced by the slider-agent are available in the `command-log` folder relative to `${AGENT_LOG_ROOT}/app`
-
-Note that the *fish* shell is convenient for debugging, as `cat log/**/slider-out.txt` will find the relevant output file
-irrespective of what the path leading to it is.
-
-## IDE-based remote debugging of the Application Master
-
-For situations in which the logging does not yield enough information to debug an issue,
-the user has the option of specifying JVM command line options for the
-Application Master that enable attaching to the running process with a debugger
-(e.g. the remote debugging facilities in Eclipse or Intellij IDEA).
-In order to specify the JVM options, edit the application configuration file
-(the file specified as the `--template` argument value on the command line for cluster creation)
-and specify the `jvm.opts` property for the `slider-appmaster` component:
-
- `"components": {
- "slider-appmaster": {
- "jvm.heapsize": "256M",
- "jvm.opts": "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
- },
- ...`
-
-You may specify `suspend=y` in the line above if you wish to have the application master process wait for the debugger to attach before beginning its processing.