You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@slider.apache.org by bi...@apache.org on 2014/06/24 21:54:21 UTC
[2/6] SLIDER-121 removed site documentation from git source
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/slider_specs/application_definition.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/application_definition.md b/src/site/markdown/slider_specs/application_definition.md
deleted file mode 100644
index 85846ac..0000000
--- a/src/site/markdown/slider_specs/application_definition.md
+++ /dev/null
@@ -1,182 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider Application Definition
-
-App definition is a declarative definition of a YARN application describing its content. The AppDefinition is used in conjunction with the [AppPackage](application_package.md). Application definition is an XML file and is packaged as metainfo.xml.
-
-## Structure
-
-*Non-mandatory fields are described in **italics**.*
-
-The fields to describe an application is as follows:
-
-* **name**: the name of the application
-
-* **version**: the version of the application. name and version together uniquely identify an application.
-
-* **type**: the type of the application. "YARN-APP" identifies an application definition suitable for YARN.
-
-* **minHadoopVersion**: the minimum version of hadoop on which the app can run
-
-* **components**: the list of component that the application is comprised of
-
-* **osSpecifics**: OS specific package information for the application
-
-* *commandScript*: application wide commands may also be defined. The command is executed on a component instance that is a client
-
-* *dependencies*: application can define a list of dependencies. Dependencies can be on the base services such as HDFS, ZOOKEEPER, YARN which are infrastructure services or GANGLIA, NAGIOS, etc. which are monitoring/alert services. The dependencies are parsed by the management infrastructure to provide the necessary configurations to allow the app to access the services. For example, a HDFS folder could be requested by the app to store its data, a ZOOKEEPER node to co-ordinate among components.
-
-An application contains several component. The fields associated with a component are:
-
-* **name**: name of the component
-
-* **category**: type of the component - MASTER, SLAVE, and CLIENT
-
-* **minInstanceCount**: the minimum number of instances required for this component
-
-* *maxInstanceCount*: maximum number of instances allowed for a component
-
-* **commandScript**: the script that implements the commands.
-
- * **script**: the script location - relative to the AppPackage root
-
- * **scriptType**: type of the script
-
- * **timeout**: default timeout of the script
-
-* *customCommands*: any additional commands available for the component and their implementation
-
-An application definition also includes the package used to install the application. Its typically a tarball or some other form of package that does not require root access to install. The details of what happens during install is captured in the command script.
-
-* **osSpecific**: details on a per OS basis
-
-* **osType**: "any" refers to any OS ~ typical for tarballs
-
-* **packages**: list of packages that needs to be deployed
-
-* **type**: type of package
-
-* **name**: name of the package
-
-* **location**: location of the package (can be a relative folder within the parent AppPackage)
-
-Application can define a an order of activation which dictates if some component activation must follow the successful activation of other components.
-
-* **command**: specifies the component and the command in the form component-command *currently, START is the only valid command*
-
-* **requires**: specifies component and their state that the command depends on, provided in the form component-state *currently, STARTED is the only valid state*
-
-Applications can also advertise a set of properties (typically urls) that can only be bound when the application components are active. One such item can be the jmx endpoint. The properties to be advertised are organized as export groups (exportGroup) and each group can export one or more properties organized as a property bag. These values are visible through the registry service.
-
-* **name**: specifies the name of the export group
-
-Each exportGroup contains one or more exports.
-
-* **name**: the name of the export
-
-* **value**: the template that will be populated by Slider and then exported
-
-
- <metainfo>
- <schemaVersion>2.0</schemaVersion>
- <application>
- <name>HBASE</name>
- <version>0.96.0.2.1.1</version>
- <type>YARN-APP</type>
- <minHadoopVersion>2.1.0</minHadoopVersion>
- <components>
- <component>
- <name>HBASE_MASTER</name>
- <category>MASTER</category>
- <minInstanceCount>1</minInstanceCount>
- <maxInstanceCount>2</maxInstanceCount>
- <commandScript>
- <script>scripts/hbase_master.py</script>
- <scriptType>PYTHON</scriptType>
- <timeout>600</timeout>
- </commandScript>
- <customCommands>
- <customCommand>
- <name>GRACEFUL_STOP</name>
- <commandScript>
- <script>scripts/hbase_master.py</script>
- <scriptType>PYTHON</scriptType>
- <timeout>1800</timeout>
- </commandScript>
- </customCommand>
- </customCommands>
- </component>
-
- <component>
- <name>HBASE_REGIONSERVER</name>
- <category>SLAVE</category>
- <minInstanceCount>1</minInstanceCount>
- ...
- </component>
-
- <component>
- <name>HBASE_CLIENT</name>
- <category>CLIENT</category>
- ...
- </components>
-
- <osSpecifics>
- <osSpecific>
- <osType>any</osType>
- <packages>
- <package>
- <type>tarball</type>
- <name>hbase-0.96.1-tar.gz</name>
- <location>package/files</location>
- </package>
- </packages>
- </osSpecific>
- </osSpecifics>
-
- <commandOrders>
- <commandOrder>
- <command>HBASE_REGIONSERVER-START</command>
- <requires>HBASE_MASTER-STARTED</requires>
- </commandOrder>
- </commandOrders>
-
- <exportGroups>
- <exportGroup>
- <name>QuickLinks</name>
- <exports>
- <export>
- <name>JMX_Endpoint</name>
- <value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/jmx</value>
- </export>
- <export>
- <name>Master_Status</name>
- <value>http://${HBASE_MASTER_HOST}:${site.hbase-site.hbase.master.info.port}/master-status</value>
- </export>
- </exports>
- </exportGroup>
- </exportGroups>
-
- <commandScript>
- <script>scripts/app_health_check.py</script>
- <scriptType>PYTHON</scriptType>
- <timeout>300</timeout>
- </commandScript>
-
- </application>
- </metainfo>
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/slider_specs/application_instance_configuration.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/application_instance_configuration.md b/src/site/markdown/slider_specs/application_instance_configuration.md
deleted file mode 100644
index d37a3db..0000000
--- a/src/site/markdown/slider_specs/application_instance_configuration.md
+++ /dev/null
@@ -1,104 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider App Instance Configuration
-
-App Instance Configuration is the configuration override provided by the application owner when creating an application instance using Slider. This configuration values override the default configuration available in the App Package.
-
-Instance configuration is a JSON formatted doc in the following form:
-
-
- {
- "schema": "http://example.org/specification/v2.0.0",
- "metadata": {
- },
- "global": {
- },
- }
-
-An appConfig.json contains the application configuration. The sample below shows configuration for HBase.
-
-
- {
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- "config_types": "core-site,hdfs-site,hbase-site",
-
- "java_home": "/usr/jdk64/jdk1.7.0_45",
- "package_list": "files/hbase-0.96.1-hadoop2-bin.tar",
-
- "site.global.app_user": "yarn",
- "site.global.app_log_dir": "${AGENT_LOG_ROOT}/app/log",
- "site.global.app_pid_dir": "${AGENT_WORK_ROOT}/app/run",
- "site.global.security_enabled": "false",
-
- "site.hbase-site.hbase.hstore.flush.retries.number": "120",
- "site.hbase-site.hbase.client.keyvalue.maxsize": "10485760",
- "site.hbase-site.hbase.hstore.compactionThreshold": "3",
- "site.hbase-site.hbase.rootdir": "${NN_URI}/apps/hbase/data",
- "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT}/work/app/tmp",
- "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}",
- "site.hbase-site.hbase.regionserver.port": "0",
-
- "site.core-site.fs.defaultFS": "${NN_URI}",
- "site.hdfs-site.dfs.namenode.https-address": "${NN_HOST}:50470",
- "site.hdfs-site.dfs.namenode.http-address": "${NN_HOST}:50070"
- }
- }
-
-appConf.json allows you to pass in arbitrary set of configuration that Slider will forward to the application component instances.
-
-**Variable naming convention**
-In order to understand how the naming convention work, lets look at how the config is passed on to component commands. Slider agent recevies a structured bag of commands as input for all commands, INSTALL, CONFIGURE, START, etc. The command includes a section "configuration" which has config properties arranged into named property bags.
-
-* Variables of the form `site.xx.yy` translates to variables by the name `yy` within the group `xx` and are typically converted to site config files by the name `xx` containing variable `yy`. For example, `"site.hbase-site.hbase.regionserver.port":""` will be sent to the Slider-Agent as `"hbase-site" : { "hbase.regionserver.port": ""}` and app definition scripts can access all variables under `hbase-site` as a single property bag.
-* Similarly, `site.core-site.fs.defaultFS` allows you to pass in the default fs. *This specific variable is automatically made available by Slider but its shown here as an example.*
-* Variables of the form `site.global.zz` are sent in the same manner as other site variables except these variables are not expected to get translated to a site xml file. Usually, variables needed for template or other filter conditions (such as security_enabled = true/false) can be sent in as "global variable".
-
-**slider variables**
-
-* Any config not of the form `site.xx.yy` are consumed by Slider itself. Some of the manadatory configuration are:
- * `agent.conf`: location of the agent config file (typically, "/slider/agent/conf/agent.ini")
- * `application.def`: location of the application definition package (typically, "/slider/hbase_v096.zip")
- * `config_types`: list of config types sent to the containers (e.g. "core-site,hdfs-site,hbase-site")
- * `java_home`: java home path (e.g. "/usr/jdk64/jdk1.7.0_45")
- * `package_list`: location of the package relative to the root where AppPackage is installed (e.g. "files/hbase-0.96.1-hadoop2-bin.tar.gz"
-
-**dynamically allocated ports**
-
-Apps can ask port to be dynamically assigned by Slider or they can leave it as "0". If there is a need for advertising any listening endpoint then the ports can be marked such.
-
-For example, HBase master info port needs to be advertised so that jmx endpoint can be accessed. This is indicated by using a special value, of the form, ${COMPONENT_NAME.ALLOCATED_PORT}. E.g. "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}"
-
-[Application Definition](application_definition.md) describes how to advertise arbitrary set of properties that are dynamically finalized when application is activated.
-
-**configuraing an app for ganglia metrics**
-
-There is no set guideline for doing so. How an application emits metrics and how the metrics are emitted to the right place is completely defined by the application. In the following example, we hso how HBase app is configured to emit metrics to a ganglia server.
-
-Ganglia server lifecycle is not controlled by the app instance. So the app instance only needs to know where to emit the metrics. This is achieved by three global variables
-
-* "site.global.ganglia_server_host": "gangliaserver.my.org"
-* "site.global.ganglia_server_port": "8663"
-* "site.global.ganglia_server_id": "HBaseApplicationCluster3"
-
-All three variable values are user provided. It is also expected that a gmond server is available on host gangliaserver.my.org listening for metrics at port 8663 and is named "HBaseApplicationCluster3". Its the reponsibility of the ganglia server admin to ensure that the server is unique and is only receving metrics from the application instance.
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/slider_specs/application_needs.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/application_needs.md b/src/site/markdown/slider_specs/application_needs.md
deleted file mode 100644
index 10ccf49..0000000
--- a/src/site/markdown/slider_specs/application_needs.md
+++ /dev/null
@@ -1,140 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider's needs of an application
-
-Slider installs and runs applications in a YARN cluster -applications that
-do not need to be written for YARN.
-
-What they do need to be is deployable by Slider, which means installable by YARN,
-configurable by Slider, and, finally, executable by YARN. YARN will kill the
-executed process when destroying a container, so the deployed application
-must expect this to happen and be able to start up from a kill-initiated
-shutdown without any manual recovery process.
-
-They need to locate each other dynamically, both at startup and during execution,
-because the location of processes will be unknown at startup, and may change
-due to server and process failures.
-
-## Must
-
-* Install and run from a tarball -and be run from a user that is not root.
-
-* Be self contained or have all dependencies pre-installed.
-
-* Support dynamic discovery of nodes -such as via ZK.
-
-* Nodes to rebind themselves dynamically -so if nodes are moved, the application
-can continue
-
-* Handle kill as a normal shutdown mechanism.
-
-* Support multiple instances of the application running in the same cluster,
- with processes from different application instances sharing
-the same servers.
-
-* Operate correctly when more than one role instance in the application is
-deployed on the same physical host. (If YARN adds anti-affinity options in
-container requests this will no longer be a requirement)
-
-* Dynamically allocate any RPC or web ports -such as supporting 0 as the number
-of the port to listen on in configuration options.
-
-* Be trusted. YARN does not run code in a sandbox.
-
-* If it talks to HDFS or other parts of Hadoop, be built against/ship with
-libaries compatible with the version of Hadoop running on the cluster.
-
-* Store persistent data in HDFS (directly or indirectly) with the exact storage location
-configurable. Specifically: not to the local filesystem, and not in a hard coded location
-such as `hdfs://app/data`. Slider creates per-instance directories for
-persistent data.
-
-* Be configurable as to where any configuration directory is (or simply relative
-to the tarball). The application must not require it to be in a hard-coded
-location such as `/etc`.
-
-* Not have a fixed location for log output -such as `/var/log/something`
-
-* Run until explicitly terminated. Slider treats an application termination
-(which triggers a container release) as a failure -and reacts to it by restarting
-the container.
-
-
-
-## MUST NOT
-
-* Require human intervention at startup or termination.
-
-## SHOULD
-
-These are the features that we'd like from a service:
-
-* Publish the actual RPC and HTTP ports in a way that can be picked up, such as via ZK
-or an admin API.
-
-* Be configurable via the standard Hadoop mechanisms: text files and XML configuration files.
-If not, custom parsers/configuration generators will be required.
-
-* Support an explicit parameter to define the configuration directory.
-
-* Take late bindings params via -D args or similar
-
-* Be possible to exec without running a complex script, so that process inheritance works everywhere, including (for testing) OS/X
-
-* Provide a way for Slider to get list of nodes in cluster and status. This will let Slider detect failed worker nodes and react to it.
-
-* FUTURE: If a graceful decommissioning is preferred, have an RPC method that a Slider provider can call to invoke this.
-
-* Be location aware from startup. Example: worker nodes to be allocated tables to serve based on which tables are
-stored locally/in-rack, rather than just randomly. This will accelerate startup time.
-
-* Support simple liveness probes (such as an HTTP GET operations).
-
-* Return a well documented set of exit codes, so that failures can be propagated
- and understood.
-
-* Support cluster size flexing: the dynamic addition and removal of nodes.
-
-
-* Support a management platform such as Apache Ambari -so that the operational
-state of a Slider application can be monitored.
-
-## MAY
-
-* Include a single process that will run at a fixed location and whose termination
-can trigger application termination. Such a process will be executed
-in the same container as the Slider AM, and so known before all other containers
-are requested. If a live cluster is unable to handle restart/migration of
-such a process, then the Slider application will be unable to handle
-Slider AM restarts.
-
-* Ideally: report on load/cost of decommissioning.
- E.g amount of data; app load.
-
-
-## MAY NOT
-
-* Be written for YARN.
-
-* Be (pure) Java. If the tarball contains native binaries for the cluster's hardware & OS,
- they should be executable.
-
-* Be dynamically reconfigurable, except for the special requirement of handling
-movement of manager/peer containers in an application-specific manner.
-
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/slider_specs/application_package.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/application_package.md b/src/site/markdown/slider_specs/application_package.md
deleted file mode 100644
index 0469e6c..0000000
--- a/src/site/markdown/slider_specs/application_package.md
+++ /dev/null
@@ -1,147 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider App Package
-
-All application artifacts, app definition, app configuration, scripts are packaged into a structured single package that can be handed off to any YARN application deployment tool including Slider
-
-## Overall Structure
-
-App package is a zip package containing all application artifacts. App package contains the following items:
-
-* **app definition file**
-application structure, content, definition, supported platforms, version, etc.
-
-* **default configurations folder**
-various configurations and configuration files associated with the application
-
-* **cmd_impl folder**
-management operations for the application/component
-
- * **scripts folder**
-various scripts that implement management operations
-
- * **templates folder**
-various templates used by the application
-
- * **files folder**
-other scripts, txt files, tarballs, etc.
-
-
-
-
-The example above shows a semi-expanded view of an application "HBASE-YARN-APP" and the package structure for OOZIE command scripts.
-
-## app definition
-
-App definition is a file named "metainfo.xml". The file contains application definition as described in [Application Definition](application_definition.md).
-
-## default configurations
-
-This folder consists of various config files containing default configuration as described in [App Configuration](application_configuration.md).
-
-## package folder
-
-package includes the "implementation" of all management operations. The folders within are divided into scripts, templates, and files.
-
-### scripts folder
-
-Scripts are the implementation of management operations. There are five default operations and a composite operation. "restart" can be redefined to have a custom implementation.
-
-1. install
-
-2. configure
-
-3. start
-
-4. stop
-
-5. status
-
-6. restart (by default calls stop + start)
-
-The script specified in the metainfo is expected to understand the command. It can choose to call other scripts based on how the application author organizes the code base. For example:
-
-
- class OozieServer(Script):
- def install(self, env):
- self.install_packages(env)
-
- def configure(self, env):
- import params
- env.set_params(params)
- oozie(is_server=True)
-
- def start(self, env):
- import params
- env.set_params(params)
- self.configure(env)
- oozie_service(action='start')
-
- def stop(self, env):
- import params
- env.set_params(params)
- oozie_service(action='stop')
-
- def status(self, env):
- import status_params
- env.set_params(status_params)
- check_process_status(status_params.pid_file)
-
-
-
-The scripts are invoked in the following manner:
-
- python SCRIPT COMMAND JSON_FILE PACKAGE_ROOT STRUCTURED_OUT_FILE
-
-* SCRIPT is the top level script that implements the commands for the component.
-
-* COMMAND is one of the six commands listed above or can be a custom command as defined in Application Definition
-
-* JSON_FILE includes all configuration parameters and the values
-
-* PACKAGE_ROOT is the root folder of the package. From this folder, its possible to access files, scripts, templates, packages (e.g. tarballs), etc. The App author has complete control over the structure of the package
-
-* STRUCTURED_OUT_FILE is the file where the script can output structured data. The management infrastructure is expected to automatically reports back STD_OUT and STD_ERR.
-
-A separate document (link TBD) discusses how the scripts are developed and the structure of the JSON_FILE containing the parameters.
-
-### templates folder
-
-templates are configurable text files that are NOT regular config files. *A library has been developed that can materialize a complete site configuration file from a property bag and therefore are not generated from templates.* Other files such as env sh files, log4j properties file, etc. may be derived from a template. Again, the implementor can choose to create these files from scratch and not use templates. The following operations are allowed during template expansion:
-
-* variable expansion
-
-* if condition
-
-* for loop
-
-* ...
-
-Sample template file for dfs.exclude file to list excluded/decommissioned hosts. hdfs_exclude_files in the property defined in params.py which is populated from config parameters defined in JSON_FILE.
-
- {% if hdfs_exclude_file %}
- {% for host in hdfs_exclude_file %}
- {{host}}
- {% endfor %}
- {% endif %}
-
-
-### files folder
-
-files is a directory to store any other files that are needed for management operations. Sample files stored here are tarballs used to install the application, shell scripts used by various operations.
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/slider_specs/canonical_scenarios.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/canonical_scenarios.md b/src/site/markdown/slider_specs/canonical_scenarios.md
deleted file mode 100644
index a877d97..0000000
--- a/src/site/markdown/slider_specs/canonical_scenarios.md
+++ /dev/null
@@ -1,165 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider Guidelines for Clients and Client Applications
-
-This document will define the canonical scenarios for the deployment and management of Slider hosted applications. It will define the types of applications supported, the sequence of events for deploying the application types, and the management facilities exposed for the deployed applications.
-
-## Deployable Application Types
-
-The server-side components of an application (alternatively referred to as the application components) will be deployed and managed using a fairly uniform mechanism. However, applications can be distinguished by their associated client implementation. Specifically, each different client application type can yield different development, deployment, and management approaches. There are two primary application client types:
-
-1. **New application client** - the application deployed to Slider will interact with a newly coded application client leveraging the Slider supplied client API and management facilities. The client API will make use of distributed management facilities such as Zookeeper to provide the application client with the configuration information required for invoking remote application components. For example, the application client (or an application hosted component, e.g. a web application) will leverage the API to lookup the appropriate host(s) and port(s) for RPC based communication. Alternatively, if annotation libraries or an "app container" environment is provided, the appropriate values will be injected into the client process.
-
-2. **Existing, configurable client** - the application client predates deployment in the Slider environment, but is implemented in a manner that can be integrated with the Slider application client support facilities (APIs etc). This case is probably very similar in nature to the new application client in terms of the mechanisms it will use for component discovery, but is distinguished by the fact that it’s development pre-dates Slider. There are two possible variants of application client in this category:
-
- 1. A client that is static - the client is dependent on existing configuration properties to communicate with master components and the existing code can not been altered (at least in the short term). This type of client would require a support infrastructure that may be relatively complex (e.g. templating of configuration files, proxying of server components).
-
- 2. A client that can be enhanced - a client that can have its code altered can leverage a number of mechanisms (REST APIs, Zookeeper, a provided client discover API) to obtain the information required to invoke the master components of the application.
-
-## Deployment Scenarios
-
-There are two primary deployment mechanisms to examine: application component and client-side (application client) component deployment.
-
-## Application Component Deployment
-
-Applications generally are composed of one or more components. In the deployment steps below, be advised that there may be a need to repeat some of the configuration/definition steps for each component.
-
-The process of deploying applications (specifically, the non-client components of an application) is:
-
-1. Compose an application package that contains:
-
- 1. An application definition that provides the following items:
-
- 1. name and version of application
-
- 2. component type(s) and role(s)
-
- 3. system requirements (RAM, CPUs, disk space etc)
-
- 4. ports required for RPC, UI
-
- 5. software dependencies (HDP deployed services required in the cluster, JDK versions, python versions, etc)
-
- 6. configurable properties including:
-
- 1. application-specific properties. If the properties designate port numbers, the general recommendation would be to set them to zero to allow for the system to assign a dynamically selected port number that subsequently can be published to zookeeper. Other properties designating remote components running on other hosts may need some additional support (floating IPs, discovery APIs, etc). The processing of these properties by the infrastructure requires agreed upon mechanisms for identification (e.g. all properties beginning with the application name are passed through as application properties with the application name prefix stripped)
-
- 2. Slider-specific properties. The following Slider properties are currently available:
-
- 1. yarn.memory
-
- 2. Environment variables specified with a "env." prefix (e.g. env.MALLOC_ARENA_MAX)
-
- 3. role.instances
-
- 4. role.name
-
- 5. yarn.vcores
-
- 2. Other application artifacts (binaries etc)
-
-2. Install the application
-
- 1. The application package will be made available in a designated HDFS location
-
- 2. If there is a managed application client component it will be deployed to selected nodes as defined in the cluster specification.
-
- 3. Slider interacts with yarn (resource manager and node manager(s)) to populate the local resources on the designated nodes.
-
- 4. Some properties important for facilitating remote interaction with the deployed components are advertised via zookeeper (though this may actually take place during component start up as the assignment of ports is a late-binding operation. Alternatively, developers may be encouraged to store these values in a registry rather than as direct-application configuration properties).
-
-## Client Application Deployment
-
-Although application clients are components that are deployed using mechanisms similar to other application components (especially in the managed case), there are a number of features that distinguish them:
-
-1. **configuration** - client applications generally require some information (host names, ports, etc) to ascertain the correct networking values required to communicate with the application's server components. In order to work in a yarn deployment, this configuration may need to be manipulated to allow proper operation (e.g. the configuration files updated with correct values, the configuration properties ascertained dynamically from services such as Zookeeper)
-
-2. **execution context **- it may be necessary to provide an execution environment for application clients that allows for discovery mechanisms (dependency injection, annotation libraries, etc)
-
-For each of these application client types there are two possible deployment modes:
-
-* **Managed** - the application client is deployed via Slider mechanisms. Clients, in this context, differ from the other application components in that they are not running, daemon processes. However, in a managed environment there is the expectation that the appropriate binaries and application elements will be distributed to the designated client hosts, and the configuration on those hosts will be updated to allow for execution of requests to the application’s master/server components. Therefore, client components should be defined in the application specification as elements that the management infrastructure supports (Figure 1).
-
-
-Figure 1 - Managed Application Client and associated Slider Application
-
-* **Unmanaged** - the application client is run as a process outside of Slider/yarn, although it may leverage Slider provided libraries that allow for server component discovery etc (Figure 2). These libraries would primarily be client bindings providing access to the registry leveraged by Slider (e.g. Java and python bindings to Zookeeper)
-
-
-Figure 2 - Unmanaged Application Client and associated Slider Application
-
-### Managed Application Client
-
-A managed application client is a component defined as part of the Slider/yarn application (i.e. it is part of the application definition presented to Slider). As such, it is deployed and managed via standard Slider/yarn mechanisms. This type of application client is more than likely already configured and written to work in a yarn environment.
-
-There are two primary needs to be met for a properly functioning client:
-
-1. **Discovery** - as a client, it is important that the client application retrieve the information it requires in order to communicate with the remote application components. As application components are spawned they (or the associated container agent) will advertise the relevant information using zookeeper. It will be up to the client (or the associated Slider client library) to contact zookeeper and retrieve the requested information.
-
-2. **Configuration** - there may be use cases in which a large number of configuration items are required by the client for its processing. In such cases it is more appropriate for a client to perform a bulk download of the application component(s) configuration as a JSON or XML document (via zookeeper or Slider-app comm?)
-
-Whether ascertained via discovery or bulk configuration retrieval, the attributes that the client obtains will more than likely need to be populated into the client’s configuration files. Therefore, a templating facility or the like should be provided to allow for such configuration file manipulation.
-
-### Unmanaged Application Client
-
-An unmanaged application client is a standalone application that leverages application components deployed into the Slider/yarn environment. It is not possible to predict the deployment mechanisms or network topology employed for an unmanaged application. However, it is feasible to provide some guidance and/or software (APIs etc) to allow for application component discovery and communication.
-
-## Application Management
-
-Post deployment, the Slider infrastructure will provide the requisite set of administrative facilities required for application management, including the ability to start/stop the application, monitor the application, and reconfigure the application.
-
-### General Management
-
-There is one general management command:
-
-* List Yarn Apps - returns a listing of deployed yarn apps and associated information:
-
- * name and version
-
- * dependencies (required HDP services and versions, etc)
-
- * configuration properties
-
- * components/roles and associated configuration
-
-### Application/Component Management
-
-The following administrative functions are supported for applications:
-
-* Install the application - the installation command will take care of the population of the application resources into the pre-determined application resource directory in HDFS. The required configuration and binary directories will also be created.
-
-* start/thaw the application - Slider/Yarn runtime negotiates and instantiates the number of component containers designated by the cluster description and the components are started.
-
-* stop/freeze the application - similar to stopping, applicaiton (or a subset of their components) can be stopped.
-
-* get application status - the retrieval of application status may take a number of forms, including:
-
- * liveness of service components
-
- * operational metrics (application-level or component-level)
-
- * viewing of logs
-
-* get application configuration - the configuration of application components is retrieved (JSON or XML form)
-
-* get cluster configuration - the cluster configuration is retrieved (number of various application components, associated hosts etc)
-
-* get cluster history
-
-* re-configure cluster
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/slider_specs/creating_app_definitions.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/creating_app_definitions.md b/src/site/markdown/slider_specs/creating_app_definitions.md
deleted file mode 100644
index ab9f51c..0000000
--- a/src/site/markdown/slider_specs/creating_app_definitions.md
+++ /dev/null
@@ -1,135 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider AppPackage
-
-Slider AppPackages are a declarative definition of an application for application management. AppPackage is not a packaging scheme for application binaries and artifacts. Tarball, zip files, rpms etc. are available for that purpose. Instead AppPackage includes the application binaries along with other artifacts necessary for application management.
-
-An application instance consists of several active component such as one or more masters and several slaves. There may be a number of accompanying processes in addition to the basic master/slave processes - lets refer to all processes as app component instances. When run in the context of Yarn, the application specific processes are activated within individual Yarn Container. If you pry into an Yarn container (created through Slider) it will be apparent as to what is the role of Slider-Agent and the actual application components. The following image provides an high-level view. Within a container there are at least two processes - and instance of a slider agent and an instance of an application component. The application can itself spawn more procsses if needed.
-
-
-
-Figure 1 - High-level view of a container
-
-For example:
-
- yarn 8849 -- python ./infra/agent/slider-agent/agent/main.py --label container_1397675825552_0011_01_000003___HBASE_REGIONSERVER --host AM_HOST --port 47830
- yarn 9085 -- bash /hadoop/yarn/local/usercache/yarn/appcache/application_1397675825552_0011/ ... internal_start regionserver
- yarn 9114 -- /usr/jdk64/jdk1.7.0_45/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=...
-
-The above list shows three processes, the Slider-Agent process, the bash script to start HBase Region Server and the HBase Region server itself. *Three of these together constitute the container*.
-
-## Using an AppPackage
-The following command creates an HBase application using the AppPackage for HBase.
-
- ./slider create cl1 --image hdfs://NN:8020/slider/agent/slider-agent.tar.gz --template /work/appConf.json --resources /work/resources.json
-
-Lets analyze various parameters from the perspective of app creation:
-
-* `--image`: its the slider agent tarball
-* `--template`: app configuration
-* `--resources`: yarn resource requests
-* … other parameters are described in accompanying docs.
-
-### AppPackage
-The structure of an AppPackage is described at [AppPackage](application_package.md).
-
-In the enlistment, there are three example AppPackages:
-
-* `app-packages/hbase-v0_96`
-* `app-packages/accumulo`
-* `app-packages/storm-v0_91`
-
-The above folders, with minor edits, can be packaged as *zip* files to get the corresponding AppPackages. The application tarball file, containing the binaries/artifacts of the application itself is a component within the AppPackage. They are:
-
-* For hbase - `app-packages/hbase-v0_96/package/files/hbase-0.96.1-hadoop2-bin.tar.gz.REPLACE`
-* For accumulo - `app-packages/accumulo/package/files/` (no file present)
-* For storm - `app-packages/storm-v0_91/package/files/apache-storm-0.9.1.2.1.1.0-237.tar.gz.placeholder`
-
-**They are placehoder files**, mostly because the files themselves are too large as well as users are free to use their own version of the package. To create a Slider AppPackage - replace the file with an actual application tarball and then ensure that the metainfo.xml has the correct file name. After that create a zip file using standard zip commands and ensure that the package has the metainfo.xml file at the root folder.
-
-For Accumulo, this is automated in maven by running
-
- mvn clean package -DskipTests -Paccumulo-app-package
-
-Additional configuration parameters are described in `app-packages/accumulo/README.txt`.
-
-For other app packages, follow this example:
-
-* cd slider/app-packages/hbase
-* zip -r hbase-v098.zip .
-* Looking at the content through unzip -l "$@" hbase-v098.zip
-
-```
-Archive: hbase-v098.zip
- Length Date Time Name
---------- ---------- ----- ----
- 3158 06-19-2014 22:08 appConfig.json
- 0 06-19-2014 22:15 configuration/
- 5077 06-19-2014 22:15 configuration/global.xml
- 5248 06-19-2014 22:15 configuration/hbase-log4j.xml
- 2250 06-19-2014 22:15 configuration/hbase-policy.xml
- 14705 06-19-2014 22:15 configuration/hbase-site.xml
- 2142 06-19-2014 22:17 jmx_metrics.json
- 3991 06-19-2014 22:45 metainfo.xml
- 0 06-19-2014 22:33 package/
- 0 06-19-2014 22:31 package/files/
- 83278591 06-19-2014 22:31 package/files/hbase-0.98.3-hadoop2-bin.tar.gz
- 0 06-19-2014 22:29 package/scripts/
- 787 06-19-2014 22:27 package/scripts/__init__.py
- 1378 06-19-2014 22:27 package/scripts/functions.py
- 3738 06-19-2014 22:27 package/scripts/hbase.py
- 1205 06-19-2014 22:27 package/scripts/hbase_client.py
- 1640 06-19-2014 22:27 package/scripts/hbase_master.py
- 1764 06-19-2014 22:27 package/scripts/hbase_regionserver.py
- 1482 06-19-2014 22:27 package/scripts/hbase_service.py
- 5089 06-19-2014 22:27 package/scripts/params.py
- 973 06-19-2014 22:27 package/scripts/status_params.py
- 0 06-19-2014 22:34 package/templates/
- 2799 06-19-2014 22:34 package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
- 2799 06-19-2014 22:34 package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
- 3878 06-19-2014 22:34 package/templates/hbase-env.sh.j2
- 909 06-19-2014 22:34 package/templates/hbase_client_jaas.conf.j2
- 989 06-19-2014 22:34 package/templates/hbase_master_jaas.conf.j2
- 1001 06-19-2014 22:34 package/templates/hbase_regionserver_jaas.conf.j2
- 837 06-19-2014 22:34 package/templates/regionservers.j2
- 3486 06-19-2014 22:39 pom.xml
- 1921 06-19-2014 21:53 README.txt
- 358 06-19-2014 22:36 resources.json
- 0 06-19-2014 22:40 src/
- 0 06-19-2014 23:02 src/assembly/
- 2400 06-19-2014 22:45 src/assembly/hbase.xml
---------- -------
- 83354595 35 files
-```
-
-Sample **resources.json** and **appConfig.json** files are also included in the enlistment. These are samples and are typically tested on one node test installations.
-
-
-### --template appConfig.json
-An appConfig.json contains the application configuration. See [Specifications InstanceConfiguration](application_instance_configuration.md) for details on how to create a template config file. The enlistment includes sample config files for HBase, Accumulo, and Storm.
-
-
-### --resources resources.json
-Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application. [Specification of Resources](resource_specification.html) describes how to write a resource config json file. The enlistment includes sample config files for HBase, Accumulo, and Storm.
-
-
-## Scripting for AppPackage
-Refer to [App Command Scripts](writing_app_command_scripts) for details on how to write scripts for a AppPackage. These scripts are in the package/script folder within the AppPackage. *Use the checked in samples for HBase/Storm/Accumulo as reference for script development.*
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/slider_specs/index.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/index.md b/src/site/markdown/slider_specs/index.md
deleted file mode 100644
index 953229e..0000000
--- a/src/site/markdown/slider_specs/index.md
+++ /dev/null
@@ -1,53 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider
-
-##Introduction
-
-**SLIDER: A collection of tools and technologies to simplify the packaging, deployment and management of long-running applications on YARN.**
-
-- Availability (always-on) - YARN works with the application to ensure recovery or restart of running application components.
-- Flexibility (dynamic scaling) - YARN provides the application with the facilities to allow for scale-up or scale-down
-- Resource Mgmt (optimization) - YARN handles allocation of cluster resources.
-
-## Terminology
-
-### Apps on YARN
- - Application written to run directly on YARN
- - Packaging, deployment and lifecycle management are custom built for each application
-
-### Slider Apps
- - Applications deployed and managed on YARN using Slider
- - Use of slider minimizes custom code for deployment + lifecycle management
- - Requires apps to follow Slider guidelines and packaging ("Sliderize")
-
-Specifications
----
-
-The entry points to leverage Slider are:
-
-- [Application Needs](application_needs.html) What it takes to be deployable by Slider.
-- [Slider AppPackage](creating_app_definitions.html) Overview of how to create an Slider AppPackage.
-- [Specifications for AppPackage](application_package.html) Describes the structure of an AppPackage
-- [Specifications for Application Definition](application_definition.html) How to write metainfo.xml?
-- [Specifications for Configuration](application_configuration.html) How to create a template config file for an app?
-- [Specification of Resources](resource_specification.html) How to write a resource spec for an app?
-- [Specifications InstanceConfiguration](application_instance_configuration.html) How to write a template config for an app?
-- [Guidelines for Clients and Client Applications](canonical_scenarios.html)
-- [Documentation for "General Developer Guidelines"](../developing/index.html)
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/slider_specs/resource_specification.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/resource_specification.md b/src/site/markdown/slider_specs/resource_specification.md
deleted file mode 100644
index cdcfb02..0000000
--- a/src/site/markdown/slider_specs/resource_specification.md
+++ /dev/null
@@ -1,53 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider Resource Specification
-Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application.
-
-An example resource requirement for an application that has two components "master" and "worker" is as follows. Slider will automatically add the requirements for the AppMaster for the application. This compoent is named "slider-appmaster".
-
-Some parameters that can be specified for a component instance include:
-
-* `yarn.memory`: amount of memory requried for the component instance
-* `yarn.vcores`: number of vcores requested
-* `yarn.role.priority`: each component must be assigned unique priority. Component with higher priority come up earlier than components with lower priority
-* `yarn.component.instances`: number of instances for this component type
-
-Sample:
-
- {
- "schema" : "http://example.org/specification/v2.0.0",
- "metadata" : {
- },
- "global" : {
- },
- "components" : {
- "HBASE_MASTER" : {
- "yarn.role.priority" : "1",
- "yarn.component.instances" : "1"
- "yarn.memory" : "768",
- "yarn.vcores" : "1"
- },
- "slider-appmaster" : {
- },
- "HBASE_REGIONSERVER" : {
- "yarn.role.priority" : "2",
- "yarn.component.instances" : "1"
- }
- }
- }
-
http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/209cee43/src/site/markdown/slider_specs/writing_app_command_scripts.md
----------------------------------------------------------------------
diff --git a/src/site/markdown/slider_specs/writing_app_command_scripts.md b/src/site/markdown/slider_specs/writing_app_command_scripts.md
deleted file mode 100644
index e59f0b9..0000000
--- a/src/site/markdown/slider_specs/writing_app_command_scripts.md
+++ /dev/null
@@ -1,211 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-# Apache Slider: Developing App Command Scripts
-
-App command implementations follow a standard structure so that they can be invoked in an uniform manner. For any command, the python scripts are invoked as:
-
- python SCRIPT COMMAND JSON_FILE PACKAGE_ROOT STRUCTURED_OUT_FILE
-
-* SCRIPT is the top level script that implements the commands for the component.
-
-* COMMAND is one of the following default commands - START, STOP, INSTALL, CONFIG, RESTART, STATUS or any custom commands.
-
-* JSON_FILE includes all configuration parameters and the values.
-
-* PACKAGE_ROOT is the root folder of the package. From this folder, its possible to access files, scripts, templates, packages (e.g. tarballs), etc. The Yarn-App author has complete control over the structure of the package as long as the PACKAGE_ROOT and SCRIPT path is known to the management tool.
-
-* STRUCTURED_OUT_FILE is the file where the script can output structured data.
-
-The management infrastructure is expected to automatically reports back STD_OUT and STD_ERR.
-
-Sample:
-
- python /apps/HBASE_ON_YARN/package/scripts/hbase_regionserver.py START /apps/commands/cmd_332/command.json /apps/HBASE_ON_YARN/package /apps/commands/cmd_332/strout.txt
-
-**Note**: The above is how Slider-Agent invokes the scripts. Its provided as a reference for developing the scripts themselves as well as a way to test/debug the scripts.
-
-## Structure of JSON formatted parameter
-
-The parameters are organized as multi-layer name-value pairs.
-
- {
- "commandId": "Command Id as assigned by Slider",
- "command": "Command being executed",
- "commandType": "Type of command",
- "clusterName": "Name of the cluster",
- "appName": "Name of the app",
- "component": "Name of the component",
- "hostname": "Name of the host",
- "public_hostname": "FQDN of the host",
- "hostParams": {
- "host specific parameters common to all commands"
- },
- "componentParams": {
- "component specific parameters, if any"
- },
- "commandParams": {
- "command specific parameters, usually used in case of custom commands"
- },
- "configurations": {
- "app-global-config": {
- },
- "config-type-2": {
- },
- "config-type-2": {
- }
- }
- }
-
-
-## Sample configuration parameters
-
- {
- "commandId": "2-2",
- "command": "START",
- "commandType": "EXECUTION_COMMAND",
- "clusterName": "c1",
- "appName": "HBASE",
- "componentName": "HBASE_MASTER",
- "hostParams": {
- "java_home": "/usr/jdk64/jdk1.7.0_45"
- },
- "componentParams": {},
- "commandParams": {},
- "hostname": "c6403.ambari.apache.org",
- "public_hostname": "c6403.ambari.apache.org",
- "configurations": {
- "hbase-log4j": {
- "log4j.threshold": "ALL",
- "log4j.rootLogger": "${hbase.root.logger}",
- "log4j.logger.org.apache.zookeeper": "INFO",
- "log4j.logger.org.apache.hadoop.hbase": "DEBUG",
- "log4j.logger.org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher": "INFO",
- "log4j.logger.org.apache.hadoop.hbase.zookeeper.ZKUtil": "INFO",
- "log4j.category.SecurityLogger": "${hbase.security.logger}",
- "log4j.appender.console": "org.apache.log4j.ConsoleAppender",
- "log4j.appender.console.target": "System.err",
- "log4j.appender.console.layout": "org.apache.log4j.PatternLayout",
- "log4j.appender.console.layout.ConversionPattern": "%d{ISO8601} %-5p [%t] %c{2}: %m%n",
- "log4j.appender.RFAS": "org.apache.log4j.RollingFileAppender",
- "log4j.appender.RFAS.layout": "org.apache.log4j.PatternLayout",
- "log4j.appender.RFAS.layout.ConversionPattern": "%d{ISO8601} %p %c: %m%n",
- "log4j.appender.RFAS.MaxFileSize": "${hbase.security.log.maxfilesize}",
- "log4j.appender.RFAS.MaxBackupIndex": "${hbase.security.log.maxbackupindex}",
- "log4j.appender.RFAS.File": "${hbase.log.dir}/${hbase.security.log.file}",
- "log4j.appender.RFA": "org.apache.log4j.RollingFileAppender",
- "log4j.appender.RFA.layout": "org.apache.log4j.PatternLayout",
- "log4j.appender.RFA.layout.ConversionPattern": "%d{ISO8601} %-5p [%t] %c{2}: %m%n",
- "log4j.appender.RFA.MaxFileSize": "${hbase.log.maxfilesize}",
- "log4j.appender.RFA.MaxBackupIndex": "${hbase.log.maxbackupindex}",
- "log4j.appender.RFA.File": "${hbase.log.dir}/${hbase.log.file}",
- "log4j.appender.NullAppender": "org.apache.log4j.varia.NullAppender",
- "log4j.appender.DRFA": "org.apache.log4j.DailyRollingFileAppender",
- "log4j.appender.DRFA.layout": "org.apache.log4j.PatternLayout",
- "log4j.appender.DRFA.layout.ConversionPattern": "%d{ISO8601} %-5p [%t] %c{2}: %m%n",
- "log4j.appender.DRFA.File": "${hbase.log.dir}/${hbase.log.file}",
- "log4j.appender.DRFA.DatePattern": ".yyyy-MM-dd",
- "log4j.additivity.SecurityLogger": "false",
- "hbase.security.logger": "INFO,console",
- "hbase.security.log.maxfilesize": "256MB",
- "hbase.security.log.maxbackupindex": "20",
- "hbase.security.log.file": "SecurityAuth.audit",
- "hbase.root.logger": "INFO,console",
- "hbase.log.maxfilesize": "256MB",
- "hbase.log.maxbackupindex": "20",
- "hbase.log.file": "hbase.log",
- "hbase.log.dir": "."
- },
- "app-global-config": {
- "security_enabled": "false",
- "pid_dir": "/hadoop/yarn/log/application_1394053491953_0003/run",
- "log_dir": "/hadoop/yarn/log/application_1394053491953_0003/log",
- "tmp_dir": "/hadoop/yarn/log/application_1394053491953_0003/tmp",
- "user_group": "hadoop",
- "user": "hbase",
- "hbase_regionserver_heapsize": "1024m",
- "hbase_master_heapsize": "1024m",
- "fs_default_name": "hdfs://c6403.ambari.apache.org:8020",
- "hdfs_root": "/apps/hbase/instances/01",
- "zookeeper_node": "/apps/hbase/instances/01",
- "zookeeper_quorom_hosts": "c6403.ambari.apache.org",
- "zookeeper_port": "2181",
- },
- "hbase-site": {
- "hbase.hstore.flush.retries.number": "120",
- "hbase.client.keyvalue.maxsize": "10485760",
- "hbase.hstore.compactionThreshold": "3",
- "hbase.rootdir": "hdfs://c6403.ambari.apache.org:8020/apps/hbase/instances/01/data",
- "hbase.stagingdir": "hdfs://c6403.ambari.apache.org:8020/apps/hbase/instances/01/staging",
- "hbase.regionserver.handler.count": "60",
- "hbase.regionserver.global.memstore.lowerLimit": "0.38",
- "hbase.hregion.memstore.block.multiplier": "2",
- "hbase.hregion.memstore.flush.size": "134217728",
- "hbase.superuser": "yarn",
- "hbase.zookeeper.property.clientPort": "2181",
- "hbase.regionserver.global.memstore.upperLimit": "0.4",
- "zookeeper.session.timeout": "30000",
- "hbase.tmp.dir": "/hadoop/yarn/log/application_1394053491953_0003/tmp",
- "hbase.hregion.max.filesize": "10737418240",
- "hfile.block.cache.size": "0.40",
- "hbase.security.authentication": "simple",
- "hbase.defaults.for.version.skip": "true",
- "hbase.zookeeper.quorum": "c6403.ambari.apache.org",
- "zookeeper.znode.parent": "/apps/hbase/instances/01",
- "hbase.hstore.blockingStoreFiles": "10",
- "hbase.hregion.majorcompaction": "86400000",
- "hbase.security.authorization": "false",
- "hbase.cluster.distributed": "true",
- "hbase.hregion.memstore.mslab.enabled": "true",
- "hbase.client.scanner.caching": "100",
- "hbase.zookeeper.useMulti": "true",
- "hbase.regionserver.info.port": "",
- "hbase.master.info.port": "60010"
- }
- }
- }
-
-
-## Sample command script
-
- class OozieServer(Script):
- def install(self, env):
- self.install_packages(env)
-
- def configure(self, env):
- import params
- env.set_params(params)
- oozie(is_server=True)
-
- def start(self, env):
- import params
- env.set_params(params)
- self.configure(env)
- oozie_service(action='start')
-
- def stop(self, env):
- import params
- env.set_params(params)
- oozie_service(action='stop')
-
- def status(self, env):
- import status_params
- env.set_params(status_params)
- check_process_status(status_params.pid_file)
-
-
-