You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by zh...@apache.org on 2015/02/23 20:36:50 UTC
[02/52] [abbrv] hadoop git commit: HADOOP-11593. Convert site
documentation from apt to markdown (stragglers) (Masatake Iwasaki via aw)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-sls/src/site/apt/SchedulerLoadSimulator.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-sls/src/site/apt/SchedulerLoadSimulator.apt.vm b/hadoop-tools/hadoop-sls/src/site/apt/SchedulerLoadSimulator.apt.vm
deleted file mode 100644
index a8b408c..0000000
--- a/hadoop-tools/hadoop-sls/src/site/apt/SchedulerLoadSimulator.apt.vm
+++ /dev/null
@@ -1,439 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License.
-
- ---
- Yarn Scheduler Load Simulator (SLS)
- ---
- ---
- ${maven.build.timestamp}
-
-Yarn Scheduler Load Simulator (SLS)
-
-%{toc|section=1|fromDepth=0}
-
-* Overview
-
-** Overview
-
- The Yarn scheduler is a fertile area of interest with different
- implementations, e.g., Fifo, Capacity and Fair schedulers. Meanwhile, several
- optimizations are also made to improve scheduler performance for different
- scenarios and workload. Each scheduler algorithm has its own set of features,
- and drives scheduling decisions by many factors, such as fairness, capacity
- guarantee, resource availability, etc. It is very important to evaluate a
- scheduler algorithm very well before we deploy in a production cluster.
- Unfortunately, currently it is non-trivial to evaluate a scheduler algorithm.
- Evaluating in a real cluster is always time and cost consuming, and it is
- also very hard to find a large-enough cluster. Hence, a simulator which can
- predict how well a scheduler algorithm for some specific workload would be
- quite useful.
-
- The Yarn Scheduler Load Simulator (SLS) is such a tool, which can simulate
- large-scale Yarn clusters and application loads in a single machine.This
- simulator would be invaluable in furthering Yarn by providing a tool for
- researchers and developers to prototype new scheduler features and predict
- their behavior and performance with reasonable amount of confidence,
- thereby aiding rapid innovation.
-
- The simulator will exercise the real Yarn <<<ResourceManager>>> removing the
- network factor by simulating <<<NodeManagers>>> and <<<ApplicationMasters>>>
- via handling and dispatching <<<NM>>>/<<<AMs>>> heartbeat events from within
- the same JVM. To keep tracking of scheduler behavior and performance, a
- scheduler wrapper will wrap the real scheduler.
-
- The size of the cluster and the application load can be loaded from
- configuration files, which are generated from job history files directly by
- adopting {{{https://hadoop.apache.org/docs/stable/rumen.html}Apache Rumen}}.
-
- The simulator will produce real time metrics while executing, including:
-
- * Resource usages for whole cluster and each queue, which can be utilized to
- configure cluster and queue's capacity.
-
- * The detailed application execution trace (recorded in relation to simulated
- time), which can be analyzed to understand/validate the scheduler behavior
- (individual jobs turn around time, throughput, fairness, capacity guarantee,
- etc.).
-
- * Several key metrics of scheduler algorithm, such as time cost of each
- scheduler operation (allocate, handle, etc.), which can be utilized by Hadoop
- developers to find the code spots and scalability limits.
-
-** Goals
-
- * Exercise the scheduler at scale without a real cluster using real job
- traces.
-
- * Being able to simulate real workloads.
-
-** Architecture
-
- The following figure illustrates the implementation architecture of the
- simulator.
-
-[images/sls_arch.png] The architecture of the simulator
-
- The simulator takes input of workload traces, and fetches the cluster and
- applications information. For each NM and AM, the simulator builds a simulator
- to simulate their running. All NM/AM simulators run in a thread pool. The
- simulator reuses Yarn Resource Manager, and builds a wrapper out of the
- scheduler. The Scheduler Wrapper can track the scheduler behaviors and
- generates several logs, which are the outputs of the simulator and can be
- further analyzed.
-
-** Usecases
-
- * Engineering
-
- * Verify correctness of scheduler algorithm under load
-
- * Cheap/practical way for finding code hotspots/critical-path.
-
- * Validate the impact of changes and new features.
-
- * Determine what drives the scheduler scalability limits.
-
- []
-
- * QA
-
- * Validate scheduler behavior for "large" clusters and several workload
- profiles.
-
- * Solutions/Sales.
-
- * Sizing model for predefined/typical workloads.
-
- * Cluster sizing tool using real customer data (job traces).
-
- * Determine minimum SLAs under a particular workload.
-
-* Usage
-
- This section will show how to use the simulator. Here let <<<$HADOOP_ROOT>>>
- represent the Hadoop install directory. If you build Hadoop yourself,
- <<<$HADOOP_ROOT>>> is <<<hadoop-dist/target/hadoop-$VERSION>>>. The simulator
- is located at <<<$HADOOP_ROOT/share/hadoop/tools/sls>>>. The fold <<<sls>>>
- containers four directories: <<<bin>>>, <<<html>>>, <<<sample-conf>>>, and
- <<<sample-data>>>
-
- * <<<bin>>>: contains running scripts for the simulator.
-
- * <<<html>>>: contains several html/css/js files we needed for real-time
- tracking.
-
- * <<<sample-conf>>>: specifies the simulator configurations.
-
- * <<<sample-data>>>: provides an example rumen trace, which can be used to
- generate inputs of the simulator.
-
- []
-
- The following sections will describe how to use the simulator step by step.
- Before start, make sure that command <<<hadoop>>> is included in your
- <<<$PATH>>> environment parameter.
-
-** Step 1: Configure Hadoop and the simulator
-
- Before we start, make sure Hadoop and the simulator are configured well.
- All configuration files for Hadoop and the simulator should be placed in
- directory <<<$HADOOP_ROOT/etc/hadoop>>>, where the <<<ResourceManager>>>
- and Yarn scheduler load their configurations. Directory
- <<<$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/>>> provides several
- example configurations, that can be used to start a demo.
-
- For configuration of Hadoop and Yarn scheduler, users can refer to Yarn’s
- website ({{{http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/}
- http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/}}).
-
- For the simulator, it loads configuration information from file
- <<<$HADOOP_ROOT/etc/hadoop/sls-runner.xml>>>.
-
- Here we illustrate each configuration parameter in <<<sls-runner.xml>>>.
- Note that <<<$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/sls-runner.xml>>>
- contains all the default values for these configuration parameters.
-
- * <<<yarn.sls.runner.pool.size>>>
-
- The simulator uses a thread pool to simulate the <<<NM>>> and <<<AM>>> running
- , and this parameter specifies the number of threads in the pool.
-
- * <<<yarn.sls.nm.memory.mb>>>
-
- The total memory for each <<<NMSimulator>>>.
-
- * <<<yarn.sls.nm.vcores>>>
-
- The total vCores for each <<<NMSimulator>>>.
-
- * <<<yarn.sls.nm.heartbeat.interval.ms>>>
-
- The heartbeat interval for each <<<NMSimulator>>>.
-
- * <<<yarn.sls.am.heartbeat.interval.ms>>>
-
- The heartbeat interval for each <<<AMSimulator>>>.
-
- * <<<yarn.sls.am.type.mapreduce>>>
-
- The <<<AMSimulator>>> implementation for MapReduce-like applications.
- Users can specify implementations for other type of applications.
-
- * <<<yarn.sls.container.memory.mb>>>
-
- The memory required for each container simulator.
-
- * <<<yarn.sls.container.vcores>>>
-
- The vCores required for each container simulator.
-
- * <<<yarn.sls.runner.metrics.switch>>>
-
- The simulator introduces {{{http://metrics.codahale.com/}Metrics}} to measure
- the behaviors of critical components and operations. This field specifies
- whether we open (<<<ON>>>) or close (<<<OFF>>>) the Metrics running.
-
- * <<<yarn.sls.metrics.web.address.port>>>
-
- The port used by simulator to provide real-time tracking. The default value is
- 10001.
-
- * <<<org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler>>>
-
- The implementation of scheduler metrics of Fifo Scheduler.
-
- * <<<org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler>>>
-
- The implementation of scheduler metrics of Fair Scheduler.
-
- * <<<org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler>>>
-
- The implementation of scheduler metrics of Capacity Scheduler.
-
-** Step 2: Run the simulator
-
- The simulator supports two types of input files: the rumen traces and its own
- input traces. The script to start the simulator is <<<slsrun.sh>>>.
-
-+----+
-$ cd $HADOOP_ROOT/share/hadoop/tools/sls
-$ bin/slsrun.sh
- --input-rumen|--input-sls=<TRACE_FILE1,TRACE_FILE2,...>
- --output-dir=<SLS_SIMULATION_OUTPUT_DIRECTORY> [--nodes=<SLS_NODES_FILE>]
- [--track-jobs=<JOBID1,JOBID2,...>] [--print-simulation]
-+----+
-
- * <<<--input-rumen>>>: The input rumen trace files. Users can input multiple
- files, separated by comma. One example trace is provided in
- <<<$HADOOP_ROOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json>>>.
-
- * <<<--input-sls>>>: Simulator its own file format. The simulator also
- provides a tool to convert rumen traces to sls traces (<<<rumen2sls.sh>>>).
- Refer to appendix for an example of sls input json file.
-
- * <<<--output-dir>>>: The output directory for generated running logs and
- metrics.
-
- * <<<--nodes>>>: The cluster topology. By default, the simulator will use the
- topology fetched from the input json files. Users can specifies a new topology
- by setting this parameter. Refer to the appendix for the topology file format.
-
- * <<<--track-jobs>>>: The particular jobs that will be tracked during
- simulator running, spearated by comma.
-
- * <<<--print-simulation>>>: Whether to print out simulation information
- before simulator running, including number of nodes, applications, tasks,
- and information for each application.
-
- In comparison to rumen format, here the sls format is much simpler and users
- can easily generate various workload. The simulator also provides a tool to
- convert rumen traces to sls traces.
-
-+----+
-$ bin/rumen2sls.sh
- --rumen-file=<RUMEN_FILE>
- --output-dir=<SLS_OUTPUT_DIRECTORY>
- [--output-prefix=<SLS_FILE_PREFIX>]
-+----+
-
- * <<<--rumen-file>>>: The rumen format file. One example trace is provided
- in directory <<<sample-data>>>.
-
- * <<<--output-dir>>>: The output directory of generated simulation traces.
- Two files will be generated in this output directory, including one trace
- file including all job and task information, and another file showing the
- topology information.
-
- * <<<--output-prefix>>>: The prefix of the generated files. The default value
- is ”sls”, and the two generated files are <<<sls-jobs.json>>> and
- <<<sls-nodes.json>>>.
-
-* Metrics
-
- The Yarn Scheduler Load Simulator has integrated
- {{{http://metrics.codahale.com/}Metrics}} to measure the behaviors of critical
- components and operations, including running applications and containers,
- cluster available resources, scheduler operation timecost, et al. If the
- switch <<<yarn.sls.runner.metrics.switch>>> is set <<<ON>>>, <<<Metrics>>>
- will run and output it logs in <<<--output-dir>>> directory specified by users.
- Users can track these information during simulator running, and can also
- analyze these logs after running to evaluate the scheduler performance.
-
-** Real-time Tracking
-
- The simulator provides an interface for tracking its running in real-time.
- Users can go to <<<http://host:port/simulate>>> to track whole running,
- and <<<http://host:port/track>>> to track a particular job or queue. Here
- the <<<host>>> is the place when we run the simulator, and <<<port>>> is
- the value configured by <<<yarn.sls.metrics.web.address.port>>> (default value
- is 10001).
-
- Here we'll illustrate each chart shown in the webpage.
-
- The first figure describes the number of running applications and containers.
-
-[images/sls_running_apps_containers.png] Number of running applications/containers
-
- The second figure describes the allocated and available resources (memory)
- in the cluster.
-
-[images/sls_cluster_memory.png] Cluster Resource (Memory)
-
- The third figure describes the allocated resource for each queue. Here we have
- three queues: sls_queue_1, sls_queue_2, and sls_queue_3.The first two queues
- are configured with 25% share, while the last one has 50% share.
-
-[images/sls_queue_allocated_memory.png] Queue Allocated Resource (Memory)
-
- The fourth figure describes the timecost for each scheduler operation.
-
-[images/sls_scheduler_operation_timecost.png] Scheduler Opertion Timecost
-
- Finally, we measure the memory used by the simulator.
-
-[images/sls_JVM.png] JVM Memory
-
- The simulator also provides an interface for tracking some particular
- jobs and queues. Go to <<<http://<Host>:<Port>/track>>> to get these
- information.
-
- Here the first figure illustrates the resource usage information for queue
- <<<SLS_Queue_1>>>.
-
-[images/sls_track_queue.png] Tracking Queue <<<sls_queue_3>>>
-
- The second figure illustrates the resource usage information for job
- <<<job_1369942127770_0653>>>.
-
-[images/sls_track_job.png] Tracking Job <<<job_1369942127770_0653>>>
-
-** Offline Analysis
-
- After the simulator finishes, all logs are saved in the output directory
- specified by <<<--output-dir>>> in
- <<<$HADOOP_ROOT/share/hadoop/tools/sls/bin/slsrun.sh>>>.
-
- * File <<<realtimetrack.json>>>: records all real-time tracking logs every 1
- second.
-
- * File <<<jobruntime.csv>>>: records all jobs’ start and end time in the
- simulator.
-
- * Folder <<<metrics>>>: logs generated by the Metrics.
-
- []
-
- Users can also reproduce those real-time tracking charts in offline mode.
- Just upload the <<<realtimetrack.json>>> to
- <<<$HADOOP_ROOT/share/hadoop/tools/sls/html/showSimulationTrace.html>>>.
- For browser security problem, need to put files <<<realtimetrack.json>>> and
- <<<showSimulationTrace.html>>> in the same directory.
-
-* Appendix
-
-** Resources
-
- {{{https://issues.apache.org/jira/browse/YARN-1021}YARN-1021}} is the main
- JIRA that introduces Yarn Scheduler Load Simulator to Hadoop Yarn project.
-
-** SLS JSON input file format
-
- Here we provide an example format of the sls json file, which contains 2 jobs.
- The first job has 3 map tasks and the second one has 2 map tasks.
-
-+----+
-{
- "am.type" : "mapreduce",
- "job.start.ms" : 0,
- "job.end.ms" : 95375,
- "job.queue.name" : "sls_queue_1",
- "job.id" : "job_1",
- "job.user" : "default",
- "job.tasks" : [ {
- "container.host" : "/default-rack/node1",
- "container.start.ms" : 6664,
- "container.end.ms" : 23707,
- "container.priority" : 20,
- "container.type" : "map"
- }, {
- "container.host" : "/default-rack/node3",
- "container.start.ms" : 6665,
- "container.end.ms" : 21593,
- "container.priority" : 20,
- "container.type" : "map"
- }, {
- "container.host" : "/default-rack/node2",
- "container.start.ms" : 68770,
- "container.end.ms" : 86613,
- "container.priority" : 20,
- "container.type" : "map"
- } ]
-}
-{
- "am.type" : "mapreduce",
- "job.start.ms" : 105204,
- "job.end.ms" : 197256,
- "job.queue.name" : "sls_queue_2",
- "job.id" : "job_2",
- "job.user" : "default",
- "job.tasks" : [ {
- "container.host" : "/default-rack/node1",
- "container.start.ms" : 111822,
- "container.end.ms" : 133985,
- "container.priority" : 20,
- "container.type" : "map"
- }, {
- "container.host" : "/default-rack/node2",
- "container.start.ms" : 111788,
- "container.end.ms" : 131377,
- "container.priority" : 20,
- "container.type" : "map"
- } ]
-}
-+----+
-
-** Simulator input topology file format
-
- Here is an example input topology file which has 3 nodes organized in 1 rack.
-
-+----+
-{
- "rack" : "default-rack",
- "nodes" : [ {
- "node" : "node1"
- }, {
- "node" : "node2"
- }, {
- "node" : "node3"
- }]
-}
-+----+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md b/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md
new file mode 100644
index 0000000..ca179ee
--- /dev/null
+++ b/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md
@@ -0,0 +1,357 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+Yarn Scheduler Load Simulator (SLS)
+===================================
+
+* [Yarn Scheduler Load Simulator (SLS)](#Yarn_Scheduler_Load_Simulator_SLS)
+ * [Overview](#Overview)
+ * [Overview](#Overview)
+ * [Goals](#Goals)
+ * [Architecture](#Architecture)
+ * [Usecases](#Usecases)
+ * [Usage](#Usage)
+ * [Step 1: Configure Hadoop and the simulator](#Step_1:_Configure_Hadoop_and_the_simulator)
+ * [Step 2: Run the simulator](#Step_2:_Run_the_simulator)
+ * [Metrics](#Metrics)
+ * [Real-time Tracking](#Real-time_Tracking)
+ * [Offline Analysis](#Offline_Analysis)
+ * [Appendix](#Appendix)
+ * [Resources](#Resources)
+ * [SLS JSON input file format](#SLS_JSON_input_file_format)
+ * [Simulator input topology file format](#Simulator_input_topology_file_format)
+
+Overview
+--------
+
+### Overview
+
+The Yarn scheduler is a fertile area of interest with different implementations, e.g., Fifo, Capacity and Fair schedulers. Meanwhile, several optimizations are also made to improve scheduler performance for different scenarios and workload. Each scheduler algorithm has its own set of features, and drives scheduling decisions by many factors, such as fairness, capacity guarantee, resource availability, etc. It is very important to evaluate a scheduler algorithm very well before we deploy in a production cluster. Unfortunately, currently it is non-trivial to evaluate a scheduler algorithm. Evaluating in a real cluster is always time and cost consuming, and it is also very hard to find a large-enough cluster. Hence, a simulator which can predict how well a scheduler algorithm for some specific workload would be quite useful.
+
+The Yarn Scheduler Load Simulator (SLS) is such a tool, which can simulate large-scale Yarn clusters and application loads in a single machine.This simulator would be invaluable in furthering Yarn by providing a tool for researchers and developers to prototype new scheduler features and predict their behavior and performance with reasonable amount of confidence, thereby aiding rapid innovation.
+o
+The simulator will exercise the real Yarn `ResourceManager` removing the network factor by simulating `NodeManagers` and `ApplicationMasters` via handling and dispatching `NM`/`AMs` heartbeat events from within the same JVM. To keep tracking of scheduler behavior and performance, a scheduler wrapper will wrap the real scheduler.
+
+The size of the cluster and the application load can be loaded from configuration files, which are generated from job history files directly by adopting [Apache Rumen](https://hadoop.apache.org/docs/stable/rumen.html).
+
+The simulator will produce real time metrics while executing, including:
+
+* Resource usages for whole cluster and each queue, which can be utilized to
+ configure cluster and queue's capacity.
+
+* The detailed application execution trace (recorded in relation to simulated
+ time), which can be analyzed to understand/validate the scheduler behavior
+ (individual jobs turn around time, throughput, fairness, capacity guarantee,
+ etc.).
+
+* Several key metrics of scheduler algorithm, such as time cost of each
+ scheduler operation (allocate, handle, etc.), which can be utilized by Hadoop
+ developers to find the code spots and scalability limits.
+
+### Goals
+
+* Exercise the scheduler at scale without a real cluster using real job
+ traces.
+
+* Being able to simulate real workloads.
+
+### Architecture
+
+The following figure illustrates the implementation architecture of the simulator.
+
+
+
+The simulator takes input of workload traces, and fetches the cluster and applications information. For each NM and AM, the simulator builds a simulator to simulate their running. All NM/AM simulators run in a thread pool. The simulator reuses Yarn Resource Manager, and builds a wrapper out of the scheduler. The Scheduler Wrapper can track the scheduler behaviors and generates several logs, which are the outputs of the simulator and can be further analyzed.
+
+### Usecases
+
+* Engineering
+ * Verify correctness of scheduler algorithm under load
+ * Cheap/practical way for finding code hotspots/critical-path.
+ * Validate the impact of changes and new features.
+ * Determine what drives the scheduler scalability limits.
+
+* QA
+ * Validate scheduler behavior for "large" clusters and several workload profiles.
+
+* Solutions/Sales.
+ * Sizing model for predefined/typical workloads.
+ * Cluster sizing tool using real customer data (job traces).
+ * Determine minimum SLAs under a particular workload.
+
+Usage
+-----
+
+This section will show how to use the simulator. Here let `$HADOOP_ROOT` represent the Hadoop install directory. If you build Hadoop yourself, `$HADOOP_ROOT` is `hadoop-dist/target/hadoop-$VERSION`. The simulator is located at `$HADOOP_ROOT/share/hadoop/tools/sls`. The fold `sls` containers four directories: `bin`, `html`, `sample-conf`, and `sample-data`
+
+* `bin`: contains running scripts for the simulator.
+
+* `html`: contains several html/css/js files we needed for real-time tracking.
+
+* `sample-conf`: specifies the simulator configurations.
+
+* `sample-data`: provides an example rumen trace, which can be used to
+ generate inputs of the simulator.
+
+The following sections will describe how to use the simulator step by step. Before start, make sure that command `hadoop` is included in your `$PATH` environment parameter.
+
+### Step 1: Configure Hadoop and the simulator
+
+Before we start, make sure Hadoop and the simulator are configured well. All configuration files for Hadoop and the simulator should be placed in directory `$HADOOP_ROOT/etc/hadoop`, where the `ResourceManager` and Yarn scheduler load their configurations. Directory `$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/` provides several example configurations, that can be used to start a demo.
+
+For configuration of Hadoop and Yarn scheduler, users can refer to Yarn’s website (<http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/>).
+
+For the simulator, it loads configuration information from file `$HADOOP_ROOT/etc/hadoop/sls-runner.xml`.
+
+Here we illustrate each configuration parameter in `sls-runner.xml`. Note that `$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/sls-runner.xml` contains all the default values for these configuration parameters.
+
+* `yarn.sls.runner.pool.size`
+
+ The simulator uses a thread pool to simulate the `NM` and `AM` running,
+ and this parameter specifies the number of threads in the pool.
+
+* `yarn.sls.nm.memory.mb`
+
+ The total memory for each `NMSimulator`.
+
+* `yarn.sls.nm.vcores`
+
+ The total vCores for each `NMSimulator`.
+
+* `yarn.sls.nm.heartbeat.interval.ms`
+
+ The heartbeat interval for each `NMSimulator`.
+
+* `yarn.sls.am.heartbeat.interval.ms`
+
+ The heartbeat interval for each `AMSimulator`.
+
+* `yarn.sls.am.type.mapreduce`
+
+ The `AMSimulator` implementation for MapReduce-like applications.
+ Users can specify implementations for other type of applications.
+
+* `yarn.sls.container.memory.mb`
+
+ The memory required for each container simulator.
+
+* `yarn.sls.container.vcores`
+
+ The vCores required for each container simulator.
+
+* `yarn.sls.runner.metrics.switch`
+
+ The simulator introduces [Metrics](http://metrics.codahale.com/) to measure
+ the behaviors of critical components and operations. This field specifies
+ whether we open (`ON`) or close (`OFF`) the Metrics running.
+
+* `yarn.sls.metrics.web.address.port`
+
+ The port used by simulator to provide real-time tracking. The default value is
+ 10001.
+
+* `org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler`
+
+ The implementation of scheduler metrics of Fifo Scheduler.
+
+* `org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler`
+
+ The implementation of scheduler metrics of Fair Scheduler.
+
+* `org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler`
+
+ The implementation of scheduler metrics of Capacity Scheduler.
+
+### Step 2: Run the simulator
+
+The simulator supports two types of input files: the rumen traces and its own input traces. The script to start the simulator is `slsrun.sh`.
+
+ $ cd $HADOOP_ROOT/share/hadoop/tools/sls
+ $ bin/slsrun.sh
+ --input-rumen |--input-sls=<TRACE_FILE1,TRACE_FILE2,...>
+ --output-dir=<SLS_SIMULATION_OUTPUT_DIRECTORY> [--nodes=<SLS_NODES_FILE>]
+ [--track-jobs=<JOBID1,JOBID2,...>] [--print-simulation]
+
+* `--input-rumen`: The input rumen trace files. Users can input multiple
+ files, separated by comma. One example trace is provided in
+ `$HADOOP_ROOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json`.
+
+* `--input-sls`: Simulator its own file format. The simulator also
+ provides a tool to convert rumen traces to sls traces (`rumen2sls.sh`).
+ Refer to appendix for an example of sls input json file.
+
+* `--output-dir`: The output directory for generated running logs and
+ metrics.
+
+* `--nodes`: The cluster topology. By default, the simulator will use the
+ topology fetched from the input json files. Users can specifies a new topology
+ by setting this parameter. Refer to the appendix for the topology file format.
+
+* `--track-jobs`: The particular jobs that will be tracked during
+ simulator running, spearated by comma.
+
+* `--print-simulation`: Whether to print out simulation information
+ before simulator running, including number of nodes, applications, tasks,
+ and information for each application.
+
+ In comparison to rumen format, here the sls format is much simpler and users
+ can easily generate various workload. The simulator also provides a tool to
+ convert rumen traces to sls traces.
+
+ $ bin/rumen2sls.sh
+ --rumen-file=<RUMEN_FILE>
+ --output-dir=<SLS_OUTPUT_DIRECTORY>
+ [--output-prefix=<SLS_FILE_PREFIX>]
+
+* `--rumen-file`: The rumen format file. One example trace is provided
+ in directory `sample-data`.
+
+* `--output-dir`: The output directory of generated simulation traces.
+ Two files will be generated in this output directory, including one trace
+ file including all job and task information, and another file showing the
+ topology information.
+
+* `--output-prefix`: The prefix of the generated files. The default value
+ is "sls", and the two generated files are `sls-jobs.json` and
+ `sls-nodes.json`.
+
+Metrics
+-------
+
+The Yarn Scheduler Load Simulator has integrated [Metrics](http://metrics.codahale.com/) to measure the behaviors of critical components and operations, including running applications and containers, cluster available resources, scheduler operation timecost, et al. If the switch `yarn.sls.runner.metrics.switch` is set `ON`, `Metrics` will run and output it logs in `--output-dir` directory specified by users. Users can track these information during simulator running, and can also analyze these logs after running to evaluate the scheduler performance.
+
+### Real-time Tracking
+
+The simulator provides an interface for tracking its running in real-time. Users can go to `http://host:port/simulate` to track whole running, and `http://host:port/track` to track a particular job or queue. Here the `host` is the place when we run the simulator, and `port` is the value configured by `yarn.sls.metrics.web.address.port` (default value is 10001).
+
+Here we'll illustrate each chart shown in the webpage.
+
+The first figure describes the number of running applications and containers.
+
+
+
+The second figure describes the allocated and available resources (memory) in the cluster.
+
+
+
+The third figure describes the allocated resource for each queue. Here we have three queues: sls\_queue\_1, sls\_queue\_2, and sls\_queue\_3.The first two queues are configured with 25% share, while the last one has 50% share.
+
+
+
+The fourth figure describes the timecost for each scheduler operation.
+
+
+
+Finally, we measure the memory used by the simulator.
+
+
+
+The simulator also provides an interface for tracking some particular jobs and queues. Go to `http://<Host>:<Port>/track` to get these information.
+
+Here the first figure illustrates the resource usage information for queue `SLS_Queue_1`.
+
+
+
+The second figure illustrates the resource usage information for job `job_1369942127770_0653`.
+
+
+
+### Offline Analysis
+
+After the simulator finishes, all logs are saved in the output directory specified by `--output-dir` in `$HADOOP_ROOT/share/hadoop/tools/sls/bin/slsrun.sh`.
+
+* File `realtimetrack.json`: records all real-time tracking logs every 1
+ second.
+
+* File `jobruntime.csv`: records all jobs’ start and end time in the
+ simulator.
+
+* Folder `metrics`: logs generated by the Metrics.
+
+Users can also reproduce those real-time tracking charts in offline mode. Just upload the `realtimetrack.json` to `$HADOOP_ROOT/share/hadoop/tools/sls/html/showSimulationTrace.html`. For browser security problem, need to put files `realtimetrack.json` and `showSimulationTrace.html` in the same directory.
+
+Appendix
+--------
+
+### Resources
+
+[YARN-1021](https://issues.apache.org/jira/browse/YARN-1021) is the main JIRA that introduces Yarn Scheduler Load Simulator to Hadoop Yarn project.
+
+### SLS JSON input file format
+
+Here we provide an example format of the sls json file, which contains 2 jobs. The first job has 3 map tasks and the second one has 2 map tasks.
+
+ {
+ "am.type" : "mapreduce",
+ "job.start.ms" : 0,
+ "job.end.ms" : 95375,
+ "job.queue.name" : "sls_queue_1",
+ "job.id" : "job_1",
+ "job.user" : "default",
+ "job.tasks" : [ {
+ "container.host" : "/default-rack/node1",
+ "container.start.ms" : 6664,
+ "container.end.ms" : 23707,
+ "container.priority" : 20,
+ "container.type" : "map"
+ }, {
+ "container.host" : "/default-rack/node3",
+ "container.start.ms" : 6665,
+ "container.end.ms" : 21593,
+ "container.priority" : 20,
+ "container.type" : "map"
+ }, {
+ "container.host" : "/default-rack/node2",
+ "container.start.ms" : 68770,
+ "container.end.ms" : 86613,
+ "container.priority" : 20,
+ "container.type" : "map"
+ } ]
+ }
+ {
+ "am.type" : "mapreduce",
+ "job.start.ms" : 105204,
+ "job.end.ms" : 197256,
+ "job.queue.name" : "sls_queue_2",
+ "job.id" : "job_2",
+ "job.user" : "default",
+ "job.tasks" : [ {
+ "container.host" : "/default-rack/node1",
+ "container.start.ms" : 111822,
+ "container.end.ms" : 133985,
+ "container.priority" : 20,
+ "container.type" : "map"
+ }, {
+ "container.host" : "/default-rack/node2",
+ "container.start.ms" : 111788,
+ "container.end.ms" : 131377,
+ "container.priority" : 20,
+ "container.type" : "map"
+ } ]
+ }
+
+### Simulator input topology file format
+
+Here is an example input topology file which has 3 nodes organized in 1 rack.
+
+ {
+ "rack" : "default-rack",
+ "nodes" : [ {
+ "node" : "node1"
+ }, {
+ "node" : "node2"
+ }, {
+ "node" : "node3"
+ }]
+ }
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm b/hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm
deleted file mode 100644
index 8be92b5..0000000
--- a/hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm
+++ /dev/null
@@ -1,792 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop Streaming
- ---
- ---
- ${maven.build.timestamp}
-
-Hadoop Streaming
-
-%{toc|section=1|fromDepth=0|toDepth=4}
-
-* Hadoop Streaming
-
- Hadoop streaming is a utility that comes with the Hadoop distribution. The
- utility allows you to create and run Map/Reduce jobs with any executable or
- script as the mapper and/or the reducer. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input myInputDirs \
- -output myOutputDir \
- -mapper /bin/cat \
- -reducer /usr/bin/wc
-+---+
-
-* How Streaming Works
-
- In the above example, both the mapper and the reducer are executables that
- read the input from stdin (line by line) and emit the output to stdout. The
- utility will create a Map/Reduce job, submit the job to an appropriate
- cluster, and monitor the progress of the job until it completes.
-
- When an executable is specified for mappers, each mapper task will launch the
- executable as a separate process when the mapper is initialized. As the
- mapper task runs, it converts its inputs into lines and feed the lines to the
- stdin of the process. In the meantime, the mapper collects the line oriented
- outputs from the stdout of the process and converts each line into a
- key/value pair, which is collected as the output of the mapper. By default,
- the <prefix of a line up to the first tab character> is the <<<key>>> and the
- rest of the line (excluding the tab character) will be the <<<value>>>. If
- there is no tab character in the line, then entire line is considered as key
- and the value is null. However, this can be customized by setting
- <<<-inputformat>>> command option, as discussed later.
-
- When an executable is specified for reducers, each reducer task will launch
- the executable as a separate process then the reducer is initialized. As the
- reducer task runs, it converts its input key/values pairs into lines and
- feeds the lines to the stdin of the process. In the meantime, the reducer
- collects the line oriented outputs from the stdout of the process, converts
- each line into a key/value pair, which is collected as the output of the
- reducer. By default, the prefix of a line up to the first tab character is
- the key and the rest of the line (excluding the tab character) is the value.
- However, this can be customized by setting <<<-outputformat>>> command
- option, as discussed later.
-
- This is the basis for the communication protocol between the Map/Reduce
- framework and the streaming mapper/reducer.
-
- User can specify <<<stream.non.zero.exit.is.failure>>> as <<<true>>> or
- <<<false>>> to make a streaming task that exits with a non-zero status to be
- <<<Failure>>> or <<<Success>>> respectively. By default, streaming tasks
- exiting with non-zero status are considered to be failed tasks.
-
-* Streaming Command Options
-
- Streaming supports streaming command options as well as
- {{{Generic_Command_Options}generic command options}}. The general command
- line syntax is shown below.
-
- <<Note:>> Be sure to place the generic options before the streaming options,
- otherwise the command will fail. For an example, see
- {{{Making_Archives_Available_to_Tasks}Making Archives Available to Tasks}}.
-
-+---+
-hadoop command [genericOptions] [streamingOptions]
-+---+
-
- The Hadoop streaming command options are listed here:
-
-*-------------*--------------------*------------------------------------------*
-|| Parameter || Optional/Required || Description |
-*-------------+--------------------+------------------------------------------+
-| -input directoryname or filename | Required | Input location for mapper
-*-------------+--------------------+------------------------------------------+
-| -output directoryname | Required | Output location for reducer
-*-------------+--------------------+------------------------------------------+
-| -mapper executable or JavaClassName | Required | Mapper executable
-*-------------+--------------------+------------------------------------------+
-| -reducer executable or JavaClassName | Required | Reducer executable
-*-------------+--------------------+------------------------------------------+
-| -file filename | Optional | Make the mapper, reducer, or combiner executable
-| | | available locally on the compute nodes
-*-------------+--------------------+------------------------------------------+
-| -inputformat JavaClassName | Optional | Class you supply should return
-| | | key/value pairs of Text class. If not
-| | | specified, TextInputFormat is used as
-| | | the default
-*-------------+--------------------+------------------------------------------+
-| -outputformat JavaClassName | Optional | Class you supply should take
-| | | key/value pairs of Text class. If
-| | | not specified, TextOutputformat is
-| | | used as the default
-*-------------+--------------------+------------------------------------------+
-| -partitioner JavaClassName | Optional | Class that determines which reduce a
-| | | key is sent to
-*-------------+--------------------+------------------------------------------+
-| -combiner streamingCommand | Optional | Combiner executable for map output
-| or JavaClassName | |
-*-------------+--------------------+------------------------------------------+
-| -cmdenv name=value | Optional | Pass environment variable to streaming
-| | | commands
-*-------------+--------------------+------------------------------------------+
-| -inputreader | Optional | For backwards-compatibility: specifies a record
-| | | reader class (instead of an input format class)
-*-------------+--------------------+------------------------------------------+
-| -verbose | Optional | Verbose output
-*-------------+--------------------+------------------------------------------+
-| -lazyOutput | Optional | Create output lazily. For example, if the output
-| | | format is based on FileOutputFormat, the output file
-| | | is created only on the first call to Context.write
-*-------------+--------------------+------------------------------------------+
-| -numReduceTasks | Optional | Specify the number of reducers
-*-------------+--------------------+------------------------------------------+
-| -mapdebug | Optional | Script to call when map task fails
-*-------------+--------------------+------------------------------------------+
-| -reducedebug | Optional | Script to call when reduce task fails
-*-------------+--------------------+------------------------------------------+
-
-** Specifying a Java Class as the Mapper/Reducer
-
- You can supply a Java class as the mapper and/or the reducer.
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input myInputDirs \
- -output myOutputDir \
- -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat \
- -mapper org.apache.hadoop.mapred.lib.IdentityMapper \
- -reducer /usr/bin/wc
-+---+
-
- You can specify <<<stream.non.zero.exit.is.failure>>> as <<<true>>> or
- <<<false>>> to make a streaming task that exits with a non-zero status to be
- <<<Failure>>> or <<<Success>>> respectively. By default, streaming tasks
- exiting with non-zero status are considered to be failed tasks.
-
-** Packaging Files With Job Submissions
-
- You can specify any executable as the mapper and/or the reducer. The
- executables do not need to pre-exist on the machines in the cluster; however,
- if they don't, you will need to use "-file" option to tell the framework to
- pack your executable files as a part of job submission. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input myInputDirs \
- -output myOutputDir \
- -mapper myPythonScript.py \
- -reducer /usr/bin/wc \
- -file myPythonScript.py
-+---+
-
- The above example specifies a user defined Python executable as the mapper.
- The option "-file myPythonScript.py" causes the python executable shipped
- to the cluster machines as a part of job submission.
-
- In addition to executable files, you can also package other auxiliary files
- (such as dictionaries, configuration files, etc) that may be used by the
- mapper and/or the reducer. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input myInputDirs \
- -output myOutputDir \
- -mapper myPythonScript.py \
- -reducer /usr/bin/wc \
- -file myPythonScript.py \
- -file myDictionary.txt
-+---+
-
-** Specifying Other Plugins for Jobs
-
- Just as with a normal Map/Reduce job, you can specify other plugins for a
- streaming job:
-
-+---+
- -inputformat JavaClassName
- -outputformat JavaClassName
- -partitioner JavaClassName
- -combiner streamingCommand or JavaClassName
-+---+
-
- The class you supply for the input format should return key/value pairs of
- Text class. If you do not specify an input format class, the TextInputFormat
- is used as the default. Since the TextInputFormat returns keys of
- LongWritable class, which are actually not part of the input data, the keys
- will be discarded; only the values will be piped to the streaming mapper.
-
- The class you supply for the output format is expected to take key/value
- pairs of Text class. If you do not specify an output format class, the
- TextOutputFormat is used as the default.
-
-** Setting Environment Variables
-
- To set an environment variable in a streaming command use:
-
-+---+
- -cmdenv EXAMPLE_DIR=/home/example/dictionaries/
-+---+
-
-* Generic Command Options
-
- Streaming supports {{{Streaming_Command_Options}streaming command options}}
- as well as generic command options. The general command line syntax is shown
- below.
-
- <<Note:>> Be sure to place the generic options before the streaming options,
- otherwise the command will fail. For an example, see
- {{{Making_Archives_Available_to_Tasks}Making Archives Available to Tasks}}.
-
-+---+
-hadoop command [genericOptions] [streamingOptions]
-+---+
-
- The Hadoop generic command options you can use with streaming are listed
- here:
-
-*-------------*--------------------*------------------------------------------*
-|| Parameter || Optional/Required || Description |
-*-------------+--------------------+------------------------------------------+
-| -conf configuration_file | Optional | Specify an application configuration
-| | | file
-*-------------+--------------------+------------------------------------------+
-| -D property=value | Optional | Use value for given property
-*-------------+--------------------+------------------------------------------+
-| -fs host:port or local | Optional | Specify a namenode
-*-------------+--------------------+------------------------------------------+
-| -files | Optional | Specify comma-separated files to be copied to the
-| | | Map/Reduce cluster
-*-------------+--------------------+------------------------------------------+
-| -libjars | Optional | Specify comma-separated jar files to include in the
-| | | classpath
-*-------------+--------------------+------------------------------------------+
-| -archives | Optional | Specify comma-separated archives to be unarchived on
-| | | the compute machines
-*-------------+--------------------+------------------------------------------+
-
-** Specifying Configuration Variables with the -D Option
-
- You can specify additional configuration variables by using
- "-D \<property\>=\<value\>".
-
-*** Specifying Directories
-
- To change the local temp directory use:
-
-+---+
- -D dfs.data.dir=/tmp
-+---+
-
- To specify additional local temp directories use:
-
-+---+
- -D mapred.local.dir=/tmp/local
- -D mapred.system.dir=/tmp/system
- -D mapred.temp.dir=/tmp/temp
-+---+
-
- <<Note:>> For more details on job configuration parameters see:
- {{{./mapred-default.xml}mapred-default.xml}}
-
-*** Specifying Map-Only Jobs
-
- Often, you may want to process input data using a map function only. To do
- this, simply set <<<mapreduce.job.reduces>>> to zero. The Map/Reduce
- framework will not create any reducer tasks. Rather, the outputs of the
- mapper tasks will be the final output of the job.
-
-+---+
- -D mapreduce.job.reduces=0
-+---+
-
- To be backward compatible, Hadoop Streaming also supports the "-reducer NONE"
- option, which is equivalent to "-D mapreduce.job.reduces=0".
-
-*** Specifying the Number of Reducers
-
- To specify the number of reducers, for example two, use:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -D mapreduce.job.reduces=2 \
- -input myInputDirs \
- -output myOutputDir \
- -mapper /bin/cat \
- -reducer /usr/bin/wc
-+---+
-
-*** Customizing How Lines are Split into Key/Value Pairs
-
- As noted earlier, when the Map/Reduce framework reads a line from the stdout
- of the mapper, it splits the line into a key/value pair. By default, the
- prefix of the line up to the first tab character is the key and the rest of
- the line (excluding the tab character) is the value.
-
- However, you can customize this default. You can specify a field separator
- other than the tab character (the default), and you can specify the nth
- (n >= 1) character rather than the first character in a line (the default) as
- the separator between the key and value. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -D stream.map.output.field.separator=. \
- -D stream.num.map.output.key.fields=4 \
- -input myInputDirs \
- -output myOutputDir \
- -mapper /bin/cat \
- -reducer /bin/cat
-+---+
-
- In the above example, "-D stream.map.output.field.separator=." specifies "."
- as the field separator for the map outputs, and the prefix up to the fourth
- "." in a line will be the key and the rest of the line (excluding the fourth
- ".") will be the value. If a line has less than four "."s, then the whole
- line will be the key and the value will be an empty Text object (like the one
- created by new Text("")).
-
- Similarly, you can use "-D stream.reduce.output.field.separator=SEP" and
- "-D stream.num.reduce.output.fields=NUM" to specify the nth field separator
- in a line of the reduce outputs as the separator between the key and the
- value.
-
- Similarly, you can specify "stream.map.input.field.separator" and
- "stream.reduce.input.field.separator" as the input separator for Map/Reduce
- inputs. By default the separator is the tab character.
-
-** Working with Large Files and Archives
-
- The -files and -archives options allow you to make files and archives
- available to the tasks. The argument is a URI to the file or archive that you
- have already uploaded to HDFS. These files and archives are cached across
- jobs. You can retrieve the host and fs_port values from the fs.default.name
- config variable.
-
- <<Note:>> The -files and -archives options are generic options. Be sure to
- place the generic options before the command options, otherwise the command
- will fail.
-
-*** Making Files Available to Tasks
-
- The -files option creates a symlink in the current working directory of the
- tasks that points to the local copy of the file.
-
- In this example, Hadoop automatically creates a symlink named testfile.txt in
- the current working directory of the tasks. This symlink points to the local
- copy of testfile.txt.
-
-+---+
--files hdfs://host:fs_port/user/testfile.txt
-+---+
-
- User can specify a different symlink name for -files using #.
-
-+---+
--files hdfs://host:fs_port/user/testfile.txt#testfile
-+---+
-
- Multiple entries can be specified like this:
-
-+---+
--files hdfs://host:fs_port/user/testfile1.txt,hdfs://host:fs_port/user/testfile2.txt
-+---+
-
-*** Making Archives Available to Tasks
-
- The -archives option allows you to copy jars locally to the current working
- directory of tasks and automatically unjar the files.
-
- In this example, Hadoop automatically creates a symlink named testfile.jar in
- the current working directory of tasks. This symlink points to the directory
- that stores the unjarred contents of the uploaded jar file.
-
-+---+
--archives hdfs://host:fs_port/user/testfile.jar
-+---+
-
- User can specify a different symlink name for -archives using #.
-
-+---+
--archives hdfs://host:fs_port/user/testfile.tgz#tgzdir
-+---+
-
- In this example, the input.txt file has two lines specifying the names of the
- two files: cachedir.jar/cache.txt and cachedir.jar/cache2.txt. "cachedir.jar"
- is a symlink to the archived directory, which has the files "cache.txt" and
- "cache2.txt".
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -archives 'hdfs://hadoop-nn1.example.com/user/me/samples/cachefile/cachedir.jar' \
- -D mapreduce.job.maps=1 \
- -D mapreduce.job.reduces=1 \
- -D mapreduce.job.name="Experiment" \
- -input "/user/me/samples/cachefile/input.txt" \
- -output "/user/me/samples/cachefile/out" \
- -mapper "xargs cat" \
- -reducer "cat"
-
-$ ls test_jar/
-cache.txt cache2.txt
-
-$ jar cvf cachedir.jar -C test_jar/ .
-added manifest
-adding: cache.txt(in = 30) (out= 29)(deflated 3%)
-adding: cache2.txt(in = 37) (out= 35)(deflated 5%)
-
-$ hdfs dfs -put cachedir.jar samples/cachefile
-
-$ hdfs dfs -cat /user/me/samples/cachefile/input.txt
-cachedir.jar/cache.txt
-cachedir.jar/cache2.txt
-
-$ cat test_jar/cache.txt
-This is just the cache string
-
-$ cat test_jar/cache2.txt
-This is just the second cache string
-
-$ hdfs dfs -ls /user/me/samples/cachefile/out
-Found 2 items
--rw-r--r-- 1 me supergroup 0 2013-11-14 17:00 /user/me/samples/cachefile/out/_SUCCESS
--rw-r--r-- 1 me supergroup 69 2013-11-14 17:00 /user/me/samples/cachefile/out/part-00000
-
-$ hdfs dfs -cat /user/me/samples/cachefile/out/part-00000
-This is just the cache string
-This is just the second cache string
-+---+
-
-* More Usage Examples
-
-** Hadoop Partitioner Class
-
- Hadoop has a library class,
- {{{../../api/org/apache/hadoop/mapred/lib/KeyFieldBasedPartitioner.html}
- KeyFieldBasedPartitioner}}, that is useful for many applications. This class
- allows the Map/Reduce framework to partition the map outputs based on certain
- key fields, not the whole keys. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -D stream.map.output.field.separator=. \
- -D stream.num.map.output.key.fields=4 \
- -D map.output.key.field.separator=. \
- -D mapreduce.partition.keypartitioner.options=-k1,2 \
- -D mapreduce.job.reduces=12 \
- -input myInputDirs \
- -output myOutputDir \
- -mapper /bin/cat \
- -reducer /bin/cat \
- -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
-+---+
-
- Here, <-D stream.map.output.field.separator=.> and
- <-D stream.num.map.output.key.fields=4> are as explained in previous example.
- The two variables are used by streaming to identify the key/value pair of
- mapper.
-
- The map output keys of the above Map/Reduce job normally have four fields
- separated by ".". However, the Map/Reduce framework will partition the map
- outputs by the first two fields of the keys using the
- <-D mapred.text.key.partitioner.options=-k1,2> option. Here,
- <-D map.output.key.field.separator=.> specifies the separator for the
- partition. This guarantees that all the key/value pairs with the same first
- two fields in the keys will be partitioned into the same reducer.
-
- <This is effectively equivalent to specifying the first two fields as the
- primary key and the next two fields as the secondary. The primary key is used
- for partitioning, and the combination of the primary and secondary keys is
- used for sorting.> A simple illustration is shown here:
-
- Output of map (the keys)
-
-+---+
-11.12.1.2
-11.14.2.3
-11.11.4.1
-11.12.1.1
-11.14.2.2
-+---+
-
- Partition into 3 reducers (the first 2 fields are used as keys for partition)
-
-+---+
-11.11.4.1
------------
-11.12.1.2
-11.12.1.1
------------
-11.14.2.3
-11.14.2.2
-+---+
-
- Sorting within each partition for the reducer(all 4 fields used for sorting)
-
-+---+
-11.11.4.1
------------
-11.12.1.1
-11.12.1.2
------------
-11.14.2.2
-11.14.2.3
-+---+
-
-** Hadoop Comparator Class
-
- Hadoop has a library class,
- {{{../../api/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedComparator.html}
- KeyFieldBasedComparator}}, that is useful for many applications. This class
- provides a subset of features provided by the Unix/GNU Sort. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -D mapreduce.job.output.key.comparator.class=org.apache.hadoop.mapreduce.lib.partition.KeyFieldBasedComparator \
- -D stream.map.output.field.separator=. \
- -D stream.num.map.output.key.fields=4 \
- -D mapreduce.map.output.key.field.separator=. \
- -D mapreduce.partition.keycomparator.options=-k2,2nr \
- -D mapreduce.job.reduces=1 \
- -input myInputDirs \
- -output myOutputDir \
- -mapper /bin/cat \
- -reducer /bin/cat
-+---+
-
- The map output keys of the above Map/Reduce job normally have four fields
- separated by ".". However, the Map/Reduce framework will sort the outputs by
- the second field of the keys using the
- <-D mapreduce.partition.keycomparator.options=-k2,2nr> option. Here, <-n>
- specifies that the sorting is numerical sorting and <-r> specifies that the
- result should be reversed. A simple illustration is shown below:
-
- Output of map (the keys)
-
-+---+
-11.12.1.2
-11.14.2.3
-11.11.4.1
-11.12.1.1
-11.14.2.2
-+---+
-
- Sorting output for the reducer (where second field used for sorting)
-
-+---+
-11.14.2.3
-11.14.2.2
-11.12.1.2
-11.12.1.1
-11.11.4.1
-+---+
-
-** Hadoop Aggregate Package
-
- Hadoop has a library package called
- {{{../../org/apache/hadoop/mapred/lib/aggregate/package-summary.html}
- Aggregate}}. Aggregate provides a special reducer class and a special
- combiner class, and a list of simple aggregators that perform aggregations
- such as "sum", "max", "min" and so on over a sequence of values. Aggregate
- allows you to define a mapper plugin class that is expected to generate
- "aggregatable items" for each input key/value pair of the mappers. The
- combiner/reducer will aggregate those aggregatable items by invoking the
- appropriate aggregators.
-
- To use Aggregate, simply specify "-reducer aggregate":
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input myInputDirs \
- -output myOutputDir \
- -mapper myAggregatorForKeyCount.py \
- -reducer aggregate \
- -file myAggregatorForKeyCount.py \
-+---+
-
- The python program myAggregatorForKeyCount.py looks like:
-
-+---+
-#!/usr/bin/python
-
-import sys;
-
-def generateLongCountToken(id):
- return "LongValueSum:" + id + "\t" + "1"
-
-def main(argv):
- line = sys.stdin.readline();
- try:
- while line:
- line = line[:-1];
- fields = line.split("\t");
- print generateLongCountToken(fields[0]);
- line = sys.stdin.readline();
- except "end of file":
- return None
-if __name__ == "__main__":
- main(sys.argv)
-+---+
-
-** Hadoop Field Selection Class
-
- Hadoop has a library class,
- {{{../../api/org/apache/hadoop/mapred/lib/FieldSelectionMapReduce.html}
- FieldSelectionMapReduce}}, that effectively allows you to process text data
- like the unix "cut" utility. The map function defined in the class treats
- each input key/value pair as a list of fields. You can specify the field
- separator (the default is the tab character). You can select an arbitrary
- list of fields as the map output key, and an arbitrary list of fields as the
- map output value. Similarly, the reduce function defined in the class treats
- each input key/value pair as a list of fields. You can select an arbitrary
- list of fields as the reduce output key, and an arbitrary list of fields as
- the reduce output value. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -D mapreduce.map.output.key.field.separator=. \
- -D mapreduce.partition.keypartitioner.options=-k1,2 \
- -D mapreduce.fieldsel.data.field.separator=. \
- -D mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0- \
- -D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5- \
- -D mapreduce.map.output.key.class=org.apache.hadoop.io.Text \
- -D mapreduce.job.reduces=12 \
- -input myInputDirs \
- -output myOutputDir \
- -mapper org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
- -reducer org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
- -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
-+---+
-
- The option "-D
- mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0-" specifies
- key/value selection for the map outputs. Key selection spec and value
- selection spec are separated by ":". In this case, the map output key will
- consist of fields 6, 5, 1, 2, and 3. The map output value will consist of all
- fields (0- means field 0 and all the subsequent fields).
-
- The option "-D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5-"
- specifies key/value selection for the reduce outputs. In this case, the
- reduce output key will consist of fields 0, 1, 2 (corresponding to the
- original fields 6, 5, 1). The reduce output value will consist of all fields
- starting from field 5 (corresponding to all the original fields).
-
-* Frequently Asked Questions
-
-** How do I use Hadoop Streaming to run an arbitrary set of (semi) independent
- tasks?
-
- Often you do not need the full power of Map Reduce, but only need to run
- multiple instances of the same program - either on different parts of the
- data, or on the same data, but with different parameters. You can use Hadoop
- Streaming to do this.
-
-** How do I process files, one per map?
-
- As an example, consider the problem of zipping (compressing) a set of files
- across the hadoop cluster. You can achieve this by using Hadoop Streaming
- and custom mapper script:
-
- * Generate a file containing the full HDFS path of the input files. Each map
- task would get one file name as input.
-
- * Create a mapper script which, given a filename, will get the file to local
- disk, gzip the file and put it back in the desired output directory.
-
-** How many reducers should I use?
-
- See MapReduce Tutorial for details: {{{./MapReduceTutorial.html#Reducer}
- Reducer}}
-
-** If I set up an alias in my shell script, will that work after -mapper?
-
- For example, say I do: alias c1='cut -f1'. Will -mapper "c1" work?
-
- Using an alias will not work, but variable substitution is allowed as shown
- in this example:
-
-+---+
-$ hdfs dfs -cat /user/me/samples/student_marks
-alice 50
-bruce 70
-charlie 80
-dan 75
-
-$ c2='cut -f2'; hadoop jar hadoop-streaming-${project.version}.jar \
- -D mapreduce.job.name='Experiment' \
- -input /user/me/samples/student_marks \
- -output /user/me/samples/student_out \
- -mapper "$c2" -reducer 'cat'
-
-$ hdfs dfs -cat /user/me/samples/student_out/part-00000
-50
-70
-75
-80
-+---+
-
-** Can I use UNIX pipes?
-
- For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?
-
- Currently this does not work and gives an "java.io.IOException: Broken pipe"
- error. This is probably a bug that needs to be investigated.
-
-** What do I do if I get the "No space left on device" error?
-
- For example, when I run a streaming job by distributing large executables
- (for example, 3.6G) through the -file option, I get a "No space left on
- device" error.
-
- The jar packaging happens in a directory pointed to by the configuration
- variable stream.tmpdir. The default value of stream.tmpdir is /tmp. Set the
- value to a directory with more space:
-
-+---+
--D stream.tmpdir=/export/bigspace/...
-+---+
-
-** How do I specify multiple input directories?
-
- You can specify multiple input directories with multiple '-input' options:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input '/user/foo/dir1' -input '/user/foo/dir2' \
- (rest of the command)
-+---+
-
-** How do I generate output files with gzip format?
-
- Instead of plain text files, you can generate gzip files as your generated
- output. Pass '-D mapreduce.output.fileoutputformat.compress=true -D
- mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec'
- as option to your streaming job.
-
-** How do I provide my own input/output format with streaming?
-
- You can specify your own custom class by packing them and putting the custom
- jar to \$\{HADOOP_CLASSPATH\}.
-
-** How do I parse XML documents using streaming?
-
- You can use the record reader StreamXmlRecordReader to process XML documents.
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -inputreader "StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING" \
- (rest of the command)
-+---+
-
- Anything found between BEGIN_STRING and END_STRING would be treated as one
- record for map tasks.
-
-** How do I update counters in streaming applications?
-
- A streaming process can use the stderr to emit counter information.
- <<<reporter:counter:\<group\>,\<counter\>,\<amount\>>>> should be sent to
- stderr to update the counter.
-
-** How do I update status in streaming applications?
-
- A streaming process can use the stderr to emit status information. To set a
- status, <<<reporter:status:\<message\>>>> should be sent to stderr.
-
-** How do I get the Job variables in a streaming job's mapper/reducer?
-
- See {{{./MapReduceTutorial.html#Configured_Parameters}
- Configured Parameters}}. During the execution of a streaming job, the names
- of the "mapred" parameters are transformed. The dots ( . ) become underscores
- ( _ ). For example, mapreduce.job.id becomes mapreduce_job_id and
- mapreduce.job.jar becomes mapreduce_job_jar. In your code, use the parameter
- names with the underscores.