You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-commits@hadoop.apache.org by sh...@apache.org on 2009/09/22 00:33:12 UTC
svn commit: r817449 [6/8] - in /hadoop/hdfs/branches/HDFS-265: ./
.eclipse.templates/.launches/ lib/ src/contrib/block_forensics/
src/contrib/block_forensics/client/ src/contrib/block_forensics/ivy/
src/contrib/block_forensics/src/java/org/apache/hadoo...
Added: hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
URL: http://svn.apache.org/viewvc/hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml?rev=817449&view=auto
==============================================================================
--- hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml (added)
+++ hadoop/hdfs/branches/HDFS-265/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml Mon Sep 21 22:33:09 2009
@@ -0,0 +1,3131 @@
+<?xml version="1.0"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd">
+
+<document>
+
+ <header>
+ <title>Map/Reduce Tutorial</title>
+ </header>
+
+ <body>
+
+ <section>
+ <title>Purpose</title>
+
+ <p>This document comprehensively describes all user-facing facets of the
+ Hadoop Map/Reduce framework and serves as a tutorial.
+ </p>
+ </section>
+
+ <section>
+ <title>Pre-requisites</title>
+
+ <p>Ensure that Hadoop is installed, configured and is running. More
+ details:</p>
+ <ul>
+ <li>
+ <a href="quickstart.html">Hadoop Quick Start</a> for first-time users.
+ </li>
+ <li>
+ <a href="cluster_setup.html">Hadoop Cluster Setup</a> for large,
+ distributed clusters.
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <title>Overview</title>
+
+ <p>Hadoop Map/Reduce is a software framework for easily writing
+ applications which process vast amounts of data (multi-terabyte data-sets)
+ in-parallel on large clusters (thousands of nodes) of commodity
+ hardware in a reliable, fault-tolerant manner.</p>
+
+ <p>A Map/Reduce <em>job</em> usually splits the input data-set into
+ independent chunks which are processed by the <em>map tasks</em> in a
+ completely parallel manner. The framework sorts the outputs of the maps,
+ which are then input to the <em>reduce tasks</em>. Typically both the
+ input and the output of the job are stored in a file-system. The framework
+ takes care of scheduling tasks, monitoring them and re-executes the failed
+ tasks.</p>
+
+ <p>Typically the compute nodes and the storage nodes are the same, that is,
+ the Map/Reduce framework and the Hadoop Distributed File System (see <a href="hdfs_design.html">HDFS Architecture </a>)
+ are running on the same set of nodes. This configuration
+ allows the framework to effectively schedule tasks on the nodes where data
+ is already present, resulting in very high aggregate bandwidth across the
+ cluster.</p>
+
+ <p>The Map/Reduce framework consists of a single master
+ <code>JobTracker</code> and one slave <code>TaskTracker</code> per
+ cluster-node. The master is responsible for scheduling the jobs' component
+ tasks on the slaves, monitoring them and re-executing the failed tasks. The
+ slaves execute the tasks as directed by the master.</p>
+
+ <p>Minimally, applications specify the input/output locations and supply
+ <em>map</em> and <em>reduce</em> functions via implementations of
+ appropriate interfaces and/or abstract-classes. These, and other job
+ parameters, comprise the <em>job configuration</em>. The Hadoop
+ <em>job client</em> then submits the job (jar/executable etc.) and
+ configuration to the <code>JobTracker</code> which then assumes the
+ responsibility of distributing the software/configuration to the slaves,
+ scheduling tasks and monitoring them, providing status and diagnostic
+ information to the job-client.</p>
+
+ <p>Although the Hadoop framework is implemented in Java<sup>TM</sup>,
+ Map/Reduce applications need not be written in Java.</p>
+ <ul>
+ <li>
+ <a href="ext:api/org/apache/hadoop/streaming/package-summary">
+ Hadoop Streaming</a> is a utility which allows users to create and run
+ jobs with any executables (e.g. shell utilities) as the mapper and/or
+ the reducer.
+ </li>
+ <li>
+ <a href="ext:api/org/apache/hadoop/mapred/pipes/package-summary">
+ Hadoop Pipes</a> is a <a href="http://www.swig.org/">SWIG</a>-
+ compatible <em>C++ API</em> to implement Map/Reduce applications (non
+ JNI<sup>TM</sup> based).
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <title>Inputs and Outputs</title>
+
+ <p>The Map/Reduce framework operates exclusively on
+ <code><key, value></code> pairs, that is, the framework views the
+ input to the job as a set of <code><key, value></code> pairs and
+ produces a set of <code><key, value></code> pairs as the output of
+ the job, conceivably of different types.</p>
+
+ <p>The <code>key</code> and <code>value</code> classes have to be
+ serializable by the framework and hence need to implement the
+ <a href="ext:api/org/apache/hadoop/io/writable">Writable</a>
+ interface. Additionally, the <code>key</code> classes have to implement the
+ <a href="ext:api/org/apache/hadoop/io/writablecomparable">
+ WritableComparable</a> interface to facilitate sorting by the framework.
+ </p>
+
+ <p>Input and Output types of a Map/Reduce job:</p>
+ <p>
+ (input) <code><k1, v1></code>
+ ->
+ <strong>map</strong>
+ ->
+ <code><k2, v2></code>
+ ->
+ <strong>combine</strong>
+ ->
+ <code><k2, v2></code>
+ ->
+ <strong>reduce</strong>
+ ->
+ <code><k3, v3></code> (output)
+ </p>
+ </section>
+
+ <section>
+ <title>Example: WordCount v1.0</title>
+
+ <p>Before we jump into the details, lets walk through an example Map/Reduce
+ application to get a flavour for how they work.</p>
+
+ <p><code>WordCount</code> is a simple application that counts the number of
+ occurences of each word in a given input set.</p>
+
+ <p>This works with a local-standalone, pseudo-distributed or fully-distributed
+ Hadoop installation(see <a href="quickstart.html"> Hadoop Quick Start</a>).</p>
+
+ <section>
+ <title>Source Code</title>
+
+ <table>
+ <tr>
+ <th></th>
+ <th>WordCount.java</th>
+ </tr>
+ <tr>
+ <td>1.</td>
+ <td>
+ <code>package org.myorg;</code>
+ </td>
+ </tr>
+ <tr>
+ <td>2.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>3.</td>
+ <td>
+ <code>import java.io.IOException;</code>
+ </td>
+ </tr>
+ <tr>
+ <td>4.</td>
+ <td>
+ <code>import java.util.*;</code>
+ </td>
+ </tr>
+ <tr>
+ <td>5.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>6.</td>
+ <td>
+ <code>import org.apache.hadoop.fs.Path;</code>
+ </td>
+ </tr>
+ <tr>
+ <td>7.</td>
+ <td>
+ <code>import org.apache.hadoop.conf.*;</code>
+ </td>
+ </tr>
+ <tr>
+ <td>8.</td>
+ <td>
+ <code>import org.apache.hadoop.io.*;</code>
+ </td>
+ </tr>
+ <tr>
+ <td>9.</td>
+ <td>
+ <code>import org.apache.hadoop.mapred.*;</code>
+ </td>
+ </tr>
+ <tr>
+ <td>10.</td>
+ <td>
+ <code>import org.apache.hadoop.util.*;</code>
+ </td>
+ </tr>
+ <tr>
+ <td>11.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>12.</td>
+ <td>
+ <code>public class WordCount {</code>
+ </td>
+ </tr>
+ <tr>
+ <td>13.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>14.</td>
+ <td>
+
+ <code>
+ public static class Map extends MapReduceBase
+ implements Mapper<LongWritable, Text, Text, IntWritable> {
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>15.</td>
+ <td>
+
+ <code>
+ private final static IntWritable one = new IntWritable(1);
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>16.</td>
+ <td>
+
+ <code>private Text word = new Text();</code>
+ </td>
+ </tr>
+ <tr>
+ <td>17.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>18.</td>
+ <td>
+
+ <code>
+ public void map(LongWritable key, Text value,
+ OutputCollector<Text, IntWritable> output,
+ Reporter reporter) throws IOException {
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>19.</td>
+ <td>
+
+ <code>String line = value.toString();</code>
+ </td>
+ </tr>
+ <tr>
+ <td>20.</td>
+ <td>
+
+ <code>StringTokenizer tokenizer = new StringTokenizer(line);</code>
+ </td>
+ </tr>
+ <tr>
+ <td>21.</td>
+ <td>
+
+ <code>while (tokenizer.hasMoreTokens()) {</code>
+ </td>
+ </tr>
+ <tr>
+ <td>22.</td>
+ <td>
+
+ <code>word.set(tokenizer.nextToken());</code>
+ </td>
+ </tr>
+ <tr>
+ <td>23.</td>
+ <td>
+
+ <code>output.collect(word, one);</code>
+ </td>
+ </tr>
+ <tr>
+ <td>24.</td>
+ <td>
+
+ <code>}</code>
+ </td>
+ </tr>
+ <tr>
+ <td>25.</td>
+ <td>
+
+ <code>}</code>
+ </td>
+ </tr>
+ <tr>
+ <td>26.</td>
+ <td>
+
+ <code>}</code>
+ </td>
+ </tr>
+ <tr>
+ <td>27.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>28.</td>
+ <td>
+
+ <code>
+ public static class Reduce extends MapReduceBase implements
+ Reducer<Text, IntWritable, Text, IntWritable> {
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>29.</td>
+ <td>
+
+ <code>
+ public void reduce(Text key, Iterator<IntWritable> values,
+ OutputCollector<Text, IntWritable> output,
+ Reporter reporter) throws IOException {
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>30.</td>
+ <td>
+
+ <code>int sum = 0;</code>
+ </td>
+ </tr>
+ <tr>
+ <td>31.</td>
+ <td>
+
+ <code>while (values.hasNext()) {</code>
+ </td>
+ </tr>
+ <tr>
+ <td>32.</td>
+ <td>
+
+ <code>sum += values.next().get();</code>
+ </td>
+ </tr>
+ <tr>
+ <td>33.</td>
+ <td>
+
+ <code>}</code>
+ </td>
+ </tr>
+ <tr>
+ <td>34.</td>
+ <td>
+
+ <code>output.collect(key, new IntWritable(sum));</code>
+ </td>
+ </tr>
+ <tr>
+ <td>35.</td>
+ <td>
+
+ <code>}</code>
+ </td>
+ </tr>
+ <tr>
+ <td>36.</td>
+ <td>
+
+ <code>}</code>
+ </td>
+ </tr>
+ <tr>
+ <td>37.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>38.</td>
+ <td>
+
+ <code>
+ public static void main(String[] args) throws Exception {
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>39.</td>
+ <td>
+
+ <code>
+ JobConf conf = new JobConf(WordCount.class);
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>40.</td>
+ <td>
+
+ <code>conf.setJobName("wordcount");</code>
+ </td>
+ </tr>
+ <tr>
+ <td>41.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>42.</td>
+ <td>
+
+ <code>conf.setOutputKeyClass(Text.class);</code>
+ </td>
+ </tr>
+ <tr>
+ <td>43.</td>
+ <td>
+
+ <code>conf.setOutputValueClass(IntWritable.class);</code>
+ </td>
+ </tr>
+ <tr>
+ <td>44.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>45.</td>
+ <td>
+
+ <code>conf.setMapperClass(Map.class);</code>
+ </td>
+ </tr>
+ <tr>
+ <td>46.</td>
+ <td>
+
+ <code>conf.setCombinerClass(Reduce.class);</code>
+ </td>
+ </tr>
+ <tr>
+ <td>47.</td>
+ <td>
+
+ <code>conf.setReducerClass(Reduce.class);</code>
+ </td>
+ </tr>
+ <tr>
+ <td>48.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>49.</td>
+ <td>
+
+ <code>conf.setInputFormat(TextInputFormat.class);</code>
+ </td>
+ </tr>
+ <tr>
+ <td>50.</td>
+ <td>
+
+ <code>conf.setOutputFormat(TextOutputFormat.class);</code>
+ </td>
+ </tr>
+ <tr>
+ <td>51.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>52.</td>
+ <td>
+
+ <code>FileInputFormat.setInputPaths(conf, new Path(args[0]));</code>
+ </td>
+ </tr>
+ <tr>
+ <td>53.</td>
+ <td>
+
+ <code>FileOutputFormat.setOutputPath(conf, new Path(args[1]));</code>
+ </td>
+ </tr>
+ <tr>
+ <td>54.</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>55.</td>
+ <td>
+
+ <code>JobClient.runJob(conf);</code>
+ </td>
+ </tr>
+ <tr>
+ <td>57.</td>
+ <td>
+
+ <code>}</code>
+ </td>
+ </tr>
+ <tr>
+ <td>58.</td>
+ <td>
+ <code>}</code>
+ </td>
+ </tr>
+ <tr>
+ <td>59.</td>
+ <td></td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title>Usage</title>
+
+ <p>Assuming <code>HADOOP_HOME</code> is the root of the installation and
+ <code>HADOOP_VERSION</code> is the Hadoop version installed, compile
+ <code>WordCount.java</code> and create a jar:</p>
+ <p>
+ <code>$ mkdir wordcount_classes</code><br/>
+ <code>
+ $ javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar
+ -d wordcount_classes WordCount.java
+ </code><br/>
+ <code>$ jar -cvf /usr/joe/wordcount.jar -C wordcount_classes/ .</code>
+ </p>
+
+ <p>Assuming that:</p>
+ <ul>
+ <li>
+ <code>/usr/joe/wordcount/input</code> - input directory in HDFS
+ </li>
+ <li>
+ <code>/usr/joe/wordcount/output</code> - output directory in HDFS
+ </li>
+ </ul>
+
+ <p>Sample text-files as input:</p>
+ <p>
+ <code>$ bin/hadoop dfs -ls /usr/joe/wordcount/input/</code><br/>
+ <code>/usr/joe/wordcount/input/file01</code><br/>
+ <code>/usr/joe/wordcount/input/file02</code><br/>
+ <br/>
+ <code>$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file01</code><br/>
+ <code>Hello World Bye World</code><br/>
+ <br/>
+ <code>$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file02</code><br/>
+ <code>Hello Hadoop Goodbye Hadoop</code>
+ </p>
+
+ <p>Run the application:</p>
+ <p>
+ <code>
+ $ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount
+ /usr/joe/wordcount/input /usr/joe/wordcount/output
+ </code>
+ </p>
+
+ <p>Output:</p>
+ <p>
+ <code>
+ $ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
+ </code>
+ <br/>
+ <code>Bye 1</code><br/>
+ <code>Goodbye 1</code><br/>
+ <code>Hadoop 2</code><br/>
+ <code>Hello 2</code><br/>
+ <code>World 2</code><br/>
+ </p>
+
+ <p> Applications can specify a comma separated list of paths which
+ would be present in the current working directory of the task
+ using the option <code>-files</code>. The <code>-libjars</code>
+ option allows applications to add jars to the classpaths of the maps
+ and reduces. The <code>-archives</code> allows them to pass archives
+ as arguments that are unzipped/unjarred and a link with name of the
+ jar/zip are created in the current working directory of tasks. More
+ details about the command line options are available at
+ <a href="commands_manual.html"> Hadoop Command Guide.</a></p>
+
+ <p>Running <code>wordcount</code> example with
+ <code>-libjars</code> and <code>-files</code>:<br/>
+ <code> hadoop jar hadoop-examples.jar wordcount -files cachefile.txt
+ -libjars mylib.jar input output </code>
+ </p>
+ </section>
+
+ <section>
+ <title>Walk-through</title>
+
+ <p>The <code>WordCount</code> application is quite straight-forward.</p>
+
+ <p>The <code>Mapper</code> implementation (lines 14-26), via the
+ <code>map</code> method (lines 18-25), processes one line at a time,
+ as provided by the specified <code>TextInputFormat</code> (line 49).
+ It then splits the line into tokens separated by whitespaces, via the
+ <code>StringTokenizer</code>, and emits a key-value pair of
+ <code>< <word>, 1></code>.</p>
+
+ <p>
+ For the given sample input the first map emits:<br/>
+ <code>< Hello, 1></code><br/>
+ <code>< World, 1></code><br/>
+ <code>< Bye, 1></code><br/>
+ <code>< World, 1></code><br/>
+ </p>
+
+ <p>
+ The second map emits:<br/>
+ <code>< Hello, 1></code><br/>
+ <code>< Hadoop, 1></code><br/>
+ <code>< Goodbye, 1></code><br/>
+ <code>< Hadoop, 1></code><br/>
+ </p>
+
+ <p>We'll learn more about the number of maps spawned for a given job, and
+ how to control them in a fine-grained manner, a bit later in the
+ tutorial.</p>
+
+ <p><code>WordCount</code> also specifies a <code>combiner</code> (line
+ 46). Hence, the output of each map is passed through the local combiner
+ (which is same as the <code>Reducer</code> as per the job
+ configuration) for local aggregation, after being sorted on the
+ <em>key</em>s.</p>
+
+ <p>
+ The output of the first map:<br/>
+ <code>< Bye, 1></code><br/>
+ <code>< Hello, 1></code><br/>
+ <code>< World, 2></code><br/>
+ </p>
+
+ <p>
+ The output of the second map:<br/>
+ <code>< Goodbye, 1></code><br/>
+ <code>< Hadoop, 2></code><br/>
+ <code>< Hello, 1></code><br/>
+ </p>
+
+ <p>The <code>Reducer</code> implementation (lines 28-36), via the
+ <code>reduce</code> method (lines 29-35) just sums up the values,
+ which are the occurence counts for each key (i.e. words in this example).
+ </p>
+
+ <p>
+ Thus the output of the job is:<br/>
+ <code>< Bye, 1></code><br/>
+ <code>< Goodbye, 1></code><br/>
+ <code>< Hadoop, 2></code><br/>
+ <code>< Hello, 2></code><br/>
+ <code>< World, 2></code><br/>
+ </p>
+
+ <p>The <code>run</code> method specifies various facets of the job, such
+ as the input/output paths (passed via the command line), key/value
+ types, input/output formats etc., in the <code>JobConf</code>.
+ It then calls the <code>JobClient.runJob</code> (line 55) to submit the
+ and monitor its progress.</p>
+
+ <p>We'll learn more about <code>JobConf</code>, <code>JobClient</code>,
+ <code>Tool</code> and other interfaces and classes a bit later in the
+ tutorial.</p>
+ </section>
+ </section>
+
+ <section>
+ <title>Map/Reduce - User Interfaces</title>
+
+ <p>This section provides a reasonable amount of detail on every user-facing
+ aspect of the Map/Reduce framwork. This should help users implement,
+ configure and tune their jobs in a fine-grained manner. However, please
+ note that the javadoc for each class/interface remains the most
+ comprehensive documentation available; this is only meant to be a tutorial.
+ </p>
+
+ <p>Let us first take the <code>Mapper</code> and <code>Reducer</code>
+ interfaces. Applications typically implement them to provide the
+ <code>map</code> and <code>reduce</code> methods.</p>
+
+ <p>We will then discuss other core interfaces including
+ <code>JobConf</code>, <code>JobClient</code>, <code>Partitioner</code>,
+ <code>OutputCollector</code>, <code>Reporter</code>,
+ <code>InputFormat</code>, <code>OutputFormat</code>,
+ <code>OutputCommitter</code> and others.</p>
+
+ <p>Finally, we will wrap up by discussing some useful features of the
+ framework such as the <code>DistributedCache</code>,
+ <code>IsolationRunner</code> etc.</p>
+
+ <section>
+ <title>Payload</title>
+
+ <p>Applications typically implement the <code>Mapper</code> and
+ <code>Reducer</code> interfaces to provide the <code>map</code> and
+ <code>reduce</code> methods. These form the core of the job.</p>
+
+ <section>
+ <title>Mapper</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/mapper">
+ Mapper</a> maps input key/value pairs to a set of intermediate
+ key/value pairs.</p>
+
+ <p>Maps are the individual tasks that transform input records into
+ intermediate records. The transformed intermediate records do not need
+ to be of the same type as the input records. A given input pair may
+ map to zero or many output pairs.</p>
+
+ <p>The Hadoop Map/Reduce framework spawns one map task for each
+ <code>InputSplit</code> generated by the <code>InputFormat</code> for
+ the job.</p>
+
+ <p>Overall, <code>Mapper</code> implementations are passed the
+ <code>JobConf</code> for the job via the
+ <a href="ext:api/org/apache/hadoop/mapred/jobconfigurable/configure">
+ JobConfigurable.configure(JobConf)</a> method and override it to
+ initialize themselves. The framework then calls
+ <a href="ext:api/org/apache/hadoop/mapred/mapper/map">
+ map(WritableComparable, Writable, OutputCollector, Reporter)</a> for
+ each key/value pair in the <code>InputSplit</code> for that task.
+ Applications can then override the
+ <a href="ext:api/org/apache/hadoop/io/closeable/close">
+ Closeable.close()</a> method to perform any required cleanup.</p>
+
+
+ <p>Output pairs do not need to be of the same types as input pairs. A
+ given input pair may map to zero or many output pairs. Output pairs
+ are collected with calls to
+ <a href="ext:api/org/apache/hadoop/mapred/outputcollector/collect">
+ OutputCollector.collect(WritableComparable,Writable)</a>.</p>
+
+ <p>Applications can use the <code>Reporter</code> to report
+ progress, set application-level status messages and update
+ <code>Counters</code>, or just indicate that they are alive.</p>
+
+ <p>All intermediate values associated with a given output key are
+ subsequently grouped by the framework, and passed to the
+ <code>Reducer</code>(s) to determine the final output. Users can
+ control the grouping by specifying a <code>Comparator</code> via
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setoutputkeycomparatorclass">
+ JobConf.setOutputKeyComparatorClass(Class)</a>.</p>
+
+ <p>The <code>Mapper</code> outputs are sorted and then
+ partitioned per <code>Reducer</code>. The total number of partitions is
+ the same as the number of reduce tasks for the job. Users can control
+ which keys (and hence records) go to which <code>Reducer</code> by
+ implementing a custom <code>Partitioner</code>.</p>
+
+ <p>Users can optionally specify a <code>combiner</code>, via
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setcombinerclass">
+ JobConf.setCombinerClass(Class)</a>, to perform local aggregation of
+ the intermediate outputs, which helps to cut down the amount of data
+ transferred from the <code>Mapper</code> to the <code>Reducer</code>.
+ </p>
+
+ <p>The intermediate, sorted outputs are always stored in a simple
+ (key-len, key, value-len, value) format.
+ Applications can control if, and how, the
+ intermediate outputs are to be compressed and the
+ <a href="ext:api/org/apache/hadoop/io/compress/compressioncodec">
+ CompressionCodec</a> to be used via the <code>JobConf</code>.
+ </p>
+
+ <section>
+ <title>How Many Maps?</title>
+
+ <p>The number of maps is usually driven by the total size of the
+ inputs, that is, the total number of blocks of the input files.</p>
+
+ <p>The right level of parallelism for maps seems to be around 10-100
+ maps per-node, although it has been set up to 300 maps for very
+ cpu-light map tasks. Task setup takes awhile, so it is best if the
+ maps take at least a minute to execute.</p>
+
+ <p>Thus, if you expect 10TB of input data and have a blocksize of
+ <code>128MB</code>, you'll end up with 82,000 maps, unless
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setnummaptasks">
+ setNumMapTasks(int)</a> (which only provides a hint to the framework)
+ is used to set it even higher.</p>
+ </section>
+ </section>
+
+ <section>
+ <title>Reducer</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/reducer">
+ Reducer</a> reduces a set of intermediate values which share a key to
+ a smaller set of values.</p>
+
+ <p>The number of reduces for the job is set by the user
+ via <a href="ext:api/org/apache/hadoop/mapred/jobconf/setnumreducetasks">
+ JobConf.setNumReduceTasks(int)</a>.</p>
+
+ <p>Overall, <code>Reducer</code> implementations are passed the
+ <code>JobConf</code> for the job via the
+ <a href="ext:api/org/apache/hadoop/mapred/jobconfigurable/configure">
+ JobConfigurable.configure(JobConf)</a> method and can override it to
+ initialize themselves. The framework then calls
+ <a href="ext:api/org/apache/hadoop/mapred/reducer/reduce">
+ reduce(WritableComparable, Iterator, OutputCollector, Reporter)</a>
+ method for each <code><key, (list of values)></code>
+ pair in the grouped inputs. Applications can then override the
+ <a href="ext:api/org/apache/hadoop/io/closeable/close">
+ Closeable.close()</a> method to perform any required cleanup.</p>
+
+ <p><code>Reducer</code> has 3 primary phases: shuffle, sort and reduce.
+ </p>
+
+ <section>
+ <title>Shuffle</title>
+
+ <p>Input to the <code>Reducer</code> is the sorted output of the
+ mappers. In this phase the framework fetches the relevant partition
+ of the output of all the mappers, via HTTP.</p>
+ </section>
+
+ <section>
+ <title>Sort</title>
+
+ <p>The framework groups <code>Reducer</code> inputs by keys (since
+ different mappers may have output the same key) in this stage.</p>
+
+ <p>The shuffle and sort phases occur simultaneously; while
+ map-outputs are being fetched they are merged.</p>
+
+ <section>
+ <title>Secondary Sort</title>
+
+ <p>If equivalence rules for grouping the intermediate keys are
+ required to be different from those for grouping keys before
+ reduction, then one may specify a <code>Comparator</code> via
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setoutputvaluegroupingcomparator">
+ JobConf.setOutputValueGroupingComparator(Class)</a>. Since
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setoutputkeycomparatorclass">
+ JobConf.setOutputKeyComparatorClass(Class)</a> can be used to
+ control how intermediate keys are grouped, these can be used in
+ conjunction to simulate <em>secondary sort on values</em>.</p>
+ </section>
+ </section>
+
+ <section>
+ <title>Reduce</title>
+
+ <p>In this phase the
+ <a href="ext:api/org/apache/hadoop/mapred/reducer/reduce">
+ reduce(WritableComparable, Iterator, OutputCollector, Reporter)</a>
+ method is called for each <code><key, (list of values)></code>
+ pair in the grouped inputs.</p>
+
+ <p>The output of the reduce task is typically written to the
+ <a href="ext:api/org/apache/hadoop/fs/filesystem">
+ FileSystem</a> via
+ <a href="ext:api/org/apache/hadoop/mapred/outputcollector/collect">
+ OutputCollector.collect(WritableComparable, Writable)</a>.</p>
+
+ <p>Applications can use the <code>Reporter</code> to report
+ progress, set application-level status messages and update
+ <code>Counters</code>, or just indicate that they are alive.</p>
+
+ <p>The output of the <code>Reducer</code> is <em>not sorted</em>.</p>
+ </section>
+
+ <section>
+ <title>How Many Reduces?</title>
+
+ <p>The right number of reduces seems to be <code>0.95</code> or
+ <code>1.75</code> multiplied by (<<em>no. of nodes</em>> *
+ <code>mapred.tasktracker.reduce.tasks.maximum</code>).</p>
+
+ <p>With <code>0.95</code> all of the reduces can launch immediately
+ and start transfering map outputs as the maps finish. With
+ <code>1.75</code> the faster nodes will finish their first round of
+ reduces and launch a second wave of reduces doing a much better job
+ of load balancing.</p>
+
+ <p>Increasing the number of reduces increases the framework overhead,
+ but increases load balancing and lowers the cost of failures.</p>
+
+ <p>The scaling factors above are slightly less than whole numbers to
+ reserve a few reduce slots in the framework for speculative-tasks and
+ failed tasks.</p>
+ </section>
+
+ <section>
+ <title>Reducer NONE</title>
+
+ <p>It is legal to set the number of reduce-tasks to <em>zero</em> if
+ no reduction is desired.</p>
+
+ <p>In this case the outputs of the map-tasks go directly to the
+ <code>FileSystem</code>, into the output path set by
+ <a href="ext:api/org/apache/hadoop/mapred/fileoutputformat/setoutputpath">
+ setOutputPath(Path)</a>. The framework does not sort the
+ map-outputs before writing them out to the <code>FileSystem</code>.
+ </p>
+ </section>
+ </section>
+
+ <section>
+ <title>Partitioner</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/partitioner">
+ Partitioner</a> partitions the key space.</p>
+
+ <p>Partitioner controls the partitioning of the keys of the
+ intermediate map-outputs. The key (or a subset of the key) is used to
+ derive the partition, typically by a <em>hash function</em>. The total
+ number of partitions is the same as the number of reduce tasks for the
+ job. Hence this controls which of the <code>m</code> reduce tasks the
+ intermediate key (and hence the record) is sent to for reduction.</p>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/lib/hashpartitioner">
+ HashPartitioner</a> is the default <code>Partitioner</code>.</p>
+ </section>
+
+ <section>
+ <title>Reporter</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/reporter">
+ Reporter</a> is a facility for Map/Reduce applications to report
+ progress, set application-level status messages and update
+ <code>Counters</code>.</p>
+
+ <p><code>Mapper</code> and <code>Reducer</code> implementations can use
+ the <code>Reporter</code> to report progress or just indicate
+ that they are alive. In scenarios where the application takes a
+ significant amount of time to process individual key/value pairs,
+ this is crucial since the framework might assume that the task has
+ timed-out and kill that task. Another way to avoid this is to
+ set the configuration parameter <code>mapred.task.timeout</code> to a
+ high-enough value (or even set it to <em>zero</em> for no time-outs).
+ </p>
+
+ <p>Applications can also update <code>Counters</code> using the
+ <code>Reporter</code>.</p>
+ </section>
+
+ <section>
+ <title>OutputCollector</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/outputcollector">
+ OutputCollector</a> is a generalization of the facility provided by
+ the Map/Reduce framework to collect data output by the
+ <code>Mapper</code> or the <code>Reducer</code> (either the
+ intermediate outputs or the output of the job).</p>
+ </section>
+
+ <p>Hadoop Map/Reduce comes bundled with a
+ <a href="ext:api/org/apache/hadoop/mapred/lib/package-summary">
+ library</a> of generally useful mappers, reducers, and partitioners.</p>
+ </section>
+
+ <section>
+ <title>Job Configuration</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/jobconf">
+ JobConf</a> represents a Map/Reduce job configuration.</p>
+
+ <p><code>JobConf</code> is the primary interface for a user to describe
+ a Map/Reduce job to the Hadoop framework for execution. The framework
+ tries to faithfully execute the job as described by <code>JobConf</code>,
+ however:</p>
+ <ul>
+ <li>f
+ Some configuration parameters may have been marked as
+ <a href="ext:api/org/apache/hadoop/conf/configuration/final_parameters">
+ final</a> by administrators and hence cannot be altered.
+ </li>
+ <li>
+ While some job parameters are straight-forward to set (e.g.
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setnumreducetasks">
+ setNumReduceTasks(int)</a>), other parameters interact subtly with
+ the rest of the framework and/or job configuration and are
+ more complex to set (e.g.
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setnummaptasks">
+ setNumMapTasks(int)</a>).
+ </li>
+ </ul>
+
+ <p><code>JobConf</code> is typically used to specify the
+ <code>Mapper</code>, combiner (if any), <code>Partitioner</code>,
+ <code>Reducer</code>, <code>InputFormat</code>,
+ <code>OutputFormat</code> and <code>OutputCommitter</code>
+ implementations. <code>JobConf</code> also
+ indicates the set of input files
+ (<a href="ext:api/org/apache/hadoop/mapred/fileinputformat/setinputpaths">setInputPaths(JobConf, Path...)</a>
+ /<a href="ext:api/org/apache/hadoop/mapred/fileinputformat/addinputpath">addInputPath(JobConf, Path)</a>)
+ and (<a href="ext:api/org/apache/hadoop/mapred/fileinputformat/setinputpathstring">setInputPaths(JobConf, String)</a>
+ /<a href="ext:api/org/apache/hadoop/mapred/fileinputformat/addinputpathstring">addInputPaths(JobConf, String)</a>)
+ and where the output files should be written
+ (<a href="ext:api/org/apache/hadoop/mapred/fileoutputformat/setoutputpath">setOutputPath(Path)</a>).</p>
+
+ <p>Optionally, <code>JobConf</code> is used to specify other advanced
+ facets of the job such as the <code>Comparator</code> to be used, files
+ to be put in the <code>DistributedCache</code>, whether intermediate
+ and/or job outputs are to be compressed (and how), debugging via
+ user-provided scripts
+ (<a href="ext:api/org/apache/hadoop/mapred/jobconf/setmapdebugscript">setMapDebugScript(String)</a>/<a href="ext:api/org/apache/hadoop/mapred/jobconf/setreducedebugscript">setReduceDebugScript(String)</a>)
+ , whether job tasks can be executed in a <em>speculative</em> manner
+ (<a href="ext:api/org/apache/hadoop/mapred/jobconf/setmapspeculativeexecution">setMapSpeculativeExecution(boolean)</a>)/(<a href="ext:api/org/apache/hadoop/mapred/jobconf/setreducespeculativeexecution">setReduceSpeculativeExecution(boolean)</a>)
+ , maximum number of attempts per task
+ (<a href="ext:api/org/apache/hadoop/mapred/jobconf/setmaxmapattempts">setMaxMapAttempts(int)</a>/<a href="ext:api/org/apache/hadoop/mapred/jobconf/setmaxreduceattempts">setMaxReduceAttempts(int)</a>)
+ , percentage of tasks failure which can be tolerated by the job
+ (<a href="ext:api/org/apache/hadoop/mapred/jobconf/setmaxmaptaskfailurespercent">setMaxMapTaskFailuresPercent(int)</a>/<a href="ext:api/org/apache/hadoop/mapred/jobconf/setmaxreducetaskfailurespercent">setMaxReduceTaskFailuresPercent(int)</a>)
+ etc.</p>
+
+ <p>Of course, users can use
+ <a href="ext:api/org/apache/hadoop/conf/configuration/set">set(String, String)</a>/<a href="ext:api/org/apache/hadoop/conf/configuration/get">get(String, String)</a>
+ to set/get arbitrary parameters needed by applications. However, use the
+ <code>DistributedCache</code> for large amounts of (read-only) data.</p>
+ </section>
+
+ <section>
+ <title>Task Execution & Environment</title>
+
+ <p>The <code>TaskTracker</code> executes the <code>Mapper</code>/
+ <code>Reducer</code> <em>task</em> as a child process in a separate jvm.
+ </p>
+
+ <p>The child-task inherits the environment of the parent
+ <code>TaskTracker</code>. The user can specify additional options to the
+ child-jvm via the <code>mapred.child.java.opts</code> configuration
+ parameter in the <code>JobConf</code> such as non-standard paths for the
+ run-time linker to search shared libraries via
+ <code>-Djava.library.path=<></code> etc. If the
+ <code>mapred.child.java.opts</code> contains the symbol <em>@taskid@</em>
+ it is interpolated with value of <code>taskid</code> of the map/reduce
+ task.</p>
+
+ <p>Here is an example with multiple arguments and substitutions,
+ showing jvm GC logging, and start of a passwordless JVM JMX agent so that
+ it can connect with jconsole and the likes to watch child memory,
+ threads and get thread dumps. It also sets the maximum heap-size of the
+ child jvm to 512MB and adds an additional path to the
+ <code>java.library.path</code> of the child-jvm.</p>
+
+ <p>
+ <code><property></code><br/>
+ <code><name>mapred.child.java.opts</name></code><br/>
+ <code><value></code><br/>
+ <code>
+ -Xmx512M -Djava.library.path=/home/mycompany/lib
+ -verbose:gc -Xloggc:/tmp/@taskid@.gc</code><br/>
+ <code>
+ -Dcom.sun.management.jmxremote.authenticate=false
+ -Dcom.sun.management.jmxremote.ssl=false</code><br/>
+ <code></value></code><br/>
+ <code></property></code>
+ </p>
+
+ <section>
+ <title> Memory management</title>
+ <p>Users/admins can also specify the maximum virtual memory
+ of the launched child-task, and any sub-process it launches
+ recursively, using <code>mapred.child.ulimit</code>. Note that
+ the value set here is a per process limit.
+ The value for <code>mapred.child.ulimit</code> should be specified
+ in kilo bytes (KB). And also the value must be greater than
+ or equal to the -Xmx passed to JavaVM, else the VM might not start.
+ </p>
+
+ <p>Note: <code>mapred.child.java.opts</code> are used only for
+ configuring the launched child tasks from task tracker. Configuring
+ the memory options for daemons is documented in
+ <a href="cluster_setup.html#Configuring+the+Environment+of+the+Hadoop+Daemons">
+ cluster_setup.html </a></p>
+
+ <p>The memory available to some parts of the framework is also
+ configurable. In map and reduce tasks, performance may be influenced
+ by adjusting parameters influencing the concurrency of operations and
+ the frequency with which data will hit disk. Monitoring the filesystem
+ counters for a job- particularly relative to byte counts from the map
+ and into the reduce- is invaluable to the tuning of these
+ parameters.</p>
+
+ <p>Users can choose to override default limits of Virtual Memory and RAM
+ enforced by the task tracker, if memory management is enabled.
+ Users can set the following parameter per job:</p>
+
+ <table>
+ <tr><th>Name</th><th>Type</th><th>Description</th></tr>
+ <tr><td><code>mapred.task.maxvmem</code></td><td>int</td>
+ <td>A number, in bytes, that represents the maximum Virtual Memory
+ task-limit for each task of the job. A task will be killed if
+ it consumes more Virtual Memory than this number.
+ </td></tr>
+ <tr><td>mapred.task.maxpmem</td><td>int</td>
+ <td>A number, in bytes, that represents the maximum RAM task-limit
+ for each task of the job. This number can be optionally used by
+ Schedulers to prevent over-scheduling of tasks on a node based
+ on RAM needs.
+ </td></tr>
+ </table>
+ </section>
+ <section>
+ <title>Map Parameters</title>
+
+ <p>A record emitted from a map will be serialized into a buffer and
+ metadata will be stored into accounting buffers. As described in the
+ following options, when either the serialization buffer or the
+ metadata exceed a threshold, the contents of the buffers will be
+ sorted and written to disk in the background while the map continues
+ to output records. If either buffer fills completely while the spill
+ is in progress, the map thread will block. When the map is finished,
+ any remaining records are written to disk and all on-disk segments
+ are merged into a single file. Minimizing the number of spills to
+ disk can decrease map time, but a larger buffer also decreases the
+ memory available to the mapper.</p>
+
+ <table>
+ <tr><th>Name</th><th>Type</th><th>Description</th></tr>
+ <tr><td>io.sort.mb</td><td>int</td>
+ <td>The cumulative size of the serialization and accounting
+ buffers storing records emitted from the map, in megabytes.
+ </td></tr>
+ <tr><td>io.sort.record.percent</td><td>float</td>
+ <td>The ratio of serialization to accounting space can be
+ adjusted. Each serialized record requires 16 bytes of
+ accounting information in addition to its serialized size to
+ effect the sort. This percentage of space allocated from
+ <code>io.sort.mb</code> affects the probability of a spill to
+ disk being caused by either exhaustion of the serialization
+ buffer or the accounting space. Clearly, for a map outputting
+ small records, a higher value than the default will likely
+ decrease the number of spills to disk.</td></tr>
+ <tr><td>io.sort.spill.percent</td><td>float</td>
+ <td>This is the threshold for the accounting and serialization
+ buffers. When this percentage of either buffer has filled,
+ their contents will be spilled to disk in the background. Let
+ <code>io.sort.record.percent</code> be <em>r</em>,
+ <code>io.sort.mb</code> be <em>x</em>, and this value be
+ <em>q</em>. The maximum number of records collected before the
+ collection thread will spill is <code>r * x * q * 2^16</code>.
+ Note that a higher value may decrease the number of- or even
+ eliminate- merges, but will also increase the probability of
+ the map task getting blocked. The lowest average map times are
+ usually obtained by accurately estimating the size of the map
+ output and preventing multiple spills.</td></tr>
+ </table>
+
+ <p>Other notes</p>
+ <ul>
+ <li>If either spill threshold is exceeded while a spill is in
+ progress, collection will continue until the spill is finished.
+ For example, if <code>io.sort.buffer.spill.percent</code> is set
+ to 0.33, and the remainder of the buffer is filled while the spill
+ runs, the next spill will include all the collected records, or
+ 0.66 of the buffer, and will not generate additional spills. In
+ other words, the thresholds are defining triggers, not
+ blocking.</li>
+ <li>A record larger than the serialization buffer will first
+ trigger a spill, then be spilled to a separate file. It is
+ undefined whether or not this record will first pass through the
+ combiner.</li>
+ </ul>
+ </section>
+
+ <section>
+ <title>Shuffle/Reduce Parameters</title>
+
+ <p>As described previously, each reduce fetches the output assigned
+ to it by the Partitioner via HTTP into memory and periodically
+ merges these outputs to disk. If intermediate compression of map
+ outputs is turned on, each output is decompressed into memory. The
+ following options affect the frequency of these merges to disk prior
+ to the reduce and the memory allocated to map output during the
+ reduce.</p>
+
+ <table>
+ <tr><th>Name</th><th>Type</th><th>Description</th></tr>
+ <tr><td>io.sort.factor</td><td>int</td>
+ <td>Specifies the number of segments on disk to be merged at
+ the same time. It limits the number of open files and
+ compression codecs during the merge. If the number of files
+ exceeds this limit, the merge will proceed in several passes.
+ Though this limit also applies to the map, most jobs should be
+ configured so that hitting this limit is unlikely
+ there.</td></tr>
+ <tr><td>mapred.inmem.merge.threshold</td><td>int</td>
+ <td>The number of sorted map outputs fetched into memory
+ before being merged to disk. Like the spill thresholds in the
+ preceding note, this is not defining a unit of partition, but
+ a trigger. In practice, this is usually set very high (1000)
+ or disabled (0), since merging in-memory segments is often
+ less expensive than merging from disk (see notes following
+ this table). This threshold influences only the frequency of
+ in-memory merges during the shuffle.</td></tr>
+ <tr><td>mapred.job.shuffle.merge.percent</td><td>float</td>
+ <td>The memory threshold for fetched map outputs before an
+ in-memory merge is started, expressed as a percentage of
+ memory allocated to storing map outputs in memory. Since map
+ outputs that can't fit in memory can be stalled, setting this
+ high may decrease parallelism between the fetch and merge.
+ Conversely, values as high as 1.0 have been effective for
+ reduces whose input can fit entirely in memory. This parameter
+ influences only the frequency of in-memory merges during the
+ shuffle.</td></tr>
+ <tr><td>mapred.job.shuffle.input.buffer.percent</td><td>float</td>
+ <td>The percentage of memory- relative to the maximum heapsize
+ as typically specified in <code>mapred.child.java.opts</code>-
+ that can be allocated to storing map outputs during the
+ shuffle. Though some memory should be set aside for the
+ framework, in general it is advantageous to set this high
+ enough to store large and numerous map outputs.</td></tr>
+ <tr><td>mapred.job.reduce.input.buffer.percent</td><td>float</td>
+ <td>The percentage of memory relative to the maximum heapsize
+ in which map outputs may be retained during the reduce. When
+ the reduce begins, map outputs will be merged to disk until
+ those that remain are under the resource limit this defines.
+ By default, all map outputs are merged to disk before the
+ reduce begins to maximize the memory available to the reduce.
+ For less memory-intensive reduces, this should be increased to
+ avoid trips to disk.</td></tr>
+ </table>
+
+ <p>Other notes</p>
+ <ul>
+ <li>If a map output is larger than 25 percent of the memory
+ allocated to copying map outputs, it will be written directly to
+ disk without first staging through memory.</li>
+ <li>When running with a combiner, the reasoning about high merge
+ thresholds and large buffers may not hold. For merges started
+ before all map outputs have been fetched, the combiner is run
+ while spilling to disk. In some cases, one can obtain better
+ reduce times by spending resources combining map outputs- making
+ disk spills small and parallelizing spilling and fetching- rather
+ than aggressively increasing buffer sizes.</li>
+ <li>When merging in-memory map outputs to disk to begin the
+ reduce, if an intermediate merge is necessary because there are
+ segments to spill and at least <code>io.sort.factor</code>
+ segments already on disk, the in-memory map outputs will be part
+ of the intermediate merge.</li>
+ </ul>
+
+ </section>
+
+ <section>
+ <title> Directory Structure </title>
+ <p>The task tracker has local directory,
+ <code> ${mapred.local.dir}/taskTracker/</code> to create localized
+ cache and localized job. It can define multiple local directories
+ (spanning multiple disks) and then each filename is assigned to a
+ semi-random local directory. When the job starts, task tracker
+ creates a localized job directory relative to the local directory
+ specified in the configuration. Thus the task tracker directory
+ structure looks the following: </p>
+ <ul>
+ <li><code>${mapred.local.dir}/taskTracker/archive/</code> :
+ The distributed cache. This directory holds the localized distributed
+ cache. Thus localized distributed cache is shared among all
+ the tasks and jobs </li>
+ <li><code>${mapred.local.dir}/taskTracker/jobcache/$jobid/</code> :
+ The localized job directory
+ <ul>
+ <li><code>${mapred.local.dir}/taskTracker/jobcache/$jobid/work/</code>
+ : The job-specific shared directory. The tasks can use this space as
+ scratch space and share files among them. This directory is exposed
+ to the users through the configuration property
+ <code>job.local.dir</code>. The directory can accessed through
+ api <a href="ext:api/org/apache/hadoop/mapred/jobconf/getjoblocaldir">
+ JobConf.getJobLocalDir()</a>. It is available as System property also.
+ So, users (streaming etc.) can call
+ <code>System.getProperty("job.local.dir")</code> to access the
+ directory.</li>
+ <li><code>${mapred.local.dir}/taskTracker/jobcache/$jobid/jars/</code>
+ : The jars directory, which has the job jar file and expanded jar.
+ The <code>job.jar</code> is the application's jar file that is
+ automatically distributed to each machine. It is expanded in jars
+ directory before the tasks for the job start. The job.jar location
+ is accessible to the application through the api
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/getjar">
+ JobConf.getJar() </a>. To access the unjarred directory,
+ JobConf.getJar().getParent() can be called.</li>
+ <li><code>${mapred.local.dir}/taskTracker/jobcache/$jobid/job.xml</code>
+ : The job.xml file, the generic job configuration, localized for
+ the job. </li>
+ <li><code>${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid</code>
+ : The task directory for each task attempt. Each task directory
+ again has the following structure :
+ <ul>
+ <li><code>${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid/job.xml</code>
+ : A job.xml file, task localized job configuration, Task localization
+ means that properties have been set that are specific to
+ this particular task within the job. The properties localized for
+ each task are described below.</li>
+ <li><code>${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid/output</code>
+ : A directory for intermediate output files. This contains the
+ temporary map reduce data generated by the framework
+ such as map output files etc. </li>
+ <li><code>${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid/work</code>
+ : The curernt working directory of the task.
+ With <a href="#Task+JVM+Reuse">jvm reuse</a> enabled for tasks, this
+ directory will be the directory on which the jvm has started</li>
+ <li><code>${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid/work/tmp</code>
+ : The temporary directory for the task.
+ (User can specify the property <code>mapred.child.tmp</code> to set
+ the value of temporary directory for map and reduce tasks. This
+ defaults to <code>./tmp</code>. If the value is not an absolute path,
+ it is prepended with task's working directory. Otherwise, it is
+ directly assigned. The directory will be created if it doesn't exist.
+ Then, the child java tasks are executed with option
+ <code>-Djava.io.tmpdir='the absolute path of the tmp dir'</code>.
+ Anp pipes and streaming are set with environment variable,
+ <code>TMPDIR='the absolute path of the tmp dir'</code>). This
+ directory is created, if <code>mapred.child.tmp</code> has the value
+ <code>./tmp</code> </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <title>Task JVM Reuse</title>
+ <p>Jobs can enable task JVMs to be reused by specifying the job
+ configuration <code>mapred.job.reuse.jvm.num.tasks</code>. If the
+ value is 1 (the default), then JVMs are not reused
+ (i.e. 1 task per JVM). If it is -1, there is no limit to the number
+ of tasks a JVM can run (of the same job). One can also specify some
+ value greater than 1 using the api
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setnumtaskstoexecuteperjvm">
+ JobConf.setNumTasksToExecutePerJvm(int)</a></p>
+ </section>
+
+ <p>The following properties are localized in the job configuration
+ for each task's execution: </p>
+ <table>
+ <tr><th>Name</th><th>Type</th><th>Description</th></tr>
+ <tr><td>mapred.job.id</td><td>String</td><td>The job id</td></tr>
+ <tr><td>mapred.jar</td><td>String</td>
+ <td>job.jar location in job directory</td></tr>
+ <tr><td>job.local.dir</td><td> String</td>
+ <td> The job specific shared scratch space</td></tr>
+ <tr><td>mapred.tip.id</td><td> String</td>
+ <td> The task id</td></tr>
+ <tr><td>mapred.task.id</td><td> String</td>
+ <td> The task attempt id</td></tr>
+ <tr><td>mapred.task.is.map</td><td> boolean </td>
+ <td>Is this a map task</td></tr>
+ <tr><td>mapred.task.partition</td><td> int </td>
+ <td>The id of the task within the job</td></tr>
+ <tr><td>map.input.file</td><td> String</td>
+ <td> The filename that the map is reading from</td></tr>
+ <tr><td>map.input.start</td><td> long</td>
+ <td> The offset of the start of the map input split</td></tr>
+ <tr><td>map.input.length </td><td>long </td>
+ <td>The number of bytes in the map input split</td></tr>
+ <tr><td>mapred.work.output.dir</td><td> String </td>
+ <td>The task's temporary output directory</td></tr>
+ </table>
+
+ <p>The standard output (stdout) and error (stderr) streams of the task
+ are read by the TaskTracker and logged to
+ <code>${HADOOP_LOG_DIR}/userlogs</code></p>
+
+ <p>The <a href="#DistributedCache">DistributedCache</a> can also be used
+ to distribute both jars and native libraries for use in the map
+ and/or reduce tasks. The child-jvm always has its
+ <em>current working directory</em> added to the
+ <code>java.library.path</code> and <code>LD_LIBRARY_PATH</code>.
+ And hence the cached libraries can be loaded via
+ <a href="http://java.sun.com/javase/6/docs/api/java/lang/System.html#loadLibrary(java.lang.String)">
+ System.loadLibrary</a> or
+ <a href="http://java.sun.com/javase/6/docs/api/java/lang/System.html#load(java.lang.String)">
+ System.load</a>. More details on how to load shared libraries through
+ distributed cache are documented at
+ <a href="native_libraries.html#Loading+native+libraries+through+DistributedCache">
+ native_libraries.html</a></p>
+ </section>
+
+ <section>
+ <title>Job Submission and Monitoring</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/jobclient">
+ JobClient</a> is the primary interface by which user-job interacts
+ with the <code>JobTracker</code>.</p>
+
+ <p><code>JobClient</code> provides facilities to submit jobs, track their
+ progress, access component-tasks' reports and logs, get the Map/Reduce
+ cluster's status information and so on.</p>
+
+ <p>The job submission process involves:</p>
+ <ol>
+ <li>Checking the input and output specifications of the job.</li>
+ <li>Computing the <code>InputSplit</code> values for the job.</li>
+ <li>
+ Setting up the requisite accounting information for the
+ <code>DistributedCache</code> of the job, if necessary.
+ </li>
+ <li>
+ Copying the job's jar and configuration to the Map/Reduce system
+ directory on the <code>FileSystem</code>.
+ </li>
+ <li>
+ Submitting the job to the <code>JobTracker</code> and optionally
+ monitoring it's status.
+ </li>
+ </ol>
+ <p> Job history files are also logged to user specified directory
+ <code>hadoop.job.history.user.location</code>
+ which defaults to job output directory. The files are stored in
+ "_logs/history/" in the specified directory. Hence, by default they
+ will be in mapred.output.dir/_logs/history. User can stop
+ logging by giving the value <code>none</code> for
+ <code>hadoop.job.history.user.location</code></p>
+
+ <p> User can view the history logs summary in specified directory
+ using the following command <br/>
+ <code>$ bin/hadoop job -history output-dir</code><br/>
+ This command will print job details, failed and killed tip
+ details. <br/>
+ More details about the job such as successful tasks and
+ task attempts made for each task can be viewed using the
+ following command <br/>
+ <code>$ bin/hadoop job -history all output-dir</code><br/></p>
+
+ <p> User can use
+ <a href="ext:api/org/apache/hadoop/mapred/outputlogfilter">OutputLogFilter</a>
+ to filter log files from the output directory listing. </p>
+
+ <p>Normally the user creates the application, describes various facets
+ of the job via <code>JobConf</code>, and then uses the
+ <code>JobClient</code> to submit the job and monitor its progress.</p>
+
+ <section>
+ <title>Job Control</title>
+
+ <p>Users may need to chain Map/Reduce jobs to accomplish complex
+ tasks which cannot be done via a single Map/Reduce job. This is fairly
+ easy since the output of the job typically goes to distributed
+ file-system, and the output, in turn, can be used as the input for the
+ next job.</p>
+
+ <p>However, this also means that the onus on ensuring jobs are
+ complete (success/failure) lies squarely on the clients. In such
+ cases, the various job-control options are:</p>
+ <ul>
+ <li>
+ <a href="ext:api/org/apache/hadoop/mapred/jobclient/runjob">
+ runJob(JobConf)</a> : Submits the job and returns only after the
+ job has completed.
+ </li>
+ <li>
+ <a href="ext:api/org/apache/hadoop/mapred/jobclient/submitjob">
+ submitJob(JobConf)</a> : Only submits the job, then poll the
+ returned handle to the
+ <a href="ext:api/org/apache/hadoop/mapred/runningjob">
+ RunningJob</a> to query status and make scheduling decisions.
+ </li>
+ <li>
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setjobendnotificationuri">
+ JobConf.setJobEndNotificationURI(String)</a> : Sets up a
+ notification upon job-completion, thus avoiding polling.
+ </li>
+ </ul>
+ </section>
+ </section>
+
+ <section>
+ <title>Job Input</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/inputformat">
+ InputFormat</a> describes the input-specification for a Map/Reduce job.
+ </p>
+
+ <p>The Map/Reduce framework relies on the <code>InputFormat</code> of
+ the job to:</p>
+ <ol>
+ <li>Validate the input-specification of the job.</li>
+ <li>
+ Split-up the input file(s) into logical <code>InputSplit</code>
+ instances, each of which is then assigned to an individual
+ <code>Mapper</code>.
+ </li>
+ <li>
+ Provide the <code>RecordReader</code> implementation used to
+ glean input records from the logical <code>InputSplit</code> for
+ processing by the <code>Mapper</code>.
+ </li>
+ </ol>
+
+ <p>The default behavior of file-based <code>InputFormat</code>
+ implementations, typically sub-classes of
+ <a href="ext:api/org/apache/hadoop/mapred/fileinputformat">
+ FileInputFormat</a>, is to split the input into <em>logical</em>
+ <code>InputSplit</code> instances based on the total size, in bytes, of
+ the input files. However, the <code>FileSystem</code> blocksize of the
+ input files is treated as an upper bound for input splits. A lower bound
+ on the split size can be set via <code>mapred.min.split.size</code>.</p>
+
+ <p>Clearly, logical splits based on input-size is insufficient for many
+ applications since record boundaries must be respected. In such cases,
+ the application should implement a <code>RecordReader</code>, who is
+ responsible for respecting record-boundaries and presents a
+ record-oriented view of the logical <code>InputSplit</code> to the
+ individual task.</p>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/textinputformat">
+ TextInputFormat</a> is the default <code>InputFormat</code>.</p>
+
+ <p>If <code>TextInputFormat</code> is the <code>InputFormat</code> for a
+ given job, the framework detects input-files with the <em>.gz</em>
+ extensions and automatically decompresses them using the
+ appropriate <code>CompressionCodec</code>. However, it must be noted that
+ compressed files with the above extensions cannot be <em>split</em> and
+ each compressed file is processed in its entirety by a single mapper.</p>
+
+ <section>
+ <title>InputSplit</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/inputsplit">
+ InputSplit</a> represents the data to be processed by an individual
+ <code>Mapper</code>.</p>
+
+ <p>Typically <code>InputSplit</code> presents a byte-oriented view of
+ the input, and it is the responsibility of <code>RecordReader</code>
+ to process and present a record-oriented view.</p>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/filesplit">
+ FileSplit</a> is the default <code>InputSplit</code>. It sets
+ <code>map.input.file</code> to the path of the input file for the
+ logical split.</p>
+ </section>
+
+ <section>
+ <title>RecordReader</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/recordreader">
+ RecordReader</a> reads <code><key, value></code> pairs from an
+ <code>InputSplit</code>.</p>
+
+ <p>Typically the <code>RecordReader</code> converts the byte-oriented
+ view of the input, provided by the <code>InputSplit</code>, and
+ presents a record-oriented to the <code>Mapper</code> implementations
+ for processing. <code>RecordReader</code> thus assumes the
+ responsibility of processing record boundaries and presents the tasks
+ with keys and values.</p>
+ </section>
+ </section>
+
+ <section>
+ <title>Job Output</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/outputformat">
+ OutputFormat</a> describes the output-specification for a Map/Reduce
+ job.</p>
+
+ <p>The Map/Reduce framework relies on the <code>OutputFormat</code> of
+ the job to:</p>
+ <ol>
+ <li>
+ Validate the output-specification of the job; for example, check that
+ the output directory doesn't already exist.
+ </li>
+ <li>
+ Provide the <code>RecordWriter</code> implementation used to
+ write the output files of the job. Output files are stored in a
+ <code>FileSystem</code>.
+ </li>
+ </ol>
+
+ <p><code>TextOutputFormat</code> is the default
+ <code>OutputFormat</code>.</p>
+
+ <section>
+ <title>Lazy Output Creation</title>
+ <p>It is possible to delay creation of output until the first write attempt
+ by using <a href="ext:api/org/apache/hadoop/mapred/lib/lazyoutputformat">
+ LazyOutputFormat</a>. This is particularly useful in preventing the
+ creation of zero byte files when there is no call to output.collect
+ (or Context.write). This is achieved by calling the static method
+ <code>setOutputFormatClass</code> of <code>LazyOutputFormat</code>
+ with the intended <code>OutputFormat</code> as the argument. The following example
+ shows how to delay creation of files when using the <code>TextOutputFormat</code>
+ </p>
+
+ <p>
+ <code> import org.apache.hadoop.mapred.lib.LazyOutputFormat;</code> <br/>
+ <code> LazyOutputFormat.setOutputFormatClass(conf, TextOutputFormat.class);</code>
+ </p>
+
+ </section>
+
+ <section>
+ <title>OutputCommitter</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/outputcommitter">
+ OutputCommitter</a> describes the commit of task output for a
+ Map/Reduce job.</p>
+
+ <p>The Map/Reduce framework relies on the <code>OutputCommitter</code>
+ of the job to:</p>
+ <ol>
+ <li>
+ Setup the job during initialization. For example, create
+ the temporary output directory for the job during the
+ initialization of the job.
+ Job setup is done by a separate task when the job is
+ in PREP state and after initializing tasks. Once the setup task
+ completes, the job will be moved to RUNNING state.
+ </li>
+ <li>
+ Cleanup the job after the job completion. For example, remove the
+ temporary output directory after the job completion.
+ Job cleanup is done by a separate task at the end of the job.
+ Job is declared SUCCEDED/FAILED/KILLED after the cleanup
+ task completes.
+ </li>
+ <li>
+ Setup the task temporary output.
+ Task setup is done as part of the same task, during task initialization.
+ </li>
+ <li>
+ Check whether a task needs a commit. This is to avoid the commit
+ procedure if a task does not need commit.
+ </li>
+ <li>
+ Commit of the task output.
+ Once task is done, the task will commit it's output if required.
+ </li>
+ <li>
+ Discard the task commit.
+ If the task has been failed/killed, the output will be cleaned-up.
+ If task could not cleanup (in exception block), a separate task
+ will be launched with same attempt-id to do the cleanup.
+ </li>
+ </ol>
+ <p><code>FileOutputCommitter</code> is the default
+ <code>OutputCommitter</code>. Job setup/cleanup tasks occupy
+ map or reduce slots, whichever is free on the TaskTracker. And
+ JobCleanup task, TaskCleanup tasks and JobSetup task have the highest
+ priority, and in that order.</p>
+ </section>
+
+ <section>
+ <title>Task Side-Effect Files</title>
+
+ <p>In some applications, component tasks need to create and/or write to
+ side-files, which differ from the actual job-output files.</p>
+
+ <p>In such cases there could be issues with two instances of the same
+ <code>Mapper</code> or <code>Reducer</code> running simultaneously (for
+ example, speculative tasks) trying to open and/or write to the same
+ file (path) on the <code>FileSystem</code>. Hence the
+ application-writer will have to pick unique names per task-attempt
+ (using the attemptid, say <code>attempt_200709221812_0001_m_000000_0</code>),
+ not just per task.</p>
+
+ <p>To avoid these issues the Map/Reduce framework, when the
+ <code>OutputCommitter</code> is <code>FileOutputCommitter</code>,
+ maintains a special
+ <code>${mapred.output.dir}/_temporary/_${taskid}</code> sub-directory
+ accessible via <code>${mapred.work.output.dir}</code>
+ for each task-attempt on the <code>FileSystem</code> where the output
+ of the task-attempt is stored. On successful completion of the
+ task-attempt, the files in the
+ <code>${mapred.output.dir}/_temporary/_${taskid}</code> (only)
+ are <em>promoted</em> to <code>${mapred.output.dir}</code>. Of course,
+ the framework discards the sub-directory of unsuccessful task-attempts.
+ This process is completely transparent to the application.</p>
+
+ <p>The application-writer can take advantage of this feature by
+ creating any side-files required in <code>${mapred.work.output.dir}</code>
+ during execution of a task via
+ <a href="ext:api/org/apache/hadoop/mapred/fileoutputformat/getworkoutputpath">
+ FileOutputFormat.getWorkOutputPath()</a>, and the framework will promote them
+ similarly for succesful task-attempts, thus eliminating the need to
+ pick unique paths per task-attempt.</p>
+
+ <p>Note: The value of <code>${mapred.work.output.dir}</code> during
+ execution of a particular task-attempt is actually
+ <code>${mapred.output.dir}/_temporary/_{$taskid}</code>, and this value is
+ set by the Map/Reduce framework. So, just create any side-files in the
+ path returned by
+ <a href="ext:api/org/apache/hadoop/mapred/fileoutputformat/getworkoutputpath">
+ FileOutputFormat.getWorkOutputPath() </a>from map/reduce
+ task to take advantage of this feature.</p>
+
+ <p>The entire discussion holds true for maps of jobs with
+ reducer=NONE (i.e. 0 reduces) since output of the map, in that case,
+ goes directly to HDFS.</p>
+ </section>
+
+ <section>
+ <title>RecordWriter</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/recordwriter">
+ RecordWriter</a> writes the output <code><key, value></code>
+ pairs to an output file.</p>
+
+ <p>RecordWriter implementations write the job outputs to the
+ <code>FileSystem</code>.</p>
+ </section>
+ </section>
+
+ <section>
+ <title>Other Useful Features</title>
+
+ <section>
+ <title>Submitting Jobs to Queues</title>
+ <p>Users submit jobs to Queues. Queues, as collection of jobs,
+ allow the system to provide specific functionality. For example,
+ queues use ACLs to control which users
+ who can submit jobs to them. Queues are expected to be primarily
+ used by Hadoop Schedulers. </p>
+
+ <p>Hadoop comes configured with a single mandatory queue, called
+ 'default'. Queue names are defined in the
+ <code>mapred.queue.names</code> property of the Hadoop site
+ configuration. Some job schedulers, such as the
+ <a href="capacity_scheduler.html">Capacity Scheduler</a>,
+ support multiple queues.</p>
+
+ <p>A job defines the queue it needs to be submitted to through the
+ <code>mapred.job.queue.name</code> property, or through the
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setqueuename">setQueueName(String)</a>
+ API. Setting the queue name is optional. If a job is submitted
+ without an associated queue name, it is submitted to the 'default'
+ queue.</p>
+ </section>
+ <section>
+ <title>Counters</title>
+
+ <p><code>Counters</code> represent global counters, defined either by
+ the Map/Reduce framework or applications. Each <code>Counter</code> can
+ be of any <code>Enum</code> type. Counters of a particular
+ <code>Enum</code> are bunched into groups of type
+ <code>Counters.Group</code>.</p>
+
+ <p>Applications can define arbitrary <code>Counters</code> (of type
+ <code>Enum</code>) and update them via
+ <a href="ext:api/org/apache/hadoop/mapred/reporter/incrcounterEnum">
+ Reporter.incrCounter(Enum, long)</a> or
+ <a href="ext:api/org/apache/hadoop/mapred/reporter/incrcounterString">
+ Reporter.incrCounter(String, String, long)</a>
+ in the <code>map</code> and/or
+ <code>reduce</code> methods. These counters are then globally
+ aggregated by the framework.</p>
+ </section>
+
+ <section>
+ <title>DistributedCache</title>
+
+ <p><a href="ext:api/org/apache/hadoop/filecache/distributedcache">
+ DistributedCache</a> distributes application-specific, large, read-only
+ files efficiently.</p>
+
+ <p><code>DistributedCache</code> is a facility provided by the
+ Map/Reduce framework to cache files (text, archives, jars and so on)
+ needed by applications.</p>
+
+ <p>Applications specify the files to be cached via urls (hdfs://)
+ in the <code>JobConf</code>. The <code>DistributedCache</code>
+ assumes that the files specified via hdfs:// urls are already present
+ on the <code>FileSystem</code>.</p>
+
+ <p>The framework will copy the necessary files to the slave node
+ before any tasks for the job are executed on that node. Its
+ efficiency stems from the fact that the files are only copied once
+ per job and the ability to cache archives which are un-archived on
+ the slaves.</p>
+
+ <p><code>DistributedCache</code> tracks the modification timestamps of
+ the cached files. Clearly the cache files should not be modified by
+ the application or externally while the job is executing.</p>
+
+ <p><code>DistributedCache</code> can be used to distribute simple,
+ read-only data/text files and more complex types such as archives and
+ jars. Archives (zip, tar, tgz and tar.gz files) are
+ <em>un-archived</em> at the slave nodes. Files
+ have <em>execution permissions</em> set. </p>
+
+ <p>The files/archives can be distributed by setting the property
+ <code>mapred.cache.{files|archives}</code>. If more than one
+ file/archive has to be distributed, they can be added as comma
+ separated paths. The properties can also be set by APIs
+ <a href="ext:api/org/apache/hadoop/filecache/distributedcache/addcachefile">
+ DistributedCache.addCacheFile(URI,conf)</a>/
+ <a href="ext:api/org/apache/hadoop/filecache/distributedcache/addcachearchive">
+ DistributedCache.addCacheArchive(URI,conf)</a> and
+ <a href="ext:api/org/apache/hadoop/filecache/distributedcache/setcachefiles">
+ DistributedCache.setCacheFiles(URIs,conf)</a>/
+ <a href="ext:api/org/apache/hadoop/filecache/distributedcache/setcachearchives">
+ DistributedCache.setCacheArchives(URIs,conf)</a>
+ where URI is of the form
+ <code>hdfs://host:port/absolute-path#link-name</code>.
+ In Streaming, the files can be distributed through command line
+ option <code>-cacheFile/-cacheArchive</code>.</p>
+
+ <p>Optionally users can also direct the <code>DistributedCache</code>
+ to <em>symlink</em> the cached file(s) into the <code>current working
+ directory</code> of the task via the
+ <a href="ext:api/org/apache/hadoop/filecache/distributedcache/createsymlink">
+ DistributedCache.createSymlink(Configuration)</a> api. Or by setting
+ the configuration property <code>mapred.create.symlink</code>
+ as <code>yes</code>. The DistributedCache will use the
+ <code>fragment</code> of the URI as the name of the symlink.
+ For example, the URI
+ <code>hdfs://namenode:port/lib.so.1#lib.so</code>
+ will have the symlink name as <code>lib.so</code> in task's cwd
+ for the file <code>lib.so.1</code> in distributed cache.</p>
+
+ <p>The <code>DistributedCache</code> can also be used as a
+ rudimentary software distribution mechanism for use in the
+ map and/or reduce tasks. It can be used to distribute both
+ jars and native libraries. The
+ <a href="ext:api/org/apache/hadoop/filecache/distributedcache/addarchivetoclasspath">
+ DistributedCache.addArchiveToClassPath(Path, Configuration)</a> or
+ <a href="ext:api/org/apache/hadoop/filecache/distributedcache/addfiletoclasspath">
+ DistributedCache.addFileToClassPath(Path, Configuration)</a> api
+ can be used to cache files/jars and also add them to the
+ <em>classpath</em> of child-jvm. The same can be done by setting
+ the configuration properties
+ <code>mapred.job.classpath.{files|archives}</code>. Similarly the
+ cached files that are symlinked into the working directory of the
+ task can be used to distribute native libraries and load them.</p>
+
+ </section>
+
+ <section>
+ <title>Tool</title>
+
+ <p>The <a href="ext:api/org/apache/hadoop/util/tool">Tool</a>
+ interface supports the handling of generic Hadoop command-line options.
+ </p>
+
+ <p><code>Tool</code> is the standard for any Map/Reduce tool or
+ application. The application should delegate the handling of
+ standard command-line options to
+ <a href="ext:api/org/apache/hadoop/util/genericoptionsparser">
+ GenericOptionsParser</a> via
+ <a href="ext:api/org/apache/hadoop/util/toolrunner/run">
+ ToolRunner.run(Tool, String[])</a> and only handle its custom
+ arguments.</p>
+
+ <p>
+ The generic Hadoop command-line options are:<br/>
+ <code>
+ -conf <configuration file>
+ </code>
+ <br/>
+ <code>
+ -D <property=value>
+ </code>
+ <br/>
+ <code>
+ -fs <local|namenode:port>
+ </code>
+ <br/>
+ <code>
+ -jt <local|jobtracker:port>
+ </code>
+ </p>
+ </section>
+
+ <section>
+ <title>IsolationRunner</title>
+
+ <p><a href="ext:api/org/apache/hadoop/mapred/isolationrunner">
+ IsolationRunner</a> is a utility to help debug Map/Reduce programs.</p>
+
+ <p>To use the <code>IsolationRunner</code>, first set
+ <code>keep.failed.tasks.files</code> to <code>true</code>
+ (also see <code>keep.tasks.files.pattern</code>).</p>
+
+ <p>
+ Next, go to the node on which the failed task ran and go to the
+ <code>TaskTracker</code>'s local directory and run the
+ <code>IsolationRunner</code>:<br/>
+ <code>$ cd <local path>/taskTracker/${taskid}/work</code><br/>
+ <code>
+ $ bin/hadoop org.apache.hadoop.mapred.IsolationRunner ../job.xml
+ </code>
+ </p>
+
+ <p><code>IsolationRunner</code> will run the failed task in a single
+ jvm, which can be in the debugger, over precisely the same input.</p>
+ </section>
+
+ <section>
+ <title>Profiling</title>
+ <p>Profiling is a utility to get a representative (2 or 3) sample
+ of built-in java profiler for a sample of maps and reduces. </p>
+
+ <p>User can specify whether the system should collect profiler
+ information for some of the tasks in the job by setting the
+ configuration property <code>mapred.task.profile</code>. The
+ value can be set using the api
+ <a href="ext:api/org/apache/hadoop/mapred/jobconf/setprofileenabled">
+ JobConf.setProfileEnabled(boolean)</a>. If the value is set
+ <code>true</code>, the task profiling is enabled. The profiler
[... 1207 lines stripped ...]