You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by dd...@apache.org on 2008/03/21 21:40:39 UTC
svn commit: r639818 [1/3] - in /hadoop/core/branches/branch-0.16: ./ docs/
src/docs/src/documentation/content/xdocs/
src/examples/org/apache/hadoop/examples/ src/java/org/apache/hadoop/mapred/
src/test/org/apache/hadoop/io/ src/test/org/apache/hadoop/m...
Author: ddas
Date: Fri Mar 21 13:40:33 2008
New Revision: 639818
URL: http://svn.apache.org/viewvc?rev=639818&view=rev
Log:
HADOOP-3041. Deprecates getOutputPath and defines two new APIs getCurrentOutputPath and getFinalOutputPath. Contributed by Amareshwari Sriramadasu.
Modified:
hadoop/core/branches/branch-0.16/CHANGES.txt
hadoop/core/branches/branch-0.16/docs/mapred_tutorial.html
hadoop/core/branches/branch-0.16/docs/mapred_tutorial.pdf
hadoop/core/branches/branch-0.16/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
hadoop/core/branches/branch-0.16/src/docs/src/documentation/content/xdocs/site.xml
hadoop/core/branches/branch-0.16/src/examples/org/apache/hadoop/examples/RandomWriter.java
hadoop/core/branches/branch-0.16/src/examples/org/apache/hadoop/examples/Sort.java
hadoop/core/branches/branch-0.16/src/java/org/apache/hadoop/mapred/JobConf.java
hadoop/core/branches/branch-0.16/src/java/org/apache/hadoop/mapred/JobInProgress.java
hadoop/core/branches/branch-0.16/src/java/org/apache/hadoop/mapred/LocalJobRunner.java
hadoop/core/branches/branch-0.16/src/java/org/apache/hadoop/mapred/MapFileOutputFormat.java
hadoop/core/branches/branch-0.16/src/java/org/apache/hadoop/mapred/OutputFormatBase.java
hadoop/core/branches/branch-0.16/src/java/org/apache/hadoop/mapred/SequenceFileOutputFormat.java
hadoop/core/branches/branch-0.16/src/java/org/apache/hadoop/mapred/Task.java
hadoop/core/branches/branch-0.16/src/java/org/apache/hadoop/mapred/TaskTracker.java
hadoop/core/branches/branch-0.16/src/java/org/apache/hadoop/mapred/TextOutputFormat.java
hadoop/core/branches/branch-0.16/src/test/org/apache/hadoop/io/FileBench.java
hadoop/core/branches/branch-0.16/src/test/org/apache/hadoop/mapred/GenericMRLoadGenerator.java
hadoop/core/branches/branch-0.16/src/test/org/apache/hadoop/mapred/MRBench.java
hadoop/core/branches/branch-0.16/src/test/org/apache/hadoop/mapred/SortValidator.java
hadoop/core/branches/branch-0.16/src/test/org/apache/hadoop/mapred/ThreadedMapBenchmark.java
Modified: hadoop/core/branches/branch-0.16/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.16/CHANGES.txt?rev=639818&r1=639817&r2=639818&view=diff
==============================================================================
--- hadoop/core/branches/branch-0.16/CHANGES.txt (original)
+++ hadoop/core/branches/branch-0.16/CHANGES.txt Fri Mar 21 13:40:33 2008
@@ -27,6 +27,10 @@
HADOOP-3007. Tolerate mirror failures while DataNode is replicating
blocks as it used to before. (rangadi)
+ HADOOP-3041. Deprecates getOutputPath and defines two new APIs
+ getCurrentOutputPath and getFinalOutputPath.
+ (Amareshwari Sriramadasu via ddas)
+
Release 0.16.1 - 2008-03-13
INCOMPATIBLE CHANGES
Modified: hadoop/core/branches/branch-0.16/docs/mapred_tutorial.html
URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.16/docs/mapred_tutorial.html?rev=639818&r1=639817&r2=639818&view=diff
==============================================================================
--- hadoop/core/branches/branch-0.16/docs/mapred_tutorial.html (original)
+++ hadoop/core/branches/branch-0.16/docs/mapred_tutorial.html Fri Mar 21 13:40:33 2008
@@ -283,7 +283,7 @@
<a href="#Example%3A+WordCount+v2.0">Example: WordCount v2.0</a>
<ul class="minitoc">
<li>
-<a href="#Source+Code-N10BBE">Source Code</a>
+<a href="#Source+Code-N10BC1">Source Code</a>
</li>
<li>
<a href="#Sample+Runs">Sample Runs</a>
@@ -1731,11 +1731,15 @@
<p>The application-writer can take advantage of this feature by
creating any side-files required in <span class="codefrag">${mapred.output.dir}</span>
during execution of a task via
- <a href="api/org/apache/hadoop/mapred/JobConf.html#getOutputPath()">
- JobConf.getOutputPath()</a>, and the framework will promote them
+ <a href="api/org/apache/hadoop/mapred/JobConf.html#getCurrentOutputPath()">
+ JobConf.getCurrentOutputPath()</a>, and the framework will promote them
similarly for succesful task-attempts, thus eliminating the need to
- pick unique paths per task-attempt.</p>
-<a name="N10A31"></a><a name="RecordWriter"></a>
+ pick unique paths per task-attempt. She can get the actual configured
+ path (final output path) via
+ <a href="api/org/apache/hadoop/mapred/JobConf.html#getFinalOutputPath()">
+ JobConf.getFinalOutputPath()</a>
+</p>
+<a name="N10A34"></a><a name="RecordWriter"></a>
<h4>RecordWriter</h4>
<p>
<a href="api/org/apache/hadoop/mapred/RecordWriter.html">
@@ -1743,9 +1747,9 @@
pairs to an output file.</p>
<p>RecordWriter implementations write the job outputs to the
<span class="codefrag">FileSystem</span>.</p>
-<a name="N10A48"></a><a name="Other+Useful+Features"></a>
+<a name="N10A4B"></a><a name="Other+Useful+Features"></a>
<h3 class="h4">Other Useful Features</h3>
-<a name="N10A4E"></a><a name="Counters"></a>
+<a name="N10A51"></a><a name="Counters"></a>
<h4>Counters</h4>
<p>
<span class="codefrag">Counters</span> represent global counters, defined either by
@@ -1759,7 +1763,7 @@
Reporter.incrCounter(Enum, long)</a> in the <span class="codefrag">map</span> and/or
<span class="codefrag">reduce</span> methods. These counters are then globally
aggregated by the framework.</p>
-<a name="N10A79"></a><a name="DistributedCache"></a>
+<a name="N10A7C"></a><a name="DistributedCache"></a>
<h4>DistributedCache</h4>
<p>
<a href="api/org/apache/hadoop/filecache/DistributedCache.html">
@@ -1792,7 +1796,7 @@
<a href="api/org/apache/hadoop/filecache/DistributedCache.html#createSymlink(org.apache.hadoop.conf.Configuration)">
DistributedCache.createSymlink(Path, Configuration)</a> api. Files
have <em>execution permissions</em> set.</p>
-<a name="N10AB7"></a><a name="Tool"></a>
+<a name="N10ABA"></a><a name="Tool"></a>
<h4>Tool</h4>
<p>The <a href="api/org/apache/hadoop/util/Tool.html">Tool</a>
interface supports the handling of generic Hadoop command-line options.
@@ -1832,7 +1836,7 @@
</span>
</p>
-<a name="N10AE9"></a><a name="IsolationRunner"></a>
+<a name="N10AEC"></a><a name="IsolationRunner"></a>
<h4>IsolationRunner</h4>
<p>
<a href="api/org/apache/hadoop/mapred/IsolationRunner.html">
@@ -1856,13 +1860,13 @@
<p>
<span class="codefrag">IsolationRunner</span> will run the failed task in a single
jvm, which can be in the debugger, over precisely the same input.</p>
-<a name="N10B1C"></a><a name="JobControl"></a>
+<a name="N10B1F"></a><a name="JobControl"></a>
<h4>JobControl</h4>
<p>
<a href="api/org/apache/hadoop/mapred/jobcontrol/package-summary.html">
JobControl</a> is a utility which encapsulates a set of Map-Reduce jobs
and their dependencies.</p>
-<a name="N10B29"></a><a name="Data+Compression"></a>
+<a name="N10B2C"></a><a name="Data+Compression"></a>
<h4>Data Compression</h4>
<p>Hadoop Map-Reduce provides facilities for the application-writer to
specify compression for both intermediate map-outputs and the
@@ -1876,7 +1880,7 @@
codecs for reasons of both performance (zlib) and non-availability of
Java libraries (lzo). More details on their usage and availability are
available <a href="native_libraries.html">here</a>.</p>
-<a name="N10B49"></a><a name="Intermediate+Outputs"></a>
+<a name="N10B4C"></a><a name="Intermediate+Outputs"></a>
<h5>Intermediate Outputs</h5>
<p>Applications can control compression of intermediate map-outputs
via the
@@ -1897,7 +1901,7 @@
<a href="api/org/apache/hadoop/mapred/JobConf.html#setMapOutputCompressionType(org.apache.hadoop.io.SequenceFile.CompressionType)">
JobConf.setMapOutputCompressionType(SequenceFile.CompressionType)</a>
api.</p>
-<a name="N10B75"></a><a name="Job+Outputs"></a>
+<a name="N10B78"></a><a name="Job+Outputs"></a>
<h5>Job Outputs</h5>
<p>Applications can control compression of job-outputs via the
<a href="api/org/apache/hadoop/mapred/OutputFormatBase.html#setCompressOutput(org.apache.hadoop.mapred.JobConf,%20boolean)">
@@ -1917,7 +1921,7 @@
</div>
-<a name="N10BA4"></a><a name="Example%3A+WordCount+v2.0"></a>
+<a name="N10BA7"></a><a name="Example%3A+WordCount+v2.0"></a>
<h2 class="h3">Example: WordCount v2.0</h2>
<div class="section">
<p>Here is a more complete <span class="codefrag">WordCount</span> which uses many of the
@@ -1927,7 +1931,7 @@
<a href="quickstart.html#SingleNodeSetup">pseudo-distributed</a> or
<a href="quickstart.html#Fully-Distributed+Operation">fully-distributed</a>
Hadoop installation.</p>
-<a name="N10BBE"></a><a name="Source+Code-N10BBE"></a>
+<a name="N10BC1"></a><a name="Source+Code-N10BC1"></a>
<h3 class="h4">Source Code</h3>
<table class="ForrestTable" cellspacing="1" cellpadding="4">
@@ -3137,7 +3141,7 @@
</tr>
</table>
-<a name="N11320"></a><a name="Sample+Runs"></a>
+<a name="N11323"></a><a name="Sample+Runs"></a>
<h3 class="h4">Sample Runs</h3>
<p>Sample text-files as input:</p>
<p>
@@ -3305,7 +3309,7 @@
<br>
</p>
-<a name="N113F4"></a><a name="Highlights"></a>
+<a name="N113F7"></a><a name="Highlights"></a>
<h3 class="h4">Highlights</h3>
<p>The second version of <span class="codefrag">WordCount</span> improves upon the
previous one by using some features offered by the Map-Reduce framework: