You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by jo...@apache.org on 2009/03/05 20:07:38 UTC
svn commit: r750552 - in /hadoop/core/trunk: CHANGES.txt
src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
src/docs/src/documentation/content/xdocs/site.xml
src/docs/src/documentation/content/xdocs/streaming.xml
Author: johan
Date: Thu Mar 5 19:07:37 2009
New Revision: 750552
URL: http://svn.apache.org/viewvc?rev=750552&view=rev
Log:
HADOOP-5317. Provide documentation for LazyOutput Feature. (Jothi Padmanabhan via johan)
Modified:
hadoop/core/trunk/CHANGES.txt
hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml
hadoop/core/trunk/src/docs/src/documentation/content/xdocs/streaming.xml
Modified: hadoop/core/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=750552&r1=750551&r2=750552&view=diff
==============================================================================
--- hadoop/core/trunk/CHANGES.txt (original)
+++ hadoop/core/trunk/CHANGES.txt Thu Mar 5 19:07:37 2009
@@ -146,6 +146,9 @@
HADOOP-4546. Fix DF reporting for AIX. (Bill Habermaas via cdouglas)
HADOOP-5023. Add Tomcat support to HdfsProxy. (Zhiyong Zhang via cdouglas)
+
+ HADOOP-5317. Provide documentation for LazyOutput Feature.
+ (Jothi Padmanabhan via johan)
OPTIMIZATIONS
Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml?rev=750552&r1=750551&r2=750552&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml Thu Mar 5 19:07:37 2009
@@ -1579,6 +1579,25 @@
<code>OutputFormat</code>.</p>
<section>
+ <title>Lazy Output Creation</title>
+ <p>It is possible to delay creation of output until the first write attempt
+ by using <a href="ext:api/org/apache/hadoop/mapred/lib/lazyoutputformat">
+ LazyOutputFormat</a>. This is particularly useful in preventing the
+ creation of zero byte files when there is no call to output.collect
+ (or Context.write). This is achieved by calling the static method
+ <code>setOutputFormatClass</code> of <code>LazyOutputFormat</code>
+ with the intended <code>OutputFormat</code> as the argument. The following example
+ shows how to delay creation of files when using the <code>TextOutputFormat</code>
+ </p>
+
+ <p>
+ <code> import org.apache.hadoop.mapred.lib.LazyOutputFormat;</code> <br/>
+ <code> LazyOutputFormat.setOutputFormatClass(conf, TextOutputFormat.class);</code>
+ </p>
+
+ </section>
+
+ <section>
<title>OutputCommitter</title>
<p><a href="ext:api/org/apache/hadoop/mapred/outputcommitter">
Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml?rev=750552&r1=750551&r2=750552&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml Thu Mar 5 19:07:37 2009
@@ -236,6 +236,7 @@
<hashpartitioner href="HashPartitioner.html" />
<keyfieldbasedpartitioner href="KeyFieldBasedPartitioner.html" />
<keyfieldbasedcomparator href="KeyFieldBasedComparator.html" />
+ <lazyoutputformat href="LazyOutputFormat.html" />
<aggregate href="aggregate/">
<package-summary href="package-summary.html" />
</aggregate>
Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/streaming.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/streaming.xml?rev=750552&r1=750551&r2=750552&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/streaming.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/streaming.xml Thu Mar 5 19:07:37 2009
@@ -239,6 +239,7 @@
<tr><td> -inputreader JavaClassName </td><td> Optional </td><td> For backwards-compatibility: specifies a record reader class (instead of an input format class) </td></tr>
<tr><td> -verbose </td><td> Optional </td><td> Verbose output </td></tr>
+<tr><td> -lazyOutput </td><td> Optional </td><td> Create output lazily. For example, if the output format is based on FileOutputFormat, the output file is created only on the first call to output.collect (or Context.write)</td></tr>
</table>
<p>
Streaming support Hadoop generic command line options.