problem using Hadoop 0.18.3 with lzo 2.03

[Bill Au <bi...@gmail.com>]

I am using Hadoop 0.18.3 with lzo 2.03.  I am able to compile Hadoop's
native code and load lzo's native library.  I am trying to run the grep
example in examples.jar on a lzo-compressed file.  I am getting an
OutOfMemoryError on the Java heap space.  My input file is 1628727 bytes
which compressed into 157879 bytes.  My map/reduce child java process is
running with a 2000m heap.  So I think that ought to be enough.  What am I
doing wrong?
Bill

<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx2000m</value>
  <description>Java opts for the task tracker child processes.
  The following symbol, if present, will be interpolated: @taskid@ is
replaced
  by current TaskID. Any other occurrences of '@' will go unchanged.
  For example, to enable verbose gc logging to a file named for the taskid
in
  /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
        -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc

  The configuration variable mapred.child.ulimit can be used to control the
  maximum virtual memory of the child processes.
  </description>
</property>


2009-08-21 14:48:44,043 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2009-08-21 14:48:44,158 INFO org.apache.hadoop.mapred.MapTask:
numReduceTasks: 1
2009-08-21 14:48:44,175 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2009-08-21 14:48:44,286 INFO org.apache.hadoop.mapred.MapTask: data buffer =
79691776/99614720
2009-08-21 14:48:44,286 INFO org.apache.hadoop.mapred.MapTask: record buffer
= 262144/327680
2009-08-21 14:48:44,379 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded
the native-hadoop library
2009-08-21 14:48:44,382 INFO org.apache.hadoop.io.compress.LzoCodec:
Successfully loaded & initialized native-lzo library
2009-08-21 14:48:44,591 WARN org.apache.hadoop.mapred.TaskTracker: Error
running child
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:100)
at
org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at java.io.InputStream.read(InputStream.java:85)
at
org.apache.hadoop.mapred.LineRecordReader$LineReader.backfill(LineRecordReader.java:94)
at
org.apache.hadoop.mapred.LineRecordReader$LineReader.readLine(LineRecordReader.java:124)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:266)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:165)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)