You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Pedro Costa <ps...@gmail.com> on 2011/02/02 14:50:12 UTC
how set compression in the map output?
Hi,
I'm running the wordcount example, but I would like compress the map output.
I set the following properties in the mapred-site.xml
[code]
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>gzip</value>
</property>
[/code]
but I still got the error:
java.lang.IllegalArgumentException: Compression codec gzip was not found.
at org.apache.hadoop.mapred.JobConf.getMapOutputCompressorClass(JobConf.java:512)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:846)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:531)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:627)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.Child.main(Child.java:190)
I run the wordcount with the command:
[code]
bin/hadoop jar build/hadoop-0.20.2-dev-examples.jar wordcount
gutenberg gutenberg-output
[/code]
How I set the compression in the map output?
Thanks,
--
Pedro
Re: how set compression in the map output?
Posted by Koji Noguchi <kn...@yahoo-inc.com>.
Try
org.apache.hadoop.io.compress.GzipCodec
Koji
On 2/2/11 5:50 AM, "Pedro Costa" <ps...@gmail.com> wrote:
Hi,
I'm running the wordcount example, but I would like compress the map output.
I set the following properties in the mapred-site.xml
[code]
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>gzip</value>
</property>
[/code]
but I still got the error:
java.lang.IllegalArgumentException: Compression codec gzip was not found.
at org.apache.hadoop.mapred.JobConf.getMapOutputCompressorClass(JobConf.java:512)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:846)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:531)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:627)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.Child.main(Child.java:190)
I run the wordcount with the command:
[code]
bin/hadoop jar build/hadoop-0.20.2-dev-examples.jar wordcount
gutenberg gutenberg-output
[/code]
How I set the compression in the map output?
Thanks,
--
Pedro