You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Pedro Costa <ps...@gmail.com> on 2011/02/02 14:50:12 UTC

how set compression in the map output?

Hi,

I'm running the wordcount example, but I would like compress the map output.

I set the following properties in the mapred-site.xml

[code]
<property>
        <name>mapred.compress.map.output</name>
        <value>true</value>
</property>

<property>
        <name>mapred.map.output.compression.codec</name>
        <value>gzip</value>
</property>
[/code]

but I still got the error:

java.lang.IllegalArgumentException: Compression codec gzip was not found.
	at org.apache.hadoop.mapred.JobConf.getMapOutputCompressorClass(JobConf.java:512)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:846)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:531)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:627)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.Child.main(Child.java:190)



I run the wordcount with the command:

[code]
bin/hadoop jar build/hadoop-0.20.2-dev-examples.jar wordcount
gutenberg gutenberg-output
[/code]

How I set the compression in the map output?

Thanks,
-- 
Pedro

Re: how set compression in the map output?

Posted by Koji Noguchi <kn...@yahoo-inc.com>.
Try
org.apache.hadoop.io.compress.GzipCodec

Koji


On 2/2/11 5:50 AM, "Pedro Costa" <ps...@gmail.com> wrote:

Hi,

I'm running the wordcount example, but I would like compress the map output.

I set the following properties in the mapred-site.xml

[code]
<property>
        <name>mapred.compress.map.output</name>
        <value>true</value>
</property>

<property>
        <name>mapred.map.output.compression.codec</name>
        <value>gzip</value>
</property>
[/code]

but I still got the error:

java.lang.IllegalArgumentException: Compression codec gzip was not found.
        at org.apache.hadoop.mapred.JobConf.getMapOutputCompressorClass(JobConf.java:512)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:846)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:531)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:627)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.Child.main(Child.java:190)



I run the wordcount with the command:

[code]
bin/hadoop jar build/hadoop-0.20.2-dev-examples.jar wordcount
gutenberg gutenberg-output
[/code]

How I set the compression in the map output?

Thanks,
--
Pedro