Custom compression codec integration

[Kévin Poupon <po...@gmail.com>]

Hello,

I developped a custom compression codec for a 2-nodes Hadoop cluster. Let's
call it my.custom.ComprCodec.
It works well in Eclipse test environnement but I have troubles integrating
it to Hadoop.

I updated the following properties via Cloudera:
io.compression.codecs: I added my.custom.ComprCodec
mapreduce.output.fileoutputformat.compress: set to true (ckeckmarked)
mapreduce.output.fileoutputformat.compress.codec: set to
my.custom.ComprCodec
I then placed the codec.jar on the HDFS NameNode machine in
/opt/cloudera/parcels/CDH-5.0.2-1.cdh5.0.2.p0.13/lib/hadoop/lib/, next to
Snappy codec.

Then to try the codec I launch a streaming job:

hadoop jar
/opt/cloudera/parcels/CDH-5.0.2-1.cdh5.0.2.p0.13/lib/hadoop-mapreduce/hadoop-streaming.jar
-input /user/hadoop/compr/ -output /user/hadoop/decompr/ -mapper cat
-reducer cat

but the job fails:

14/07/09 14:43:46 INFO mapreduce.Job:  map 0% reduce 0%
14/07/09 14:43:55 INFO mapreduce.Job:  map 50% reduce 0%
14/07/09 14:44:02 INFO mapreduce.Job:  map 100% reduce 0%
14/07/09 14:44:09 INFO mapreduce.Job: Task Id :
attempt_1404901272360_0003_r_000000_0, Status : FAILED
14/07/09 14:44:15 INFO mapreduce.Job: Task Id :
attempt_1404901272360_0003_r_000001_0, Status : FAILED
14/07/09 14:44:21 INFO mapreduce.Job: Task Id :
attempt_1404901272360_0003_r_000000_1, Status : FAILED
14/07/09 14:44:27 INFO mapreduce.Job: Task Id :
attempt_1404901272360_0003_r_000001_1, Status : FAILED
14/07/09 14:44:33 INFO mapreduce.Job: Task Id :
attempt_1404901272360_0003_r_000000_2, Status : FAILED
14/07/09 14:44:38 INFO mapreduce.Job: Task Id :
attempt_1404901272360_0003_r_000001_2, Status : FAILED
14/07/09 14:44:44 INFO mapreduce.Job:  map 100% reduce 100%
14/07/09 14:44:44 INFO mapreduce.Job: Job job_1404901272360_0003 failed
with state FAILED due to: Task failed task_1404901272360_0003_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1

Note that the job works well when the compression codec to use is set to be
Snappy.

How to integrate a custom codec to Hadoop? What did I forget ?

Thank you