You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Marc Sturlese <ma...@gmail.com> on 2012/03/02 04:35:55 UTC
Re: LZO exception decompressing (returned -8)
Tried but still getting the error 0.4.15. Really lost with this.
My hadoop release is 0.20.2 from more than a year ago. Could this be related
to the problem?
--
View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3792484.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: LZO exception decompressing (returned -8)
Posted by Marc Sturlese <ma...@gmail.com>.
Absolutely. In case I don't find the root of the problem soon I'll definitely
try it.
--
View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3792531.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: LZO exception decompressing (returned -8)
Posted by Marc Sturlese <ma...@gmail.com>.
Absolutely. In case I don't find the root of the problem soon I'll definitely
try it.
--
View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3792530.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: LZO exception decompressing (returned -8)
Posted by Joey Echeverria <jo...@cloudera.com>.
I know this doesn't fix lzo, but have you considered Snappy for the
intermediate output compression? It gets similar compression ratios
and compress/decompress speed, but arguably has better Hadoop
integration.
-Joey
On Thu, Mar 1, 2012 at 10:01 PM, Marc Sturlese <ma...@gmail.com> wrote:
> I use to have 2.05 but now as I said I installed 2.06
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3792511.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
Re: LZO exception decompressing (returned -8)
Posted by Marc Sturlese <ma...@gmail.com>.
I use to have 2.05 but now as I said I installed 2.06
--
View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3792511.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: LZO exception decompressing (returned -8)
Posted by Marc Sturlese <ma...@gmail.com>.
Yes, The steps I followed where:
1-Intall lzo 2.06 in a machine with the same kernel as my nodes.
2-Compile there lzo 0.4.15 (in /lib replaced cdh3u3 per my hadoop 0.20.2
release)
3-Replace hadoop-lzo-0.4.9.jar for the now compiled hadoop-lzo-0.4.15.jar in
the hadoop lib directory of all my nodes and master
4-Put de generated native files in the native lib directory of all the nodes
and master
5-In my jar job, replaced the jar library hadoop-lzo-0.4.9.jar for
hadoop-lzo-0.4.15.jar
And sometimes when a job is running I get (4 times so the job gets killed):
...org.apache.hadoop.mapred.ReduceTask: Shuffling 3188320 bytes (1025174 raw
bytes) into RAM from attempt_201202291221_1501_m_000480_0
2012-03-02 02:32:55,496 INFO org.apache.hadoop.mapred.ReduceTask: Task
attempt_201202291221_1501_r_000105_0: Failed fetch #1 from
attempt_201202291221_1501_m_000046_0
2012-03-02 02:32:55,496 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_201202291221_1501_r_000105_0 adding host hadoop-01.backend to
penalty box, next contact in 4 seconds
2012-03-02 02:32:55,496 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201202291221_1501_r_000105_0: Got 1 map-outputs from previous
failures
2012-03-02 02:32:55,497 FATAL org.apache.hadoop.mapred.TaskRunner:
attempt_201202291221_1501_r_000105_0 : Map output copy failure :
java.lang.InternalError: lzo1x_decompress returned: -8
at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect(Native
Method)
at
com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:305)
at
org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:76)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1553)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216)
--
View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3792505.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: LZO exception decompressing (returned -8)
Posted by Harsh J <ha...@cloudera.com>.
Marc,
Was the lzo libs on your server upgraded to a higher version recently?
Also, when you deployed a built copy of 0.4.15, did you ensure you
replaced the older native libs for hadoop-lzo as well?
On Fri, Mar 2, 2012 at 9:05 AM, Marc Sturlese <ma...@gmail.com> wrote:
> Tried but still getting the error 0.4.15. Really lost with this.
> My hadoop release is 0.20.2 from more than a year ago. Could this be related
> to the problem?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3792484.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
--
Harsh J