RE: Hbase loading error -- Trailer 'header' is wrong; does the trailer size match content

[Sriram Muthuswamy Chittathoor <sr...@ivycomptech.com>]

Thanks.  I disabled speculative execution and it seems to have worked.
Will try it on bigger sizes to see if it is consistent.  Currently I
write some 20 milli  rows using a MR job  (very simple row though).  

 

________________________________

From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
stack
Sent: Friday, January 08, 2010 10:42 AM
To: Sriram Muthuswamy Chittathoor
Cc: hbase-user@hadoop.apache.org
Subject: Re: Hbase loading error -- Trailer 'header' is wrong; does the
trailer size match content

 

I think the failed tasks are leaving around incomplete hfiles.  Search
for empty hfiles, try and correlate them to the failed tasks.  If they
match, just remove them before running loadtable.rb.  Otherwise, try and
figure how the bad hfiles were written.  Maybe the task log will give us
a clue?

 

Did you disable speculative execution?  

St.Ack

 

On Thu, Jan 7, 2010 at 5:49 AM, Sriram Muthuswamy Chittathoor
<sr...@ivycomptech.com> wrote:

Hi:

I am trying to run a MR job to output HFiles directly containing 10
million records (very simple 1 column family and very small).  The job
completes with some mention about killed jobs (reduce Failed/Killed
Task Attempts > 0) .  Then I use the script loadtable.rb to load my
hfiles into hbase and get the error stack given below.

I tried all combinations of settings in the mapred-site.xml. Also tried
the suggestions given in this chain which talks about similar problem.
http://www.mail-archive.com/hbase-user@hadoop.apache.org/msg07668.html


If I reduce the number of records it works.

Thanks for any help

Sriram C



org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
java.io.IOException: Trailer 'header' is  wrong; does the trailer size
match content? (NativeException)
       from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in
`readTrailer'
       from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
`loadFileInfo'
       from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
       from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
       from sun/reflect/DelegatingMethodAccessorImpl.java:25:in
`invoke'
       from java/lang/reflect/Method.java:597:in `invoke'
       from org/jruby/javasupport/JavaMethod.java:298:in
`invokeWithExceptionHandling'
       from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
        ... 19 levels...
       from org/jruby/Main.java:94:in `main'
       from bin/loadtable.rb:83:in `each'
       from bin/loadtable.rb:83
Complete Java stackTrace
java.io.IOException: Trailer 'header' is wrong; does the trailer size
match content?
       at
org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFil
e.java:1335)
       at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813
)
       at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:75
8)



This email is sent for and on behalf of Ivy Comptech Private Limited.
Ivy Comptech Private Limited is a limited liability company.

This email and any attachments are confidential, and may be legally
privileged and protected by copyright. If you are not the intended
recipient dissemination or copying of this email is prohibited. If you
have received this in error, please notify the sender by replying by
email and then delete the email completely from your system.
Any views or opinions are solely those of the sender.  This
communication is not intended to form a binding contract on behalf of
Ivy Comptech Private Limited unless expressly indicated to the contrary
and properly authorised. Any actions taken on the basis of this email
are at the recipient's own risk.

Registered office:
Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
Registered in India. A list of members' names is available for
inspection at the registered office.