You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Jean-Daniel Cryans <jd...@apache.org> on 2010/01/06 19:42:23 UTC

SF HBase User Group Meetup Jan. 27th @ StumbleUpon

Hi all,

This year's first San Francisco HBase User Group meetup takes place on
January 27th at StumbleUpon. The first talk will be about the upcoming
versions, others to be announced.

RSVP at: http://su.pr/6Cldz7

See you there!

J-D

Re: Hbase table querying using 'like' concept of SQL

Posted by Jeff Zhang <zj...@gmail.com>.

I think you can use PrefixFilter to do the like operation.



On Fri, Jan 8, 2010 at 2:27 PM, Sriram Muthuswamy Chittathoor <
sriramc@ivycomptech.com> wrote:

> Hi:
>
>
>
> I am trying to evaluate this for a hbase table which has around 30
> million records.  The table stores user information like
>
> {accountName , screenName, firstName , lastName, dataOfBirth} etc.  The
> idea is to be able to query the rows using  "like" clause.
>
>
>
> So something like      select  * from userProfile  with ....  where
> accountName like 'po*'  and  screenName like 'scr*'
>
>
>
> What would be the most performant way to do this --  can I use some
> tricks or do I need to get the rows and filter it myself  or  not a good
> use case for hbase ?
>
>
>
>
>
> Thanks for the help
>
>
>
> Sriram C
>
>
>
>
> This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy
> Comptech Private Limited is a limited liability company.
>
> This email and any attachments are confidential, and may be legally
> privileged and protected by copyright. If you are not the intended recipient
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then delete
> the email completely from your system.
> Any views or opinions are solely those of the sender.  This communication
> is not intended to form a binding contract on behalf of Ivy Comptech Private
> Limited unless expressly indicated to the contrary and properly authorised.
> Any actions taken on the basis of this email are at the recipient's own
> risk.
>
> Registered office:
> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> Registered in India. A list of members' names is available for inspection at
> the registered office.
>
>


-- 
Best Regards

Jeff Zhang

Hbase table querying using 'like' concept of SQL

Posted by Sriram Muthuswamy Chittathoor <sr...@ivycomptech.com>.

Hi:

I am trying to evaluate this for a hbase table which has around 30
million records. The table stores user information like

{accountName , screenName, firstName , lastName, dataOfBirth} etc. The
idea is to be able to query the rows using "like" clause.

So something like select * from userProfile with .... where
accountName like 'po*' and screenName like 'scr*'

What would be the most performant way to do this -- can I use some
tricks or do I need to get the rows and filter it myself or not a good
use case for hbase ?

Thanks for the help

Sriram C

This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy Comptech Private Limited is a limited liability company.

This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system.
Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract on behalf of Ivy Comptech Private Limited unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.

Registered office:
Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills, Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994. Registered in India. A list of members' names is available for inspection at the registered office.

RE: Hbase loading error -- Trailer 'header' is wrong; does the trailer size match content

Posted by Sriram Muthuswamy Chittathoor <sr...@ivycomptech.com>.

Thanks.  I disabled speculative execution and it seems to have worked.
Will try it on bigger sizes to see if it is consistent.  Currently I
write some 20 milli  rows using a MR job  (very simple row though).  

 

________________________________

From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
stack
Sent: Friday, January 08, 2010 10:42 AM
To: Sriram Muthuswamy Chittathoor
Cc: hbase-user@hadoop.apache.org
Subject: Re: Hbase loading error -- Trailer 'header' is wrong; does the
trailer size match content

 

I think the failed tasks are leaving around incomplete hfiles.  Search
for empty hfiles, try and correlate them to the failed tasks.  If they
match, just remove them before running loadtable.rb.  Otherwise, try and
figure how the bad hfiles were written.  Maybe the task log will give us
a clue?

 

Did you disable speculative execution?  

St.Ack

 

On Thu, Jan 7, 2010 at 5:49 AM, Sriram Muthuswamy Chittathoor
<sr...@ivycomptech.com> wrote:

Hi:

I am trying to run a MR job to output HFiles directly containing 10
million records (very simple 1 column family and very small).  The job
completes with some mention about killed jobs (reduce Failed/Killed
Task Attempts > 0) .  Then I use the script loadtable.rb to load my
hfiles into hbase and get the error stack given below.

I tried all combinations of settings in the mapred-site.xml. Also tried
the suggestions given in this chain which talks about similar problem.
http://www.mail-archive.com/hbase-user@hadoop.apache.org/msg07668.html


If I reduce the number of records it works.

Thanks for any help

Sriram C



org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
java.io.IOException: Trailer 'header' is  wrong; does the trailer size
match content? (NativeException)
       from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in
`readTrailer'
       from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
`loadFileInfo'
       from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
       from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
       from sun/reflect/DelegatingMethodAccessorImpl.java:25:in
`invoke'
       from java/lang/reflect/Method.java:597:in `invoke'
       from org/jruby/javasupport/JavaMethod.java:298:in
`invokeWithExceptionHandling'
       from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
        ... 19 levels...
       from org/jruby/Main.java:94:in `main'
       from bin/loadtable.rb:83:in `each'
       from bin/loadtable.rb:83
Complete Java stackTrace
java.io.IOException: Trailer 'header' is wrong; does the trailer size
match content?
       at
org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFil
e.java:1335)
       at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813
)
       at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:75
8)



This email is sent for and on behalf of Ivy Comptech Private Limited.
Ivy Comptech Private Limited is a limited liability company.

This email and any attachments are confidential, and may be legally
privileged and protected by copyright. If you are not the intended
recipient dissemination or copying of this email is prohibited. If you
have received this in error, please notify the sender by replying by
email and then delete the email completely from your system.
Any views or opinions are solely those of the sender.  This
communication is not intended to form a binding contract on behalf of
Ivy Comptech Private Limited unless expressly indicated to the contrary
and properly authorised. Any actions taken on the basis of this email
are at the recipient's own risk.

Registered office:
Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
Registered in India. A list of members' names is available for
inspection at the registered office.

Re: Hbase loading error -- Trailer 'header' is wrong; does the trailer size match content

Posted by stack <st...@duboce.net>.

I think the failed tasks are leaving around incomplete hfiles.  Search for
empty hfiles, try and correlate them to the failed tasks.  If they match,
just remove them before running loadtable.rb.  Otherwise, try and figure how
the bad hfiles were written.  Maybe the task log will give us a clue?

Did you disable speculative execution?

St.Ack

On Thu, Jan 7, 2010 at 5:49 AM, Sriram Muthuswamy Chittathoor <
sriramc@ivycomptech.com> wrote:

> Hi:
>
> I am trying to run a MR job to output HFiles directly containing 10
> million records (very simple 1 column family and very small).  The job
> completes with some mention about killed jobs (reduce Failed/Killed
> Task Attempts > 0) .  Then I use the script loadtable.rb to load my
> hfiles into hbase and get the error stack given below.
>
> I tried all combinations of settings in the mapred-site.xml. Also tried
> the suggestions given in this chain which talks about similar problem.
> http://www.mail-archive.com/hbase-user@hadoop.apache.org/msg07668.html
>
>
> If I reduce the number of records it works.
>
> Thanks for any help
>
> Sriram C
>
>
>
> org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> java.io.IOException: Trailer 'header' is  wrong; does the trailer size
> match content? (NativeException)
>        from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in
> `readTrailer'
>        from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> `loadFileInfo'
>        from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
>        from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
>        from sun/reflect/DelegatingMethodAccessorImpl.java:25:in
> `invoke'
>        from java/lang/reflect/Method.java:597:in `invoke'
>        from org/jruby/javasupport/JavaMethod.java:298:in
> `invokeWithExceptionHandling'
>        from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>         ... 19 levels...
>        from org/jruby/Main.java:94:in `main'
>        from bin/loadtable.rb:83:in `each'
>        from bin/loadtable.rb:83
> Complete Java stackTrace
> java.io.IOException: Trailer 'header' is wrong; does the trailer size
> match content?
>        at
> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFil
> e.java:1335)
>        at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813
> )
>        at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:75
> 8)
>
>
>
> This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy
> Comptech Private Limited is a limited liability company.
>
> This email and any attachments are confidential, and may be legally
> privileged and protected by copyright. If you are not the intended recipient
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then delete
> the email completely from your system.
> Any views or opinions are solely those of the sender.  This communication
> is not intended to form a binding contract on behalf of Ivy Comptech Private
> Limited unless expressly indicated to the contrary and properly authorised.
> Any actions taken on the basis of this email are at the recipient's own
> risk.
>
> Registered office:
> Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills,
> Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994.
> Registered in India. A list of members' names is available for inspection at
> the registered office.
>
>

Hbase loading error -- Trailer 'header' is wrong; does the trailer size match content

Posted by Sriram Muthuswamy Chittathoor <sr...@ivycomptech.com>.

Hi:

I am trying to run a MR job to output HFiles directly containing 10
million records (very simple 1 column family and very small). The job
completes with some mention about killed jobs (reduce Failed/Killed
Task Attempts > 0) . Then I use the script loadtable.rb to load my
hfiles into hbase and get the error stack given below.

I tried all combinations of settings in the mapred-site.xml. Also tried
the suggestions given in this chain which talks about similar problem.
http://www.mail-archive.com/hbase-user@hadoop.apache.org/msg07668.html

If I reduce the number of records it works.

Thanks for any help

Sriram C

org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
java.io.IOException: Trailer 'header' is wrong; does the trailer size
match content? (NativeException)
from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in
`readTrailer'
from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
`loadFileInfo'
from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in
`invoke'
from java/lang/reflect/Method.java:597:in `invoke'
from org/jruby/javasupport/JavaMethod.java:298:in
`invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
... 19 levels...
from org/jruby/Main.java:94:in `main'
from bin/loadtable.rb:83:in `each'
from bin/loadtable.rb:83
Complete Java stackTrace
java.io.IOException: Trailer 'header' is wrong; does the trailer size
match content?
at
org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFil
e.java:1335)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813
)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:75
8)

This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy Comptech Private Limited is a limited liability company.

Hbase loading error -- Trailer 'header' is wrong; does the trailer size match content

Posted by Sriram Muthuswamy Chittathoor <sr...@ivycomptech.com>.

Hi:

If I reduce the number of records it works.

Thanks for any help

Sriram C

This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy Comptech Private Limited is a limited liability company.

Hbase loading error -- Trailer 'header' is wrong; does the trailer size match content

Posted by Sriram Muthuswamy Chittathoor <sr...@ivycomptech.com>.

Hi:

If I reduce the number of records it works.

Thanks for any help

Sriram C

This email is sent for and on behalf of Ivy Comptech Private Limited. Ivy Comptech Private Limited is a limited liability company.