You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by llpind <so...@hotmail.com> on 2009/05/15 22:11:29 UTC

Loading large resultset into HBase

Hey all,

I'm loading data from a DB into HBase.  I have a single java process
iterating over a ResultSet.  After about 10,000 rows i do a BatchUpdate. 
I've changed the Heap size of both Hadoop & HBase to 2000.  

Setup:  0.19.1.  1 box with master and secondary.  3 boxes with
HRegionServer.

Problem 1:  The load seems to be unblananced:  
 Address Start Code Load 
1:60020 1242415770566 requests=0, regions=1, usedHeap=39, maxHeap=1777 
2:60020 1242415770417 requests=3, regions=3, usedHeap=52, maxHeap=1777 
3:60020 1242415770273 requests=1, regions=3, usedHeap=604, maxHeap=1777 
Total:  servers: 3   requests=4, regions=7 

Problem 2:  Around 10 Million rows, the upload starts to slow down.  

The upload is still going, so i'll update on what happens.  
-- 
View this message in context: http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23566568.html
Sent from the HBase User mailing list archive at Nabble.com.


Re: Loading large resultset into HBase

Posted by Andrew Purtell <ap...@apache.org>.
I've seen this too. Seems to correlate with uneven block placement in HDFS, so maybe it is cache of a hot region.




________________________________
From: stack <st...@duboce.net>
To: hbase-user@hadoop.apache.org
Sent: Friday, May 15, 2009 1:38:22 PM
Subject: Re: Loading large resultset into HBase

On Fri, May 15, 2009 at 1:11 PM, llpind <so...@hotmail.com> wrote:

>
> Hey all,
>
> I'm loading data from a DB into HBase.  I have a single java process
> iterating over a ResultSet.  After about 10,000 rows i do a BatchUpdate.
> I've changed the Heap size of both Hadoop & HBase to 2000.
>
> Setup:  0.19.1.  1 box with master and secondary.  3 boxes with
> HRegionServer.
>
> Problem 1:  The load seems to be unblananced:
>  Address Start Code Load
> 1:60020 1242415770566 requests=0, regions=1, usedHeap=39, maxHeap=1777
> 2:60020 1242415770417 requests=3, regions=3, usedHeap=52, maxHeap=1777
> 3:60020 1242415770273 requests=1, regions=3, usedHeap=604, maxHeap=1777
> Total:  servers: 3   requests=4, regions=7
>



Yeah, it looks unbalanced memory-wise but not too off regions wise.  What
happens if you load more into your cluster?  And used heap fluctuates,
doesn't it?



>
> Problem 2:  Around 10 Million rows, the upload starts to slow down.
>


Turn on DEBUG logging -- see FAQ on wiki for how -- and then tail one of
your regionserver logs.  Whats going on?

It might be the fact that the commit log buffer is small in hbase 0.19.1.  I
found this issue recently in
https://issues.apache.org/jira/browse/HBASE-1394.   Change
hbase.regionserver.hlog.blocksize from its 1MB size up to say, 64MB?



>
> The upload is still going, so i'll update on what happens.


Please do.

St.Ack



>
> --
> View this message in context:
> http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23566568.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>



      

Re: Loading large resultset into HBase

Posted by llpind <so...@hotmail.com>.
Thanks.  How can I handle millions of columns?  .20?  If not, how can I
handle a data model similar to:
http://www.nabble.com/HBase-Data-Model-td23511426.html


-- 
View this message in context: http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23620679.html
Sent from the HBase User mailing list archive at Nabble.com.


Re: Loading large resultset into HBase

Posted by stack <st...@duboce.net>.
On Sat, May 16, 2009 at 7:02 AM, llpind <so...@hotmail.com> wrote:

>
> @Ryan
>
> Thanks.  how do i turn off authflushing?



http://hadoop.apache.org/hbase/docs/r0.19.2/api/org/apache/hadoop/hbase/client/HTable.html#setAutoFlush(boolean)



>  also, do you have an example of
> map/reduce which uses a Resultset?



See the mapred package in src.


>
>
> Yes. for a given row key, i have potentially millions of columns.  so i was
> doing 10K cols per row in the batch update.



This will give you grief in 0.19.x hbase:
https://issues.apache.org/jira/browse/HBASE-867.  The more columns you have,
currently, the slower hbase will run.

St.Ack



>
> --
> View this message in context:
> http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23574048.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>

Re: Loading large resultset into HBase

Posted by llpind <so...@hotmail.com>.
@Ryan

Thanks.  how do i turn off authflushing?  also, do you have an example of
map/reduce which uses a Resultset?  

Yes. for a given row key, i have potentially millions of columns.  so i was
doing 10K cols per row in the batch update.
-- 
View this message in context: http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23574048.html
Sent from the HBase User mailing list archive at Nabble.com.


Re: Loading large resultset into HBase

Posted by Ryan Rawson <ry...@gmail.com>.
A batch update holds the puts for a single row, are you really updating 5k
cols per row?

generally i turn autoflushing off, use a 1-12mb client buffer.

even so, you can see 'cant find address' or 'not serving region exception'
during a transition of regions.  check the region historian for the region
in question to see if there were moves.

one other thing, are you getting exceptions throw from table.commit() ?  the
existing client will retry to smooth over region transitions, so you can get
exceptions in the logfile but not actually be thrown from table.commit().

One thing I do is do my uploads using java programs and map reduces - this
ensures maximal performance and lets me do automated retries.  24m rows
should import in no time.  I can do about 500m rows in 4 hours...

good luck!
-ryan

On Fri, May 15, 2009 at 4:53 PM, llpind <so...@hotmail.com> wrote:

>
> Hey All,
>
> Finally finished.  But didn't complete (~ between 24,770,000 - 25,760,000
> records processed).  Got the following exception before java upload app
> died:
>
> org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address
> listed in .META. for region
>
> I believe it has to do with "Problem: DFS instability and/or regionserver
> lease timeouts" (http://wiki.apache.org/hadoop/Hbase/Troubleshooting).  So
> i
> was going to try to put some JVM flags in for GC.  Also, when creating the
> table, is there any column options I should use that will help in upload?
>
> When i try to get results from within HBase shell it gives the following
> exception:
>
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException:
> <TABLE_NAME>,<ROWID>,1242429308026
>        at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2071)
>        at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1534)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>
>
>        from org/apache/hadoop/hbase/client/HConnectionManager.java:858:in
> `getRegionServerWithRetries'
>        from org/apache/hadoop/hbase/client/HTable.java:608:in `getRow'
>        from org/apache/hadoop/hbase/client/HTable.java:583:in `getRow'
>        from org/apache/hadoop/hbase/client/HTable.java:478:in `getRow'
>        from org/apache/hadoop/hbase/client/HTable.java:425:in `getRow'
>        from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
>        from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
>        from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>        from java/lang/reflect/Method.java:597:in `invoke'
>        from org/jruby/javasupport/JavaMethod.java:298:in
> `invokeWithExceptionHandling'
>        from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>        from org/jruby/java/invokers/InstanceMethodInvoker.java:44:in `call'
>        from org/jruby/runtime/callsite/CachingCallSite.java:280:in
> `cacheAndCall'
>        from org/jruby/runtime/callsite/CachingCallSite.java:116:in `call'
>        from org/jruby/ast/CallOneArgNode.java:57:in `interpret'
>        from org/jruby/ast/LocalAsgnNode.java:120:in `interpret'
> ... 120 levels...
>        from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in
> `call'
>        from org/jruby/internal/runtime/methods/CompiledMethod.java:216:in
> `call'
>        from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in
> `call'
>        from org/jruby/runtime/callsite/CachingCallSite.java:260:in
> `cacheAndCall'
>        from org/jruby/runtime/callsite/CachingCallSite.java:75:in `call'
>        from home/hadoop/hbase19/bin/$_dot_dot_/bin/hirb.rb:441:in
> `__file__'
>        from home/hadoop/hbase19/bin/$_dot_dot_/bin/hirb.rb:-1:in `__file__'
>        from home/hadoop/hbase19/bin/$_dot_dot_/bin/hirb.rb:-1:in `load'
>        from org/jruby/Ruby.java:564:in `runScript'
>        from org/jruby/Ruby.java:467:in `runNormally'
>        from org/jruby/Ruby.java:340:in `runFromMain'
>        from org/jruby/Main.java:214:in `run'
>        from org/jruby/Main.java:100:in `run'
>        from org/jruby/Main.java:84:in `main'
>        from /home/hadoop/hbase19/bin/../bin/hirb.rb:338:in `get'
>
>
>
> I am open to any other ideas.  Should my batch update be smaller than
> 10,000? ...5,000?  I'd like to make any adjustments before i start this
> long
> upload again.  Thanks.
> --
> View this message in context:
> http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23568944.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>

Re: Loading large resultset into HBase

Posted by llpind <so...@hotmail.com>.
Hey All,

Finally finished.  But didn't complete (~ between 24,770,000 - 25,760,000
records processed).  Got the following exception before java upload app
died:

org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in .META. for region

I believe it has to do with "Problem: DFS instability and/or regionserver
lease timeouts" (http://wiki.apache.org/hadoop/Hbase/Troubleshooting).  So i
was going to try to put some JVM flags in for GC.  Also, when creating the
table, is there any column options I should use that will help in upload?

When i try to get results from within HBase shell it gives the following
exception:

org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
<TABLE_NAME>,<ROWID>,1242429308026
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2071)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1534)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)


        from org/apache/hadoop/hbase/client/HConnectionManager.java:858:in
`getRegionServerWithRetries'
        from org/apache/hadoop/hbase/client/HTable.java:608:in `getRow'
        from org/apache/hadoop/hbase/client/HTable.java:583:in `getRow'
        from org/apache/hadoop/hbase/client/HTable.java:478:in `getRow'
        from org/apache/hadoop/hbase/client/HTable.java:425:in `getRow'
        from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
        from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
        from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
        from java/lang/reflect/Method.java:597:in `invoke'
        from org/jruby/javasupport/JavaMethod.java:298:in
`invokeWithExceptionHandling'
        from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
        from org/jruby/java/invokers/InstanceMethodInvoker.java:44:in `call'
        from org/jruby/runtime/callsite/CachingCallSite.java:280:in
`cacheAndCall'
        from org/jruby/runtime/callsite/CachingCallSite.java:116:in `call'
        from org/jruby/ast/CallOneArgNode.java:57:in `interpret'
        from org/jruby/ast/LocalAsgnNode.java:120:in `interpret'
... 120 levels...
        from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in
`call'
        from org/jruby/internal/runtime/methods/CompiledMethod.java:216:in
`call'
        from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in
`call'
        from org/jruby/runtime/callsite/CachingCallSite.java:260:in
`cacheAndCall'
        from org/jruby/runtime/callsite/CachingCallSite.java:75:in `call'
        from home/hadoop/hbase19/bin/$_dot_dot_/bin/hirb.rb:441:in
`__file__'
        from home/hadoop/hbase19/bin/$_dot_dot_/bin/hirb.rb:-1:in `__file__'
        from home/hadoop/hbase19/bin/$_dot_dot_/bin/hirb.rb:-1:in `load'
        from org/jruby/Ruby.java:564:in `runScript'
        from org/jruby/Ruby.java:467:in `runNormally'
        from org/jruby/Ruby.java:340:in `runFromMain'
        from org/jruby/Main.java:214:in `run'
        from org/jruby/Main.java:100:in `run'
        from org/jruby/Main.java:84:in `main'
        from /home/hadoop/hbase19/bin/../bin/hirb.rb:338:in `get'



I am open to any other ideas.  Should my batch update be smaller than
10,000? ...5,000?  I'd like to make any adjustments before i start this long
upload again.  Thanks.
-- 
View this message in context: http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23568944.html
Sent from the HBase User mailing list archive at Nabble.com.


Re: Loading large resultset into HBase

Posted by llpind <so...@hotmail.com>.
Okay.  Thanks. 

This exception was thrown two times so far in the java process:

2009-05-15 12:42:28,080 DEBUG [main]
client.HConnectionManager$TableServers(966): Reloading region XX,,XX
location because regionserver didn't accept updates; tries=0 of max=10,
waiting=2000ms
2009-05-15 12:42:30,080 DEBUG [main]
client.HConnectionManager$TableServers(691): Removed XX for tableName=XX
from cache because of XX
2009-05-15 12:42:30,111 DEBUG [main]
client.HConnectionManager$TableServers(796): Found ROOT at
xxx.xxx.xxx.xxx:60020
2009-05-15 12:42:30,127 DEBUG [main]
client.HConnectionManager$TableServers(570): locateRegionInMeta attempt 0 of
10 failed; retrying after sleep of 2000
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in .META. for region XX,XX
	at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:548)
	at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:478)
	at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:447)
	at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:292)
	at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocationForRowWithRetries(HConnectionManager.java:902)
	at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:979)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1372)
	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1316)
	at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1296)
	at
com.apptechsys.hbase.HBaseLoaderMain.populateEntities(HBaseLoaderMain.java:200)
	at com.apptechsys.hbase.HBaseLoaderMain.main(HBaseLoaderMain.java:103)

-- 
View this message in context: http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23567478.html
Sent from the HBase User mailing list archive at Nabble.com.


Re: Loading large resultset into HBase

Posted by stack <st...@duboce.net>.
If you read the FAQ on setting DEBUG, you can do it from UI but to have it
persist across restarts, you need to edit your log4j.properties.

You'll need to restart if you want the cluster to notice the
'hbase.regionserver.hlog.blocksize' property.

St.Ack

On Fri, May 15, 2009 at 2:15 PM, llpind <so...@hotmail.com> wrote:

>
> Yes, as more data is loaded the 'used heap' fluctuates.  (note: box 1 now
> is
> now fluctuates around 200, box2 is still around 40.  3/3/2 regions).
>
> Thanks I will update 'hbase.regionserver.hlog.blocksize' property.  Will i
> need to restart the job if I turn on debugging?
>
> The job is still going (@ ~15.8 M rows)
> --
> View this message in context:
> http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23567378.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>

Re: Loading large resultset into HBase

Posted by llpind <so...@hotmail.com>.
Yes, as more data is loaded the 'used heap' fluctuates.  (note: box 1 now is
now fluctuates around 200, box2 is still around 40.  3/3/2 regions).

Thanks I will update 'hbase.regionserver.hlog.blocksize' property.  Will i
need to restart the job if I turn on debugging? 

The job is still going (@ ~15.8 M rows)
-- 
View this message in context: http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23567378.html
Sent from the HBase User mailing list archive at Nabble.com.


Re: Loading large resultset into HBase

Posted by stack <st...@duboce.net>.
On Fri, May 15, 2009 at 1:11 PM, llpind <so...@hotmail.com> wrote:

>
> Hey all,
>
> I'm loading data from a DB into HBase.  I have a single java process
> iterating over a ResultSet.  After about 10,000 rows i do a BatchUpdate.
> I've changed the Heap size of both Hadoop & HBase to 2000.
>
> Setup:  0.19.1.  1 box with master and secondary.  3 boxes with
> HRegionServer.
>
> Problem 1:  The load seems to be unblananced:
>  Address Start Code Load
> 1:60020 1242415770566 requests=0, regions=1, usedHeap=39, maxHeap=1777
> 2:60020 1242415770417 requests=3, regions=3, usedHeap=52, maxHeap=1777
> 3:60020 1242415770273 requests=1, regions=3, usedHeap=604, maxHeap=1777
> Total:  servers: 3   requests=4, regions=7
>



Yeah, it looks unbalanced memory-wise but not too off regions wise.  What
happens if you load more into your cluster?  And used heap fluctuates,
doesn't it?



>
> Problem 2:  Around 10 Million rows, the upload starts to slow down.
>


Turn on DEBUG logging -- see FAQ on wiki for how -- and then tail one of
your regionserver logs.  Whats going on?

It might be the fact that the commit log buffer is small in hbase 0.19.1.  I
found this issue recently in
https://issues.apache.org/jira/browse/HBASE-1394.   Change
hbase.regionserver.hlog.blocksize from its 1MB size up to say, 64MB?



>
> The upload is still going, so i'll update on what happens.


Please do.

St.Ack



>
> --
> View this message in context:
> http://www.nabble.com/Loading-large-resultset-into-HBase-tp23566568p23566568.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>