You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Bing Li <lb...@gmail.com> on 2012/03/10 04:20:48 UTC
Memory Requirements
Dear all,
Today I ran HBase in the pseudo-distributed mode. Since I need to save the
meta information about Web pages crawled from more than 10,000 Web sites
simultaneously and periodically. I believe the data size is large. After
running for about 30 minutes, it got the following exceptions.
[java] ...............2012-3-10 10:09:04
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
processBatchCallback
[java] Warning: Failed all from
region=PlayTable,,1331343367325.2bf6bf580051da504c347c19eb08f9d5.,
hostname=greatfreeinpain, port=60020
[java] java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
[java] at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
[java] at java.util.concurrent.FutureTask.get(FutureTask.java:83)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1557)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
[java] at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:900)
[java] at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:777)
[java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:760)
[java] at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
[java] at
com.greatfree.hbase.AuthorityPersister.InsertInitialPlayRecord(AuthorityPersister.java:128)
[java] at
com.greatfree.hbase.RandomPersistDispatcher.run(RandomPersistDispatcher.java:92)
[java] at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[java] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[java] at java.lang.Thread.run(Thread.java:662)
[java] Caused by: java.lang.RuntimeException:
java.lang.OutOfMemoryError: Java heap space
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1371)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1383)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1381)
[java] at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[java] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[java] ... 3 more
[java] Caused by: java.lang.OutOfMemoryError: Java heap space
[java] at java.util.Arrays.copyOf(Arrays.java:2786)
[java] at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
[java] at java.io.DataOutputStream.write(DataOutputStream.java:90)
[java] at org.apache.hadoop.hbase.client.Put.write(Put.java:422)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:464)
[java] at org.apache.hadoop.hbase.client.Action.write(Action.java:94)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:464)
[java] at
org.apache.hadoop.hbase.client.MultiAction.write(MultiAction.java:102)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:464)
[java] at
org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:138)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:537)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:879)
[java] at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
[java] at $Proxy6.multi(Unknown Source)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1386)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1384)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1365)
[java] ... 7 more
The machine running the HBase has 1G memory only over Ubuntu 10. According
to the book, HBase the Definitive Guide, 64-bit OS and 4GB are required. I
need to upgrade my system?
I change the parameter, HBASE_HEAPSIZE, from 1000 to 2048. The exceptions
still occurred. When the value was changed to 3072, it got the following
errors when I ran start-hbase.sh.
Could not create the Java virtual machine.
Could not create the Java virtual machine.
starting master, logging to
/opt/hbase-0.92.0/bin/../logs/hbase-libing-master-greatfreeinpain.out
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
localhost: starting regionserver, logging to
/opt/hbase-0.92.0/bin/../logs/hbase-libing-regionserver-greatfreeinpain.out
localhost: Error occurred during initialization of VM
localhost: Could not reserve enough space for object heap
localhost: Could not create the Java virtual machine.
It seems that I need to increase the memory size? Do I need to install a
64-bit Ubuntu Linux?
Thanks so much for your help!
Best regards,
Bing
Re: Memory Requirements
Posted by Doug Meil <do...@explorysmedical.com>.
Hi there-
Here are the recommendations from the HBase RefGuide:
http://hbase.apache.org/book.html#perf.os
... and they are consistently with what the book says (recommends 64-bit
OS and more memory).
Also, keep this in mind...
http://hbase.apache.org/book.html#arch.overview
... the standalone deployment should be considered a development config
only.
On 3/9/12 10:20 PM, "Bing Li" <lb...@gmail.com> wrote:
>Dear all,
>
>Today I ran HBase in the pseudo-distributed mode. Since I need to save the
>meta information about Web pages crawled from more than 10,000 Web sites
>simultaneously and periodically. I believe the data size is large. After
>running for about 30 minutes, it got the following exceptions.
>
> [java] ...............2012-3-10 10:09:04
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n
>processBatchCallback
> [java] Warning: Failed all from
>region=PlayTable,,1331343367325.2bf6bf580051da504c347c19eb08f9d5.,
>hostname=greatfreeinpain, port=60020
> [java] java.util.concurrent.ExecutionException:
>java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
> [java] at
>java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> [java] at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n.processBatchCallback(HConnectionManager.java:1557)
> [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n.processBatch(HConnectionManager.java:1409)
> [java] at
>org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:900)
> [java] at
>org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:777)
> [java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:760)
> [java] at
>org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java
>:402)
> [java] at
>com.greatfree.hbase.AuthorityPersister.InsertInitialPlayRecord(AuthorityPe
>rsister.java:128)
> [java] at
>com.greatfree.hbase.RandomPersistDispatcher.run(RandomPersistDispatcher.ja
>va:92)
> [java] at
>java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.
>java:886)
> [java] at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:908)
> [java] at java.lang.Thread.run(Thread.java:662)
> [java] Caused by: java.lang.RuntimeException:
>java.lang.OutOfMemoryError: Java heap space
> [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n.getRegionServerWithoutRetries(HConnectionManager.java:1371)
> [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n$3.call(HConnectionManager.java:1383)
> [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n$3.call(HConnectionManager.java:1381)
> [java] at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [java] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> [java] ... 3 more
> [java] Caused by: java.lang.OutOfMemoryError: Java heap space
> [java] at java.util.Arrays.copyOf(Arrays.java:2786)
> [java] at
>java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> [java] at java.io.DataOutputStream.write(DataOutputStream.java:90)
> [java] at org.apache.hadoop.hbase.client.Put.write(Put.java:422)
> [java] at
>org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWrit
>able.java:464)
> [java] at org.apache.hadoop.hbase.client.Action.write(Action.java:94)
> [java] at
>org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWrit
>able.java:464)
> [java] at
>org.apache.hadoop.hbase.client.MultiAction.write(MultiAction.java:102)
> [java] at
>org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWrit
>able.java:464)
> [java] at
>org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:138)
> [java] at
>org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.j
>ava:537)
> [java] at
>org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:879)
> [java] at
>org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEn
>gine.java:150)
> [java] at $Proxy6.multi(Unknown Source)
> [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n$3$1.call(HConnectionManager.java:1386)
> [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n$3$1.call(HConnectionManager.java:1384)
> [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n.getRegionServerWithoutRetries(HConnectionManager.java:1365)
> [java] ... 7 more
>
>The machine running the HBase has 1G memory only over Ubuntu 10. According
>to the book, HBase the Definitive Guide, 64-bit OS and 4GB are required. I
>need to upgrade my system?
>
>I change the parameter, HBASE_HEAPSIZE, from 1000 to 2048. The exceptions
>still occurred. When the value was changed to 3072, it got the following
>errors when I ran start-hbase.sh.
>
> Could not create the Java virtual machine.
> Could not create the Java virtual machine.
> starting master, logging to
>/opt/hbase-0.92.0/bin/../logs/hbase-libing-master-greatfreeinpain.out
> Error occurred during initialization of VM
> Could not reserve enough space for object heap
> Could not create the Java virtual machine.
> localhost: starting regionserver, logging to
>/opt/hbase-0.92.0/bin/../logs/hbase-libing-regionserver-greatfreeinpain.ou
>t
> localhost: Error occurred during initialization of VM
> localhost: Could not reserve enough space for object heap
> localhost: Could not create the Java virtual machine.
>
>It seems that I need to increase the memory size? Do I need to install a
>64-bit Ubuntu Linux?
>
>Thanks so much for your help!
>
>Best regards,
>Bing