You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Peter Falk <pe...@bugsoft.nu> on 2010/03/24 11:51:48 UTC
Problems with region server OOME
Hi,
We have a cluster of four nodes that run hadoop 0.20.1 data nodes and hbase
0.20.2 region servers. We occasionally loose region servers with an OOME
like the following.
2010-03-24 04:22:03,027 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: OutOfMemoryError,
aborting.
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.hbase.client.Result.write(Result.java:496)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:333)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.write(HbaseObjectWritable.java:213)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:937)
We would appreciate tips/information of how to change the configuration so
that OOME probability is minimized. I have attached the current hbase-site
config below. The nodes configuration is 8 GB RAM, 4 2.66 GHz Intel Xeon
processors, and two 1 TB discs. The region servers have 2 GB heap, and other
server have 1 GB heap.
Relevant hbase-site.xml config:
<property>
<name>hbase.client.scanner.caching</name>
<value>20</value>
</property>
<property>
<name>dfs.datanode.socket.write.timeout</name>
<value>0</value>
</property>
<property>
<name>hbase.regionserver.handler.count</name>
<value>10</value>
</property>
<property>
<name>hbase.hregion.memstore.flush.size</name>
<value>33554432</value>
<description>
Memstore will be flushed to disk if size of the memstore
exceeds this number of bytes. Value is checked by a thread that runs
every hbase.server.thread.wakefrequency.
</description>
</property>
<property>
<name>hbase.server.thread.wakefrequency</name>
<value>5000</value>
<description>Time to sleep in between searches for work (in
milliseconds).
Used as sleep interval by service threads such as META scanner and log
roller.
</description>
</property>
<property>
<name>hbase.regionserver.global.memstore.upperLimit</name>
<value>0.35</value>
<description>Maximum size of all memstores in a region server before new
updates are blocked and flushes are forced. Defaults to 40% of heap
</description>
</property>
<property>
<name>hbase.hstore.blockingStoreFiles</name>
<value>5</value>
<description>
If more than this number of StoreFiles in any one Store
(one StoreFile is written per flush of MemStore) then updates are
blocked for this HRegion until a compaction is completed, or
until hbase.hstore.blockingWaitTime has been exceeded.
</description>
</property>
<property>
<name>hbase.hstore.blockingWaitTime</name>
<value>360000</value>
<description>
The time an HRegion will block updates for after hitting the StoreFile
limit defined by hbase.hstore.blockingStoreFiles.
After this time has elapsed, the HRegion will stop blocking updates even
if a compaction has not been completed. Default: 90 seconds.
</description>
</property>
<property>
<name>hfile.block.cache.size</name>
<value>0.15</value>
<description>
Percentage of maximum heap (-Xmx setting) to allocate to block
cache
used by HFile/StoreFile. Default of 0.2 means allocate 20%.
Set to 0 to disable.
</description>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>240000</value>
<description>ZooKeeper session timeout. This option is not used by HBase
directly, it is for the internals of ZooKeeper. HBase merely passes it
in
whenever a connection is established to ZooKeeper. It is used by
ZooKeeper
for hearbeats. In milliseconds.
</description>
</property>
TIA,
Peter
Re: Problems with region server OOME
Posted by Ted Yu <yu...@gmail.com>.
Minor correction for the VM option syntax:
-XX:+UseCompressedOops
On Wed, Mar 24, 2010 at 9:24 AM, Andrew Purtell <ap...@apache.org> wrote:
> > We would appreciate tips/information of how to change the
> > configuration so that OOME probability is minimized.
>
> Try running with 4GB heaps if you can.
>
> On recent JVMs -- but don't use 1.6.0_18! -- you can have the JVM compress
> 64 bit object references into 32 bits. This will save heap at minor
> performance cost. Add '-XX:UseCompressedOops' to HBASE_OPTS in hbase-env.sh.
> For more information on compressed OOPS:
> http://wikis.sun.com/display/HotSpotInternals/CompressedOops
>
> Also see this page on the HBase wiki for some related information:
> http://wiki.apache.org/hadoop/PerformanceTuning, the section "HBase JVM
> and GC"
>
> Hope this helps,
>
> - Andy
>
>
>
>
>
>
Re: Problems with region server OOME
Posted by Edward Capriolo <ed...@gmail.com>.
On Thu, Mar 25, 2010 at 8:44 AM, Peter Falk <pe...@bugsoft.nu> wrote:
> Thanks Andrew, Edward, Jonathan for the help. I will try to increase RAM
> and
> try the compressed oops suggestion.
>
> As I said, we are running data nodes and region servers on all nodes. Also,
> we are running the name node and master on one of these four, secondary
> name
> node on two of them, and zoo keepers on the two other. We tried to save RAM
> for the file cache in the system, to improve data node performance. There
> are currently about 3 - 4 GB os file cache on the nodes, except on the one
> running master and name node where it is less. Any thoughts about the size
> of the file cache for such nodes?
>
> Regarding the type of rows/columns, the biggest table and the one that also
> is most used, has few columns but large cells (about 5 kB in average). This
> table is also LZO compressed.
>
> BTW, is there any way to debug or log each find/mutate operation that is
> performed by the cluster? Even with DEBUG log level on master and region
> server, it does not seem to exist any such log message.
>
> Thanks again,
> Peter
>
> On Wed, Mar 24, 2010 at 17:24, Andrew Purtell <ap...@apache.org> wrote:
>
> > > We would appreciate tips/information of how to change the
> > > configuration so that OOME probability is minimized.
> >
> > Try running with 4GB heaps if you can.
> >
> > On recent JVMs -- but don't use 1.6.0_18! -- you can have the JVM
> compress
> > 64 bit object references into 32 bits. This will save heap at minor
> > performance cost. Add '-XX:UseCompressedOops' to HBASE_OPTS in
> hbase-env.sh.
> > For more information on compressed OOPS:
> > http://wikis.sun.com/display/HotSpotInternals/CompressedOops
> >
> > Also see this page on the HBase wiki for some related information:
> > http://wiki.apache.org/hadoop/PerformanceTuning, the section "HBase JVM
> > and GC"
> >
> > Hope this helps,
> >
> > - Andy
> >
> >
> >
> >
> >
> >
>
>>There
are currently about 3 - 4 GB os file cache on the nodes, except on the one
running master and name node where it is less. Any thoughts about the size
of the file cache for such nodes?
HBase has its own block cache that can be configured to be a percentage of
hbase memory. So generally it is better to give hbase the memory directly
since the two are essentially doing the same thing.
Re: Problems with region server OOME
Posted by Peter Falk <pe...@bugsoft.nu>.
Thanks Andrew, Edward, Jonathan for the help. I will try to increase RAM and
try the compressed oops suggestion.
As I said, we are running data nodes and region servers on all nodes. Also,
we are running the name node and master on one of these four, secondary name
node on two of them, and zoo keepers on the two other. We tried to save RAM
for the file cache in the system, to improve data node performance. There
are currently about 3 - 4 GB os file cache on the nodes, except on the one
running master and name node where it is less. Any thoughts about the size
of the file cache for such nodes?
Regarding the type of rows/columns, the biggest table and the one that also
is most used, has few columns but large cells (about 5 kB in average). This
table is also LZO compressed.
BTW, is there any way to debug or log each find/mutate operation that is
performed by the cluster? Even with DEBUG log level on master and region
server, it does not seem to exist any such log message.
Thanks again,
Peter
On Wed, Mar 24, 2010 at 17:24, Andrew Purtell <ap...@apache.org> wrote:
> > We would appreciate tips/information of how to change the
> > configuration so that OOME probability is minimized.
>
> Try running with 4GB heaps if you can.
>
> On recent JVMs -- but don't use 1.6.0_18! -- you can have the JVM compress
> 64 bit object references into 32 bits. This will save heap at minor
> performance cost. Add '-XX:UseCompressedOops' to HBASE_OPTS in hbase-env.sh.
> For more information on compressed OOPS:
> http://wikis.sun.com/display/HotSpotInternals/CompressedOops
>
> Also see this page on the HBase wiki for some related information:
> http://wiki.apache.org/hadoop/PerformanceTuning, the section "HBase JVM
> and GC"
>
> Hope this helps,
>
> - Andy
>
>
>
>
>
>
RE: Problems with region server OOME
Posted by Andrew Purtell <ap...@apache.org>.
> We would appreciate tips/information of how to change the
> configuration so that OOME probability is minimized.
Try running with 4GB heaps if you can.
On recent JVMs -- but don't use 1.6.0_18! -- you can have the JVM compress 64 bit object references into 32 bits. This will save heap at minor performance cost. Add '-XX:UseCompressedOops' to HBASE_OPTS in hbase-env.sh. For more information on compressed OOPS: http://wikis.sun.com/display/HotSpotInternals/CompressedOops
Also see this page on the HBase wiki for some related information: http://wiki.apache.org/hadoop/PerformanceTuning, the section "HBase JVM and GC"
Hope this helps,
- Andy
RE: Problems with region server OOME
Posted by Jonathan Gray <jg...@facebook.com>.
As Edward said, try increasing HBase RegionServer heap to 4GB. Look around the wiki for GC tuning information.
What does your data look like and what is your read/write pattern? Do you have large rows or columns?
> -----Original Message-----
> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
> Sent: Wednesday, March 24, 2010 6:57 AM
> To: hbase-user@hadoop.apache.org; peter@bugsoft.nu
> Subject: Re: Problems with region server OOME
>
> On Wed, Mar 24, 2010 at 6:51 AM, Peter Falk <pe...@bugsoft.nu> wrote:
>
> > Hi,
> >
> > We have a cluster of four nodes that run hadoop 0.20.1 data nodes and
> hbase
> > 0.20.2 region servers. We occasionally loose region servers with an
> OOME
> > like the following.
> >
> > 2010-03-24 04:22:03,027 FATAL
> > org.apache.hadoop.hbase.regionserver.HRegionServer: OutOfMemoryError,
> > aborting.
> > java.lang.OutOfMemoryError: Java heap space
> > at java.util.Arrays.copyOf(Arrays.java:2786)
> > at
> > java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> > at java.io.DataOutputStream.write(DataOutputStream.java:90)
> > at
> org.apache.hadoop.hbase.client.Result.write(Result.java:496)
> > at
> >
> >
> org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectW
> ritable.java:333)
> > at
> >
> >
> org.apache.hadoop.hbase.io.HbaseObjectWritable.write(HbaseObjectWritabl
> e.java:213)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:93
> 7)
> >
> > We would appreciate tips/information of how to change the
> configuration so
> > that OOME probability is minimized. I have attached the current
> hbase-site
> > config below. The nodes configuration is 8 GB RAM, 4 2.66 GHz Intel
> Xeon
> > processors, and two 1 TB discs. The region servers have 2 GB heap,
> and
> > other
> > server have 1 GB heap.
> >
> > Relevant hbase-site.xml config:
> >
> > <property>
> > <name>hbase.client.scanner.caching</name>
> > <value>20</value>
> > </property>
> >
> > <property>
> > <name>dfs.datanode.socket.write.timeout</name>
> > <value>0</value>
> > </property>
> >
> > <property>
> > <name>hbase.regionserver.handler.count</name>
> > <value>10</value>
> > </property>
> >
> > <property>
> > <name>hbase.hregion.memstore.flush.size</name>
> > <value>33554432</value>
> > <description>
> > Memstore will be flushed to disk if size of the memstore
> > exceeds this number of bytes. Value is checked by a thread that
> runs
> > every hbase.server.thread.wakefrequency.
> > </description>
> > </property>
> >
> > <property>
> > <name>hbase.server.thread.wakefrequency</name>
> > <value>5000</value>
> > <description>Time to sleep in between searches for work (in
> > milliseconds).
> > Used as sleep interval by service threads such as META scanner and
> log
> > roller.
> > </description>
> > </property>
> >
> > <property>
> > <name>hbase.regionserver.global.memstore.upperLimit</name>
> > <value>0.35</value>
> > <description>Maximum size of all memstores in a region server
> before new
> > updates are blocked and flushes are forced. Defaults to 40% of
> heap
> > </description>
> > </property>
> >
> > <property>
> > <name>hbase.hstore.blockingStoreFiles</name>
> > <value>5</value>
> > <description>
> > If more than this number of StoreFiles in any one Store
> > (one StoreFile is written per flush of MemStore) then updates are
> > blocked for this HRegion until a compaction is completed, or
> > until hbase.hstore.blockingWaitTime has been exceeded.
> > </description>
> > </property>
> >
> > <property>
> > <name>hbase.hstore.blockingWaitTime</name>
> > <value>360000</value>
> > <description>
> > The time an HRegion will block updates for after hitting the
> StoreFile
> > limit defined by hbase.hstore.blockingStoreFiles.
> > After this time has elapsed, the HRegion will stop blocking
> updates even
> > if a compaction has not been completed. Default: 90 seconds.
> > </description>
> > </property>
> >
> > <property>
> > <name>hfile.block.cache.size</name>
> > <value>0.15</value>
> > <description>
> > Percentage of maximum heap (-Xmx setting) to allocate to
> block
> > cache
> > used by HFile/StoreFile. Default of 0.2 means allocate 20%.
> > Set to 0 to disable.
> > </description>
> > </property>
> >
> > <property>
> > <name>zookeeper.session.timeout</name>
> > <value>240000</value>
> > <description>ZooKeeper session timeout. This option is not used by
> HBase
> > directly, it is for the internals of ZooKeeper. HBase merely
> passes it
> > in
> > whenever a connection is established to ZooKeeper. It is used by
> > ZooKeeper
> > for hearbeats. In milliseconds.
> > </description>
> > </property>
> >
> >
> > TIA,
> > Peter
> >
>
> If you have 8Gb of RAM you should give as much of it as you can to
> hbase.
Re: Problems with region server OOME
Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Mar 24, 2010 at 6:51 AM, Peter Falk <pe...@bugsoft.nu> wrote:
> Hi,
>
> We have a cluster of four nodes that run hadoop 0.20.1 data nodes and hbase
> 0.20.2 region servers. We occasionally loose region servers with an OOME
> like the following.
>
> 2010-03-24 04:22:03,027 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: OutOfMemoryError,
> aborting.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.hbase.client.Result.write(Result.java:496)
> at
>
> org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:333)
> at
>
> org.apache.hadoop.hbase.io.HbaseObjectWritable.write(HbaseObjectWritable.java:213)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:937)
>
> We would appreciate tips/information of how to change the configuration so
> that OOME probability is minimized. I have attached the current hbase-site
> config below. The nodes configuration is 8 GB RAM, 4 2.66 GHz Intel Xeon
> processors, and two 1 TB discs. The region servers have 2 GB heap, and
> other
> server have 1 GB heap.
>
> Relevant hbase-site.xml config:
>
> <property>
> <name>hbase.client.scanner.caching</name>
> <value>20</value>
> </property>
>
> <property>
> <name>dfs.datanode.socket.write.timeout</name>
> <value>0</value>
> </property>
>
> <property>
> <name>hbase.regionserver.handler.count</name>
> <value>10</value>
> </property>
>
> <property>
> <name>hbase.hregion.memstore.flush.size</name>
> <value>33554432</value>
> <description>
> Memstore will be flushed to disk if size of the memstore
> exceeds this number of bytes. Value is checked by a thread that runs
> every hbase.server.thread.wakefrequency.
> </description>
> </property>
>
> <property>
> <name>hbase.server.thread.wakefrequency</name>
> <value>5000</value>
> <description>Time to sleep in between searches for work (in
> milliseconds).
> Used as sleep interval by service threads such as META scanner and log
> roller.
> </description>
> </property>
>
> <property>
> <name>hbase.regionserver.global.memstore.upperLimit</name>
> <value>0.35</value>
> <description>Maximum size of all memstores in a region server before new
> updates are blocked and flushes are forced. Defaults to 40% of heap
> </description>
> </property>
>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>5</value>
> <description>
> If more than this number of StoreFiles in any one Store
> (one StoreFile is written per flush of MemStore) then updates are
> blocked for this HRegion until a compaction is completed, or
> until hbase.hstore.blockingWaitTime has been exceeded.
> </description>
> </property>
>
> <property>
> <name>hbase.hstore.blockingWaitTime</name>
> <value>360000</value>
> <description>
> The time an HRegion will block updates for after hitting the StoreFile
> limit defined by hbase.hstore.blockingStoreFiles.
> After this time has elapsed, the HRegion will stop blocking updates even
> if a compaction has not been completed. Default: 90 seconds.
> </description>
> </property>
>
> <property>
> <name>hfile.block.cache.size</name>
> <value>0.15</value>
> <description>
> Percentage of maximum heap (-Xmx setting) to allocate to block
> cache
> used by HFile/StoreFile. Default of 0.2 means allocate 20%.
> Set to 0 to disable.
> </description>
> </property>
>
> <property>
> <name>zookeeper.session.timeout</name>
> <value>240000</value>
> <description>ZooKeeper session timeout. This option is not used by HBase
> directly, it is for the internals of ZooKeeper. HBase merely passes it
> in
> whenever a connection is established to ZooKeeper. It is used by
> ZooKeeper
> for hearbeats. In milliseconds.
> </description>
> </property>
>
>
> TIA,
> Peter
>
If you have 8Gb of RAM you should give as much of it as you can to hbase.