You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Chang Chen <ba...@gmail.com> on 2015/08/03 14:18:02 UTC

Compaction queue size constantly increase.

Hi All

We use HBase 0.96.1.1, and find a very strange issue which looks
like CompactSplitThread is waiting for something!  The following logs are
from problematic RegionServer:


2015-07-29 10:15:39,702 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
compaction: Request =
regionName=realtime_rec,901943112,1426251108696.3a3c8d1c888f98518ce42f94b4033a39.,
storeName=usr, fileCount=3, fileSize=2.4 M, priority=12,
time=21663559757818028; duration=0sec

2015-07-29 19:37:38,566 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Aborted
compaction: Request =
regionName=full_user_profile,4U5BN9EUHMZXZA58NX811Q64XRBQMNZF2KH2,1406419093595.e850bec0a113f4fb6c7ed5a136bc630e.,
storeName=cat, fileCount=3, fileSize=55.2 M (31.6 M, 11.5 M, 12.1 M),
priority=12, time=21697278778022407; duration=0sec


Between 10:15 and 19:37, I didn't find any logs generated by
CompactSplitThread, and we restarted HBase cluster at time of 19:37. The
waiting of compaction thread made compaction queue size constantly
increase, and eventually slow down HTable.get

Our major and minor compaction thread are set to 1, and I guess
CompactSplitThread may be blocked in selectCompaction, since the aborted
compaction only spent 0 seconds!

Is there any clue which can help me to find more information?

Thanks
Chang

Re: Compaction queue size constantly increase.

Posted by Ted Yu <yu...@gmail.com>.
Can you take jstack of the region server next time this happens ?

Btw please consider upgrading. 
0.96.x is too old. 



> On Aug 3, 2015, at 5:18 AM, Chang Chen <ba...@gmail.com> wrote:
> 
> Hi All
> 
> We use HBase 0.96.1.1, and find a very strange issue which looks
> like CompactSplitThread is waiting for something!  The following logs are
> from problematic RegionServer:
> 
> 
> 2015-07-29 10:15:39,702 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
> compaction: Request =
> regionName=realtime_rec,901943112,1426251108696.3a3c8d1c888f98518ce42f94b4033a39.,
> storeName=usr, fileCount=3, fileSize=2.4 M, priority=12,
> time=21663559757818028; duration=0sec
> 
> 2015-07-29 19:37:38,566 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Aborted
> compaction: Request =
> regionName=full_user_profile,4U5BN9EUHMZXZA58NX811Q64XRBQMNZF2KH2,1406419093595.e850bec0a113f4fb6c7ed5a136bc630e.,
> storeName=cat, fileCount=3, fileSize=55.2 M (31.6 M, 11.5 M, 12.1 M),
> priority=12, time=21697278778022407; duration=0sec
> 
> 
> Between 10:15 and 19:37, I didn't find any logs generated by
> CompactSplitThread, and we restarted HBase cluster at time of 19:37. The
> waiting of compaction thread made compaction queue size constantly
> increase, and eventually slow down HTable.get
> 
> Our major and minor compaction thread are set to 1, and I guess
> CompactSplitThread may be blocked in selectCompaction, since the aborted
> compaction only spent 0 seconds!
> 
> Is there any clue which can help me to find more information?
> 
> Thanks
> Chang