High CPU usage on region server

[Xuan Li <xu...@gmail.com>]

Hey,
I am seeing a wired issue on my hbase cluster: randomly, one or two region
servers in the cluster will have very high CPU utilization(close to 100%), even
these servers aren’t doing that much qps. Restart the region server will solve
it, but after a while the hot cpu usage problem will come back again on other
region servers.
I am running hbase 0.94.7, with a wide schema design(like couple hundreds
column). The cluster serves both read/write requests, with about 5K read/500K
write.
Here is the jstack dump on high cpu usage region server: Preview image
[https://res.cloudinary.com/mixmax/image/fetch/w_800,h_600,c_fill,q_90,fl_progressive,g_faces:center/http%3A%2F%2Fpastebin.com%2Fi%2Ffb2.jpg]
[http://pastebin.com/UQr2FPze] RegionServer jstack [http://pastebin.com/UQr2FPze] ** Pastebin PRO Accounts Spring Special ** Get 40% discount for a limited time
only! [http://pastebin.com/UQr2FPze] pastebin.com [http://pastebin.com/UQr2FPze] Mixmax [https://emailapps.mixmax.com/img/badge_mixmax.png]
[https://mixmax.com/r/3vvevSdBLFaEGWEf6]
Seems the cpu is busy with:
org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1535)
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:214)

Any idea why it would be like this? Thanks!

Xuan[https://app.mixmax.com/api/track/v2/R0EY3UBi7HS5emeW5/xuan.dreamer%40gmail.com/IyZy9mLlh2YhBXYuU2chJGaAJXZzVnI/?sc=false]