[jira] [Commented] (HBASE-20673) Reduce the number of Cell implementations; the profusion is distracting to users and JIT

["zhuobin zheng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-20673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004384#comment-17004384 ] 

zhuobin zheng commented on HBASE-20673:
---------------------------------------

Hi, [~stack] 
 I also encountered similar problems in my production environment: *Too many KeyValue implementation types confused JIT* .

And cresult in a large amount of CPU waste, resulting in full cpu usage of singal server(48 core, cpu 4800%), and a significant decrease in read and write speeds.

Through Profiler analysis, it is found that itable stub takes a lot of CPU time (almost 70%)
Through Jstack: A large number of read and write threads are stuck in several special places where CellComparator compares: 

[https://github.com/apache/hbase/blob/branch-1.2/hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java#L186]

[https://github.com/apache/hbase/blob/branch-1.2/hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java#L157]

The Jstack really confused me. Because this line of code does nothing but resolve the actual calling code address.

But combining the above profiler result (70% cpu in itable stub).

Now, I think there are two possible reasons for the crazy use of cpu.
 # Too many Cell implements confuesd JIT. Result jvm interface call to original itable scan.
 # KeyValue implements too many interface, cause itable too long.

 
{code:java}
//代码占位符
public class KeyValue implements Cell, HeapSize, Cloneable, SettableSequenceId, SettableTimestamp{code}
 

I think this situation will occur when the two types of KeyValue / NoTagKeyValue are more evenly distributed. 

But unfortunately, although it has always appeared in the production environment, I cannot reproduce it in the test environment, so I cannot provide a better test solution.

Now I will try to modify a part of the code so that a large number of Cell implementations are of one type, and put it into the production environment to see if it can solve the 100% CPU problem.

 

 

> Reduce the number of Cell implementations; the profusion is distracting to users and JIT
> ----------------------------------------------------------------------------------------
>
>                 Key: HBASE-20673
>                 URL: https://issues.apache.org/jira/browse/HBASE-20673
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Performance
>            Reporter: Michael Stack
>            Assignee: Michael Stack
>            Priority: Major
>         Attachments: 0001-current.patch, 0001-current.patch, hits.20673.png
>
>
> We have a wild blossom of Cell implementations in hbase. Purge the bulk of them. Make it so we do one type well. JIT gets confused if it has an abstract or an interface and then the instantiated classes are myriad (megamorphic).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)