[jira] [Commented] (HBASE-22134) JIT deoptimization in Cell.write

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-22134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16805632#comment-16805632 ] 

Todd Lipcon commented on HBASE-22134:
-------------------------------------

For future reference, the stap script I used was:

{code}
probe process("/usr/lib64/libc-2.17.so").function("_IO_vsnprintf").return {
        printf("%s\n", user_string_n_quoted(@entry($string), $return));
}
{code}

(because I noticed that there was a call to vsnprintf in the deoptimization path, I guessed it was log-message-related, and bingo!). Likely you could get the same info out of LogCompilation but that flag can't be switched at runtime.

> JIT deoptimization in Cell.write
> --------------------------------
>
>                 Key: HBASE-22134
>                 URL: https://issues.apache.org/jira/browse/HBASE-22134
>             Project: HBase
>          Issue Type: Bug
>          Components: Performance
>    Affects Versions: 2.0.2
>            Reporter: Todd Lipcon
>            Priority: Major
>
> I was looking at a profile of a workload which was running compaction very slowly, and saw that the top CPU consumers were from JVM internals regarding deoptimization. I managed to write a little systemtap script to extract the deoptimization log and got the following in a tight loop:
> "Uncommon trap: trap_request=0xffffff67 fr.pc=0x00007f85bcdb8644"
> "Uncommon trap: reason=unstable_if action=none pc=0x00007f85bcdb8644 method=org.apache.hadoop.hbase.io.encoding.NoneEncoder.write(Lorg/apache/hadoop/hbase/Cell;)I @ 67"
> "DEOPT PACKING pc=0x00007f85bcdb8644 sp=0x00007f84d3d83080"
> "DEOPT UNPACKING pc=0x00007f85b5005229 sp=0x00007f84d3d82f30 mode 2"
> The java stack is spending most of its time at:
>   java.lang.Thread.State: RUNNABLE
> 	at org.apache.hadoop.hbase.io.encoding.NoneEncoder.write(NoneEncoder.java:57)
> 	at org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.encode(NoOpDataBlockEncoder.java:55)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.write(HFileBlock.java:983)
> 	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.append(HFileWriterImpl.java:740)
> This was with Oracle JDK 1.8.0_112. Likely a JDK bug but perhaps some reorganization of this code path could help avoid triggering the bug.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)