[jira] [Created] (HBASE-26024) Region server JVM Crash - A fatal error has been detected by the Java Runtime Environment

["Mohamed Mohideen Meeran (Jira)" <ji...@apache.org>]

Mohamed Mohideen Meeran created HBASE-26024:
-----------------------------------------------

             Summary: Region server JVM Crash - A fatal error has been detected by the Java Runtime Environment
                 Key: HBASE-26024
                 URL: https://issues.apache.org/jira/browse/HBASE-26024
             Project: HBase
          Issue Type: Bug
    Affects Versions: 2.1.9
            Reporter: Mohamed Mohideen Meeran
         Attachments: error.log

Our production Region servers JVM crashed with the following error logs.

 

Register to memory mapping:

RAX=0x000000002eea8f42 is an unknown value

RBX=0x00007f1f8c7900d6 is an unknown value

RCX=0x0000000000000021 is an unknown value

RDX=0x0000000000000000 is an unknown value

RSP=0x00007f1fe3092200 is pointing into the stack for thread: 0x00007f29775fb000

RBP=0x00007f1fe3092200 is pointing into the stack for thread: 0x00007f29775fb000

RSI=0x00007f1f8c7900cc is an unknown value

RDI=0x000000002eea8f38 is an unknown value

R8 =0x00007f28e14a3a38 is an oop

java.nio.DirectByteBuffer

 - klass: 'java/nio/DirectByteBuffer'

R9 =0x00007f1f8c790094 is an unknown value

R10=0x00007f2965053400 is at begin+0 in a stub

StubRoutines::unsafe_arraycopy [0x00007f2965053400, 0x00007f296505343b[ (59 bytes)

R11=0x00007f28e14a3a38 is an oop

java.nio.DirectByteBuffer

 - klass: 'java/nio/DirectByteBuffer'

R12=

[error occurred during error reporting (printing register info), id 0xb]

 

Stack: [0x00007f1fe2f93000,0x00007f1fe3094000],  sp=0x00007f1fe3092200,  free space=1020k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

v  ~StubRoutines::jshort_disjoint_arraycopy

J 18388 C2 org.apache.hadoop.hbase.io.ByteBufferListOutputStream.write(Ljava/nio/ByteBuffer;II)V (53 bytes) @ 0x00007f2967fa0ea2 [0x00007f2967fa0d40+0x162]

J 11722 C2 org.apache.hadoop.hbase.util.ByteBufferUtils.copyBufferToStream(Ljava/io/OutputStream;Ljava/nio/ByteBuffer;II)V (75 bytes) @ 0x00007f29670aa0fc [0x00007f29670a9fa0+0x15c]

J 13251 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I (21 bytes) @ 0x00007f2965cbe87c [0x00007f2965cbe820+0x5c]

J 8703 C2 org.apache.hadoop.hbase.KeyValueUtil.oswrite(Lorg/apache/hadoop/hbase/Cell;Ljava/io/OutputStream;Z)I (259 bytes) @ 0x00007f296684a2d4 [0x00007f296684a140+0x194]

J 15474 C2 org.apache.hadoop.hbase.ipc.CellBlockBuilder.buildCellBlockStream(Lorg/apache/hadoop/hbase/codec/Codec;Lorg/apache/hadoop/io/compress/CompressionCodec;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/io/ByteBufferPool;)Lorg/apache/hadoop/hbase/io/ByteBufferListOutputStream; (75 bytes) @ 0x00007f29675f9dc8 [0x00007f29675f7c80+0x2148]

J 14260 C2 org.apache.hadoop.hbase.ipc.ServerCall.setResponse(Lorg/apache/hbase/thirdparty/com/google/protobuf/Message;Lorg/apache/hadoop/hbase/CellScanner;Ljava/lang/Throwable;Ljava/lang/String;)V (408 bytes) @ 0x00007f29678ad11c [0x00007f29678acec0+0x25c]

J 14732 C2 org.apache.hadoop.hbase.ipc.CallRunner.run()V (1376 bytes) @ 0x00007f296797f690 [0x00007f296797e6a0+0xff0]

J 14293 C2 org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(Lorg/apache/hadoop/hbase/ipc/CallRunner;)V (268 bytes) @ 0x00007f29667b7464 [0x00007f29667b72e0+0x184]

J 17796% C1 org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run()V (72 bytes) @ 0x00007f2967c9cbe4 [0x00007f2967c9caa0+0x144]

v  ~StubRoutines::call_stub

V  [libjvm.so+0x65ebbb]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x108b

V  [libjvm.so+0x65ffd7]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x2f7

V  [libjvm.so+0x660497]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47

V  [libjvm.so+0x6ada71]  thread_entry(JavaThread*, Thread*)+0x91

V  [libjvm.so+0x9f24f1]  JavaThread::thread_main_inner()+0xf1

V  [libjvm.so+0x9f26d8]  JavaThread::run()+0x1b8

V  [libjvm.so+0x8af502]  java_start(Thread*)+0x122

C  [libpthread.so.0+0x7dc5]  start_thread+0xc5

 

we used the workaround by switching *hbase.rpc.server.impl* back to SimpleRpcServer as mentioned in the following JIRA

https://issues.apache.org/jira/browse/HBASE-22539?focusedCommentId=16855688&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16855688

Also, attached the error logs during JVM crash. Any help?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)