You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Anoop Sam John (Jira)" <ji...@apache.org> on 2021/03/10 06:08:00 UTC
[jira] [Commented] (HBASE-25628) Regionserver crash occasionally
due to SIGSEGV
[ https://issues.apache.org/jira/browse/HBASE-25628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298545#comment-17298545 ]
Anoop Sam John commented on HBASE-25628:
----------------------------------------
[~zhangduo] I remember seeing a Jira which speaks abt buffer corruption during writes to WAL (AsyncFSWAL?) Seems that was some issue in an async write path where we reuse the buffer before completing the write or so. Not able to remember. Or was it related to asycn mode of wal write?
> Regionserver crash occasionally due to SIGSEGV
> ----------------------------------------------
>
> Key: HBASE-25628
> URL: https://issues.apache.org/jira/browse/HBASE-25628
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 2.1.0
> Reporter: SweetCandy
> Priority: Major
> Attachments: hs_err_pid15967.log, hs_err_pid3375.log
>
>
> Regionserver had a problem which is JVM crash after running for a period of time. From the error dump logs, the main reason is due to SIGSEGV singal.
> BTW, from the start of the regionserver to the crash, there is no error record in the hbase log
> After many crash, there are some common points.
> # Thread is AsyncFSWAL-0.
> # Problematic frame is v ~StubRoutines::jshort_disjoint_arraycopy
> # Error method is org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter$OutputStreamWrapper.write.
> jdk: Oracle jdk-1.8_181
> os: CentOS Linux release 7.4.1708 (Core)
> hbase: 2.1.0-cdh6.3.0
>
> {code:java}
> # JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 )
> # Problematic frame:# v ~StubRoutines::jshort_disjoint_arraycopy
> {code}
>
> {code:java}
> --------------- T H R E A D ---------------
> Current thread (0x00007f4758820800): JavaThread "AsyncFSWAL-0" daemon [_thread_in_Java, id=17396, stack(0x00007f36f0aa3000,0x00007f36f0ae4000)]
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f36bc3dd93a
> Stack: [0x00007f36f0aa3000,0x00007f36f0ae4000], sp=0x00007f36f0ae2310, free space=252kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> v ~StubRoutines::jshort_disjoint_arraycopy
> J 13350 C2 org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter$OutputStreamWrapper.write(Ljava/nio/ByteBuffer;II)V (34 bytes) @ 0x00007f475f113fd1 [0x00007f475f113cc0+0x311]
> J 16839 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I (21 bytes) @ 0x00007f475f4329dc [0x00007f475f432960+0x7c]
> J 12594 C2 org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$EnsureKvEncoder.write(Lorg/apache/hadoop/hbase/Cell;)V (27 bytes) @ 0x00007f475ef80414 [0x00007f475ef7fc20+0x7f4]
> VM Arguments:
> jvm_args:
> -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p
> -Djava.net.preferIPv4Stack=true -Xms33285996544 -Xmx33285996544 -Xmx64g -Xms32g -Xmn6g -Xss256k
> -XX:MaxPermSize=384m -XX:SurvivorRatio=6 -XX:+UseParNewGC -XX:ParallelGCThreads=10 -XX:+UseConcMarkSweepGC
> -XX:ParallelCMSThreads=16 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSMaxAbortablePrecleanTime=500 -XX:CMSFullGCsBeforeCompaction=5 -XX:+CMSClassUnloadingEnabled
> -XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/log/tmp/hbase/hbase_hbase-REGIONSERVER-58f025f30c5505722d8dee21c92b9469_pid15967.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh -Dhbase.log.dir=/log/hbase -Dhbase.log.file=hbase-cmf-hbase-REGIONSERVER-spbsjzy05.log.out -Dhbase.home.dir=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hbase -Dhbase.id.str= -Dhbase.root.logger=INFO,RFA -Djava.library.path=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hbase/lib/native/Linux-amd64-64 -Dhbase.security.logger=INFO,RFAS
> java_command: org.apache.hadoop.hbase.regionserver.HRegionServer start
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)