You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "donatelloOo (via GitHub)" <gi...@apache.org> on 2024/02/06 16:16:11 UTC
[I] SIGSEGV errors while ingesting realtime data [pinot]
donatelloOo opened a new issue, #12376:
URL: https://github.com/apache/pinot/issues/12376
While ingesting real-time data in a single table we are getting this SIGSEGV fatal error periodically on all servers.
This table is composed of:
- a few dimension fields (SV with inverted indexes)
- a lot of metric fields (SV with forward index only)
- a timestamp field
Below is a short extract of the core dump log (full one is attached).
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f9ba3e89056, pid=1, tid=247
#
# JRE version: OpenJDK Runtime Environment Corretto-11.0.20.9.1 (11.0.20.1+9) (build 11.0.20.1+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM Corretto-11.0.20.9.1 (11.0.20.1+9-LTS, mixed mode, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# J 3816 c2 org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReader.getValue(I)Ljava/lang/Object; (648 bytes) @ 0x00007f9ba3e89056 [0x00007f9ba3e88f20+0x0000000000000136]
#
# Core dump will be written. Default location: /opt/pinot/core.1
#
# If you would like to submit a bug report, please visit:
# https://github.com/corretto/corretto-11/issues/
#
--------------- S U M M A R Y ------------
Command Line: -Xms512M -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ErrorFileToStdout -XX:-TieredCompilation -Xlog:gc*:file=/opt/pinot/gc-pinot-server.log -javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Dlog4j2.configurationFile=/opt/pinot/etc/conf/pinot-server-log4j2.xml -Dplugins.dir=/opt/pinot/plugins -Dplugins.dir=/opt/pinot/plugins -Dapp.name=pinot-admin -Dapp.pid=1 -Dapp.repo=/opt/pinot/lib -Dapp.home=/opt/pinot -Dbasedir=/opt/pinot org.apache.pinot.tools.admin.PinotAdministrator StartServer -clusterName pinot -zkAddress pinot-zookeeper:2181 -configFileName /var/pinot/server/config/pinot-server.conf
Host: Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz, 16 cores, 32G, Amazon Linux release 2 (Karoo)
Time: Tue Feb 6 12:41:51 2024 UTC elapsed time: 2970.773692 seconds (0d 0h 49m 30s)
--------------- T H R E A D ---------------
Current thread (0x00007f9a04062800): JavaThread "obf_50d04beb9d1d306c5a5e45167656a565abc036da0763183eb9f074402fbd4f2c__0__3__20240206T1214Z" daemon [_thread_in_Java, id=247, stack(0x00007f9a264f6000,0x00007f9a265f7000)]
Stack: [0x00007f9a264f6000,0x00007f9a265f7000], sp=0x00007f9a265f55e0, free space=1021k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 3816 c2 org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReader.getValue(I)Ljava/lang/Object; (648 bytes) @ 0x00007f9ba3e89056 [0x00007f9ba3e88f20+0x0000000000000136]
J 3819 c2 org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader.getRecord(ILorg/apache/pinot/spi/data/readers/GenericRow;)V (106 bytes) @ 0x00007f9ba3e726ac [0x00007f9ba3e72080+0x000000000000062c]
J 4599 c2 org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build()V (376 bytes) @ 0x00007f9ba3dc04c4 [0x00007f9ba3dbfc60+0x0000000000000864]
j org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(Lorg/apache/pinot/segment/spi/creator/SegmentVersion;Lorg/apache/pinot/common/metrics/ServerMetrics;)V+282
j org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.buildSegmentInternal(Z)Lorg/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager$SegmentBuildDescriptor;+206
j org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.buildSegmentAndReplace()Z+2
j org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run()V+561
j java.lang.Thread.run()V+11 java.base@11.0.20.1
v ~StubRoutines::call_stub
V [libjvm.so+0x8e13bb] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x39b
V [libjvm.so+0x8df37d] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x1ed
V [libjvm.so+0x98ae7c] thread_entry(JavaThread*, Thread*)+0x6c
V [libjvm.so+0xedf730] JavaThread::run()+0x280
V [libjvm.so+0xedc0ff] Thread::call_run()+0x14f
V [libjvm.so+0xc78ea6] thread_native_entry(Thread*)+0xe6
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f95f0137ac8
```
> - Sensitive data has been obfuscated.
> - Tiered Compilation was disabled to try fixing this kind of SIGSEGV errors while invoking MutableForwardIndex / OffHeapMutableDictionary
See full core dump here:
[SIGSEGV-obf.log](https://github.com/apache/pinot/files/14183217/SIGSEGV-obf.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] SIGSEGV errors while ingesting realtime data [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #12376:
URL: https://github.com/apache/pinot/issues/12376#issuecomment-1935442118
May be related to #12286. I would suggest the same thing I suggested there. Could you try again but running Pinot with Java 17 or 21? Alternatively, could you change `pinot.offheap.buffer.factory` to `org.apache.pinot.segment.spi.memory.unsafe.UnsafePinotBufferFactory`? That change should be applied in the pinot-`server.conf` file.
This change would probably not fix the issue but may prevent the SIGSEV.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] SIGSEGV errors while ingesting realtime data [pinot]
Posted by "donatelloOo (via GitHub)" <gi...@apache.org>.
donatelloOo commented on issue #12376:
URL: https://github.com/apache/pinot/issues/12376#issuecomment-1935506167
Hi @gortiz, thanks for your answer.
We already tried using java-17 amazon corretto but we got same issue.
I will try with the UnsafePinotBufferFactory and provide the feedback soon.
@snleee I can work on a reproducer with shareable data/schema, I will let you know when it's available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] SIGSEGV errors while ingesting realtime data [pinot]
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee commented on issue #12376:
URL: https://github.com/apache/pinot/issues/12376#issuecomment-1934763716
@donatelloOo This looks to be some low level bug where we write & read the data from the segment. Is it possible for us to get some extra information on how to reproduce issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] SIGSEGV errors while ingesting realtime data [pinot]
Posted by "donatelloOo (via GitHub)" <gi...@apache.org>.
donatelloOo commented on issue #12376:
URL: https://github.com/apache/pinot/issues/12376#issuecomment-1938685024
The only way to limit such errors seems to add JIT compiler exclusions like below:
```
-XX:CompileCommand=exclude,org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReader::getValue
-XX:CompileCommand=exclude,org.apache.pinot.segment.spi.index.mutable.MutableForwardIndex::getDictId
-XX:CompileCommand=exclude,org.apache.pinot.segment.spi.memory.PinotByteBuffer::getInt
-XX:CompileCommand=exclude,org.apache.pinot.segment.local.realtime.impl.forward.FixedByteSVMutableForwardIndex::getDictId
-XX:CompileCommand=exclude,org.apache.pinot.segment.local.realtime.impl.forward.FixedByteSVMutableForwardIndex$ReaderWithOffset::getInt
-XX:CompileCommand=exclude,org.apache.pinot.segment.local.realtime.impl.forward.FixedByteSVMutableForwardIndex::getLong
-XX:CompileCommand=exclude,org.apache.pinot.segment.local.realtime.impl.dictionary.LongOffHeapMutableDictionary::get
-XX:CompileCommand=exclude,org.apache.pinot.segment.local.realtime.impl.dictionary.LongOffHeapMutableDictionary::getLongValue
-XX:CompileCommand=exclude,org.apache.pinot.segment.local.io.reader.impl.FixedByteSingleValueMultiColReader::getInt
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org