You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "dragondgold (via GitHub)" <gi...@apache.org> on 2024/01/18 19:25:30 UTC

[I] SIGSEGV error when creating inverted index in MV column from large parquet files [pinot]

dragondgold opened a new issue, #12286:
URL: https://github.com/apache/pinot/issues/12286

   When creating and inverted index in a large MV column (3000 integer values on average) from a parquet file with many rows (2 million rows) I get a `SIGSEV` error:
   
   ```
   #
   # A fatal error has been detected by the Java Runtime Environment:
   #
   #  SIGSEGV (0xb) at pc=0x00007f4d7fd1c9d8, pid=671, tid=808
   #
   # JRE version: OpenJDK Runtime Environment Corretto-11.0.20.9.1 (11.0.20.1+9) (build 11.0.20.1+9-LTS)
   # Java VM: OpenJDK 64-Bit Server VM Corretto-11.0.20.9.1 (11.0.20.1+9-LTS, mixed mode, tiered, serial gc, linux-amd64)
   # Problematic frame:
   # J 10100 c2 org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.indexRow(Lorg/apache/pinot/spi/data/readers/GenericRow;)V (228 bytes) @ 0x00007f4d7fd1c9d8 [0x00007f4d7fd1b580+0x0000000000001458]
   #
   # Core dump will be written. Default location: /opt/pinot/core.671
   #
   # If you would like to submit a bug report, please visit:
   #   https://github.com/corretto/corretto-11/issues/
   #
   
   ---------------  S U M M A R Y ------------
   
   Command Line: -Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx110G -XX:+UseSerialGC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log -Dplugins.dir=/opt/pinot/plugins -Dapp.name=pinot-admin -Dapp.pid=671 -Dapp.repo=/opt/pinot/lib -Dapp.home=/opt/pinot -Dbasedir=/opt/pinot -Dorg.slf4j.simpleLogger.defaultLogLevel=debug -Dlog4j2.configurationFile=/opt/pinot/etc/conf/pinot-controller-log4j2.xml org.apache.pinot.tools.admin.PinotAdministrator LaunchDataIngestionJob -jobSpecFile /opt/data/job.yml
   
   Host: AMD EPYC 7R13 Processor, 32 cores, 123G, Amazon Linux release 2 (Karoo)
   Time: Fri Dec 22 15:56:34 2023 UTC elapsed time: 1015.369777 seconds (0d 0h 16m 55s)
   
   ---------------  T H R E A D  ---------------
   
   Current thread (0x00007f4d90241000):  JavaThread "pool-3-thread-1" [_thread_in_Java, id=808, stack(0x00007f31da549000,0x00007f31da64a000)]
   
   Stack: [0x00007f31da549000,0x00007f31da64a000],  sp=0x00007f31da6484f0,  free space=1021k
   Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
   J 10100 c2 org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.indexRow(Lorg/apache/pinot/spi/data/readers/GenericRow;)V (228 bytes) @ 0x00007f4d7fd1c9d8 [0x00007f4d7fd1b580+0x0000000000001458]
   J 10106% c2 org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build()V (376 bytes) @ 0x00007f4d7fd247a0 [0x00007f4d7fd246a0+0x0000000000000100]
   j  org.apache.pinot.plugin.ingestion.batch.common.SegmentGenerationTaskRunner.run()Ljava/lang/String;+228
   j  org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$submitSegmentGenTask$1(Lorg/apache/pinot/spi/ingestion/batch/spec/SegmentGenerationTaskSpec;Ljava/io/File;Ljava/net/URI;Ljava/io/File;)V+18
   j  org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner$$Lambda$780.run()V+20
   j  java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4 java.base@11.0.20.1
   j  java.util.concurrent.FutureTask.run()V+39 java.base@11.0.20.1
   j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 java.base@11.0.20.1
   j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@11.0.20.1
   j  java.lang.Thread.run()V+11 java.base@11.0.20.1
   v  ~StubRoutines::call_stub
   V  [libjvm.so+0x8e13bb]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x39b
   V  [libjvm.so+0x8df37d]  JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x1ed
   V  [libjvm.so+0x98ae7c]  thread_entry(JavaThread*, Thread*)+0x6c
   V  [libjvm.so+0xedf730]  JavaThread::run()+0x280
   V  [libjvm.so+0xedc0ff]  Thread::call_run()+0x14f
   V  [libjvm.so+0xc78ea6]  thread_native_entry(Thread*)+0xe6
   
   
   siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007f31602ff000
   
   Register to memory mapping:
   
   RAX=0x0 is NULL
   RBX=0x00007f31602ff000 is an unknown value
   RCX=0x00007f3c21365d58 is an oop: java.util.HashMap 
   {0x00007f3c21365d58} - klass: 'java/util/HashMap'
    - ---- fields (total size 8 words):
    - transient strict 'keySet' 'Ljava/util/Set;' @16  NULL (0 0)
    - transient strict 'values' 'Ljava/util/Collection;' @24  a 'java/util/HashMap$Values'{0x00007f3c21370760} (21370760 7f3c)
    - transient 'size' 'I' @32  1
    - transient 'modCount' 'I' @36  1
    - 'threshold' 'I' @40  12 (c)
    - final 'loadFactor' 'F' @44  0.750000 (3f400000)
    - transient strict 'table' '[Ljava/util/HashMap$Node;' @48  a 'java/util/HashMap$Node'[16] {0x00007f3c21370778} (21370778 7f3c)
    - transient strict 'entrySet' 'Ljava/util/Set;' @56  NULL (0 0)
   RDX=0x00007f31602ff000 is an unknown value
   RSP=0x00007f31da6484f0 is pointing into the stack for thread: 0x00007f4d90241000
   RBP=0x00007f319473b358 is a pointer to class: 
   xerial.larray.mmap.MMapBuffer {0x00007f319473b358}
    - instance size:     9
    - klass size:        122
    - access:            public synchronized 
    - state:             fully_initialized
    - name:              'xerial/larray/mmap/MMapBuffer'
    - super:             'xerial/larray/buffer/LBufferAPI'
    - sub:               
    - arrays:            NULL
    - methods:           Array<T>(0x00007f319473ad68)
    - method ordering:   Array<T>(0x00007f31e0155018)
    - default_methods:   Array<T>(0x0000000000000000)
    - local interfaces:  Array<T>(0x00007f31e0155060)
    - trans. interfaces: Array<T>(0x00007f31e0155060)
    - constants:         constant pool [227] {0x00007f319473a500} for 'xerial/larray/mmap/MMapBuffer' cache=0x00007f319473d190
    - class loader data:  loader data: 0x00007f4d90101eb0 for instance a 'jdk/internal/loader/ClassLoaders$AppClassLoader'{0x00007f3b223090f8}
    - host class:        NULL
    - source file:       'MMapBuffer.java'
    - class annotations:       Array<T>(0x0000000000000000)
    - class type annotations:  Array<T>(0x0000000000000000)
    - field annotations:       Array<T>(0x0000000000000000)
    - field type annotations:  Array<T>(0x0000000000000000)
    - inner classes:     Array<T>(0x00007f31e0155030)
    - nest members:     Array<T>(0x00007f31e0155030)
    - java mirror:       a 'java/lang/Class'{0x00007f3ba37080b8} = 'xerial/larray/mmap/MMapBuffer'
    - vtable length      60  (start addr: 0x00007f319473b528)
    - itable length      2 (start addr: 0x00007f319473b708)
    - ---- static fields (0 words):
    - ---- non-static fields (7 words):
    - public 'm' 'Lxerial/larray/buffer/Memory;' @16 
    - private final 'fd' 'J' @24 
    - private final 'address' 'J' @32
   ```
   
   Reducing the parquet file from 2M rows to 1.2M rows results in an index out of bounds error instead of `SIGSEV`. Reducing the row count event more to 700k rows works as expected.
   
   My guess, when `number_of_rows * MV_column_length` is slightly over `Integer.MAX_VALUE` I get an index out of bounds error, when it goes over `Integer.MAX_VALUE` for a lot (i don't know how much exactly) I get a `SIGSEV` error, so I think the issue is when using an inverted inde and `number_of_rows * MV_column_length > Integer.MAX_VALUE` , probably because a 32-bit roaring bitmap is being used?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] SIGSEGV error when creating inverted index in MV column from large parquet files [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #12286:
URL: https://github.com/apache/pinot/issues/12286#issuecomment-2314997416

   Larray can be used to create buffers larger than >2GBs, but Larray is not maintained and not safe (see more in https://github.com/apache/pinot/issues/12810). With not safe I mean that there is no offset check when accessing memory with LArray, which means that:
   1. it can be used to execute buffer overflow attacks (something we are not used in Java)
   2. an error in an offset usually means that an illegal memory address is accessed which means that a SEGSEV is produced.
   
   For the context I'm not 100% sure this is the actual reason that produces the specific error reported here. In fact in some very strange scenarios we have seen SEGSEV errors even when using ByteBuffers when code is compiled with C2. But in general we recommend to do not use LArray and in fact Pinot 1.2.0 does not use LArray by default.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] SIGSEGV error when creating inverted index in MV column from large parquet files [pinot]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12286:
URL: https://github.com/apache/pinot/issues/12286#issuecomment-5642138220

   This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] SIGSEGV error when creating inverted index in MV column from large parquet files [pinot]

Posted by "dd-willgan (via GitHub)" <gi...@apache.org>.
dd-willgan commented on issue #12286:
URL: https://github.com/apache/pinot/issues/12286#issuecomment-2305568237

   Hey @gortiz , why would using > 2GB LArray buffers be an issue? Looking at their repo https://github.com/xerial/larray it seems like the first thing they advertise is that > 2GB buffers can be supported?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] SIGSEGV error when creating inverted index in MV column from large parquet files [pinot]

Posted by "ksnijjer (via GitHub)" <gi...@apache.org>.
ksnijjer commented on issue #12286:
URL: https://github.com/apache/pinot/issues/12286#issuecomment-1934779531

   @snleee ^


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] SIGSEGV error when creating inverted index in MV column from large parquet files [pinot]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12286:
URL: https://github.com/apache/pinot/issues/12286#issuecomment-5274341982

   This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] SIGSEGV error when creating inverted index in MV column from large parquet files [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #12286:
URL: https://github.com/apache/pinot/issues/12286#issuecomment-1935438524

   This kind of problems should not produce a SIGSEV. I think this may be related to using LArray buffers when the index is larger than 2GBs.
   
   One of the issues of LArray is that it doesn't check memory offsets and that may produce SIGSEVs. The other issue of LArrays is that it doesn't work in Java > 15. Therefore we created our own library to be able to run in modern Java versions.
   
   We can tests whether the issue is fired by LArray by changing the library used. This is not going to fix the issue, but it is not going to kill the process in case it happens. Could you run the same job on a Pinot cluster using Java 17 or 21? Alternatively our library can be used in Java 11 by changing the value of pinot server property `pinot.offheap.buffer.factory`.
   
   ```
   pinot.offheap.buffer.factory = org.apache.pinot.segment.spi.memory.unsafe
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] SIGSEGV error when creating inverted index in MV column from large parquet files [pinot]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12286: SIGSEGV error when creating inverted index in MV column from large parquet files
URL: https://github.com/apache/pinot/issues/12286


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org