You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2016/07/05 18:02:11 UTC
[jira] [Comment Edited] (LUCENE-7183) JVM crash at
ForUtil.readBlock
[ https://issues.apache.org/jira/browse/LUCENE-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362883#comment-15362883 ]
Uwe Schindler edited comment on LUCENE-7183 at 7/5/16 6:01 PM:
---------------------------------------------------------------
Hi,
What all three stack traces have in common: It looks like you are trying to search on an already closed IndexReader. Lucene tries to catch this, but under highly concurrent cases, it may happen that you try to access stuff that was already unmapped: Lucene memory maps the index into virtual address space, but forcefully unmaps this when closing indexreader. If you do this and e.g. another search is still running while closing (in another thread), the forceful unmapping will cause SIGSEGV or SIGBUS.
To debug this you can do the following:
- Change the FSDirectory implementation (which defaults to MMapDirectory) on 64 bit platforms to NIOFSDirectory. This will slow down searches, but does not have the risk of segfaults. Instead it will throw stuff like AlreadyClosedExceptions
- Check your code that you don't suppress any exceptions. It looks like your code meight do this, so you don't see the root cause (closing index while searches are running)
Once you found the bad code parts and fixed the issues in your code (closing IndexReader while searches are running), you can switch back to MMapDirectory.
was (Author: thetaphi):
Hi,
What all three stack traces have in common: It looks like you are trying to search on an already closed IndexReader. Lucene tries to ctahc this, but under highly concurrent cases, it may happen that you try to access stuff that was already unmapped: Lucene memory maps the index into virtual address space, but forcefully unmaps this when closing indexreader. If you do this and e.g. another search is still running while closing (in another thread), the forceful unmapping will cause SIGSEGV or SIGBUS.
To debug this you can do the following:
- Change the FSDirectory implementation (which defaults to MMapDirectory) on 64 bit platforms to NIOFSDirectory. This will slow down searches, but does not have the risk of segfaults. Instead it will throw stuff like AlreadyClosedExceptions
- Check your code that you don't suppress any exceptions. It looks like your code meight do this, so you don't see the root cause (closing index while searches are running)
Once you found the bad code parts and fixed the issues in your code (closing IndexReader while searches are running), you can switch back to MMapDirectory.
> JVM crash at ForUtil.readBlock
> ------------------------------
>
> Key: LUCENE-7183
> URL: https://issues.apache.org/jira/browse/LUCENE-7183
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
> Affects Versions: 4.9
> Environment: linux 64bit, java build 1.8.0_71-b15
> Reporter: David Rainer
> Attachments: hs_err_pid22587.log, hs_err_pid7567.log, hs_err_pid8645.log
>
>
> The JVM crashes occasionaly when performing IndexSearcher.search()
> Do you have any suggestions?
> Best Regards
> David
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00007ff64cea7bdd, pid=8645, tid=140693552637696
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_71-b15) (build 1.8.0_71-b15)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.71-b15 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # J 8876 C2 org.apache.lucene.codecs.lucene41.ForUtil.readBlock(Lorg/apache/lucene/store/IndexInput;[B[I)V (130 bytes) @ 0x00007ff64cea7bdd [0x00007ff64cea7b40+0x9d]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> #
> --------------- T H R E A D ---------------
> Current thread (0x00007ff620073000): JavaThread "http-nio-8081-exec-695" daemon [_thread_in_Java, id=31060, stack(0x00007ff5c52a2000,0x00007ff5c53a3000)]
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007ff5cef8f848
> Register to memory mapping:
> RAX=0x0000000756653d80 is an oop
> org.apache.lucene.codecs.lucene41.ForUtil
> - klass: 'org/apache/lucene/codecs/lucene41/ForUtil'
> RBX=0x000000000040b848 is an unknown value
> RCX=0x00007ff5ceb84000 is an unknown value
> RDX=0x00007ff5ceb84000 is an unknown value
> RSP=0x00007ff5c53a0bb0 is pointing into the stack for thread: 0x00007ff620073000
> RBP=0x00000007a3f4dd08 is an oop
> org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum
> - klass: 'org/apache/lucene/codecs/lucene41/Lucene41PostingsReader$BlockDocsEnum'
> RSI=0x0000000756653d80 is an oop
> org.apache.lucene.codecs.lucene41.ForUtil
> - klass: 'org/apache/lucene/codecs/lucene41/ForUtil'
> RDI=0x00007ff5cef8f849 is an unknown value
> R8 =0x000000000040b849 is an unknown value
> R9 =0x000000000040b849 is an unknown value
> R10=0x000000000040b848 is an unknown value
> R11=0x00000000005d52d6 is an unknown value
> R12=0x0000000000000000 is an unknown value
> R13=0x0000000000000007 is an unknown value
> R14=0x00000007a3f4e4e0 is an oop
> java.nio.DirectByteBufferR
> - klass: 'java/nio/DirectByteBufferR'
> R15=0x00007ff620073000 is a thread
> Stack: [0x00007ff5c52a2000,0x00007ff5c53a3000], sp=0x00007ff5c53a0bb0, free space=1018k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> J 8876 C2 org.apache.lucene.codecs.lucene41.ForUtil.readBlock(Lorg/apache/lucene/store/IndexInput;[B[I)V (130 bytes) @ 0x00007ff64cea7bdd [0x00007ff64cea7b40+0x9d]
> J 10134 C2 org.apache.lucene.search.TermScorer.nextDoc()I (8 bytes) @ 0x00007ff64d51dd2c [0x00007ff64d51db20+0x20c]
> J 8905 C2 org.apache.lucene.search.Weight$DefaultBulkScorer.score(Lorg/apache/lucene/search/Collector;I)Z (55 bytes) @ 0x00007ff64cc94d18 [0x00007ff64cc949c0+0x358]
> J 12718 C2 org.apache.lucene.search.IndexSearcher.search(Ljava/util/List;Lorg/apache/lucene/search/Weight;Lorg/apache/lucene/search/Collector;)V (92 bytes) @ 0x00007ff64dbfe5ac [0x00007ff64dbfe0c0+0x4ec]
> J 21599 C2 com.gentics.cr.lucene.autocomplete.Autocompleter.suggestWords(Lcom/gentics/cr/CRRequest;)Ljava/util/Collection; (316 bytes) @ 0x00007ff64f65b708 [0x00007ff64f65a660+0x10a8]
> J 15153 C2 com.gentics.cr.lucene.autocomplete.AutocompleteRequestProcessor.getObjects(Lcom/gentics/cr/CRRequest;Z)Ljava/util/Collection; (28 bytes) @ 0x00007ff64dd2fa40 [0x00007ff64dd2fa00+0x40]
> J 14318 C2 com.gentics.cr.rest.RESTSimpleContainer.processService(Lcom/gentics/cr/CRRequest;Lcom/gentics/cr/util/ContentRepositoryConfig;Ljava/util/Map;Ljava/io/OutputStream;Lcom/gentics/cr/util/response/IResponseTypeSetter;Z)V (407 bytes) @ 0x00007ff64e0aed74 [0x00007ff64e0ad860+0x1514]
> J 14370 C2 com.gentics.cr.rest.RESTServlet.doServiceSafe(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (354 bytes) @ 0x00007ff64e0d8344 [0x00007ff64e0d5940+0x2a04]
> J 14237 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (574 bytes) @ 0x00007ff64d6b8568 [0x00007ff64d6b7e80+0x6e8]
> J 14237 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (574 bytes) @ 0x00007ff64d6b7fc0 [0x00007ff64d6b7e80+0x140]
> J 22020 C2 org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (1414 bytes) @ 0x00007ff64c4f41c8 [0x00007ff64c4f36a0+0xb28]
> J 15254 C2 org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (903 bytes) @ 0x00007ff64cd819f0 [0x00007ff64cd806e0+0x1310]
> J 21449 C2 org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (406 bytes) @ 0x00007ff64d36ad64 [0x00007ff64d36abc0+0x1a4]
> J 22032 C2 org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V (660 bytes) @ 0x00007ff64ea97964 [0x00007ff64ea97420+0x544]
> J 22013 C2 org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (1102 bytes) @ 0x00007ff64f5df144 [0x00007ff64f5de940+0x804]
> J 21897 C2 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (825 bytes) @ 0x00007ff64f6e9b8c [0x00007ff64f6e9640+0x54c]
> J 22016 C2 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(Ljava/nio/channels/SelectionKey;Lorg/apache/tomcat/util/net/NioEndpoint$KeyAttachment;)V (792 bytes) @ 0x00007ff64e6ed848 [0x00007ff64e6ed780+0xc8]
> J 14471 C2 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run()V (106 bytes) @ 0x00007ff64c074b28 [0x00007ff64c074980+0x1a8]
> J 20138 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007ff64f2327b8 [0x00007ff64f2324c0+0x2f8]
> J 21511 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x00007ff64d456b04 [0x00007ff64d456a00+0x104]
> J 22048 C1 org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V (25 bytes) @ 0x00007ff64edee04c [0x00007ff64ededf40+0x10c]
> J 23078 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007ff64ee532ec [0x00007ff64ee532a0+0x4c]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x68c656] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
> V [libjvm.so+0x68cb61] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
> V [libjvm.so+0x68d007] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
> V [libjvm.so+0x723dc0] thread_entry(JavaThread*, Thread*)+0xa0
> V [libjvm.so+0xa69cff] JavaThread::thread_main_inner()+0xdf
> V [libjvm.so+0xa69e2c] JavaThread::run()+0x11c
> V [libjvm.so+0x91d918] java_start(Thread*)+0x108
> C [libpthread.so.0+0x7aa1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org