You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Lasse Aagren <aa...@dtic.dk> on 2011/11/24 13:45:51 UTC
inconsistent JVM crash with version 4.0-SNAPSHOT
Hi,
We are running Solr-Lucene 4.0-SNAPSHOT (1199777M - hudson - 2011-11-09 14:58:50) on severel servers running:
64bit Debian Squeeze (6.0.3)
OpenJDK6 (b18-1.8.9-0.1~squeeze1)
Tomcat 6.028 (6.0.28-9+squeeze1)
Some of the servers have 48G RAM and in that case java have 16G (-Xmx16g) and some of the servers have 96G RAM and in that case java have 48G (-Xmx48G).
We are seeing some inconsistent crashes of tomcat's JVM under different Solr/Lucene operations/circumstances. Sadly we can't replicate it.
It doesn't happen often, but often enough that we can't rely on it in production.
When it happens, something like the following appears in the logs:
==========================================================================
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f6c318d0902, pid=16516, tid=139772378892032
#
# JRE version: 6.0_18-b18
# Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
# Derivative: IcedTea6 1.8.9
# Distribution: Debian GNU/Linux 6.0.2 (squeeze), package 6b18-1.8.9-0.1~squeeze1
# Problematic frame:
# j org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(Lorg/apache/lucene/index/IndexReader$AtomicReaderContext;Lorg/apache/lucene/util/Bits;)Lorg/apache/lucene/search/DocIdSet;+193
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid16516.log
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
#
==========================================================================
Every time it happens the problematic frame is:
Problematic frame:
# j org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(Lorg/apache/lucene/index/IndexReader$AtomicReaderContext;Lorg/apache/lucene/util/Bits;
)Lorg/apache/lucene/search/DocIdSet;+193
And /tmp/hs_err_pid16516.log is attached to this mail.
Has anyone seen this before?
Please don't hesitate to ask for further specification about our setup.
Best regards,
--
Lasse Aagren
DTU Library
---------------------------------------
Technical University of Denmark
Technical Information Center of Denmark
Anker Engelunds Vej 1
Building 101D
2800 Kgs. Lyngby
Direct +45 45257229
Mobile +45 40516542
laa@dtic.dtu.dk
http://www.dtic.dtu.dk/
Re: inconsistent JVM crash with version 4.0-SNAPSHOT
Posted by Lasse Aagren <aa...@dtic.dk>.
Thank you for the answer!
Shortly after writing to the list, we found that ticket ourselves. We think that it most likely is the cause (and solution) to our problems.
Hopefully we wont't see the crashes again.
Best regards,
Lasse Aagren
On Nov 26, 2011, at 04:41 , Erick Erickson wrote:
> Don't know if its this particular issue, but have you seen:
> https://issues.apache.org/jira/browse/LUCENE-3588
>
> Best
> Erick
>
> On Fri, Nov 25, 2011 at 4:59 PM, Justin Caratzas
> <ju...@gmail.com> wrote:
>> Lasse Aagren <aa...@dtic.dk> writes:
>>
>>> Hi,
>>>
>>> We are running Solr-Lucene 4.0-SNAPSHOT (1199777M - hudson - 2011-11-09 14:58:50) on severel servers running:
>>>
>>> 64bit Debian Squeeze (6.0.3)
>>> OpenJDK6 (b18-1.8.9-0.1~squeeze1)
>>> Tomcat 6.028 (6.0.28-9+squeeze1)
>>>
>>> Some of the servers have 48G RAM and in that case java have 16G (-Xmx16g) and some of the servers have 96G RAM and in that case java have 48G (-Xmx48G).
>>>
>>> We are seeing some inconsistent crashes of tomcat's JVM under different Solr/Lucene operations/circumstances. Sadly we can't replicate it.
>>>
>>> It doesn't happen often, but often enough that we can't rely on it in production.
>>>
>>> When it happens, something like the following appears in the logs:
>>>
>>> ==========================================================================
>>> #
>>> # A fatal error has been detected by the Java Runtime Environment:
>>> #
>>> # SIGSEGV (0xb) at pc=0x00007f6c318d0902, pid=16516, tid=139772378892032
>>> #
>>> # JRE version: 6.0_18-b18
>>> # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
>>> # Derivative: IcedTea6 1.8.9
>>> # Distribution: Debian GNU/Linux 6.0.2 (squeeze), package 6b18-1.8.9-0.1~squeeze1
>>> # Problematic frame:
>>> # j org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(Lorg/apache/lucene/index/IndexReader$AtomicReaderContext;Lorg/apache/lucene/util/Bits;)Lorg/apache/lucene/search/DocIdSet;+193
>>> #
>>> # An error report file with more information is saved as:
>>> # /tmp/hs_err_pid16516.log
>>> #
>>> # If you would like to submit a bug report, please include
>>> # instructions how to reproduce the bug and visit:
>>> # http://icedtea.classpath.org/bugzilla
>>> #
>>> ==========================================================================
>>>
>>> Every time it happens the problematic frame is:
>>>
>>> Problematic frame:
>>> # j org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(Lorg/apache/lucene/index/IndexReader$AtomicReaderContext;Lorg/apache/lucene/util/Bits;
>>> )Lorg/apache/lucene/search/DocIdSet;+193
>>>
>>> And /tmp/hs_err_pid16516.log is attached to this mail.
>>>
>>> Has anyone seen this before?
>>>
>>> Please don't hesitate to ask for further specification about our setup.
>>>
>>> Best regards,
>>
>> I seem to remember a recent java released fixed seemingly random
>> SIGSEGV's causing Solr/Lucene to crash non-deterministicly.
>>
>> http://lucene.apache.org/solr/#26+October+2011+-+Java+7u1+fixes+index+corruption+and+crash+bugs+in+Apache+Lucene+Core+and+Apache+Solr
>>
>> Hopefully this will provide you with some answers. If not, please let
>> the list know.
>>
>> justin
>>
>>
--
Lasse Aagren
DTU Library
---------------------------------------
Technical University of Denmark
Technical Information Center of Denmark
Anker Engelunds Vej 1
Building 101D
2800 Kgs. Lyngby
Direct +45 45257229
Mobile +45 40516542
laa@dtic.dtu.dk
http://www.dtic.dtu.dk/
Re: inconsistent JVM crash with version 4.0-SNAPSHOT
Posted by Erick Erickson <er...@gmail.com>.
Don't know if its this particular issue, but have you seen:
https://issues.apache.org/jira/browse/LUCENE-3588
Best
Erick
On Fri, Nov 25, 2011 at 4:59 PM, Justin Caratzas
<ju...@gmail.com> wrote:
> Lasse Aagren <aa...@dtic.dk> writes:
>
>> Hi,
>>
>> We are running Solr-Lucene 4.0-SNAPSHOT (1199777M - hudson - 2011-11-09 14:58:50) on severel servers running:
>>
>> 64bit Debian Squeeze (6.0.3)
>> OpenJDK6 (b18-1.8.9-0.1~squeeze1)
>> Tomcat 6.028 (6.0.28-9+squeeze1)
>>
>> Some of the servers have 48G RAM and in that case java have 16G (-Xmx16g) and some of the servers have 96G RAM and in that case java have 48G (-Xmx48G).
>>
>> We are seeing some inconsistent crashes of tomcat's JVM under different Solr/Lucene operations/circumstances. Sadly we can't replicate it.
>>
>> It doesn't happen often, but often enough that we can't rely on it in production.
>>
>> When it happens, something like the following appears in the logs:
>>
>> ==========================================================================
>> #
>> # A fatal error has been detected by the Java Runtime Environment:
>> #
>> # SIGSEGV (0xb) at pc=0x00007f6c318d0902, pid=16516, tid=139772378892032
>> #
>> # JRE version: 6.0_18-b18
>> # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
>> # Derivative: IcedTea6 1.8.9
>> # Distribution: Debian GNU/Linux 6.0.2 (squeeze), package 6b18-1.8.9-0.1~squeeze1
>> # Problematic frame:
>> # j org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(Lorg/apache/lucene/index/IndexReader$AtomicReaderContext;Lorg/apache/lucene/util/Bits;)Lorg/apache/lucene/search/DocIdSet;+193
>> #
>> # An error report file with more information is saved as:
>> # /tmp/hs_err_pid16516.log
>> #
>> # If you would like to submit a bug report, please include
>> # instructions how to reproduce the bug and visit:
>> # http://icedtea.classpath.org/bugzilla
>> #
>> ==========================================================================
>>
>> Every time it happens the problematic frame is:
>>
>> Problematic frame:
>> # j org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(Lorg/apache/lucene/index/IndexReader$AtomicReaderContext;Lorg/apache/lucene/util/Bits;
>> )Lorg/apache/lucene/search/DocIdSet;+193
>>
>> And /tmp/hs_err_pid16516.log is attached to this mail.
>>
>> Has anyone seen this before?
>>
>> Please don't hesitate to ask for further specification about our setup.
>>
>> Best regards,
>
> I seem to remember a recent java released fixed seemingly random
> SIGSEGV's causing Solr/Lucene to crash non-deterministicly.
>
> http://lucene.apache.org/solr/#26+October+2011+-+Java+7u1+fixes+index+corruption+and+crash+bugs+in+Apache+Lucene+Core+and+Apache+Solr
>
> Hopefully this will provide you with some answers. If not, please let
> the list know.
>
> justin
>
>
Re: inconsistent JVM crash with version 4.0-SNAPSHOT
Posted by Justin Caratzas <ju...@gmail.com>.
Lasse Aagren <aa...@dtic.dk> writes:
> Hi,
>
> We are running Solr-Lucene 4.0-SNAPSHOT (1199777M - hudson - 2011-11-09 14:58:50) on severel servers running:
>
> 64bit Debian Squeeze (6.0.3)
> OpenJDK6 (b18-1.8.9-0.1~squeeze1)
> Tomcat 6.028 (6.0.28-9+squeeze1)
>
> Some of the servers have 48G RAM and in that case java have 16G (-Xmx16g) and some of the servers have 96G RAM and in that case java have 48G (-Xmx48G).
>
> We are seeing some inconsistent crashes of tomcat's JVM under different Solr/Lucene operations/circumstances. Sadly we can't replicate it.
>
> It doesn't happen often, but often enough that we can't rely on it in production.
>
> When it happens, something like the following appears in the logs:
>
> ==========================================================================
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00007f6c318d0902, pid=16516, tid=139772378892032
> #
> # JRE version: 6.0_18-b18
> # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
> # Derivative: IcedTea6 1.8.9
> # Distribution: Debian GNU/Linux 6.0.2 (squeeze), package 6b18-1.8.9-0.1~squeeze1
> # Problematic frame:
> # j org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(Lorg/apache/lucene/index/IndexReader$AtomicReaderContext;Lorg/apache/lucene/util/Bits;)Lorg/apache/lucene/search/DocIdSet;+193
> #
> # An error report file with more information is saved as:
> # /tmp/hs_err_pid16516.log
> #
> # If you would like to submit a bug report, please include
> # instructions how to reproduce the bug and visit:
> # http://icedtea.classpath.org/bugzilla
> #
> ==========================================================================
>
> Every time it happens the problematic frame is:
>
> Problematic frame:
> # j org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(Lorg/apache/lucene/index/IndexReader$AtomicReaderContext;Lorg/apache/lucene/util/Bits;
> )Lorg/apache/lucene/search/DocIdSet;+193
>
> And /tmp/hs_err_pid16516.log is attached to this mail.
>
> Has anyone seen this before?
>
> Please don't hesitate to ask for further specification about our setup.
>
> Best regards,
I seem to remember a recent java released fixed seemingly random
SIGSEGV's causing Solr/Lucene to crash non-deterministicly.
http://lucene.apache.org/solr/#26+October+2011+-+Java+7u1+fixes+index+corruption+and+crash+bugs+in+Apache+Lucene+Core+and+Apache+Solr
Hopefully this will provide you with some answers. If not, please let
the list know.
justin