You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Nicolae Mihalache <xp...@yahoo.com> on 2009/07/30 21:54:41 UTC
How to get a stack trace
Hello,
I'm a new user of solr but I have worked a bit with Lucene before. I get some out of memory exception when optimizing the index through Solr and I would like to find out why.
However, the only message I get on standard output is:
Jul 30, 2009 9:20:22 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
Is there a way to get a stack trace for this exception? I had a look into the java.util.logging options and didn't find anything.
My solr runs in some standard configuration inside jetty.
Any suggestion would be appreciated.
Thanks,
nicolae
Re: How to get a stack trace
Posted by Andrzej Bialecki <ab...@getopt.org>.
Chris Hostetter wrote:
> : I'm a new user of solr but I have worked a bit with Lucene before. I get
> : some out of memory exception when optimizing the index through Solr and
> : I would like to find out why. However, the only message I get on
> : standard output is: Jul 30, 2009 9:20:22 PM
> : org.apache.solr.common.SolrException log SEVERE:
> : java.lang.OutOfMemoryError: Java heap space
> :
> : Is there a way to get a stack trace for this exception? I had a look
> : into the java.util.logging options and didn't find anything.
>
> FWIW #1: OutOfMemoryError is a java "Error" not an "Exception" ...
> Exceptions and Errors are both Throwable, but an Error is not an
> Exception. this is a really importatn distinction (see below)
>
> FWIW #2: when dealing with an OOM, a stack trace is almost never useful.
> as mentioned in other threads, a heapdump is the most useful diagnostic
> tool
>
> FWIW #3: the formatting of Throwables in log files is 100% dependent
> on the configuration of the log manager -- the client code doing
> the logging just specifies the Throwable object -- it's up to the
> Formatter to decide how to output it.
>
> Ok .. on to the meat of hte issue...
>
> OOM Errors are a particularly devious class of errors: they don't
> neccessarily have stack traces (depending on your VM impl, and the state
> of the VM when it tries to log the OOM) ....
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4753347
> http://blogs.sun.com/alanb/entry/outofmemoryerror_looks_a_bit_better
>
> ...on any *Exception* you should get a detailed stacktrace in the logs
> (unless you have a really screwed up LogManger configs), but when dealing
> with *Errors* like OutOfMemoryError, all bets are off as to what hte VM
> can give you.
I had some success in debugging this type of problems when I would
generate a heap dump on OOM (it's a JVM flag) and then use a tool like
HAT to find largest objects and references to them.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: How to get a stack trace
Posted by Chris Hostetter <ho...@fucit.org>.
: I'm a new user of solr but I have worked a bit with Lucene before. I get
: some out of memory exception when optimizing the index through Solr and
: I would like to find out why. However, the only message I get on
: standard output is: Jul 30, 2009 9:20:22 PM
: org.apache.solr.common.SolrException log SEVERE:
: java.lang.OutOfMemoryError: Java heap space
:
: Is there a way to get a stack trace for this exception? I had a look
: into the java.util.logging options and didn't find anything.
FWIW #1: OutOfMemoryError is a java "Error" not an "Exception" ...
Exceptions and Errors are both Throwable, but an Error is not an
Exception. this is a really importatn distinction (see below)
FWIW #2: when dealing with an OOM, a stack trace is almost never useful.
as mentioned in other threads, a heapdump is the most useful diagnostic
tool
FWIW #3: the formatting of Throwables in log files is 100% dependent
on the configuration of the log manager -- the client code doing
the logging just specifies the Throwable object -- it's up to the
Formatter to decide how to output it.
Ok .. on to the meat of hte issue...
OOM Errors are a particularly devious class of errors: they don't
neccessarily have stack traces (depending on your VM impl, and the state
of the VM when it tries to log the OOM) ....
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4753347
http://blogs.sun.com/alanb/entry/outofmemoryerror_looks_a_bit_better
...on any *Exception* you should get a detailed stacktrace in the logs
(unless you have a really screwed up LogManger configs), but when dealing
with *Errors* like OutOfMemoryError, all bets are off as to what hte VM
can give you.
-Hoss
Re: How to get a stack trace
Posted by Nicolae Mihalache <xp...@gmail.com>.
Hello,
Thanks for your answers and suggestions. I can get a heap dump also with the
jmap command. The resulting file is so big that jhat gets out of memory
errors itself when reading the dump.
I traced back my problem (using the heap dump analyser from yourkit.com) to
the FieldCache. In fact it had nothing to do with the index optimization but
with some queries running in background from cron doing faceted search. I
will come back with another email about that.
But nevertheless, wouldn't be a good idea to print the stack trace when
SEVERE errors are encountered?
nicolae
On Mon, Aug 3, 2009 at 3:20 AM, Bill Au <bi...@gmail.com> wrote:
> Your heap may be just too small or you may have a memory leak. A stack
> trace may not help you since the thread encountered the OutOfMemoryError
> may
> not be where the memory leak is. A heap dump will tell you what's using up
> all the memory in your heap.
> Bill
>
> On Thu, Jul 30, 2009 at 3:54 PM, Nicolae Mihalache <xpromache@yahoo.com
> >wrote:
>
> > Hello,
> >
> > I'm a new user of solr but I have worked a bit with Lucene before. I get
> > some out of memory exception when optimizing the index through Solr and I
> > would like to find out why.
> > However, the only message I get on standard output is:
> > Jul 30, 2009 9:20:22 PM org.apache.solr.common.SolrException log
> > SEVERE: java.lang.OutOfMemoryError: Java heap space
> >
> > Is there a way to get a stack trace for this exception? I had a look into
> > the java.util.logging options and didn't find anything.
> >
> > My solr runs in some standard configuration inside jetty.
> > Any suggestion would be appreciated.
> >
> > Thanks,
> > nicolae
> >
> >
> >
> >
>
Re: How to get a stack trace
Posted by Bill Au <bi...@gmail.com>.
Your heap may be just too small or you may have a memory leak. A stack
trace may not help you since the thread encountered the OutOfMemoryError may
not be where the memory leak is. A heap dump will tell you what's using up
all the memory in your heap.
Bill
On Thu, Jul 30, 2009 at 3:54 PM, Nicolae Mihalache <xp...@yahoo.com>wrote:
> Hello,
>
> I'm a new user of solr but I have worked a bit with Lucene before. I get
> some out of memory exception when optimizing the index through Solr and I
> would like to find out why.
> However, the only message I get on standard output is:
> Jul 30, 2009 9:20:22 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
>
> Is there a way to get a stack trace for this exception? I had a look into
> the java.util.logging options and didn't find anything.
>
> My solr runs in some standard configuration inside jetty.
> Any suggestion would be appreciated.
>
> Thanks,
> nicolae
>
>
>
>
RE: How to get a stack trace
Posted by "Smiley, David W." <ds...@mitre.org>.
I much prefer "jstack" (which comes with the JDK) which is for exactly this purpose. I always forget the right number for using kill. When using jstack, the stack goes to the current terminal session, not Solr's output -- very convenient.
~ David Smiley
________________________________________
From: Nicolae Mihalache [xpromache@gmail.com]
Sent: Monday, August 03, 2009 5:21 AM
To: solr-user@lucene.apache.org
Subject: Re: How to get a stack trace
Thanks, I will try maybe together with the -XX:OnOutOfMemoryError="<cmd
args>; <cmd args>":
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
But now I know where the problem came from, so maybe next time.
On Mon, Aug 3, 2009 at 11:08 AM, Chantal Ackermann <
chantal.ackermann@btelligent.de> wrote:
> kill -3 [JVM-process-id] (SIGQUIT)
> sends a signal to the JVM to dump all thread stacks. It does NOT kill it.
> It's only for outputting the stacks. You can see whether there are any
> threads that lock up.
> This produces a LOT of output in the main logfile (e.g. catalina.out for
> Tomcat).
>
> (see for example
>
> http://www.unixville.com/~moazam/stories/2004/05/18/debuggingHangsInTheJvm.html<http://www.unixville.com/%7Emoazam/stories/2004/05/18/debuggingHangsInTheJvm.html>
> and others)
>
>
> Otis Gospodnetic schrieb:
>
>> Nicolae,
>>
>> You may be able to figure things out from the heap dump. You'll need to
>> start the JVM like this, for example:
>>
>> java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap .......
>>
>> Otis
>> --
>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>>
>>
>>
>> ----- Original Message ----
>>
>>> From: Nicolae Mihalache <xp...@yahoo.com>
>>> To: solr-user@lucene.apache.org
>>> Sent: Thursday, July 30, 2009 3:54:41 PM
>>> Subject: How to get a stack trace
>>>
>>> Hello,
>>>
>>> I'm a new user of solr but I have worked a bit with Lucene before. I get
>>> some
>>> out of memory exception when optimizing the index through Solr and I
>>> would like
>>> to find out why.
>>> However, the only message I get on standard output is:
>>> Jul 30, 2009 9:20:22 PM org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.OutOfMemoryError: Java heap space
>>>
>>> Is there a way to get a stack trace for this exception? I had a look into
>>> the
>>> java.util.logging options and didn't find anything.
>>>
>>> My solr runs in some standard configuration inside jetty.
>>> Any suggestion would be appreciated.
>>>
>>> Thanks,
>>> nicolae
>>>
>>
>>
> --
> Chantal Ackermann
> Consultant
>
> mobil +49 (176) 10 00 09 45
> email chantal.ackermann@btelligent.de
>
>
> --------------------------------------------------------------------------------------------------------
>
> b.telligent GmbH & Co. KG
> Lichtenbergstraße 8
> D-85748 Garching / München
>
> fon +49 (89) 54 84 25 60
> fax +49 (89) 54 84 25 69
> web www.btelligent.de
>
> Registered in Munich: HRA 84393
> Managing Director: b.telligent Verwaltungs GmbH, HRB 153164 represented by
> Sebastian Amtage and Klaus Blaschek
> USt.Id.-Nr. DE814054803
>
>
>
> Confidentiality Note
> This email is intended only for the use of the individual or entity to
> which it is addressed, and may contain information that is privileged,
> confidential and exempt from disclosure under applicable law. If the reader
> of this email message is not the intended recipient, or the employee or
> agent responsible for delivery of the message to the intended recipient, you
> are hereby notified that any dissemination, distribution or copying of this
> communication is prohibited. If you have received this email in error,
> please notify us immediately by telephone at +49 (0) 89 54 84 25 60. Thank
> you.
>
Re: How to get a stack trace
Posted by Nicolae Mihalache <xp...@gmail.com>.
Thanks, I will try maybe together with the -XX:OnOutOfMemoryError="<cmd
args>; <cmd args>":
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
But now I know where the problem came from, so maybe next time.
On Mon, Aug 3, 2009 at 11:08 AM, Chantal Ackermann <
chantal.ackermann@btelligent.de> wrote:
> kill -3 [JVM-process-id] (SIGQUIT)
> sends a signal to the JVM to dump all thread stacks. It does NOT kill it.
> It's only for outputting the stacks. You can see whether there are any
> threads that lock up.
> This produces a LOT of output in the main logfile (e.g. catalina.out for
> Tomcat).
>
> (see for example
>
> http://www.unixville.com/~moazam/stories/2004/05/18/debuggingHangsInTheJvm.html<http://www.unixville.com/%7Emoazam/stories/2004/05/18/debuggingHangsInTheJvm.html>
> and others)
>
>
> Otis Gospodnetic schrieb:
>
>> Nicolae,
>>
>> You may be able to figure things out from the heap dump. You'll need to
>> start the JVM like this, for example:
>>
>> java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap .......
>>
>> Otis
>> --
>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>>
>>
>>
>> ----- Original Message ----
>>
>>> From: Nicolae Mihalache <xp...@yahoo.com>
>>> To: solr-user@lucene.apache.org
>>> Sent: Thursday, July 30, 2009 3:54:41 PM
>>> Subject: How to get a stack trace
>>>
>>> Hello,
>>>
>>> I'm a new user of solr but I have worked a bit with Lucene before. I get
>>> some
>>> out of memory exception when optimizing the index through Solr and I
>>> would like
>>> to find out why.
>>> However, the only message I get on standard output is:
>>> Jul 30, 2009 9:20:22 PM org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.OutOfMemoryError: Java heap space
>>>
>>> Is there a way to get a stack trace for this exception? I had a look into
>>> the
>>> java.util.logging options and didn't find anything.
>>>
>>> My solr runs in some standard configuration inside jetty.
>>> Any suggestion would be appreciated.
>>>
>>> Thanks,
>>> nicolae
>>>
>>
>>
> --
> Chantal Ackermann
> Consultant
>
> mobil +49 (176) 10 00 09 45
> email chantal.ackermann@btelligent.de
>
>
> --------------------------------------------------------------------------------------------------------
>
> b.telligent GmbH & Co. KG
> Lichtenbergstraße 8
> D-85748 Garching / München
>
> fon +49 (89) 54 84 25 60
> fax +49 (89) 54 84 25 69
> web www.btelligent.de
>
> Registered in Munich: HRA 84393
> Managing Director: b.telligent Verwaltungs GmbH, HRB 153164 represented by
> Sebastian Amtage and Klaus Blaschek
> USt.Id.-Nr. DE814054803
>
>
>
> Confidentiality Note
> This email is intended only for the use of the individual or entity to
> which it is addressed, and may contain information that is privileged,
> confidential and exempt from disclosure under applicable law. If the reader
> of this email message is not the intended recipient, or the employee or
> agent responsible for delivery of the message to the intended recipient, you
> are hereby notified that any dissemination, distribution or copying of this
> communication is prohibited. If you have received this email in error,
> please notify us immediately by telephone at +49 (0) 89 54 84 25 60. Thank
> you.
>
Re: How to get a stack trace
Posted by Chantal Ackermann <ch...@btelligent.de>.
kill -3 [JVM-process-id] (SIGQUIT)
sends a signal to the JVM to dump all thread stacks. It does NOT kill
it. It's only for outputting the stacks. You can see whether there are
any threads that lock up.
This produces a LOT of output in the main logfile (e.g. catalina.out for
Tomcat).
(see for example
http://www.unixville.com/~moazam/stories/2004/05/18/debuggingHangsInTheJvm.html
and others)
Otis Gospodnetic schrieb:
> Nicolae,
>
> You may be able to figure things out from the heap dump. You'll need to start the JVM like this, for example:
>
> java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap .......
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
>
> ----- Original Message ----
>> From: Nicolae Mihalache <xp...@yahoo.com>
>> To: solr-user@lucene.apache.org
>> Sent: Thursday, July 30, 2009 3:54:41 PM
>> Subject: How to get a stack trace
>>
>> Hello,
>>
>> I'm a new user of solr but I have worked a bit with Lucene before. I get some
>> out of memory exception when optimizing the index through Solr and I would like
>> to find out why.
>> However, the only message I get on standard output is:
>> Jul 30, 2009 9:20:22 PM org.apache.solr.common.SolrException log
>> SEVERE: java.lang.OutOfMemoryError: Java heap space
>>
>> Is there a way to get a stack trace for this exception? I had a look into the
>> java.util.logging options and didn't find anything.
>>
>> My solr runs in some standard configuration inside jetty.
>> Any suggestion would be appreciated.
>>
>> Thanks,
>> nicolae
>
--
Chantal Ackermann
Consultant
mobil +49 (176) 10 00 09 45
email chantal.ackermann@btelligent.de
--------------------------------------------------------------------------------------------------------
b.telligent GmbH & Co. KG
Lichtenbergstraße 8
D-85748 Garching / München
fon +49 (89) 54 84 25 60
fax +49 (89) 54 84 25 69
web www.btelligent.de
Registered in Munich: HRA 84393
Managing Director: b.telligent Verwaltungs GmbH, HRB 153164 represented
by Sebastian Amtage and Klaus Blaschek
USt.Id.-Nr. DE814054803
Confidentiality Note
This email is intended only for the use of the individual or entity to
which it is addressed, and may contain information that is privileged,
confidential and exempt from disclosure under applicable law. If the
reader of this email message is not the intended recipient, or the
employee or agent responsible for delivery of the message to the
intended recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is prohibited. If you have
received this email in error, please notify us immediately by telephone
at +49 (0) 89 54 84 25 60. Thank you.
Re: How to get a stack trace
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Nicolae,
You may be able to figure things out from the heap dump. You'll need to start the JVM like this, for example:
java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap .......
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
----- Original Message ----
> From: Nicolae Mihalache <xp...@yahoo.com>
> To: solr-user@lucene.apache.org
> Sent: Thursday, July 30, 2009 3:54:41 PM
> Subject: How to get a stack trace
>
> Hello,
>
> I'm a new user of solr but I have worked a bit with Lucene before. I get some
> out of memory exception when optimizing the index through Solr and I would like
> to find out why.
> However, the only message I get on standard output is:
> Jul 30, 2009 9:20:22 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
>
> Is there a way to get a stack trace for this exception? I had a look into the
> java.util.logging options and didn't find anything.
>
> My solr runs in some standard configuration inside jetty.
> Any suggestion would be appreciated.
>
> Thanks,
> nicolae