solr/tomcat stops responding

[Kevin Lewandowski <ke...@gmail.com>]

My solr installation has been running fine for a few weeks but now
after a server reboot it starts and runs for a few seconds, then stops
responding. I don't see any errors in the logfiles, apart from
snapinstaller not being able to issue a commit. Also, the process is
using 100% cpu and stops responding to http requests (admin interface
and queries).

Is there anything in solr that would cause this? It also seems like it
could be a tomcat problem but I wanted to ask here first and see if
anyone has encountered this.

thanks,
Kevin

Re: solr/tomcat stops responding

[roz dev <ro...@gmail.com>]

You are referring to a very old thread

Did you take any heap dump and thread dumo?  They can help you get more
insight.

-Saroj


On Tue, Jul 31, 2012 at 9:04 AM, Suneel <pa...@gmail.com> wrote:

> Hello Kevin,
>
> I am also facing same problem After few hours or  few day my solr server
> getting crash.
> I try  to download following patch but its not accessible now. i am using
> 3.1 version of solr.
>
> http://people.apache.org/~yonik/solr/current/solr.war
>
>
>
> -----
> Regards,
>
> Suneel Pandey
> Sr. Software Developer
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/solr-tomcat-stops-responding-tp474577p3998435.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: solr/tomcat stops responding

[Suneel <pa...@gmail.com>]

Hello Kevin,

I am also facing same problem After few hours or  few day my solr server
getting crash. 
I try  to download following patch but its not accessible now. i am using
3.1 version of solr.

http://people.apache.org/~yonik/solr/current/solr.war



-----
Regards,

Suneel Pandey
Sr. Software Developer
--
View this message in context: http://lucene.472066.n3.nabble.com/solr-tomcat-stops-responding-tp474577p3998435.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr/tomcat stops responding

[Kevin Lewandowski <ke...@gmail.com>]

> OK, this may fix it:
> https://issues.apache.org/jira/browse/SOLR-77
>
> A war with this patch included is here:
> http://people.apache.org/~yonik/solr/current/solr.war
>
> You also need to configure some queries to be done on the firstSearcher event
> in solrconfig.xml.  Uncomment and customize the example one included.

Yonik, thanks very much for that patch! I've been running it since
yesterday and it looks like it's solved the problem.

thanks,
Kevin

Re: solr/tomcat stops responding

[Yonik Seeley <yo...@apache.org>]

On 12/2/06, Kevin Lewandowski <ke...@gmail.com> wrote:
> I've done a temporary fix for this by running a proxy in front of
> tomcat. Then I updated my system startup to start solr, wait 20
> seconds, run a few queries, wait 20 seconds, then start the proxy.
> This is working fine now. But I'd still like to fix the real problem.

OK, this may fix it:
https://issues.apache.org/jira/browse/SOLR-77

A war with this patch included is here:
http://people.apache.org/~yonik/solr/current/solr.war

You also need to configure some queries to be done on the firstSearcher event
in solrconfig.xml.  Uncomment and customize the example one included.

A single query should do, just make sure to include a sort that
includes *all* of the fields that are commonly sorted on (the new
lucene build with the sort fix hasn't been included yet).  If you use
any filters (fq param) or faceted browsing commands, then include
those too.

This will cause all searchers to block and wait for initial warming to
be completed.
If the load balancer is sending traffic during this time, these
requests will pile up for a short time, but should complete OK after
that.

-Yonik

Re: solr/tomcat stops responding

[Yonik Seeley <yo...@apache.org>]

On 12/3/06, Kevin Lewandowski <ke...@gmail.com> wrote:
> Okay, I figured out how to get the thread dump. It was in the tomcat
> logfile. I'm attaching it here.

Nothing sticks out to me as problematic, except for the 150 concurrent
requests in progress!
Perhaps the real problem already passed, and the result is the pileup
of all these requests which thrash eachother and have a hard time
making progress. Also throw the memory consumed by all these threads
might cause GC to churn, also lowering the amount of CPU available to
complete them.

Hopefully the patch I just provided, along with warming, will prevent
getting into this situation.

-Yonik

Re: solr/tomcat stops responding

[Kevin Lewandowski <ke...@gmail.com>]

> Hmmm, on most Linux/UNIX systems, sending the QUIT signal does nothing
> else but generate a stack trace to the console or a log file.  If you
> don't start tomcat by hand, the stack trace may go somewhere else I
> suppose.  This would be useful to learn how to do on your particular
> system (and we should add it to a debugging/troubleshooting wiki too).

Okay, I figured out how to get the thread dump. It was in the tomcat
logfile. I'm attaching it here.


> Are you load-balancing at all, or is this your only search server?
> FYI, I'm looking into something that will help.

I'm load balancing two solr servers.

thanks,
Kevin

Re: solr/tomcat stops responding

[Yonik Seeley <yo...@apache.org>]

On 12/2/06, Kevin Lewandowski <ke...@gmail.com> wrote:
> > accept connections for 3 or 4 hours ... did you try taking some thread
> > dumps like yonik suggested to see what all the threads were doing?
>
> A kill -3 will not kill the process. It does nothing and there's no
> thread dump on the console. kill -9 does kill it though.

Hmmm, on most Linux/UNIX systems, sending the QUIT signal does nothing
else but generate a stack trace to the console or a log file.  If you
don't start tomcat by hand, the stack trace may go somewhere else I
suppose.  This would be useful to learn how to do on your particular
system (and we should add it to a debugging/troubleshooting wiki too).

> btw, this has been a bigger problem for me because there's a separate
> hardware issue and the system freezes about every 12 hours. So I have
> to reboot it. After that I noticed solr not responding.

Are you load-balancing at all, or is this your only search server?
FYI, I'm looking into something that will help.

-Yonik

Re: solr/tomcat stops responding

[Kevin Lewandowski <ke...@gmail.com>]

> accept connections for 3 or 4 hours ... did you try taking some thread
> dumps like yonik suggested to see what all the threads were doing?

A kill -3 will not kill the process. It does nothing and there's no
thread dump on the console. kill -9 does kill it though.

btw, this has been a bigger problem for me because there's a separate
hardware issue and the system freezes about every 12 hours. So I have
to reboot it. After that I noticed solr not responding.

I've done a temporary fix for this by running a proxy in front of
tomcat. Then I updated my system startup to start solr, wait 20
seconds, run a few queries, wait 20 seconds, then start the proxy.
This is working fine now. But I'd still like to fix the real problem.
Let me know if there's anything else I can test or information I can
provide.

thanks,
Kevin

Re: solr/tomcat stops responding

[Chris Hostetter <ho...@fucit.org>]

: Okay, I did that. Shut off traffic to the server, restarted solr, ran
: a few queries against it, then turned traffic back on, and it's
: running fine now. So maybe the initial flood of requests has something
: to do with it?

an initial flood of requests to a completley cold index can definitely
hang while the index gets loaded, pages get swapped in, and FieldCaches
get constructed ... but i'm really amazed that it just flat out wouldn't
accept connections for 3 or 4 hours ... did you try taking some thread
dumps like yonik suggested to see what all the threads were doing?



-Hoss

Re: solr/tomcat stops responding

[Kevin Lewandowski <ke...@gmail.com>]

> > My solr installation has been running fine for a few weeks but now
> > after a server reboot it starts and runs for a few seconds, then stops
> > responding. I don't see any errors in the logfiles, apart from
> > snapinstaller not being able to issue a commit. Also, the process is
> > using 100% cpu and stops responding to http requests (admin interface
> > and queries).

Okay, some more happened after I sent this email. About 3 hours after
the reboot solr started running normally again. Then I rebooted it to
see if I could reproduce it. This time solr remained in the
not-responding state for about 4 hours but I did not wait longer to
see if it would come back.


> - check what got changed after the server reboot... anything?

Nothing had been changed on the server.


> Part of the fix for this has recently been committed into Lucene
> (multiple threads won't generate the same FieldCache entry).

Has that been added to solr yet? I'm running solr-2006-11-20.


> To see if this is your problem, restart the server and make sure no
> traffic goes to it.
> Then run some queries of the same type that will be hitting it to warm
> it up, then turn on normal traffic.

Okay, I did that. Shut off traffic to the server, restarted solr, ran
a few queries against it, then turned traffic back on, and it's
running fine now. So maybe the initial flood of requests has something
to do with it?

thanks,
Kevin

Re: solr/tomcat stops responding

[Yonik Seeley <yo...@apache.org>]

On 12/1/06, Kevin Lewandowski <ke...@gmail.com> wrote:
> My solr installation has been running fine for a few weeks but now
> after a server reboot it starts and runs for a few seconds, then stops
> responding. I don't see any errors in the logfiles, apart from
> snapinstaller not being able to issue a commit. Also, the process is
> using 100% cpu and stops responding to http requests (admin interface
> and queries).

- check what got changed after the server reboot... anything?
- turn on GC logging to see if you are running out of memory and the
GC system is taking up the CPU:  -Xloggc:gc.log
- get a JVM thread dump via kill -3 (assuming Linux/Unix)
- try rebuilding the index to see if it's something funky there

> Is there anything in solr that would cause this?

Yes, if it's normally a heavily loaded server that was restarted,
there can be a problem
of too many search requests hitting it all at once.  Many of the
requests may want to sort by the same field, and all of them start
generating the same fieldcache entry or filter entries at the same
time.  This causes a lot of memory to quickly be allocated, GC starts
churning trying to free up memory, more threads pile up, etc...

Part of the fix for this has recently been committed into Lucene
(multiple threads won't generate the same FieldCache entry).  We could
also do more in Solr by not releasing the first searcher until it has
been warmed by the static solrconfig.xml entries.

To see if this is your problem, restart the server and make sure no
traffic goes to it.
Then run some queries of the same type that will be hitting it to warm
it up, then turn on normal traffic.

-Yonik