You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chris Cowan <Ch...@plus3network.com> on 2011/06/01 21:04:08 UTC
Debugging a Solr/Jetty Hung Process
About once a day a Solr/Jetty process gets hung on my server consuming 100% of one of the CPU's. Once this happens the server no longer responds to requests. I've looked through the logs to try and see if anything stands out but so far I've found nothing out of the ordinary.
My current remedy is to log in and just kill the single processes that's hung. Once that happens everything goes back to normal and I'm good for a day or so. I'm currently the running following:
solr-jetty-1.4.0+ds1-1ubuntu1
which is comprised of
Solr 1.4.0
Jetty 6.1.22
on Unbuntu 10.10
I'm pretty new to managing a Jetty/Solr instance so at this point I'm just looking for advice on how I should go about trouble shooting this problem.
Chris
Re: Debugging a Solr/Jetty Hung Process
Posted by Chris Cowan <Ch...@plus3network.com>.
Sorry ... I just found it. I will try that next time. I have a feeling it wont work since the server usually stops accepting connections.
Chris
On Jun 1, 2011, at 12:12 PM, Chris Cowan wrote:
> I'm pretty green... is that something I can do while the event is happening or is there something I need to configure to capture the dump ahead of time.
>
> I've tried to reproduce the problem by putting the server under load but that doesn't seem to be the issue.
>
> Chris
>
> On Jun 1, 2011, at 12:06 PM, Bill Au wrote:
>
>> Taking a thread dump will take you what's going.
>>
>> Bill
>>
>> On Wed, Jun 1, 2011 at 3:04 PM, Chris Cowan <Ch...@plus3network.com>wrote:
>>
>>> About once a day a Solr/Jetty process gets hung on my server consuming 100%
>>> of one of the CPU's. Once this happens the server no longer responds to
>>> requests. I've looked through the logs to try and see if anything stands out
>>> but so far I've found nothing out of the ordinary.
>>>
>>> My current remedy is to log in and just kill the single processes that's
>>> hung. Once that happens everything goes back to normal and I'm good for a
>>> day or so. I'm currently the running following:
>>>
>>> solr-jetty-1.4.0+ds1-1ubuntu1
>>>
>>> which is comprised of
>>>
>>> Solr 1.4.0
>>> Jetty 6.1.22
>>> on Unbuntu 10.10
>>>
>>> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just
>>> looking for advice on how I should go about trouble shooting this problem.
>>>
>>> Chris
>
Re: Debugging a Solr/Jetty Hung Process
Posted by Chris Cowan <Ch...@plus3network.com>.
I'm pretty green... is that something I can do while the event is happening or is there something I need to configure to capture the dump ahead of time.
I've tried to reproduce the problem by putting the server under load but that doesn't seem to be the issue.
Chris
On Jun 1, 2011, at 12:06 PM, Bill Au wrote:
> Taking a thread dump will take you what's going.
>
> Bill
>
> On Wed, Jun 1, 2011 at 3:04 PM, Chris Cowan <Ch...@plus3network.com>wrote:
>
>> About once a day a Solr/Jetty process gets hung on my server consuming 100%
>> of one of the CPU's. Once this happens the server no longer responds to
>> requests. I've looked through the logs to try and see if anything stands out
>> but so far I've found nothing out of the ordinary.
>>
>> My current remedy is to log in and just kill the single processes that's
>> hung. Once that happens everything goes back to normal and I'm good for a
>> day or so. I'm currently the running following:
>>
>> solr-jetty-1.4.0+ds1-1ubuntu1
>>
>> which is comprised of
>>
>> Solr 1.4.0
>> Jetty 6.1.22
>> on Unbuntu 10.10
>>
>> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just
>> looking for advice on how I should go about trouble shooting this problem.
>>
>> Chris
Re: Debugging a Solr/Jetty Hung Process
Posted by Bill Au <bi...@gmail.com>.
Taking a thread dump will take you what's going.
Bill
On Wed, Jun 1, 2011 at 3:04 PM, Chris Cowan <Ch...@plus3network.com>wrote:
> About once a day a Solr/Jetty process gets hung on my server consuming 100%
> of one of the CPU's. Once this happens the server no longer responds to
> requests. I've looked through the logs to try and see if anything stands out
> but so far I've found nothing out of the ordinary.
>
> My current remedy is to log in and just kill the single processes that's
> hung. Once that happens everything goes back to normal and I'm good for a
> day or so. I'm currently the running following:
>
> solr-jetty-1.4.0+ds1-1ubuntu1
>
> which is comprised of
>
> Solr 1.4.0
> Jetty 6.1.22
> on Unbuntu 10.10
>
> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just
> looking for advice on how I should go about trouble shooting this problem.
>
> Chris
Re: Debugging a Solr/Jetty Hung Process
Posted by Michael Sokolov <so...@ifactory.com>.
If you have an SNMP infrastructure available (nagios or similar) you
should be able to set up a polling monitor that will keep statistics on
the number of threads in your jvm and even allow you to inspect their
stacks remotely. You can set alarms so you will be notified if cpu
thread count or other metrics exceed a configurable threshold and then
take a look at the process before it goes off the deep end. It is a
fair amount of work to set this up, but really useful if you need to
support a critical system.
-Mike
On 6/1/2011 3:42 PM, Jonathan Rochkind wrote:
> First guess (and it really is just a guess) would be Java garbage
> collection taking over. There are some JVM parameters you can use to
> tune the GC process, especially if the machine is multi-core, making
> sure GC happens in a seperate thread is helpful.
>
> But figuring out exactly what's going on requires confusing JVM
> debugging of which I am no expert at either.
>
> On 6/1/2011 3:04 PM, Chris Cowan wrote:
>> About once a day a Solr/Jetty process gets hung on my server
>> consuming 100% of one of the CPU's. Once this happens the server no
>> longer responds to requests. I've looked through the logs to try and
>> see if anything stands out but so far I've found nothing out of the
>> ordinary.
>>
>> My current remedy is to log in and just kill the single processes
>> that's hung. Once that happens everything goes back to normal and I'm
>> good for a day or so. I'm currently the running following:
>>
>> solr-jetty-1.4.0+ds1-1ubuntu1
>>
>> which is comprised of
>>
>> Solr 1.4.0
>> Jetty 6.1.22
>> on Unbuntu 10.10
>>
>> I'm pretty new to managing a Jetty/Solr instance so at this point I'm
>> just looking for advice on how I should go about trouble shooting
>> this problem.
>>
>> Chris
Re: Debugging a Solr/Jetty Hung Process
Posted by Chris Cowan <Ch...@plus3network.com>.
OK... The fix I thought would fix it didn't fix it (which was to use the commitWithin feature). What I can gather from `ps` is that the thread has pages locked in memory. Currently I'm using native locking for Solr. Would switching to simple help alleviate this problem?
Chris
On Jun 4, 2011, at 2:48 PM, Chris Cowan wrote:
> I found this thread that looks similar to what's happening on my system. I think what happens is there are multiple commits happening at once from the clients and it's causing the same issue. I'm going to use the commitWithin argument to the updates to see if that fixes the problem. I will report back with any findings.
>
> Chris
>
> On Jun 1, 2011, at 12:42 PM, Jonathan Rochkind wrote:
>
>> First guess (and it really is just a guess) would be Java garbage
>> collection taking over. There are some JVM parameters you can use to
>> tune the GC process, especially if the machine is multi-core, making
>> sure GC happens in a seperate thread is helpful.
>>
>> But figuring out exactly what's going on requires confusing JVM
>> debugging of which I am no expert at either.
>>
>> On 6/1/2011 3:04 PM, Chris Cowan wrote:
>>> About once a day a Solr/Jetty process gets hung on my server consuming 100% of one of the CPU's. Once this happens the server no longer responds to requests. I've looked through the logs to try and see if anything stands out but so far I've found nothing out of the ordinary.
>>>
>>> My current remedy is to log in and just kill the single processes that's hung. Once that happens everything goes back to normal and I'm good for a day or so. I'm currently the running following:
>>>
>>> solr-jetty-1.4.0+ds1-1ubuntu1
>>>
>>> which is comprised of
>>>
>>> Solr 1.4.0
>>> Jetty 6.1.22
>>> on Unbuntu 10.10
>>>
>>> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just looking for advice on how I should go about trouble shooting this problem.
>>>
>>> Chris
>
Re: Debugging a Solr/Jetty Hung Process
Posted by Chris Cowan <Ch...@plus3network.com>.
oops...
http://search.lucidimagination.com/search/document/bf43af733e898424/busywait_hang_using_extracting_update_handler_on_trunk
Chris
On Jun 4, 2011, at 2:48 PM, Chris Cowan wrote:
> I found this thread that looks similar to what's happening on my system. I think what happens is there are multiple commits happening at once from the clients and it's causing the same issue. I'm going to use the commitWithin argument to the updates to see if that fixes the problem. I will report back with any findings.
>
> Chris
>
> On Jun 1, 2011, at 12:42 PM, Jonathan Rochkind wrote:
>
>> First guess (and it really is just a guess) would be Java garbage
>> collection taking over. There are some JVM parameters you can use to
>> tune the GC process, especially if the machine is multi-core, making
>> sure GC happens in a seperate thread is helpful.
>>
>> But figuring out exactly what's going on requires confusing JVM
>> debugging of which I am no expert at either.
>>
>> On 6/1/2011 3:04 PM, Chris Cowan wrote:
>>> About once a day a Solr/Jetty process gets hung on my server consuming 100% of one of the CPU's. Once this happens the server no longer responds to requests. I've looked through the logs to try and see if anything stands out but so far I've found nothing out of the ordinary.
>>>
>>> My current remedy is to log in and just kill the single processes that's hung. Once that happens everything goes back to normal and I'm good for a day or so. I'm currently the running following:
>>>
>>> solr-jetty-1.4.0+ds1-1ubuntu1
>>>
>>> which is comprised of
>>>
>>> Solr 1.4.0
>>> Jetty 6.1.22
>>> on Unbuntu 10.10
>>>
>>> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just looking for advice on how I should go about trouble shooting this problem.
>>>
>>> Chris
>
Re: Debugging a Solr/Jetty Hung Process
Posted by Chris Cowan <Ch...@plus3network.com>.
I found this thread that looks similar to what's happening on my system. I think what happens is there are multiple commits happening at once from the clients and it's causing the same issue. I'm going to use the commitWithin argument to the updates to see if that fixes the problem. I will report back with any findings.
Chris
On Jun 1, 2011, at 12:42 PM, Jonathan Rochkind wrote:
> First guess (and it really is just a guess) would be Java garbage
> collection taking over. There are some JVM parameters you can use to
> tune the GC process, especially if the machine is multi-core, making
> sure GC happens in a seperate thread is helpful.
>
> But figuring out exactly what's going on requires confusing JVM
> debugging of which I am no expert at either.
>
> On 6/1/2011 3:04 PM, Chris Cowan wrote:
>> About once a day a Solr/Jetty process gets hung on my server consuming 100% of one of the CPU's. Once this happens the server no longer responds to requests. I've looked through the logs to try and see if anything stands out but so far I've found nothing out of the ordinary.
>>
>> My current remedy is to log in and just kill the single processes that's hung. Once that happens everything goes back to normal and I'm good for a day or so. I'm currently the running following:
>>
>> solr-jetty-1.4.0+ds1-1ubuntu1
>>
>> which is comprised of
>>
>> Solr 1.4.0
>> Jetty 6.1.22
>> on Unbuntu 10.10
>>
>> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just looking for advice on how I should go about trouble shooting this problem.
>>
>> Chris
Re: Debugging a Solr/Jetty Hung Process
Posted by Jonathan Rochkind <ro...@jhu.edu>.
First guess (and it really is just a guess) would be Java garbage
collection taking over. There are some JVM parameters you can use to
tune the GC process, especially if the machine is multi-core, making
sure GC happens in a seperate thread is helpful.
But figuring out exactly what's going on requires confusing JVM
debugging of which I am no expert at either.
On 6/1/2011 3:04 PM, Chris Cowan wrote:
> About once a day a Solr/Jetty process gets hung on my server consuming 100% of one of the CPU's. Once this happens the server no longer responds to requests. I've looked through the logs to try and see if anything stands out but so far I've found nothing out of the ordinary.
>
> My current remedy is to log in and just kill the single processes that's hung. Once that happens everything goes back to normal and I'm good for a day or so. I'm currently the running following:
>
> solr-jetty-1.4.0+ds1-1ubuntu1
>
> which is comprised of
>
> Solr 1.4.0
> Jetty 6.1.22
> on Unbuntu 10.10
>
> I'm pretty new to managing a Jetty/Solr instance so at this point I'm just looking for advice on how I should go about trouble shooting this problem.
>
> Chris