You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by John Russell <jj...@gmail.com> on 2010/12/06 23:10:46 UTC

high CPU usage and SelectCannelConnector threads used a lot

Hi,
I'm using solr and have been load testing it for around 4 days.  We use the
solrj client to communicate with a separate jetty based solr process on the
same box.

After a few days solr's CPU% is now consistently at or above 100% (multiple
processors available) and the application using it is mostly not responding
because it times out talking to solr. I connected visual VM to the solr JVM
and found that out of the many btpool-# threads there are 4 that are pretty
much stuck in the running state 100% of the time. Their names are

btpool0-1-Acceptor1 SelectChannelConnector @0.0.0.0:9983
btpool0-2-Acceptor2 SelectChannelConnector @0.0.0.0:9983
btpool0-3-Acceptor3 SelectChannelConnector @0.0.0.0:9983
btpool0-9-Acceptor0 SelectChannelConnector @0.0.0.0:9983



The stacks are all the same

    "btpool0-2 - Acceptor2 SelectChannelConnector @ 0.0.0.0:9983" - Thread
t@27
    java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <106a644> (a sun.nio.ch.Util$1)
        - locked <18dd381> (a java.util.Collections$UnmodifiableSet)
        - locked <38d07d> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:419)
        at
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:169)
        at
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:516)
        at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

       Locked ownable synchronizers:
        - None

All of the other idle thread pool threads are just waiting for new tasks.
The active threads never seem to change, its always these 4.  The selector
channel appears to be in the jetty code, receiving requests from our other
process through the solrj client.

Does anyone know what this might mean or how to address it? Are these
running all the time because they are blocked on IO so not actually
consuming CPU? If so, what else might be? Is there a better way to figure
out what is pinning the CPU?

Some more info that might be useful.

32 bit machine ( I know, I know)
2.7GB of RAM for solr process ~2.5 is "used"
According to visual VM around 25% of CPU time is spent in GC with the rest
in application.

Thanks for the help.

John

Re: high CPU usage and SelectCannelConnector threads used a lot

Posted by Kent Fitch <ke...@gmail.com>.
Hi John,

sounds like this bug in NIO:

http://jira.codehaus.org/browse/JETTY-937

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6403933

I think recent versions of jetty work around this bug, or maybe try
the non-NIO socket connector

Kent

On Tue, Dec 7, 2010 at 9:10 AM, John Russell <jj...@gmail.com> wrote:
> Hi,
> I'm using solr and have been load testing it for around 4 days.  We use the
> solrj client to communicate with a separate jetty based solr process on the
> same box.
>
> After a few days solr's CPU% is now consistently at or above 100% (multiple
> processors available) and the application using it is mostly not responding
> because it times out talking to solr. I connected visual VM to the solr JVM
> and found that out of the many btpool-# threads there are 4 that are pretty
> much stuck in the running state 100% of the time. Their names are
>
> btpool0-1-Acceptor1 SelectChannelConnector @0.0.0.0:9983
> btpool0-2-Acceptor2 SelectChannelConnector @0.0.0.0:9983
> btpool0-3-Acceptor3 SelectChannelConnector @0.0.0.0:9983
> btpool0-9-Acceptor0 SelectChannelConnector @0.0.0.0:9983
>
>
>
> The stacks are all the same
>
>    "btpool0-2 - Acceptor2 SelectChannelConnector @ 0.0.0.0:9983" - Thread
> t@27
>    java.lang.Thread.State: RUNNABLE
>        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>        - locked <106a644> (a sun.nio.ch.Util$1)
>        - locked <18dd381> (a java.util.Collections$UnmodifiableSet)
>        - locked <38d07d> (a sun.nio.ch.EPollSelectorImpl)
>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>        at
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:419)
>        at
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:169)
>        at
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>        at
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:516)
>        at
> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>
>       Locked ownable synchronizers:
>        - None
>
> All of the other idle thread pool threads are just waiting for new tasks.
> The active threads never seem to change, its always these 4.  The selector
> channel appears to be in the jetty code, receiving requests from our other
> process through the solrj client.
>
> Does anyone know what this might mean or how to address it? Are these
> running all the time because they are blocked on IO so not actually
> consuming CPU? If so, what else might be? Is there a better way to figure
> out what is pinning the CPU?
>
> Some more info that might be useful.
>
> 32 bit machine ( I know, I know)
> 2.7GB of RAM for solr process ~2.5 is "used"
> According to visual VM around 25% of CPU time is spent in GC with the rest
> in application.
>
> Thanks for the help.
>
> John
>