You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Roxana Angheluta <an...@yahoo.com> on 2010/11/01 14:30:46 UTC
Re: solr stuck in writing to inexisting sockets
Hi,
Yes, sometimes it takes >5 minutes for a query. I agree this is not desirable. However, if the application has no control over the input queries other that closing the socket after a while, solr should not continue writing the response, but terminate the thread.
In general, is there a way to quantify the complexity of a given query on a certain index? Some general guidelines which can be used by non-technical people?
Thanks a lot,
roxana
--- On Sun, 10/31/10, Erick Erickson <er...@gmail.com> wrote:
> From: Erick Erickson <er...@gmail.com>
> Subject: Re: solr stuck in writing to inexisting sockets
> To: solr-user@lucene.apache.org
> Date: Sunday, October 31, 2010, 2:29 AM
> Are you saying that your Solr server
> is at times taking 5 minutes to
> complete? If so,
> I'd get to the bottom of that first off. My first guess
> would be you're
> either hitting
> memory issues and swapping horribly or..well, that would be
> my first guess.
>
> Best
> Erick
>
> On Thu, Oct 28, 2010 at 5:23 AM, Roxana Angheluta <an...@yahoo.com>wrote:
>
> > Hi all,
> >
> > We are using Solr over Jetty with a large index,
> sharded and distributed
> > over multiple machines. Our queries are quite long,
> involving boolean and
> > proximity operators. We cut the connection at the
> client side after 5
> > minutes. Also, we are using parameter timeAllowed to
> stop executing it on
> > the server after a while.
> > We quite often run into situations when solr "blocks".
> The load on the
> > server increases and a thread dump on the solr process
> shows many threads
> > like below:
> >
> >
> > "btpool0-49" prio=10 tid=0x00007f73afe1d000 nid=0x3581
> runnable
> > [0x00000000451a0000]
> > java.lang.Thread.State: RUNNABLE
> > at
> java.io.PrintWriter.write(PrintWriter.java:362)
> > at
> org.apache.solr.common.util.XML.escape(XML.java:206)
> > at
> org.apache.solr.common.util.XML.escapeCharData(XML.java:79)
> > at
> org.apache.solr.request.XMLWriter.writePrim(XMLWriter.java:832)
> > at
> org.apache.solr.request.XMLWriter.writeStr(XMLWriter.java:684)
> > at
> org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:564)
> > at
> org.apache.solr.request.XMLWriter.writeDoc(XMLWriter.java:435)
> > at
> org.apache.solr.request.XMLWriter$2.writeDocs(XMLWriter.java:514)
> > at
> >
> org.apache.solr.request.XMLWriter.writeDocuments(XMLWriter.java:485)
> > at
> >
> org.apache.solr.request.XMLWriter.writeSolrDocumentList(XMLWriter.java:494)
> > at
> org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:588)
> > at
> >
> org.apache.solr.request.XMLWriter.writeResponse(XMLWriter.java:130)
> > at
> >
> org.apache.solr.request.XMLResponseWriter.write(XMLResponseWriter.java:34)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:325)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
> > at
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
> > at
> >
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
> > at
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> > at
> >
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> > at
> >
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
> > ......................................
> >
> >
> > A netstat on the machine shows sockets in state
> CLOSE_WAIT. However, they
> > are fewer than the number of RUNNABLE threads as the
> above.
> >
> > Why is this happening? Is there anything we can do to
> avoid getting in
> > these situations?
> >
> > Thanks,
> > roxana
> >
> >
> >
> >
>
Re: solr stuck in writing to inexisting sockets
Posted by Lance Norskog <go...@gmail.com>.
> Besides, I don't know how you'd stop Solr processing a query mid-way
> through,
> I don't know of any way to make that happen.
The timeAllowed parameter causes a timeout in the Solr server to kill
the searching thread. They uses that now.
But, yes, Erick is right- there is a fundamental problem you should
solve. Since they are all stuck in returning XML results, there is
something wrong in reading back results.
It is possible that there is a bug in timeAllowed, where the
kill-this-thread hits while returning the results and the handler for
this does not work correctly when returning results. It would be great
if someone wrote a unit test for this (not me) and posted it.
On Mon, Nov 1, 2010 at 8:44 AM, Erick Erickson <er...@gmail.com> wrote:
> I'm going to nudge you in the direction of understanding why the queries
> take so long in the first place rather than going toward the blunt approach
> of cutting them off after some time. The fact that you don't control the
> queries submitted doesn't prevent you from trying to understand what
> is taking so long.
>
> The first thing I'd look for is whether the system is memory starved. What
> JVM are you using and what memory parameters are you giving it? What
> version of Solr are you using? Have you tried any performance monitoring
> to determine what is happening?
>
> The reason I'm pushing in this direction is that 5 minute searches are
> pathological. Once you're up in that range, virtually any fix you come up
> with will simply mask the underlying problems, and you'll be forever
> chasing the next manifestation of the underlying problem.
>
> Besides, I don't know how you'd stop Solr processing a query mid-way
> through,
> I don't know of any way to make that happen.
>
> Best
> Erick
>
> On Mon, Nov 1, 2010 at 9:30 AM, Roxana Angheluta <an...@yahoo.com>wrote:
>
>> Hi,
>>
>> Yes, sometimes it takes >5 minutes for a query. I agree this is not
>> desirable. However, if the application has no control over the input queries
>> other that closing the socket after a while, solr should not continue
>> writing the response, but terminate the thread.
>>
>> In general, is there a way to quantify the complexity of a given query on a
>> certain index? Some general guidelines which can be used by non-technical
>> people?
>>
>> Thanks a lot,
>> roxana
>>
>> --- On Sun, 10/31/10, Erick Erickson <er...@gmail.com> wrote:
>>
>> > From: Erick Erickson <er...@gmail.com>
>> > Subject: Re: solr stuck in writing to inexisting sockets
>> > To: solr-user@lucene.apache.org
>> > Date: Sunday, October 31, 2010, 2:29 AM
>> > Are you saying that your Solr server
>> > is at times taking 5 minutes to
>> > complete? If so,
>> > I'd get to the bottom of that first off. My first guess
>> > would be you're
>> > either hitting
>> > memory issues and swapping horribly or..well, that would be
>> > my first guess.
>> >
>> > Best
>> > Erick
>> >
>> > On Thu, Oct 28, 2010 at 5:23 AM, Roxana Angheluta <anghelutar@yahoo.com
>> >wrote:
>> >
>> > > Hi all,
>> > >
>> > > We are using Solr over Jetty with a large index,
>> > sharded and distributed
>> > > over multiple machines. Our queries are quite long,
>> > involving boolean and
>> > > proximity operators. We cut the connection at the
>> > client side after 5
>> > > minutes. Also, we are using parameter timeAllowed to
>> > stop executing it on
>> > > the server after a while.
>> > > We quite often run into situations when solr "blocks".
>> > The load on the
>> > > server increases and a thread dump on the solr process
>> > shows many threads
>> > > like below:
>> > >
>> > >
>> > > "btpool0-49" prio=10 tid=0x00007f73afe1d000 nid=0x3581
>> > runnable
>> > > [0x00000000451a0000]
>> > > java.lang.Thread.State: RUNNABLE
>> > > at
>> > java.io.PrintWriter.write(PrintWriter.java:362)
>> > > at
>> > org.apache.solr.common.util.XML.escape(XML.java:206)
>> > > at
>> > org.apache.solr.common.util.XML.escapeCharData(XML.java:79)
>> > > at
>> > org.apache.solr.request.XMLWriter.writePrim(XMLWriter.java:832)
>> > > at
>> > org.apache.solr.request.XMLWriter.writeStr(XMLWriter.java:684)
>> > > at
>> > org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:564)
>> > > at
>> > org.apache.solr.request.XMLWriter.writeDoc(XMLWriter.java:435)
>> > > at
>> > org.apache.solr.request.XMLWriter$2.writeDocs(XMLWriter.java:514)
>> > > at
>> > >
>> > org.apache.solr.request.XMLWriter.writeDocuments(XMLWriter.java:485)
>> > > at
>> > >
>> >
>> org.apache.solr.request.XMLWriter.writeSolrDocumentList(XMLWriter.java:494)
>> > > at
>> > org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:588)
>> > > at
>> > >
>> > org.apache.solr.request.XMLWriter.writeResponse(XMLWriter.java:130)
>> > > at
>> > >
>> >
>> org.apache.solr.request.XMLResponseWriter.write(XMLResponseWriter.java:34)
>> > > at
>> > >
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:325)
>> > > at
>> > >
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
>> > > at
>> > >
>> >
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
>> > > at
>> > >
>> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
>> > > at
>> > >
>> >
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> > > at
>> > >
>> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> > > at
>> > >
>> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
>> > > ......................................
>> > >
>> > >
>> > > A netstat on the machine shows sockets in state
>> > CLOSE_WAIT. However, they
>> > > are fewer than the number of RUNNABLE threads as the
>> > above.
>> > >
>> > > Why is this happening? Is there anything we can do to
>> > avoid getting in
>> > > these situations?
>> > >
>> > > Thanks,
>> > > roxana
>> > >
>> > >
>> > >
>> > >
>> >
>>
>>
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: solr stuck in writing to inexisting sockets
Posted by Erick Erickson <er...@gmail.com>.
I'm going to nudge you in the direction of understanding why the queries
take so long in the first place rather than going toward the blunt approach
of cutting them off after some time. The fact that you don't control the
queries submitted doesn't prevent you from trying to understand what
is taking so long.
The first thing I'd look for is whether the system is memory starved. What
JVM are you using and what memory parameters are you giving it? What
version of Solr are you using? Have you tried any performance monitoring
to determine what is happening?
The reason I'm pushing in this direction is that 5 minute searches are
pathological. Once you're up in that range, virtually any fix you come up
with will simply mask the underlying problems, and you'll be forever
chasing the next manifestation of the underlying problem.
Besides, I don't know how you'd stop Solr processing a query mid-way
through,
I don't know of any way to make that happen.
Best
Erick
On Mon, Nov 1, 2010 at 9:30 AM, Roxana Angheluta <an...@yahoo.com>wrote:
> Hi,
>
> Yes, sometimes it takes >5 minutes for a query. I agree this is not
> desirable. However, if the application has no control over the input queries
> other that closing the socket after a while, solr should not continue
> writing the response, but terminate the thread.
>
> In general, is there a way to quantify the complexity of a given query on a
> certain index? Some general guidelines which can be used by non-technical
> people?
>
> Thanks a lot,
> roxana
>
> --- On Sun, 10/31/10, Erick Erickson <er...@gmail.com> wrote:
>
> > From: Erick Erickson <er...@gmail.com>
> > Subject: Re: solr stuck in writing to inexisting sockets
> > To: solr-user@lucene.apache.org
> > Date: Sunday, October 31, 2010, 2:29 AM
> > Are you saying that your Solr server
> > is at times taking 5 minutes to
> > complete? If so,
> > I'd get to the bottom of that first off. My first guess
> > would be you're
> > either hitting
> > memory issues and swapping horribly or..well, that would be
> > my first guess.
> >
> > Best
> > Erick
> >
> > On Thu, Oct 28, 2010 at 5:23 AM, Roxana Angheluta <anghelutar@yahoo.com
> >wrote:
> >
> > > Hi all,
> > >
> > > We are using Solr over Jetty with a large index,
> > sharded and distributed
> > > over multiple machines. Our queries are quite long,
> > involving boolean and
> > > proximity operators. We cut the connection at the
> > client side after 5
> > > minutes. Also, we are using parameter timeAllowed to
> > stop executing it on
> > > the server after a while.
> > > We quite often run into situations when solr "blocks".
> > The load on the
> > > server increases and a thread dump on the solr process
> > shows many threads
> > > like below:
> > >
> > >
> > > "btpool0-49" prio=10 tid=0x00007f73afe1d000 nid=0x3581
> > runnable
> > > [0x00000000451a0000]
> > > java.lang.Thread.State: RUNNABLE
> > > at
> > java.io.PrintWriter.write(PrintWriter.java:362)
> > > at
> > org.apache.solr.common.util.XML.escape(XML.java:206)
> > > at
> > org.apache.solr.common.util.XML.escapeCharData(XML.java:79)
> > > at
> > org.apache.solr.request.XMLWriter.writePrim(XMLWriter.java:832)
> > > at
> > org.apache.solr.request.XMLWriter.writeStr(XMLWriter.java:684)
> > > at
> > org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:564)
> > > at
> > org.apache.solr.request.XMLWriter.writeDoc(XMLWriter.java:435)
> > > at
> > org.apache.solr.request.XMLWriter$2.writeDocs(XMLWriter.java:514)
> > > at
> > >
> > org.apache.solr.request.XMLWriter.writeDocuments(XMLWriter.java:485)
> > > at
> > >
> >
> org.apache.solr.request.XMLWriter.writeSolrDocumentList(XMLWriter.java:494)
> > > at
> > org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:588)
> > > at
> > >
> > org.apache.solr.request.XMLWriter.writeResponse(XMLWriter.java:130)
> > > at
> > >
> >
> org.apache.solr.request.XMLResponseWriter.write(XMLResponseWriter.java:34)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:325)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
> > > at
> > >
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
> > > at
> > >
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
> > > at
> > >
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> > > at
> > >
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> > > at
> > >
> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
> > > ......................................
> > >
> > >
> > > A netstat on the machine shows sockets in state
> > CLOSE_WAIT. However, they
> > > are fewer than the number of RUNNABLE threads as the
> > above.
> > >
> > > Why is this happening? Is there anything we can do to
> > avoid getting in
> > > these situations?
> > >
> > > Thanks,
> > > roxana
> > >
> > >
> > >
> > >
> >
>
>
>
>