You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Burton-West, Tom" <tb...@umich.edu> on 2008/11/07 22:23:29 UTC
Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory
Hello,
We are testing Solr with a simulation of 30 concurrent users. We are
getting socket timeouts and the thread dump from the admin tool shows
about 100+ threads with a similar message about a lock. (Message
appended below).
We supsect this may have something to do with one or more phrase queries
containing common terms since our index is very large and we suspect one
or more very large segments of the position index need to be read into
memory.
Can someone point us to either the possible cause of this problem or
what we might change to reduce/eliminate it?
Tom
Tom Burton-West
Information Retrieval Programmer
Digital Library Production Services
University of Michigan Library
tburtonw@umich.edu
----------
'http-8080-Processor54' Id=71, BLOCKED on
lock=org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor@1352447
, total cpu time=2070.0000ms user time=1460.0000ms
at
org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
y.java:532)
at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
a:93)
at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
ava:34)
at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:57)
at
org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(SegmentTe
rmPositions.java:70)
at
org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermPos
itions.java:66)
at
org.apache.lucene.search.PhrasePositions.nextPosition(PhrasePositions.ja
va:76)
at
org.apache.lucene.search.ExactPhraseScorer.phraseFreq(ExactPhraseScorer.
java:45)
at org.apache.lucene.search.PhraseScorer.doNext(PhraseScorer.java:94)
at org.apache.lucene.search.PhraseScorer.next(PhraseScorer.java:81)
at org.apache.lucene.search.Scorer.score(Scorer.java:48)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
at org.apache.lucene.search.Searcher.search(Searcher.java:118)
at org.apache.lucene.search.Searcher.search(Searcher.java:97)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.
java:888)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.j
ava:805)
at
org.apache.solr.search.SolrIndexSearcher.getDocList(SolrIndexSearcher.ja
va:698)
at
org.apache.solr.request.StandardRequestHandler.handleRequestBody(Standar
dRequestHandler.java:122)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB
ase.java:77)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:658)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja
va:191)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j
ava:159)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:215)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:174)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548
)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:1
74)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:87
4)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.proc
essConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint
.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollow
erWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool
.java:689)
at java.lang.Thread.run(Thread.java:619)
Re: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory
Posted by Yonik Seeley <yo...@apache.org>.
On Tue, Nov 11, 2008 at 3:35 PM, Burton-West, Tom <tb...@umich.edu> wrote:
> Hi Yonik,
>
> Thanks for the NIO suggestion. We are using Linux, but our indexes are
> NFS mounted.
That may be your biggest performance problem right there - try on a
local index and see what the difference is.
> I thought I saw something about problems with NIO and NFS,
> but am fuzzy on the details.
>
> These results are with Solr 1.2 and I'm wondering if even without the
> NIO change, upgrading to Solr 1.3 might help.
You would need to try the latest nightly build to get the NIO stuff.
> What confuses me is why
> multiple searchers are locking the prx index file. I would think that
> searching is a read-only operation.
Reading a chunk of a file requires a seek() and a read()... that needs
to be synchronized so the read can execute before another seek()
happens in another thread.
NIO has a read(position, length) call and we can thus avoid synchronization.
-Yonik
Re: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Tom, have you tried copying your index to the local disk to see if that resolves the locking problem?
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
________________________________
From: "Burton-West, Tom" <tb...@umich.edu>
To: solr-user@lucene.apache.org
Cc: "Farber, Phillip" <pf...@umich.edu>
Sent: Tuesday, November 11, 2008 3:35:20 PM
Subject: RE: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory
Hi Yonik,
Thanks for the NIO suggestion. We are using Linux, but our indexes are
NFS mounted. I thought I saw something about problems with NIO and NFS,
but am fuzzy on the details.
These results are with Solr 1.2 and I'm wondering if even without the
NIO change, upgrading to Solr 1.3 might help. What confuses me is why
multiple searchers are locking the prx index file. I would think that
searching is a read-only operation.
Perhaps we need to change something to tell Solr we aren't updating the
index?
Tom
-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
Seeley
Sent: Friday, November 07, 2008 8:25 PM
To: solr-user@lucene.apache.org
Cc: Farber, Phillip
Subject: Re: Solr locking issue? BLOCKED on
lock=org.apache.lucene.store.FSDirectory
Hi Tom, if you're on a non Windows box, could you perhaps try your test
on the latest Solr nightly build? We've recently improved this through
the use of NIO.
-Yonik
On Fri, Nov 7, 2008 at 4:23 PM, Burton-West, Tom <tb...@umich.edu>
wrote:
> Hello,
>
> We are testing Solr with a simulation of 30 concurrent users. We are
> getting socket timeouts and the thread dump from the admin tool shows
> about 100+ threads with a similar message about a lock. (Message
> appended below).
>
> We supsect this may have something to do with one or more phrase
> queries containing common terms since our index is very large and we
> suspect one or more very large segments of the position index need to
> be read into memory.
>
> Can someone point us to either the possible cause of this problem or
> what we might change to reduce/eliminate it?
>
> Tom
>
> Tom Burton-West
> Information Retrieval Programmer
> Digital Library Production Services
> University of Michigan Library
> tburtonw@umich.edu
>
> ----------
>
> 'http-8080-Processor54' Id=71, BLOCKED on
> lock=org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor@13524
> 47 , total cpu time=2070.0000ms user time=1460.0000ms at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirect
> or
> y.java:532)
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.j
> av
> a:93)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput
> .j
> ava:34)
> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:57)
> at
> org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(Segment
> Te
> rmPositions.java:70)
> at
> org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermP
> os
> itions.java:66)
> at
> org.apache.lucene.search.PhrasePositions.nextPosition(PhrasePositions.
> ja
> va:76)
> at
>
org.apache.lucene.search.ExactPhraseScorer.phraseFreq(ExactPhraseScorer.
> java:45)
> at org.apache.lucene.search.PhraseScorer.doNext(PhraseScorer.java:94)
> at org.apache.lucene.search.PhraseScorer.next(PhraseScorer.java:81)
> at org.apache.lucene.search.Scorer.score(Scorer.java:48)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
> at org.apache.lucene.search.Searcher.search(Searcher.java:97)
> at
>
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.
> java:888)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher
> .j
> ava:805)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocList(SolrIndexSearcher.
> ja
> va:698)
> at
> org.apache.solr.request.StandardRequestHandler.handleRequestBody(Stand
> ar
> dRequestHandler.java:122)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle
> rB
> ase.java:77)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:658)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.
> ja
> va:191)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter
> .j
> ava:159)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> ca
> tionFilterChain.java:215)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
> lt
> erChain.java:188)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
> lv
> e.java:213)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
> lv
> e.java:174)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:5
> 48
> )
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
> va
> :127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
> va
> :117)
> at
>
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
> java:108)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
> :1
> 74)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
> 87
> 4)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.pr
> oc
> essConnection(Http11BaseProtocol.java:665)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoi
> nt
> .java:528)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll
> ow
> erWorkerThread.java:81)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
> ol
> .java:689)
> at java.lang.Thread.run(Thread.java:619)
>
>
RE: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory
Posted by "Burton-West, Tom" <tb...@umich.edu>.
Hi Yonik,
Thanks for the NIO suggestion. We are using Linux, but our indexes are
NFS mounted. I thought I saw something about problems with NIO and NFS,
but am fuzzy on the details.
These results are with Solr 1.2 and I'm wondering if even without the
NIO change, upgrading to Solr 1.3 might help. What confuses me is why
multiple searchers are locking the prx index file. I would think that
searching is a read-only operation.
Perhaps we need to change something to tell Solr we aren't updating the
index?
Tom
-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
Seeley
Sent: Friday, November 07, 2008 8:25 PM
To: solr-user@lucene.apache.org
Cc: Farber, Phillip
Subject: Re: Solr locking issue? BLOCKED on
lock=org.apache.lucene.store.FSDirectory
Hi Tom, if you're on a non Windows box, could you perhaps try your test
on the latest Solr nightly build? We've recently improved this through
the use of NIO.
-Yonik
On Fri, Nov 7, 2008 at 4:23 PM, Burton-West, Tom <tb...@umich.edu>
wrote:
> Hello,
>
> We are testing Solr with a simulation of 30 concurrent users. We are
> getting socket timeouts and the thread dump from the admin tool shows
> about 100+ threads with a similar message about a lock. (Message
> appended below).
>
> We supsect this may have something to do with one or more phrase
> queries containing common terms since our index is very large and we
> suspect one or more very large segments of the position index need to
> be read into memory.
>
> Can someone point us to either the possible cause of this problem or
> what we might change to reduce/eliminate it?
>
> Tom
>
> Tom Burton-West
> Information Retrieval Programmer
> Digital Library Production Services
> University of Michigan Library
> tburtonw@umich.edu
>
> ----------
>
> 'http-8080-Processor54' Id=71, BLOCKED on
> lock=org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor@13524
> 47 , total cpu time=2070.0000ms user time=1460.0000ms at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirect
> or
> y.java:532)
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.j
> av
> a:93)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput
> .j
> ava:34)
> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:57)
> at
> org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(Segment
> Te
> rmPositions.java:70)
> at
> org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermP
> os
> itions.java:66)
> at
> org.apache.lucene.search.PhrasePositions.nextPosition(PhrasePositions.
> ja
> va:76)
> at
>
org.apache.lucene.search.ExactPhraseScorer.phraseFreq(ExactPhraseScorer.
> java:45)
> at org.apache.lucene.search.PhraseScorer.doNext(PhraseScorer.java:94)
> at org.apache.lucene.search.PhraseScorer.next(PhraseScorer.java:81)
> at org.apache.lucene.search.Scorer.score(Scorer.java:48)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
> at org.apache.lucene.search.Searcher.search(Searcher.java:97)
> at
>
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.
> java:888)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher
> .j
> ava:805)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocList(SolrIndexSearcher.
> ja
> va:698)
> at
> org.apache.solr.request.StandardRequestHandler.handleRequestBody(Stand
> ar
> dRequestHandler.java:122)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle
> rB
> ase.java:77)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:658)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.
> ja
> va:191)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter
> .j
> ava:159)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> ca
> tionFilterChain.java:215)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
> lt
> erChain.java:188)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
> lv
> e.java:213)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
> lv
> e.java:174)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:5
> 48
> )
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
> va
> :127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
> va
> :117)
> at
>
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
> java:108)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
> :1
> 74)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
> 87
> 4)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.pr
> oc
> essConnection(Http11BaseProtocol.java:665)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoi
> nt
> .java:528)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll
> ow
> erWorkerThread.java:81)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
> ol
> .java:689)
> at java.lang.Thread.run(Thread.java:619)
>
>
Re: Solr locking issue? BLOCKED on lock=org.apache.lucene.store.FSDirectory
Posted by Yonik Seeley <yo...@apache.org>.
Hi Tom, if you're on a non Windows box, could you perhaps try your
test on the latest Solr nightly build? We've recently improved this
through the use of NIO.
-Yonik
On Fri, Nov 7, 2008 at 4:23 PM, Burton-West, Tom <tb...@umich.edu> wrote:
> Hello,
>
> We are testing Solr with a simulation of 30 concurrent users. We are
> getting socket timeouts and the thread dump from the admin tool shows
> about 100+ threads with a similar message about a lock. (Message
> appended below).
>
> We supsect this may have something to do with one or more phrase queries
> containing common terms since our index is very large and we suspect one
> or more very large segments of the position index need to be read into
> memory.
>
> Can someone point us to either the possible cause of this problem or
> what we might change to reduce/eliminate it?
>
> Tom
>
> Tom Burton-West
> Information Retrieval Programmer
> Digital Library Production Services
> University of Michigan Library
> tburtonw@umich.edu
>
> ----------
>
> 'http-8080-Processor54' Id=71, BLOCKED on
> lock=org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor@1352447
> , total cpu time=2070.0000ms user time=1460.0000ms
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:532)
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
> a:93)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
> ava:34)
> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:57)
> at
> org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(SegmentTe
> rmPositions.java:70)
> at
> org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermPos
> itions.java:66)
> at
> org.apache.lucene.search.PhrasePositions.nextPosition(PhrasePositions.ja
> va:76)
> at
> org.apache.lucene.search.ExactPhraseScorer.phraseFreq(ExactPhraseScorer.
> java:45)
> at org.apache.lucene.search.PhraseScorer.doNext(PhraseScorer.java:94)
> at org.apache.lucene.search.PhraseScorer.next(PhraseScorer.java:81)
> at org.apache.lucene.search.Scorer.score(Scorer.java:48)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
> at org.apache.lucene.search.Searcher.search(Searcher.java:97)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.
> java:888)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.j
> ava:805)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocList(SolrIndexSearcher.ja
> va:698)
> at
> org.apache.solr.request.StandardRequestHandler.handleRequestBody(Standar
> dRequestHandler.java:122)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB
> ase.java:77)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:658)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja
> va:191)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j
> ava:159)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
> tionFilterChain.java:215)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
> erChain.java:188)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
> e.java:213)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
> e.java:174)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548
> )
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
> :127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
> :117)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
> java:108)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:1
> 74)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:87
> 4)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.proc
> essConnection(Http11BaseProtocol.java:665)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint
> .java:528)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollow
> erWorkerThread.java:81)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool
> .java:689)
> at java.lang.Thread.run(Thread.java:619)
>
>