You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by mike anderson <sa...@gmail.com> on 2010/01/25 22:14:40 UTC
Re: Lock problems: Lock obtain timed out
I am getting this exception as well, but disk space is not my problem. What
else can I do to debug this? The solr log doesn't appear to lend any other
clues..
Jan 25, 2010 4:02:22 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={} status=500 QTime=1990
Jan 25, 2010 4:02:22 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed
out: NativeFSLock@
/solr8984/index/lucene-98c1cb272eb9e828b1357f68112231e0-write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:85)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1402)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:190)
at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:98)
at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:220)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Should I consider changing the lock timeout settings (currently set to
defaults)? If so, I'm not sure what to base these values on.
Thanks in advance,
mike
On Wed, Nov 4, 2009 at 8:27 PM, Lance Norskog <go...@gmail.com> wrote:
> This will not ever work reliably. You should have 2x total disk space
> for the index. Optimize, for one, requires this.
>
> On Wed, Nov 4, 2009 at 6:37 AM, Jérôme Etévé <je...@gmail.com>
> wrote:
> > Hi,
> >
> > It seems this situation is caused by some No space left on device
> exeptions:
> > SEVERE: java.io.IOException: No space left on device
> > at java.io.RandomAccessFile.writeBytes(Native Method)
> > at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
> > at
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.flushBuffer(SimpleFSDirectory.java:192)
> > at
> org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:96)
> >
> >
> > I'd better try to set my maxMergeDocs and mergeFactor to more
> > adequates values for my app (I'm indexing ~15 Gb of data on 20Gb
> > device, so I guess there's problem when solr tries to merge the index
> > bits being build.
> >
> > At the moment, they are set to <mergeFactor>100</mergeFactor> and
> > <maxMergeDocs>2147483647</maxMergeDocs>
> >
> > Jerome.
> >
> > --
> > Jerome Eteve.
> > http://www.eteve.net
> > jerome@eteve.net
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: Lock problems: Lock obtain timed out
Posted by Chris Hostetter <ho...@fucit.org>.
: Can anyone think of a reason why these locks would hang around for more than
: 2 hours?
:
: I have been monitoring them and they look like they are very short lived.
Typically the lock files are only left arround for more then a few seconds
when there was a fatal crash of some kind ... an OOM Error for example, or
as already mentioned in this thread...
: >> > > SEVERE: java.io.IOException: No space left on device
...if you check your solr logs for messages in the immediate time frame
following the the lastModified time of the lock file you'll probably find
something interesting.
-Hoss
Re: Lock problems: Lock obtain timed out
Posted by Ian Connor <ia...@gmail.com>.
Can anyone think of a reason why these locks would hang around for more than
2 hours?
I have been monitoring them and they look like they are very short lived.
On Tue, Jan 26, 2010 at 10:15 AM, Ian Connor <ia...@gmail.com> wrote:
> We traced one of the lock files, and it had been around for 3 hours. A
> restart removed it - but is 3 hours normal for one of these locks?
>
> Ian.
>
>
> On Mon, Jan 25, 2010 at 4:14 PM, mike anderson <sa...@gmail.com>wrote:
>
>> I am getting this exception as well, but disk space is not my problem.
>> What
>> else can I do to debug this? The solr log doesn't appear to lend any other
>> clues..
>>
>> Jan 25, 2010 4:02:22 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/update params={} status=500 QTime=1990
>> Jan 25, 2010 4:02:22 PM org.apache.solr.common.SolrException log
>> SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain
>> timed
>> out: NativeFSLock@
>> /solr8984/index/lucene-98c1cb272eb9e828b1357f68112231e0-write.lock
>> at org.apache.lucene.store.Lock.obtain(Lock.java:85)
>> at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545)
>> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1402)
>> at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:190)
>> at
>>
>> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:98)
>> at
>>
>> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173)
>> at
>>
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:220)
>> at
>>
>> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61)
>> at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
>> at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>> at
>>
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
>> at
>>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>> at
>>
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>> at
>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>> at
>>
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
>> at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
>> at
>>
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
>> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
>> at
>>
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
>> at
>>
>> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>> at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
>> at org.mortbay.jetty.Server.handle(Server.java:285)
>> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
>> at
>>
>> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
>> at
>>
>> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
>> at
>>
>> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>>
>>
>> Should I consider changing the lock timeout settings (currently set to
>> defaults)? If so, I'm not sure what to base these values on.
>>
>> Thanks in advance,
>> mike
>>
>>
>> On Wed, Nov 4, 2009 at 8:27 PM, Lance Norskog <go...@gmail.com> wrote:
>>
>> > This will not ever work reliably. You should have 2x total disk space
>> > for the index. Optimize, for one, requires this.
>> >
>> > On Wed, Nov 4, 2009 at 6:37 AM, Jérôme Etévé <je...@gmail.com>
>> > wrote:
>> > > Hi,
>> > >
>> > > It seems this situation is caused by some No space left on device
>> > exeptions:
>> > > SEVERE: java.io.IOException: No space left on device
>> > > at java.io.RandomAccessFile.writeBytes(Native Method)
>> > > at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
>> > > at
>> >
>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.flushBuffer(SimpleFSDirectory.java:192)
>> > > at
>> >
>> org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:96)
>> > >
>> > >
>> > > I'd better try to set my maxMergeDocs and mergeFactor to more
>> > > adequates values for my app (I'm indexing ~15 Gb of data on 20Gb
>> > > device, so I guess there's problem when solr tries to merge the index
>> > > bits being build.
>> > >
>> > > At the moment, they are set to <mergeFactor>100</mergeFactor> and
>> > > <maxMergeDocs>2147483647</maxMergeDocs>
>> > >
>> > > Jerome.
>> > >
>> > > --
>> > > Jerome Eteve.
>> > > http://www.eteve.net
>> > > jerome@eteve.net
>> > >
>> >
>> >
>> >
>> > --
>> > Lance Norskog
>> > goksron@gmail.com
>> >
>>
>
--
Regards,
Ian Connor
1 Leighton St #723
Cambridge, MA 02141
Call Center Phone: +1 (714) 239 3875 (24 hrs)
Fax: +1(770) 818 5697
Skype: ian.connor
Re: Lock problems: Lock obtain timed out
Posted by Ian Connor <ia...@gmail.com>.
We traced one of the lock files, and it had been around for 3 hours. A
restart removed it - but is 3 hours normal for one of these locks?
Ian.
On Mon, Jan 25, 2010 at 4:14 PM, mike anderson <sa...@gmail.com>wrote:
> I am getting this exception as well, but disk space is not my problem. What
> else can I do to debug this? The solr log doesn't appear to lend any other
> clues..
>
> Jan 25, 2010 4:02:22 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/update params={} status=500 QTime=1990
> Jan 25, 2010 4:02:22 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain
> timed
> out: NativeFSLock@
> /solr8984/index/lucene-98c1cb272eb9e828b1357f68112231e0-write.lock
> at org.apache.lucene.store.Lock.obtain(Lock.java:85)
> at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1402)
> at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:190)
> at
>
> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:98)
> at
>
> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173)
> at
>
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:220)
> at
>
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61)
> at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
> at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
> at
>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
> at
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at
>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
> at
>
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
> at
>
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
> at org.mortbay.jetty.Server.handle(Server.java:285)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
> at
>
> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
> at
>
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
> at
>
> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>
>
> Should I consider changing the lock timeout settings (currently set to
> defaults)? If so, I'm not sure what to base these values on.
>
> Thanks in advance,
> mike
>
>
> On Wed, Nov 4, 2009 at 8:27 PM, Lance Norskog <go...@gmail.com> wrote:
>
> > This will not ever work reliably. You should have 2x total disk space
> > for the index. Optimize, for one, requires this.
> >
> > On Wed, Nov 4, 2009 at 6:37 AM, Jérôme Etévé <je...@gmail.com>
> > wrote:
> > > Hi,
> > >
> > > It seems this situation is caused by some No space left on device
> > exeptions:
> > > SEVERE: java.io.IOException: No space left on device
> > > at java.io.RandomAccessFile.writeBytes(Native Method)
> > > at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
> > > at
> >
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.flushBuffer(SimpleFSDirectory.java:192)
> > > at
> >
> org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:96)
> > >
> > >
> > > I'd better try to set my maxMergeDocs and mergeFactor to more
> > > adequates values for my app (I'm indexing ~15 Gb of data on 20Gb
> > > device, so I guess there's problem when solr tries to merge the index
> > > bits being build.
> > >
> > > At the moment, they are set to <mergeFactor>100</mergeFactor> and
> > > <maxMergeDocs>2147483647</maxMergeDocs>
> > >
> > > Jerome.
> > >
> > > --
> > > Jerome Eteve.
> > > http://www.eteve.net
> > > jerome@eteve.net
> > >
> >
> >
> >
> > --
> > Lance Norskog
> > goksron@gmail.com
> >
>