LockObtainFailedException and open finalizing IndexWriters

[Michael Kuhlmann <so...@kuli.org>]

Hi,

we are running Solr 3.2.0 on Jetty for a web application. Since we just
went online and are still in beta tests, we don't have very much load on
our servers (indeed, they're currently much oversized for the current
usage), and our index size on file system ist just 1.1 MB.

We have one dedicated Solr instance for updates, and two replicated
read-only servers for requests. The update server gets filled by three
different Java web servers, each has a distinct Quartz job for its
updates. Every such Quartz job takes all collected updates, sends them
via Solrj's addBeans() method, and from time to time, they send an
additional commit() after that. Each update job has a
CommonHTTPSolrServer instance, which is a Spring controlled singleton.

We already had LockObtainFailedExceptions before, raising every few
days. Sometimes, we had such an exception before:
org.apache.solr.common.SolrException: java.io.IOException: directory
'/data/solr/data/index' exists and is a directory, but cannot be listed:
list() returned null

This looks like if there were no more file handles from the operating
system. This is strange, since the only index directory never had more
than 100 files, if ever. However, we raised ulimit -n from 1024 to 4096,
and reduced mergeFactor from 10 to 5, which firsted helped us with our
problem. Until yesterday.

Again, we had this:
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed
out: SimpleFSLock@solr/main/data/index/write.lock        at
org.apache.lucene.store.Lock.obtain(Lock.java:84)    at
org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1114)    at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:83)
     at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:101)
  at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:174)
       at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:222)
   at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61)

.....


When we deleted the write.lock file without restarting Solr, several
hours later we had 441 same log entries:

Jul 18, 2011 7:20:29 AM org.apache.solr.update.SolrIndexWriter finalize
SEVERE: SolrIndexWriter was not closed prior to finalize(), indicates a
bug -- POSSIBLE RESOURCE LEAK!!!

Wow, if there really were 441 open IndexWriters trying to access the
index directory, it's no wonder that there will be Lock timeouts sooner
or later! However, I have no clue why there are so many IndexWriters
opened and never closed. The only accessing Solr instances are pure Java
applications using Solrj. Each application only has one SolrServer
instance - and even of not, this shouldn't harm, AFAIK. The update job
is started every five seconds. The installation is a pure 3.2.0 Solr,
without additional jars. And all jars are of the correct revision. The
solrconfig.xml is based on the example configuration, with nothing
special. We currently don't have any own extensions running. There is
absolutely only one jetty instance running on the machine. And I checked
the solr.xml, it's only one core defined, and we don't do any additional
core administration.

I'm using Solr since the beginning of 2010, but never had such a
problem. Any help is welcome.

Greetings,
Kuli