You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Michael Thessel <mi...@gossamer-threads.com> on 2007/07/04 00:30:30 UTC
Re: Processor load
Hello UG
I couldn't solve the processor load problem. I decided to switch to
release 1.2. I currently run some tests (indexing 9 Million documents
and sending different search queries as fast as possible while indexing)
on the server and got some problems:
------------------------------------------
208973 SEVERE: Error during auto-warming of
key:org.apache.solr.search.QueryResultKey@75ca81e8:java
208974 at
org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:97)
208975 at
org.apache.lucene.search.ConjunctionScorer.next(ConjunctionScorer.java:54)
208976 at
org.apache.lucene.search.BooleanScorer2.next(BooleanScorer2.java:338)
208977 at org.apache.lucene.search.BooleanScorer
$SubScorer.<init>(BooleanScorer.java:56)
208978 at
org.apache.lucene.search.BooleanScorer.add(BooleanScorer.java:84)
208979 at
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:294)
208980 at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
208981 at
org.apache.lucene.search.Searcher.search(Searcher.java:118)
208982 at
org.apache.lucene.search.Searcher.search(Searcher.java:97)
208983 at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:888)
208984 at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:805)
208985 at org.apache.solr.search.SolrIndexSearcher.access
$100(SolrIndexSearcher.java:60)
208986 at org.apache.solr.search.SolrIndexSearcher
$2.regenerateItem(SolrIndexSearcher.java:251)
208987 at org.apache.solr.search.LRUCache.warm(LRUCache.java:193)
208988 at
org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1385)
208989 at org.apache.solr.core.SolrCore$1.call(SolrCore.java:488)
208990 at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:269)
208991 at java.util.concurrent.FutureTask.run(FutureTask.java:123)
208992 at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:650)
208993 at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:675)
208994 at java.lang.Thread.run(Thread.java:595)
------------------------------------------
After this error message repeated for about 100 times I get this error:
------------------------------------------
SEVERE: java.util.concurrent.ExecutionException:
java.lang.NullPointerException
210622 at java.util.concurrent.FutureTask
$Sync.innerGet(FutureTask.java:205)
210623 at java.util.concurrent.FutureTask.get(FutureTask.java:80)
210624 at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:542)
210625 at org.apache.solr.update.DirectUpdateHandler2
$CommitTracker.run(DirectUpdateHandler2.ja
210626 at java.util.concurrent.Executors
$RunnableAdapter.call(Executors.java:417)
210627 at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:269)
210628 at java.util.concurrent.FutureTask.run(FutureTask.java:123)
210629 at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$301(Sched
210630 at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThr
210631 at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:650)
210632 at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:675)
210633 at java.lang.Thread.run(Thread.java:595)
210634 Caused by: java.lang.NullPointerException
210635 at
org.apache.solr.core.SolrException.log(SolrException.java:67)
210636 at
org.apache.solr.core.SolrException.logOnce(SolrException.java:85)
210637 at org.apache.solr.core.SolrCore$4.call(SolrCore.java:546)
210638 at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:269)
210639 at java.util.concurrent.FutureTask.run(FutureTask.java:123)
210640 ... 3 more
------------------------------------------
And finally it ends up with a LockObtainFailedException:
------------------------------------------
SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain
timed out: SimpleFSLo
211054 at org.apache.lucene.store.Lock.obtain(Lock.java:70)
211055 at
org.apache.lucene.index.IndexWriter.init(IndexWriter.java:579)
211056 at
org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:341)
211057 at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:65)
211058 at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:120)
211059 at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:181)
211060 at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:259)
211061 at
org.apache.solr.handler.XmlUpdateRequestHandler.update(XmlUpdateRequestHandler.java:1
211062 at
org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHan
211063 at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:77)
211064 at org.apache.solr.core.SolrCore.execute(SolrCore.java:658)
211065 at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:191)
211066 at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:159)
211067 at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterCha
211068 at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
211069 at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:228)
211070 at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
211071 at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
211072 at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
211073 at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
211074 at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:216)
211075 at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
211076 at org.apache.coyote.http11.Http11Protocol
$Http11ConnectionHandler.process(Http11Protoco
211077 at org.apache.tomcat.util.net.JIoEndpoint
$Worker.run(JIoEndpoint.java:445)
211078 at java.lang.Thread.run(Thread.java:595)
------------------------------------------
The first two errors only appear when I send queries while I'm indexing.
The LockObtainFailedException always comes up after 1-2 hours of
indexing.
I found this patch for the LockObtainFailedException problem. Is this
currently the official way of fixing the problem? Is this patch already
available with one of the nightly builds?
https://issues.apache.org/jira/browse/SOLR-240
Thanks and Cheers,
Michael
On Thu, 2007-28-06 at 16:31 -0700, Otis Gospodnetic wrote:
> I think you'll need to add this to your Java command line (but you'll need to kill your JVM first):
> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9889 ....
>
> Otis
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Simpy -- http://www.simpy.com/ - Tag - Search - Share
>
> ----- Original Message ----
> From: Mike Klaas <mi...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Friday, June 29, 2007 1:01:03 AM
> Subject: Re: Processor load
>
> On 28-Jun-07, at 10:52 AM, Michael Thessel wrote:
>
> > Hi Mike
> >
> > Thanks for your quick response.
> > The restarting the server hasn't any effect. Im not a Java expert and
> > not really familiar with the java tools.
>
> I'm no expert in this matter either. Have you tried kill -QUIT? (it
> prints a thread dump to stderr).
>
> -Mike
>
> > Running jconsole on the 100% process (as the same user as the
> > server is
> > running):
> > # jconsole 23248
> > Unable to attach to 23248: Could not map vmid to user Name
> >
> > Running jps:
> > # jps -mlv
> > 11824 sun.tools.jps.Jps -mlv
> > -Dapplication.home=/var/home/solr/src/jdk1.5.0_11 -Xms8m
> > 23228 org.apache.catalina.startup.Bootstrap start -Xms1024M -Xmx2048M
> > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> > -Djava.util.logging.config.file=/home/solr/tomcat/conf/
> > logging.properties -Djava.endorsed.dirs=/home/solr/tomcat/endorsed -
> > Dcatalina.base=/home/solr/tomcat -Dcatalina.home=/home/solr/tomcat -
> > Djava.io.tmpdir=/home/solr/tomcat/temp
> >
> > Running jconsole on the parent process:
> > # jconsole 23228
> > 23228 is not a managed VM.
> >
> >
> > Cheers,
> >
> > Michael
> >
> >
> > On Thu, 2007-28-06 at 10:14 -0700, Mike Klaas wrote:
> >> On 28-Jun-07, at 9:36 AM, Michael Thessel wrote:
> >>
> >>> Hello UG
> >>>
> >>> I got a solr server running to index the posts of a forum. The
> >>> server is
> >>> running in a tomcat installation on a 4-processor Opteron server.
> >>> One of
> >>> the threads is eating up 100% of one processor all the time. As you
> >>> can
> >>> see in the ps output it is a fork of one of the childs:
> >>
> >> have you tried looking at a thread dump using jConsole/kill -QUIT?
> >>
> >> -Mike
> > --
> > Michael Thessel <mi...@gossamer-threads.com>
> > Gossamer Threads Inc. http://www.gossamer-threads.com/
> > Tel: (604) 687-5804 Fax: (604) 687-5806
> >
--
Michael Thessel <mi...@gossamer-threads.com>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806
Re: Processor load
Posted by Michael Thessel <mi...@gossamer-threads.com>.
On Wed, 2007-04-07 at 10:37 -0400, Yonik Seeley wrote:
> On 7/3/07, Michael Thessel <mi...@gossamer-threads.com> wrote:
> > ------------------------------------------
> > 208973 SEVERE: Error during auto-warming of
> > key:org.apache.solr.search.QueryResultKey@75ca81e8:java
> > 208974 at
> > org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:97)
> > 208975 at
>
> The Exception and exception message seem to be missing from the stack trace.
> Could you check your log and see if they are all like this?
>
> -Yonik
The problem was the autocommit every 10 s. I increased the autocommit
time to 60s and now the process runs only at 100% for 30-40s per minute.
Thanks a lot,
Michael
--
Michael Thessel <mi...@gossamer-threads.com>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806
Re: Processor load
Posted by Yonik Seeley <yo...@apache.org>.
On 7/3/07, Michael Thessel <mi...@gossamer-threads.com> wrote:
> ------------------------------------------
> 208973 SEVERE: Error during auto-warming of
> key:org.apache.solr.search.QueryResultKey@75ca81e8:java
> 208974 at
> org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:97)
> 208975 at
The Exception and exception message seem to be missing from the stack trace.
Could you check your log and see if they are all like this?
-Yonik