You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sunil <su...@yahoo.com> on 2008/12/02 19:05:19 UTC
OOM on commit after few days
I have been facing this issue since long in production environment and wanted to know if anybody came across can share their thoughts.
Appreciate your help.
Environment
2 GB index file
3.5 million documents
15 mins. time interval for committing 100 to 400 document updates
Commit happens once in 15 mins.
3.5 GB of RAM available for JVM
Solr Version 1.3 ; (nightly build of oct 18, 2008)
MDB - Message Driven Bean
I am Not using solr's replication mecahnism. Also don't use xml post update since the amount of data is too much.
I have bundled a MDB that receives messages for data updates and uses solr's update handler to update and commit the index.
Optimize happens once a day.
Everything runs fine for 2-3 days; after that I keep getting following exceptions.
Exception
org.apache.solr.common.SolrException log java.lang.OutOfMemoryError:
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:350)
at org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:596)
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:92)
at org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:907)
at org.apache.lucene.index.MultiSegmentReader.norms(MultiSegmentReader.java:338)
at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:131)
at org.apache.lucene.search.Searcher.search(Searcher.java:126)
at org.apache.lucene.search.Searcher.search(Searcher.java:105)
at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1170)
at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:856)
at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:283)
at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:170)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1302)
at org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:51)
at org.apache.solr.core.SolrCore$4.call(SolrCore.java:1128)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
at java.lang.Thread.run(Thread.java:810)
Re: OOM on commit after few days
Posted by Sunil <su...@yahoo.com>.
Thanks Yonik
The main search still happens through SolrDispatchFilter so SolrQueryRequest is getting closed implicitly.
But I do use direct api in following cases; So pl suggest any more possible resource issues
1. update and commit;
core.getUpdateHanlder();
Here I close the updateHandler once update/commits are done
2. Searching in other cores from current core writer
I have requirement to aggregate the data from multiple indexes and send single xml response.
otherCore.getSearcher() and call search method to get reference to Hits
I do call decref() on refCounted once done with processing result
3. Also call reload core after commit ; This brings down the ram usage but does not solve the main issue; With the reload I don't see any leaks but the OOM error occurs after 2-3 days time.
Do you think any other resource not getting closed ?
Sunil
--- On Tue, 12/2/08, Yonik Seeley <yo...@apache.org> wrote:
> From: Yonik Seeley <yo...@apache.org>
> Subject: Re: OOM on commit after few days
> To: solr-user@lucene.apache.org
> Date: Tuesday, December 2, 2008, 1:13 PM
> Using embedded is always more error prone...you're
> probably forgetting
> to close some resource.
> Make sure to close all SolrQueryRequest objects.
> Start with a memory profiler or heap dump to try and figure
> out what's
> taking up all the memory.
>
> -Yonik
>
> On Tue, Dec 2, 2008 at 1:05 PM, Sunil
> <su...@yahoo.com> wrote:
> > I have been facing this issue since long in production
> environment and wanted to know if anybody came across can
> share their thoughts.
> > Appreciate your help.
> >
> > Environment
> > 2 GB index file
> > 3.5 million documents
> > 15 mins. time interval for committing 100 to 400
> document updates
> > Commit happens once in 15 mins.
> > 3.5 GB of RAM available for JVM
> > Solr Version 1.3 ; (nightly build of oct 18, 2008)
> >
> > MDB - Message Driven Bean
> > I am Not using solr's replication mecahnism. Also
> don't use xml post update since the amount of data is
> too much.
> > I have bundled a MDB that receives messages for data
> updates and uses solr's update handler to update and
> commit the index.
> > Optimize happens once a day.
> >
> > Everything runs fine for 2-3 days; after that I keep
> getting following exceptions.
> >
> > Exception
> > org.apache.solr.common.SolrException log
> java.lang.OutOfMemoryError:
> > at java.io.RandomAccessFile.readBytes(Native
> Method)
> > at
> java.io.RandomAccessFile.read(RandomAccessFile.java:350)
> > at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:596)
> > at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)
> > at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:92)
> > at
> org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:907)
> > at
> org.apache.lucene.index.MultiSegmentReader.norms(MultiSegmentReader.java:338)
> > at
> org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
> > at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:131)
> > at
> org.apache.lucene.search.Searcher.search(Searcher.java:126)
> > at
> org.apache.lucene.search.Searcher.search(Searcher.java:105)
> > at
> org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1170)
> > at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:856)
> > at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:283)
> > at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
> > at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:170)
> > at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> > at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:1302)
> > at
> org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:51)
> > at
> org.apache.solr.core.SolrCore$4.call(SolrCore.java:1128)
> > at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284)
> > at
> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
> > at java.lang.Thread.run(Thread.java:810)
> >
> >
Re: OOM on commit after few days
Posted by Yonik Seeley <yo...@apache.org>.
Using embedded is always more error prone...you're probably forgetting
to close some resource.
Make sure to close all SolrQueryRequest objects.
Start with a memory profiler or heap dump to try and figure out what's
taking up all the memory.
-Yonik
On Tue, Dec 2, 2008 at 1:05 PM, Sunil <su...@yahoo.com> wrote:
> I have been facing this issue since long in production environment and wanted to know if anybody came across can share their thoughts.
> Appreciate your help.
>
> Environment
> 2 GB index file
> 3.5 million documents
> 15 mins. time interval for committing 100 to 400 document updates
> Commit happens once in 15 mins.
> 3.5 GB of RAM available for JVM
> Solr Version 1.3 ; (nightly build of oct 18, 2008)
>
> MDB - Message Driven Bean
> I am Not using solr's replication mecahnism. Also don't use xml post update since the amount of data is too much.
> I have bundled a MDB that receives messages for data updates and uses solr's update handler to update and commit the index.
> Optimize happens once a day.
>
> Everything runs fine for 2-3 days; after that I keep getting following exceptions.
>
> Exception
> org.apache.solr.common.SolrException log java.lang.OutOfMemoryError:
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(RandomAccessFile.java:350)
> at org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:596)
> at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)
> at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:92)
> at org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:907)
> at org.apache.lucene.index.MultiSegmentReader.norms(MultiSegmentReader.java:338)
> at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:131)
> at org.apache.lucene.search.Searcher.search(Searcher.java:126)
> at org.apache.lucene.search.Searcher.search(Searcher.java:105)
> at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1170)
> at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:856)
> at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:283)
> at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:170)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1302)
> at org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:51)
> at org.apache.solr.core.SolrCore$4.call(SolrCore.java:1128)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:284)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
> at java.lang.Thread.run(Thread.java:810)
>
>