Searching with too many clauses + Out of Memory

[Harini Raghavan <ha...@insideview.com>]

 Hi Everyone,

I am using Compass 1.1 M2 which supports Lucene 2.2 to store & search huge
amount of company, executive and employment data. There are some usecases
where I need to search for executives/employments on the result set of
company search. But when I try to create a compass query to search for
executives for over 1 lac company ids, it runs out of memory as the query is
huge. Here is the exception stack trace:

java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.lucene.index.SegmentReader.termDocs(SegmentReader.java:342)
at org.apache.lucene.index.MultiTermDocs.termDocs(MultiReader.java:435)
at org.apache.lucene.index.MultiTermDocs.termDocs(MultiReader.java:428)
at org.apache.lucene.index.MultiTermDocs.read(MultiReader.java:393)
at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
at org.apache.lucene.util.ScorerDocQueue.topNextAndAdjustElsePop(
ScorerDocQueue.ja va:116)
at org.apache.lucene.search.DisjunctionSumScorer.advanceAfterCurrent(DisjunctionSu
mScorer.java:175)
at org.apache.lucene.search.DisjunctionSumScorer.next(
DisjunctionSumScorer.java:14 6)
at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:327)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:124)
at org.apache.lucene.search.MultiSearcher.search(MultiSearcher.java:232)
at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:74)
at org.apache.lucene.search.Hits.<init>(Hits.java:61)
at org.apache.lucene.search.Searcher.search(Searcher.java:55)
at
org.compass.core.lucene.engine.transaction.ReadCommittedTransaction.findByQuery(
ReadCommittedTransaction.java:469)
at
org.compass.core.lucene.engine.transaction.ReadCommittedTransaction.doFind(Read
CommittedTransaction.java:426)
at org.compass.core.lucene.engine.transaction.AbstractTransaction.find(AbstractTra
nsaction.java:91)
at org.compass.core.lucene.engine.LuceneSearchEngine.find(
LuceneSearchEngine.java: 379)
at org.compass.core.lucene.engine.LuceneSearchEngineQuery.hits(LuceneSearchEngineQ
uery.java:151)
at org.compass.core.impl.DefaultCompassQuery.hits(DefaultCompassQuery.java:133)

at org.compass.core.support.search.CompassSearchHelper.performSearch(CompassSearch
Helper.java:144)
at org.compass.core.support.search.CompassSearchHelper$1.doInCompass(CompassSearch
Helper.java:89)
at org.compass.core.CompassTemplate.execute(CompassTemplate.java:137)
at org.compass.core.support.search.CompassSearchHelper.search(CompassSearchHelper.
java:86)

It looks like this error is actually in the lucene code. It would be great
if anyone in this group has an idea about this kind of usecase and has some
suggestions.

Thanks,
Harini

RE: Searching with too many clauses + Out of Memory

[Chandan Tamrakar <ch...@ccnep.com.np>]

What is the size of heap u r allocating for your app ?

-----Original Message-----
From: Harini Raghavan [mailto:harini.raghavan@insideview.com] 
Sent: Wednesday, August 01, 2007 2:29 PM
To: java-user@lucene.apache.org
Subject: Searching with too many clauses + Out of Memory

 Hi Everyone,

I am using Compass 1.1 M2 which supports Lucene 2.2 to store & search huge
amount of company, executive and employment data. There are some usecases
where I need to search for executives/employments on the result set of
company search. But when I try to create a compass query to search for
executives for over 1 lac company ids, it runs out of memory as the query is
huge. Here is the exception stack trace:

java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.lucene.index.SegmentReader.termDocs(SegmentReader.java:342)
at org.apache.lucene.index.MultiTermDocs.termDocs(MultiReader.java:435)
at org.apache.lucene.index.MultiTermDocs.termDocs(MultiReader.java:428)
at org.apache.lucene.index.MultiTermDocs.read(MultiReader.java:393)
at org.apache.lucene.search.TermScorer.next(TermScorer.java:106)
at org.apache.lucene.util.ScorerDocQueue.topNextAndAdjustElsePop(
ScorerDocQueue.ja va:116)
at
org.apache.lucene.search.DisjunctionSumScorer.advanceAfterCurrent(Disjunctio
nSu
mScorer.java:175)
at org.apache.lucene.search.DisjunctionSumScorer.next(
DisjunctionSumScorer.java:14 6)
at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:327)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:146)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:124)
at org.apache.lucene.search.MultiSearcher.search(MultiSearcher.java:232)
at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:74)
at org.apache.lucene.search.Hits.<init>(Hits.java:61)
at org.apache.lucene.search.Searcher.search(Searcher.java:55)
at
org.compass.core.lucene.engine.transaction.ReadCommittedTransaction.findByQu
ery(
ReadCommittedTransaction.java:469)
at
org.compass.core.lucene.engine.transaction.ReadCommittedTransaction.doFind(R
ead
CommittedTransaction.java:426)
at
org.compass.core.lucene.engine.transaction.AbstractTransaction.find(Abstract
Tra
nsaction.java:91)
at org.compass.core.lucene.engine.LuceneSearchEngine.find(
LuceneSearchEngine.java: 379)
at
org.compass.core.lucene.engine.LuceneSearchEngineQuery.hits(LuceneSearchEngi
neQ
uery.java:151)
at
org.compass.core.impl.DefaultCompassQuery.hits(DefaultCompassQuery.java:133)

at
org.compass.core.support.search.CompassSearchHelper.performSearch(CompassSea
rch
Helper.java:144)
at
org.compass.core.support.search.CompassSearchHelper$1.doInCompass(CompassSea
rch
Helper.java:89)
at org.compass.core.CompassTemplate.execute(CompassTemplate.java:137)
at
org.compass.core.support.search.CompassSearchHelper.search(CompassSearchHelp
er.
java:86)

It looks like this error is actually in the lucene code. It would be great
if anyone in this group has an idea about this kind of usecase and has some
suggestions.

Thanks,
Harini



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org