You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by danomano <ds...@earthlink.net> on 2011/03/17 19:39:35 UTC
Segments and Memory Correlate?
Hi folks, I ran into problem today where I am no longer able to execute any
queries :( due to Out of Memory issues.
I am in the process of investigating the use of different mergeFactors, or
even different merge policies all together.
My question is if I have many segments (i.e. smaller sized segments), will
that also reduce the total memory in RAM required for searching? (my System
is currently allocated 8GB ram and has a ~255GB index). (I'm not fully up
on the 'default merge policy' but I believe with a mergeFactor of 10, that
would mean each segment should be approaching about 25Gb? with ~543 million
documents
of note: this is all running on 1 server.
As seen below.
SEVERE: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.search.cache.LongValuesCreator.fillLongValues(LongValuesCreator.java:141)
at
org.apache.lucene.search.cache.LongValuesCreator.validate(LongValuesCreator.java:84)
at
org.apache.lucene.search.cache.LongValuesCreator.create(LongValuesCreator.java:74)
at
org.apache.lucene.search.cache.LongValuesCreator.create(LongValuesCreator.java:37)
at
org.apache.lucene.search.FieldCacheImpl$Cache.createValue(FieldCacheImpl.java:155)
at
org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:188)
at
org.apache.lucene.search.FieldCacheImpl.getLongs(FieldCacheImpl.java:337)
at
org.apache.lucene.search.FieldComparator$LongComparator.setNextReader(FieldComparator.java:504)
at
org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:97)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:207)
at org.apache.lucene.search.Searcher.search(Searcher.java:101)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1389)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1285)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:344)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:273)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:210)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1324)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:337)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:240)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at
com.openmarket.servletfilters.LogToCSVFilter.doFilter(LogToCSVFilter.java:89)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at
com.openmarket.servletfilters.GZipAutoDeflateFilter.doFilter(GZipAutoDeflateFilter.java:66)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
...etc
--
View this message in context: http://lucene.472066.n3.nabble.com/Segments-and-Memory-Correlate-tp2694747p2694747.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Segments and Memory Correlate?
Posted by danomano <ds...@earthlink.net>.
yes we are having physical memory issues, but do does anyone know for a fact
if there is direct correlation between be segment counts and RAM - memory?
i.e. when the system begins a search on a segment, does it load that
segments full index. Or does the system load All index data for all
Segments?
i.e. if I had for example 1000 segments, and it only loaded the index for
each segment it was searching, (assuming it 'indexes' through the
segments..hence this is why it would take longer to query on system with
many segments..as it must go through each segment one at a time), I would
assume this would require significantly less memory then loading 1 full
index on a system with only 10 segments. (assuming same total Index size).
Also I am considering switching to Zoie, but have encountered some
issues..(will posting on that shortly).
Dan
--
View this message in context: http://lucene.472066.n3.nabble.com/Segments-and-Memory-Correlate-tp2694747p2710586.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Segments and Memory Correlate?
Posted by Erick Erickson <er...@gmail.com>.
543M documents? On a single machine? How big is the index anyway?
I think you're running up against physical memory limitations,
the number of segments is a red herring. You're at a point where
you need to shard your index to multiple machines I'd guess. Or perhaps
that point was some time ago <G>.....
Best
Erick
On Thu, Mar 17, 2011 at 2:39 PM, danomano <ds...@earthlink.net> wrote:
> Hi folks, I ran into problem today where I am no longer able to execute any
> queries :( due to Out of Memory issues.
>
> I am in the process of investigating the use of different mergeFactors, or
> even different merge policies all together.
> My question is if I have many segments (i.e. smaller sized segments), will
> that also reduce the total memory in RAM required for searching? (my System
> is currently allocated 8GB ram and has a ~255GB index). (I'm not fully up
> on the 'default merge policy' but I believe with a mergeFactor of 10, that
> would mean each segment should be approaching about 25Gb? with ~543 million
> documents
>
> of note: this is all running on 1 server.
>
> As seen below.
>
> SEVERE: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.lucene.search.cache.LongValuesCreator.fillLongValues(LongValuesCreator.java:141)
> at
> org.apache.lucene.search.cache.LongValuesCreator.validate(LongValuesCreator.java:84)
> at
> org.apache.lucene.search.cache.LongValuesCreator.create(LongValuesCreator.java:74)
> at
> org.apache.lucene.search.cache.LongValuesCreator.create(LongValuesCreator.java:37)
> at
> org.apache.lucene.search.FieldCacheImpl$Cache.createValue(FieldCacheImpl.java:155)
> at
> org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:188)
> at
> org.apache.lucene.search.FieldCacheImpl.getLongs(FieldCacheImpl.java:337)
> at
> org.apache.lucene.search.FieldComparator$LongComparator.setNextReader(FieldComparator.java:504)
> at
> org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:97)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:207)
> at org.apache.lucene.search.Searcher.search(Searcher.java:101)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1389)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1285)
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:344)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:273)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:210)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1324)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:337)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:240)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
> at
> com.openmarket.servletfilters.LogToCSVFilter.doFilter(LogToCSVFilter.java:89)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
> at
> com.openmarket.servletfilters.GZipAutoDeflateFilter.doFilter(GZipAutoDeflateFilter.java:66)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
> ...etc
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Segments-and-Memory-Correlate-tp2694747p2694747.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>