You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by nonrenewable <no...@gmail.com> on 2009/10/14 18:26:18 UTC
Solr/Lucene keeps eating up memory while idling
I'm curious why this is occurring and whether i can prevent it. This is my
scenario:
Locally I have an idle running solr 1.3 service using lucene 2.4.1 which has
an index of ~330K documents containing ~10 fields each(total size ~12GB).
Currently I've turned off all caching, lazy field loading, however i do have
facet fields set for some request handlers.
What i'm seeing is heap space usage increasing by ~1.2MB per 2 sec (by
java.lang.String objects). I'm assuming they're being used by lucene but i
may be wrong about that, since i have no actual data to confirm it. Why
exactly is this happening, considering no requests are being serviced?
Shouldn't the memory usage stabilise with a certain set of information and
only be affected on requests? Additionally there is a full GC every half
hour, which seems very unreasonable on a machine that isn't actually being
used as a service.
I really hope there's just a certain setting that i've overlooked, or a
concept i'm not understanding because otherwise this behaviour seems very
unreasonable...
Thanks beforehand,
Tony
--
View this message in context: http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25894357.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr/Lucene keeps eating up memory while idling
Posted by Yonik Seeley <yo...@lucidimagination.com>.
I just did some allocation profiling on the stock Solr example... it's
not completely idle when no requests are being made.
There's only one thing allocating memory: org.mortbay.util.Scanner.scanFiles()
That must be Jetty looking to see if any of the files under webapps has changed.
It's really nothing to worry about - there's no memory leaks, and the
activity is extremely minimal, but if you want to shut it off, it
would be a Jetty config option somewhere.
-Yonik
http://www.lucidimagination.com
On Wed, Oct 14, 2009 at 12:26 PM, nonrenewable <no...@gmail.com> wrote:
>
> I'm curious why this is occurring and whether i can prevent it. This is my
> scenario:
>
> Locally I have an idle running solr 1.3 service using lucene 2.4.1 which has
> an index of ~330K documents containing ~10 fields each(total size ~12GB).
> Currently I've turned off all caching, lazy field loading, however i do have
> facet fields set for some request handlers.
>
> What i'm seeing is heap space usage increasing by ~1.2MB per 2 sec (by
> java.lang.String objects). I'm assuming they're being used by lucene but i
> may be wrong about that, since i have no actual data to confirm it. Why
> exactly is this happening, considering no requests are being serviced?
> Shouldn't the memory usage stabilise with a certain set of information and
> only be affected on requests? Additionally there is a full GC every half
> hour, which seems very unreasonable on a machine that isn't actually being
> used as a service.
>
> I really hope there's just a certain setting that i've overlooked, or a
> concept i'm not understanding because otherwise this behaviour seems very
> unreasonable...
>
> Thanks beforehand,
> Tony
> --
> View this message in context: http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25894357.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: Solr/Lucene keeps eating up memory while idling
Posted by nonrenewable <no...@gmail.com>.
Here is exactly half an hour from roughly the beginning of logging. There's
nothing to see really because no requests are sent, you just see the GC
behaviour:
[Full GC 211987K->208493K(432448K), 0.6273480 secs]
[GC 276333K->212269K(438720K), 0.0929710 secs]
[GC 289133K->216269K(439936K), 0.1019780 secs]
[GC 293133K->220205K(436672K), 0.1128410 secs]
[GC 304301K->224429K(441472K), 0.1358250 secs]
[GC 308525K->228685K(431744K), 0.1559950 secs]
[GC 317197K->233069K(437312K), 0.1642160 secs]
[GC 321581K->237613K(432832K), 0.1772830 secs]
[GC 329197K->242093K(435136K), 0.1896270 secs]
[GC 333677K->246701K(436352K), 0.2039880 secs]
[GC 274165K->247917K(437760K), 0.2022640 secs]
[Full GC 247917K->208726K(437760K), 0.7195200 secs]
The heap is set to 1400m so it'll take it awhile to hit the roof. I also
haven't tested to see if it stabilises but i'll leave it running now and see
what happens to it overnight. I assume that when(if) it reaches the heap
limit i'll just do full GCs more often.
Grant Ingersoll-6 wrote:
>
> Please send a log covering at least the 2.5 minutes you discuss, but
> upwards of 5 minutes would be good.
>
--
View this message in context: http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25916348.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr/Lucene keeps eating up memory while idling
Posted by Grant Ingersoll <gs...@apache.org>.
Please send a log covering at least the 2.5 minutes you discuss, but
upwards of 5 minutes would be good.
On Oct 15, 2009, at 1:26 PM, nonrenewable wrote:
>
>> Did I read that right? 330K docs == 12 GB index.
>
> Ops, missed the dot - 1.2GB, but i don't think that should really
> make the
> difference in this case. Even if it was 12 GB it would just have
> some really
> juicy documents, right? :)
>
>> Can you share the Solr logs and/or your config? Is this happening
>> around a commit or some warming process? After startup, with no
>> requests hitting it and no warming/commits/indexing, I don't see why
>> it would be growing. Do you have custom code?
>
> There is custom code around the solrj API however it does not
> explain this
> behaviour because of the lack of requests coming through it. There
> are no
> indexing, commits or queries sent to the server after it's started up,
> except for the initial 2 warming queries (can those be to blame for
> this
> even with no caches present??). Here are these in the log (it's on
> it's
> default verbosity so i'll refrain from posting the whole start up
> until
> necessary) After the initial start up, what you see in the log is GC
> every
> 2.5 min and Full GC every 30min. No actual activity is present.
>
> Oct 15, 2009 1:13:36 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=null path=null params={start=0&q=fast_warm&rows=10}
> hits=0
> status=0 QTime=16853
> Oct 15, 2009 1:13:36 PM org.apache.solr.core.QuerySenderListener
> newSearcher
> INFO: QuerySenderListener done.
> Oct 15, 2009 1:13:36 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=null path=null
> params={q=static+firstSearcher+warming+query+from+solrconfig.xml}
> hits=0
> status=0 QTime=204
> Oct 15, 2009 1:13:36 PM org.apache.solr.core.QuerySenderListener
> newSearcher
> INFO: QuerySenderListener done
>
>
> here is the config on it:
>
> <config>
>
> <abortOnConfigurationError>${solr.abortOnConfigurationError:true}</
> abortOnConfigurationError>
> <dataDir>/r9/flare1.data/solr/data</dataDir>
> <indexDefaults>
> <useCompoundFile>false</useCompoundFile>
> <mergeFactor>10</mergeFactor>
> <ramBufferSizeMB>32</ramBufferSizeMB>
> <maxMergeDocs>2147483647</maxMergeDocs>
> <maxFieldLength>10000</maxFieldLength>
> <writeLockTimeout>1000</writeLockTimeout>
> <commitLockTimeout>10000</commitLockTimeout>
> <lockType>single</lockType>
> </indexDefaults>
>
> <mainIndex>
> <useCompoundFile>false</useCompoundFile>
> <ramBufferSizeMB>32</ramBufferSizeMB>
> <mergeFactor>10</mergeFactor>
> <maxMergeDocs>2147483647</maxMergeDocs>
> <maxFieldLength>10000</maxFieldLength>
> <unlockOnStartup>false</unlockOnStartup>
> </mainIndex>
> <jmx />
>
> <updateHandler class="solr.DirectUpdateHandler2">
> </updateHandler>
>
>
> <query>
> <maxBooleanClauses>1024</maxBooleanClauses>
> <queryResultWindowSize>50</queryResultWindowSize>
> <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
> <HashDocSet maxSize="3000" loadFactor="0.75"/>
> <listener event="newSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst> <str name="q">solr</str> <str name="start">0</str> <str
> name="rows">10</str> </lst>
> <lst> <str name="q">rocks</str> <str name="start">0</str> <str
> name="rows">10</str> </lst>
> <lst><str name="q">static newSearcher warming query from
> solrconfig.xml</str></lst>
> </arr>
> </listener>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst> <str name="q">fast_warm</str> <str name="start">0</str>
> <str
> name="rows">10</str> </lst>
> <lst><str name="q">static firstSearcher warming query from
> solrconfig.xml</str></lst>
> </arr>
> </listener>
> <useColdSearcher>false</useColdSearcher>
> <maxWarmingSearchers>2</maxWarmingSearchers>
> </query>
>
> <requestDispatcher handleSelect="true" >
> <requestParsers enableRemoteStreaming="false"
> multipartUploadLimitInKB="2048" />
> </requestDispatcher>
>
> <requestHandler name="standard" class="solr.SearchHandler"
> default="true">
> <lst name="defaults">
> <str name="echoParams">explicit</str>
> </lst>
> </requestHandler>
>
> <requestHandler name="dismax" class="solr.SearchHandler" >
> <lst name="defaults">
> <str name="defType">dismax</str>
> <str name="echoParams">explicit</str>
> <float name="tie">0.01</float>
> <str name="qf">
> text^0.5 address_t^2.0 name^1.5 brand^1.1 airport_name_t^1.0
> </str>
> <str name="pf">
> text^0.2 address_t^1.1 name^1.5 brand^1.4 brand_exact^1.9
> airport_name_t^1.0
> </str>
> <str name="fl">
> id,name,price,score
> </str>
> <int name="ps">100</int>
> <str name="q.alt">*:*</str>
> <str name="hl.fl">text features name</str>
> <str name="f.name.hl.fragsize">0</str>
> <str name="f.name.hl.alternateField">name</str>
> <str name="f.text.hl.fragmenter">regex</str> <!-- defined below
> -->
> <str name="spellcheck">true</str>
> <str name="spellcheck.extendedResults">true</str>
> <str name="spellcheck.collate">true</str>
> <str name="spellcheck.count">5</str>
> </lst>
> <arr name="last-components">
> <str>spellcheck</str>
> </arr>
> </requestHandler>
> <requestHandler name="partitioned" class="solr.SearchHandler" >
> <lst name="defaults">
> <str name="defType">dismax</str>
> <str name="echoParams">explicit</str>
> <str name="qf">text^0.5 features^1.0 name^1.2 id^10.0</str>
> <str name="mm">2<-1 5<-2 6<90%</str>
> <str name="bq">incubationdate_dt:[* TO NOW/DAY-1MONTH]^2.2</str>
> </lst>
> <lst name="appends">
> <str name="fq">inStock:true</str>
> </lst>
> </requestHandler>
>
> <updateRequestProcessor>
> <factory name="standard" class="solr.ChainedUpdateProcessorFactory"
> default="true">
> <chain
> class="com.pjaol.search.solr.update.LocalUpdateProcessorFactory">
> <str name="latField">lat</str>
> <str name="lngField">lng</str>
> <int name="startTier">9</int>
> <int name="endTier">17</int>
> </chain>
> <chain class="solr.LogUpdateProcessorFactory" >
> </chain>
> <chain class="solr.RunUpdateProcessorFactory" />
> </factory>
> </updateRequestProcessor>
>
> <requestHandler name="geo"
> class="com.pjaol.search.solr.LocalSolrRequestHandler">
> <str name="latField">lat</str>
> <str name="lngField">lng</str>
> </requestHandler>
>
> <searchComponent name="spellcheck" class="solr.SpellCheckComponent">
>
> <str name="queryAnalyzerFieldType">textSpell</str>
>
> <lst name="spellchecker">
> <str name="name">default</str>
> <str name="field">spell</str>
> <str name="spellcheckIndexDir">./spellchecker1</str>
>
> </lst>
> <lst name="spellchecker">
> <str name="name">jarowinkler</str>
> <str name="field">spell</str>
> <str
> name
> =
> "distanceMeasure
> ">org.apache.lucene.search.spell.JaroWinklerDistance</str>
> <str name="spellcheckIndexDir">./spellchecker2</str>
>
> </lst>
> </searchComponent>
>
> <requestHandler name="/spellCheckCompRH" class="solr.SearchHandler">
> <lst name="defaults">
> <str name="spellcheck.onlyMorePopular">false</str>
> <str name="spellcheck.extendedResults">false</str>
> <str name="spellcheck.count">1</str>
> </lst>
> <arr name="last-components">
> <str>spellcheck</str>
> </arr>
> </requestHandler>
>
> <searchComponent name="elevator"
> class="solr.QueryElevationComponent" >
> <str name="queryFieldType">string</str>
> <str name="config-file">elevate.xml</str>
> </searchComponent>
>
> <requestHandler name="/elevate" class="solr.SearchHandler"
> startup="lazy">
> <lst name="defaults">
> <str name="echoParams">explicit</str>
> </lst>
> <arr name="last-components">
> <str>elevator</str>
> </arr>
> </requestHandler>
>
>
> <requestHandler name="/update"
> class="solr.XmlUpdateRequestHandler" />
> <requestHandler name="/analysis"
> class="solr.AnalysisRequestHandler" />
>
>
> <requestHandler name="/update/csv" class="solr.CSVRequestHandler"
> startup="lazy" />
>
>
> <requestHandler name="/admin/"
> class="org.apache.solr.handler.admin.AdminHandlers" />
>
> <requestHandler name="/admin/ping" class="PingRequestHandler">
> <lst name="defaults">
> <str name="qt">standard</str>
> <str name="q">solrpingquery</str>
> <str name="echoParams">all</str>
> </lst>
> </requestHandler>
>
> <requestHandler name="/debug/dump" class="solr.DumpRequestHandler" >
> <lst name="defaults">
> <str name="echoParams">explicit</str>
> <str name="echoHandler">true</str>
> </lst>
> </requestHandler>
>
> <highlighting>
> <fragmenter name="gap"
> class="org.apache.solr.highlight.GapFragmenter"
> default="true">
> <lst name="defaults">
> <int name="hl.fragsize">100</int>
> </lst>
> </fragmenter>
>
> <fragmenter name="regex"
> class="org.apache.solr.highlight.RegexFragmenter">
> <lst name="defaults">
> <int name="hl.fragsize">70</int>
> <float name="hl.regex.slop">0.5</float>
> <str name="hl.regex.pattern">[-\w ,/\n\"']{20,200}</str>
> </lst>
> </fragmenter>
>
> <formatter name="html"
> class="org.apache.solr.highlight.HtmlFormatter"
> default="true">
> <lst name="defaults">
> <str name="hl.simple.pre"><![CDATA[<em>]]></str>
> <str name="hl.simple.post"><![CDATA[</em>]]></str>
> </lst>
> </formatter>
> </highlighting>
>
> <queryResponseWriter name="xslt"
> class="org.apache.solr.request.XSLTResponseWriter">
> <int name="xsltCacheLifetimeSeconds">5</int>
> </queryResponseWriter>
>
> <admin>
> <defaultQuery>solr</defaultQuery>
>
> </admin>
>
> </config>
> --
> View this message in context: http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25912727.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
Re: Solr/Lucene keeps eating up memory while idling
Posted by nonrenewable <no...@gmail.com>.
>Did I read that right? 330K docs == 12 GB index.
Ops, missed the dot - 1.2GB, but i don't think that should really make the
difference in this case. Even if it was 12 GB it would just have some really
juicy documents, right? :)
>Can you share the Solr logs and/or your config? Is this happening
>around a commit or some warming process? After startup, with no
>requests hitting it and no warming/commits/indexing, I don't see why
>it would be growing. Do you have custom code?
There is custom code around the solrj API however it does not explain this
behaviour because of the lack of requests coming through it. There are no
indexing, commits or queries sent to the server after it's started up,
except for the initial 2 warming queries (can those be to blame for this
even with no caches present??). Here are these in the log (it's on it's
default verbosity so i'll refrain from posting the whole start up until
necessary) After the initial start up, what you see in the log is GC every
2.5 min and Full GC every 30min. No actual activity is present.
Oct 15, 2009 1:13:36 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=null path=null params={start=0&q=fast_warm&rows=10} hits=0
status=0 QTime=16853
Oct 15, 2009 1:13:36 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Oct 15, 2009 1:13:36 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=null path=null
params={q=static+firstSearcher+warming+query+from+solrconfig.xml} hits=0
status=0 QTime=204
Oct 15, 2009 1:13:36 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done
here is the config on it:
<config>
<abortOnConfigurationError>${solr.abortOnConfigurationError:true}</abortOnConfigurationError>
<dataDir>/r9/flare1.data/solr/data</dataDir>
<indexDefaults>
<useCompoundFile>false</useCompoundFile>
<mergeFactor>10</mergeFactor>
<ramBufferSizeMB>32</ramBufferSizeMB>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength>
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
<lockType>single</lockType>
</indexDefaults>
<mainIndex>
<useCompoundFile>false</useCompoundFile>
<ramBufferSizeMB>32</ramBufferSizeMB>
<mergeFactor>10</mergeFactor>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength>
<unlockOnStartup>false</unlockOnStartup>
</mainIndex>
<jmx />
<updateHandler class="solr.DirectUpdateHandler2">
</updateHandler>
<query>
<maxBooleanClauses>1024</maxBooleanClauses>
<queryResultWindowSize>50</queryResultWindowSize>
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
<HashDocSet maxSize="3000" loadFactor="0.75"/>
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst> <str name="q">solr</str> <str name="start">0</str> <str
name="rows">10</str> </lst>
<lst> <str name="q">rocks</str> <str name="start">0</str> <str
name="rows">10</str> </lst>
<lst><str name="q">static newSearcher warming query from
solrconfig.xml</str></lst>
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst> <str name="q">fast_warm</str> <str name="start">0</str> <str
name="rows">10</str> </lst>
<lst><str name="q">static firstSearcher warming query from
solrconfig.xml</str></lst>
</arr>
</listener>
<useColdSearcher>false</useColdSearcher>
<maxWarmingSearchers>2</maxWarmingSearchers>
</query>
<requestDispatcher handleSelect="true" >
<requestParsers enableRemoteStreaming="false"
multipartUploadLimitInKB="2048" />
</requestDispatcher>
<requestHandler name="standard" class="solr.SearchHandler" default="true">
<lst name="defaults">
<str name="echoParams">explicit</str>
</lst>
</requestHandler>
<requestHandler name="dismax" class="solr.SearchHandler" >
<lst name="defaults">
<str name="defType">dismax</str>
<str name="echoParams">explicit</str>
<float name="tie">0.01</float>
<str name="qf">
text^0.5 address_t^2.0 name^1.5 brand^1.1 airport_name_t^1.0
</str>
<str name="pf">
text^0.2 address_t^1.1 name^1.5 brand^1.4 brand_exact^1.9
airport_name_t^1.0
</str>
<str name="fl">
id,name,price,score
</str>
<int name="ps">100</int>
<str name="q.alt">*:*</str>
<str name="hl.fl">text features name</str>
<str name="f.name.hl.fragsize">0</str>
<str name="f.name.hl.alternateField">name</str>
<str name="f.text.hl.fragmenter">regex</str> <!-- defined below -->
<str name="spellcheck">true</str>
<str name="spellcheck.extendedResults">true</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.count">5</str>
</lst>
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
<requestHandler name="partitioned" class="solr.SearchHandler" >
<lst name="defaults">
<str name="defType">dismax</str>
<str name="echoParams">explicit</str>
<str name="qf">text^0.5 features^1.0 name^1.2 id^10.0</str>
<str name="mm">2<-1 5<-2 6<90%</str>
<str name="bq">incubationdate_dt:[* TO NOW/DAY-1MONTH]^2.2</str>
</lst>
<lst name="appends">
<str name="fq">inStock:true</str>
</lst>
</requestHandler>
<updateRequestProcessor>
<factory name="standard" class="solr.ChainedUpdateProcessorFactory"
default="true">
<chain
class="com.pjaol.search.solr.update.LocalUpdateProcessorFactory">
<str name="latField">lat</str>
<str name="lngField">lng</str>
<int name="startTier">9</int>
<int name="endTier">17</int>
</chain>
<chain class="solr.LogUpdateProcessorFactory" >
</chain>
<chain class="solr.RunUpdateProcessorFactory" />
</factory>
</updateRequestProcessor>
<requestHandler name="geo"
class="com.pjaol.search.solr.LocalSolrRequestHandler">
<str name="latField">lat</str>
<str name="lngField">lng</str>
</requestHandler>
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">textSpell</str>
<lst name="spellchecker">
<str name="name">default</str>
<str name="field">spell</str>
<str name="spellcheckIndexDir">./spellchecker1</str>
</lst>
<lst name="spellchecker">
<str name="name">jarowinkler</str>
<str name="field">spell</str>
<str
name="distanceMeasure">org.apache.lucene.search.spell.JaroWinklerDistance</str>
<str name="spellcheckIndexDir">./spellchecker2</str>
</lst>
</searchComponent>
<requestHandler name="/spellCheckCompRH" class="solr.SearchHandler">
<lst name="defaults">
<str name="spellcheck.onlyMorePopular">false</str>
<str name="spellcheck.extendedResults">false</str>
<str name="spellcheck.count">1</str>
</lst>
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
<searchComponent name="elevator" class="solr.QueryElevationComponent" >
<str name="queryFieldType">string</str>
<str name="config-file">elevate.xml</str>
</searchComponent>
<requestHandler name="/elevate" class="solr.SearchHandler"
startup="lazy">
<lst name="defaults">
<str name="echoParams">explicit</str>
</lst>
<arr name="last-components">
<str>elevator</str>
</arr>
</requestHandler>
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler" />
<requestHandler name="/analysis" class="solr.AnalysisRequestHandler" />
<requestHandler name="/update/csv" class="solr.CSVRequestHandler"
startup="lazy" />
<requestHandler name="/admin/"
class="org.apache.solr.handler.admin.AdminHandlers" />
<requestHandler name="/admin/ping" class="PingRequestHandler">
<lst name="defaults">
<str name="qt">standard</str>
<str name="q">solrpingquery</str>
<str name="echoParams">all</str>
</lst>
</requestHandler>
<requestHandler name="/debug/dump" class="solr.DumpRequestHandler" >
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="echoHandler">true</str>
</lst>
</requestHandler>
<highlighting>
<fragmenter name="gap" class="org.apache.solr.highlight.GapFragmenter"
default="true">
<lst name="defaults">
<int name="hl.fragsize">100</int>
</lst>
</fragmenter>
<fragmenter name="regex"
class="org.apache.solr.highlight.RegexFragmenter">
<lst name="defaults">
<int name="hl.fragsize">70</int>
<float name="hl.regex.slop">0.5</float>
<str name="hl.regex.pattern">[-\w ,/\n\"']{20,200}</str>
</lst>
</fragmenter>
<formatter name="html" class="org.apache.solr.highlight.HtmlFormatter"
default="true">
<lst name="defaults">
<str name="hl.simple.pre"><![CDATA[<em>]]></str>
<str name="hl.simple.post"><![CDATA[</em>]]></str>
</lst>
</formatter>
</highlighting>
<queryResponseWriter name="xslt"
class="org.apache.solr.request.XSLTResponseWriter">
<int name="xsltCacheLifetimeSeconds">5</int>
</queryResponseWriter>
<admin>
<defaultQuery>solr</defaultQuery>
</admin>
</config>
--
View this message in context: http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25912727.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr/Lucene keeps eating up memory while idling
Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 14, 2009, at 12:26 PM, nonrenewable wrote:
>
> I'm curious why this is occurring and whether i can prevent it. This
> is my
> scenario:
>
> Locally I have an idle running solr 1.3 service using lucene 2.4.1
> which has
> an index of ~330K documents containing ~10 fields each(total size
> ~12GB).
Did I read that right? 330K docs == 12 GB index.
> Currently I've turned off all caching, lazy field loading, however i
> do have
> facet fields set for some request handlers.
>
> What i'm seeing is heap space usage increasing by ~1.2MB per 2 sec (by
> java.lang.String objects). I'm assuming they're being used by lucene
> but i
> may be wrong about that, since i have no actual data to confirm it.
> Why
> exactly is this happening, considering no requests are being serviced?
> Shouldn't the memory usage stabilise with a certain set of
> information and
> only be affected on requests? Additionally there is a full GC every
> half
> hour, which seems very unreasonable on a machine that isn't actually
> being
> used as a service.
Can you share the Solr logs and/or your config? Is this happening
around a commit or some warming process? After startup, with no
requests hitting it and no warming/commits/indexing, I don't see why
it would be growing. Do you have custom code?
>
> I really hope there's just a certain setting that i've overlooked,
> or a
> concept i'm not understanding because otherwise this behaviour seems
> very
> unreasonable...
>
> Thanks beforehand,
> Tony
> --
> View this message in context: http://www.nabble.com/Solr-Lucene-keeps-eating-up-memory-while-idling-tp25894357p25894357.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search