You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Samuel García Martínez <sa...@gmail.com> on 2013/08/20 10:24:32 UTC
High memory usage on solr 3.6
Hi all, we are facing a high memory usage in our Solr 3.6 master (not in
the slaves) even during "idle" (non indexing) periods.
container: Tomcat 6.0.29
maxthreads: 1.5k (i think this setting is wrong, 300 would be enough)
solr: solr 3.6.0
setup: multitenant environment with 120+ cores and near 5M docs spread over
all indexes.
uptime: 100+ days
qps on this machine: 10 qps
used heap: 8gb
JVM params:
-Djava.util.logging.config.file=/opt/opensearch/tomcat/conf/logging.properties
-Dmaster=true
-Dmaster.url=""
-Xms5G
-Xmx10G
-XX:MaxPermSize=256m
-XX:SurvivorRatio=5
-XX:NewRatio=2
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=22222
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.endorsed.dirs=/opt/opensearch/tomcat/endorsed
-Dcatalina.base=/opt/opensearch/tomcat
-Dcatalina.home=/opt/opensearch/tomcat
-Djava.io.tmpdir=/opt/opensearch/tomcat/temp
I think we have a high memory usage due to the next report:
char[] 3.1M instances 5.3Gb -----> this is what i'm talking about.
LinkedHashMap$Entry 2M instances 121Mb
String 1.48M 53Mb
Checking for "cg roots" at all these instances, i found that almost all
these references are contained in ISOLatin1AccentFilter.output ->
TokenStreamImpl - Map$Entry -> Map -> CloseableThreadLocal -> TokenizerChain
Do we had to add any CMS param to the JVM params? Or is this a memory leak
due to the ThreadLocal's?
I verified that those char[] didn't belong to FieldCache.default, so this
high memory usage is not due the faceting and high cadinality values.
PS: we reduced the number of threads and memory (and char[] instances)
decreased significantly.
--
Un saludo,
Samuel García.
Re: High memory usage on solr 3.6
Posted by Samuel García Martínez <sa...@gmail.com>.
You were right. I've attached VisualVM to the process and forced a
System.gc(): used memory went down to near 1.8gb.
So, i don't understand VisualVM dump reports. It said that all those char[]
references have a SolrDispatchFilter instance as CG root.
Another example (1M+ references with the exact same size 32792 [default
buffer size]):
https://dl.dropboxusercontent.com/u/4740964/solr_references.png
Can you give any hint on how to read this kind of reference graph?
Thanks!
On Tue, Aug 20, 2013 at 7:20 PM, Samuel García Martínez <
samuelgmartinez@gmail.com> wrote:
> Thanks for the quick answer.
>
> We are experiencing slower indexing speed (x2/x2.5 time consumed) and the
> memory consumed during this process oscilates between 10g (max allowed, so
> JVM performs full gc's) and 8gb, but never goes under 8gb.
>
> I'll try your suggestions and see how it goes.
>
> Thanks!
> El 20/08/2013 13:29, "Erick Erickson" <er...@gmail.com> escribió:
>
> Hmmm, first time I've seen this, but are you seeing a problem
>> or is this just a theoretical concern? There are some
>> references that are held around until GC, these could be
>> perfectly legitimate, unused but un-collected references.
>>
>> What I'd try before worrying much:
>> 1> attach jconsole and press the GC button and see if lots of those
>> go away.
>> 2> _reduce_ the memory allocated to Solr and see if you can get by.
>> In this case, reduce it to, say, 4G. If your memory consumption
>> hits 4G and you continue running, you're probably seeing uncollected
>> memory.
>>
>> Best
>> Erick
>>
>>
>> On Tue, Aug 20, 2013 at 4:24 AM, Samuel García Martínez <
>> samuelgmartinez@gmail.com> wrote:
>>
>> > Hi all, we are facing a high memory usage in our Solr 3.6 master (not in
>> > the slaves) even during "idle" (non indexing) periods.
>> >
>> > container: Tomcat 6.0.29
>> > maxthreads: 1.5k (i think this setting is wrong, 300 would be enough)
>> > solr: solr 3.6.0
>> > setup: multitenant environment with 120+ cores and near 5M docs spread
>> over
>> > all indexes.
>> > uptime: 100+ days
>> > qps on this machine: 10 qps
>> > used heap: 8gb
>> > JVM params:
>> >
>> >
>> -Djava.util.logging.config.file=/opt/opensearch/tomcat/conf/logging.properties
>> > -Dmaster=true
>> > -Dmaster.url=""
>> > -Xms5G
>> > -Xmx10G
>> > -XX:MaxPermSize=256m
>> > -XX:SurvivorRatio=5
>> > -XX:NewRatio=2
>> > -XX:+UseParNewGC
>> > -XX:+UseConcMarkSweepGC
>> > -Dcom.sun.management.jmxremote
>> > -Dcom.sun.management.jmxremote.port=22222
>> > -Dcom.sun.management.jmxremote.authenticate=false
>> > -Dcom.sun.management.jmxremote.ssl=false
>> > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> > -Djava.endorsed.dirs=/opt/opensearch/tomcat/endorsed
>> > -Dcatalina.base=/opt/opensearch/tomcat
>> > -Dcatalina.home=/opt/opensearch/tomcat
>> > -Djava.io.tmpdir=/opt/opensearch/tomcat/temp
>> >
>> >
>> > I think we have a high memory usage due to the next report:
>> > char[] 3.1M instances 5.3Gb -----> this is what i'm talking about.
>> > LinkedHashMap$Entry 2M instances 121Mb
>> > String 1.48M 53Mb
>> >
>> >
>> > Checking for "cg roots" at all these instances, i found that almost all
>> > these references are contained in ISOLatin1AccentFilter.output ->
>> > TokenStreamImpl - Map$Entry -> Map -> CloseableThreadLocal ->
>> > TokenizerChain
>> >
>> > Do we had to add any CMS param to the JVM params? Or is this a memory
>> leak
>> > due to the ThreadLocal's?
>> >
>> > I verified that those char[] didn't belong to FieldCache.default, so
>> this
>> > high memory usage is not due the faceting and high cadinality values.
>> >
>> > PS: we reduced the number of threads and memory (and char[] instances)
>> > decreased significantly.
>> > --
>> > Un saludo,
>> > Samuel García.
>> >
>>
>
--
Un saludo,
Samuel García.
Re: High memory usage on solr 3.6
Posted by Samuel García Martínez <sa...@gmail.com>.
Thanks for the quick answer.
We are experiencing slower indexing speed (x2/x2.5 time consumed) and the
memory consumed during this process oscilates between 10g (max allowed, so
JVM performs full gc's) and 8gb, but never goes under 8gb.
I'll try your suggestions and see how it goes.
Thanks!
El 20/08/2013 13:29, "Erick Erickson" <er...@gmail.com> escribió:
> Hmmm, first time I've seen this, but are you seeing a problem
> or is this just a theoretical concern? There are some
> references that are held around until GC, these could be
> perfectly legitimate, unused but un-collected references.
>
> What I'd try before worrying much:
> 1> attach jconsole and press the GC button and see if lots of those
> go away.
> 2> _reduce_ the memory allocated to Solr and see if you can get by.
> In this case, reduce it to, say, 4G. If your memory consumption
> hits 4G and you continue running, you're probably seeing uncollected
> memory.
>
> Best
> Erick
>
>
> On Tue, Aug 20, 2013 at 4:24 AM, Samuel García Martínez <
> samuelgmartinez@gmail.com> wrote:
>
> > Hi all, we are facing a high memory usage in our Solr 3.6 master (not in
> > the slaves) even during "idle" (non indexing) periods.
> >
> > container: Tomcat 6.0.29
> > maxthreads: 1.5k (i think this setting is wrong, 300 would be enough)
> > solr: solr 3.6.0
> > setup: multitenant environment with 120+ cores and near 5M docs spread
> over
> > all indexes.
> > uptime: 100+ days
> > qps on this machine: 10 qps
> > used heap: 8gb
> > JVM params:
> >
> >
> -Djava.util.logging.config.file=/opt/opensearch/tomcat/conf/logging.properties
> > -Dmaster=true
> > -Dmaster.url=""
> > -Xms5G
> > -Xmx10G
> > -XX:MaxPermSize=256m
> > -XX:SurvivorRatio=5
> > -XX:NewRatio=2
> > -XX:+UseParNewGC
> > -XX:+UseConcMarkSweepGC
> > -Dcom.sun.management.jmxremote
> > -Dcom.sun.management.jmxremote.port=22222
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> > -Djava.endorsed.dirs=/opt/opensearch/tomcat/endorsed
> > -Dcatalina.base=/opt/opensearch/tomcat
> > -Dcatalina.home=/opt/opensearch/tomcat
> > -Djava.io.tmpdir=/opt/opensearch/tomcat/temp
> >
> >
> > I think we have a high memory usage due to the next report:
> > char[] 3.1M instances 5.3Gb -----> this is what i'm talking about.
> > LinkedHashMap$Entry 2M instances 121Mb
> > String 1.48M 53Mb
> >
> >
> > Checking for "cg roots" at all these instances, i found that almost all
> > these references are contained in ISOLatin1AccentFilter.output ->
> > TokenStreamImpl - Map$Entry -> Map -> CloseableThreadLocal ->
> > TokenizerChain
> >
> > Do we had to add any CMS param to the JVM params? Or is this a memory
> leak
> > due to the ThreadLocal's?
> >
> > I verified that those char[] didn't belong to FieldCache.default, so this
> > high memory usage is not due the faceting and high cadinality values.
> >
> > PS: we reduced the number of threads and memory (and char[] instances)
> > decreased significantly.
> > --
> > Un saludo,
> > Samuel García.
> >
>
Re: High memory usage on solr 3.6
Posted by Erick Erickson <er...@gmail.com>.
Hmmm, first time I've seen this, but are you seeing a problem
or is this just a theoretical concern? There are some
references that are held around until GC, these could be
perfectly legitimate, unused but un-collected references.
What I'd try before worrying much:
1> attach jconsole and press the GC button and see if lots of those
go away.
2> _reduce_ the memory allocated to Solr and see if you can get by.
In this case, reduce it to, say, 4G. If your memory consumption
hits 4G and you continue running, you're probably seeing uncollected
memory.
Best
Erick
On Tue, Aug 20, 2013 at 4:24 AM, Samuel García Martínez <
samuelgmartinez@gmail.com> wrote:
> Hi all, we are facing a high memory usage in our Solr 3.6 master (not in
> the slaves) even during "idle" (non indexing) periods.
>
> container: Tomcat 6.0.29
> maxthreads: 1.5k (i think this setting is wrong, 300 would be enough)
> solr: solr 3.6.0
> setup: multitenant environment with 120+ cores and near 5M docs spread over
> all indexes.
> uptime: 100+ days
> qps on this machine: 10 qps
> used heap: 8gb
> JVM params:
>
> -Djava.util.logging.config.file=/opt/opensearch/tomcat/conf/logging.properties
> -Dmaster=true
> -Dmaster.url=""
> -Xms5G
> -Xmx10G
> -XX:MaxPermSize=256m
> -XX:SurvivorRatio=5
> -XX:NewRatio=2
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=22222
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djava.endorsed.dirs=/opt/opensearch/tomcat/endorsed
> -Dcatalina.base=/opt/opensearch/tomcat
> -Dcatalina.home=/opt/opensearch/tomcat
> -Djava.io.tmpdir=/opt/opensearch/tomcat/temp
>
>
> I think we have a high memory usage due to the next report:
> char[] 3.1M instances 5.3Gb -----> this is what i'm talking about.
> LinkedHashMap$Entry 2M instances 121Mb
> String 1.48M 53Mb
>
>
> Checking for "cg roots" at all these instances, i found that almost all
> these references are contained in ISOLatin1AccentFilter.output ->
> TokenStreamImpl - Map$Entry -> Map -> CloseableThreadLocal ->
> TokenizerChain
>
> Do we had to add any CMS param to the JVM params? Or is this a memory leak
> due to the ThreadLocal's?
>
> I verified that those char[] didn't belong to FieldCache.default, so this
> high memory usage is not due the faceting and high cadinality values.
>
> PS: we reduced the number of threads and memory (and char[] instances)
> decreased significantly.
> --
> Un saludo,
> Samuel García.
>