Memory + WeakIdentityMap

[Harish Agarwal <ha...@gmail.com>]

I'm transitioning my index from a 3.x version to >4.6.  I'm running a large
heap (20G), primarily to accomodate a large facet cache (~5G), but have
been able to run it on 3.x stably.

On 4.6.0 after stress testing I'm finding that all of my shards are
spending all of their time in GC.  After taking a heap dump and analyzing,
it appears that org.apache.lucene.util.WeakIdentityMap is using many Gs of
memory.  Does anyone have any insight into which Solr component(s) use this
and whether this kind of memory consumption is to be expected?

Thank You,
-Harish

Re: Memory + WeakIdentityMap

[jim ferenczi <ji...@gmail.com>]

Hi,
If you are not on windows, you can try to disable the tracking of clones in
the MMapDirectory by setting unmap to false in your solrconfig.xml:


*<directoryFactory name="DirectoryFactory"
class="solr.MMapDirectoryFactory}"> *

*  <bool name="unmap">false</bool>*
*</directoryFactory>*
The MMapDirectory keeps track of all clones in a weak map and forces the
unmapping of the buffers on close. This was added because on Windows
mmapped files cannot be modified or deleted. If unmap is false the weak map
is not created and the weak references you see in your heap should disapear
as well.
You can find more informations here:
https://issues.apache.org/jira/browse/LUCENE-4740

Thanks,
Jim






2014-03-21 6:56 GMT+01:00 Shawn Heisey <so...@elyograg.org>:

> On 3/20/2014 6:54 PM, Harish Agarwal wrote:
> > I'm transitioning my index from a 3.x version to >4.6.  I'm running a
> large
> > heap (20G), primarily to accomodate a large facet cache (~5G), but have
> > been able to run it on 3.x stably.
> >
> > On 4.6.0 after stress testing I'm finding that all of my shards are
> > spending all of their time in GC.  After taking a heap dump and
> analyzing,
> > it appears that org.apache.lucene.util.WeakIdentityMap is using many Gs
> of
> > memory.  Does anyone have any insight into which Solr component(s) use
> this
> > and whether this kind of memory consumption is to be expected?
>
> I can't really say what WeakIdentityMap is doing.  I can trace the only
> usage in Lucene to MMapDirectory, but it doesn't make a lot of sense for
> this to use a lot of memory, unless this is the source of the memory
> misreporting that Java 7 seems to do with MMap.  See this message in a
> recent thread on this mailing list:
>
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201403.mbox/%3C53285CA1.9000300@elyograg.org%3E
>
> If you have a lot of facets, one approach for performance is to use
> facet.method=enum so that your Java heap does not need to be super large.
>
> This does not actually reduce the overall system memory requirements.
> It just shifts the responsibility for caching to the operating system
> instead of Solr, and requires that you have enough memory to put a
> majority of the index into the OS disk cache.  Ideally, there would be
> enough RAM for the entire index to fit.
>
> http://wiki.apache.org/solr/SolrPerformanceProblems
>
> Another option for facet memory optimization is docValues.  One caveat:
> It is my understanding that the docValues content is the same as a
> stored field.  Depending on your schema definition, this may be
> different than the indexed values that facets normally use.  The
> docValues feature also helps with sorting.
>
> Thanks,
> Shawn
>
>

Re: Memory + WeakIdentityMap

[Shawn Heisey <so...@elyograg.org>]

On 3/20/2014 6:54 PM, Harish Agarwal wrote:
> I'm transitioning my index from a 3.x version to >4.6.  I'm running a large
> heap (20G), primarily to accomodate a large facet cache (~5G), but have
> been able to run it on 3.x stably.
> 
> On 4.6.0 after stress testing I'm finding that all of my shards are
> spending all of their time in GC.  After taking a heap dump and analyzing,
> it appears that org.apache.lucene.util.WeakIdentityMap is using many Gs of
> memory.  Does anyone have any insight into which Solr component(s) use this
> and whether this kind of memory consumption is to be expected?

I can't really say what WeakIdentityMap is doing.  I can trace the only
usage in Lucene to MMapDirectory, but it doesn't make a lot of sense for
this to use a lot of memory, unless this is the source of the memory
misreporting that Java 7 seems to do with MMap.  See this message in a
recent thread on this mailing list:

http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201403.mbox/%3C53285CA1.9000300@elyograg.org%3E

If you have a lot of facets, one approach for performance is to use
facet.method=enum so that your Java heap does not need to be super large.

This does not actually reduce the overall system memory requirements.
It just shifts the responsibility for caching to the operating system
instead of Solr, and requires that you have enough memory to put a
majority of the index into the OS disk cache.  Ideally, there would be
enough RAM for the entire index to fit.

http://wiki.apache.org/solr/SolrPerformanceProblems

Another option for facet memory optimization is docValues.  One caveat:
It is my understanding that the docValues content is the same as a
stored field.  Depending on your schema definition, this may be
different than the indexed values that facets normally use.  The
docValues feature also helps with sorting.

Thanks,
Shawn