You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Damien Kamerman <da...@gmail.com> on 2017/05/02 04:52:49 UTC
Suggester uses lots of 'Page cache' memory
Hi all,
I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each replica
uses about 14GB disk usage. I'm using Solaris 11 and I see the 'Page cache'
grow by about 7GB for each suggester replica I build. The suggester index
itself is very small. The 'Page cache' memory is freed when the node is
stopped.
I guess the Suggester component is mmap'ing the entire Lucene index into
memory and holding it? Is this expected behavior? Is there a workaround?
I use this command to build the suggester for just the replica
'target1_shard1_replica1':
curl "
http://localhost:8983/solr/collection1/suggest?suggest.dictionary=mySuggester&suggest.build=true&shards=localhost:8983/solr/target1_shard1_replica1
"
BTW: Without the 'shards' param the distributed request will randomly hit
half the replicas.
From my solrconfig.xml:
<searchComponent name="suggest" class="solr.SuggestComponent">
<lst name="suggester">
<str name="name">mySuggester</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="indexPath">mySuggester</str>
<str name="dictionaryImpl">DocumentDictionaryFactory</str>
<str name="field">mySuggest</str>
<str name="contextField">x</str>
<str name="suggestAnalyzerFieldType">suggestTypeLc</str>
<str name="buildOnStartup">false</str>
</lst>
</searchComponent>
Cheers,
Damien.
Re: Suggester uses lots of 'Page cache' memory
Posted by Damien Kamerman <da...@gmail.com>.
Memory/cache aside, the fundamental Solr issue is that the Suggester build
operation will read the entire index, even though very few docs have the
relevant fields.
Is there a way to set a 'fq' on the Suggester build?
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:135)
at
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.document(CompressingStoredFieldsReader.java:560)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:576)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:583)
at org.apache.lucene.index.CodecReader.document(CodecReader.java:88)
at
org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:411)
at
org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:411)
at
org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:118)
at
org.apache.lucene.index.IndexReader.document(IndexReader.java:381)
at
org.apache.lucene.search.suggest.DocumentDictionary$DocumentInputIterator.next(DocumentDictionary.java:165)
at
org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.build(AnalyzingInfixSuggester.java:300)
- locked <0x00000004b8f29260> (a java.lang.Object)
at org.apache.lucene.search.suggest.Lookup.build(Lookup.java:190)
at
org.apache.solr.spelling.suggest.SolrSuggester.build(SolrSuggester.java:178)
at
org.apache.solr.handler.component.SuggestComponent.prepare(SuggestComponent.java:179)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:269)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2299)
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
On 3 May 2017 at 12:47, Damien Kamerman <da...@gmail.com> wrote:
> Thanks Shawn, I'll have to look closer into this.
>
> On 3 May 2017 at 12:10, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 5/2/2017 6:46 PM, Damien Kamerman wrote:
>> > Shalin, yes I think it's a case of the Suggester build hitting the index
>> > all at once. I'm thinking it's hitting all docs, even the ones without
>> > fields relevant to the suggester.
>> >
>> > Shawn, I am using ZFS, though I think it's comparable to other setups.
>> > mmap() should still be faster, while the ZFS ARC cache may prefer more
>> > memory that other OS disk caches.
>> >
>> > So, it sounds like I enough memory/swap to hold the entire index. When
>> will
>> > the memory be released? On a commit?
>> > https://lucene.apache.org/core/6_5_0/core/org/apache/lucene/
>> store/MMapDirectory.html
>> > talks about a bug on the close().
>>
>> What I'm going to describe below is how things *normally* work on most
>> operating systems (think Linux or Windows) with most filesystems. If
>> ZFS is different, and it sounds like it might be, then that's something
>> for you to discuss with Oracle.
>>
>> Normally, MMap doesn't *allocate* any memory -- so there's nothing to
>> release later. It asks the operating system to map the file's contents
>> to a section of virtual memory, and then the program accesses that
>> memory block directly.
>>
>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>
>> A typical OS takes care of translating accesses to MMap virtual memory
>> into disk accesses, and uses available system memory to cache the data
>> that's read so a subsequent access of the same data is super fast.
>>
>> On most operating systems, memory in the disk cache is always available
>> to programs that request it for an allocation.
>>
>> ZFS uses a completely separate piece of memory for caching -- the ARC
>> cache. I do not know if the OS is able to release memory from that
>> cache when a program requests it. My experience with ZFS on Linux (not
>> with Solr) suggests that the ARC cache holds onto memory a lot tighter
>> than the standard OS disk cache. ZFS on Solaris might be a different
>> animal, though.
>>
>> I'm finding conflicting information regarding MMap problems on ZFS.
>> Some sources say that memory usage is doubled (data in both the standard
>> page cache and the arc cache), some say that this is not a general
>> problem. This is probably a question for Oracle to answer.
>>
>> You don't want to count swap space when looking at how much memory you
>> have. Swap performance is REALLY bad.
>>
>> Thanks,
>> Shawn
>>
>>
>
Re: Suggester uses lots of 'Page cache' memory
Posted by Damien Kamerman <da...@gmail.com>.
Thanks Shawn, I'll have to look closer into this.
On 3 May 2017 at 12:10, Shawn Heisey <ap...@elyograg.org> wrote:
> On 5/2/2017 6:46 PM, Damien Kamerman wrote:
> > Shalin, yes I think it's a case of the Suggester build hitting the index
> > all at once. I'm thinking it's hitting all docs, even the ones without
> > fields relevant to the suggester.
> >
> > Shawn, I am using ZFS, though I think it's comparable to other setups.
> > mmap() should still be faster, while the ZFS ARC cache may prefer more
> > memory that other OS disk caches.
> >
> > So, it sounds like I enough memory/swap to hold the entire index. When
> will
> > the memory be released? On a commit?
> > https://lucene.apache.org/core/6_5_0/core/org/apache/
> lucene/store/MMapDirectory.html
> > talks about a bug on the close().
>
> What I'm going to describe below is how things *normally* work on most
> operating systems (think Linux or Windows) with most filesystems. If
> ZFS is different, and it sounds like it might be, then that's something
> for you to discuss with Oracle.
>
> Normally, MMap doesn't *allocate* any memory -- so there's nothing to
> release later. It asks the operating system to map the file's contents
> to a section of virtual memory, and then the program accesses that
> memory block directly.
>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> A typical OS takes care of translating accesses to MMap virtual memory
> into disk accesses, and uses available system memory to cache the data
> that's read so a subsequent access of the same data is super fast.
>
> On most operating systems, memory in the disk cache is always available
> to programs that request it for an allocation.
>
> ZFS uses a completely separate piece of memory for caching -- the ARC
> cache. I do not know if the OS is able to release memory from that
> cache when a program requests it. My experience with ZFS on Linux (not
> with Solr) suggests that the ARC cache holds onto memory a lot tighter
> than the standard OS disk cache. ZFS on Solaris might be a different
> animal, though.
>
> I'm finding conflicting information regarding MMap problems on ZFS.
> Some sources say that memory usage is doubled (data in both the standard
> page cache and the arc cache), some say that this is not a general
> problem. This is probably a question for Oracle to answer.
>
> You don't want to count swap space when looking at how much memory you
> have. Swap performance is REALLY bad.
>
> Thanks,
> Shawn
>
>
Re: Suggester uses lots of 'Page cache' memory
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/2/2017 6:46 PM, Damien Kamerman wrote:
> Shalin, yes I think it's a case of the Suggester build hitting the index
> all at once. I'm thinking it's hitting all docs, even the ones without
> fields relevant to the suggester.
>
> Shawn, I am using ZFS, though I think it's comparable to other setups.
> mmap() should still be faster, while the ZFS ARC cache may prefer more
> memory that other OS disk caches.
>
> So, it sounds like I enough memory/swap to hold the entire index. When will
> the memory be released? On a commit?
> https://lucene.apache.org/core/6_5_0/core/org/apache/lucene/store/MMapDirectory.html
> talks about a bug on the close().
What I'm going to describe below is how things *normally* work on most
operating systems (think Linux or Windows) with most filesystems. If
ZFS is different, and it sounds like it might be, then that's something
for you to discuss with Oracle.
Normally, MMap doesn't *allocate* any memory -- so there's nothing to
release later. It asks the operating system to map the file's contents
to a section of virtual memory, and then the program accesses that
memory block directly.
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
A typical OS takes care of translating accesses to MMap virtual memory
into disk accesses, and uses available system memory to cache the data
that's read so a subsequent access of the same data is super fast.
On most operating systems, memory in the disk cache is always available
to programs that request it for an allocation.
ZFS uses a completely separate piece of memory for caching -- the ARC
cache. I do not know if the OS is able to release memory from that
cache when a program requests it. My experience with ZFS on Linux (not
with Solr) suggests that the ARC cache holds onto memory a lot tighter
than the standard OS disk cache. ZFS on Solaris might be a different
animal, though.
I'm finding conflicting information regarding MMap problems on ZFS.
Some sources say that memory usage is doubled (data in both the standard
page cache and the arc cache), some say that this is not a general
problem. This is probably a question for Oracle to answer.
You don't want to count swap space when looking at how much memory you
have. Swap performance is REALLY bad.
Thanks,
Shawn
Re: Suggester uses lots of 'Page cache' memory
Posted by Damien Kamerman <da...@gmail.com>.
Shalin, yes I think it's a case of the Suggester build hitting the index
all at once. I'm thinking it's hitting all docs, even the ones without
fields relevant to the suggester.
Shawn, I am using ZFS, though I think it's comparable to other setups.
mmap() should still be faster, while the ZFS ARC cache may prefer more
memory that other OS disk caches.
So, it sounds like I enough memory/swap to hold the entire index. When will
the memory be released? On a commit?
https://lucene.apache.org/core/6_5_0/core/org/apache/lucene/store/MMapDirectory.html
talks about a bug on the close().
On 2 May 2017 at 23:07, Shawn Heisey <ap...@elyograg.org> wrote:
> On 5/1/2017 10:52 PM, Damien Kamerman wrote:
> > I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each
> > replica uses about 14GB disk usage. I'm using Solaris 11 and I see the
> > 'Page cache' grow by about 7GB for each suggester replica I build. The
> > suggester index itself is very small. The 'Page cache' memory is freed
> > when the node is stopped. I guess the Suggester component is mmap'ing
> > the entire Lucene index into memory and holding it? Is this expected
> > behavior? Is there a workaround?
>
> I found the following. The last comment on the answer, the one about
> mmap causing double-buffering with ZFS, is possibly relevant:
>
> https://serverfault.com/a/270604
>
> What filesystem are your indexes on? If it's ZFS, it could completely
> explain the behavior. If it's not ZFS, then the only part of it that I
> cannot explain is the fact that the page cache is freed when Solr stops.
>
> If this double-buffering actually means that the memory is allocated
> twice, then I think that ZFS is probably the wrong filesystem to run
> Solr on, unless you have a LOT of spare memory. You could try changing
> the directory factory to one that doesn't use MMAP, but the suggester
> index factory probably cannot be easily changed. This is too bad --
> normally MMAP is far more efficient than "standard" filesystem access.
>
> I could be reaching completely wrong conclusions based on the limited
> research I did.
>
> Thanks,
> Shawn
>
>
Re: Suggester uses lots of 'Page cache' memory
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/1/2017 10:52 PM, Damien Kamerman wrote:
> I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each
> replica uses about 14GB disk usage. I'm using Solaris 11 and I see the
> 'Page cache' grow by about 7GB for each suggester replica I build. The
> suggester index itself is very small. The 'Page cache' memory is freed
> when the node is stopped. I guess the Suggester component is mmap'ing
> the entire Lucene index into memory and holding it? Is this expected
> behavior? Is there a workaround?
I found the following. The last comment on the answer, the one about
mmap causing double-buffering with ZFS, is possibly relevant:
https://serverfault.com/a/270604
What filesystem are your indexes on? If it's ZFS, it could completely
explain the behavior. If it's not ZFS, then the only part of it that I
cannot explain is the fact that the page cache is freed when Solr stops.
If this double-buffering actually means that the memory is allocated
twice, then I think that ZFS is probably the wrong filesystem to run
Solr on, unless you have a LOT of spare memory. You could try changing
the directory factory to one that doesn't use MMAP, but the suggester
index factory probably cannot be easily changed. This is too bad --
normally MMAP is far more efficient than "standard" filesystem access.
I could be reaching completely wrong conclusions based on the limited
research I did.
Thanks,
Shawn
Re: Suggester uses lots of 'Page cache' memory
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Tue, May 2, 2017 at 10:22 AM, Damien Kamerman <da...@gmail.com> wrote:
> Hi all,
>
> I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each replica
> uses about 14GB disk usage. I'm using Solaris 11 and I see the 'Page cache'
> grow by about 7GB for each suggester replica I build. The suggester index
> itself is very small. The 'Page cache' memory is freed when the node is
> stopped.
>
> I guess the Suggester component is mmap'ing the entire Lucene index into
> memory and holding it? Is this expected behavior? Is there a workaround?
>
Yes, this is expected. The suggester opens the index using Lucene's
MMapDirectory which ends up memory mapping the index. But the memory
mapped pages can be shared across everyone using the same index which
basically means that the replica's usual index searcher can also use
these pages. If you were not building the suggester index, even then
the replica's search index would have been mmapped but perhaps only on
demand instead of all at once.
> I use this command to build the suggester for just the replica
> 'target1_shard1_replica1':
> curl "
> http://localhost:8983/solr/collection1/suggest?suggest.dictionary=mySuggester&suggest.build=true&shards=localhost:8983/solr/target1_shard1_replica1
> "
>
> BTW: Without the 'shards' param the distributed request will randomly hit
> half the replicas.
Yes, this is a problem. I recently opened
https://issues.apache.org/jira/browse/SOLR-10532 but unfortunately I
don't have the time to fix it soon. Patches are always welcome.
>
> From my solrconfig.xml:
> <searchComponent name="suggest" class="solr.SuggestComponent">
> <lst name="suggester">
> <str name="name">mySuggester</str>
> <str name="lookupImpl">AnalyzingInfixLookupFactory</str>
> <str name="indexPath">mySuggester</str>
> <str name="dictionaryImpl">DocumentDictionaryFactory</str>
> <str name="field">mySuggest</str>
> <str name="contextField">x</str>
> <str name="suggestAnalyzerFieldType">suggestTypeLc</str>
> <str name="buildOnStartup">false</str>
> </lst>
> </searchComponent>
>
> Cheers,
> Damien.
--
Regards,
Shalin Shekhar Mangar.