You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by SOLR4189 <Kl...@yandex.ru> on 2017/02/23 08:51:19 UTC
maxwarmingSearchers and memory leak
We have maxwarmingSearchers set to 2 and field value cache set to initial
size of 64. We saw that by taking a heap dump that our caches consume 70% of
the heap size, by looking into the dump we saw that fieldValueCache has 6
occurences of org.apache.solr.util.concurrentCache.
When we have maxWarmingSearches=2 we would expect to have only 3 (maybe 4
before GC has been launched).
What can it be? We use solr4.10.1
--
View this message in context: http://lucene.472066.n3.nabble.com/maxwarmingSearchers-and-memory-leak-tp4321937.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: maxwarmingSearchers and memory leak
Posted by SOLR4189 <Kl...@yandex.ru>.
1) We've actually got 60 to 80 GB of index on the machine (in the image below
you can see that size of index on the machine 82GB, because all index is in
path /opt/solr):
<http://lucene.472066.n3.nabble.com/file/n4323509/size.jpg>
2) Our commits runs: autoSoftCommit - each 15 minutes and autoHardCommit -
each 30 minutes
and our commits take 10 seconds only
3) ConcurrentLFUCaches (that you saw in the image in the previous message)
aren't filterCaches, they are fieldValueCaches
4) Solr top:
<http://lucene.472066.n3.nabble.com/file/n4323509/top.jpg>
5) We don't know if this related to problem, but all our SOLR servers are
virtual servers.
--
View this message in context: http://lucene.472066.n3.nabble.com/maxwarmingSearchers-and-memory-leak-tp4321937p4323509.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: maxwarmingSearchers and memory leak
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/26/2017 6:40 AM, SOLR4189 wrote:
> Shawn, you are right.
> * OS vendor and version
> CentosOS 6.5
>
> * Java vendor and version
> OpenJDK version 1.8.0_20
> OpenJDK 64-bit Server VM (build 25.20-b23)
>
> * Servlet container used to start Solr.
> Catalina(tomcat7)
>
> * Total amount of memory in the server.
> 30 GB
>
> * Max heap size for Solr.
> 8GB(JVM)
>
> * An idea of exactly what is running on the server.
> On our servers runs solr service only and splunk forwarder
>
> * Total index size and document count being handled by Solr (add up all
> indexes).
> 60GB and 2.6 milion on one shard
You say that you've got 60GB of index, but the screenshot seems to
indicate that you've actually got 160 to 180GB of index on the machine.
With approximately 22GB of memory available for caching (30GB total
minus 8GB for Solr's heap), you don't have enough memory for good
performance. Your commits are probabl taking so long to finish that
additional commits are coming in and trying to open new searchers before
the previous commits are done. Increasing the memory or splitting the
index onto more machines might help performance.
With 2.6 million documents in an index shard, whenever the system
creates a filterCache entry for that shard, it will be 325000 bytes. If
enough of these entries are created, a huge amount of heap memory will
be required. It will not be a memory leak, though.
You've got an early Java 8 release. There have been some memory leaks
in Java itself fixed in later releases. Consider upgrading to the
latest Java 8.
The only thing I can say about the container (tomcat) is that it is an
untested environment. The only container that actually gets tested is
Jetty. It's not very likely that running in Tomcat is the problm, though.
> * A screen shot of a process list sorted by memory usage.
> <http://lucene.472066.n3.nabble.com/file/n4322362/20170226_102812.jpg>
The display for htop is NOT the same as top. If I had wanted htop, that
would have been what I mentioned. The standard top utility shows
everything I was wanting to see. The display for htop can be useful,
and has answered one question, but doesn't contain everything that I was
after.
Can you share a screenshot of the Solr dashboard, and one of the
standard top utility sorted by memory usage?
> * A screenshot showing total system memory allocations
> <http://lucene.472066.n3.nabble.com/file/n4322362/20170226_102007.jpg>
This file is not available. Nabble says "file not found."
Thanks,
Shawn
Re: maxwarmingSearchers and memory leak
Posted by SOLR4189 <Kl...@yandex.ru>.
Shawn, you are right.
* OS vendor and version
CentosOS 6.5
* Java vendor and version
OpenJDK version 1.8.0_20
OpenJDK 64-bit Server VM (build 25.20-b23)
* Servlet container used to start Solr.
Catalina(tomcat7)
* Total amount of memory in the server.
30 GB
* Max heap size for Solr.
8GB(JVM)
* An idea of exactly what is running on the server.
On our servers runs solr service only and splunk forwarder
* Total index size and document count being handled by Solr (add up all
indexes).
60GB and 2.6 milion on one shard
* A screen shot of a process list sorted by memory usage.
<http://lucene.472066.n3.nabble.com/file/n4322362/20170226_102812.jpg>
* A screenshot showing total system memory allocations
<http://lucene.472066.n3.nabble.com/file/n4322362/20170226_102007.jpg>
--
View this message in context: http://lucene.472066.n3.nabble.com/maxwarmingSearchers-and-memory-leak-tp4321937p4322362.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: maxwarmingSearchers and memory leak
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/23/2017 1:51 AM, SOLR4189 wrote:
> We have maxwarmingSearchers set to 2 and field value cache set to
> initial size of 64. We saw that by taking a heap dump that our caches
> consume 70% of the heap size, by looking into the dump we saw that
> fieldValueCache has 6 occurences of
> org.apache.solr.util.concurrentCache. When we have
> maxWarmingSearches=2 we would expect to have only 3 (maybe 4 before GC
> has been launched). What can it be? We use solr4.10.1
There are no specific details here about your setup except for
maxWarmingSearchers (which is at the typical default value) and
fieldValueCache, which is not explicitly configured in any example, and
probably should not be explicitly configured. The version number is not
provided. Heap size is not provided. Installation method is not
provided. Other details about your OS, Solr install, configuration, and
index are not available.
Version 4.10.x can be installed in a variety of ways, most of which are
outside the project's control. 5.0 and later are typically only
installed using the built-in container (jetty) and scripts.
Here's a starting list for additional info that we may need:
* OS vendor and version
* Java vendor and version
* Servlet container used to start Solr.
* Total amount of memory in the server.
* Max heap size for Solr.
* An idea of exactly what is running on the server.
* Total index size and document count being handled by Solr (add up all
indexes).
* A screen shot of a process list sorted by memory usage.
* A screenshot showing total system memory allocations.
Those last two are typically available with one screen on most operating
systems that have the "top" utility, if it allows sorting by memory by
pressing shift-M.
Although a memory leak is certainly possible, I am not aware of any
known problems with memory leaks in 4.10 or any of the current code
branches. Very likely you simply need a larger heap for Solr to do the
work that it has been asked to do. This link may be helpful:
https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
There is no "ConcurrentCache" class in Solr. There is
ConcurrentLRUCache, though. I have no idea how large cache entries are
in the fieldValueCache, though I would expect them to be fairly small.
One cache that typically has very large entry sizes is filterCache.
Each entry in that cache has a byte size equal to the maxDoc value of
the index divided by eight. So an index with one million documents in
it has filterCache entries which are each 125000 bytes. An index with
100 million documents has filterCache entries which are each 12.5
million bytes.
Thanks,
Shawn