You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by rohit <ro...@gmail.com> on 2015/08/11 01:05:17 UTC
SOLR Physical Memory leading to OOM
Hi All,
I have just started with a new project and using solrCloud and during my
performance testing, i have been finding OOM issues. The thing which i
notice more is physical memory keeps on increasing and never comes to
original state.
I'm indexing 10 million documents and have 4 nodes as leader and 4 replicas.
I have updated heap memory to 4 GB out of total 8 GB.
Currently using StandardDirectoryFactory and
<autoCommit>
<maxDocs>25000</maxDocs>
<maxTime>${solr.autoCommit.maxTime:1000000}</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxDocs>25000</maxDocs>
<maxTime>${solr.autoSoftCommit.maxTime:1000000}</maxTime>
</autoSoftCommit>
Can anyone help to find where im going wrong.
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-Physical-Memory-leading-to-OOM-tp4222299.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOLR Physical Memory leading to OOM
Posted by rohit <ro...@gmail.com>.
Thanks Shawn a lot !!.
Just wanted to clarify we have solrCloud so when doing my testing its not a
single server where im hitting. I have multiple servers. At a time we have 4
leaders n 4 replicas which are communicated using zookeeper.
So, in total we have 8 servers and zookeeper is install on 5 of them.
As per your other article , we are planning to move from 1.7 java either to
1.7.0_72-b14 or java 8
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-Physical-Memory-leading-to-OOM-tp4222299p4222434.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOLR Physical Memory leading to OOM
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/10/2015 7:07 PM, rohit wrote:
> Thanks Shawn. I was looking at SOLR Admin UI and also using top command on
> server.
The amount of free memory shown by tools like that is not a very good
way to determine what's happening with your memory. As I said before,
it's completely normal for the OS to utilize almost all of your physical
memory, even if your programs only require a fraction of it. It is not
a meaningful metric for success.
> Im running a endurance for 4 hr with 50/sec TPS and i see the physical
> memory keeps on increasing during the time and if we have schedule delta
> import during that time frame which can import upto 4 million docs. After
> the import I see again the memory increases and their comes a point when
> their is no more memory left which in turn leads to OOM.
If you're hitting OOM, then you need to increase the heap size. Solr is
requiring more memory than you have assigned to the heap. This will
reduce the amount of memory available for the OS disk cache, which may
reduce performance.
There may be ways you can reduce heap usage by adjusting your
configuration or the way you use Solr.
http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
There is other good information on that page. I would encourage you to
go to the top of the page and read all of it.
50 queries per second is a lot of load for a single server. I would
only expect to see success with that many queries per second if the
index fits entirely into the OS disk cache. It may also be necessary
for the index to be relatively small.
> I have seen one more thing if their is no activity on server no import , no
> search going. I have not seen the memory coming down from the state which
> was created after test.
In general, once Java grabs memory, it doesn't let it go. As I
previously mentioned, it cannot grab more than you ask, plus some
overhead. The overhead may be a few hundred mb, which is not very much
when you're talking about multiple gigabytes.
> Couple of things to notice:
>
> 1. We are storing data and indexing also. (not sure if that is causing
> problem).
> 2. Is 8 GB enuf for 10 million or more data to index.
> 3. We have custom handler which extend solr handlers to return data when
> client calls solr handler.
I couldn't tell you whether 8GB is enough for 10 million documents.
That depends on what's in those documents, what your schema.xml says,
how you query Solr, and a few other factors. Even if you tell me the
answers to these questions, I *still* may not be able to say whether
it's enough. I *might* be able to tell you that it's NOT enough,
though. The only way to be absolutely sure is to prototype -- actually
try it out.
https://lucidworks.com/blog/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
8GB of RAM is a *very* small system in the world of Solr. My systems
have 64GB of RAM, and I frequently wish that was 256GB. My indexes are
somewhat larger than yours, though.
Thanks,
Shawn
Re: SOLR Physical Memory leading to OOM
Posted by rohit <ro...@gmail.com>.
Thanks Shawn. I was looking at SOLR Admin UI and also using top command on
server.
Im running a endurance for 4 hr with 50/sec TPS and i see the physical
memory keeps on increasing during the time and if we have schedule delta
import during that time frame which can import upto 4 million docs. After
the import I see again the memory increases and their comes a point when
their is no more memory left which in turn leads to OOM.
I have seen one more thing if their is no activity on server no import , no
search going. I have not seen the memory coming down from the state which
was created after test.
Couple of things to notice:
1. We are storing data and indexing also. (not sure if that is causing
problem).
2. Is 8 GB enuf for 10 million or more data to index.
3. We have custom handler which extend solr handlers to return data when
client calls solr handler.
Let me know, if we are missing configuration pieces.
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-Physical-Memory-leading-to-OOM-tp4222299p4222314.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOLR Physical Memory leading to OOM
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/10/2015 5:05 PM, rohit wrote:
> I have just started with a new project and using solrCloud and during
> my performance testing, i have been finding OOM issues. The thing
> which i notice more is physical memory keeps on increasing and never
> comes to original state.
>
> I'm indexing 10 million documents and have 4 nodes as leader and 4
> replicas. I have updated heap memory to 4 GB out of total 8 GB.
>
> Currently using StandardDirectoryFactory and
>
> <autoCommit> <maxDocs>25000</maxDocs>
> <maxTime>${solr.autoCommit.maxTime:1000000}</maxTime>
> <openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit>
> <maxDocs>25000</maxDocs>
> <maxTime>${solr.autoSoftCommit.maxTime:1000000}</maxTime>
> </autoSoftCommit> Can anyone help to find where im going wrong.
If you've told java that the max heap is 4GB, then Java cannot use more
than 4GB plus a very small amount extra (probably several megabytes) as
overhead. If more is used, then that's a bug in *Java* ... not Solr.
What numbers are you looking at which show a memory problem and where
are you looking at them?
It is 100 percent completely normal for a computer to use all the
physical memory. This is simply how a modern operating system works.
https://en.wikipedia.org/wiki/Page_cache
FYI, you should use the NRTCachingDirectoryFactory with 4.0 and later.
This should be the default in Solr's examples.
Thanks,
Shawn