You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Zheng Lin Edwin Yeo <ed...@gmail.com> on 2015/12/30 03:35:50 UTC
Memory Usage increases by a lot during and after optimization .
Hi,
I am facing a situation, when I do an optimization by clicking on the
"Optimized" button on the Solr Admin Overview UI, the memory usage of the
server increases gradually, until it reaches near the maximum memory
available. There is 64GB of memory available in the server.
Even after the optimized is completed, the memory usage stays near the 100%
range, and could not be reduced until I stop Solr. Why could this be
happening?
Also, I don't think the optimization is completed, as the admin page says
the index is not optimized again after I go back to the Overview page, even
though I did not do any updates to the index.
I am using Solr 5.3.0, with 1 shard and 2 replica. My index size is 183GB.
Regards,
Edwin
Re: Memory Usage increases by a lot during and after optimization .
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Shawn,
I am using Java Version 8 Update 45 (build 1.8.0_45-b15). It is a 64-bit
Java.
Thank you.
Regards,
Edwin
On 8 January 2016 at 00:15, Shawn Heisey <ap...@elyograg.org> wrote:
> On 1/7/2016 12:53 AM, Zheng Lin Edwin Yeo wrote:
> >> Subtracting SHR from RES (or in your case, Shareable from Working)
> >> reveals the actual memory being used, and I believe you can see this
> >> actual number in the Private column, which is approximately the
> >> difference between Working and Shareable. If I'm right, this means that
> >> the actual memory usage is almost 14GB lower than Windows is reporting.
> > Does this means that sometimes when I see the high memory usage (can be
> up
> > to 100%), it is just a memory reporting error by Windows, but Solr is
> > working exactly as it should?
>
> I believe so, yes. It should be completely impossible for Java to
> allocate more memory than you ask it to ... and I believe that Java is
> acting correctly in this regard, but the reporting is wrong.
>
> I do not know whether anything can be done about the reporting problem.
> The fact that it happens with Java on two different operating systems
> (that I know of) suggests that the problem is in Java itself.
>
> I suspect that it is related to the fact that the index files are
> accessed via MMap, and part of that virtual memory is misreported as
> shared memory. I am subscribed to the hotspot mailing list for
> openjdk. I don't know whether the message will be on-topic for the
> list, but I will ask about it there. For this message I will need to
> know the precise version of Java you are using, including whether it's
> 32 or 64-bit.
>
> Thanks,
> Shawn
>
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/7/2016 12:53 AM, Zheng Lin Edwin Yeo wrote:
>> Subtracting SHR from RES (or in your case, Shareable from Working)
>> reveals the actual memory being used, and I believe you can see this
>> actual number in the Private column, which is approximately the
>> difference between Working and Shareable. If I'm right, this means that
>> the actual memory usage is almost 14GB lower than Windows is reporting.
> Does this means that sometimes when I see the high memory usage (can be up
> to 100%), it is just a memory reporting error by Windows, but Solr is
> working exactly as it should?
I believe so, yes. It should be completely impossible for Java to
allocate more memory than you ask it to ... and I believe that Java is
acting correctly in this regard, but the reporting is wrong.
I do not know whether anything can be done about the reporting problem.
The fact that it happens with Java on two different operating systems
(that I know of) suggests that the problem is in Java itself.
I suspect that it is related to the fact that the index files are
accessed via MMap, and part of that virtual memory is misreported as
shared memory. I am subscribed to the hotspot mailing list for
openjdk. I don't know whether the message will be on-topic for the
list, but I will ask about it there. For this message I will need to
know the precise version of Java you are using, including whether it's
32 or 64-bit.
Thanks,
Shawn
Re: Memory Usage increases by a lot during and after optimization .
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Shawn,
Thank you for your explanation.
Yes, both of the top two processes are Solr. I have two Solr processes on
one machine now, as the second one is a replica of the first one. In the
future, the plan is to have them on separate machine.
>Subtracting SHR from RES (or in your case, Shareable from Working)
>reveals the actual memory being used, and I believe you can see this
>actual number in the Private column, which is approximately the
>difference between Working and Shareable. If I'm right, this means that
>the actual memory usage is almost 14GB lower than Windows is reporting.
Does this means that sometimes when I see the high memory usage (can be up
to 100%), it is just a memory reporting error by Windows, but Solr is
working exactly as it should?
Regards,
Edwin
On 7 January 2016 at 04:42, Shawn Heisey <ap...@elyograg.org> wrote:
> On 1/5/2016 11:50 PM, Zheng Lin Edwin Yeo wrote:
> > Here is the new screenshot of the Memory tab of the Resource Monitor.
> > https://www.dropbox.com/s/w4bnrb66r16lpx1/Resource%20Monitor.png?dl=0
> >
> > Yes, I found that the value under the "Working Set" column is much higher
> > than the others. Also, the value which I was previously looking at under
> > the Task Manager is under the Private Column here.
> > It says that I have about 14GB of available memory, but the "Free" number
> > is much lower, at 79MB.
>
> You'll probably think I'm nuts, but I believe everything is working
> exactly as it should.
>
> The first two processes, which I assume are Solr processes, show a
> Shareable size near 7GB each. I have seen something similar happen on
> Linux where SHR memory is huge for the Solr process, and when this
> happens, the combination of memory numbers would turn out to be
> impossible, so I think it's a memory reporting bug related to Java, one
> that affects both Linux and Windows.
>
> Subtracting SHR from RES (or in your case, Shareable from Working)
> reveals the actual memory being used, and I believe you can see this
> actual number in the Private column, which is approximately the
> difference between Working and Shareable. If I'm right, this means that
> the actual memory usage is almost 14GB lower than Windows is reporting.
>
> If both of the top processes are Solr, I'm not sure why you have two
> Solr processes on one machine. One Solr instance can handle multiple
> indexes with no problem.
>
> As evidence that I'm not insane, consider the following screenshot, from
> another of my servers:
>
> https://www.dropbox.com/s/64en3sar4cr1ytj/linux-solr-mem-high-shr.png?dl=0
>
> On the screenshot, the solr process shows RES size of 22GB ... which is
> highly unusual, because this Solr install has a max heap of 8GB ... but
> notice that SHR is 13GB. The difference between 22GB and 13GB is 9GB,
> which is much more reasonable, and if we assume that the 22GB is rounded
> up and/or the 13GB is rounded down, then the difference is much closer
> to 8GB. Looking at some other numbers, the "cached" value is 48GB. If
> you add the 48GB cache allocation to the *reported* resident size of
> 22GB for Solr, you get a total of 70GB ... which is more memory than the
> machine even has (64GB). This is why I am sure that when SHR is really
> high on a Java process, it is a memory reporting error.
>
> Thanks,
> Shawn
>
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/5/2016 11:50 PM, Zheng Lin Edwin Yeo wrote:
> Here is the new screenshot of the Memory tab of the Resource Monitor.
> https://www.dropbox.com/s/w4bnrb66r16lpx1/Resource%20Monitor.png?dl=0
>
> Yes, I found that the value under the "Working Set" column is much higher
> than the others. Also, the value which I was previously looking at under
> the Task Manager is under the Private Column here.
> It says that I have about 14GB of available memory, but the "Free" number
> is much lower, at 79MB.
You'll probably think I'm nuts, but I believe everything is working
exactly as it should.
The first two processes, which I assume are Solr processes, show a
Shareable size near 7GB each. I have seen something similar happen on
Linux where SHR memory is huge for the Solr process, and when this
happens, the combination of memory numbers would turn out to be
impossible, so I think it's a memory reporting bug related to Java, one
that affects both Linux and Windows.
Subtracting SHR from RES (or in your case, Shareable from Working)
reveals the actual memory being used, and I believe you can see this
actual number in the Private column, which is approximately the
difference between Working and Shareable. If I'm right, this means that
the actual memory usage is almost 14GB lower than Windows is reporting.
If both of the top processes are Solr, I'm not sure why you have two
Solr processes on one machine. One Solr instance can handle multiple
indexes with no problem.
As evidence that I'm not insane, consider the following screenshot, from
another of my servers:
https://www.dropbox.com/s/64en3sar4cr1ytj/linux-solr-mem-high-shr.png?dl=0
On the screenshot, the solr process shows RES size of 22GB ... which is
highly unusual, because this Solr install has a max heap of 8GB ... but
notice that SHR is 13GB. The difference between 22GB and 13GB is 9GB,
which is much more reasonable, and if we assume that the 22GB is rounded
up and/or the 13GB is rounded down, then the difference is much closer
to 8GB. Looking at some other numbers, the "cached" value is 48GB. If
you add the 48GB cache allocation to the *reported* resident size of
22GB for Solr, you get a total of 70GB ... which is more memory than the
machine even has (64GB). This is why I am sure that when SHR is really
high on a Java process, it is a memory reporting error.
Thanks,
Shawn
Re: Memory Usage increases by a lot during and after optimization .
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Shawn,
Here is the new screenshot of the Memory tab of the Resource Monitor.
https://www.dropbox.com/s/w4bnrb66r16lpx1/Resource%20Monitor.png?dl=0
Yes, I found that the value under the "Working Set" column is much higher
than the others. Also, the value which I was previously looking at under
the Task Manager is under the Private Column here.
It says that I have about 14GB of available memory, but the "Free" number
is much lower, at 79MB.
Regards.
Edwin
On 6 January 2016 at 03:11, Shawn Heisey <ap...@elyograg.org> wrote:
> On 1/5/2016 9:59 AM, Zheng Lin Edwin Yeo wrote:
> > I have uploaded the screenshot here
> > https://www.dropbox.com/s/l5itfbaus1c9793/Memmory%20Usage.png?dl=0
> >
> > Basically, Java(TM) Platform SE Library, which Solr is running on, is
> only
> > using about 22GB currently. However, the memory usage at the top says it
> is
> > using 73% now (which I think is already higher than the figures, given
> that
> > I have 64GB of RAM), and it could potentially go up to 100%, even though
> > the memory usage of Java(TM) Platform SE Library remains around 22GB, and
> > there is no other new task which uses alot of memory are running. The
> > figure is sorted according to the memory usage already.
>
> I would bet that the 73 percent refers to all memory allocated for *any*
> purpose, including the disk cache.
>
> The screenshot that you provided is not the best way to see everything
> that's happening in memory. With the task manager open, click on the
> "Performance" tab, then click on "Open Resource Monitor" down at the
> bottom of the window. This will open a whole new program. Once that's
> open, click on the Memory tab, then click on the "Working Set" column
> header to sort by that column. Increase the size of the window so that
> a large number of processes with the memory utilization can be seen,
> adjust the column widths so the information is clear, and make sure that
> the "Physical Memory" graph and its legend are fully visible. Then grab
> a new screenshot.
>
> I believe that you will find the "Available" number is quite high, even
> though the "Free" number is very small ... and the difference will be
> similar to the "Cached" number. If this is what you find, then
> everything is working exactly as it is supposed to be working.
>
> Thanks,
> Shawn
>
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/5/2016 9:59 AM, Zheng Lin Edwin Yeo wrote:
> I have uploaded the screenshot here
> https://www.dropbox.com/s/l5itfbaus1c9793/Memmory%20Usage.png?dl=0
>
> Basically, Java(TM) Platform SE Library, which Solr is running on, is only
> using about 22GB currently. However, the memory usage at the top says it is
> using 73% now (which I think is already higher than the figures, given that
> I have 64GB of RAM), and it could potentially go up to 100%, even though
> the memory usage of Java(TM) Platform SE Library remains around 22GB, and
> there is no other new task which uses alot of memory are running. The
> figure is sorted according to the memory usage already.
I would bet that the 73 percent refers to all memory allocated for *any*
purpose, including the disk cache.
The screenshot that you provided is not the best way to see everything
that's happening in memory. With the task manager open, click on the
"Performance" tab, then click on "Open Resource Monitor" down at the
bottom of the window. This will open a whole new program. Once that's
open, click on the Memory tab, then click on the "Working Set" column
header to sort by that column. Increase the size of the window so that
a large number of processes with the memory utilization can be seen,
adjust the column widths so the information is clear, and make sure that
the "Physical Memory" graph and its legend are fully visible. Then grab
a new screenshot.
I believe that you will find the "Available" number is quite high, even
though the "Free" number is very small ... and the difference will be
similar to the "Cached" number. If this is what you find, then
everything is working exactly as it is supposed to be working.
Thanks,
Shawn
Re: Memory Usage increases by a lot during and after optimization .
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Shawn,
Thanks for your reply.
I have uploaded the screenshot here
https://www.dropbox.com/s/l5itfbaus1c9793/Memmory%20Usage.png?dl=0
Basically, Java(TM) Platform SE Library, which Solr is running on, is only
using about 22GB currently. However, the memory usage at the top says it is
using 73% now (which I think is already higher than the figures, given that
I have 64GB of RAM), and it could potentially go up to 100%, even though
the memory usage of Java(TM) Platform SE Library remains around 22GB, and
there is no other new task which uses alot of memory are running. The
figure is sorted according to the memory usage already.
Regards,
Edwin
On 5 January 2016 at 08:16, Shawn Heisey <ap...@elyograg.org> wrote:
> On 1/3/2016 7:05 PM, Zheng Lin Edwin Yeo wrote:
> > A) Before I start the optimization, the server's memory usage
> > is consistent at around 16GB, when Solr startsup and we did some
> searching.
> > However, when I click on the optimization button, the memory usage
> > increases gradually, until it reaches the maximum of 64GB which the
> server
> > has. But this only happens to the collection with index of 200GB, and not
> > other collections which has smaller index size (they are at most 1GB at
> the
> > moment).
>
> <snip>
>
> > A) I am quite curious at this also, because in the Task Manager of the
> > server, the amount of memory usage stated does not tally with the
> > percentage of memory usage. When I start optimizatoin, the memory usage
> > states the JVM is only using 14GB, but the percentage of memory usage is
> > almost 100%, when I have 64GB RAM. I have check the other processes
> running
> > in the server, and did not found any other processes that takes up a
> large
> > amount of memory, and the total amount of memory usage for the whole
> sever
> > is only around 16GB.
>
> Toke's reply is spot on.
>
> In your first answer above, you didn't really answer my question, which
> was "What *exactly* are you looking at that says Solr is using all your
> memory?" You've said "the server's memory usage" but haven't described
> how you got that number.
>
> Here's a screenshot of "top" on one of my Solr servers, with the list
> sorted by memory usage:
>
> https://www.dropbox.com/s/i49s2uyfetwo3xq/solr-mem-prod-8g-heap.png?dl=0
>
> This machine has 165GB (base 2 number) of index data on it, and 64GB of
> memory. Solr has been assigned an 8GB heap. Here's more specific info
> about the size of the index data:
>
> root@idxb3:/index/solr5/data# du -hs data
> 165G data
> root@idxb3:/index/solr5/data# du -s data
> 172926520 data
>
> You can see that the VIRT memory size of the Solr process is
> approximately the same as the total index size (165GB) plus the max heap
> (8GB), which adds up to 173GB. The RES memory size of the java process
> is 8.3GB -- just a little bit larger than the max heap.
>
> At the OS level, my server shows 46GB used out of 64GB total ... which
> probably seems excessive, until you consider the 36 million kilobytes in
> the "cached" statistic. This is the amount of memory being used for the
> page cache. If you subtract that memory, then you can see that this
> server has only allocated about 10GB of RAM total -- exactly what I
> would expect for a Linux machine dedicated to Solr with the max heap at
> 8GB.
>
> Although my server is indicating about 18GB of memory free, I have seen
> perfectly functioning servers with that number very close to zero. It
> is completely normal for the "free" memory statistic on Linux and
> Windows to show a few megabytes or less, especially when you optimize a
> Solr index, which reads (and writes) all of the index data, and will
> fill up the page cache.
>
> So, I will ask something very similar to my initial question. Where
> *exactly* are you looking to see the memory usage that you believe is a
> problem? A screenshot would be very helpful.
>
> Here's a screenshot from my Windows client. This machine is NOT running
> Solr, but the situation with free and cached memory is similar.
>
> https://www.dropbox.com/s/wex1gbj7e45g8ed/windows7-mem-usage.png?dl=0
>
> I am not doing anything particularly unusual with this machine, but it
> says there is *zero* free memory, out of 16GB total. There is 9GB of
> memory in the page cache, though -- memory that the OS will instantly
> give up if any program requests it, which you can see because the
> "available" stat is also about 9GB. This Windows machine is doing
> perfectly fine as far as memory.
>
> Thanks,
> Shawn
>
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/3/2016 7:05 PM, Zheng Lin Edwin Yeo wrote:
> A) Before I start the optimization, the server's memory usage
> is consistent at around 16GB, when Solr startsup and we did some searching.
> However, when I click on the optimization button, the memory usage
> increases gradually, until it reaches the maximum of 64GB which the server
> has. But this only happens to the collection with index of 200GB, and not
> other collections which has smaller index size (they are at most 1GB at the
> moment).
<snip>
> A) I am quite curious at this also, because in the Task Manager of the
> server, the amount of memory usage stated does not tally with the
> percentage of memory usage. When I start optimizatoin, the memory usage
> states the JVM is only using 14GB, but the percentage of memory usage is
> almost 100%, when I have 64GB RAM. I have check the other processes running
> in the server, and did not found any other processes that takes up a large
> amount of memory, and the total amount of memory usage for the whole sever
> is only around 16GB.
Toke's reply is spot on.
In your first answer above, you didn't really answer my question, which
was "What *exactly* are you looking at that says Solr is using all your
memory?" You've said "the server's memory usage" but haven't described
how you got that number.
Here's a screenshot of "top" on one of my Solr servers, with the list
sorted by memory usage:
https://www.dropbox.com/s/i49s2uyfetwo3xq/solr-mem-prod-8g-heap.png?dl=0
This machine has 165GB (base 2 number) of index data on it, and 64GB of
memory. Solr has been assigned an 8GB heap. Here's more specific info
about the size of the index data:
root@idxb3:/index/solr5/data# du -hs data
165G data
root@idxb3:/index/solr5/data# du -s data
172926520 data
You can see that the VIRT memory size of the Solr process is
approximately the same as the total index size (165GB) plus the max heap
(8GB), which adds up to 173GB. The RES memory size of the java process
is 8.3GB -- just a little bit larger than the max heap.
At the OS level, my server shows 46GB used out of 64GB total ... which
probably seems excessive, until you consider the 36 million kilobytes in
the "cached" statistic. This is the amount of memory being used for the
page cache. If you subtract that memory, then you can see that this
server has only allocated about 10GB of RAM total -- exactly what I
would expect for a Linux machine dedicated to Solr with the max heap at 8GB.
Although my server is indicating about 18GB of memory free, I have seen
perfectly functioning servers with that number very close to zero. It
is completely normal for the "free" memory statistic on Linux and
Windows to show a few megabytes or less, especially when you optimize a
Solr index, which reads (and writes) all of the index data, and will
fill up the page cache.
So, I will ask something very similar to my initial question. Where
*exactly* are you looking to see the memory usage that you believe is a
problem? A screenshot would be very helpful.
Here's a screenshot from my Windows client. This machine is NOT running
Solr, but the situation with free and cached memory is similar.
https://www.dropbox.com/s/wex1gbj7e45g8ed/windows7-mem-usage.png?dl=0
I am not doing anything particularly unusual with this machine, but it
says there is *zero* free memory, out of 16GB total. There is 9GB of
memory in the page cache, though -- memory that the OS will instantly
give up if any program requests it, which you can see because the
"available" stat is also about 9GB. This Windows machine is doing
perfectly fine as far as memory.
Thanks,
Shawn
Re: Memory Usage increases by a lot during and after optimization .
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Toke,
I read the server's memory usage from the Task manager under Windows,
Regards,
Edwin
On 4 January 2016 at 17:17, Toke Eskildsen <te...@statsbiblioteket.dk> wrote:
> On Mon, 2016-01-04 at 10:05 +0800, Zheng Lin Edwin Yeo wrote:
> > A) Before I start the optimization, the server's memory usage
> > is consistent at around 16GB, when Solr startsup and we did some
> searching.
>
> How do you read this number?
>
> > However, when I click on the optimization button, the memory usage
> > increases gradually, until it reaches the maximum of 64GB which the
> server
> > has.
>
> There are multiple ways of looking at memory. The most relevant ones in
> this context are
>
> - Total memory on the system
> This appears to be 64GB.
>
> - Free memory on the system
> Usually determined by 'top' under Linux or Task Manager under Windows.
>
> - Memory used for caching on the system
> Usually determined by 'top' under Linux or Task Manager under Windows.
>
> - JVM memory usage
> Usually determined by 'top' under Linux or Task Manager under Windows.
> Look for "Res" (resident) for the task in Linux. It might be called
> "physical" under Windows.
>
>
> - Maximum JVM heap (Xmx)
> Lightest grey in "JVM-Memory" in the Solr Admin interface Dashboard.
>
> - Allocated JVM heap (Xmx)
> Medium grey in "JVM-Memory" in the Solr Admin interface Dashboard.
>
> - Active JVM heap (Xmx)
> Dark grey in "JVM-Memory" in the Solr Admin interface Dashboard.
>
>
> I am guessing that the number you are talking about is "Free memory on
> the system" and as Shawn and Erick points out, a full allocation there
> is expected behaviour.
>
> What we are interested in are the JVM heap numbers.
>
> - Toke Eskildsen, State and University Library, Denmark
>
>
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2016-01-04 at 10:05 +0800, Zheng Lin Edwin Yeo wrote:
> A) Before I start the optimization, the server's memory usage
> is consistent at around 16GB, when Solr startsup and we did some searching.
How do you read this number?
> However, when I click on the optimization button, the memory usage
> increases gradually, until it reaches the maximum of 64GB which the server
> has.
There are multiple ways of looking at memory. The most relevant ones in
this context are
- Total memory on the system
This appears to be 64GB.
- Free memory on the system
Usually determined by 'top' under Linux or Task Manager under Windows.
- Memory used for caching on the system
Usually determined by 'top' under Linux or Task Manager under Windows.
- JVM memory usage
Usually determined by 'top' under Linux or Task Manager under Windows.
Look for "Res" (resident) for the task in Linux. It might be called
"physical" under Windows.
- Maximum JVM heap (Xmx)
Lightest grey in "JVM-Memory" in the Solr Admin interface Dashboard.
- Allocated JVM heap (Xmx)
Medium grey in "JVM-Memory" in the Solr Admin interface Dashboard.
- Active JVM heap (Xmx)
Dark grey in "JVM-Memory" in the Solr Admin interface Dashboard.
I am guessing that the number you are talking about is "Free memory on
the system" and as Shawn and Erick points out, a full allocation there
is expected behaviour.
What we are interested in are the JVM heap numbers.
- Toke Eskildsen, State and University Library, Denmark
Re: Memory Usage increases by a lot during and after optimization .
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Thanks for the reply Shawn and Erick.
What *exactly* are you looking at that says Solr is using all your
memory? You must be extremely specific when answering this question.
This will determine whether we should be looking for a bug or not.
A) Before I start the optimization, the server's memory usage
is consistent at around 16GB, when Solr startsup and we did some searching.
However, when I click on the optimization button, the memory usage
increases gradually, until it reaches the maximum of 64GB which the server
has. But this only happens to the collection with index of 200GB, and not
other collections which has smaller index size (they are at most 1GB at the
moment).
In another message thread, you indicated that your max heap was set to
14GB. Java will only ever use that much memory for the program that is
being run, plus a relatively small amount so that Java itself can
operate. Any significantly large resident memory allocation beyond the
max heap would be an indication of a bug in Java, not a bug in Solr.
A) I am quite curious at this also, because in the Task Manager of the
server, the amount of memory usage stated does not tally with the
percentage of memory usage. When I start optimizatoin, the memory usage
states the JVM is only using 14GB, but the percentage of memory usage is
almost 100%, when I have 64GB RAM. I have check the other processes running
in the server, and did not found any other processes that takes up a large
amount of memory, and the total amount of memory usage for the whole sever
is only around 16GB.
Regards,
Edwin
On 3 January 2016 at 01:24, Erick Erickson <er...@gmail.com> wrote:
> If you happen to be looking at "top" or the like, you
> might be seeing virtual memory, see Uwe's
> excellent blog here:
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> Best,
> Erick
>
> On Fri, Jan 1, 2016 at 11:46 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> > On 12/31/2015 8:03 PM, Zheng Lin Edwin Yeo wrote:
> >> But the problem I'm facing now is that during optimizing, the memory
> usage
> >> of the server hit the maximum of 64GB, and I believe the optimization
> could
> >> not be completed fully as there is not enough memory, so when I check
> the
> >> index again, it says that it is not optimized. Before the optimization,
> the
> >> memory usage was less than 16GB, so the optimization actually uses up
> more
> >> than 48GB of memory.
> >>
> >> Is it normal for an index size of 200GB to use up so much memory during
> >> optimization?
> >
> > What *exactly* are you looking at that says Solr is using all your
> > memory? You must be extremely specific when answering this question.
> > This will determine whether we should be looking for a bug or not.
> >
> > It is completely normal for all modern operating systems to use all the
> > memory when the amount of data being handled is large. Some of the
> > memory will be allocated to programs like Java/Solr, and the operating
> > system will use everything else to cache data from I/O operations on the
> > disk. This is called the page cache. For Solr to perform well, the
> > page cache must be large enough to effectively cache your index data.
> >
> > https://en.wikipedia.org/wiki/Page_cache
> >
> > In another message thread, you indicated that your max heap was set to
> > 14GB. Java will only ever use that much memory for the program that is
> > being run, plus a relatively small amount so that Java itself can
> > operate. Any significantly large resident memory allocation beyond the
> > max heap would be an indication of a bug in Java, not a bug in Solr.
> >
> > With the index size at 200GB, I would hope to have at least 128GB of
> > memory in the server, but I would *want* 256GB. 64GB may not be enough
> > for good performance.
> >
> > Thanks,
> > Shawn
> >
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Erick Erickson <er...@gmail.com>.
If you happen to be looking at "top" or the like, you
might be seeing virtual memory, see Uwe's
excellent blog here:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best,
Erick
On Fri, Jan 1, 2016 at 11:46 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 12/31/2015 8:03 PM, Zheng Lin Edwin Yeo wrote:
>> But the problem I'm facing now is that during optimizing, the memory usage
>> of the server hit the maximum of 64GB, and I believe the optimization could
>> not be completed fully as there is not enough memory, so when I check the
>> index again, it says that it is not optimized. Before the optimization, the
>> memory usage was less than 16GB, so the optimization actually uses up more
>> than 48GB of memory.
>>
>> Is it normal for an index size of 200GB to use up so much memory during
>> optimization?
>
> What *exactly* are you looking at that says Solr is using all your
> memory? You must be extremely specific when answering this question.
> This will determine whether we should be looking for a bug or not.
>
> It is completely normal for all modern operating systems to use all the
> memory when the amount of data being handled is large. Some of the
> memory will be allocated to programs like Java/Solr, and the operating
> system will use everything else to cache data from I/O operations on the
> disk. This is called the page cache. For Solr to perform well, the
> page cache must be large enough to effectively cache your index data.
>
> https://en.wikipedia.org/wiki/Page_cache
>
> In another message thread, you indicated that your max heap was set to
> 14GB. Java will only ever use that much memory for the program that is
> being run, plus a relatively small amount so that Java itself can
> operate. Any significantly large resident memory allocation beyond the
> max heap would be an indication of a bug in Java, not a bug in Solr.
>
> With the index size at 200GB, I would hope to have at least 128GB of
> memory in the server, but I would *want* 256GB. 64GB may not be enough
> for good performance.
>
> Thanks,
> Shawn
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/31/2015 8:03 PM, Zheng Lin Edwin Yeo wrote:
> But the problem I'm facing now is that during optimizing, the memory usage
> of the server hit the maximum of 64GB, and I believe the optimization could
> not be completed fully as there is not enough memory, so when I check the
> index again, it says that it is not optimized. Before the optimization, the
> memory usage was less than 16GB, so the optimization actually uses up more
> than 48GB of memory.
>
> Is it normal for an index size of 200GB to use up so much memory during
> optimization?
What *exactly* are you looking at that says Solr is using all your
memory? You must be extremely specific when answering this question.
This will determine whether we should be looking for a bug or not.
It is completely normal for all modern operating systems to use all the
memory when the amount of data being handled is large. Some of the
memory will be allocated to programs like Java/Solr, and the operating
system will use everything else to cache data from I/O operations on the
disk. This is called the page cache. For Solr to perform well, the
page cache must be large enough to effectively cache your index data.
https://en.wikipedia.org/wiki/Page_cache
In another message thread, you indicated that your max heap was set to
14GB. Java will only ever use that much memory for the program that is
being run, plus a relatively small amount so that Java itself can
operate. Any significantly large resident memory allocation beyond the
max heap would be an indication of a bug in Java, not a bug in Solr.
With the index size at 200GB, I would hope to have at least 128GB of
memory in the server, but I would *want* 256GB. 64GB may not be enough
for good performance.
Thanks,
Shawn
Re: Memory Usage increases by a lot during and after optimization .
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Yonik,
Yes, the plan is to do the optimizing at night after indexing, when there
are lesser user who will use the system.
But the problem I'm facing now is that during optimizing, the memory usage
of the server hit the maximum of 64GB, and I believe the optimization could
not be completed fully as there is not enough memory, so when I check the
index again, it says that it is not optimized. Before the optimization, the
memory usage was less than 16GB, so the optimization actually uses up more
than 48GB of memory.
Is it normal for an index size of 200GB to use up so much memory during
optimization?
Regards,
Edwin
On 30 December 2015 at 11:49, Yonik Seeley <ys...@gmail.com> wrote:
> Some people also want to control when major segment merges happen, and
> optimizing at a known time helps prevent a major merge at an unknown
> time (which can be equivalent to an optimize/forceMerge).
>
> The benefits of optimizing (and having fewer segments to search
> across) will vary depending on the requests.
> Normal full-text searches will see little benefit (merging a few terms
> across many segments is not expensive), while other operations that
> need to deal with many terms, like faceting, may see bigger speedups.
>
> -Yonik
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by William Bell <bi...@gmail.com>.
Question: does anyone have example good merge settings for solrconfig ? To
keep the number of segments small like 6?
On Tue, Dec 29, 2015 at 8:49 PM, Yonik Seeley <ys...@gmail.com> wrote:
> Some people also want to control when major segment merges happen, and
> optimizing at a known time helps prevent a major merge at an unknown
> time (which can be equivalent to an optimize/forceMerge).
>
> The benefits of optimizing (and having fewer segments to search
> across) will vary depending on the requests.
> Normal full-text searches will see little benefit (merging a few terms
> across many segments is not expensive), while other operations that
> need to deal with many terms, like faceting, may see bigger speedups.
>
> -Yonik
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
Re: Memory Usage increases by a lot during and after optimization .
Posted by Yonik Seeley <ys...@gmail.com>.
Some people also want to control when major segment merges happen, and
optimizing at a known time helps prevent a major merge at an unknown
time (which can be equivalent to an optimize/forceMerge).
The benefits of optimizing (and having fewer segments to search
across) will vary depending on the requests.
Normal full-text searches will see little benefit (merging a few terms
across many segments is not expensive), while other operations that
need to deal with many terms, like faceting, may see bigger speedups.
-Yonik
Re: Memory Usage increases by a lot during and after optimization .
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Wouldn't collection swapping be a better strategy in that case?
Load and optimise in a separate server, then swap it in.
On 30 Dec 2015 10:08 am, "Walter Underwood" <wu...@wunderwood.org> wrote:
> The only time that a force merge might be useful is when you reindex all
> content every night or every week, then do not make any changes until the
> next reindex. But even then, it probably does not matter.
>
> Just let Solr do its thing. Solr is pretty smart.
>
> A long time ago (1996-2006), I worked on an enterprise search engine with
> the same merging algorithm as Solr (Ultraseek Server). We always had
> customers asking about force-merge/optimize. It never made a useful
> difference. Even with twenty servers at irs.gov <http://irs.gov/>, it
> didn’t make a difference.
>
> wunder
> K6WRU
> Walter Underwood
> CM87wj
> http://observer.wunderwood.org/ (my blog)
>
> > On Dec 29, 2015, at 6:59 PM, Zheng Lin Edwin Yeo <ed...@gmail.com>
> wrote:
> >
> > Hi Walter,
> >
> > Thanks for your reply.
> >
> > Then how about optimization after indexing?
> > Normally the index size is much larger after indexing, then after
> > optimization, the index size reduces. Do we still need to do that?
> >
> > Regards,
> > Edwin
> >
> > On 30 December 2015 at 10:45, Walter Underwood <wu...@wunderwood.org>
> > wrote:
> >
> >> Do not “optimize".
> >>
> >> It is a forced merge, not an optimization. It was a mistake to ever name
> >> it “optimize”. Solr automatically merges as needed. There are a few
> >> situations where a force merge might make a small difference. Maybe 10%
> or
> >> 20%, no one had bothered to measure it.
> >>
> >> If your index is continually updated, clicking that is a complete waste
> of
> >> resources. Don’t do it.
> >>
> >> wunder
> >> Walter Underwood
> >> wunder@wunderwood.org
> >> http://observer.wunderwood.org/ (my blog)
> >>
> >>> On Dec 29, 2015, at 6:35 PM, Zheng Lin Edwin Yeo <edwinyeozl@gmail.com
> >
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am facing a situation, when I do an optimization by clicking on the
> >>> "Optimized" button on the Solr Admin Overview UI, the memory usage of
> the
> >>> server increases gradually, until it reaches near the maximum memory
> >>> available. There is 64GB of memory available in the server.
> >>>
> >>> Even after the optimized is completed, the memory usage stays near the
> >> 100%
> >>> range, and could not be reduced until I stop Solr. Why could this be
> >>> happening?
> >>>
> >>> Also, I don't think the optimization is completed, as the admin page
> says
> >>> the index is not optimized again after I go back to the Overview page,
> >> even
> >>> though I did not do any updates to the index.
> >>>
> >>> I am using Solr 5.3.0, with 1 shard and 2 replica. My index size is
> >> 183GB.
> >>>
> >>> Regards,
> >>> Edwin
> >>
> >>
>
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Thanks for the information.
Another thing I like to confirm is, will the Java Heap size setting affect
the optimization process or the memory usage?
Is the any recommended setting that we can use, for an index size of 200GB?
Regards,
Edwin
On 30 December 2015 at 11:07, Walter Underwood <wu...@wunderwood.org>
wrote:
> The only time that a force merge might be useful is when you reindex all
> content every night or every week, then do not make any changes until the
> next reindex. But even then, it probably does not matter.
>
> Just let Solr do its thing. Solr is pretty smart.
>
> A long time ago (1996-2006), I worked on an enterprise search engine with
> the same merging algorithm as Solr (Ultraseek Server). We always had
> customers asking about force-merge/optimize. It never made a useful
> difference. Even with twenty servers at irs.gov <http://irs.gov/>, it
> didn’t make a difference.
>
> wunder
> K6WRU
> Walter Underwood
> CM87wj
> http://observer.wunderwood.org/ (my blog)
>
> > On Dec 29, 2015, at 6:59 PM, Zheng Lin Edwin Yeo <ed...@gmail.com>
> wrote:
> >
> > Hi Walter,
> >
> > Thanks for your reply.
> >
> > Then how about optimization after indexing?
> > Normally the index size is much larger after indexing, then after
> > optimization, the index size reduces. Do we still need to do that?
> >
> > Regards,
> > Edwin
> >
> > On 30 December 2015 at 10:45, Walter Underwood <wu...@wunderwood.org>
> > wrote:
> >
> >> Do not “optimize".
> >>
> >> It is a forced merge, not an optimization. It was a mistake to ever name
> >> it “optimize”. Solr automatically merges as needed. There are a few
> >> situations where a force merge might make a small difference. Maybe 10%
> or
> >> 20%, no one had bothered to measure it.
> >>
> >> If your index is continually updated, clicking that is a complete waste
> of
> >> resources. Don’t do it.
> >>
> >> wunder
> >> Walter Underwood
> >> wunder@wunderwood.org
> >> http://observer.wunderwood.org/ (my blog)
> >>
> >>> On Dec 29, 2015, at 6:35 PM, Zheng Lin Edwin Yeo <edwinyeozl@gmail.com
> >
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am facing a situation, when I do an optimization by clicking on the
> >>> "Optimized" button on the Solr Admin Overview UI, the memory usage of
> the
> >>> server increases gradually, until it reaches near the maximum memory
> >>> available. There is 64GB of memory available in the server.
> >>>
> >>> Even after the optimized is completed, the memory usage stays near the
> >> 100%
> >>> range, and could not be reduced until I stop Solr. Why could this be
> >>> happening?
> >>>
> >>> Also, I don't think the optimization is completed, as the admin page
> says
> >>> the index is not optimized again after I go back to the Overview page,
> >> even
> >>> though I did not do any updates to the index.
> >>>
> >>> I am using Solr 5.3.0, with 1 shard and 2 replica. My index size is
> >> 183GB.
> >>>
> >>> Regards,
> >>> Edwin
> >>
> >>
>
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Walter Underwood <wu...@wunderwood.org>.
The only time that a force merge might be useful is when you reindex all content every night or every week, then do not make any changes until the next reindex. But even then, it probably does not matter.
Just let Solr do its thing. Solr is pretty smart.
A long time ago (1996-2006), I worked on an enterprise search engine with the same merging algorithm as Solr (Ultraseek Server). We always had customers asking about force-merge/optimize. It never made a useful difference. Even with twenty servers at irs.gov <http://irs.gov/>, it didn’t make a difference.
wunder
K6WRU
Walter Underwood
CM87wj
http://observer.wunderwood.org/ (my blog)
> On Dec 29, 2015, at 6:59 PM, Zheng Lin Edwin Yeo <ed...@gmail.com> wrote:
>
> Hi Walter,
>
> Thanks for your reply.
>
> Then how about optimization after indexing?
> Normally the index size is much larger after indexing, then after
> optimization, the index size reduces. Do we still need to do that?
>
> Regards,
> Edwin
>
> On 30 December 2015 at 10:45, Walter Underwood <wu...@wunderwood.org>
> wrote:
>
>> Do not “optimize".
>>
>> It is a forced merge, not an optimization. It was a mistake to ever name
>> it “optimize”. Solr automatically merges as needed. There are a few
>> situations where a force merge might make a small difference. Maybe 10% or
>> 20%, no one had bothered to measure it.
>>
>> If your index is continually updated, clicking that is a complete waste of
>> resources. Don’t do it.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>> On Dec 29, 2015, at 6:35 PM, Zheng Lin Edwin Yeo <ed...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I am facing a situation, when I do an optimization by clicking on the
>>> "Optimized" button on the Solr Admin Overview UI, the memory usage of the
>>> server increases gradually, until it reaches near the maximum memory
>>> available. There is 64GB of memory available in the server.
>>>
>>> Even after the optimized is completed, the memory usage stays near the
>> 100%
>>> range, and could not be reduced until I stop Solr. Why could this be
>>> happening?
>>>
>>> Also, I don't think the optimization is completed, as the admin page says
>>> the index is not optimized again after I go back to the Overview page,
>> even
>>> though I did not do any updates to the index.
>>>
>>> I am using Solr 5.3.0, with 1 shard and 2 replica. My index size is
>> 183GB.
>>>
>>> Regards,
>>> Edwin
>>
>>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Walter,
Thanks for your reply.
Then how about optimization after indexing?
Normally the index size is much larger after indexing, then after
optimization, the index size reduces. Do we still need to do that?
Regards,
Edwin
On 30 December 2015 at 10:45, Walter Underwood <wu...@wunderwood.org>
wrote:
> Do not “optimize".
>
> It is a forced merge, not an optimization. It was a mistake to ever name
> it “optimize”. Solr automatically merges as needed. There are a few
> situations where a force merge might make a small difference. Maybe 10% or
> 20%, no one had bothered to measure it.
>
> If your index is continually updated, clicking that is a complete waste of
> resources. Don’t do it.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
> > On Dec 29, 2015, at 6:35 PM, Zheng Lin Edwin Yeo <ed...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I am facing a situation, when I do an optimization by clicking on the
> > "Optimized" button on the Solr Admin Overview UI, the memory usage of the
> > server increases gradually, until it reaches near the maximum memory
> > available. There is 64GB of memory available in the server.
> >
> > Even after the optimized is completed, the memory usage stays near the
> 100%
> > range, and could not be reduced until I stop Solr. Why could this be
> > happening?
> >
> > Also, I don't think the optimization is completed, as the admin page says
> > the index is not optimized again after I go back to the Overview page,
> even
> > though I did not do any updates to the index.
> >
> > I am using Solr 5.3.0, with 1 shard and 2 replica. My index size is
> 183GB.
> >
> > Regards,
> > Edwin
>
>
Re: Memory Usage increases by a lot during and after optimization .
Posted by Walter Underwood <wu...@wunderwood.org>.
Do not “optimize".
It is a forced merge, not an optimization. It was a mistake to ever name it “optimize”. Solr automatically merges as needed. There are a few situations where a force merge might make a small difference. Maybe 10% or 20%, no one had bothered to measure it.
If your index is continually updated, clicking that is a complete waste of resources. Don’t do it.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Dec 29, 2015, at 6:35 PM, Zheng Lin Edwin Yeo <ed...@gmail.com> wrote:
>
> Hi,
>
> I am facing a situation, when I do an optimization by clicking on the
> "Optimized" button on the Solr Admin Overview UI, the memory usage of the
> server increases gradually, until it reaches near the maximum memory
> available. There is 64GB of memory available in the server.
>
> Even after the optimized is completed, the memory usage stays near the 100%
> range, and could not be reduced until I stop Solr. Why could this be
> happening?
>
> Also, I don't think the optimization is completed, as the admin page says
> the index is not optimized again after I go back to the Overview page, even
> though I did not do any updates to the index.
>
> I am using Solr 5.3.0, with 1 shard and 2 replica. My index size is 183GB.
>
> Regards,
> Edwin