You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Susheel Kumar <su...@gmail.com> on 2018/02/07 19:01:31 UTC
Solr Swap space
Hello,
Just trying to find where do we set swap space available to Solr process. I
see in our 6.0 instances it was set to 2GB on and on 6.6 instances its set
to 16GB.
Thanks,
Susheel
Re: Solr Swap space
Posted by Susheel Kumar <su...@gmail.com>.
Cool, Thanks, Shawn. I was also looking the swapiness and it is set to
60. Will try this out and let you know. Thanks, again.
On Thu, Feb 22, 2018 at 10:55 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 2/21/2018 7:58 PM, Susheel Kumar wrote:
>
>> Below output for prod machine based on the steps you described. Please
>> take a look. The solr searches are returning fine and no issue with
>> performance but since last 4 months swap space started going up. After
>> restart, it comes down to zero and then few weeks, it utilization reaches
>> to 40-50% and thus requires restart of solr process.
>>
>
> I bet that if you run this command, it will show you a value of 60:
>
> cat /proc/sys/vm/swappiness
>
> This makes the OS very aggressive about using swap, even when there is
> absolutely no need for it to do so.
>
> If you type the following series of commands, it should fix the problem
> and prevent it from happening again until you reboot the system:
>
> echo "0" > /proc/sys/vm/swappiness
> swapoff -a
> swapon -a
>
> Note that when the swapoff command runs, it will force the OS to read all
> the swapped data back into memory. It will take several minutes for this
> to occur, because it must read nearly a gigabyte of data and figure out how
> to put it back in memory. Both of the command outputs you included say that
> there is over 20GB of free memory. So I do not anticipate the system
> having problems from running these commands. It will slow the machine down
> temporarily, though -- so only do it during a quiet time for your Solr
> install.
>
> To make this setting survive a reboot, find the sysctl.conf file somewhere
> in your /etc directory and add this line to it:
>
> vm.swappiness = 0
>
> This setting does not completely disable swap. If the system finds itself
> with real memory pressure and actually does NEED to use swap, it still will
> ... it just won't swap anything out before it's actually required.
>
> I do not think the behavior you are seeing is actually causing problems,
> based on your system load and CPU usage. But what I've shared should fix
> it for you.
>
> Thanks,
> Shawn
>
>
Re: Solr Swap space
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/21/2018 7:58 PM, Susheel Kumar wrote:
> Below output for prod machine based on the steps you described. Please
> take a look. The solr searches are returning fine and no issue with
> performance but since last 4 months swap space started going up. After
> restart, it comes down to zero and then few weeks, it utilization reaches
> to 40-50% and thus requires restart of solr process.
I bet that if you run this command, it will show you a value of 60:
cat /proc/sys/vm/swappiness
This makes the OS very aggressive about using swap, even when there is
absolutely no need for it to do so.
If you type the following series of commands, it should fix the problem
and prevent it from happening again until you reboot the system:
echo "0" > /proc/sys/vm/swappiness
swapoff -a
swapon -a
Note that when the swapoff command runs, it will force the OS to read
all the swapped data back into memory. It will take several minutes for
this to occur, because it must read nearly a gigabyte of data and figure
out how to put it back in memory. Both of the command outputs you
included say that there is over 20GB of free memory. So I do not
anticipate the system having problems from running these commands. It
will slow the machine down temporarily, though -- so only do it during a
quiet time for your Solr install.
To make this setting survive a reboot, find the sysctl.conf file
somewhere in your /etc directory and add this line to it:
vm.swappiness = 0
This setting does not completely disable swap. If the system finds
itself with real memory pressure and actually does NEED to use swap, it
still will ... it just won't swap anything out before it's actually
required.
I do not think the behavior you are seeing is actually causing problems,
based on your system load and CPU usage. But what I've shared should
fix it for you.
Thanks,
Shawn
Re: Solr Swap space
Posted by Susheel Kumar <su...@gmail.com>.
Hi Shawn,
Below output for prod machine based on the steps you described. Please
take a look. The solr searches are returning fine and no issue with
performance but since last 4 months swap space started going up. After
restart, it comes down to zero and then few weeks, it utilization reaches
to 40-50% and thus requires restart of solr process.
top - 21:31:37 up 116 days, 20:09, 3 users, load average: 0.24, 0.24, 0.16
Tasks: 202 total, 1 running, 201 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.4 us, 0.0 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem: 65976996 total, 43846008 used, 22130988 free, 84 buffers
KiB Swap: 2097148 total, 942228 used, 1154920 free. 39754116 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
SWAP
20327 solr 20 0 53.791g 0.021t 0.017t S 10.60 33.61 5109:53
java
547524
481 root 20 0 267688 153312 9012 S 0.662 0.232 47:44.63
splunkd
34816
1293 root 20 0 94676 5536 5168 S 0.331 0.008 99:01.45
vmtoolsd
932
2699 root 20 0 21212 2740 2112 S 0.331 0.004 6:17.38
cvfwd
236
17706 kumarsu 20 0 46936 7600 5472 R 0.331 0.012 0:00.18 top
0
1 root 20 0 37532 5360 3896 S 0.000 0.008 6:07.28
systemd
248
2 root 20 0 0 0 0 S 0.000 0.000 0:04.66
kthreadd
0
3 root 20 0 0 0 0 S 0.000 0.000 0:38.21
ksoftirqd/0
0
5 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/0:0H
0
7 root 20 0 0 0 0 S 0.000 0.000 115:56.38
rcu_sched
0
8 root 20 0 0 0 0 S 0.000 0.000 0:00.00
rcu_bh
0
9 root rt 0 0 0 0 S 0.000 0.000 0:28.61
migration/0
0
10 root rt 0 0 0 0 S 0.000 0.000 0:24.68
watchdog/0
0
11 root rt 0 0 0 0 S 0.000 0.000 0:23.78
watchdog/1
0
12 root rt 0 0 0 0 S 0.000 0.000 0:29.63
migration/1
0
13 root 20 0 0 0 0 S 0.000 0.000 1:49.79
ksoftirqd/1
0
15 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/1:0H
0
16 root rt 0 0 0 0 S 0.000 0.000 0:24.00
watchdog/2
0
17 root rt 0 0 0 0 S 0.000 0.000 0:28.67
migration/2
0
18 root 20 0 0 0 0 S 0.000 0.000 0:26.13
ksoftirqd/2
0
20 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/2:0H
0
21 root rt 0 0 0 0 S 0.000 0.000 0:26.61
watchdog/3
0
22 root rt 0 0 0 0 S 0.000 0.000 0:27.92
migration/3
0
23 root 20 0 0 0 0 S 0.000 0.000 0:23.08
ksoftirqd/3
0
25 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/3:0H
0
26 root rt 0 0 0 0 S 0.000 0.000 0:28.02
watchdog/4
0
27 root rt 0 0 0 0 S 0.000 0.000 0:28.01
migration/4
0
28 root 20 0 0 0 0 S 0.000 0.000 0:51.07
ksoftirqd/4
0
31 root rt 0 0 0 0 S 0.000 0.000 0:25.29
watchdog/5
0
32 root rt 0 0 0 0 S 0.000 0.000 0:31.65
migration/5
0
33 root 20 0 0 0 0 S 0.000 0.000 0:22.52
ksoftirqd/5
0
35 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/5:0H
0
36 root rt 0 0 0 0 S 0.000 0.000 0:25.90
watchdog/6
0
37 root rt 0 0 0 0 S 0.000 0.000 0:27.94
migration/6
0
38 root 20 0 0 0 0 S 0.000 0.000 1:07.91
ksoftirqd/6
0
40 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/6:0H
0
>free -m
total used free shared buffers cached
Mem: 64430 42816 21614 132 0 38823
-/+ buffers/cache: 3992 60437
Swap: 2047 920 1127
On Wed, Feb 21, 2018 at 8:45 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 2/21/2018 1:30 PM, Susheel Kumar wrote:
> > I did go thru your posts on swap usage http://lucene.472066.n3.
> > nabble.com/Solr-4-3-1-memory-swapping-td4126641.html and my situation is
> > also similar. Below is top output from our prod and performance test
> > machine and as you can see the swap utilization on Prod machine is 44%
> > while on test machines it is zero.
>
> Are those "top" outputs sorted by the default (CPU usage) or by memory?
> To make any useful determination, it needs to be by memory. Press
> shift-M to sort by memory if your top program supports that key. Also,
> the list only shows the first few processes. More of the list needs to
> be visible.
>
> The system load on the second "top" output is quite low, and doesn't
> show much used CPU percentage. So it looks like the system is not
> actually suffering due to the swap usage, which probably means that it
> is not actively swapping. The machine has plenty of memory available --
> even though almost all of the memory is allocated, the vast majority of
> what's allocated is in the "cached" state -- used by the OS disk cache.
> The OS will instantly give up this memory if a program requests it.
>
> I've learned how to use top to show which processes are using swap.
> What I've described below should work on recent version of gnu top. If
> the top is from another software provider, it may not support this.
>
> http://northernmost.org/blog/swap-usage-5-years-later/
>
> These steps are not precisely as described in that blog post: Run top,
> press f, press p, press space, then press the right angle bracket (>)
> key three times. If top is running with default settings, these
> keypresses should enable the SWAP column and move the sort to that
> column. The list should be sorted by swap usage. If a .toprc file
> exists in your home directory, then the program may be running with very
> different settings than default, and these keypresses might not work as
> expected.
>
> Thanks,
> Shawn
>
>
Re: Solr Swap space
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/21/2018 1:30 PM, Susheel Kumar wrote:
> I did go thru your posts on swap usage http://lucene.472066.n3.
> nabble.com/Solr-4-3-1-memory-swapping-td4126641.html and my situation is
> also similar. Below is top output from our prod and performance test
> machine and as you can see the swap utilization on Prod machine is 44%
> while on test machines it is zero.
Are those "top" outputs sorted by the default (CPU usage) or by memory?
To make any useful determination, it needs to be by memory. Press
shift-M to sort by memory if your top program supports that key. Also,
the list only shows the first few processes. More of the list needs to
be visible.
The system load on the second "top" output is quite low, and doesn't
show much used CPU percentage. So it looks like the system is not
actually suffering due to the swap usage, which probably means that it
is not actively swapping. The machine has plenty of memory available --
even though almost all of the memory is allocated, the vast majority of
what's allocated is in the "cached" state -- used by the OS disk cache.
The OS will instantly give up this memory if a program requests it.
I've learned how to use top to show which processes are using swap.
What I've described below should work on recent version of gnu top. If
the top is from another software provider, it may not support this.
http://northernmost.org/blog/swap-usage-5-years-later/
These steps are not precisely as described in that blog post: Run top,
press f, press p, press space, then press the right angle bracket (>)
key three times. If top is running with default settings, these
keypresses should enable the SWAP column and move the sort to that
column. The list should be sorted by swap usage. If a .toprc file
exists in your home directory, then the program may be running with very
different settings than default, and these keypresses might not work as
expected.
Thanks,
Shawn
Re: Solr Swap space
Posted by Susheel Kumar <su...@gmail.com>.
Hello Shawn,
I did go thru your posts on swap usage http://lucene.472066.n3.
nabble.com/Solr-4-3-1-memory-swapping-td4126641.html and my situation is
also similar. Below is top output from our prod and performance test
machine and as you can see the swap utilization on Prod machine is 44%
while on test machines it is zero.
I haven't been able to figure out what/where exactly is the problem. Both
test and prod machines, in terms of Java version are same 1.8.0_91. What
could help to further understand and debug the issue?
Thanks,
Susheel
*Performance test machine *
top - 15:03:52 up 37 days, 5:49, 1 user, load average: 0.06, 0.09, 0.04
Tasks: 191 total, 1 running, 190 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.0 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem: 65977076 total, 60079356 used, 5897720 free, 1080 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 56416304 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
433 solr 20 0 71.813g 3.484g 311548 S 1.667 5.538 105:37.51
java
1 root 20 0 37504 5672 4020 S 0.000 0.009 3:43.54
systemd
2 root 20 0 0 0 0 S 0.000 0.000 0:04.91
kthreadd
...
...
*Prod machine*
op - 15:10:20 up 116 days, 13:47, 3 users, load average: 0.16, 0.20, 0.18
Tasks: 200 total, 1 running, 199 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem: 65976996 total, 65460256 used, 516740 free, 84 buffers
KiB Swap: 2097148 total, *942288* used, 1154860 free. 61361628 cached
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
20327 solr 20 0 53.792g 0.021t 0.017t S 15.95 33.42 5051:07 java
481 root 20 0 267688 152256 9012 S 0.332 0.231 46:47.60
splunkd
1 root 20 0 37532 5360 3896 S 0.000 0.008 6:06.47
systemd
...
...
On Fri, Feb 9, 2018 at 5:44 PM, Emir Arnautović <
emir.arnautovic@sematext.com> wrote:
> No worries, I don't mind being confused with Erick ;)
>
> Emir
>
> On Feb 9, 2018 9:16 PM, "Susheel Kumar" <su...@gmail.com> wrote:
>
> > Sorry, I meant Emir.
> >
> > On Fri, Feb 9, 2018 at 3:15 PM, Susheel Kumar <su...@gmail.com>
> > wrote:
> >
> > > Thanks, Shawn, Eric. I see that same using swapon -s. Looks like
> during
> > > the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6)
> > >
> > > Our 6.0 instance has been running since 1+ year but recently our monit
> > > started reporting swap usage above 30% and Solr dashboard showing the
> > > same. I haven't been able to find what causes / why Solr is using the
> > swap
> > > space. The index size is well within memory size to fit and we have
> > double
> > > the index size on our performance test machines but there is no usage
> of
> > > swap space.
> > >
> > > Thanks,
> > > Susheel
> > >
> > > On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey <ap...@elyograg.org>
> > wrote:
> > >
> > >> On 2/7/2018 12:01 PM, Susheel Kumar wrote:
> > >>
> > >>> Just trying to find where do we set swap space available to Solr
> > >>> process. I
> > >>> see in our 6.0 instances it was set to 2GB on and on 6.6 instances
> its
> > >>> set
> > >>> to 16GB.
> > >>>
> > >>
> > >> Solr has absolutely no involvement or control over swap space.
> Neither
> > >> does Java. This is a function of your operating system's memory
> > >> management, and is typically set up when you first install your OS.
> > >>
> > >> https://www.linux.com/news/all-about-linux-swap-space
> > >> https://en.wikipedia.org/wiki/Paging#Windows_NT
> > >>
> > >> If your system is using swap space, it's a strong indication that you
> > >> don't have enough memory installed. If any of the memory that Solr
> > uses is
> > >> swapped out to disk, Solr performance is going to be REALLY bad.
> > >>
> > >> Thanks,
> > >> Shawn
> > >>
> > >
> > >
> >
>
Re: Solr Swap space
Posted by Emir Arnautović <em...@sematext.com>.
No worries, I don't mind being confused with Erick ;)
Emir
On Feb 9, 2018 9:16 PM, "Susheel Kumar" <su...@gmail.com> wrote:
> Sorry, I meant Emir.
>
> On Fri, Feb 9, 2018 at 3:15 PM, Susheel Kumar <su...@gmail.com>
> wrote:
>
> > Thanks, Shawn, Eric. I see that same using swapon -s. Looks like during
> > the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6)
> >
> > Our 6.0 instance has been running since 1+ year but recently our monit
> > started reporting swap usage above 30% and Solr dashboard showing the
> > same. I haven't been able to find what causes / why Solr is using the
> swap
> > space. The index size is well within memory size to fit and we have
> double
> > the index size on our performance test machines but there is no usage of
> > swap space.
> >
> > Thanks,
> > Susheel
> >
> > On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey <ap...@elyograg.org>
> wrote:
> >
> >> On 2/7/2018 12:01 PM, Susheel Kumar wrote:
> >>
> >>> Just trying to find where do we set swap space available to Solr
> >>> process. I
> >>> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its
> >>> set
> >>> to 16GB.
> >>>
> >>
> >> Solr has absolutely no involvement or control over swap space. Neither
> >> does Java. This is a function of your operating system's memory
> >> management, and is typically set up when you first install your OS.
> >>
> >> https://www.linux.com/news/all-about-linux-swap-space
> >> https://en.wikipedia.org/wiki/Paging#Windows_NT
> >>
> >> If your system is using swap space, it's a strong indication that you
> >> don't have enough memory installed. If any of the memory that Solr
> uses is
> >> swapped out to disk, Solr performance is going to be REALLY bad.
> >>
> >> Thanks,
> >> Shawn
> >>
> >
> >
>
Re: Solr Swap space
Posted by Susheel Kumar <su...@gmail.com>.
Sorry, I meant Emir.
On Fri, Feb 9, 2018 at 3:15 PM, Susheel Kumar <su...@gmail.com> wrote:
> Thanks, Shawn, Eric. I see that same using swapon -s. Looks like during
> the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6)
>
> Our 6.0 instance has been running since 1+ year but recently our monit
> started reporting swap usage above 30% and Solr dashboard showing the
> same. I haven't been able to find what causes / why Solr is using the swap
> space. The index size is well within memory size to fit and we have double
> the index size on our performance test machines but there is no usage of
> swap space.
>
> Thanks,
> Susheel
>
> On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 2/7/2018 12:01 PM, Susheel Kumar wrote:
>>
>>> Just trying to find where do we set swap space available to Solr
>>> process. I
>>> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its
>>> set
>>> to 16GB.
>>>
>>
>> Solr has absolutely no involvement or control over swap space. Neither
>> does Java. This is a function of your operating system's memory
>> management, and is typically set up when you first install your OS.
>>
>> https://www.linux.com/news/all-about-linux-swap-space
>> https://en.wikipedia.org/wiki/Paging#Windows_NT
>>
>> If your system is using swap space, it's a strong indication that you
>> don't have enough memory installed. If any of the memory that Solr uses is
>> swapped out to disk, Solr performance is going to be REALLY bad.
>>
>> Thanks,
>> Shawn
>>
>
>
Re: Solr Swap space
Posted by Susheel Kumar <su...@gmail.com>.
Thanks, Shawn, Eric. I see that same using swapon -s. Looks like during
the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6)
Our 6.0 instance has been running since 1+ year but recently our monit
started reporting swap usage above 30% and Solr dashboard showing the
same. I haven't been able to find what causes / why Solr is using the swap
space. The index size is well within memory size to fit and we have double
the index size on our performance test machines but there is no usage of
swap space.
Thanks,
Susheel
On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 2/7/2018 12:01 PM, Susheel Kumar wrote:
>
>> Just trying to find where do we set swap space available to Solr process.
>> I
>> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its set
>> to 16GB.
>>
>
> Solr has absolutely no involvement or control over swap space. Neither
> does Java. This is a function of your operating system's memory
> management, and is typically set up when you first install your OS.
>
> https://www.linux.com/news/all-about-linux-swap-space
> https://en.wikipedia.org/wiki/Paging#Windows_NT
>
> If your system is using swap space, it's a strong indication that you
> don't have enough memory installed. If any of the memory that Solr uses is
> swapped out to disk, Solr performance is going to be REALLY bad.
>
> Thanks,
> Shawn
>
Re: Solr Swap space
Posted by JDISTim <la...@johndeere.com>.
Thank you for this information!
I am working on a client PC where Solr Admin showed 11.64 GB/13.03 GB Swap
Space, and was losing my mind trying to track down the issue!
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr Swap space
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/7/2018 12:01 PM, Susheel Kumar wrote:
> Just trying to find where do we set swap space available to Solr process. I
> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its set
> to 16GB.
Solr has absolutely no involvement or control over swap space. Neither
does Java. This is a function of your operating system's memory
management, and is typically set up when you first install your OS.
https://www.linux.com/news/all-about-linux-swap-space
https://en.wikipedia.org/wiki/Paging#Windows_NT
If your system is using swap space, it's a strong indication that you
don't have enough memory installed. If any of the memory that Solr uses
is swapped out to disk, Solr performance is going to be REALLY bad.
Thanks,
Shawn
Re: Solr Swap space
Posted by Emir Arnautović <em...@sematext.com>.
Hi Susheel,
Swap space is OS thing, not Solr thing. You should see how to disable swap space or at least set swappiness to some low number on your OS.
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 7 Feb 2018, at 20:01, Susheel Kumar <su...@gmail.com> wrote:
>
> Hello,
>
> Just trying to find where do we set swap space available to Solr process. I
> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its set
> to 16GB.
>
> Thanks,
> Susheel