Re: Solr 6.6 using swap space causes recovery?

[Shawn Heisey <ap...@elyograg.org>]

On 12/15/2017 10:53 AM, Bill Oconnor wrote:
> The recovering server has a much larger swap usage than the other servers in the cluster. We think this this related to the mmap files used for indexes. The server eventually recovers but it triggers alerts for devops which are annoying.
>
> I have found a previous mail  list question (Shawn responded to) with almost an identical problem from 2014 but there is no suggested remedy. ( http://lucene.472066.n3.nabble.com/Solr-4-3-1-memory-swapping-td4126641.html)

Solr itself cannot influence swap usage.  This is handled by the
operating system.  I have no idea whether Java can influence swap usage,
but if it can, this is also outside our control and I have no idea what
to tell you.  My guess is that Java is unlikely to influence swap usage,
but only a developer on the JDK team could tell you that for sure.

Assuming we're dealing with a Linux machine, my recommendation would be
to set vm.swappiness to 0 or 1, so that the OS is not aggressively
deciding that data should be swapped.  The default for vm.swappiness on
Linux is 60, which is quite aggressive.

> Questions :
>
> Is there progress regarding this?

As mentioned previously, there's nothing Solr or Lucene can do to help,
because it's software completely outside our ability to influence.  The
operating system will make those decisions.

If your software tries to use more memory than the machine has, then
swap is going to get used no matter how the OS is configured, and when
that happens, performance will suffer greatly.  In the case of
SolrCloud, it would make basic operation go so slowly that timeouts
would get exceeded, and Solr would initiate recovery.

If the OS is Linux, I would like to see a screenshot from the "top"
program.  (not htop, or anything else, the program needs to be top). 
Run the program, press shift-M to sort the list by memory, and grab a
screenshot.

Thanks,
Shawn

Re: Solr 6.6 using swap space causes recovery?

[Erick Erickson <er...@gmail.com>]

One mechanism that comes to mind is if the swapping slows down an update.

Here's the process
- Leader sends doc to follower
- follower times out
- leader says "that replica must be sick, I'll tell it to recover"

The smoking gun here is if you see any messages about
"leader-initiated recovery". grep for leader and initiated, because
some of the text is inconsistent, may be "leader initiated" or
"leader-initiated". And grep on both leader and follower.

On the leader you should also see messages in the log about the update
timing out.

The follower won't have any errors at all; the doc indexed
successfully it just took a long time.

This is a total shot in the dark BTW. I'm rather surprised that it
takes 3 or more hours for 200K docs. If they're really big and you're
sending them in batches you may just be taking a while to process the
batch and swapping may have nothing to do with it. Perhaps smaller
batches would help if that's the case.

Long GC pauses can also cause this to happen BTW, ZooKeeper will
periodically ping the node to see if it's up. If it times out ZK can
cause the node to go into recovery (actually, I think the leader gets
a message and puts the follower into recovery). Examining the GC logs
should tell you whether that's possible. 3-4 second stop-the-world GC
pauses (and those are excessive IMO) shouldn't hurt. Although with
heaps that small I wouldn't expect much in the way of stop-the-world
GC pauses.

Best,
Erick

On Fri, Dec 15, 2017 at 10:29 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 12/15/2017 10:53 AM, Bill Oconnor wrote:
>> The recovering server has a much larger swap usage than the other servers in the cluster. We think this this related to the mmap files used for indexes. The server eventually recovers but it triggers alerts for devops which are annoying.
>>
>> I have found a previous mail  list question (Shawn responded to) with almost an identical problem from 2014 but there is no suggested remedy. ( http://lucene.472066.n3.nabble.com/Solr-4-3-1-memory-swapping-td4126641.html)
>
> Solr itself cannot influence swap usage.  This is handled by the
> operating system.  I have no idea whether Java can influence swap usage,
> but if it can, this is also outside our control and I have no idea what
> to tell you.  My guess is that Java is unlikely to influence swap usage,
> but only a developer on the JDK team could tell you that for sure.
>
> Assuming we're dealing with a Linux machine, my recommendation would be
> to set vm.swappiness to 0 or 1, so that the OS is not aggressively
> deciding that data should be swapped.  The default for vm.swappiness on
> Linux is 60, which is quite aggressive.
>
>> Questions :
>>
>> Is there progress regarding this?
>
> As mentioned previously, there's nothing Solr or Lucene can do to help,
> because it's software completely outside our ability to influence.  The
> operating system will make those decisions.
>
> If your software tries to use more memory than the machine has, then
> swap is going to get used no matter how the OS is configured, and when
> that happens, performance will suffer greatly.  In the case of
> SolrCloud, it would make basic operation go so slowly that timeouts
> would get exceeded, and Solr would initiate recovery.
>
> If the OS is Linux, I would like to see a screenshot from the "top"
> program.  (not htop, or anything else, the program needs to be top).
> Run the program, press shift-M to sort the list by memory, and grab a
> screenshot.
>
> Thanks,
> Shawn
>