Re: Dangers of "sudo swapoff --all"

[Robert Coli <rc...@eventbrite.com>]

On Thu, Feb 13, 2014 at 2:57 PM, Donald Smith <
Donald.Smith@audiencescience.com> wrote:

>  I followed the recommendations at
> http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/install/installRecommendSettings.htmland did:
>
>
>
> $ sudo swapoff -all
>
>
>
> on each of the cassandra servers in my test cluster.
>
>
>
> I noticed, though, that sometimes the cassandra server and other processes
> on one of the nodes suddenly crashed, with no messages indicating why.
>

That is by design; as you have mentioned, you have run out of memory and
the OOM (out of memory) killer has picked some processes to kill.

Cassandra stores an assortment of data (bloom filters, index summaries,
caches) off the (mlockall-ed, and therefore not subject to swapping [1]
heap). If you have swap defined on the system, the kernel *will* swap some
of this data. Once it has been swapped to disk, accesses which Cassandra
expects to be to RAM and at RAM speeds will be served from disk. You
presumably do not want this while trying to produce predictable performance
from a production Cassandra node.

The solution to your immediate problem is to reduce memory consumption.
Swap could be ok on QA or dev or developer machine, but if you absolutely
must use it, use a kernel 2.6.36 or later so that you can use oom_score_adj
to avoid killing things like sshd. [2]

=Rob
[1] https://issues.apache.org/jira/browse/CASSANDRA-1214
[2]
https://github.com/mirrors/linux-2.6/commit/a63d83f427fbce97a6cea0db2e64b0eb8435cd10#include/linux/oom.h