You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Adi <ad...@gmail.com> on 2011/03/22 20:36:15 UTC
Advice on mmap related swapping issue
I have been going through the mailing list and compiling suggestions to
address the swapping due to mmap issue.
1) Use JNA (done but)
Are these steps also required:
- Start Cassandra with CAP_IPC_LOCK (or as "root"). (not done)
grep Unevictable /proc/meminfo
- set /proc/sys/vm/swappiness to 0 (not done)
- set initial -Xms max -Xmx heapsize to same value(done)
2) use mmap_index_only
3) Disable swap completely
4) use 'standard' disk access mode and try mmap only if more performance is
needed
I was wondering if there an order of preference for these options based on
experience of folks who have tried these suggestions before I trial and
error my way to what works for us otherwise I am assuming the above order of
preference.
Our cluster Info:
Linux 2.6.18-194.32.1.el5.centos.plus #1 SMP x86_64 x86_64 x86_64 GNU/Linux
Cassandra v 0.7.2 , 4 nodes (JNA enabled), 16GB RAM, 8GB
heapsize(initial/max computed by cassandra-env.sh) , 5 GB swap
Swap space got swamped on all four nodes with a light write only load of
around 5-10 MBps at load size of about 5-6 GB on each node. Heap usage was
not high(around 50%). We are running a 4 node hadoop cluster as client(4
mappers). The nodes were responding well but the swapping is sending alerts
and waking up our sys admin and we want to configure this correctly before
it becomes a problem.
INFO [main] 2011-03-14 12:36:20,276 AbstractCassandraDaemon.java (line 97)
Heap size: 8328839168/8329887744
INFO [main] 2011-03-14 12:36:24,141 CLibrary.java (line 106) JNA mlockall
successful
INFO [main] 2011-03-14 12:36:24,151 DatabaseDescriptor.java (line 121)
Loading settings from
file:/usr/local/apache-cassandra-0.7.2/conf/cassandra.yaml
INFO [main] 2011-03-14 12:36:24,308 DatabaseDescriptor.java (line 181)
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
Re: Advice on mmap related swapping issue
Posted by Daniel Doubleday <da...@gmx.net>.
FWIW: For whatever reason jna memlockall does not work for us. jna call is successful but cassandra process swaps anyway.
see: http://www.mail-archive.com/user@cassandra.apache.org/msg11235.html
We disabled swap entirely.
On Mar 22, 2011, at 8:56 PM, Chris Goffinet wrote:
> The easiest way to get memlock to work after putting the jna jar in your classpath is just run this before:
>
> ulimit -a unlimited
>
> in your init script or before starting cassandra. The default for max locked memory is 32KB on older kernels, and 64KB on newer ones.
>
> -Chris
>
> On Mar 22, 2011, at 12:52 PM, Adi wrote:
>
>> On Tue, Mar 22, 2011 at 3:44 PM, ruslan usifov <ru...@gmail.com> wrote:
>>
>>
>> 2011/3/22 Adi <ad...@gmail.com>
>> I have been going through the mailing list and compiling suggestions to address the swapping due to mmap issue.
>>
>> 1) Use JNA (done but)
>> Are these steps also required:
>> - Start Cassandra with CAP_IPC_LOCK (or as "root"). (not done)
>>
>> And what is CAP_IPC_LOCK?
>>
>> I saw that suggestion in
>> https://issues.apache.org/jira/browse/CASSANDRA-1214.
>>
>> I do not yet know how to run cassandra or a java process with that privilege, still researching and hoping my sysadmin knows better.
>>
>> http://www.lids.org/lids-howto/node50.html
>> Allow locking of shared memory segments
>> Allow mlock and mlockall (which doesn't really have anything to do with IPC)
>>
>
Re: Advice on mmap related swapping issue
Posted by Chris Goffinet <cg...@chrisgoffinet.com>.
The easiest way to get memlock to work after putting the jna jar in your classpath is just run this before:
ulimit -a unlimited
in your init script or before starting cassandra. The default for max locked memory is 32KB on older kernels, and 64KB on newer ones.
-Chris
On Mar 22, 2011, at 12:52 PM, Adi wrote:
> On Tue, Mar 22, 2011 at 3:44 PM, ruslan usifov <ru...@gmail.com> wrote:
>
>
> 2011/3/22 Adi <ad...@gmail.com>
> I have been going through the mailing list and compiling suggestions to address the swapping due to mmap issue.
>
> 1) Use JNA (done but)
> Are these steps also required:
> - Start Cassandra with CAP_IPC_LOCK (or as "root"). (not done)
>
> And what is CAP_IPC_LOCK?
>
> I saw that suggestion in
> https://issues.apache.org/jira/browse/CASSANDRA-1214.
>
> I do not yet know how to run cassandra or a java process with that privilege, still researching and hoping my sysadmin knows better.
>
> http://www.lids.org/lids-howto/node50.html
> Allow locking of shared memory segments
> Allow mlock and mlockall (which doesn't really have anything to do with IPC)
>
Re: Advice on mmap related swapping issue
Posted by Adi <ad...@gmail.com>.
On Tue, Mar 22, 2011 at 3:44 PM, ruslan usifov <ru...@gmail.com>wrote:
>
>
> 2011/3/22 Adi <ad...@gmail.com>
>
>> I have been going through the mailing list and compiling suggestions to
>> address the swapping due to mmap issue.
>>
>> 1) Use JNA (done but)
>> Are these steps also required:
>> - Start Cassandra with CAP_IPC_LOCK (or as "root"). (not done)
>>
>
> And what is CAP_IPC_LOCK?
I saw that suggestion in
https://issues.apache.org/jira/browse/CASSANDRA-1214.
I do not yet know how to run cassandra or a java process with that
privilege, still researching and hoping my sysadmin knows better.
http://www.lids.org/lids-howto/node50.html
- Allow locking of shared memory segments
- Allow mlock and mlockall (which doesn't really have anything to do with
IPC)
Re: Advice on mmap related swapping issue
Posted by ruslan usifov <ru...@gmail.com>.
2011/3/22 Adi <ad...@gmail.com>
> I have been going through the mailing list and compiling suggestions to
> address the swapping due to mmap issue.
>
> 1) Use JNA (done but)
> Are these steps also required:
> - Start Cassandra with CAP_IPC_LOCK (or as "root"). (not done)
>
And what is CAP_IPC_LOCK?