hdfs swapping

[Panagiotis Atmatzidis <at...@convalesco.org>]

Hello,

I have a hadoop cluster where each server has 128 GB RAM. I noticed that in some servers hdfs (java process) swaps. To give an idea:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
$ for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done|grep java
java 14352 kB
java 39956 kB
java 148416 kB
java 16848 kB

$ free -m
             total       used       free     shared    buffers     cached
Mem:        128835     128181        653         11        105      90162
-/+ buffers/cache:      37913      90921
Swap:          511        233        278

$ vmstat -w -S M
procs ---------------memory-------------- ---swap-- -----io---- -system-- ------cpu-----
 r  b     swpd     free     buff    cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      233      673      105    90140    0    0   863  1644    0    0  6  2 91  1  0

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

In cloudera’s documentation says[1] that swappiness should be setup to ‘0’ while a more recent blogpost[1] says it’s safer to set it up to ‘1’.

Are there any recommendations and/or best practices? I’d like to know what other people use in their clusters.

According to this SO[3] thread if a java app is swapping, there’s not much I can do. In my case there’s lots of extra RAM, are there any other options I missed, I can tweak to avoid such behaviour while keeping swampiness to ‘1’?

Thanks,


[1] https://www.cloudera.com/documentation/enterprise/5-2-x/topics/cdh_admin_performance.html#xd_583c10bfdbd326ba-7dae4aa6-147c30d0933--7fd5__section_ovb_wcy_kq <https://www.cloudera.com/documentation/enterprise/5-2-x/topics/cdh_admin_performance.html#xd_583c10bfdbd326ba-7dae4aa6-147c30d0933--7fd5__section_ovb_wcy_kq>

[2] http://blog.cloudera.com/blog/2015/01/how-to-deploy-apache-hadoop-clusters-like-a-boss/ <http://blog.cloudera.com/blog/2015/01/how-to-deploy-apache-hadoop-clusters-like-a-boss/>

[3] http://stackoverflow.com/a/9306054

Panagiotis (atmosx) Atmatzidis

email:	atma@convalesco.org
URL:	http://www.convalesco.org
GnuPG ID: 0x1A7BFEC5
gpg --keyserver pgp.mit.edu --recv-keys 1A7BFEC5

"Everyone thinks of changing the world, but no one thinks of changing himself.” - Leo Tolstoy