You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Abdul I Mohammed <or...@gmail.com> on 2015/03/25 21:45:22 UTC
Swap requirements
Hello all,
I am trying to figure out what are the swap requirements for name node and data nodes? Some vendor docs that I read says to set vm.swappiness to 0 which is telling kernel to not to use swap?
What is the default settings the community is using for there clusters?
Also does the below parameter uses swap memory or regular memory in yarn?
Yarn.nodemanager.Vmem-pmem-ratio
RE: Swap requirements
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Hi,
I do not think DataNodes (DN) require any swapping as swap is only used for paging in case of running out of memory.
NameNodes (NN) behave like an in-memory database IMDB (like Oracle TimesTen) and uses a disk file system called the FsImage to load the metadata as startup. For the remaining period until HDFS is shutdown or otherwise NN will use the in memory cache to access metadata. Now the only occasion I see the swap space will be used is when NN runs out of memory and start swapping to disk. That is using swap space like memory which is undesirable. Not very clear to me.
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
-----Original Message-----
From: Abdul I Mohammed [mailto:oracle.blog3@gmail.com]
Sent: 25 March 2015 20:45
To: HDP mailing list
Subject: Swap requirements
Hello all,
I am trying to figure out what are the swap requirements for name node and data nodes? Some vendor docs that I read says to set vm.swappiness to 0 which is telling kernel to not to use swap?
What is the default settings the community is using for there clusters?
Also does the below parameter uses swap memory or regular memory in yarn?
Yarn.nodemanager.Vmem-pmem-ratio
RE: Swap requirements
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Hi,
I do not think DataNodes (DN) require any swapping as swap is only used for paging in case of running out of memory.
NameNodes (NN) behave like an in-memory database IMDB (like Oracle TimesTen) and uses a disk file system called the FsImage to load the metadata as startup. For the remaining period until HDFS is shutdown or otherwise NN will use the in memory cache to access metadata. Now the only occasion I see the swap space will be used is when NN runs out of memory and start swapping to disk. That is using swap space like memory which is undesirable. Not very clear to me.
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
-----Original Message-----
From: Abdul I Mohammed [mailto:oracle.blog3@gmail.com]
Sent: 25 March 2015 20:45
To: HDP mailing list
Subject: Swap requirements
Hello all,
I am trying to figure out what are the swap requirements for name node and data nodes? Some vendor docs that I read says to set vm.swappiness to 0 which is telling kernel to not to use swap?
What is the default settings the community is using for there clusters?
Also does the below parameter uses swap memory or regular memory in yarn?
Yarn.nodemanager.Vmem-pmem-ratio
RE: Swap requirements
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Hi,
I do not think DataNodes (DN) require any swapping as swap is only used for paging in case of running out of memory.
NameNodes (NN) behave like an in-memory database IMDB (like Oracle TimesTen) and uses a disk file system called the FsImage to load the metadata as startup. For the remaining period until HDFS is shutdown or otherwise NN will use the in memory cache to access metadata. Now the only occasion I see the swap space will be used is when NN runs out of memory and start swapping to disk. That is using swap space like memory which is undesirable. Not very clear to me.
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
-----Original Message-----
From: Abdul I Mohammed [mailto:oracle.blog3@gmail.com]
Sent: 25 March 2015 20:45
To: HDP mailing list
Subject: Swap requirements
Hello all,
I am trying to figure out what are the swap requirements for name node and data nodes? Some vendor docs that I read says to set vm.swappiness to 0 which is telling kernel to not to use swap?
What is the default settings the community is using for there clusters?
Also does the below parameter uses swap memory or regular memory in yarn?
Yarn.nodemanager.Vmem-pmem-ratio
RE: Swap requirements
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Hi,
I do not think DataNodes (DN) require any swapping as swap is only used for paging in case of running out of memory.
NameNodes (NN) behave like an in-memory database IMDB (like Oracle TimesTen) and uses a disk file system called the FsImage to load the metadata as startup. For the remaining period until HDFS is shutdown or otherwise NN will use the in memory cache to access metadata. Now the only occasion I see the swap space will be used is when NN runs out of memory and start swapping to disk. That is using swap space like memory which is undesirable. Not very clear to me.
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
-----Original Message-----
From: Abdul I Mohammed [mailto:oracle.blog3@gmail.com]
Sent: 25 March 2015 20:45
To: HDP mailing list
Subject: Swap requirements
Hello all,
I am trying to figure out what are the swap requirements for name node and data nodes? Some vendor docs that I read says to set vm.swappiness to 0 which is telling kernel to not to use swap?
What is the default settings the community is using for there clusters?
Also does the below parameter uses swap memory or regular memory in yarn?
Yarn.nodemanager.Vmem-pmem-ratio