You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by gu...@zte.com.cn on 2017/08/18 05:19:48 UTC
JVM OPTS about HDFS
SGkgQWxsLA0KDQoNCiAgICBIREZTIEZlZGVyYXRpb24gd2l0aCBQQisgcmVzdCBkYXRhIChTaW5n
bGUgTmFtZSBTZXJ2aWNlIGlzIEhBLCBCYXNlZCBvbiBRSk0pICwgIEFwYWNoZSAyLjcuMyBvbiBS
ZWRoYXQgNi41IHdpdGggSkRLMS43Lg0KDQoNCg0KDQoNCg0KICAgIDEuUGxhbiB0byBkZXBsb3kg
Tk4gb24gc2VydmVyKDMyY29yZXMsIDUxMkcpICwgYW55IHByZWNpb3VzIGFkdmljZSBhYm91dCBK
Vk0gT1BUUz8gIElmIHNldCBoZWFwIHNpemUgdG8gYWJvdXQgNDAwRyB3aXRoIENNUyBHQyBjb2xs
ZWN0b3IsIGFueSBvYnZpb3VzIHByb2JsZW1zPw0KDQoNCg0KDQoNCg0KICAgIDIuSWYgdGhlcmUg
YXJlIG1hbnkgZ3JvdXBzIG9mIE5hbWUgU2VydmljZXMsIGlzIGl0IG1vcmUgZWZmaWNpZW50IHRo
YXQgcGFydCBvZiBOYW1lIFNlcnZpY2VzIHNoYXJlIG9uZSBncm91cCBvZiBKTnM/IEFueSBhZHZp
Y2UgdG8gSk4/DQoNCg0KDQoNCg0KDQogICAgMy5XZWxjb21lIGFueSB3b3JkcyB0byBGZWRlcmF0
aW9uLCB0aGFua3MhDQoNCg0KDQoNCg0KDQoNClRoYW5rcyBpbiBhZHZhbmNlLA0KDQoNCkRvcmlz
Re: JVM OPTS about HDFS
Posted by Akash Mishra <ak...@gmail.com>.
I am currently supporting Single Name Service is HA, Based on QJM with 0.9
PT data with 55-58 Million Object [ files + Blocks ] with 36G of JVM heap
with G1GC.
I would recommend starting with 16G and scale depending on your blocks with
G1GC Garbage collection.
Thanks,
On Fri, Aug 18, 2017 at 4:42 PM, Gurmukh Singh <
gurmukh.dhillon@yahoo.com.invalid> wrote:
> 400GB as heap space for Namenode is bit high. The GC pause time will be
> very high.
>
> For a cluster with about 6PB, approx 20GB is decent memory.
>
> As you mentioned it is HA, so it is safe to assume that the fsimage is
> check pointed at regular intervals and we do not need to worry during a
> manual restart of namenode, about the memory to play edits into fsimage.
> But still it is good to account for as delta. But still not 400GB.
>
>
> A good way to estimate:
>
> Some of my tests:
>
> writing about 2TB of data on HDFS with block size = 128 MB, replication 3
> - creates about 18k blocks (18051).
>
> *Memory needed for that blocks:*
>
> hdfs oiv -p XML -printToScreen -i /mnt/namenode/current/fsimage_0000000000000051228
> | egrep "block|inode" | wc -l | awk '{printf "Objects=%d : Suggested
> Xms=%0dm Xmx=%0dm\n", $1, (($1 / 1000000 )*1024), (($1 / 1000000 )*1024)}'
>
> Objects=18051 : Suggested Xms=18m Xmx=18m
>
>
>
> *Maths for Cluster:*
> ----------------
>
> 150 bytes per object(object is block, file, directory)
>
> 24 TB x 2000 nodes = 48000 TB
>
> Block size = 128 MB
>
> Total blocks = 48000TB/128MB = 393216000 Blocks
>
> Adjusting for replication factor, which is 3 by default. As each
> replicated block just takes about 16 bytes in memory of namenode.
>
> 393216000/3 = 131072000 x 150 + (16 bytes x 131072000 blocks) =
> 19660800000 + 2097152000 =*20.23 GB*
>
> In addition to this memory is needed for namespace metadata -> Each file
> name will also be accounted for 150 bytes of Namenode memory
>
>
>
> On 18/8/17 3:19 pm, gu.yizhou@zte.com.cn wrote:
>
> Hi All,
>
> HDFS Federation with PB+ rest data (Single Name Service is HA, Based
> on QJM) , Apache 2.7.3 on Redhat 6.5 with JDK1.7.
>
>
> 1.Plan to deploy NN on server(32cores, 512G) , any precious advice
> about JVM OPTS? If set heap size to about 400G with CMS GC collector, any
> obvious problems?
>
>
> 2.If there are many groups of Name Services, is it more efficient
> that part of Name Services share one group of JNs? Any advice to JN?
>
>
> 3.Welcome any words to Federation, thanks!
>
>
> Thanks in advance,
>
> Doris
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>
>
--
Regards,
Akash Mishra.
"It's not our abilities that make us, but our decisions."--Albus Dumbledore
Re: JVM OPTS about HDFS
Posted by Gurmukh Singh <gu...@yahoo.com.INVALID>.
400GB as heap space for Namenode is bit high. The GC pause time will be
very high.
For a cluster with about 6PB, approx 20GB is decent memory.
As you mentioned it is HA, so it is safe to assume that the fsimage is
check pointed at regular intervals and we do not need to worry during a
manual restart of namenode, about the memory to play edits into fsimage.
But still it is good to account for as delta. But still not 400GB.
A good way to estimate:
Some of my tests:
writing about 2TB of data on HDFS with block size = 128 MB, replication
3 - creates about 18k blocks (18051).
*Memory needed for that blocks:*
hdfs oiv -p XML -printToScreen -i
/mnt/namenode/current/fsimage_0000000000000051228 | egrep "block|inode"
| wc -l | awk '{printf "Objects=%d : Suggested Xms=%0dm Xmx=%0dm\n", $1,
(($1 / 1000000 )*1024), (($1 / 1000000 )*1024)}'
Objects=18051 : Suggested Xms=18m Xmx=18m
*Maths for Cluster:*
----------------
150 bytes per object(object is block, file, directory)
24 TB x 2000 nodes = 48000 TB
Block size = 128 MB
Total blocks = 48000TB/128MB = 393216000 Blocks
Adjusting for replication factor, which is 3 by default. As each
replicated block just takes about 16 bytes in memory of namenode.
393216000/3 = 131072000 x 150 + (16 bytes x 131072000 blocks) =
19660800000 + 2097152000 =*20.23 GB*
In addition to this memory is needed for namespace metadata -> Each file
name will also be accounted for 150 bytes of Namenode memory
On 18/8/17 3:19 pm, gu.yizhou@zte.com.cn wrote:
>
> Hi All,
>
> HDFS Federation with PB+ rest data (Single Name Service is HA, Based
> on QJM) , Apache 2.7.3 on Redhat 6.5 with JDK1.7.
>
>
> 1.Plan to deploy NN on server(32cores, 512G) , any precious advice
> about JVM OPTS? If set heap size to about 400G with CMS GC collector,
> any obvious problems?
>
>
> 2.If there are many groups of Name Services, is it more efficient
> that part of Name Services share one group of JNs? Any advice to JN?
>
>
> 3.Welcome any words to Federation, thanks!
>
>
> Thanks in advance,
>
> Doris
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org