Ramdisk for dataDir and dataLogDir?

[jlindwall <jl...@yahoo.com>]

[Sorry for the repeat message. The website-based mailing list subscription
malfunctioned and I needed to manually subscribe]

Is it possible to use a ramdisk for dataDir and/or dataLogDir? 

My sysadmins are not interested in setting up special disk volumes for
zookeeper (sigh).  They want to use a ramdisk for zookeeper's data storage. 

The ensemble will be split across physical machines of course, as much as
possible.  If a node fails, obviously the contents of the ramdisk are lost. 
When we restart that machine and fire up zoookeeper, will that zookeeper
node synch up properly? 

The zookeeper admin guide suggest so; even saying to delete version-2 dirs
from dataDir and dataLogDir if database corruption occurs on a node.  The
caveat being: the other servers in the ensemble are still up. 

Assuming this will work functionally, what are the performance and stability
implications of this choice?  Any special risks?   

Thanks! 



--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/Ramdisk-for-dataDir-and-dataLogDir-tp7580323.html
Sent from the zookeeper-user mailing list archive at Nabble.com.

RE: Ramdisk for dataDir and dataLogDir?

[Rakesh R <ra...@huawei.com>]

I'd like to share one note.

Say, you have a very large datastore(depends on your system). During startup, it can take considerable amount time for the leader to communicate the znode data to the followers(considering the n/w transfer rate). initLimit is the amount of time the leader will allow the followers to get the data from the leader and get to a point where they are ready to serve requests to clients. If this time is exceeded the leader will close the connection to the follower and the quorum process will restart. 

Please have a look at the 'initLimit' and 'syncLimit' parameters in zookeeper and tune it.
Probably you can have a test with large datastore(in GBs). I'm interested to know few things like:
1) zk version
2) are you using ramfs or tmpfs(I think doesn't matter to zk service, but for my understanding)

Also, it would be great if you can share the results in the community. All the best!

Cheers,
Rakesh
-----Original Message-----
From: jlindwall [mailto:jlindwall@yahoo.com] 
Sent: 25 September 2014 00:21
To: zookeeper-user@hadoop.apache.org
Subject: Re: Ramdisk for dataDir and dataLogDir?

I simulated this situation by bringing down one node, deleting its version-2 directory tree, and restarting the node.  It rejoined and synched up to the existing ensemble just fine.

>From that evidence and the verbiage in the admin guide, I'll trust in this approach to work.  Of course, I realize when we bring up a failed node in this manner it needs to do a full sync with the leader which will involve a single SNAP transfer and some transaction log replays.  We are willing to accept that price.



--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/Ramdisk-for-dataDir-and-dataLogDir-tp7580323p7580331.html
Sent from the zookeeper-user mailing list archive at Nabble.com.

Re: Ramdisk for dataDir and dataLogDir?

[jlindwall <jl...@yahoo.com>]

I simulated this situation by bringing down one node, deleting its version-2
directory tree, and restarting the node.  It rejoined and synched up to the
existing ensemble just fine.

>From that evidence and the verbiage in the admin guide, I'll trust in this
approach to work.  Of course, I realize when we bring up a failed node in
this manner it needs to do a full sync with the leader which will involve a
single SNAP transfer and some transaction log replays.  We are willing to
accept that price.



--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/Ramdisk-for-dataDir-and-dataLogDir-tp7580323p7580331.html
Sent from the zookeeper-user mailing list archive at Nabble.com.