You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sreeram <sr...@gmail.com> on 2017/05/19 05:31:25 UTC
How to improve HBase replication throughput between data centers ?
Hi All,
I have setup HBase replication between two clusters containing 25
nodes each. The inter-data center network link has a capacity of 500
MBPS.
I have been running some tests to understand the speed of replication.
I am observing that the replication speed does not go more than 5
MBPS.
On reading up regarding the same, I understand that the speed of data
transfer depends on OS level TCP socket read and write buffer sizes.
Below are the OS level parameters that I see for the socket size
# cat /proc/sys/net/ipv4/tcp_wmem
4096 (min) 16384 (default) 4194304 (max)
# cat /proc/sys/net/ipv4/tcp_rmem
4096 (min) 87380 (default) 6291456 (max)
The default write buffer size for sockets is 16KB and the read buffer
size is around 85KB.
There are suggestions [1] to set higher values for the default read
and write buffers to fully utilize the link capacity.
But I am not sure how to influence HBase to use higher values for the
socket read/write buffers when it does replication.
Any thoughts from the community on the same?
Thanks
Sreeram
[1] http://www.onlamp.com/pub/a/onlamp/2005/11/17/tcp_tuning.html