Best way to throttle hadoop network bandwidth?

[raymond <rg...@163.com>]

Hi

We have a need to throttle hadoop network bandwidth across nodes. You might ask why, the reason is that we need to limit the overall bandwidth consumed by Hadoop cluster so that it can live happily with other servers located under the same switches . we know it can also be solved by rearrange server and carefully plan the network topology, or a reasonable config on the switch might works. But we have trouble to apply those approaching. For now, we need a solution to cut down the network bandwidth consumed by hadoop to say 60MB/s instead of the full bandwidth 120MB/s .

We have tried many approaches, but none of them works well.


1. we try to use TC to throttle network interface’s bandwidth.

we tried something like :

tc qdisc add dev eth0 root tbf rate 480Mbit burst 1M latency 10ms , on centos6.x, we hope by control every node’s outbound bandwidth to reach our goal.

but it seems that it doesn’t works quite well with hadoop workload. A simple scp test run’s well, but with a heavy loaded hadoop cluster, we encounter several random kernel crashes in just a couple hours, and the core dump show something like:

<1>BUG: unable to handle kernel NULL pointer dereference at 0000000000000010

or

<0>Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2 

in the latter case, it seems that a dequeue operation on qdisc last too long time, also the interrupt are masked. then trigger the watchdog.

it seems like a kernel’s bug ( not quite sure, but some clue shows that 3.1 kernel fixed it)   or we don’t made the right config. or hadoop workload just doesn’t work well under this case. we know a lot of people using tc to do the traffic control, but we just failed to make it run long enough.


2. we try to use iptables to limit the bandwidth like:

iptables -A OUTPUT -m limit  --limit 10000/sec --limit-burst 10000 -j ACCEPT
iptables -A OUTPUT -j REJECT

however, the —limit parameter only accept numbers smaller than 10k , thus it can only provide 15MB/s bandwidth. 

3. we try to tune tcp data transfer win size by adjust tcp sending / receiving buffer etc. 

It can somehow throttle the bandwidth, but it is not accurate, there is no obvious relation between the buffer size and transfer speed. And I believe it depends on cpu load and the actual throughput can vary a lot. So I doubt this way can work well.


4. turn to Hadoop itself, 

there were several issue related to data node bandwidth control 

https://issues.apache.org/jira/browse/HDFS-9796 <https://issues.apache.org/jira/browse/HDFS-9796>
https://issues.apache.org/jira/browse/HDFS-7270 <https://issues.apache.org/jira/browse/HDFS-7270>
https://issues.apache.org/jira/browse/HDFS-7265 <https://issues.apache.org/jira/browse/HDFS-7265>

but they seems like applied on disk IO etc. none of them can solve our problem.

And

https://issues.apache.org/jira/browse/HDFS-8683 <https://issues.apache.org/jira/browse/HDFS-8683>

this one aim to do flow control on top of HTTP2, and not yet done.

Anyway, seems no quick solution for our problem. But I believe this should be able to be done from the data node side. 

So, Can anyone give us some advices? Any?

-------
Raymond