You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Runping Qi <ru...@yahoo-inc.com> on 2008/12/18 01:16:22 UTC

Warning on turning on ipv6 on your Hadoop clusters


If you may have turned on ipv6 on your hadoop cluster, it may cause severe
performance hit!

When I ran the gridmix2 benchmark on a newly constructed cluster, it took
30% more time than the baseline time that was obtained on a similar cluster.

I noticed that some task processes on some machines took 3+ minutes to
initialize.
After examining these processes in details, I found that they were stuck at
socket initialization tile, as shown in the following stack:

"main" prio=10 tid=0x0805b400 nid=0x4681 runnable [0xf7fbb000..0xf7fbc208]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.initProto(Native Method)
    at java.net.PlainSocketImpl.<clinit>(PlainSocketImpl.java:84)
    at java.net.Socket.setImpl(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:68)
    at sun.nio.ch.SocketAdaptor.<init>(SocketAdaptor.java:50)
    at sun.nio.ch.SocketAdaptor.create(SocketAdaptor.java:55)
    at sun.nio.ch.SocketChannelImpl.socket(SocketChannelImpl.java:105)
    - locked <0xf17a38c8> (a java.lang.Object)
    at 
org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFacto
ry.java:58)
    at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:298)
    - locked <0xf1795db0> (a org.apache.hadoop.ipc.Client$Connection)
    at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)
    at org.apache.hadoop.ipc.Client.call(Client.java:705)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
    at org.apache.hadoop.mapred.$Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372)
    at 
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2188)


I did a search on the web and found that that was due to a known bug for
Java related to ipv6.

More information about the bug can be found at the following two pages:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6483406
http://edocs.bea.com/jrockit/releases/5026x/relnotes/relnotes.html

{quote}
    Slow startup because of a hang in java.net.PlainSocketImpl.initProto(),
which typically is called when creating the first Socket or    ServerSocket.
    In BEA JRockit 5.0 R26 the network stack is configured so that IPv6 is
used in preference to IPv4 when it is present.
    During initialization of the network stack, the network code connects a
socket to its own loopback interface to set up some data   structures.
Blocking this connection (e.g. with a firewall) will cause the
initialization code to wait for a socket timeout, after which   the system
falls back on using IPv4.


{quote}

Suggested Workaround:

Either set -Djava.net.preferIPv4Stack=true for the child process option,
which forces Java to use IPv4
instead, or you disable IPv6 entirely in the system. The proper fix is to
allow IPv6 traffic from localhost to localhost.
For more information, see the Sun documentation:
http://java.sun.com/j2se/1.4.2/docs/guide/net/ipv6_guide/#ipv6-networking



Runping

Re: Warning on turning on ipv6 on your Hadoop clusters

Posted by Steve Loughran <st...@apache.org>.
Runping Qi wrote:
> 
> If you may have turned on ipv6 on your hadoop cluster, it may cause severe
> performance hit!
> 

> Suggested Workaround:
> 
> Either set -Djava.net.preferIPv4Stack=true for the child process option,
> which forces Java to use IPv4
> instead, or you disable IPv6 entirely in the system. The proper fix is to
> allow IPv6 traffic from localhost to localhost.
> For more information, see the Sun documentation:
> http://java.sun.com/j2se/1.4.2/docs/guide/net/ipv6_guide/#ipv6-networking
> 
> 

Seems to me, disabling IPv6 is a legit action on any network that thinks 
addresses should fit in 32 bits. Linux desktop perf with IPv6 turned on 
(including firefox performance) is worse, hence turning it off is a well 
known action, one of the simple ways to speed up your box

http://ubuntuforums.org/showthread.php?t=6841


my own modprobe.d/aliases files hint that the test for ipv6 live is the 
line:

  ip a | grep inet6

Re: Warning on turning on ipv6 on your Hadoop clusters

Posted by Steve Loughran <st...@apache.org>.
Runping Qi wrote:
> 
> If you may have turned on ipv6 on your hadoop cluster, it may cause severe
> performance hit!
> 

> Suggested Workaround:
> 
> Either set -Djava.net.preferIPv4Stack=true for the child process option,
> which forces Java to use IPv4
> instead, or you disable IPv6 entirely in the system. The proper fix is to
> allow IPv6 traffic from localhost to localhost.
> For more information, see the Sun documentation:
> http://java.sun.com/j2se/1.4.2/docs/guide/net/ipv6_guide/#ipv6-networking
> 
> 

Seems to me, disabling IPv6 is a legit action on any network that thinks 
addresses should fit in 32 bits. Linux desktop perf with IPv6 turned on 
(including firefox performance) is worse, hence turning it off is a well 
known action, one of the simple ways to speed up your box

http://ubuntuforums.org/showthread.php?t=6841


my own modprobe.d/aliases files hint that the test for ipv6 live is the 
line:

  ip a | grep inet6