Netty connection errors

[Piotr Kozikowski <pi...@liveramp.com>]

Hi all,

We're are experiencing transient connection errors like the one below. It
doesn't seem to be a firewall issue since sometimes the job succeeds. Is
this a common symptom with a known cause/fix? Also, it seems very strange
to me that there is no wait period between connection attempts, and I can't
find any setting to modify this. The default number of 1000 attempts seems
pretty generous, but if they all happen one right after another with no
sleep in between it's effectively not much better than a single attempt.

Any thoughts?

Thank you,
PIotr

2016-02-11 18:02:35,002 WARN [main]
org.apache.giraph.comm.netty.NettyClient: connectAllAddresses: Future
failed to connect with ds0604.liveramp.net/10.100.132.96:31030 with
1999 failures because of java.net.ConnectException: Connection
refused: ds0604.liveramp.net/10.100.132.96:31030
2016-02-11 18:02:35,002 WARN [main]
org.apache.giraph.comm.netty.NettyClient: connectAllAddresses: Future
failed to connect with ds0604.liveramp.net/10.100.132.96:31440 with
2000 failures because of java.net.ConnectException: Connection
refused: ds0604.liveramp.net/10.100.132.96:31440
2016-02-11 18:02:35,002 INFO [netty-client-worker-3]
org.apache.giraph.comm.netty.NettyClient: Using Netty without
authentication.
2016-02-11 18:02:35,002 INFO [main]
org.apache.giraph.comm.netty.NettyClient: connectAllAddresses:
Successfully added 0 connections, (1997 total connected) 3 failed,
2001 failures total.
2016-02-11 18:02:35,002 INFO [netty-client-worker-0]
org.apache.giraph.comm.netty.NettyClient: Using Netty without
authentication.
2016-02-11 18:02:35,003 ERROR [main]
org.apache.giraph.graph.GraphMapper: Caught an unrecoverable exception
connectAllAddresses: Too many failures (2001).
java.lang.IllegalStateException: connectAllAddresses: Too many failures (2001).
	at org.apache.giraph.comm.netty.NettyClient.connectAllAddresses(NettyClient.java:488)
	at org.apache.giraph.comm.netty.NettyWorkerClient.openConnections(NettyWorkerClient.java:132)
	at org.apache.giraph.comm.netty.NettyWorkerClient.setup(NettyWorkerClient.java:168)
	at org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:584)
	at org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:284)
	at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:93)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)