You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by "von Domarus, Marven" <ma...@rwth-aachen.de> on 2015/09/04 18:53:16 UTC
NettyClient java.net.ConnectException: Connection refused
Hi everyone,
I have already run the SimpleShortestPathsComputation on single node pseudo-distribution and I am currently trying to run it on a 2 node cluster on Ubuntu 14.04.
But every time I got the following Error: (Complete Logs below)
"ERROR org.apache.giraph.master.MasterThread: masterThread: Master algorithm failed with IllegalStateException java.lang.IllegalStateException: connectAllAddresses: Too many failures (1000)."
Which seems to be the result of the NettyClient not being able to connect with the other node on port 30001 resulting in "java.net.ConnectException: Connection refused: rofti/192.168.0.103:30001".
I already tried to open the port on both nodes as described here: http://mail-archives.apache.org/mod_mbox/giraph-user/201310.mbox/%3CCAAcgib35tQyROodNQrE3Se_39JsT5MiQVP=4v==X10jtOUfSVA@mail.gmail.com%3E
and made sure that all host-names are written in only lowercase letters. But that didn't change anything. I really appreciate any help you guys can offer.
Thanks,
Marven
Output of iptables -L :
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere state NEW tcp spt:30001
ACCEPT tcp -- anywhere anywhere state NEW tcp dpts:30000:30010
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:2181
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
File: /etc/hosts on master(rofti-ex58-ud3r):
127.0.0.1 localhost
127.0.1.1 rofti-ex58-ud3r
192.168.0.101 master
192.168.0.103 slave rofti
File: conf/zoo.cfg
tickTime=2000
dataDir=/home/hduser/mydata/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=192.168.0.101:2888:3888
server.2=192.168.0.103:2888:3888
Syslog logs 1:
2015-09-04 17:43:46,416 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2015-09-04 17:43:46,661 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2015-09-04 17:43:46,784 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2015-09-04 17:43:46,787 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3c1e6a1d
2015-09-04 17:43:46,884 INFO org.apache.hadoop.mapred.MapTask: Processing split: 'org.apache.giraph.bsp.BspInputSplit, index=-1, num=-1
2015-09-04 17:43:46,927 INFO org.apache.giraph.graph.GraphTaskManager: setup: Log level remains at info
2015-09-04 17:43:46,967 INFO org.apache.giraph.graph.GraphTaskManager: setup: Starting up BspServiceMaster (master thread)...
2015-09-04 17:43:46,982 INFO org.apache.giraph.bsp.BspService: BspService: Path to create to halt is /_hadoopBsp/job_201509041740_0002/_haltComputation
2015-09-04 17:43:46,982 INFO org.apache.giraph.bsp.BspService: BspService: Connecting to ZooKeeper with job job_201509041740_0002, 0 on master:2181,slave:2181
2015-09-04 17:43:46,987 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2015-09-04 17:43:46,987 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=rofti-ex58-ud3r
2015-09-04 17:43:46,987 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.7.0_79
2015-09-04 17:43:46,987 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/jars/classes:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/jars:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000000_0/work:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.2.1.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.11.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.11.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-3.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar:/usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000000_0/work
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000000_0/work/tmp
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.16.0-38-generic
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hduser
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hduser
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000000_0/work
2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave:2181 sessionTimeout=60000 watcher=org.apache.giraph.master.BspServiceMaster@77084cb5
2015-09-04 17:43:47,003 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server slave/192.168.0.103:2181. Will not attempt to authenticate using SASL (unknown error)
2015-09-04 17:43:47,005 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to slave/192.168.0.103:2181, initiating session
2015-09-04 17:43:47,225 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server slave/192.168.0.103:2181, sessionid = 0x24f9906beb90000, negotiated timeout = 40000
2015-09-04 17:43:47,227 INFO org.apache.giraph.bsp.BspService: process: Asynchronous connection complete.
2015-09-04 17:43:47,232 INFO org.apache.giraph.graph.GraphTaskManager: map: No need to do anything when not a worker
2015-09-04 17:43:47,232 INFO org.apache.giraph.graph.GraphTaskManager: cleanup: Starting for MASTER_ONLY
2015-09-04 17:43:47,420 INFO org.apache.giraph.master.BspServiceMaster: becomeMaster: First child is '/_hadoopBsp/job_201509041740_0002/_masterElectionDir/rofti-ex58-ud3r_00000000000' and my bid is '/_hadoopBsp/job_201509041740_0002/_masterElectionDir/rofti-ex58-ud3r_00000000000'
2015-09-04 17:43:47,490 INFO org.apache.giraph.comm.netty.NettyServer: NettyServer: Using execution group with 8 threads for requestFrameDecoder.
2015-09-04 17:43:47,539 INFO org.apache.giraph.comm.netty.NettyServer: start: Started server communication server: rofti-ex58-ud3r/127.0.1.1:30000 with up to 16 threads on bind attempt 0 with sendBufferSize = 32768 receiveBufferSize = 524288
2015-09-04 17:43:47,543 INFO org.apache.giraph.comm.netty.NettyClient: NettyClient: Using execution handler with 8 threads after request-encoder.
2015-09-04 17:43:47,546 INFO org.apache.giraph.master.BspServiceMaster: becomeMaster: I am now the master!
2015-09-04 17:43:47,823 INFO org.apache.giraph.bsp.BspService: process: applicationAttemptChanged signaled
2015-09-04 17:43:48,013 WARN org.apache.giraph.bsp.BspService: process: Unknown and unprocessed event (path=/_hadoopBsp/job_201509041740_0002/_applicationAttemptsDir/0/_superstepDir, type=NodeChildrenChanged, state=SyncConnected)
2015-09-04 17:43:54,587 INFO org.apache.giraph.io.formats.GiraphFileInputFormat: Total input paths to process : 1
2015-09-04 17:43:54,597 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2015-09-04 17:43:54,598 INFO org.apache.giraph.master.BspServiceMaster: generateVertexInputSplits: Got 1 input splits for 1 input threads
2015-09-04 17:43:54,598 INFO org.apache.giraph.master.BspServiceMaster: createVertexInputSplits: Starting to write input split data to zookeeper with 1 threads
2015-09-04 17:43:54,668 INFO org.apache.giraph.master.BspServiceMaster: createVertexInputSplits: Done writing input split data to zookeeper
2015-09-04 17:43:54,729 INFO org.apache.giraph.comm.netty.NettyClient: Using Netty without authentication.
2015-09-04 17:43:54,741 WARN org.apache.giraph.comm.netty.NettyClient: connectAllAddresses: Future failed to connect with rofti/192.168.0.103:30001 with 0 failures because of java.net.ConnectException: Connection refused: rofti/192.168.0.103:30001
2015-09-04 17:43:54,742 INFO org.apache.giraph.comm.netty.NettyClient: connectAllAddresses: Successfully added 0 connections, (0 total connected) 1 failed, 1 failures total.
2015-09-04 17:43:54,742 INFO org.apache.giraph.comm.netty.NettyClient: Using Netty without authentication.
2015-09-04 17:43:54,745 WARN org.apache.giraph.comm.netty.NettyClient: connectAllAddresses: Future failed to connect with rofti/192.168.0.103:30001 with 1 failures because of java.net.ConnectException: Connection refused: rofti/192.168.0.103:30001
[....]
2015-09-04 17:44:12,314 INFO org.apache.giraph.comm.netty.NettyClient: connectAllAddresses: Successfully added 0 connections, (0 total connected) 1 failed, 999 failures total.
2015-09-04 17:44:12,314 INFO org.apache.giraph.comm.netty.NettyClient: Using Netty without authentication.
2015-09-04 17:44:12,316 WARN org.apache.giraph.comm.netty.NettyClient: connectAllAddresses: Future failed to connect with rofti/192.168.0.103:30001 with 999 failures because of java.net.ConnectException: Connection refused: rofti/192.168.0.103:30001
2015-09-04 17:44:12,317 INFO org.apache.giraph.comm.netty.NettyClient: connectAllAddresses: Successfully added 0 connections, (0 total connected) 1 failed, 1000 failures total.
2015-09-04 17:44:12,317 INFO org.apache.giraph.comm.netty.NettyClient: Using Netty without authentication.
2015-09-04 17:44:12,317 ERROR org.apache.giraph.master.MasterThread: masterThread: Master algorithm failed with IllegalStateException
java.lang.IllegalStateException: connectAllAddresses: Too many failures (1000).
at org.apache.giraph.comm.netty.NettyClient.connectAllAddresses(NettyClient.java:488)
at org.apache.giraph.comm.netty.NettyMasterClient.openConnections(NettyMasterClient.java:76)
at org.apache.giraph.master.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1605)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:124)
2015-09-04 17:44:12,318 FATAL org.apache.giraph.graph.GraphTaskManager: uncaughtException: OverrideExceptionHandler on thread org.apache.giraph.master.MasterThread, msg = java.lang.IllegalStateException: connectAllAddresses: Too many failures (1000)., exiting...
java.lang.IllegalStateException: java.lang.IllegalStateException: connectAllAddresses: Too many failures (1000).
at org.apache.giraph.master.MasterThread.run(MasterThread.java:194)
Caused by: java.lang.IllegalStateException: connectAllAddresses: Too many failures (1000).
at org.apache.giraph.comm.netty.NettyClient.connectAllAddresses(NettyClient.java:488)
at org.apache.giraph.comm.netty.NettyMasterClient.openConnections(NettyMasterClient.java:76)
at org.apache.giraph.master.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1605)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:124)
Syslog logs 2:
2015-09-04 17:43:52,111 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2015-09-04 17:43:52,797 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2015-09-04 17:43:53,162 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2015-09-04 17:43:53,170 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@320ca662
2015-09-04 17:43:53,407 INFO org.apache.hadoop.mapred.MapTask: Processing split: 'org.apache.giraph.bsp.BspInputSplit, index=-1, num=-1
2015-09-04 17:43:53,544 INFO org.apache.giraph.graph.GraphTaskManager: setup: Log level remains at info
2015-09-04 17:43:53,782 INFO org.apache.giraph.graph.GraphTaskManager: setup: Starting up BspServiceWorker...
2015-09-04 17:43:53,828 INFO org.apache.giraph.bsp.BspService: BspService: Path to create to halt is /_hadoopBsp/job_201509041740_0002/_haltComputation
2015-09-04 17:43:53,829 INFO org.apache.giraph.bsp.BspService: BspService: Connecting to ZooKeeper with job job_201509041740_0002, 1 on master:2181,slave:2181
2015-09-04 17:43:53,856 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2015-09-04 17:43:53,856 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=rofti
2015-09-04 17:43:53,856 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.7.0_79
2015-09-04 17:43:53,856 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2015-09-04 17:43:53,856 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/jars/classes:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/jars:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000001_0/work:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.2.1.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.11.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.11.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-3.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar:/usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar
2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000001_0/work
2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000001_0/work/tmp
2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.19.0-26-generic
2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=hduser
2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/hduser
2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000001_0/work
2015-09-04 17:43:53,859 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=master:2181,slave:2181 sessionTimeout=60000 watcher=org.apache.giraph.worker.BspServiceWorker@5fc3d61f
2015-09-04 17:43:53,899 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server slave/192.168.0.103:2181. Will not attempt to authenticate using SASL (unknown error)
2015-09-04 17:43:53,899 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to slave/192.168.0.103:2181, initiating session
2015-09-04 17:43:53,987 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server slave/192.168.0.103:2181, sessionid = 0x24f9906beb90001, negotiated timeout = 40000
2015-09-04 17:43:53,990 INFO org.apache.giraph.bsp.BspService: process: Asynchronous connection complete.
2015-09-04 17:43:54,107 INFO org.apache.giraph.comm.netty.NettyServer: NettyServer: Using execution group with 8 threads for requestFrameDecoder.
2015-09-04 17:43:54,215 INFO org.apache.giraph.comm.netty.NettyServer: start: Started server communication server: rofti/127.0.1.1:30001 with up to 16 threads on bind attempt 0 with sendBufferSize = 32768 receiveBufferSize = 524288
2015-09-04 17:43:54,223 INFO org.apache.giraph.comm.netty.NettyClient: NettyClient: Using execution handler with 8 threads after request-encoder.
2015-09-04 17:43:54,238 INFO org.apache.giraph.graph.GraphTaskManager: setup: Registering health of this worker...
2015-09-04 17:43:54,304 INFO org.apache.giraph.bsp.BspService: getJobState: Job state already exists (/_hadoopBsp/job_201509041740_0002/_masterJobState)
2015-09-04 17:43:54,323 INFO org.apache.giraph.bsp.BspService: getApplicationAttempt: Node /_hadoopBsp/job_201509041740_0002/_applicationAttemptsDir already exists!
2015-09-04 17:43:54,339 INFO org.apache.giraph.bsp.BspService: getApplicationAttempt: Node /_hadoopBsp/job_201509041740_0002/_applicationAttemptsDir already exists!
2015-09-04 17:43:54,359 INFO org.apache.giraph.worker.BspServiceWorker: registerHealth: Created my health node for attempt=0, superstep=-1 with /_hadoopBsp/job_201509041740_0002/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir/rofti_1 and workerInfo= Worker(hostname=rofti, MRtaskID=1, port=30001)
Re: NettyClient java.net.ConnectException: Connection refused
Posted by Flavian Jacquot <fl...@gmail.com>.
Hi Marven,
did you tried to ssh from the master to the slaves?
You may need to add the slave to the "known host" file.
In order to do that you can use that cmf:
ssh -n -o StrictHostKeyChecking=no $destinationhostname hostname
Don't forget to replace $destinationhostname by the hostname of the slaves.
This script can do it for all the slaves :
http://pastebin.com/raw.php?i=Lj54BWZD
Regards,
Flavain
2015-09-04 18:53 GMT+02:00 von Domarus, Marven <
marven.von.domarus@rwth-aachen.de>:
> Hi everyone,
>
> I have already run the SimpleShortestPathsComputation on single node
> pseudo-distribution and I am currently trying to run it on a 2 node cluster
> on Ubuntu 14.04.
> But every time I got the following Error: (Complete Logs below)
>
> "ERROR org.apache.giraph.master.MasterThread: masterThread: Master
> algorithm failed with IllegalStateException
> java.lang.IllegalStateException: connectAllAddresses: Too many failures
> (1000)."
>
> Which seems to be the result of the NettyClient not being able to connect
> with the other node on port 30001 resulting in "java.net.ConnectException:
> Connection refused: rofti/192.168.0.103:30001".
> I already tried to open the port on both nodes as described here:
> http://mail-archives.apache.org/mod_mbox/giraph-user/201310.mbox/%3CCAAcgib35tQyROodNQrE3Se_39JsT5MiQVP=4v==X10jtOUfSVA@mail.gmail.com%3E
> and made sure that all host-names are written in only lowercase letters.
> But that didn't change anything. I really appreciate any help you guys can
> offer.
>
> Thanks,
> Marven
>
> Output of iptables -L :
> Chain INPUT (policy ACCEPT)
> target prot opt source destination
> ACCEPT tcp -- anywhere anywhere state NEW
> tcp spt:30001
> ACCEPT tcp -- anywhere anywhere state NEW
> tcp dpts:30000:30010
> ACCEPT tcp -- anywhere anywhere state NEW
> tcp dpt:2181
> Chain FORWARD (policy ACCEPT)
> target prot opt source destination
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>
>
> File: /etc/hosts on master(rofti-ex58-ud3r):
> 127.0.0.1 localhost
> 127.0.1.1 rofti-ex58-ud3r
> 192.168.0.101 master
> 192.168.0.103 slave rofti
>
>
> File: conf/zoo.cfg
> tickTime=2000
> dataDir=/home/hduser/mydata/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.1=192.168.0.101:2888:3888
> server.2=192.168.0.103:2888:3888
>
>
> Syslog logs 1:
> 2015-09-04 17:43:46,416 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2015-09-04 17:43:46,661 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2015-09-04 17:43:46,784 INFO org.apache.hadoop.util.ProcessTree: setsid
> exited with exit code 0
> 2015-09-04 17:43:46,787 INFO org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3c1e6a1d
> 2015-09-04 17:43:46,884 INFO org.apache.hadoop.mapred.MapTask: Processing
> split: 'org.apache.giraph.bsp.BspInputSplit, index=-1, num=-1
> 2015-09-04 17:43:46,927 INFO org.apache.giraph.graph.GraphTaskManager:
> setup: Log level remains at info
> 2015-09-04 17:43:46,967 INFO org.apache.giraph.graph.GraphTaskManager:
> setup: Starting up BspServiceMaster (master thread)...
> 2015-09-04 17:43:46,982 INFO org.apache.giraph.bsp.BspService: BspService:
> Path to create to halt is /_hadoopBsp/job_201509041740_0002/_haltComputation
> 2015-09-04 17:43:46,982 INFO org.apache.giraph.bsp.BspService: BspService:
> Connecting to ZooKeeper with job job_201509041740_0002, 0 on
> master:2181,slave:2181
> 2015-09-04 17:43:46,987 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
> 2015-09-04 17:43:46,987 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:host.name=rofti-ex58-ud3r
> 2015-09-04 17:43:46,987 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.version=1.7.0_79
> 2015-09-04 17:43:46,987 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.vendor=Oracle Corporation
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.class.path=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/jars/classes:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/jars:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000000_0/work:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.2.1.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.11.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.11.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-3.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar:/usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.library.path=/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000000_0/work
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000000_0/work/tmp
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.name=Linux
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.version=3.16.0-38-generic
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.name=hduser
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.home=/home/hduser
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.dir=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000000_0/work
> 2015-09-04 17:43:46,988 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=master:2181,slave:2181
> sessionTimeout=60000
> watcher=org.apache.giraph.master.BspServiceMaster@77084cb5
> 2015-09-04 17:43:47,003 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server slave/192.168.0.103:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-09-04 17:43:47,005 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to slave/192.168.0.103:2181, initiating session
> 2015-09-04 17:43:47,225 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server slave/192.168.0.103:2181, sessionid =
> 0x24f9906beb90000, negotiated timeout = 40000
> 2015-09-04 17:43:47,227 INFO org.apache.giraph.bsp.BspService: process:
> Asynchronous connection complete.
> 2015-09-04 17:43:47,232 INFO org.apache.giraph.graph.GraphTaskManager:
> map: No need to do anything when not a worker
> 2015-09-04 17:43:47,232 INFO org.apache.giraph.graph.GraphTaskManager:
> cleanup: Starting for MASTER_ONLY
> 2015-09-04 17:43:47,420 INFO org.apache.giraph.master.BspServiceMaster:
> becomeMaster: First child is
> '/_hadoopBsp/job_201509041740_0002/_masterElectionDir/rofti-ex58-ud3r_00000000000'
> and my bid is
> '/_hadoopBsp/job_201509041740_0002/_masterElectionDir/rofti-ex58-ud3r_00000000000'
> 2015-09-04 17:43:47,490 INFO org.apache.giraph.comm.netty.NettyServer:
> NettyServer: Using execution group with 8 threads for requestFrameDecoder.
> 2015-09-04 17:43:47,539 INFO org.apache.giraph.comm.netty.NettyServer:
> start: Started server communication server: rofti-ex58-ud3r/
> 127.0.1.1:30000 with up to 16 threads on bind attempt 0 with
> sendBufferSize = 32768 receiveBufferSize = 524288
> 2015-09-04 17:43:47,543 INFO org.apache.giraph.comm.netty.NettyClient:
> NettyClient: Using execution handler with 8 threads after request-encoder.
> 2015-09-04 17:43:47,546 INFO org.apache.giraph.master.BspServiceMaster:
> becomeMaster: I am now the master!
> 2015-09-04 17:43:47,823 INFO org.apache.giraph.bsp.BspService: process:
> applicationAttemptChanged signaled
> 2015-09-04 17:43:48,013 WARN org.apache.giraph.bsp.BspService: process:
> Unknown and unprocessed event
> (path=/_hadoopBsp/job_201509041740_0002/_applicationAttemptsDir/0/_superstepDir,
> type=NodeChildrenChanged, state=SyncConnected)
> 2015-09-04 17:43:54,587 INFO
> org.apache.giraph.io.formats.GiraphFileInputFormat: Total input paths to
> process : 1
> 2015-09-04 17:43:54,597 WARN
> org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not
> loaded
> 2015-09-04 17:43:54,598 INFO org.apache.giraph.master.BspServiceMaster:
> generateVertexInputSplits: Got 1 input splits for 1 input threads
> 2015-09-04 17:43:54,598 INFO org.apache.giraph.master.BspServiceMaster:
> createVertexInputSplits: Starting to write input split data to zookeeper
> with 1 threads
> 2015-09-04 17:43:54,668 INFO org.apache.giraph.master.BspServiceMaster:
> createVertexInputSplits: Done writing input split data to zookeeper
> 2015-09-04 17:43:54,729 INFO org.apache.giraph.comm.netty.NettyClient:
> Using Netty without authentication.
> 2015-09-04 17:43:54,741 WARN org.apache.giraph.comm.netty.NettyClient:
> connectAllAddresses: Future failed to connect with rofti/
> 192.168.0.103:30001 with 0 failures because of java.net.ConnectException:
> Connection refused: rofti/192.168.0.103:30001
> 2015-09-04 17:43:54,742 INFO org.apache.giraph.comm.netty.NettyClient:
> connectAllAddresses: Successfully added 0 connections, (0 total connected)
> 1 failed, 1 failures total.
> 2015-09-04 17:43:54,742 INFO org.apache.giraph.comm.netty.NettyClient:
> Using Netty without authentication.
> 2015-09-04 17:43:54,745 WARN org.apache.giraph.comm.netty.NettyClient:
> connectAllAddresses: Future failed to connect with rofti/
> 192.168.0.103:30001 with 1 failures because of java.net.ConnectException:
> Connection refused: rofti/192.168.0.103:30001
> [....]
> 2015-09-04 17:44:12,314 INFO org.apache.giraph.comm.netty.NettyClient:
> connectAllAddresses: Successfully added 0 connections, (0 total connected)
> 1 failed, 999 failures total.
> 2015-09-04 17:44:12,314 INFO org.apache.giraph.comm.netty.NettyClient:
> Using Netty without authentication.
> 2015-09-04 17:44:12,316 WARN org.apache.giraph.comm.netty.NettyClient:
> connectAllAddresses: Future failed to connect with rofti/
> 192.168.0.103:30001 with 999 failures because of
> java.net.ConnectException: Connection refused: rofti/192.168.0.103:30001
> 2015-09-04 17:44:12,317 INFO org.apache.giraph.comm.netty.NettyClient:
> connectAllAddresses: Successfully added 0 connections, (0 total connected)
> 1 failed, 1000 failures total.
> 2015-09-04 17:44:12,317 INFO org.apache.giraph.comm.netty.NettyClient:
> Using Netty without authentication.
> 2015-09-04 17:44:12,317 ERROR org.apache.giraph.master.MasterThread:
> masterThread: Master algorithm failed with IllegalStateException
> java.lang.IllegalStateException: connectAllAddresses: Too many failures
> (1000).
> at
> org.apache.giraph.comm.netty.NettyClient.connectAllAddresses(NettyClient.java:488)
> at
> org.apache.giraph.comm.netty.NettyMasterClient.openConnections(NettyMasterClient.java:76)
> at
> org.apache.giraph.master.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1605)
> at org.apache.giraph.master.MasterThread.run(MasterThread.java:124)
> 2015-09-04 17:44:12,318 FATAL org.apache.giraph.graph.GraphTaskManager:
> uncaughtException: OverrideExceptionHandler on thread
> org.apache.giraph.master.MasterThread, msg =
> java.lang.IllegalStateException: connectAllAddresses: Too many failures
> (1000)., exiting...
> java.lang.IllegalStateException: java.lang.IllegalStateException:
> connectAllAddresses: Too many failures (1000).
> at org.apache.giraph.master.MasterThread.run(MasterThread.java:194)
> Caused by: java.lang.IllegalStateException: connectAllAddresses: Too many
> failures (1000).
> at
> org.apache.giraph.comm.netty.NettyClient.connectAllAddresses(NettyClient.java:488)
> at
> org.apache.giraph.comm.netty.NettyMasterClient.openConnections(NettyMasterClient.java:76)
> at
> org.apache.giraph.master.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1605)
> at org.apache.giraph.master.MasterThread.run(MasterThread.java:124)
>
>
>
> Syslog logs 2:
> 2015-09-04 17:43:52,111 INFO org.apache.hadoop.util.NativeCodeLoader:
> Loaded the native-hadoop library
> 2015-09-04 17:43:52,797 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2015-09-04 17:43:53,162 INFO org.apache.hadoop.util.ProcessTree: setsid
> exited with exit code 0
> 2015-09-04 17:43:53,170 INFO org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@320ca662
> 2015-09-04 17:43:53,407 INFO org.apache.hadoop.mapred.MapTask: Processing
> split: 'org.apache.giraph.bsp.BspInputSplit, index=-1, num=-1
> 2015-09-04 17:43:53,544 INFO org.apache.giraph.graph.GraphTaskManager:
> setup: Log level remains at info
> 2015-09-04 17:43:53,782 INFO org.apache.giraph.graph.GraphTaskManager:
> setup: Starting up BspServiceWorker...
> 2015-09-04 17:43:53,828 INFO org.apache.giraph.bsp.BspService: BspService:
> Path to create to halt is /_hadoopBsp/job_201509041740_0002/_haltComputation
> 2015-09-04 17:43:53,829 INFO org.apache.giraph.bsp.BspService: BspService:
> Connecting to ZooKeeper with job job_201509041740_0002, 1 on
> master:2181,slave:2181
> 2015-09-04 17:43:53,856 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
> 2015-09-04 17:43:53,856 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:host.name=rofti
> 2015-09-04 17:43:53,856 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.version=1.7.0_79
> 2015-09-04 17:43:53,856 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.vendor=Oracle Corporation
> 2015-09-04 17:43:53,856 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
> 2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.class.path=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/jars/classes:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/jars:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000001_0/work:/usr/local/hadoop/libexec/../conf:/usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar:/usr/local/hadoop/libexec/..:/usr/local/hadoop/libexec/../hadoop-core-1.2.1.jar:/usr/local/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop/libexec/../lib/aspectjrt-1.6.11.jar:/usr/local/hadoop/libexec/../lib/aspectjtools-1.6.11.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop/libexec/../lib/commons-net-3.1.jar:/usr/local/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/usr/local/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop/libexec/../lib/slf4j-api-1.4.3.jar:/usr/local/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar:/usr/local/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar
> 2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.library.path=/usr/local/hadoop/libexec/../lib/native/Linux-amd64-64:/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000001_0/work
> 2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000001_0/work/tmp
> 2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.name=Linux
> 2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.version=3.19.0-26-generic
> 2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.name=hduser
> 2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.home=/home/hduser
> 2015-09-04 17:43:53,857 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.dir=/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201509041740_0002/attempt_201509041740_0002_m_000001_0/work
> 2015-09-04 17:43:53,859 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=master:2181,slave:2181
> sessionTimeout=60000
> watcher=org.apache.giraph.worker.BspServiceWorker@5fc3d61f
> 2015-09-04 17:43:53,899 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server slave/192.168.0.103:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-09-04 17:43:53,899 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to slave/192.168.0.103:2181, initiating session
> 2015-09-04 17:43:53,987 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server slave/192.168.0.103:2181, sessionid =
> 0x24f9906beb90001, negotiated timeout = 40000
> 2015-09-04 17:43:53,990 INFO org.apache.giraph.bsp.BspService: process:
> Asynchronous connection complete.
> 2015-09-04 17:43:54,107 INFO org.apache.giraph.comm.netty.NettyServer:
> NettyServer: Using execution group with 8 threads for requestFrameDecoder.
> 2015-09-04 17:43:54,215 INFO org.apache.giraph.comm.netty.NettyServer:
> start: Started server communication server: rofti/127.0.1.1:30001 with up
> to 16 threads on bind attempt 0 with sendBufferSize = 32768
> receiveBufferSize = 524288
> 2015-09-04 17:43:54,223 INFO org.apache.giraph.comm.netty.NettyClient:
> NettyClient: Using execution handler with 8 threads after request-encoder.
> 2015-09-04 17:43:54,238 INFO org.apache.giraph.graph.GraphTaskManager:
> setup: Registering health of this worker...
> 2015-09-04 17:43:54,304 INFO org.apache.giraph.bsp.BspService:
> getJobState: Job state already exists
> (/_hadoopBsp/job_201509041740_0002/_masterJobState)
> 2015-09-04 17:43:54,323 INFO org.apache.giraph.bsp.BspService:
> getApplicationAttempt: Node
> /_hadoopBsp/job_201509041740_0002/_applicationAttemptsDir already exists!
> 2015-09-04 17:43:54,339 INFO org.apache.giraph.bsp.BspService:
> getApplicationAttempt: Node
> /_hadoopBsp/job_201509041740_0002/_applicationAttemptsDir already exists!
> 2015-09-04 17:43:54,359 INFO org.apache.giraph.worker.BspServiceWorker:
> registerHealth: Created my health node for attempt=0, superstep=-1 with
> /_hadoopBsp/job_201509041740_0002/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir/rofti_1
> and workerInfo= Worker(hostname=rofti, MRtaskID=1, port=30001)
>