You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hama.apache.org by Francisco Sanches <sa...@gmail.com> on 2012/12/26 14:00:42 UTC

What is the best configuration for Cluster hama distributed mode

First I would like to know if anyone has some documentation a bit more
comprehensive cluster configuration hama?

I would also like some information about the cluster configuration HAMA as:

1) I have a cluster with 12 computers in HDFS which the optimal
configuration of replication? configured to create 3 replicas of files,
this is the best?

2) In my hama-site.xml for the best cluster configuration parameter
hama.zookeeper.quorum? 1 node 2 nodes, 3 nodes.

3) When I process my graph with just over 65 000 vertices got the following
error:
attempt_201212260904_0005_000031_0: Exception in thread "pool-2-thread-1"
java.lang.OutOfMemoryError: GC overhead limit exceeded
attempt_201212260904_0005_000031_0: Exception in thread "Thread-1"
java.lang.OutOfMemoryError: GC overhead limit exceeded

Is there any parameter I change more increase the memory limit? Or my
cluster will not be able to process this amount of information? With
smaller graphs it works correctly. I'm working with the all-pairs problem.

-- 
Francisco Sanches

Re: What is the best configuration for Cluster hama distributed mode

Posted by Suraj Menon <su...@apache.org>.

Hi Francis,

First I would like to know if anyone has some documentation a bit more
> comprehensive cluster configuration hama?
>

On AWS EC2, you can use Apache Whirr to configure the cluster. Edward may
share his procedures on maintaining his Hama cluster on Oracle BDA.


> I would also like some information about the cluster configuration HAMA as:
>
> 1) I have a cluster with 12 computers in HDFS which the optimal
> configuration of replication? configured to create 3 replicas of files,
> this is the best?
>
>
This depends on your availability requirements and the capacity of your
cluster. 3 would be good if you cannot tolerate data-loss. You would have
to work this out depending on the size of data and the capacity of your
cluster.


> 2) In my hama-site.xml for the best cluster configuration parameter
> hama.zookeeper.quorum? 1 node 2 nodes, 3 nodes.
>
> Once again this depends on your availability requirements and the usage of
cluster.


> 3) When I process my graph with just over 65 000 vertices got the following
> error:
> attempt_201212260904_0005_000031_0: Exception in thread "pool-2-thread-1"
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> attempt_201212260904_0005_000031_0: Exception in thread "Thread-1"
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Is there any parameter I change more increase the memory limit? Or my
> cluster will not be able to process this amount of information? With
> smaller graphs it works correctly. I'm working with the all-pairs problem.
>

As reported recently by other users, Hama is facing scalability issues. I
am trying to close - https://issues.apache.org/jira/browse/HAMA-559 and
some other message object lifecycle issues.(Today we create a new Writable
object for every message read and received.) Also , we keep all the
vertices in the memory.

However, you can change your JVM arguments. Please look at what you can do
with the configuration parameter - bsp.child.java.opts. The default value
could be found in hama-default.xml.

Regards,
Suraj

Re: What is the best configuration for Cluster hama distributed mode

Posted by "Edward J. Yoon" <ed...@apache.org>.

Francisco, there's a problem that requires huge memory and the number
of BSP tasks can't be increased manually in Hama 0.6.0 release.

We will release 0.6.1 soon. Please wait or use the TRUNK version.

On Fri, Dec 28, 2012 at 10:11 PM, Francisco Sanches
<sa...@gmail.com> wrote:
> Thanks for the reply Edward J. Yoon.
>
> I made the changes in the configuration suggested that Edward Yoon
> continues with more problems. What could observe is that for smaller
> graphs, with 2000 vertices My application works perfectly. But when I put
> larger graphs got the following error:
>
> hduser@mestre:/usr/local/hama$ bin/hama jar aplicmenosmsg.jar 8719
> /hdfs/disco/grafonewc.txt /hdfs/disco/grafoout 36
> 12/12/28 10:00:54 INFO bsp.FileInputFormat: Total input paths to process : 1
> 12/12/28 10:00:54 INFO bsp.FileInputFormat: Total # of splits: 36
> 12/12/28 10:00:57 INFO bsp.BSPJobClient: Running job: job_201212280959_0001
> 12/12/28 10:01:00 INFO bsp.BSPJobClient: Current supersteps number: 0
> 12/12/28 10:01:30 INFO bsp.BSPJobClient: Current supersteps number: 2
> 12/12/28 10:01:33 INFO bsp.BSPJobClient: Current supersteps number: 4
> 12/12/28 10:01:39 INFO bsp.BSPJobClient: Current supersteps number: 5
> 12/12/28 10:02:03 INFO bsp.BSPJobClient: Current supersteps number: 6
> 12/12/28 10:02:09 INFO bsp.BSPJobClient: Current supersteps number: 0
> attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO
> zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969,
> built on 02/23/2011 22:27 GMT
> attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO
> zookeeper.ZooKeeper: Client environment:host.name=node01
> attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO
> zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24
> attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO
> zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
> attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO
> zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-6-openjdk-i386/jre
> .
> .
> .
> attempt_201212280959_0001_000027_0: 12/12/28 10:01:50 INFO
> ipc.NettyTransceiver: [id: 0x0129c445, /10.16.1.201:46065 => node08/
> 10.16.1.208:61001] INTEREST_CHANGED
> 12/12/28 10:02:12 INFO bsp.BSPJobClient: Job failed.
>
>
> Has anyone had this problem? Got Resolve? If I increase the number of
> cluster nodes is possible to solve this problem? Is this is a limitation of
> the Graph API Hama? or my application?
>
>
> 2012/12/27 Edward J. Yoon <ed...@apache.org>
>
>> You should increase max heap size of child JVMs, instead of HAMA_HEAPSIZE.
>>
>> Add below property to hama-site.xml.
>>
>>   <property>
>>     <name>bsp.child.java.opts</name>
>>     <value>-Xmx2048m</value>
>>     <description>Java opts for the groom server child processes.
>>     The following symbol, if present, will be interpolated: @taskid@
>> is replaced
>>     by current TaskID. Any other occurrences of '@' will go unchanged.
>>     For example, to enable verbose gc logging to a file named for the
>> taskid in
>>     /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
>>           -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
>>     The configuration variable bsp.child.ulimit can be used to control the
>>     maximum virtual memory of the child processes.
>>     </description>
>>   </property>
>>
>> On Thu, Dec 27, 2012 at 5:11 AM, Francisco Sanches
>> <sa...@gmail.com> wrote:
>> > Thanks for the replies:
>> >
>> > Excuse me by the size of the log. But after I changed the parameter of
>> > hama-env.sh to "export HAMA_HEAPSIZE = 2000" My program stopped giving
>> the
>> > memory error and displays the error below. Remembering that I am running
>> > for a graph with just over 65 000 vertices. What is it like to understand
>> > is that if I increase the number of nodes in the cluster can run my
>> > application with this graph of 65 000 vertices?
>> >
>> > LOG:
>> >
>> > hduser@mestre:/usr/local/hama$ bin/hama jar aplicpronto.jar 65537
>> > /hdfs/disco/grafogrande /hdfs/disco/grafoout 36
>> > 12/12/26 17:00:19 INFO bsp.FileInputFormat: Total input paths to process
>> : 1
>> > 12/12/26 17:00:19 INFO bsp.FileInputFormat: Total # of splits: 36
>> > 12/12/26 17:00:23 INFO bsp.BSPJobClient: Running job:
>> job_201212261658_0001
>> > 12/12/26 17:00:26 INFO bsp.BSPJobClient: Current supersteps number: 0
>> > 12/12/26 17:00:56 INFO bsp.BSPJobClient: Current supersteps number: 1
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969,
>> > built on 02/23/2011 22:27 GMT
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:host.name=mestre
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client
>> > environment:java.home=/usr/lib/jvm/java-6-openjdk-i386/jre
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client
>> >
>> environment:java.class.path=/usr/local/hama/bin/../conf:/usr/lib/jvm/java-6-openjdk-i386/lib/tools.jar:/usr/local/hama/bin/../hama-core-0.5.0.jar:/usr/local/hama/bin/../hama-examples-0.5.0.jar:/usr/local/hama/bin/../hama-graph-0.5.0.jar:/usr/local/hama/bin/../lib/ant-1.7.1.jar:/usr/local/hama/bin/../lib/ant-launcher-1.7.1.jar:/usr/local/hama/bin/../lib/avro-1.6.0.jar:/usr/local/hama/bin/../lib/avro-ipc-1.6.0.jar:/usr/local/hama/bin/../lib/commons-cli-1.2.jar:/usr/local/hama/bin/../lib/commons-configuration-1.7.jar:/usr/local/hama/bin/../lib/commons-lang-2.6.jar:/usr/local/hama/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hama/bin/../lib/commons-math3-3.0.jar:/usr/local/hama/bin/../lib/guava-10.0.1.jar:/usr/local/hama/bin/../lib/hadoop-core-1.0.0.jar:/usr/local/hama/bin/../lib/hadoop-test-1.0.0.jar:/usr/local/hama/bin/../lib/jackson-core-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jackson-mapper-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jetty-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-annotations-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-util-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/junit-4.8.1.jar:/usr/local/hama/bin/../lib/log4j-1.2.16.jar:/usr/local/hama/bin/../lib/netty-3.2.6.Final.jar:/usr/local/hama/bin/../lib/servlet-api-6.0.32.jar:/usr/local/hama/bin/../lib/slf4j-api-1.5.8.jar:/usr/local/hama/bin/../lib/slf4j-log4j12-1.5.8.jar:/usr/local/hama/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hama/bin/../lib/zookeeper-3.3.3.jar::/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/classes:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client
>> >
>> environment:java.library.path=/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386:/usr/lib/jvm/java-6-openjdk-i386/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/i386-linux-gnu/jni:/lib/i386-linux-gnu:/usr/lib/i386-linux-gnu:/usr/lib/jni:/lib:/usr/lib
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:os.name=Linux
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:os.arch=i386
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:os.version=3.2.0-32-generic-pae
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:user.name=hduser
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client environment:user.home=/home/hduser
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Client
>> >
>> environment:user.dir=/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ZooKeeper: Initiating client connection,
>> > connectString=node02:21810,node01:21810,mestre:21810
>> sessionTimeout=1200000
>> > watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImpl@1deeb40
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ClientCnxn: Opening socket connection to server mestre/
>> > 10.16.1.200:21810
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > zookeeper.ClientCnxn: Socket connection established to mestre/
>> > 10.16.1.200:21810, initiating session
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
>> > sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At mestre/
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:32 INFO
>> > zookeeper.ClientCnxn: Session establishment complete on server mestre/
>> > 10.16.1.200:21810, sessionid = 0x3bd8cbb48a0006, negotiated timeout =
>> > 1200000
>> > attempt_201212261658_0001_000000_0: SLF4J: Class path contains multiple
>> > SLF4J bindings.
>> > attempt_201212261658_0001_000000_0: SLF4J: Found binding in
>> >
>> [jar:file:/usr/local/hama/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > attempt_201212261658_0001_000000_0: SLF4J: Found binding in
>> >
>> [file:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/org/slf4j/impl/StaticLoggerBinder.class]
>> > attempt_201212261658_0001_000000_0: SLF4J: See
>> > http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x002f8116] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/
>> > 10.16.1.203:61003] BOUND: /10.16.1.200:47685
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/
>> > 10.16.1.203:61003] CONNECTED: node03/10.16.1.203:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00d8ca48] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/
>> > 10.16.1.205:61001] BOUND: /10.16.1.200:52329
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/
>> > 10.16.1.205:61001] CONNECTED: node05/10.16.1.205:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x018adae2] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/
>> > 10.16.1.204:61002] BOUND: /10.16.1.200:59179
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/
>> > 10.16.1.204:61002] CONNECTED: node04/10.16.1.204:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x003a835d] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/
>> > 10.16.1.206:61001] BOUND: /10.16.1.200:45011
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/
>> > 10.16.1.206:61001] CONNECTED: node06/10.16.1.206:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00061cd2] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/
>> > 10.16.1.204:61003] BOUND: /10.16.1.200:54578
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/
>> > 10.16.1.204:61003] CONNECTED: node04/10.16.1.204:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00fb6c5f] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/
>> > 10.16.1.205:61002] BOUND: /10.16.1.200:52964
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/
>> > 10.16.1.205:61002] CONNECTED: node05/10.16.1.205:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00cb42cf] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/
>> > 10.16.1.203:61001] BOUND: /10.16.1.200:36765
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/
>> > 10.16.1.203:61001] CONNECTED: node03/10.16.1.203:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x018bbf55] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/
>> > 10.16.1.201:61003] BOUND: /10.16.1.200:45734
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/
>> > 10.16.1.201:61003] CONNECTED: node01/10.16.1.201:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00ed5d9d] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/
>> > 10.16.1.202:61002] BOUND: /10.16.1.200:60724
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/
>> > 10.16.1.202:61002] CONNECTED: node02/10.16.1.202:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x014b84c7] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/
>> > 10.16.1.203:61002] BOUND: /10.16.1.200:45839
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/
>> > 10.16.1.203:61002] CONNECTED: node03/10.16.1.203:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x0160c4b0] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/
>> > 10.16.1.202:61003] BOUND: /10.16.1.200:44757
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/
>> > 10.16.1.202:61003] CONNECTED: node02/10.16.1.202:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x0029ea31] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/
>> > 10.16.1.204:61001] BOUND: /10.16.1.200:40846
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/
>> > 10.16.1.204:61001] CONNECTED: node04/10.16.1.204:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.210:36396
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.200:41941
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.206:44493
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.202:48243
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.200:41945
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00a86d12] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/
>> > 10.16.1.201:61001] BOUND: /10.16.1.200:35763
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/
>> > 10.16.1.201:61001] CONNECTED: node01/10.16.1.201:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x008932e8] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.200:41949
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/
>> > 10.16.1.200:61002] BOUND: /10.16.1.200:41949
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/
>> > 10.16.1.200:61002] CONNECTED: mestre/10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00f2164f] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.209:43193
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/
>> > 10.16.1.201:61002] BOUND: /10.16.1.200:44908
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> > ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/
>> > 10.16.1.201:61002] CONNECTED: node01/10.16.1.201:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.207:57476
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.207:57478
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.210:36402
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.201:37455
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.207:57480
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.205:56188
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.206:44506
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.205:56189
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.202:48251
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.205:56191
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.208:50918
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.208:50917
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.209:43200
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
>> ipc.NettyServer:
>> > [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.209:43201
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.208:50924
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.211:44099
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.206:44511
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.211:44100
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.211:44101
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x019c4091] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.201:37460
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/
>> > 10.16.1.202:61001] BOUND: /10.16.1.200:54333
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/
>> > 10.16.1.202:61001] CONNECTED: node02/10.16.1.202:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.203:46568
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.203:46569
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01e80761] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/
>> > 10.16.1.200:61003] BOUND: /10.16.1.200:56327
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/
>> > 10.16.1.200:61003] CONNECTED: mestre/10.16.1.200:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01dbb27d] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.203:46573
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/
>> > 10.16.1.200:61001] BOUND: /10.16.1.200:51740
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/
>> > 10.16.1.200:61001] CONNECTED: mestre/10.16.1.200:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.204:35237
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.204:35239
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01971eb3] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.202:48260
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/
>> > 10.16.1.211:61003] BOUND: /10.16.1.200:57399
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/
>> > 10.16.1.211:61003] CONNECTED: node11/10.16.1.211:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.201:37463
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x005b78cf] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/
>> > 10.16.1.211:61001] BOUND: /10.16.1.200:41092
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/
>> > 10.16.1.211:61001] CONNECTED: node11/10.16.1.211:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] BOUND: /
>> > 10.16.1.200:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> ipc.NettyServer:
>> > [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] CONNECTED: /
>> > 10.16.1.204:35245
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01f51e5c] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/
>> > 10.16.1.210:61002] BOUND: /10.16.1.200:48018
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/
>> > 10.16.1.210:61002] CONNECTED: node10/10.16.1.210:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00d36ff3] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/
>> > 10.16.1.209:61003] BOUND: /10.16.1.200:34916
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/
>> > 10.16.1.209:61003] CONNECTED: node09/10.16.1.209:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x018e899c] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/
>> > 10.16.1.211:61002] BOUND: /10.16.1.200:58261
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/
>> > 10.16.1.211:61002] CONNECTED: node11/10.16.1.211:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x014bc4e6] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/
>> > 10.16.1.210:61003] BOUND: /10.16.1.200:39073
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/
>> > 10.16.1.210:61003] CONNECTED: node10/10.16.1.210:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01be1041] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/
>> > 10.16.1.208:61002] BOUND: /10.16.1.200:39758
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/
>> > 10.16.1.208:61002] CONNECTED: node08/10.16.1.208:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01e090ee] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/
>> > 10.16.1.209:61001] BOUND: /10.16.1.200:56029
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/
>> > 10.16.1.209:61001] CONNECTED: node09/10.16.1.209:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x0187f9f1] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/
>> > 10.16.1.207:61003] BOUND: /10.16.1.200:35151
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/
>> > 10.16.1.207:61003] CONNECTED: node07/10.16.1.207:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00c3358b] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/
>> > 10.16.1.208:61003] BOUND: /10.16.1.200:35301
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/
>> > 10.16.1.208:61003] CONNECTED: node08/10.16.1.208:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01e82b72] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/
>> > 10.16.1.210:61001] BOUND: /10.16.1.200:34819
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/
>> > 10.16.1.210:61001] CONNECTED: node10/10.16.1.210:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00d0deb5] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/
>> > 10.16.1.209:61002] BOUND: /10.16.1.200:34684
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/
>> > 10.16.1.209:61002] CONNECTED: node09/10.16.1.209:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01505258] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/
>> > 10.16.1.206:61002] BOUND: /10.16.1.200:35962
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/
>> > 10.16.1.206:61002] CONNECTED: node06/10.16.1.206:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01d8417c] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/
>> > 10.16.1.207:61001] BOUND: /10.16.1.200:41009
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/
>> > 10.16.1.207:61001] CONNECTED: node07/10.16.1.207:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x017c398e] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/
>> > 10.16.1.205:61003] BOUND: /10.16.1.200:53654
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/
>> > 10.16.1.205:61003] CONNECTED: node05/10.16.1.205:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00e13e7b] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/
>> > 10.16.1.208:61001] BOUND: /10.16.1.200:55138
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/
>> > 10.16.1.208:61001] CONNECTED: node08/10.16.1.208:61001
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01503458] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/
>> > 10.16.1.206:61003] BOUND: /10.16.1.200:40540
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/
>> > 10.16.1.206:61003] CONNECTED: node06/10.16.1.206:61003
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01b963c4] OPEN
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/
>> > 10.16.1.207:61002] BOUND: /10.16.1.200:42816
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
>> > ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/
>> > 10.16.1.207:61002] CONNECTED: node07/10.16.1.207:61002
>> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:53 INFO
>> > graph.GraphJobRunner: Loading finished at 1 steps.
>> > 12/12/26 17:03:17 INFO bsp.BSPJobClient: Job failed.
>> >
>> >
>> >
>> > 2012/12/26 Edward J. Yoon <ed...@apache.org>
>> >
>> >> BTW, This log looks like you ran out of memory while job was running
>> >> as a local mode.
>> >>
>> >> You can check the cluster status via web UI.
>> >>
>> >> 1. http://wiki.apache.org/hama/GettingStarted#Hama_Web_Interfaces
>> >>
>> >> On Wed, Dec 26, 2012 at 10:00 PM, Francisco Sanches
>> >> <sa...@gmail.com> wrote:
>> >> > attempt_201212260904_0005_000031_0: Exception in thread "Thread-1"
>> >> > java.lang.OutOfMemoryError: GC overhead limit exceeded
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> @eddieyoon
>> >>
>> >
>> >
>> >
>> > --
>> > Francisco Sanches
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Francisco Sanches



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: What is the best configuration for Cluster hama distributed mode

Posted by Francisco Sanches <sa...@gmail.com>.

Thanks for the reply Edward J. Yoon.

I made the changes in the configuration suggested that Edward Yoon
continues with more problems. What could observe is that for smaller
graphs, with 2000 vertices My application works perfectly. But when I put
larger graphs got the following error:

hduser@mestre:/usr/local/hama$ bin/hama jar aplicmenosmsg.jar 8719
/hdfs/disco/grafonewc.txt /hdfs/disco/grafoout 36
12/12/28 10:00:54 INFO bsp.FileInputFormat: Total input paths to process : 1
12/12/28 10:00:54 INFO bsp.FileInputFormat: Total # of splits: 36
12/12/28 10:00:57 INFO bsp.BSPJobClient: Running job: job_201212280959_0001
12/12/28 10:01:00 INFO bsp.BSPJobClient: Current supersteps number: 0
12/12/28 10:01:30 INFO bsp.BSPJobClient: Current supersteps number: 2
12/12/28 10:01:33 INFO bsp.BSPJobClient: Current supersteps number: 4
12/12/28 10:01:39 INFO bsp.BSPJobClient: Current supersteps number: 5
12/12/28 10:02:03 INFO bsp.BSPJobClient: Current supersteps number: 6
12/12/28 10:02:09 INFO bsp.BSPJobClient: Current supersteps number: 0
attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO
zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969,
built on 02/23/2011 22:27 GMT
attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO
zookeeper.ZooKeeper: Client environment:host.name=node01
attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO
zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24
attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO
zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO
zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-6-openjdk-i386/jre
.
.
.
attempt_201212280959_0001_000027_0: 12/12/28 10:01:50 INFO
ipc.NettyTransceiver: [id: 0x0129c445, /10.16.1.201:46065 => node08/
10.16.1.208:61001] INTEREST_CHANGED
12/12/28 10:02:12 INFO bsp.BSPJobClient: Job failed.


Has anyone had this problem? Got Resolve? If I increase the number of
cluster nodes is possible to solve this problem? Is this is a limitation of
the Graph API Hama? or my application?


2012/12/27 Edward J. Yoon <ed...@apache.org>

> You should increase max heap size of child JVMs, instead of HAMA_HEAPSIZE.
>
> Add below property to hama-site.xml.
>
>   <property>
>     <name>bsp.child.java.opts</name>
>     <value>-Xmx2048m</value>
>     <description>Java opts for the groom server child processes.
>     The following symbol, if present, will be interpolated: @taskid@
> is replaced
>     by current TaskID. Any other occurrences of '@' will go unchanged.
>     For example, to enable verbose gc logging to a file named for the
> taskid in
>     /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
>           -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
>     The configuration variable bsp.child.ulimit can be used to control the
>     maximum virtual memory of the child processes.
>     </description>
>   </property>
>
> On Thu, Dec 27, 2012 at 5:11 AM, Francisco Sanches
> <sa...@gmail.com> wrote:
> > Thanks for the replies:
> >
> > Excuse me by the size of the log. But after I changed the parameter of
> > hama-env.sh to "export HAMA_HEAPSIZE = 2000" My program stopped giving
> the
> > memory error and displays the error below. Remembering that I am running
> > for a graph with just over 65 000 vertices. What is it like to understand
> > is that if I increase the number of nodes in the cluster can run my
> > application with this graph of 65 000 vertices?
> >
> > LOG:
> >
> > hduser@mestre:/usr/local/hama$ bin/hama jar aplicpronto.jar 65537
> > /hdfs/disco/grafogrande /hdfs/disco/grafoout 36
> > 12/12/26 17:00:19 INFO bsp.FileInputFormat: Total input paths to process
> : 1
> > 12/12/26 17:00:19 INFO bsp.FileInputFormat: Total # of splits: 36
> > 12/12/26 17:00:23 INFO bsp.BSPJobClient: Running job:
> job_201212261658_0001
> > 12/12/26 17:00:26 INFO bsp.BSPJobClient: Current supersteps number: 0
> > 12/12/26 17:00:56 INFO bsp.BSPJobClient: Current supersteps number: 1
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969,
> > built on 02/23/2011 22:27 GMT
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:host.name=mestre
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client
> > environment:java.home=/usr/lib/jvm/java-6-openjdk-i386/jre
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client
> >
> environment:java.class.path=/usr/local/hama/bin/../conf:/usr/lib/jvm/java-6-openjdk-i386/lib/tools.jar:/usr/local/hama/bin/../hama-core-0.5.0.jar:/usr/local/hama/bin/../hama-examples-0.5.0.jar:/usr/local/hama/bin/../hama-graph-0.5.0.jar:/usr/local/hama/bin/../lib/ant-1.7.1.jar:/usr/local/hama/bin/../lib/ant-launcher-1.7.1.jar:/usr/local/hama/bin/../lib/avro-1.6.0.jar:/usr/local/hama/bin/../lib/avro-ipc-1.6.0.jar:/usr/local/hama/bin/../lib/commons-cli-1.2.jar:/usr/local/hama/bin/../lib/commons-configuration-1.7.jar:/usr/local/hama/bin/../lib/commons-lang-2.6.jar:/usr/local/hama/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hama/bin/../lib/commons-math3-3.0.jar:/usr/local/hama/bin/../lib/guava-10.0.1.jar:/usr/local/hama/bin/../lib/hadoop-core-1.0.0.jar:/usr/local/hama/bin/../lib/hadoop-test-1.0.0.jar:/usr/local/hama/bin/../lib/jackson-core-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jackson-mapper-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jetty-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-annotations-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-util-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/junit-4.8.1.jar:/usr/local/hama/bin/../lib/log4j-1.2.16.jar:/usr/local/hama/bin/../lib/netty-3.2.6.Final.jar:/usr/local/hama/bin/../lib/servlet-api-6.0.32.jar:/usr/local/hama/bin/../lib/slf4j-api-1.5.8.jar:/usr/local/hama/bin/../lib/slf4j-log4j12-1.5.8.jar:/usr/local/hama/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hama/bin/../lib/zookeeper-3.3.3.jar::/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/classes:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client
> >
> environment:java.library.path=/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386:/usr/lib/jvm/java-6-openjdk-i386/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/i386-linux-gnu/jni:/lib/i386-linux-gnu:/usr/lib/i386-linux-gnu:/usr/lib/jni:/lib:/usr/lib
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:os.name=Linux
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:os.arch=i386
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:os.version=3.2.0-32-generic-pae
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:user.name=hduser
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client environment:user.home=/home/hduser
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Client
> >
> environment:user.dir=/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ZooKeeper: Initiating client connection,
> > connectString=node02:21810,node01:21810,mestre:21810
> sessionTimeout=1200000
> > watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImpl@1deeb40
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ClientCnxn: Opening socket connection to server mestre/
> > 10.16.1.200:21810
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > zookeeper.ClientCnxn: Socket connection established to mestre/
> > 10.16.1.200:21810, initiating session
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> > sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At mestre/
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:32 INFO
> > zookeeper.ClientCnxn: Session establishment complete on server mestre/
> > 10.16.1.200:21810, sessionid = 0x3bd8cbb48a0006, negotiated timeout =
> > 1200000
> > attempt_201212261658_0001_000000_0: SLF4J: Class path contains multiple
> > SLF4J bindings.
> > attempt_201212261658_0001_000000_0: SLF4J: Found binding in
> >
> [jar:file:/usr/local/hama/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > attempt_201212261658_0001_000000_0: SLF4J: Found binding in
> >
> [file:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/org/slf4j/impl/StaticLoggerBinder.class]
> > attempt_201212261658_0001_000000_0: SLF4J: See
> > http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x002f8116] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/
> > 10.16.1.203:61003] BOUND: /10.16.1.200:47685
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/
> > 10.16.1.203:61003] CONNECTED: node03/10.16.1.203:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00d8ca48] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/
> > 10.16.1.205:61001] BOUND: /10.16.1.200:52329
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/
> > 10.16.1.205:61001] CONNECTED: node05/10.16.1.205:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x018adae2] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/
> > 10.16.1.204:61002] BOUND: /10.16.1.200:59179
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/
> > 10.16.1.204:61002] CONNECTED: node04/10.16.1.204:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x003a835d] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/
> > 10.16.1.206:61001] BOUND: /10.16.1.200:45011
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/
> > 10.16.1.206:61001] CONNECTED: node06/10.16.1.206:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00061cd2] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/
> > 10.16.1.204:61003] BOUND: /10.16.1.200:54578
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/
> > 10.16.1.204:61003] CONNECTED: node04/10.16.1.204:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00fb6c5f] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/
> > 10.16.1.205:61002] BOUND: /10.16.1.200:52964
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/
> > 10.16.1.205:61002] CONNECTED: node05/10.16.1.205:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00cb42cf] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/
> > 10.16.1.203:61001] BOUND: /10.16.1.200:36765
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/
> > 10.16.1.203:61001] CONNECTED: node03/10.16.1.203:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x018bbf55] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/
> > 10.16.1.201:61003] BOUND: /10.16.1.200:45734
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/
> > 10.16.1.201:61003] CONNECTED: node01/10.16.1.201:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00ed5d9d] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/
> > 10.16.1.202:61002] BOUND: /10.16.1.200:60724
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/
> > 10.16.1.202:61002] CONNECTED: node02/10.16.1.202:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x014b84c7] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/
> > 10.16.1.203:61002] BOUND: /10.16.1.200:45839
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/
> > 10.16.1.203:61002] CONNECTED: node03/10.16.1.203:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x0160c4b0] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/
> > 10.16.1.202:61003] BOUND: /10.16.1.200:44757
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/
> > 10.16.1.202:61003] CONNECTED: node02/10.16.1.202:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x0029ea31] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/
> > 10.16.1.204:61001] BOUND: /10.16.1.200:40846
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/
> > 10.16.1.204:61001] CONNECTED: node04/10.16.1.204:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.210:36396
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.200:41941
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.206:44493
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.202:48243
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.200:41945
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00a86d12] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/
> > 10.16.1.201:61001] BOUND: /10.16.1.200:35763
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/
> > 10.16.1.201:61001] CONNECTED: node01/10.16.1.201:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x008932e8] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.200:41949
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/
> > 10.16.1.200:61002] BOUND: /10.16.1.200:41949
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/
> > 10.16.1.200:61002] CONNECTED: mestre/10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00f2164f] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.209:43193
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/
> > 10.16.1.201:61002] BOUND: /10.16.1.200:44908
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> > ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/
> > 10.16.1.201:61002] CONNECTED: node01/10.16.1.201:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.207:57476
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.207:57478
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.210:36402
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.201:37455
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.207:57480
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.205:56188
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.206:44506
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.205:56189
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.202:48251
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.205:56191
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.208:50918
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.208:50917
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.209:43200
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyServer:
> > [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.209:43201
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.208:50924
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.211:44099
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.206:44511
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.211:44100
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.211:44101
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x019c4091] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.201:37460
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/
> > 10.16.1.202:61001] BOUND: /10.16.1.200:54333
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/
> > 10.16.1.202:61001] CONNECTED: node02/10.16.1.202:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.203:46568
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.203:46569
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01e80761] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/
> > 10.16.1.200:61003] BOUND: /10.16.1.200:56327
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/
> > 10.16.1.200:61003] CONNECTED: mestre/10.16.1.200:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01dbb27d] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.203:46573
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/
> > 10.16.1.200:61001] BOUND: /10.16.1.200:51740
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/
> > 10.16.1.200:61001] CONNECTED: mestre/10.16.1.200:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.204:35237
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.204:35239
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01971eb3] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.202:48260
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/
> > 10.16.1.211:61003] BOUND: /10.16.1.200:57399
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/
> > 10.16.1.211:61003] CONNECTED: node11/10.16.1.211:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.201:37463
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x005b78cf] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/
> > 10.16.1.211:61001] BOUND: /10.16.1.200:41092
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/
> > 10.16.1.211:61001] CONNECTED: node11/10.16.1.211:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] BOUND: /
> > 10.16.1.200:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyServer:
> > [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] CONNECTED: /
> > 10.16.1.204:35245
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01f51e5c] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/
> > 10.16.1.210:61002] BOUND: /10.16.1.200:48018
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/
> > 10.16.1.210:61002] CONNECTED: node10/10.16.1.210:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00d36ff3] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/
> > 10.16.1.209:61003] BOUND: /10.16.1.200:34916
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/
> > 10.16.1.209:61003] CONNECTED: node09/10.16.1.209:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x018e899c] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/
> > 10.16.1.211:61002] BOUND: /10.16.1.200:58261
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/
> > 10.16.1.211:61002] CONNECTED: node11/10.16.1.211:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x014bc4e6] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/
> > 10.16.1.210:61003] BOUND: /10.16.1.200:39073
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/
> > 10.16.1.210:61003] CONNECTED: node10/10.16.1.210:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01be1041] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/
> > 10.16.1.208:61002] BOUND: /10.16.1.200:39758
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/
> > 10.16.1.208:61002] CONNECTED: node08/10.16.1.208:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01e090ee] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/
> > 10.16.1.209:61001] BOUND: /10.16.1.200:56029
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/
> > 10.16.1.209:61001] CONNECTED: node09/10.16.1.209:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x0187f9f1] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/
> > 10.16.1.207:61003] BOUND: /10.16.1.200:35151
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/
> > 10.16.1.207:61003] CONNECTED: node07/10.16.1.207:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00c3358b] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/
> > 10.16.1.208:61003] BOUND: /10.16.1.200:35301
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/
> > 10.16.1.208:61003] CONNECTED: node08/10.16.1.208:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01e82b72] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/
> > 10.16.1.210:61001] BOUND: /10.16.1.200:34819
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/
> > 10.16.1.210:61001] CONNECTED: node10/10.16.1.210:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00d0deb5] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/
> > 10.16.1.209:61002] BOUND: /10.16.1.200:34684
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/
> > 10.16.1.209:61002] CONNECTED: node09/10.16.1.209:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01505258] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/
> > 10.16.1.206:61002] BOUND: /10.16.1.200:35962
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/
> > 10.16.1.206:61002] CONNECTED: node06/10.16.1.206:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01d8417c] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/
> > 10.16.1.207:61001] BOUND: /10.16.1.200:41009
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/
> > 10.16.1.207:61001] CONNECTED: node07/10.16.1.207:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x017c398e] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/
> > 10.16.1.205:61003] BOUND: /10.16.1.200:53654
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/
> > 10.16.1.205:61003] CONNECTED: node05/10.16.1.205:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00e13e7b] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/
> > 10.16.1.208:61001] BOUND: /10.16.1.200:55138
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/
> > 10.16.1.208:61001] CONNECTED: node08/10.16.1.208:61001
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01503458] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/
> > 10.16.1.206:61003] BOUND: /10.16.1.200:40540
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/
> > 10.16.1.206:61003] CONNECTED: node06/10.16.1.206:61003
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01b963c4] OPEN
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/
> > 10.16.1.207:61002] BOUND: /10.16.1.200:42816
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> > ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/
> > 10.16.1.207:61002] CONNECTED: node07/10.16.1.207:61002
> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:53 INFO
> > graph.GraphJobRunner: Loading finished at 1 steps.
> > 12/12/26 17:03:17 INFO bsp.BSPJobClient: Job failed.
> >
> >
> >
> > 2012/12/26 Edward J. Yoon <ed...@apache.org>
> >
> >> BTW, This log looks like you ran out of memory while job was running
> >> as a local mode.
> >>
> >> You can check the cluster status via web UI.
> >>
> >> 1. http://wiki.apache.org/hama/GettingStarted#Hama_Web_Interfaces
> >>
> >> On Wed, Dec 26, 2012 at 10:00 PM, Francisco Sanches
> >> <sa...@gmail.com> wrote:
> >> > attempt_201212260904_0005_000031_0: Exception in thread "Thread-1"
> >> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
> >
> >
> >
> > --
> > Francisco Sanches
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>



-- 
Francisco Sanches

Re: What is the best configuration for Cluster hama distributed mode

Posted by "Edward J. Yoon" <ed...@apache.org>.

You should increase max heap size of child JVMs, instead of HAMA_HEAPSIZE.

Add below property to hama-site.xml.

  <property>
    <name>bsp.child.java.opts</name>
    <value>-Xmx2048m</value>
    <description>Java opts for the groom server child processes.
    The following symbol, if present, will be interpolated: @taskid@
is replaced
    by current TaskID. Any other occurrences of '@' will go unchanged.
    For example, to enable verbose gc logging to a file named for the taskid in
    /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
          -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
    The configuration variable bsp.child.ulimit can be used to control the
    maximum virtual memory of the child processes.
    </description>
  </property>

On Thu, Dec 27, 2012 at 5:11 AM, Francisco Sanches
<sa...@gmail.com> wrote:
> Thanks for the replies:
>
> Excuse me by the size of the log. But after I changed the parameter of
> hama-env.sh to "export HAMA_HEAPSIZE = 2000" My program stopped giving the
> memory error and displays the error below. Remembering that I am running
> for a graph with just over 65 000 vertices. What is it like to understand
> is that if I increase the number of nodes in the cluster can run my
> application with this graph of 65 000 vertices?
>
> LOG:
>
> hduser@mestre:/usr/local/hama$ bin/hama jar aplicpronto.jar 65537
> /hdfs/disco/grafogrande /hdfs/disco/grafoout 36
> 12/12/26 17:00:19 INFO bsp.FileInputFormat: Total input paths to process : 1
> 12/12/26 17:00:19 INFO bsp.FileInputFormat: Total # of splits: 36
> 12/12/26 17:00:23 INFO bsp.BSPJobClient: Running job: job_201212261658_0001
> 12/12/26 17:00:26 INFO bsp.BSPJobClient: Current supersteps number: 0
> 12/12/26 17:00:56 INFO bsp.BSPJobClient: Current supersteps number: 1
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969,
> built on 02/23/2011 22:27 GMT
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:host.name=mestre
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-6-openjdk-i386/jre
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client
> environment:java.class.path=/usr/local/hama/bin/../conf:/usr/lib/jvm/java-6-openjdk-i386/lib/tools.jar:/usr/local/hama/bin/../hama-core-0.5.0.jar:/usr/local/hama/bin/../hama-examples-0.5.0.jar:/usr/local/hama/bin/../hama-graph-0.5.0.jar:/usr/local/hama/bin/../lib/ant-1.7.1.jar:/usr/local/hama/bin/../lib/ant-launcher-1.7.1.jar:/usr/local/hama/bin/../lib/avro-1.6.0.jar:/usr/local/hama/bin/../lib/avro-ipc-1.6.0.jar:/usr/local/hama/bin/../lib/commons-cli-1.2.jar:/usr/local/hama/bin/../lib/commons-configuration-1.7.jar:/usr/local/hama/bin/../lib/commons-lang-2.6.jar:/usr/local/hama/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hama/bin/../lib/commons-math3-3.0.jar:/usr/local/hama/bin/../lib/guava-10.0.1.jar:/usr/local/hama/bin/../lib/hadoop-core-1.0.0.jar:/usr/local/hama/bin/../lib/hadoop-test-1.0.0.jar:/usr/local/hama/bin/../lib/jackson-core-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jackson-mapper-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jetty-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-annotations-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-util-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/junit-4.8.1.jar:/usr/local/hama/bin/../lib/log4j-1.2.16.jar:/usr/local/hama/bin/../lib/netty-3.2.6.Final.jar:/usr/local/hama/bin/../lib/servlet-api-6.0.32.jar:/usr/local/hama/bin/../lib/slf4j-api-1.5.8.jar:/usr/local/hama/bin/../lib/slf4j-log4j12-1.5.8.jar:/usr/local/hama/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hama/bin/../lib/zookeeper-3.3.3.jar::/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/classes:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client
> environment:java.library.path=/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386:/usr/lib/jvm/java-6-openjdk-i386/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/i386-linux-gnu/jni:/lib/i386-linux-gnu:/usr/lib/i386-linux-gnu:/usr/lib/jni:/lib:/usr/lib
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:os.name=Linux
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:os.arch=i386
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:os.version=3.2.0-32-generic-pae
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:user.name=hduser
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client environment:user.home=/home/hduser
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Client
> environment:user.dir=/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ZooKeeper: Initiating client connection,
> connectString=node02:21810,node01:21810,mestre:21810 sessionTimeout=1200000
> watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImpl@1deeb40
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ClientCnxn: Opening socket connection to server mestre/
> 10.16.1.200:21810
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> zookeeper.ClientCnxn: Socket connection established to mestre/
> 10.16.1.200:21810, initiating session
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
> sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At mestre/
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:32 INFO
> zookeeper.ClientCnxn: Session establishment complete on server mestre/
> 10.16.1.200:21810, sessionid = 0x3bd8cbb48a0006, negotiated timeout =
> 1200000
> attempt_201212261658_0001_000000_0: SLF4J: Class path contains multiple
> SLF4J bindings.
> attempt_201212261658_0001_000000_0: SLF4J: Found binding in
> [jar:file:/usr/local/hama/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> attempt_201212261658_0001_000000_0: SLF4J: Found binding in
> [file:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/org/slf4j/impl/StaticLoggerBinder.class]
> attempt_201212261658_0001_000000_0: SLF4J: See
> http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x002f8116] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/
> 10.16.1.203:61003] BOUND: /10.16.1.200:47685
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/
> 10.16.1.203:61003] CONNECTED: node03/10.16.1.203:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00d8ca48] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/
> 10.16.1.205:61001] BOUND: /10.16.1.200:52329
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/
> 10.16.1.205:61001] CONNECTED: node05/10.16.1.205:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018adae2] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/
> 10.16.1.204:61002] BOUND: /10.16.1.200:59179
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/
> 10.16.1.204:61002] CONNECTED: node04/10.16.1.204:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x003a835d] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/
> 10.16.1.206:61001] BOUND: /10.16.1.200:45011
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/
> 10.16.1.206:61001] CONNECTED: node06/10.16.1.206:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00061cd2] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/
> 10.16.1.204:61003] BOUND: /10.16.1.200:54578
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/
> 10.16.1.204:61003] CONNECTED: node04/10.16.1.204:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00fb6c5f] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/
> 10.16.1.205:61002] BOUND: /10.16.1.200:52964
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/
> 10.16.1.205:61002] CONNECTED: node05/10.16.1.205:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00cb42cf] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/
> 10.16.1.203:61001] BOUND: /10.16.1.200:36765
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/
> 10.16.1.203:61001] CONNECTED: node03/10.16.1.203:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018bbf55] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/
> 10.16.1.201:61003] BOUND: /10.16.1.200:45734
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/
> 10.16.1.201:61003] CONNECTED: node01/10.16.1.201:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00ed5d9d] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/
> 10.16.1.202:61002] BOUND: /10.16.1.200:60724
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/
> 10.16.1.202:61002] CONNECTED: node02/10.16.1.202:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x014b84c7] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/
> 10.16.1.203:61002] BOUND: /10.16.1.200:45839
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/
> 10.16.1.203:61002] CONNECTED: node03/10.16.1.203:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0160c4b0] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/
> 10.16.1.202:61003] BOUND: /10.16.1.200:44757
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/
> 10.16.1.202:61003] CONNECTED: node02/10.16.1.202:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0029ea31] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/
> 10.16.1.204:61001] BOUND: /10.16.1.200:40846
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/
> 10.16.1.204:61001] CONNECTED: node04/10.16.1.204:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.210:36396
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.200:41941
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.206:44493
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.202:48243
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.200:41945
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00a86d12] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/
> 10.16.1.201:61001] BOUND: /10.16.1.200:35763
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/
> 10.16.1.201:61001] CONNECTED: node01/10.16.1.201:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x008932e8] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.200:41949
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/
> 10.16.1.200:61002] BOUND: /10.16.1.200:41949
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/
> 10.16.1.200:61002] CONNECTED: mestre/10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00f2164f] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.209:43193
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/
> 10.16.1.201:61002] BOUND: /10.16.1.200:44908
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
> ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/
> 10.16.1.201:61002] CONNECTED: node01/10.16.1.201:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.207:57476
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.207:57478
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.210:36402
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.201:37455
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.207:57480
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.205:56188
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.206:44506
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.205:56189
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.202:48251
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.205:56191
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.208:50918
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.208:50917
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.209:43200
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
> [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.209:43201
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.208:50924
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.211:44099
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.206:44511
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.211:44100
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.211:44101
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x019c4091] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.201:37460
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/
> 10.16.1.202:61001] BOUND: /10.16.1.200:54333
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/
> 10.16.1.202:61001] CONNECTED: node02/10.16.1.202:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.203:46568
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.203:46569
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e80761] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/
> 10.16.1.200:61003] BOUND: /10.16.1.200:56327
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/
> 10.16.1.200:61003] CONNECTED: mestre/10.16.1.200:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01dbb27d] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.203:46573
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/
> 10.16.1.200:61001] BOUND: /10.16.1.200:51740
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/
> 10.16.1.200:61001] CONNECTED: mestre/10.16.1.200:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.204:35237
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.204:35239
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01971eb3] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.202:48260
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/
> 10.16.1.211:61003] BOUND: /10.16.1.200:57399
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/
> 10.16.1.211:61003] CONNECTED: node11/10.16.1.211:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.201:37463
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x005b78cf] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/
> 10.16.1.211:61001] BOUND: /10.16.1.200:41092
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/
> 10.16.1.211:61001] CONNECTED: node11/10.16.1.211:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] BOUND: /
> 10.16.1.200:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
> [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] CONNECTED: /
> 10.16.1.204:35245
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01f51e5c] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/
> 10.16.1.210:61002] BOUND: /10.16.1.200:48018
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/
> 10.16.1.210:61002] CONNECTED: node10/10.16.1.210:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d36ff3] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/
> 10.16.1.209:61003] BOUND: /10.16.1.200:34916
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/
> 10.16.1.209:61003] CONNECTED: node09/10.16.1.209:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x018e899c] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/
> 10.16.1.211:61002] BOUND: /10.16.1.200:58261
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/
> 10.16.1.211:61002] CONNECTED: node11/10.16.1.211:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x014bc4e6] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/
> 10.16.1.210:61003] BOUND: /10.16.1.200:39073
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/
> 10.16.1.210:61003] CONNECTED: node10/10.16.1.210:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01be1041] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/
> 10.16.1.208:61002] BOUND: /10.16.1.200:39758
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/
> 10.16.1.208:61002] CONNECTED: node08/10.16.1.208:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e090ee] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/
> 10.16.1.209:61001] BOUND: /10.16.1.200:56029
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/
> 10.16.1.209:61001] CONNECTED: node09/10.16.1.209:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x0187f9f1] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/
> 10.16.1.207:61003] BOUND: /10.16.1.200:35151
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/
> 10.16.1.207:61003] CONNECTED: node07/10.16.1.207:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00c3358b] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/
> 10.16.1.208:61003] BOUND: /10.16.1.200:35301
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/
> 10.16.1.208:61003] CONNECTED: node08/10.16.1.208:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e82b72] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/
> 10.16.1.210:61001] BOUND: /10.16.1.200:34819
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/
> 10.16.1.210:61001] CONNECTED: node10/10.16.1.210:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d0deb5] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/
> 10.16.1.209:61002] BOUND: /10.16.1.200:34684
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/
> 10.16.1.209:61002] CONNECTED: node09/10.16.1.209:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01505258] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/
> 10.16.1.206:61002] BOUND: /10.16.1.200:35962
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/
> 10.16.1.206:61002] CONNECTED: node06/10.16.1.206:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01d8417c] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/
> 10.16.1.207:61001] BOUND: /10.16.1.200:41009
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/
> 10.16.1.207:61001] CONNECTED: node07/10.16.1.207:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x017c398e] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/
> 10.16.1.205:61003] BOUND: /10.16.1.200:53654
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/
> 10.16.1.205:61003] CONNECTED: node05/10.16.1.205:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00e13e7b] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/
> 10.16.1.208:61001] BOUND: /10.16.1.200:55138
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/
> 10.16.1.208:61001] CONNECTED: node08/10.16.1.208:61001
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01503458] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/
> 10.16.1.206:61003] BOUND: /10.16.1.200:40540
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/
> 10.16.1.206:61003] CONNECTED: node06/10.16.1.206:61003
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01b963c4] OPEN
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/
> 10.16.1.207:61002] BOUND: /10.16.1.200:42816
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
> ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/
> 10.16.1.207:61002] CONNECTED: node07/10.16.1.207:61002
> attempt_201212261658_0001_000000_0: 12/12/26 17:00:53 INFO
> graph.GraphJobRunner: Loading finished at 1 steps.
> 12/12/26 17:03:17 INFO bsp.BSPJobClient: Job failed.
>
>
>
> 2012/12/26 Edward J. Yoon <ed...@apache.org>
>
>> BTW, This log looks like you ran out of memory while job was running
>> as a local mode.
>>
>> You can check the cluster status via web UI.
>>
>> 1. http://wiki.apache.org/hama/GettingStarted#Hama_Web_Interfaces
>>
>> On Wed, Dec 26, 2012 at 10:00 PM, Francisco Sanches
>> <sa...@gmail.com> wrote:
>> > attempt_201212260904_0005_000031_0: Exception in thread "Thread-1"
>> > java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Francisco Sanches



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: What is the best configuration for Cluster hama distributed mode

Posted by Francisco Sanches <sa...@gmail.com>.

Thanks for the replies:

Excuse me by the size of the log. But after I changed the parameter of
hama-env.sh to "export HAMA_HEAPSIZE = 2000" My program stopped giving the
memory error and displays the error below. Remembering that I am running
for a graph with just over 65 000 vertices. What is it like to understand
is that if I increase the number of nodes in the cluster can run my
application with this graph of 65 000 vertices?

LOG:

hduser@mestre:/usr/local/hama$ bin/hama jar aplicpronto.jar 65537
/hdfs/disco/grafogrande /hdfs/disco/grafoout 36
12/12/26 17:00:19 INFO bsp.FileInputFormat: Total input paths to process : 1
12/12/26 17:00:19 INFO bsp.FileInputFormat: Total # of splits: 36
12/12/26 17:00:23 INFO bsp.BSPJobClient: Running job: job_201212261658_0001
12/12/26 17:00:26 INFO bsp.BSPJobClient: Current supersteps number: 0
12/12/26 17:00:56 INFO bsp.BSPJobClient: Current supersteps number: 1
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969,
built on 02/23/2011 22:27 GMT
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:host.name=mestre
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-6-openjdk-i386/jre
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client
environment:java.class.path=/usr/local/hama/bin/../conf:/usr/lib/jvm/java-6-openjdk-i386/lib/tools.jar:/usr/local/hama/bin/../hama-core-0.5.0.jar:/usr/local/hama/bin/../hama-examples-0.5.0.jar:/usr/local/hama/bin/../hama-graph-0.5.0.jar:/usr/local/hama/bin/../lib/ant-1.7.1.jar:/usr/local/hama/bin/../lib/ant-launcher-1.7.1.jar:/usr/local/hama/bin/../lib/avro-1.6.0.jar:/usr/local/hama/bin/../lib/avro-ipc-1.6.0.jar:/usr/local/hama/bin/../lib/commons-cli-1.2.jar:/usr/local/hama/bin/../lib/commons-configuration-1.7.jar:/usr/local/hama/bin/../lib/commons-lang-2.6.jar:/usr/local/hama/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hama/bin/../lib/commons-math3-3.0.jar:/usr/local/hama/bin/../lib/guava-10.0.1.jar:/usr/local/hama/bin/../lib/hadoop-core-1.0.0.jar:/usr/local/hama/bin/../lib/hadoop-test-1.0.0.jar:/usr/local/hama/bin/../lib/jackson-core-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jackson-mapper-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jetty-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-annotations-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-util-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/junit-4.8.1.jar:/usr/local/hama/bin/../lib/log4j-1.2.16.jar:/usr/local/hama/bin/../lib/netty-3.2.6.Final.jar:/usr/local/hama/bin/../lib/servlet-api-6.0.32.jar:/usr/local/hama/bin/../lib/slf4j-api-1.5.8.jar:/usr/local/hama/bin/../lib/slf4j-log4j12-1.5.8.jar:/usr/local/hama/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hama/bin/../lib/zookeeper-3.3.3.jar::/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/classes:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386:/usr/lib/jvm/java-6-openjdk-i386/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/i386-linux-gnu/jni:/lib/i386-linux-gnu:/usr/lib/i386-linux-gnu:/usr/lib/jni:/lib:/usr/lib
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:os.name=Linux
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:os.arch=i386
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:os.version=3.2.0-32-generic-pae
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:user.name=hduser
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client environment:user.home=/home/hduser
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Client
environment:user.dir=/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ZooKeeper: Initiating client connection,
connectString=node02:21810,node01:21810,mestre:21810 sessionTimeout=1200000
watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImpl@1deeb40
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ClientCnxn: Opening socket connection to server mestre/
10.16.1.200:21810
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
zookeeper.ClientCnxn: Socket connection established to mestre/
10.16.1.200:21810, initiating session
attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO
sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At mestre/
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:32 INFO
zookeeper.ClientCnxn: Session establishment complete on server mestre/
10.16.1.200:21810, sessionid = 0x3bd8cbb48a0006, negotiated timeout =
1200000
attempt_201212261658_0001_000000_0: SLF4J: Class path contains multiple
SLF4J bindings.
attempt_201212261658_0001_000000_0: SLF4J: Found binding in
[jar:file:/usr/local/hama/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201212261658_0001_000000_0: SLF4J: Found binding in
[file:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201212261658_0001_000000_0: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x002f8116] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/
10.16.1.203:61003] BOUND: /10.16.1.200:47685
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/
10.16.1.203:61003] CONNECTED: node03/10.16.1.203:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00d8ca48] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/
10.16.1.205:61001] BOUND: /10.16.1.200:52329
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/
10.16.1.205:61001] CONNECTED: node05/10.16.1.205:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x018adae2] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/
10.16.1.204:61002] BOUND: /10.16.1.200:59179
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/
10.16.1.204:61002] CONNECTED: node04/10.16.1.204:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x003a835d] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/
10.16.1.206:61001] BOUND: /10.16.1.200:45011
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/
10.16.1.206:61001] CONNECTED: node06/10.16.1.206:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00061cd2] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/
10.16.1.204:61003] BOUND: /10.16.1.200:54578
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/
10.16.1.204:61003] CONNECTED: node04/10.16.1.204:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00fb6c5f] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/
10.16.1.205:61002] BOUND: /10.16.1.200:52964
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/
10.16.1.205:61002] CONNECTED: node05/10.16.1.205:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00cb42cf] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/
10.16.1.203:61001] BOUND: /10.16.1.200:36765
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/
10.16.1.203:61001] CONNECTED: node03/10.16.1.203:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x018bbf55] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/
10.16.1.201:61003] BOUND: /10.16.1.200:45734
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/
10.16.1.201:61003] CONNECTED: node01/10.16.1.201:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00ed5d9d] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/
10.16.1.202:61002] BOUND: /10.16.1.200:60724
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/
10.16.1.202:61002] CONNECTED: node02/10.16.1.202:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x014b84c7] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/
10.16.1.203:61002] BOUND: /10.16.1.200:45839
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/
10.16.1.203:61002] CONNECTED: node03/10.16.1.203:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x0160c4b0] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/
10.16.1.202:61003] BOUND: /10.16.1.200:44757
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/
10.16.1.202:61003] CONNECTED: node02/10.16.1.202:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x0029ea31] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/
10.16.1.204:61001] BOUND: /10.16.1.200:40846
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/
10.16.1.204:61001] CONNECTED: node04/10.16.1.204:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] CONNECTED: /
10.16.1.210:36396
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] CONNECTED: /
10.16.1.200:41941
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] CONNECTED: /
10.16.1.206:44493
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] CONNECTED: /
10.16.1.202:48243
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] CONNECTED: /
10.16.1.200:41945
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00a86d12] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/
10.16.1.201:61001] BOUND: /10.16.1.200:35763
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/
10.16.1.201:61001] CONNECTED: node01/10.16.1.201:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x008932e8] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] CONNECTED: /
10.16.1.200:41949
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/
10.16.1.200:61002] BOUND: /10.16.1.200:41949
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/
10.16.1.200:61002] CONNECTED: mestre/10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00f2164f] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] CONNECTED: /
10.16.1.209:43193
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/
10.16.1.201:61002] BOUND: /10.16.1.200:44908
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO
ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/
10.16.1.201:61002] CONNECTED: node01/10.16.1.201:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] CONNECTED: /
10.16.1.207:57476
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] CONNECTED: /
10.16.1.207:57478
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] CONNECTED: /
10.16.1.210:36402
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] CONNECTED: /
10.16.1.201:37455
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] CONNECTED: /
10.16.1.207:57480
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] CONNECTED: /
10.16.1.205:56188
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] CONNECTED: /
10.16.1.206:44506
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] CONNECTED: /
10.16.1.205:56189
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] CONNECTED: /
10.16.1.202:48251
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] CONNECTED: /
10.16.1.205:56191
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] CONNECTED: /
10.16.1.208:50918
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] CONNECTED: /
10.16.1.208:50917
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] CONNECTED: /
10.16.1.209:43200
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO ipc.NettyServer:
[id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] CONNECTED: /
10.16.1.209:43201
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] CONNECTED: /
10.16.1.208:50924
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] CONNECTED: /
10.16.1.211:44099
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] CONNECTED: /
10.16.1.206:44511
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] CONNECTED: /
10.16.1.211:44100
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] CONNECTED: /
10.16.1.211:44101
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x019c4091] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] CONNECTED: /
10.16.1.201:37460
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/
10.16.1.202:61001] BOUND: /10.16.1.200:54333
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/
10.16.1.202:61001] CONNECTED: node02/10.16.1.202:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] CONNECTED: /
10.16.1.203:46568
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] CONNECTED: /
10.16.1.203:46569
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01e80761] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/
10.16.1.200:61003] BOUND: /10.16.1.200:56327
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/
10.16.1.200:61003] CONNECTED: mestre/10.16.1.200:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01dbb27d] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] CONNECTED: /
10.16.1.203:46573
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/
10.16.1.200:61001] BOUND: /10.16.1.200:51740
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/
10.16.1.200:61001] CONNECTED: mestre/10.16.1.200:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] CONNECTED: /
10.16.1.204:35237
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] CONNECTED: /
10.16.1.204:35239
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01971eb3] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] CONNECTED: /
10.16.1.202:48260
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/
10.16.1.211:61003] BOUND: /10.16.1.200:57399
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/
10.16.1.211:61003] CONNECTED: node11/10.16.1.211:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] CONNECTED: /
10.16.1.201:37463
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x005b78cf] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/
10.16.1.211:61001] BOUND: /10.16.1.200:41092
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/
10.16.1.211:61001] CONNECTED: node11/10.16.1.211:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] BOUND: /
10.16.1.200:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO ipc.NettyServer:
[id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] CONNECTED: /
10.16.1.204:35245
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01f51e5c] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/
10.16.1.210:61002] BOUND: /10.16.1.200:48018
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/
10.16.1.210:61002] CONNECTED: node10/10.16.1.210:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00d36ff3] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/
10.16.1.209:61003] BOUND: /10.16.1.200:34916
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/
10.16.1.209:61003] CONNECTED: node09/10.16.1.209:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x018e899c] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/
10.16.1.211:61002] BOUND: /10.16.1.200:58261
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/
10.16.1.211:61002] CONNECTED: node11/10.16.1.211:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x014bc4e6] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/
10.16.1.210:61003] BOUND: /10.16.1.200:39073
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/
10.16.1.210:61003] CONNECTED: node10/10.16.1.210:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01be1041] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/
10.16.1.208:61002] BOUND: /10.16.1.200:39758
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/
10.16.1.208:61002] CONNECTED: node08/10.16.1.208:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01e090ee] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/
10.16.1.209:61001] BOUND: /10.16.1.200:56029
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/
10.16.1.209:61001] CONNECTED: node09/10.16.1.209:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x0187f9f1] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/
10.16.1.207:61003] BOUND: /10.16.1.200:35151
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/
10.16.1.207:61003] CONNECTED: node07/10.16.1.207:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00c3358b] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/
10.16.1.208:61003] BOUND: /10.16.1.200:35301
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/
10.16.1.208:61003] CONNECTED: node08/10.16.1.208:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01e82b72] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/
10.16.1.210:61001] BOUND: /10.16.1.200:34819
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/
10.16.1.210:61001] CONNECTED: node10/10.16.1.210:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00d0deb5] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/
10.16.1.209:61002] BOUND: /10.16.1.200:34684
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/
10.16.1.209:61002] CONNECTED: node09/10.16.1.209:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01505258] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/
10.16.1.206:61002] BOUND: /10.16.1.200:35962
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/
10.16.1.206:61002] CONNECTED: node06/10.16.1.206:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01d8417c] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/
10.16.1.207:61001] BOUND: /10.16.1.200:41009
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/
10.16.1.207:61001] CONNECTED: node07/10.16.1.207:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x017c398e] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/
10.16.1.205:61003] BOUND: /10.16.1.200:53654
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/
10.16.1.205:61003] CONNECTED: node05/10.16.1.205:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00e13e7b] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/
10.16.1.208:61001] BOUND: /10.16.1.200:55138
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/
10.16.1.208:61001] CONNECTED: node08/10.16.1.208:61001
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01503458] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/
10.16.1.206:61003] BOUND: /10.16.1.200:40540
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/
10.16.1.206:61003] CONNECTED: node06/10.16.1.206:61003
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01b963c4] OPEN
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/
10.16.1.207:61002] BOUND: /10.16.1.200:42816
attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO
ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/
10.16.1.207:61002] CONNECTED: node07/10.16.1.207:61002
attempt_201212261658_0001_000000_0: 12/12/26 17:00:53 INFO
graph.GraphJobRunner: Loading finished at 1 steps.
12/12/26 17:03:17 INFO bsp.BSPJobClient: Job failed.



2012/12/26 Edward J. Yoon <ed...@apache.org>

> BTW, This log looks like you ran out of memory while job was running
> as a local mode.
>
> You can check the cluster status via web UI.
>
> 1. http://wiki.apache.org/hama/GettingStarted#Hama_Web_Interfaces
>
> On Wed, Dec 26, 2012 at 10:00 PM, Francisco Sanches
> <sa...@gmail.com> wrote:
> > attempt_201212260904_0005_000031_0: Exception in thread "Thread-1"
> > java.lang.OutOfMemoryError: GC overhead limit exceeded
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>



-- 
Francisco Sanches

Re: What is the best configuration for Cluster hama distributed mode

Posted by "Edward J. Yoon" <ed...@apache.org>.

BTW, This log looks like you ran out of memory while job was running
as a local mode.

You can check the cluster status via web UI.

1. http://wiki.apache.org/hama/GettingStarted#Hama_Web_Interfaces

On Wed, Dec 26, 2012 at 10:00 PM, Francisco Sanches
<sa...@gmail.com> wrote:
> attempt_201212260904_0005_000031_0: Exception in thread "Thread-1"
> java.lang.OutOfMemoryError: GC overhead limit exceeded



-- 
Best Regards, Edward J. Yoon
@eddieyoon