Problem partitioner? Aggregator?

[Francisco Sanches <sa...@gmail.com>]

I have tried to perform my job but after some updates in Hama I have
received the following message:

Messages must never be behind the vertex in ID! Current Message ID: 13064
vs. 13079

I'm using the Hama-Trunk. The complete log is below.


hduser @ node100: ~ / hama-2 $ time bin / hama jar FastGDC_4.jar 40001 /
hdfs/disco/dados/v40mila200mil / hdfs/disco/v40mila200_out 18 30
Mortbay.log 04/13/17 09:02:19 INFO: Logging to
org.slf4j.impl.Log4jLoggerAdapter (org.mortbay.log) via
org.mortbay.log.Slf4jLog
Bsp.FileInputFormat 04/13/17 09:02:20 INFO: Total input paths to process: 1
04/13/17 09:02:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform ... using builtin-java classes where applicable
04/13/17 09:02:20 WARN snappy.LoadSnappy: Snappy native library not loaded
Bsp.FileInputFormat 04/13/17 09:02:20 INFO: Total input paths to process: 1
Bsp.BSPJobClient 04/13/17 09:02:27 INFO: Running job: job_201304170858_0002
04/13/17 09:02:30 INFO bsp.BSPJobClient: Current supersteps number: 0
04/13/17 09:02:39 INFO bsp.BSPJobClient: Current supersteps number: 2
04/13/17 09:02:42 INFO bsp.BSPJobClient: The overall number of supersteps: 2
Bsp.BSPJobClient 04/13/17 09:02:42 INFO: Counters: 6
04/13/17 09:02:42 INFO bsp.BSPJobClient: $
org.apache.hama.bsp.JobInProgress JobCounter
04/13/17 09:02:42 INFO bsp.BSPJobClient: SUPERSTEPS = 2
04/13/17 09:02:42 INFO bsp.BSPJobClient: LAUNCHED_TASKS = 1
04/13/17 09:02:42 INFO bsp.BSPJobClient: $ org.apache.hama.bsp.BSPPeerImpl
PeerCounter
04/13/17 09:02:42 INFO bsp.BSPJobClient: SUPERSTEP_SUM = 2
04/13/17 09:02:42 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS = 68
04/13/17 09:02:42 INFO bsp.BSPJobClient: IO_BYTES_READ = 1411919
04/13/17 09:02:42 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS = 39688
Bsp.FileInputFormat 04/13/17 09:02:42 INFO: Total input paths to process: 18
Bsp.BSPJobClient 04/13/17 09:02:45 INFO: Running job: job_201304170858_0001
04/13/17 09:02:48 INFO bsp.BSPJobClient: Current supersteps number: 0
04/13/17 09:03:15 INFO bsp.BSPJobClient: Current supersteps number: 1
04/13/17 09:05:34 INFO bsp.BSPJobClient: Current supersteps number: 2
attempt_201304170858_0001_000003_0: SLF4J: Class path contains multiple
SLF4J bindings.
attempt_201304170858_0001_000003_0: SLF4J: Found binding in [jar: file :/
home/hduser/hama-2/lib/slf4j-log4j12-1.5.8.jar! /
org/slf4j/impl/StaticLoggerBinder.class]
attempt_201304170858_0001_000003_0: SLF4J: Found binding in
attempt_201304170858_0001_000003_0: SLF4J: See
http://www.slf4j.org/codes.html # multiple_bindings for an explanation.
attempt_201304170858_0001_000003_0: mortbay.log 04/13/17 08:35:42 INFO:
Logging to org.slf4j.impl.Log4jLoggerAdapter (org.mortbay.log) via
org.mortbay.log.Slf4jLog
attempt_201304170858_0001_000003_0: sync.ZKSyncClient 04/13/17 08:35:51
INFO: Initializing ZK Sync Client
attempt_201304170858_0001_000003_0: 04/13/17 08:35:51 INFO
sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At node109/
10.16.5.109: 61001
attempt_201304170858_0001_000003_0: 04/13/17 08:35:52 ERROR
sync.ZooKeeperSyncClientImpl: $ org.apache.zookeeper.KeeperException
NodeExistsException: KeeperErrorCode = NodeExists for /
bsp/job_201304170858_0001/peers
attempt_201304170858_0001_000003_0: ipc.Server 04/13/17 08:35:54 INFO:
Starting SocketReader
attempt_201304170858_0001_000003_0: 04/13/17 08:35:54 INFO ipc.Server: IPC
Server Reply: starting
attempt_201304170858_0001_000003_0: 04/13/17 08:35:54 INFO ipc.Server: IPC
Server listener on 61001: starting
attempt_201304170858_0001_000003_0: 04/13/17 08:35:54 INFO
message.HadoopMessageManagerImpl: BSPPeer address: port node109: 61001
attempt_201304170858_0001_000003_0: 04/13/17 08:35:54 INFO ipc.Server: IPC
Server handler 0 on 61001: starting
attempt_201304170858_0001_000003_0: 04/13/17 08:36:06 INFO
graph.GraphJobRunner: 2200 vertices are loaded into node109: 61001
attempt_201304170858_0001_000003_0: bsp.BSPTask 04/13/17 08:38:23 ERROR:
Error running bsp bsp setup and function.
attempt_201304170858_0001_000003_0: java.lang.IllegalArgumentException:
Messages must never be behind the vertex in ID! Current Message ID: 13064
vs. 13079
attempt_201304170858_0001_000003_0: at
org.apache.hama.graph.GraphJobRunner.iterate (GraphJobRunner.java: 280)
attempt_201304170858_0001_000003_0: at
org.apache.hama.graph.GraphJobRunner.doSuperstep (GraphJobRunner.java: 226)
attempt_201304170858_0001_000003_0: at
org.apache.hama.graph.GraphJobRunner.bsp (GraphJobRunner.java: 130)
attempt_201304170858_0001_000003_0: at org.apache.hama.bsp.BSPTask.runBSP
(BSPTask.java: 177)
attempt_201304170858_0001_000003_0: at org.apache.hama.bsp.BSPTask.run
(BSPTask.java: 146)
attempt_201304170858_0001_000003_0: org.apache.hama.bsp.GroomServer at $
BSPPeerChild.main (GroomServer.java: 1262)
attempt_201304170858_0001_000003_0: ipc.Server 04/13/17 08:38:29 INFO:
Stopping server on 61001
attempt_201304170858_0001_000003_0: ipc.Server 04/13/17 08:38:29 INFO:
Stopping IPC Server listener on 61001
attempt_201304170858_0001_000003_0: 04/13/17 08:38:29 INFO ipc.Server: IPC
Server handler 0 on 61001: exiting
attempt_201304170858_0001_000003_0: 04/13/17 08:38:29 INFO
metrics.RpcInstrumentation: shut down
attempt_201304170858_0001_000003_0: 04/13/17 08:38:29 ERROR bsp.BSPTask:
Shutting down ping service.
attempt_201304170858_0001_000003_0: 04/13/17 08:38:29 bsp.GroomServer
FATAL: Error running child
attempt_201304170858_0001_000003_0: java.lang.IllegalArgumentException:
Messages must never be behind the vertex in ID! Current Message ID: 13064
vs. 13079
attempt_201304170858_0001_000003_0: at
org.apache.hama.graph.GraphJobRunner.iterate (GraphJobRunner.java: 280)
attempt_201304170858_0001_000003_0: at
org.apache.hama.graph.GraphJobRunner.doSuperstep (GraphJobRunner.java: 226)
attempt_201304170858_0001_000003_0: at
org.apache.hama.graph.GraphJobRunner.bsp (GraphJobRunner.java: 130)
attempt_201304170858_0001_000003_0: at org.apache.hama.bsp.BSPTask.runBSP
(BSPTask.java: 177)
attempt_201304170858_0001_000003_0: at org.apache.hama.bsp.BSPTask.run
(BSPTask.java: 146)
attempt_201304170858_0001_000003_0: org.apache.hama.bsp.GroomServer at $
BSPPeerChild.main (GroomServer.java: 1262)
attempt_201304170858_0001_000003_0: ipc.Server 04/13/17 08:38:29 INFO:
Stopping IPC Server Reply
attempt_201304170858_0001_000003_0: java.lang.IllegalArgumentException:
Messages must never be behind the vertex in ID! Current Message ID: 13064
vs. 13079
attempt_201304170858_0001_000003_0: at
org.apache.hama.graph.GraphJobRunner.iterate (GraphJobRunner.java: 280)
attempt_201304170858_0001_000003_0: at
org.apache.hama.graph.GraphJobRunner.doSuperstep (GraphJobRunner.java: 226)
attempt_201304170858_0001_000003_0: at
org.apache.hama.graph.GraphJobRunner.bsp (GraphJobRunner.java: 130)
attempt_201304170858_0001_000003_0: at org.apache.hama.bsp.BSPTask.runBSP
(BSPTask.java: 177)
attempt_201304170858_0001_000003_0: at org.apache.hama.bsp.BSPTask.run
(BSPTask.java: 146)
attempt_201304170858_0001_000003_0: org.apache.hama.bsp.GroomServer at $
BSPPeerChild.main (GroomServer.java: 1262)
04/13/17 09:06:42 INFO bsp.BSPJobClient: Job failed.

real 4m28.022s

-- 
Francisco Sanches