BUG ? SequenceFileVertexInputFormat with custom Writable : can not iterate over vertices

[ol...@orange.com]

Hi,

first of all many thanks to the community for the great work you are doing.


with giraph 1.1-SNAPSHOT
built for hadoop 2.4.0


I am stalled on a bug using custom SequenceFile Vertex Input format.


I have opened a project on github to allow you to run/play/view the bug

https://github.com/ovarene/giraph_SequenceFileVertexInput


you will find everything needed :D



I try to load/create my vertices from a SequenceFile<LongWritable,MyWritable>
"LongWritable" being the vertexId
"MyWritable" being a custom value writable used to generate VertexValue and its edges


Vertices are created correctly and loaded (as seen on the log)

2014-09-04 10:44:44,683 DEBUG [load-0] MyReader (SequenceFileLongMyVertexInputFormat.java:getCurrentVertex(108)) - Create Vertex from "1.0 0.0 0:" to : Vertex(id=1,value=1.0,#edges=0) 2014-09-04 10:44:44,694 DEBUG [load-0] MyReader (SequenceFileLongMyVertexInputFormat.java:getCurrentVertex(108)) - Create Vertex from "2.0 0.0 2:1|6|" to : Vertex(id=2,value=2.0,#edges=2) 2014-09-04 10:44:44,695 DEBUG [load-0] MyReader (SequenceFileLongMyVertexInputFormat.java:getCurrentVertex(108)) - Create Vertex from "5.0 0.0 3:4|1|6|" to : Vertex(id=5,value=5.0,#edges=3)

2014-09-04 10:44:44,697 INFO [load-0] worker.InputSplitsCallable (InputSplitsCallable.java:call(235)) - call: Loaded 1 input splits in 0.12198963 secs, (v=3, e=5) 24.592255 vertices/sec, 40.98709 edges/sec


But alas, when trying to iterate over them during a compute session, I get the following error

ERROR : But when reading in PrintVertex (BasicComputation): 2014-09-04 10:44:44,701 ERROR [load-0] utils.LogStacktraceCallable (LogStacktraceCallable.java:call(57)) - Execution of callable failed java.lang.IllegalStateException: next: IOException at org.apache.giraph.utils.VertexIterator.next(VertexIterator.java:101) at org.apache.giraph.partition.BasicPartition.addPartitionVertices(BasicPartition.java:99) at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:110) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.doRequest(NettyWorkerClientRequestProcessor.java:482) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.flush(NettyWorkerClientRequestProcessor.java:428) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:241) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60) at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: ensureRemaining: Only 5 bytes remaining, trying to read 8 at org.apache.giraph.utils.UnsafeByteArrayInputStream.ensureRemaining(UnsafeByteArrayInputStream.java:114) at org.apache.giraph.utils.UnsafeByteArrayInputStream.readLong(UnsafeByteArrayInputStream.java:197) at org.apache.hadoop.io.LongWritable.readFields(LongWritable.java:47) at org.apache.giraph.utils.WritableUtils.reinitializeVertexFromDataInput(WritableUtils.java:522) at org.apache.giraph.utils.VertexIterator.next(VertexIterator.java:98) ... 11 more


Any idea ?

Thanks a lot

Olivier










_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.