You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by ol...@orange.com on 2014/09/04 15:31:19 UTC
BUG ? SequenceFileVertexInputFormat with custom Writable : can not
iterate over vertices
Hi,
first of all many thanks to the community for the great work you are doing.
with giraph 1.1-SNAPSHOT
built for hadoop 2.4.0
I am stalled on a bug using custom SequenceFile Vertex Input format.
I have opened a project on github to allow you to run/play/view the bug
https://github.com/ovarene/giraph_SequenceFileVertexInput
you will find everything needed :D
I try to load/create my vertices from a SequenceFile<LongWritable,MyWritable>
"LongWritable" being the vertexId
"MyWritable" being a custom value writable used to generate VertexValue and its edges
Vertices are created correctly and loaded (as seen on the log)
2014-09-04 10:44:44,683 DEBUG [load-0] MyReader (SequenceFileLongMyVertexInputFormat.java:getCurrentVertex(108)) - Create Vertex from "1.0 0.0 0:" to : Vertex(id=1,value=1.0,#edges=0) 2014-09-04 10:44:44,694 DEBUG [load-0] MyReader (SequenceFileLongMyVertexInputFormat.java:getCurrentVertex(108)) - Create Vertex from "2.0 0.0 2:1|6|" to : Vertex(id=2,value=2.0,#edges=2) 2014-09-04 10:44:44,695 DEBUG [load-0] MyReader (SequenceFileLongMyVertexInputFormat.java:getCurrentVertex(108)) - Create Vertex from "5.0 0.0 3:4|1|6|" to : Vertex(id=5,value=5.0,#edges=3)
2014-09-04 10:44:44,697 INFO [load-0] worker.InputSplitsCallable (InputSplitsCallable.java:call(235)) - call: Loaded 1 input splits in 0.12198963 secs, (v=3, e=5) 24.592255 vertices/sec, 40.98709 edges/sec
But alas, when trying to iterate over them during a compute session, I get the following error
ERROR : But when reading in PrintVertex (BasicComputation): 2014-09-04 10:44:44,701 ERROR [load-0] utils.LogStacktraceCallable (LogStacktraceCallable.java:call(57)) - Execution of callable failed java.lang.IllegalStateException: next: IOException at org.apache.giraph.utils.VertexIterator.next(VertexIterator.java:101) at org.apache.giraph.partition.BasicPartition.addPartitionVertices(BasicPartition.java:99) at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:110) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.doRequest(NettyWorkerClientRequestProcessor.java:482) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.flush(NettyWorkerClientRequestProcessor.java:428) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:241) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60) at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: ensureRemaining: Only 5 bytes remaining, trying to read 8 at org.apache.giraph.utils.UnsafeByteArrayInputStream.ensureRemaining(UnsafeByteArrayInputStream.java:114) at org.apache.giraph.utils.UnsafeByteArrayInputStream.readLong(UnsafeByteArrayInputStream.java:197) at org.apache.hadoop.io.LongWritable.readFields(LongWritable.java:47) at org.apache.giraph.utils.WritableUtils.reinitializeVertexFromDataInput(WritableUtils.java:522) at org.apache.giraph.utils.VertexIterator.next(VertexIterator.java:98) ... 11 more
Any idea ?
Thanks a lot
Olivier
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.