You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Michal Haris <mi...@visualdna.com> on 2015/02/11 15:43:15 UTC

problem with removeEdges mutation

Hi all, I have managed to build a production giraph application off the
hadoop2 profile (but only mapreduce as yarn has few unresolved bugs). I
think I have now come across a bug which prevents me from using mutations
to do  de-duplicate edges in the first superstep of the computation.

Basically whether I use requestRemoveEdges(..) or vertex.removeEdges() the
program falls over with this exception:

java.lang.IllegalStateException: next: Failed on pos 16 edge
(targetVertexId = , value = dataxu)

at org.apache.giraph.edge.ByteArrayEdges$ByteArrayEdgeIterator.next(
ByteArrayEdges.java:170)at org.apache.giraph.edge.ByteArrayEdges.remove(
ByteArrayEdges.java:111)at
org.apache.giraph.graph.DefaultVertex.removeEdges(DefaultVertex.java:232)at
net.imagini.giraph.EdgeDeduplicationJob.compute(EdgeDeduplicationJob.java:34)at
org.apache.giraph.graph.ComputeCallable.computePartition(
ComputeCallable.java:247)at org.apache.giraph.graph.ComputeCallable.call(
ComputeCallable.java:168)at org.apache.giraph.graph.ComputeCallable.call(
ComputeCallable.java:1)at
org.apache.giraph.utils.LogStacktraceCallable.call(
LogStacktraceCallable.java:51)at java.util.concurrent.FutureTask.run(
FutureTask.java:262)at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)at
java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:744)

I have been using my own class for VertexID but even after reverting to
simple Text to see if problem is with my writable implementation I still
get what seems a corrupt state of the serialised byte array  edges after
requesting removal.

There is no mention of this problem in jira and googling yields nothing.
Anybody tried using removal of edges as part of superstep mutation ?
Anybody seen something similar?

I am seeing the problem with trunk or release-1.1, haven't tried older.



-- 
Michal Haris
Technical Architect
direct line: +44 (0) 207 749 0229
www.visualdna.com | t: +44 (0) 207 734 7033,

Re: problem with removeEdges mutation

Posted by José Luis Larroque <la...@gmail.com>.
Hi Michal, i'm facing some strange things with the state of
serializedEdgesByteUsed (i send an email explaining this issue a few days
ago), and maybe both problems are related. Could you find any solution for
this problem?

Thanks in advance!
Jose

2015-02-11 11:43 GMT-03:00 Michal Haris <mi...@visualdna.com>:

> Hi all, I have managed to build a production giraph application off the
> hadoop2 profile (but only mapreduce as yarn has few unresolved bugs). I
> think I have now come across a bug which prevents me from using mutations
> to do  de-duplicate edges in the first superstep of the computation.
>
> Basically whether I use requestRemoveEdges(..) or vertex.removeEdges() the
> program falls over with this exception:
>
> java.lang.IllegalStateException: next: Failed on pos 16 edge
> (targetVertexId = , value = dataxu)
>
> at org.apache.giraph.edge.ByteArrayEdges$ByteArrayEdgeIterator.next(
> ByteArrayEdges.java:170)at org.apache.giraph.edge.ByteArrayEdges.remove(
> ByteArrayEdges.java:111)at
> org.apache.giraph.graph.DefaultVertex.removeEdges(DefaultVertex.java:232)at
> net.imagini.giraph.EdgeDeduplicationJob.compute(
> EdgeDeduplicationJob.java:34)at
> org.apache.giraph.graph.ComputeCallable.computePartition(
> ComputeCallable.java:247)at org.apache.giraph.graph.ComputeCallable.call(
> ComputeCallable.java:168)at org.apache.giraph.graph.ComputeCallable.call(
> ComputeCallable.java:1)at
> org.apache.giraph.utils.LogStacktraceCallable.call(
> LogStacktraceCallable.java:51)at java.util.concurrent.FutureTask.run(
> FutureTask.java:262)at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:744)
>
> I have been using my own class for VertexID but even after reverting to
> simple Text to see if problem is with my writable implementation I still
> get what seems a corrupt state of the serialised byte array  edges after
> requesting removal.
>
> There is no mention of this problem in jira and googling yields nothing.
> Anybody tried using removal of edges as part of superstep mutation ?
> Anybody seen something similar?
>
> I am seeing the problem with trunk or release-1.1, haven't tried older.
>
>
>
> --
> Michal Haris
> Technical Architect
> direct line: +44 (0) 207 749 0229
> www.visualdna.com | t: +44 (0) 207 734 7033,
>