You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by Tommaso Teofili <to...@gmail.com> on 2013/07/17 09:44:35 UTC
Issue while using DiskVerticesInfo
Hi all,
I was trying to run the TestSubmitGraphJob with DiskVerticesInfo and I got
this :
13/07/17 09:21:45 INFO graph.GraphJobRunner: 7 vertices are loaded into
192.168.1.4:61001
13/07/17 09:21:45 ERROR bsp.BSPTask: Error running bsp setup and bsp
function.
java.lang.IllegalArgumentException: Messages must never be behind the
vertex in ID! Current Message ID: facebook.com vs. stackoverflow.com
at org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:281)
at org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:229)
at org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:133)
at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:177)
at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1262)
13/07/17 09:21:45 INFO server.PrepRequestProcessor: Processed session
termination for sessionid: 0x13feb81547f0003
13/07/17 09:21:45 INFO server.NIOServerCnxn: Closed socket connection for
client /0:0:0:0:0:0:0:1%0:51900 which had sessionid 0x13feb81547f0003
Does anyone know what could be the root cause of such a failure?
Thanks a lot in advance,
Tommaso
Re: Issue while using DiskVerticesInfo
Posted by Tommaso Teofili <to...@gmail.com>.
thanks a lot Suraj, that'd be great.
Tommaso
2013/7/18 Suraj Menon <su...@apache.org>
> I can take a look at it this weekend.
>
> -Suraj
>
>
> On Wed, Jul 17, 2013 at 6:06 AM, Tommaso Teofili
> <to...@gmail.com>wrote:
>
> > Yes, I assumed the DiskVerticesInfo implementation was sorting vertices
> to
> > accomplish that so I wonder if we have a bug there (or, for example, if
> > there's something else to configure).
> >
> > Maybe @Suraj could help?
> >
> > Thanks in advance,
> > Tommaso
> >
> > 2013/7/17 Edward J. Yoon <ed...@apache.org>
> >
> > > Hi,
> > >
> > > Our graph package is a pregel-like vertex-centric programming model,
> > > and it allows to communication between "vertices".
> > >
> > > Internally, each BSP processor performs computations for all assigned
> > > vertices.
> > >
> > > /**
> > > * The user-defined function
> > > */
> > > public void compute(Iterable<M> messages) throws IOException;
> > >
> > > To avoid grouping messages in received queue by vertex ID, we uses
> > > Sorted Message Queue, and calls user-defined function for each vertex
> > > sequentially. By using this sequential processing approach, we reduce
> > > the memory usage.
> > >
> > > The current problem is vertex loading phase (or partitioner). The
> > > loaded vertices in memory of each BSP processor should already be
> > > sorted by vertex ID. In ListVerticesInfo case,
> > >
> > > @Override
> > > public void finishAdditions() {
> > > Collections.sort(vertices);
> > > }
> > >
> > > This is quick-fix solution. We have to sort the vertices by vertex ID
> > > at partitioning or loading phase.
> > >
> > > On Wed, Jul 17, 2013 at 4:44 PM, Tommaso Teofili
> > > <to...@gmail.com> wrote:
> > > > Hi all,
> > > >
> > > > I was trying to run the TestSubmitGraphJob with DiskVerticesInfo and
> I
> > > got
> > > > this :
> > > >
> > > > 13/07/17 09:21:45 INFO graph.GraphJobRunner: 7 vertices are loaded
> into
> > > > 192.168.1.4:61001
> > > >
> > > > 13/07/17 09:21:45 ERROR bsp.BSPTask: Error running bsp setup and bsp
> > > > function.
> > > > java.lang.IllegalArgumentException: Messages must never be behind the
> > > > vertex in ID! Current Message ID: facebook.com vs. stackoverflow.com
> > > > at
> > org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:281)
> > > > at
> > >
> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:229)
> > > > at org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:133)
> > > > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:177)
> > > > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> > > > at
> > >
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1262)
> > > >
> > > > 13/07/17 09:21:45 INFO server.PrepRequestProcessor: Processed session
> > > > termination for sessionid: 0x13feb81547f0003
> > > >
> > > > 13/07/17 09:21:45 INFO server.NIOServerCnxn: Closed socket connection
> > for
> > > > client /0:0:0:0:0:0:0:1%0:51900 which had sessionid 0x13feb81547f0003
> > > >
> > > >
> > > > Does anyone know what could be the root cause of such a failure?
> > > >
> > > > Thanks a lot in advance,
> > > > Tommaso
> > >
> > >
> > >
> > > --
> > > Best Regards, Edward J. Yoon
> > > @eddieyoon
> > >
> >
>
Re: Issue while using DiskVerticesInfo
Posted by Suraj Menon <su...@apache.org>.
I can take a look at it this weekend.
-Suraj
On Wed, Jul 17, 2013 at 6:06 AM, Tommaso Teofili
<to...@gmail.com>wrote:
> Yes, I assumed the DiskVerticesInfo implementation was sorting vertices to
> accomplish that so I wonder if we have a bug there (or, for example, if
> there's something else to configure).
>
> Maybe @Suraj could help?
>
> Thanks in advance,
> Tommaso
>
> 2013/7/17 Edward J. Yoon <ed...@apache.org>
>
> > Hi,
> >
> > Our graph package is a pregel-like vertex-centric programming model,
> > and it allows to communication between "vertices".
> >
> > Internally, each BSP processor performs computations for all assigned
> > vertices.
> >
> > /**
> > * The user-defined function
> > */
> > public void compute(Iterable<M> messages) throws IOException;
> >
> > To avoid grouping messages in received queue by vertex ID, we uses
> > Sorted Message Queue, and calls user-defined function for each vertex
> > sequentially. By using this sequential processing approach, we reduce
> > the memory usage.
> >
> > The current problem is vertex loading phase (or partitioner). The
> > loaded vertices in memory of each BSP processor should already be
> > sorted by vertex ID. In ListVerticesInfo case,
> >
> > @Override
> > public void finishAdditions() {
> > Collections.sort(vertices);
> > }
> >
> > This is quick-fix solution. We have to sort the vertices by vertex ID
> > at partitioning or loading phase.
> >
> > On Wed, Jul 17, 2013 at 4:44 PM, Tommaso Teofili
> > <to...@gmail.com> wrote:
> > > Hi all,
> > >
> > > I was trying to run the TestSubmitGraphJob with DiskVerticesInfo and I
> > got
> > > this :
> > >
> > > 13/07/17 09:21:45 INFO graph.GraphJobRunner: 7 vertices are loaded into
> > > 192.168.1.4:61001
> > >
> > > 13/07/17 09:21:45 ERROR bsp.BSPTask: Error running bsp setup and bsp
> > > function.
> > > java.lang.IllegalArgumentException: Messages must never be behind the
> > > vertex in ID! Current Message ID: facebook.com vs. stackoverflow.com
> > > at
> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:281)
> > > at
> > org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:229)
> > > at org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:133)
> > > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:177)
> > > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> > > at
> > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1262)
> > >
> > > 13/07/17 09:21:45 INFO server.PrepRequestProcessor: Processed session
> > > termination for sessionid: 0x13feb81547f0003
> > >
> > > 13/07/17 09:21:45 INFO server.NIOServerCnxn: Closed socket connection
> for
> > > client /0:0:0:0:0:0:0:1%0:51900 which had sessionid 0x13feb81547f0003
> > >
> > >
> > > Does anyone know what could be the root cause of such a failure?
> > >
> > > Thanks a lot in advance,
> > > Tommaso
> >
> >
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
>
Re: Issue while using DiskVerticesInfo
Posted by Tommaso Teofili <to...@gmail.com>.
Yes, I assumed the DiskVerticesInfo implementation was sorting vertices to
accomplish that so I wonder if we have a bug there (or, for example, if
there's something else to configure).
Maybe @Suraj could help?
Thanks in advance,
Tommaso
2013/7/17 Edward J. Yoon <ed...@apache.org>
> Hi,
>
> Our graph package is a pregel-like vertex-centric programming model,
> and it allows to communication between "vertices".
>
> Internally, each BSP processor performs computations for all assigned
> vertices.
>
> /**
> * The user-defined function
> */
> public void compute(Iterable<M> messages) throws IOException;
>
> To avoid grouping messages in received queue by vertex ID, we uses
> Sorted Message Queue, and calls user-defined function for each vertex
> sequentially. By using this sequential processing approach, we reduce
> the memory usage.
>
> The current problem is vertex loading phase (or partitioner). The
> loaded vertices in memory of each BSP processor should already be
> sorted by vertex ID. In ListVerticesInfo case,
>
> @Override
> public void finishAdditions() {
> Collections.sort(vertices);
> }
>
> This is quick-fix solution. We have to sort the vertices by vertex ID
> at partitioning or loading phase.
>
> On Wed, Jul 17, 2013 at 4:44 PM, Tommaso Teofili
> <to...@gmail.com> wrote:
> > Hi all,
> >
> > I was trying to run the TestSubmitGraphJob with DiskVerticesInfo and I
> got
> > this :
> >
> > 13/07/17 09:21:45 INFO graph.GraphJobRunner: 7 vertices are loaded into
> > 192.168.1.4:61001
> >
> > 13/07/17 09:21:45 ERROR bsp.BSPTask: Error running bsp setup and bsp
> > function.
> > java.lang.IllegalArgumentException: Messages must never be behind the
> > vertex in ID! Current Message ID: facebook.com vs. stackoverflow.com
> > at org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:281)
> > at
> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:229)
> > at org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:133)
> > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:177)
> > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> > at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1262)
> >
> > 13/07/17 09:21:45 INFO server.PrepRequestProcessor: Processed session
> > termination for sessionid: 0x13feb81547f0003
> >
> > 13/07/17 09:21:45 INFO server.NIOServerCnxn: Closed socket connection for
> > client /0:0:0:0:0:0:0:1%0:51900 which had sessionid 0x13feb81547f0003
> >
> >
> > Does anyone know what could be the root cause of such a failure?
> >
> > Thanks a lot in advance,
> > Tommaso
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
Re: Issue while using DiskVerticesInfo
Posted by "Edward J. Yoon" <ed...@apache.org>.
Hi,
Our graph package is a pregel-like vertex-centric programming model,
and it allows to communication between "vertices".
Internally, each BSP processor performs computations for all assigned vertices.
/**
* The user-defined function
*/
public void compute(Iterable<M> messages) throws IOException;
To avoid grouping messages in received queue by vertex ID, we uses
Sorted Message Queue, and calls user-defined function for each vertex
sequentially. By using this sequential processing approach, we reduce
the memory usage.
The current problem is vertex loading phase (or partitioner). The
loaded vertices in memory of each BSP processor should already be
sorted by vertex ID. In ListVerticesInfo case,
@Override
public void finishAdditions() {
Collections.sort(vertices);
}
This is quick-fix solution. We have to sort the vertices by vertex ID
at partitioning or loading phase.
On Wed, Jul 17, 2013 at 4:44 PM, Tommaso Teofili
<to...@gmail.com> wrote:
> Hi all,
>
> I was trying to run the TestSubmitGraphJob with DiskVerticesInfo and I got
> this :
>
> 13/07/17 09:21:45 INFO graph.GraphJobRunner: 7 vertices are loaded into
> 192.168.1.4:61001
>
> 13/07/17 09:21:45 ERROR bsp.BSPTask: Error running bsp setup and bsp
> function.
> java.lang.IllegalArgumentException: Messages must never be behind the
> vertex in ID! Current Message ID: facebook.com vs. stackoverflow.com
> at org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:281)
> at org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:229)
> at org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:133)
> at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:177)
> at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1262)
>
> 13/07/17 09:21:45 INFO server.PrepRequestProcessor: Processed session
> termination for sessionid: 0x13feb81547f0003
>
> 13/07/17 09:21:45 INFO server.NIOServerCnxn: Closed socket connection for
> client /0:0:0:0:0:0:0:1%0:51900 which had sessionid 0x13feb81547f0003
>
>
> Does anyone know what could be the root cause of such a failure?
>
> Thanks a lot in advance,
> Tommaso
--
Best Regards, Edward J. Yoon
@eddieyoon