You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Nick West <ni...@benchmarksolutions.com> on 2012/07/26 00:29:51 UTC
Heterogeneous Vertex Types
Hi,
I'm working on implementing a belief propagation algorithm over Giraph. (Do you know if anyone has done this before?) This requires having (at least) two different types of vertices implemented (values and factors) and different types of messages sent between different vertices.
I've been able to set up and run my own vertices (with my own custom readers and writers), however, whenever I try to extend this to a more complex case I run into problems. For example, suppose aI have the following two vertex types:
class FooVertex() extends BasicVertex[IntWritable, IntWritable, Text, IntWritable]
class BarVertex() extends BasicVertex[IntWritable, IntWritable, Text, IntWritable]
(both of which run fine in the basic set up), and I then configure the GiraphJob with the following,
job.setVertexClass(classOf[BasicVertex[IntWritable, IntWritable, Text, IntWritable]])
(as opposed to with FooVertex or BarVertex).
I get a null pointer error in GraphMapper (at line 204 in the version of the code that I am using - just after the log statement about the input format vertex index type not being known, and at the line where it checks if the vertextIndexTypes are the same).
Do you know what is causing this? (The stack trace is below.) Have you tried to instantiate different classes for different vertices in the past?
One last piece of info that might explain things: I'm developing in scala, which I have seen behave a little bit poorly with java reflection - could that be a cause?
Thanks,
Nick West
Stack Trace:
./bin/hadoop jar ~/code/watchdog/modules/inference/target/inference-assembly-1.0.jar bmrk.inference.giraph.IdiotRunner ising ising_out 4
Input arguments:
[0] = ising
[1] = ising_out
[2] = 4
setting vertex class to: org.apache.giraph.graph.BasicVertex
setting vertex input class to: bmrk.inference.giraph.IdiotVertexInputFormat
setting vertex output class to: bmrk.inference.giraph.IdiotVertexOutputFormat
12/07/25 17:48:54 INFO mapred.JobClient: Running job: job_201207250932_0012
12/07/25 17:48:55 INFO mapred.JobClient: map 0% reduce 0%
12/07/25 17:52:43 INFO mapred.JobClient: Task Id : attempt_201207250932_0012_m_000000_0, Status : FAILED
java.lang.NullPointerException
at org.apache.giraph.graph.GraphMapper.determineClassTypes(GraphMapper.java:204)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:392)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:655)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Benchmark Solutions
101 Park Avenue - 7th Floor
New York, NY 10178
Tel +1.212.220.4739 | Mobile +1.646.267.4324
www.benchmarksolutions.com <http://www.benchmarksolutions.com/>
[cid:image001.png@01CCA50E.43B4A860]
Re: Heterogeneous Vertex Types
Posted by Etienne Dumoulin <et...@idiro.com>.
Hi,
I am not sure to say something stupid or not but if you have two kind of
vertexes,
meaning is there are different in there values... I would looking at doing
something
this way. Have a Writable type which offer options and different
functionalities.
Accessing it from a getVertexValue(). You can load correctly what you need
in the read methods of Writable...
Regards,
Étienne
On 27 July 2012 14:32, Nick West <ni...@benchmarksolutions.com> wrote:
> Thanks for the quick reply.
>
> That (using one uber-class) was my initial thought when I ran into this
> problem, however I was wondering if there was a solution that would use the
> type hierarchy (which admittedly is a bit more flexible in scala than java,
> so this may not be possible). Is there something in the Giraph
> architecture that precludes this?
>
> Thanks,
> Nick
>
> On Jul 25, 2012, at 7:11 PM, Jonathan Bishop wrote:
>
> Nick,
>
> You may want to reconsider your approach as I don't think Giraph will be
> happy with this.
>
> How about using a new vertex value class with BasicVertex which can do
> both behaviours for you? Same for edge and message classes. This should be
> enough do what you want.
>
> Jon
>
> On Wed, Jul 25, 2012 at 3:29 PM, Nick West <
> nick.west@benchmarksolutions.com> wrote:
>
>> Hi,
>>
>> I'm working on implementing a belief propagation algorithm over Giraph.
>> (Do you know if anyone has done this before?) This requires having (at
>> least) two different types of vertices implemented (values and factors) and
>> different types of messages sent between different vertices.
>>
>> I've been able to set up and run my own vertices (with my own custom
>> readers and writers), however, whenever I try to extend this to a more
>> complex case I run into problems. For example, suppose aI have the
>> following two vertex types:
>>
>> class FooVertex() extends BasicVertex[IntWritable, IntWritable, Text,
>> IntWritable]
>> class BarVertex() extends BasicVertex[IntWritable, IntWritable, Text,
>> IntWritable]
>>
>> (both of which run fine in the basic set up), and I then configure the
>> GiraphJob with the following,
>>
>> job.setVertexClass(classOf[BasicVertex[IntWritable, IntWritable, Text,
>> IntWritable]])
>>
>> (as opposed to with FooVertex or BarVertex).
>>
>> I get a null pointer error in GraphMapper (at line 204 in the version
>> of the code that I am using - just after the log statement about the input
>> format vertex index type not being known, and at the line where it checks
>> if the vertextIndexTypes are the same).
>>
>> Do you know what is causing this? (The stack trace is below.) Have you
>> tried to instantiate different classes for different vertices in the past?
>>
>> One last piece of info that might explain things: I'm developing in
>> scala, which I have seen behave a little bit poorly with java reflection -
>> could that be a cause?
>>
>> Thanks,
>> *Nick West
>> **
>> *
>> Stack Trace:
>>
>> ./bin/hadoop jar
>> ~/code/watchdog/modules/inference/target/inference-assembly-1.0.jar
>> bmrk.inference.giraph.IdiotRunner ising ising_out 4
>> Input arguments:
>> [0] = ising
>> [1] = ising_out
>> [2] = 4
>> setting vertex class to: org.apache.giraph.graph.BasicVertex
>> setting vertex input class to:
>> bmrk.inference.giraph.IdiotVertexInputFormat
>> setting vertex output class to:
>> bmrk.inference.giraph.IdiotVertexOutputFormat
>> 12/07/25 17:48:54 INFO mapred.JobClient: Running job:
>> job_201207250932_0012
>> 12/07/25 17:48:55 INFO mapred.JobClient: map 0% reduce 0%
>> 12/07/25 17:52:43 INFO mapred.JobClient: Task Id :
>> attempt_201207250932_0012_m_000000_0, Status : FAILED
>> java.lang.NullPointerException
>> at
>> org.apache.giraph.graph.GraphMapper.determineClassTypes(GraphMapper.java:204)
>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:392)
>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:655)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>
>>
>> Benchmark Solutions
>> 101 Park Avenue - 7th Floor
>> New York, NY 10178
>> Tel +1.212.220.4739 | Mobile +1.646.267.4324
>> *www.benchmarksolutions.com * <http://www.benchmarksolutions.com/>
>> **
>> *<image001.png>
>>
>>
>>
>> *
>> **
>>
>
>
> *
> Nick West
> *
> *
> *Benchmark Solutions
> 101 Park Avenue - 7th Floor
> New York, NY 10178
> Tel +1.212.220.4739 | Mobile +1.646.267.4324
> *www.benchmarksolutions.com * <http://www.benchmarksolutions.com/>
> **
> *
>
>
>
> *
> **
>
Re: Heterogeneous Vertex Types
Posted by Nick West <ni...@benchmarksolutions.com>.
Joh, Etienne and Avery,
Thank you all for your replies. I have settled on using a container class with different functionalities implemented with in it, and one message type that wraps all the different types of messages.
Avery and Jon: I ended up with the same conclusion that a parametrized type for a vertex would be difficult for the de/serialization process.
Thanks for all the feedback.
-Nick
On Jul 28, 2012, at 3:43 AM, Avery Ching wrote:
Hi Nick,
Giraph needs to have be able to instantiate the vertex id, value, edges, and messages and there can only be a single type for each of them.
That being said, since all the types are user chosen, you can basically implement types that support multiple types inside of it. As a simple, stupid example
class MessageA {
...
}
class MessageB {
...
}
class MultiMessage implements Writable {
private MessageA msgA;
private MessageB msgB;
private boolean useMsgA;
pubic MessageA getMsgA(){
return msgA;
}
pubic MessageB getMsgB(){
return msgB;
}
...
}
I didn't try and compile it, but hopefully you get the idea. =)
On 7/27/12 9:52 AM, Jonathan Bishop wrote:
Nick,
I am guessing it is because Giraph needs to move vertices around and needs to construct them before it reads in their serialization. I am new to Giraph myself so this may be incorrect. Maybe Avery could comment.
Jon
On Fri, Jul 27, 2012 at 6:32 AM, Nick West <ni...@benchmarksolutions.com>> wrote:
Thanks for the quick reply.
That (using one uber-class) was my initial thought when I ran into this problem, however I was wondering if there was a solution that would use the type hierarchy (which admittedly is a bit more flexible in scala than java, so this may not be possible). Is there something in the Giraph architecture that precludes this?
Thanks,
Nick
On Jul 25, 2012, at 7:11 PM, Jonathan Bishop wrote:
Nick,
You may want to reconsider your approach as I don't think Giraph will be happy with this.
How about using a new vertex value class with BasicVertex which can do both behaviours for you? Same for edge and message classes. This should be enough do what you want.
Jon
On Wed, Jul 25, 2012 at 3:29 PM, Nick West <ni...@benchmarksolutions.com>> wrote:
Hi,
I'm working on implementing a belief propagation algorithm over Giraph. (Do you know if anyone has done this before?) This requires having (at least) two different types of vertices implemented (values and factors) and different types of messages sent between different vertices.
I've been able to set up and run my own vertices (with my own custom readers and writers), however, whenever I try to extend this to a more complex case I run into problems. For example, suppose aI have the following two vertex types:
class FooVertex() extends BasicVertex[IntWritable, IntWritable, Text, IntWritable]
class BarVertex() extends BasicVertex[IntWritable, IntWritable, Text, IntWritable]
(both of which run fine in the basic set up), and I then configure the GiraphJob with the following,
job.setVertexClass(classOf[BasicVertex[IntWritable, IntWritable, Text, IntWritable]])
(as opposed to with FooVertex or BarVertex).
I get a null pointer error in GraphMapper (at line 204 in the version of the code that I am using - just after the log statement about the input format vertex index type not being known, and at the line where it checks if the vertextIndexTypes are the same).
Do you know what is causing this? (The stack trace is below.) Have you tried to instantiate different classes for different vertices in the past?
One last piece of info that might explain things: I'm developing in scala, which I have seen behave a little bit poorly with java reflection - could that be a cause?
Thanks,
Nick West
Stack Trace:
./bin/hadoop jar ~/code/watchdog/modules/inference/target/inference-assembly-1.0.jar bmrk.inference.giraph.IdiotRunner ising ising_out 4
Input arguments:
[0] = ising
[1] = ising_out
[2] = 4
setting vertex class to: org.apache.giraph.graph.BasicVertex
setting vertex input class to: bmrk.inference.giraph.IdiotVertexInputFormat
setting vertex output class to: bmrk.inference.giraph.IdiotVertexOutputFormat
12/07/25 17:48:54 INFO mapred.JobClient: Running job: job_201207250932_0012
12/07/25 17:48:55 INFO mapred.JobClient: map 0% reduce 0%
12/07/25 17:52:43 INFO mapred.JobClient: Task Id : attempt_201207250932_0012_m_000000_0, Status : FAILED
java.lang.NullPointerException
at org.apache.giraph.graph.GraphMapper.determineClassTypes(GraphMapper.java:204)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:392)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:655)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Benchmark Solutions
101 Park Avenue - 7th Floor
New York, NY 10178
Tel +1.212.220.4739<tel:%2B1.212.220.4739> | Mobile +1.646.267.4324<tel:%2B1.646.267.4324>
www.benchmarksolutions.com <http://www.benchmarksolutions.com/>
<image001.png>
Nick West
Benchmark Solutions
101 Park Avenue - 7th Floor
New York, NY 10178
Tel +1.212.220.4739<tel:%2B1.212.220.4739> | Mobile +1.646.267.4324<tel:%2B1.646.267.4324>
www.benchmarksolutions.com <http://www.benchmarksolutions.com/>
<Mail Attachment.png>
Nick West
Benchmark Solutions
101 Park Avenue - 7th Floor
New York, NY 10178
Tel +1.212.220.4739 | Mobile +1.646.267.4324
www.benchmarksolutions.com <http://www.benchmarksolutions.com/>
[cid:image001.png@01CCA50E.43B4A860]
Re: Heterogeneous Vertex Types
Posted by Avery Ching <ac...@apache.org>.
Hi Nick,
Giraph needs to have be able to instantiate the vertex id, value, edges,
and messages and there can only be a single type for each of them.
That being said, since all the types are user chosen, you can basically
implement types that support multiple types inside of it. As a simple,
stupid example
class MessageA {
...
}
class MessageB {
...
}
class MultiMessage implements Writable {
private MessageA msgA;
private MessageB msgB;
private boolean useMsgA;
pubic MessageA getMsgA(){
return msgA;
}
pubic MessageB getMsgB(){
return msgB;
}
...
}
I didn't try and compile it, but hopefully you get the idea. =)
On 7/27/12 9:52 AM, Jonathan Bishop wrote:
> Nick,
>
> I am guessing it is because Giraph needs to move vertices around and
> needs to construct them before it reads in their serialization. I am
> new to Giraph myself so this may be incorrect. Maybe Avery could comment.
>
> Jon
>
> On Fri, Jul 27, 2012 at 6:32 AM, Nick West
> <nick.west@benchmarksolutions.com
> <ma...@benchmarksolutions.com>> wrote:
>
> Thanks for the quick reply.
>
> That (using one uber-class) was my initial thought when I ran into
> this problem, however I was wondering if there was a solution that
> would use the type hierarchy (which admittedly is a bit more
> flexible in scala than java, so this may not be possible). Is
> there something in the Giraph architecture that precludes this?
>
> Thanks,
> Nick
>
> On Jul 25, 2012, at 7:11 PM, Jonathan Bishop wrote:
>
>> Nick,
>>
>> You may want to reconsider your approach as I don't think Giraph
>> will be happy with this.
>>
>> How about using a new vertex value class with BasicVertex which
>> can do both behaviours for you? Same for edge and message
>> classes. This should be enough do what you want.
>>
>> Jon
>>
>> On Wed, Jul 25, 2012 at 3:29 PM, Nick West
>> <nick.west@benchmarksolutions.com
>> <ma...@benchmarksolutions.com>> wrote:
>>
>> Hi,
>>
>> I'm working on implementing a belief propagation algorithm
>> over Giraph. (Do you know if anyone has done this before?)
>> This requires having (at least) two different types of
>> vertices implemented (values and factors) and different types
>> of messages sent between different vertices.
>>
>> I've been able to set up and run my own vertices (with my own
>> custom readers and writers), however, whenever I try to
>> extend this to a more complex case I run into problems. For
>> example, suppose aI have the following two vertex types:
>>
>> class FooVertex() extends BasicVertex[IntWritable,
>> IntWritable, Text, IntWritable]
>> class BarVertex() extends BasicVertex[IntWritable,
>> IntWritable, Text, IntWritable]
>>
>> (both of which run fine in the basic set up), and I then
>> configure the GiraphJob with the following,
>>
>> job.setVertexClass(classOf[BasicVertex[IntWritable,
>> IntWritable, Text, IntWritable]])
>>
>> (as opposed to with FooVertex or BarVertex).
>>
>> I get a null pointer error in GraphMapper (at line 204 in the
>> version of the code that I am using - just after the log
>> statement about the input format vertex index type not being
>> known, and at the line where it checks if the
>> vertextIndexTypes are the same).
>>
>> Do you know what is causing this? (The stack trace is below.)
>> Have you tried to instantiate different classes for
>> different vertices in the past?
>>
>> One last piece of info that might explain things: I'm
>> developing in scala, which I have seen behave a little bit
>> poorly with java reflection - could that be a cause?
>>
>> Thanks,
>> *Nick West
>> **
>> *
>> Stack Trace:
>>
>> ./bin/hadoop jar
>> ~/code/watchdog/modules/inference/target/inference-assembly-1.0.jar
>> bmrk.inference.giraph.IdiotRunner ising ising_out 4
>> Input arguments:
>> [0] = ising
>> [1] = ising_out
>> [2] = 4
>> setting vertex class to: org.apache.giraph.graph.BasicVertex
>> setting vertex input class to:
>> bmrk.inference.giraph.IdiotVertexInputFormat
>> setting vertex output class to:
>> bmrk.inference.giraph.IdiotVertexOutputFormat
>> 12/07/25 17:48:54 INFO mapred.JobClient: Running job:
>> job_201207250932_0012
>> 12/07/25 17:48:55 INFO mapred.JobClient: map 0% reduce 0%
>> 12/07/25 17:52:43 INFO mapred.JobClient: Task Id :
>> attempt_201207250932_0012_m_000000_0, Status : FAILED
>> java.lang.NullPointerException
>> at
>> org.apache.giraph.graph.GraphMapper.determineClassTypes(GraphMapper.java:204)
>> at
>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:392)
>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:655)
>> at
>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>
>>
>> Benchmark Solutions
>> 101 Park Avenue - 7th Floor
>> New York, NY 10178
>> Tel +1.212.220.4739 <tel:%2B1.212.220.4739> | Mobile
>> +1.646.267.4324 <tel:%2B1.646.267.4324>
>> *www.benchmarksolutions.com *
>> <http://www.benchmarksolutions.com/>
>> *<image001.png>
>>
>>
>> *
>>
>>
>
> *
> Nick West
> *
> *
> *Benchmark Solutions
> 101 Park Avenue - 7th Floor
> New York, NY 10178
> Tel +1.212.220.4739 <tel:%2B1.212.220.4739> | Mobile
> +1.646.267.4324 <tel:%2B1.646.267.4324>
> *www.benchmarksolutions.com * <http://www.benchmarksolutions.com/>
> *
>
>
> *
>
>
Re: Heterogeneous Vertex Types
Posted by Jonathan Bishop <jb...@gmail.com>.
Nick,
I am guessing it is because Giraph needs to move vertices around and needs
to construct them before it reads in their serialization. I am new to
Giraph myself so this may be incorrect. Maybe Avery could comment.
Jon
On Fri, Jul 27, 2012 at 6:32 AM, Nick West <nick.west@benchmarksolutions.com
> wrote:
> Thanks for the quick reply.
>
> That (using one uber-class) was my initial thought when I ran into this
> problem, however I was wondering if there was a solution that would use the
> type hierarchy (which admittedly is a bit more flexible in scala than java,
> so this may not be possible). Is there something in the Giraph
> architecture that precludes this?
>
> Thanks,
> Nick
>
> On Jul 25, 2012, at 7:11 PM, Jonathan Bishop wrote:
>
> Nick,
>
> You may want to reconsider your approach as I don't think Giraph will be
> happy with this.
>
> How about using a new vertex value class with BasicVertex which can do
> both behaviours for you? Same for edge and message classes. This should be
> enough do what you want.
>
> Jon
>
> On Wed, Jul 25, 2012 at 3:29 PM, Nick West <
> nick.west@benchmarksolutions.com> wrote:
>
>> Hi,
>>
>> I'm working on implementing a belief propagation algorithm over Giraph.
>> (Do you know if anyone has done this before?) This requires having (at
>> least) two different types of vertices implemented (values and factors) and
>> different types of messages sent between different vertices.
>>
>> I've been able to set up and run my own vertices (with my own custom
>> readers and writers), however, whenever I try to extend this to a more
>> complex case I run into problems. For example, suppose aI have the
>> following two vertex types:
>>
>> class FooVertex() extends BasicVertex[IntWritable, IntWritable, Text,
>> IntWritable]
>> class BarVertex() extends BasicVertex[IntWritable, IntWritable, Text,
>> IntWritable]
>>
>> (both of which run fine in the basic set up), and I then configure the
>> GiraphJob with the following,
>>
>> job.setVertexClass(classOf[BasicVertex[IntWritable, IntWritable, Text,
>> IntWritable]])
>>
>> (as opposed to with FooVertex or BarVertex).
>>
>> I get a null pointer error in GraphMapper (at line 204 in the version
>> of the code that I am using - just after the log statement about the input
>> format vertex index type not being known, and at the line where it checks
>> if the vertextIndexTypes are the same).
>>
>> Do you know what is causing this? (The stack trace is below.) Have you
>> tried to instantiate different classes for different vertices in the past?
>>
>> One last piece of info that might explain things: I'm developing in
>> scala, which I have seen behave a little bit poorly with java reflection -
>> could that be a cause?
>>
>> Thanks,
>> *Nick West
>> **
>> *
>> Stack Trace:
>>
>> ./bin/hadoop jar
>> ~/code/watchdog/modules/inference/target/inference-assembly-1.0.jar
>> bmrk.inference.giraph.IdiotRunner ising ising_out 4
>> Input arguments:
>> [0] = ising
>> [1] = ising_out
>> [2] = 4
>> setting vertex class to: org.apache.giraph.graph.BasicVertex
>> setting vertex input class to:
>> bmrk.inference.giraph.IdiotVertexInputFormat
>> setting vertex output class to:
>> bmrk.inference.giraph.IdiotVertexOutputFormat
>> 12/07/25 17:48:54 INFO mapred.JobClient: Running job:
>> job_201207250932_0012
>> 12/07/25 17:48:55 INFO mapred.JobClient: map 0% reduce 0%
>> 12/07/25 17:52:43 INFO mapred.JobClient: Task Id :
>> attempt_201207250932_0012_m_000000_0, Status : FAILED
>> java.lang.NullPointerException
>> at
>> org.apache.giraph.graph.GraphMapper.determineClassTypes(GraphMapper.java:204)
>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:392)
>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:655)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>
>>
>> Benchmark Solutions
>> 101 Park Avenue - 7th Floor
>> New York, NY 10178
>> Tel +1.212.220.4739 | Mobile +1.646.267.4324
>> *www.benchmarksolutions.com * <http://www.benchmarksolutions.com/>
>> **
>> *<image001.png>
>>
>>
>>
>> *
>> **
>>
>
>
> *
> Nick West
> *
> *
> *Benchmark Solutions
> 101 Park Avenue - 7th Floor
> New York, NY 10178
> Tel +1.212.220.4739 | Mobile +1.646.267.4324
> *www.benchmarksolutions.com * <http://www.benchmarksolutions.com/>
> **
> *
>
>
>
> *
> **
>
Re: Heterogeneous Vertex Types
Posted by Nick West <ni...@benchmarksolutions.com>.
Thanks for the quick reply.
That (using one uber-class) was my initial thought when I ran into this problem, however I was wondering if there was a solution that would use the type hierarchy (which admittedly is a bit more flexible in scala than java, so this may not be possible). Is there something in the Giraph architecture that precludes this?
Thanks,
Nick
On Jul 25, 2012, at 7:11 PM, Jonathan Bishop wrote:
Nick,
You may want to reconsider your approach as I don't think Giraph will be happy with this.
How about using a new vertex value class with BasicVertex which can do both behaviours for you? Same for edge and message classes. This should be enough do what you want.
Jon
On Wed, Jul 25, 2012 at 3:29 PM, Nick West <ni...@benchmarksolutions.com>> wrote:
Hi,
I'm working on implementing a belief propagation algorithm over Giraph. (Do you know if anyone has done this before?) This requires having (at least) two different types of vertices implemented (values and factors) and different types of messages sent between different vertices.
I've been able to set up and run my own vertices (with my own custom readers and writers), however, whenever I try to extend this to a more complex case I run into problems. For example, suppose aI have the following two vertex types:
class FooVertex() extends BasicVertex[IntWritable, IntWritable, Text, IntWritable]
class BarVertex() extends BasicVertex[IntWritable, IntWritable, Text, IntWritable]
(both of which run fine in the basic set up), and I then configure the GiraphJob with the following,
job.setVertexClass(classOf[BasicVertex[IntWritable, IntWritable, Text, IntWritable]])
(as opposed to with FooVertex or BarVertex).
I get a null pointer error in GraphMapper (at line 204 in the version of the code that I am using - just after the log statement about the input format vertex index type not being known, and at the line where it checks if the vertextIndexTypes are the same).
Do you know what is causing this? (The stack trace is below.) Have you tried to instantiate different classes for different vertices in the past?
One last piece of info that might explain things: I'm developing in scala, which I have seen behave a little bit poorly with java reflection - could that be a cause?
Thanks,
Nick West
Stack Trace:
./bin/hadoop jar ~/code/watchdog/modules/inference/target/inference-assembly-1.0.jar bmrk.inference.giraph.IdiotRunner ising ising_out 4
Input arguments:
[0] = ising
[1] = ising_out
[2] = 4
setting vertex class to: org.apache.giraph.graph.BasicVertex
setting vertex input class to: bmrk.inference.giraph.IdiotVertexInputFormat
setting vertex output class to: bmrk.inference.giraph.IdiotVertexOutputFormat
12/07/25 17:48:54 INFO mapred.JobClient: Running job: job_201207250932_0012
12/07/25 17:48:55 INFO mapred.JobClient: map 0% reduce 0%
12/07/25 17:52:43 INFO mapred.JobClient: Task Id : attempt_201207250932_0012_m_000000_0, Status : FAILED
java.lang.NullPointerException
at org.apache.giraph.graph.GraphMapper.determineClassTypes(GraphMapper.java:204)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:392)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:655)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Benchmark Solutions
101 Park Avenue - 7th Floor
New York, NY 10178
Tel +1.212.220.4739<tel:%2B1.212.220.4739> | Mobile +1.646.267.4324<tel:%2B1.646.267.4324>
www.benchmarksolutions.com <http://www.benchmarksolutions.com/>
<image001.png>
Nick West
Benchmark Solutions
101 Park Avenue - 7th Floor
New York, NY 10178
Tel +1.212.220.4739 | Mobile +1.646.267.4324
www.benchmarksolutions.com <http://www.benchmarksolutions.com/>
[cid:image001.png@01CCA50E.43B4A860]
Re: Heterogeneous Vertex Types
Posted by Jonathan Bishop <jb...@gmail.com>.
Nick,
You may want to reconsider your approach as I don't think Giraph will be
happy with this.
How about using a new vertex value class with BasicVertex which can do both
behaviours for you? Same for edge and message classes. This should be
enough do what you want.
Jon
On Wed, Jul 25, 2012 at 3:29 PM, Nick West <nick.west@benchmarksolutions.com
> wrote:
> Hi,
>
> I'm working on implementing a belief propagation algorithm over Giraph.
> (Do you know if anyone has done this before?) This requires having (at
> least) two different types of vertices implemented (values and factors) and
> different types of messages sent between different vertices.
>
> I've been able to set up and run my own vertices (with my own custom
> readers and writers), however, whenever I try to extend this to a more
> complex case I run into problems. For example, suppose aI have the
> following two vertex types:
>
> class FooVertex() extends BasicVertex[IntWritable, IntWritable, Text,
> IntWritable]
> class BarVertex() extends BasicVertex[IntWritable, IntWritable, Text,
> IntWritable]
>
> (both of which run fine in the basic set up), and I then configure the
> GiraphJob with the following,
>
> job.setVertexClass(classOf[BasicVertex[IntWritable, IntWritable, Text,
> IntWritable]])
>
> (as opposed to with FooVertex or BarVertex).
>
> I get a null pointer error in GraphMapper (at line 204 in the version of
> the code that I am using - just after the log statement about the input
> format vertex index type not being known, and at the line where it checks
> if the vertextIndexTypes are the same).
>
> Do you know what is causing this? (The stack trace is below.) Have you
> tried to instantiate different classes for different vertices in the past?
>
> One last piece of info that might explain things: I'm developing in
> scala, which I have seen behave a little bit poorly with java reflection -
> could that be a cause?
>
> Thanks,
> *Nick West
> **
> *
> Stack Trace:
>
> ./bin/hadoop jar
> ~/code/watchdog/modules/inference/target/inference-assembly-1.0.jar
> bmrk.inference.giraph.IdiotRunner ising ising_out 4
> Input arguments:
> [0] = ising
> [1] = ising_out
> [2] = 4
> setting vertex class to: org.apache.giraph.graph.BasicVertex
> setting vertex input class to: bmrk.inference.giraph.IdiotVertexInputFormat
> setting vertex output class to:
> bmrk.inference.giraph.IdiotVertexOutputFormat
> 12/07/25 17:48:54 INFO mapred.JobClient: Running job: job_201207250932_0012
> 12/07/25 17:48:55 INFO mapred.JobClient: map 0% reduce 0%
> 12/07/25 17:52:43 INFO mapred.JobClient: Task Id :
> attempt_201207250932_0012_m_000000_0, Status : FAILED
> java.lang.NullPointerException
> at
> org.apache.giraph.graph.GraphMapper.determineClassTypes(GraphMapper.java:204)
> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:392)
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:655)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> Benchmark Solutions
> 101 Park Avenue - 7th Floor
> New York, NY 10178
> Tel +1.212.220.4739 | Mobile +1.646.267.4324
> *www.benchmarksolutions.com * <http://www.benchmarksolutions.com/>
> ***
>
>
>
> *
> **
>