You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Kyle Orlando <ky...@gmail.com> on 2013/08/01 01:13:37 UTC
Getting Giraph to work with HBase
Hello,
I've been having some problems getting Giraph to work with a table
that is stored in HBase. I've created my own classes that extend
HBaseVertexInputFormat and HBaseVertexReader (respectively) and
implemented nextVertex() and getVertex() accordingly, but I still
can't get it to work. Here is what I typed into the command line after
specifying the GiraphRunner class:
org.apache.giraph.examples.SimplePageRankComputation -vif
HBasePractice.LongDoubleFloatHBaseVertexInputFormat -of
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /hbaseOut
-w 2 -mc org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
This is the error that I'm getting from the logs:
2013-07-31 17:20:28,880 ERROR org.apache.giraph.master.MasterThread:
masterThread: Master algorithm failed with NullPointerException
java.lang.NullPointerException
at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:94)
at org.apache.giraph.io.hbase.HBaseVertexInputFormat.getSplits(HBaseVertexInputFormat.java:169)
at org.apache.giraph.io.internal.WrappedVertexInputFormat.getSplits(WrappedVertexInputFormat.java:72)
at org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMaster.java:315)
at org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:627)
at org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:694)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:100)
2013-07-31 17:20:28,881 FATAL org.apache.giraph.graph.GraphMapper:
uncaughtException: OverrideExceptionHandler on thread
org.apache.giraph.master.MasterThread, msg =
java.lang.NullPointerException, exiting...
java.lang.IllegalStateException: java.lang.NullPointerException
at org.apache.giraph.master.MasterThread.run(MasterThread.java:185)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:94)
at org.apache.giraph.io.hbase.HBaseVertexInputFormat.getSplits(HBaseVertexInputFormat.java:169)
at org.apache.giraph.io.internal.WrappedVertexInputFormat.getSplits(WrappedVertexInputFormat.java:72)
at org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMaster.java:315)
at org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:627)
at org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:694)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:100)
I don't know how/why a NullPointerException is being thrown. Do I
need to specify the -vip option? I didn't think so, because in my code
for my class that extends HBaseVertexInputFormat I do the following:
private ImmutableClassesGiraphConfiguration<LongWritable,
DoubleWritable, FloatWritable> conf;
@Override
public void setConf(ImmutableClassesGiraphConfiguration<LongWritable,
DoubleWritable, FloatWritable> conf) {
conf.set(TableInputFormat.INPUT_TABLE, "edges");
this.conf = conf;
}
Help would be much appreciated. I actually was a bit hesitant to post
this at first; I was determined to figure it out on my own, but then I
noticed that there have been virtually no questions asked regarding
HBase + Giraph on this mailing list, so I figured that posting this
might help someone out with similar problems later on in the future.
Thanks,
--
Kyle Orlando
Computer Engineering Major
University of Maryland
Re: Getting Giraph to work with HBase
Posted by Kyle Orlando <ky...@gmail.com>.
Update:
The NullPointerException being thrown was an error on my part: I had
accidentally deleted the table I was using minutes prior. But
exceptions were still being thrown after I fixed that. Eventually, I
found this in the logs:
FATAL org.apache.giraph.graph.GraphMapper: uncaughtException:
OverrideExceptionHandler on thread
org.apache.giraph.master.MasterThread, msg =
com/google/protobuf/Message, exiting...
java.lang.NoClassDefFoundError: com/google/protobuf/Message
I fixed this by adding protobuf-java-2.4.0a.jar in $HBASE_HOME/lib to
the hadoop classpath. In my case, I just copied the jar into
$HADOOP_HOME/lib. I started her up again, and it worked fine. Also, I
needed to add this to my class that extends HBaseVertexInputFormat to
make it work:
private ImmutableClassesGiraphConfiguration<I, V, E> conf;
@Override
public void setConf(ImmutableClassesGiraphConfiguration<I, V, E> conf) {
conf.set(TableInputFormat.INPUT_TABLE, "edges");
this.conf = conf;
}
@Override
public ImmutableClassesGiraphConfiguration<I, V, E> getConf() {
return conf;
}
On Wed, Jul 31, 2013 at 7:13 PM, Kyle Orlando <ky...@gmail.com> wrote:
> Hello,
>
> I've been having some problems getting Giraph to work with a table
> that is stored in HBase. I've created my own classes that extend
> HBaseVertexInputFormat and HBaseVertexReader (respectively) and
> implemented nextVertex() and getVertex() accordingly, but I still
> can't get it to work. Here is what I typed into the command line after
> specifying the GiraphRunner class:
>
> org.apache.giraph.examples.SimplePageRankComputation -vif
> HBasePractice.LongDoubleFloatHBaseVertexInputFormat -of
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /hbaseOut
> -w 2 -mc org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>
> This is the error that I'm getting from the logs:
>
> 2013-07-31 17:20:28,880 ERROR org.apache.giraph.master.MasterThread:
> masterThread: Master algorithm failed with NullPointerException
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:94)
> at org.apache.giraph.io.hbase.HBaseVertexInputFormat.getSplits(HBaseVertexInputFormat.java:169)
> at org.apache.giraph.io.internal.WrappedVertexInputFormat.getSplits(WrappedVertexInputFormat.java:72)
> at org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMaster.java:315)
> at org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:627)
> at org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:694)
> at org.apache.giraph.master.MasterThread.run(MasterThread.java:100)
> 2013-07-31 17:20:28,881 FATAL org.apache.giraph.graph.GraphMapper:
> uncaughtException: OverrideExceptionHandler on thread
> org.apache.giraph.master.MasterThread, msg =
> java.lang.NullPointerException, exiting...
> java.lang.IllegalStateException: java.lang.NullPointerException
> at org.apache.giraph.master.MasterThread.run(MasterThread.java:185)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:94)
> at org.apache.giraph.io.hbase.HBaseVertexInputFormat.getSplits(HBaseVertexInputFormat.java:169)
> at org.apache.giraph.io.internal.WrappedVertexInputFormat.getSplits(WrappedVertexInputFormat.java:72)
> at org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMaster.java:315)
> at org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:627)
> at org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:694)
> at org.apache.giraph.master.MasterThread.run(MasterThread.java:100)
>
> I don't know how/why a NullPointerException is being thrown. Do I
> need to specify the -vip option? I didn't think so, because in my code
> for my class that extends HBaseVertexInputFormat I do the following:
>
> private ImmutableClassesGiraphConfiguration<LongWritable,
> DoubleWritable, FloatWritable> conf;
>
> @Override
> public void setConf(ImmutableClassesGiraphConfiguration<LongWritable,
> DoubleWritable, FloatWritable> conf) {
> conf.set(TableInputFormat.INPUT_TABLE, "edges");
> this.conf = conf;
> }
>
> Help would be much appreciated. I actually was a bit hesitant to post
> this at first; I was determined to figure it out on my own, but then I
> noticed that there have been virtually no questions asked regarding
> HBase + Giraph on this mailing list, so I figured that posting this
> might help someone out with similar problems later on in the future.
>
> Thanks,
> --
> Kyle Orlando
> Computer Engineering Major
> University of Maryland
--
Kyle Orlando
Computer Engineering Major
University of Maryland