You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by david <dg...@bbn.com> on 2013/07/16 23:05:42 UTC
SequenceFileVertexInputFormatClass
Does anyone have a simple example of how to use it? Do I need to extend it
for my classes or can I just use it like I would
SequenceFileInputFormat in hadoop. I'm getting the following error;
java.lang.IllegalStateException: generateVertexInputSplits: Got IOException
at
org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMast
er.java:316)
at
org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster
.java:626)
at
org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspService
Master.java:693)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:99)
Caused by: java.io.IOException: No input paths specified in job
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFo
rmat.java:189)
at
org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(Seq
uenceFileInputFormat.java:55)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFor
mat.java:248)
at
org.apache.giraph.io.formats.SequenceFileVertexInputFormat.getSplits(Sequenc
eFileVertexInputFormat.java:52)
at
org.apache.giraph.io.internal.WrappedVertexInputFormat.getSplits(WrappedVert
exInputFormat.java:66)
at
org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMast
er.java:314)
But I think the input path is correctly set. Thanks, I appreciate it,
David
Question concerning Aggregators
Posted by Pascal Jäger <pa...@pascaljaeger.de>.
Hi everyone,
I am trying to use an Aggregator I have written.
Unfortunately I do get an IOException.
As far as I can see the exception occurs when things are send from the
Aggregator to the Master, during a readFields of one of my classes.
The writes and readFields of my classes do not seem to be the problem
because a lot of messages get passed without any problem.
But shortly before it comes to the exception the read methods get values
that do not match to my application, e.g. It reads Long.MAX_Value / 2
instead of 3.
An then suddenly the EOF exception occurs during the readFields() of one
of my classes.
I checked if my code accidentally produces the errors by checking what
gets written out and read in later, but this seems to be okay - except for
the case below.
Do you have any idea?
Regards
Pascal
2013-07-17 15:52:09,087 INFO org.apache.giraph.master.BspServiceMaster:
aggregateWorkerStats: Aggregation found
(vtx=6,finVtx=0,edges=16,msgCount=8,haltComputation=false) on superstep = 1
2013-07-17 15:52:09,088 INFO org.apache.giraph.master.BspServiceMaster:
coordinateSuperstep: Cleaning up old Superstep
/_hadoopBsp/job_201307171551_0001/_applicationAttemptsDir/0/_superstepDir/0
2013-07-17 15:52:09,108 INFO org.apache.giraph.master.MasterThread:
masterThread: Coordination of superstep 1 took 0.07 seconds ended with
state THIS_SUPERSTEP_DONE and is now on superstep 2
2013-07-17 15:52:09,112 INFO org.apache.giraph.comm.netty.NettyClient:
connectAllAddresses: Successfully added 0 connections, (0 total connected)
0 failed, 0 failures total.
2013-07-17 15:52:09,112 INFO
org.apache.giraph.partition.PartitionBalancer:
balancePartitionsAcrossWorkers: Using algorithm static
2013-07-17 15:52:09,112 INFO org.apache.giraph.partition.PartitionUtils:
analyzePartitionStats: Vertices - Mean: 6, Min: Worker(hostname=127.0.0.1,
MRtaskID=1, port=30001) - 6, Max: Worker(hostname=127.0.0.1, MRtaskID=1,
port=30001) - 6
2013-07-17 15:52:09,112 INFO org.apache.giraph.partition.PartitionUtils:
analyzePartitionStats: Edges - Mean: 16, Min: Worker(hostname=127.0.0.1,
MRtaskID=1, port=30001) - 16, Max: Worker(hostname=127.0.0.1, MRtaskID=1,
port=30001) - 16
2013-07-17 15:52:09,119 INFO org.apache.giraph.master.BspServiceMaster:
barrierOnWorkerList: 0 out of 1 workers finished on superstep 2 on path
/_hadoopBsp/job_201307171551_0001/_applicationAttemptsDir/0/_superstepDir/2
/_workerFinishedDir
2013-07-17 15:52:09,119 INFO org.apache.giraph.master.BspServiceMaster:
barrierOnWorkerList: Waiting on [127.0.0.1_1]
2013-07-17 15:52:09,144 WARN
org.apache.giraph.comm.netty.handler.RequestServerHandler:
exceptionCaught: Channel failed with remote address /127.0.0.1:60439
java.lang.IllegalStateException: doRequest: IOException occurred while
processing request
at
org.apache.giraph.comm.requests.SendAggregatorsToMasterRequest.doRequest(Se
ndAggregatorsToMasterRequest.java:52)
at
org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequ
est(MasterRequestServerHandler.java:51)
at
org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequ
est(MasterRequestServerHandler.java:27)
at
org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(R
equestServerHandler.java:106)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOn
eDecoder.java:71)
at
org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(Channe
lUpstreamEventRunnable.java:45)
at
org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunn
able.java:69)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j
ava:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
918)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at java.io.DataInputStream.readDouble(DataInputStream.java:451)
at mystuff.maxflow.ExcessPath.readFields(ExcessPath.java:71)
at mystuff.maxflow.MFMessage.readFields(MFMessage.java:42)
at
org.apache.giraph.master.MasterAggregatorHandler.acceptAggregatedValues(Mas
terAggregatorHandler.java:253)
at
org.apache.giraph.comm.requests.SendAggregatorsToMasterRequest.doRequest(Se
ndAggregatorsToMasterRequest.java:50)
... 10 more
2013-07-17 15:57:09,151 INFO org.apache.giraph.master.BspServiceMaster:
barrierOnWorkerList: 0 out of 1 workers finished on superstep 2 on path
/_hadoopBsp/job_201307171551_0001/_applicationAttemptsDir/0/_superstepDir/2
/_workerFinishedDir
2013-07-17 15:57:09,151 INFO org.apache.giraph.master.BspServiceMaster:
barrierOnWorkerList: Waiting on [127.0.0.1_1]
RE: SequenceFileVertexInputFormatClass
Posted by "Nalezenec, Lukas" <Lu...@firma.seznam.cz>.
Hi,
I am using
FileInputFormat.addInputPath(job.getInternalJob(), new Path(args[0]));
its not best solution but it works.
Lukas
________________________________________
Od: david [dgainer@bbn.com]
Odesláno: 16. července 2013 23:11
Komu: user@giraph.apache.org
Předmět: RE: SequenceFileVertexInputFormatClass
Actually I'm guessing that I'm just not setting the input path correctly - does this work:
GiraphConfiguration conf = job.getConfiguration();
...
GiraphFileInputFormat.addVertexInputPath(conf, new Path(args[0]));
conf.setVertexInputFormatClass(SequenceFileVertexInputFormat.class);
(in 1.1.0 snapshot)
Thanks,
David
________________________________
From: david [mailto:dgainer@bbn.com]
Sent: Tuesday, July 16, 2013 5:06 PM
To: user@giraph.apache.org
Subject: SequenceFileVertexInputFormatClass
Does anyone have a simple example of how to use it? Do I need to extend it for my classes or can I just use it like I would
SequenceFileInputFormat in hadoop. I'm getting the following error;
java.lang.IllegalStateException: generateVertexInputSplits: Got IOException
at org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMaster.java:316)
at org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:626)
at org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:693)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:99)
Caused by: java.io.IOException: No input paths specified in job
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:189)
at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:55)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248)
at org.apache.giraph.io.formats.SequenceFileVertexInputFormat.getSplits(SequenceFileVertexInputFormat.java:52)
at org.apache.giraph.io.internal.WrappedVertexInputFormat.getSplits(WrappedVertexInputFormat.java:66)
at org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMaster.java:314)
But I think the input path is correctly set. Thanks, I appreciate it,
David
RE: SequenceFileVertexInputFormatClass
Posted by david <dg...@bbn.com>.
Actually I'm guessing that I'm just not setting the input path correctly -
does this work:
GiraphConfiguration conf = job.getConfiguration();
...
GiraphFileInputFormat.addVertexInputPath(conf, new Path(args[0]));
conf.setVertexInputFormatClass(SequenceFileVertexInputFormat.class);
(in 1.1.0 snapshot)
Thanks,
David
_____
From: david [mailto:dgainer@bbn.com]
Sent: Tuesday, July 16, 2013 5:06 PM
To: user@giraph.apache.org
Subject: SequenceFileVertexInputFormatClass
Does anyone have a simple example of how to use it? Do I need to extend it
for my classes or can I just use it like I would
SequenceFileInputFormat in hadoop. I'm getting the following error;
java.lang.IllegalStateException: generateVertexInputSplits: Got IOException
at
org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMast
er.java:316)
at
org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster
.java:626)
at
org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspService
Master.java:693)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:99)
Caused by: java.io.IOException: No input paths specified in job
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFo
rmat.java:189)
at
org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(Seq
uenceFileInputFormat.java:55)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFor
mat.java:248)
at
org.apache.giraph.io.formats.SequenceFileVertexInputFormat.getSplits(Sequenc
eFileVertexInputFormat.java:52)
at
org.apache.giraph.io.internal.WrappedVertexInputFormat.getSplits(WrappedVert
exInputFormat.java:66)
at
org.apache.giraph.master.BspServiceMaster.generateInputSplits(BspServiceMast
er.java:314)
But I think the input path is correctly set. Thanks, I appreciate it,
David