You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Jyoti Yadav <ra...@gmail.com> on 2013/11/13 16:20:48 UTC
About vertex input format..
Hi..
I am trying to execute SingleSourceShortestPath example with some changes
in this.. While sending message,i am sending two values-(sender vertex
value+edge weight),sender_id..
For this message to send,i created my own message file
MyMessageWritable.java.
The graph input is given in following form..
[0,0,[[1,1],[3,3]]][1,0,[[0,1],[2,2],[3,1]]][2,0,[[1,2],[4,4]]][3,0,[[0,3],[1,1],[4,4]]][4,0,[[3,4],[2,4]]]
here vertex id is -long
vertex value-long
edge value-long
Message is (long,long)
what should be the vertex input format for this??
Any help is really appreciated...
Regards
Jyoti
Re: About vertex input format..
Posted by Jyoti Yadav <ra...@gmail.com>.
Thanks a lot Ameya....Your ideas really helped me a lot..I got my program
executed...
Nice job..:)
On Sat, Nov 16, 2013 at 1:11 AM, Ameya Vilankar <am...@gmail.com>wrote:
> No, I said you should not be bothered about the Message Type in your
> Vertex Input Format Class. It only needs your Vertex id type, Vertex Value
> type and the Edge Value type. It does not need the type of messages you are
> sending.
>
> You can put the VertexInputFormat File anywhere you want. Preferably keep
> it in the same location as you Computation Class. I know that in giraph,
> all the input formats are in the io.formats package in giraph-core. You can
> keep it there if you want and it will work in your case, since you vertex
> value is just LongWritable. But if you are writing a more complex giraph
> program where you vertex value stores all kinds of information and you hava
> a custom Writable Class. Then you would have to keep your vertex value
> class somewhere in the giraph-core package just so that your InputFormat
> class that is in the io package has access to it. So its best if you keep
> your vertex input format class in the same directory as your Computation
> Class and your Vertex value class.
> Have look at the BrachaToeugComputation Example in giraph-examples. It
> defines its own io formats in giraph-examples.
>
>
> On Wed, Nov 13, 2013 at 9:56 PM, Jyoti Yadav <ra...@gmail.com>wrote:
>
>> Thanks Ameya..
>> I am just newbie to giraph... Would you please let me know the path
>> where should i include this file ,so that i can compile it.
>>
>> Also you mentioned that i need not to bothered about
>> VertexInputFormat,then while giving the jar command to execute the program
>> ,what about -vif option?? Should i ignore it??
>>
>> Thanks
>> Jyoti
>>
>>
>> On Thu, Nov 14, 2013 at 12:20 AM, Ameya Vilankar <
>> ameya.vilankar@gmail.com> wrote:
>>
>>> Here is class that you need. I haven't tested it. You don't need to
>>> worry about the message type in InputFormatClass. This code compiled with
>>> the most updated trunk branch of giraph.
>>>
>>>
>>> import com.google.common.collect.Lists;
>>> import org.apache.giraph.edge.Edge;
>>> import org.apache.giraph.edge.EdgeFactory;
>>> import org.apache.giraph.graph.Vertex;
>>> import org.apache.giraph.io.formats.TextVertexInputFormat;
>>> import org.apache.hadoop.io.LongWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.hadoop.mapreduce.InputSplit;
>>> import org.apache.hadoop.mapreduce.TaskAttemptContext;
>>> import org.json.JSONArray;
>>> import org.json.JSONException;
>>>
>>> import java.io.IOException;
>>> import java.util.List;
>>>
>>> public class ShortestPathVertexInputFormat extends
>>> TextVertexInputFormat<LongWritable, LongWritable, LongWritable> {
>>>
>>> @Override
>>> public TextVertexReader createVertexReader(InputSplit split,
>>> TaskAttemptContext
>>> context) {
>>> return new ShortestPathVertexReader();
>>> }
>>>
>>> class ShortestPathVertexReader
>>> extends
>>> TextVertexReaderFromEachLineProcessedHandlingExceptions<JSONArray,
>>> JSONException> {
>>>
>>> @Override
>>> protected JSONArray preprocessLine(Text line) throws
>>> JSONException {
>>> return new JSONArray(line.toString());
>>> }
>>>
>>> @Override
>>> protected LongWritable getId(JSONArray jsonVertex) throws
>>> JSONException,
>>> IOException {
>>>
>>> // System.out.println("Got Vertex Id: " +
>>> jsonVertex.getLong(0));
>>> return new LongWritable(jsonVertex.getLong(0));
>>> }
>>>
>>> @Override
>>> protected LongWritable getValue(JSONArray jsonVertex)
>>> throws JSONException, IOException {
>>>
>>> return new LongWritable(0L);
>>> }
>>>
>>> @Override
>>> protected Iterable<Edge<LongWritable, LongWritable>>
>>> getEdges(JSONArray jsonVertex) throws JSONException, IOException
>>> {
>>>
>>> // Get the Edge array
>>> JSONArray jsonEdgeArray = jsonVertex.getJSONArray(2);
>>>
>>> List<Edge<LongWritable, LongWritable>> edges =
>>>
>>> Lists.newArrayListWithCapacity(jsonEdgeArray.length());
>>>
>>> // Get the indiviudal edges from the edge array
>>> for (int i = 0; i < jsonEdgeArray.length(); ++i) {
>>> JSONArray jsonEdge = jsonEdgeArray.getJSONArray(i);
>>> edges.add(EdgeFactory.create(new
>>> LongWritable(jsonEdge.getLong(0)), new LongWritable(jsonEdge.getLong(1))));
>>> }
>>>
>>> //System.out.println("Got The Edges for the Vertex: " +
>>> jsonVertex.getLong(0));
>>> return edges;
>>> }
>>>
>>> @Override
>>> protected Vertex<LongWritable, LongWritable,
>>> LongWritable> handleException(Text line, JSONArray
>>> jsonVertex,
>>> JSONException e) {
>>>
>>> throw new IllegalArgumentException(
>>> "Couldn't get vertex from line " + line, e);
>>> }
>>> }
>>> }
>>>
>>>
>>> On Wed, Nov 13, 2013 at 7:20 AM, Jyoti Yadav <rao.jyoti26yadav@gmail.com
>>> > wrote:
>>>
>>>> Hi..
>>>> I am trying to execute SingleSourceShortestPath example with some
>>>> changes in this.. While sending message,i am sending two values-(sender
>>>> vertex value+edge weight),sender_id..
>>>> For this message to send,i created my own message file
>>>> MyMessageWritable.java.
>>>> The graph input is given in following form..
>>>>
>>>> [0,0,[[1,1],[3,3]]][1,0,[[0,1],[2,2],[3,1]]][2,0,[[1,2],[4,4]]][3,0,[[0,3],[1,1],[4,4]]][4,0,[[3,4],[2,4]]]
>>>>
>>>>
>>>> here vertex id is -long
>>>>
>>>> vertex value-long
>>>>
>>>> edge value-long
>>>>
>>>> Message is (long,long)
>>>>
>>>> what should be the vertex input format for this??
>>>>
>>>> Any help is really appreciated...
>>>>
>>>>
>>>>
>>>>
>>>> Regards
>>>>
>>>> Jyoti
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: About vertex input format..
Posted by Ameya Vilankar <am...@gmail.com>.
No, I said you should not be bothered about the Message Type in your Vertex
Input Format Class. It only needs your Vertex id type, Vertex Value type
and the Edge Value type. It does not need the type of messages you are
sending.
You can put the VertexInputFormat File anywhere you want. Preferably keep
it in the same location as you Computation Class. I know that in giraph,
all the input formats are in the io.formats package in giraph-core. You can
keep it there if you want and it will work in your case, since you vertex
value is just LongWritable. But if you are writing a more complex giraph
program where you vertex value stores all kinds of information and you hava
a custom Writable Class. Then you would have to keep your vertex value
class somewhere in the giraph-core package just so that your InputFormat
class that is in the io package has access to it. So its best if you keep
your vertex input format class in the same directory as your Computation
Class and your Vertex value class.
Have look at the BrachaToeugComputation Example in giraph-examples. It
defines its own io formats in giraph-examples.
On Wed, Nov 13, 2013 at 9:56 PM, Jyoti Yadav <ra...@gmail.com>wrote:
> Thanks Ameya..
> I am just newbie to giraph... Would you please let me know the path
> where should i include this file ,so that i can compile it.
>
> Also you mentioned that i need not to bothered about
> VertexInputFormat,then while giving the jar command to execute the program
> ,what about -vif option?? Should i ignore it??
>
> Thanks
> Jyoti
>
>
> On Thu, Nov 14, 2013 at 12:20 AM, Ameya Vilankar <ameya.vilankar@gmail.com
> > wrote:
>
>> Here is class that you need. I haven't tested it. You don't need to worry
>> about the message type in InputFormatClass. This code compiled with the
>> most updated trunk branch of giraph.
>>
>>
>> import com.google.common.collect.Lists;
>> import org.apache.giraph.edge.Edge;
>> import org.apache.giraph.edge.EdgeFactory;
>> import org.apache.giraph.graph.Vertex;
>> import org.apache.giraph.io.formats.TextVertexInputFormat;
>> import org.apache.hadoop.io.LongWritable;
>> import org.apache.hadoop.io.Text;
>> import org.apache.hadoop.mapreduce.InputSplit;
>> import org.apache.hadoop.mapreduce.TaskAttemptContext;
>> import org.json.JSONArray;
>> import org.json.JSONException;
>>
>> import java.io.IOException;
>> import java.util.List;
>>
>> public class ShortestPathVertexInputFormat extends
>> TextVertexInputFormat<LongWritable, LongWritable, LongWritable> {
>>
>> @Override
>> public TextVertexReader createVertexReader(InputSplit split,
>> TaskAttemptContext
>> context) {
>> return new ShortestPathVertexReader();
>> }
>>
>> class ShortestPathVertexReader
>> extends
>> TextVertexReaderFromEachLineProcessedHandlingExceptions<JSONArray,
>> JSONException> {
>>
>> @Override
>> protected JSONArray preprocessLine(Text line) throws
>> JSONException {
>> return new JSONArray(line.toString());
>> }
>>
>> @Override
>> protected LongWritable getId(JSONArray jsonVertex) throws
>> JSONException,
>> IOException {
>>
>> // System.out.println("Got Vertex Id: " +
>> jsonVertex.getLong(0));
>> return new LongWritable(jsonVertex.getLong(0));
>> }
>>
>> @Override
>> protected LongWritable getValue(JSONArray jsonVertex)
>> throws JSONException, IOException {
>>
>> return new LongWritable(0L);
>> }
>>
>> @Override
>> protected Iterable<Edge<LongWritable, LongWritable>>
>> getEdges(JSONArray jsonVertex) throws JSONException, IOException {
>>
>> // Get the Edge array
>> JSONArray jsonEdgeArray = jsonVertex.getJSONArray(2);
>>
>> List<Edge<LongWritable, LongWritable>> edges =
>>
>> Lists.newArrayListWithCapacity(jsonEdgeArray.length());
>>
>> // Get the indiviudal edges from the edge array
>> for (int i = 0; i < jsonEdgeArray.length(); ++i) {
>> JSONArray jsonEdge = jsonEdgeArray.getJSONArray(i);
>> edges.add(EdgeFactory.create(new
>> LongWritable(jsonEdge.getLong(0)), new LongWritable(jsonEdge.getLong(1))));
>> }
>>
>> //System.out.println("Got The Edges for the Vertex: " +
>> jsonVertex.getLong(0));
>> return edges;
>> }
>>
>> @Override
>> protected Vertex<LongWritable, LongWritable,
>> LongWritable> handleException(Text line, JSONArray
>> jsonVertex,
>> JSONException e) {
>>
>> throw new IllegalArgumentException(
>> "Couldn't get vertex from line " + line, e);
>> }
>> }
>> }
>>
>>
>> On Wed, Nov 13, 2013 at 7:20 AM, Jyoti Yadav <ra...@gmail.com>wrote:
>>
>>> Hi..
>>> I am trying to execute SingleSourceShortestPath example with some
>>> changes in this.. While sending message,i am sending two values-(sender
>>> vertex value+edge weight),sender_id..
>>> For this message to send,i created my own message file
>>> MyMessageWritable.java.
>>> The graph input is given in following form..
>>>
>>> [0,0,[[1,1],[3,3]]][1,0,[[0,1],[2,2],[3,1]]][2,0,[[1,2],[4,4]]][3,0,[[0,3],[1,1],[4,4]]][4,0,[[3,4],[2,4]]]
>>>
>>>
>>> here vertex id is -long
>>>
>>> vertex value-long
>>>
>>> edge value-long
>>>
>>> Message is (long,long)
>>>
>>> what should be the vertex input format for this??
>>>
>>> Any help is really appreciated...
>>>
>>>
>>>
>>> Regards
>>>
>>> Jyoti
>>>
>>>
>>>
>>
>
Re: About vertex input format..
Posted by Jyoti Yadav <ra...@gmail.com>.
Thanks Ameya..
I am just newbie to giraph... Would you please let me know the path where
should i include this file ,so that i can compile it.
Also you mentioned that i need not to bothered about VertexInputFormat,then
while giving the jar command to execute the program ,what about -vif
option?? Should i ignore it??
Thanks
Jyoti
On Thu, Nov 14, 2013 at 12:20 AM, Ameya Vilankar
<am...@gmail.com>wrote:
> Here is class that you need. I haven't tested it. You don't need to worry
> about the message type in InputFormatClass. This code compiled with the
> most updated trunk branch of giraph.
>
>
> import com.google.common.collect.Lists;
> import org.apache.giraph.edge.Edge;
> import org.apache.giraph.edge.EdgeFactory;
> import org.apache.giraph.graph.Vertex;
> import org.apache.giraph.io.formats.TextVertexInputFormat;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapreduce.InputSplit;
> import org.apache.hadoop.mapreduce.TaskAttemptContext;
> import org.json.JSONArray;
> import org.json.JSONException;
>
> import java.io.IOException;
> import java.util.List;
>
> public class ShortestPathVertexInputFormat extends
> TextVertexInputFormat<LongWritable, LongWritable, LongWritable> {
>
> @Override
> public TextVertexReader createVertexReader(InputSplit split,
> TaskAttemptContext context)
> {
> return new ShortestPathVertexReader();
> }
>
> class ShortestPathVertexReader
> extends
> TextVertexReaderFromEachLineProcessedHandlingExceptions<JSONArray,
> JSONException> {
>
> @Override
> protected JSONArray preprocessLine(Text line) throws JSONException
> {
> return new JSONArray(line.toString());
> }
>
> @Override
> protected LongWritable getId(JSONArray jsonVertex) throws
> JSONException,
> IOException {
>
> // System.out.println("Got Vertex Id: " +
> jsonVertex.getLong(0));
> return new LongWritable(jsonVertex.getLong(0));
> }
>
> @Override
> protected LongWritable getValue(JSONArray jsonVertex)
> throws JSONException, IOException {
>
> return new LongWritable(0L);
> }
>
> @Override
> protected Iterable<Edge<LongWritable, LongWritable>>
> getEdges(JSONArray jsonVertex) throws JSONException, IOException {
>
> // Get the Edge array
> JSONArray jsonEdgeArray = jsonVertex.getJSONArray(2);
>
> List<Edge<LongWritable, LongWritable>> edges =
> Lists.newArrayListWithCapacity(jsonEdgeArray.length());
>
> // Get the indiviudal edges from the edge array
> for (int i = 0; i < jsonEdgeArray.length(); ++i) {
> JSONArray jsonEdge = jsonEdgeArray.getJSONArray(i);
> edges.add(EdgeFactory.create(new
> LongWritable(jsonEdge.getLong(0)), new LongWritable(jsonEdge.getLong(1))));
> }
>
> //System.out.println("Got The Edges for the Vertex: " +
> jsonVertex.getLong(0));
> return edges;
> }
>
> @Override
> protected Vertex<LongWritable, LongWritable,
> LongWritable> handleException(Text line, JSONArray
> jsonVertex,
> JSONException e) {
>
> throw new IllegalArgumentException(
> "Couldn't get vertex from line " + line, e);
> }
> }
> }
>
>
> On Wed, Nov 13, 2013 at 7:20 AM, Jyoti Yadav <ra...@gmail.com>wrote:
>
>> Hi..
>> I am trying to execute SingleSourceShortestPath example with some changes
>> in this.. While sending message,i am sending two values-(sender vertex
>> value+edge weight),sender_id..
>> For this message to send,i created my own message file
>> MyMessageWritable.java.
>> The graph input is given in following form..
>>
>> [0,0,[[1,1],[3,3]]][1,0,[[0,1],[2,2],[3,1]]][2,0,[[1,2],[4,4]]][3,0,[[0,3],[1,1],[4,4]]][4,0,[[3,4],[2,4]]]
>>
>>
>> here vertex id is -long
>>
>> vertex value-long
>>
>> edge value-long
>>
>> Message is (long,long)
>>
>> what should be the vertex input format for this??
>>
>> Any help is really appreciated...
>>
>>
>> Regards
>>
>> Jyoti
>>
>>
>>
>
Re: About vertex input format..
Posted by Ameya Vilankar <am...@gmail.com>.
Here is class that you need. I haven't tested it. You don't need to worry
about the message type in InputFormatClass. This code compiled with the
most updated trunk branch of giraph.
import com.google.common.collect.Lists;
import org.apache.giraph.edge.Edge;
import org.apache.giraph.edge.EdgeFactory;
import org.apache.giraph.graph.Vertex;
import org.apache.giraph.io.formats.TextVertexInputFormat;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.json.JSONArray;
import org.json.JSONException;
import java.io.IOException;
import java.util.List;
public class ShortestPathVertexInputFormat extends
TextVertexInputFormat<LongWritable, LongWritable, LongWritable> {
@Override
public TextVertexReader createVertexReader(InputSplit split,
TaskAttemptContext context) {
return new ShortestPathVertexReader();
}
class ShortestPathVertexReader
extends
TextVertexReaderFromEachLineProcessedHandlingExceptions<JSONArray,
JSONException> {
@Override
protected JSONArray preprocessLine(Text line) throws JSONException {
return new JSONArray(line.toString());
}
@Override
protected LongWritable getId(JSONArray jsonVertex) throws
JSONException,
IOException {
// System.out.println("Got Vertex Id: " +
jsonVertex.getLong(0));
return new LongWritable(jsonVertex.getLong(0));
}
@Override
protected LongWritable getValue(JSONArray jsonVertex)
throws JSONException, IOException {
return new LongWritable(0L);
}
@Override
protected Iterable<Edge<LongWritable, LongWritable>>
getEdges(JSONArray jsonVertex) throws JSONException, IOException {
// Get the Edge array
JSONArray jsonEdgeArray = jsonVertex.getJSONArray(2);
List<Edge<LongWritable, LongWritable>> edges =
Lists.newArrayListWithCapacity(jsonEdgeArray.length());
// Get the indiviudal edges from the edge array
for (int i = 0; i < jsonEdgeArray.length(); ++i) {
JSONArray jsonEdge = jsonEdgeArray.getJSONArray(i);
edges.add(EdgeFactory.create(new
LongWritable(jsonEdge.getLong(0)), new LongWritable(jsonEdge.getLong(1))));
}
//System.out.println("Got The Edges for the Vertex: " +
jsonVertex.getLong(0));
return edges;
}
@Override
protected Vertex<LongWritable, LongWritable,
LongWritable> handleException(Text line, JSONArray
jsonVertex,
JSONException e) {
throw new IllegalArgumentException(
"Couldn't get vertex from line " + line, e);
}
}
}
On Wed, Nov 13, 2013 at 7:20 AM, Jyoti Yadav <ra...@gmail.com>wrote:
> Hi..
> I am trying to execute SingleSourceShortestPath example with some changes
> in this.. While sending message,i am sending two values-(sender vertex
> value+edge weight),sender_id..
> For this message to send,i created my own message file
> MyMessageWritable.java.
> The graph input is given in following form..
>
> [0,0,[[1,1],[3,3]]][1,0,[[0,1],[2,2],[3,1]]][2,0,[[1,2],[4,4]]][3,0,[[0,3],[1,1],[4,4]]][4,0,[[3,4],[2,4]]]
>
>
> here vertex id is -long
>
> vertex value-long
>
> edge value-long
>
> Message is (long,long)
>
> what should be the vertex input format for this??
>
> Any help is really appreciated...
>
> Regards
>
> Jyoti
>
>
>