You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Yasser Altowim <ya...@ericsson.com> on 2013/08/15 23:07:03 UTC
MultiVertexInputFormat
Hi,
I am implementing an algorithm using Giraph. My algorithm needs to read input data from two files, each has its own format. My questions are:
1. How can I use the MultiVertexInputFormat class? Is there any example that shows how this class can be used?
2. How can I specify this class when running my job using the Giraph Runner or using a driver class?
Thanks in advance.
Best,
Yasser
RE: MultiVertexInputFormat
Posted by Yasser Altowim <ya...@ericsson.com>.
Thanks Maja for your response. That works but as I told you I had to modify the implementation of the MultiVertexInputFormat. I am posting my fix here in case someone runs into a similar problem.
@Override
public VertexReader<I, V, E> createVertexReader(InputSplit inputSplit,
TaskAttemptContext context) throws IOException {
if (inputSplit instanceof InputSplitWithInputFormatIndex) {
// When multithreaded input is used we need to make sure other threads
// don't change context's configuration while we use it
synchronized (context) {
InputSplitWithInputFormatIndex split =
(InputSplitWithInputFormatIndex) inputSplit;
VertexInputFormat<I, V, E> vertexInputFormat =
vertexInputFormats.get(split.getInputFormatIndex());
VertexReader<I, V, E> vertexReader =
vertexInputFormat.createVertexReader(split.getSplit(), context);
return new WrappedVertexReader<I, V, E>(
vertexReader, vertexInputFormat.getConf()) {
@Override
public void initialize(InputSplit inputSplit,
TaskAttemptContext context) throws IOException,
InterruptedException {
// When multithreaded input is used we need to make sure other
// threads don't change context's configuration while we use it
synchronized (context) {
super.initialize(inputSplit, context);
}
}
};
}
} else {
throw new IllegalStateException("createVertexReader: Got InputSplit " +
"which was not created by this class: " +
inputSplit.getClass().getName());
}
}
I changed the line in red above to the following:
super.initialize(((InputSplitWithInputFormatIndex) inputSplit).getSplit(), context);
Best,
Yasser
From: Maja Kabiljo [mailto:majakabiljo@fb.com]
Sent: Wednesday, August 21, 2013 8:24 PM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: Re: MultiVertexInputFormat
Hi Yasser,
You can do this through the Configuration parameters. You should call:
description1.addParameter("myApplication.vertexInputPath", "file1.txt");
and
description2.addParameter("myApplication.vertexInputPath", "file2.txt");
Then from the code of your InputFormat class you can get this parameter from Configuration. If it's not already, make sure your InputFormat implements ImmutableClassesGiraphConfigurable, and configuration is going to be set in it automatically.
You can also take a look at HiveGiraphRunner which uses multiple inputs and sets parameters user passes from command line.
Hope this helps,
Maja
From: Yasser Altowim <ya...@ericsson.com>>
Reply-To: "user@giraph.apache.org<ma...@giraph.apache.org>" <us...@giraph.apache.org>>
Date: Monday, August 19, 2013 9:16 AM
To: "user@giraph.apache.org<ma...@giraph.apache.org>" <us...@giraph.apache.org>>
Subject: RE: MultiVertexInputFormat
Hi Guys,
Any help on this will be appreciated. I am repeating my question and my code below:
I am implementing an algorithm in Giraph that reads the vertex values from two input files, each has its own format. I am not using any EdgeInputFormatClass. I am now using VertexInputFormatDescription along with MultiVertexInputFormats, but still could not figure out how to set the Vertex input path for each Input Format Class. Can you please take a look at my code below and show me how to set the Vertex Input Path? I have taken a look at HiveGiraphRunner but still no luck. Thanks
if (null == getConf()) {
conf = new Configuration();
}
GiraphConfiguration gconf = new GiraphConfiguration(getConf());
int workers = Integer.parseInt(arg0[2]);
gconf.setWorkerConfiguration(workers, workers, 100.0f);
List<VertexInputFormatDescription> vertexInputDescriptions = Lists.newArrayList();
// Input one
VertexInputFormatDescription description1 = new VertexInputFormatDescription(UseCase1FirstVertexInputFormat.class);
// how to set the vertex input path? i.e. how to say that I want to read file1.txt using this input format class
vertexInputDescriptions.add(description1);
// Input two
VertexInputFormatDescription description2 = new VertexInputFormatDescription(UseCase1SecondVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description2);
GiraphConstants.VERTEX_INPUT_FORMAT_CLASS.set(gconf,
MultiVertexInputFormat.class);
VertexInputFormatDescription.VERTEX_INPUT_FORMAT_DESCRIPTIONS.set(gconf,InputFormatDescription.toJsonString(vertexInputDescriptions));
gconf.setVertexOutputFormatClass(UseCase1OutputFormat.class);
gconf.setComputationClass(UseCase1Vertex.class);
GiraphJob job = new GiraphJob(gconf, "Use Case 1");
FileOutputFormat.setOutputPath(job.getInternalJob(), new Path(arg0[1]));
return job.run(true) ? 0 : -1;
Thanks in advance.
Best,
Yasser
From: Yasser Altowim [mailto:yasser.altowim@ericsson.com]
Sent: Friday, August 16, 2013 11:36 AM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: RE: MultiVertexInputFormat
Thanks a lot Avery for your response. I am now using VertexInputFormatDescription, but still could not figure out how to set the Vertex input path. I just need to read the vertex values from two different files, each with its own format. I am not using any EdgeInputFormatClass.
Can you please take a look at my code below and show me how to set the Vertex Input Path? Thanks
if (null == getConf()) {
conf = new Configuration();
}
GiraphConfiguration gconf = new GiraphConfiguration(getConf());
int workers = Integer.parseInt(arg0[2]);
gconf.setWorkerConfiguration(workers, workers, 100.0f);
List<VertexInputFormatDescription> vertexInputDescriptions = Lists.newArrayList();
// Input one
VertexInputFormatDescription description1 = new VertexInputFormatDescription(UseCase1FirstVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description1);
// Input two
VertexInputFormatDescription description2 = new VertexInputFormatDescription(UseCase1SecondVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description2);
VertexInputFormatDescription.VERTEX_INPUT_FORMAT_DESCRIPTIONS.set(gconf,InputFormatDescription.toJsonString(vertexInputDescriptions));
gconf.setVertexOutputFormatClass(UseCase1OutputFormat.class);
gconf.setComputationClass(UseCase1Vertex.class);
GiraphJob job = new GiraphJob(gconf, "Use Case 1");
FileOutputFormat.setOutputPath(job.getInternalJob(), new Path(arg0[1]));
return job.run(true) ? 0 : -1;
Best,
Yasser
From: Avery Ching [mailto:aching@apache.org]
Sent: Friday, August 16, 2013 9:50 AM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: Re: MultiVertexInputFormat
This is doable in Giraph, you can use as many vertex or edge input formats as you like (via GIRAPH-639). You just need to choose MultiVertexInputFormat and/or MultiEdgeInputFromat
See VertexInputFormatDescription for vertex input formats
/**
* VertexInputFormats description - JSON array containing a JSON array for
* each vertex input. Vertex input JSON arrays contain one or two elements -
* first one is the name of vertex input class, and second one is JSON object
* with all specific parameters for this vertex input. For example:
* [["VIF1",{"p":"v1"}],["VIF2",{"p":"v2","q":"v"}]]
*/
public static final StrConfOption VERTEX_INPUT_FORMAT_DESCRIPTIONS =
new StrConfOption("giraph.multiVertexInput.descriptions", null,
"VertexInputFormats description - JSON array containing a JSON " +
"array for each vertex input. Vertex input JSON arrays contain " +
"one or two elements - first one is the name of vertex input " +
"class, and second one is JSON object with all specific parameters " +
"for this vertex input. For example: [[\"VIF1\",{\"p\":\"v1\"}]," +
"[\"VIF2\",{\"p\":\"v2\",\"q\":\"v\"}]]\"");
See EdgeInputFormatDescription for edge input formats
/**
* EdgeInputFormats description - JSON array containing a JSON array for
* each edge input. Edge input JSON arrays contain one or two elements -
* first one is the name of edge input class, and second one is JSON object
* with all specific parameters for this edge input. For example:
* [["EIF1",{"p":"v1"}],["EIF2",{"p":"v2","q":"v"}]]
*/
public static final StrConfOption EDGE_INPUT_FORMAT_DESCRIPTIONS =
new StrConfOption("giraph.multiEdgeInput.descriptions", null,
"EdgeInputFormats description - JSON array containing a JSON array " +
"for each edge input. Edge input JSON arrays contain one or two " +
"elements - first one is the name of edge input class, and second " +
"one is JSON object with all specific parameters for this edge " +
"input. For example: [[\"EIF1\",{\"p\":\"v1\"}]," +
"[\"EIF2\",{\"p\":\"v2\",\"q\":\"v\"}]]");
Hope that helps,
Avery
On 8/16/13 8:45 AM, Yasser Altowim wrote:
Guys, any help with this will be appreciated. Thanks.
From: Yasser Altowim [mailto:yasser.altowim@ericsson.com]
Sent: Thursday, August 15, 2013 2:07 PM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: MultiVertexInputFormat
Hi,
I am implementing an algorithm using Giraph. My algorithm needs to read input data from two files, each has its own format. My questions are:
1. How can I use the MultiVertexInputFormat class? Is there any example that shows how this class can be used?
2. How can I specify this class when running my job using the Giraph Runner or using a driver class?
Thanks in advance.
Best,
Yasser
Re: MultiVertexInputFormat
Posted by Maja Kabiljo <ma...@fb.com>.
Hi Yasser,
You can do this through the Configuration parameters. You should call:
description1.addParameter("myApplication.vertexInputPath", "file1.txt");
and
description2.addParameter("myApplication.vertexInputPath", "file2.txt");
Then from the code of your InputFormat class you can get this parameter from Configuration. If it's not already, make sure your InputFormat implements ImmutableClassesGiraphConfigurable, and configuration is going to be set in it automatically.
You can also take a look at HiveGiraphRunner which uses multiple inputs and sets parameters user passes from command line.
Hope this helps,
Maja
From: Yasser Altowim <ya...@ericsson.com>>
Reply-To: "user@giraph.apache.org<ma...@giraph.apache.org>" <us...@giraph.apache.org>>
Date: Monday, August 19, 2013 9:16 AM
To: "user@giraph.apache.org<ma...@giraph.apache.org>" <us...@giraph.apache.org>>
Subject: RE: MultiVertexInputFormat
Hi Guys,
Any help on this will be appreciated. I am repeating my question and my code below:
I am implementing an algorithm in Giraph that reads the vertex values from two input files, each has its own format. I am not using any EdgeInputFormatClass. I am now using VertexInputFormatDescription along with MultiVertexInputFormats, but still could not figure out how to set the Vertex input path for each Input Format Class. Can you please take a look at my code below and show me how to set the Vertex Input Path? I have taken a look at HiveGiraphRunner but still no luck. Thanks
if (null == getConf()) {
conf = new Configuration();
}
GiraphConfiguration gconf = new GiraphConfiguration(getConf());
int workers = Integer.parseInt(arg0[2]);
gconf.setWorkerConfiguration(workers, workers, 100.0f);
List<VertexInputFormatDescription> vertexInputDescriptions = Lists.newArrayList();
// Input one
VertexInputFormatDescription description1 = new VertexInputFormatDescription(UseCase1FirstVertexInputFormat.class);
// how to set the vertex input path? i.e. how to say that I want to read file1.txt using this input format class
vertexInputDescriptions.add(description1);
// Input two
VertexInputFormatDescription description2 = new VertexInputFormatDescription(UseCase1SecondVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description2);
GiraphConstants.VERTEX_INPUT_FORMAT_CLASS.set(gconf,
MultiVertexInputFormat.class);
VertexInputFormatDescription.VERTEX_INPUT_FORMAT_DESCRIPTIONS.set(gconf,InputFormatDescription.toJsonString(vertexInputDescriptions));
gconf.setVertexOutputFormatClass(UseCase1OutputFormat.class);
gconf.setComputationClass(UseCase1Vertex.class);
GiraphJob job = new GiraphJob(gconf, "Use Case 1");
FileOutputFormat.setOutputPath(job.getInternalJob(), new Path(arg0[1]));
return job.run(true) ? 0 : -1;
Thanks in advance.
Best,
Yasser
From: Yasser Altowim [mailto:yasser.altowim@ericsson.com]
Sent: Friday, August 16, 2013 11:36 AM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: RE: MultiVertexInputFormat
Thanks a lot Avery for your response. I am now using VertexInputFormatDescription, but still could not figure out how to set the Vertex input path. I just need to read the vertex values from two different files, each with its own format. I am not using any EdgeInputFormatClass.
Can you please take a look at my code below and show me how to set the Vertex Input Path? Thanks
if (null == getConf()) {
conf = new Configuration();
}
GiraphConfiguration gconf = new GiraphConfiguration(getConf());
int workers = Integer.parseInt(arg0[2]);
gconf.setWorkerConfiguration(workers, workers, 100.0f);
List<VertexInputFormatDescription> vertexInputDescriptions = Lists.newArrayList();
// Input one
VertexInputFormatDescription description1 = new VertexInputFormatDescription(UseCase1FirstVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description1);
// Input two
VertexInputFormatDescription description2 = new VertexInputFormatDescription(UseCase1SecondVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description2);
VertexInputFormatDescription.VERTEX_INPUT_FORMAT_DESCRIPTIONS.set(gconf,InputFormatDescription.toJsonString(vertexInputDescriptions));
gconf.setVertexOutputFormatClass(UseCase1OutputFormat.class);
gconf.setComputationClass(UseCase1Vertex.class);
GiraphJob job = new GiraphJob(gconf, "Use Case 1");
FileOutputFormat.setOutputPath(job.getInternalJob(), new Path(arg0[1]));
return job.run(true) ? 0 : -1;
Best,
Yasser
From: Avery Ching [mailto:aching@apache.org]
Sent: Friday, August 16, 2013 9:50 AM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: Re: MultiVertexInputFormat
This is doable in Giraph, you can use as many vertex or edge input formats as you like (via GIRAPH-639). You just need to choose MultiVertexInputFormat and/or MultiEdgeInputFromat
See VertexInputFormatDescription for vertex input formats
/**
* VertexInputFormats description - JSON array containing a JSON array for
* each vertex input. Vertex input JSON arrays contain one or two elements -
* first one is the name of vertex input class, and second one is JSON object
* with all specific parameters for this vertex input. For example:
* [["VIF1",{"p":"v1"}],["VIF2",{"p":"v2","q":"v"}]]
*/
public static final StrConfOption VERTEX_INPUT_FORMAT_DESCRIPTIONS =
new StrConfOption("giraph.multiVertexInput.descriptions", null,
"VertexInputFormats description - JSON array containing a JSON " +
"array for each vertex input. Vertex input JSON arrays contain " +
"one or two elements - first one is the name of vertex input " +
"class, and second one is JSON object with all specific parameters " +
"for this vertex input. For example: [[\"VIF1\",{\"p\":\"v1\"}]," +
"[\"VIF2\",{\"p\":\"v2\",\"q\":\"v\"}]]\"");
See EdgeInputFormatDescription for edge input formats
/**
* EdgeInputFormats description - JSON array containing a JSON array for
* each edge input. Edge input JSON arrays contain one or two elements -
* first one is the name of edge input class, and second one is JSON object
* with all specific parameters for this edge input. For example:
* [["EIF1",{"p":"v1"}],["EIF2",{"p":"v2","q":"v"}]]
*/
public static final StrConfOption EDGE_INPUT_FORMAT_DESCRIPTIONS =
new StrConfOption("giraph.multiEdgeInput.descriptions", null,
"EdgeInputFormats description - JSON array containing a JSON array " +
"for each edge input. Edge input JSON arrays contain one or two " +
"elements - first one is the name of edge input class, and second " +
"one is JSON object with all specific parameters for this edge " +
"input. For example: [[\"EIF1\",{\"p\":\"v1\"}]," +
"[\"EIF2\",{\"p\":\"v2\",\"q\":\"v\"}]]");
Hope that helps,
Avery
On 8/16/13 8:45 AM, Yasser Altowim wrote:
Guys, any help with this will be appreciated. Thanks.
From: Yasser Altowim [mailto:yasser.altowim@ericsson.com]
Sent: Thursday, August 15, 2013 2:07 PM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: MultiVertexInputFormat
Hi,
I am implementing an algorithm using Giraph. My algorithm needs to read input data from two files, each has its own format. My questions are:
1. How can I use the MultiVertexInputFormat class? Is there any example that shows how this class can be used?
2. How can I specify this class when running my job using the Giraph Runner or using a driver class?
Thanks in advance.
Best,
Yasser
RE: MultiVertexInputFormat
Posted by Yasser Altowim <ya...@ericsson.com>.
Hi Guys,
Any help on this will be appreciated. I am repeating my question and my code below:
I am implementing an algorithm in Giraph that reads the vertex values from two input files, each has its own format. I am not using any EdgeInputFormatClass. I am now using VertexInputFormatDescription along with MultiVertexInputFormats, but still could not figure out how to set the Vertex input path for each Input Format Class. Can you please take a look at my code below and show me how to set the Vertex Input Path? I have taken a look at HiveGiraphRunner but still no luck. Thanks
if (null == getConf()) {
conf = new Configuration();
}
GiraphConfiguration gconf = new GiraphConfiguration(getConf());
int workers = Integer.parseInt(arg0[2]);
gconf.setWorkerConfiguration(workers, workers, 100.0f);
List<VertexInputFormatDescription> vertexInputDescriptions = Lists.newArrayList();
// Input one
VertexInputFormatDescription description1 = new VertexInputFormatDescription(UseCase1FirstVertexInputFormat.class);
// how to set the vertex input path? i.e. how to say that I want to read file1.txt using this input format class
vertexInputDescriptions.add(description1);
// Input two
VertexInputFormatDescription description2 = new VertexInputFormatDescription(UseCase1SecondVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description2);
GiraphConstants.VERTEX_INPUT_FORMAT_CLASS.set(gconf,
MultiVertexInputFormat.class);
VertexInputFormatDescription.VERTEX_INPUT_FORMAT_DESCRIPTIONS.set(gconf,InputFormatDescription.toJsonString(vertexInputDescriptions));
gconf.setVertexOutputFormatClass(UseCase1OutputFormat.class);
gconf.setComputationClass(UseCase1Vertex.class);
GiraphJob job = new GiraphJob(gconf, "Use Case 1");
FileOutputFormat.setOutputPath(job.getInternalJob(), new Path(arg0[1]));
return job.run(true) ? 0 : -1;
Thanks in advance.
Best,
Yasser
From: Yasser Altowim [mailto:yasser.altowim@ericsson.com]
Sent: Friday, August 16, 2013 11:36 AM
To: user@giraph.apache.org
Subject: RE: MultiVertexInputFormat
Thanks a lot Avery for your response. I am now using VertexInputFormatDescription, but still could not figure out how to set the Vertex input path. I just need to read the vertex values from two different files, each with its own format. I am not using any EdgeInputFormatClass.
Can you please take a look at my code below and show me how to set the Vertex Input Path? Thanks
if (null == getConf()) {
conf = new Configuration();
}
GiraphConfiguration gconf = new GiraphConfiguration(getConf());
int workers = Integer.parseInt(arg0[2]);
gconf.setWorkerConfiguration(workers, workers, 100.0f);
List<VertexInputFormatDescription> vertexInputDescriptions = Lists.newArrayList();
// Input one
VertexInputFormatDescription description1 = new VertexInputFormatDescription(UseCase1FirstVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description1);
// Input two
VertexInputFormatDescription description2 = new VertexInputFormatDescription(UseCase1SecondVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description2);
VertexInputFormatDescription.VERTEX_INPUT_FORMAT_DESCRIPTIONS.set(gconf,InputFormatDescription.toJsonString(vertexInputDescriptions));
gconf.setVertexOutputFormatClass(UseCase1OutputFormat.class);
gconf.setComputationClass(UseCase1Vertex.class);
GiraphJob job = new GiraphJob(gconf, "Use Case 1");
FileOutputFormat.setOutputPath(job.getInternalJob(), new Path(arg0[1]));
return job.run(true) ? 0 : -1;
Best,
Yasser
From: Avery Ching [mailto:aching@apache.org]
Sent: Friday, August 16, 2013 9:50 AM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: Re: MultiVertexInputFormat
This is doable in Giraph, you can use as many vertex or edge input formats as you like (via GIRAPH-639). You just need to choose MultiVertexInputFormat and/or MultiEdgeInputFromat
See VertexInputFormatDescription for vertex input formats
/**
* VertexInputFormats description - JSON array containing a JSON array for
* each vertex input. Vertex input JSON arrays contain one or two elements -
* first one is the name of vertex input class, and second one is JSON object
* with all specific parameters for this vertex input. For example:
* [["VIF1",{"p":"v1"}],["VIF2",{"p":"v2","q":"v"}]]
*/
public static final StrConfOption VERTEX_INPUT_FORMAT_DESCRIPTIONS =
new StrConfOption("giraph.multiVertexInput.descriptions", null,
"VertexInputFormats description - JSON array containing a JSON " +
"array for each vertex input. Vertex input JSON arrays contain " +
"one or two elements - first one is the name of vertex input " +
"class, and second one is JSON object with all specific parameters " +
"for this vertex input. For example: [[\"VIF1\",{\"p\":\"v1\"}]," +
"[\"VIF2\",{\"p\":\"v2\",\"q\":\"v\"}]]\"");
See EdgeInputFormatDescription for edge input formats
/**
* EdgeInputFormats description - JSON array containing a JSON array for
* each edge input. Edge input JSON arrays contain one or two elements -
* first one is the name of edge input class, and second one is JSON object
* with all specific parameters for this edge input. For example:
* [["EIF1",{"p":"v1"}],["EIF2",{"p":"v2","q":"v"}]]
*/
public static final StrConfOption EDGE_INPUT_FORMAT_DESCRIPTIONS =
new StrConfOption("giraph.multiEdgeInput.descriptions", null,
"EdgeInputFormats description - JSON array containing a JSON array " +
"for each edge input. Edge input JSON arrays contain one or two " +
"elements - first one is the name of edge input class, and second " +
"one is JSON object with all specific parameters for this edge " +
"input. For example: [[\"EIF1\",{\"p\":\"v1\"}]," +
"[\"EIF2\",{\"p\":\"v2\",\"q\":\"v\"}]]");
Hope that helps,
Avery
On 8/16/13 8:45 AM, Yasser Altowim wrote:
Guys, any help with this will be appreciated. Thanks.
From: Yasser Altowim [mailto:yasser.altowim@ericsson.com]
Sent: Thursday, August 15, 2013 2:07 PM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: MultiVertexInputFormat
Hi,
I am implementing an algorithm using Giraph. My algorithm needs to read input data from two files, each has its own format. My questions are:
1. How can I use the MultiVertexInputFormat class? Is there any example that shows how this class can be used?
2. How can I specify this class when running my job using the Giraph Runner or using a driver class?
Thanks in advance.
Best,
Yasser
RE: MultiVertexInputFormat
Posted by Yasser Altowim <ya...@ericsson.com>.
Thanks a lot Avery for your response. I am now using VertexInputFormatDescription, but still could not figure out how to set the Vertex input path. I just need to read the vertex values from two different files, each with its own format. I am not using any EdgeInputFormatClass.
Can you please take a look at my code below and show me how to set the Vertex Input Path? Thanks
if (null == getConf()) {
conf = new Configuration();
}
GiraphConfiguration gconf = new GiraphConfiguration(getConf());
int workers = Integer.parseInt(arg0[2]);
gconf.setWorkerConfiguration(workers, workers, 100.0f);
List<VertexInputFormatDescription> vertexInputDescriptions = Lists.newArrayList();
// Input one
VertexInputFormatDescription description1 = new VertexInputFormatDescription(UseCase1FirstVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description1);
// Input two
VertexInputFormatDescription description2 = new VertexInputFormatDescription(UseCase1SecondVertexInputFormat.class);
// how to set the vertex input path?
vertexInputDescriptions.add(description2);
VertexInputFormatDescription.VERTEX_INPUT_FORMAT_DESCRIPTIONS.set(gconf,InputFormatDescription.toJsonString(vertexInputDescriptions));
gconf.setVertexOutputFormatClass(UseCase1OutputFormat.class);
gconf.setComputationClass(UseCase1Vertex.class);
GiraphJob job = new GiraphJob(gconf, "Use Case 1");
FileOutputFormat.setOutputPath(job.getInternalJob(), new Path(arg0[1]));
return job.run(true) ? 0 : -1;
Best,
Yasser
From: Avery Ching [mailto:aching@apache.org]
Sent: Friday, August 16, 2013 9:50 AM
To: user@giraph.apache.org
Subject: Re: MultiVertexInputFormat
This is doable in Giraph, you can use as many vertex or edge input formats as you like (via GIRAPH-639). You just need to choose MultiVertexInputFormat and/or MultiEdgeInputFromat
See VertexInputFormatDescription for vertex input formats
/**
* VertexInputFormats description - JSON array containing a JSON array for
* each vertex input. Vertex input JSON arrays contain one or two elements -
* first one is the name of vertex input class, and second one is JSON object
* with all specific parameters for this vertex input. For example:
* [["VIF1",{"p":"v1"}],["VIF2",{"p":"v2","q":"v"}]]
*/
public static final StrConfOption VERTEX_INPUT_FORMAT_DESCRIPTIONS =
new StrConfOption("giraph.multiVertexInput.descriptions", null,
"VertexInputFormats description - JSON array containing a JSON " +
"array for each vertex input. Vertex input JSON arrays contain " +
"one or two elements - first one is the name of vertex input " +
"class, and second one is JSON object with all specific parameters " +
"for this vertex input. For example: [[\"VIF1\",{\"p\":\"v1\"}]," +
"[\"VIF2\",{\"p\":\"v2\",\"q\":\"v\"}]]\"");
See EdgeInputFormatDescription for edge input formats
/**
* EdgeInputFormats description - JSON array containing a JSON array for
* each edge input. Edge input JSON arrays contain one or two elements -
* first one is the name of edge input class, and second one is JSON object
* with all specific parameters for this edge input. For example:
* [["EIF1",{"p":"v1"}],["EIF2",{"p":"v2","q":"v"}]]
*/
public static final StrConfOption EDGE_INPUT_FORMAT_DESCRIPTIONS =
new StrConfOption("giraph.multiEdgeInput.descriptions", null,
"EdgeInputFormats description - JSON array containing a JSON array " +
"for each edge input. Edge input JSON arrays contain one or two " +
"elements - first one is the name of edge input class, and second " +
"one is JSON object with all specific parameters for this edge " +
"input. For example: [[\"EIF1\",{\"p\":\"v1\"}]," +
"[\"EIF2\",{\"p\":\"v2\",\"q\":\"v\"}]]");
Hope that helps,
Avery
On 8/16/13 8:45 AM, Yasser Altowim wrote:
Guys, any help with this will be appreciated. Thanks.
From: Yasser Altowim [mailto:yasser.altowim@ericsson.com]
Sent: Thursday, August 15, 2013 2:07 PM
To: user@giraph.apache.org<ma...@giraph.apache.org>
Subject: MultiVertexInputFormat
Hi,
I am implementing an algorithm using Giraph. My algorithm needs to read input data from two files, each has its own format. My questions are:
1. How can I use the MultiVertexInputFormat class? Is there any example that shows how this class can be used?
2. How can I specify this class when running my job using the Giraph Runner or using a driver class?
Thanks in advance.
Best,
Yasser
Re: MultiVertexInputFormat
Posted by Avery Ching <ac...@apache.org>.
This is doable in Giraph, you can use as many vertex or edge input
formats as you like (via GIRAPH-639). You just need to choose
MultiVertexInputFormat and/or MultiEdgeInputFromat
See VertexInputFormatDescription for vertex input formats
/**
* VertexInputFormats description - JSON array containing a JSON
array for
* each vertex input. Vertex input JSON arrays contain one or two
elements -
* first one is the name of vertex input class, and second one is
JSON object
* with all specific parameters for this vertex input. For example:
* [["VIF1",{"p":"v1"}],["VIF2",{"p":"v2","q":"v"}]]
*/
public static final StrConfOption VERTEX_INPUT_FORMAT_DESCRIPTIONS =
new StrConfOption("giraph.multiVertexInput.descriptions", null,
"VertexInputFormats description - JSON array containing a
JSON " +
"array for each vertex input. Vertex input JSON arrays
contain " +
"one or two elements - first one is the name of vertex input " +
"class, and second one is JSON object with all specific
parameters " +
"for this vertex input. For example:
[[\"VIF1\",{\"p\":\"v1\"}]," +
"[\"VIF2\",{\"p\":\"v2\",\"q\":\"v\"}]]\"");
See EdgeInputFormatDescription for edge input formats
/**
* EdgeInputFormats description - JSON array containing a JSON array for
* each edge input. Edge input JSON arrays contain one or two elements -
* first one is the name of edge input class, and second one is JSON
object
* with all specific parameters for this edge input. For example:
* [["EIF1",{"p":"v1"}],["EIF2",{"p":"v2","q":"v"}]]
*/
public static final StrConfOption EDGE_INPUT_FORMAT_DESCRIPTIONS =
new StrConfOption("giraph.multiEdgeInput.descriptions", null,
"EdgeInputFormats description - JSON array containing a JSON
array " +
"for each edge input. Edge input JSON arrays contain one or
two " +
"elements - first one is the name of edge input class, and
second " +
"one is JSON object with all specific parameters for this
edge " +
"input. For example: [[\"EIF1\",{\"p\":\"v1\"}]," +
"[\"EIF2\",{\"p\":\"v2\",\"q\":\"v\"}]]");
Hope that helps,
Avery
On 8/16/13 8:45 AM, Yasser Altowim wrote:
>
> Guys, any help with this will be appreciated. Thanks.
>
> *From:*Yasser Altowim [mailto:yasser.altowim@ericsson.com]
> *Sent:* Thursday, August 15, 2013 2:07 PM
> *To:* user@giraph.apache.org
> *Subject:* MultiVertexInputFormat
>
> Hi,
>
> I am implementing an algorithm using Giraph. My
> algorithm needs to read input data from two files, each has its own
> format. My questions are:
>
> 1.How can I use the MultiVertexInputFormat class? Is there any example
> that shows how this class can be used?
>
> 2.How can I specify this class when running my job using the Giraph
> Runner or using a driver class?
>
> Thanks in advance.
>
> *Best,*
>
> *Yasser*
>
RE: MultiVertexInputFormat
Posted by Yasser Altowim <ya...@ericsson.com>.
Guys, any help with this will be appreciated. Thanks.
From: Yasser Altowim [mailto:yasser.altowim@ericsson.com]
Sent: Thursday, August 15, 2013 2:07 PM
To: user@giraph.apache.org
Subject: MultiVertexInputFormat
Hi,
I am implementing an algorithm using Giraph. My algorithm needs to read input data from two files, each has its own format. My questions are:
1. How can I use the MultiVertexInputFormat class? Is there any example that shows how this class can be used?
2. How can I specify this class when running my job using the Giraph Runner or using a driver class?
Thanks in advance.
Best,
Yasser