You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Kyle Orlando <ky...@gmail.com> on 2013/07/24 17:19:28 UTC
How to retrieve and display the values aggregated by the aggregators?
Hello,
I am new to Giraph and was just wondering how one could retrieve and
display the certain global values/statistics that the aggregators keep
track of. What classes and methods would I use, and would this be
done in a class that extends VertexOutputFormat, or would it be done
elsewhere?
As an example, in the provided SimplePageRankComputation in
org.apache.giraph.examples, there are three aggregators: sum, min, and
max. I would like to display all of their final values (after the
final superstep) in some way, such as writing them to a text file.
--
Kyle Orlando
Computer Engineering Major
University of Maryland
Re: How to retrieve and display the values aggregated by the aggregators?
Posted by Kyle Orlando <ky...@gmail.com>.
Will do, I just need to figure out how to submit the patch exactly, as
well as having some extra time.
And that worked, actually. I'd already figured it out without
realizing it; the text file with the aggregator values was created, it
was just placed somewhere on my HDFS that I wasn't expecting.
On Wed, Jul 31, 2013 at 12:46 PM, Claudio Martella
<cl...@gmail.com> wrote:
> Hi Kyle,
>
> good catch. ALWAYS should be set to 1. Want to write a patch to fix this?
> Try to set the property on the command line by putting -D
> giraph.textAggregatorWriter.frequency=-1 right after the GiraphRunner class
> in your command line.
>
> Hope this helps.
>
> Best,
> Claudio
>
>
> On Wed, Jul 24, 2013 at 10:31 PM, Kyle Orlando <ky...@gmail.com>
> wrote:
>>
>> Hi Claudio,
>>
>> So I checked out TextAggregatorWriter and was, initially, still a bit
>> confused on how to use it to write to a text file. That's when I
>> noticed that, in org.apache.giraph.utils.ConfigurationUtils, there is
>> an option "aw", which corresponds to an AggregatorWriterClass. I
>> tried this out when running the SimplePageRankComputation program
>> using my data as input by specifying this as an option:
>>
>> -aw org.apache.giraph.aggregators.TextAggregatorWriter.
>>
>> Here's the full command:
>>
>> hadoop jar /home/hduser/Documents/combined.jar
>> org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimplePageRankComputation -eif
>> StackExchangeParsee.StackExchangeLongFloatTextEdgeInput -vif
>> StackExchangeParsee.StackExchangeLongDoubleTextVertexValueInput -eip
>> /in/gaming_edges.txt -vip /in/gaming_vertices.txt -of
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -aw
>> org.apache.giraph.aggregators.TextAggregatorWriter -op /outPR -w 2 -mc
>>
>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>>
>> According to TextAggregatorWriter, it, by default, writes to a file
>> called "aggregatorValues". I checked my HDFS, and did not see that
>> particular file. That's when I noticed that there is a configuration
>> "giraph.textAggregatorWriter.frequency", and that by default the
>> frequency is set to "NEVER", which means that nothing is ever
>> created/written to a file for the aggregators. The other two
>> frequencies are "AT_THE_END" and "ALWAYS", which strangely both
>> correspond to the same integer: -1. Could someone explain why this is
>> so?
>>
>> Ignoring the above uncertainty, I surmised that the property
>> "giraph.textAggregatorWriter.frequency" was to be added to my
>> giraph-site.xml. I wanted the "AT_THE_END" frequency, which
>> corresponds to the value of -1. Here's the contents of my
>> giraph-site.xml file:
>>
>> <configuration>
>> <property>
>> <name>giraph.textAggregatorWriter.frequency</name>
>> <value>-1</value>
>> </property>
>> </configuration>
>>
>> I ran the SimplePageRankComputation program again (using the verbose
>> "hadoop jar" command above), and still, I couldn't find
>> "aggregatorValues" on my HDFS.
>>
>> Could someone help me out, or at the very least rectify any
>> misconceptions and uncertainties that I have?
>>
>>
>>
>> On Wed, Jul 24, 2013 at 12:25 PM, Claudio Martella
>> <cl...@gmail.com> wrote:
>> > Hi Kyle,
>> >
>> > you can check out the AggregatorWriter interface which allows you to do
>> > that. As a matter of fact there is already a class that implements what
>> > you
>> > need (org.apache.giraph.aggregators.TextAggregatorWriter).
>> >
>> > Hope it helps.
>> >
>> >
>> > On Wed, Jul 24, 2013 at 5:19 PM, Kyle Orlando <ky...@gmail.com>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> I am new to Giraph and was just wondering how one could retrieve and
>> >> display the certain global values/statistics that the aggregators keep
>> >> track of. What classes and methods would I use, and would this be
>> >> done in a class that extends VertexOutputFormat, or would it be done
>> >> elsewhere?
>> >>
>> >> As an example, in the provided SimplePageRankComputation in
>> >> org.apache.giraph.examples, there are three aggregators: sum, min, and
>> >> max. I would like to display all of their final values (after the
>> >> final superstep) in some way, such as writing them to a text file.
>> >>
>> >> --
>> >> Kyle Orlando
>> >> Computer Engineering Major
>> >> University of Maryland
>> >
>> >
>> >
>> >
>> > --
>> > Claudio Martella
>> > claudio.martella@gmail.com
>>
>>
>>
>> --
>> Kyle Orlando
>> Computer Engineering Major
>> University of Maryland
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
--
Kyle Orlando
Computer Engineering Major
University of Maryland
Re: How to retrieve and display the values aggregated by the aggregators?
Posted by Claudio Martella <cl...@gmail.com>.
Hi Kyle,
good catch. ALWAYS should be set to 1. Want to write a patch to fix this?
Try to set the property on the command line by putting -D
giraph.textAggregatorWriter.frequency=-1 right after the GiraphRunner class
in your command line.
Hope this helps.
Best,
Claudio
On Wed, Jul 24, 2013 at 10:31 PM, Kyle Orlando <ky...@gmail.com>wrote:
> Hi Claudio,
>
> So I checked out TextAggregatorWriter and was, initially, still a bit
> confused on how to use it to write to a text file. That's when I
> noticed that, in org.apache.giraph.utils.ConfigurationUtils, there is
> an option "aw", which corresponds to an AggregatorWriterClass. I
> tried this out when running the SimplePageRankComputation program
> using my data as input by specifying this as an option:
>
> -aw org.apache.giraph.aggregators.TextAggregatorWriter.
>
> Here's the full command:
>
> hadoop jar /home/hduser/Documents/combined.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimplePageRankComputation -eif
> StackExchangeParsee.StackExchangeLongFloatTextEdgeInput -vif
> StackExchangeParsee.StackExchangeLongDoubleTextVertexValueInput -eip
> /in/gaming_edges.txt -vip /in/gaming_vertices.txt -of
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -aw
> org.apache.giraph.aggregators.TextAggregatorWriter -op /outPR -w 2 -mc
>
> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
>
> According to TextAggregatorWriter, it, by default, writes to a file
> called "aggregatorValues". I checked my HDFS, and did not see that
> particular file. That's when I noticed that there is a configuration
> "giraph.textAggregatorWriter.frequency", and that by default the
> frequency is set to "NEVER", which means that nothing is ever
> created/written to a file for the aggregators. The other two
> frequencies are "AT_THE_END" and "ALWAYS", which strangely both
> correspond to the same integer: -1. Could someone explain why this is
> so?
>
> Ignoring the above uncertainty, I surmised that the property
> "giraph.textAggregatorWriter.frequency" was to be added to my
> giraph-site.xml. I wanted the "AT_THE_END" frequency, which
> corresponds to the value of -1. Here's the contents of my
> giraph-site.xml file:
>
> <configuration>
> <property>
> <name>giraph.textAggregatorWriter.frequency</name>
> <value>-1</value>
> </property>
> </configuration>
>
> I ran the SimplePageRankComputation program again (using the verbose
> "hadoop jar" command above), and still, I couldn't find
> "aggregatorValues" on my HDFS.
>
> Could someone help me out, or at the very least rectify any
> misconceptions and uncertainties that I have?
>
>
>
> On Wed, Jul 24, 2013 at 12:25 PM, Claudio Martella
> <cl...@gmail.com> wrote:
> > Hi Kyle,
> >
> > you can check out the AggregatorWriter interface which allows you to do
> > that. As a matter of fact there is already a class that implements what
> you
> > need (org.apache.giraph.aggregators.TextAggregatorWriter).
> >
> > Hope it helps.
> >
> >
> > On Wed, Jul 24, 2013 at 5:19 PM, Kyle Orlando <ky...@gmail.com>
> > wrote:
> >>
> >> Hello,
> >>
> >> I am new to Giraph and was just wondering how one could retrieve and
> >> display the certain global values/statistics that the aggregators keep
> >> track of. What classes and methods would I use, and would this be
> >> done in a class that extends VertexOutputFormat, or would it be done
> >> elsewhere?
> >>
> >> As an example, in the provided SimplePageRankComputation in
> >> org.apache.giraph.examples, there are three aggregators: sum, min, and
> >> max. I would like to display all of their final values (after the
> >> final superstep) in some way, such as writing them to a text file.
> >>
> >> --
> >> Kyle Orlando
> >> Computer Engineering Major
> >> University of Maryland
> >
> >
> >
> >
> > --
> > Claudio Martella
> > claudio.martella@gmail.com
>
>
>
> --
> Kyle Orlando
> Computer Engineering Major
> University of Maryland
>
--
Claudio Martella
claudio.martella@gmail.com
Re: How to retrieve and display the values aggregated by the aggregators?
Posted by Kyle Orlando <ky...@gmail.com>.
Hi Claudio,
So I checked out TextAggregatorWriter and was, initially, still a bit
confused on how to use it to write to a text file. That's when I
noticed that, in org.apache.giraph.utils.ConfigurationUtils, there is
an option "aw", which corresponds to an AggregatorWriterClass. I
tried this out when running the SimplePageRankComputation program
using my data as input by specifying this as an option:
-aw org.apache.giraph.aggregators.TextAggregatorWriter.
Here's the full command:
hadoop jar /home/hduser/Documents/combined.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimplePageRankComputation -eif
StackExchangeParsee.StackExchangeLongFloatTextEdgeInput -vif
StackExchangeParsee.StackExchangeLongDoubleTextVertexValueInput -eip
/in/gaming_edges.txt -vip /in/gaming_vertices.txt -of
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -aw
org.apache.giraph.aggregators.TextAggregatorWriter -op /outPR -w 2 -mc
org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute
According to TextAggregatorWriter, it, by default, writes to a file
called "aggregatorValues". I checked my HDFS, and did not see that
particular file. That's when I noticed that there is a configuration
"giraph.textAggregatorWriter.frequency", and that by default the
frequency is set to "NEVER", which means that nothing is ever
created/written to a file for the aggregators. The other two
frequencies are "AT_THE_END" and "ALWAYS", which strangely both
correspond to the same integer: -1. Could someone explain why this is
so?
Ignoring the above uncertainty, I surmised that the property
"giraph.textAggregatorWriter.frequency" was to be added to my
giraph-site.xml. I wanted the "AT_THE_END" frequency, which
corresponds to the value of -1. Here's the contents of my
giraph-site.xml file:
<configuration>
<property>
<name>giraph.textAggregatorWriter.frequency</name>
<value>-1</value>
</property>
</configuration>
I ran the SimplePageRankComputation program again (using the verbose
"hadoop jar" command above), and still, I couldn't find
"aggregatorValues" on my HDFS.
Could someone help me out, or at the very least rectify any
misconceptions and uncertainties that I have?
On Wed, Jul 24, 2013 at 12:25 PM, Claudio Martella
<cl...@gmail.com> wrote:
> Hi Kyle,
>
> you can check out the AggregatorWriter interface which allows you to do
> that. As a matter of fact there is already a class that implements what you
> need (org.apache.giraph.aggregators.TextAggregatorWriter).
>
> Hope it helps.
>
>
> On Wed, Jul 24, 2013 at 5:19 PM, Kyle Orlando <ky...@gmail.com>
> wrote:
>>
>> Hello,
>>
>> I am new to Giraph and was just wondering how one could retrieve and
>> display the certain global values/statistics that the aggregators keep
>> track of. What classes and methods would I use, and would this be
>> done in a class that extends VertexOutputFormat, or would it be done
>> elsewhere?
>>
>> As an example, in the provided SimplePageRankComputation in
>> org.apache.giraph.examples, there are three aggregators: sum, min, and
>> max. I would like to display all of their final values (after the
>> final superstep) in some way, such as writing them to a text file.
>>
>> --
>> Kyle Orlando
>> Computer Engineering Major
>> University of Maryland
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
--
Kyle Orlando
Computer Engineering Major
University of Maryland
Re: How to retrieve and display the values aggregated by the aggregators?
Posted by Claudio Martella <cl...@gmail.com>.
Hi Kyle,
you can check out the AggregatorWriter interface which allows you to do
that. As a matter of fact there is already a class that implements what you
need (org.apache.giraph.aggregators.TextAggregatorWriter).
Hope it helps.
On Wed, Jul 24, 2013 at 5:19 PM, Kyle Orlando <ky...@gmail.com>wrote:
> Hello,
>
> I am new to Giraph and was just wondering how one could retrieve and
> display the certain global values/statistics that the aggregators keep
> track of. What classes and methods would I use, and would this be
> done in a class that extends VertexOutputFormat, or would it be done
> elsewhere?
>
> As an example, in the provided SimplePageRankComputation in
> org.apache.giraph.examples, there are three aggregators: sum, min, and
> max. I would like to display all of their final values (after the
> final superstep) in some way, such as writing them to a text file.
>
> --
> Kyle Orlando
> Computer Engineering Major
> University of Maryland
>
--
Claudio Martella
claudio.martella@gmail.com