Re: InputFormat for the example SimpleMasterComputeVertex

[Eli Reisman <ap...@gmail.com>]

Thanks for doing this Maja!

On Thu, Feb 21, 2013 at 5:12 PM, Maja Kabiljo <ma...@fb.com> wrote:

>  I wrote up a basic info about aggregators:
> https://cwiki.apache.org/confluence/display/GIRAPH/Aggregators. Please
> take a look, and let me know if something needs to be changed / improved.
>
>   From: Eli Reisman <ap...@gmail.com>
> Reply-To: "user@giraph.apache.org" <us...@giraph.apache.org>
> Date: Thursday, February 21, 2013 4:21 PM
>
> To: "user@giraph.apache.org" <us...@giraph.apache.org>
> Subject: Re: InputFormat for the example SimpleMasterComputeVertex
>
>  That sounds great to me, maybe just a mention in the wiki that the two
> functionalities are tied together will help the idea click for people.
> Either way this will be a big help I think.
>
>
> On Thu, Feb 21, 2013 at 3:24 PM, Maja Kabiljo <ma...@fb.com> wrote:
>
>>  Eli, that's an interesting idea, we could have some class which user
>> extends and which is there only for aggregator registration. Sometimes we
>> want to register some aggregators later on during the computation, so we
>> need to keep allowing registration from masterCompute too.
>>
>>  But I think for users the biggest problem is to realize that they have
>> to extend and set MasterCompute/this new class in order to use aggregators.
>> Currently, if user tries to aggregate a value to unregistered aggregator he
>> will get an exception, but if he tries to get the value of unregistered
>> aggregator he will just get null. So maybe adding a warning message in that
>> case, together with a wiki page, might be enough? What do you think?
>>
>>   From: Eli Reisman <ap...@gmail.com>
>> Reply-To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>  Date: Thursday, February 21, 2013 10:25 AM
>>
>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>> Subject: Re: InputFormat for the example SimpleMasterComputeVertex
>>
>>  Thanks for the explanation, that makes sense. I would love to see a
>> wiki page at some point, you have so much knowledge of this piece of Giraph
>> from all your dev work on it and have also the additional bonus of
>> experience running big cluster jobs using these features so you have a lot
>> of insight to share.
>>
>>  Would there be any point to a future JIRA to break out the aggregator
>> registration from the master compute stuff, at least from the user's view?
>> Or is it not that confusing once you've used them a bit?
>>
>>
>> On Thu, Feb 14, 2013 at 4:52 PM, Maja Kabiljo <ma...@fb.com> wrote:
>>
>>>  Progressable exception can be caused by many different reasons (it's
>>> totally unrelated to aggregators), and when looking at which exception it's
>>> caused by users should get better sense about what's going on.
>>> What you are suggesting about providing default master compute is not
>>> doable, since the part which needs to be done there is aggregator
>>> registration. We can't know what kind of aggregators (names and types) an
>>> application needs.
>>> I remember I was talking about writing a short tutorial for aggregators
>>> long time ago, sorry for not doing that, will try to get to it soon.
>>>
>>>   From: Eli Reisman <ap...@gmail.com>
>>> Reply-To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>> Date: Thursday, February 14, 2013 2:23 PM
>>>
>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>> Subject: Re: InputFormat for the example SimpleMasterComputeVertex
>>>
>>>   Other folks on the list are also having this problem with the
>>> progressable utile exception & job failures. I don't know much about master
>>> compute usage but if it is needed to make the aggregators work, maybe we
>>> should have a default dummy class that just handles aggregators if no other
>>> master compute is specified? Or a wiki page? The progressable error message
>>> does not lead us to this conclusion directly.
>>>
>>> On Wed, Feb 13, 2013 at 3:04 AM, Maria Stylianou <ma...@gmail.com>wrote:
>>>
>>>> Hey,
>>>>
>>>>  I am trying to run the example SimpleMasterComputeVertex, but no
>>>> matter which Input Format and graph I give, it doesn't work. Each worker
>>>> gives the error:
>>>>
>>>> Caused by: java.lang.*NullPointerException*
>>>> 	at org.apache.giraph.examples.*SimpleMasterComputeVertex.compute*(SimpleMasterComputeVertex.java:*42*)
>>>>
>>>> This line 42 is the first line of the compute()
>>>> public void compute(Iterable<DoubleWritable> messages){
>>>>
>>>>  So I guess, the initialization is not done correctly, because the
>>>> input file does not have the correct format.
>>>>
>>>>  Any help would be appreciated,
>>>> Thanks!
>>>>  Maria
>>>> --
>>>> Maria Stylianou
>>>> Intern at Telefonica, Barcelona, Spain
>>>> Master Student of European Master in Distributed Computing<http://www.kth.se/en/studies/programmes/master/em/emdc>
>>>> Universitat Politècnica de Catalunya - BarcelonaTech, Barcelona, Spain
>>>>  KTH Royal Institute of Technology, Stockholm, Sweden
>>>>
>>>>
>>>
>>
>

Re: InputFormat for the example SimpleMasterComputeVertex

[Eli Reisman <ap...@gmail.com>]

Its fantastic thanks again! We need more wiki pages like that!!! This will
save so many emails on the list too! I need to (and will) update the graph
representations page to say we support multigraph, edge and vertex IO, etc.
soon. A lot of our wiki info is old enough to be confusing people now.

On Sat, Mar 2, 2013 at 1:57 PM, Eli Reisman <ap...@gmail.com>wrote:

> Thanks for doing this Maja!
>
>
> On Thu, Feb 21, 2013 at 5:12 PM, Maja Kabiljo <ma...@fb.com> wrote:
>
>>  I wrote up a basic info about aggregators:
>> https://cwiki.apache.org/confluence/display/GIRAPH/Aggregators. Please
>> take a look, and let me know if something needs to be changed / improved.
>>
>>   From: Eli Reisman <ap...@gmail.com>
>> Reply-To: "user@giraph.apache.org" <us...@giraph.apache.org>
>> Date: Thursday, February 21, 2013 4:21 PM
>>
>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>> Subject: Re: InputFormat for the example SimpleMasterComputeVertex
>>
>>  That sounds great to me, maybe just a mention in the wiki that the two
>> functionalities are tied together will help the idea click for people.
>> Either way this will be a big help I think.
>>
>>
>> On Thu, Feb 21, 2013 at 3:24 PM, Maja Kabiljo <ma...@fb.com> wrote:
>>
>>>  Eli, that's an interesting idea, we could have some class which user
>>> extends and which is there only for aggregator registration. Sometimes we
>>> want to register some aggregators later on during the computation, so we
>>> need to keep allowing registration from masterCompute too.
>>>
>>>  But I think for users the biggest problem is to realize that they have
>>> to extend and set MasterCompute/this new class in order to use aggregators.
>>> Currently, if user tries to aggregate a value to unregistered aggregator he
>>> will get an exception, but if he tries to get the value of unregistered
>>> aggregator he will just get null. So maybe adding a warning message in that
>>> case, together with a wiki page, might be enough? What do you think?
>>>
>>>   From: Eli Reisman <ap...@gmail.com>
>>> Reply-To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>  Date: Thursday, February 21, 2013 10:25 AM
>>>
>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>> Subject: Re: InputFormat for the example SimpleMasterComputeVertex
>>>
>>>  Thanks for the explanation, that makes sense. I would love to see a
>>> wiki page at some point, you have so much knowledge of this piece of Giraph
>>> from all your dev work on it and have also the additional bonus of
>>> experience running big cluster jobs using these features so you have a lot
>>> of insight to share.
>>>
>>>  Would there be any point to a future JIRA to break out the aggregator
>>> registration from the master compute stuff, at least from the user's view?
>>> Or is it not that confusing once you've used them a bit?
>>>
>>>
>>> On Thu, Feb 14, 2013 at 4:52 PM, Maja Kabiljo <ma...@fb.com>wrote:
>>>
>>>>  Progressable exception can be caused by many different reasons (it's
>>>> totally unrelated to aggregators), and when looking at which exception it's
>>>> caused by users should get better sense about what's going on.
>>>> What you are suggesting about providing default master compute is not
>>>> doable, since the part which needs to be done there is aggregator
>>>> registration. We can't know what kind of aggregators (names and types) an
>>>> application needs.
>>>> I remember I was talking about writing a short tutorial for aggregators
>>>> long time ago, sorry for not doing that, will try to get to it soon.
>>>>
>>>>   From: Eli Reisman <ap...@gmail.com>
>>>> Reply-To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>> Date: Thursday, February 14, 2013 2:23 PM
>>>>
>>>> To: "user@giraph.apache.org" <us...@giraph.apache.org>
>>>> Subject: Re: InputFormat for the example SimpleMasterComputeVertex
>>>>
>>>>   Other folks on the list are also having this problem with the
>>>> progressable utile exception & job failures. I don't know much about master
>>>> compute usage but if it is needed to make the aggregators work, maybe we
>>>> should have a default dummy class that just handles aggregators if no other
>>>> master compute is specified? Or a wiki page? The progressable error message
>>>> does not lead us to this conclusion directly.
>>>>
>>>> On Wed, Feb 13, 2013 at 3:04 AM, Maria Stylianou <ma...@gmail.com>wrote:
>>>>
>>>>> Hey,
>>>>>
>>>>>  I am trying to run the example SimpleMasterComputeVertex, but no
>>>>> matter which Input Format and graph I give, it doesn't work. Each worker
>>>>> gives the error:
>>>>>
>>>>> Caused by: java.lang.*NullPointerException*
>>>>> 	at org.apache.giraph.examples.*SimpleMasterComputeVertex.compute*(SimpleMasterComputeVertex.java:*42*)
>>>>>
>>>>> This line 42 is the first line of the compute()
>>>>> public void compute(Iterable<DoubleWritable> messages){
>>>>>
>>>>>  So I guess, the initialization is not done correctly, because the
>>>>> input file does not have the correct format.
>>>>>
>>>>>  Any help would be appreciated,
>>>>> Thanks!
>>>>>  Maria
>>>>> --
>>>>> Maria Stylianou
>>>>> Intern at Telefonica, Barcelona, Spain
>>>>> Master Student of European Master in Distributed Computing<http://www.kth.se/en/studies/programmes/master/em/emdc>
>>>>> Universitat Politècnica de Catalunya - BarcelonaTech, Barcelona, Spain
>>>>>  KTH Royal Institute of Technology, Stockholm, Sweden
>>>>>
>>>>>
>>>>
>>>
>>
>