Giraph still running after mappers are 100% finished ?

[Benjamin Heitmann <be...@deri.org>]

Hello, 

under which circumstances is it possible that a Giraph job, will report that he is 100% finished will his mappers, 
but the job is still running ? 

I can see that it is still running from: 
* debugging messages, 
* list of running threads (in top)
* the hadoop jobtracker web site (it reports 100% completion of mappers, but also that all mappers are running, and none is complete)

I am currently running some benchmarks to get a handle on the scalability of giraph and of my algorithm implementation. 
And the results up to now are very confusing...


Looking forward to any answers, cheers, Benjamin. 

Re: Giraph still running after mappers are 100% finished ?

[Benjamin Heitmann <be...@deri.org>]

Hello Avery, 

On 1 May 2012, at 15:45, Avery Ching wrote:

> I wonder if the issues you are seeing are related to https://issues.apache.org/jira/browse/GIRAPH-169.
> 
> This shouldn't happen.

Good to know that that should not happen. 

For my specific algorithm it happens all the time. 
For small amounts of processing the job finishes 2 minutes after the mappers report a 100%. 
For larger amounts it can take 20 minutes or so. So there is definitively a connection between the expected length of processing the job, 
and the amount of time which passes after the mappers report 100%. 

I even had a pretty extreme case where most of the workers where restarted after an hour, and I killed the job after 90 minutes.

In addition, the "100% map" always comes about 14-15 minutes after starting the job, independent of the total processing time. 
That might be due to the time it takes to read in the data, which is always around 11 minutes for the "vertex input superstep". 
(The data (and its size) which my job reads in order to construct the graph is always the same. Only the "configuration" of the algorithm changes. 
In my case, the configuration consists of the set of start nodes, and the association between different start nodes and user ids). 


Should I attach a zip file of the log directory for the job which restarted most of its workers after an hour ? 
I can attach that to the JIRA issue. 

Re: Giraph still running after mappers are 100% finished ?

[Avery Ching <ac...@apache.org>]

Benjamin,

I wonder if the issues you are seeing are related to 
https://issues.apache.org/jira/browse/GIRAPH-169.

This shouldn't happen.

Avery

On 5/1/12 7:29 AM, Benjamin Heitmann wrote:
> Hello,
>
> under which circumstances is it possible that a Giraph job, will report that he is 100% finished will his mappers,
> but the job is still running ?
>
> I can see that it is still running from:
> * debugging messages,
> * list of running threads (in top)
> * the hadoop jobtracker web site (it reports 100% completion of mappers, but also that all mappers are running, and none is complete)
>
> I am currently running some benchmarks to get a handle on the scalability of giraph and of my algorithm implementation.
> And the results up to now are very confusing...
>
>
> Looking forward to any answers, cheers, Benjamin.