You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amar Kamat (JIRA)" <ji...@apache.org> on 2008/08/22 11:52:44 UTC

[jira] Created: (HADOOP-4000) Reduce progress in the Reducer crosses 1

Reduce progress in the Reducer crosses 1
----------------------------------------

                 Key: HADOOP-4000
                 URL: https://issues.apache.org/jira/browse/HADOOP-4000
             Project: Hadoop Core
          Issue Type: Bug
          Components: mapred
            Reporter: Amar Kamat


I printed the progress in the {{Progress.java}} for every progress made and I found this scenario
{noformat}
update called for reduce with progress 0.99998355
update called for reduce with progress 0.99998355
update called for reduce with progress 1.0000043
update called for reduce with progress 1.0000043
update called for reduce with progress 1.0000244
{noformat}
Following are the lines I added to get this 
{code:title=Progress.java|borderStyle=solid}
/** Called during execution on a leaf node to set its progress. */
  public synchronized void set(float progress) {
    System.out.println("update called for " + status + " with progress " + progress);
    this.progress = progress;
  }
{code}

This needs investigation and will also explain why occasionally users see jobs that are complete and their progress less that 100%.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-4000) Reduce progress in the Reducer crosses 1

Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672848#action_12672848 ] 

Billy Pearson commented on HADOOP-4000:
---------------------------------------

I see this also when using streaming. the reduce map disappears because over 100% task

> Reduce progress in the Reducer crosses 1
> ----------------------------------------
>
>                 Key: HADOOP-4000
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4000
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Amar Kamat
>
> I printed the progress in the {{Progress.java}} for every progress made and I found this scenario
> {noformat}
> update called for reduce with progress 0.99998355
> update called for reduce with progress 0.99998355
> update called for reduce with progress 1.0000043
> update called for reduce with progress 1.0000043
> update called for reduce with progress 1.0000244
> {noformat}
> Following are the lines I added to get this 
> {code:title=Progress.java|borderStyle=solid}
> /** Called during execution on a leaf node to set its progress. */
>   public synchronized void set(float progress) {
>     System.out.println("update called for " + status + " with progress " + progress);
>     this.progress = progress;
>   }
> {code}
> This needs investigation and will also explain why occasionally users see jobs that are complete and their progress less that 100%.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.