[jira] Created: (HADOOP-223) Should be able to re-run jobs, collecting only missing output

["Bryan Pendleton (JIRA)" <ji...@apache.org>]

Should be able to re-run jobs, collecting only missing output
-------------------------------------------------------------

         Key: HADOOP-223
         URL: http://issues.apache.org/jira/browse/HADOOP-223
     Project: Hadoop
        Type: New Feature

  Components: mapred  
    Reporter: Bryan Pendleton
    Priority: Minor


For jobs with no side effects (roughly == jobs with speculative execution enabled), if partial output has been generated, it should be possible to re-run the job, and fill in the missing pieces. I have now run the same job twice, once finishing 42 of 44 reduce tasks, another time finishing only 17. Each time, many nodes have failed, causing many many tasks to fail ( in one case, 5k failures from 15k map tasks, 23 failures from 44 reduces), but some valid output was generated. Since the output is only dependent on the input, and both jobs used the same input, I will now be able to combine these two failed task outputs to get a completed job's output. This should be something that can be more automatic.

In particular, it should be possible to resubmit a job, with a list of partitions that should be ignored. A special Combiner, or pre-Combiner, would throw out any map output for partitions that have already been successfully completed, thus reducing the amount of data that needs to be reduced to complete the job. It would, of course, be nice to support "filling in" existing outputs, rather than having to do a move operation on completed outputs.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

[jira] Commented: (HADOOP-223) Should be able to re-run jobs, collecting only missing output

["Bryan Pendleton (JIRA)" <ji...@apache.org>]

    [ http://issues.apache.org/jira/browse/HADOOP-223?page=comments#action_12432196 ] 
            
Bryan Pendleton commented on HADOOP-223:
----------------------------------------

This issue still plagues me. I have a large enough dataset, that processing through it and storing temporary outputs often causes many failures. I still have my local fix for re-running (very cumbersome). I'd love to see a general solution supported.

What I currently do is to inject a special combiner, that only produces output for partitions that didn't complete in a previous run. Thus, if 20% of my job finishes (completes writing reduce output to DFS), I need 20% less working space to run the subsequent job attempt. Usually, this is enough progress.

However, that suffers from a lot of limits:
1) There's no auto-magic storage of what partitions were generated.
2) This loses the ability to have a Combiner doing something else
3) It also requires a new output directory, and hand-reassembly of the completed tasks.

It seems like it should be solvable more generally by:
1) Storing the partitions that are generated when a job is initially run.
2) Keeping track of what partitions have completed.
3) Allowing a job to be re-submitted, using the previous partitions (and not failing on the output existance check)

Anyone else run into these issues? I'd be happy to write up how I actually work around the problem now, but it seems like a more general solution would be helpful to a wider audience.

> Should be able to re-run jobs, collecting only missing output
> -------------------------------------------------------------
>
>                 Key: HADOOP-223
>                 URL: http://issues.apache.org/jira/browse/HADOOP-223
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Bryan Pendleton
>            Priority: Minor
>
> For jobs with no side effects (roughly == jobs with speculative execution enabled), if partial output has been generated, it should be possible to re-run the job, and fill in the missing pieces. I have now run the same job twice, once finishing 42 of 44 reduce tasks, another time finishing only 17. Each time, many nodes have failed, causing many many tasks to fail ( in one case, 5k failures from 15k map tasks, 23 failures from 44 reduces), but some valid output was generated. Since the output is only dependent on the input, and both jobs used the same input, I will now be able to combine these two failed task outputs to get a completed job's output. This should be something that can be more automatic.
> In particular, it should be possible to resubmit a job, with a list of partitions that should be ignored. A special Combiner, or pre-Combiner, would throw out any map output for partitions that have already been successfully completed, thus reducing the amount of data that needs to be reduced to complete the job. It would, of course, be nice to support "filling in" existing outputs, rather than having to do a move operation on completed outputs.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira