[jira] [Created] (CRUNCH-237) Improper job dependencies for certain types of long pipelines

["Josh Wills (JIRA)" <ji...@apache.org>]

Josh Wills created CRUNCH-237:
---------------------------------

             Summary: Improper job dependencies for certain types of long pipelines
                 Key: CRUNCH-237
                 URL: https://issues.apache.org/jira/browse/CRUNCH-237
             Project: Crunch
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.6.0
            Reporter: Josh Wills
            Assignee: Josh Wills
             Fix For: 0.7.0


The Crunch planner analyzes the dependencies between different phases of a MapReduce pipeline and uses those dependencies to ensure that the MapReduce jobs in the pipeline are executed in the correct sequence. For certain kinds of long pipelines, it's possible for the planner to miss a necessary dependency as follows:

Pipeline spec: [Input] -> GBK -> [Out1] -> (GBK) -> (Out2) -> GBK -> [Out3]

This pipeline has two explicit outputs (Out1 and Out3) and one implicit output (Out2). Additionally, assume that there is a map-side join between Out1 that happens in the map stage of the job that creates Out2. For this pipeline, the planner will mark a dependency between the job that creates Out1 and the job that creates Out3, but NOT between the job that creates Out1 and the job that creates Out2. This makes it possible for the Out2 job to run before Out1 is created, causing a failure.

The easiest way to fix this I could see was to add a step to the dependency chain such that every job that is created in a later stage of pipeline creation depends on all of the jobs in the earlier stages having been run, which is what the attached patch and example integration test demonstrate.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira