You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Srikanth Sundarrajan <sr...@hotmail.com> on 2015/07/02 10:17:30 UTC

Re: Review Request 34161: Falcon Native Scheduler - Refactor existing workflow/coord/bundle builder

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34161/#review89668
-----------------------------------------------------------



oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java (line 180)
<https://reviews.apache.org/r/34161/#comment142328>

    Why is generic entity builder concerned about replication. Is there no other way to handle this ?



oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java (line 181)
<https://reviews.apache.org/r/34161/#comment143206>

    Why is generic entity builder concerned about replication. Is there no other way to handle this ?


- Srikanth Sundarrajan


On June 30, 2015, 4:40 a.m., Pallavi Rao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34161/
> -----------------------------------------------------------
> 
> (Updated June 30, 2015, 4:40 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1170
>     https://issues.apache.org/jira/browse/FALCON-1170
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> The current set of builders assume that a bundle is going to be submitted to Oozie, and hence, populate configuration properties at the coord level. When a native scheduler is built out, we will submitting workflows to Oozie (not coords/bundles) as the scheduling of the workflows will be done within Falcon.
> 
> Re-factored the builder code as follows:
> 1. Generate a config-default.xml along with workflow.xml whenever a workflow is built. The default config file will have most of the configurations set in it. Note that the config values can be overridden during run/re-run using job.properties.
> 2. Move all workflow related configuration to config-default.xml and keep only coord related variables (input, output, timestamp, startTime, endTime, frequency) in the coordinator.xml
> 3. Make bundle.xml really lean. Only app path, entity name and path will be set there.
> 4. Mostly code (methods) moved around to ensure the config is populated at the right level.
> 
> 
> Diffs
> -----
> 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 03063f4 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java e5d75fb 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java f00290e 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 49f9e07 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java 1d97204 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java de6f373 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java aa936ad 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java c896d5a 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java 51e081f 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java 72bbca4 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java 60f9fe1 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 8b18ecc 
>   oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 0ae229c 
>   oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java b223447 
>   oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java 6488682 
>   oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 4e5c3f0 
> 
> Diff: https://reviews.apache.org/r/34161/diff/
> 
> 
> Testing
> -------
> 
> UTs modified. ITs run and manual testing done using falcon examples.
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>