You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Srikanth Sundarrajan <sr...@hotmail.com> on 2015/07/02 10:17:30 UTC
Re: Review Request 34161: Falcon Native Scheduler - Refactor existing
workflow/coord/bundle builder
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34161/#review89668
-----------------------------------------------------------
oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java (line 180)
<https://reviews.apache.org/r/34161/#comment142328>
Why is generic entity builder concerned about replication. Is there no other way to handle this ?
oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java (line 181)
<https://reviews.apache.org/r/34161/#comment143206>
Why is generic entity builder concerned about replication. Is there no other way to handle this ?
- Srikanth Sundarrajan
On June 30, 2015, 4:40 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34161/
> -----------------------------------------------------------
>
> (Updated June 30, 2015, 4:40 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1170
> https://issues.apache.org/jira/browse/FALCON-1170
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The current set of builders assume that a bundle is going to be submitted to Oozie, and hence, populate configuration properties at the coord level. When a native scheduler is built out, we will submitting workflows to Oozie (not coords/bundles) as the scheduling of the workflows will be done within Falcon.
>
> Re-factored the builder code as follows:
> 1. Generate a config-default.xml along with workflow.xml whenever a workflow is built. The default config file will have most of the configurations set in it. Note that the config values can be overridden during run/re-run using job.properties.
> 2. Move all workflow related configuration to config-default.xml and keep only coord related variables (input, output, timestamp, startTime, endTime, frequency) in the coordinator.xml
> 3. Make bundle.xml really lean. Only app path, entity name and path will be set there.
> 4. Mostly code (methods) moved around to ensure the config is populated at the right level.
>
>
> Diffs
> -----
>
> oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 03063f4
> oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java e5d75fb
> oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java f00290e
> oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 49f9e07
> oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java 1d97204
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java de6f373
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java aa936ad
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java c896d5a
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java 51e081f
> oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java 72bbca4
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java 60f9fe1
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 8b18ecc
> oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 0ae229c
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java b223447
> oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java 6488682
> oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 4e5c3f0
>
> Diff: https://reviews.apache.org/r/34161/diff/
>
>
> Testing
> -------
>
> UTs modified. ITs run and manual testing done using falcon examples.
>
>
> Thanks,
>
> Pallavi Rao
>
>