You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by sh...@inmobi.com on 2014/07/23 13:36:45 UTC

Review Request 23844: FALCON-481 Simplify process parent workflow

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/
-----------------------------------------------------------

Review request for Falcon.


Bugs: FALCON-481
    https://issues.apache.org/jira/browse/FALCON-481


Repository: falcon-git


Description
-------

Currently, process parent workflow contains 2 decision nodes which take up oozie processing and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing actions only once so that the definitions are consistent


Diffs
-----

  oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
  oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
  oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 0b582ef 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9 
  oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883 
  oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1 
  oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java e46ae6e 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 2e3a5c1 
  oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
  oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
  oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
  oozie/src/main/resources/workflow/feed/falcon-table-export.hql PRE-CREATION 
  oozie/src/main/resources/workflow/feed/falcon-table-import.hql PRE-CREATION 
  oozie/src/main/resources/workflow/feed/fs-replication-workflow.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/hcat-replication-workflow.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/replication-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/retention-workflow.xml PRE-CREATION 
  oozie/src/main/resources/workflow/post-process.xml PRE-CREATION 
  oozie/src/main/resources/workflow/pre-process.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
  oozie/src/main/resources/workflow/process/hive-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/oozie-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/parent-workflow.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/pig-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
  oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
  oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c 
  oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
  oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 3655af9 
  src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 

Diff: https://reviews.apache.org/r/23844/diff/


Testing
-------

UTs + end to end tests for fs usecases

Pending: hcat testing


Thanks,

shwethags

Re: Review Request 23844: FALCON-481 Simplify process parent workflow

Posted by sh...@inmobi.com.

> On July 30, 2014, 4:30 p.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java, line 172
> > <https://reviews.apache.org/r/23844/diff/4/?file=645654#file645654line172>
> >
> >     Every Action unmarshal seems to be accompanied with a code to retry decoration. Can be moved inside with unmarshal action being renamed appropriately ?

retry decoration is just for pre processing and post processing


> On July 30, 2014, 4:30 p.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/resources/workflow/feed/falcon-table-export.hql, line 1
> > <https://reviews.apache.org/r/23844/diff/4/?file=645670#file645670line1>
> >
> >     Can be moved to an apt folder instead of workflow/feed

will do


> On July 30, 2014, 4:30 p.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/resources/workflow/post-process.xml, line 1
> > <https://reviews.apache.org/r/23844/diff/4/?file=645676#file645676line1>
> >
> >     Seems to be an action definition. Perhaps can be moved to action resource folder

will do


- shwethags


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/#review49115
-----------------------------------------------------------


On July 30, 2014, 11:11 a.m., shwethags wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23844/
> -----------------------------------------------------------
> 
> (Updated July 30, 2014, 11:11 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-481
>     https://issues.apache.org/jira/browse/FALCON-481
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Currently, process parent workflow contains 2 decision nodes which take up oozie processing and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing actions only once so that the definitions are consistent
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java 6f50829 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 62d95fa 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 0b582ef 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883 
>   oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1 
>   oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java e46ae6e 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 2e3a5c1 
>   oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
>   oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
>   oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
>   oozie/src/main/resources/workflow/feed/eviction-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/falcon-table-export.hql PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/falcon-table-import.hql PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/replication-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/table-cleanup.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/table-export.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/table-import.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/post-process.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/pre-process.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
>   oozie/src/main/resources/workflow/process/hive-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/oozie-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/pig-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
>   oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
>   oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c 
>   oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
>   oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 3655af9 
>   src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 
>   src/main/examples/entity/hcat/hcat-pig-process.xml 6ee17eb 
> 
> Diff: https://reviews.apache.org/r/23844/diff/
> 
> 
> Testing
> -------
> 
> UTs + end to end tests for fs usecases
> 
> Pending: hcat testing
> 
> 
> Thanks,
> 
> shwethags
> 
>

Re: Review Request 23844: FALCON-481 Simplify process parent workflow

Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/#review49115
-----------------------------------------------------------



oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
<https://reviews.apache.org/r/23844/#comment85956>

    Every Action unmarshal seems to be accompanied with a code to retry decoration. Can be moved inside with unmarshal action being renamed appropriately ?



oozie/src/main/resources/workflow/feed/falcon-table-export.hql
<https://reviews.apache.org/r/23844/#comment85958>

    Can be moved to an apt folder instead of workflow/feed



oozie/src/main/resources/workflow/post-process.xml
<https://reviews.apache.org/r/23844/#comment85957>

    Seems to be an action definition. Perhaps can be moved to action resource folder


- Srikanth Sundarrajan


On July 30, 2014, 11:11 a.m., shwethags wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23844/
> -----------------------------------------------------------
> 
> (Updated July 30, 2014, 11:11 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-481
>     https://issues.apache.org/jira/browse/FALCON-481
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Currently, process parent workflow contains 2 decision nodes which take up oozie processing and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing actions only once so that the definitions are consistent
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java 6f50829 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 62d95fa 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 0b582ef 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883 
>   oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1 
>   oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java e46ae6e 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 2e3a5c1 
>   oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
>   oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
>   oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
>   oozie/src/main/resources/workflow/feed/eviction-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/falcon-table-export.hql PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/falcon-table-import.hql PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/replication-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/table-cleanup.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/table-export.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/table-import.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/post-process.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/pre-process.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
>   oozie/src/main/resources/workflow/process/hive-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/oozie-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/pig-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
>   oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
>   oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c 
>   oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
>   oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 3655af9 
>   src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 
>   src/main/examples/entity/hcat/hcat-pig-process.xml 6ee17eb 
> 
> Diff: https://reviews.apache.org/r/23844/diff/
> 
> 
> Testing
> -------
> 
> UTs + end to end tests for fs usecases
> 
> Pending: hcat testing
> 
> 
> Thanks,
> 
> shwethags
> 
>

Re: Review Request 23844: FALCON-481 Simplify process parent workflow

Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/
-----------------------------------------------------------

(Updated July 31, 2014, 10:19 a.m.)


Review request for Falcon.


Changes
-------

review comments incorporated


Bugs: FALCON-481
    https://issues.apache.org/jira/browse/FALCON-481


Repository: falcon-git


Description
-------

Currently, process parent workflow contains 2 decision nodes which take up oozie processing and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing actions only once so that the definitions are consistent


Diffs (updated)
-----

  common/src/main/java/org/apache/falcon/entity/EntityUtil.java 6f50829 
  oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 62d95fa 
  oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
  oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
  oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 0b582ef 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9 
  oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883 
  oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1 
  oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java e46ae6e 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 2e3a5c1 
  oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
  oozie/src/main/resources/action/feed/eviction-action.xml PRE-CREATION 
  oozie/src/main/resources/action/feed/falcon-table-export.hql PRE-CREATION 
  oozie/src/main/resources/action/feed/falcon-table-import.hql PRE-CREATION 
  oozie/src/main/resources/action/feed/replication-action.xml PRE-CREATION 
  oozie/src/main/resources/action/feed/table-cleanup.xml PRE-CREATION 
  oozie/src/main/resources/action/feed/table-export.xml PRE-CREATION 
  oozie/src/main/resources/action/feed/table-import.xml PRE-CREATION 
  oozie/src/main/resources/action/post-process.xml PRE-CREATION 
  oozie/src/main/resources/action/pre-process.xml PRE-CREATION 
  oozie/src/main/resources/action/process/hive-action.xml PRE-CREATION 
  oozie/src/main/resources/action/process/oozie-action.xml PRE-CREATION 
  oozie/src/main/resources/action/process/pig-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
  oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
  oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
  oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
  oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
  oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c 
  oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
  oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 3655af9 
  src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 
  src/main/examples/entity/hcat/hcat-pig-process.xml 6ee17eb 

Diff: https://reviews.apache.org/r/23844/diff/


Testing
-------

UTs + end to end tests for fs usecases

Pending: hcat testing


Thanks,

shwethags

Re: Review Request 23844: FALCON-481 Simplify process parent workflow

Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/
-----------------------------------------------------------

(Updated July 30, 2014, 11:11 a.m.)


Review request for Falcon.


Changes
-------

fixed hcat path issues and example process


Bugs: FALCON-481
    https://issues.apache.org/jira/browse/FALCON-481


Repository: falcon-git


Description
-------

Currently, process parent workflow contains 2 decision nodes which take up oozie processing and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing actions only once so that the definitions are consistent


Diffs (updated)
-----

  common/src/main/java/org/apache/falcon/entity/EntityUtil.java 6f50829 
  oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 62d95fa 
  oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
  oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
  oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 0b582ef 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9 
  oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883 
  oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1 
  oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java e46ae6e 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 2e3a5c1 
  oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
  oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
  oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
  oozie/src/main/resources/workflow/feed/eviction-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/falcon-table-export.hql PRE-CREATION 
  oozie/src/main/resources/workflow/feed/falcon-table-import.hql PRE-CREATION 
  oozie/src/main/resources/workflow/feed/replication-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/table-cleanup.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/table-export.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/table-import.xml PRE-CREATION 
  oozie/src/main/resources/workflow/post-process.xml PRE-CREATION 
  oozie/src/main/resources/workflow/pre-process.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
  oozie/src/main/resources/workflow/process/hive-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/oozie-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/pig-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
  oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
  oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c 
  oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
  oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 3655af9 
  src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 
  src/main/examples/entity/hcat/hcat-pig-process.xml 6ee17eb 

Diff: https://reviews.apache.org/r/23844/diff/


Testing
-------

UTs + end to end tests for fs usecases

Pending: hcat testing


Thanks,

shwethags

Re: Review Request 23844: FALCON-481 Simplify process parent workflow

Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/
-----------------------------------------------------------

(Updated July 29, 2014, 8:41 a.m.)


Review request for Falcon.


Changes
-------

Refactored to build workflow on the fly and only action templates are checked in(no workflows). 
Removed caching of templates (Every consumer of loaded action template modified the action. So, everyone needs their own copy of the loaded action template)


Bugs: FALCON-481
    https://issues.apache.org/jira/browse/FALCON-481


Repository: falcon-git


Description
-------

Currently, process parent workflow contains 2 decision nodes which take up oozie processing and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing actions only once so that the definitions are consistent


Diffs (updated)
-----

  common/src/main/java/org/apache/falcon/entity/EntityUtil.java 6f50829 
  oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 62d95fa 
  oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
  oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
  oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 0b582ef 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9 
  oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883 
  oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1 
  oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java e46ae6e 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 2e3a5c1 
  oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
  oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
  oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
  oozie/src/main/resources/workflow/feed/eviction-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/falcon-table-export.hql PRE-CREATION 
  oozie/src/main/resources/workflow/feed/falcon-table-import.hql PRE-CREATION 
  oozie/src/main/resources/workflow/feed/replication-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/table-cleanup.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/table-export.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/table-import.xml PRE-CREATION 
  oozie/src/main/resources/workflow/post-process.xml PRE-CREATION 
  oozie/src/main/resources/workflow/pre-process.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
  oozie/src/main/resources/workflow/process/hive-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/oozie-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/pig-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
  oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
  oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c 
  oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
  oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 3655af9 
  src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 

Diff: https://reviews.apache.org/r/23844/diff/


Testing
-------

UTs + end to end tests for fs usecases

Pending: hcat testing


Thanks,

shwethags

Re: Review Request 23844: FALCON-481 Simplify process parent workflow

Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/#review48722
-----------------------------------------------------------



oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
<https://reviews.apache.org/r/23844/#comment85478>

    The reason order is not important is because the inter-dependency between actions are also wired up here. Does it make sense to just build the post processing actions here and stitch them in the correct flow sequence in the calling function ?


- Srikanth Sundarrajan


On July 24, 2014, 11:51 a.m., shwethags wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23844/
> -----------------------------------------------------------
> 
> (Updated July 24, 2014, 11:51 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-481
>     https://issues.apache.org/jira/browse/FALCON-481
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Currently, process parent workflow contains 2 decision nodes which take up oozie processing and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing actions only once so that the definitions are consistent
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/falcon/entity/EntityUtil.java 6f50829 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 62d95fa 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 0b582ef 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883 
>   oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1 
>   oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java e46ae6e 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 2e3a5c1 
>   oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
>   oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
>   oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
>   oozie/src/main/resources/workflow/feed/falcon-table-export.hql PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/falcon-table-import.hql PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/fs-replication-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/hcat-replication-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/replication-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/retention-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/post-process.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/pre-process.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
>   oozie/src/main/resources/workflow/process/hive-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/oozie-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/parent-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/pig-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
>   oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
>   oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c 
>   oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
>   oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 3655af9 
>   src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 
> 
> Diff: https://reviews.apache.org/r/23844/diff/
> 
> 
> Testing
> -------
> 
> UTs + end to end tests for fs usecases
> 
> Pending: hcat testing
> 
> 
> Thanks,
> 
> shwethags
> 
>

Re: Review Request 23844: FALCON-481 Simplify process parent workflow

Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/
-----------------------------------------------------------

(Updated July 24, 2014, 11:51 a.m.)


Review request for Falcon.


Changes
-------

addressed review comments


Bugs: FALCON-481
    https://issues.apache.org/jira/browse/FALCON-481


Repository: falcon-git


Description
-------

Currently, process parent workflow contains 2 decision nodes which take up oozie processing and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing actions only once so that the definitions are consistent


Diffs (updated)
-----

  common/src/main/java/org/apache/falcon/entity/EntityUtil.java 6f50829 
  oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 62d95fa 
  oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
  oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
  oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 0b582ef 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9 
  oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION 
  oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883 
  oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1 
  oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java e46ae6e 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 2e3a5c1 
  oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
  oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
  oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
  oozie/src/main/resources/workflow/feed/falcon-table-export.hql PRE-CREATION 
  oozie/src/main/resources/workflow/feed/falcon-table-import.hql PRE-CREATION 
  oozie/src/main/resources/workflow/feed/fs-replication-workflow.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/hcat-replication-workflow.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/replication-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/retention-workflow.xml PRE-CREATION 
  oozie/src/main/resources/workflow/post-process.xml PRE-CREATION 
  oozie/src/main/resources/workflow/pre-process.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
  oozie/src/main/resources/workflow/process/hive-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/oozie-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/parent-workflow.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/pig-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
  oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
  oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c 
  oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
  oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 3655af9 
  src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 

Diff: https://reviews.apache.org/r/23844/diff/


Testing
-------

UTs + end to end tests for fs usecases

Pending: hcat testing


Thanks,

shwethags

Re: Review Request 23844: FALCON-481 Simplify process parent workflow

Posted by sh...@inmobi.com.

> On July 24, 2014, 9:33 a.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java, line 68
> > <https://reviews.apache.org/r/23844/diff/1/?file=640491#file640491line68>
> >
> >     Why is coord func in generic entity builder ? Theoretically entity builder can be used to build bundle and this constant has no relevance there.

I was trying out something, but it's not required now. Will move it out


> On July 24, 2014, 9:33 a.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java, line 171
> > <https://reviews.apache.org/r/23844/diff/1/?file=640491#file640491line171>
> >
> >     Perhaps this is more appropriate for EntityUtil instead of EntityBuilder ?

will move


> On July 24, 2014, 9:33 a.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java, line 176
> > <https://reviews.apache.org/r/23844/diff/1/?file=640491#file640491line176>
> >
> >     EntityUtil perhaps ?

will move


> On July 24, 2014, 9:33 a.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java, line 134
> > <https://reviews.apache.org/r/23844/diff/1/?file=640492#file640492line134>
> >
> >     Is the caller expected to retrieve items in order assuming first is success & next is failed. Seems a bit fragile.
> >     
> >     Does it make sense to pass a bunch of templates and return an action for each template in a map ?

ordering is not expected


> On July 24, 2014, 9:33 a.m., Srikanth Sundarrajan wrote:
> > oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java, line 344
> > <https://reviews.apache.org/r/23844/diff/1/?file=640522#file640522line344>
> >
> >     Can we add similar tests for retention & process as well ?

OozieProcessWorkflowBuilderTest has process tests. OozieFeedWorkflowBuilderTest.java has both retention and replication


> On July 24, 2014, 9:33 a.m., Srikanth Sundarrajan wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java, line 189
> > <https://reviews.apache.org/r/23844/diff/1/?file=640492#file640492line189>
> >
> >     There is no reason to do this repeatedly since the action xml doesn't change across successive calls. Cache ?

can be cached, will check


- shwethags


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/#review48638
-----------------------------------------------------------


On July 23, 2014, 11:36 a.m., shwethags wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23844/
> -----------------------------------------------------------
> 
> (Updated July 23, 2014, 11:36 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-481
>     https://issues.apache.org/jira/browse/FALCON-481
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Currently, process parent workflow contains 2 decision nodes which take up oozie processing and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing actions only once so that the definitions are consistent
> 
> 
> Diffs
> -----
> 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 0b582ef 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883 
>   oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1 
>   oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java e46ae6e 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 2e3a5c1 
>   oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
>   oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
>   oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
>   oozie/src/main/resources/workflow/feed/falcon-table-export.hql PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/falcon-table-import.hql PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/fs-replication-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/hcat-replication-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/replication-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/retention-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/post-process.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/pre-process.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
>   oozie/src/main/resources/workflow/process/hive-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/oozie-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/parent-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/pig-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
>   oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
>   oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c 
>   oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
>   oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 3655af9 
>   src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 
> 
> Diff: https://reviews.apache.org/r/23844/diff/
> 
> 
> Testing
> -------
> 
> UTs + end to end tests for fs usecases
> 
> Pending: hcat testing
> 
> 
> Thanks,
> 
> shwethags
> 
>

Re: Review Request 23844: FALCON-481 Simplify process parent workflow

Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/#review48638
-----------------------------------------------------------



oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java
<https://reviews.apache.org/r/23844/#comment85355>

    Why is coord func in generic entity builder ? Theoretically entity builder can be used to build bundle and this constant has no relevance there.



oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java
<https://reviews.apache.org/r/23844/#comment85356>

    Perhaps this is more appropriate for EntityUtil instead of EntityBuilder ?



oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java
<https://reviews.apache.org/r/23844/#comment85357>

    EntityUtil perhaps ?



oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
<https://reviews.apache.org/r/23844/#comment85358>

    Is the caller expected to retrieve items in order assuming first is success & next is failed. Seems a bit fragile.
    
    Does it make sense to pass a bunch of templates and return an action for each template in a map ?



oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
<https://reviews.apache.org/r/23844/#comment85359>

    There is no reason to do this repeatedly since the action xml doesn't change across successive calls. Cache ?



oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
<https://reviews.apache.org/r/23844/#comment85360>

    Can we add similar tests for retention & process as well ?


- Srikanth Sundarrajan


On July 23, 2014, 11:36 a.m., shwethags wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23844/
> -----------------------------------------------------------
> 
> (Updated July 23, 2014, 11:36 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-481
>     https://issues.apache.org/jira/browse/FALCON-481
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Currently, process parent workflow contains 2 decision nodes which take up oozie processing and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing actions only once so that the definitions are consistent
> 
> 
> Diffs
> -----
> 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 0b582ef 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION 
>   oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883 
>   oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1 
>   oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java e46ae6e 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 2e3a5c1 
>   oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
>   oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
>   oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
>   oozie/src/main/resources/workflow/feed/falcon-table-export.hql PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/falcon-table-import.hql PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/fs-replication-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/hcat-replication-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/replication-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/feed/retention-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/post-process.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/pre-process.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
>   oozie/src/main/resources/workflow/process/hive-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/oozie-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/parent-workflow.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/process/pig-action.xml PRE-CREATION 
>   oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
>   oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
>   oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c 
>   oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
>   oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 3655af9 
>   src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 
> 
> Diff: https://reviews.apache.org/r/23844/diff/
> 
> 
> Testing
> -------
> 
> UTs + end to end tests for fs usecases
> 
> Pending: hcat testing
> 
> 
> Thanks,
> 
> shwethags
> 
>