You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by sh...@inmobi.com on 2014/07/15 13:57:07 UTC
Review Request 23483: FALCON-353 enable dry run feature of oozie for
schedule and update
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23483/
-----------------------------------------------------------
Review request for Falcon.
Bugs: FALCON-353
https://issues.apache.org/jira/browse/FALCON-353
Repository: falcon-git
Description
-------
Entity schedule creates a bundle in oozie. But bundle submit is asynchronous and coord create can fail and creates bad user experience. So, this patch uses dryRun feature of oozie to validate the entity before scheduling in oozie.
DryRun is used in validate(), schedule() and update()
Diffs
-----
common/src/main/java/org/apache/falcon/entity/EntityUtil.java a38e553
common/src/main/java/org/apache/falcon/update/UpdateHelper.java 7af77d0
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java eedd81f
common/src/test/java/org/apache/falcon/update/UpdateHelperTest.java e532b24
oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 2018db2
oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5a29683
oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java bb8dfcc
oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java ac78297
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 6917f4e
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 3226cf2
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 00fab99
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 4393c94
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java 4a7f96b
oozie/src/main/java/org/apache/falcon/oozie/process/ProcessBundleBuilder.java 86cea93
oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java c87bc86
oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 0d9abdb
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 38be792
oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java 542634d
oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java 54a2ea7
oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 5ceea75
prism/src/main/java/org/apache/falcon/FalconWebException.java d552c07
prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java c135470
webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java e214392
webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseySmokeIT.java dceb2f2
webapp/src/test/java/org/apache/falcon/resource/TestContext.java bc79609
Diff: https://reviews.apache.org/r/23483/diff/
Testing
-------
UTs + manual end to end test of feed retention and process schedule and update
Thanks,
shwethags
Re: Review Request 23483: FALCON-353 enable dry run feature of oozie for
schedule and update
Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23483/
-----------------------------------------------------------
(Updated July 22, 2014, 10:09 a.m.)
Review request for Falcon.
Changes
-------
Addressed review comments
Bugs: FALCON-353
https://issues.apache.org/jira/browse/FALCON-353
Repository: falcon-git
Description
-------
Entity schedule creates a bundle in oozie. But bundle submit is asynchronous and coord create can fail and creates bad user experience. So, this patch uses dryRun feature of oozie to validate the entity before scheduling in oozie.
DryRun is used in validate(), schedule() and update()
Diffs (updated)
-----
common/src/main/java/org/apache/falcon/entity/EntityUtil.java a38e553
common/src/main/java/org/apache/falcon/update/UpdateHelper.java 7af77d0
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java eedd81f
common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java 520541a
common/src/test/java/org/apache/falcon/update/UpdateHelperTest.java e532b24
common/src/test/resources/config/process/process-0.1.xml 0aa249b
oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 2018db2
oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5a29683
oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java bb8dfcc
oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java ac78297
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 6917f4e
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 3226cf2
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 00fab99
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 4393c94
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java 4a7f96b
oozie/src/main/java/org/apache/falcon/oozie/process/ProcessBundleBuilder.java 86cea93
oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java c87bc86
oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 0d9abdb
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 38be792
oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java 542634d
oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java 54a2ea7
oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 5ceea75
prism/src/main/java/org/apache/falcon/FalconWebException.java d552c07
prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java c135470
webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java e214392
webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseySmokeIT.java dceb2f2
webapp/src/test/java/org/apache/falcon/resource/TestContext.java bc79609
Diff: https://reviews.apache.org/r/23483/diff/
Testing
-------
UTs + manual end to end test of feed retention and process schedule and update
Thanks,
shwethags
Re: Review Request 23483: FALCON-353 enable dry run feature of oozie for
schedule and update
Posted by sh...@inmobi.com.
> On July 22, 2014, 6 a.m., Suhas Vasu wrote:
> > common/src/main/java/org/apache/falcon/entity/EntityUtil.java, line 593
> > <https://reviews.apache.org/r/23483/diff/1/?file=632249#file632249line593>
> >
> > we should log the error, will help in debugging in case of issues.
will log
> On July 22, 2014, 6 a.m., Suhas Vasu wrote:
> > common/src/main/java/org/apache/falcon/update/UpdateHelper.java, line 201
> > <https://reviews.apache.org/r/23483/diff/1/?file=632250#file632250line201>
> >
> > it should return false here
good catch. will fix
> On July 22, 2014, 6 a.m., Suhas Vasu wrote:
> > common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java, line 91
> > <https://reviews.apache.org/r/23483/diff/1/?file=632251#file632251line91>
> >
> > Changing the return type won't cause backward compatibility issues later ?
> > my concern is because this is a server call.
Its internal to falcon, not an externally exposed API. So, its not an issue
> On July 22, 2014, 6 a.m., Suhas Vasu wrote:
> > oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java, line 1238
> > <https://reviews.apache.org/r/23483/diff/1/?file=632265#file632265line1238>
> >
> > Shouldn't it be clone rather than entity ?
yes, should be. will fix
> On July 22, 2014, 6 a.m., Suhas Vasu wrote:
> > oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java, line 1242
> > <https://reviews.apache.org/r/23483/diff/1/?file=632265#file632265line1242>
> >
> > Shouldn't it be clone rather than entity ?
> >
the name won't change in clone. So, not an issue
> On July 22, 2014, 6 a.m., Suhas Vasu wrote:
> > oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java, line 447
> > <https://reviews.apache.org/r/23483/diff/1/?file=632266#file632266line447>
> >
> > isn't new Path(wfPath) redundant as wfPath is already new Path()
its new Path(wfPath + "/scripts"), not just new Path(wfPath). Haven't changed it
> On July 22, 2014, 6 a.m., Suhas Vasu wrote:
> > oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java, line 997
> > <https://reviews.apache.org/r/23483/diff/1/?file=632265#file632265line997>
> >
> > problem if update fails just before updating the config store, it will go through all the steps again on retry.
will fix
> On July 22, 2014, 6 a.m., Suhas Vasu wrote:
> > oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java, line 354
> > <https://reviews.apache.org/r/23483/diff/1/?file=632265#file632265line354>
> >
> > what if we get back a empty list for delete/kill ?
It means the entity is not scheduled and does nothing
- shwethags
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23483/#review48206
-----------------------------------------------------------
On July 15, 2014, 11:57 a.m., shwethags wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23483/
> -----------------------------------------------------------
>
> (Updated July 15, 2014, 11:57 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-353
> https://issues.apache.org/jira/browse/FALCON-353
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Entity schedule creates a bundle in oozie. But bundle submit is asynchronous and coord create can fail and creates bad user experience. So, this patch uses dryRun feature of oozie to validate the entity before scheduling in oozie.
>
> DryRun is used in validate(), schedule() and update()
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/EntityUtil.java a38e553
> common/src/main/java/org/apache/falcon/update/UpdateHelper.java 7af77d0
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java eedd81f
> common/src/test/java/org/apache/falcon/update/UpdateHelperTest.java e532b24
> oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 2018db2
> oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5a29683
> oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java bb8dfcc
> oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java ac78297
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 6917f4e
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 3226cf2
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 00fab99
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 4393c94
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java 4a7f96b
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessBundleBuilder.java 86cea93
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java c87bc86
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 0d9abdb
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 38be792
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java 542634d
> oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java 54a2ea7
> oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 5ceea75
> prism/src/main/java/org/apache/falcon/FalconWebException.java d552c07
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java c135470
> webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java e214392
> webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseySmokeIT.java dceb2f2
> webapp/src/test/java/org/apache/falcon/resource/TestContext.java bc79609
>
> Diff: https://reviews.apache.org/r/23483/diff/
>
>
> Testing
> -------
>
> UTs + manual end to end test of feed retention and process schedule and update
>
>
> Thanks,
>
> shwethags
>
>
Re: Review Request 23483: FALCON-353 enable dry run feature of oozie for
schedule and update
Posted by Suhas Vasu <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23483/#review48206
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/entity/EntityUtil.java
<https://reviews.apache.org/r/23483/#comment84559>
we should log the error, will help in debugging in case of issues.
common/src/main/java/org/apache/falcon/update/UpdateHelper.java
<https://reviews.apache.org/r/23483/#comment84566>
it should return false here
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java
<https://reviews.apache.org/r/23483/#comment84560>
Changing the return type won't cause backward compatibility issues later ?
my concern is because this is a server call.
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java
<https://reviews.apache.org/r/23483/#comment84562>
what if we get back a empty list for delete/kill ?
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java
<https://reviews.apache.org/r/23483/#comment84567>
problem if update fails just before updating the config store, it will go through all the steps again on retry.
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java
<https://reviews.apache.org/r/23483/#comment84563>
Shouldn't it be clone rather than entity ?
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java
<https://reviews.apache.org/r/23483/#comment84564>
Shouldn't it be clone rather than entity ?
oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
<https://reviews.apache.org/r/23483/#comment84565>
isn't new Path(wfPath) redundant as wfPath is already new Path()
- Suhas Vasu
On July 15, 2014, 11:57 a.m., shwethags wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23483/
> -----------------------------------------------------------
>
> (Updated July 15, 2014, 11:57 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-353
> https://issues.apache.org/jira/browse/FALCON-353
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Entity schedule creates a bundle in oozie. But bundle submit is asynchronous and coord create can fail and creates bad user experience. So, this patch uses dryRun feature of oozie to validate the entity before scheduling in oozie.
>
> DryRun is used in validate(), schedule() and update()
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/EntityUtil.java a38e553
> common/src/main/java/org/apache/falcon/update/UpdateHelper.java 7af77d0
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java eedd81f
> common/src/test/java/org/apache/falcon/update/UpdateHelperTest.java e532b24
> oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 2018db2
> oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5a29683
> oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java bb8dfcc
> oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java ac78297
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java 6917f4e
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java 3226cf2
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 00fab99
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 4393c94
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java 4a7f96b
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessBundleBuilder.java 86cea93
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java c87bc86
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java 0d9abdb
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 38be792
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java 542634d
> oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java 54a2ea7
> oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java 5ceea75
> prism/src/main/java/org/apache/falcon/FalconWebException.java d552c07
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java c135470
> webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java e214392
> webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseySmokeIT.java dceb2f2
> webapp/src/test/java/org/apache/falcon/resource/TestContext.java bc79609
>
> Diff: https://reviews.apache.org/r/23483/diff/
>
>
> Testing
> -------
>
> UTs + manual end to end test of feed retention and process schedule and update
>
>
> Thanks,
>
> shwethags
>
>