You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Pallavi Rao <pa...@inmobi.com> on 2015/11/03 06:50:04 UTC
Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/
-----------------------------------------------------------
Review request for Falcon.
Bugs: FALCON-1233
https://issues.apache.org/jira/browse/FALCON-1233
Repository: falcon-git
Description
-------
The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
Diffs
-----
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
docs/src/site/twiki/FalconCLI.twiki e001a7f
prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java b2f9e59
scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
src/conf/startup.properties ce6e91f
Diff: https://reviews.apache.org/r/39844/diff/
Testing
-------
UTs added. Manual testing done.
Thanks,
Pallavi Rao
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Pallavi Rao <pa...@inmobi.com>.
> On Nov. 20, 2015, 6:53 a.m., Peeyush Bishnoi wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line 46
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114449#file1114449line46>
> >
> > Any particular reason for naming WF Engine PRIMARY and SECONDARY.
Yep. confusing. Addressed in the new patch.
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review107316
-----------------------------------------------------------
On Nov. 25, 2015, 6:32 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Nov. 25, 2015, 6:32 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 7b36b11
> docs/src/site/twiki/falconcli/Schedule.twiki 42192c7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java d7d157f
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java bff92c9
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> unit/pom.xml 8d9f443
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Peeyush Bishnoi <bp...@yahoo.co.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review107316
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java (line 46)
<https://reviews.apache.org/r/39844/#comment166380>
Any particular reason for naming WF Engine PRIMARY and SECONDARY.
- Peeyush Bishnoi
On Nov. 3, 2015, 5:50 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2015, 5:50 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> docs/src/site/twiki/FalconCLI.twiki e001a7f
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java b2f9e59
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> src/conf/startup.properties ce6e91f
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Pallavi Rao <pa...@inmobi.com>.
> On Nov. 26, 2015, 10:50 a.m., Peeyush Bishnoi wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line 69
> > <https://reviews.apache.org/r/39844/diff/2/?file=1146487#file1146487line69>
> >
> > In this function getWorkflowEngine() is getting return when Entity is null and props is null. Is it not possible to check both the conditions once and return once.
Yes. The conditions can be clubbed. Will do that.
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review108112
-----------------------------------------------------------
On Nov. 25, 2015, 6:32 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Nov. 25, 2015, 6:32 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 7b36b11
> docs/src/site/twiki/falconcli/Schedule.twiki 42192c7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java d7d157f
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java bff92c9
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> unit/pom.xml 8d9f443
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Peeyush Bishnoi <bp...@yahoo.co.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review108112
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java (line 69)
<https://reviews.apache.org/r/39844/#comment167477>
In this function getWorkflowEngine() is getting return when Entity is null and props is null. Is it not possible to check both the conditions once and return once.
- Peeyush Bishnoi
On Nov. 25, 2015, 6:32 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Nov. 25, 2015, 6:32 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 7b36b11
> docs/src/site/twiki/falconcli/Schedule.twiki 42192c7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java d7d157f
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java bff92c9
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> unit/pom.xml 8d9f443
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Peeyush Bishnoi <bp...@yahoo.co.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review108460
-----------------------------------------------------------
Ship it!
Ship It!
- Peeyush Bishnoi
On Dec. 1, 2015, 5:15 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Dec. 1, 2015, 5:15 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 7b36b11
> docs/src/site/twiki/falconcli/Schedule.twiki 42192c7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java 5c7bf91
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java 2a9fbce
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> unit/pom.xml 8d9f443
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Pallavi Rao <pa...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/
-----------------------------------------------------------
(Updated Dec. 1, 2015, 5:15 a.m.)
Review request for Falcon.
Changes
-------
Rebased and Review comments addressed.
Bugs: FALCON-1233
https://issues.apache.org/jira/browse/FALCON-1233
Repository: falcon-git
Description
-------
The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
Diffs (updated)
-----
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 7b36b11
docs/src/site/twiki/falconcli/Schedule.twiki 42192c7
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646
prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java 5c7bf91
scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java 2a9fbce
scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
unit/pom.xml 8d9f443
Diff: https://reviews.apache.org/r/39844/diff/
Testing
-------
UTs added. Manual testing done.
Thanks,
Pallavi Rao
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Pallavi Rao <pa...@inmobi.com>.
> On Nov. 26, 2015, 11:02 a.m., Peeyush Bishnoi wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line 107
> > <https://reviews.apache.org/r/39844/diff/2/?file=1146487#file1146487line107>
> >
> > Should not be the variable configuredWorkflowEngine declared volatile.
A WorkflowEngine holds no state and does not need to be a singleton. Even, if we return multiple instances (earlier that is what used to happen) there is no harm. Caching the instance is just for optimization. Hence, it does not need any thread-safety measure.
> On Nov. 26, 2015, 11:02 a.m., Peeyush Bishnoi wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line 113
> > <https://reviews.apache.org/r/39844/diff/2/?file=1146487#file1146487line113>
> >
> > Should not be the variable nativeWorkflowEngine declared volatile.
A WorkflowEngine holds no state and does not need to be a singleton. Even, if we return multiple instances (earlier that is what used to happen) there is no harm. Caching the instance is just for optimization. Hence, it does not need any thread-safety measure.
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review108113
-----------------------------------------------------------
On Nov. 25, 2015, 6:32 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Nov. 25, 2015, 6:32 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 7b36b11
> docs/src/site/twiki/falconcli/Schedule.twiki 42192c7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java d7d157f
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java bff92c9
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> unit/pom.xml 8d9f443
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Peeyush Bishnoi <bp...@yahoo.co.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review108113
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java (line 107)
<https://reviews.apache.org/r/39844/#comment167478>
Should not be the variable configuredWorkflowEngine declared volatile.
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java (line 113)
<https://reviews.apache.org/r/39844/#comment167479>
Should not be the variable nativeWorkflowEngine declared volatile.
- Peeyush Bishnoi
On Nov. 25, 2015, 6:32 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Nov. 25, 2015, 6:32 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 7b36b11
> docs/src/site/twiki/falconcli/Schedule.twiki 42192c7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java d7d157f
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java bff92c9
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> unit/pom.xml 8d9f443
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Pallavi Rao <pa...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/
-----------------------------------------------------------
(Updated Nov. 25, 2015, 6:32 a.m.)
Review request for Falcon.
Changes
-------
Addressed review comments
Bugs: FALCON-1233
https://issues.apache.org/jira/browse/FALCON-1233
Repository: falcon-git
Description
-------
The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
Diffs (updated)
-----
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 7b36b11
docs/src/site/twiki/falconcli/Schedule.twiki 42192c7
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646
prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java d7d157f
scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java bff92c9
scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
unit/pom.xml 8d9f443
Diff: https://reviews.apache.org/r/39844/diff/
Testing
-------
UTs added. Manual testing done.
Thanks,
Pallavi Rao
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Pallavi Rao <pa...@inmobi.com>.
> On Nov. 4, 2015, 7:01 a.m., pavan kumar kolamuri wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line 84
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114449#file1114449line84>
> >
> > Once entity is migrated to Native Scheduler. What will happen if user wants to get status of instances of an entity with start time before migration time. (startTime < migrationTime). Will user gets the consolidated view as per doc says ?
With this patch, the assumption is that user will delete older entity, create a new one and schedule it on native scheduler. So, no history will be available. However, in the next phase, I do plan to provide an option to move an existing entity to native scheduler which will preserve the instance history.
> On Nov. 4, 2015, 7:01 a.m., pavan kumar kolamuri wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line 102
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114449#file1114449line102>
> >
> > If entity is null shouldn't we throw exception ? In which case entity can be null and still we will return Primary Engine.
With the previous API, the usage was WorkflowEngineFactory.getWorkflowEngine(). Entity was not an argument at all. Just to preserve the backward compatibility of API for such usage, if no entity is supplied, the default oozie engine is returned.
> On Nov. 4, 2015, 7:01 a.m., pavan kumar kolamuri wrote:
> > scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java, line 103
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114456#file1114456line103>
> >
> > Other case props set to secondary should be added right ?
Ideally .. yes.. But, there is check in there where I invoke OozieWorkflowEngine.isAlive() that actually makes a oozie call and it was too much pain to Mock the entire class. Didn't think it was worth it.
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review105036
-----------------------------------------------------------
On Nov. 25, 2015, 6:32 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Nov. 25, 2015, 6:32 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 7b36b11
> docs/src/site/twiki/falconcli/Schedule.twiki 42192c7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java d7d157f
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java bff92c9
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> unit/pom.xml 8d9f443
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by pavan kumar kolamuri <pa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review105036
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java (line 84)
<https://reviews.apache.org/r/39844/#comment163399>
Once entity is migrated to Native Scheduler. What will happen if user wants to get status of instances of an entity with start time before migration time. (startTime < migrationTime). Will user gets the consolidated view as per doc says ?
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java (line 102)
<https://reviews.apache.org/r/39844/#comment163401>
If entity is null shouldn't we throw exception ? In which case entity can be null and still we will return Primary Engine.
scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java (line 103)
<https://reviews.apache.org/r/39844/#comment163404>
Other case props set to secondary should be added right ?
- pavan kumar kolamuri
On Nov. 3, 2015, 5:50 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2015, 5:50 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> docs/src/site/twiki/FalconCLI.twiki e001a7f
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java b2f9e59
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> src/conf/startup.properties ce6e91f
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Pallavi Rao <pa...@inmobi.com>.
> On Nov. 20, 2015, 5:08 a.m., Ajay Yadava wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line 100
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114449#file1114449line100>
> >
> > Why is a map required to be sent when we are using only one property?
Just future-proofing. In case we introduce other schedule properties.
> On Nov. 20, 2015, 5:08 a.m., Ajay Yadava wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line 140
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114449#file1114449line140>
> >
> > nit: Isn't it already available in WFENGINE.PRIMARY.getEngine()?
Yep.. Will use the cached instance.
> On Nov. 20, 2015, 5:08 a.m., Ajay Yadava wrote:
> > docs/src/site/twiki/FalconCLI.twiki, line 58
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114451#file1114451line58>
> >
> > Can you please add a little bit of description on the use case for this command? The name secondary doesn't convey much.
Addressed it.
> On Nov. 20, 2015, 5:08 a.m., Ajay Yadava wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java, line 90
> > <https://reviews.apache.org/r/39844/diff/1/?file=1114449#file1114449line90>
> >
> > nit: can you please log the entity for which it is returned as well?
done.
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review105231
-----------------------------------------------------------
On Nov. 25, 2015, 6:32 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Nov. 25, 2015, 6:32 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 7b36b11
> docs/src/site/twiki/falconcli/Schedule.twiki 42192c7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 724f646
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java d7d157f
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java bff92c9
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> unit/pom.xml 8d9f443
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 39844: FALCON-1233 Support co-existence of Oozie
scheduler (coord) and Falcon native scheduler.
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39844/#review105231
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java (line 90)
<https://reviews.apache.org/r/39844/#comment166348>
nit: can you please log the entity for which it is returned as well?
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java (line 100)
<https://reviews.apache.org/r/39844/#comment163720>
Why is a map required to be sent when we are using only one property?
common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java (line 140)
<https://reviews.apache.org/r/39844/#comment166344>
nit: Isn't it already available in WFENGINE.PRIMARY.getEngine()?
docs/src/site/twiki/FalconCLI.twiki (line 57)
<https://reviews.apache.org/r/39844/#comment163719>
Can you please add a little bit of description on the use case for this command? The name secondary doesn't convey much.
- Ajay Yadava
On Nov. 3, 2015, 5:50 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39844/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2015, 5:50 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1233
> https://issues.apache.org/jira/browse/FALCON-1233
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> The migration to Falcon native scheduler should be as seamless as possible and the user should be able to migrate to the native scheduler in a phased manner (entity by entity).
>
> This patch introduces the notion of primary and secondary workflow engines. It allows users to schedule an entity on the secondary engine (primary, by default). The idea is that initially, users will use Falcon native scheduler as the secondary engine to migrate their entities one by one. When most entities have been migrated, the Oozie scheduler becomes secondary (and eventually removed).
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 49592ac
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java 9d96fa3
> docs/src/site/twiki/FalconCLI.twiki e001a7f
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 16ef83a
> prism/src/main/java/org/apache/falcon/resource/AbstractInstanceManager.java fea2989
> prism/src/main/java/org/apache/falcon/resource/AbstractSchedulableEntityManager.java d317aa1
> scheduler/src/test/java/org/apache/falcon/execution/FalconExecutionServiceTest.java b2f9e59
> scheduler/src/test/java/org/apache/falcon/workflow/engine/WorkflowEngineFactoryTest.java PRE-CREATION
> src/conf/startup.properties ce6e91f
>
> Diff: https://reviews.apache.org/r/39844/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>