You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by pavan kumar kolamuri <pa...@gmail.com> on 2015/12/15 16:08:55 UTC
Review Request 41401: Rerun API for Falcon Native Scheduler
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/
-----------------------------------------------------------
Review request for Falcon.
Repository: falcon-git
Description
-------
Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
Diffs
-----
client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
Diff: https://reviews.apache.org/r/41401/diff/
Testing
-------
Thanks,
pavan kumar kolamuri
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by pavan kumar kolamuri <pa...@gmail.com>.
> On Dec. 18, 2015, 5:20 a.m., Ajay Yadava wrote:
> > scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java, line 438
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168361#file1168361line438>
> >
> > instance is already cast to ProcessExecutionInstance in the above statement, so checking here is redundant. May be this check should go up.
Will fix
> On Dec. 18, 2015, 5:20 a.m., Ajay Yadava wrote:
> > scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java, line 34
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168363#file1168363line34>
> >
> > Similarly can we make change here also to accept only InstanceID?
Sure
> On Dec. 18, 2015, 5:20 a.m., Ajay Yadava wrote:
> > scheduler/src/main/java/org/apache/falcon/state/InstanceState.java, line 53
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168366#file1168366line53>
> >
> > Won't RERUN be more intuitive instead of EXTERNAL_TRIGGER?
We want to use same logic for extrenal triggers also which can be triggered by external service.
> On Dec. 18, 2015, 5:20 a.m., Ajay Yadava wrote:
> > scheduler/src/main/java/org/apache/falcon/state/InstanceState.java, line 54
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168366#file1168366line54>
> >
> > Shouldn't rerun only be valid for terminal states? By that logic for waiting instances, rerun should throw invalid state transition exception, right?
This is to make sure consistency with trigger and makes idempotent. As it is starting state
> On Dec. 18, 2015, 5:20 a.m., Ajay Yadava wrote:
> > scheduler/src/main/java/org/apache/falcon/state/InstanceState.java, line 196
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168366#file1168366line196>
> >
> > Will it be more intuitive to name it rerun?
Explained
> On Dec. 18, 2015, 5:20 a.m., Ajay Yadava wrote:
> > scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java, line 42
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168367#file1168367line42>
> >
> > Will it be more intuitive to rename it to onRerun?
Explained the reason
> On Dec. 18, 2015, 5:20 a.m., Ajay Yadava wrote:
> > scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java, line 145
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168369#file1168369line145>
> >
> > Isn't it instanceID?
No it is workflowID
- pavan kumar
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review110948
-----------------------------------------------------------
On Dec. 17, 2015, 9 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2015, 9 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review110948
-----------------------------------------------------------
client/src/main/java/org/apache/falcon/resource/InstancesResult.java (line 37)
<https://reviews.apache.org/r/41401/#comment171052>
Where is it getting used? Under what conditions is a workflow said to be in READY state?
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java (line 437)
<https://reviews.apache.org/r/41401/#comment171057>
instance is already cast to ProcessExecutionInstance in the above statement, so checking here is redundant. May be this check should go up.
scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java (line 28)
<https://reviews.apache.org/r/41401/#comment171000>
Won't it be just InstanceID?
scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java (line 34)
<https://reviews.apache.org/r/41401/#comment171001>
Similarly can we make change here also to accept only InstanceID?
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java (line 53)
<https://reviews.apache.org/r/41401/#comment170966>
Won't RERUN be more intuitive instead of EXTERNAL_TRIGGER?
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java (line 54)
<https://reviews.apache.org/r/41401/#comment171003>
Shouldn't rerun only be valid for terminal states? By that logic for waiting instances, rerun should throw invalid state transition exception, right?
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java (line 196)
<https://reviews.apache.org/r/41401/#comment171105>
Will it be more intuitive to name it rerun?
scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java (line 42)
<https://reviews.apache.org/r/41401/#comment171106>
Will it be more intuitive to rename it to onRerun?
scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java (line 145)
<https://reviews.apache.org/r/41401/#comment171107>
Isn't it instanceID?
- Ajay Yadava
On Dec. 17, 2015, 9 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2015, 9 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111151
-----------------------------------------------------------
Ship it!
FALCON-1674 has been created to address the remaining points. To unblock testing of native scheduler, I will commit this patch.
- Ajay Yadava
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Pallavi Rao <pa...@inmobi.com>.
> On Dec. 18, 2015, 8:01 a.m., Ajay Yadava wrote:
> > After an offline discussion with @pavan kumar kolamuri, here are some key points which I am not sure are the best way to do it.
> >
> > 1. A new WorkflowStatus has been added, since we are changing only scheduler, a new status for workflow(READY) looks odd to me. It is not used anywhere in OozieWorkflowEngine, so the two engines api are now out of sync.
> > 2. There is a bad coupling between two Enums - InstanceState.STATE, and InstancesResult.WorkflowStatus e.g. in the following code (It may not be the only place where this is being done, but I am taking this as I used it for discussion with Pavan Kumar Kolamuri)
> >
> > ```case RERUN:
> > executor.rerun(instance, userProps, isForced);
> > instanceInfo.status =
> > InstancesResult.WorkflowStatus.valueOf(STATE_STORE
> > .getExecutionInstance(instance.getId()).getCurrentState().name());
> > break;
> > ```
> > STATE_STORE.getExecutionInstance(instance.getId()).getCurrentState() returns InstanceState.STATE and it's name is used to generate an enum of type InstancesResult.WorkflowStatus. Both the enums have some states which don't make sense for others.
> >
> > InstancesResult.WorkflowStatus was earlier used to return workflow status(which was never ready), now it seems to be used to return status for what earlier were Coordinator Action status which were. I think we should maintain that separation for the sake of compatibility across two engines.
+1 to your 2nd point.
Regarding your 1st point, we will need to expose a READY status to users when we absorb the scheduling. If we do not add READY to WorkflowStatus and introduce a new Status (lets say, InstanceStatus), the InstanceResult will need to be changed to handle this new type and then we'll have to worry about how we merge these 2 when we send it to the user. It is easier to preserve backward-compatibility with the addition of READY.
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111143
-----------------------------------------------------------
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Ajay Yadava <aj...@gmail.com>.
> On Dec. 18, 2015, 8:01 a.m., Ajay Yadava wrote:
> > After an offline discussion with @pavan kumar kolamuri, here are some key points which I am not sure are the best way to do it.
> >
> > 1. A new WorkflowStatus has been added, since we are changing only scheduler, a new status for workflow(READY) looks odd to me. It is not used anywhere in OozieWorkflowEngine, so the two engines api are now out of sync.
> > 2. There is a bad coupling between two Enums - InstanceState.STATE, and InstancesResult.WorkflowStatus e.g. in the following code (It may not be the only place where this is being done, but I am taking this as I used it for discussion with Pavan Kumar Kolamuri)
> >
> > ```case RERUN:
> > executor.rerun(instance, userProps, isForced);
> > instanceInfo.status =
> > InstancesResult.WorkflowStatus.valueOf(STATE_STORE
> > .getExecutionInstance(instance.getId()).getCurrentState().name());
> > break;
> > ```
> > STATE_STORE.getExecutionInstance(instance.getId()).getCurrentState() returns InstanceState.STATE and it's name is used to generate an enum of type InstancesResult.WorkflowStatus. Both the enums have some states which don't make sense for others.
> >
> > InstancesResult.WorkflowStatus was earlier used to return workflow status(which was never ready), now it seems to be used to return status for what earlier were Coordinator Action status which were. I think we should maintain that separation for the sake of compatibility across two engines.
>
> Pallavi Rao wrote:
> +1 to your 2nd point.
>
> Regarding your 1st point, we will need to expose a READY status to users when we absorb the scheduling. If we do not add READY to WorkflowStatus and introduce a new Status (lets say, InstanceStatus), the InstanceResult will need to be changed to handle this new type and then we'll have to worry about how we merge these 2 when we send it to the user. It is easier to preserve backward-compatibility with the addition of READY.
>
> Pallavi Rao wrote:
> WorkflowStatus itself is a wrong variable name in the first place, coz., WAITING is not really not a "Workflow" status either. It is more of a co-ordinator function. It should have ideally been "InstanceStatus"
Makes sense. But since we will now be showing Users a new status "READY" we should document it(all?) with detailed explanation.
- Ajay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111143
-----------------------------------------------------------
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Pallavi Rao <pa...@inmobi.com>.
> On Dec. 18, 2015, 8:01 a.m., Ajay Yadava wrote:
> > After an offline discussion with @pavan kumar kolamuri, here are some key points which I am not sure are the best way to do it.
> >
> > 1. A new WorkflowStatus has been added, since we are changing only scheduler, a new status for workflow(READY) looks odd to me. It is not used anywhere in OozieWorkflowEngine, so the two engines api are now out of sync.
> > 2. There is a bad coupling between two Enums - InstanceState.STATE, and InstancesResult.WorkflowStatus e.g. in the following code (It may not be the only place where this is being done, but I am taking this as I used it for discussion with Pavan Kumar Kolamuri)
> >
> > ```case RERUN:
> > executor.rerun(instance, userProps, isForced);
> > instanceInfo.status =
> > InstancesResult.WorkflowStatus.valueOf(STATE_STORE
> > .getExecutionInstance(instance.getId()).getCurrentState().name());
> > break;
> > ```
> > STATE_STORE.getExecutionInstance(instance.getId()).getCurrentState() returns InstanceState.STATE and it's name is used to generate an enum of type InstancesResult.WorkflowStatus. Both the enums have some states which don't make sense for others.
> >
> > InstancesResult.WorkflowStatus was earlier used to return workflow status(which was never ready), now it seems to be used to return status for what earlier were Coordinator Action status which were. I think we should maintain that separation for the sake of compatibility across two engines.
>
> Pallavi Rao wrote:
> +1 to your 2nd point.
>
> Regarding your 1st point, we will need to expose a READY status to users when we absorb the scheduling. If we do not add READY to WorkflowStatus and introduce a new Status (lets say, InstanceStatus), the InstanceResult will need to be changed to handle this new type and then we'll have to worry about how we merge these 2 when we send it to the user. It is easier to preserve backward-compatibility with the addition of READY.
WorkflowStatus itself is a wrong variable name in the first place, coz., WAITING is not really not a "Workflow" status either. It is more of a co-ordinator function. It should have ideally been "InstanceStatus"
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111143
-----------------------------------------------------------
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111143
-----------------------------------------------------------
After an offline discussion with @pavan kumar kolamuri, here are some key points which I am not sure are the best way to do it.
1. A new WorkflowStatus has been added, since we are changing only scheduler, a new status for workflow(READY) looks odd to me. It is not used anywhere in OozieWorkflowEngine, so the two engines api are now out of sync.
2. There is a bad coupling between two Enums - InstanceState.STATE, and InstancesResult.WorkflowStatus e.g. in the following code (It may not be the only place where this is being done, but I am taking this as I used it for discussion with Pavan Kumar Kolamuri)
```case RERUN:
executor.rerun(instance, userProps, isForced);
instanceInfo.status =
InstancesResult.WorkflowStatus.valueOf(STATE_STORE
.getExecutionInstance(instance.getId()).getCurrentState().name());
break;
```
STATE_STORE.getExecutionInstance(instance.getId()).getCurrentState() returns InstanceState.STATE and it's name is used to generate an enum of type InstancesResult.WorkflowStatus. Both the enums have some states which don't make sense for others.
InstancesResult.WorkflowStatus was earlier used to return workflow status(which was never ready), now it seems to be used to return status for what earlier were Coordinator Action status which were. I think we should maintain that separation for the sake of compatibility across two engines.
- Ajay Yadava
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Pallavi Rao <pa...@inmobi.com>.
> On Dec. 18, 2015, 8:09 a.m., Ajay Yadava wrote:
> > This is how the mapping was done earlier
> > ```private String mapActionStatus(String status) {
> > if (CoordinatorAction.Status.READY.toString().equals(status)
> > || CoordinatorAction.Status.WAITING.toString().equals(status)
> > || CoordinatorAction.Status.SUBMITTED.toString().equals(status)) {
> > return InstancesResult.WorkflowStatus.WAITING.name();
> > } else if (CoordinatorAction.Status.IGNORED.toString().equals(status)) {
> > return InstancesResult.WorkflowStatus.KILLED.name();
> > } else if (CoordinatorAction.Status.TIMEDOUT.toString().equals(status)) {
> > return InstancesResult.WorkflowStatus.FAILED.name();
> > } else if (WorkflowJob.Status.PREP.toString().equals(status)) {
> > return InstancesResult.WorkflowStatus.RUNNING.name();
> > } else {
> > return status;
> > }
> > }
> >
> > ```
> >
> > I think we should use the same approach and maintain consistency between the two engines.
+1 to that. We should have a similar method that translates a STATE to WorkflowStatus
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111144
-----------------------------------------------------------
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111144
-----------------------------------------------------------
This is how the mapping was done earlier
```private String mapActionStatus(String status) {
if (CoordinatorAction.Status.READY.toString().equals(status)
|| CoordinatorAction.Status.WAITING.toString().equals(status)
|| CoordinatorAction.Status.SUBMITTED.toString().equals(status)) {
return InstancesResult.WorkflowStatus.WAITING.name();
} else if (CoordinatorAction.Status.IGNORED.toString().equals(status)) {
return InstancesResult.WorkflowStatus.KILLED.name();
} else if (CoordinatorAction.Status.TIMEDOUT.toString().equals(status)) {
return InstancesResult.WorkflowStatus.FAILED.name();
} else if (WorkflowJob.Status.PREP.toString().equals(status)) {
return InstancesResult.WorkflowStatus.RUNNING.name();
} else {
return status;
}
}
```
I think we should use the same approach and maintain consistency between the two engines.
- Ajay Yadava
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by pavan kumar kolamuri <pa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/
-----------------------------------------------------------
(Updated Dec. 18, 2015, 9:58 a.m.)
Review request for Falcon.
Changes
-------
Rebased Patch
Repository: falcon-git
Description
-------
Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
Diffs (updated)
-----
client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java 88d88c1
scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 5f96d3f
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java cff4a73
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java ace8444
scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java ebc05ec
scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java 5c7bf91
scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java ca2010b
scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java 087114f
scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
Diff: https://reviews.apache.org/r/41401/diff/
Testing
-------
Thanks,
pavan kumar kolamuri
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by pavan kumar kolamuri <pa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/
-----------------------------------------------------------
(Updated Dec. 18, 2015, 6:45 a.m.)
Review request for Falcon.
Changes
-------
Addressed Comments
Repository: falcon-git
Description
-------
Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
Diffs (updated)
-----
client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
Diff: https://reviews.apache.org/r/41401/diff/
Testing
-------
Thanks,
pavan kumar kolamuri
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Ajay Yadava <aj...@gmail.com>.
> On Dec. 18, 2015, 6:37 a.m., pavan kumar kolamuri wrote:
> > scheduler/src/main/java/org/apache/falcon/state/InstanceState.java, line 53
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168366#file1168366line53>
> >
> > We want to use same logic for extrenal triggers also which can be triggered by external service.
It is still advisable to have two different events, we can still avoid code duplication by having multiple event types in case statement or calling same methods but the flows will be much clearer and they will also be easy to separate later on if you want special handling in case of only rerun.
> On Dec. 18, 2015, 6:37 a.m., pavan kumar kolamuri wrote:
> > client/src/main/java/org/apache/falcon/resource/InstancesResult.java, line 37
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168357#file1168357line37>
> >
> > After rerunning instances, we are displaying result with status. That time instance might be in Ready state. Please check in Falconworkflowengine rerun api.
My question is not on code flow but more on conditions. For example workflow status can be waiting - when workflow is waiting for data to be available. So what is the difference between READY and waiting? Is it that it is not running because of concurrency constraints?
- Ajay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111125
-----------------------------------------------------------
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Ajay Yadava <aj...@gmail.com>.
> On Dec. 18, 2015, 6:37 a.m., pavan kumar kolamuri wrote:
> > scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java, line 145
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168369#file1168369line145>
> >
> > No, its actual externalID workflowId. We have already method for ExternalID
In FalconWorkflowEngine.java
```InstancesResult.WorkflowStatus.valueOf(STATE_STORE
.getExecutionInstance(instance.getId()).getCurrentState().name());```
instance.getID() returns InstanceID.
- Ajay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111125
-----------------------------------------------------------
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by pavan kumar kolamuri <pa...@gmail.com>.
> On Dec. 18, 2015, 6:37 a.m., pavan kumar kolamuri wrote:
> > scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java, line 145
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168369#file1168369line145>
> >
> > No, its actual externalID workflowId. We have already method for ExternalID
>
> Ajay Yadava wrote:
> In FalconWorkflowEngine.java
> ```InstancesResult.WorkflowStatus.valueOf(STATE_STORE
> .getExecutionInstance(instance.getId()).getCurrentState().name());```
> instance.getID() returns InstanceID.
This is not the place this is used , this is used in
@Override
public void reRun(String cluster, String jobId, Properties props, boolean isForced) throws FalconException {
InstanceState instanceState = STATE_STORE.getExecutionInstance(jobId);
- pavan kumar
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111125
-----------------------------------------------------------
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Ajay Yadava <aj...@gmail.com>.
- Ajay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111125
-----------------------------------------------------------
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Ajay Yadava <aj...@gmail.com>.
> On Dec. 18, 2015, 6:37 a.m., pavan kumar kolamuri wrote:
> > scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java, line 145
> > <https://reviews.apache.org/r/41401/diff/3/?file=1168369#file1168369line145>
> >
> > No, its actual externalID workflowId. We have already method for ExternalID
>
> Ajay Yadava wrote:
> In FalconWorkflowEngine.java
> ```InstancesResult.WorkflowStatus.valueOf(STATE_STORE
> .getExecutionInstance(instance.getId()).getCurrentState().name());```
> instance.getID() returns InstanceID.
>
> pavan kumar kolamuri wrote:
> This is not the place this is used , this is used in
> @Override
> public void reRun(String cluster, String jobId, Properties props, boolean isForced) throws FalconException {
> InstanceState instanceState = STATE_STORE.getExecutionInstance(jobId);
So you are using the same method for fetching through instanceID and through externalID as well?
- Ajay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111125
-----------------------------------------------------------
On Dec. 18, 2015, 6:45 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 6:45 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by pavan kumar kolamuri <pa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review111125
-----------------------------------------------------------
client/src/main/java/org/apache/falcon/resource/InstancesResult.java (line 37)
<https://reviews.apache.org/r/41401/#comment171115>
After rerunning instances, we are displaying result with status. That time instance might be in Ready state. Please check in Falconworkflowengine rerun api.
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java (line 437)
<https://reviews.apache.org/r/41401/#comment171116>
Makes sense will remove
scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java (line 28)
<https://reviews.apache.org/r/41401/#comment171118>
Sure will change
scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java (line 34)
<https://reviews.apache.org/r/41401/#comment171119>
Sure will change
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java (line 53)
<https://reviews.apache.org/r/41401/#comment171120>
We want to use same logic for extrenal triggers also which can be triggered by external service.
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java (line 54)
<https://reviews.apache.org/r/41401/#comment171123>
This is to make sure consistency with trigger and makes idempotent. As it is starting state
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java (line 196)
<https://reviews.apache.org/r/41401/#comment171124>
Explained in previous comments
scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java (line 42)
<https://reviews.apache.org/r/41401/#comment171126>
Explained
scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java (line 145)
<https://reviews.apache.org/r/41401/#comment171127>
No, its actual externalID workflowId. We have already method for ExternalID
- pavan kumar kolamuri
On Dec. 17, 2015, 9 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2015, 9 a.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by pavan kumar kolamuri <pa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/
-----------------------------------------------------------
(Updated Dec. 17, 2015, 9 a.m.)
Review request for Falcon.
Changes
-------
Added description to Rerun API as said in comments
Repository: falcon-git
Description
-------
Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
Diffs (updated)
-----
client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
Diff: https://reviews.apache.org/r/41401/diff/
Testing
-------
Thanks,
pavan kumar kolamuri
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Pallavi Rao <pa...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review110913
-----------------------------------------------------------
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java (line 273)
<https://reviews.apache.org/r/41401/#comment170909>
Nit: Please add some comments to indicate that this API relies on the DAGEngine to maintain a details of reruns and no rerun information is stored in Falcon.
- Pallavi Rao
On Dec. 16, 2015, 12:30 p.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2015, 12:30 p.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Pallavi Rao <pa...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review110914
-----------------------------------------------------------
Ship it!
Please add the additional details (comments) that I have requested for. Looks good otherwise.
- Pallavi Rao
On Dec. 16, 2015, 12:30 p.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2015, 12:30 p.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by pavan kumar kolamuri <pa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/
-----------------------------------------------------------
(Updated Dec. 16, 2015, 12:30 p.m.)
Review request for Falcon.
Changes
-------
Addressed comments
Repository: falcon-git
Description
-------
Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
Diffs (updated)
-----
client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
Diff: https://reviews.apache.org/r/41401/diff/
Testing
-------
Thanks,
pavan kumar kolamuri
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by pavan kumar kolamuri <pa...@gmail.com>.
> On Dec. 16, 2015, 7:29 a.m., Pallavi Rao wrote:
> > scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java, line 112
> > <https://reviews.apache.org/r/41401/diff/1/?file=1164741#file1164741line112>
> >
> > Not a good idea to expose this directly. It should remain a private method. If you need this to be invoked, you can add another method (rerun, may be) and have it invoke this.
Sure will fix
> On Dec. 16, 2015, 7:29 a.m., Pallavi Rao wrote:
> > scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java, line 284
> > <https://reviews.apache.org/r/41401/diff/1/?file=1164742#file1164742line284>
> >
> > If we update the existing instance, we will be relying on Oozie to give us all the runs for an instance and details on each run.
> >
> > Have you evaluated this approach against creating a brand new instance every time and Falcon having the information that of runs and run-info?
As discussed offline, We are using existing workflow it needed to run from failed actions and we will get retry count from oozie
> On Dec. 16, 2015, 7:29 a.m., Pallavi Rao wrote:
> > scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java, line 286
> > <https://reviews.apache.org/r/41401/diff/1/?file=1164742#file1164742line286>
> >
> > You don't need to update the STATE_STORE explicitly. It is the responsibility of the StateService to do that (handleStateChange method). Just update the cache let the State transition take care of updating the state.
Sure will fix
> On Dec. 16, 2015, 7:29 a.m., Pallavi Rao wrote:
> > scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java, line 441
> > <https://reviews.apache.org/r/41401/diff/1/?file=1164742#file1164742line441>
> >
> > As mentioned above, create a new "trigger" method in ProcessExecutionInstance and have that invoke this method.
Ok will fix
> On Dec. 16, 2015, 7:29 a.m., Pallavi Rao wrote:
> > scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java, line 282
> > <https://reviews.apache.org/r/41401/diff/1/?file=1164745#file1164745line282>
> >
> > We already have the instance, why are we retrieving it again from state store?
> >
> > If the rerun method of ProcessExecutor updates its cache, the instance will contain the properties.
Will do
> On Dec. 16, 2015, 7:29 a.m., Pallavi Rao wrote:
> > scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java, line 286
> > <https://reviews.apache.org/r/41401/diff/1/?file=1164745#file1164745line286>
> >
> > Shouldn't this be FALCON_FORCE_RERUN?
Yes my bad missed it
> On Dec. 16, 2015, 7:29 a.m., Pallavi Rao wrote:
> > scheduler/src/main/java/org/apache/falcon/state/StateService.java, line 133
> > <https://reviews.apache.org/r/41401/diff/1/?file=1164749#file1164749line133>
> >
> > Shouldn't this be EVENT.EXTERNAL_TRIGGER?
Yes thanks for catching
> On Dec. 16, 2015, 7:29 a.m., Pallavi Rao wrote:
> > scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java, line 275
> > <https://reviews.apache.org/r/41401/diff/1/?file=1164754#file1164754line275>
> >
> > Add empty properties check too.
Ok will add
> On Dec. 16, 2015, 7:29 a.m., Pallavi Rao wrote:
> > scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java, line 246
> > <https://reviews.apache.org/r/41401/diff/1/?file=1164757#file1164757line246>
> >
> > Why remove these? These shouldn't be present in the first place, right?
We are rerunning by taking user params also, we are just making sure it will rerun the workflow always even though it wont be set in code.
- pavan kumar
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review110629
-----------------------------------------------------------
On Dec. 15, 2015, 3:08 p.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2015, 3:08 p.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by Pallavi Rao <pa...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review110629
-----------------------------------------------------------
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java (line 112)
<https://reviews.apache.org/r/41401/#comment170609>
Not a good idea to expose this directly. It should remain a private method. If you need this to be invoked, you can add another method (rerun, may be) and have it invoke this.
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java (line 284)
<https://reviews.apache.org/r/41401/#comment170636>
If we update the existing instance, we will be relying on Oozie to give us all the runs for an instance and details on each run.
Have you evaluated this approach against creating a brand new instance every time and Falcon having the information that of runs and run-info?
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java (line 286)
<https://reviews.apache.org/r/41401/#comment170627>
You don't need to update the STATE_STORE explicitly. It is the responsibility of the StateService to do that (handleStateChange method). Just update the cache let the State transition take care of updating the state.
scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java (line 440)
<https://reviews.apache.org/r/41401/#comment170610>
As mentioned above, create a new "trigger" method in ProcessExecutionInstance and have that invoke this method.
scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java (line 281)
<https://reviews.apache.org/r/41401/#comment170619>
We already have the instance, why are we retrieving it again from state store?
If the rerun method of ProcessExecutor updates its cache, the instance will contain the properties.
scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java (line 285)
<https://reviews.apache.org/r/41401/#comment170612>
Shouldn't this be FALCON_FORCE_RERUN?
scheduler/src/main/java/org/apache/falcon/state/StateService.java (line 133)
<https://reviews.apache.org/r/41401/#comment170618>
Shouldn't this be EVENT.EXTERNAL_TRIGGER?
scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java (line 275)
<https://reviews.apache.org/r/41401/#comment170631>
Add empty properties check too.
scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java (line 246)
<https://reviews.apache.org/r/41401/#comment170634>
Why remove these? These shouldn't be present in the first place, right?
- Pallavi Rao
On Dec. 15, 2015, 3:08 p.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2015, 3:08 p.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by pavan kumar kolamuri <pa...@gmail.com>.
> On Dec. 16, 2015, 9:57 a.m., sandeep samudrala wrote:
> > scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java, line 250
> > <https://reviews.apache.org/r/41401/diff/1/?file=1164757#file1164757line250>
> >
> > Can details be added regarding the what entity and rerun being issued for which instance?
> >
> > may be instanceId?
Yes Makes sense . Thanks will add for other methods also
- pavan kumar
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review110642
-----------------------------------------------------------
On Dec. 15, 2015, 3:08 p.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2015, 3:08 p.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 41401: Rerun API for Falcon Native Scheduler
Posted by sandeep samudrala <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41401/#review110642
-----------------------------------------------------------
scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java (line 250)
<https://reviews.apache.org/r/41401/#comment170658>
Can details be added regarding the what entity and rerun being issued for which instance?
may be instanceId?
- sandeep samudrala
On Dec. 15, 2015, 3:08 p.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41401/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2015, 3:08 p.m.)
>
>
> Review request for Falcon.
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently rerun API was not supported in case of Native Scheduler in Falcon. Rerun of instances should be implemented in FalconWorkflowEngine.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java e05eeeb
> scheduler/src/main/java/org/apache/falcon/execution/EntityExecutor.java c9c0f42
> scheduler/src/main/java/org/apache/falcon/execution/ExecutionInstance.java 3cc8a25
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutionInstance.java f3beabc
> scheduler/src/main/java/org/apache/falcon/execution/ProcessExecutor.java e446069
> scheduler/src/main/java/org/apache/falcon/notification/service/event/EventType.java 59f5cba
> scheduler/src/main/java/org/apache/falcon/notification/service/event/RerunEvent.java PRE-CREATION
> scheduler/src/main/java/org/apache/falcon/notification/service/impl/SchedulerService.java fb11091
> scheduler/src/main/java/org/apache/falcon/predicate/Predicate.java 164fb0e
> scheduler/src/main/java/org/apache/falcon/state/InstanceState.java 7f2bda9
> scheduler/src/main/java/org/apache/falcon/state/InstanceStateChangeHandler.java 1f69fab
> scheduler/src/main/java/org/apache/falcon/state/StateService.java c702cc3
> scheduler/src/main/java/org/apache/falcon/state/store/InMemoryStateStore.java 2f3aa3a
> scheduler/src/main/java/org/apache/falcon/state/store/InstanceStateStore.java f1d1931
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/BeanMapperUtil.java 4bee269
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/InstanceBean.java 0e3dfa9
> scheduler/src/main/java/org/apache/falcon/state/store/jdbc/JDBCStateStore.java b2f8e80
> scheduler/src/main/java/org/apache/falcon/workflow/engine/DAGEngine.java e0d2a0e
> scheduler/src/main/java/org/apache/falcon/workflow/engine/FalconWorkflowEngine.java c19cada
> scheduler/src/main/java/org/apache/falcon/workflow/engine/OozieDAGEngine.java a26eb77
> scheduler/src/test/java/org/apache/falcon/execution/MockDAGEngine.java d274ad7
> scheduler/src/test/java/org/apache/falcon/state/service/store/TestJDBCStateStore.java 6d5bd49
> webapp/src/test/java/org/apache/falcon/resource/AbstractSchedulerManagerJerseyIT.java f5bcc54
> webapp/src/test/java/org/apache/falcon/resource/InstanceSchedulerManagerJerseyIT.java 7959b63
> webapp/src/test/resources/local-process-noinputs-template.xml aabdc6a
>
> Diff: https://reviews.apache.org/r/41401/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>