You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Pallavi Rao <pa...@inmobi.com> on 2015/09/23 08:10:30 UTC
Re: Review Request 37520: FALCON-1231 Reliable notifications for Oozie
workflow status
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37520/
-----------------------------------------------------------
(Updated Sept. 23, 2015, 6:10 a.m.)
Review request for Falcon.
Changes
-------
Incorporated the additional requirement of notifying on wait and timeout.
Bugs: FALCON-1231
https://issues.apache.org/jira/browse/FALCON-1231
Repository: falcon-git
Description
-------
Workflow completion is currently dependent on post processing action putting messages (success/failure) to a JMS queue.
When failures occur on Hadoop / Oozie, the post processing action may not get executed either. In such cases, Falcon will be blind to the status of the workflow and re-tries will not happen.
Instead, use JMS notifications of Oozie to get job completion notifications.
The patch is backward compatible. It will handle notifications from post-processing and Oozie. That way users are not forced to perform any migration steps, but, can move to Oozie notifications in a phased manner.
Diffs (updated)
-----
client/src/main/java/org/apache/falcon/resource/InstancesResult.java 76bb4b0
common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java d0b09df
common/src/main/java/org/apache/falcon/entity/WorkflowNameBuilder.java 6890594
common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java ef9da45
common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java 9456fb9
common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239
common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionListener.java 2d3a477
common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java c4f8843
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 0b560bb
common/src/test/java/org/apache/falcon/entity/TestWorkflowNameBuilder.java 6060731
common/src/test/java/org/apache/falcon/workflow/WorkflowExecutionContextTest.java 65a057d
common/src/test/java/org/apache/falcon/workflow/WorkflowJobEndNotificationServiceTest.java b7df443
messaging/pom.xml 6528278
messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java d3178fb
messaging/src/test/java/org/apache/falcon/messaging/JMSMessageConsumerTest.java 5bbcc87
oozie-el-extensions/src/main/conf/oozie-site.xml 0925b41
oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java cff1187
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 0441f7c
oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java 9d31d17
rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java 785dce8
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java b952bbe
webapp/src/conf/oozie/conf/oozie-site.xml 8545ef9
Diff: https://reviews.apache.org/r/37520/diff/
Testing
-------
UTs added. Manual testing done.
Thanks,
Pallavi Rao
Re: Review Request 37520: FALCON-1231 Reliable notifications for Oozie
workflow status
Posted by Pallavi Rao <pa...@inmobi.com>.
> On Sept. 29, 2015, 12:20 a.m., Ying Zheng wrote:
> > messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java, line 225
> > <https://reviews.apache.org/r/37520/diff/3/?file=1082055#file1082055line225>
> >
> > As I experimented, coordinator actions can capture all status changes including the ones provided by workflow jobs. The notification from WORKFLOW_JOB doesn't seem necessary here.
The WORKFLOW_JOB is given preference and is used for 2 reasons:
1. Doesn't need a second call to Oozie to get the workflow id and corresponding conf.
2. When we move to native scheduler, Oozie coordinator is not going to be used. So, keeping the patch a little future looking.
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37520/#review100904
-----------------------------------------------------------
On Sept. 23, 2015, 6:10 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37520/
> -----------------------------------------------------------
>
> (Updated Sept. 23, 2015, 6:10 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1231
> https://issues.apache.org/jira/browse/FALCON-1231
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Workflow completion is currently dependent on post processing action putting messages (success/failure) to a JMS queue.
> When failures occur on Hadoop / Oozie, the post processing action may not get executed either. In such cases, Falcon will be blind to the status of the workflow and re-tries will not happen.
> Instead, use JMS notifications of Oozie to get job completion notifications.
>
> The patch is backward compatible. It will handle notifications from post-processing and Oozie. That way users are not forced to perform any migration steps, but, can move to Oozie notifications in a phased manner.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java 76bb4b0
> common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java d0b09df
> common/src/main/java/org/apache/falcon/entity/WorkflowNameBuilder.java 6890594
> common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java ef9da45
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java 9456fb9
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionListener.java 2d3a477
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java c4f8843
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 0b560bb
> common/src/test/java/org/apache/falcon/entity/TestWorkflowNameBuilder.java 6060731
> common/src/test/java/org/apache/falcon/workflow/WorkflowExecutionContextTest.java 65a057d
> common/src/test/java/org/apache/falcon/workflow/WorkflowJobEndNotificationServiceTest.java b7df443
> messaging/pom.xml 6528278
> messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java d3178fb
> messaging/src/test/java/org/apache/falcon/messaging/JMSMessageConsumerTest.java 5bbcc87
> oozie-el-extensions/src/main/conf/oozie-site.xml 0925b41
> oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java cff1187
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 0441f7c
> oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java 9d31d17
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java 785dce8
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java b952bbe
> webapp/src/conf/oozie/conf/oozie-site.xml 8545ef9
>
> Diff: https://reviews.apache.org/r/37520/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 37520: FALCON-1231 Reliable notifications for Oozie
workflow status
Posted by Ying Zheng <yz...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37520/#review100904
-----------------------------------------------------------
messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java (line 223)
<https://reviews.apache.org/r/37520/#comment158164>
As I experimented, coordinator actions can capture all status changes including the ones provided by workflow jobs. The notification from WORKFLOW_JOB doesn't seem necessary here.
- Ying Zheng
On Sept. 23, 2015, 6:10 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37520/
> -----------------------------------------------------------
>
> (Updated Sept. 23, 2015, 6:10 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1231
> https://issues.apache.org/jira/browse/FALCON-1231
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Workflow completion is currently dependent on post processing action putting messages (success/failure) to a JMS queue.
> When failures occur on Hadoop / Oozie, the post processing action may not get executed either. In such cases, Falcon will be blind to the status of the workflow and re-tries will not happen.
> Instead, use JMS notifications of Oozie to get job completion notifications.
>
> The patch is backward compatible. It will handle notifications from post-processing and Oozie. That way users are not forced to perform any migration steps, but, can move to Oozie notifications in a phased manner.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java 76bb4b0
> common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java d0b09df
> common/src/main/java/org/apache/falcon/entity/WorkflowNameBuilder.java 6890594
> common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java ef9da45
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java 9456fb9
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionListener.java 2d3a477
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java c4f8843
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 0b560bb
> common/src/test/java/org/apache/falcon/entity/TestWorkflowNameBuilder.java 6060731
> common/src/test/java/org/apache/falcon/workflow/WorkflowExecutionContextTest.java 65a057d
> common/src/test/java/org/apache/falcon/workflow/WorkflowJobEndNotificationServiceTest.java b7df443
> messaging/pom.xml 6528278
> messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java d3178fb
> messaging/src/test/java/org/apache/falcon/messaging/JMSMessageConsumerTest.java 5bbcc87
> oozie-el-extensions/src/main/conf/oozie-site.xml 0925b41
> oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java cff1187
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 0441f7c
> oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java 9d31d17
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java 785dce8
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java b952bbe
> webapp/src/conf/oozie/conf/oozie-site.xml 8545ef9
>
> Diff: https://reviews.apache.org/r/37520/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 37520: FALCON-1231 Reliable notifications for Oozie
workflow status
Posted by sandeep samudrala <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37520/#review101876
-----------------------------------------------------------
Ship it!
Ship It!
- sandeep samudrala
On Sept. 23, 2015, 6:10 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37520/
> -----------------------------------------------------------
>
> (Updated Sept. 23, 2015, 6:10 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1231
> https://issues.apache.org/jira/browse/FALCON-1231
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Workflow completion is currently dependent on post processing action putting messages (success/failure) to a JMS queue.
> When failures occur on Hadoop / Oozie, the post processing action may not get executed either. In such cases, Falcon will be blind to the status of the workflow and re-tries will not happen.
> Instead, use JMS notifications of Oozie to get job completion notifications.
>
> The patch is backward compatible. It will handle notifications from post-processing and Oozie. That way users are not forced to perform any migration steps, but, can move to Oozie notifications in a phased manner.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java 76bb4b0
> common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java d0b09df
> common/src/main/java/org/apache/falcon/entity/WorkflowNameBuilder.java 6890594
> common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java ef9da45
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java 9456fb9
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionListener.java 2d3a477
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java c4f8843
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 0b560bb
> common/src/test/java/org/apache/falcon/entity/TestWorkflowNameBuilder.java 6060731
> common/src/test/java/org/apache/falcon/workflow/WorkflowExecutionContextTest.java 65a057d
> common/src/test/java/org/apache/falcon/workflow/WorkflowJobEndNotificationServiceTest.java b7df443
> messaging/pom.xml 6528278
> messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java d3178fb
> messaging/src/test/java/org/apache/falcon/messaging/JMSMessageConsumerTest.java 5bbcc87
> oozie-el-extensions/src/main/conf/oozie-site.xml 0925b41
> oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java cff1187
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 0441f7c
> oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java 9d31d17
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java 785dce8
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java b952bbe
> webapp/src/conf/oozie/conf/oozie-site.xml 8545ef9
>
> Diff: https://reviews.apache.org/r/37520/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 37520: FALCON-1231 Reliable notifications for Oozie
workflow status
Posted by Pallavi Rao <pa...@inmobi.com>.
> On Sept. 28, 2015, 11:29 p.m., Ying Zheng wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java, line 139
> > <https://reviews.apache.org/r/37520/diff/3/?file=1082047#file1082047line139>
> >
> > Name it as "hasWorkflowKilled", to be consistent with "hasWorkflowFailed" "hasWorkflowTimedOut"?
Didn't sound right grammatically :-).. It should be hasWorkflowBeenKilled.. But, I can rename it for consistency.
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37520/#review100879
-----------------------------------------------------------
On Sept. 23, 2015, 6:10 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37520/
> -----------------------------------------------------------
>
> (Updated Sept. 23, 2015, 6:10 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1231
> https://issues.apache.org/jira/browse/FALCON-1231
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Workflow completion is currently dependent on post processing action putting messages (success/failure) to a JMS queue.
> When failures occur on Hadoop / Oozie, the post processing action may not get executed either. In such cases, Falcon will be blind to the status of the workflow and re-tries will not happen.
> Instead, use JMS notifications of Oozie to get job completion notifications.
>
> The patch is backward compatible. It will handle notifications from post-processing and Oozie. That way users are not forced to perform any migration steps, but, can move to Oozie notifications in a phased manner.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java 76bb4b0
> common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java d0b09df
> common/src/main/java/org/apache/falcon/entity/WorkflowNameBuilder.java 6890594
> common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java ef9da45
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java 9456fb9
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionListener.java 2d3a477
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java c4f8843
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 0b560bb
> common/src/test/java/org/apache/falcon/entity/TestWorkflowNameBuilder.java 6060731
> common/src/test/java/org/apache/falcon/workflow/WorkflowExecutionContextTest.java 65a057d
> common/src/test/java/org/apache/falcon/workflow/WorkflowJobEndNotificationServiceTest.java b7df443
> messaging/pom.xml 6528278
> messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java d3178fb
> messaging/src/test/java/org/apache/falcon/messaging/JMSMessageConsumerTest.java 5bbcc87
> oozie-el-extensions/src/main/conf/oozie-site.xml 0925b41
> oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java cff1187
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 0441f7c
> oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java 9d31d17
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java 785dce8
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java b952bbe
> webapp/src/conf/oozie/conf/oozie-site.xml 8545ef9
>
> Diff: https://reviews.apache.org/r/37520/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 37520: FALCON-1231 Reliable notifications for Oozie
workflow status
Posted by Ajay Yadava <aj...@gmail.com>.
> On Sept. 28, 2015, 11:29 p.m., Ying Zheng wrote:
> > common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java, line 139
> > <https://reviews.apache.org/r/37520/diff/3/?file=1082047#file1082047line139>
> >
> > Name it as "hasWorkflowKilled", to be consistent with "hasWorkflowFailed" "hasWorkflowTimedOut"?
>
> Pallavi Rao wrote:
> Didn't sound right grammatically :-).. It should be hasWorkflowBeenKilled.. But, I can rename it for consistency.
Just for the kicks, I recommend "hasWorkflowKickedTheBucket" (pun intended)! :P
- Ajay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37520/#review100879
-----------------------------------------------------------
On Sept. 23, 2015, 6:10 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37520/
> -----------------------------------------------------------
>
> (Updated Sept. 23, 2015, 6:10 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1231
> https://issues.apache.org/jira/browse/FALCON-1231
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Workflow completion is currently dependent on post processing action putting messages (success/failure) to a JMS queue.
> When failures occur on Hadoop / Oozie, the post processing action may not get executed either. In such cases, Falcon will be blind to the status of the workflow and re-tries will not happen.
> Instead, use JMS notifications of Oozie to get job completion notifications.
>
> The patch is backward compatible. It will handle notifications from post-processing and Oozie. That way users are not forced to perform any migration steps, but, can move to Oozie notifications in a phased manner.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java 76bb4b0
> common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java d0b09df
> common/src/main/java/org/apache/falcon/entity/WorkflowNameBuilder.java 6890594
> common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java ef9da45
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java 9456fb9
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionListener.java 2d3a477
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java c4f8843
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 0b560bb
> common/src/test/java/org/apache/falcon/entity/TestWorkflowNameBuilder.java 6060731
> common/src/test/java/org/apache/falcon/workflow/WorkflowExecutionContextTest.java 65a057d
> common/src/test/java/org/apache/falcon/workflow/WorkflowJobEndNotificationServiceTest.java b7df443
> messaging/pom.xml 6528278
> messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java d3178fb
> messaging/src/test/java/org/apache/falcon/messaging/JMSMessageConsumerTest.java 5bbcc87
> oozie-el-extensions/src/main/conf/oozie-site.xml 0925b41
> oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java cff1187
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 0441f7c
> oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java 9d31d17
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java 785dce8
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java b952bbe
> webapp/src/conf/oozie/conf/oozie-site.xml 8545ef9
>
> Diff: https://reviews.apache.org/r/37520/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 37520: FALCON-1231 Reliable notifications for Oozie
workflow status
Posted by Ying Zheng <yz...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37520/#review100879
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java (line 139)
<https://reviews.apache.org/r/37520/#comment158131>
Name it as "hasWorkflowKilled", to be consistent with "hasWorkflowFailed" "hasWorkflowTimedOut"?
- Ying Zheng
On Sept. 23, 2015, 6:10 a.m., Pallavi Rao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37520/
> -----------------------------------------------------------
>
> (Updated Sept. 23, 2015, 6:10 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1231
> https://issues.apache.org/jira/browse/FALCON-1231
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Workflow completion is currently dependent on post processing action putting messages (success/failure) to a JMS queue.
> When failures occur on Hadoop / Oozie, the post processing action may not get executed either. In such cases, Falcon will be blind to the status of the workflow and re-tries will not happen.
> Instead, use JMS notifications of Oozie to get job completion notifications.
>
> The patch is backward compatible. It will handle notifications from post-processing and Oozie. That way users are not forced to perform any migration steps, but, can move to Oozie notifications in a phased manner.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/resource/InstancesResult.java 76bb4b0
> common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java d0b09df
> common/src/main/java/org/apache/falcon/entity/WorkflowNameBuilder.java 6890594
> common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java ef9da45
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java 9456fb9
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionListener.java 2d3a477
> common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java c4f8843
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 0b560bb
> common/src/test/java/org/apache/falcon/entity/TestWorkflowNameBuilder.java 6060731
> common/src/test/java/org/apache/falcon/workflow/WorkflowExecutionContextTest.java 65a057d
> common/src/test/java/org/apache/falcon/workflow/WorkflowJobEndNotificationServiceTest.java b7df443
> messaging/pom.xml 6528278
> messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java d3178fb
> messaging/src/test/java/org/apache/falcon/messaging/JMSMessageConsumerTest.java 5bbcc87
> oozie-el-extensions/src/main/conf/oozie-site.xml 0925b41
> oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java cff1187
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 0441f7c
> oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java 9d31d17
> rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java 785dce8
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java b952bbe
> webapp/src/conf/oozie/conf/oozie-site.xml 8545ef9
>
> Diff: https://reviews.apache.org/r/37520/diff/
>
>
> Testing
> -------
>
> UTs added. Manual testing done.
>
>
> Thanks,
>
> Pallavi Rao
>
>
Re: Review Request 37520: FALCON-1231 Reliable notifications for Oozie
workflow status
Posted by Pallavi Rao <pa...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37520/
-----------------------------------------------------------
(Updated Oct. 8, 2015, 10:24 a.m.)
Review request for Falcon.
Changes
-------
Addressed review comments
Bugs: FALCON-1231
https://issues.apache.org/jira/browse/FALCON-1231
Repository: falcon-git
Description
-------
Workflow completion is currently dependent on post processing action putting messages (success/failure) to a JMS queue.
When failures occur on Hadoop / Oozie, the post processing action may not get executed either. In such cases, Falcon will be blind to the status of the workflow and re-tries will not happen.
Instead, use JMS notifications of Oozie to get job completion notifications.
The patch is backward compatible. It will handle notifications from post-processing and Oozie. That way users are not forced to perform any migration steps, but, can move to Oozie notifications in a phased manner.
Diffs (updated)
-----
client/src/main/java/org/apache/falcon/resource/InstancesResult.java 76bb4b0
common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java d0b09df
common/src/main/java/org/apache/falcon/entity/WorkflowNameBuilder.java 6890594
common/src/main/java/org/apache/falcon/metadata/MetadataMappingService.java ef9da45
common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionArgs.java 9456fb9
common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 4454239
common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionListener.java 2d3a477
common/src/main/java/org/apache/falcon/workflow/WorkflowJobEndNotificationService.java c4f8843
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 0b560bb
common/src/test/java/org/apache/falcon/entity/TestWorkflowNameBuilder.java 6060731
common/src/test/java/org/apache/falcon/workflow/WorkflowExecutionContextTest.java 65a057d
common/src/test/java/org/apache/falcon/workflow/WorkflowJobEndNotificationServiceTest.java b7df443
messaging/pom.xml 918de63
messaging/src/main/java/org/apache/falcon/messaging/JMSMessageConsumer.java d3178fb
messaging/src/test/java/org/apache/falcon/messaging/JMSMessageConsumerTest.java 7356ee3
oozie-el-extensions/src/main/conf/oozie-site.xml 0925b41
oozie/src/main/java/org/apache/falcon/workflow/FalconPostProcessing.java cff1187
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 6660af1
oozie/src/test/java/org/apache/falcon/oozie/workflow/FalconPostProcessingTest.java 9d31d17
rerun/src/main/java/org/apache/falcon/rerun/handler/LateRerunHandler.java 785dce8
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java b952bbe
webapp/src/conf/oozie/conf/oozie-site.xml 8545ef9
Diff: https://reviews.apache.org/r/37520/diff/
Testing
-------
UTs added. Manual testing done.
Thanks,
Pallavi Rao