You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Balu Vellanki <bv...@hortonworks.com> on 2015/09/03 02:29:10 UTC
Review Request 38082: Status of scheduled Process entity is shown as
submitted in corner case
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38082/
-----------------------------------------------------------
Review request for Falcon, Ajay Yadava, Sowmya Ramesh, and Venkat Ranganathan.
Bugs: FALCON-1371
https://issues.apache.org/jira/browse/FALCON-1371
Repository: falcon-git
Description
-------
Create a process entity with
- frequency of 1 hour
- validity start as "2012-07-01T00:00Z"
- and validity end as "2012-07-01T01:02Z"
Now list the process entity after one hour. The process coordinator status in Oozie is "succeeded". But Falcon shows the status as submitted.
Diffs
-----
common/pom.xml 42facc9
common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 4d45cc7
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 7e6cd6c
prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 78964dd
Diff: https://reviews.apache.org/r/38082/diff/
Testing
-------
This patch was tested end2end for the following scenarios.
1. When a bundle completed running successfully, status is now shown as "Succeeded" instead of "Submitted"
2. When the following feed is submitted, a coordinator is created for replication on backupCluster, but no coordinator is created on primaryCluster for retention because the feed validity on primaryCluster is in the past. There was a bug in code which was causing feed status to be shown as "submitted" if any of the clusters have a MISSING bundle. After this fix, the status will be shown as running.
{code}
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="cleansedEmailFeed" description="Cleansed customer emails" xmlns="uri:falcon:feed:0.1">
<groups>churnAnalysisDataPipeline</groups>
<frequency>hours(1)</frequency>
<timezone>UTC</timezone>
<late-arrival cut-off="hours(4)"/>
<clusters>
<cluster name="primaryCluster" type="source">
<validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
<retention limit="hours(90)" action="delete"/>
<locations>
<location type="data" path="/user/ambari-qa/falcon/demo/primary/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
<location type="stats" path="/"/>
<location type="meta" path="/"/>
</locations>
</cluster>
<cluster name="backupCluster" type="target">
<validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
<retention limit="hours(90)" action="delete"/>
<locations>
<location type="data" path="/falcon/demo/bcp/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
<location type="stats" path="/"/>
<location type="meta" path="/"/>
</locations>
</cluster>
</clusters>
<locations>
<location type="data" path="/user/ambari-qa/falcon/demo/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
<location type="stats" path="/"/>
<location type="meta" path="/"/>
</locations>
<ACL owner="ambari-qa" group="users" permission="0x755"/>
<schema location="/none" provider="/none"/>
</feed>
{code}
3. If a bundle is killed using oozie command line, the status of entity will be now shown as KILLED
4. Added support for FAILED.
5. Added methods to avoid calling findLatestBundle(...) repeatedly.
Thanks,
Balu Vellanki
Re: Review Request 38082: Status of scheduled Process entity is shown
as submitted in corner case
Posted by Pallavi Rao <pa...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38082/#review97609
-----------------------------------------------------------
Ship it!
Ship It!
- Pallavi Rao
On Sept. 3, 2015, 12:29 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38082/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2015, 12:29 a.m.)
>
>
> Review request for Falcon, Ajay Yadava, Sowmya Ramesh, and Venkat Ranganathan.
>
>
> Bugs: FALCON-1371
> https://issues.apache.org/jira/browse/FALCON-1371
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Create a process entity with
> - frequency of 1 hour
> - validity start as "2012-07-01T00:00Z"
> - and validity end as "2012-07-01T01:02Z"
>
> Now list the process entity after one hour. The process coordinator status in Oozie is "succeeded". But Falcon shows the status as submitted.
>
>
> Diffs
> -----
>
> common/pom.xml 42facc9
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 4d45cc7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 7e6cd6c
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 78964dd
>
> Diff: https://reviews.apache.org/r/38082/diff/
>
>
> Testing
> -------
>
> This patch was tested end2end for the following scenarios.
>
> 1. When a bundle completed running successfully, status is now shown as "Succeeded" instead of "Submitted"
>
> 2. When the following feed is submitted, a coordinator is created for replication on backupCluster, but no coordinator is created on primaryCluster for retention because the feed validity on primaryCluster is in the past. There was a bug in code which was causing feed status to be shown as "submitted" if any of the clusters have a MISSING bundle. After this fix, the status will be shown as running.
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <feed name="cleansedEmailFeed" description="Cleansed customer emails" xmlns="uri:falcon:feed:0.1">
> <groups>churnAnalysisDataPipeline</groups>
> <frequency>hours(1)</frequency>
> <timezone>UTC</timezone>
> <late-arrival cut-off="hours(4)"/>
> <clusters>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
> <retention limit="hours(90)" action="delete"/>
> <locations>
> <location type="data" path="/user/ambari-qa/falcon/demo/primary/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> </cluster>
> <cluster name="backupCluster" type="target">
> <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
> <retention limit="hours(90)" action="delete"/>
> <locations>
> <location type="data" path="/falcon/demo/bcp/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> </cluster>
> </clusters>
> <locations>
> <location type="data" path="/user/ambari-qa/falcon/demo/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> <ACL owner="ambari-qa" group="users" permission="0x755"/>
> <schema location="/none" provider="/none"/>
> </feed>
> {code}
>
> 3. If a bundle is killed using oozie command line, the status of entity will be now shown as KILLED
>
> 4. Added support for FAILED.
>
> 5. Added methods to avoid calling findLatestBundle(...) repeatedly.
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 38082: Status of scheduled Process entity is shown
as submitted in corner case
Posted by Sowmya Ramesh <sr...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38082/#review98267
-----------------------------------------------------------
Ship it!
Ship It!
- Sowmya Ramesh
On Sept. 3, 2015, 12:29 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38082/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2015, 12:29 a.m.)
>
>
> Review request for Falcon, Ajay Yadava, Sowmya Ramesh, and Venkat Ranganathan.
>
>
> Bugs: FALCON-1371
> https://issues.apache.org/jira/browse/FALCON-1371
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Create a process entity with
> - frequency of 1 hour
> - validity start as "2012-07-01T00:00Z"
> - and validity end as "2012-07-01T01:02Z"
>
> Now list the process entity after one hour. The process coordinator status in Oozie is "succeeded". But Falcon shows the status as submitted.
>
>
> Diffs
> -----
>
> common/pom.xml 42facc9
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 4d45cc7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 7e6cd6c
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 78964dd
>
> Diff: https://reviews.apache.org/r/38082/diff/
>
>
> Testing
> -------
>
> This patch was tested end2end for the following scenarios.
>
> 1. When a bundle completed running successfully, status is now shown as "Succeeded" instead of "Submitted"
>
> 2. When the following feed is submitted, a coordinator is created for replication on backupCluster, but no coordinator is created on primaryCluster for retention because the feed validity on primaryCluster is in the past. There was a bug in code which was causing feed status to be shown as "submitted" if any of the clusters have a MISSING bundle. After this fix, the status will be shown as running.
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <feed name="cleansedEmailFeed" description="Cleansed customer emails" xmlns="uri:falcon:feed:0.1">
> <groups>churnAnalysisDataPipeline</groups>
> <frequency>hours(1)</frequency>
> <timezone>UTC</timezone>
> <late-arrival cut-off="hours(4)"/>
> <clusters>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
> <retention limit="hours(90)" action="delete"/>
> <locations>
> <location type="data" path="/user/ambari-qa/falcon/demo/primary/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> </cluster>
> <cluster name="backupCluster" type="target">
> <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
> <retention limit="hours(90)" action="delete"/>
> <locations>
> <location type="data" path="/falcon/demo/bcp/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> </cluster>
> </clusters>
> <locations>
> <location type="data" path="/user/ambari-qa/falcon/demo/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> <ACL owner="ambari-qa" group="users" permission="0x755"/>
> <schema location="/none" provider="/none"/>
> </feed>
> {code}
>
> 3. If a bundle is killed using oozie command line, the status of entity will be now shown as KILLED
>
> 4. Added support for FAILED.
>
> 5. Added methods to avoid calling findLatestBundle(...) repeatedly.
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 38082: Status of scheduled Process entity is shown
as submitted in corner case
Posted by Balu Vellanki <bv...@hortonworks.com>.
> On Sept. 3, 2015, 2:42 a.m., Venkat Ranganathan wrote:
> > Good work Balu. Can you file the follow on JIRA for the issue with validity issue?
Hi Venkat - Falcon-1339 handles this issue.
- Balu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38082/#review97600
-----------------------------------------------------------
On Sept. 3, 2015, 12:29 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38082/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2015, 12:29 a.m.)
>
>
> Review request for Falcon, Ajay Yadava, Sowmya Ramesh, and Venkat Ranganathan.
>
>
> Bugs: FALCON-1371
> https://issues.apache.org/jira/browse/FALCON-1371
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Create a process entity with
> - frequency of 1 hour
> - validity start as "2012-07-01T00:00Z"
> - and validity end as "2012-07-01T01:02Z"
>
> Now list the process entity after one hour. The process coordinator status in Oozie is "succeeded". But Falcon shows the status as submitted.
>
>
> Diffs
> -----
>
> common/pom.xml 42facc9
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 4d45cc7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 7e6cd6c
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 78964dd
>
> Diff: https://reviews.apache.org/r/38082/diff/
>
>
> Testing
> -------
>
> This patch was tested end2end for the following scenarios.
>
> 1. When a bundle completed running successfully, status is now shown as "Succeeded" instead of "Submitted"
>
> 2. When the following feed is submitted, a coordinator is created for replication on backupCluster, but no coordinator is created on primaryCluster for retention because the feed validity on primaryCluster is in the past. There was a bug in code which was causing feed status to be shown as "submitted" if any of the clusters have a MISSING bundle. After this fix, the status will be shown as running.
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <feed name="cleansedEmailFeed" description="Cleansed customer emails" xmlns="uri:falcon:feed:0.1">
> <groups>churnAnalysisDataPipeline</groups>
> <frequency>hours(1)</frequency>
> <timezone>UTC</timezone>
> <late-arrival cut-off="hours(4)"/>
> <clusters>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
> <retention limit="hours(90)" action="delete"/>
> <locations>
> <location type="data" path="/user/ambari-qa/falcon/demo/primary/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> </cluster>
> <cluster name="backupCluster" type="target">
> <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
> <retention limit="hours(90)" action="delete"/>
> <locations>
> <location type="data" path="/falcon/demo/bcp/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> </cluster>
> </clusters>
> <locations>
> <location type="data" path="/user/ambari-qa/falcon/demo/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> <ACL owner="ambari-qa" group="users" permission="0x755"/>
> <schema location="/none" provider="/none"/>
> </feed>
> {code}
>
> 3. If a bundle is killed using oozie command line, the status of entity will be now shown as KILLED
>
> 4. Added support for FAILED.
>
> 5. Added methods to avoid calling findLatestBundle(...) repeatedly.
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 38082: Status of scheduled Process entity is shown
as submitted in corner case
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38082/#review97600
-----------------------------------------------------------
Ship it!
Good work Balu. Can you file the follow on JIRA for the issue with validity issue?
- Venkat Ranganathan
On Sept. 2, 2015, 5:29 p.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38082/
> -----------------------------------------------------------
>
> (Updated Sept. 2, 2015, 5:29 p.m.)
>
>
> Review request for Falcon, Ajay Yadava, Sowmya Ramesh, and Venkat Ranganathan.
>
>
> Bugs: FALCON-1371
> https://issues.apache.org/jira/browse/FALCON-1371
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Create a process entity with
> - frequency of 1 hour
> - validity start as "2012-07-01T00:00Z"
> - and validity end as "2012-07-01T01:02Z"
>
> Now list the process entity after one hour. The process coordinator status in Oozie is "succeeded". But Falcon shows the status as submitted.
>
>
> Diffs
> -----
>
> common/pom.xml 42facc9
> common/src/main/java/org/apache/falcon/workflow/engine/AbstractWorkflowEngine.java 4d45cc7
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 7e6cd6c
> prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 78964dd
>
> Diff: https://reviews.apache.org/r/38082/diff/
>
>
> Testing
> -------
>
> This patch was tested end2end for the following scenarios.
>
> 1. When a bundle completed running successfully, status is now shown as "Succeeded" instead of "Submitted"
>
> 2. When the following feed is submitted, a coordinator is created for replication on backupCluster, but no coordinator is created on primaryCluster for retention because the feed validity on primaryCluster is in the past. There was a bug in code which was causing feed status to be shown as "submitted" if any of the clusters have a MISSING bundle. After this fix, the status will be shown as running.
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <feed name="cleansedEmailFeed" description="Cleansed customer emails" xmlns="uri:falcon:feed:0.1">
> <groups>churnAnalysisDataPipeline</groups>
> <frequency>hours(1)</frequency>
> <timezone>UTC</timezone>
> <late-arrival cut-off="hours(4)"/>
> <clusters>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
> <retention limit="hours(90)" action="delete"/>
> <locations>
> <location type="data" path="/user/ambari-qa/falcon/demo/primary/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> </cluster>
> <cluster name="backupCluster" type="target">
> <validity start="2015-07-22T01:00Z" end="2015-07-22T03:00Z"/>
> <retention limit="hours(90)" action="delete"/>
> <locations>
> <location type="data" path="/falcon/demo/bcp/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> </cluster>
> </clusters>
> <locations>
> <location type="data" path="/user/ambari-qa/falcon/demo/processed/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
> <location type="stats" path="/"/>
> <location type="meta" path="/"/>
> </locations>
> <ACL owner="ambari-qa" group="users" permission="0x755"/>
> <schema location="/none" provider="/none"/>
> </feed>
> {code}
>
> 3. If a bundle is killed using oozie command line, the status of entity will be now shown as KILLED
>
> 4. Added support for FAILED.
>
> 5. Added methods to avoid calling findLatestBundle(...) repeatedly.
>
>
> Thanks,
>
> Balu Vellanki
>
>