You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Balu Vellanki <bv...@hortonworks.com> on 2015/12/03 05:38:39 UTC
Review Request 40892: FALCON-1644 Retention : Some feed instances are
never deleted by retention jobs.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/
-----------------------------------------------------------
Review request for Falcon and Venkat Ranganathan.
Bugs: Falcon-1644
https://issues.apache.org/jira/browse/Falcon-1644
Repository: falcon-git
Description
-------
Here is a sample feed xml.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="rawEmailFeed" description="Raw customer email feed" xmlns="uri:falcon:feed:0.1">
<tags>externalSystem=USWestEmailServers</tags>
<groups>churnAnalysisDataPipeline</groups>
<frequency>hours(1)</frequency>
<timezone>UTC</timezone>
<late-arrival cut-off="hours(1)"/>
<clusters>
<cluster name="primaryCluster" type="source">
<validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
<retention limit="hours(10)" action="delete"/>
</cluster>
</clusters>
<locations>
<location type="data" path="/user/ambari-qa/falcon/demo/primary/input/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
<location type="stats" path="/"/>
<location type="meta" path="/"/>
</locations>
<ACL owner="ambari-qa" group="users" permission="0x755"/>
<schema location="/none" provider="/none"/>
</feed>
In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
Diffs
-----
common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
Diff: https://reviews.apache.org/r/40892/diff/
Testing
-------
End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
Thanks,
Balu Vellanki
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Ying Zheng <yz...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review108926
-----------------------------------------------------------
Ship it!
Ship It!
- Ying Zheng
On Dec. 3, 2015, 4:39 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2015, 4:39 a.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Balu Vellanki <bv...@hortonworks.com>.
> On Dec. 4, 2015, 11:45 p.m., Ajay Yadava wrote:
> > common/src/main/java/org/apache/falcon/entity/FeedHelper.java, line 1015
> > <https://reviews.apache.org/r/40892/diff/1/?file=1152598#file1152598line1015>
> >
> > There is a FeedHelper method "getRetentionFrequency". You need to use that method.
>
> Balu Vellanki wrote:
> Yes, I will use this method in FeedHelper.
@Ajay : The method "getRetentionFrequency" returns the frequency at which retention coord should run. getRetentionLimitInSeconds method returns the actual time limit upto which files should be retained. These two are different and hence I will have to use getRetentionLimitInSeconds.
- Balu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review109054
-----------------------------------------------------------
On Dec. 3, 2015, 4:39 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2015, 4:39 a.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Balu Vellanki <bv...@hortonworks.com>.
> On Dec. 4, 2015, 11:45 p.m., Ajay Yadava wrote:
> > common/src/main/java/org/apache/falcon/entity/FeedHelper.java, line 1015
> > <https://reviews.apache.org/r/40892/diff/1/?file=1152598#file1152598line1015>
> >
> > There is a FeedHelper method "getRetentionFrequency". You need to use that method.
Yes, I will use this method in FeedHelper.
> On Dec. 4, 2015, 11:45 p.m., Ajay Yadava wrote:
> > oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java, line 681
> > <https://reviews.apache.org/r/40892/diff/1/?file=1152600#file1152600line681>
> >
> > Is it required, now that it is getting overriden just after one line?
agreed, will remove.
> On Dec. 4, 2015, 11:45 p.m., Ajay Yadava wrote:
> > common/src/main/java/org/apache/falcon/entity/FeedHelper.java, line 1023
> > <https://reviews.apache.org/r/40892/diff/1/?file=1152598#file1152598line1023>
> >
> > This is an unreachable code.
IDE throws a warning here, saying that feedCluster can be null. But, if I use "getRetentionFrequency" method in FeedHelper, this code can be removed all together.
- Balu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review109054
-----------------------------------------------------------
On Dec. 3, 2015, 4:39 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2015, 4:39 a.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Ajay Yadava <aj...@gmail.com>.
> On Dec. 4, 2015, 11:45 p.m., Ajay Yadava wrote:
> > common/src/main/java/org/apache/falcon/entity/FeedHelper.java, line 1015
> > <https://reviews.apache.org/r/40892/diff/1/?file=1152598#file1152598line1015>
> >
> > There is a FeedHelper method "getRetentionFrequency". You need to use that method.
>
> Balu Vellanki wrote:
> Yes, I will use this method in FeedHelper.
>
> Balu Vellanki wrote:
> @Ajay : The method "getRetentionFrequency" returns the frequency at which retention coord should run. getRetentionLimitInSeconds method returns the actual time limit upto which files should be retained. These two are different and hence I will have to use getRetentionLimitInSeconds.
The correct position for that comment was the line below it, but I couldn't comment there as there was already another comment on that line. Intention was to highlight that in getRetentionLimitInSeconds you need to use FeedHelper's method, as getting retention frequency for a cluster is not as straight forward as ```feedCluster.getLifecycle().getRetentionStage().getFrequency();```
- Ajay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review109054
-----------------------------------------------------------
On Dec. 3, 2015, 4:39 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2015, 4:39 a.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review109054
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/entity/FeedHelper.java (line 1015)
<https://reviews.apache.org/r/40892/#comment168454>
There is a FeedHelper method "getRetentionFrequency". You need to use that method.
common/src/main/java/org/apache/falcon/entity/FeedHelper.java (line 1023)
<https://reviews.apache.org/r/40892/#comment168455>
This is an unreachable code.
oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java (line 681)
<https://reviews.apache.org/r/40892/#comment168456>
Is it required, now that it is getting overriden just after one line?
- Ajay Yadava
On Dec. 3, 2015, 4:39 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2015, 4:39 a.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review108780
-----------------------------------------------------------
Ship it!
Thanks for fixing it Balu
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java (line 64)
<https://reviews.apache.org/r/40892/#comment168250>
I think this is a good change but it is a different behavior. Initially thought it would be good to to provide a runtime property to say whether we want to use this behavior (which we can make as default or the old behavior.
But I think this is more of a bug fix and should be OK without any runtime property to switch to old behavior. May be others have different opinion.
- Venkat Ranganathan
On Dec. 2, 2015, 8:39 p.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2015, 8:39 p.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review110265
-----------------------------------------------------------
Changes look good. However you need to make the same change for end date in coordinator built by lifecycle as well.
- Ajay Yadava
On Dec. 11, 2015, 9:30 p.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 11, 2015, 9:30 p.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java 29daff3
> common/src/main/resources/runtime.properties f499dd9
> docs/src/site/twiki/FalconDocumentation.twiki f384a42
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
> src/conf/runtime.properties 4bc1bc0
> webapp/src/test/resources/runtime.properties 1da0ca7
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review110375
-----------------------------------------------------------
Ship it!
Ship It!
- Ajay Yadava
On Dec. 14, 2015, 7:18 p.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 7:18 p.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java 29daff3
> common/src/main/resources/runtime.properties f499dd9
> docs/src/site/twiki/FalconDocumentation.twiki f384a42
> lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/AgeBasedCoordinatorBuilder.java 47ab2fc
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
> src/conf/runtime.properties 4bc1bc0
> webapp/src/test/resources/runtime.properties 1da0ca7
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Balu Vellanki <bv...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 7:18 p.m.)
Review request for Falcon and Venkat Ranganathan.
Changes
-------
Thanks for pointing it out Ajay. I updated the patch and tested it end2end.
Bugs: Falcon-1644
https://issues.apache.org/jira/browse/Falcon-1644
Repository: falcon-git
Description
-------
Here is a sample feed validity.
<cluster name="primaryCluster" type="source">
<validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
<retention limit="hours(10)" action="delete"/>
</cluster>
In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
Diffs (updated)
-----
common/src/main/java/org/apache/falcon/entity/FeedHelper.java 29daff3
common/src/main/resources/runtime.properties f499dd9
docs/src/site/twiki/FalconDocumentation.twiki f384a42
lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/AgeBasedCoordinatorBuilder.java 47ab2fc
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
src/conf/runtime.properties 4bc1bc0
webapp/src/test/resources/runtime.properties 1da0ca7
Diff: https://reviews.apache.org/r/40892/diff/
Testing
-------
End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
Thanks,
Balu Vellanki
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Balu Vellanki <bv...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/
-----------------------------------------------------------
(Updated Dec. 11, 2015, 9:30 p.m.)
Review request for Falcon and Venkat Ranganathan.
Changes
-------
Use "retention.policy.agebaseddelete.limit" property. Incorporated Ajay's feedback.
Bugs: Falcon-1644
https://issues.apache.org/jira/browse/Falcon-1644
Repository: falcon-git
Description
-------
Here is a sample feed validity.
<cluster name="primaryCluster" type="source">
<validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
<retention limit="hours(10)" action="delete"/>
</cluster>
In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
Diffs (updated)
-----
common/src/main/java/org/apache/falcon/entity/FeedHelper.java 29daff3
common/src/main/resources/runtime.properties f499dd9
docs/src/site/twiki/FalconDocumentation.twiki f384a42
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
src/conf/runtime.properties 4bc1bc0
webapp/src/test/resources/runtime.properties 1da0ca7
Diff: https://reviews.apache.org/r/40892/diff/
Testing
-------
End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
Thanks,
Balu Vellanki
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Balu Vellanki <bv...@hortonworks.com>.
> On Dec. 10, 2015, 11:36 a.m., Ajay Yadava wrote:
> > common/src/main/java/org/apache/falcon/entity/FeedHelper.java, line 1015
> > <https://reviews.apache.org/r/40892/diff/2/?file=1156251#file1156251line1015>
> >
> > I just noticed that you are using frequency for lifecycle and limit for old retention. I think limit is correct for both. For lifecycle this is available in the property: "retention.policy.agebaseddelete.limit"
Good catch. I missed it before, I will fix it. thank you.
- Balu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review109743
-----------------------------------------------------------
On Dec. 8, 2015, 7:20 p.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 8, 2015, 7:20 p.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> common/src/main/resources/runtime.properties f499dd9
> docs/src/site/twiki/FalconDocumentation.twiki f384a42
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
> src/conf/runtime.properties 4bc1bc0
> webapp/src/test/resources/runtime.properties 1da0ca7
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review109743
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/entity/FeedHelper.java (line 1015)
<https://reviews.apache.org/r/40892/#comment169367>
I just noticed that you are using frequency for lifecycle and limit for old retention. I think limit is correct for both. For lifecycle this is available in the property: "retention.policy.agebaseddelete.limit"
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java (line 68)
<https://reviews.apache.org/r/40892/#comment169368>
We need same change for lifecycle retention stage also.
- Ajay Yadava
On Dec. 8, 2015, 7:20 p.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 8, 2015, 7:20 p.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> common/src/main/resources/runtime.properties f499dd9
> docs/src/site/twiki/FalconDocumentation.twiki f384a42
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
> src/conf/runtime.properties 4bc1bc0
> webapp/src/test/resources/runtime.properties 1da0ca7
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Balu Vellanki <bv...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/
-----------------------------------------------------------
(Updated Dec. 8, 2015, 7:20 p.m.)
Review request for Falcon and Venkat Ranganathan.
Changes
-------
Attaching a new patch that incorporates feedback provided by Falcon team. This patch has
- Incorporated feedback made in review board.
- Introduced a new runtime property "falcon.retention.keep.instances.beyond.validity", set to true by default. This retains existing behavior. Users can set this property to false, and ensure deletion of all instances.
- Updated retention documentation.
Bugs: Falcon-1644
https://issues.apache.org/jira/browse/Falcon-1644
Repository: falcon-git
Description
-------
Here is a sample feed validity.
<cluster name="primaryCluster" type="source">
<validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
<retention limit="hours(10)" action="delete"/>
</cluster>
In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
Diffs (updated)
-----
common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
common/src/main/resources/runtime.properties f499dd9
docs/src/site/twiki/FalconDocumentation.twiki f384a42
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
src/conf/runtime.properties 4bc1bc0
webapp/src/test/resources/runtime.properties 1da0ca7
Diff: https://reviews.apache.org/r/40892/diff/
Testing
-------
End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
Thanks,
Balu Vellanki
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Balu Vellanki <bv...@hortonworks.com>.
> On Dec. 4, 2015, 5:08 p.m., sandeep samudrala wrote:
> > common/src/main/java/org/apache/falcon/entity/FeedHelper.java, line 1016
> > <https://reviews.apache.org/r/40892/diff/1/?file=1152598#file1152598line1016>
> >
> > Shouldn't lifecycle's retention be given preference in case of both retentions being mentioned in the feed definition?
My understanding is that the retention specified under feedCluster gets higher priority than the lifecycle's retention. I might be wrong, let me confirm this.
> On Dec. 4, 2015, 5:08 p.m., sandeep samudrala wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java, line 65
> > <https://reviews.apache.org/r/40892/diff/1/?file=1152599#file1152599line65>
> >
> > Can you use calendar rather than DateUtils?
Agreed, I will make this change.
- Balu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review108995
-----------------------------------------------------------
On Dec. 3, 2015, 4:39 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2015, 4:39 a.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Ajay Yadava <aj...@gmail.com>.
> On Dec. 4, 2015, 5:08 p.m., sandeep samudrala wrote:
> > common/src/main/java/org/apache/falcon/entity/FeedHelper.java, line 1016
> > <https://reviews.apache.org/r/40892/diff/1/?file=1152598#file1152598line1016>
> >
> > Shouldn't lifecycle's retention be given preference in case of both retentions being mentioned in the feed definition?
>
> Balu Vellanki wrote:
> My understanding is that the retention specified under feedCluster gets higher priority than the lifecycle's retention. I might be wrong, let me confirm this.
Retention specified under lifecycle is given higher priority.
- Ajay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review108995
-----------------------------------------------------------
On Dec. 3, 2015, 4:39 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2015, 4:39 a.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Balu Vellanki <bv...@hortonworks.com>.
> On Dec. 4, 2015, 5:08 p.m., sandeep samudrala wrote:
> > oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java, line 65
> > <https://reviews.apache.org/r/40892/diff/1/?file=1152599#file1152599line65>
> >
> > Can you use calendar rather than DateUtils?
>
> Balu Vellanki wrote:
> Agreed, I will make this change.
DateUtils is cleaner to use when working with java.util.Date objects, it is better to use Calendar when working with calendar instances. Since we are using Date class, dont you think it is better to use DateUtils?
- Balu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review108995
-----------------------------------------------------------
On Dec. 3, 2015, 4:39 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2015, 4:39 a.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by sandeep samudrala <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review108995
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/entity/FeedHelper.java (line 1016)
<https://reviews.apache.org/r/40892/#comment168414>
Shouldn't lifecycle's retention be given preference in case of both retentions being mentioned in the feed definition?
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java (line 65)
<https://reviews.apache.org/r/40892/#comment168415>
Can you use calendar rather than DateUtils?
- sandeep samudrala
On Dec. 3, 2015, 4:39 a.m., Balu Vellanki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2015, 4:39 a.m.)
>
>
> Review request for Falcon and Venkat Ranganathan.
>
>
> Bugs: Falcon-1644
> https://issues.apache.org/jira/browse/Falcon-1644
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Here is a sample feed validity.
>
> <cluster name="primaryCluster" type="source">
> <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
> <retention limit="hours(10)" action="delete"/>
> </cluster>
>
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
> oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
> oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
>
> Diff: https://reviews.apache.org/r/40892/diff/
>
>
> Testing
> -------
>
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
>
>
> Thanks,
>
> Balu Vellanki
>
>
Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
are never deleted by retention jobs.
Posted by Balu Vellanki <bv...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/
-----------------------------------------------------------
(Updated Dec. 3, 2015, 4:39 a.m.)
Review request for Falcon and Venkat Ranganathan.
Bugs: Falcon-1644
https://issues.apache.org/jira/browse/Falcon-1644
Repository: falcon-git
Description (updated)
-------
Here is a sample feed validity.
<cluster name="primaryCluster" type="source">
<validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
<retention limit="hours(10)" action="delete"/>
</cluster>
In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
Diffs
-----
common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d
oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3
oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a
Diff: https://reviews.apache.org/r/40892/diff/
Testing
-------
End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
Thanks,
Balu Vellanki