You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by "Balu Vellanki (JIRA)" <ji...@apache.org> on 2015/11/09 21:13:11 UTC
[jira] [Comment Edited] (FALCON-1372) Retention does not work in
corner cases
[ https://issues.apache.org/jira/browse/FALCON-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991908#comment-14991908 ]
Balu Vellanki edited comment on FALCON-1372 at 11/9/15 8:12 PM:
----------------------------------------------------------------
This seems to be an invalid bug. My retention coord is running on Nov 4th, 10 AM in San Francisco. Feed frequency is monthly, Timezone is UTC and retention time is 3 days. So, anything older than 2015 Nov 1st 10 AM San Francisco time will be evicted. The tester is expecting 2015/10 to be deleted and 2015/11 to not be evicted. However 2015/11 is in UTC and it translates to 2015-10-31T16:00 pacific time, so it will be evicted.
A few more cases below.
- If feed frequency is daily and timezone is UTC, 2015/11/01 will translate to 2015-10-31T16:00 pacific time and will be evicted.
- If feed frequency is hourly and timezone is UTC, 2015/11/01/10 will translate to 2015-11-01T02:00 pacific time and will be evicted.
- If feed frequency is hourly and timezone is UTC, 2015/11/01/19 will translate to 2015-11-01T11:00 pacific time and will NOT be evicted.
was (Author: bvellanki):
This seems to be an invalid bug. My retention coord is running on Nov 4th, 10 AM in San Francisco. Feed frequency is monthly, Timezone is UTC and retention time is 4days. So, anything older than 2015 Nov 1st 10 AM San Francisco time will be evicted. The tester is expecting 2015/10 to be deleted and 2015/11 to not be evicted. However 2015/11 is in UTC and it translates to 2015-10-31T16:00 pacific time, so it will be evicted.
A few more cases below.
- If feed frequency is daily and timezone is UTC, 2015/11/01 will translate to 2015-10-31T16:00 pacific time and will be evicted.
- If feed frequency is hourly and timezone is UTC, 2015/11/01/10 will translate to 2015-11-01T02:00 pacific time and will be evicted.
- If feed frequency is hourly and timezone is UTC, 2015/11/01/19 will translate to 2015-11-01T11:00 pacific time and will NOT be evicted.
> Retention does not work in corner cases
> ----------------------------------------
>
> Key: FALCON-1372
> URL: https://issues.apache.org/jira/browse/FALCON-1372
> Project: Falcon
> Issue Type: Bug
> Affects Versions: 0.6.1
> Reporter: Balu Vellanki
> Assignee: Balu Vellanki
>
> Originally reported by [~rostafiychuk].
> RetentionTest#testRetention (24, DAYS, true, MONTHLY, true)
> Uploaded feed:
> {code:xml}
> <?xml version="1.0" encoding="UTF-8"?><feed xmlns="uri:falcon:feed:0.1" name="A852708b0-3057547e" description="clicks log">
> <frequency>hours(1)</frequency>
> <timezone>UTC</timezone>
> <late-arrival cut-off="hours(6)"/>
> <clusters>
> <cluster name="A852708b0-cd30589f" type="source">
> <validity start="2012-01-30T00:00Z" end="2099-03-31T23:59Z"/>
> <retention limit="days(24)" action="delete"/>
> </cluster>
> </clusters>
> <locations>
> <location type="data" path="/tmp/falcon-regression/RetentionTest/testFolders/${YEAR}/${MONTH}"/>
> <location type="stats" path="/projects/falcon/clicksStats"/>
> <location type="meta" path="/projects/falcon/clicksMetaData"/>
> </locations>
> <ACL owner="hrt_qa" group="users" permission="*"/>
> <schema location="/schema/clicks" provider="protobuf"/>
> <properties/>
> </feed>
> {code}
> Created dirs:
> {code}
> dataDates = [2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01, 2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11, 2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09, 2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05, 2015/06, 2015/07, 2015/08, 2015/09, 2015/10, 2015/11, 2015/12, 2016/01, 2016/02, 2016/03, 2016/04, 2016/05, 2016/06, 2016/07, 2016/08, 2016/09, 2016/10, 2016/11, 2016/12, 2017/01, 2017/02, 2017/03, 2017/04, 2017/05, 2017/06, 2017/07, 2017/08, 2017/09, 2017/10, 2017/11, 2017/12, 2018/01, 2018/02, 2018/03, 2018/04, 2018/05, 2018/06, 2018/07] {code}
> expected for deletion:
> {code} [2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01, 2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11, 2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09, 2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05]
> {code}
> deleted:
> {code:javascript}[2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01, 2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11, 2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09, 2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05, 2015/06]
> {code}
> 24 days ago was 31th of May, but *2015/06* was deleted too.
> Another case:
> RetentionTest#testRetention (10080, HOURS, false, MONTHLY, true)
> {code:xml}
> <retention limit="hours(10080)" action="delete"/>
> {code}
> 10080 hours ago was 30th of April, but dir *"2014/05"* was deleted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)