You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by "Pragya Mittal (JIRA)" <ji...@apache.org> on 2015/10/09 16:33:26 UTC
[jira] [Updated] (FALCON-1526) Update happens twice in case of feed
containing lifecycle
[ https://issues.apache.org/jira/browse/FALCON-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pragya Mittal updated FALCON-1526:
----------------------------------
Description:
If a feed containing lifecycle tag is updated then two bundles are created corresponding to retention(source, target) and replication(target).
Feed definition is :
{code}
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="retention-feed1" description="Input File" xmlns="uri:falcon:feed:0.1">
<partitions>
<partition name="colo"/>
<partition name="eventTime"/>
<partition name="impressionHour"/>
<partition name="pricingModel"/>
</partitions>
<frequency>minutes(5)</frequency>
<timezone>UTC</timezone>
<late-arrival cut-off="minutes(2)"/>
<clusters>
<cluster name="A969761bb-af20fc5f" type="source">
<validity start="2015-10-09T14:00Z" end="2015-10-09T18:00Z"/>
<retention limit="days(1000000)" action="delete"/>
</cluster>
<cluster name="A969761bb-69e50f1d" type="target">
<validity start="2015-10-09T14:00Z" end="2015-10-09T15:00Z"/>
<retention limit="days(1000000)" action="delete"/>
<locations>
<location type="data" path="/tmp/falcon-regression/LifecycleRetentionAndReplicationTest/target/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
</locations>
</cluster>
</clusters>
<locations>
<location type="data" path="/tmp/falcon-regression/LifecycleRetentionAndReplicationTest/source/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
<location type="stats" path="/data/regression/fetlrc/billing/stats"/>
<location type="meta" path="/data/regression/fetlrc/billing/metadata"/>
</locations>
<ACL owner="pragya" group="dataqa" permission="*"/>
<schema location="/databus/streams_local/click_rr/schema/" provider="protobuf"/>
<properties>
<property name="field1" value="value1"/>
<property name="field2" value="value2"/>
</properties>
<lifecycle>
<retention-stage>
<frequency>minutes(10)</frequency>
<queue>default</queue>
<priority>NORMAL</priority>
<properties>
<property name="retention.policy.agebaseddelete.limit" value="minutes(15)"/>
</properties>
</retention-stage>
</lifecycle>
</feed>
{code}
Update response log is :
{code}
dataqa@8RPCG32:/mnt/users/pragya/lifecycle$ fef -name retention-feed1 -update -file feed.xml
falcon/update/ua1/Updated successfully(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000053-151009121634425-oozie-oozi-B. Old coordinator id: 0000054-151009121634425-oozie-oozi-C. New bundle id: 0000064-151009121634425-oozie-oozi-B
(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000039-151009142953874-oozie-oozi-B. Old coordinator id: 0000041-151009142953874-oozie-oozi-C,0000040-151009142953874-oozie-oozi-C. New bundle id: 0000053-151009142953874-oozie-oozi-B
ua2/Updated successfully(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000053-151009142953874-oozie-oozi-B. Old coordinator id: 0000054-151009142953874-oozie-oozi-C,0000055-151009142953874-oozie-oozi-C. New bundle id: 0000059-151009142953874-oozie-oozi-B
(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000064-151009121634425-oozie-oozi-B. Old coordinator id: 0000065-151009121634425-oozie-oozi-C. New coordinator id: 0000069-151009121634425-oozie-oozi-C
{code}
was:
If a feed containing lifecycle tag is updated then two bundles are created corresponding to retention(source, target) and replication(target).
Feed definition is :
{code}
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="retention-feed1" description="Input File" xmlns="uri:falcon:feed:0.1">
<partitions>
<partition name="colo"/>
<partition name="eventTime"/>
<partition name="impressionHour"/>
<partition name="pricingModel"/>
</partitions>
<frequency>minutes(5)</frequency>
<timezone>UTC</timezone>
<late-arrival cut-off="minutes(2)"/>
<clusters>
<cluster name="A969761bb-af20fc5f" type="source">
<validity start="2015-10-09T14:00Z" end="2015-10-09T18:00Z"/>
<retention limit="days(1000000)" action="delete"/>
</cluster>
<cluster name="A969761bb-69e50f1d" type="target">
<validity start="2015-10-09T14:00Z" end="2015-10-09T15:00Z"/>
<retention limit="days(1000000)" action="delete"/>
<locations>
<location type="data" path="/tmp/falcon-regression/LifecycleRetentionAndReplicationTest/target/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
</locations>
</cluster>
</clusters>
<locations>
<location type="data" path="/tmp/falcon-regression/LifecycleRetentionAndReplicationTest/source/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
<location type="stats" path="/data/regression/fetlrc/billing/stats"/>
<location type="meta" path="/data/regression/fetlrc/billing/metadata"/>
</locations>
<ACL owner="pragya" group="dataqa" permission="*"/>
<schema location="/databus/streams_local/click_rr/schema/" provider="protobuf"/>
<properties>
<property name="field1" value="value1"/>
<property name="field2" value="value2"/>
</properties>
<lifecycle>
<retention-stage>
<frequency>minutes(10)</frequency>
<queue>default</queue>
<priority>NORMAL</priority>
<properties>
<property name="retention.policy.agebaseddelete.limit" value="minutes(15)"/>
</properties>
</retention-stage>
</lifecycle>
</feed>
{code}
Updated log is :
{code}
dataqa@8RPCG32:/mnt/users/pragya/lifecycle$ fef -name retention-feed1 -update -file feed.xml
falcon/update/ua1/Updated successfully(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000053-151009121634425-oozie-oozi-B. Old coordinator id: 0000054-151009121634425-oozie-oozi-C. New bundle id: 0000064-151009121634425-oozie-oozi-B
(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000039-151009142953874-oozie-oozi-B. Old coordinator id: 0000041-151009142953874-oozie-oozi-C,0000040-151009142953874-oozie-oozi-C. New bundle id: 0000053-151009142953874-oozie-oozi-B
ua2/Updated successfully(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000053-151009142953874-oozie-oozi-B. Old coordinator id: 0000054-151009142953874-oozie-oozi-C,0000055-151009142953874-oozie-oozi-C. New bundle id: 0000059-151009142953874-oozie-oozi-B
(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000064-151009121634425-oozie-oozi-B. Old coordinator id: 0000065-151009121634425-oozie-oozi-C. New coordinator id: 0000069-151009121634425-oozie-oozi-C
{code}
> Update happens twice in case of feed containing lifecycle
> ---------------------------------------------------------
>
> Key: FALCON-1526
> URL: https://issues.apache.org/jira/browse/FALCON-1526
> Project: Falcon
> Issue Type: Bug
> Components: prism
> Affects Versions: 0.8
> Reporter: Pragya Mittal
>
> If a feed containing lifecycle tag is updated then two bundles are created corresponding to retention(source, target) and replication(target).
> Feed definition is :
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <feed name="retention-feed1" description="Input File" xmlns="uri:falcon:feed:0.1">
> <partitions>
> <partition name="colo"/>
> <partition name="eventTime"/>
> <partition name="impressionHour"/>
> <partition name="pricingModel"/>
> </partitions>
> <frequency>minutes(5)</frequency>
> <timezone>UTC</timezone>
> <late-arrival cut-off="minutes(2)"/>
> <clusters>
> <cluster name="A969761bb-af20fc5f" type="source">
> <validity start="2015-10-09T14:00Z" end="2015-10-09T18:00Z"/>
> <retention limit="days(1000000)" action="delete"/>
> </cluster>
> <cluster name="A969761bb-69e50f1d" type="target">
> <validity start="2015-10-09T14:00Z" end="2015-10-09T15:00Z"/>
> <retention limit="days(1000000)" action="delete"/>
> <locations>
> <location type="data" path="/tmp/falcon-regression/LifecycleRetentionAndReplicationTest/target/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
> </locations>
> </cluster>
> </clusters>
> <locations>
> <location type="data" path="/tmp/falcon-regression/LifecycleRetentionAndReplicationTest/source/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
> <location type="stats" path="/data/regression/fetlrc/billing/stats"/>
> <location type="meta" path="/data/regression/fetlrc/billing/metadata"/>
> </locations>
> <ACL owner="pragya" group="dataqa" permission="*"/>
> <schema location="/databus/streams_local/click_rr/schema/" provider="protobuf"/>
> <properties>
> <property name="field1" value="value1"/>
> <property name="field2" value="value2"/>
> </properties>
> <lifecycle>
> <retention-stage>
> <frequency>minutes(10)</frequency>
> <queue>default</queue>
> <priority>NORMAL</priority>
> <properties>
> <property name="retention.policy.agebaseddelete.limit" value="minutes(15)"/>
> </properties>
> </retention-stage>
> </lifecycle>
> </feed>
> {code}
> Update response log is :
> {code}
> dataqa@8RPCG32:/mnt/users/pragya/lifecycle$ fef -name retention-feed1 -update -file feed.xml
> falcon/update/ua1/Updated successfully(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000053-151009121634425-oozie-oozi-B. Old coordinator id: 0000054-151009121634425-oozie-oozi-C. New bundle id: 0000064-151009121634425-oozie-oozi-B
> (feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000039-151009142953874-oozie-oozi-B. Old coordinator id: 0000041-151009142953874-oozie-oozi-C,0000040-151009142953874-oozie-oozi-C. New bundle id: 0000053-151009142953874-oozie-oozi-B
> ua2/Updated successfully(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000053-151009142953874-oozie-oozi-B. Old coordinator id: 0000054-151009142953874-oozie-oozi-C,0000055-151009142953874-oozie-oozi-C. New bundle id: 0000059-151009142953874-oozie-oozi-B
> (feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000064-151009121634425-oozie-oozi-B. Old coordinator id: 0000065-151009121634425-oozie-oozi-C. New coordinator id: 0000069-151009121634425-oozie-oozi-C
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)