You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by "Pallavi Rao (JIRA)" <ji...@apache.org> on 2016/10/25 11:02:58 UTC
[jira] [Resolved] (FALCON-2163) Feed replication based on partition
does not produce expected output
[ https://issues.apache.org/jira/browse/FALCON-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pallavi Rao resolved FALCON-2163.
---------------------------------
Resolution: Fixed
Assignee: Pallavi Rao
Fix Version/s: trunk
> Feed replication based on partition does not produce expected output
> --------------------------------------------------------------------
>
> Key: FALCON-2163
> URL: https://issues.apache.org/jira/browse/FALCON-2163
> Project: Falcon
> Issue Type: Bug
> Components: feed, prism
> Affects Versions: trunk
> Reporter: Pragya Mittal
> Assignee: Pallavi Rao
> Fix For: trunk
>
>
> Feed :
> {noformat}
> <?xml version="1.0" encoding="UTF-8"?><feed xmlns="uri:falcon:feed:0.1" name="PrismFeedReplicationPartitionExpTest-InputFeed-021eefe6" description="Input File">
> <partitions>
> <partition name="colo"/>
> <partition name="eventTime"/>
> <partition name="impressionHour"/>
> <partition name="pricingModel"/>
> </partitions>
> <frequency>minutes(5)</frequency>
> <late-arrival cut-off="days(100000)"/>
> <clusters>
> <cluster name="PrismFeedReplicationPartitionExpTest-falcon2-stg4-5698a59c" type="source" partition="${cluster.colo}">
> <validity start="2012-10-01T12:00Z" end="2099-10-01T12:10Z"/>
> <retention limit="days(1000000)" action="delete"/>
> <locations>
> <location type="data" path="/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
> </locations>
> </cluster>
> <cluster name="PrismFeedReplicationPartitionExpTest-falcon2-stg4-7a96289d" type="target">
> <validity start="2012-10-01T12:00Z" end="2099-10-01T12:25Z"/>
> <retention limit="days(1000000)" action="delete"/>
> <locations>
> <location type="data" path="/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
> </locations>
> </cluster>
> <cluster name="PrismFeedReplicationPartitionExpTest-falcon2-stg4-d4fbf2d9" type="source" partition="${cluster.colo}">
> <validity start="2012-10-01T12:00Z" end="2099-01-01T00:00Z"/>
> <retention limit="days(1000000)" action="delete"/>
> <locations>
> <location type="data" path="/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/localDC/rc/billing/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
> </locations>
> </cluster>
> </clusters>
> <locations>
> <location type="data" path="/data/regression/fetlrc/billing/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
> <location type="stats" path="/data/regression/fetlrc/billing/stats"/>
> <location type="meta" path="/data/regression/fetlrc/billing/metadata"/>
> </locations>
> <ACL owner="pragya" group="dataqa" permission="*"/>
> <schema location="/databus/streams_local/click_rr/schema/" provider="protobuf"/>
> <properties>
> <property name="field1" value="value1"/>
> <property name="field2" value="value2"/>
> </properties>
> </feed>
> {noformat}
> Cluster 1 data :
> {noformat}
> dataqa@falcon1001:~$ hlr /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/*/*
> 16/09/28 10:53:02 INFO util.Shell: ===Loading custom Shell class JIRA:GRIDOPS-4518 ==
> lsr: DEPRECATED: Please use 'ls -R' instead.
> -rw-r--r-- 3 pragya hdfs 2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/00/ua1/dataFile.xml
> -rw-r--r-- 3 pragya hdfs 2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/00/ua2/dataFile.xml
> -rw-r--r-- 3 pragya hdfs 2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/00/ua3/dataFile.xml
> -rw-r--r-- 3 pragya hdfs 922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/05/ua1/id.pig
> -rw-r--r-- 3 pragya hdfs 922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/05/ua2/id.pig
> -rw-r--r-- 3 pragya hdfs 922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/05/ua3/id.pig
> {noformat}
> Cluster 3 data :
> {noformat}
> dataqa@hydra:~$ hlr /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/*/*
> lsr: DEPRECATED: Please use 'ls -R' instead.
> -rw-r--r-- 3 pragya hdfs 2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/00/ua1/dataFile.xml
> -rw-r--r-- 3 pragya hdfs 2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/00/ua2/dataFile.xml
> -rw-r--r-- 3 pragya hdfs 2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/00/ua3/dataFile.xml
> -rw-r--r-- 3 pragya hdfs 922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/05/ua1/id.pig
> -rw-r--r-- 3 pragya hdfs 922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/05/ua2/id.pig
> -rw-r--r-- 3 pragya hdfs 922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/05/ua3/id.pig
> -rw-r--r-- 3 pragya hdfs 1711 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/10/ua1/cluster-0.1.xml
> -rw-r--r-- 3 pragya hdfs 1711 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/10/ua2/cluster-0.1.xml
> -rw-r--r-- 3 pragya hdfs 1711 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/10/ua3/cluster-0.1.xml
> -rw-r--r-- 3 pragya hdfs 1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/15/ua1/dataFile.properties
> -rw-r--r-- 3 pragya hdfs 1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/15/ua2/dataFile.properties
> -rw-r--r-- 3 pragya hdfs 1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/15/ua3/dataFile.properties
> -rw-r--r-- 3 pragya hdfs 1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/20/ua1/dataFile.properties
> -rw-r--r-- 3 pragya hdfs 1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/20/ua2/dataFile.properties
> -rw-r--r-- 3 pragya hdfs 1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/20/ua3/dataFile.properties
> {noformat}
> Target data :
> {noformat}
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/dataFile.xml
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/id.pig
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/cluster-0.1.xml
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/dataFile.properties
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/dataFile.properties
> {noformat}
> Expected output :
> {noformat}
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/dataFile.xml
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/id.pig
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua3/dataFile.xml
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua3/id.pig
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/cluster-0.1.xml
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/dataFile.properties
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/_SUCCESS
> /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/dataFile.properties
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)