Replication Problem

[prashant madaan <pr...@gmail.com>]

Hi team,
Hope you are doing good.

I was doing some testing on falcon but stuck up with below issue. looking
for some quick response on this.

The issue I am facing are :

* Replication Issues :*

I have a process that runs an oozie workflow and passes the time parameters
of 7 hours before the current time of falcon job . Below is the value i
pass :
<property name="feed_date" value="${formatTime(dateOffset(instanceTime(),
-7, 'HOUR'),'yyyy-MM-dd')}" />

<property name="feed_hour" value="${formatTime(dateOffset(instanceTime(),
-7, 'HOUR'),'HH')}" />

I have built a feed for replication and mentioned the late arrival cutoff
to be hours(10).

But the issue is that the the data comes late, approximately 7 hours from
the current running time and the feed entity runs on the current time . The
late cut off works for 4 5 iterations of this but then it stops the
replication .

Note: The frequency of the feed entity as well as the process entity  is
hours(1).

Example :

Suppose Current time is 10 AM

The data comes for 3 AM hour.

I schedule this process at 10 am and the data gets picked up and is
processed and the feed replication runs fine .

Similar things happen for the next 3 4 iterations , but after that the feed
replication does not kick off at all (Thus i cannot check the logs) but the
process entity runs fine .


Please let me know how to fix this issue . Also can we make falcon run 7
hours late from the current time . For example

Feed Replication Path contains /xyz/${YEAR}-${MONTH}-${DAY}/${HOUR}/abc

This picks up the current falcon time . But i want it to always run 7 hours
behind the current time .


Any Help is appreciated .

Thanks and Regards

Prashant Madaan

Re: Replication Problem

[Ajay Yadav <aj...@inmobi.com>]

I am not sure if I understood your problem correctly but it looks like the
data comes late and hence the replication doesn't happen. You can try
specifying the "delay" attribute on source cluster in feed definition to
start replication with delay. e.g.

<cluster name="corp1" type="source" delay="hours(7)">
    <validity start="2010-01-01T00:00Z" end="2020-01-01T02:00Z"/>
    <retention limit="days(5)" action="delete"/>
</cluster>
<cluster name="corp2" type="target">
    <validity start="2010-01-01T00:00Z" end="2020-01-01T02:00Z"/>
    <retention limit="days(7)" action="delete"/>
</cluster>



Cheers
Ajay Yadava

On Thu, Nov 26, 2015 at 8:54 PM, prashant madaan <
prashantmadaan123@gmail.com> wrote:

> Hi team,
> Hope you are doing good.
>
> I was doing some testing on falcon but stuck up with below issue. looking
> for some quick response on this.
>
> The issue I am facing are :
>
> * Replication Issues :*
>
> I have a process that runs an oozie workflow and passes the time parameters
> of 7 hours before the current time of falcon job . Below is the value i
> pass :
> <property name="feed_date" value="${formatTime(dateOffset(instanceTime(),
> -7, 'HOUR'),'yyyy-MM-dd')}" />
>
> <property name="feed_hour" value="${formatTime(dateOffset(instanceTime(),
> -7, 'HOUR'),'HH')}" />
>
> I have built a feed for replication and mentioned the late arrival cutoff
> to be hours(10).
>
> But the issue is that the the data comes late, approximately 7 hours from
> the current running time and the feed entity runs on the current time . The
> late cut off works for 4 5 iterations of this but then it stops the
> replication .
>
> Note: The frequency of the feed entity as well as the process entity  is
> hours(1).
>
> Example :
>
> Suppose Current time is 10 AM
>
> The data comes for 3 AM hour.
>
> I schedule this process at 10 am and the data gets picked up and is
> processed and the feed replication runs fine .
>
> Similar things happen for the next 3 4 iterations , but after that the feed
> replication does not kick off at all (Thus i cannot check the logs) but the
> process entity runs fine .
>
>
> Please let me know how to fix this issue . Also can we make falcon run 7
> hours late from the current time . For example
>
> Feed Replication Path contains /xyz/${YEAR}-${MONTH}-${DAY}/${HOUR}/abc
>
> This picks up the current falcon time . But i want it to always run 7 hours
> behind the current time .
>
>
> Any Help is appreciated .
>
> Thanks and Regards
>
> Prashant Madaan
>

-- 
_____________________________________________________________
The information contained in this communication is intended solely for the 
use of the individual or entity to whom it is addressed and others 
authorized to receive it. It may contain confidential or legally privileged 
information. If you are not the intended recipient you are hereby notified 
that any disclosure, copying, distribution or taking any action in reliance 
on the contents of this information is strictly prohibited and may be 
unlawful. If you have received this communication in error, please notify 
us immediately by responding to this email and then delete it from your 
system. The firm is neither liable for the proper and complete transmission 
of the information contained in this communication nor for any delay in its 
receipt.