You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Venkat R <ve...@yahoo.com> on 2014/05/25 21:03:03 UTC

Representing hourly and daily datasets

Hi All,

is there a way to logically associate the hourly and daily versions of the feeds? 
Or do they have to be two different feeds in falcon feed definition?

Wondering is it possible to create a flow definition so that it consumes daily version of the feed instances by default, but hourly versions for those instances missing daily? 

Thanks
Venkat   

Re: Representing hourly and daily datasets

Posted by Venkat R <ve...@yahoo.com>.
Ok. if that being the case, is it possible to refer daily and hourly dynamically in the workflow definition?
The use case:
A job is reading 30 days of data and the daily data is available for all except the last day -- for which only hourly is available. In this case, either the job waits for the last day daily data to become available or consume all the 24 hours and let the job run. 

Thanks
Venkat



On Tuesday, May 27, 2014 4:55 AM, Srikanth Sundarrajan <sr...@hotmail.com> wrote:
 


Currently hourly and daily are independent feeds. It might make sense to model hourly as a partition of daily but that would require reasonable work.


Sent from my iPhone

On 26-May-2014, at 12:33 am, Venkat R <ve...@yahoo.com> wrote:


Hi All,
>
>
>is there a way to logically associate the hourly and daily versions of the feeds? 
>Or do they have to be two different feeds in falcon feed definition?
>
>
>Wondering is it possible to create a flow definition so that it consumes daily version of the feed instances by default, but hourly versions for those instances missing daily? 
>
>
>Thanks
>Venkat   

Re: Representing hourly and daily datasets

Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
Currently hourly and daily are independent feeds. It might make sense to model hourly as a partition of daily but that would require reasonable work.


Sent from my iPhone

> On 26-May-2014, at 12:33 am, Venkat R <ve...@yahoo.com> wrote:
> 
> Hi All,
> 
> is there a way to logically associate the hourly and daily versions of the feeds? 
> Or do they have to be two different feeds in falcon feed definition?
> 
> Wondering is it possible to create a flow definition so that it consumes daily version of the feed instances by default, but hourly versions for those instances missing daily? 
> 
> Thanks
> Venkat