You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Eric Czech <ec...@gmail.com> on 2015/04/01 16:28:10 UTC
Forced Execution of Late Input Feed Instances
Apologies if this isn't the right place for a question like this, but I
thought it was worth asking before diving into the source code any more
(and I'm not sure where else to go).
Is there a way to force a process to run if one or more of its required
input feed instances aren't yet available? By this I don't mean that the
input feed instances arrive late, I mean that they're not present and they
never will be (e.g. for sparse events recorded on a fairly small time scale
like seconds or minutes).
I'd like to tie a process that runs every 6 hours to an input feed with an
hourly frequency but I'm not sure how to gracefully manage cases where one
of the input hours is missing in an expected way, without causing the other
5 hours worth of data to be ignored.
Is there a good place in the source code to look for where that behavior
would be managed (to answer the question myself perhaps)?
Thanks and sorry again for posing a user question to a dev list!
Re: Forced Execution of Late Input Feed Instances
Posted by Eric Czech <ec...@gmail.com>.
Ah ok, good to know! Not to hard to work around that for now, and I
appreciate the insight.
On Thu, Apr 2, 2015 at 5:34 AM, Srikanth Sundarrajan <sr...@hotmail.com>
wrote:
> This is something that we intend to build, but we dont have it yet in the
> system.
>
> Regards
> Srikanth Sundarrajan
>
> ----------------------------------------
> > From: eczech52@gmail.com
> > Date: Wed, 1 Apr 2015 12:23:19 -0400
> > Subject: Re: Forced Execution of Late Input Feed Instances
> > To: dev@falcon.apache.org
> >
> > Thanks Srikanth,
> >
> > That helps for sure, though I'll probably have to delay things
> > unnecessarily to do that.
> >
> > Would a future gating primitive in the works cover a case like this?
> >
> > - We want a process instance to run right away if all data is present
> > - We want the process to wait up to X minutes for late data before
> running
> > anyways
> >
> > I hope I'm not missing anything but it seems like with optional inputs
> that
> > I'll just have to make "right away" be some time further in the future
> > (i.e. the time I'd like the process to run plus the X minute delay).
> >
> > Think there's any hope to pull that off with existing functionality?
> >
> > Thanks again!
> >
> >
> >
> > On Wed, Apr 1, 2015 at 11:54 AM, Srikanth Sundarrajan <
> sriksun@hotmail.com>
> > wrote:
> >
> >> This is a case where one or more inputs is optional. This can be managed
> >> by configured the input as optional. The job in this case would run
> every 6
> >> hours with whatever input is available without actually waiting for any
> >> inputs. We are currently discussing about more gating primitives. Please
> >> see if the existing capability suffices. See "Optional Inputs" sections
> in
> >> http://falcon.apache.org/EntitySpecification.html for additional
> details.
> >>
> >> @dev is the right mailing list. We dont have the @user list created.
> This
> >> is to allow all the dev to help with user queries as well.
> >> Regards
> >> Srikanth Sundarrajan
> >>
> >> ----------------------------------------
> >>> From: eczech52@gmail.com
> >>> Date: Wed, 1 Apr 2015 10:28:10 -0400
> >>> Subject: Forced Execution of Late Input Feed Instances
> >>> To: dev@falcon.apache.org
> >>>
> >>> Apologies if this isn't the right place for a question like this, but I
> >>> thought it was worth asking before diving into the source code any more
> >>> (and I'm not sure where else to go).
> >>>
> >>> Is there a way to force a process to run if one or more of its required
> >>> input feed instances aren't yet available? By this I don't mean that
> the
> >>> input feed instances arrive late, I mean that they're not present and
> >> they
> >>> never will be (e.g. for sparse events recorded on a fairly small time
> >> scale
> >>> like seconds or minutes).
> >>>
> >>> I'd like to tie a process that runs every 6 hours to an input feed with
> >> an
> >>> hourly frequency but I'm not sure how to gracefully manage cases where
> >> one
> >>> of the input hours is missing in an expected way, without causing the
> >> other
> >>> 5 hours worth of data to be ignored.
> >>>
> >>> Is there a good place in the source code to look for where that
> behavior
> >>> would be managed (to answer the question myself perhaps)?
> >>>
> >>> Thanks and sorry again for posing a user question to a dev list!
> >>
> >>
>
>
RE: Forced Execution of Late Input Feed Instances
Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
This is something that we intend to build, but we dont have it yet in the system.
Regards
Srikanth Sundarrajan
----------------------------------------
> From: eczech52@gmail.com
> Date: Wed, 1 Apr 2015 12:23:19 -0400
> Subject: Re: Forced Execution of Late Input Feed Instances
> To: dev@falcon.apache.org
>
> Thanks Srikanth,
>
> That helps for sure, though I'll probably have to delay things
> unnecessarily to do that.
>
> Would a future gating primitive in the works cover a case like this?
>
> - We want a process instance to run right away if all data is present
> - We want the process to wait up to X minutes for late data before running
> anyways
>
> I hope I'm not missing anything but it seems like with optional inputs that
> I'll just have to make "right away" be some time further in the future
> (i.e. the time I'd like the process to run plus the X minute delay).
>
> Think there's any hope to pull that off with existing functionality?
>
> Thanks again!
>
>
>
> On Wed, Apr 1, 2015 at 11:54 AM, Srikanth Sundarrajan <sr...@hotmail.com>
> wrote:
>
>> This is a case where one or more inputs is optional. This can be managed
>> by configured the input as optional. The job in this case would run every 6
>> hours with whatever input is available without actually waiting for any
>> inputs. We are currently discussing about more gating primitives. Please
>> see if the existing capability suffices. See "Optional Inputs" sections in
>> http://falcon.apache.org/EntitySpecification.html for additional details.
>>
>> @dev is the right mailing list. We dont have the @user list created. This
>> is to allow all the dev to help with user queries as well.
>> Regards
>> Srikanth Sundarrajan
>>
>> ----------------------------------------
>>> From: eczech52@gmail.com
>>> Date: Wed, 1 Apr 2015 10:28:10 -0400
>>> Subject: Forced Execution of Late Input Feed Instances
>>> To: dev@falcon.apache.org
>>>
>>> Apologies if this isn't the right place for a question like this, but I
>>> thought it was worth asking before diving into the source code any more
>>> (and I'm not sure where else to go).
>>>
>>> Is there a way to force a process to run if one or more of its required
>>> input feed instances aren't yet available? By this I don't mean that the
>>> input feed instances arrive late, I mean that they're not present and
>> they
>>> never will be (e.g. for sparse events recorded on a fairly small time
>> scale
>>> like seconds or minutes).
>>>
>>> I'd like to tie a process that runs every 6 hours to an input feed with
>> an
>>> hourly frequency but I'm not sure how to gracefully manage cases where
>> one
>>> of the input hours is missing in an expected way, without causing the
>> other
>>> 5 hours worth of data to be ignored.
>>>
>>> Is there a good place in the source code to look for where that behavior
>>> would be managed (to answer the question myself perhaps)?
>>>
>>> Thanks and sorry again for posing a user question to a dev list!
>>
>>
Re: Forced Execution of Late Input Feed Instances
Posted by Eric Czech <ec...@gmail.com>.
Thanks Srikanth,
That helps for sure, though I'll probably have to delay things
unnecessarily to do that.
Would a future gating primitive in the works cover a case like this?
- We want a process instance to run right away if all data is present
- We want the process to wait up to X minutes for late data before running
anyways
I hope I'm not missing anything but it seems like with optional inputs that
I'll just have to make "right away" be some time further in the future
(i.e. the time I'd like the process to run plus the X minute delay).
Think there's any hope to pull that off with existing functionality?
Thanks again!
On Wed, Apr 1, 2015 at 11:54 AM, Srikanth Sundarrajan <sr...@hotmail.com>
wrote:
> This is a case where one or more inputs is optional. This can be managed
> by configured the input as optional. The job in this case would run every 6
> hours with whatever input is available without actually waiting for any
> inputs. We are currently discussing about more gating primitives. Please
> see if the existing capability suffices. See "Optional Inputs" sections in
> http://falcon.apache.org/EntitySpecification.html for additional details.
>
> @dev is the right mailing list. We dont have the @user list created. This
> is to allow all the dev to help with user queries as well.
> Regards
> Srikanth Sundarrajan
>
> ----------------------------------------
> > From: eczech52@gmail.com
> > Date: Wed, 1 Apr 2015 10:28:10 -0400
> > Subject: Forced Execution of Late Input Feed Instances
> > To: dev@falcon.apache.org
> >
> > Apologies if this isn't the right place for a question like this, but I
> > thought it was worth asking before diving into the source code any more
> > (and I'm not sure where else to go).
> >
> > Is there a way to force a process to run if one or more of its required
> > input feed instances aren't yet available? By this I don't mean that the
> > input feed instances arrive late, I mean that they're not present and
> they
> > never will be (e.g. for sparse events recorded on a fairly small time
> scale
> > like seconds or minutes).
> >
> > I'd like to tie a process that runs every 6 hours to an input feed with
> an
> > hourly frequency but I'm not sure how to gracefully manage cases where
> one
> > of the input hours is missing in an expected way, without causing the
> other
> > 5 hours worth of data to be ignored.
> >
> > Is there a good place in the source code to look for where that behavior
> > would be managed (to answer the question myself perhaps)?
> >
> > Thanks and sorry again for posing a user question to a dev list!
>
>
RE: Forced Execution of Late Input Feed Instances
Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
This is a case where one or more inputs is optional. This can be managed by configured the input as optional. The job in this case would run every 6 hours with whatever input is available without actually waiting for any inputs. We are currently discussing about more gating primitives. Please see if the existing capability suffices. See "Optional Inputs" sections in http://falcon.apache.org/EntitySpecification.html for additional details.
@dev is the right mailing list. We dont have the @user list created. This is to allow all the dev to help with user queries as well.
Regards
Srikanth Sundarrajan
----------------------------------------
> From: eczech52@gmail.com
> Date: Wed, 1 Apr 2015 10:28:10 -0400
> Subject: Forced Execution of Late Input Feed Instances
> To: dev@falcon.apache.org
>
> Apologies if this isn't the right place for a question like this, but I
> thought it was worth asking before diving into the source code any more
> (and I'm not sure where else to go).
>
> Is there a way to force a process to run if one or more of its required
> input feed instances aren't yet available? By this I don't mean that the
> input feed instances arrive late, I mean that they're not present and they
> never will be (e.g. for sparse events recorded on a fairly small time scale
> like seconds or minutes).
>
> I'd like to tie a process that runs every 6 hours to an input feed with an
> hourly frequency but I'm not sure how to gracefully manage cases where one
> of the input hours is missing in an expected way, without causing the other
> 5 hours worth of data to be ignored.
>
> Is there a good place in the source code to look for where that behavior
> would be managed (to answer the question myself perhaps)?
>
> Thanks and sorry again for posing a user question to a dev list!