You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Ajay Yadav <aj...@gmail.com> on 2015/09/08 20:27:03 UTC
[DISCUSS] Lifecycle design closure
Hello everyone,
During our previous sync up we discussed feed lifecycle. I have
incorporated the suggestions in the updated design doc (attached). In this
email I will try to answer the remaining unresolved questions.
*Target Audience?*
Falcon users are the people who write falcon processes and feeds and use
Falcon CLI/ Rest API to maintain/operate them. Falcon developers are the
people who work on enhancing falcon, regardless of the their decision to
contribute it back to the Apache Falcon, though it is highly encouraged.
Lifecycle extensions are meant for falcon developers to extend.
Through the lifecycle feature we aim to provide clean extension points for
extending feed management. It will be easier to understand falcon feed
lifecycle with the analogy of oozie-el-extensions.
*Is a restart of falcon required for adding a new lifecycle policy?*
Yes, a restart is required. Lifecycle policies are applicable for all feeds
in the falcon and we want addition of new lifecycle policies to be an
explicit and authorised action. Moreover, addition of new policy for a
lifecycle stage is a rare requirement and shouldn't be a hindrance in it's
adoption.
*Will lifecycle policies run as a separate process?*
No. Execution of lifecycle policies happen outside of falcon. Some parts
like preparing the workflows and validation are done inside falcon at the
time of submission of a feed. Again with the analogy of
oozie-el-extensions, this is an extension of falcon and is different from
user submitted workflows/process definitions and running it in a separate
JVM will be an overkill.
I have attached the latest doc and xsd patch in the JIRA for your
consideration. Would like to hear your thoughts. Once we have closure over
design, I will work towards providing a complete patch for base framework.
Cheers
Ajay Yadava
Re: [DISCUSS] Lifecycle design closure
Posted by Peeyush Bishnoi <bp...@yahoo.co.in>.
Ajay,
Thanks for initiating the discussion.
You have mentioned that Users can create new Lifecycle by extending it, so it is like custom implementation using Lifecycle API. I am of the opinion Falcon should not be restarted for user custom implementation.
It is better to provide the way to the user to plug and use their custom Lifecycle implementation to entities withoutrestarting Falcon server.
If user contribute their implemented Lifecycle to Falcon, after thorough review, it should be made part of existing Falcon lifecycles to increase the adoption.
Thanks,---Peeyush
On Tuesday, 8 September 2015 11:57 PM, Ajay Yadav <aj...@gmail.com> wrote:
Hello everyone,
During our previous sync up we discussed feed lifecycle. I have
incorporated the suggestions in the updated design doc (attached). In this
email I will try to answer the remaining unresolved questions.
*Target Audience?*
Falcon users are the people who write falcon processes and feeds and use
Falcon CLI/ Rest API to maintain/operate them. Falcon developers are the
people who work on enhancing falcon, regardless of the their decision to
contribute it back to the Apache Falcon, though it is highly encouraged.
Lifecycle extensions are meant for falcon developers to extend.
Through the lifecycle feature we aim to provide clean extension points for
extending feed management. It will be easier to understand falcon feed
lifecycle with the analogy of oozie-el-extensions.
*Is a restart of falcon required for adding a new lifecycle policy?*
Yes, a restart is required. Lifecycle policies are applicable for all feeds
in the falcon and we want addition of new lifecycle policies to be an
explicit and authorised action. Moreover, addition of new policy for a
lifecycle stage is a rare requirement and shouldn't be a hindrance in it's
adoption.
*Will lifecycle policies run as a separate process?*
No. Execution of lifecycle policies happen outside of falcon. Some parts
like preparing the workflows and validation are done inside falcon at the
time of submission of a feed. Again with the analogy of
oozie-el-extensions, this is an extension of falcon and is different from
user submitted workflows/process definitions and running it in a separate
JVM will be an overkill.
I have attached the latest doc and xsd patch in the JIRA for your
consideration. Would like to hear your thoughts. Once we have closure over
design, I will work towards providing a complete patch for base framework.
Cheers
Ajay Yadava
Re: [DISCUSS] Lifecycle design closure
Posted by Balu Vellanki <bv...@hortonworks.com>.
This is https://issues.apache.org/jira/browse/FALCON-965
On 9/8/15, 11:29 AM, "Seetharam Venkatesh" <ve...@innerzeal.com> wrote:
>It would help to add the Jira link and/or paste the design document here.
>
>On Tue, Sep 8, 2015 at 11:27 AM Ajay Yadav <aj...@gmail.com> wrote:
>
>> Hello everyone,
>>
>> During our previous sync up we discussed feed lifecycle. I have
>> incorporated the suggestions in the updated design doc (attached). In
>>this
>> email I will try to answer the remaining unresolved questions.
>>
>> *Target Audience?*
>> Falcon users are the people who write falcon processes and feeds and use
>> Falcon CLI/ Rest API to maintain/operate them. Falcon developers are the
>> people who work on enhancing falcon, regardless of the their decision to
>> contribute it back to the Apache Falcon, though it is highly encouraged.
>> Lifecycle extensions are meant for falcon developers to extend.
>>
>> Through the lifecycle feature we aim to provide clean extension points
>>for
>> extending feed management. It will be easier to understand falcon feed
>> lifecycle with the analogy of oozie-el-extensions.
>>
>>
>> *Is a restart of falcon required for adding a new lifecycle policy?*
>> Yes, a restart is required. Lifecycle policies are applicable for all
>>feeds
>> in the falcon and we want addition of new lifecycle policies to be an
>> explicit and authorised action. Moreover, addition of new policy for a
>> lifecycle stage is a rare requirement and shouldn't be a hindrance in
>>it's
>> adoption.
>>
>>
>> *Will lifecycle policies run as a separate process?*
>> No. Execution of lifecycle policies happen outside of falcon. Some parts
>> like preparing the workflows and validation are done inside falcon at
>>the
>> time of submission of a feed. Again with the analogy of
>> oozie-el-extensions, this is an extension of falcon and is different
>>from
>> user submitted workflows/process definitions and running it in a
>>separate
>> JVM will be an overkill.
>>
>>
>> I have attached the latest doc and xsd patch in the JIRA for your
>> consideration. Would like to hear your thoughts. Once we have closure
>>over
>> design, I will work towards providing a complete patch for base
>>framework.
>>
>>
>> Cheers
>> Ajay Yadava
>>
Re: [DISCUSS] Lifecycle design closure
Posted by Seetharam Venkatesh <ve...@innerzeal.com>.
It would help to add the Jira link and/or paste the design document here.
On Tue, Sep 8, 2015 at 11:27 AM Ajay Yadav <aj...@gmail.com> wrote:
> Hello everyone,
>
> During our previous sync up we discussed feed lifecycle. I have
> incorporated the suggestions in the updated design doc (attached). In this
> email I will try to answer the remaining unresolved questions.
>
> *Target Audience?*
> Falcon users are the people who write falcon processes and feeds and use
> Falcon CLI/ Rest API to maintain/operate them. Falcon developers are the
> people who work on enhancing falcon, regardless of the their decision to
> contribute it back to the Apache Falcon, though it is highly encouraged.
> Lifecycle extensions are meant for falcon developers to extend.
>
> Through the lifecycle feature we aim to provide clean extension points for
> extending feed management. It will be easier to understand falcon feed
> lifecycle with the analogy of oozie-el-extensions.
>
>
> *Is a restart of falcon required for adding a new lifecycle policy?*
> Yes, a restart is required. Lifecycle policies are applicable for all feeds
> in the falcon and we want addition of new lifecycle policies to be an
> explicit and authorised action. Moreover, addition of new policy for a
> lifecycle stage is a rare requirement and shouldn't be a hindrance in it's
> adoption.
>
>
> *Will lifecycle policies run as a separate process?*
> No. Execution of lifecycle policies happen outside of falcon. Some parts
> like preparing the workflows and validation are done inside falcon at the
> time of submission of a feed. Again with the analogy of
> oozie-el-extensions, this is an extension of falcon and is different from
> user submitted workflows/process definitions and running it in a separate
> JVM will be an overkill.
>
>
> I have attached the latest doc and xsd patch in the JIRA for your
> consideration. Would like to hear your thoughts. Once we have closure over
> design, I will work towards providing a complete patch for base framework.
>
>
> Cheers
> Ajay Yadava
>