[jira] [Commented] (FALCON-634) Add recipes in Falcon

["Venkat Ranganathan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FALCON-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120698#comment-15120698 ] 

Venkat Ranganathan commented on FALCON-634:
-------------------------------------------

Myself, [~sowmyaramesh] and [~yzheng-hortonworks]  discussed this today.

Basically
*  There should be two classes of recipes - core recipes - are those that are designed, developed and released as part of Falcon - these are essentially upto-date with the release of Falcon, have regression and integration tests that are run as part of the falcon release and thoroughly tested.    These qualify to run within the Falcon server and typically live under the _addons_ directory.   Examples are the HDFS and Hive mirroring, and future such addons

* Optional recipes are those that are customer/administrator written, not shipped with the product and may need updates with newer versions of the product.   Their execution model  (whether they get scheduled in a managed workflow or on a separate process) are things that need further discussion and clarity.

The packaging of the recipes for execution, their directory structure, how they can be introspected so that a UI can render the invocation of these REST APIs are common to both core and optional recipes.

There will be a new design document that we will  publish.   Just providing the gist of the discussion to kick off early discussions.

> Add recipes in Falcon
> ---------------------
>
>                 Key: FALCON-634
>                 URL: https://issues.apache.org/jira/browse/FALCON-634
>             Project: Falcon
>          Issue Type: New Feature
>    Affects Versions: 0.6
>            Reporter: Venkatesh Seetharam
>              Labels: recipes
>
> Falcon offers many services OOTB and caters to a wide array of use cases. However, there has been many asks that does not fit the functionality offered by Falcon. I'm proposing that we add recipes to Falcon which is similar to recipes in Whirr and other management solutions such as puppet and chef.
> Overview:
> A recipe essentially is a static process template with parameterized workflow to realize a specific use case. For example:
> * replicating directories from one HDFS cluster to another (not timed partitions)
> * replicating hive metadata (database, table, views, etc.)
> * replicating between HDFS and Hive - either way
> * anonymization of data based on schema
> * data masking
> * etc.
> Proposal:
> Falcon provides a Process abstraction that encapsulates the configuration 
> for a user workflow with scheduling controls. All recipes can be modeled 
> as a Process with in Falcon which executes the user workflow 
> periodically. The process and its associated workflow are parameterized. The user will provide a properties file with name value pairs that are substituted by falcon before scheduling it.
> This is a client side concept. The server does not know about a recipe but only accepts the cooked recipe as a process entity. 
> The CLI would look something like this:
> falcon -recipe $recipe_name -properties $properties_file
> Recipes will reside inside addons (contrib) with source code and will have an option to package 'em.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)