You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@falcon.apache.org by aj...@apache.org on 2015/07/09 14:09:12 UTC
svn commit: r1690074 [14/15] - in /falcon: site/ site/0.3-incubating/
site/0.4-incubating/ site/0.5-incubating/ site/0.6-incubating/ site/0.6.1/
site/0.6.1/css/ site/0.6.1/images/ site/0.6.1/images/logos/
site/0.6.1/images/profiles/ site/0.6.1/img/ sit...
Added: falcon/trunk/releases/0.6.1/src/site/twiki/Recipes.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/Recipes.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/Recipes.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/Recipes.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,77 @@
+---+ Falcon Recipes
+
+---++ Overview
+
+A Falcon recipe is a static process template with parameterized workflow to realize a specific use case. Recipes are
+defined in user space. Recipes will not have support for update or lifecycle management.
+
+For example:
+
+ * Replicating directories from one HDFS cluster to another (not timed partitions)
+ * Replicating hive metadata (database, table, views, etc.)
+ * Replicating between HDFS and Hive - either way
+ * Data masking etc.
+
+---++ Proposal
+
+Falcon provides a Process abstraction that encapsulates the configuration for a user workflow with scheduling
+controls. All recipes can be modeled as a Process with in Falcon which executes the user workflow periodically. The
+process and its associated workflow are parameterized. The user will provide a properties file with name value pairs
+that are substituted by falcon before scheduling it. Falcon translates these recipes as a process entity by
+replacing the parameters in the workflow definition.
+
+---++ Falcon CLI recipe support
+
+Falcon CLI functionality to support recipes has been added.
+<a href="./FalconCLI.html">Recipe command usage is defined here.</a>
+
+CLI accepts recipe option with a recipe name and optional tool and does the following:
+ * Validates the options; name option is mandatory and tool is optional and should be provided if user wants to override the base recipe tool
+ * Looks for <name>-workflow.xml, <name>-template.xml and <name>.properties file in the path specified by falcon.recipe.path in client.properties. If files cannot be found then Falcon CLI will fail
+ * Invokes a Tool to substitute the properties in the templated process for the recipe. By default invokes base tool if tool option is not passed. Tool is responsible for generating process entity at the path specified by FalconCLI
+ * Validates the generated entity
+ * Submit and schedule this entity
+ * Generated process entity files are stored in tmp directory
+
+---++ Base Recipe tool
+
+Falcon provides a base tool that recipes can override. Base Recipe tool does the following:
+ * Expects recipe template file path, recipe properties file path and path where process entity to be submitted should be generated. Validates these arguments
+ * Validates the artifacts i.e. workflow and/or lib files specified in the recipe template exists on local filesystem or HDFS at the specified path else returns error
+ * Copies if the artifacts exists on local filesystem
+ * If workflow is on local FS then falcon.recipe.workflow.path in recipe property file is mandatory for it to be copied to HDFS. If templated process requires custom libs falcon.recipe.workflow.lib.path property is mandatory for them to be copied from Local FS to HDFS. Recipe tool will copy the local artifacts only if these properties are set in properties file
+ * Looks for the patten ##[A-Za-z0-9_.]*## in the templated process and substitutes it with the properties. Process entity generated after the substitution is written to the empty file passed by FalconCLI
+
+---++ Recipe template file format
+
+ * Any templatized string should be in the format ##[A-Za-z0-9_.]*##.
+ * There should be a corresponding entry in the recipe properties file "falcon.recipe.<templatized-string> = <value to be substituted>"
+
+<verbatim>
+Example: If the entry in recipe template is <workflow name="##workflow.name##"> there should be a corresponding entry in the recipe properties file falcon.recipe.workflow.name=hdfs-dr-workflow
+</verbatim>
+
+---++ Recipe properties file format
+
+ * Regular key value pair properties file
+ * Property key should be prefixed by "falcon.recipe."
+
+<verbatim>
+Example: falcon.recipe.workflow.name=hdfs-dr-workflow
+Recipe template will have <workflow name="##workflow.name##">. Recipe tool will look for the patten ##workflow.name##
+and replace it with the property value "hdfs-dr-workflow". Substituted template will have <workflow name="hdfs-dr-workflow">
+</verbatim>
+
+---++ Managing the scheduled recipe process
+ * Scheduled recipe process is similar to regular process
+ * List : falcon entity -type process -name <recipe-process-name> -list
+ * Status : falcon entity -type process -name <recipe-process-name> -status
+ * Delete : falcon entity -type process -name <recipe-process-name> -delete
+
+---++ Sample recipes
+
+ * Sample recipes are published in addons/recipes
+
+---++ Packaging
+
+ * There is no packaging for recipes at this time but will be added soon.
Added: falcon/trunk/releases/0.6.1/src/site/twiki/Security.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/Security.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/Security.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/Security.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,384 @@
+---+ Securing Falcon
+
+---++ Overview
+
+Apache Falcon enforces authentication and authorization which are detailed below. Falcon also
+provides transport level security ensuring data confidentiality and integrity.
+
+
+---++ Authentication (User Identity)
+
+Apache Falcon enforces authentication on protected resources. Once authentication has been established it sets a
+signed HTTP Cookie that contains an authentication token with the user name, user principal,
+authentication type and expiration time.
+
+It does so by using [[http://hadoop.apache .org/docs/current/hadoop-auth/index.html][Hadoop Auth]].
+Hadoop Auth is a Java library consisting of a client and a server components to enable Kerberos SPNEGO authentication
+for HTTP. Hadoop Auth also supports additional authentication mechanisms on the client and the server side via 2
+simple interfaces.
+
+
+---+++ Authentication Methods
+
+It supports 2 authentication methods, simple and kerberos out of the box.
+
+---++++ Pseudo/Simple Authentication
+
+Falcon authenticates the user by simply trusting the value of the query string parameter 'user.name'. This is the
+default mode Falcon is configured with.
+
+---++++ Kerberos Authentication
+
+Falcon uses HTTP Kerberos SPNEGO to authenticate the user.
+
+
+---++ Authorization
+
+Falcon also enforces authorization on Entities using ACLs (Access Control Lists). ACLs are useful
+for implementing permission requirements and provide a way to set different permissions for
+specific users or named groups.
+
+By default, support for authorization is disabled and can be enabled in startup.properties.
+
+---+++ ACLs in Entity
+
+All Entities now have ACL which needs to be present if authorization is enabled. Only owners who
+own or created the entity will be allowed to update or delete their entities.
+
+An entity has ACLs (Access Control Lists) that are useful for implementing permission requirements
+and provide a way to set different permissions for specific users or named groups.
+<verbatim>
+ <ACL owner="test-user" group="test-group" permission="*"/>
+</verbatim>
+ACL indicates the Access control list for this cluster.
+owner is the Owner of this entity.
+group is the one which has access to read.
+permission indicates the rwx is not enforced at this time.
+
+---+++ Super-User
+
+The super-user is the user with the same identity as falcon process itself. Loosely, if you
+started the falcon, then you are the super-user. The super-user can do anything in that
+permissions checks never fail for the super-user. There is no persistent notion of who was the
+super-user; when the falcon is started the process identity determines who is the super-user
+for now. The Falcon super-user does not have to be the super-user of the falcon host, nor is it
+necessary that all clusters have the same super-user. Also, an experimenter running Falcon on a
+personal workstation, conveniently becomes that installation's super-user without any configuration.
+
+Falcon also allows users to configure a super user group and allows users belonging to this
+group to be a super user.
+
+ACL owner and group must be valid even if the authenticated user is a super-user.
+
+---+++ Group Memberships
+
+Once a user has been authenticated and a username has been determined, the list of groups is
+determined by a group mapping service, configured by the hadoop.security.group.mapping property
+in Hadoop. The default implementation, org.apache.hadoop.security.ShellBasedUnixGroupsMapping,
+will shell out to the Unix bash -c groups command to resolve a list of groups for a user.
+
+Note that Falcon stores the user and group of an Entity as strings; there is no
+conversion from user and group identity numbers as is conventional in Unix.
+
+The only limitation is that a user cannot add a group in ACL that he does not belong to.
+
+---+++ Authorization Provider
+
+Falcon provides a plugin-able provider interface for Authorization. It also ships with a default
+implementation that enforces the following authorization policy.
+
+---++++ Entity and Instance Management Operations Policy
+
+ * All Entity and Instance operations are authorized for users who created them, Owners and users with group memberships
+ * Reference to entities with in a feed or process is allowed with out enforcing permissions
+
+Any Feed or Process can refer to a Cluster entity not owned by the Feed or Process owner. Any Process can refer to a Feed entity not owned by the Process owner
+
+The authorization is enforced in the following way:
+
+ * if admin resource,
+ * If authenticated user name matches the admin users configuration
+ * Else if groups of the authenticated user matches the admin groups configuration
+ * Else authorization exception is thrown
+ * Else if entities or instance resource
+ * If the authenticated user matches the owner in ACL for the entity
+ * Else if the groups of the authenticated user matches the group in ACL for the entity
+ * Else authorization exception is thrown
+ * Else if lineage resource
+ * All have read-only permissions, reason being folks should be able to examine the dependency and allow reuse
+
+To authenticate user for REST api calls, user should append "user.name=<username>" to the query.
+
+*operations on Entity Resource*
+
+| *Resource* | *Description* | *Authorization* |
+| [[./restapi/EntityValidate][api/entities/validate/:entity-type]] | Validate the entity | Owner/Group |
+| [[./restapi/EntitySubmit][api/entities/submit/:entity-type]] | Submit the entity | Owner/Group |
+| [[./restapi/EntityUpdate][api/entities/update/:entity-type/:entity-name]] | Update the entity | Owner/Group |
+| [[./restapi/EntitySubmitAndSchedule][api/entities/submitAndSchedule/:entity-type]] | Submit & Schedule the entity | Owner/Group |
+| [[./restapi/EntitySchedule][api/entities/schedule/:entity-type/:entity-name]] | Schedule the entity | Owner/Group |
+| [[./restapi/EntitySuspend][api/entities/suspend/:entity-type/:entity-name]] | Suspend the entity | Owner/Group |
+| [[./restapi/EntityResume][api/entities/resume/:entity-type/:entity-name]] | Resume the entity | Owner/Group |
+| [[./restapi/EntityDelete][api/entities/delete/:entity-type/:entity-name]] | Delete the entity | Owner/Group |
+| [[./restapi/EntityStatus][api/entities/status/:entity-type/:entity-name]] | Get the status of the entity | Owner/Group |
+| [[./restapi/EntityDefinition][api/entities/definition/:entity-type/:entity-name]] | Get the definition of the entity | Owner/Group |
+| [[./restapi/EntityList][api/entities/list/:entity-type?fields=:fields]] | Get the list of entities | Owner/Group |
+| [[./restapi/EntityDependencies][api/entities/dependencies/:entity-type/:entity-name]] | Get the dependencies of the entity | Owner/Group |
+
+*REST Call on Feed and Process Instances*
+
+| *Resource* | *Description* | *Authorization* |
+| [[./restapi/InstanceRunning][api/instance/running/:entity-type/:entity-name]] | List of running instances. | Owner/Group |
+| [[./restapi/InstanceStatus][api/instance/status/:entity-type/:entity-name]] | Status of a given instance | Owner/Group |
+| [[./restapi/InstanceKill][api/instance/kill/:entity-type/:entity-name]] | Kill a given instance | Owner/Group |
+| [[./restapi/InstanceSuspend][api/instance/suspend/:entity-type/:entity-name]] | Suspend a running instance | Owner/Group |
+| [[./restapi/InstanceResume][api/instance/resume/:entity-type/:entity-name]] | Resume a given instance | Owner/Group |
+| [[./restapi/InstanceRerun][api/instance/rerun/:entity-type/:entity-name]] | Rerun a given instance | Owner/Group |
+| [[InstanceLogs][api/instance/logs/:entity-type/:entity-name]] | Get logs of a given instance | Owner/Group |
+
+---++++ Admin Resources Policy
+
+Only users belonging to admin users or groups have access to this resource. Admin membership is
+determined by a static configuration parameter.
+
+| *Resource* | *Description* | *Authorization* |
+| [[./restapi/AdminVersion][api/admin/version]] | Get version of the server | No restriction |
+| [[./restapi/AdminStack][api/admin/stack]] | Get stack of the server | Admin User/Group |
+| [[./restapi/AdminConfig][api/admin/config/:config-type]] | Get configuration information of the server | Admin User/Group |
+
+
+---++++ Lineage Resource Policy
+
+Lineage is read-only and hence all users can look at lineage for their respective entities.
+*Note:* This gap will be fixed in a later release.
+
+
+---++ Authentication Configuration
+
+Following is the Server Side Configuration Setup for Authentication.
+
+---+++ Common Configuration Parameters
+
+<verbatim>
+# Authentication type must be specified: simple|kerberos
+*.falcon.authentication.type=kerberos
+</verbatim>
+
+---+++ Kerberos Configuration
+
+<verbatim>
+##### Service Configuration
+
+# Indicates the Kerberos principal to be used in Falcon Service.
+*.falcon.service.authentication.kerberos.principal=falcon/_HOST@EXAMPLE.COM
+
+# Location of the keytab file with the credentials for the Service principal.
+*.falcon.service.authentication.kerberos.keytab=/etc/security/keytabs/falcon.service.keytab
+
+# name node principal to talk to config store
+*.dfs.namenode.kerberos.principal=nn/_HOST@EXAMPLE.COM
+
+##### SPNEGO Configuration
+
+# Authentication type must be specified: simple|kerberos|<class>
+# org.apache.falcon.security.RemoteUserInHeaderBasedAuthenticationHandler can be used for backwards compatibility
+*.falcon.http.authentication.type=kerberos
+
+# Indicates how long (in seconds) an authentication token is valid before it has to be renewed.
+*.falcon.http.authentication.token.validity=36000
+
+# The signature secret for signing the authentication tokens.
+*.falcon.http.authentication.signature.secret=falcon
+
+# The domain to use for the HTTP cookie that stores the authentication token.
+*.falcon.http.authentication.cookie.domain=
+
+# Indicates if anonymous requests are allowed when using 'simple' authentication.
+*.falcon.http.authentication.simple.anonymous.allowed=true
+
+# Indicates the Kerberos principal to be used for HTTP endpoint.
+# The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.
+*.falcon.http.authentication.kerberos.principal=HTTP/_HOST@EXAMPLE.COM
+
+# Location of the keytab file with the credentials for the HTTP principal.
+*.falcon.http.authentication.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab
+
+# The kerberos names rules is to resolve kerberos principal names, refer to Hadoop's KerberosName for more details.
+*.falcon.http.authentication.kerberos.name.rules=DEFAULT
+
+# Comma separated list of black listed users
+*.falcon.http.authentication.blacklisted.users=
+
+# Increase Jetty request buffer size to accommodate the generated Kerberos token
+*.falcon.jetty.request.buffer.size=16192
+</verbatim>
+
+---+++ Pseudo/Simple Configuration
+
+<verbatim>
+##### SPNEGO Configuration
+
+# Authentication type must be specified: simple|kerberos|<class>
+# org.apache.falcon.security.RemoteUserInHeaderBasedAuthenticationHandler can be used for backwards compatibility
+*.falcon.http.authentication.type=simple
+
+# Indicates how long (in seconds) an authentication token is valid before it has to be renewed.
+*.falcon.http.authentication.token.validity=36000
+
+# The signature secret for signing the authentication tokens.
+*.falcon.http.authentication.signature.secret=falcon
+
+# The domain to use for the HTTP cookie that stores the authentication token.
+*.falcon.http.authentication.cookie.domain=
+
+# Indicates if anonymous requests are allowed when using 'simple' authentication.
+*.falcon.http.authentication.simple.anonymous.allowed=true
+
+# Comma separated list of black listed users
+*.falcon.http.authentication.blacklisted.users=
+</verbatim>
+
+---++ Authorization Configuration
+
+---+++ Enabling Authorization
+By default, support for authorization is disabled and specifying ACLs in entities are optional.
+To enable support for authorization, set falcon.security.authorization.enabled to true in the
+startup configuration.
+
+<verbatim>
+# Authorization Enabled flag: false|true
+*.falcon.security.authorization.enabled=true
+</verbatim>
+
+---+++ Authorization Provider
+
+Falcon provides a basic implementation for Authorization bundled, org.apache.falcon.security .DefaultFalconAuthorizationProvider.
+This can be overridden by custom implementations in the startup configuration.
+
+<verbatim>
+# Authorization Provider Fully Qualified Class Name
+*.falcon.security.authorization.provider=org.apache.falcon.security.DefaultAuthorizationProvider
+</verbatim>
+
+---+++ Super User Group
+
+Super user group is determined by the configuration:
+
+<verbatim>
+# The name of the group of super-users
+*.falcon.security.authorization.superusergroup=falcon
+</verbatim>
+
+---+++ Admin Membership
+
+Administrative users are determined by the configuration:
+
+<verbatim>
+# Admin Users, comma separated users
+*.falcon.security.authorization.admin.users=falcon,ambari-qa,seetharam
+</verbatim>
+
+Administrative groups are determined by the configuration:
+
+<verbatim>
+# Admin Group Membership, comma separated users
+*.falcon.security.authorization.admin.groups=falcon,testgroup,staff
+</verbatim>
+
+
+---++ SSL
+
+Falcon provides transport level security ensuring data confidentiality and integrity. This is
+enabled by default for communicating over HTTP between the client and the server.
+
+---+++ SSL Configuration
+
+<verbatim>
+*.falcon.enableTLS=true
+*.keystore.file=/path/to/keystore/file
+*.keystore.password=password
+</verbatim>
+
+---+++ Distributed Falcon Setup
+
+Falcon should be configured to communicate with Prism over TLS in secure mode. Its not enabled by default.
+
+
+---++ Changes to ownership and permissions of directories managed by Falcon
+
+| *Directory* | *Location* | *Owner* | *Permissions* |
+| Configuration Store | ${config.store.uri} | falcon | 700 |
+| Cluster Staging Location | ${cluster.staging-location} | falcon | 777 |
+| Cluster Working Location | ${cluster.working-location} | falcon | 755 |
+| Shared libs | {cluster.working}/{lib,libext} | falcon | 755 |
+| Oozie coord/bundle XMLs | ${cluster.staging-location}/workflows/{entity}/{entity-name} | $user | cluster umask |
+| App logs | ${cluster.staging-location}/workflows/{entity}/{entity-name}/logs | $user | cluster umask |
+
+*Note:* Please note that the cluster staging and working locations MUST be created prior to
+submitting a cluster entity to Falcon. Also, note that the the parent dirs must have execute
+permissions.
+
+
+---++ Backwards compatibility
+
+---+++ Scheduled Entities
+
+Entities already scheduled with an earlier version of Falcon are not compatible with this version
+
+---+++ Falcon Clients
+
+Older Falcon clients are backwards compatible wrt Authentication and user information sent as part of the HTTP
+header, Remote-User is still honoured when the authentication type is configured as below:
+
+<verbatim>
+*.falcon.http.authentication.type=org.apache.falcon.security.RemoteUserInHeaderBasedAuthenticationHandler
+</verbatim>
+
+---+++ Blacklisted super users for authentication
+
+The blacklist users used to have the following super users: hdfs, mapreduce, oozie, and falcon.
+The list is externalized from code into Startup.properties file and is empty now and needs to be
+configured specifically in the file.
+
+
+---+++ Falcon Dashboard
+
+To initialize the current user for dashboard, user should append query param "user.name=<username>" to the REST api call.
+
+If dashboard user wishes to change the current user, they should do the following.
+ * delete the hadoop.auth cookie from browser cache.
+ * append query param "user.name=<new_user>" to the next REST API call.
+
+In Kerberos method, the browser must support HTTP Kerberos SPNEGO.
+
+
+---++ Known Limitations
+
+ * ActiveMQ topics are not secure but will be in the near future
+ * Entities already scheduled with an earlier version of Falcon are not compatible with this version as new
+ workflow parameters are being passed back into Falcon such as the user are required
+ * Use of hftp as the scheme for read only interface in cluster entity [[https://issues.apache.org/jira/browse/HADOOP-10215][will not work in Oozie]]
+ The alternative is to use webhdfs scheme instead and its been tested with DistCp.
+
+
+---++ Examples
+
+---+++ Accessing the server using Falcon CLI (Java client)
+
+There is no change in the way the CLI is used. The CLI has been changed to work with the configured authentication
+method.
+
+---+++ Accessing the server using curl
+
+Try accessing protected resources using curl. The protected resources are:
+
+<verbatim>
+$ kinit
+Please enter the password for venkatesh@LOCALHOST:
+
+$ curl http://localhost:15000/api/admin/version
+
+$ curl http://localhost:15000/api/admin/version?user.name=venkatesh
+
+$ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt curl http://localhost:15000/api/admin/version
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/index.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/index.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/index.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/index.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,19 @@
+---+++ Contents
+
+ * <a href="./InstallationSteps.html">Simple setup</a>
+
+ * <a href="./FalconArchitecture.html">Overview</a>
+
+ * <a href="./HiveIntegration.html">Hive Integration</a>
+
+ * <a href="./Security.html">Security</a>
+
+ * <a href="./OnBoarding.html">On boarding</a>
+
+ * <a href="./EntitySpecification.html">Entity specification</a>
+
+ * <a href="./FalconCLI.html">CLI</a>
+
+ * <a href="./Operability.html">Operability</a>
+
+ * <a href="./restapi/ResourceList.html">Rest API</a>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdjacentVertices.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdjacentVertices.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdjacentVertices.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdjacentVertices.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,70 @@
+---++ GET api/metadata/lineage/vertices/:id/:direction
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get a list of adjacent vertices or edges with a direction.
+
+---++ Parameters
+ * :id is the id of the vertex.
+ * :direction is the direction associated with the edges.
+
+ To get the adjacent out vertices of vertex pass direction as out, in to get adjacent in vertices
+ and both to get both in and out adjacent vertices. Similarly to get the out edges of vertex
+ pass outE, inE to get in edges and bothE to get the both in and out edges of vertex.
+
+ * out : get the adjacent out vertices of vertex
+ * in : get the adjacent in vertices of vertex
+ * both : get the both adjacent in and out vertices of vertex
+ * outCount : get the number of out vertices of vertex
+ * inCount : get the number of in vertices of vertex
+ * bothCount : get the number of adjacent in and out vertices of vertex
+ * outIds : get the identifiers of out vertices of vertex
+ * inIds : get the identifiers of in vertices of vertex
+ * bothIds : get the identifiers of adjacent in and out vertices of vertex
+
+---++ Results
+Adjacent vertices of the vertex for the specified direction.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/metadata/lineage/vertices/4/out
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "results": [
+ {
+ "timestamp":"2014-04-21T20:55Z",
+ "name":"sampleFeed",
+ "type":"feed-instance",
+ "_id":8,
+ "_type":"vertex"
+ }
+ ],
+ "totalSize":1}
+}
+</verbatim>
+
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/metadata/lineage/vertices/4/bothE
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "results":[
+ {
+ "_id":"Q5V-4-5g",
+ "_type":"edge",
+ "_outV":4,
+ "_inV":8,
+ "_label":"output"
+ }
+ ],
+ "totalSize":1
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminConfig.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminConfig.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminConfig.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminConfig.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,35 @@
+---++ GET /api/admin/config/:config-type
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get configuration information of the falcon server.
+
+---++ Parameters
+ * :config-type can be build, deploy, startup or runtime
+
+---++ Results
+Configuration information of the server.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/admin/config/deploy
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "properties": [
+ {
+ "value": "embedded",
+ "key": "deploy.mode"
+ },
+ {
+ "value": "all",
+ "key": "domain"
+ }
+ ]
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminStack.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminStack.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminStack.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminStack.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,39 @@
+---++ GET /api/admin/stack
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get stack trace of the falcon server.
+
+---++ Parameters
+None.
+---++ Results
+Stack trace of the server.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/admin/stack
+</verbatim>
+---+++ Result
+<verbatim>
+Reference Handler
+State: WAITING
+java.lang.Object.wait(Native Method)
+java.lang.Object.wait(Object.java:485)
+java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)Finalizer
+
+...
+
+State: TIMED_WAITING
+sun.misc.Unsafe.park(Native Method)
+java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
+java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
+java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
+java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
+java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:955)
+java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
+java.lang.Thread.run(Thread.java:695)
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminVersion.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminVersion.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminVersion.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AdminVersion.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,34 @@
+---++ GET /api/admin/version
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get version of the falcon server.
+
+---++ Parameters
+None.
+---++ Results
+Version of the server.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/admin/version
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "properties":[
+ {
+ "key":"Version",
+ "value":"0.4-incubating-SNAPSHOT-rb47788d1112fcf949c22a3860934167237b395b0"
+ },
+ {
+ "key":"Mode",
+ "value":"embedded"
+ }
+ ]
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AllEdges.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AllEdges.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AllEdges.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AllEdges.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,42 @@
+---++ GET pi/metadata/lineage//edges/all
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get all edges.
+
+---++ Parameters
+None.
+
+---++ Results
+All edges in lineage graph.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/metadata/lineage/edges/all
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "results": [
+ {
+ "_id":"Q5V-4-5g",
+ "_type":"edge",
+ "_outV":4,
+ "_inV":8,
+ "_label":"output"
+ },
+ {
+ "_id":"Q6t-c-5g",
+ "_type":"edge",
+ "_outV":12,
+ "_inV":16,
+ "_label":"output"
+ }
+ ],
+ "totalSize": 2
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AllVertices.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AllVertices.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AllVertices.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/AllVertices.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,43 @@
+---++ GET api/metadata/lineage/vertices/all
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get all vertices.
+
+---++ Parameters
+None.
+
+---++ Results
+All vertices in lineage graph.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/metadata/lineage/vertices/all
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "results": [
+ {
+ "timestamp":"2014-04-21T20:55Z",
+ "name":"sampleIngestProcess\/2014-03-01T10:00Z",
+ "type":"process-instance",
+ "version":"2.0.0",
+ "_id":4,
+ "_type":"vertex"
+ },
+ {
+ "timestamp":"2014-04-21T20:55Z",
+ "name":"rawEmailFeed\/2014-03-01T10:00Z",
+ "type":"feed-instance",
+ "_id":8,
+ "_type":"vertex"
+ }
+ ],
+ "totalSize": 2
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/Edge.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/Edge.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/Edge.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/Edge.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,33 @@
+---++ GET api/metadata/lineage/edges/:id
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Gets the edge with specified id.
+
+---++ Parameters
+ * :id is the unique id of the edge.
+
+---++ Results
+Edge with the specified id.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/metadata/lineage/edges/Q6t-c-5g
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "results":
+ {
+ "_id":"Q6t-c-5g",
+ "_type":"edge",
+ "_outV":12,
+ "_inV":16,
+ "_label":"output"
+ }
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDefinition.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDefinition.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDefinition.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDefinition.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,52 @@
+---++ GET /api/entities/definition/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get definition of the entity.
+
+---++ Parameters
+ * :entity-type can be cluster, feed or process.
+ * :entity-name is name of the entity.
+
+---++ Results
+Definition of the entity.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/entities/definition/process/SampleProcess
+</verbatim>
+---+++ Result
+<verbatim>
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<process name="SampleProcess" xmlns="uri:falcon:process:0.1">
+ <clusters>
+ <cluster name="primary-cluster">
+ <validity start="2012-04-03T06:00Z" end="2022-12-30T00:00Z"/>
+ </cluster>
+ </clusters>
+ <parallel>1</parallel>
+ <order>FIFO</order>
+ <frequency>hours(1)</frequency>
+ <timezone>UTC</timezone>
+ <inputs>
+ <input name="input" feed="SampleInput" start="yesterday(0,0)" end="today(-1,0)"/>
+ </inputs>
+ <outputs>
+ <output name="output" feed="SampleOutput" instance="yesterday(0,0)"/>
+ </outputs>
+ <properties>
+ <property name="queueName" value="default"/>
+ <property name="ssh.host" value="localhost"/>
+ <property name="fileTimestamp" value="${coord:formatTime(coord:nominalTime(), 'yyyy-MM-dd')}"/>
+ </properties>
+ <workflow engine="oozie" path="/examples/apps/aggregator"/>
+ <retry policy="exp-backoff" delay="minutes(5)" attempts="3"/>
+ <late-process policy="exp-backoff" delay="hours(1)">
+ <late-input input="input" workflow-path="/projects/bootcamp/workflow/lateinput"/>
+ </late-process>
+</process>
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDelete.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDelete.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDelete.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDelete.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,30 @@
+---++ DELETE /api/entities/delete/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Delete the specified entity.
+
+---++ Parameters
+ * :entity-type can be feed or process.
+ * :entity-name is name of the feed or process.
+
+---++ Results
+Results of the delete operation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+DELETE http://localhost:15000/api/entities/delete/cluster/SampleProcess
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "requestId": "falcon\/17ff6ca6-1c8a-459f-9ba8-8fec480e384a\n",
+ "message": "falcon\/SampleProcess(cluster) removed successfully\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
+
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDependencies.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDependencies.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDependencies.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityDependencies.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,42 @@
+---++ GET /api/entities/dependencies/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get dependencies of the entity.
+
+---++ Parameters
+ * :entity-type can be cluster, feed or process.
+ * :entity-name is name of the entity.
+
+---++ Results
+Dependenciess of the entity.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/entities/dependencies/process/SampleProcess
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "entity": [
+ {
+ "name": "SampleInput",
+ "type": "feed",
+ "tag": [Input]
+ },
+ {
+ "name": "SampleOutput",
+ "type": "feed"
+ "tag": [Output]
+ },
+ {
+ "name": "primary-cluster",
+ "type": "cluster"
+ }
+ ]
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityLineage.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityLineage.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityLineage.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityLineage.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,39 @@
+---++ GET api/metadata/lineage/entities?pipeline=:pipeline
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+It returns the graph depicting the relationship between the various processes and feeds in a given pipeline.
+
+---++ Parameters
+ * :pipeline is the name of the pipeline
+
+---++ Results
+It returns a json graph
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/metadata/lineage/entities?pipeline=my-pipeline
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "vertices": ["my-minutely-process", "my-hourly-process"],
+ "edges":
+ [
+ {
+ "from" : "my-minutely-process",
+ "to" : "my-hourly-process",
+ "label" : "my-minutely-feed"
+ },
+ {
+ "from" : "my-hourly-process",
+ "to" : "my-minutely-process",
+ "label" : "my-hourly-feedback"
+ }
+ ]
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityList.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityList.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityList.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityList.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,125 @@
+---++ GET /api/entities/list/:entity-type?fields=:fields
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get list of the entities.
+
+---++ Parameters
+ * :entity-type Valid options are cluster, feed or process.
+ * fields <optional param> Fields of entity that the user wants to view, separated by commas.
+ * Valid options are STATUS, TAGS, PIPELINES.
+ * nameseq <optional param> Subsequence of entity name. Not case sensitive.
+ * The entity name needs to contain all the characters in the subsequence in the same order.
+ * Example 1: "sample1" will match the entity named "SampleFeed1-2".
+ * Example 2: "mhs" will match the entity named "New-My-Hourly-Summary".
+ * filterBy <optional param> Filter results by list of field:value pairs. Example: filterBy=STATUS:RUNNING,PIPELINES:clickLogs
+ * Supported filter fields are NAME, STATUS, PIPELINES, CLUSTER.
+ * Query will do an AND among filterBy fields.
+ * tags <optional param> Return list of entities that have specified tags, separated by a comma. Query will do AND on tag values.
+ * Example: tags=consumer=consumer@xyz.com,owner=producer@xyz.com
+ * orderBy <optional param> Field by which results should be ordered.
+ * Supports ordering by "name".
+ * sortOrder <optional param> Valid options are "asc" and "desc"
+ * offset <optional param> Show results from the offset, used for pagination. Defaults to 0.
+ * numResults <optional param> Number of results to show per request, used for pagination. Only integers > 0 are valid, Default is 10.
+
+---++ Results
+List of the entities.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/entities/list/feed
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "entity": [
+ {
+ "name": "SampleOutput",
+ "type": "feed"
+ },
+ {
+ "name": "SampleInput",
+ "type": "feed"
+ }
+ ]
+}
+</verbatim>
+
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/entities/list/feed?fields=status
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "entity": [
+ {
+ "name" : "SampleOutput",
+ "type" : "feed",
+ "status": "RUNNING"
+ },
+ {
+ "name": "SampleInput",
+ "type": "feed",
+ "status": "RUNNING"
+ }
+ ]
+}
+</verbatim>
+
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/entities/list/process?filterBy=STATUS:RUNNING,PIPELINES:dataReplication&fields=status,pipelines,tags&tags=consumer=consumer@xyz.com&orderBy=name&offset=2&numResults=2
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "entity": [
+ {
+ "name" : "SampleProcess1",
+ "type" : "process",
+ "status": "RUNNING",
+ "pipelines": "dataReplication",
+ "tags": "consumer=consumer@xyz.com"
+ },
+ {
+ "name": "SampleProcess3",
+ "type": "process",
+ "status": "RUNNING",
+ "pipelines": "dataReplication",
+ "tags": "consumer=consumer@xyz.com,owner=producer@xyz.com"
+ }
+ ]
+}
+</verbatim>
+
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/entities/list/feed?nameseq=samplebill&numResults=2&offset=1&fields=status,clusters,tags
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "entityâ:[
+ {
+ "type":"FEEDâ,
+ "name":"SampleUSHealthBillâ,
+ "status":"SUBMITTEDâ,
+ "tagsâ: {"tag":["related=ushealthcare","department=billingDepartment"]},
+ "clusters": {"cluster":["SampleCluster1","primaryClusterâ]}
+ },
+ {
+ "type":"FEEDâ,
+ "name":"SampleHealthBillâ,
+ "status":"SUBMITTEDâ,
+ "tagsâ: {"tag":["related=healthcare","department=billingDepartment"]},
+ "clusters": {"cluster":"primaryClusterâ}
+ }
+ ]
+}
+</verbatim>
\ No newline at end of file
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityResume.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityResume.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityResume.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityResume.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,29 @@
+---++ POST /api/entities/resume/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Resume a supended entity.
+
+---++ Parameters
+ * :entity-type can either be a feed or a process.
+ * :entity-name is name of the entity.
+
+---++ Results
+Result of the resume command.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/resume/process/SampleProcess
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "requestId": "default\/106582a9-130f-4903-8b8f-f95d7b286c30\n",
+ "message": "default\/SampleProcess(process) resumed successfully\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySchedule.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySchedule.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySchedule.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySchedule.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,29 @@
+---++ POST /api/entities/schedule/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Schedule an entity.
+
+---++ Parameters
+ * :entity-type can either be a feed or a process.
+ * :entity-name is name of the entity.
+
+---++ Results
+Result of the schedule command.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/schedule/process/SampleProcess
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "requestId": "default\/ee735c95-98bd-41b8-a705-2e78bcfcdcd9\n",
+ "message": "default\/SampleProcess(process) scheduled successfully\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityStatus.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityStatus.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityStatus.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityStatus.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,29 @@
+---++ GET /api/entities/status/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get status of the entity.
+
+---++ Parameters
+ * :entity-type can be cluster, feed or process.
+ * :entity-name is name of the entity.
+
+---++ Results
+Status of the entity.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/entities/status/process/SampleProcess
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "requestId": "default\/4d35b382-852a-4bc7-9972-b9db3493322a\n",
+ "message": "default\/SUBMITTED\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySubmit.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySubmit.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySubmit.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySubmit.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,104 @@
+---++ POST api/entities/submit/:entity-type
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Submit the given entity.
+
+---++ Parameters
+:entity-type can be cluster, feed or process.
+
+---++ Results
+Result of the submission.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/submit/feed
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Hourly sample input data -->
+
+<feed description="sample input data"
+ name="SampleInput" xmlns="uri:falcon:feed:0.1"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+ <groups>group</groups>
+ <frequency>hours(1)</frequency>
+ <late-arrival cut-off="hours(6)" />
+ <clusters>
+ <cluster name="primary-cluster" type="source">
+ <!--validity start="2009-01-01T00:00Z" end="2099-12-31T00:00Z" timezone="UTC" /-->
+ <validity start="2009-01-01T00:00Z" end="2099-12-31T00:00Z" />
+ <retention limit="months(24)" action="delete" />
+ </cluster>
+ </clusters>
+
+ <locations>
+ <location type="data" path="/projects/bootcamp/data/${YEAR}-${MONTH}-${DAY}-${HOUR}/SampleInput" />
+ <location type="stats" path="/projects/bootcamp/stats/SampleInput" />
+ <location type="meta" path="/projects/bootcamp/meta/SampleInput" />
+ </locations>
+
+ <ACL owner="suser" group="users" permission="0755" />
+
+ <schema location="/none" provider="none" />
+</feed>
+</verbatim>
+
+---+++ Result
+<verbatim>
+{
+ "requestId": "default\/d72a41f7-6420-487b-8199-62d66e492e35\n",
+ "message": "default\/Submit successful (feed) SampleInput\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
+
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/submit/process
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Daily sample process. Runs at 6th hour every day. Input - last day's hourly data. Generates output for yesterday -->
+<process xmlns="uri:falcon:process:0.1" name="SampleProcess" >
+ <clusters>
+ <cluster name="primary-cluster">
+ <validity start="2012-04-03T06:00Z" end="2022-12-30T00:00Z" />
+ </cluster>
+ </clusters>
+
+ <parallel>1</parallel>
+ <order>FIFO</order>
+ <frequency>hours(1)</frequency>
+
+ <inputs>
+ <input name="input" feed="SampleInput" start="yesterday(0,0)" end="today(-1,0)" />
+ </inputs>
+
+ <outputs>
+ <output name="output" feed="SampleOutput" instance="yesterday(0,0)" />
+ </outputs>
+
+ <properties>
+ <property name="queueName" value="default" />
+ <property name="ssh.host" value="localhost" />
+ <property name="fileTimestamp" value="${coord:formatTime(coord:nominalTime(), 'yyyy-MM-dd')}" />
+ </properties>
+
+ <workflow engine="oozie" path="/examples/apps/aggregator" />
+ <retry policy="exp-backoff" delay="minutes(5)" attempts="3" />
+
+ <late-process policy="exp-backoff" delay="hours(1)">
+ <late-input input="input" workflow-path="/projects/bootcamp/workflow/lateinput" />
+ </late-process>
+</process>
+</verbatim>
+
+---+++ Result
+<verbatim>
+{
+ "requestId": "default\/e5cc8230-f356-4566-9b65-536abdff8aa3\n",
+ "message": "default\/Submit successful (process) SampleProcess\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySubmitAndSchedule.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySubmitAndSchedule.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySubmitAndSchedule.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySubmitAndSchedule.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,62 @@
+---++ POST /api/entities/submitAndSchedule/:entity-type
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Submits and schedules an entity.
+
+---++ Parameters
+ * :entity-type can either be a feed or a process.
+
+---++ Results
+Result of the submit and schedule command.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/submitAndSchedule/process
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Daily sample process. Runs at 6th hour every day. Input - last day's hourly data. Generates output for yesterday -->
+<process xmlns="uri:falcon:process:0.1" name="SampleProcess" >
+ <clusters>
+ <cluster name="primary-cluster">
+ <validity start="2012-04-03T06:00Z" end="2022-12-30T00:00Z" />
+ </cluster>
+ </clusters>
+
+ <parallel>1</parallel>
+ <order>FIFO</order>
+ <frequency>hours(1)</frequency>
+
+ <inputs>
+ <input name="input" feed="SampleInput" start="yesterday(0,0)" end="today(-1,0)" />
+ </inputs>
+
+ <outputs>
+ <output name="output" feed="SampleOutput" instance="yesterday(0,0)" />
+ </outputs>
+
+ <properties>
+ <property name="queueName" value="default" />
+ <property name="ssh.host" value="localhost" />
+ <property name="fileTimestamp" value="${coord:formatTime(coord:nominalTime(), 'yyyy-MM-dd')}" />
+ </properties>
+
+ <workflow engine="oozie" path="/examples/apps/aggregator" />
+ <retry policy="exp-backoff" delay="minutes(5)" attempts="3" />
+
+ <late-process policy="exp-backoff" delay="hours(1)">
+ <late-input input="input" workflow-path="/projects/bootcamp/workflow/lateinput" />
+ </late-process>
+</process>
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "requestId": "schedule\/default\/b5b40931-175b-4b15-8f2b-02ef2e66f06b\n\nsubmit\/default\/b5b40931-175b-4b15-8f2b-02ef2e66f06b\n\n",
+ "message": "schedule\/default\/SampleProcess(process) scheduled successfully\n\nsubmit\/default\/Submit successful (process) SampleProcess\n\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySummary.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySummary.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySummary.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySummary.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,73 @@
+---++ GET /api/entities/summary/:entity-type
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Given an EntityType and cluster, get list of entities along with summary of N recent instances of each entity
+
+---++ Parameters
+ * :entity-type Valid options are feed or process.
+ * cluster Show entities that belong to this cluster.
+ * start <optional param> Show entity summaries from this date. Date format is yyyy-MM-dd'T'HH:mm'Z'.
+ * By default, it is set to (end - 2 days).
+ * end <optional param> Show entity summary up to this date. Date format is yyyy-MM-dd'T'HH:mm'Z'.
+ * Default is set to now.
+ * fields <optional param> Fields of entity that the user wants to view, separated by commas.
+ * Valid options are STATUS, TAGS, PIPELINES.
+ * filterBy <optional param> Filter results by list of field:value pairs. Example: filterBy=STATUS:RUNNING,PIPELINES:clickLogs
+ * Supported filter fields are NAME, STATUS, PIPELINES, CLUSTER.
+ * Query will do an AND among filterBy fields.
+ * tags <optional param> Return list of entities that have specified tags, separated by a comma. Query will do AND on tag values.
+ * Example: tags=consumer=consumer@xyz.com,owner=producer@xyz.com
+ * orderBy <optional param> Field by which results should be ordered.
+ * Supports ordering by "name".
+ * sortOrder <optional param> Valid options are "asc" and "desc"
+ * offset <optional param> Show results from the offset, used for pagination. Defaults to 0.
+ * numResults <optional param> Number of results to show per request, used for pagination. Only integers > 0 are valid, Default is 10.
+ * numInstances <optional param> Number of recent instances to show per entity. Only integers > 0 are valid, Default is 7.
+
+---++ Results
+Show entities along with summary of N instances for each entity.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/entities/summary/feed?cluster=primary-cluster&filterBy=STATUS:RUNNING&fields=status&tags=consumer=consumer@xyz.com&orderBy=name&offset=0&numResults=1&numInstances=2
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "entitySummary": [
+ {
+ "name" : "SampleOutput",
+ "type" : "feed",
+ "status": "RUNNING",
+ "instances": [
+ {
+ "details": "",
+ "endTime": "2013-10-21T14:40:26-07:00",
+ "startTime": "2013-10-21T14:39:56-07:00",
+ "cluster": "primary-cluster",
+ "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W",
+ "status": "RUNNING",
+ "instance": "2012-04-03T07:00Z"
+ },
+ {
+ "details": "",
+ "endTime": "2013-10-21T14:42:27-07:00",
+ "startTime": "2013-10-21T14:41:57-07:00",
+ "cluster": "primary-cluster",
+ "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933397-oozie-rgau-W",
+ "status": "RUNNING",
+ "instance": "2012-04-03T08:00Z"
+ },
+ ]
+ }
+ ]
+ "requestId": "default\/e15bb378-d09f-4911-9df2-5334a45153d2\n",
+ "message": "default\/STATUS\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySuspend.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySuspend.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySuspend.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntitySuspend.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,29 @@
+---++ POST /api/entities/suspend/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Suspend an entity.
+
+---++ Parameters
+ * :entity-type can either be a feed or a process.
+ * :entity-name is name of the entity.
+
+---++ Results
+Status of the entity.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/suspend/process/SampleProcess
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "requestId": "default\/fe5f2b6c-1f2e-49fc-af3a-342079f0b46b\n",
+ "message": "default\/SampleProcess(process) suspended successfully\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityTouch.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityTouch.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityTouch.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityTouch.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,29 @@
+---++ POST api/entities/touch/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Force updates the entity.
+
+---++ Parameters
+ * :entity-type can be feed or process.
+ * :entity-name is name of the feed or process.
+
+---++ Results
+Result of the validation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/touch/process/SampleProcess
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "requestId": "touch\/default\/d6aaa328-6836-4818-a212-515bb43d8b86\n\n",
+ "message": "touch\/default\/SampleProcess updated successfully\n\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
\ No newline at end of file
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityUpdate.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityUpdate.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityUpdate.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityUpdate.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,64 @@
+---++ POST api/entities/update/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Updates the submitted entity.
+
+---++ Parameters
+ * :entity-type can be feed or process.
+ * :entity-name is name of the feed or process.
+
+---++ Results
+Result of the validation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/update/process/SampleProcess
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Daily sample process. Runs at 6th hour every day. Input - last day's hourly data. Generates output for yesterday -->
+<process xmlns="uri:falcon:process:0.1" name="SampleProcess" >
+ <clusters>
+ <cluster name="primary-cluster">
+ <validity start="2012-04-03T06:00Z" end="2022-12-30T00:00Z" />
+ </cluster>
+ </clusters>
+
+ <parallel>1</parallel>
+ <order>FIFO</order>
+ <frequency>hours(1)</frequency>
+
+ <inputs>
+ <input name="input" feed="SampleInput" start="yesterday(0,0)" end="today(-1,0)" />
+ </inputs>
+
+ <outputs>
+ <output name="output" feed="SampleOutput" instance="yesterday(0,0)" />
+ </outputs>
+
+ <properties>
+ <property name="queueName" value="default" />
+ <property name="ssh.host" value="localhost" />
+ <property name="fileTimestamp" value="${coord:formatTime(coord:nominalTime(), 'yyyy-MM-dd')}" />
+ </properties>
+
+ <workflow engine="oozie" path="/examples/apps/aggregator" />
+ <retry policy="exp-backoff" delay="minutes(5)" attempts="3" />
+
+ <late-process policy="exp-backoff" delay="hours(1)">
+ <late-input input="input" workflow-path="/projects/bootcamp/workflow/lateinput" />
+ </late-process>
+</process>
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "requestId": "update\/default\/d6aaa328-6836-4818-a212-515bb43d8b86\n\n",
+ "message": "update\/default\/SampleProcess updated successfully\n\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
+
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityValidate.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityValidate.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityValidate.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/EntityValidate.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,168 @@
+---++ POST api/entities/validate/entity-type
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Validates the submitted entity.
+
+---++ Parameters
+:entity-type can be cluster, feed or process.
+
+---++ Results
+Result of the validation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/validate/cluster
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<cluster xmlns="uri:falcon:cluster:0.1" name="primary-cluster" description="Primary Cluster" colo="west-coast">
+ <interfaces>
+ <interface type="readonly" endpoint="hftp://localhost:50070" version="1.1.1"/>
+ <interface type="write" endpoint="hdfs://localhost:9000" version="1.1.1"/>
+ <interface type="execute" endpoint="localhost:9001" version="1.1.1"/>
+ <interface type="workflow" endpoint="http://localhost:11000/oozie/" version="4.0.0"/>
+ <interface type="messaging" endpoint="tcp://localhost:61616?daemon=true" version="5.4.3"/>
+ </interfaces>
+ <locations>
+ <location name="staging" path="/apps/falcon/staging"/>
+ <location name="temp" path="/tmp"/>
+ <location name="working" path="/apps/falcon/working"/>
+ </locations>
+</cluster>
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "requestId": "dd3f6c3a-a6f1-4c50-97fb-3f9a3f698e10",
+ "message": "Validated successfully (CLUSTER) primary-cluster",
+ "status": "SUCCEEDED"
+}
+</verbatim>
+
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/validate/feed
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Hourly sample input data -->
+
+<feed description="sample input data"
+ name="SampleInput" xmlns="uri:falcon:feed:0.1"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+ <groups>group</groups>
+ <frequency>hours(1)</frequency>
+ <late-arrival cut-off="hours(6)" />
+ <clusters>
+ <cluster name="primary-cluster" type="source">
+ <!--validity start="2009-01-01T00:00Z" end="2099-12-31T00:00Z" timezone="UTC" /-->
+ <validity start="2009-01-01T00:00Z" end="2099-12-31T00:00Z" />
+ <retention limit="months(24)" action="delete" />
+ </cluster>
+ </clusters>
+
+ <locations>
+ <location type="data" path="/projects/bootcamp/data/${YEAR}-${MONTH}-${DAY}-${HOUR}/SampleInput" />
+ <location type="stats" path="/projects/bootcamp/stats/SampleInput" />
+ <location type="meta" path="/projects/bootcamp/meta/SampleInput" />
+ </locations>
+
+ <ACL owner="suser" group="users" permission="0755" />
+
+ <schema location="/none" provider="none" />
+</feed>
+</verbatim>
+
+---+++ Result
+<verbatim>
+{
+ "requestId": "c85b190e-e653-493a-a863-d62de9c2e3b0",
+ "message": "Validated successfully (FEED) SampleInput",
+ "status": "SUCCEEDED"
+}
+</verbatim>
+
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/validate/feed
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Daily sample output data -->
+
+<feed description="sample output data" name="SampleOutput" xmlns="uri:falcon:feed:0.1"
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+ <groups>group</groups>
+ <frequency>hours(1)</frequency>
+ <late-arrival cut-off="hours(6)" />
+ <clusters>
+ <cluster name="primary-cluster" type="source">
+ <!--validity start="2009-01-01T00:00Z" end="2099-12-31T00:00Z" timezone="UTC" /-->
+ <validity start="2009-01-01T00:00Z" end="2099-12-31T00:00Z" />
+ <retention limit="months(24)" action="delete" />
+ </cluster>
+ </clusters>
+ <locations>
+ <location type="data" path="/projects/bootcamp/output/${YEAR}-${MONTH}-${DAY}-${HOUR}/SampleOutput" />
+ <location type="stats" path="/projects/bootcamp/stats/SampleOutput" />
+ <location type="meta" path="/projects/bootcamp/meta/SampleOutput" />
+ </locations>
+ <ACL owner="suser" group="users" permission="0755" />
+ <schema location="/none" provider="none" />
+</feed>
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "requestId": "60781732-460e-4c6c-ba86-a75fae574b05",
+ "message": "Validated successfully (FEED) SampleOutput",
+ "status": "SUCCEEDED"
+}
+</verbatim>
+
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/entities/validate/process
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Daily sample process. Runs at 6th hour every day. Input - last day's hourly data. Generates output for yesterday -->
+<process xmlns="uri:falcon:process:0.1" name="SampleProcess" >
+ <clusters>
+ <cluster name="primary-cluster">
+ <validity start="2012-04-03T06:00Z" end="2022-12-30T00:00Z" />
+ </cluster>
+ </clusters>
+
+ <parallel>1</parallel>
+ <order>FIFO</order>
+ <frequency>hours(1)</frequency>
+
+ <inputs>
+ <input name="input" feed="SampleInput" start="yesterday(0,0)" end="today(-1,0)" />
+ </inputs>
+
+ <outputs>
+ <output name="output" feed="SampleOutput" instance="yesterday(0,0)" />
+ </outputs>
+
+ <properties>
+ <property name="queueName" value="default" />
+ <property name="ssh.host" value="localhost" />
+ <property name="fileTimestamp" value="${coord:formatTime(coord:nominalTime(), 'yyyy-MM-dd')}" />
+ </properties>
+
+ <workflow engine="oozie" path="/examples/apps/aggregator" />
+ <retry policy="exp-backoff" delay="minutes(5)" attempts="3" />
+
+ <late-process policy="exp-backoff" delay="hours(1)">
+ <late-input input="input" workflow-path="/projects/bootcamp/workflow/lateinput" />
+ </late-process>
+</process>
+</verbatim>
+
+---+++ Result
+<verbatim>
+{
+ "requestId": "e4a965c6-c7a2-41d9-ba08-2e77f1c43f57",
+ "message": "Validated successfully (PROCESS) SampleProcess",
+ "status": "SUCCEEDED"
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/FeedInstanceListing.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/FeedInstanceListing.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/FeedInstanceListing.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/FeedInstanceListing.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,45 @@
+---++ GET /api/instance/listing/feed/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get falcon feed instance availability.
+
+---++ Parameters
+ * :entity-name Name of the entity.
+ * start <optional param> Show instances from this date. Date format is yyyy-MM-dd'T'HH:mm'Z'.
+ * By default, it is set to (end - (10 * entityFrequency)).
+ * end <optional param> Show instances up to this date. Date format is yyyy-MM-dd'T'HH:mm'Z'.
+ * Default is set to now.
+ * colo <optional param> Colo on which the query should be run.
+
+---++ Results
+Feed instance availability status
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/instance/listing/feed/SampleFeed?colo=*&start=2012-04-03T07:00Z
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "instances": [
+ {
+ "size": "450231212222",
+ "creationTime": "1236679827365",
+ "cluster": "primary-cluster",
+ "uri": "/data/SampleFeed/2012-04-03",
+ "status": "AVAILABLE",
+ "instance": "2012-04-03T07:00Z"
+ }
+ ],
+ "requestId": "default\/3527038e-8334-4e50-8173-76c4fa430d0b\n",
+ "message": "default\/STATUS\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
+
+
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/FeedLookup.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/FeedLookup.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/FeedLookup.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/FeedLookup.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,36 @@
+---++ GET api/entities/lookup/feed
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+
+---++ Parameters
+ * path path of the instance for which you want to determine the feed. e.g. /data/project1/2014/10/10/23/
+ Path has to be the complete path and can't be a part of it.
+
+---++ Results
+Returns the name of the feed along with the location type(meta/data/stats) and cluster on which the given path belongs to this feed.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/entities/lookup/feed?path=/data/project1/2014/10/10/23
+</verbatim>
+---+++ Result
+{
+ "feeds":
+ [
+ {
+ "feedName": "My-Feed1",
+ "locationType": "DATA",
+ "clusterName": "My-cluster1"
+ },
+ {
+ "feedName": "My-Feed2",
+ "locationType": "DATA",
+ "clusterName": "My-cluster2"
+ }
+ ]
+}
\ No newline at end of file
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/Graph.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/Graph.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/Graph.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/Graph.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,22 @@
+---++ GET api/metadata/lineage/serialize
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Dump the graph.
+
+---++ Parameters
+None.
+
+---++ Results
+Serialize graph to a file configured using *.falcon.graph.serialize.path in Custom startup.properties.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/metadata/lineage/serialize
+</verbatim>
+---+++ Result
+None.
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceKill.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceKill.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceKill.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceKill.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,43 @@
+---++ POST /api/instance/kill/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Kill currently running instance(s) of an entity.
+
+---++ Parameters
+ * :entity-type can either be a feed or a process.
+ * :entity-name is name of the entity.
+ * start is the start time of the instance(s) that you want to refer to
+ * end is the end time of the instance(s) that you want to refer to
+ * lifecycle <optional param> can be Eviction/Replication(default) for feed and Execution(default) for process.
+
+---++ Results
+Result of the kill operation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/instance/kill/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&end=2014-04-03T07:00Z
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "instances": [
+ {
+ "details": "",
+ "endTime": "2013-10-21T15:26:59-07:00",
+ "startTime": "2013-10-21T15:19:57-07:00",
+ "cluster": "primary-cluster",
+ "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W",
+ "status": "KILLED",
+ "instance": "2012-04-03T07:00Z"
+ }
+ ],
+ "requestId": "default\/23b3cfee-ee22-40c0-825d-39c322587d5f\n",
+ "message": "default\/KILL\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceList.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceList.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceList.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceList.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,89 @@
+---++ GET /api/instance/list/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get list of all instances of a given entity.
+
+---++ Parameters
+ * :entity-type Valid options are cluster, feed or process.
+ * :entity-name Name of the entity.
+ * start <optional param> Show instances from this date. Date format is yyyy-MM-dd'T'HH:mm'Z'.
+ * By default, it is set to (end - (10 * entityFrequency)).
+ * end <optional param> Show instances up to this date. Date format is yyyy-MM-dd'T'HH:mm'Z'.
+ * Default is set to now.
+ * colo <optional param> Colo on which the query should be run.
+ * lifecycle <optional param> Valid lifecycles for feed are Eviction/Replication(default) and for process is Execution(default).
+ * filterBy <optional param> Filter results by list of field:value pairs. Example: filterBy=STATUS:RUNNING,CLUSTER:primary-cluster
+ * Supported filter fields are STATUS, CLUSTER, SOURCECLUSTER, STARTEDAFTER.
+ * Query will do an AND among filterBy fields.
+ * orderBy <optional param> Field by which results should be ordered.
+ * Supports ordering by "status","startTime","endTime","cluster".
+ * sortOrder <optional param> Valid options are "asc" and "desc"
+ * offset <optional param> Show results from the offset, used for pagination. Defaults to 0.
+ * numResults <optional param> Number of results to show per request, used for pagination. Only integers > 0 are valid, Default is 10.
+
+---++ Results
+List of instances of given entity.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/instance/list/process/SampleProcess?colo=*&start=2012-04-03T07:00Z
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "instances": [
+ {
+ "details": "",
+ "endTime": "2013-10-21T14:40:26-07:00",
+ "startTime": "2013-10-21T14:39:56-07:00",
+ "cluster": "primary-cluster",
+ "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W",
+ "status": "SUCCEEDED",
+ "instance": "2012-04-03T07:00Z"
+ }
+ ],
+ "requestId": "default\/e15bb378-d09f-4911-9df2-5334a45153d2\n",
+ "message": "default\/STATUS\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
+
+
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/instance/list/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&filterBy=STATUS:SUCCEEDED,CLUSTER:primary-cluster&orderBy=startTime&offset=2&numResults=2
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "instances": [
+ {
+ "details": "",
+ "endTime": "2013-10-21T14:40:26-07:00",
+ "startTime": "2013-10-21T14:39:56-07:00",
+ "cluster": "primary-cluster",
+ "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W",
+ "status": "SUCCEEDED",
+ "instance": "2012-04-03T07:00Z"
+ },
+ {
+ "details": "",
+ "endTime": "2013-10-21T14:42:26-07:00",
+ "startTime": "2013-10-21T14:41:56-07:00",
+ "cluster": "primary-cluster",
+ "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933397-oozie-rgau-W",
+ "status": "SUCCEEDED",
+ "instance": "2012-04-03T08:00Z"
+ },
+ ],
+
+ "requestId": "default\/e15bb378-d09f-4911-9df2-5334a45153d2\n",
+ "message": "default\/STATUS\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
Added: falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceLogs.twiki
URL: http://svn.apache.org/viewvc/falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceLogs.twiki?rev=1690074&view=auto
==============================================================================
--- falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceLogs.twiki (added)
+++ falcon/trunk/releases/0.6.1/src/site/twiki/restapi/InstanceLogs.twiki Thu Jul 9 12:09:09 2015
@@ -0,0 +1,112 @@
+---++ GET /api/instance/logs/:entity-type/:entity-name
+ * <a href="#Description">Description</a>
+ * <a href="#Parameters">Parameters</a>
+ * <a href="#Results">Results</a>
+ * <a href="#Examples">Examples</a>
+
+---++ Description
+Get log of a specific instance of an entity.
+
+---++ Parameters
+ * :entity-type Valid options are cluster, feed or process.
+ * :entity-name Name of the entity.
+ * start <optional param> Show instances from this date. Date format is yyyy-MM-dd'T'HH:mm'Z'.
+ * By default, it is set to (end - (10 * entityFrequency)).
+ * end <optional param> Show instances up to this date. Date format is yyyy-MM-dd'T'HH:mm'Z'.
+ * Default is set to now.
+ * colo <optional param> Colo on which the query should be run.
+ * runId <optional param> Run Id.
+ * lifecycle <optional param> Valid lifecycles for feed are Eviction/Replication(default) and for process is Execution(default).
+ * filterBy <optional param> Filter results by list of field:value pairs. Example: filterBy=STATUS:RUNNING,CLUSTER:primary-cluster
+ * Supported filter fields are STATUS, CLUSTER, SOURCECLUSTER, STARTEDAFTER.
+ * Query will do an AND among filterBy fields.
+ * orderBy <optional param> Field by which results should be ordered.
+ * Supports ordering by "status","startTime","endTime","cluster".
+ * sortOrder <optional param> Valid options are "asc" and "desc"
+ * offset <optional param> Show results from the offset, used for pagination. Defaults to 0.
+ * numResults <optional param> Number of results to show per request, used for pagination. Only integers > 0 are valid, Default is 10.
+
+---++ Results
+Log of specified instance.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/instance/logs/process/SampleProcess?colo=*&start=2012-04-03T07:00Z
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "instances": [
+ {
+ "actions": [
+ {
+ "logFile": "http:\/\/localhost:50070\/data\/apps\/falcon\/staging\/falcon\/workflows\/process\/SampleProcess\/logs\/job-2012-04-03-07-00\/000\/pig_SUCCEEDED.log",
+ "status": "SUCCEEDED",
+ "action": "pig"
+ }
+ ],
+ "details": "",
+ "endTime": "2013-10-21T14:40:26-07:00",
+ "startTime": "2013-10-21T14:39:56-07:00",
+ "cluster": "primary-cluster",
+ "logFile": "http:\/\/localhost:50070\/data\/apps\/falcon\/staging\/falcon\/workflows\/process\/SampleProcess\/logs\/job-2012-04-03-07-00\/000\/oozie.log",
+ "status": "SUCCEEDED",
+ "instance": "2012-04-03T07:00Z"
+ }
+ ],
+ "requestId": "default\/3527038e-8334-4e50-8173-76c4fa430d0b\n",
+ "message": "default\/STATUS\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
+
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/instance/logs/process/SampleProcess?colo=*&start=2012-04-03T07:00Z&filterBy=STATUS:SUCCEEDED,CLUSTER:primary-cluster&orderBy=startTime&offset=2&numResults=2
+</verbatim>
+---+++ Result
+<verbatim>
+{
+ "instances": [
+ {
+ "actions": [
+ {
+ "logFile": "http:\/\/localhost:50070\/data\/apps\/falcon\/staging\/falcon\/workflows\/process\/SampleProcess\/logs\/job-2012-04-03-07-00\/000\/pig_SUCCEEDED.log",
+ "status": "SUCCEEDED",
+ "action": "pig"
+ }
+ ],
+ "details": "",
+ "endTime": "2013-10-21T14:40:26-07:00",
+ "startTime": "2013-10-21T14:39:56-07:00",
+ "cluster": "primary-cluster",
+ "logFile": "http:\/\/localhost:50070\/data\/apps\/falcon\/staging\/falcon\/workflows\/process\/SampleProcess\/logs\/job-2012-04-03-07-00\/000\/oozie.log",
+ "status": "SUCCEEDED",
+ "instance": "2012-04-03T07:00Z"
+ },
+ {
+ "actions": [
+ {
+ "logFile": "http:\/\/localhost:50070\/data\/apps\/falcon\/staging\/falcon\/workflows\/process\/SampleProcess\/logs\/job-2012-04-03-07-00\/001\/pig_SUCCEEDED.log",
+ "status": "SUCCEEDED",
+ "action": "pig"
+ }
+ ],
+ "details": "",
+ "endTime": "2013-10-21T14:42:27-07:00",
+ "startTime": "2013-10-21T14:41:57-07:00",
+ "cluster": "primary-cluster",
+ "logFile": "http:\/\/localhost:50070\/data\/apps\/falcon\/staging\/falcon\/workflows\/process\/SampleProcess\/logs\/job-2012-04-03-07-00\/001\/oozie.log",
+ "status": "SUCCEEDED",
+ "instance": "2012-04-03T08:00Z"
+ }
+ ],
+ "requestId": "default\/3527038e-8334-4e50-8173-76c4fa430d0b\n",
+ "message": "default\/STATUS\n",
+ "status": "SUCCEEDED"
+}
+</verbatim>
+
+
+