You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@falcon.apache.org by ve...@apache.org on 2014/05/01 20:21:02 UTC
[3/4] git commit: FALCON-364 Process specification contains a mistake in the parallel section. Contributed by Jean-Baptiste Onofré
FALCON-364 Process specification contains a mistake in the parallel section. Contributed by Jean-Baptiste Onofré
Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/97f06ac0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/97f06ac0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/97f06ac0
Branch: refs/heads/master
Commit: 97f06ac0c9e87d5ff86b233e666fb448d193d012
Parents: b765d32
Author: Venkatesh Seetharam <ve...@apache.org>
Authored: Thu May 1 11:18:07 2014 -0700
Committer: Venkatesh Seetharam <ve...@apache.org>
Committed: Thu May 1 11:18:07 2014 -0700
----------------------------------------------------------------------
CHANGES.txt | 5 ++++-
docs/src/site/twiki/EntitySpecification.twiki | 17 +++++++++--------
docs/src/site/twiki/FalconCLI.twiki | 14 ++++++++++----
docs/src/site/twiki/FalconDocumentation.twiki | 15 ++++++++-------
4 files changed, 31 insertions(+), 20 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/97f06ac0/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 10cfc0f..363e277 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -116,7 +116,10 @@ Trunk (Unreleased)
FALCON-123 Improve build speeds in falcon. (Srikanth Sundarrajan via Shwetha GS)
BUG FIXES
- FALCON-398 Null pointer Exception on submitting HCat Feed. (Sowmya Ramesh
+ FALCON-364 Process specification contains a mistake in the parallel section
+ (Jean-Baptiste Onofré via Venkatesh Seetharam)
+
+ FALCON-398 Null pointer Exception on submitting HCat Feed (Sowmya Ramesh
via Venkatesh Seetharam)
FALCON-390 falcon HCatProcess tests are failing in secure clusters
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/97f06ac0/docs/src/site/twiki/EntitySpecification.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/EntitySpecification.twiki b/docs/src/site/twiki/EntitySpecification.twiki
index 447a435..d387c9c 100644
--- a/docs/src/site/twiki/EntitySpecification.twiki
+++ b/docs/src/site/twiki/EntitySpecification.twiki
@@ -128,7 +128,7 @@ Location specifies where the feed is available on this cluster. This is an optio
A location tag specifies the type of location like data, meta, stats and the corresponding paths for them.
A feed should at least define the location for type data, which specifies the HDFS path pattern where the feed is generated
periodically. ex: type="data" path="/projects/TrafficHourly/${YEAR}-${MONTH}-${DAY}/traffic"
-The granularity of date pattern in the path should be atleast that of a frequency of a feed.
+The granularity of date pattern in the path should be at least that of a frequency of a feed.
Other location type which are supported are stats and meta paths, if a process references a feed then the meta and stats
paths are available as a property in a process.
@@ -177,7 +177,7 @@ Examples:
A feed can define multiple partitions, if a referenced cluster defines partitions then the number of partitions in feed has to be equal to or more than the cluster partitions.
*Note:* This will only apply for !FileSystem storage but not Table storage as partitions are defined and maintained in
-Hive (Hcatalog) registry.
+Hive (HCatalog) registry.
---+++ Groups
@@ -232,7 +232,7 @@ upto 8 hours then late-arrival's cut-off="hours(8)"
</properties>
</verbatim>
A key-value pair, which are propagated to the workflow engine. "queueName" and "jobPriority" are special properties
-available to user to specify the hadoop job queue and priority, the same value is used by Falcons launcher job.
+available to user to specify the Hadoop job queue and priority, the same value is used by Falcons launcher job.
"timeout" and "parallel" are other special properties which decides replication instance's timeout value while
waiting for the feed instance and parallel decides the concurrent replication instances that can run at any given time. "maxMaps" represents the maximum number of maps used during replication. "mapBandwidthKB" represents the bandwidth in KB/s used by each mapper during replication.
@@ -271,12 +271,13 @@ Syntax:
</verbatim>
---++++ Parallel
-Parallel defines how many instances of the workflow can run concurrently. It should be a positive interger > 0. For example, concurrency of 1 ensures that only one instance of the workflow can run at a time. The next instance will start only after the running instance completes.
+Parallel defines how many instances of the workflow can run concurrently. It should be a positive integer > 0.
+For example, parallel of 1 ensures that only one instance of the workflow can run at a time. The next instance will start only after the running instance completes.
Syntax:
<verbatim>
<process name="[process name]">
...
- <concurrency>[concurrency]</concurrency>
+ <parallel>[parallel]</parallel>
...
</process>
</verbatim>
@@ -563,19 +564,19 @@ Example workflow configuration:
---++++ Properties
The properties are key value pairs that are passed to the workflow. These properties are optional and can be used
in workflow to parameterize the workflow.
-Synatx:
+Syntax:
<verbatim>
<process name="[process name]">
...
<properties>
- <propery name=[key] value=[value]/>
+ <property name=[key] value=[value]/>
...
</properties>
...
</process>
</verbatim>
-queueName and jobPriority are special properites, which when present are used by the Falcon's launcher job, the same property is also availalble in workflow which can be used to propogate to pig or M/R job.
+queueName and jobPriority are special properties, which when present are used by the Falcon's launcher job, the same property is also available in workflow which can be used to propagate to pig or M/R job.
<verbatim>
<property name="queueName" value="hadoopQueue"/>
<property name="jobPriority" value="VERY_HIGH"/>
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/97f06ac0/docs/src/site/twiki/FalconCLI.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/FalconCLI.twiki b/docs/src/site/twiki/FalconCLI.twiki
index 149aaa3..c649cd4 100644
--- a/docs/src/site/twiki/FalconCLI.twiki
+++ b/docs/src/site/twiki/FalconCLI.twiki
@@ -25,7 +25,7 @@ $FALCON_HOME/bin/falcon entity -type process -name sampleProcess -schedule
---+++Suspend
-Suspend on an entity results in suspension of the oozie bundle that was scheduled earlier through the schedule function. No further instances are executed on a suspended entity. Only schedulable entities(process/feed) can be suspended.
+Suspend on an entity results in suspension of the oozie bundle that was scheduled earlier through the schedule function. No further instances are executed on a suspended entity. Only schedule-able entities(process/feed) can be suspended.
Usage:
$FALCON_HOME/bin/falcon entity -type [feed|process] -name <<name>> -suspend
@@ -170,10 +170,16 @@ $FALCON_HOME/bin/falcon instance -type <<feed/process>> -name <<name>> -logs -st
---+++Help
Usage:
-$FALCON_HOME/bin/falcon admin -version
+$FALCON_HOME/bin/falcon admin -help
---+++Version
-Version returns the current verion of Falcon installed.
+Version returns the current version of Falcon installed.
Usage:
-$FALCON_HOME/bin/falcon admin -help
+$FALCON_HOME/bin/falcon admin -version
+
+---+++Status
+
+Status returns the current state of Falcon (running or stopped).
+Usage:
+$FALCON_HOME/bin/falcon admin -status
http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/97f06ac0/docs/src/site/twiki/FalconDocumentation.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/FalconDocumentation.twiki b/docs/src/site/twiki/FalconDocumentation.twiki
index 91bf53f..c78765e 100644
--- a/docs/src/site/twiki/FalconDocumentation.twiki
+++ b/docs/src/site/twiki/FalconDocumentation.twiki
@@ -54,7 +54,7 @@ As the name suggests Falcon Prism splits the request it gets to the Falcon Serve
Stand alone mode is useful when the hadoop jobs and relevant data processing involves only one hadoop cluster. In this mode there is single Falcon server that contacts with oozie to schedule jobs on Hadoop. All the process / feed request like submit, schedule, suspend, kill are sent to this server only. For running in this mode one should use the falcon which has been built for standalone mode, or build using standalone option if using source code.
---+++ Distributed Mode
-Distributed mode is the mode which you might me using most of the time. This is for orgs which have multiple instances of hadoop clusters, and multiple workflow schedulers to handle them. Here we have 2 components: Prism and Server. Both Prism and server have there own setup (runtime and startup properties) and there config locations.
+Distributed mode is the mode which you might me using most of the time. This is for organisations which have multiple instances of hadoop clusters, and multiple workflow schedulers to handle them. Here we have 2 components: Prism and Server. Both Prism and server have there own setup (runtime and startup properties) and there config locations.
In this mode Prism acts as a contact point for Falcon servers. Below are the requests that can be sent to prism and server in this mode:
Prism: submit, schedule, submitAndSchedule, Suspend, Resume, Kill, instance management
@@ -197,7 +197,7 @@ performed in Oozie to realize an update:
* Resume the suspended coordinators
* Schedule as per the new process/feed definition with the start time as "now"
-Update optionally takes effective time as a parameter which is used as the endtime of previously scheduled coordinator.
+Update optionally takes effective time as a parameter which is used as the end time of previously scheduled coordinator.
So, the updated configuration will be effective since the given timestamp.
@@ -474,12 +474,12 @@ feed="raaw-logs16" name="inputData"/>
*Feed xml:*
<verbatim>
-<feed description="clicks log" name="raaw-logs16"....
+<feed description="clicks log" name="raw-logs16"....
</verbatim>
* The time interpretation for corresponding tags indicating the start and end instances for a
-particular input feed in the process xml should lie well within the timespan of the period specified in
+particular input feed in the process xml should lie well within the time span of the period specified in
<validity> tag of the particular feed.
*Example:*
@@ -489,7 +489,7 @@ particular input feed in the process xml should lie well within the timespan of
*Process XML:*
<verbatim>
<input end-instance="now(0,20)" start-instance="now(0,-60)"
- feed="raaw-logs16" name="inputData"/>
+ feed="raw-logs16" name="inputData"/>
</verbatim>
*Feed XML:*
<verbatim>
@@ -597,7 +597,7 @@ The metric logged for an event has the following properties:
1. Action - Name of the event.
2. Dimensions - A list of name/value pairs of various attributes for a given action.
3. Status- Status of an action FAILED/SUCCEEDED.
- 4. Time-taken - Time taken in nano seconds for a given action.
+ 4. Time-taken - Time taken in nanoseconds for a given action.
An example for an event logged for a submit of a new process definition:
@@ -696,11 +696,12 @@ Falcon currently support following ELs:
* 3. *yesterday(hours,minutes)*: As the name suggest EL yesterday picks up feed instances with respect to start of day yesterday. Hours and minutes are added to the 00 hours starting yesterday, Example: yesterday(24,30) will actually correspond to 00:30 am of today, for 2010-01-02T01:30Z this would mean 2010-01-02:00:30 feed.
+ * 7. *lastYear(month,day,hour,minute)*: This is exactly similarly to currentYear in usage> only difference being start reference is taken to start of previous year. For example: lastYear(4,2,2,20) will correspond to feed instance created at 2009-05-03T02:20Z and lastYear(12,2,2,20) will correspond to feed at 2010-01-03T02:20Z.
* 4. *currentMonth(day,hour,minute)*: Current month takes the reference to start of the month with respect to instance start time. One thing to keep in mind is that day is added to the first day of the month. So the value of day is the number of days you want to add to the first day of the month. For example: for instance start time 2010-01-12T01:30Z and El as currentMonth(3,2,40) will correspond to feed created at 2010-01-04T02:40Z and currentMonth(0,0,0) will mean 2010-01-01T00:00Z.
* 5. *lastMonth(day,hour,minute)*: Parameters for lastMonth is same as currentMonth, only difference being the reference is shifted to one month back. For instance start 2010-01-12T01:30Z lastMonth(2,3,30) will correspond to feed instance at 2009-12-03:T03:30Z
- * 6. *currentYear(month,day,hour,minute)*: The month,day,hour, minutes in the pareamter are added with reference to the start of year of instance start time. For our exmple start time 2010-01-02:00:30 reference will go back to 2010-01-01:T00:00Z. Also similar to days, months are added to the 1st month that Jan. So currentYear(0,2,2,20) will mean 2010-01-03T02:20Z while currentYear(11,2,2,20) will mean 2010-12-03T02:20Z
+ * 6. *currentYear(month,day,hour,minute)*: The month,day,hour, minutes in the parameter are added with reference to the start of year of instance start time. For our example start time 2010-01-02:00:30 reference will go back to 2010-01-01:T00:00Z. Also similar to days, months are added to the 1st month that Jan. So currentYear(0,2,2,20) will mean 2010-01-03T02:20Z while currentYear(11,2,2,20) will mean 2010-12-03T02:20Z
* 7. *lastYear(month,day,hour,minute)*: This is exactly similarly to currentYear in usage> only difference being start reference is taken to start of previous year. For example: lastYear(4,2,2,20) will corrospond to feed insatnce created at 2009-05-03T02:20Z and lastYear(12,2,2,20) will corrospond to feed at 2010-01-03T02:20Z.