You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@falcon.apache.org by ba...@apache.org on 2016/08/08 23:16:03 UTC
[29/49] falcon git commit: FALCON-2006 Update documentation on site
for 0.10 release
http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/content/Configuration.html
----------------------------------------------------------------------
diff --git a/content/Configuration.html b/content/Configuration.html
index fdfeb79..f04d8b6 100644
--- a/content/Configuration.html
+++ b/content/Configuration.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2016-03-11
+ | Generated by Apache Maven Doxia at 2016-08-08
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20160311" />
+ <meta name="Date-Revision-yyyymmdd" content="20160808" />
<meta http-equiv="Content-Language" content="en" />
<title>Falcon - Configuring Falcon</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,7 +30,7 @@
<div id="banner">
<div class="pull-left">
<a href="../index.html" id="bannerLeft">
- <img src="images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/>
+ <img src="images/falcon-logo.png" alt="Apache Falcon" width="200px" height="45px"/>
</a>
</div>
<div class="pull-right"> <a href="http://www.apache.org" id="bannerRight">
@@ -58,7 +58,7 @@
- <li id="publishDate" class="pull-right">Last Published: 2016-03-11</li>
+ <li id="publishDate" class="pull-right">Last Published: 2016-08-08</li>
</ul>
</div>
@@ -123,6 +123,13 @@
<li>
+ <a href="MigrationInstructions.html" title="Migrate to 0.10">
+ <i class="none"></i>
+ Migrate to 0.10</a>
+ </li>
+
+ <li>
+
<a href="Operability.html" title="Operability">
<i class="none"></i>
Operability</a>
@@ -141,7 +148,7 @@
<i class="none"></i>
Client (Falcon CLI)</a>
</li>
-
+
<li>
<a href="restapi/ResourceList.html" title="Rest API">
@@ -158,6 +165,13 @@
<li>
+ <a href="Extensions.html" title="Server side Extensions">
+ <i class="none"></i>
+ Server side Extensions</a>
+ </li>
+
+ <li>
+
<a href="Security.html" title="Security">
<i class="none"></i>
Security</a>
@@ -251,6 +265,13 @@
<li>
+ <a href="http://www.apache.org/dyn/closer.lua/falcon/0.10" class="externalLink" title="0.10">
+ <i class="none"></i>
+ 0.10</a>
+ </li>
+
+ <li>
+
<a href="http://www.apache.org/dyn/closer.lua/falcon/0.9" class="externalLink" title="0.9">
<i class="none"></i>
0.9</a>
@@ -315,9 +336,16 @@
<li>
- <a href="0.9/index.html" title="0.9 (Current)">
+ <a href="0.10/index.html" title="0.10 (Current)">
+ <i class="none"></i>
+ 0.10 (Current)</a>
+ </li>
+
+ <li>
+
+ <a href="0.9/index.html" title="0.9">
<i class="none"></i>
- 0.9 (Current)</a>
+ 0.9</a>
</li>
<li>
@@ -462,6 +490,9 @@
# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir.
#export FALCON_EXPANDED_WEBAPP_DIR=
+# Any additional classpath elements to be added to the Falcon server/client classpath
+#export FALCON_EXTRA_CLASS_PATH=
+
</pre></div></div>
<div class="section">
<h3>Advanced Configurations<a name="Advanced_Configurations"></a></h3></div>
@@ -507,15 +538,337 @@ export FALCON_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.re
<li>falcon.embeddedmq.data=<path> - Data path for embedded active mq, default {package dir}/logs/data</li></ul></div>
<div class="section">
<h4>Falcon System Notifications<a name="Falcon_System_Notifications"></a></h4>
-<p>Some Falcon features such as late data handling, retries, metadata service, depend on JMS notifications sent when the Oozie workflow completes. These system notifications are sent as part of Falcon Post Processing action. Given that the post processing action is also a job, it is prone to failures and in case of failures, Falcon is blind to the status of the workflow. To alleviate this problem and make the notifications more reliable, you can enable Oozie's JMS notification feature and disable Falcon post-processing notification by making the following changes:</p>
+<p>Some Falcon features such as late data handling, retries, metadata service, depend on JMS notifications sent when the Oozie workflow completes. Falcon listens to Oozie notification via JMS. You need to enable Oozie JMS notification as explained below. Falcon post processing feature continues to only send user notifications so enabling Oozie JMS notification is important.</p></div>
+<div class="section">
+<h4>Enable Oozie JMS notification<a name="Enable_Oozie_JMS_notification"></a></h4>
+<p></p>
+<ul>
+<li>Please add/change the following properties in oozie-site.xml in the oozie installation dir.</li></ul>
+<div class="source">
+<pre>
+ <property>
+ <name>oozie.jms.producer.connection.properties</name>
+ <value>java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://<activemq-host>:<port></value>
+ </property>
+
+ <property>
+ <name>oozie.service.EventHandlerService.event.listeners</name>
+ <value>org.apache.oozie.jms.JMSJobEventListener</value>
+ </property>
+
+ <property>
+ <name>oozie.service.JMSTopicService.topic.name</name>
+ <value>WORKFLOW=ENTITY.TOPIC,COORDINATOR=ENTITY.TOPIC</value>
+ </property>
+
+ <property>
+ <name>oozie.service.JMSTopicService.topic.prefix</name>
+ <value>FALCON.</value>
+ </property>
+
+ <!-- add org.apache.oozie.service.JMSAccessorService to the other existing services if any -->
+ <property>
+ <name>oozie.services.ext</name>
+ <value>org.apache.oozie.service.JMSAccessorService,org.apache.oozie.service.PartitionDependencyManagerService,org.apache.oozie.service.HCatAccessorService</value>
+ </property>
+
+</pre></div>
+<p></p>
+<ul>
+<li>In falcon startup.properties, set JMS broker url to be the same as the one set in oozie-site.xml property</li></ul>oozie.jms.producer.connection.properties (see above)
+<div class="source">
+<pre>
+ *.broker.url=tcp://<activemq-host>:<port>
+
+</pre></div></div>
+<div class="section">
+<h4>Configuring Oozie for Falcon<a name="Configuring_Oozie_for_Falcon"></a></h4>
+<p>Falcon uses HCatalog for data availability notification when Hive tables are replicated. Make the following configuration changes to Oozie to ensure Hive table replication in Falcon:</p>
+<p></p>
+<ul>
+<li>Stop the Oozie service on all Falcon clusters. Run the following commands on the Oozie host machine.</li></ul>
+<div class="source">
+<pre>
+su - $OOZIE_USER
+
+<oozie-install-dir>/bin/oozie-stop.sh
+
+where $OOZIE_USER is the Oozie user. For example, oozie.
+
+</pre></div>
+<p></p>
+<ul>
+<li>Copy each cluster's hadoop conf directory to a different location. For example, if you have two clusters, copy one to /etc/hadoop/conf-1 and the other to /etc/hadoop/conf-2.</li></ul>
+<p></p>
+<ul>
+<li>For each oozie-site.xml file, modify the oozie.service.HadoopAccessorService.hadoop.configurations property, specifying clusters, the RPC ports of the NameNodes, and HostManagers accordingly. For example, if Falcon connects to three clusters, specify:</li></ul>
+<div class="source">
+<pre>
+
+<property>
+ <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
+ <value>*=/etc/hadoop/conf,$NameNode:$rpcPortNN=$hadoopConfDir1,$ResourceManager1:$rpcPortRM=$hadoopConfDir1,$NameNode2=$hadoopConfDir2,$ResourceManager2:$rpcPortRM=$hadoopConfDir2,$NameNode3 :$rpcPortNN =$hadoopConfDir3,$ResourceManager3 :$rpcPortRM =$hadoopConfDir3</value>
+ <description>
+ Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
+ the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
+ used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
+ the relevant Hadoop *-site.xml files. If the path is relative is looked within
+ the Oozie configuration directory; though the path can be absolute (i.e. to point
+ to Hadoop client conf/ directories in the local filesystem.
+ </description>
+</property>
+
+
+</pre></div>
+<p></p>
+<ul>
+<li>Add the following properties to the /etc/oozie/conf/oozie-site.xml file:</li></ul>
+<div class="source">
+<pre>
+
+<property>
+ <name>oozie.service.ProxyUserService.proxyuser.falcon.hosts</name>
+ <value>*</value>
+</property>
+
+<property>
+ <name>oozie.service.ProxyUserService.proxyuser.falcon.groups</name>
+ <value>*</value>
+</property>
+
+<property>
+ <name>oozie.service.URIHandlerService.uri.handlers</name>
+ <value>org.apache.oozie.dependency.FSURIHandler, org.apache.oozie.dependency.HCatURIHandler</value>
+</property>
+
+<property>
+ <name>oozie.services.ext</name>
+ <value>org.apache.oozie.service.JMSAccessorService, org.apache.oozie.service.PartitionDependencyManagerService,
+ org.apache.oozie.service.HCatAccessorService</value>
+</property>
+
+<!-- Coord EL Functions Properties -->
+
+<property>
+ <name>oozie.service.ELService.ext.functions.coord-job-submit-instances</name>
+ <value>now=org.apache.oozie.extensions.OozieELExtensions#ph1_now_echo,
+ today=org.apache.oozie.extensions.OozieELExtensions#ph1_today_echo,
+ yesterday=org.apache.oozie.extensions.OozieELExtensions#ph1_yesterday_echo,
+ currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_currentMonth_echo,
+ lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph1_lastMonth_echo,
+ currentYear=org.apache.oozie.extensions.OozieELExtensions#ph1_currentYear_echo,
+ lastYear=org.apache.oozie.extensions.OozieELExtensions#ph1_lastYear_echo,
+ formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo,
+ latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo,
+ future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo
+ </value>
+</property>
+
+<property>
+ <name>oozie.service.ELService.ext.functions.coord-action-create-inst</name>
+ <value>now=org.apache.oozie.extensions.OozieELExtensions#ph2_now_inst,
+ today=org.apache.oozie.extensions.OozieELExtensions#ph2_today_inst,
+ yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday_inst,
+ currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth_inst,
+ lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth_inst,
+ currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear_inst,
+ lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear_inst,
+ latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo,
+ future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo,
+ formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime,
+ user=org.apache.oozie.coord.CoordELFunctions#coord_user
+ </value>
+</property>
+
+<property>
+<name>oozie.service.ELService.ext.functions.coord-action-start</name>
+<value>
+now=org.apache.oozie.extensions.OozieELExtensions#ph2_now,
+today=org.apache.oozie.extensions.OozieELExtensions#ph2_today,
+yesterday=org.apache.oozie.extensions.OozieELExtensions#ph2_yesterday,
+currentMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_currentMonth,
+lastMonth=org.apache.oozie.extensions.OozieELExtensions#ph2_lastMonth,
+currentYear=org.apache.oozie.extensions.OozieELExtensions#ph2_currentYear,
+lastYear=org.apache.oozie.extensions.OozieELExtensions#ph2_lastYear,
+latest=org.apache.oozie.coord.CoordELFunctions#ph3_coord_latest,
+future=org.apache.oozie.coord.CoordELFunctions#ph3_coord_future,
+dataIn=org.apache.oozie.extensions.OozieELExtensions#ph3_dataIn,
+instanceTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_nominalTime,
+dateOffset=org.apache.oozie.coord.CoordELFunctions#ph3_coord_dateOffset,
+formatTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_formatTime,
+user=org.apache.oozie.coord.CoordELFunctions#coord_user
+</value>
+</property>
+
+<property>
+ <name>oozie.service.ELService.ext.functions.coord-sla-submit</name>
+ <value>
+ instanceTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_fixed,
+ user=org.apache.oozie.coord.CoordELFunctions#coord_user
+ </value>
+</property>
+
+<property>
+ <name>oozie.service.ELService.ext.functions.coord-sla-create</name>
+ <value>
+ instanceTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_nominalTime,
+ user=org.apache.oozie.coord.CoordELFunctions#coord_user
+ </value>
+</property>
+
+
+</pre></div>
+<p></p>
+<ul>
+<li>Copy the existing Oozie WAR file to <oozie-install-dir>/oozie.war. This will ensure that all existing items in the WAR file are still present after the current update.</li></ul>
+<div class="source">
+<pre>
+su - root
+cp $CATALINA_BASE/webapps/oozie.war <oozie-install-dir>/oozie.war
+
+where $CATALINA_BASE is the path for the Oozie web app. By default, $CATALINA_BASE is: <oozie-install-dir>
+
+</pre></div>
+<p></p>
+<ul>
+<li>Add the Falcon EL extensions to Oozie.</li></ul>
+<p>Copy the extension JAR files provided with the Falcon Server to a temporary directory on the Oozie server. For example, if your standalone Falcon Server is on the same machine as your Oozie server, you can just copy the JAR files.</p>
+<div class="source">
+<pre>
+
+mkdir /tmp/falcon-oozie-jars
+cp <falcon-install-dir>/oozie/ext/falcon-oozie-el-extension-<$version>.jar /tmp/falcon-oozie-jars
+cp /tmp/falcon-oozie-jars/falcon-oozie-el-extension-<$version>.jar <oozie-install-dir>/libext
+
+
+</pre></div>
+<p></p>
+<ul>
+<li>Package the Oozie WAR file as the Oozie user</li></ul>
+<div class="source">
+<pre>
+su - $OOZIE_USER
+cd <oozie-install-dir>/bin
+./oozie-setup.sh prepare-war
+
+Where $OOZIE_USER is the Oozie user. For example, oozie.
+
+</pre></div>
+<p></p>
<ul>
-<li>In Falcon runtime.properties, set *.falcon.jms.notification.enabled to false. This will turn off JMS notification in post-processing.</li>
-<li>Copy notification related properties in oozie/conf/oozie-site.xml to oozie-site.xml of the Oozie installation. Restart Oozie so changes get reflected.</li></ul>
-<p><b>NOTE : If you disable Falcon post-processing JMS notification and not enable Oozie JMS notification, features such as failure retry, late data handling and metadata service will be disabled for all entities on the server.</b></p></div>
+<li>Start the Oozie service on all Falcon clusters. Run these commands on the Oozie host machine.</li></ul>
+<div class="source">
+<pre>
+su - $OOZIE_USER
+<oozie-install-dir>/bin/oozie-start.sh
+
+Where $OOZIE_USER is the Oozie user. For example, oozie.
+
+</pre></div></div>
<div class="section">
<h4>Enabling Falcon Native Scheudler<a name="Enabling_Falcon_Native_Scheudler"></a></h4>
<p><verbatim>$FALCON_HOME/conf/startup.properties</verbatim> before starting the Falcon Server. For details on the same, refer to <a href="./FalconNativeScheduler.html">Falcon Native Scheduler</a></p></div>
<div class="section">
+<h4>Titan GraphDB backend<a name="Titan_GraphDB_backend"></a></h4>
+<p>GraphDB backend needs to be configured to properly start Falcon server. You can either choose to use 5.0.73 version of berkeleydb (the default for Falcon for the last few releases) or 1.1.x or later version HBase as the backend database. Falcon in its release distributions will have the titan storage plugins for both BerkeleyDB and HBase.</p>
+<p>----++++Using BerkeleyDB backend Falcon distributions may not package berkeley db artifacts (je-5.0.73.jar) based on build profiles. If Berkeley DB is not packaged, you can download the Berkeley DB jar file from the URL:</p>
+<div class="source">
+<pre>http://download.oracle.com/otn/berkeley-db/je-5.0.73.zip
+</pre></div>
+<p>The following properties describe an example berkeley db graph storage backend that can be specified in the configuration file</p>
+<div class="source">
+<pre>$FALCON_HOME/conf/startup.properties
+</pre></div>
+<div class="source">
+<pre>
+# Graph Storage
+*.falcon.graph.storage.directory=${user.dir}/target/graphdb
+*.falcon.graph.storage.backend=berkeleyje
+*.falcon.graph.serialize.path=${user.dir}/target/graphdb
+
+</pre></div></div>
+<div class="section">
+<h5>Using HBase backend<a name="Using_HBase_backend"></a></h5>
+<p><verbatim>hbase-site.xml</verbatim> is provided, which can be used to start the standalone mode HBase enviornment for development/testing purposes.</p></div>
+<div class="section">
+<h5>Basic configuration<a name="Basic_configuration"></a></h5>
+<div class="source">
+<pre>
+##### Falcon startup.properties
+*.falcon.graph.storage.backend=hbase
+#For standalone mode , specify localhost
+#for distributed mode, specify zookeeper quorum here - For more information refer http://s3.thinkaurelius.com/docs/titan/current/hbase.html#_remote_server_mode_2
+*.falcon.graph.storage.hostname=<ZooKeeper Quorum>
+
+</pre></div>
+<p><verbatim>FALCON_EXTRA_CLASS_PATH</verbatim> in <verbatim>$FALCON_HOME/bin/falcon-env.sh</verbatim>. Additionally the correct hbase client libraries need to be added. For example,</p>
+<div class="source">
+<pre>
+export FALCON_EXTRA_CLASS_PATH=`${HBASE_HOME}/bin/hbase classpath`
+
+</pre></div>
+<p>Table name We recommend that in the startup config the tablename for titan storage be named <verbatim>falcon_titan<verbatim> so that multiple applications using Titan can share the same HBase cluster. This can be set by specifying the tablename using the startup property given below. The default value is shown.</p>
+<div class="source">
+<pre>
+*.falcon.graph.storage.hbase.table=falcon_titan
+
+</pre></div></div>
+<div class="section">
+<h5>Starting standalone HBase for testing<a name="Starting_standalone_HBase_for_testing"></a></h5>
+<p>HBase can be started in stand alone mode for testing as a backend for Titan. The following steps outline the config changes required:</p>
+<div class="source">
+<pre>
+1. Build Falcon as below to package hbase binaries
+ $ export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m" && mvn clean assembly:assembly -Ppackage-standalone-hbase
+2. Configure HBase
+ a. When falcon tar file is expanded, HBase binaries are under ${FALCON_HOME}/hbase
+ b. Copy ${FALCON_HOME}/conf/hbase-site.xml.template into hbase conf dir in ${FALCON_HOME}/hbase/conf/hbase-site.xml
+ c. Set {hbase_home} property to point to a local dir
+ d. Standalone HBase starts zookeeper on the default port (2181). This port can be changed by adding the following to hbase-site.xml
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2223</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ e. set JAVA_HOME to point to Java 1.7 or above
+ f. Start hbase as ${FALCON_HOME}/hbase/bin/start-hbase.sh
+3. Configure Falcon
+ a. In ${FALCON_HOME}/conf/startup.properties, uncomment the following to enable HBase as the backend
+ *.falcon.graph.storage.backend=hbase
+ ### specify the zookeeper host and port name with which standalone hbase is started (see step 2)
+ ### by default, it will be localhost and port 2181
+ *.falcon.graph.storage.hostname=<zookeeper-host-name>:<zookeeper-host-port>
+ *.falcon.graph.serialize.path=${user.dir}/target/graphdb
+ *.falcon.graph.storage.hbase.table=falcon_titan
+ *.falcon.graph.storage.transactions=false
+4. Add HBase jars to Falcon classpath in ${FALCON_HOME}/conf/falcon-env.sh as:
+ FALCON_EXTRA_CLASS_PATH=`${FALCON_HOME}/hbase/bin/hbase classpath`
+5. Set the following in ${FALCON_HOME}/conf/startup.properties to disable SSL if needed
+ *.falcon.enableTLS=false
+6. Start Falcon
+
+</pre></div></div>
+<div class="section">
+<h5>Permissions<a name="Permissions"></a></h5>
+<p><verbatim>falcon</verbatim> user for the <verbatim>falcon_titan</verbatim> table (or whateven tablename was specified for the property <verbatim>*.falcon.graph.storage.hbase.table</verbatim></p>
+<p><verbatim>falcon_titan</verbatim>.</p>
+<p>Without Ranger, HBase shell can be used to set the permissions.</p>
+<div class="source">
+<pre>
+ su hbase
+ kinit -k -t <hbase keytab> <hbase principal>
+ echo "grant 'falcon', 'RWXCA', 'falcon_titan'" | hbase shell
+
+</pre></div></div>
+<div class="section">
+<h5>Advanced configuration<a name="Advanced_configuration"></a></h5>
+<p><verbatim>$FALCON_HOME/conf/startup.properties</verbatim>, by prefixing the Titan property with <verbatim>*.falcon.graph</verbatim> prefix.</p>
+<p><verbatim>http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html#_storage</verbatim> for generic storage properties, <verbatim>http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html#_storage_berkeleydb</verbatim> for berkeley db properties and <verbatim><a class="externalLink" href="http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html#_storage_hbase">http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html#_storage_hbase</a></verbatim> for hbase storage backend properties.</p></div>
+<div class="section">
<h4>Adding Extension Libraries<a name="Adding_Extension_Libraries"></a></h4>
<p>Library extensions allows users to add custom libraries to entity lifecycles such as feed retention, feed replication and process execution. This is useful for usecases such as adding filesystem extensions. To enable this, add the following configs to startup.properties: *.libext.paths=<paths to be added to all entity lifecycles></p>
<p>*.libext.feed.paths=<paths to be added to all feed lifecycles></p>
http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/content/DataReplicationAzure.html
----------------------------------------------------------------------
diff --git a/content/DataReplicationAzure.html b/content/DataReplicationAzure.html
new file mode 100644
index 0000000..4578449
--- /dev/null
+++ b/content/DataReplicationAzure.html
@@ -0,0 +1,523 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2016-08-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20160808" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Falcon - Data Replication between On-premise Hadoop Clusters and Azure Cloud</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="../index.html" id="bannerLeft">
+ <img src="images/falcon-logo.png" alt="Apache Falcon" width="200px" height="45px"/>
+ </a>
+ </div>
+ <div class="pull-right"> <a href="http://www.apache.org" id="bannerRight">
+ <img src="images/apache-feather-tm.gif" alt="Falcon" height="45px"/>
+ </a>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="index.html" title="Falcon">
+ Falcon</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Data Replication between On-premise Hadoop Clusters and Azure Cloud</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2016-08-08</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Falcon</li>
+
+ <li>
+
+ <a href="index.html" title="About">
+ <i class="none"></i>
+ About</a>
+ </li>
+
+ <li>
+
+ <a href="slides/falcon-overview.html" title="Overview">
+ <i class="none"></i>
+ Overview</a>
+ </li>
+
+ <li>
+
+ <a href="slides/falcon-user-guide.html" title="User Guide">
+ <i class="none"></i>
+ User Guide</a>
+ </li>
+
+ <li>
+
+ <a href="GettingStarted.html" title="Getting Started">
+ <i class="none"></i>
+ Getting Started</a>
+ </li>
+
+ <li>
+
+ <a href="FalconDocumentation.html" title="Architecture">
+ <i class="none"></i>
+ Architecture</a>
+ </li>
+
+ <li>
+
+ <a href="InstallationSteps.html" title="Installation">
+ <i class="none"></i>
+ Installation</a>
+ </li>
+
+ <li>
+
+ <a href="OnBoarding.html" title="On Boarding">
+ <i class="none"></i>
+ On Boarding</a>
+ </li>
+
+ <li>
+
+ <a href="MigrationInstructions.html" title="Migrate to 0.10">
+ <i class="none"></i>
+ Migrate to 0.10</a>
+ </li>
+
+ <li>
+
+ <a href="Operability.html" title="Operability">
+ <i class="none"></i>
+ Operability</a>
+ </li>
+
+ <li>
+
+ <a href="EntitySpecification.html" title="Entity Specification">
+ <i class="none"></i>
+ Entity Specification</a>
+ </li>
+
+ <li>
+
+ <a href="falconcli/FalconCLI.html" title="Client (Falcon CLI)">
+ <i class="none"></i>
+ Client (Falcon CLI)</a>
+ </li>
+
+ <li>
+
+ <a href="restapi/ResourceList.html" title="Rest API">
+ <i class="icon-chevron-right"></i>
+ Rest API</a>
+ </li>
+
+ <li>
+
+ <a href="HiveIntegration.html" title="Hive Integration">
+ <i class="none"></i>
+ Hive Integration</a>
+ </li>
+
+ <li>
+
+ <a href="Extensions.html" title="Server side Extensions">
+ <i class="none"></i>
+ Server side Extensions</a>
+ </li>
+
+ <li>
+
+ <a href="Security.html" title="Security">
+ <i class="none"></i>
+ Security</a>
+ </li>
+ <li class="nav-header">Project Information</li>
+
+ <li>
+
+ <a href="project-info.html" title="Summary">
+ <i class="none"></i>
+ Summary</a>
+ </li>
+
+ <li>
+
+ <a href="mail-lists.html" title="Mailing Lists">
+ <i class="none"></i>
+ Mailing Lists</a>
+ </li>
+
+ <li>
+
+ <a href="http://webchat.freenode.net?channels=apachefalcon&uio=d4" class="externalLink" title="IRC">
+ <i class="none"></i>
+ IRC</a>
+ </li>
+
+ <li>
+
+ <a href="team-list.html" title="Team">
+ <i class="none"></i>
+ Team</a>
+ </li>
+
+ <li>
+
+ <a href="issue-tracking.html" title="Issue Tracking">
+ <i class="none"></i>
+ Issue Tracking</a>
+ </li>
+
+ <li>
+
+ <a href="source-repository.html" title="Source Repository">
+ <i class="none"></i>
+ Source Repository</a>
+ </li>
+
+ <li>
+
+ <a href="https://cwiki.apache.org/confluence/display/FALCON/Index" class="externalLink" title="Wiki">
+ <i class="none"></i>
+ Wiki</a>
+ </li>
+
+ <li>
+
+ <a href="license.html" title="License">
+ <i class="none"></i>
+ License</a>
+ </li>
+
+ <li>
+
+ <a href="https://cwiki.apache.org/confluence/display/FALCON/News" class="externalLink" title="News">
+ <i class="none"></i>
+ News</a>
+ </li>
+
+ <li>
+
+ <a href="https://cwiki.apache.org/confluence/display/FALCON/PoweredBy" class="externalLink" title="Powered by">
+ <i class="none"></i>
+ Powered by</a>
+ </li>
+
+ <li>
+
+ <a href="https://cwiki.apache.org/confluence/display/FALCON/Acknowledgements" class="externalLink" title="Acknowledgements">
+ <i class="none"></i>
+ Acknowledgements</a>
+ </li>
+
+ <li>
+
+ <a href="http://blogs.apache.org/falcon/" class="externalLink" title="Blog">
+ <i class="none"></i>
+ Blog</a>
+ </li>
+ <li class="nav-header">Releases</li>
+
+ <li>
+
+ <a href="http://www.apache.org/dyn/closer.lua/falcon/0.10" class="externalLink" title="0.10">
+ <i class="none"></i>
+ 0.10</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.apache.org/dyn/closer.lua/falcon/0.9" class="externalLink" title="0.9">
+ <i class="none"></i>
+ 0.9</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.apache.org/dyn/closer.lua/falcon/0.8" class="externalLink" title="0.8">
+ <i class="none"></i>
+ 0.8</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.apache.org/dyn/closer.lua/falcon/0.7" class="externalLink" title="0.7">
+ <i class="none"></i>
+ 0.7</a>
+ </li>
+
+ <li>
+
+ <a href="http://archive.apache.org/dist/falcon/0.6.1" class="externalLink" title="0.6.1">
+ <i class="none"></i>
+ 0.6.1</a>
+ </li>
+
+ <li>
+
+ <a href="http://archive.apache.org/dist/incubator/falcon/0.6-incubating" class="externalLink" title="0.6-incubating">
+ <i class="none"></i>
+ 0.6-incubating</a>
+ </li>
+
+ <li>
+
+ <a href="http://archive.apache.org/dist/incubator/falcon/0.5-incubating" class="externalLink" title="0.5-incubating">
+ <i class="none"></i>
+ 0.5-incubating</a>
+ </li>
+
+ <li>
+
+ <a href="http://archive.apache.org/dist/incubator/falcon/0.4-incubating" class="externalLink" title="0.4-incubating">
+ <i class="none"></i>
+ 0.4-incubating</a>
+ </li>
+
+ <li>
+
+ <a href="http://archive.apache.org/dist/incubator/falcon/0.3-incubating" class="externalLink" title="0.3-incubating">
+ <i class="none"></i>
+ 0.3-incubating</a>
+ </li>
+
+ <li>
+
+ <a href="https://cwiki.apache.org/confluence/display/FALCON/Roadmap" class="externalLink" title="Coming soon">
+ <i class="none"></i>
+ Coming soon</a>
+ </li>
+ <li class="nav-header">Documentation</li>
+
+ <li>
+
+ <a href="0.10/index.html" title="0.10 (Current)">
+ <i class="none"></i>
+ 0.10 (Current)</a>
+ </li>
+
+ <li>
+
+ <a href="0.9/index.html" title="0.9">
+ <i class="none"></i>
+ 0.9</a>
+ </li>
+
+ <li>
+
+ <a href="0.8/index.html" title="0.8">
+ <i class="none"></i>
+ 0.8</a>
+ </li>
+
+ <li>
+
+ <a href="0.7/index.html" title="0.7">
+ <i class="none"></i>
+ 0.7</a>
+ </li>
+
+ <li>
+
+ <a href="0.6.1/index.html" title="0.6.1">
+ <i class="none"></i>
+ 0.6.1</a>
+ </li>
+
+ <li>
+
+ <a href="0.6-incubating/index.html" title="0.6-incubating">
+ <i class="none"></i>
+ 0.6-incubating</a>
+ </li>
+
+ <li>
+
+ <a href="0.5-incubating/index.html" title="0.5-incubating">
+ <i class="none"></i>
+ 0.5-incubating</a>
+ </li>
+
+ <li>
+
+ <a href="0.4-incubating/index.html" title="0.4-incubating">
+ <i class="none"></i>
+ 0.4-incubating</a>
+ </li>
+
+ <li>
+
+ <a href="0.3-incubating/index.html" title="0.3-incubating">
+ <i class="none"></i>
+ 0.3-incubating</a>
+ </li>
+ <li class="nav-header">ASF</li>
+
+ <li>
+
+ <a href="http://www.apache.org/foundation/how-it-works.html" class="externalLink" title="How Apache Works">
+ <i class="none"></i>
+ How Apache Works</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.apache.org/foundation/" class="externalLink" title="Foundation">
+ <i class="none"></i>
+ Foundation</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.apache.org/foundation/sponsorship.html" class="externalLink" title="Sponsoring Apache">
+ <i class="none"></i>
+ Sponsoring Apache</a>
+ </li>
+
+ <li>
+
+ <a href="http://www.apache.org/foundation/thanks.html" class="externalLink" title="Thanks">
+ <i class="none"></i>
+ Thanks</a>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <div class="section">
+<h2>Data Replication between On-premise Hadoop Clusters and Azure Cloud<a name="Data_Replication_between_On-premise_Hadoop_Clusters_and_Azure_Cloud"></a></h2></div>
+<div class="section">
+<h3>Overview<a name="Overview"></a></h3>
+<p>Falcon provides an easy way to replicate data between on-premise Hadoop clusters and Azure cloud. With this feature, users would be able to build a hybrid data pipeline, e.g. processing sensitive data on-premises for privacy and compliance reasons while leverage cloud for elastic scale and online services (e.g. Azure machine learning) with non-sensitive data.</p></div>
+<div class="section">
+<h3>Use Case<a name="Use_Case"></a></h3>
+<p>1. Copy data from on-premise Hadoop clusters to Azure cloud 2. Copy data from Azure cloud to on-premise Hadoop clusters 3. Copy data within Azure cloud (i.e. from one Azure location to another).</p></div>
+<div class="section">
+<h3>Usage<a name="Usage"></a></h3></div>
+<div class="section">
+<h4>Set Up Azure Blob Credentials<a name="Set_Up_Azure_Blob_Credentials"></a></h4>
+<p>To move data to/from Azure blobs, we need to add Azure blob credentials in HDFS. This can be done by adding the credential property through Ambari HDFS configs, and HDFS needs to be restarted after adding the credential. You can also add the credential property to core-site.xml directly, but make sure you restart HDFS from command line instead of Ambari. Otherwise, Ambari will take the previous HDFS configuration without your Azure blob credentials.</p>
+<div class="source">
+<pre>
+<property>
+ <name>fs.azure.account.key.{AZURE_BLOB_ACCOUNT_NAME}.blob.core.windows.net</name>
+ <value>{AZURE_BLOB_ACCOUNT_KEY}</value>
+</property>
+
+</pre></div>
+<p>To verify you set up Azure credential properly, you can check if you are able to access Azure blob through HDFS, e.g.</p>
+<div class="source">
+<pre>
+hadoop fs ­ls wasb://{AZURE_BLOB_CONTAINER}@{AZURE_BLOB_ACCOUNT_NAME}.blob.core.windows.net/
+
+</pre></div></div>
+<div class="section">
+<h4>Replication Feed<a name="Replication_Feed"></a></h4>
+<p><a href="./EntitySpecification.html">Falcon replication feed</a> can be used for data replication to/from Azure cloud. You can specify WASB (i.e. Windows Azure Storage Blob) url in source or target locations. See below for an example of data replication from Hadoop cluster to Azure blob. Note that the clusters for the source and the target need to be different. Analogously, if you want to copy data from Azure blob, you can add Azure blob location to the source.</p>
+<div class="source">
+<pre>
+<?xml version="1.0" encoding="UTF-8"?>
+<feed name="AzureReplication" xmlns="uri:falcon:feed:0.1">
+ <frequency>months(1)</frequency>
+ <clusters>
+ <cluster name="SampleCluster1" type="source">
+ <validity start="2010-06-01T00:00Z" end="2010-06-02T00:00Z"/>
+ <retention limit="days(90)" action="delete"/>
+ </cluster>
+ <cluster name="SampleCluster2" type="target">
+ <validity start="2010-06-01T00:00Z" end="2010-06-02T00:00Z"/>
+ <retention limit="days(90)" action="delete"/>
+ <locations>
+ <location type="data" path="wasb://replication-test@mystorage.blob.core.windows.net/replicated-${YEAR}-${MONTH}"/>
+ </locations>
+ </cluster>
+ </clusters>
+ <locations>
+ <location type="data" path="/apps/falcon/demo/data-${YEAR}-${MONTH}" />
+ </locations>
+ <ACL owner="ambari-qa" group="users" permission="0755"/>
+ <schema location="hcat" provider="hcat"/>
+</feed>
+
+</pre></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2013-2016
+ <a href="http://www.apache.org">Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/content/Distributed-mode.html
----------------------------------------------------------------------
diff --git a/content/Distributed-mode.html b/content/Distributed-mode.html
index 08ba627..3f8416f 100644
--- a/content/Distributed-mode.html
+++ b/content/Distributed-mode.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2016-03-11
+ | Generated by Apache Maven Doxia at 2016-08-08
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20160311" />
+ <meta name="Date-Revision-yyyymmdd" content="20160808" />
<meta http-equiv="Content-Language" content="en" />
<title>Falcon - Distributed Mode</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,7 +30,7 @@
<div id="banner">
<div class="pull-left">
<a href="../index.html" id="bannerLeft">
- <img src="images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/>
+ <img src="images/falcon-logo.png" alt="Apache Falcon" width="200px" height="45px"/>
</a>
</div>
<div class="pull-right"> <a href="http://www.apache.org" id="bannerRight">
@@ -58,7 +58,7 @@
- <li id="publishDate" class="pull-right">Last Published: 2016-03-11</li>
+ <li id="publishDate" class="pull-right">Last Published: 2016-08-08</li>
</ul>
</div>
@@ -123,6 +123,13 @@
<li>
+ <a href="MigrationInstructions.html" title="Migrate to 0.10">
+ <i class="none"></i>
+ Migrate to 0.10</a>
+ </li>
+
+ <li>
+
<a href="Operability.html" title="Operability">
<i class="none"></i>
Operability</a>
@@ -141,7 +148,7 @@
<i class="none"></i>
Client (Falcon CLI)</a>
</li>
-
+
<li>
<a href="restapi/ResourceList.html" title="Rest API">
@@ -158,6 +165,13 @@
<li>
+ <a href="Extensions.html" title="Server side Extensions">
+ <i class="none"></i>
+ Server side Extensions</a>
+ </li>
+
+ <li>
+
<a href="Security.html" title="Security">
<i class="none"></i>
Security</a>
@@ -251,6 +265,13 @@
<li>
+ <a href="http://www.apache.org/dyn/closer.lua/falcon/0.10" class="externalLink" title="0.10">
+ <i class="none"></i>
+ 0.10</a>
+ </li>
+
+ <li>
+
<a href="http://www.apache.org/dyn/closer.lua/falcon/0.9" class="externalLink" title="0.9">
<i class="none"></i>
0.9</a>
@@ -315,9 +336,16 @@
<li>
- <a href="0.9/index.html" title="0.9 (Current)">
+ <a href="0.10/index.html" title="0.10 (Current)">
+ <i class="none"></i>
+ 0.10 (Current)</a>
+ </li>
+
+ <li>
+
+ <a href="0.9/index.html" title="0.9">
<i class="none"></i>
- 0.9 (Current)</a>
+ 0.9</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/content/Embedded-mode.html
----------------------------------------------------------------------
diff --git a/content/Embedded-mode.html b/content/Embedded-mode.html
index 65e6b6a..0c1cf2e 100644
--- a/content/Embedded-mode.html
+++ b/content/Embedded-mode.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2016-03-11
+ | Generated by Apache Maven Doxia at 2016-08-08
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20160311" />
+ <meta name="Date-Revision-yyyymmdd" content="20160808" />
<meta http-equiv="Content-Language" content="en" />
<title>Falcon - Embedded Mode</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,7 +30,7 @@
<div id="banner">
<div class="pull-left">
<a href="../index.html" id="bannerLeft">
- <img src="images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/>
+ <img src="images/falcon-logo.png" alt="Apache Falcon" width="200px" height="45px"/>
</a>
</div>
<div class="pull-right"> <a href="http://www.apache.org" id="bannerRight">
@@ -58,7 +58,7 @@
- <li id="publishDate" class="pull-right">Last Published: 2016-03-11</li>
+ <li id="publishDate" class="pull-right">Last Published: 2016-08-08</li>
</ul>
</div>
@@ -123,6 +123,13 @@
<li>
+ <a href="MigrationInstructions.html" title="Migrate to 0.10">
+ <i class="none"></i>
+ Migrate to 0.10</a>
+ </li>
+
+ <li>
+
<a href="Operability.html" title="Operability">
<i class="none"></i>
Operability</a>
@@ -141,7 +148,7 @@
<i class="none"></i>
Client (Falcon CLI)</a>
</li>
-
+
<li>
<a href="restapi/ResourceList.html" title="Rest API">
@@ -158,6 +165,13 @@
<li>
+ <a href="Extensions.html" title="Server side Extensions">
+ <i class="none"></i>
+ Server side Extensions</a>
+ </li>
+
+ <li>
+
<a href="Security.html" title="Security">
<i class="none"></i>
Security</a>
@@ -251,6 +265,13 @@
<li>
+ <a href="http://www.apache.org/dyn/closer.lua/falcon/0.10" class="externalLink" title="0.10">
+ <i class="none"></i>
+ 0.10</a>
+ </li>
+
+ <li>
+
<a href="http://www.apache.org/dyn/closer.lua/falcon/0.9" class="externalLink" title="0.9">
<i class="none"></i>
0.9</a>
@@ -315,9 +336,16 @@
<li>
- <a href="0.9/index.html" title="0.9 (Current)">
+ <a href="0.10/index.html" title="0.10 (Current)">
+ <i class="none"></i>
+ 0.10 (Current)</a>
+ </li>
+
+ <li>
+
+ <a href="0.9/index.html" title="0.9">
<i class="none"></i>
- 0.9 (Current)</a>
+ 0.9</a>
</li>
<li>
@@ -583,6 +611,7 @@ $hadoop fs -put examples/app /
<pre>
$bin/falcon entity -submitAndSchedule -type process -file examples/entity/filesystem/oozie-mr-process.xml
$bin/falcon entity -submitAndSchedule -type process -file examples/entity/filesystem/pig-process.xml
+$bin/falcon entity -submitAndSchedule -type process -file examples/entity/spark/spark-process.xml
</pre></div>
<p>Generate input data:</p>
@@ -597,7 +626,7 @@ $examples/data/generate.sh <<hdfs endpoint>>
$bin/falcon instance -status -type process -name oozie-mr-process -start 2013-11-15T00:05Z -end 2013-11-15T01:00Z
</pre></div>
-<p>HCat based example entities are in examples/entity/hcat.</p></div>
+<p>HCat based example entities are in examples/entity/hcat. Spark based example entities are in examples/entity/spark.</p></div>
<div class="section">
<h4>Stopping Falcon Server<a name="Stopping_Falcon_Server"></a></h4>
<div class="source">
http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/content/EntitySpecification.html
----------------------------------------------------------------------
diff --git a/content/EntitySpecification.html b/content/EntitySpecification.html
index 2fff61f..6847a48 100644
--- a/content/EntitySpecification.html
+++ b/content/EntitySpecification.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2016-03-11
+ | Generated by Apache Maven Doxia at 2016-08-08
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20160311" />
+ <meta name="Date-Revision-yyyymmdd" content="20160808" />
<meta http-equiv="Content-Language" content="en" />
<title>Falcon - Contents</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,7 +30,7 @@
<div id="banner">
<div class="pull-left">
<a href="../index.html" id="bannerLeft">
- <img src="images/falcon-logo.png" alt="Falcon" width="200px" height="45px"/>
+ <img src="images/falcon-logo.png" alt="Apache Falcon" width="200px" height="45px"/>
</a>
</div>
<div class="pull-right"> <a href="http://www.apache.org" id="bannerRight">
@@ -58,7 +58,7 @@
- <li id="publishDate" class="pull-right">Last Published: 2016-03-11</li>
+ <li id="publishDate" class="pull-right">Last Published: 2016-08-08</li>
</ul>
</div>
@@ -123,6 +123,13 @@
<li>
+ <a href="MigrationInstructions.html" title="Migrate to 0.10">
+ <i class="none"></i>
+ Migrate to 0.10</a>
+ </li>
+
+ <li>
+
<a href="Operability.html" title="Operability">
<i class="none"></i>
Operability</a>
@@ -139,7 +146,7 @@
<i class="none"></i>
Client (Falcon CLI)</a>
</li>
-
+
<li>
<a href="restapi/ResourceList.html" title="Rest API">
@@ -156,6 +163,13 @@
<li>
+ <a href="Extensions.html" title="Server side Extensions">
+ <i class="none"></i>
+ Server side Extensions</a>
+ </li>
+
+ <li>
+
<a href="Security.html" title="Security">
<i class="none"></i>
Security</a>
@@ -249,6 +263,13 @@
<li>
+ <a href="http://www.apache.org/dyn/closer.lua/falcon/0.10" class="externalLink" title="0.10">
+ <i class="none"></i>
+ 0.10</a>
+ </li>
+
+ <li>
+
<a href="http://www.apache.org/dyn/closer.lua/falcon/0.9" class="externalLink" title="0.9">
<i class="none"></i>
0.9</a>
@@ -313,9 +334,16 @@
<li>
- <a href="0.9/index.html" title="0.9 (Current)">
+ <a href="0.10/index.html" title="0.10 (Current)">
+ <i class="none"></i>
+ 0.10 (Current)</a>
+ </li>
+
+ <li>
+
+ <a href="0.9/index.html" title="0.9">
<i class="none"></i>
- 0.9 (Current)</a>
+ 0.9</a>
</li>
<li>
@@ -465,7 +493,12 @@
<interface type="registry" endpoint="thrift://localhost:9083" version="0.11.0" />
</pre></div>
-<p>A registry interface specifies the interface for metadata catalog, such as Hive Metastore (or HCatalog). Falcon uses this interface to register/de-register partitions for a given database and table. Also, uses this information to schedule data availability events based on partitions in the workflow engine. Although Hive metastore supports both RPC and HTTP, Falcon comes with an implementation for RPC over thrift.</p>
+<p>A registry interface specifies the interface for metadata catalog, such as Hive Metastore (or HCatalog). Falcon uses this interface to register/de-register partitions for a given database and table. Also, uses this information to schedule data availability events based on partitions in the workflow engine. Although Hive metastore supports both RPC and HTTP, Falcon comes with an implementation for RPC over thrift. For Hive HA mode, make sure the uris are separated with comma and you only add protocol "thrift://" at the beginning. See below for an example of Hive HA mode:</p>
+<div class="source">
+<pre>
+<interface type="registry" endpoint="thrift://c6402.ambari.apache.org:9083,c6403.ambari.apache.org:9083" version="0.11.0" />
+
+</pre></div>
<div class="source">
<pre>
<interface type="messaging" endpoint="tcp://localhost:61616?daemon=true" version="5.4.6" />
@@ -1017,7 +1050,7 @@ Sqoop options can be specified via the arguments.
</pre></div></div>
<div class="section">
<h4>Optional Inputs<a name="Optional_Inputs"></a></h4>
-<p>User can mention one or more inputs as optional inputs. In such cases the job does not wait on those inputs which are mentioned as optional. If they are present it considers them otherwise continue with the compulsory ones. Example:</p>
+<p>User can mention one or more inputs as optional inputs. In such cases the job does not wait on those inputs which are mentioned as optional. If they are present it considers them otherwise continues with the mandatory ones. If some instances of the optional feed are present for the given data window, those are considered and passed on to the process. While checking for presence of an feed instance, Falcon looks for <b><i>availabilityFlag</i></b> in the directory, if specified in the feed definition. If no <b><i>availabilityFlag</i></b> is specified, presence of the instance directory is treated as indication of availability of data. Example:</p>
<div class="source">
<pre>
<feed name="feed1">
@@ -1155,10 +1188,10 @@ Sqoop options can be specified via the arguments.
<h4>Workflow<a name="Workflow"></a></h4>
<p>The workflow defines the workflow engine that should be used and the path to the workflow on hdfs. Libraries required can be specified using lib attribute in the workflow element and will be comma separated HDFS paths. The workflow definition on hdfs contains the actual job that should run and it should confirm to the workflow specification of the engine specified. The libraries required by the workflow should be in lib folder inside the workflow path.</p>
<p>The properties defined in the cluster and cluster properties(nameNode and jobTracker) will also be available for the workflow.</p>
-<p>There are 3 engines supported today.</p></div>
+<p>There are 4 engines supported today.</p></div>
<div class="section">
<h5>Oozie<a name="Oozie"></a></h5>
-<p>As part of oozie workflow engine support, users can embed a oozie workflow. Refer to oozie <a class="externalLink" href="http://oozie.apache.org/docs/4.0.1/DG_Overview.html">workflow overview</a> and <a class="externalLink" href="http://oozie.apache.org/docs/4.0.1/WorkflowFunctionalSpec.html">workflow specification</a> for details.</p>
+<p>As part of oozie workflow engine support, users can embed a oozie workflow. Refer to oozie <a class="externalLink" href="http://oozie.apache.org/docs/4.2.0/DG_Overview.html">workflow overview</a> and <a class="externalLink" href="http://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html">workflow specification</a> for details.</p>
<p>Syntax:</p>
<div class="source">
<pre>
@@ -1217,6 +1250,44 @@ Sqoop options can be specified via the arguments.
<pre>$input_filter
</pre></div></div>
<div class="section">
+<h5>Spark<a name="Spark"></a></h5>
+<p>Falcon also adds the Spark engine as part of Spark Integration which enables users to run the Java/Python Spark application as a process. When "spark" workflow engine is mentioned spark related parameters must be provided through <spark-attributes> Examples:</p>
+<div class="source">
+<pre>
+<process name="spark-process">
+...
+ <workflow engine="spark" path="/resources/action">
+ <spark-attributes>
+ <master>local</master>
+ <name>Spark WordCount</name>
+ <class>org.examples.WordCount</class>
+ <jar>/resources/action/lib/spark-application.jar</jar>
+ <spark-opts>--num-executors 1 --driver-memory 512m</spark-opts>
+ </spark-attributes>
+...
+</process>
+
+</pre></div>
+<p>This defines the workflow engine to be spark and Java/Python Spark application must be defined with "jar" option that need to be executed. There is flexibility to override the Spark master through process entity either to "yarn-client" or "yarn-cluster", if spark interface is already defined in cluster entity. Input and Output data to the Spark application will be set as argument when Spark workflow will be generated, if input and output feed entity is defined in the process entity. In the set of arguments, first argument will always correspond to input feed, second argument will always correspond to output feed and then user's provided argument will be set.</p>
+<p>For running the Spark SQL process entity, that read and write the data stored on Hive, the datanucleus jars under the $HIVE_HOME/lib directory and hive-site.xml under $SPARK_HOME/conf/ directory need to be available on the driver and all executors launched by the YARN cluster. The convenient way to do this is adding them through the --jars option and --file option of the spark-opts attribute. Example:</p>
+<div class="source">
+<pre>
+<process name="spark-process">
+...
+ <workflow engine="spark" path="/resources/action">
+ <spark-attributes>
+ <master>local</master>
+ <name>Spark SQL</name>
+ <class>org.examples.SparkSQLProcessTable</class>
+ <jar>/resources/action/lib/spark-application.jar</jar>
+ <spark-opts>--num-executors 1 --driver-memory 512m --jars /usr/local/hive/lib/datanucleus-rdbms.jar,/usr/local/hive/lib/datanucleus-core.jar,/usr/local/hive/lib/datanucleus-api-jdo.jar --files /usr/local/spark/conf/hive-site.xml</spark-opts>
+ </spark-attributes>
+...
+</process>
+
+</pre></div>
+<p>Input and Output to the Spark SQL application will be set as argument when Spark workflow will be generated, if input and output feed entity is defined in the process entity. If input feed is of table type, then input table partition, table name and database name will be set as input arguments. If output feed is of table type, then output table partition, table name and database name will be set as output arguments. Once input and output arguments is set, then user's provided argument will be set.</p></div>
+<div class="section">
<h4>Retry<a name="Retry"></a></h4>
<p>Retry policy defines how the workflow failures should be handled. Three retry policies are defined: periodic, exp-backoff(exponential backoff) and final. Depending on the delay and number of attempts, the workflow is re-tried after specific intervals. If user sets the onTimeout attribute to "true", retries will happen for TIMED_OUT instances. Syntax:</p>
<div class="source">
@@ -1243,10 +1314,10 @@ Sqoop options can be specified via the arguments.
<p>To enable retries for instances for feeds, user will have to set the following properties in runtime.properties</p>
<div class="source">
<pre>
-falcon.recipe.retry.policy=periodic
-falcon.recipe.retry.delay=minutes(30)
-falcon.recipe.retry.attempts=3
-falcon.recipe.retry.onTimeout=false
+falcon.retry.policy=periodic
+falcon.retry.delay=minutes(30)
+falcon.retry.attempts=3
+falcon.retry.onTimeout=false
<verbatim>
---+++ Late data
Late data handling defines how the late data should be handled. Each feed is defined with a late cut-off value which specifies the time till which late data is valid. For example, late cut-off of hours(6) means that data for nth hour can get delayed by upto 6 hours. Late data specification in process defines how this late data is handled.