You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@falcon.apache.org by so...@apache.org on 2015/11/06 20:48:22 UTC
falcon git commit: FALCON-1585 Documentation for HDFS and Hive DR.
Contributed by Peeyush Bishnoi
Repository: falcon
Updated Branches:
refs/heads/0.8 9d201294f -> 517718bfb
FALCON-1585 Documentation for HDFS and Hive DR. Contributed by Peeyush Bishnoi
Project: http://git-wip-us.apache.org/repos/asf/falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/falcon/commit/517718bf
Tree: http://git-wip-us.apache.org/repos/asf/falcon/tree/517718bf
Diff: http://git-wip-us.apache.org/repos/asf/falcon/diff/517718bf
Branch: refs/heads/0.8
Commit: 517718bfbdd30b09785de303b7b8b9a8f35dae29
Parents: 9d20129
Author: Sowmya Ramesh <sr...@hortonworks.com>
Authored: Fri Nov 6 11:48:19 2015 -0800
Committer: Sowmya Ramesh <sr...@hortonworks.com>
Committed: Fri Nov 6 11:48:19 2015 -0800
----------------------------------------------------------------------
CHANGES.txt | 2 ++
docs/src/site/twiki/HDFSDR.twiki | 21 +++++++++++
docs/src/site/twiki/HiveDR.twiki | 64 ++++++++++++++++++++++++++++++++++
docs/src/site/twiki/Recipes.twiki | 4 +++
4 files changed, 91 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 9eec6ea..7bf823d 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -27,6 +27,8 @@ Release version: 0.8
FALCON-1027 Falcon proxy user support(Sowmya Ramesh)
IMPROVEMENTS
+ FALCON-1585 Documentation for HDFS and Hive DR(Peeyush Bishnoi via Sowmya Ramesh)
+
FALCON-1582 Documentation for globally disabling retries (Pallavi Rao)
FALCON-1574 Document behavior of lifecycle retention stage frequency behavior(Sowmya Ramesh via Ajay Yadava).
http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/docs/src/site/twiki/HDFSDR.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/HDFSDR.twiki b/docs/src/site/twiki/HDFSDR.twiki
new file mode 100644
index 0000000..4a42c4c
--- /dev/null
+++ b/docs/src/site/twiki/HDFSDR.twiki
@@ -0,0 +1,21 @@
+---+ HDFS DR Recipe
+---++ Overview
+Falcon supports HDFS DR recipe to replicate data from source cluster to destination cluster.
+
+---++ Usage
+---+++ Setup cluster definition.
+ <verbatim>
+ $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml
+ </verbatim>
+
+---+++ Update recipes properties
+ Update recipe properties file in addons/recipes/hdfs-replication with required attributes for replicating
+ data from source cluster to destination cluster.
+
+---+++ Submit HDFS DR recipe
+ <verbatim>
+ $FALCON_HOME/bin/falcon recipe -name hdfs-replication -operation HDFS_REPLICATION
+ </verbatim>
+
+Recipe templates for HDFS DR is available in addons/recipes/hdfs-replication and copy it to
+recipe path (*falcon.recipe.path=<recipe directory path>*) by specifying in client.properties.
http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/docs/src/site/twiki/HiveDR.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/HiveDR.twiki b/docs/src/site/twiki/HiveDR.twiki
new file mode 100644
index 0000000..ca039ce
--- /dev/null
+++ b/docs/src/site/twiki/HiveDR.twiki
@@ -0,0 +1,64 @@
+---+Hive Disaster Recovery
+
+
+---++Overview
+Falcon provides feature to replicate Hive metadata and data events from source cluster
+to destination cluster. This is supported for secure and unsecure cluster through Falcon Recipes.
+
+
+---++Prerequisites
+Following is the prerequisites to use Hive DR
+
+ * *Hive 1.2.0+*
+ * *Oozie 4.2.0+*
+
+*Note:* Set following properties in hive-site.xml for replicating the Hive events on source and destination Hive cluster:
+<verbatim>
+ <property>
+ <name>hive.metastore.event.listeners</name>
+ <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value>
+ <description>event listeners that are notified of any metastore changes</description>
+ </property>
+
+ <property>
+ <name>hive.metastore.dml.events</name>
+ <value>true</value>
+ </property>
+</verbatim>
+
+---++ Usage
+---+++ Bootstrap
+ Perform initial bootstrap of Table and Database from source cluster to destination cluster
+ * *Database Bootstrap*
+ For bootstrapping DB replication, first destination DB should be created. This step is expected,
+ since DB replication definitions can be set up by users only on pre-existing DB’s. Second, Export all tables in
+ the source db and Import it in the destination db, as described in Table bootstrap.
+
+ * *Table Bootstrap*
+ For bootstrapping table replication, essentially after having turned on the !DbNotificationListener
+ on the source db, perform an Export of the table, distcp the Export over to the destination
+ warehouse and do an Import over there. Check the following [[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport][Hive Export-Import]] for syntax details
+ and examples.
+ This will set up the destination table so that the events on the source cluster that modify the table
+ will then be replicated.
+
+---+++ Setup cluster definition
+ <verbatim>
+ $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml
+ </verbatim>
+
+---+++ Update recipes properties
+ Update recipe properties file in addons/recipes/hive-disaster-recovery with required attributes for replicating
+ Hive data and metadata from source cluster to destination cluster.
+
+---+++ Submit Hive DR recipe
+ <verbatim>
+ $FALCON_HOME/bin/falcon recipe -name hive-disaster-recovery -operation HIVE_DISASTER_RECOVERY
+ </verbatim>
+
+
+Recipe templates for Hive DR is available in addons/recipes/hive-disaster-recovery and copy it to
+recipe path (*falcon.recipe.path=<recipe directory path>*) by specifying in client.properties.
+
+*Note:* If kerberos security is enabled on cluster, use the secure templates for Hive DR from
+ addons/recipes/hive-disaster-recovery
http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/docs/src/site/twiki/Recipes.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Recipes.twiki b/docs/src/site/twiki/Recipes.twiki
index ea29d94..792bf1b 100644
--- a/docs/src/site/twiki/Recipes.twiki
+++ b/docs/src/site/twiki/Recipes.twiki
@@ -76,6 +76,10 @@ GraphDB for display on UI.
* Sample recipes are published in addons/recipes
+---++ Types of recipes
+ * [[HDFSDR][HDFS Recipe]]
+ * [[HiveDR][HiveDR Recipe]]
+
---++ Packaging
* There is no packaging for recipes at this time but will be added soon.