You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@atlas.apache.org by ni...@apache.org on 2018/09/18 12:00:57 UTC
[36/39] atlas-website git commit: ATLAS-2868_Atlas_website changes
for version 1.1.0-release
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/9ae0ebb0/1.1.0/Hook-Falcon.html
----------------------------------------------------------------------
diff --git a/1.1.0/Hook-Falcon.html b/1.1.0/Hook-Falcon.html
new file mode 100644
index 0000000..e2ee95a
--- /dev/null
+++ b/1.1.0/Hook-Falcon.html
@@ -0,0 +1,197 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Hook-Falcon.twiki at 2018-09-06
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20180906" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Atlas – Falcon Atlas Bridge</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
+ </head>
+ <body class="topBarEnabled">
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container" style="width: 68%;"><div class="nav-collapse">
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache Atlas <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="index.html" title="Overview">Overview</a></li>
+ <li><a href="license.html" title="License">License</a></li>
+ <li><a href="Downloads.html" title="Downloads">Downloads</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a></li>
+ <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git" title="Git">Git</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a></li>
+ <li><a href="https://reviews.apache.org/groups/atlas/?sort=-time_added" title="Review Board">Review Board</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="project-info.html" title="Summary">Summary</a></li>
+ <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li>
+ <li><a href="team-list.html" title="Team">Team</a></li>
+ <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li>
+ <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Downloads <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="Downloads.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="Downloads.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="Downloads.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="Downloads.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="Downloads.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="Downloads.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="Downloads.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="../index.html" title="latest">latest</a></li>
+ <li><a href="../1.1.0/index.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="../1.0.0/index.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a></li>
+ <li><a href="https://www.apache.org/events/current-event" title="Events">Events</a></li>
+ <li><a href="https://www.apache.org/licenses/" title="License">License</a></li>
+ <li><a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li>
+ </ul>
+ </li>
+ </ul>
+<form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" >
+ <input value="http://atlas.apache.org" name="sitesearch" type="hidden"/>
+ <input class="search-query" name="q" id="query" type="text" />
+</form>
+<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script>
+ <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"
+ scrolling="no" frameborder="0"
+ style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe>
+ <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script>
+ <ul class="nav pull-right"><li style="margin-top: 10px;">
+ <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div>
+ </li></ul>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li>
+ <li class="active ">Falcon Atlas Bridge</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-06</li>
+ <li id="projectVersion" class="pull-right">Version: 1.1.0-SNAPSHOT</li>
+ </ul>
+ </div>
+ <div id="bodyColumn" >
+<div class="section">
+<h2><a name="Falcon_Atlas_Bridge"></a>Falcon Atlas Bridge</h2></div>
+<div class="section">
+<h3><a name="Falcon_Model"></a>Falcon Model</h3>
+<p>The default hive model includes the following types:</p>
+<ul>
+<li>Entity types:
+<ul>
+<li>falcon_cluster
+<ul>
+<li>super-types: Infrastructure</li>
+<li>attributes: timestamp, colo, owner, tags</li></ul></li>
+<li>falcon_feed
+<ul>
+<li>super-types: DataSet</li>
+<li>attributes: timestamp, stored-in, owner, groups, tags</li></ul></li>
+<li>falcon_feed_creation
+<ul>
+<li>super-types: Process</li>
+<li>attributes: timestamp, stored-in, owner</li></ul></li>
+<li>falcon_feed_replication
+<ul>
+<li>super-types: Process</li>
+<li>attributes: timestamp, owner</li></ul></li>
+<li>falcon_process
+<ul>
+<li>super-types: Process</li>
+<li>attributes: timestamp, runs-on, owner, tags, pipelines, workflow-properties</li></ul></li></ul></li></ul>
+<p>One falcon_process entity is created for every cluster that the falcon process is defined for.</p>
+<p>The entities are created and de-duped using unique qualifiedName attribute. They provide namespace and can be used for querying/lineage as well. The unique attributes are:</p>
+<ul>
+<li>falcon_process.qualifiedName - <process name>@<cluster name></li>
+<li>falcon_cluster.qualifiedName - <cluster name></li>
+<li>falcon_feed.qualifiedName - <feed name>@<cluster name></li>
+<li>falcon_feed_creation.qualifiedName - <feed name></li>
+<li>falcon_feed_replication.qualifiedName - <feed name></li></ul></div>
+<div class="section">
+<h3><a name="Falcon_Hook"></a>Falcon Hook</h3>
+<p>Falcon supports listeners on falcon entity submission. This is used to add entities in Atlas using the model detailed above. Follow the instructions below to setup Atlas hook in Falcon:</p>
+<ul>
+<li>Add 'org.apache.atlas.falcon.service.AtlasService' to application.services in <falcon-conf>/startup.properties</li>
+<li>untar apache-atlas-${project.version}-falcon-hook.tar.gz</li>
+<li>cd apache-atlas-falcon-hook-${project.version}</li>
+<li>Copy entire contents of folder apache-atlas-falcon-hook-${project.version}/hook/falcon to <atlas-home>/hook/falcon</li>
+<li>Link Atlas hook jars in Falcon classpath - 'ln -s <atlas-home>/hook/falcon/* <falcon-home>/server/webapp/falcon/WEB-INF/lib/'</li>
+<li>In <falcon_conf>/falcon-env.sh, set an environment variable as follows:</li></ul>
+<div class="source"><pre class="prettyprint">
+ export FALCON_SERVER_OPTS="<atlas_home>/hook/falcon/*:$FALCON_SERVER_OPTS"
+</pre></div>
+<p>The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:</p>
+<ul>
+<li>atlas.hook.falcon.synchronous - boolean, true to run the hook synchronously. default false</li>
+<li>atlas.hook.falcon.numRetries - number of retries for notification failure. default 3</li>
+<li>atlas.hook.falcon.minThreads - core number of threads. default 5</li>
+<li>atlas.hook.falcon.maxThreads - maximum number of threads. default 5</li>
+<li>atlas.hook.falcon.keepAliveTime - keep alive time in msecs. default 10</li>
+<li>atlas.hook.falcon.queueSize - queue size for the threadpool. default 10000</li></ul>
+<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div>
+<div class="section">
+<h3><a name="NOTES"></a>NOTES</h3>
+<p></p>
+<ul>
+<li>In falcon cluster entity, cluster name used should be uniform across components like hive, falcon, sqoop etc. If used with ambari, ambari cluster name should be used for cluster entity</li></ul></div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container">
+ <div class="row">
+<p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="20" width="20" /></a>Copyright © 2011-2018 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>.<br/>
+Apache Atlas, Atlas, Apache, the Apache feather logo are trademarks of the <a href="https://www.apache.org">Apache Software Foundation</a>.<br/>
+All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+ <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a>
+</p>
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/9ae0ebb0/1.1.0/Hook-HBase.html
----------------------------------------------------------------------
diff --git a/1.1.0/Hook-HBase.html b/1.1.0/Hook-HBase.html
new file mode 100644
index 0000000..49b49aa
--- /dev/null
+++ b/1.1.0/Hook-HBase.html
@@ -0,0 +1,212 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Hook-HBase.twiki at 2018-09-06
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20180906" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Atlas – Apache Atlas Hook & Bridge for Apache HBase</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
+ </head>
+ <body class="topBarEnabled">
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container" style="width: 68%;"><div class="nav-collapse">
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache Atlas <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="index.html" title="Overview">Overview</a></li>
+ <li><a href="license.html" title="License">License</a></li>
+ <li><a href="Downloads.html" title="Downloads">Downloads</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a></li>
+ <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git" title="Git">Git</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a></li>
+ <li><a href="https://reviews.apache.org/groups/atlas/?sort=-time_added" title="Review Board">Review Board</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="project-info.html" title="Summary">Summary</a></li>
+ <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li>
+ <li><a href="team-list.html" title="Team">Team</a></li>
+ <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li>
+ <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Downloads <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="Downloads.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="Downloads.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="Downloads.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="Downloads.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="Downloads.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="Downloads.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="Downloads.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="../index.html" title="latest">latest</a></li>
+ <li><a href="../1.1.0/index.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="../1.0.0/index.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a></li>
+ <li><a href="https://www.apache.org/events/current-event" title="Events">Events</a></li>
+ <li><a href="https://www.apache.org/licenses/" title="License">License</a></li>
+ <li><a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li>
+ </ul>
+ </li>
+ </ul>
+<form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" >
+ <input value="http://atlas.apache.org" name="sitesearch" type="hidden"/>
+ <input class="search-query" name="q" id="query" type="text" />
+</form>
+<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script>
+ <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"
+ scrolling="no" frameborder="0"
+ style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe>
+ <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script>
+ <ul class="nav pull-right"><li style="margin-top: 10px;">
+ <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div>
+ </li></ul>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li>
+ <li class="active ">Apache Atlas Hook & Bridge for Apache HBase</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-06</li>
+ <li id="projectVersion" class="pull-right">Version: 1.1.0-SNAPSHOT</li>
+ </ul>
+ </div>
+ <div id="bodyColumn" >
+<div class="section">
+<h2><a name="Apache_Atlas_Hook_.26_Bridge_for_Apache_HBase"></a>Apache Atlas Hook & Bridge for Apache HBase</h2></div>
+<div class="section">
+<h3><a name="HBase_Model"></a>HBase Model</h3>
+<p>HBase model includes the following types:</p>
+<ul>
+<li>Entity types:
+<ul>
+<li>hbase_namespace
+<ul>
+<li>super-types: !Asset</li>
+<li>attributes: qualifiedName, name, description, owner, clusterName, parameters, createTime, modifiedTime</li></ul></li>
+<li>hbase_table
+<ul>
+<li>super-types: DataSet</li>
+<li>attributes: qualifiedName, name, description, owner, namespace, column_families, uri, parameters, createtime, modifiedtime, maxfilesize, isReadOnly, isCompactionEnabled, isNormalizationEnabled, ReplicaPerRegion, Durability</li></ul></li>
+<li>hbase_column_family
+<ul>
+<li>super-types: DataSet</li>
+<li>attributes: qualifiedName, name, description, owner, columns, createTime, bloomFilterType, compressionType, compactionCompressionType, encryptionType, inMemoryCompactionPolicy, keepDeletedCells, maxversions, minVersions, datablockEncoding, storagePolicy, ttl, blockCachedEnabled, cacheBloomsOnWrite, cacheDataOnWrite, evictBlocksOnClose, prefetchBlocksOnOpen, newVersionsBehavior, isMobEnabled, mobCompactPartitionPolicy</li></ul></li></ul></li></ul>
+<p>HBase entities are created and de-duped in Atlas using unique attribute qualifiedName, whose value should be formatted as detailed below. Note that namespaceName, tableName and columnFamilyName should be in lower case.</p>
+<div class="source"><pre class="prettyprint">
+ hbase_namespace.qualifiedName: <namespaceName>@<clusterName>
+ hbase_table.qualifiedName: <namespaceName>:<tableName>@<clusterName>
+ hbase_column_family.qualifiedName: <namespaceName>:<tableName>.<columnFamilyName>@<clusterName>
+
+</pre></div></div>
+<div class="section">
+<h3><a name="HBase_Hook"></a>HBase Hook</h3>
+<p>Atlas HBase hook registers with HBase master as a co-processor. On detecting changes to HBase namespaces/tables/column-families, Atlas hook updates the metadata in Atlas via Kafka notifications. Follow the instructions below to setup Atlas hook in HBase:</p>
+<ul>
+<li>Register Atlas hook in hbase-site.xml by adding the following:</li></ul>
+<div class="source"><pre class="prettyprint">
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.atlas.hbase.hook.HBaseAtlasCoprocessor</value>
+ </property>
+</pre></div>
+<p></p>
+<ul>
+<li>untar apache-atlas-${project.version}-hbase-hook.tar.gz</li>
+<li>cd apache-atlas-hbase-hook-${project.version}</li>
+<li>Copy entire contents of folder apache-atlas-hbase-hook-${project.version}/hook/hbase to <atlas package>/hook/hbase</li>
+<li>Link Atlas hook jars in HBase classpath - 'ln -s <atlas package>/hook/hbase/* <hbase-home>/lib/'</li>
+<li>Copy <atlas-conf>/atlas-application.properties to the HBase conf directory.</li></ul>
+<p>The following properties in atlas-application.properties control the thread pool and notification details:</p>
+<div class="source"><pre class="prettyprint">
+atlas.hook.hbase.synchronous=false # whether to run the hook synchronously. false recommended to avoid delays in HBase operations. Default: false
+atlas.hook.hbase.numRetries=3 # number of retries for notification failure. Default: 3
+atlas.hook.hbase.queueSize=10000 # queue size for the threadpool. Default: 10000
+
+atlas.cluster.name=primary # clusterName to use in qualifiedName of entities. Default: primary
+
+atlas.kafka.zookeeper.connect= # Zookeeper connect URL for Kafka. Example: localhost:2181
+atlas.kafka.zookeeper.connection.timeout.ms=30000 # Zookeeper connection timeout. Default: 30000
+atlas.kafka.zookeeper.session.timeout.ms=60000 # Zookeeper session timeout. Default: 60000
+atlas.kafka.zookeeper.sync.time.ms=20 # Zookeeper sync time. Default: 20
+
+</pre></div>
+<p>Other configurations for Kafka notification producer can be specified by prefixing the configuration name with "atlas.kafka.". For list of configuration supported by Kafka producer, please refer to <a class="externalLink" href="http://kafka.apache.org/documentation/#producerconfigs">Kafka Producer Configs</a></p></div>
+<div class="section">
+<h3><a name="NOTES"></a>NOTES</h3>
+<p></p>
+<ul>
+<li>Only the namespace, table and column-family create/update/ delete operations are captured by Atlas HBase hook. Changes to columns are be captured.</li></ul></div>
+<div class="section">
+<h3><a name="Importing_HBase_Metadata"></a>Importing HBase Metadata</h3>
+<p>Apache Atlas provides a command-line utility, import-hbase.sh, to import metadata of Apache HBase namespaces and tables into Apache Atlas. This utility can be used to initialize Apache Atlas with namespaces/tables present in a Apache HBase cluster. This utility supports importing metadata of a specific table, tables in a specific namespace or all tables.</p>
+<div class="source"><pre class="prettyprint">
+Usage 1: <atlas package>/hook-bin/import-hbase.sh
+Usage 2: <atlas package>/hook-bin/import-hbase.sh [-n <namespace regex> OR --namespace <namespace regex>] [-t <table regex> OR --table <table regex>]
+Usage 3: <atlas package>/hook-bin/import-hbase.sh [-f <filename>]
+ File Format:
+ namespace1:tbl1
+ namespace1:tbl2
+ namespace2:tbl1
+
+</pre></div></div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container">
+ <div class="row">
+<p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="20" width="20" /></a>Copyright © 2011-2018 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>.<br/>
+Apache Atlas, Atlas, Apache, the Apache feather logo are trademarks of the <a href="https://www.apache.org">Apache Software Foundation</a>.<br/>
+All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+ <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a>
+</p>
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/9ae0ebb0/1.1.0/Hook-Hive.html
----------------------------------------------------------------------
diff --git a/1.1.0/Hook-Hive.html b/1.1.0/Hook-Hive.html
new file mode 100644
index 0000000..2072147
--- /dev/null
+++ b/1.1.0/Hook-Hive.html
@@ -0,0 +1,288 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Hook-Hive.twiki at 2018-09-06
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20180906" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Atlas – Apache Atlas Hook & Bridge for Apache Hive</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
+ </head>
+ <body class="topBarEnabled">
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container" style="width: 68%;"><div class="nav-collapse">
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache Atlas <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="index.html" title="Overview">Overview</a></li>
+ <li><a href="license.html" title="License">License</a></li>
+ <li><a href="Downloads.html" title="Downloads">Downloads</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a></li>
+ <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git" title="Git">Git</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a></li>
+ <li><a href="https://reviews.apache.org/groups/atlas/?sort=-time_added" title="Review Board">Review Board</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="project-info.html" title="Summary">Summary</a></li>
+ <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li>
+ <li><a href="team-list.html" title="Team">Team</a></li>
+ <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li>
+ <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Downloads <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="Downloads.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="Downloads.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="Downloads.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="Downloads.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="Downloads.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="Downloads.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="Downloads.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="../index.html" title="latest">latest</a></li>
+ <li><a href="../1.1.0/index.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="../1.0.0/index.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a></li>
+ <li><a href="https://www.apache.org/events/current-event" title="Events">Events</a></li>
+ <li><a href="https://www.apache.org/licenses/" title="License">License</a></li>
+ <li><a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li>
+ </ul>
+ </li>
+ </ul>
+<form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" >
+ <input value="http://atlas.apache.org" name="sitesearch" type="hidden"/>
+ <input class="search-query" name="q" id="query" type="text" />
+</form>
+<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script>
+ <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"
+ scrolling="no" frameborder="0"
+ style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe>
+ <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script>
+ <ul class="nav pull-right"><li style="margin-top: 10px;">
+ <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div>
+ </li></ul>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li>
+ <li class="active ">Apache Atlas Hook & Bridge for Apache Hive</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-06</li>
+ <li id="projectVersion" class="pull-right">Version: 1.1.0-SNAPSHOT</li>
+ </ul>
+ </div>
+ <div id="bodyColumn" >
+<div class="section">
+<h2><a name="Apache_Atlas_Hook_.26_Bridge_for_Apache_Hive"></a>Apache Atlas Hook & Bridge for Apache Hive</h2></div>
+<div class="section">
+<h3><a name="Hive_Model"></a>Hive Model</h3>
+<p>Hive model includes the following types:</p>
+<ul>
+<li>Entity types:
+<ul>
+<li>hive_db
+<ul>
+<li>super-types: !Asset</li>
+<li>attributes: qualifiedName, name, description, owner, clusterName, location, parameters, ownerName</li></ul></li>
+<li>hive_table
+<ul>
+<li>super-types: DataSet</li>
+<li>attributes: qualifiedName, name, description, owner, db, createTime, lastAccessTime, comment, retention, sd, partitionKeys, columns, aliases, parameters, viewOriginalText, viewExpandedText, tableType, temporary</li></ul></li>
+<li>hive_column
+<ul>
+<li>super-types: DataSet</li>
+<li>attributes: qualifiedName, name, description, owner, type, comment, table</li></ul></li>
+<li>hive_storagedesc
+<ul>
+<li>super-types: Referenceable</li>
+<li>attributes: qualifiedName, table, location, inputFormat, outputFormat, compressed, numBuckets, serdeInfo, bucketCols, sortCols, parameters, storedAsSubDirectories</li></ul></li>
+<li>hive_process
+<ul>
+<li>super-types: Process</li>
+<li>attributes: qualifiedName, name, description, owner, inputs, outputs, startTime, endTime, userName, operationType, queryText, queryPlan, queryId, clusterName</li></ul></li>
+<li>hive_column_lineage
+<ul>
+<li>super-types: Process</li>
+<li>attributes: qualifiedName, name, description, owner, inputs, outputs, query, depenendencyType, expression</li></ul></li></ul></li></ul>
+<p></p>
+<ul>
+<li>Enum types:
+<ul>
+<li>hive_principal_type
+<ul>
+<li>values: USER, ROLE, GROUP</li></ul></li></ul></li></ul>
+<p></p>
+<ul>
+<li>Struct types:
+<ul>
+<li>hive_order
+<ul>
+<li>attributes: col, order</li></ul></li>
+<li>hive_serde
+<ul>
+<li>attributes: name, serializationLib, parameters</li></ul></li></ul></li></ul>
+<p>Hive entities are created and de-duped in Atlas using unique attribute qualifiedName, whose value should be formatted as detailed below. Note that dbName, tableName and columnName should be in lower case.</p>
+<div class="source"><pre class="prettyprint">
+ hive_db.qualifiedName: <dbName>@<clusterName>
+ hive_table.qualifiedName: <dbName>.<tableName>@<clusterName>
+ hive_column.qualifiedName: <dbName>.<tableName>.<columnName>@<clusterName>
+ hive_process.queryString: trimmed query string in lower case
+
+</pre></div></div>
+<div class="section">
+<h3><a name="Hive_Hook"></a>Hive Hook</h3>
+<p>Atlas Hive hook registers with Hive to listen for create/update/delete operations and updates the metadata in Atlas, via Kafka notifications, for the changes in Hive. Follow the instructions below to setup Atlas hook in Hive:</p>
+<ul>
+<li>Set-up Atlas hook in hive-site.xml by adding the following:</li></ul>
+<div class="source"><pre class="prettyprint">
+ <property>
+ <name>hive.exec.post.hooks</name>
+ <value>org.apache.atlas.hive.hook.HiveHook</value>
+ </property>
+</pre></div>
+<p></p>
+<ul>
+<li>untar apache-atlas-${project.version}-hive-hook.tar.gz</li>
+<li>cd apache-atlas-hive-hook-${project.version}</li>
+<li>Copy entire contents of folder apache-atlas-hive-hook-${project.version}/hook/hive to <atlas package>/hook/hive</li>
+<li>Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in hive-env.sh of your hive configuration</li>
+<li>Copy <atlas-conf>/atlas-application.properties to the hive conf directory.</li></ul>
+<p>The following properties in atlas-application.properties control the thread pool and notification details:</p>
+<div class="source"><pre class="prettyprint">
+atlas.hook.hive.synchronous=false # whether to run the hook synchronously. false recommended to avoid delays in Hive query completion. Default: false
+atlas.hook.hive.numRetries=3 # number of retries for notification failure. Default: 3
+atlas.hook.hive.queueSize=10000 # queue size for the threadpool. Default: 10000
+
+atlas.cluster.name=primary # clusterName to use in qualifiedName of entities. Default: primary
+
+atlas.kafka.zookeeper.connect= # Zookeeper connect URL for Kafka. Example: localhost:2181
+atlas.kafka.zookeeper.connection.timeout.ms=30000 # Zookeeper connection timeout. Default: 30000
+atlas.kafka.zookeeper.session.timeout.ms=60000 # Zookeeper session timeout. Default: 60000
+atlas.kafka.zookeeper.sync.time.ms=20 # Zookeeper sync time. Default: 20
+
+</pre></div>
+<p>Other configurations for Kafka notification producer can be specified by prefixing the configuration name with "atlas.kafka.". For list of configuration supported by Kafka producer, please refer to <a class="externalLink" href="http://kafka.apache.org/documentation/#producerconfigs">Kafka Producer Configs</a></p></div>
+<div class="section">
+<h3><a name="Column_Level_Lineage"></a>Column Level Lineage</h3>
+<p>Starting from 0.8-incubating version of Atlas, Column level lineage is captured in Atlas. Below are the details</p></div>
+<div class="section">
+<h4><a name="Model"></a>Model</h4>
+<p></p>
+<ul>
+<li>ColumnLineageProcess type is a subtype of Process</li></ul>
+<p></p>
+<ul>
+<li>This relates an output Column to a set of input Columns or the Input Table</li></ul>
+<p></p>
+<ul>
+<li>The lineage also captures the kind of dependency, as listed below:
+<ul>
+<li>SIMPLE: output column has the same value as the input</li>
+<li>EXPRESSION: output column is transformed by some expression at runtime (for e.g. a Hive SQL expression) on the Input Columns.</li>
+<li>SCRIPT: output column is transformed by a user provided script.</li></ul></li></ul>
+<p></p>
+<ul>
+<li>In case of EXPRESSION dependency the expression attribute contains the expression in string form</li></ul>
+<p></p>
+<ul>
+<li>Since Process links input and output DataSets, Column is a subtype of DataSet</li></ul></div>
+<div class="section">
+<h4><a name="Examples"></a>Examples</h4>
+<p>For a simple CTAS below:</p>
+<div class="source"><pre class="prettyprint">
+create table t2 as select id, name from T1
+</pre></div>
+<p>The lineage is captured as</p>
+<p><img src="images/column_lineage_ex1.png" alt="" /></p></div>
+<div class="section">
+<h4><a name="Extracting_Lineage_from_Hive_commands"></a>Extracting Lineage from Hive commands</h4>
+<p>* The HiveHook maps the LineageInfo in the HookContext to Column lineage instances</p>
+<p>* The LineageInfo in Hive provides column-level lineage for the final FileSinkOperator, linking them to the input columns in the Hive Query</p></div>
+<div class="section">
+<h3><a name="NOTES"></a>NOTES</h3>
+<p></p>
+<ul>
+<li>Column level lineage works with Hive version 1.2.1 after the patch for <a href="https://issues.apache.org/jira/browse/HIVE-13112">HIVE-13112</a> is applied to Hive source</li>
+<li>Since database name, table name and column names are case insensitive in hive, the corresponding names in entities are lowercase. So, any search APIs should use lowercase while querying on the entity names</li>
+<li>The following hive operations are captured by hive hook currently
+<ul>
+<li>create database</li>
+<li>create table/view, create table as select</li>
+<li>load, import, export</li>
+<li>DMLs (insert)</li>
+<li>alter database</li>
+<li>alter table (skewed table information, stored as, protection is not supported)</li>
+<li>alter view</li></ul></li></ul></div>
+<div class="section">
+<h3><a name="Importing_Hive_Metadata"></a>Importing Hive Metadata</h3>
+<p>Apache Atlas provides a command-line utility, import-hive.sh, to import metadata of Apache Hive databases and tables into Apache Atlas. This utility can be used to initialize Apache Atlas with databases/tables present in Apache Hive. This utility supports importing metadata of a specific table, tables in a specific database or all databases and tables.</p>
+<div class="source"><pre class="prettyprint">
+Usage 1: <atlas package>/hook-bin/import-hive.sh
+Usage 2: <atlas package>/hook-bin/import-hive.sh [-d <database regex> OR --database <database regex>] [-t <table regex> OR --table <table regex>]
+Usage 3: <atlas package>/hook-bin/import-hive.sh [-f <filename>]
+ File Format:
+ database1:tbl1
+ database1:tbl2
+ database2:tbl1
+
+</pre></div></div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container">
+ <div class="row">
+<p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="20" width="20" /></a>Copyright © 2011-2018 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>.<br/>
+Apache Atlas, Atlas, Apache, the Apache feather logo are trademarks of the <a href="https://www.apache.org">Apache Software Foundation</a>.<br/>
+All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+ <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a>
+</p>
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/9ae0ebb0/1.1.0/Hook-Sqoop.html
----------------------------------------------------------------------
diff --git a/1.1.0/Hook-Sqoop.html b/1.1.0/Hook-Sqoop.html
new file mode 100644
index 0000000..afd6a84
--- /dev/null
+++ b/1.1.0/Hook-Sqoop.html
@@ -0,0 +1,208 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Hook-Sqoop.twiki at 2018-09-06
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20180906" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Atlas – Apache Atlas Hook for Apache Sqoop</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
+ </head>
+ <body class="topBarEnabled">
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container" style="width: 68%;"><div class="nav-collapse">
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache Atlas <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="index.html" title="Overview">Overview</a></li>
+ <li><a href="license.html" title="License">License</a></li>
+ <li><a href="Downloads.html" title="Downloads">Downloads</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a></li>
+ <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git" title="Git">Git</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a></li>
+ <li><a href="https://reviews.apache.org/groups/atlas/?sort=-time_added" title="Review Board">Review Board</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="project-info.html" title="Summary">Summary</a></li>
+ <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li>
+ <li><a href="team-list.html" title="Team">Team</a></li>
+ <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li>
+ <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Downloads <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="Downloads.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="Downloads.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="Downloads.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="Downloads.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="Downloads.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="Downloads.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="Downloads.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="../index.html" title="latest">latest</a></li>
+ <li><a href="../1.1.0/index.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="../1.0.0/index.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a></li>
+ <li><a href="https://www.apache.org/events/current-event" title="Events">Events</a></li>
+ <li><a href="https://www.apache.org/licenses/" title="License">License</a></li>
+ <li><a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li>
+ </ul>
+ </li>
+ </ul>
+<form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" >
+ <input value="http://atlas.apache.org" name="sitesearch" type="hidden"/>
+ <input class="search-query" name="q" id="query" type="text" />
+</form>
+<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script>
+ <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"
+ scrolling="no" frameborder="0"
+ style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe>
+ <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script>
+ <ul class="nav pull-right"><li style="margin-top: 10px;">
+ <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div>
+ </li></ul>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li>
+ <li class="active ">Apache Atlas Hook for Apache Sqoop</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-06</li>
+ <li id="projectVersion" class="pull-right">Version: 1.1.0-SNAPSHOT</li>
+ </ul>
+ </div>
+ <div id="bodyColumn" >
+<div class="section">
+<h2><a name="Apache_Atlas_Hook_for_Apache_Sqoop"></a>Apache Atlas Hook for Apache Sqoop</h2></div>
+<div class="section">
+<h3><a name="Sqoop_Model"></a>Sqoop Model</h3>
+<p>Sqoop model includes the following types:</p>
+<ul>
+<li>Entity types:
+<ul>
+<li>sqoop_process
+<ul>
+<li>super-types: Process</li>
+<li>attributes: qualifiedName, name, description, owner, inputs, outputs, operation, commandlineOpts, startTime, endTime, userName</li></ul></li>
+<li>sqoop_dbdatastore
+<ul>
+<li>super-types: DataSet</li>
+<li>attributes: qualifiedName, name, description, owner, dbStoreType, storeUse, storeUri, source</li></ul></li></ul></li></ul>
+<p></p>
+<ul>
+<li>Enum types:
+<ul>
+<li>sqoop_operation_type
+<ul>
+<li>values: IMPORT, EXPORT, EVAL</li></ul></li>
+<li>sqoop_dbstore_usage
+<ul>
+<li>values: TABLE, QUERY, PROCEDURE, OTHER</li></ul></li></ul></li></ul>
+<p>Sqoop entities are created and de-duped in Atlas using unique attribute qualifiedName, whose value should be formatted as detailed below.</p>
+<div class="source"><pre class="prettyprint">
+ sqoop_process.qualifiedName: sqoop <operation> --connect <url> {[--table <tableName>] || [--database <databaseName>]} [--query <storeQuery>]
+ sqoop_dbdatastore.qualifiedName: <storeType> --url <storeUri> {[--table <tableName>] || [--database <databaseName>]} [--query <storeQuery>] --hive-<operation> --hive-database <databaseName> [--hive-table <tableName>] --hive-cluster <clusterName>
+
+</pre></div></div>
+<div class="section">
+<h3><a name="Sqoop_Hook"></a>Sqoop Hook</h3>
+<p>Sqoop added a SqoopJobDataPublisher that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in SqoopHook. This is used to add entities in Atlas using the model detailed above.</p>
+<p>Follow the instructions below to setup Atlas hook in Hive:</p>
+<p>Add the following properties to to enable Atlas hook in Sqoop:</p>
+<ul>
+<li>Set-up Atlas hook in <sqoop-conf>/sqoop-site.xml by adding the following:</li></ul>
+<div class="source"><pre class="prettyprint">
+ <property>
+ <name>sqoop.job.data.publish.class</name>
+ <value>org.apache.atlas.sqoop.hook.SqoopHook</value>
+ </property>
+</pre></div>
+<p></p>
+<ul>
+<li>untar apache-atlas-${project.version}-sqoop-hook.tar.gz</li>
+<li>cd apache-atlas-sqoop-hook-${project.version}</li>
+<li>Copy entire contents of folder apache-atlas-sqoop-hook-${project.version}/hook/sqoop to <atlas package>/hook/sqoop</li>
+<li>Copy <atlas-conf>/atlas-application.properties to to the sqoop conf directory <sqoop-conf>/</li>
+<li>Link <atlas package>/hook/sqoop/*.jar in sqoop lib</li></ul>
+<p>The following properties in atlas-application.properties control the thread pool and notification details:</p>
+<div class="source"><pre class="prettyprint">
+atlas.hook.sqoop.synchronous=false # whether to run the hook synchronously. false recommended to avoid delays in Sqoop operation completion. Default: false
+atlas.hook.sqoop.numRetries=3 # number of retries for notification failure. Default: 3
+atlas.hook.sqoop.queueSize=10000 # queue size for the threadpool. Default: 10000
+
+atlas.cluster.name=primary # clusterName to use in qualifiedName of entities. Default: primary
+
+atlas.kafka.zookeeper.connect= # Zookeeper connect URL for Kafka. Example: localhost:2181
+atlas.kafka.zookeeper.connection.timeout.ms=30000 # Zookeeper connection timeout. Default: 30000
+atlas.kafka.zookeeper.session.timeout.ms=60000 # Zookeeper session timeout. Default: 60000
+atlas.kafka.zookeeper.sync.time.ms=20 # Zookeeper sync time. Default: 20
+
+</pre></div>
+<p>Other configurations for Kafka notification producer can be specified by prefixing the configuration name with "atlas.kafka.". For list of configuration supported by Kafka producer, please refer to <a class="externalLink" href="http://kafka.apache.org/documentation/#producerconfigs">Kafka Producer Configs</a></p></div>
+<div class="section">
+<h3><a name="NOTES"></a>NOTES</h3>
+<p></p>
+<ul>
+<li>Only the following sqoop operations are captured by sqoop hook currently
+<ul>
+<li>hiveImport</li></ul></li></ul></div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container">
+ <div class="row">
+<p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="20" width="20" /></a>Copyright © 2011-2018 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>.<br/>
+Apache Atlas, Atlas, Apache, the Apache feather logo are trademarks of the <a href="https://www.apache.org">Apache Software Foundation</a>.<br/>
+All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+ <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a>
+</p>
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/9ae0ebb0/1.1.0/Hook-Storm.html
----------------------------------------------------------------------
diff --git a/1.1.0/Hook-Storm.html b/1.1.0/Hook-Storm.html
new file mode 100644
index 0000000..d5e2b0e
--- /dev/null
+++ b/1.1.0/Hook-Storm.html
@@ -0,0 +1,207 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Hook-Storm.twiki at 2018-09-06
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20180906" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Atlas – Apache Atlas Hook for Apache Storm</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
+ </head>
+ <body class="topBarEnabled">
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container" style="width: 68%;"><div class="nav-collapse">
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache Atlas <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="index.html" title="Overview">Overview</a></li>
+ <li><a href="license.html" title="License">License</a></li>
+ <li><a href="Downloads.html" title="Downloads">Downloads</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a></li>
+ <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git" title="Git">Git</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a></li>
+ <li><a href="https://reviews.apache.org/groups/atlas/?sort=-time_added" title="Review Board">Review Board</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="project-info.html" title="Summary">Summary</a></li>
+ <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li>
+ <li><a href="team-list.html" title="Team">Team</a></li>
+ <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li>
+ <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Downloads <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="Downloads.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="Downloads.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="Downloads.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="Downloads.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="Downloads.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="Downloads.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="Downloads.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="../index.html" title="latest">latest</a></li>
+ <li><a href="../1.1.0/index.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="../1.0.0/index.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a></li>
+ <li><a href="https://www.apache.org/events/current-event" title="Events">Events</a></li>
+ <li><a href="https://www.apache.org/licenses/" title="License">License</a></li>
+ <li><a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li>
+ </ul>
+ </li>
+ </ul>
+<form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" >
+ <input value="http://atlas.apache.org" name="sitesearch" type="hidden"/>
+ <input class="search-query" name="q" id="query" type="text" />
+</form>
+<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script>
+ <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"
+ scrolling="no" frameborder="0"
+ style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe>
+ <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script>
+ <ul class="nav pull-right"><li style="margin-top: 10px;">
+ <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div>
+ </li></ul>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li>
+ <li class="active ">Apache Atlas Hook for Apache Storm</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-06</li>
+ <li id="projectVersion" class="pull-right">Version: 1.1.0-SNAPSHOT</li>
+ </ul>
+ </div>
+ <div id="bodyColumn" >
+<div class="section">
+<h2><a name="Apache_Atlas_Hook_for_Apache_Storm"></a>Apache Atlas Hook for Apache Storm</h2></div>
+<div class="section">
+<h3><a name="Introduction"></a>Introduction</h3>
+<p>Apache Storm is a distributed real-time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop did for batch processing. The process is essentially a DAG of nodes, which is called <b>topology</b>.</p>
+<p>Apache Atlas is a metadata repository that enables end-to-end data lineage, search and associate business classification.</p>
+<p>The goal of this integration is to push the operational topology metadata along with the underlying data source(s), target(s), derivation processes and any available business context so Atlas can capture the lineage for this topology.</p>
+<p>There are 2 parts in this process detailed below:</p>
+<ul>
+<li>Data model to represent the concepts in Storm</li>
+<li>Storm Atlas Hook to update metadata in Atlas</li></ul></div>
+<div class="section">
+<h3><a name="Storm_Data_Model"></a>Storm Data Model</h3>
+<p>A data model is represented as Types in Atlas. It contains the descriptions of various nodes in the topology graph, such as spouts and bolts and the corresponding producer and consumer types.</p>
+<p>The following types are added in Atlas.</p>
+<p></p>
+<ul>
+<li>storm_topology - represents the coarse-grained topology. A storm_topology derives from an Atlas Process type and hence can be used to inform Atlas about lineage.</li>
+<li>Following data sets are added - kafka_topic, jms_topic, hbase_table, hdfs_data_set. These all derive from an Atlas Dataset type and hence form the end points of a lineage graph.</li>
+<li>storm_spout - Data Producer having outputs, typically Kafka, JMS</li>
+<li>storm_bolt - Data Consumer having inputs and outputs, typically Hive, HBase, HDFS, etc.</li></ul>
+<p>The Storm Atlas hook auto registers dependent models like the Hive data model if it finds that these are not known to the Atlas server.</p>
+<p>The data model for each of the types is described in the class definition at org.apache.atlas.storm.model.StormDataModel.</p></div>
+<div class="section">
+<h3><a name="Storm_Atlas_Hook"></a>Storm Atlas Hook</h3>
+<p>Atlas is notified when a new topology is registered successfully in Storm. Storm provides a hook, backtype.storm.ISubmitterHook, at the Storm client used to submit a storm topology.</p>
+<p>The Storm Atlas hook intercepts the hook post execution and extracts the metadata from the topology and updates Atlas using the types defined. Atlas implements the Storm client hook interface in org.apache.atlas.storm.hook.StormAtlasHook.</p></div>
+<div class="section">
+<h3><a name="Limitations"></a>Limitations</h3>
+<p>The following apply for the first version of the integration.</p>
+<p></p>
+<ul>
+<li>Only new topology submissions are registered with Atlas, any lifecycle changes are not reflected in Atlas.</li>
+<li>The Atlas server needs to be online when a Storm topology is submitted for the metadata to be captured.</li>
+<li>The Hook currently does not support capturing lineage for custom spouts and bolts.</li></ul></div>
+<div class="section">
+<h3><a name="Installation"></a>Installation</h3>
+<p>The Storm Atlas Hook needs to be manually installed in Storm on the client side.</p>
+<ul>
+<li>untar apache-atlas-${project.version}-storm-hook.tar.gz</li>
+<li>cd apache-atlas-storm-hook-${project.version}</li>
+<li>Copy entire contents of folder apache-atlas-storm-hook-${project.version}/hook/storm to $ATLAS_PACKAGE/hook/storm</li></ul>
+<p>Storm Atlas hook jars in $ATLAS_PACKAGE/hook/storm need to be copied to $STORM_HOME/extlib. Replace STORM_HOME with storm installation path.</p>
+<p>Restart all daemons after you have installed the atlas hook into Storm.</p></div>
+<div class="section">
+<h3><a name="Configuration"></a>Configuration</h3></div>
+<div class="section">
+<h4><a name="Storm_Configuration"></a>Storm Configuration</h4>
+<p>The Storm Atlas Hook needs to be configured in Storm client config in <b>$STORM_HOME/conf/storm.yaml</b> as:</p>
+<div class="source"><pre class="prettyprint">
+storm.topology.submission.notifier.plugin.class: "org.apache.atlas.storm.hook.StormAtlasHook"
+
+</pre></div>
+<p>Also set a 'cluster name' that would be used as a namespace for objects registered in Atlas. This name would be used for namespacing the Storm topology, spouts and bolts.</p>
+<p>The other objects like data sets should ideally be identified with the cluster name of the components that generate them. For e.g. Hive tables and databases should be identified using the cluster name set in Hive. The Storm Atlas hook will pick this up if the Hive configuration is available in the Storm topology jar that is submitted on the client and the cluster name is defined there. This happens similarly for HBase data sets. In case this configuration is not available, the cluster name set in the Storm configuration will be used.</p>
+<div class="source"><pre class="prettyprint">
+atlas.cluster.name: "cluster_name"
+
+</pre></div>
+<p>In <b>$STORM_HOME/conf/storm_env.ini</b>, set an environment variable as follows:</p>
+<div class="source"><pre class="prettyprint">
+STORM_JAR_JVM_OPTS:"-Datlas.conf=$ATLAS_HOME/conf/"
+
+</pre></div>
+<p>where ATLAS_HOME is pointing to where ATLAS is installed.</p>
+<p>You could also set this up programatically in Storm Config as:</p>
+<div class="source"><pre class="prettyprint">
+ Config stormConf = new Config();
+ ...
+ stormConf.put(Config.STORM_TOPOLOGY_SUBMISSION_NOTIFIER_PLUGIN,
+ org.apache.atlas.storm.hook.StormAtlasHook.class.getName());
+
+</pre></div></div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container">
+ <div class="row">
+<p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="20" width="20" /></a>Copyright © 2011-2018 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>.<br/>
+Apache Atlas, Atlas, Apache, the Apache feather logo are trademarks of the <a href="https://www.apache.org">Apache Software Foundation</a>.<br/>
+All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+ <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a>
+</p>
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/9ae0ebb0/1.1.0/Import-API-Options.html
----------------------------------------------------------------------
diff --git a/1.1.0/Import-API-Options.html b/1.1.0/Import-API-Options.html
new file mode 100644
index 0000000..7b23531
--- /dev/null
+++ b/1.1.0/Import-API-Options.html
@@ -0,0 +1,265 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Import-API-Options.twiki at 2018-09-06
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20180906" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Atlas – Import API Options</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
+ </head>
+ <body class="topBarEnabled">
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container" style="width: 68%;"><div class="nav-collapse">
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Apache Atlas <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="index.html" title="Overview">Overview</a></li>
+ <li><a href="license.html" title="License">License</a></li>
+ <li><a href="Downloads.html" title="Downloads">Downloads</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a></li>
+ <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git" title="Git">Git</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a></li>
+ <li><a href="https://reviews.apache.org/groups/atlas/?sort=-time_added" title="Review Board">Review Board</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="project-info.html" title="Summary">Summary</a></li>
+ <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li>
+ <li><a href="team-list.html" title="Team">Team</a></li>
+ <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li>
+ <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Downloads <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="Downloads.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="Downloads.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="Downloads.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="Downloads.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="Downloads.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="Downloads.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="Downloads.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="Downloads.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="../index.html" title="latest">latest</a></li>
+ <li><a href="../1.1.0/index.html" title="1.1.0">1.1.0</a></li>
+ <li><a href="../1.0.0/index.html" title="1.0.0">1.0.0</a></li>
+ <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a></li>
+ <li><a href="https://www.apache.org/events/current-event" title="Events">Events</a></li>
+ <li><a href="https://www.apache.org/licenses/" title="License">License</a></li>
+ <li><a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li>
+ </ul>
+ </li>
+ </ul>
+<form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" >
+ <input value="http://atlas.apache.org" name="sitesearch" type="hidden"/>
+ <input class="search-query" name="q" id="query" type="text" />
+</form>
+<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script>
+ <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"
+ scrolling="no" frameborder="0"
+ style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe>
+ <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script>
+ <ul class="nav pull-right"><li style="margin-top: 10px;">
+ <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div>
+ </li></ul>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li>
+ <li class="active ">Import API Options</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-09-06</li>
+ <li id="projectVersion" class="pull-right">Version: 1.1.0-SNAPSHOT</li>
+ </ul>
+ </div>
+ <div id="bodyColumn" >
+<div class="section">
+<h2><a name="Import_API_Options"></a>Import API Options</h2>
+<p>Import API options are specified as <i>options</i> JSON. Since the API accepts multi-part form data, it is possible to sepecify multipls input streams within the CURL call.</p></div>
+<div class="section">
+<h4><a name="Examples_Using_CURL_Calls"></a>Examples Using CURL Calls</h4>
+<div class="source"><pre class="prettyprint">
+curl -g -X POST -u adminuser:password -H "Content-Type: multipart/form-data"
+ -H "Cache-Control: no-cache"
+ -F request=@importOptions.json
+ -F data=@quickStartDB.zip
+ "http://localhost:21000/api/atlas/admin/import"
+
+</pre></div>
+<p>To use the defaults, set the contents of <i>importOptions.json</i> to:</p>
+<div class="source"><pre class="prettyprint">
+{
+ "options": {
+ }
+}
+
+</pre></div></div>
+<div class="section">
+<h4><a name="Options"></a>Options</h4>
+<p>Following options are supported for Import process:</p>
+<p></p>
+<ul>
+<li>Specify transforms during import operation.</li>
+<li>Resume import by specifying starting entity guid.</li>
+<li>Optionally import type definition.</li></ul></div>
+<div class="section">
+<h5><a name="Transforms"></a>Transforms</h5>
+<p>During the import process, the attribute value of the incoming entity can be changed.</p>
+<p>This is possible by specifying entity type and at attribute to be modified and then the manner in which it needs to be modified.</p>
+<p>Right now these are the transforms that can be applied:</p>
+<ul>
+<li><i>lowercase</i> Converts the attribute value to lower case.</li>
+<li><i>replace</i> This performs a string find and replace operation. It takes two parameters, the first is the string to search for and the next one is the string to replace it with.</li></ul>
+<p>Example:</p>
+<p>The example below applies couple of transforms to the the <i>qualifiedName</i> attribute of hive_table. It converts the value to lower case, then searches for 'cl1', if found, replaces it with 'cl2'.</p>
+<p>To use the option, set the contents of <i>importOptions.json</i> to:</p>
+<div class="source"><pre class="prettyprint">
+{
+ "options": {
+ "transforms": "{ \"hive_table\": { \"qualifiedName\": [ \"replace:@cl1:@cl2\" ] }, \"hive_db\": { \"qualifiedName\": [ \"replace:@cl1:@cl2\" ] } }"
+ }
+}
+
+</pre></div>
+<p>Please refer to <a class="externalLink" href="https://issues.apache.org/jira/browse/ATLAS-1825">ATLAS-1825</a> for details scenarios when this option could be used.</p></div>
+<div class="section">
+<h5><a name="Start_Guid_or_Start_Index"></a>Start Guid or Start Index</h5>
+<p>When an import operation is in progress and the server goes down, it would be possible to resume import from the last successfully imported entity. This would allow the import to resume from where it left off.</p>
+<p>Server-side logging is improved to display the detail of the last successfully imported entity, this includes the index within the import list and the entity's guid. Either can be used specify the point to resume import.</p>
+<p>To use the option, set the contents of <i>importOptions.json</i> to:</p>
+<div class="source"><pre class="prettyprint">
+{
+
+ "options": {
+ "startGuid": "bd97c78e-3fa5-4f9c-9f48-3683ca3d1fb1"
+ }
+}
+
+</pre></div>
+<p>To use <i>startPosition</i>, use the following in the <i>importOptions.json</i>:</p>
+<div class="source"><pre class="prettyprint">
+{
+
+ "options": {
+ "startPosition": "332"
+ }
+}
+
+</pre></div>
+<p>Steps to use the behavior:</p>
+<ul>
+<li>Start an import (using the CURL) that is fairly long, say about 1000+ entities.</li>
+<li>While the import is in progress, stop atlas server (using atlas_stop.py).</li>
+<li>From the log file located at <i>/var/log/atlas/application.log</i> get the last successfully imported entity GUID or index position.</li>
+<li>Update the <i>importOptions.json</i> with the guid.</li>
+<li>Restart import.</li></ul></div>
+<div class="section">
+<h5><a name="Optional_Importing_Type_Definition"></a>Optional Importing Type Definition</h5>
+<p>The output of Export has <i>atlas-typedef.json</i> that contains the type definitions for the entities exported.</p>
+<p>By default (that is if no options is specified), the type definitions are imported and applied to the system being imported to. The entity import is performed after this.</p>
+<p>In some cases, you would not want to modify the type definitions. Import may be better off failing than the types be modified.</p>
+<p>This option allows for optionally importing of type definition. The option is set to <i>true</i> by default, which means that type definition is imported. With this option set to <i>false</i>, type definitions preseneraent in the source will not be imported. In case of mismatch between the entities being imported the types present in the system where the import is being performed, the operation will fail.</p>
+<p>Table below enumerates the conditions that get addressed as part of type definition import:</p>
+<p></p>
+<table border="0" class="table table-striped">
+<tr class="a">
+<th>Condition</th>
+<th>Action</th></tr>
+<tr class="b">
+<td>Incoming type does not exist in target system</td>
+<td>Type is created.</td></tr>
+<tr class="a">
+<td>Type to be imported and type in target system are same</td>
+<td>No change</td></tr>
+<tr class="b">
+<td>Type to be imported and type in target system differ by some attributes</td>
+<td>Target system type is updated to the attributes present in the source. It is possible that the target system will have attributes in addition to the one present in the source. In that case, the target system's type attributes will be an union of the attributes. Attributes in target system will not be deleted to match the source. If the type of the attribute differ, import process will be aborted and exception logged.</td></tr></table>To use the option, set the contents of <i>importOptions.json</i> to:
+<div class="source"><pre class="prettyprint">
+{
+
+ "options": {
+ "updateTypeDefinition": true
+ }
+}
+
+</pre></div></div>
+<div class="section">
+<h5><a name="Specifying_File_to_be_Imported_From_Server_Location"></a>Specifying File to be Imported From Server Location</h5>
+<p>In scenario where the file to be imported is present at a location on the server, the <i>importfile</i> API can be used. It behaves like the Import API.</p>
+<p>To use the option, set the contents of <i>importOptions.json</i> to:</p>
+<div class="source"><pre class="prettyprint">
+{
+
+ "options": {
+ "fileName": "/root/fileToBeImported.zip"
+ }
+}
+
+</pre></div>
+<p><i>CURL</i></p>
+<div class="source"><pre class="prettyprint">
+curl -g -X POST -u adminuser:password -H "Content-Type: application/json"
+ -H "Cache-Control: no-cache"
+ -d r@importOptions.json
+ "http://localhost:21000/api/atlas/admin/importfile"
+
+</pre></div></div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container">
+ <div class="row">
+<p><a href="https://www.apache.org/foundation/contributing"><img src="https://www.apache.org/images/SupportApache-small.png" alt="Support the ASF" id="asf-logo" height="20" width="20" /></a>Copyright © 2011-2018 The Apache Software Foundation. Licensed under the <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>.<br/>
+Apache Atlas, Atlas, Apache, the Apache feather logo are trademarks of the <a href="https://www.apache.org">Apache Software Foundation</a>.<br/>
+All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
+ </div>
+ <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a>
+</p>
+ </div>
+ </footer>
+ </body>
+</html>