You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@atlas.apache.org by ni...@apache.org on 2018/11/01 14:40:49 UTC
[43/86] atlas-website git commit: ATLAS-2913 Update Atlas Website
with Release Details 0.8.3
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/HighAvailability.html
----------------------------------------------------------------------
diff --git a/0.8.3/HighAvailability.html b/0.8.3/HighAvailability.html
new file mode 100644
index 0000000..9ea4275
--- /dev/null
+++ b/0.8.3/HighAvailability.html
@@ -0,0 +1,296 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/HighAvailability.twiki at 2018-10-31
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20181031" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Atlas – Fault Tolerance and High Availability Options</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
+ </head>
+ <body class="topBarEnabled">
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container" style="width: 68%;"><div class="nav-collapse">
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="index.html" title="About">About</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="News">News</a></li>
+ <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git" title="Git">Git</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy" title="Powered by">Powered by</a></li>
+ <li><a href="http://blogs.apache.org/atlas/" title="Blog">Blog</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="project-info.html" title="Summary">Summary</a></li>
+ <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li>
+ <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4" title="IRC">IRC</a></li>
+ <li><a href="team-list.html" title="Team">Team</a></li>
+ <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li>
+ <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li>
+ <li><a href="license.html" title="License">License</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/" title="0.8.2">0.8.2</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/" title="0.8.1">0.8.1</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="../index.html" title="latest">latest</a></li>
+ <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a></li>
+ <li><a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li>
+ </ul>
+ </li>
+ </ul>
+<form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" >
+ <input value="http://atlas.apache.org" name="sitesearch" type="hidden"/>
+ <input class="search-query" name="q" id="query" type="text" />
+</form>
+<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script>
+ <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"
+ scrolling="no" frameborder="0"
+ style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe>
+ <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script>
+ <ul class="nav pull-right"><li style="margin-top: 10px;">
+ <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div>
+ </li></ul>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li>
+ <li class="active ">Fault Tolerance and High Availability Options</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li>
+ <li id="projectVersion" class="pull-right">Version: 0.8.3</li>
+ </ul>
+ </div>
+ <div id="bodyColumn" >
+<div class="section">
+<h2><a name="Fault_Tolerance_and_High_Availability_Options"></a>Fault Tolerance and High Availability Options</h2></div>
+<div class="section">
+<h3><a name="Introduction"></a>Introduction</h3>
+<p>Apache Atlas uses and interacts with a variety of systems to provide metadata management and data lineage to data administrators. By choosing and configuring these dependencies appropriately, it is possible to achieve a high degree of service availability with Atlas. This document describes the state of high availability support in Atlas, including its capabilities and current limitations, and also the configuration required for achieving this level of high availability.</p>
+<p><a href="./Architecture.html">The architecture page</a> in the wiki gives an overview of the various components that make up Atlas. The options mentioned below for various components derive context from the above page, and would be worthwhile to review before proceeding to read this page.</p></div>
+<div class="section">
+<h3><a name="Atlas_Web_Service"></a>Atlas Web Service</h3>
+<p>Currently, the Atlas Web Service has a limitation that it can only have one active instance at a time. In earlier releases of Atlas, a backup instance could be provisioned and kept available. However, a manual failover was required to make this backup instance active.</p>
+<p>From this release, Atlas will support multiple instances of the Atlas Web service in an active/passive configuration with automated failover. This means that users can deploy and start multiple instances of the Atlas Web Service on different physical hosts at the same time. One of these instances will be automatically selected as an 'active' instance to service user requests. The others will automatically be deemed 'passive'. If the 'active' instance becomes unavailable either because it is deliberately stopped, or due to unexpected failures, one of the other instances will automatically be elected as an 'active' instance and start to service user requests.</p>
+<p>An 'active' instance is the only instance that can respond to user requests correctly. It can create, delete, modify or respond to queries on metadata objects. A 'passive' instance will accept user requests, but will redirect them using HTTP redirect to the currently known 'active' instance. Specifically, a passive instance will not itself respond to any queries on metadata objects. However, all instances (both active and passive), will respond to admin requests that return information about that instance.</p>
+<p>When configured in a High Availability mode, users can get the following operational benefits:</p>
+<p></p>
+<ul>
+<li><b>Uninterrupted service during maintenance intervals</b>: If an active instance of the Atlas Web Service needs to be brought down for maintenance, another instance would automatically become active and can service requests.</li>
+<li><b>Uninterrupted service in event of unexpected failures</b>: If an active instance of the Atlas Web Service fails due to software or hardware errors, another instance would automatically become active and can service requests.</li></ul>
+<p>In the following sub-sections, we describe the steps required to setup High Availability for the Atlas Web Service. We also describe how the deployment and client can be designed to take advantage of this capability. Finally, we describe a few details of the underlying implementation.</p></div>
+<div class="section">
+<h4><a name="Setting_up_the_High_Availability_feature_in_Atlas"></a>Setting up the High Availability feature in Atlas</h4>
+<p>The following pre-requisites must be met for setting up the High Availability feature.</p>
+<p></p>
+<ul>
+<li>Ensure that you install Apache Zookeeper on a cluster of machines (a minimum of 3 servers is recommended for production).</li>
+<li>Select 2 or more physical machines to run the Atlas Web Service instances on. These machines define what we refer to as a 'server ensemble' for Atlas.</li></ul>
+<p>To setup High Availability in Atlas, a few configuration options must be defined in the <tt>atlas-application.properties</tt> file. While the complete list of configuration items are defined in the <a href="./Configuration.html">Configuration Page</a>, this section lists a few of the main options.</p>
+<p></p>
+<ul>
+<li>High Availability is an optional feature in Atlas. Hence, it must be enabled by setting the configuration option <tt>atlas.server.ha.enabled</tt> to true.</li>
+<li>Next, define a list of identifiers, one for each physical machine you have selected for the Atlas Web Service instance. These identifiers can be simple strings like <tt>id1</tt>, <tt>id2</tt> etc. They should be unique and should not contain a comma.</li>
+<li>Define a comma separated list of these identifiers as the value of the option <tt>atlas.server.ids</tt>.</li>
+<li>For each physical machine, list the IP Address/hostname and port as the value of the configuration <tt>atlas.server.address.id</tt>, where <tt>id</tt> refers to the identifier string for this physical machine.
+<ul>
+<li>For e.g., if you have selected 2 machines with hostnames <tt>host1.company.com</tt> and <tt>host2.company.com</tt>, you can define the configuration options as below:</li></ul></li></ul>
+<div class="source"><pre class="prettyprint">
+ atlas.server.ids=id1,id2
+ atlas.server.address.id1=host1.company.com:21000
+ atlas.server.address.id2=host2.company.com:21000
+
+</pre></div>
+<p></p>
+<ul>
+<li>Define the Zookeeper quorum which will be used by the Atlas High Availability feature.</li></ul>
+<div class="source"><pre class="prettyprint">
+ atlas.server.ha.zookeeper.connect=zk1.company.com:2181,zk2.company.com:2181,zk3.company.com:2181
+
+</pre></div>
+<p></p>
+<ul>
+<li>You can review other configuration options that are defined for the High Availability feature, and set them up as desired in the <tt>atlas-application.properties</tt> file.</li>
+<li>For production environments, the components that Atlas depends on must also be set up in High Availability mode. This is described in detail in the following sections. Follow those instructions to setup and configure them.</li>
+<li>Install the Atlas software on the selected physical machines.</li>
+<li>Copy the <tt>atlas-application.properties</tt> file created using the steps above to the configuration directory of all the machines.</li>
+<li>Start the dependent components.</li>
+<li>Start each instance of the Atlas Web Service.</li></ul>
+<p>To verify that High Availability is working, run the following script on each of the instances where Atlas Web Service is installed.</p>
+<div class="source"><pre class="prettyprint">
+$ATLAS_HOME/bin/atlas_admin.py -status
+
+</pre></div>
+<p>This script can print one of the values below as response:</p>
+<p></p>
+<ul>
+<li><b>ACTIVE</b>: This instance is active and can respond to user requests.</li>
+<li><b>PASSIVE</b>: This instance is PASSIVE. It will redirect any user requests it receives to the current active instance.</li>
+<li><b>BECOMING_ACTIVE</b>: This would be printed if the server is transitioning to become an ACTIVE instance. The server cannot service any metadata user requests in this state.</li>
+<li><b>BECOMING_PASSIVE</b>: This would be printed if the server is transitioning to become a PASSIVE instance. The server cannot service any metadata user requests in this state.</li></ul>
+<p>Under normal operating circumstances, only one of these instances should print the value <b>ACTIVE</b> as response to the script, and the others would print <b>PASSIVE</b>.</p></div>
+<div class="section">
+<h4><a name="Configuring_clients_to_use_the_High_Availability_feature"></a>Configuring clients to use the High Availability feature</h4>
+<p>The Atlas Web Service can be accessed in two ways:</p>
+<p></p>
+<ul>
+<li><b>Using the Atlas Web UI</b>: This is a browser based client that can be used to query the metadata stored in Atlas.</li>
+<li><b>Using the Atlas REST API</b>: As Atlas exposes a RESTful API, one can use any standard REST client including libraries in other applications. In fact, Atlas ships with a client called AtlasClient that can be used as an example to build REST client access.</li></ul>
+<p>In order to take advantage of the High Availability feature in the clients, there are two options possible.</p></div>
+<div class="section">
+<h5><a name="Using_an_intermediate_proxy"></a>Using an intermediate proxy</h5>
+<p>The simplest solution to enable highly available access to Atlas is to install and configure some intermediate proxy that has a capability to transparently switch services based on status. One such proxy solution is <a class="externalLink" href="http://www.haproxy.org/">HAProxy</a>.</p>
+<p>Here is an example HAProxy configuration that can be used. Note this is provided for illustration only, and not as a recommended production configuration. For that, please refer to the HAProxy documentation for appropriate instructions.</p>
+<div class="source"><pre class="prettyprint">
+frontend atlas_fe
+ bind *:41000
+ default_backend atlas_be
+
+backend atlas_be
+ mode http
+ option httpchk get /api/atlas/admin/status
+ http-check expect string ACTIVE
+ balance roundrobin
+ server host1_21000 host1:21000 check
+ server host2_21000 host2:21000 check backup
+
+listen atlas
+ bind localhost:42000
+
+</pre></div>
+<p>The above configuration binds HAProxy to listen on port 41000 for incoming client connections. It then routes the connections to either of the hosts host1 or host2 depending on a HTTP status check. The status check is done using a HTTP GET on the REST URL <tt>/api/atlas/admin/status</tt>, and is deemed successful only if the HTTP response contains the string ACTIVE.</p></div>
+<div class="section">
+<h5><a name="Using_automatic_detection_of_active_instance"></a>Using automatic detection of active instance</h5>
+<p>If one does not want to setup and manage a separate proxy, then the other option to use the High Availability feature is to build a client application that is capable of detecting status and retrying operations. In such a setting, the client application can be launched with the URLs of all Atlas Web Service instances that form the ensemble. The client should then call the REST URL <tt>/api/atlas/admin/status</tt> on each of these to determine which is the active instance. The response from the Active instance would be of the form <tt>{Status:ACTIVE}</tt>. Also, when the client faces any exceptions in the course of an operation, it should again determine which of the remaining URLs is active and retry the operation.</p>
+<p>The AtlasClient class that ships with Atlas can be used as an example client library that implements the logic for working with an ensemble and selecting the right Active server instance.</p>
+<p>Utilities in Atlas, like <tt>quick_start.py</tt> and <tt>import-hive.sh</tt> can be configured to run with multiple server URLs. When launched in this mode, the AtlasClient automatically selects and works with the current active instance. If a proxy is set up in between, then its address can be used when running quick_start.py or import-hive.sh.</p></div>
+<div class="section">
+<h4><a name="Implementation_Details_of_Atlas_High_Availability"></a>Implementation Details of Atlas High Availability</h4>
+<p>The Atlas High Availability work is tracked under the master JIRA <a class="externalLink" href="https://issues.apache.org/jira/browse/ATLAS-510">ATLAS-510</a>. The JIRAs filed under it have detailed information about how the High Availability feature has been implemented. At a high level the following points can be called out:</p>
+<p></p>
+<ul>
+<li>The automatic selection of an Active instance, as well as automatic failover to a new Active instance happen through a leader election algorithm.</li>
+<li>For leader election, we use the <a class="externalLink" href="http://curator.apache.org/curator-recipes/leader-latch.html">Leader Latch Recipe</a> of <a class="externalLink" href="http://curator.apache.org">Apache Curator</a>.</li>
+<li>The Active instance is the only one which initializes, modifies or reads state in the backend stores to keep them consistent.</li>
+<li>Also, when an instance is elected as Active, it refreshes any cached information from the backend stores to get up to date.</li>
+<li>A servlet filter ensures that only the active instance services user requests. If a passive instance receives these requests, it automatically redirects them to the current active instance.</li></ul></div>
+<div class="section">
+<h3><a name="Metadata_Store"></a>Metadata Store</h3>
+<p>As described above, Atlas uses Titan to store the metadata it manages. By default, Atlas uses a standalone HBase instance as the backing store for Titan. In order to provide HA for the metadata store, we recommend that Atlas be configured to use distributed HBase as the backing store for Titan. Doing this implies that you could benefit from the HA guarantees HBase provides. In order to configure Atlas to use HBase in HA mode, do the following:</p>
+<p></p>
+<ul>
+<li>Choose an existing HBase cluster that is set up in HA mode to configure in Atlas (OR) Set up a new HBase cluster in <a class="externalLink" href="http://hbase.apache.org/book.html#quickstart_fully_distributed">HA mode</a>.
+<ul>
+<li>If setting up HBase for Atlas, please following instructions listed for setting up HBase in the <a href="./InstallationSteps.html">Installation Steps</a>.</li></ul></li>
+<li>We recommend using more than one HBase masters (at least 2) in the cluster on different physical hosts that use Zookeeper for coordination to provide redundancy and high availability of HBase.
+<ul>
+<li>Refer to the <a href="./Configuration.html">Configuration page</a> for the options to configure in atlas.properties to setup Atlas with HBase.</li></ul></li></ul></div>
+<div class="section">
+<h3><a name="Index_Store"></a>Index Store</h3>
+<p>As described above, Atlas indexes metadata through Titan to support full text search queries. In order to provide HA for the index store, we recommend that Atlas be configured to use Solr as the backing index store for Titan. In order to configure Atlas to use Solr in HA mode, do the following:</p>
+<p></p>
+<ul>
+<li>Choose an existing SolrCloud cluster setup in HA mode to configure in Atlas (OR) Set up a new <a class="externalLink" href="https://cwiki.apache.org/confluence/display/solr/SolrCloud">SolrCloud cluster</a>.
+<ul>
+<li>Ensure Solr is brought up on at least 2 physical hosts for redundancy, and each host runs a Solr node.</li>
+<li>We recommend the number of replicas to be set to at least 2 for redundancy.</li></ul></li>
+<li>Create the SolrCloud collections required by Atlas, as described in <a href="./InstallationSteps.html">Installation Steps</a></li>
+<li>Refer to the <a href="./Configuration.html">Configuration page</a> for the options to configure in atlas.properties to setup Atlas with Solr.</li></ul></div>
+<div class="section">
+<h3><a name="Notification_Server"></a>Notification Server</h3>
+<p>Metadata notification events from Hooks are sent to Atlas by writing them to a Kafka topic called <b>ATLAS_HOOK</b>. Similarly, events from Atlas to other integrating components like Ranger, are written to a Kafka topic called <b>ATLAS_ENTITIES</b>. Since Kafka persists these messages, the events will not be lost even if the consumers are down as the events are being sent. In addition, we recommend Kafka is also setup for fault tolerance so that it has higher availability guarantees. In order to configure Atlas to use Kafka in HA mode, do the following:</p>
+<p></p>
+<ul>
+<li>Choose an existing Kafka cluster set up in HA mode to configure in Atlas (OR) Set up a new Kafka cluster.</li>
+<li>We recommend that there are more than one Kafka brokers in the cluster on different physical hosts that use Zookeeper for coordination to provide redundancy and high availability of Kafka.
+<ul>
+<li>Setup at least 2 physical hosts for redundancy, each hosting a Kafka broker.</li></ul></li>
+<li>Set up Kafka topics for Atlas usage:
+<ul>
+<li>The number of partitions for the ATLAS topics should be set to 1 (numPartitions)</li>
+<li>Decide number of replicas for Kafka topic: Set this to at least 2 for redundancy.</li>
+<li>Run the following commands:</li></ul></li></ul>
+<div class="source"><pre class="prettyprint">
+ $KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper <list of zookeeper host:port entries> --topic ATLAS_HOOK --replication-factor <numReplicas> --partitions 1
+ $KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper <list of zookeeper host:port entries> --topic ATLAS_ENTITIES --replication-factor <numReplicas> --partitions 1
+ Here KAFKA_HOME points to the Kafka installation directory.
+
+</pre></div>
+<p></p>
+<ul>
+<li>In atlas-application.properties, set the following configuration:</li></ul>
+<div class="source"><pre class="prettyprint">
+ atlas.notification.embedded=false
+ atlas.kafka.zookeeper.connect=<comma separated list of servers forming Zookeeper quorum used by Kafka>
+ atlas.kafka.bootstrap.servers=<comma separated list of Kafka broker endpoints in host:port form> - Give at least 2 for redundancy.
+
+</pre></div></div>
+<div class="section">
+<h3><a name="Known_Issues"></a>Known Issues</h3>
+<p></p>
+<ul>
+<li>If the HBase region servers hosting the Atlas ‘titan’ HTable are down, Atlas would not be able to store or retrieve metadata from HBase until they are brought back online.</li></ul></div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container">
+ <div class="row">
+Copyright © 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.
+ </div>
+ <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a>
+</p>
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/Import-API-Options.html
----------------------------------------------------------------------
diff --git a/0.8.3/Import-API-Options.html b/0.8.3/Import-API-Options.html
new file mode 100644
index 0000000..1f892ff
--- /dev/null
+++ b/0.8.3/Import-API-Options.html
@@ -0,0 +1,259 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Import-API-Options.twiki at 2018-10-31
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20181031" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Atlas – Import API Options</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
+ </head>
+ <body class="topBarEnabled">
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container" style="width: 68%;"><div class="nav-collapse">
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="index.html" title="About">About</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="News">News</a></li>
+ <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git" title="Git">Git</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy" title="Powered by">Powered by</a></li>
+ <li><a href="http://blogs.apache.org/atlas/" title="Blog">Blog</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="project-info.html" title="Summary">Summary</a></li>
+ <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li>
+ <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4" title="IRC">IRC</a></li>
+ <li><a href="team-list.html" title="Team">Team</a></li>
+ <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li>
+ <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li>
+ <li><a href="license.html" title="License">License</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/" title="0.8.2">0.8.2</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/" title="0.8.1">0.8.1</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="../index.html" title="latest">latest</a></li>
+ <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a></li>
+ <li><a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li>
+ </ul>
+ </li>
+ </ul>
+<form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" >
+ <input value="http://atlas.apache.org" name="sitesearch" type="hidden"/>
+ <input class="search-query" name="q" id="query" type="text" />
+</form>
+<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script>
+ <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"
+ scrolling="no" frameborder="0"
+ style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe>
+ <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script>
+ <ul class="nav pull-right"><li style="margin-top: 10px;">
+ <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div>
+ </li></ul>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li>
+ <li class="active ">Import API Options</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li>
+ <li id="projectVersion" class="pull-right">Version: 0.8.3</li>
+ </ul>
+ </div>
+ <div id="bodyColumn" >
+<div class="section">
+<h2><a name="Import_API_Options"></a>Import API Options</h2>
+<p>Import API options are specified as <i>options</i> JSON. Since the API accepts multi-part form data, it is possible to sepecify multipls input streams within the CURL call.</p></div>
+<div class="section">
+<h4><a name="Examples_Using_CURL_Calls"></a>Examples Using CURL Calls</h4>
+<div class="source"><pre class="prettyprint">
+curl -g -X POST -u adminuser:password -H "Content-Type: multipart/form-data"
+ -H "Cache-Control: no-cache"
+ -F request=@importOptions.json
+ -F data=@quickStartDB.zip
+ "http://localhost:21000/api/atlas/admin/import"
+
+</pre></div>
+<p>To use the defaults, set the contents of <i>importOptions.json</i> to:</p>
+<div class="source"><pre class="prettyprint">
+{
+ "options": {
+ }
+}
+
+</pre></div></div>
+<div class="section">
+<h4><a name="Options"></a>Options</h4>
+<p>Following options are supported for Import process:</p>
+<p></p>
+<ul>
+<li>Specify transforms during import operation.</li>
+<li>Resume import by specifying starting entity guid.</li>
+<li>Optionally import type definition.</li></ul></div>
+<div class="section">
+<h5><a name="Transforms"></a>Transforms</h5>
+<p>During the import process, the attribute value of the incoming entity can be changed.</p>
+<p>This is possible by specifying entity type and at attribute to be modified and then the manner in which it needs to be modified.</p>
+<p>Right now these are the transforms that can be applied:</p>
+<ul>
+<li><i>lowercase</i> Converts the attribute value to lower case.</li>
+<li><i>replace</i> This performs a string find and replace operation. It takes two parameters, the first is the string to search for and the next one is the string to replace it with.</li></ul>
+<p>Example:</p>
+<p>The example below applies couple of transforms to the the <i>qualifiedName</i> attribute of hive_table. It converts the value to lower case, then searches for 'cl1', if found, replaces it with 'cl2'.</p>
+<p>To use the option, set the contents of <i>importOptions.json</i> to:</p>
+<div class="source"><pre class="prettyprint">
+{
+ "options": {
+ "transforms": "{ \"hive_table\": { \"qualifiedName\": [ \"replace:@cl1:@cl2\" ] }, \"hive_db\": { \"qualifiedName\": [ \"replace:@cl1:@cl2\" ] } }"
+ }
+}
+
+</pre></div>
+<p>Please refer to <a class="externalLink" href="https://issues.apache.org/jira/browse/ATLAS-1825">ATLAS-1825</a> for details scenarios when this option could be used.</p></div>
+<div class="section">
+<h5><a name="Start_Guid_or_Start_Index"></a>Start Guid or Start Index</h5>
+<p>When an import operation is in progress and the server goes down, it would be possible to resume import from the last successfully imported entity. This would allow the import to resume from where it left off.</p>
+<p>Server-side logging is improved to display the detail of the last successfully imported entity, this includes the index within the import list and the entity's guid. Either can be used specify the point to resume import.</p>
+<p>To use the option, set the contents of <i>importOptions.json</i> to:</p>
+<div class="source"><pre class="prettyprint">
+{
+
+ "options": {
+ "startGuid": "bd97c78e-3fa5-4f9c-9f48-3683ca3d1fb1"
+ }
+}
+
+</pre></div>
+<p>To use <i>startPosition</i>, use the following in the <i>importOptions.json</i>:</p>
+<div class="source"><pre class="prettyprint">
+{
+
+ "options": {
+ "startPosition": "332"
+ }
+}
+
+</pre></div>
+<p>Steps to use the behavior:</p>
+<ul>
+<li>Start an import (using the CURL) that is fairly long, say about 1000+ entities.</li>
+<li>While the import is in progress, stop atlas server (using atlas_stop.py).</li>
+<li>From the log file located at <i>/var/log/atlas/application.log</i> get the last successfully imported entity GUID or index position.</li>
+<li>Update the <i>importOptions.json</i> with the guid.</li>
+<li>Restart import.</li></ul></div>
+<div class="section">
+<h5><a name="Optional_Importing_Type_Definition"></a>Optional Importing Type Definition</h5>
+<p>The output of Export has <i>atlas-typedef.json</i> that contains the type definitions for the entities exported.</p>
+<p>By default (that is if no options is specified), the type definitions are imported and applied to the system being imported to. The entity import is performed after this.</p>
+<p>In some cases, you would not want to modify the type definitions. Import may be better off failing than the types be modified.</p>
+<p>This option allows for optionally importing of type definition. The option is set to <i>true</i> by default, which means that type definition is imported. With this option set to <i>false</i>, type definitions preseneraent in the source will not be imported. In case of mismatch between the entities being imported the types present in the system where the import is being performed, the operation will fail.</p>
+<p>Table below enumerates the conditions that get addressed as part of type definition import:</p>
+<p></p>
+<table border="0" class="table table-striped">
+<tr class="a">
+<th>Condition</th>
+<th>Action</th></tr>
+<tr class="b">
+<td>Incoming type does not exist in target system</td>
+<td>Type is created.</td></tr>
+<tr class="a">
+<td>Type to be imported and type in target system are same</td>
+<td>No change</td></tr>
+<tr class="b">
+<td>Type to be imported and type in target system differ by some attributes</td>
+<td>Target system type is updated to the attributes present in the source. It is possible that the target system will have attributes in addition to the one present in the source. In that case, the target system's type attributes will be an union of the attributes. Attributes in target system will not be deleted to match the source. If the type of the attribute differ, import process will be aborted and exception logged.</td></tr></table>To use the option, set the contents of <i>importOptions.json</i> to:
+<div class="source"><pre class="prettyprint">
+{
+
+ "options": {
+ "updateTypeDefinition": true
+ }
+}
+
+</pre></div></div>
+<div class="section">
+<h5><a name="Specifying_File_to_be_Imported_From_Server_Location"></a>Specifying File to be Imported From Server Location</h5>
+<p>In scenario where the file to be imported is present at a location on the server, the <i>importfile</i> API can be used. It behaves like the Import API.</p>
+<p>To use the option, set the contents of <i>importOptions.json</i> to:</p>
+<div class="source"><pre class="prettyprint">
+{
+
+ "options": {
+ "fileName": "/root/fileToBeImported.zip"
+ }
+}
+
+</pre></div>
+<p><i>CURL</i></p>
+<div class="source"><pre class="prettyprint">
+curl -g -X POST -u adminuser:password -H "Content-Type: application/json"
+ -H "Cache-Control: no-cache"
+ -d r@importOptions.json
+ "http://localhost:21000/api/atlas/admin/importfile"
+
+</pre></div></div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container">
+ <div class="row">
+Copyright © 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.
+ </div>
+ <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a>
+</p>
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/Import-API.html
----------------------------------------------------------------------
diff --git a/0.8.3/Import-API.html b/0.8.3/Import-API.html
new file mode 100644
index 0000000..e02dd93
--- /dev/null
+++ b/0.8.3/Import-API.html
@@ -0,0 +1,285 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Import-API.twiki at 2018-10-31
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20181031" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Atlas – Import API</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
+ </head>
+ <body class="topBarEnabled">
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container" style="width: 68%;"><div class="nav-collapse">
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="index.html" title="About">About</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="News">News</a></li>
+ <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git" title="Git">Git</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy" title="Powered by">Powered by</a></li>
+ <li><a href="http://blogs.apache.org/atlas/" title="Blog">Blog</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="project-info.html" title="Summary">Summary</a></li>
+ <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li>
+ <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4" title="IRC">IRC</a></li>
+ <li><a href="team-list.html" title="Team">Team</a></li>
+ <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li>
+ <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li>
+ <li><a href="license.html" title="License">License</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/" title="0.8.2">0.8.2</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/" title="0.8.1">0.8.1</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="../index.html" title="latest">latest</a></li>
+ <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a></li>
+ <li><a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li>
+ </ul>
+ </li>
+ </ul>
+<form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" >
+ <input value="http://atlas.apache.org" name="sitesearch" type="hidden"/>
+ <input class="search-query" name="q" id="query" type="text" />
+</form>
+<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script>
+ <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"
+ scrolling="no" frameborder="0"
+ style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe>
+ <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script>
+ <ul class="nav pull-right"><li style="margin-top: 10px;">
+ <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div>
+ </li></ul>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li>
+ <li class="active ">Import API</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li>
+ <li id="projectVersion" class="pull-right">Version: 0.8.3</li>
+ </ul>
+ </div>
+ <div id="bodyColumn" >
+<div class="section">
+<h2><a name="Import_API"></a>Import API</h2>
+<p>The general approach is:</p>
+<ul>
+<li>Consumer makes a ZIP file available for import operation. See details below for the 2 flavors of the API.</li>
+<li>The API if successful, will return the results of the operation.</li>
+<li>Error will be returned on failure of the call.</li></ul></div>
+<div class="section">
+<h4><a name="Import_ZIP_File_Using_POST"></a>Import ZIP File Using POST</h4>
+<p></p>
+<table border="0" class="table table-striped">
+<tr class="a">
+<th>Title</th>
+<th>Import API</th></tr>
+<tr class="b">
+<td><i>Example</i></td>
+<td>See Examples sections below.</td></tr>
+<tr class="a">
+<td><i>Description</i></td>
+<td>Provide the contents of the file to be imported in the request body.</td></tr>
+<tr class="b">
+<td><i>URL</i></td>
+<td><i>api/atlas/admin/import</i></td></tr>
+<tr class="a">
+<td><i>Method</i></td>
+<td><i>POST</i></td></tr>
+<tr class="b">
+<td><i>URL Parameters</i></td>
+<td><i>None</i></td></tr>
+<tr class="a">
+<td><i>Data Parameters</i></td>
+<td><i>None</i></td></tr>
+<tr class="b">
+<td><i>Success Response</i></td>
+<td><i>AtlasImporResult</i> is returned as JSON. See details below.</td></tr>
+<tr class="a">
+<td><i>Error Response</i></td>
+<td>Errors that are handled within the system will be returned as <i>AtlasBaseException</i>.</td></tr></table></div>
+<div class="section">
+<h4><a name="Import_ZIP_File_Available_on_Server"></a>Import ZIP File Available on Server</h4>
+<p></p>
+<table border="0" class="table table-striped">
+<tr class="a">
+<th>Title</th>
+<th>Import API</th></tr>
+<tr class="b">
+<td><i>Example</i></td>
+<td>See Examples sections below.</td></tr>
+<tr class="a">
+<td><i>Description</i></td>
+<td>Provide the path of the file to be imported.</td></tr>
+<tr class="b">
+<td><i>URL</i></td>
+<td><i>api/atlas/admin/importfile</i></td></tr>
+<tr class="a">
+<td><i>Method</i></td>
+<td><i>POST</i></td></tr>
+<tr class="b">
+<td><i>URL Parameters</i></td>
+<td><i>None</i></td></tr>
+<tr class="a">
+<td><i>Data Parameters</i></td>
+<td><i>None</i></td></tr>
+<tr class="b">
+<td><i>Success Response</i></td>
+<td><i>AtlasImporResult</i> is returned as JSON. See details below.</td></tr>
+<tr class="a">
+<td><i>Error Response</i></td>
+<td>Errors that are handled within the system will be returned as <i>AtlasBaseException</i>.</td></tr>
+<tr class="b">
+<td><i>Notes</i></td>
+<td>The file to be imported needs to be present on the server at the location specified by the <i>FILENAME</i> parameter.</td></tr></table><b><i>Method Signature for Import</i></b>
+<div class="source"><pre class="prettyprint">
+@POST
+@Path("/import")
+@Produces("application/json; charset=UTF-8")
+@Consumes("multipart/form-data")
+
+</pre></div>
+<p><b><i>Method Signature for Import File</i></b></p>
+<div class="source"><pre class="prettyprint">
+@POST
+@Path("/importfile")
+@Produces("application/json; charset=UTF-8")
+@Consumes("application/json")
+
+</pre></div>
+<p><b><i>Import Options</i></b> Please see <b><i><a href="./Import-API-Options.html">here</a></i></b> for the available options during import process.</p>
+<p><b><i>AtlasImportResult Response</i></b> The API will return the results of the import operation in the format defined by the <i>AtlasImportResult</i>:</p>
+<ul>
+<li><i>AtlasImportParameters</i>: This contains a collection of name value pair of the options that are applied during the import operation.</li>
+<li><i>Metrics</i>: Operation metrics. These include details on the number of types imported, number of entities imported, etc.</li>
+<li><i>Processed Entities</i>: Contains list of GUIDs for the entities that were processed.</li>
+<li><i>Operation Status</i>: Overall status of the operation. Values are <i>SUCCESS</i>, PARTIAL_SUCCESS, <i>FAIL</i>.</li></ul></div>
+<div class="section">
+<h4><a name="Examples_Using_CURL_Calls"></a>Examples Using CURL Calls</h4>
+<p>The call below performs Import of <i>QuickStart</i> database using POST.</p>
+<div class="source"><pre class="prettyprint">
+curl -g -X POST -u adminuser:password -H "Content-Type: multipart/form-data"
+ -H "Cache-Control: no-cache"
+ -F request=@importOptions.json
+ -F data=@quickStartDB.zip
+ "http://localhost:21000/api/atlas/admin/import"
+
+</pre></div>
+<p>The <i>request</i> parameter is optional. If import has to be run without any options use:</p>
+<div class="source"><pre class="prettyprint">
+curl -g -X POST -u adminuser:password -H "Content-Type: multipart/form-data"
+ -H "Cache-Control: no-cache"
+ -F data=@quickStartDB.zip
+ "http://localhost:21000/api/atlas/admin/import"
+
+</pre></div>
+<p>The call below performs Import of <i>QuickStart</i> database using a ZIP file available on server.</p>
+<div class="source"><pre class="prettyprint">
+curl -X POST -u adminuser:password -H "Cache-Control: no-cache" -d ./importOptions.json
+"http://localhost:21000/api/atlas/admin/importFile" > quickStartDB-import-result.json
+
+</pre></div>
+<p>Below is the <i>AtlasImportResult</i> JSON for an import that contains <i>hive</i>db_.</p>
+<p>The <i>processedEntities</i> contains the <i>guids</i> of all the entities imported.</p>
+<p>The <i>metrics</i> contain a breakdown of the types and entities imported along with the operation performed on them viz. <i>created</i> or <i>updated</i>.</p>
+<div class="source"><pre class="prettyprint">
+{
+ "request": {
+ "options": {}
+ },
+ "userName": "admin",
+ "clientIpAddress": "10.0.2.2",
+ "hostName": "10.0.2.15",
+ "timeStamp": 1491285622823,
+ "metrics": {
+ "duration": 9143,
+ "typedef:enum": 0,
+ "typedef:struct": 0,
+ "entity:hive_column:created": 461,
+ "entity:hive_storagedesc:created": 20,
+ "entity:hive_process:created": 12,
+ "entity:hive_db:created": 5,
+ "entity:hive_table:created": 20,
+ "entity:hdfs_path:created": 2,
+ "typedef:entitydef": 0,
+ "typedef:classification": 3
+ },
+ "processedEntities": [
+ "2c4aa713-030b-4fb3-98b1-1cab23d9ac81",
+ "e4aa71ed-70fd-4fa7-9dfb-8250a573e293",
+
+ ...
+
+ "ea0f9bdb-1dfc-4e48-9848-a006129929f9",
+ "b5e2cb41-3e7d-4468-84e1-d87c320e75f9"
+ ],
+ "operationStatus": "SUCCESS"
+}
+
+</pre></div></div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container">
+ <div class="row">
+Copyright © 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.
+ </div>
+ <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a>
+</p>
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/Import-Export-API.html
----------------------------------------------------------------------
diff --git a/0.8.3/Import-Export-API.html b/0.8.3/Import-Export-API.html
new file mode 100644
index 0000000..a3f4fb1
--- /dev/null
+++ b/0.8.3/Import-Export-API.html
@@ -0,0 +1,160 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Import-Export-API.twiki at 2018-10-31
+ | Rendered using Apache Maven Fluido Skin 1.7
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20181031" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Apache Atlas – Export & Import REST APIs</title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script>
+ </head>
+ <body class="topBarEnabled">
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container" style="width: 68%;"><div class="nav-collapse">
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="index.html" title="About">About</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="Wiki">Wiki</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS" title="News">News</a></li>
+ <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git" title="Git">Git</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/ATLAS" title="Jira">Jira</a></li>
+ <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy" title="Powered by">Powered by</a></li>
+ <li><a href="http://blogs.apache.org/atlas/" title="Blog">Blog</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="project-info.html" title="Summary">Summary</a></li>
+ <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li>
+ <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4" title="IRC">IRC</a></li>
+ <li><a href="team-list.html" title="Team">Team</a></li>
+ <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li>
+ <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li>
+ <li><a href="license.html" title="License">License</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/" title="0.8.2">0.8.2</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/" title="0.8.1">0.8.1</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="../index.html" title="latest">latest</a></li>
+ <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li>
+ <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li>
+ <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li>
+ <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li>
+ <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li>
+ <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li>
+ <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html" title="How Apache Works">How Apache Works</a></li>
+ <li><a href="http://www.apache.org/foundation/" title="Foundation">Foundation</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html" title="Sponsoring Apache">Sponsoring Apache</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li>
+ </ul>
+ </li>
+ </ul>
+<form id="search-form" action="https://www.google.com/search" method="get" class="navbar-search pull-right" >
+ <input value="http://atlas.apache.org" name="sitesearch" type="hidden"/>
+ <input class="search-query" name="q" id="query" type="text" />
+</form>
+<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script>
+ <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"
+ scrolling="no" frameborder="0"
+ style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe>
+ <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script>
+ <ul class="nav pull-right"><li style="margin-top: 10px;">
+ <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs" data-size="medium" width="60px" align="right" ></div>
+ </li></ul>
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="container">
+ <div id="banner">
+ <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div>
+ <div class="pull-right"></div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+ <li class=""><a href="http://www.apache.org" class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li>
+ <li class="active ">Export & Import REST APIs</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li>
+ <li id="projectVersion" class="pull-right">Version: 0.8.3</li>
+ </ul>
+ </div>
+ <div id="bodyColumn" >
+<div class="section">
+<h2><a name="Export_.26_Import_REST_APIs"></a>Export & Import REST APIs</h2></div>
+<div class="section">
+<h4><a name="Background"></a>Background</h4>
+<p>The Import-Export APIs for Atlas facilitate transfer of data to and from a cluster that has Atlas provisioned.</p>
+<p>The APIs when integrated with backup and/or disaster recovery process will ensure participation of Atlas.</p></div>
+<div class="section">
+<h4><a name="Introduction"></a>Introduction</h4>
+<p>There are 2 broad categories viz. Export & Import. The details of the APIs are discussed below.</p>
+<p>The APIs are available only to <i>admin</i> user.</p>
+<p>Only a single import or export operation can be performed at a given time. The operations have a potential for generating large amount. They can also put pressure on resources. This restriction tries to alleviate this problem.</p>
+<p>For Import-Export APIs relating to HDFS path, can be found <a href="./Import-Export-HDFS-Path.html">here</a>.</p>
+<p>For additional information please refer to the following:</p>
+<ul>
+<li><a class="externalLink" href="https://issues.apache.org/jira/browse/ATLAS-1503">ATLAS-1503</a> Original Import-Export API requirements.</li>
+<li><a class="externalLink" href="https://issues.apache.org/jira/browse/ATLAS-1618">ATLAS-1618</a> Export API Scope Specification.</li></ul></div>
+<div class="section">
+<h4><a name="Errors"></a>Errors</h4>
+<p>If an import or export operation is initiated while another is in progress, the consumer will receive this error:</p>
+<div class="source"><pre class="prettyprint">
+"ATLAS5005E": "Another import or export is in progress. Please try again."
+
+</pre></div>
+<p>Unhandled errors will be returned as Internal error code 500.</p></div>
+<div class="section">
+<h3><a name="REST_API_Reference"></a>REST API Reference</h3>
+<p></p>
+<ul>
+<li><b><i><a href="./Export-API.html">Export</a></i></b></li>
+<li><b><i><a href="./Export-HDFS-API.html">Export HDFS</a></i></b></li>
+<li><b><i><a href="./Import-API.html">Import</a></i></b></li>
+<li><b><i><a href="./Import-API-Options.html">Import Options</a></i></b></li></ul></div>
+ </div>
+ </div>
+ <hr/>
+ <footer>
+ <div class="container">
+ <div class="row">
+Copyright © 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.
+ </div>
+ <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a>
+</p>
+ </div>
+ </footer>
+ </body>
+</html>