You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@atlas.apache.org by ma...@apache.org on 2018/02/01 01:57:54 UTC
atlas git commit: ATLAS-2428: updated docs for 0.8.2 release
Repository: atlas
Updated Branches:
refs/heads/branch-0.8 15cd2bbe9 -> dabb9f4ad
ATLAS-2428: updated docs for 0.8.2 release
Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/dabb9f4a
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/dabb9f4a
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/dabb9f4a
Branch: refs/heads/branch-0.8
Commit: dabb9f4ade00e8ff3939eb459262234e6f1d38ba
Parents: 15cd2bb
Author: Madhan Neethiraj <ma...@apache.org>
Authored: Wed Jan 31 11:45:50 2018 -0800
Committer: Madhan Neethiraj <ma...@apache.org>
Committed: Wed Jan 31 17:56:51 2018 -0800
----------------------------------------------------------------------
docs/pom.xml | 2 +-
docs/src/site/site.xml | 5 +-
docs/src/site/twiki/Architecture.twiki | 9 +-
docs/src/site/twiki/Bridge-Falcon.twiki | 56 ++--
docs/src/site/twiki/Bridge-Hive.twiki | 117 ++++-----
docs/src/site/twiki/Bridge-Sqoop.twiki | 45 ++--
docs/src/site/twiki/Configuration.twiki | 223 ++++------------
docs/src/site/twiki/EclipseSetup.twiki | 2 +-
docs/src/site/twiki/InstallationSteps.twiki | 312 +++++++++--------------
docs/src/site/twiki/QuickStart.twiki | 7 +-
docs/src/site/twiki/Repository.twiki | 4 -
docs/src/site/twiki/TypeSystem.twiki | 206 +++++++--------
docs/src/site/twiki/index.twiki | 46 ++--
13 files changed, 427 insertions(+), 607 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/pom.xml
----------------------------------------------------------------------
diff --git a/docs/pom.xml b/docs/pom.xml
index e2b6e30..10d70ca 100755
--- a/docs/pom.xml
+++ b/docs/pom.xml
@@ -83,7 +83,7 @@
</dependency>
</dependencies>
<configuration>
- <port>9090</port>
+ <port>8080</port>
</configuration>
<executions>
<execution>
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/site.xml
----------------------------------------------------------------------
diff --git a/docs/src/site/site.xml b/docs/src/site/site.xml
index c4288fd..50d015b 100755
--- a/docs/src/site/site.xml
+++ b/docs/src/site/site.xml
@@ -81,8 +81,10 @@
</menu>
<menu name="Releases">
+ <item name="0.8.2"
+ href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"/>
<item name="0.8.1"
- href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.1/"/>
+ href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"/>
<item name="0.8-incubating"
href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"/>
<item name="0.7.1-incubating"
@@ -97,6 +99,7 @@
<menu name="Documentation">
<item name="latest" href="../index.html"/>
+ <item name="0.8.2" href="../0.8.2/index.html"/>
<item name="0.8.1" href="../0.8.1/index.html"/>
<item name="0.8-incubating" href="../0.8.0-incubating/index.html"/>
<item name="0.7.1-incubating" href="../0.7.1-incubating/index.html"/>
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/Architecture.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Architecture.twiki b/docs/src/site/twiki/Architecture.twiki
index c832500..818d6ca 100755
--- a/docs/src/site/twiki/Architecture.twiki
+++ b/docs/src/site/twiki/Architecture.twiki
@@ -61,6 +61,8 @@ as well. Currently, Atlas supports ingesting and managing metadata from the foll
* [[Bridge-Sqoop][Sqoop]]
* [[Bridge-Falcon][Falcon]]
* [[StormAtlasHook][Storm]]
+ * [[StormAtlasHook][Storm]]
+ * HBase - _documentation work-in-progress_
The integration implies two things:
There are metadata models that Atlas defines natively to represent objects of these components.
@@ -80,13 +82,6 @@ for the Hadoop ecosystem having wide integration with a variety of Hadoop compon
Ranger allows security administrators to define metadata driven security policies for effective governance.
Ranger is a consumer to the metadata change events notified by Atlas.
-*Business Taxonomy*: The metadata objects ingested into Atlas from the Metadata sources are primarily a form
-of technical metadata. To enhance the discoverability and governance capabilities, Atlas comes with a Business
-Taxonomy interface that allows users to first, define a hierarchical set of business terms that represent their
-business domain and associate them to the metadata entities Atlas manages. Business Taxonomy is a web application that
-is part of the Atlas Admin UI currently and integrates with Atlas using the REST API.
-
-
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/Bridge-Falcon.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Bridge-Falcon.twiki b/docs/src/site/twiki/Bridge-Falcon.twiki
index de80035..0cf1645 100644
--- a/docs/src/site/twiki/Bridge-Falcon.twiki
+++ b/docs/src/site/twiki/Bridge-Falcon.twiki
@@ -1,44 +1,52 @@
---+ Falcon Atlas Bridge
---++ Falcon Model
-The default falcon modelling is available in org.apache.atlas.falcon.model.FalconDataModelGenerator. It defines the following types:
-<verbatim>
-falcon_cluster(ClassType) - super types [Infrastructure] - attributes [timestamp, colo, owner, tags]
-falcon_feed(ClassType) - super types [DataSet] - attributes [timestamp, stored-in, owner, groups, tags]
-falcon_feed_creation(ClassType) - super types [Process] - attributes [timestamp, stored-in, owner]
-falcon_feed_replication(ClassType) - super types [Process] - attributes [timestamp, owner]
-falcon_process(ClassType) - super types [Process] - attributes [timestamp, runs-on, owner, tags, pipelines, workflow-properties]
-</verbatim>
+The default hive model includes the following types:
+ * Entity types:
+ * falcon_cluster
+ * super-types: Infrastructure
+ * attributes: timestamp, colo, owner, tags
+ * falcon_feed
+ * super-types: !DataSet
+ * attributes: timestamp, stored-in, owner, groups, tags
+ * falcon_feed_creation
+ * super-types: Process
+ * attributes: timestamp, stored-in, owner
+ * falcon_feed_replication
+ * super-types: Process
+ * attributes: timestamp, owner
+ * falcon_process
+ * super-types: Process
+ * attributes: timestamp, runs-on, owner, tags, pipelines, workflow-properties
One falcon_process entity is created for every cluster that the falcon process is defined for.
The entities are created and de-duped using unique qualifiedName attribute. They provide namespace and can be used for querying/lineage as well. The unique attributes are:
- * falcon_process - <process name>@<cluster name>
- * falcon_cluster - <cluster name>
- * falcon_feed - <feed name>@<cluster name>
- * falcon_feed_creation - <feed name>
- * falcon_feed_replication - <feed name>
+ * falcon_process.qualifiedName - <process name>@<cluster name>
+ * falcon_cluster.qualifiedName - <cluster name>
+ * falcon_feed.qualifiedName - <feed name>@<cluster name>
+ * falcon_feed_creation.qualifiedName - <feed name>
+ * falcon_feed_replication.qualifiedName - <feed name>
---++ Falcon Hook
-Falcon supports listeners on falcon entity submission. This is used to add entities in Atlas using the model defined in org.apache.atlas.falcon.model.FalconDataModelGenerator.
-The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities.
+Falcon supports listeners on falcon entity submission. This is used to add entities in Atlas using the model detailed above.
+Follow the instructions below to setup Atlas hook in Falcon:
* Add 'org.apache.atlas.falcon.service.AtlasService' to application.services in <falcon-conf>/startup.properties
- * Link falcon hook jars in falcon classpath - 'ln -s <atlas-home>/hook/falcon/* <falcon-home>/server/webapp/falcon/WEB-INF/lib/'
+ * Link Atlas hook jars in Falcon classpath - 'ln -s <atlas-home>/hook/falcon/* <falcon-home>/server/webapp/falcon/WEB-INF/lib/'
* In <falcon_conf>/falcon-env.sh, set an environment variable as follows:
<verbatim>
- export FALCON_SERVER_OPTS="<atlas_home>/hook/falcon/*:$FALCON_SERVER_OPTS"
- </verbatim>
+ export FALCON_SERVER_OPTS="<atlas_home>/hook/falcon/*:$FALCON_SERVER_OPTS"</verbatim>
The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:
- * atlas.hook.falcon.synchronous - boolean, true to run the hook synchronously. default false
- * atlas.hook.falcon.numRetries - number of retries for notification failure. default 3
- * atlas.hook.falcon.minThreads - core number of threads. default 5
- * atlas.hook.falcon.maxThreads - maximum number of threads. default 5
+ * atlas.hook.falcon.synchronous - boolean, true to run the hook synchronously. default false
+ * atlas.hook.falcon.numRetries - number of retries for notification failure. default 3
+ * atlas.hook.falcon.minThreads - core number of threads. default 5
+ * atlas.hook.falcon.maxThreads - maximum number of threads. default 5
* atlas.hook.falcon.keepAliveTime - keep alive time in msecs. default 10
- * atlas.hook.falcon.queueSize - queue size for the threadpool. default 10000
+ * atlas.hook.falcon.queueSize - queue size for the threadpool. default 10000
Refer [[Configuration][Configuration]] for notification related configurations
----++ Limitations
+---++ NOTES
* In falcon cluster entity, cluster name used should be uniform across components like hive, falcon, sqoop etc. If used with ambari, ambari cluster name should be used for cluster entity
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/Bridge-Hive.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Bridge-Hive.twiki b/docs/src/site/twiki/Bridge-Hive.twiki
index dd22b5c..7c93ecd 100644
--- a/docs/src/site/twiki/Bridge-Hive.twiki
+++ b/docs/src/site/twiki/Bridge-Hive.twiki
@@ -1,73 +1,71 @@
---+ Hive Atlas Bridge
---++ Hive Model
-The default hive modelling is available in org.apache.atlas.hive.model.HiveDataModelGenerator. It defines the following types:
-<verbatim>
-hive_db(ClassType) - super types [Referenceable] - attributes [name, clusterName, description, locationUri, parameters, ownerName, ownerType]
-hive_storagedesc(ClassType) - super types [Referenceable] - attributes [cols, location, inputFormat, outputFormat, compressed, numBuckets, serdeInfo, bucketCols, sortCols, parameters, storedAsSubDirectories]
-hive_column(ClassType) - super types [Referenceable] - attributes [name, type, comment, table]
-hive_table(ClassType) - super types [DataSet] - attributes [name, db, owner, createTime, lastAccessTime, comment, retention, sd, partitionKeys, columns, aliases, parameters, viewOriginalText, viewExpandedText, tableType, temporary]
-hive_process(ClassType) - super types [Process] - attributes [name, startTime, endTime, userName, operationType, queryText, queryPlan, queryId]
-hive_principal_type(EnumType) - values [USER, ROLE, GROUP]
-hive_order(StructType) - attributes [col, order]
-hive_serde(StructType) - attributes [name, serializationLib, parameters]
-</verbatim>
-
-The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying/lineage as well. Note that dbName, tableName and columnName should be in lower case. clusterName is explained below.
- * hive_db - attribute qualifiedName - <dbName>@<clusterName>
- * hive_table - attribute qualifiedName - <dbName>.<tableName>@<clusterName>
- * hive_column - attribute qualifiedName - <dbName>.<tableName>.<columnName>@<clusterName>
- * hive_process - attribute name - <queryString> - trimmed query string in lower case
+The default hive model includes the following types:
+ * Entity types:
+ * hive_db
+ * super-types: Referenceable
+ * attributes: name, clusterName, description, locationUri, parameters, ownerName, ownerType
+ * hive_storagedesc
+ * super-types: Referenceable
+ * attributes: cols, location, inputFormat, outputFormat, compressed, numBuckets, serdeInfo, bucketCols, sortCols, parameters, storedAsSubDirectories
+ * hive_column
+ * super-types: Referenceable
+ * attributes: name, type, comment, table
+ * hive_table
+ * super-types: !DataSet
+ * attributes: name, db, owner, createTime, lastAccessTime, comment, retention, sd, partitionKeys, columns, aliases, parameters, viewOriginalText, viewExpandedText, tableType, temporary
+ * hive_process
+ * super-types: Process
+ * attributes: name, startTime, endTime, userName, operationType, queryText, queryPlan, queryId
+ * hive_column_lineage
+ * super-types: Process
+ * attributes: query, depenendencyType, expression
+
+ * Enum types:
+ * hive_principal_type
+ * values: USER, ROLE, GROUP
+
+ * Struct types:
+ * hive_order
+ * attributes: col, order
+ * hive_serde
+ * attributes: name, serializationLib, parameters
+
+The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying/lineage as well. Note that dbName, tableName and columnName should be in lower case. clusterName is explained below.
+ * hive_db.qualifiedName - <dbName>@<clusterName>
+ * hive_table.qualifiedName - <dbName>.<tableName>@<clusterName>
+ * hive_column.qualifiedName - <dbName>.<tableName>.<columnName>@<clusterName>
+ * hive_process.queryString - trimmed query string in lower case
---++ Importing Hive Metadata
-org.apache.atlas.hive.bridge.HiveMetaStoreBridge imports the Hive metadata into Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator. import-hive.sh command can be used to facilitate this. The script needs Hadoop and Hive classpath jars.
- * For Hadoop jars, please make sure that the environment variable HADOOP_CLASSPATH is set. Another way is to set HADOOP_HOME to point to root directory of your Hadoop installation
- * Similarly, for Hive jars, set HIVE_HOME to the root of Hive installation
- * Set environment variable HIVE_CONF_DIR to Hive configuration directory
- * Copy <atlas-conf>/atlas-application.properties to the hive conf directory
-
+org.apache.atlas.hive.bridge.HiveMetaStoreBridge imports the Hive metadata into Atlas using the model defined above. import-hive.sh command can be used to facilitate this.
<verbatim>
- Usage: <atlas package>/hook-bin/import-hive.sh
- </verbatim>
+ Usage: <atlas package>/hook-bin/import-hive.sh</verbatim>
The logs are in <atlas package>/logs/import-hive.log
-If you you are importing metadata in a kerberized cluster you need to run the command like this:
-<verbatim>
-<atlas package>/hook-bin/import-hive.sh -Dsun.security.jgss.debug=true -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.conf=[krb5.conf location] -Djava.security.auth.login.config=[jaas.conf location]
-</verbatim>
- * krb5.conf is typically found at /etc/krb5.conf
- * for details about jaas.conf and a suggested location see the [[security][atlas security documentation]]
-
---++ Hive Hook
-Hive supports listeners on hive command execution using hive hooks. This is used to add/update/remove entities in Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator.
-The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities.
-Follow these instructions in your hive set-up to add hive hook for Atlas:
- * Set-up atlas hook in hive-site.xml of your hive configuration:
+Atlas Hive hook registers with Hive to listen for create/update/delete operations and updates the metadata in Atlas, via Kafka notifications, for the changes in Hive.
+Follow the instructions below to setup Atlas hook in Hive:
+ * Set-up Atlas hook in hive-site.xml by adding the following:
<verbatim>
<property>
<name>hive.exec.post.hooks</name>
<value>org.apache.atlas.hive.hook.HiveHook</value>
- </property>
- </verbatim>
- <verbatim>
- <property>
- <name>atlas.cluster.name</name>
- <value>primary</value>
- </property>
- </verbatim>
+ </property></verbatim>
* Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in hive-env.sh of your hive configuration
* Copy <atlas-conf>/atlas-application.properties to the hive conf directory.
The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:
- * atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false. Recommended to be set to false to avoid delays in hive query completion.
- * atlas.hook.hive.numRetries - number of retries for notification failure. default 3
- * atlas.hook.hive.minThreads - core number of threads. default 5
- * atlas.hook.hive.maxThreads - maximum number of threads. default 5
+ * atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false. Recommended to be set to false to avoid delays in hive query completion.
+ * atlas.hook.hive.numRetries - number of retries for notification failure. default 3
+ * atlas.hook.hive.minThreads - core number of threads. default 1
+ * atlas.hook.hive.maxThreads - maximum number of threads. default 5
* atlas.hook.hive.keepAliveTime - keep alive time in msecs. default 10
- * atlas.hook.hive.queueSize - queue size for the threadpool. default 10000
+ * atlas.hook.hive.queueSize - queue size for the threadpool. default 10000
Refer [[Configuration][Configuration]] for notification related configurations
@@ -76,24 +74,23 @@ Refer [[Configuration][Configuration]] for notification related configurations
Starting from 0.8-incubating version of Atlas, Column level lineage is captured in Atlas. Below are the details
---+++ Model
- * !ColumnLineageProcess type is a subclass of Process
+ * !ColumnLineageProcess type is a subtype of Process
* This relates an output Column to a set of input Columns or the Input Table
- * The Lineage also captures the kind of Dependency: currently the values are SIMPLE, EXPRESSION, SCRIPT
- * A SIMPLE dependency means the output column has the same value as the input
- * An EXPRESSION dependency means the output column is transformed by some expression in the runtime(for e.g. a Hive SQL expression) on the Input Columns.
- * SCRIPT means that the output column is transformed by a user provided script.
+ * The lineage also captures the kind of dependency, as listed below:
+ * SIMPLE: output column has the same value as the input
+ * EXPRESSION: output column is transformed by some expression at runtime (for e.g. a Hive SQL expression) on the Input Columns.
+ * SCRIPT: output column is transformed by a user provided script.
* In case of EXPRESSION dependency the expression attribute contains the expression in string form
- * Since Process links input and output !DataSets, we make Column a subclass of !DataSet
+ * Since Process links input and output !DataSets, Column is a subtype of !DataSet
---+++ Examples
For a simple CTAS below:
<verbatim>
-create table t2 as select id, name from T1
-</verbatim>
+create table t2 as select id, name from T1</verbatim>
The lineage is captured as
@@ -106,10 +103,8 @@ The lineage is captured as
* The !LineageInfo in Hive provides column-level lineage for the final !FileSinkOperator, linking them to the input columns in the Hive Query
----+++ NOTE
-Column level lineage works with Hive version 1.2.1 after the patch for <a href="https://issues.apache.org/jira/browse/HIVE-13112">HIVE-13112</a> is applied to Hive source
-
----++ Limitations
+---++ NOTES
+ * Column level lineage works with Hive version 1.2.1 after the patch for <a href="https://issues.apache.org/jira/browse/HIVE-13112">HIVE-13112</a> is applied to Hive source
* Since database name, table name and column names are case insensitive in hive, the corresponding names in entities are lowercase. So, any search APIs should use lowercase while querying on the entity names
* The following hive operations are captured by hive hook currently
* create database
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/Bridge-Sqoop.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Bridge-Sqoop.twiki b/docs/src/site/twiki/Bridge-Sqoop.twiki
index bf942f2..480578b 100644
--- a/docs/src/site/twiki/Bridge-Sqoop.twiki
+++ b/docs/src/site/twiki/Bridge-Sqoop.twiki
@@ -1,37 +1,42 @@
---+ Sqoop Atlas Bridge
---++ Sqoop Model
-The default Sqoop modelling is available in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. It defines the following types:
-<verbatim>
-sqoop_operation_type(EnumType) - values [IMPORT, EXPORT, EVAL]
-sqoop_dbstore_usage(EnumType) - values [TABLE, QUERY, PROCEDURE, OTHER]
-sqoop_process(ClassType) - super types [Process] - attributes [name, operation, dbStore, hiveTable, commandlineOpts, startTime, endTime, userName]
-sqoop_dbdatastore(ClassType) - super types [DataSet] - attributes [name, dbStoreType, storeUse, storeUri, source, description, ownerName]
-</verbatim>
+The default hive model includes the following types:
+ * Entity types:
+ * sqoop_process
+ * super-types: Process
+ * attributes: name, operation, dbStore, hiveTable, commandlineOpts, startTime, endTime, userName
+ * sqoop_dbdatastore
+ * super-types: !DataSet
+ * attributes: name, dbStoreType, storeUse, storeUri, source, description, ownerName
+
+ * Enum types:
+ * sqoop_operation_type
+ * values: IMPORT, EXPORT, EVAL
+ * sqoop_dbstore_usage
+ * values: TABLE, QUERY, PROCEDURE, OTHER
The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying as well:
-sqoop_process - attribute name - sqoop-dbStoreType-storeUri-endTime
-sqoop_dbdatastore - attribute name - dbStoreType-connectorUrl-source
+ * sqoop_process.qualifiedName - dbStoreType-storeUri-endTime
+ * sqoop_dbdatastore.qualifiedName - dbStoreType-storeUri-source
---++ Sqoop Hook
-Sqoop added a !SqoopJobDataPublisher that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in sqoopHook.
-This is used to add entities in Atlas using the model defined in org.apache.atlas.sqoop.model.SqoopDataModelGenerator.
-Follow these instructions in your sqoop set-up to add sqoop hook for Atlas in <sqoop-conf>/sqoop-site.xml:
+Sqoop added a !SqoopJobDataPublisher that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in !SqoopHook.
+This is used to add entities in Atlas using the model detailed above.
+
+Follow the instructions below to setup Atlas hook in Hive:
- * Sqoop Job publisher class. Currently only one publishing class is supported
+Add the following properties to to enable Atlas hook in Sqoop:
+ * Set-up Atlas hook in <sqoop-conf>/sqoop-site.xml by adding the following:
+ <verbatim>
<property>
<name>sqoop.job.data.publish.class</name>
<value>org.apache.atlas.sqoop.hook.SqoopHook</value>
- </property>
- * Atlas cluster name
- <property>
- <name>atlas.cluster.name</name>
- <value><clustername></value>
- </property>
+ </property></verbatim>
* Copy <atlas-conf>/atlas-application.properties to to the sqoop conf directory <sqoop-conf>/
* Link <atlas-home>/hook/sqoop/*.jar in sqoop lib
Refer [[Configuration][Configuration]] for notification related configurations
----++ Limitations
+---++ NOTES
* Only the following sqoop operations are captured by sqoop hook currently - hiveImport
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/Configuration.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Configuration.twiki b/docs/src/site/twiki/Configuration.twiki
index a5069ab..1da3184 100644
--- a/docs/src/site/twiki/Configuration.twiki
+++ b/docs/src/site/twiki/Configuration.twiki
@@ -5,139 +5,41 @@ All configuration in Atlas uses java properties style configuration. The main co
---++ Graph Configs
----+++ Graph persistence engine
-
-This section sets up the graph db - titan - to use a persistence engine. Please refer to
-<a href="http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html">link</a> for more
-details. The example below uses BerkeleyDBJE.
-
-<verbatim>
-atlas.graph.storage.backend=berkeleyje
-atlas.graph.storage.directory=data/berkley
-</verbatim>
-
----++++ Graph persistence engine - Hbase
-
-Basic configuration
+---+++ Graph Persistence engine - HBase
+Set the following properties to configure Atlas to use HBase as the persistence engine. Please refer to
+<a href="http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html">link</a> for more details.
<verbatim>
atlas.graph.storage.backend=hbase
-#For standalone mode , specify localhost
-#for distributed mode, specify zookeeper quorum here - For more information refer http://s3.thinkaurelius.com/docs/titan/current/hbase.html#_remote_server_mode_2
atlas.graph.storage.hostname=<ZooKeeper Quorum>
-</verbatim>
+atlas.graph.storage.hbase.table=atlas</verbatim>
-HBASE_CONF_DIR environment variable needs to be set to point to the Hbase client configuration directory which is added to classpath when Atlas starts up.
-hbase-site.xml needs to have the following properties set according to the cluster setup
-<verbatim>
-#Set below to /hbase-secure if the Hbase server is setup in secure mode
-zookeeper.znode.parent=/hbase-unsecure
-</verbatim>
-
-Advanced configuration
-
-# If you are planning to use any of the configs mentioned below, they need to be prefixed with "atlas.graph." to take effect in ATLAS
-Refer http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html#_storage_hbase
-
-Permissions
+If any further Titan configuration needs to be setup, please prefix the property name with "atlas.graph.".
-When Atlas is configured with HBase as the storage backend the graph db (titan) needs sufficient user permissions to be able to create and access an HBase table. In a secure cluster it may be necessary to grant permissions to the 'atlas' user for the 'titan' table.
+In addition to setting up configurations, please ensure that environment variable HBASE_CONF_DIR is setup to point to
+the directory containing HBase configuration file hbase-site.xml.
-With Ranger, a policy can be configured for 'titan'.
-
-Without Ranger, HBase shell can be used to set the permissions.
+---+++ Graph Search Index - Solr
+Solr installation in Cloud mode is a prerequisite for Apache Atlas use. Set the following properties to configure Titan to use Solr as the index search engine.
<verbatim>
- su hbase
- kinit -k -t <hbase keytab> <hbase principal>
- echo "grant 'atlas', 'RWXCA', 'titan'" | hbase shell
-</verbatim>
+atlas.graph.index.search.backend=solr5
+atlas.graph.index.search.solr.mode=cloud
+atlas.graph.index.search.solr.wait-searcher=true
-Note that if the embedded-hbase-solr profile is used then HBase is included in the distribution so that a standalone
-instance of HBase can be started as the default storage backend for the graph repository. Using the embedded-hbase-solr
-profile will configure Atlas so that HBase instance will be started and stopped along with the Atlas server by default.
-To use the embedded-hbase-solr profile please see "Building Atlas" in the [[InstallationSteps][Installation Steps]]
-section.
+# ZK quorum setup for solr as comma separated value. Example: 10.1.6.4:2181,10.1.6.5:2181
+atlas.graph.index.search.solr.zookeeper-url=
----+++ Graph Search Index
-This section sets up the graph db - titan - to use an search indexing system. The example
-configuration below sets up to use an embedded Elastic search indexing system.
+# SolrCloud Zookeeper Connection Timeout. Default value is 60000 ms
+atlas.graph.index.search.solr.zookeeper-connect-timeout=60000
-<verbatim>
-atlas.graph.index.search.backend=elasticsearch
-atlas.graph.index.search.directory=data/es
-atlas.graph.index.search.elasticsearch.client-only=false
-atlas.graph.index.search.elasticsearch.local-mode=true
-atlas.graph.index.search.elasticsearch.create.sleep=2000
-</verbatim>
-
----++++ Graph Search Index - Solr
-Please note that Solr installation in Cloud mode is a prerequisite before configuring Solr as the search indexing backend. Refer InstallationSteps section for Solr installation/configuration.
-
-<verbatim>
- atlas.graph.index.search.backend=solr5
- atlas.graph.index.search.solr.mode=cloud
- atlas.graph.index.search.solr.zookeeper-url=<the ZK quorum setup for solr as comma separated value> eg: 10.1.6.4:2181,10.1.6.5:2181
- atlas.graph.index.search.solr.zookeeper-connect-timeout=<SolrCloud Zookeeper Connection Timeout>. Default value is 60000 ms
- atlas.graph.index.search.solr.zookeeper-session-timeout=<SolrCloud Zookeeper Session Timeout>. Default value is 60000 ms
-</verbatim>
-
-Also note that if the embedded-hbase-solr profile is used then Solr is included in the distribution so that a standalone
-instance of Solr can be started as the default search indexing backend. Using the embedded-hbase-solr profile will
-configure Atlas so that the standalone Solr instance will be started and stopped along with the Atlas server by default.
-To use the embedded-hbase-solr profile please see "Building Atlas" in the [[InstallationSteps][Installation Steps]]
-section.
-
----+++ Choosing between Persistence and Indexing Backends
-
-Refer http://s3.thinkaurelius.com/docs/titan/0.5.4/bdb.html and http://s3.thinkaurelius.com/docs/titan/0.5.4/hbase.html for choosing between the persistence backends.
-BerkeleyDB is suitable for smaller data sets in the range of upto 10 million vertices with ACID gurantees.
-HBase on the other hand doesnt provide ACID guarantees but is able to scale for larger graphs. HBase also provides HA inherently.
-
----+++ Choosing between Persistence Backends
-
-Refer http://s3.thinkaurelius.com/docs/titan/0.5.4/bdb.html and http://s3.thinkaurelius.com/docs/titan/0.5.4/hbase.html for choosing between the persistence backends.
-BerkeleyDB is suitable for smaller data sets in the range of upto 10 million vertices with ACID gurantees.
-HBase on the other hand doesnt provide ACID guarantees but is able to scale for larger graphs. HBase also provides HA inherently.
-
----+++ Choosing between Indexing Backends
-
-Refer http://s3.thinkaurelius.com/docs/titan/0.5.4/elasticsearch.html and http://s3.thinkaurelius.com/docs/titan/0.5.4/solr.html for choosing between !ElasticSearch and Solr.
-Solr in cloud mode is the recommended setup.
-
----+++ Switching Persistence Backend
-
-For switching the storage backend from BerkeleyDB to HBase and vice versa, refer the documentation for "Graph Persistence Engine" described above and restart ATLAS.
-The data in the indexing backend needs to be cleared else there will be discrepancies between the storage and indexing backend which could result in errors during the search.
-!ElasticSearch runs by default in embedded mode and the data could easily be cleared by deleting the ATLAS_HOME/data/es directory.
-For Solr, the collections which were created during ATLAS Installation - vertex_index, edge_index, fulltext_index could be deleted which will cleanup the indexes
-
----+++ Switching Index Backend
-
-Switching the Index backend requires clearing the persistence backend data. Otherwise there will be discrepancies between the persistence and index backends since switching the indexing backend means index data will be lost.
-This leads to "Fulltext" queries not working on the existing data
-For clearing the data for BerkeleyDB, delete the ATLAS_HOME/data/berkeley directory
-For clearing the data for HBase, in Hbase shell, run 'disable titan' and 'drop titan'
-
-
----++ Lineage Configs
-
-The higher layer services like lineage, schema, etc. are driven by the type system and this section encodes the specific types for the hive data model.
-
-# This models reflects the base super types for Data and Process
-<verbatim>
-atlas.lineage.hive.table.type.name=DataSet
-atlas.lineage.hive.process.type.name=Process
-atlas.lineage.hive.process.inputs.name=inputs
-atlas.lineage.hive.process.outputs.name=outputs
-
-## Schema
-atlas.lineage.hive.table.schema.query=hive_table where name=?, columns
+# SolrCloud Zookeeper Session Timeout. Default value is 60000 ms
+atlas.graph.index.search.solr.zookeeper-session-timeout=60000
</verbatim>
---++ Search Configs
-Search APIs (DSL and full text search) support pagination and have optional limit and offset arguments. Following configs are related to search pagination
+Search APIs (DSL, basic search, full-text search) support pagination and have optional limit and offset arguments. Following configs are related to search pagination
<verbatim>
# Default limit used when limit is not specified in API
@@ -152,46 +54,27 @@ atlas.search.maxlimit=10000
Refer http://kafka.apache.org/documentation.html#configuration for Kafka configuration. All Kafka configs should be prefixed with 'atlas.kafka.'
<verbatim>
-atlas.notification.embedded=true
-atlas.kafka.data=${sys:atlas.home}/data/kafka
-atlas.kafka.zookeeper.connect=localhost:9026
-atlas.kafka.bootstrap.servers=localhost:9027
-atlas.kafka.zookeeper.session.timeout.ms=400
-atlas.kafka.zookeeper.sync.time.ms=20
-atlas.kafka.auto.commit.interval.ms=1000
-atlas.kafka.hook.group.id=atlas
-</verbatim>
+atlas.kafka.auto.commit.enable=false
-Note that Kafka group ids are specified for a specific topic. The Kafka group id configuration for entity notifications is 'atlas.kafka.entities.group.id'
+# Kafka servers. Example: localhost:6667
+atlas.kafka.bootstrap.servers=
-<verbatim>
-atlas.kafka.entities.group.id=<consumer id>
-</verbatim>
+atlas.kafka.hook.group.id=atlas
-These configuration parameters are useful for setting up Kafka topics via Atlas provided scripts, described in the
-[[InstallationSteps][Installation Steps]] page.
+# Zookeeper connect URL for Kafka. Example: localhost:2181
+atlas.kafka.zookeeper.connect=
-<verbatim>
-# Whether to create the topics automatically, default is true.
-# Comma separated list of topics to be created, default is "ATLAS_HOOK,ATLAS_ENTITES"
-atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
-# Number of replicas for the Atlas topics, default is 1. Increase for higher resilience to Kafka failures.
-atlas.notification.replicas=1
-# Enable the below two properties if Kafka is running in Kerberized mode.
-# Set this to the service principal representing the Kafka service
-atlas.notification.kafka.service.principal=kafka/_HOST@EXAMPLE.COM
-# Set this to the location of the keytab file for Kafka
-#atlas.notification.kafka.keytab.location=/etc/security/keytabs/kafka.service.keytab
-</verbatim>
+atlas.kafka.zookeeper.connection.timeout.ms=30000
+atlas.kafka.zookeeper.session.timeout.ms=60000
+atlas.kafka.zookeeper.sync.time.ms=20
-These configuration parameters are useful for saving messages in case there are issues in reaching Kafka for
-sending messages.
+# Setup the following configurations only in test deployments where Kafka is started within Atlas in embedded mode
+# atlas.notification.embedded=true
+# atlas.kafka.data=${sys:atlas.home}/data/kafka
-<verbatim>
-# Whether to save messages that failed to be sent to Kafka, default is true
-atlas.notification.log.failed.messages=true
-# If saving messages is enabled, the file name to save them to. This file will be created under the log directory of the hook's host component - like HiveServer2
-atlas.notification.failed.messages.filename=atlas_hook_failed_messages.log
+# Setup the following two properties if Kafka is running in Kerberized mode.
+# atlas.notification.kafka.service.principal=kafka/_HOST@EXAMPLE.COM
+# atlas.notification.kafka.keytab.location=/etc/security/keytabs/kafka.service.keytab
</verbatim>
---++ Client Configs
@@ -212,26 +95,28 @@ atlas.enableTLS=false
</verbatim>
---++ High Availability Properties
-
The following properties describe High Availability related configuration options:
<verbatim>
# Set the following property to true, to enable High Availability. Default = false.
atlas.server.ha.enabled=true
-# Define a unique set of strings to identify each instance that should run an Atlas Web Service instance as a comma separated list.
+# Specify the list of Atlas instances
atlas.server.ids=id1,id2
-# For each string defined above, define the host and port on which Atlas server binds to.
+# For each instance defined above, define the host and port on which Atlas server listens.
atlas.server.address.id1=host1.company.com:21000
atlas.server.address.id2=host2.company.com:31000
# Specify Zookeeper properties needed for HA.
# Specify the list of services running Zookeeper servers as a comma separated list.
atlas.server.ha.zookeeper.connect=zk1.company.com:2181,zk2.company.com:2181,zk3.company.com:2181
+
# Specify how many times should connection try to be established with a Zookeeper cluster, in case of any connection issues.
atlas.server.ha.zookeeper.num.retries=3
+
# Specify how much time should the server wait before attempting connections to Zookeeper, in case of any connection issues.
atlas.server.ha.zookeeper.retry.sleeptime.ms=1000
+
# Specify how long a session to Zookeeper should last without inactiviy to be deemed as unreachable.
atlas.server.ha.zookeeper.session.timeout.ms=20000
@@ -239,6 +124,7 @@ atlas.server.ha.zookeeper.session.timeout.ms=20000
# The format of these options is <scheme>:<identity>. For more information refer to http://zookeeper.apache.org/doc/r3.2.2/zookeeperProgrammers.html#sc_ZooKeeperAccessControl.
# The 'acl' option allows to specify a scheme, identity pair to setup an ACL for.
atlas.server.ha.zookeeper.acl=sasl:client@comany.com
+
# The 'auth' option specifies the authentication that should be used for connecting to Zookeeper.
atlas.server.ha.zookeeper.auth=sasl:client@company.com
@@ -254,14 +140,12 @@ atlas.client.ha.sleep.interval.ms=5000
</verbatim>
---++ Server Properties
-
<verbatim>
# Set the following property to true, to enable the setup steps to run on each server start. Default = false.
atlas.server.run.setup.on.start=false
</verbatim>
---++ Performance configuration items
-
The following properties can be used to tune performance of Atlas under specific circumstances:
<verbatim>
@@ -288,14 +172,19 @@ atlas.webserver.queuesize=100
</verbatim>
---+++ Recording performance metrics
-
-Atlas package should be built with '-P perf' to instrument atlas code to collect metrics. The metrics will be recorded in
-<atlas.log.dir>/metric.log, with one log line per API call. The metrics contain the number of times the instrumented methods
-are called and the total time spent in the instrumented method. Logging to metric.log is controlled through log4j configuration
-in atlas-log4j.xml. When the atlas code is instrumented, to disable logging to metric.log at runtime, set log level of METRICS logger to info level:
-<verbatim>
-<logger name="METRICS" additivity="false">
- <level value="info"/>
- <appender-ref ref="METRICS"/>
-</logger>
-</verbatim>
+To enable performance logs for various Atlas operations (like REST API calls, notification processing), setup the following in atlas-log4j.xml:
+<verbatim>
+ <appender name="perf_appender" class="org.apache.log4j.DailyRollingFileAppender">
+ <param name="File" value="/var/log/atlas/atlas_perf.log"/>
+ <param name="datePattern" value="'.'yyyy-MM-dd"/>
+ <param name="append" value="true"/>
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="%d|%t|%m%n"/>
+ </layout>
+ </appender>
+
+ <logger name="org.apache.atlas.perf" additivity="false">
+ <level value="debug"/>
+ <appender-ref ref="perf_appender"/>
+ </logger>
+ </verbatim>
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/EclipseSetup.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/EclipseSetup.twiki b/docs/src/site/twiki/EclipseSetup.twiki
index 9d21859..b9f1d5b 100644
--- a/docs/src/site/twiki/EclipseSetup.twiki
+++ b/docs/src/site/twiki/EclipseSetup.twiki
@@ -92,7 +92,7 @@ export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Hom
export M2_HOME=/Applications/apache-maven-3.3.9 # Git is installed in the system path
export PYTHON_HOME='/Applications/Python 2.7'
export PATH=$PYTHON_HOME:$M2_HOME/bin:$JAVA_HOME/bin:$PATH
-export MAVEN_OPTS="-Xmx1536m -Drat.numUnapprovedLicenses=100 -XX:MaxPermSize=256m"
+export MAVEN_OPTS="-Xmx1536m -Drat.numUnapprovedLicenses=100"
</verbatim>
* If you do not want to set Java 8 as your system java, you can use this bash script to setup the environment and run Eclipse (which you can drop in Applications and rename to neon).
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/InstallationSteps.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/InstallationSteps.twiki b/docs/src/site/twiki/InstallationSteps.twiki
index 417df11..bc70d6f 100644
--- a/docs/src/site/twiki/InstallationSteps.twiki
+++ b/docs/src/site/twiki/InstallationSteps.twiki
@@ -4,132 +4,45 @@
<verbatim>
git clone https://git-wip-us.apache.org/repos/asf/atlas.git atlas
-
cd atlas
+git checkout branch-0.8
+export MAVEN_OPTS="-Xms2g -Xmx2g"
+mvn clean -DskipTests install</verbatim></verbatim>
-export MAVEN_OPTS="-Xmx1536m -XX:MaxPermSize=512m" && mvn clean install
-</verbatim>
-
-Once the build successfully completes, artifacts can be packaged for deployment.
-
-<verbatim>
-
-mvn clean package -Pdist
-</verbatim>
-
-NOTE:
-1. Use option '-DskipTests' to skip running unit and integration tests
-2. Use option '-P perf' to instrument atlas to collect performance metrics
-
-To build a distribution that configures Atlas for external HBase and Solr, build with the external-hbase-solr profile.
+---+++ Packaging Atlas
+To create Apache Atlas package for deployment in an environment having functional HBase and Solr instances, build with the following command:
<verbatim>
+mvn clean -DskipTests package -Pdist</verbatim>
-mvn clean package -Pdist,external-hbase-solr
+ * NOTES:
+ * Remove option '-DskipTests' to run unit and integration tests
+ * To build a distribution without minified js,css file, build with skipMinify profile. By default js and css files are minified.
-</verbatim>
-Note that when the external-hbase-solr profile is used the following steps need to be completed to make Atlas functional.
+Above will build Atlas for an environment having functional HBase and Solr instances. Atlas needs to be setup with the following to run in this environment:
* Configure atlas.graph.storage.hostname (see "Graph persistence engine - HBase" in the [[Configuration][Configuration]] section).
* Configure atlas.graph.index.search.solr.zookeeper-url (see "Graph Search Index - Solr" in the [[Configuration][Configuration]] section).
* Set HBASE_CONF_DIR to point to a valid HBase config directory (see "Graph persistence engine - HBase" in the [[Configuration][Configuration]] section).
* Create the SOLR indices (see "Graph Search Index - Solr" in the [[Configuration][Configuration]] section).
-To build a distribution that packages HBase and Solr, build with the embedded-hbase-solr profile.
-
-<verbatim>
-
-mvn clean package -Pdist,embedded-hbase-solr
-</verbatim>
-
-Using the embedded-hbase-solr profile will configure Atlas so that an HBase instance and a Solr instance will be started
-and stopped along with the Atlas server by default.
-
-Atlas also supports building a distribution that can use BerkeleyDB and Elastic search as the graph and index backends.
-To build a distribution that is configured for these backends, build with the berkeley-elasticsearch profile.
+---+++ Packaging Atlas with Embedded HBase & Solr
+To create Apache Atlas package that includes HBase and Solr, build with the embedded-hbase-solr profile as shown below:
<verbatim>
+mvn clean -DskipTests package -Pdist,embedded-hbase-solr</verbatim>
-mvn clean package -Pdist,berkeley-elasticsearch
-
-</verbatim>
-
-An additional step is required for the binary built using this profile to be used along with the Atlas distribution.
-Due to licensing requirements, Atlas does not bundle the BerkeleyDB Java Edition in the tarball.
-
-You can download the Berkeley DB jar file from the URL: <verbatim>http://download.oracle.com/otn/berkeley-db/je-5.0.73.zip</verbatim>
-and copy the je-5.0.73.jar to the ${atlas_home}/libext directory.
+Using the embedded-hbase-solr profile will configure Atlas so that an HBase instance and a Solr instance will be started and stopped along with the Atlas server by default.
-Tar can be found in atlas/distro/target/apache-atlas-${project.version}-bin.tar.gz
-Tar is structured as follows
+---+++ Apache Atlas Package
+Build will create following files, which are used to install Apache Atlas.
<verbatim>
-
-|- bin
- |- atlas_start.py
- |- atlas_stop.py
- |- atlas_config.py
- |- quick_start.py
- |- cputil.py
-|- conf
- |- atlas-application.properties
- |- atlas-env.sh
- |- hbase
- |- hbase-site.xml.template
- |- log4j.xml
- |- solr
- |- currency.xml
- |- lang
- |- stopwords_en.txt
- |- protowords.txt
- |- schema.xml
- |- solrconfig.xml
- |- stopwords.txt
- |- synonyms.txt
-|- docs
-|- hbase
- |- bin
- |- conf
- ...
-|- server
- |- webapp
- |- atlas.war
-|- solr
- |- bin
- ...
-|- README
-|- NOTICE
-|- LICENSE
-|- DISCLAIMER.txt
-|- CHANGES.txt
-
-</verbatim>
-
-Note that if the embedded-hbase-solr profile is specified for the build then HBase and Solr are included in the
-distribution.
-
-In this case, a standalone instance of HBase can be started as the default storage backend for the graph repository.
-During Atlas installation the conf/hbase/hbase-site.xml.template gets expanded and moved to hbase/conf/hbase-site.xml
-for the initial standalone HBase configuration. To configure ATLAS
-graph persistence for a different HBase instance, please see "Graph persistence engine - HBase" in the
-[[Configuration][Configuration]] section.
-
-Also, a standalone instance of Solr can be started as the default search indexing backend. To configure ATLAS search
-indexing for a different Solr instance please see "Graph Search Index - Solr" in the
-[[Configuration][Configuration]] section.
-
-To build a distribution without minified js,css file, build with the skipMinify profile.
-
-<verbatim>
-
-mvn clean package -Pdist,skipMinify
-
-</verbatim>
-
-Note that by default js and css files are minified.
+distro/target/apache-atlas-${project.version}-bin.tar.gz
+distro/target/apache-atlas-${project.version}-sources.tar.gz</verbatim>
---+++ Installing & Running Atlas
@@ -137,18 +50,12 @@ Note that by default js and css files are minified.
<verbatim>
tar -xzvf apache-atlas-${project.version}-bin.tar.gz
-cd atlas-${project.version}
-</verbatim>
+cd atlas-${project.version}</verbatim>
---++++ Configuring Atlas
+By default config directory used by Atlas is {package dir}/conf. To override this set environment variable ATLAS_CONF to the path of the conf dir.
-By default config directory used by Atlas is {package dir}/conf. To override this set environment
-variable ATLAS_CONF to the path of the conf dir.
-
-atlas-env.sh has been added to the Atlas conf. This file can be used to set various environment
-variables that you need for you services. In addition you can set any other environment
-variables you might need. This file will be sourced by atlas scripts before any commands are
-executed. The following environment variables are available to set.
+Environment variables needed to run Atlas can be set in atlas-env.sh file in the conf directory. This file will be sourced by Atlas scripts before any commands are executed. The following environment variables are available to set.
<verbatim>
# The java implementation to use. If JAVA_HOME is not found we expect java and jar to be in path
@@ -182,65 +89,50 @@ executed. The following environment variables are available to set.
#export ATLAS_DATA_DIR=
# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir.
-#export ATLAS_EXPANDED_WEBAPP_DIR=
-</verbatim>
+#export ATLAS_EXPANDED_WEBAPP_DIR=</verbatim>
*Settings to support large number of metadata objects*
-If you plan to store several tens of thousands of metadata objects, it is recommended that you use values
-tuned for better GC performance of the JVM.
+If you plan to store several tens of thousands of metadata objects, it is recommended that you use values tuned for better GC performance of the JVM.
The following values are common server side options:
<verbatim>
-export ATLAS_SERVER_OPTS="-server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dumps/atlas_server.hprof -Xloggc:logs/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps"
-</verbatim>
+export ATLAS_SERVER_OPTS="-server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dumps/atlas_server.hprof -Xloggc:logs/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps"</verbatim>
-The =-XX:SoftRefLRUPolicyMSPerMB= option was found to be particularly helpful to regulate GC performance for
-query heavy workloads with many concurrent users.
+The =-XX:SoftRefLRUPolicyMSPerMB= option was found to be particularly helpful to regulate GC performance for query heavy workloads with many concurrent users.
The following values are recommended for JDK 7:
<verbatim>
-export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m -XX:MaxNewSize=3072m -XX:PermSize=100M -XX:MaxPermSize=512m"
-</verbatim>
+export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m -XX:MaxNewSize=3072m -XX:PermSize=100M -XX:MaxPermSize=512m"</verbatim>
The following values are recommended for JDK 8:
<verbatim>
-export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m -XX:MaxNewSize=5120m -XX:MetaspaceSize=100M -XX:MaxMetaspaceSize=512m"
-</verbatim>
+export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m -XX:MaxNewSize=5120m -XX:MetaspaceSize=100M -XX:MaxMetaspaceSize=512m"</verbatim>
*NOTE for Mac OS users*
If you are using a Mac OS, you will need to configure the ATLAS_SERVER_OPTS (explained above).
In {package dir}/conf/atlas-env.sh uncomment the following line
<verbatim>
-#export ATLAS_SERVER_OPTS=
-</verbatim>
+#export ATLAS_SERVER_OPTS=</verbatim>
and change it to look as below
<verbatim>
-export ATLAS_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="
-</verbatim>
+export ATLAS_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="</verbatim>
*Hbase as the Storage Backend for the Graph Repository*
-By default, Atlas uses Titan as the graph repository and is the only graph repository implementation available currently.
-The HBase versions currently supported are 1.1.x. For configuring ATLAS graph persistence on HBase, please see "Graph persistence engine - HBase" in the [[Configuration][Configuration]] section
+By default, Atlas uses Titan as the graph repository and is the only graph repository implementation available currently. The HBase versions currently supported are 1.1.x. For configuring ATLAS graph persistence on HBase, please see "Graph persistence engine - HBase" in the [[Configuration][Configuration]] section
for more details.
-Pre-requisites for running HBase as a distributed cluster
- * 3 or 5 !ZooKeeper nodes
- * Atleast 3 !RegionServer nodes. It would be ideal to run the !DataNodes on the same hosts as the Region servers for data locality.
-
-HBase tablename in Titan can be set using the following configuration in ATLAS_HOME/conf/atlas-application.properties:
+HBase tables used by Atlas can be set using the following configurations:
<verbatim>
atlas.graph.storage.hbase.table=apache_atlas_titan
-atlas.audit.hbase.tablename=apache_atlas_entity_audit
-</verbatim>
+atlas.audit.hbase.tablename=apache_atlas_entity_audit</verbatim>
*Configuring SOLR as the Indexing Backend for the Graph Repository*
-By default, Atlas uses Titan as the graph repository and is the only graph repository implementation available currently.
-For configuring Titan to work with Solr, please follow the instructions below
+By default, Atlas uses Titan as the graph repository and is the only graph repository implementation available currently. For configuring Titan to work with Solr, please follow the instructions below
* Install solr if not already running. The version of SOLR supported is 5.2.1. Could be installed from http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz
@@ -251,18 +143,14 @@ For configuring Titan to work with Solr, please follow the instructions below
* For e.g., to bring up a Solr node listening on port 8983 on a machine, you can use the command:
<verbatim>
- $SOLR_HOME/bin/solr start -c -z <zookeeper_host:port> -p 8983
- </verbatim>
+ $SOLR_HOME/bin/solr start -c -z <zookeeper_host:port> -p 8983</verbatim>
- * Run the following commands from SOLR_BIN (e.g. $SOLR_HOME/bin) directory to create collections in Solr corresponding to the indexes that Atlas uses. In the case that the ATLAS and SOLR instance are on 2 different hosts,
- first copy the required configuration files from ATLAS_HOME/conf/solr on the ATLAS instance host to the Solr instance host. SOLR_CONF in the below mentioned commands refer to the directory where the solr configuration files
- have been copied to on Solr host:
+ * Run the following commands from SOLR_BIN (e.g. $SOLR_HOME/bin) directory to create collections in Solr corresponding to the indexes that Atlas uses. In the case that the ATLAS and SOLR instance are on 2 different hosts, first copy the required configuration files from ATLAS_HOME/conf/solr on the ATLAS instance host to the Solr instance host. SOLR_CONF in the below mentioned commands refer to the directory where the solr configuration files have been copied to on Solr host:
<verbatim>
$SOLR_BIN/solr create -c vertex_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor
$SOLR_BIN/solr create -c edge_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor
- $SOLR_BIN/solr create -c fulltext_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor
-</verbatim>
+ $SOLR_BIN/solr create -c fulltext_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor</verbatim>
Note: If numShards and replicationFactor are not specified, they default to 1 which suffices if you are trying out solr with ATLAS on a single node instance.
Otherwise specify numShards according to the number of hosts that are in the Solr cluster and the maxShardsPerNode configuration.
@@ -279,8 +167,7 @@ For configuring Titan to work with Solr, please follow the instructions below
atlas.graph.index.search.solr.mode=cloud
atlas.graph.index.search.solr.zookeeper-url=<the ZK quorum setup for solr as comma separated value> eg: 10.1.6.4:2181,10.1.6.5:2181
atlas.graph.index.search.solr.zookeeper-connect-timeout=<SolrCloud Zookeeper Connection Timeout>. Default value is 60000 ms
- atlas.graph.index.search.solr.zookeeper-session-timeout=<SolrCloud Zookeeper Session Timeout>. Default value is 60000 ms
-</verbatim>
+ atlas.graph.index.search.solr.zookeeper-session-timeout=<SolrCloud Zookeeper Session Timeout>. Default value is 60000 ms</verbatim>
* Restart Atlas
@@ -323,61 +210,108 @@ in the =atlas-application.properties= file.
---++++ Starting Atlas Server
<verbatim>
-bin/atlas_start.py [-port <port>]
-</verbatim>
-
-By default,
- * To change the port, use -port option.
- * atlas server starts with conf from {package dir}/conf. To override this (to use the same conf with multiple atlas upgrades), set environment variable ATLAS_CONF to the path of conf dir
+bin/atlas_start.py [-port <port>]</verbatim>
---+++ Using Atlas
-
- * Quick start model - sample model and data
+ * Verify if the server is up and running
<verbatim>
- bin/quick_start.py [<atlas endpoint>]
-</verbatim>
+ curl -v -u username:password http://localhost:21000/api/atlas/admin/version
+ {"Version":"v0.1"}</verbatim>
- * Verify if the server is up and running
+ * Access Atlas UI using a browser: http://localhost:21000
+
+ * Quick start model - sample model and data
<verbatim>
- curl -v http://localhost:21000/api/atlas/admin/version
- {"Version":"v0.1"}
-</verbatim>
+ bin/quick_start.py [<atlas endpoint>]</verbatim>
* List the types in the repository
<verbatim>
- curl -v http://localhost:21000/api/atlas/types
- {"results":["Process","Infrastructure","DataSet"],"count":3,"requestId":"1867493731@qtp-262860041-0 - 82d43a27-7c34-4573-85d1-a01525705091"}
-</verbatim>
+ curl -v -u username:password http://localhost:21000/api/atlas/v2/types/typedefs/headers
+ [ {"guid":"fa421be8-c21b-4cf8-a226-fdde559ad598","name":"Referenceable","category":"ENTITY"},
+ {"guid":"7f3f5712-521d-450d-9bb2-ba996b6f2a4e","name":"Asset","category":"ENTITY"},
+ {"guid":"84b02fa0-e2f4-4cc4-8b24-d2371cd00375","name":"DataSet","category":"ENTITY"},
+ {"guid":"f93975d5-5a5c-41da-ad9d-eb7c4f91a093","name":"Process","category":"ENTITY"},
+ {"guid":"79dcd1f9-f350-4f7b-b706-5bab416f8206","name":"Infrastructure","category":"ENTITY"}
+ ]</verbatim>
* List the instances for a given type
<verbatim>
- curl -v http://localhost:21000/api/atlas/entities?type=hive_table
- {"requestId":"788558007@qtp-44808654-5","list":["cb9b5513-c672-42cb-8477-b8f3e537a162","ec985719-a794-4c98-b98f-0509bd23aac0","48998f81-f1d3-45a2-989a-223af5c1ed6e","a54b386e-c759-4651-8779-a099294244c4"]}
-
- curl -v http://localhost:21000/api/atlas/entities/list/hive_db
-</verbatim>
-
- * Search for entities (instances) in the repository
-<verbatim>
- curl -v http://localhost:21000/api/atlas/discovery/search/dsl?query="from hive_table"
-</verbatim>
-
-
-*Dashboard*
-
-Once atlas is started, you can view the status of atlas entities using the Web-based dashboard. You can open your browser at the corresponding port to use the web UI.
+ curl -v -u username:password http://localhost:21000/api/atlas/v2/search/basic?typeName=hive_db
+ {
+ "queryType":"BASIC",
+ "searchParameters":{
+ "typeName":"hive_db",
+ "excludeDeletedEntities":false,
+ "includeClassificationAttributes":false,
+ "includeSubTypes":true,
+ "includeSubClassifications":true,
+ "limit":100,
+ "offset":0
+ },
+ "entities":[
+ {
+ "typeName":"hive_db",
+ "guid":"5d900c19-094d-4681-8a86-4eb1d6ffbe89",
+ "status":"ACTIVE",
+ "displayText":"default",
+ "classificationNames":[],
+ "attributes":{
+ "owner":"public",
+ "createTime":null,
+ "qualifiedName":"default@cl1",
+ "name":"default",
+ "description":"Default Hive database"
+ }
+ },
+ {
+ "typeName":"hive_db",
+ "guid":"3a0b14b0-ab85-4b65-89f2-e418f3f7f77c",
+ "status":"ACTIVE",
+ "displayText":"finance",
+ "classificationNames":[],
+ "attributes":{
+ "owner":"hive",
+ "createTime":null,
+ "qualifiedName":"finance@cl1",
+ "name":"finance",
+ "description":null
+ }
+ }
+ ]
+ }</verbatim>
+
+ * Search for entities
+ curl -v -u username:password http://localhost:21000/api/atlas/v2/search/dsl?query=hive_db%20where%20name='default'
+ {
+ "queryType":"DSL",
+ "queryText":"hive_db where name='default'",
+ "entities":[
+ {
+ "typeName":"hive_db",
+ "guid":"5d900c19-094d-4681-8a86-4eb1d6ffbe89",
+ "status":"ACTIVE",
+ "displayText":"default",
+ "classificationNames":[],
+ "attributes":{
+ "owner":"public",
+ "createTime":null,
+ "qualifiedName":"default@cl1",
+ "name":"default",
+ "description":
+ "Default Hive database"
+ }
+ }
+ ]
+ }</verbatim>
---+++ Stopping Atlas Server
-
<verbatim>
-bin/atlas_stop.py
-</verbatim>
-
----+++ Known Issues
+bin/atlas_stop.py</verbatim>
----++++ Setup
+---+++ Troubleshooting
+---++++ Setup issues
If the setup of Atlas service fails due to any reason, the next run of setup (either by an explicit invocation of
=atlas_start.py -setup= or by enabling the configuration option =atlas.server.run.setup.on.start=) will fail with
a message such as =A previous setup run may not have completed cleanly.=. In such cases, you would need to manually
@@ -385,4 +319,4 @@ ensure the setup can run and delete the Zookeeper node at =/apache_atlas/setup_i
run setup again.
If the setup failed due to HBase Titan schema setup errors, it may be necessary to repair the HBase schema. If no
-data has been stored, one can also disable and drop the 'titan' schema in HBase to let setup run again.
+data has been stored, one can also disable and drop the HBase tables used by Atlas and run setup again.
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/QuickStart.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/QuickStart.twiki b/docs/src/site/twiki/QuickStart.twiki
index a3c1b1e..dd648d0 100644
--- a/docs/src/site/twiki/QuickStart.twiki
+++ b/docs/src/site/twiki/QuickStart.twiki
@@ -1,9 +1,8 @@
----+ Quick Start Guide
+---+ Quick Start
---++ Introduction
-This quick start user guide is a simple client that adds a few sample type definitions modeled
-after the example as shown below. It also adds example entities along with traits as shown in the
-instance graph below.
+Quick start is a simple client that adds a few sample type definitions modeled after the example shown below.
+It also adds sample entities along with traits as shown in the instance graph below.
---+++ Example Type Definitions
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/Repository.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Repository.twiki b/docs/src/site/twiki/Repository.twiki
deleted file mode 100755
index b84b3b3..0000000
--- a/docs/src/site/twiki/Repository.twiki
+++ /dev/null
@@ -1,4 +0,0 @@
----+ Repository
-
----++ Introduction
-
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/TypeSystem.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/TypeSystem.twiki b/docs/src/site/twiki/TypeSystem.twiki
index b658cfa..6c82711 100755
--- a/docs/src/site/twiki/TypeSystem.twiki
+++ b/docs/src/site/twiki/TypeSystem.twiki
@@ -1,7 +1,6 @@
---+ Type System
---++ Overview
-
Atlas allows users to define a model for the metadata objects they want to manage. The model is composed of definitions
called ‘types’. Instances of ‘types’ called ‘entities’ represent the actual metadata objects that are managed. The Type
System is a component that allows users to define and manage the types and entities. All metadata objects managed by
@@ -9,7 +8,6 @@ Atlas out of the box (like Hive tables, for e.g.) are modelled using types and r
types of metadata in Atlas, one needs to understand the concepts of the type system component.
---++ Types
-
A ‘Type’ in Atlas is a definition of how a particular type of metadata objects are stored and accessed. A type
represents one or a collection of attributes that define the properties for the metadata object. Users with a
development background will recognize the similarity of a type to a ‘Class’ definition of object oriented programming
@@ -19,108 +17,100 @@ An example of a type that comes natively defined with Atlas is a Hive table. A H
attributes:
<verbatim>
-Name: hive_table
-MetaType: Class
-SuperTypes: DataSet
+Name: hive_table
+TypeCategory: Entity
+SuperTypes: DataSet
Attributes:
- name: String (name of the table)
- db: Database object of type hive_db
- owner: String
- createTime: Date
- lastAccessTime: Date
- comment: String
- retention: int
- sd: Storage Description object of type hive_storagedesc
- partitionKeys: Array of objects of type hive_column
- aliases: Array of strings
- columns: Array of objects of type hive_column
- parameters: Map of String keys to String values
- viewOriginalText: String
- viewExpandedText: String
- tableType: String
- temporary: Boolean
-</verbatim>
+ name: string
+ db: hive_db
+ owner: string
+ createTime: date
+ lastAccessTime: date
+ comment: string
+ retention: int
+ sd: hive_storagedesc
+ partitionKeys: array<hive_column>
+ aliases: array<string>
+ columns: array<hive_column>
+ parameters: map<string,string>
+ viewOriginalText: string
+ viewExpandedText: string
+ tableType: string
+ temporary: boolean</verbatim>
The following points can be noted from the above example:
* A type in Atlas is identified uniquely by a ‘name’
- * A type has a metatype. A metatype represents the type of this model in Atlas. Atlas has the following metatypes:
- * Basic metatypes: E.g. Int, String, Boolean etc.
+ * A type has a metatype. Atlas has the following metatypes:
+ * Primitive metatypes: boolean, byte, short, int, long, float, double, biginteger, bigdecimal, string, date
* Enum metatypes
- * Collection metatypes: E.g. Array, Map
- * Composite metatypes: E.g. Class, Struct, Trait
- * A type can ‘extend’ from a parent type called ‘supertype’ - by virtue of this, it will get to include the attributes that are defined in the supertype as well. This allows modellers to define common attributes across a set of related types etc. This is again similar to the concept of how Object Oriented languages define super classes for a class. It is also possible for a type in Atlas to extend from multiple super types.
+ * Collection metatypes: array, map
+ * Composite metatypes: Entity, Struct, Classification
+ * Entity & Classification types can ‘extend’ from other types, called ‘supertype’ - by virtue of this, it will get to include the attributes that are defined in the supertype as well. This allows modellers to define common attributes across a set of related types etc. This is again similar to the concept of how Object Oriented languages define super classes for a class. It is also possible for a type in Atlas to extend from multiple super types.
* In this example, every hive table extends from a pre-defined supertype called a ‘DataSet’. More details about this pre-defined types will be provided later.
- * Types which have a metatype of ‘Class’, ‘Struct’ or ‘Trait’ can have a collection of attributes. Each attribute has a name (e.g. ‘name’) and some other associated properties. A property can be referred to using an expression type_name.attribute_name. It is also good to note that attributes themselves are defined using Atlas metatypes.
+ * Types which have a metatype of ‘Entity’, ‘Struct’ or ‘Classification’ can have a collection of attributes. Each attribute has a name (e.g. ‘name’) and some other associated properties. A property can be referred to using an expression type_name.attribute_name. It is also good to note that attributes themselves are defined using Atlas metatypes.
* In this example, hive_table.name is a String, hive_table.aliases is an array of Strings, hive_table.db refers to an instance of a type called hive_db and so on.
- * Type references in attributes, (like hive_table.db) are particularly interesting. Note that using such an attribute, we can define arbitrary relationships between two types defined in Atlas and thus build rich models. Note that one can also collect a list of references as an attribute type (e.g. hive_table.cols which represents a list of references from hive_table to the hive_column type)
+ * Type references in attributes, (like hive_table.db) are particularly interesting. Note that using such an attribute, we can define arbitrary relationships between two types defined in Atlas and thus build rich models. Note that one can also collect a list of references as an attribute type (e.g. hive_table.columns which represents a list of references from hive_table to hive_column type)
---++ Entities
-
-An ‘entity’ in Atlas is a specific value or instance of a Class ‘type’ and thus represents a specific metadata object
+An ‘entity’ in Atlas is a specific value or instance of an Entity ‘type’ and thus represents a specific metadata object
in the real world. Referring back to our analogy of Object Oriented Programming languages, an ‘instance’ is an
‘Object’ of a certain ‘Class’.
An example of an entity will be a specific Hive Table. Say Hive has a table called ‘customers’ in the ‘default’
-database. This table will be an ‘entity’ in Atlas of type hive_table. By virtue of being an instance of a class
+database. This table will be an ‘entity’ in Atlas of type hive_table. By virtue of being an instance of an entity
type, it will have values for every attribute that are a part of the Hive table ‘type’, such as:
<verbatim>
-id: "9ba387dd-fa76-429c-b791-ffc338d3c91f"
-typeName: “hive_table”
+guid: "9ba387dd-fa76-429c-b791-ffc338d3c91f"
+typeName: "hive_table"
+status: "ACTIVE"
values:
- name: “customers”
- db: "b42c6cfc-c1e7-42fd-a9e6-890e0adf33bc"
- owner: “admin”
- createTime: "2016-06-20T06:13:28.000Z"
- lastAccessTime: "2016-06-20T06:13:28.000Z"
- comment: null
- retention: 0
- sd: "ff58025f-6854-4195-9f75-3a3058dd8dcf"
- partitionKeys: null
- aliases: null
- columns: ["65e2204f-6a23-4130-934a-9679af6a211f", "d726de70-faca-46fb-9c99-cf04f6b579a6", ...]
- parameters: {"transient_lastDdlTime": "1466403208"}
+ name: “customers”
+ db: { "guid": "b42c6cfc-c1e7-42fd-a9e6-890e0adf33bc", "typeName": "hive_db" }
+ owner: “admin”
+ createTime: 1490761686029
+ updateTime: 1516298102877
+ comment: null
+ retention: 0
+ sd: { "guid": "ff58025f-6854-4195-9f75-3a3058dd8dcf", "typeName": "hive_storagedesc" }
+ partitionKeys: null
+ aliases: null
+ columns: [ { "guid": ""65e2204f-6a23-4130-934a-9679af6a211f", "typeName": "hive_column" }, { "guid": ""d726de70-faca-46fb-9c99-cf04f6b579a6", "typeName": "hive_column" }, ...]
+ parameters: { "transient_lastDdlTime": "1466403208"}
viewOriginalText: null
viewExpandedText: null
- tableType: “MANAGED_TABLE”
- temporary: false
-</verbatim>
+ tableType: “MANAGED_TABLE”
+ temporary: false</verbatim>
The following points can be noted from the example above:
- * Every entity that is an instance of a Class type is identified by a unique identifier, a GUID. This GUID is generated by the Atlas server when the object is defined, and remains constant for the entire lifetime of the entity. At any point in time, this particular entity can be accessed using its GUID.
+ * Every instance ofan entity type is identified by a unique identifier, a GUID. This GUID is generated by the Atlas server when the object is defined, and remains constant for the entire lifetime of the entity. At any point in time, this particular entity can be accessed using its GUID.
* In this example, the ‘customers’ table in the default database is uniquely identified by the GUID "9ba387dd-fa76-429c-b791-ffc338d3c91f"
* An entity is of a given type, and the name of the type is provided with the entity definition.
* In this example, the ‘customers’ table is a ‘hive_table.
* The values of this entity are a map of all the attribute names and their values for attributes that are defined in the hive_table type definition.
- * Attribute values will be according to the metatype of the attribute.
- * Basic metatypes: integer, String, boolean values. E.g. ‘name’ = ‘customers’, ‘Temporary’ = ‘false’
- * Collection metatypes: An array or map of values of the contained metatype. E.g. parameters = { “transient_lastDdlTime”: “1466403208”}
- * Composite metatypes: For classes, the value will be an entity with which this particular entity will have a relationship. E.g. The hive table “customers” is present in a database called “default”. The relationship between the table and database are captured via the “db” attribute. Hence, the value of the “db” attribute will be a GUID that uniquely identifies the hive_db entity called “default”
-
-With this idea on entities, we can now see the difference between Class and Struct metatypes. Classes and Structs
-both compose attributes of other types. However, entities of Class types have the Id attribute (with a GUID value) a
-nd can be referenced from other entities (like a hive_db entity is referenced from a hive_table entity). Instances of
-Struct types do not have an identity of their own. The value of a Struct type is a collection of attributes that are
+ * Attribute values will be according to the datatype of the attribute. Entity-type attributes will have value of type AtlasObjectId
+
+With this idea on entities, we can now see the difference between Entity and Struct metatypes. Entities and Structs
+both compose attributes of other types. However, instances of Entity types have an identity (with a GUID value) and can
+be referenced from other entities (like a hive_db entity is referenced from a hive_table entity). Instances of Struct
+types do not have an identity of their own. The value of a Struct type is a collection of attributes that are
‘embedded’ inside the entity itself.
---++ Attributes
-
-We already saw that attributes are defined inside composite metatypes like Class and Struct. But we simplistically
-referred to attributes as having a name and a metatype value. However, attributes in Atlas have some more properties
-that define more concepts related to the type system.
+We already saw that attributes are defined inside metatypes like Entity, Struct and Classification. But we
+implicitly referred to attributes as having a name and a metatype value. However, attributes in Atlas have some more
+properties that define more concepts related to the type system.
An attribute has the following properties:
<verbatim>
- name: string,
- dataTypeName: string,
- isComposite: boolean,
+ name: string,
+ typeName: string,
+ isOptional: boolean,
isIndexable: boolean,
- isUnique: boolean,
- multiplicity: enum,
- reverseAttributeName: string
-</verbatim>
+ isUnique: boolean,
+ cardinality: enum</verbatim>
The properties above have the following meanings:
@@ -132,7 +122,7 @@ The properties above have the following meanings:
* isIndexable -
* This flag indicates whether this property should be indexed on, so that look ups can be performed using the attribute value as a predicate and can be performed efficiently.
* isUnique -
- * This flag is again related to indexing. If specified to be unique, it means that a special index is created for this attribute in Titan that allows for equality based look ups.
+ * This flag is again related to indexing. If specified to be unique, it means that a special index is created for this attribute in JanusGraph that allows for equality based look ups.
* Any attribute with a true value for this flag is treated like a primary key to distinguish this entity from other entities. Hence care should be taken ensure that this attribute does model a unique property in real world.
* For e.g. consider the name attribute of a hive_table. In isolation, a name is not a unique attribute for a hive_table, because tables with the same name can exist in multiple databases. Even a pair of (database name, table name) is not unique if Atlas is storing metadata of hive tables amongst multiple clusters. Only a cluster location, database name and table name can be deemed unique in the physical world.
* multiplicity - indicates whether this attribute is required, optional, or could be multi-valued. If an entity’s definition of the attribute value does not match the multiplicity declaration in the type definition, this would be a constraint violation and the entity addition will fail. This field can therefore be used to define some constraints on the metadata information.
@@ -142,59 +132,55 @@ Let us look at the attribute called ‘db’ which represents the database to wh
<verbatim>
db:
- "dataTypeName": "hive_db",
- "isComposite": false,
+ "name": "db",
+ "typeName": "hive_db",
+ "isOptional": false,
"isIndexable": true,
- "isUnique": false,
- "multiplicity": "required",
- "name": "db",
- "reverseAttributeName": null
-</verbatim>
+ "isUnique": false,
+ "cardinality": "SINGLE"</verbatim>
-Note the “required” constraint on multiplicity. A table entity cannot be sent without a db reference.
+Note the “isOptional=true” constraint - a table entity cannot be created without a db reference.
<verbatim>
columns:
- "dataTypeName": "array<hive_column>",
- "isComposite": true,
+ "name": "columns",
+ "typeName": "array<hive_column>",
+ "isOptional": optional,
"isIndexable": true,
- “isUnique": false,
- "multiplicity": "optional",
- "name": "columns",
- "reverseAttributeName": null
-</verbatim>
+ “isUnique": false,
+ "constraints": [ { "type": "ownedRef" } ]</verbatim>
-Note the “isComposite” true value for columns. By doing this, we are indicating that the defined column entities should
+Note the “ownedRef” constraint for columns. By doing this, we are indicating that the defined column entities should
always be bound to the table entity they are defined with.
From this description and examples, you will be able to realize that attribute definitions can be used to influence
specific modelling behavior (constraints, indexing, etc) to be enforced by the Atlas system.
---++ System specific types and their significance
-
-Atlas comes with a few pre-defined system types. We saw one example (DataSet) in the preceding sections. In this
-section we will see all these types and understand their significance.
+Atlas comes with a few pre-defined system types. We saw one example (DataSet) in preceding sections. In this
+section we will see more of these types and understand their significance.
*Referenceable*: This type represents all entities that can be searched for using a unique attribute called
qualifiedName.
-*Asset*: This type contains attributes like name, description and owner. Name is a required attribute
-(multiplicity = required), the others are optional. The purpose of Referenceable and Asset is to provide modellers
-with way to enforce consistency when defining and querying entities of their own types. Having these fixed set of
-attributes allows applications and User interfaces to make convention based assumptions about what attributes they can
-expect of types by default.
-
-*Infrastructure*: This type extends Referenceable and Asset and typically can be used to be a common super type for
-infrastructural metadata objects like clusters, hosts etc.
-
-*!DataSet*: This type extends Referenceable and Asset. Conceptually, it can be used to represent an type that stores
-data. In Atlas, hive tables, Sqoop RDBMS tables etc are all types that extend from !DataSet. Types that extend !DataSet
-can be expected to have a Schema in the sense that they would have an attribute that defines attributes of that dataset.
-For e.g. the columns attribute in a hive_table. Also entities of types that extend !DataSet participate in data
-transformation and this transformation can be captured by Atlas via lineage (or provenance) graphs.
-
-*Process*: This type extends Referenceable and Asset. Conceptually, it can be used to represent any data transformation
-operation. For example, an ETL process that transforms a hive table with raw data to another hive table that stores
-some aggregate can be a specific type that extends the Process type. A Process type has two specific attributes,
-inputs and outputs. Both inputs and outputs are arrays of !DataSet entities. Thus an instance of a Process type can
-use these inputs and outputs to capture how the lineage of a !DataSet evolves.
\ No newline at end of file
+*Asset*: This type extends Referenceable and adds attributes like name, description and owner. Name is a required
+attribute (isOptional=false), the others are optional.
+
+The purpose of Referenceable and Asset is to provide modellers with way to enforce consistency when defining and
+querying entities of their own types. Having these fixed set of attributes allows applications and user interfaces to
+make convention based assumptions about what attributes they can expect of types by default.
+
+*Infrastructure*: This type extends Asset and typically can be used to be a common super type for infrastructural
+metadata objects like clusters, hosts etc.
+
+*!DataSet*: This type extends Referenceable. Conceptually, it can be used to represent an type that stores data. In Atlas,
+hive tables, hbase_tables etc are all types that extend from !DataSet. Types that extend !DataSet can be expected to have
+a Schema in the sense that they would have an attribute that defines attributes of that dataset. For e.g. the columns
+attribute in a hive_table. Also entities of types that extend !DataSet participate in data transformation and this
+transformation can be captured by Atlas via lineage (or provenance) graphs.
+
+*Process*: This type extends Asset. Conceptually, it can be used to represent any data transformation operation. For
+example, an ETL process that transforms a hive table with raw data to another hive table that stores some aggregate can
+be a specific type that extends the Process type. A Process type has two specific attributes, inputs and outputs. Both
+inputs and outputs are arrays of !DataSet entities. Thus an instance of a Process type can use these inputs and outputs
+to capture how the lineage of a !DataSet evolves.
http://git-wip-us.apache.org/repos/asf/atlas/blob/dabb9f4a/docs/src/site/twiki/index.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/index.twiki b/docs/src/site/twiki/index.twiki
index a8e7de9..198ac52 100755
--- a/docs/src/site/twiki/index.twiki
+++ b/docs/src/site/twiki/index.twiki
@@ -7,39 +7,49 @@ Atlas is a scalable and extensible set of core foundational governance services
enterprises to effectively and efficiently meet their compliance requirements within Hadoop and
allows integration with the whole enterprise data ecosystem.
+Apache Atlas provides open metadata management and governance capabilities for organizations
+to build a catalog of their data assets, classify and govern these assets and provide collaboration
+capabilities around these data assets for data scientists, analysts and the data governance team.
+
---++ Features
----+++ Data Classification
- * Import or define taxonomy business-oriented annotations for data
- * Define, annotate, and automate capture of relationships between data sets and underlying elements including source, target, and derivation processes
- * Export metadata to third-party systems
+---+++ Metadata types & instances
+ * Pre-defined types for various Hadoop and non-Hadoop metadata
+ * Ability to define new types for the metadata to be managed
+ * Types can have primitive attributes, complex attributes, object references; can inherit from other types
+ * Instances of types, called entities, capture metadata object details and their relationships
+ * REST APIs to work with types and instances allow easier integration
+
+---+++ Classification
+ * Ability to dynamically create classifications - like PII, EXPIRES_ON, DATA_QUALITY, SENSITIVE
+ * Classifications can include attributes - like expiry_date attribute in EXPIRES_ON classification
+ * Entities can be associated with multiple classifications, enabling easier discovery and security enforcement
----+++ Centralized Auditing
- * Capture security access information for every application, process, and interaction with data
- * Capture the operational information for execution, steps, and activities
+---+++ Lineage
+ * Intuitive UI to view lineage of data as it moves through various processes
+ * REST APIs to access and update lineage
----+++ Search & Lineage (Browse)
- * Pre-defined navigation paths to explore the data classification and audit information
- * Text-based search features locates relevant data and audit event across Data Lake quickly and accurately
- * Browse visualization of data set lineage allowing users to drill-down into operational, security, and provenance related information
+---+++ Search/Discovery
+ * Intuitive UI to search entities by type, classification, attribute value or free-text
+ * Rich REST APIs to search by complex criteria
+ * SQL like query language to search entities - Domain Specific Language (DSL)
----+++ Security & Policy Engine
- * Rationalize compliance policy at runtime based on data classification schemes, attributes and roles.
- * Advanced definition of policies for preventing data derivation based on classification (i.e. re-identification) – Prohibitions
- * Column and Row level masking based on cell values and attibutes.
+---+++ Security & Data Masking
+ * Integration with Apache Ranger enables authorization/data-masking based on classifications associated with entities in Apache Atlas. For example:
+ * who can access data classified as PII, SENSITIVE
+ * customer-service users can only see last 4 digits of columns classified as NATIONAL_ID
---++ Getting Started
- * [[InstallationSteps][Install Steps]]
- * [[QuickStart][Quick Start Guide]]
+ * [[InstallationSteps][Build & Install]]
+ * [[QuickStart][Quick Start]]
---++ Documentation
* [[Architecture][High Level Architecture]]
* [[TypeSystem][Type System]]
- * [[Repository][Metadata Repository]]
* [[Search][Search]]
* [[security][Security]]
* [[Authentication-Authorization][Authentication and Authorization]]