You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@atlas.apache.org by ma...@apache.org on 2017/11/27 04:59:41 UTC
atlas git commit: ATLAS-2280: Recovering lost/bad index data
Repository: atlas
Updated Branches:
refs/heads/branch-0.8 f6502c9e5 -> 0e80d9b37
ATLAS-2280: Recovering lost/bad index data
Signed-off-by: Madhan Neethiraj <ma...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/0e80d9b3
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/0e80d9b3
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/0e80d9b3
Branch: refs/heads/branch-0.8
Commit: 0e80d9b37df5654b341c0aeb129c86597147ed4f
Parents: f6502c9
Author: Ashutosh Mestry <am...@hortonworks.com>
Authored: Sun Nov 26 20:28:22 2017 -0800
Committer: Madhan Neethiraj <ma...@apache.org>
Committed: Sun Nov 26 20:29:43 2017 -0800
----------------------------------------------------------------------
tools/atlas-index-repair-kit/README.txt | 75 +++++++++++++++++++
.../atlas-index-repair-kit.tar.gz | Bin 0 -> 1252999 bytes
2 files changed, 75 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/atlas/blob/0e80d9b3/tools/atlas-index-repair-kit/README.txt
----------------------------------------------------------------------
diff --git a/tools/atlas-index-repair-kit/README.txt b/tools/atlas-index-repair-kit/README.txt
new file mode 100644
index 0000000..4e10bdf
--- /dev/null
+++ b/tools/atlas-index-repair-kit/README.txt
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+Follow the instructions below to rebuid Atlas index data from the data
+stored in data-store (like HBase).
+
+Please note that these steps have been verified in an Atlas environment
+with Titan 0.5.4, HBase and Solr. For other variations, please refer to
+the relevant documentation of the graph DB.
+
+1. Setup titan-0.5.4 in the host where Atlas is installed
+ - Download titan-0.5.4 Hadoop-2 distribution from http://s3.thinkaurelius.com/downloads/titan/titan-0.5.4-hadoop2.zip
+ - cd /tmp/
+ - unzip titan-0.5.4-hadoop2.zip
+
+2. Add Atlas index repair kit to titan-0.5.4 installation, by running the following commands:
+ - cd /tmp/titan-0.5.4-hadoop2
+ - tar xvf atlas-index-repair-kit.tar
+
+3. Update atlas-conf/atlas-titan.properties with details to connect to data-store (like HBase). For example:
+ storage.backend=hbase
+ storage.hostname=fqdn
+ storage.hbase.table=atlas_titan
+
+4. Update bin/atlas-gremlin.sh to set the following variables at the top. For example:
+ ATLAS_WEBAPP_DIR=/home/atlas/atlas-server/server/webapp
+ STORE_CONF_DIR=/etc/hbase/conf
+
+5. If Atlas is run in a kerberized environment, setup the following:
+ 5.1. Update atlas-conf/atlas-titan.properties with necessary kerberos details for the data-store and index. For example:
+ hbase.security.authentication=kerberos
+ hbase.security.authorization=true
+ hbase.rpc.engine=org.apache.hadoop.hbase.ipc.SecureRpcEngine
+ index.search.backend=solr5
+ index.search.solr.mode=cloud
+ index.search.solr.zookeeper-url=fqdn:2181/infra-solr
+
+ 5.2. Copy necessary configuration files from the deployment to atlas-conf folder. For example:
+ hadoop-client/conf/core-site.xml
+ hadoop-client/conf/hdfs-site.xml
+ hadoop-client/conf/yarn-site.xml
+ ambari-infra-solr/conf/infra_solr_jaas.conf
+ ambari-infra-solr/conf/security.json
+
+ 5.3. Update bin/atlas-gremlin.sh to set the following variable at the top. For example:
+ JAAS_CONF_FILE=infra_solr_jaas.conf
+
+ 5.4. Kinit as atlas user with the command like:
+ kinit -kt /etc/security/keytabs/atlas.service.keytab atlas/fqdn@EXAMPLE.COM
+
+6. Ensure home directory for 'atlas' user exists in HDFS and this directory is owned by 'atlas' user
+ su hdfs
+ hdfs dfs -mkdir /user/atlas
+ hdfs dfs -chown atlas:hdfs -R /user/atlas
+
+7. Start Gremlin shell by executing the following command:
+ bin/atlas-gremlin.sh bin/atlas-index-repair.groovy
+
+8. Start index repair by entering the following in the Gremlin shell:
+ repairAtlasIndex("atlas-conf/atlas-titan.properties")
http://git-wip-us.apache.org/repos/asf/atlas/blob/0e80d9b3/tools/atlas-index-repair-kit/atlas-index-repair-kit.tar.gz
----------------------------------------------------------------------
diff --git a/tools/atlas-index-repair-kit/atlas-index-repair-kit.tar.gz b/tools/atlas-index-repair-kit/atlas-index-repair-kit.tar.gz
new file mode 100644
index 0000000..f4d4525
Binary files /dev/null and b/tools/atlas-index-repair-kit/atlas-index-repair-kit.tar.gz differ