You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Alejandro Fernandez (JIRA)" <ji...@apache.org> on 2016/09/13 01:48:20 UTC
[jira] [Commented] (AMBARI-18368) Atlas web UI alert after
performing stack upgrade to HDP 2.5 and adding Atlas Service
[ https://issues.apache.org/jira/browse/AMBARI-18368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485944#comment-15485944 ]
Alejandro Fernandez commented on AMBARI-18368:
----------------------------------------------
To summarize, Atlas is failing because config files do not have the correct permissions as they are not managed by Ambari.
My local cluster went through HDP 2.4, EU to HDP 2.5, and then added Atlas.
Before the EU, /usr/hdp/2.5.0.0-1245/etc/ doesn't contain any atlas dirs, as it should.
After EU, I ran "yum install atlas-metadata_2_5_0_0_1245" and this creates /usr/hdp/2.5.0.0-1245/etc/atlas/conf.dist/ , which contains the "solr" subdirectory, policy-store.txt, and users-credentials.properties
At this point in time, those 2 files and the solr dir are present in two places,
1.
{code}
/usr/hdp/current/atlas-client -> /usr/hdp/2.5.0.0-1245/atlas
/usr/hdp/current/atlas-client/conf -> /etc/atlas/conf
[root@c6404 etc]# ls -la /etc/atlas/conf (this is a directory that will be backed up, since Ambari needs to make this a symlink instead)
-rw-r--r-- 1 root root 8054 Aug 26 04:49 atlas-application.properties
-rw-r--r-- 1 root root 3208 Aug 26 04:49 atlas-env.sh
-rw-r--r-- 1 root root 3912 Aug 26 04:49 atlas-log4j.xml
drwxr-xr-x 2 root root 4096 Sep 13 00:13 hbase
-rw-r--r-- 1 root root 623 Aug 26 04:49 policy-store.txt
drwxr-xr-x 3 root root 4096 Sep 13 00:13 solr
-rw-r--r-- 1 root root 207 Aug 26 04:49 users-credentials.properties
{code}
And 2 (this dir is not managed by Ambari),
{code}
[root@c6404 etc]# ls -la /usr/hdp/2.5.0.0-1245/etc/atlas/conf.dist/
-rw-r--r-- 1 root root 8054 Aug 26 04:49 atlas-application.properties
-rw-r--r-- 1 root root 3208 Aug 26 04:49 atlas-env.sh
-rw-r--r-- 1 root root 3912 Aug 26 04:49 atlas-log4j.xml
drwxr-xr-x 2 root root 4096 Sep 13 00:13 hbase
-rw-r--r-- 1 root root 623 Aug 26 04:49 policy-store.txt
drwxr-xr-x 3 root root 4096 Sep 13 00:13 solr
-rw-r--r-- 1 root root 207 Aug 26 04:49 users-credentials.properties
{code}
During the Atlas Install command, Ambari log shows this,
{code}
2016-09-13 00:22:52,189 - Seeding versioned configuration directories for atlas
2016-09-13 00:22:52,189 - Execute['ambari-sudo.sh -H -E cp -R -p -v /usr/hdp/current/atlas-client/conf/* /etc/atlas/2.5.0.0-1245/0'] {'logoutput': True}
`/usr/hdp/current/atlas-client/conf/atlas-application.properties' -> `/etc/atlas/2.5.0.0-1245/0/atlas-application.properties'
`/usr/hdp/current/atlas-client/conf/atlas-env.sh' -> `/etc/atlas/2.5.0.0-1245/0/atlas-env.sh'
`/usr/hdp/current/atlas-client/conf/atlas-log4j.xml' -> `/etc/atlas/2.5.0.0-1245/0/atlas-log4j.xml'
`/usr/hdp/current/atlas-client/conf/atlas_jaas.conf' -> `/etc/atlas/2.5.0.0-1245/0/atlas_jaas.conf'
`/usr/hdp/current/atlas-client/conf/hbase' -> `/etc/atlas/2.5.0.0-1245/0/hbase'
`/usr/hdp/current/atlas-client/conf/hbase/hbase-site.xml.template' -> `/etc/atlas/2.5.0.0-1245/0/hbase/hbase-site.xml.template'
`/usr/hdp/current/atlas-client/conf/policy-store.txt' -> `/etc/atlas/2.5.0.0-1245/0/policy-store.txt'
`/usr/hdp/current/atlas-client/conf/solr' -> `/etc/atlas/2.5.0.0-1245/0/solr'
`/usr/hdp/current/atlas-client/conf/solr/lang' -> `/etc/atlas/2.5.0.0-1245/0/solr/lang'
`/usr/hdp/current/atlas-client/conf/solr/lang/stopwords_en.txt' -> `/etc/atlas/2.5.0.0-1245/0/solr/lang/stopwords_en.txt'
`/usr/hdp/current/atlas-client/conf/solr/stopwords.txt' -> `/etc/atlas/2.5.0.0-1245/0/solr/stopwords.txt'
`/usr/hdp/current/atlas-client/conf/solr/synonyms.txt' -> `/etc/atlas/2.5.0.0-1245/0/solr/synonyms.txt'
`/usr/hdp/current/atlas-client/conf/solr/schema.xml' -> `/etc/atlas/2.5.0.0-1245/0/solr/schema.xml'
`/usr/hdp/current/atlas-client/conf/solr/currency.xml' -> `/etc/atlas/2.5.0.0-1245/0/solr/currency.xml'
`/usr/hdp/current/atlas-client/conf/solr/solrconfig.xml' -> `/etc/atlas/2.5.0.0-1245/0/solr/solrconfig.xml'
`/usr/hdp/current/atlas-client/conf/solr/protwords.txt' -> `/etc/atlas/2.5.0.0-1245/0/solr/protwords.txt'
`/usr/hdp/current/atlas-client/conf/users-credentials.properties' -> `/etc/atlas/2.5.0.0-1245/0/users-credentials.properties'
2016-09-13 00:22:52,195 - Execute['ambari-sudo.sh -H -E cp -R -p /etc/atlas/conf/* /etc/atlas/2.5.0.0-1245/0'] {'only_if': 'ls -d /etc/atlas/conf/*'}
2016-09-13 00:22:52,204 - Checking if need to create versioned conf dir /etc/atlas/2.5.0.0-1245/0
2016-09-13 00:22:52,204 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'atlas', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2016-09-13 00:22:52,224 - call returned (1, '/etc/atlas/2.5.0.0-1245/0 exist already', '')
2016-09-13 00:22:52,225 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'atlas', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False}
2016-09-13 00:22:52,243 - checked_call returned (0, '/usr/hdp/2.5.0.0-1245/atlas/conf -> /etc/atlas/2.5.0.0-1245/0')
2016-09-13 00:22:52,244 - Ensuring that atlas has the correct symlink structure
2016-09-13 00:22:52,245 - Execute[('cp', '-R', '-p', '/etc/atlas/conf', '/etc/atlas/conf.backup')] {'not_if': 'test -e /etc/atlas/conf.backup', 'sudo': True}
2016-09-13 00:22:52,247 - Skipping Execute[('cp', '-R', '-p', '/etc/atlas/conf', '/etc/atlas/conf.backup')] due to not_if
2016-09-13 00:22:52,247 - Directory['/etc/atlas/conf'] {'action': ['delete']}
2016-09-13 00:22:52,248 - Removing directory Directory['/etc/atlas/conf'] and all its content
2016-09-13 00:22:52,249 - Link['/etc/atlas/conf'] {'to': '/etc/atlas/conf.backup'}
2016-09-13 00:22:52,249 - Creating symbolic Link['/etc/atlas/conf'] to /etc/atlas/conf.backup
2016-09-13 00:22:52,249 - Link['/etc/atlas/conf'] {'action': ['delete']}
2016-09-13 00:22:52,249 - Deleting Link['/etc/atlas/conf']
2016-09-13 00:22:52,249 - Link['/etc/atlas/conf'] {'to': '/usr/hdp/current/atlas-client/conf'}
2016-09-13 00:22:52,249 - Creating symbolic Link['/etc/atlas/conf'] to /usr/hdp/current/atlas-client/conf
{code}
After it is done, the system looks like this,
/etc/atlas/conf -> /usr/hdp/current/atlas-client/conf
/usr/hdp/current/atlas-client -> /usr/hdp/2.5.0.0-1245/atlas
/usr/hdp/2.5.0.0-1245/atlas/conf -> /etc/atlas/2.5.0.0-1245/0
{code}
[root@c6404 etc]# ls -la /usr/hdp/2.5.0.0-1245/atlas/conf/
-rw-r--r-- 1 atlas hadoop 5120 Sep 13 00:24 atlas-application.properties
-rwxr-xr-x 1 atlas hadoop 1679 Sep 13 00:24 atlas-env.sh
-rw-r--r-- 1 atlas root 261 Sep 13 00:22 atlas_jaas.conf
-rw-r--r-- 1 atlas hadoop 3259 Sep 13 00:24 atlas-log4j.xml
drwxr-xr-x 2 root root 4096 Sep 13 00:13 hbase
-rw-r--r-- 1 root root 623 Aug 26 04:49 policy-store.txt
drwxr-xr-x 3 atlas hadoop 4096 Sep 13 00:13 solr
-rw-r--r-- 1 root root 207 Aug 26 04:49 users-credentials.properties
{code}
Fix was to change ownership of users-credentials.properties and policy-store.txt to atlas:hadoop
> Atlas web UI alert after performing stack upgrade to HDP 2.5 and adding Atlas Service
> -------------------------------------------------------------------------------------
>
> Key: AMBARI-18368
> URL: https://issues.apache.org/jira/browse/AMBARI-18368
> Project: Ambari
> Issue Type: Bug
> Components: stacks
> Affects Versions: 2.4.0
> Reporter: Alejandro Fernandez
> Assignee: Alejandro Fernandez
> Priority: Critical
> Fix For: trunk, 2.5.0
>
> Attachments: AMBARI-18368.patch
>
>
> Steps to Reproduce:
> * Install Ambari 2.2.2 with HDP 2.4 (HBase, Solr)
> * Kerberize the cluster
> * Perform EU/RU to HDP 2.5
> * Add Atlas Service
> Atlas Server log contains,
> {code}
> Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://natu146-ehbs-dgm10toeriesec-u14-1.openstacklocal:8886/solr: Can not find the specified config set: vertex_index
> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:577)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
> at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.createCollectionIfNotExists(Solr5Index.java:901)
> at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.register(Solr5Index.java:269)
> at com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction.register(IndexTransaction.java:83)
> at com.thinkaurelius.titan.graphdb.database.IndexSerializer.register(IndexSerializer.java:92)
> at com.thinkaurelius.titan.graphdb.database.management.ManagementSystem.addIndexKey(ManagementSystem.java:534)
> at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.enhanceMixedIndex(GraphBackedSearchIndexer.java:405)
> at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.createIndexes(GraphBackedSearchIndexer.java:334)
> at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.initialize(GraphBackedSearchIndexer.java:103)
> ... 71 more
> {code}
> Atlas tables in HBase look ok.
> {code}
> su hbase
> kinit -kt /etc/security/keytabs/hbase.headless.keytab cstm-hbase@EXAMPLE.COM
> hbase shell
> hbase(main):001:0> list
> TABLE
> ATLAS_ENTITY_AUDIT_EVENTS
> atlas_titan
> 2 row(s) in 1.4300 seconds
> => ["ATLAS_ENTITY_AUDIT_EVENTS", "atlas_titan"]
> {code}
> h4. Workaround
> 1. Stop Atlas Server
> 2. Copy solr xml files to correct config folder and chown as $atlas_user:$hadoop_group
> {code}
> cp -R /usr/hdp/2.5.0.0-####/etc/atlas/conf.dist/solr/* /etc/atlas/conf/solr/
> cp: overwrite `/etc/atlas/conf/solr/solrconfig.xml'? n
> chown atlas:hadoop /etc/atlas/conf/solr/*
> cp /usr/hdp/2.5.0.0-####/etc/atlas/conf.dist/users-credentials.properties /etc/atlas/conf/
> cp /usr/hdp/2.5.0.0-####/etc/atlas/conf.dist/policy-store.txt /etc/atlas/conf/
> chown atlas:hadoop /etc/atlas/conf/users-credentials.properties
> chown atlas:hadoop /etc/atlas/conf/policy-store.txt
> {code}
> 3. Delete zookeeper znode,
> {code}
> # kinit -kt /etc/security/keytabs/atlas.service.keytab atlas/<HOST>@<DOMAIN>
> # cd /usr/hdp/current/zookeeper-client/bin/
> # ./zkCli.sh -server <zookeepernode>:<zookeeperport>
> [ ...... (CONNECTED) ] rmr /infra-solr/configs/atlas_configs
> {code}
> 4. Ensure Atlas application-properties are present for,
> atlas.jaas.KafkaClient.option.keyTab = /etc/security/keytabs/atlas.service.keytab
> atlas.jaas.KafkaClient.option.principal = atlas/_HOST@EXAMPLE.COM
> 5. Start Atlas
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)