You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Greg Senia (JIRA)" <ji...@apache.org> on 2017/02/10 11:04:41 UTC
[jira] [Commented] (ATLAS-1546) Hive hook should choose appropriate
JAAS config if host uses kerberos ticket-cache
[ https://issues.apache.org/jira/browse/ATLAS-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861107#comment-15861107 ]
Greg Senia commented on ATLAS-1546:
-----------------------------------
I accomplished this by adding a new definition to Ambari
Here is a workaround to make HiveCLI work from Ambari... A bit involved but it works..
The work around to make Ambari generate a hive-cli-atlas-application.properties file is as follows:
curl -u username -H "X-Requested-By: ambari" -X PUT -d @atlas-hivecli.json "http://localhost:8080/api/v1/clusters/tech"
atkas-hivecli.json:
[{"Clusters":{
"desired_config":[{
"type" : "hive-cli-atlas-application.properties",
"properties" : {
"atlas.hook.hive.keepAliveTime" : "10",
"atlas.hook.hive.maxThreads" : "5",
"atlas.hook.hive.minThreads" : "5",
"atlas.hook.hive.numRetries" : "3",
"atlas.hook.hive.queueSize" : "1000",
"atlas.hook.hive.synchronous" : "false",
"atlas.jaas.KafkaClient.loginModuleControlFlag" : "required",
"atlas.jaas.KafkaClient.loginModuleName" : "com.sun.security.auth.module.Krb5LoginModule",
"atlas.jaas.KafkaClient.option.serviceName" : "kafka",
"atlas.jaas.KafkaClient.option.renewTicket" : "True",
"atlas.jaas.KafkaClient.option.storeKey" : "false",
"atlas.jaas.KafkaClient.option.useKeyTab" : "false",
"atlas.jaas.KafkaClient.option.useTicketCache" : "True"
}
}
]
}
}
]
Apply the following to /var/lib/ambari-server/resources/stacks/HDP/HDP-2.5
[username@hadoop1 ~]$ cat stacks_ambari.patch
--- /dev/null
+++ /var/lib/ambari-server/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-cli-atlas-application.properties.xml 2017-02-09 14:10:05.000000000 -0500
@@ -0,0 +1,61 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+-->
+<configuration supports_final="false">
+ <!-- These are the Atlas Hooks properties specific to this service. This file is then merged with common properties
+ that apply to all services. -->
+ <property>
+ <name>atlas.hook.hive.synchronous</name>
+ <value>false</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>atlas.hook.hive.numRetries</name>
+ <value>3</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>atlas.hook.hive.minThreads</name>
+ <value>5</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>atlas.hook.hive.maxThreads</name>
+ <value>5</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>atlas.hook.hive.keepAliveTime</name>
+ <value>10</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>atlas.hook.hive.queueSize</name>
+ <value>1000</value>
+ <description/>
+ <on-ambari-upgrade add="true"/>
+ </property>
+</configuration>
--- /var/lib/ambari-server/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml 2016-11-23 02:27:15.000000000 -0500
+++ /var/lib/ambari-server/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml 2017-02-09 09:46:52.000000000 -0500
@@ -243,6 +243,7 @@
<configuration-dependencies>
<config-type>application-properties</config-type>
<config-type>hive-atlas-application.properties</config-type>
+ <config-type>hive-cli-atlas-application.properties</config-type>
</configuration-dependencies>
</service>
</services>
/var/lib/ambari-server/resources/common-services:
[username@hadoop1 ~]$ cat common_ambari.patch
diff -Naur -x '*.pyc' -x '*.zip' -x '*.pyo' /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py /tmp/amb/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
--- /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py 2016-11-23 02:27:10.000000000 -0500
+++ /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py 2017-02-09 14:42:00.000000000 -0500
@@ -228,12 +228,15 @@
group=params.user_group,
mode=0644)
- # Generate atlas-application.properties.xml file
if has_atlas_in_cluster():
atlas_hook_filepath = os.path.join(params.hive_config_dir, params.atlas_hook_filename)
- setup_atlas_hook(SERVICE.HIVE, params.hive_atlas_application_properties, atlas_hook_filepath, params.hive_user, params.user_group)
+ setup_atlas_hook(SERVICE.HIVE, params.hive_cli_atlas_application_properties, atlas_hook_filepath, params.hive_user, params.user_group)
if name == 'hiveserver2':
+ if has_atlas_in_cluster():
+ atlas_hook_filepath = os.path.join(params.hive_server_conf_dir, params.atlas_hook_filename)
+ setup_atlas_hook(SERVICE.HIVE, params.hive_atlas_application_properties, atlas_hook_filepath, params.hive_user, params.user_group)
+
XmlConfig("hiveserver2-site.xml",
conf_dir=params.hive_server_conf_dir,
configurations=params.config['configurations']['hiveserver2-site'],
diff -Naur -x '*.pyc' -x '*.zip' -x '*.pyo' /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py /tmp/amb/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
--- /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 2016-11-23 02:27:10.000000000 -0500
+++ /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 2017-02-09 14:42:41.000000000 -0500
@@ -520,6 +520,7 @@
########################################################
#region Atlas Hooks
hive_atlas_application_properties = default('/configurations/hive-atlas-application.properties', {})
+hive_cli_atlas_application_properties = default('/configurations/hive-cli-atlas-application.properties', {})
if has_atlas_in_cluster():
atlas_hook_filename = default('/configurations/atlas-env/metadata_conf_file', 'atlas-application.properties')
diff -Naur -x '*.pyc' -x '*.zip' -x '*.pyo' /var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py /tmp/amb/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
--- /var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 2016-11-23 02:27:06.000000000 -0500
+++ /var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py 2017-02-09 15:09:15.000000000 -0500
@@ -314,7 +314,7 @@
if has_atlas_in_cluster():
atlas_hook_filepath = os.path.join(params.hive_conf_dir, params.atlas_hook_filename)
Logger.info("Has atlas in cluster, will save Atlas Hive hook into location %s" % str(atlas_hook_filepath))
- setup_atlas_hook(SERVICE.HIVE, params.hive_atlas_application_properties, atlas_hook_filepath, params.oozie_user, params.user_group)
+ setup_atlas_hook(SERVICE.HIVE, params.hive_cli_atlas_application_properties, atlas_hook_filepath, params.oozie_user, params.user_group)
Directory(params.oozie_server_dir,
owner = params.oozie_user,
diff -Naur -x '*.pyc' -x '*.zip' -x '*.pyo' /var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py /tmp/amb/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
--- /var/lib/ambari-server/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py 2016-11-23 02:27:06.000000000 -0500
+++ /var/lib/ambari-server/resources//common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py 2017-02-09 15:10:05.000000000 -0500
@@ -308,6 +308,7 @@
########################################################
#region Atlas Hooks needed by Hive on Oozie
hive_atlas_application_properties = default('/configurations/hive-atlas-application.properties', {})
+hive_cli_atlas_application_properties = default('/configurations/hive-cli-atlas-application.properties', {})
if has_atlas_in_cluster():
atlas_hook_filename = default('/configurations/atlas-env/metadata_conf_file', 'atlas-application.properties')
No API call avaialble to allow linking a config property to the serviceconfigmapping DB so it can be performed as follows:
Update Ambari DB !!! WORKING EXAMPLE!!!
insert into clusterconfigmapping (cluster_id, type_name, version_tag,create_timestamp, selected, user_name) VALUES ('2', 'hive-cli-atlas-application.properties', 'generatedTag_1', '1484066089143', '1', 'username');
select max(config_id) from clusterconfig where type_name = 'hive-cli-atlas-application.properties';
select max(version) from serviceconfig where service_name = 'HIVE';
insert into serviceconfigmapping (service_config_id, config_id) VALUES ('1203', '1552');
ambari-server restart
Restart Hive Services and Oozie ]
> Hive hook should choose appropriate JAAS config if host uses kerberos ticket-cache
> ----------------------------------------------------------------------------------
>
> Key: ATLAS-1546
> URL: https://issues.apache.org/jira/browse/ATLAS-1546
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-intg
> Affects Versions: 0.7-incubating, 0.8-incubating
> Reporter: Madhan Neethiraj
> Assignee: Nixon Rodrigues
>
> In a kerberized environment, Atlas hook uses JAAS configuration section named "KakfaClient" to authenticate with Kafka broker. In a typical Hive deployment this configuration section is set to use the keytab and principal of HiveServer2 process. The hook running in HiveCLI might fail to authenticate with Kafka if the user can't read the configured keytab.
> Given that HiveCLI users would have performed kinit, the hook in HiveCLI should use the ticket-cache generated by kinit. When ticket cache is not available (for example in HiveServer2), the hook should use the configuration provided in KafkaClient JAAS section.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)