You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hawq.apache.org by go...@apache.org on 2016/08/25 20:44:40 UTC
[1/2] incubator-hawq git commit: HAWQ-1013. Move HAWQ Ambari plugin
to Apache HAWQ
Repository: incubator-hawq
Updated Branches:
refs/heads/master b13ca83bf -> 43b8e44c5
HAWQ-1013. Move HAWQ Ambari plugin to Apache HAWQ
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq/commit/5b013baa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq/tree/5b013baa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq/diff/5b013baa
Branch: refs/heads/master
Commit: 5b013baa88553a92f8c87085f1a4f4061e5154f6
Parents: b13ca83
Author: Matt <mm...@pivotal.io>
Authored: Wed Aug 24 16:37:48 2016 -0700
Committer: Goden Yao <go...@apache.org>
Committed: Thu Aug 25 13:42:05 2016 -0700
----------------------------------------------------------------------
contrib/hawq-ambari-plugin/.gitignore | 5 +
contrib/hawq-ambari-plugin/README.md | 98 ++++
contrib/hawq-ambari-plugin/build.properties | 8 +
contrib/hawq-ambari-plugin/pom.xml | 119 +++++
.../main/resources/services/HAWQ/metainfo.xml | 27 +
.../main/resources/services/PXF/metainfo.xml | 54 ++
.../src/main/resources/utils/add-hawq.py | 499 +++++++++++++++++++
pom.xml | 2 +
8 files changed, 812 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/5b013baa/contrib/hawq-ambari-plugin/.gitignore
----------------------------------------------------------------------
diff --git a/contrib/hawq-ambari-plugin/.gitignore b/contrib/hawq-ambari-plugin/.gitignore
new file mode 100644
index 0000000..859f265
--- /dev/null
+++ b/contrib/hawq-ambari-plugin/.gitignore
@@ -0,0 +1,5 @@
+# Maven target folder
+target/
+
+# Intellij
+*.idea/
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/5b013baa/contrib/hawq-ambari-plugin/README.md
----------------------------------------------------------------------
diff --git a/contrib/hawq-ambari-plugin/README.md b/contrib/hawq-ambari-plugin/README.md
new file mode 100644
index 0000000..5aac4e0
--- /dev/null
+++ b/contrib/hawq-ambari-plugin/README.md
@@ -0,0 +1,98 @@
+# HAWQ Ambari Plugin
+
+hawq-ambari-plugin helps users install HAWQ and PXF using Ambari.
+
+To ensure that Ambari recognizes HAWQ and PXF as services that can be installed for a specific stack, the following steps are required:
+ * Add HAWQ and PXF metainfo.xml files (containing metadata about the service) under the stack to be installed.
+ * Add repositories where HAWQ and PXF rpms reside, so that Ambari can use it during installation. This requires updating repoinfo.xml under the stack HAWQ and PXF is to be added.
+ * If a stack is already installed using Ambari, add repositories to the existing stack using the Ambari REST API.
+
+The above steps are taken care of by the hawq-ambari-plugin when the ```./add-hawq.py``` command is used.
+
+## Source Code
+Source code directory structure of the hawq-ambari-plugin is as follows:
+```
+hawq-ambari-plugin
++-- README.md
++-- build.properties
++-- pom.xml
++-- src
+ +-- main
+ +-- resources
+ +-- services
+ � +-- HAWQ
+ � � +-- metainfo.xml
+ � +-- PXF
+ � +-- metainfo.xml
+ +-- utils
+ +-- add-hawq.py
+```
+
+### build.properties
+[build.properties](build.properties) contains properties required for building the plugin.
+
+### metainfo.xml
+[metainfo.xml](src/main/resources/services/HAWQ/metainfo.xml) contains the metadata about the service. The metainfo.xml specifies that the service definition is to be inherited from Ambari common-services. HAWQ and PXF common-services code can be found under [Apache Ambari repository](https://github.com/apache/ambari/tree/trunk/ambari-server/src/main/resources/common-services/).
+
+### add-hawq<i></i>.py
+[add-hawq.py](src/main/resources/utils/add-hawq.py) deploys HAWQ and PXF metainfo.xml files under the stack and adds the repositories to Ambari.
+
+
+## Building the plugin
+***Build Environment***: centos6 is the typical operating system used for building.
+
+Properties specified in the [build.properties](build.properties) file:
+
+| Property | Description | Value |
+| --- | --- | --- |
+| hawq.release.version | Release version of HAWQ | 2.0.1 |
+| hawq.common.services.version | HAWQ common services code in Ambari to be inherited | 2.0.0 |
+| pxf.release.version | Release version of PXF | 3.0.1 |
+| pxf.common.services.version | PXF common services code in Ambari to be inherited | 3.0.0 |
+| hawq.repo.prefix | Repository name for HAWQ core repository | hawq |
+| hawq.addons.repo.prefix | Repository name for HAWQ Add Ons repository | hawq-add-ons |
+| repository.version | Repository Version to be used in repository information | 2.0.1<i></i>.0 |
+| default.stack | Default stack under which, metainfo.xml and repositories have to be added | HDP-2.4 |
+
+To build the rpm for hawq-ambari-plugin, change the [build.properties](build.properties) file with the required parameters and run ```mvn install``` command under hawq-ambari-plugin directory:
+```
+$ pwd
+incubator-hawq/contrib/hawq-ambari-plugin
+$ mvn clean resources:copy-resources rpm:rpm -Dbuild_number=1
+```
+
+## Usage
+
+Installing the hawq-ambari-plugin rpm would lay down the following directory:
+```
+/var/lib/hawq
++-- add-hawq.py
++-- staging
+ +-- HAWQ
+ � +-- metainfo.xml
+ +-- PXF
+ +-- metainfo.xml
+```
+
+***Prerequisite***: Ensure that the script is run on the host where Ambari server is **running**.
+
+*Replace* ```<ambari-username>``` *and* ```<ambari-password>``` *with login Ambari credentials*.
+
+If hawq-2.0.1.0 and hawq-add-ons-2.0.1.0 repository have been set up on the Ambari server host, run the following command:
+
+```
+$ ./add-hawq.py --user <ambari-username> --password <ambari-password> --stack HDP-2.5
+```
+If ```--stack``` is not mentioned, *HDP-2.4* stack will be used as default parameter.
+
+If hawq-2.0.1.0 and hawq-add-ons-2.0.1.0 repository have been set up on a different host than the Ambari server host, run the following command:
+
+
+```
+$ ./add-hawq.py --user <ambari-username> --password <ambari-password> --stack HDP-2.5 --hawqrepo http://my.host.address/hawq-2.0.1.0 --addonsrepo http://my.host.address/hawq-add-ons-2.0.1.0
+```
+
+**Please restart ambari-server after running the script so that the changes take effect:**
+```
+$ ambari-server restart
+```
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/5b013baa/contrib/hawq-ambari-plugin/build.properties
----------------------------------------------------------------------
diff --git a/contrib/hawq-ambari-plugin/build.properties b/contrib/hawq-ambari-plugin/build.properties
new file mode 100644
index 0000000..b4c2b74
--- /dev/null
+++ b/contrib/hawq-ambari-plugin/build.properties
@@ -0,0 +1,8 @@
+hawq.release.version=2.0.1
+hawq.common.services.version=2.0.0
+pxf.release.version=3.0.1
+pxf.common.services.version=3.0.0
+hawq.repo.prefix=hawq
+hawq.addons.repo.prefix=hawq-add-ons
+repository.version=2.0.1.0
+default.stack=HDP-2.4
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/5b013baa/contrib/hawq-ambari-plugin/pom.xml
----------------------------------------------------------------------
diff --git a/contrib/hawq-ambari-plugin/pom.xml b/contrib/hawq-ambari-plugin/pom.xml
new file mode 100644
index 0000000..7b309a8
--- /dev/null
+++ b/contrib/hawq-ambari-plugin/pom.xml
@@ -0,0 +1,119 @@
+<?xml version="1.0"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.apache.hawq</groupId>
+ <artifactId>hawq-ambari-plugin</artifactId>
+ <version>2.0.1.0</version>
+ <name>hawq-ambari-plugin</name>
+ <url>http://maven.apache.org</url>
+
+ <properties>
+ <release>${project.version}</release>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <resources.temp>${basedir}/target/resources-temp</resources.temp>
+ <hawq.lib.dir>/var/lib/hawq</hawq.lib.dir>
+ <hawq.lib.staging.dir>${hawq.lib.dir}/staging</hawq.lib.staging.dir>
+ </properties>
+
+ <build>
+ <plugins>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-resources-plugin</artifactId>
+ <version>3.0.1</version>
+
+ <executions>
+ <execution>
+ <phase>none</phase>
+ <goals>
+ <goal>copy-resources</goal>
+ </goals>
+ </execution>
+ </executions>
+
+ <configuration>
+ <filters>
+ <filter>${basedir}/build.properties</filter>
+ </filters>
+ <outputDirectory>${resources.temp}</outputDirectory>
+ <resources>
+ <resource>
+ <directory>${basedir}/src/main/resources/</directory>
+ <filtering>true</filtering>
+ </resource>
+ </resources>
+ </configuration>
+ </plugin>
+
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>rpm-maven-plugin</artifactId>
+ <version>2.0.1</version>
+
+ <executions>
+ <execution>
+ <phase>none</phase>
+ <goals>
+ <goal>rpm</goal>
+ </goals>
+ </execution>
+ </executions>
+
+ <configuration>
+ <group>org.apache.hawq</group>
+ <description>
+ HAWQ plugin contains Ambari's stack definition for HAWQ.
+ When installed, Ambari will be able to support HAWQ as a service.
+ </description>
+ <needarch>x86_64</needarch>
+ <release>${build_number}%{?dist}</release>
+ <requires>
+ <require>ambari-server >= 2.2</require>
+ <require>python >= 2.6</require>
+ </requires>
+ <mappings>
+ <mapping>
+ <directory>${hawq.lib.staging.dir}</directory>
+ <sources>
+ <source>
+ <location>${resources.temp}/services</location>
+ </source>
+ </sources>
+ </mapping>
+ <mapping>
+ <directory>${hawq.lib.dir}</directory>
+ <filemode>755</filemode>
+ <sources>
+ <source>
+ <location>${resources.temp}/utils</location>
+ </source>
+ </sources>
+ </mapping>
+ </mappings>
+ </configuration>
+ </plugin>
+
+ </plugins>
+ </build>
+
+</project>
+
http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/5b013baa/contrib/hawq-ambari-plugin/src/main/resources/services/HAWQ/metainfo.xml
----------------------------------------------------------------------
diff --git a/contrib/hawq-ambari-plugin/src/main/resources/services/HAWQ/metainfo.xml b/contrib/hawq-ambari-plugin/src/main/resources/services/HAWQ/metainfo.xml
new file mode 100755
index 0000000..7ada845
--- /dev/null
+++ b/contrib/hawq-ambari-plugin/src/main/resources/services/HAWQ/metainfo.xml
@@ -0,0 +1,27 @@
+<?xml version="1.0"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<metainfo>
+ <schemaVersion>2.0</schemaVersion>
+ <services>
+ <service>
+ <name>HAWQ</name>
+ <extends>common-services/HAWQ/${hawq.common.services.version}</extends>
+ <version>${hawq.release.version}</version>
+ </service>
+ </services>
+</metainfo>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/5b013baa/contrib/hawq-ambari-plugin/src/main/resources/services/PXF/metainfo.xml
----------------------------------------------------------------------
diff --git a/contrib/hawq-ambari-plugin/src/main/resources/services/PXF/metainfo.xml b/contrib/hawq-ambari-plugin/src/main/resources/services/PXF/metainfo.xml
new file mode 100755
index 0000000..e462538
--- /dev/null
+++ b/contrib/hawq-ambari-plugin/src/main/resources/services/PXF/metainfo.xml
@@ -0,0 +1,54 @@
+<?xml version="1.0"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<metainfo>
+ <schemaVersion>2.0</schemaVersion>
+ <services>
+ <service>
+ <name>PXF</name>
+ <extends>common-services/PXF/${pxf.common.services.version}</extends>
+ <version>${pxf.release.version}</version>
+
+ <osSpecifics>
+ <osSpecific>
+ <osFamily>any</osFamily>
+ <packages>
+ <package>
+ <name>pxf-service</name>
+ </package>
+ <package>
+ <name>apache-tomcat</name>
+ </package>
+ <package>
+ <name>pxf-hive</name>
+ </package>
+ <package>
+ <name>pxf-hdfs</name>
+ </package>
+ <package>
+ <name>pxf-hbase</name>
+ </package>
+ <package>
+ <name>pxf-json</name>
+ </package>
+ </packages>
+ </osSpecific>
+ </osSpecifics>
+
+ </service>
+ </services>
+</metainfo>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/5b013baa/contrib/hawq-ambari-plugin/src/main/resources/utils/add-hawq.py
----------------------------------------------------------------------
diff --git a/contrib/hawq-ambari-plugin/src/main/resources/utils/add-hawq.py b/contrib/hawq-ambari-plugin/src/main/resources/utils/add-hawq.py
new file mode 100755
index 0000000..d8ba361
--- /dev/null
+++ b/contrib/hawq-ambari-plugin/src/main/resources/utils/add-hawq.py
@@ -0,0 +1,499 @@
+#!/usr/bin/env python
+
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import base64
+import getpass
+import json
+import os
+import shutil
+import socket
+import urllib2
+import xml.etree.ElementTree as ET
+from optparse import OptionParser
+
+PLUGIN_VERSION = '${release}'
+DEFAULT_STACK = '${default.stack}'
+SUPPORTED_OS_LIST = ['redhat6']
+HAWQ_LIB_STAGING_DIR = '${hawq.lib.staging.dir}'
+REPO_VERSION = '${repository.version}'
+HAWQ_REPO = '${hawq.repo.prefix}'
+HAWQ_ADD_ONS_REPO = '${hawq.addons.repo.prefix}'
+
+REPO_INFO = {
+ HAWQ_REPO: {
+ 'repoid': '-'.join([HAWQ_REPO, REPO_VERSION]),
+ 'input_param': '--hawqrepo',
+ 'optional': False
+ },
+ HAWQ_ADD_ONS_REPO: {
+ 'repoid': '-'.join([HAWQ_ADD_ONS_REPO, REPO_VERSION]),
+ 'input_param': '--addonsrepo',
+ 'optional': True
+ }
+}
+
+
+class APIClient:
+ """
+ Class which interacts with Ambari Server API
+ """
+
+ # Base API URL points to localhost. This script is to be executed on the Ambari Server
+ BASE_API_URL = 'http://localhost:8080/api/v1'
+
+ def __init__(self, user, password):
+ self.user = user
+ self.password = password
+ self.encoded_credentials = base64.encodestring(self.user + ':' + self.password).replace('\n', '')
+
+ def __request(self, method, url_path, headers=None, data=None):
+ """
+ Creates API requests and packages response into the following format: (response code, response body in json object)
+ """
+ headers = headers if headers is not None else {}
+ headers['Authorization'] = 'Basic {0}'.format(self.encoded_credentials)
+
+ req = urllib2.Request(self.BASE_API_URL + url_path, data, headers)
+ req.get_method = lambda: method
+ response = urllib2.urlopen(req)
+ response_str = response.read()
+
+ return response.getcode(), json.loads(response_str) if response_str else None
+
+ def verify_api_reachable(self):
+ """
+ Returns true if Ambari Server is reachable through API
+ """
+ try:
+ status_code, _ = self.__request('GET', '/stacks')
+ except Exception as e:
+ if type(e) == urllib2.HTTPError and e.code == 403:
+ raise Exception('Invalid username and/or password.')
+ elif type(e) == urllib2.URLError:
+ raise Exception('Ambari-server is not running. Please start ambari-server.')
+ else:
+ raise Exception('Unable to connect to Ambari Server.\n' + str(e))
+
+ def get_cluster_name(self):
+ """
+ Returns the name of the installed cluster
+ """
+ _, response_json = self.__request('GET', '/clusters')
+ return None if len(response_json['items']) == 0 else response_json['items'][0]['Clusters']['cluster_name']
+
+ def get_stack_info(self, cluster_name):
+ """
+ Returns stack information (stack name, stack version, repository version) of stack installed on cluster
+ """
+ _, response_json = self.__request('GET',
+ '/clusters/{0}/stack_versions?ClusterStackVersions/state.matches(CURRENT)'.format(
+ cluster_name))
+ if 'items' not in response_json or len(response_json['items']) == 0:
+ raise Exception('No Stack found to be installed on the cluster {0}'.format(cluster_name))
+ stack_versions = response_json['items'][0]['ClusterStackVersions']
+ return stack_versions['stack'], stack_versions['version'], stack_versions['repository_version']
+
+ def get_existing_repository_info(self, stack_name, stack_version, repository_version):
+ """
+ Returns existing repo information for a given stack
+ """
+ url_path = '/stacks/{0}/versions/{1}/compatible_repository_versions/{2}?fields=*,operating_systems/*,operating_systems/repositories/*'.format(
+ stack_name,
+ stack_version,
+ repository_version)
+ _, response_json = self.__request('GET', url_path)
+ return response_json
+
+ def update_existing_repo(self, stack_name, stack_version, repository_version, merged_repo_info):
+ """
+ Sends a PUT request to add new repo information to the Ambari database
+ """
+ url_path = '/stacks/{0}/versions/{1}/repository_versions/{2}'.format(stack_name, stack_version,
+ repository_version)
+
+ headers = {}
+ headers['X-Requested-By'] = 'ambari'
+ headers['Content-Type'] = 'application/x-www-form-urlencoded; charset=UTF-8'
+
+ try:
+ status_code, _ = self.__request('PUT', url_path, headers, merged_repo_info)
+ except:
+ # Ambari returns sporadic errors even if PUT succeeds
+ # Ignore any exception, because existing information from cluster will be verified after PUT request
+ return
+
+
+class RepoUtils:
+ """
+ Utility class for handling json structure to add new repo to existing repo
+ """
+
+ def __transform_repo(self, repository):
+ """
+ Extracts and returns the base_url, repo_id and repo_name for each repository
+ """
+ repo_info_json = repository['Repositories']
+ result = {}
+ result['Repositories'] = dict(
+ (k, v) for k, v in repo_info_json.iteritems() if k in ('base_url', 'repo_id', 'repo_name'))
+ return result
+
+ def __transform_os_repos(self, os_repos):
+ """
+ Constructs the json string for each operating system
+ """
+ result = {
+ 'OperatingSystems': {},
+ 'repositories': []
+ }
+ result['OperatingSystems']['os_type'] = os_repos['OperatingSystems']['os_type']
+ result['repositories'] = [self.__transform_repo(repository) for repository in os_repos['repositories']]
+ return result
+
+ def __transform(self, repository_info):
+ """
+ Constructs the json string with required repository information
+ """
+ result = {
+ 'operating_systems': []
+ }
+ result['operating_systems'] = [self.__transform_os_repos(os_repos) for os_repos in
+ repository_info['operating_systems']]
+ return result
+
+ def __create_repo_info_dict(self, repo):
+ """
+ Creates json string with new repo information
+ """
+ result = {}
+ result['Repositories'] = {
+ 'base_url': repo['baseurl'],
+ 'repo_id': repo['repoid'],
+ 'repo_name': repo['reponame']
+ }
+ return result
+
+ def verify_repos_updated(self, existing_repo_info, repos_to_add):
+ """
+ Checks if input repo exists for that os_type on the cluster
+ """
+ existing_repos = self.__transform(existing_repo_info)
+
+ all_repos_updated = True
+
+ for os_repos in existing_repos['operating_systems']:
+ if os_repos['OperatingSystems']['os_type'] in SUPPORTED_OS_LIST:
+
+ for repo_to_add in repos_to_add:
+ repo_exists = False
+
+ for existing_repo in os_repos['repositories']:
+ if existing_repo['Repositories']['repo_id'] == repo_to_add['repoid'] and \
+ existing_repo['Repositories']['repo_name'] == repo_to_add['reponame'] and \
+ url_exists(existing_repo['Repositories']['base_url'], repo_to_add['baseurl']):
+ repo_exists = True
+
+ all_repos_updated = all_repos_updated and repo_exists
+
+ return all_repos_updated
+
+ def add_to_existing_repos(self, existing_repo_info, repos_to_add):
+ """
+ Helper function for adding new repos to existing repos
+ """
+ existing_repos = self.__transform(existing_repo_info)
+
+ for os_repos in existing_repos['operating_systems']:
+ if os_repos['OperatingSystems']['os_type'] in SUPPORTED_OS_LIST:
+ for repo_to_add in repos_to_add:
+ repo_exists = False
+ for existing_repo in os_repos['repositories']:
+ if existing_repo['Repositories']['repo_id'] == repo_to_add['repoid']:
+ repo_exists = True
+ existing_repo['Repositories']['repo_name'] = repo_to_add['reponame']
+ existing_repo['Repositories']['base_url'] = repo_to_add['baseurl']
+
+ if not repo_exists:
+ os_repos['repositories'].append(self.__create_repo_info_dict(repo_to_add))
+
+ return json.dumps(existing_repos)
+
+
+class InputValidator:
+ """
+ Class containing methods for validating command line inputs
+ """
+
+ def __is_repourl_valid(self, repo_url):
+ """
+ Returns True if repo_url points to a valid repository
+ """
+ repo_url = os.path.join(repo_url, 'repodata/repomd.xml')
+ req = urllib2.Request(repo_url)
+
+ try:
+ response = urllib2.urlopen(req)
+ except urllib2.URLError:
+ return False
+
+ if response.getcode() != 200:
+ return False
+
+ return True
+
+ def verify_stack(self, stack):
+ """
+ Returns stack info of stack
+ """
+ if not stack:
+ # Use default stack
+ print 'INFO: Using default stack {0}, since --stack parameter was not specified.'.format(DEFAULT_STACK)
+ stack = DEFAULT_STACK
+
+ stack_pair = stack.split('-')
+
+ if len(stack_pair) != 2:
+ raise Exception('Specified stack {0} is not of expected format STACK_NAME-STACK_VERSION'.format(stack))
+
+ stack_name = stack_pair[0]
+ stack_version = stack_pair[1]
+
+ stack_dir = '/var/lib/ambari-server/resources/stacks/{0}/{1}'.format(stack_name, stack_version)
+
+ if not os.path.isdir(stack_dir):
+ raise Exception(
+ 'Specified stack {0} does not exist under /var/lib/ambari-server/resources/stacks'.format(stack))
+
+ return {
+ 'stack_name': stack_name,
+ 'stack_version': stack_version,
+ 'stack_dir': stack_dir
+ }
+
+ def verify_repo(self, repoid_prefix, repo_url):
+ """
+ Returns repo info of repo
+ """
+ repo_specified = True
+ if not repo_url:
+ # Use default repo_url
+ repo_url = 'http://{0}/{1}'.format(socket.getfqdn(), REPO_INFO[repoid_prefix]['repoid'])
+ repo_specified = False
+
+ if not self.__is_repourl_valid(repo_url):
+ if repo_specified:
+ raise Exception('Specified URL {0} is not a valid repository. \n'
+ 'Please specify a valid url for {1}'.format(repo_url,
+ REPO_INFO[repoid_prefix]['input_param']))
+ elif REPO_INFO[repoid_prefix]['optional']:
+ return None
+ else:
+ raise Exception(
+ 'Repository URL {0} is not valid. \nPlease ensure setup_repo.sh has been run for the {1} repository on this machine '
+ 'OR specify a valid url for {2}'.format(repo_url, REPO_INFO[repoid_prefix]['repoid'],
+ REPO_INFO[repoid_prefix]['input_param']))
+
+ return {
+ 'repoid': REPO_INFO[repoid_prefix]['repoid'],
+ 'reponame': REPO_INFO[repoid_prefix]['repoid'],
+ 'baseurl': repo_url
+ }
+
+
+def url_exists(repoA, repoB):
+ """
+ Returns True if given repourl repoA exists in repoB
+ """
+ if type(repoB) in (list, tuple):
+ return repoA.rstrip('/') in [existing_url.rstrip('/') for existing_url in repoB]
+ else:
+ return repoA.rstrip('/') == repoB.rstrip('/')
+
+
+def update_repoinfo(stack_dir, repos_to_add):
+ """
+ Updates the repoinfo.xml under the specified stack_dir
+ """
+ file_path = '{0}/repos/repoinfo.xml'.format(stack_dir)
+
+ for repo in repos_to_add:
+ repo['xmltext'] = '<repo>\n' \
+ ' <repoid>{0}</repoid>\n' \
+ ' <reponame>{1}</reponame>\n' \
+ ' <baseurl>{2}</baseurl>\n' \
+ '</repo>\n'.format(repo['repoid'], repo['reponame'], repo['baseurl'])
+
+ tree = ET.parse(file_path)
+ root = tree.getroot()
+ file_needs_update = False
+
+ for os_tag in root.findall('.//os'):
+
+ if os_tag.attrib['family'] in SUPPORTED_OS_LIST:
+ for repo_to_add in repos_to_add:
+
+ repo_needs_update = False
+ for existing_repo in os_tag.findall('.//repo'):
+
+ existing_repoid = [repoid.text for repoid in existing_repo.findall('.//repoid')][0]
+ existing_reponame = [repoid.text for repoid in existing_repo.findall('.//reponame')][0]
+ existing_baseurl = [baseurl.text for baseurl in existing_repo.findall('.//baseurl')][0]
+
+ if existing_repoid == repo_to_add['repoid']:
+ repo_needs_update = True
+ print 'INFO: Repository {0} already exists with reponame {1}, baseurl {2} in {3}'.format(
+ repo_to_add['repoid'], existing_reponame, existing_baseurl, file_path)
+
+ if existing_reponame != repo_to_add['reponame'] or existing_baseurl != repo_to_add['baseurl']:
+ os_tag.remove(existing_repo)
+ os_tag.append(ET.fromstring(repo_to_add['xmltext']))
+ print 'INFO: Repository {0} updated with reponame {1}, baseurl {2} in {3}'.format(
+ repo_to_add['repoid'], repo_to_add['reponame'], repo_to_add['baseurl'], file_path)
+ file_needs_update = True
+
+ if not repo_needs_update:
+ os_tag.append(ET.fromstring(repo_to_add['xmltext']))
+ print 'INFO: Repository {0} with baseurl {1} added to {2}'.format(repo_to_add['repoid'],
+ repo_to_add['baseurl'], file_path)
+ file_needs_update = True
+
+ if file_needs_update:
+ tree.write(file_path)
+
+
+def add_repo_to_cluster(api_client, stack, repos_to_add):
+ """
+ Adds the new repository to the existing cluster if the specified stack has been installed on that cluster
+ """
+ stack_name = stack['stack_name']
+ stack_version = stack['stack_version']
+
+ cluster_name = api_client.get_cluster_name()
+
+ # Proceed only if cluster is installed
+ if cluster_name is None:
+ return
+
+ repo_utils = RepoUtils()
+ installed_stack_name, installed_stack_version, installed_repository_version = api_client.get_stack_info(
+ cluster_name)
+
+ # Proceed only if installed stack matches input stack
+ if stack_name != installed_stack_name or stack_version != installed_stack_version:
+ return
+
+ existing_repo_info = api_client.get_existing_repository_info(stack_name, stack_version,
+ installed_repository_version)
+
+ new_repo_info = repo_utils.add_to_existing_repos(existing_repo_info, repos_to_add)
+ api_client.update_existing_repo(stack_name, stack_version, installed_repository_version, new_repo_info)
+
+ if not repo_utils.verify_repos_updated(
+ api_client.get_existing_repository_info(stack_name, stack_version, installed_repository_version),
+ repos_to_add):
+ raise Exception(
+ 'Failed to update repository information on existing cluster, {0} with stack {1}-{2}'.format(cluster_name,
+ stack_name,
+ stack_version))
+
+ print 'INFO: Repositories are available on existing cluster, {0} with stack {1}-{2}'.format(cluster_name,
+ stack_name,
+ stack_version)
+
+
+def write_service_info(stack_dir):
+ """
+ Writes the service info content to the specified stack_dir
+ """
+ stack_services = os.path.join(stack_dir, 'services')
+
+ for service in ('HAWQ', 'PXF'):
+ source_directory = os.path.join(HAWQ_LIB_STAGING_DIR, service)
+ destination_directory = os.path.join(stack_services, service)
+
+ if not os.path.exists(source_directory):
+ raise Exception('{0} directory was not found under {1}'.format(service, HAWQ_LIB_STAGING_DIR))
+
+ service_exists = False
+ if os.path.exists(destination_directory):
+ service_exists = True
+ shutil.rmtree(destination_directory)
+
+ if service_exists:
+ print 'INFO: Updating service {0}, which already exists under {1}'.format(service, stack_services)
+
+ shutil.copytree(source_directory, destination_directory)
+
+ print 'INFO: {0} directory was successfully {1}d under directory {2}'.format(service,
+ 'update' if service_exists else 'create',
+ stack_services)
+
+
+def build_parser():
+ """
+ Builds the parser required for parsing user inputs from command line
+ """
+ usage_string = 'Usage: ./add-hawq.py --user admin --password admin --stack HDP-2.4 --hawqrepo http://my.host.address/hawq-2.0.1.0/ --addonsrepo http://my.host.address/hawq-add-ons-2.0.1.0/'
+ parser = OptionParser(usage=usage_string, version='%prog {0}'.format(PLUGIN_VERSION))
+ parser.add_option('-u', '--user', dest='user', help='Ambari login username (Required)')
+ parser.add_option('-p', '--password', dest='password',
+ help='Ambari login password. Providing password through command line is not recommended.\n'
+ 'The script prompts for the password.')
+ parser.add_option('-s', '--stack', dest='stack', help='Stack Name and Version to be added.'
+ '(Eg: HDP-2.4 or HDP-2.5)')
+ parser.add_option('-r', '--hawqrepo', dest='hawqrepo', help='Repository URL which points to the HAWQ packages')
+ parser.add_option('-a', '--addonsrepo', dest='addonsrepo',
+ help='Repository URL which points to the HAWQ Add Ons packages')
+ return parser
+
+
+def main():
+ parser = build_parser()
+
+ options, _ = parser.parse_args()
+
+ user = options.user if options.user else raw_input('Enter Ambari login Username: ')
+ password = options.password if options.password else getpass.getpass('Enter Ambari login Password: ')
+
+ try:
+ # Verify if Ambari credentials are correct and API is reachable
+ api_client = APIClient(user, password)
+ api_client.verify_api_reachable()
+
+ validator = InputValidator()
+
+ stack_info = validator.verify_stack(options.stack)
+
+ repos_to_add = [validator.verify_repo(HAWQ_REPO, options.hawqrepo)]
+
+ add_ons_repo = validator.verify_repo(HAWQ_ADD_ONS_REPO, options.addonsrepo)
+ if add_ons_repo is not None:
+ repos_to_add.append(add_ons_repo)
+
+ update_repoinfo(stack_info['stack_dir'], repos_to_add)
+ add_repo_to_cluster(api_client, stack_info, repos_to_add)
+ write_service_info(stack_info['stack_dir'])
+ print '\nINFO: Please restart ambari-server for changes to take effect'
+ except Exception as e:
+ print '\nERROR: {0}'.format(str(e))
+
+
+if __name__ == '__main__':
+ main()
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/5b013baa/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 1b9a1a0..442b8c6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -132,6 +132,8 @@
<exclude>contrib/hawq-hadoop/hawq-mapreduce-tool/src/test/resources/dataset</exclude>
<exclude>contrib/hawq-hadoop/**/*.yaml</exclude>
+ <exclude>contrib/hawq-ambari-plugin/build.properties</exclude>
+ <exclude>contrib/hawq-ambari-plugin/README.md</exclude>
<exclude>src/backend/access/index/caql.files</exclude>
<exclude>src/backend/gpopt/library.ver</exclude>
[2/2] incubator-hawq git commit: HAWQ-1013. Move HAWQ Ambari plugin
to Apache HAWQ, merging from mithmatt:hawq-ambari-plugin
Posted by go...@apache.org.
HAWQ-1013. Move HAWQ Ambari plugin to Apache HAWQ, merging from mithmatt:hawq-ambari-plugin
Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq/commit/43b8e44c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq/tree/43b8e44c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq/diff/43b8e44c
Branch: refs/heads/master
Commit: 43b8e44c584e9321e4414d421e7784d30d422dc8
Parents: b13ca83 5b013ba
Author: Goden Yao <go...@apache.org>
Authored: Thu Aug 25 13:43:25 2016 -0700
Committer: Goden Yao <go...@apache.org>
Committed: Thu Aug 25 13:43:25 2016 -0700
----------------------------------------------------------------------
contrib/hawq-ambari-plugin/.gitignore | 5 +
contrib/hawq-ambari-plugin/README.md | 98 ++++
contrib/hawq-ambari-plugin/build.properties | 8 +
contrib/hawq-ambari-plugin/pom.xml | 119 +++++
.../main/resources/services/HAWQ/metainfo.xml | 27 +
.../main/resources/services/PXF/metainfo.xml | 54 ++
.../src/main/resources/utils/add-hawq.py | 499 +++++++++++++++++++
pom.xml | 2 +
8 files changed, 812 insertions(+)
----------------------------------------------------------------------