You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airavata.apache.org by sh...@apache.org on 2016/09/24 01:49:35 UTC
[1/9] airavata git commit: Initial commit
Repository: airavata
Updated Branches:
refs/heads/develop 43b3757d6 -> 7071ef173
Initial commit
Project: http://git-wip-us.apache.org/repos/asf/airavata/repo
Commit: http://git-wip-us.apache.org/repos/asf/airavata/commit/dc6012eb
Tree: http://git-wip-us.apache.org/repos/asf/airavata/tree/dc6012eb
Diff: http://git-wip-us.apache.org/repos/asf/airavata/diff/dc6012eb
Branch: refs/heads/develop
Commit: dc6012eb10f1f7de1e56523dcfcee91d97f41a69
Parents:
Author: Shameera <sh...@gmail.com>
Authored: Mon Aug 29 16:25:14 2016 -0400
Committer: Shameera <sh...@gmail.com>
Committed: Mon Aug 29 16:25:14 2016 -0400
----------------------------------------------------------------------
LICENSE | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
README.md | 2 +
2 files changed, 203 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/airavata/blob/dc6012eb/LICENSE
----------------------------------------------------------------------
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000..8dada3e
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "{}"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright {yyyy} {name of copyright owner}
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
http://git-wip-us.apache.org/repos/asf/airavata/blob/dc6012eb/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..70fcd39
--- /dev/null
+++ b/README.md
@@ -0,0 +1,2 @@
+# mesos-deployment
+Setup Apache Mesos on Cloud Infrastructures using Anisble
[4/9] airavata git commit: Added mesos master and slave yum install
ansible
Posted by sh...@apache.org.
Added mesos master and slave yum install ansible
Project: http://git-wip-us.apache.org/repos/asf/airavata/repo
Commit: http://git-wip-us.apache.org/repos/asf/airavata/commit/03db1dfe
Tree: http://git-wip-us.apache.org/repos/asf/airavata/tree/03db1dfe
Diff: http://git-wip-us.apache.org/repos/asf/airavata/diff/03db1dfe
Branch: refs/heads/develop
Commit: 03db1dfe3dfaf2aca18547b58cdaa82d3f3acf6c
Parents: 8868abc
Author: Shameera Rathnayaka <sh...@gmail.com>
Authored: Sat Sep 10 19:15:35 2016 -0400
Committer: Shameera Rathnayaka <sh...@gmail.com>
Committed: Sat Sep 10 19:15:35 2016 -0400
----------------------------------------------------------------------
.gitignore | 3 +-
group_vars/all.yml | 31 ++++++
hosts | 9 +-
roles/ec2/tasks/main.yml | 30 +++++
roles/ec2/vars/aws-credential.yml | 3 +
roles/ec2/vars/main.yml | 8 ++
roles/group_vars/all.yml | 13 ---
roles/mesos-master/tasks/backup_main.yml | 117 ++++++++++++++++++++
roles/mesos-master/tasks/main.yml | 101 +++++++++++++++++
roles/mesos-master/templates/myid.j2 | 1 +
roles/mesos-master/templates/quorum.j2 | 1 +
roles/mesos-master/templates/zk.j2 | 1 +
roles/mesos-master/templates/zoo.cfg.j2 | 32 ++++++
roles/mesos-slave/tasks/main.yml | 66 +++++++++++
roles/mesos-slave/templates/zk.j2 | 1 +
roles/mesos/tasks/RedHat.yml | 18 ---
roles/mesos/tasks/main.yml | 84 --------------
roles/test/tasks/main.yml | 2 +
roles/zookeeper/handlers/main.yml | 12 --
roles/zookeeper/tasks/main.yml | 34 ------
roles/zookeeper/templates/zoo.cfg.j2 | 28 -----
roles/zookeeper/templates/zookeeper.service.j2 | 15 ---
roles/zookeeper/vars/main.yml | 17 ---
site.yml | 10 +-
24 files changed, 409 insertions(+), 228 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
index c996e50..e1f821f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1 +1,2 @@
-*.key
+jetcloud.*
+shameera-*
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/group_vars/all.yml
----------------------------------------------------------------------
diff --git a/group_vars/all.yml b/group_vars/all.yml
new file mode 100644
index 0000000..b0a6cf2
--- /dev/null
+++ b/group_vars/all.yml
@@ -0,0 +1,31 @@
+---
+ansible_connection: ssh
+#ansible_ssh_user: root
+ansible_ssh_user: centos
+#ansible_ssh_private_key_file: /Users/syodage/Projects/mesos-deployment/jetcloud.key
+ansible_ssh_private_key_file: /Users/syodage/Projects/airavata-ansible/shameera-aws.pem.txt
+
+user: centos
+group: centos
+user_home: "/home/{{user}}"
+
+# Mesos related properties
+#mesos_version: "1.0.1"
+# mesos_version: "0.28.2"
+# mesos_name: "mesos"
+# mesos_url: "http://www.apache.org/dist/mesos/{{mesos_version}}/{{mesos_name}}-{{mesos_version}}.tar.gz"
+# mesos_deploy_dir: "{{user_home}}/{{mesos_name}}-{{mesos_version}}"
+# mesos_build_dir: "{{mesos_deploy_dir}}/build"
+
+
+#Zookeeper related properties
+zk_client_port: 2181
+# zookeeper_servers:
+# - { index: 1, ip: "", ports: "2888:3888"}
+# - { index: 2, ip: "", ports: "2888:3888"}
+# - { index: 3, ip: "", ports: "2888:3888"}
+zk_1: "{{ groups['mesos-master'][0] }}"
+zk_2: "{{ groups['mesos-master'][1] }}"
+zk_3: "{{ groups['mesos-master'][2] }}"
+
+zk_quorum: 2
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/hosts
----------------------------------------------------------------------
diff --git a/hosts b/hosts
index 949fb79..7ceb1f7 100644
--- a/hosts
+++ b/hosts
@@ -1,8 +1,9 @@
---
-[localhost]
-127.0.0.1
+# inventory fie: production
+#[localhost]
+#127.0.0.1
[mesos-master]
+52.91.222.138
-
-[mesos-agent]
+#[mesos-agent]
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/ec2/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/ec2/tasks/main.yml b/roles/ec2/tasks/main.yml
new file mode 100644
index 0000000..7a6eb3d
--- /dev/null
+++ b/roles/ec2/tasks/main.yml
@@ -0,0 +1,30 @@
+---
+- name: include credentials
+ include_vars: vars/aws-credential.yml
+
+- name: create a aws instace/s
+ ec2:
+ key_name: "{{ key_name }}"
+ region: us-east-1
+ instance_type: t2.small
+ image: "{{centos_ami_7_2}}"
+ group: "{{ security_group }}"
+ instance_tags:
+ Name: "{{ item }}"
+ assign_public_ip: yes
+ state: present
+ count: 1
+ vpc_subnet_id: subnet-54f8f70c
+ volumes:
+ - device_name: /dev/sda1
+ volume_type: io1
+ iops: 200
+ volume_size: 50
+ delete_on_termination: true
+ wait: yes
+ with_items:
+ - Mesos-master-1
+ - Mesos-master-2
+ - Mesos-master-3
+ - Mesos-slave-1
+ register: ec2
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/ec2/vars/aws-credential.yml
----------------------------------------------------------------------
diff --git a/roles/ec2/vars/aws-credential.yml b/roles/ec2/vars/aws-credential.yml
new file mode 100644
index 0000000..687cfbb
--- /dev/null
+++ b/roles/ec2/vars/aws-credential.yml
@@ -0,0 +1,3 @@
+---
+aws_access_key: AKIAIKBSNQUKEZN2VLYA
+aws_secret_key: 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
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/ec2/vars/main.yml
----------------------------------------------------------------------
diff --git a/roles/ec2/vars/main.yml b/roles/ec2/vars/main.yml
new file mode 100644
index 0000000..1f7d77a
--- /dev/null
+++ b/roles/ec2/vars/main.yml
@@ -0,0 +1,8 @@
+---
+
+key_name: shameera
+ec2_name: Mesos28
+centos_ami_7_2: ami-6d1c2007
+ubutntu_ami_16_04: ami-29f96d3e
+
+security_group: Mesos-sec
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/group_vars/all.yml
----------------------------------------------------------------------
diff --git a/roles/group_vars/all.yml b/roles/group_vars/all.yml
deleted file mode 100644
index c041787..0000000
--- a/roles/group_vars/all.yml
+++ /dev/null
@@ -1,13 +0,0 @@
----
-
-user: centos
-group: centos
-user_home: "home/{{user}}"
-
-# Mesos related properties
-mesos_dir: "{{user_home}}/mesos"
-mesos_build_dir: "{{mesos_dir}}/build"
-
-
-#Zookeeper related properties
-zookeeper_client_port: 2181
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/mesos-master/tasks/backup_main.yml
----------------------------------------------------------------------
diff --git a/roles/mesos-master/tasks/backup_main.yml b/roles/mesos-master/tasks/backup_main.yml
new file mode 100644
index 0000000..1d0abd1
--- /dev/null
+++ b/roles/mesos-master/tasks/backup_main.yml
@@ -0,0 +1,117 @@
+---
+
+- name: wait untill instance is up and running
+ local_action: wait_for port=22
+ host="{{ ansible_ssh_host | default(inventory_hostname) }}"
+ search_regex=OpenSSH delay=10
+- name: create group
+ group: name="{{group}}"
+ become: yes
+
+- name: create user
+ user: name="{{user}}"
+ become: yes
+
+# RedHat related things
+- name: install required packages
+ yum: name="{{item}}" state=latest
+ with_items:
+ - wget
+ - tar
+ - git
+ - epel-release
+ - firewalld
+ become: yes
+
+- name: updage systemd
+ yum: name=systemd state=latest
+ become: yes
+
+- name: get apache maven
+ get_url:
+ url: http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo
+ dest: /etc/yum.repos.d/epel-apache-maven.repo
+ become: yes
+
+- name: copy svn repo file
+ copy:
+ src: wandisco-svn.repo
+ dest: /etc/yum.repos.d/wandisco-svn.repo
+ become: yes
+
+- name: install development tools
+ yum: name="@Development tools" state=present
+ become: yes
+
+#Install other Mesos dependencies.
+- name: install mesos dependencies
+ yum: name="{{item}}" state=latest
+ with_items:
+ - apache-maven
+ - python-devel
+ - java-1.8.0-openjdk-devel
+ - zlib-devel
+ - libcurl-devel
+ - openssl-devel
+ - cyrus-sasl-devel
+ - cyrus-sasl-md5
+ - apr-devel
+ - apr-util-devel
+ - subversion-devel
+ become: yes
+
+- name: start firewall service
+ service: name=firewalld state=started enabled=yes
+ become: yes
+
+- name: download and unarchive mesos latest
+ unarchive:
+ src="{{mesos_url}}"
+ dest="{{user_home}}"
+ remote_src=yes
+ copy=no
+ creates="{{mesos_deploy_dir}}/configure"
+ owner="{{user}}"
+ group="{{group}}"
+
+ # Bootstrap (Only required if building from git repository).
+ # $ ./bootstrap
+
+- name: make build dir
+ file: path="{{mesos_build_dir}}" state=directory owner={{user}} group={{group}}
+
+# - name: create simlink to configure script
+# file: src="{{mesos_deploy_dir}}/configure"
+# dest="{{mesos_build_dir}}/configure"
+# state=link
+# owner={{user}}
+# group={{group}}
+# tags: test
+
+- name: configure and build mesos
+ command: ../configure chdir="{{mesos_build_dir}}"
+ register: build
+
+- name: run make
+ command: make chdir="{{mesos_build_dir}}"
+ when: build|success
+ register: make
+
+# - name: run make tests
+# command: make check chdir="{{mesos_build_dir}}"
+# when: make|success
+
+- name: open firewall port
+ firewalld: port=5050/tcp zone=public permanent=true state=enabled immediate=yes
+ become: yes
+
+- name: create working dir
+ file: path="{{mesos_work_dir}}" state=directory user={{user}} group={{group}}
+
+- name: start mesos master
+ command: ./bin/mesos-master.sh --ip={{inventory_hostname}} --work_dir={{mesos_work_dir}} &
+ chdir={{mesos_build_dir}}
+
+- name: start mesos agent
+ command: /bin/mesos-agent.sh --master={{inventory_hostname}}:5050 --work_dir={{mesos_work_dir}} &
+ chdir={{mesos_build_dir}}
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/mesos-master/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/mesos-master/tasks/main.yml b/roles/mesos-master/tasks/main.yml
new file mode 100644
index 0000000..e132915
--- /dev/null
+++ b/roles/mesos-master/tasks/main.yml
@@ -0,0 +1,101 @@
+---
+# reference: https://open.mesosphere.com/getting-started/install/
+- name: install firewalld
+ yum:
+ name: firewalld
+ state: latest
+ become: yes
+
+- name: start firewalld
+ service:
+ name: firewalld
+ state: started
+ enabled: yes
+ become: yes
+
+- name: open ports
+ firewalld:
+ port: "{{ item }}"
+ zone: public
+ permanent: true
+ state: enabled
+ immediate: yes
+ with_items:
+ - "{{ zk_client_port }}/tcp"
+ - "2888/tcp"
+ - "3888/tcp"
+ - "5050/tcp"
+ # - "marathon/tcp"
+ become: yes
+
+# Setup repository
+- name: add mesosphere rpm
+ yum:
+ name: http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
+ state: present
+ become: yes
+
+- name: install mesos , marathon and mesosphere-zookeeper
+ yum:
+ name: "{{ item }}"
+ state: latest
+ with_items:
+ - mesos
+ - marathon
+ - mesosphere-zookeeper
+ become: yes
+
+# configurations
+#setting ID
+- name: copy myid file
+ template:
+ src: myid.j2
+ dest: /var/lib/zookeeper/myid
+ force: yes
+ become: yes
+
+- name: append zookeeper ring ips
+ template:
+ src: zoo.cfg.j2
+ dest: /etc/zookeeper/conf/zoo.cfg
+ force: yes
+ become: yes
+
+- name: start Zookeeper
+ service:
+ name: zookeeper
+ state: started
+ become: yes
+
+- name: set zk file for mesos master and marathon
+ template:
+ src: zk.j2
+ dest: /etc/mesos/zk
+ force: yes
+ become: yes
+
+- name: set zookeeper quorum
+ template:
+ src: quorum.j2
+ dest: /etc/mesos-master/quorum
+ force: yes
+ become: yes
+
+- name: stop and disable mesos slave
+ service:
+ name: mesos-slave
+ enabled: no
+ state: stopped
+ become: yes
+
+- name: restart mesos master
+ service:
+ name: mesos-master
+ state: restarted
+ become: yes
+
+- name: restart marathon
+ service:
+ name: marathon
+ state: restarted
+ become: yes
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/mesos-master/templates/myid.j2
----------------------------------------------------------------------
diff --git a/roles/mesos-master/templates/myid.j2 b/roles/mesos-master/templates/myid.j2
new file mode 100644
index 0000000..78a0bd7
--- /dev/null
+++ b/roles/mesos-master/templates/myid.j2
@@ -0,0 +1 @@
+{{ my_id }}
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/mesos-master/templates/quorum.j2
----------------------------------------------------------------------
diff --git a/roles/mesos-master/templates/quorum.j2 b/roles/mesos-master/templates/quorum.j2
new file mode 100644
index 0000000..c1d54ef
--- /dev/null
+++ b/roles/mesos-master/templates/quorum.j2
@@ -0,0 +1 @@
+{{zk_quorum}}
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/mesos-master/templates/zk.j2
----------------------------------------------------------------------
diff --git a/roles/mesos-master/templates/zk.j2 b/roles/mesos-master/templates/zk.j2
new file mode 100644
index 0000000..9c7747b
--- /dev/null
+++ b/roles/mesos-master/templates/zk.j2
@@ -0,0 +1 @@
+zk://{{zk_1}}:{{zk_client_port}},{{zk_2}}:{{zk_client_port}},{{zk_3}}:{{zk_client_port}}/mesos
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/mesos-master/templates/zoo.cfg.j2
----------------------------------------------------------------------
diff --git a/roles/mesos-master/templates/zoo.cfg.j2 b/roles/mesos-master/templates/zoo.cfg.j2
new file mode 100644
index 0000000..922d0b2
--- /dev/null
+++ b/roles/mesos-master/templates/zoo.cfg.j2
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+maxClientCnxns=50
+# The number of milliseconds of each tick
+tickTime=2000
+# The number of ticks that the initial
+# synchronization phase can take
+initLimit=10
+# The number of ticks that can pass between
+# sending a request and getting an acknowledgement
+syncLimit=5
+# the directory where the snapshot is stored.
+dataDir=/var/lib/zookeeper
+# the port at which the clients will connect
+clientPort={{ zk_client_port }}
+
+server.1={{zk_1}}:2888:3888
+server.2={{zk_2}}:2888:3888
+server.3={{zk_3}}:2888:3888
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/mesos-slave/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/mesos-slave/tasks/main.yml b/roles/mesos-slave/tasks/main.yml
new file mode 100644
index 0000000..f6890fe
--- /dev/null
+++ b/roles/mesos-slave/tasks/main.yml
@@ -0,0 +1,66 @@
+---
+- name: install firewalld
+ yum:
+ name: firewalld
+ state: latest
+ update_cache: yes
+ become: yes
+
+- name: start firewalld
+ service:
+ name: firewalld
+ state: started
+ enabled: yes
+ become: yes
+
+- name: open ports
+ firewalld:
+ port: "{{ item }}"
+ zone: public
+ permanent: true
+ state: enabled
+ immediate: yes
+ with_items:
+ - "5051/tcp"
+ become: yes
+
+# setup mesos slave
+- name: add mesosphere rpm
+ yum:
+ name: http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
+ state: present
+ become: yes
+
+- name: install mesos
+ yum:
+ name: mesos
+ state: latest
+ become: yes
+
+# zookeeper will be pulled in and isntall as a depencency automatically
+# - name: disable zookeeper
+# service:
+# name: mesosphere-zookeeper
+# state: stopped
+# enabled: no
+# become: yes
+
+- name: set zk file for mesos master and marathon
+ template:
+ src: zk.j2
+ dest: /etc/mesos/zk
+ force: yes
+ become: yes
+
+- name: disable mesos-master
+ service:
+ name: mesos-master
+ state: stopped
+ enabled: no
+ become: yes
+
+- name: restart mesos slave
+ service:
+ name: mesos-slave
+ state: restarted
+ become: yes
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/mesos-slave/templates/zk.j2
----------------------------------------------------------------------
diff --git a/roles/mesos-slave/templates/zk.j2 b/roles/mesos-slave/templates/zk.j2
new file mode 100644
index 0000000..9c7747b
--- /dev/null
+++ b/roles/mesos-slave/templates/zk.j2
@@ -0,0 +1 @@
+zk://{{zk_1}}:{{zk_client_port}},{{zk_2}}:{{zk_client_port}},{{zk_3}}:{{zk_client_port}}/mesos
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/mesos/tasks/RedHat.yml
----------------------------------------------------------------------
diff --git a/roles/mesos/tasks/RedHat.yml b/roles/mesos/tasks/RedHat.yml
deleted file mode 100644
index 4c7ace9..0000000
--- a/roles/mesos/tasks/RedHat.yml
+++ /dev/null
@@ -1,18 +0,0 @@
----
-- name: create group
- group: name="{{group}}"
- become: yes
-
-- name: create user
- user: name="{{user}}"
- become: yes
-
-- name: install required packages
- yum: name="{{item}}" state=latest
- with_item:
- - wget
- - tar
- - git
- - epel-release
- - subversion
- - firewalld
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/mesos/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/mesos/tasks/main.yml b/roles/mesos/tasks/main.yml
deleted file mode 100644
index f0fa90c..0000000
--- a/roles/mesos/tasks/main.yml
+++ /dev/null
@@ -1,84 +0,0 @@
----
-- name: create group
- group: name="{{group}}"
- become: yes
-
-- name: create user
- user: name="{{user}}"
- become: yes
-
-# RedHat related things
-- name: install required packages
- yum: name="{{item}}" state=latest
- with_items:
- - wget
- - tar
- - git
- - epel-release
- - firewalld
- become: yes
-
-- name: updage systemd
- yum: name=systemd state=latest enabled=yes
- become: yes
-
-- name: install development tools
- yum: name="@Development tools" state=latest
- become: yes
-
- # Install other Mesos dependencies.
-- name: install mesos dependencies
- yum: name="{{item}}" state=latest
- with_items:
- - apache-maven
- - python-devel
- - java-1.8.0-openjdk-devel
- - zlib-devel
- - libcurl-devel
- - openssl-devel
- - cyrus-sasl-devel
- - cyrus-sasl-md5
- - apr-devel
- - apr-util-devel
- - subversion-devel
-
-- name: start firewall service
- service: name=firewalld state=started enabled=yes
- become: yes
-
-- name: download and unarchive mesos latest
- unarchive:
- src=http://www.apache.org/dist/mesos/1.0.1/mesos-1.0.1.tar.gz
- dest="{{mesos_dir}}"
- owner="{{user}}"
- group="{{group}}"
-
- # Bootstrap (Only required if building from git repository).
- # $ ./bootstrap
-
-- name: make build dir
- file: path="{{mesos_dir}}/build" state=directory user={{user}} group={{group}}
-
-- name: configure and build mesos
- command: ../configure chdir="{{mesos_build_dir}}"
- register: build
-
-- name: run make
- command: make chdir="{{mesos_build_dir}}"
- when: build|success
- register: make
-
-- name: run make tests
- command: make check chdir="{{mesos_build_dir}}"
- when: make|success
-
-- name: create working dir
- file: path="{{mesos_work_dir}}" state=directory user={{user}} group={{group}}
-
-- name: start mesos master
- command: ./bin/mesos-master.sh --ip={{inventory_hostname}} --work_dir={{mesos_work_dir}} &
- chdir={{mesos_build_dir}}
-
-- name: start mesos agent
- command: /bin/mesos-agent.sh --master={{inventory_hostname}}:5050 --work_dir={{mesos_work_dir}} &
- chdir={{mesos_build_dir}}
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/test/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/test/tasks/main.yml b/roles/test/tasks/main.yml
index b3cb5ca..f0548b1 100644
--- a/roles/test/tasks/main.yml
+++ b/roles/test/tasks/main.yml
@@ -1 +1,3 @@
- debug: msg="ding {{ inventory_hostname }}"
+
+- debug: msg="{{ inventory }}"
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/zookeeper/handlers/main.yml
----------------------------------------------------------------------
diff --git a/roles/zookeeper/handlers/main.yml b/roles/zookeeper/handlers/main.yml
deleted file mode 100644
index daefebe..0000000
--- a/roles/zookeeper/handlers/main.yml
+++ /dev/null
@@ -1,12 +0,0 @@
----
-- name: start zookeeper
- service: name=zookeeper state=started enabled=yes
- become: yes
-
-- name: stop zookeeper
- service: name=zookeeper state=stopped
- become: yes
-
-- name: restart zookeeper
- service: name=zookeeper state=restarted enabled=yes
- become: yes
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/zookeeper/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/zookeeper/tasks/main.yml b/roles/zookeeper/tasks/main.yml
deleted file mode 100644
index 78bcec2..0000000
--- a/roles/zookeeper/tasks/main.yml
+++ /dev/null
@@ -1,34 +0,0 @@
----
-################################################################################
-# Setup and run Zookeeper
-- name: Download and unarchive zookeeper
- unarchive: src="{{ zookeeper_url }}"
- dest="{{ user_home }}"
- copy=no
- owner="{{ user }}"
- group="{{ group }}"
-
-- name: open zookeeper port
- firewalld: port=2181/tcp zone=public permanent=true state=enabled immediate=yes
- become: yes
-
-- name: Copy zoo.cfg file
- template: src=zoo.cfg.j2 dest="{{ zookeeper_dir }}/conf/zoo.cfg" owner="{{ user }}" group="{{ group }}" mode="u=rw,g=r,o=r"
- notify:
- - restart zookeeper
-
-- name: Check if systemd exists
- stat: path=/usr/lib/systemd/system/
- register: systemd_check
-
-- name: Systemd script.
- template: src=zookeeper.service.j2 dest=/usr/lib/systemd/system/zookeeper.service
- when: systemd_check.stat.exists == true
- notify:
- - restart zookeeper
- become: yes
-
-- name: reload systemd daemons
- command: systemctl daemon-reload
- become: yes
-...
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/zookeeper/templates/zoo.cfg.j2
----------------------------------------------------------------------
diff --git a/roles/zookeeper/templates/zoo.cfg.j2 b/roles/zookeeper/templates/zoo.cfg.j2
deleted file mode 100644
index 8426b98..0000000
--- a/roles/zookeeper/templates/zoo.cfg.j2
+++ /dev/null
@@ -1,28 +0,0 @@
-# The number of milliseconds of each tick
-tickTime={{tick_time}}
-# The number of ticks that the initial
-# synchronization phase can take
-initLimit={{init_limit}}
-# The number of ticks that can pass between
-# sending a request and getting an acknowledgement
-syncLimit={{sync_limit}}
-# the directory where the snapshot is stored.
-# do not use /tmp for storage, /tmp here is just
-# example sakes.
-dataDir={{zookeeper_data_dir}}
-# the port at which the clients will connect
-clientPort={{ client_port }}
-# the maximum number of client connections.
-# increase this if you need to handle more clients
-#maxClientCnxns=60
-#
-# Be sure to read the maintenance section of the
-# administrator guide before turning on autopurge.
-#
-# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
-#
-# The number of snapshots to retain in dataDir
-#autopurge.snapRetainCount=3
-# Purge task interval in hours
-# Set to "0" to disable auto purge feature
-#autopurge.purgeInterval=1
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/zookeeper/templates/zookeeper.service.j2
----------------------------------------------------------------------
diff --git a/roles/zookeeper/templates/zookeeper.service.j2 b/roles/zookeeper/templates/zookeeper.service.j2
deleted file mode 100644
index 19c3718..0000000
--- a/roles/zookeeper/templates/zookeeper.service.j2
+++ /dev/null
@@ -1,15 +0,0 @@
-# {{ansible_managed}}
-
-[Unit]
-Description=ZooKeeper
-
-
-[Service]
-Type=forking
-ExecStart={{zookeeper_dir}}/bin/zkServer.sh start
-ExecStop={{ zookeeper_dir }}/bin/zkServer.sh stop
-Restart=always
-TimeoutSec=300
-
-[Install]
-WantedBy=multi-user.target
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/roles/zookeeper/vars/main.yml
----------------------------------------------------------------------
diff --git a/roles/zookeeper/vars/main.yml b/roles/zookeeper/vars/main.yml
deleted file mode 100644
index a3e10db..0000000
--- a/roles/zookeeper/vars/main.yml
+++ /dev/null
@@ -1,17 +0,0 @@
----
-#Variables associated with this role
-# zookeeper related variable
-zookeeper_version: 3.4.8
-zookeeper_url: http://www.us.apache.org/dist/zookeeper/zookeeper-{{zookeeper_version}}/zookeeper-{{zookeeper_version}}.tar.gz
-
-apt_cache_timeout: 3600
-client_port: "{{ zookeeper_client_port }}"
-init_limit: 5
-sync_limit: 2
-tick_time: 2000
-data_dir: /var/lib/zookeeper
-log_dir: /var/log/zookeeper
-zookeeper_dir: "{{ user_home }}/zookeeper-{{zookeeper_version}}"
-zookeeper_data_dir: "{{ zookeeper_dir }}/data"
-
-...
http://git-wip-us.apache.org/repos/asf/airavata/blob/03db1dfe/site.yml
----------------------------------------------------------------------
diff --git a/site.yml b/site.yml
index 2fc7293..4020f15 100644
--- a/site.yml
+++ b/site.yml
@@ -4,8 +4,14 @@
roles:
- test
-
- hosts: mesos-master
- tags: mesos-master
+ tags: mesos
roles:
- mesos
+
+- hosts: mesos-master
+ tags: mesosyum
+ roles:
+ - mesos-yum
+
+...
[8/9] airavata git commit: Updated readme to avoid known_host adding
prompt
Posted by sh...@apache.org.
Updated readme to avoid known_host adding prompt
Project: http://git-wip-us.apache.org/repos/asf/airavata/repo
Commit: http://git-wip-us.apache.org/repos/asf/airavata/commit/21545bea
Tree: http://git-wip-us.apache.org/repos/asf/airavata/tree/21545bea
Diff: http://git-wip-us.apache.org/repos/asf/airavata/diff/21545bea
Branch: refs/heads/develop
Commit: 21545bea2b338dcc8491cb15aee85787fdb38e92
Parents: d5a9c8a
Author: Shameera Rathnayaka <sh...@gmail.com>
Authored: Mon Sep 12 12:29:29 2016 -0400
Committer: Shameera Rathnayaka <sh...@gmail.com>
Committed: Mon Sep 12 12:29:29 2016 -0400
----------------------------------------------------------------------
README.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/airavata/blob/21545bea/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 794099a..9ce4e47 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,7 @@
# mesos-deployment
This project setup Apache Mesos master and slave clusters on Cloud Infrastructures using Anisble
+If you are working with new set of hosts which you haven't ssh login to those host from the machine you are suppose to run your Ansible play. It will prompt you to get permission to add to known_hosts list, if you wan't remove this prompt and you know the consequences of disabling this feature and you want tor remove this prompt behavior run this on your terminal `export ANSIBLE_HOST_KEY_CHECKING=False`
## Configurations
[9/9] airavata git commit: Imported mesos ansible scripts to airavata
repo
Posted by sh...@apache.org.
Imported mesos ansible scripts to airavata repo
Project: http://git-wip-us.apache.org/repos/asf/airavata/repo
Commit: http://git-wip-us.apache.org/repos/asf/airavata/commit/7071ef17
Tree: http://git-wip-us.apache.org/repos/asf/airavata/tree/7071ef17
Diff: http://git-wip-us.apache.org/repos/asf/airavata/diff/7071ef17
Branch: refs/heads/develop
Commit: 7071ef173edaa0d4cbddcf8e2246a4d5f0c620cf
Parents: 43b3757 21545be
Author: Shameera Rathnayaka <sh...@gmail.com>
Authored: Fri Sep 23 21:45:32 2016 -0400
Committer: Shameera Rathnayaka <sh...@gmail.com>
Committed: Fri Sep 23 21:49:08 2016 -0400
----------------------------------------------------------------------
modules/cloud/ansible-playbooks/README.md | 82 +++++++++++++
.../cloud/ansible-playbooks/group_vars/all.yml | 19 +++
modules/cloud/ansible-playbooks/hosts | 13 +++
.../ansible-playbooks/roles/ec2/tasks/main.yml | 30 +++++
.../roles/ec2/vars/aws-credential.yml | 3 +
.../ansible-playbooks/roles/ec2/vars/main.yml | 8 ++
.../ansible-playbooks/roles/java/tasks/main.yml | 21 ++++
.../ansible-playbooks/roles/java/vars/main.yml | 15 +++
.../roles/mesos-master/tasks/backup_main.yml | 117 +++++++++++++++++++
.../roles/mesos-master/tasks/main.yml | 86 ++++++++++++++
.../roles/mesos-master/templates/quorum.j2 | 1 +
.../roles/mesos-master/templates/zk.j2 | 1 +
.../roles/mesos-slave/tasks/main.yml | 66 +++++++++++
.../roles/mesos-slave/templates/zk.j2 | 1 +
.../ansible-playbooks/roles/test/tasks/main.yml | 3 +
.../roles/zookeeper/handlers/main.yml | 15 +++
.../roles/zookeeper/tasks/main.yml | 98 ++++++++++++++++
.../roles/zookeeper/templates/myid.j2 | 1 +
.../roles/zookeeper/templates/zoo.cfg.j2 | 32 +++++
.../roles/zookeeper/vars/main.yml | 15 +++
modules/cloud/ansible-playbooks/site.yml | 19 +++
21 files changed, 646 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/README.md
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/README.md
index 0000000,0000000..9ce4e47
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/README.md
@@@ -1,0 -1,0 +1,82 @@@
++# mesos-deployment
++This project setup Apache Mesos master and slave clusters on Cloud Infrastructures using Anisble
++
++If you are working with new set of hosts which you haven't ssh login to those host from the machine you are suppose to run your Ansible play. It will prompt you to get permission to add to known_hosts list, if you wan't remove this prompt and you know the consequences of disabling this feature and you want tor remove this prompt behavior run this on your terminal `export ANSIBLE_HOST_KEY_CHECKING=False`
++
++## Configurations
++
++To run this ansible script you must have at least 4 instances. 3 instances to setup Mesos master , marathon and zookeeper clusters. Along with Mesos master we install marathon and zookeeper. Another one or more instance/s to setup mesos agent/s. You can use ec2 ansible role provided with this playbook to spin up aws instances. To do that you need to set valid aws credentials.
++
++1. set valid aws credentials in `roles/ec2/vars/aws-credential.yml` if it doesn't work add folowing to ec2 task in `roles/ec2/tasks/main.yml`
++
++ `aws_access_key: <your_valid_access_key>`
++
++ `aws_secret_key: <your_valid_secret_key?`
++
++2. Set ec2 instance names under ec2 task `with_items:` configurations
++
++ After you set valid aws credentials and instance names run following ansible playbook command to spin up require aws ec2 instances.
++
++ `ansible-palybook -i hosts site.yml -t "ec2"`
++
++3. You need to know public ips of all the nodes and private ips of all mesos-master nodes. List all mesos master public ips under `[mesos-master]` section in `hosts` file. List all mesos-slave *(mesos-agent)* public ips under `[mesos-slave]` section. You need to set `my_id` hosts variable along with mesos-master ip addresses. This will use to set zookeeper id.
++
++ > [mesos-master]
++
++ > *1.2.3.4 my_id=1*
++
++ > *11.22.33.44 my_id=2*
++
++ > ...
++
++ > [mesos-slave]
++
++ > *123.123.123.123*
++
++ > ...
++
++4. Use mesos-master private ip addresses to set zookeeper servers properties in `roles/zookeeper/vars/main.yml`
++
++ > zookeeper_servers:
++
++ > \- {id: "1", ip: "172.31.24.149"}
++
++ > \- {id: "2", ip: "172.31.19.49"}
++
++ > \- {id: "3", ip: "172.31.25.80"}
++
++5. Set Mesos-master zookeeper quorum value in `group_vars/all.yml` file
++
++ > zk_quorum: 2
++
++6. Now we are set to deploy Apache Mesos cluster (Mesos-master, marthon, zookeeper, mesos-slave). Following ansible playbook command respectively setup mesos-master cluster , Mesos-slaves, and both mesos-master cluster and mesos-slave
++
++ `ansible-playbook -i hosts site.yml -t "mesos-master"`
++
++ `ansible-playbook -i hosts site.yml -t "mesos-slave"`
++
++ `ansible-playbook -i hosts site.yml -t "mesos"`
++
++ If everything works without any error, now you have running mesos cluster on aws instances.
++
++## verifying installation
++
++1. If your plays works without any error. Now you can access Mesos master console using `http://<master-ip>:5050` run following in one of Mesos master node to find which instance is the leader of cluster.
++
++ ``mesos-resolve `cat /etc/mesos/zk` `` this will return master-ip. Then use it to access console
++
++2. To access marathon console use `http://<master-ip>:8080`
++
++3. If you want run simple task on mesos. run following command on one of master node.
++
++ ``MASTER=$(mesos-resolve `cat /etc/mesos/zk`)``
++
++ `mesos-execute --master=$MASTER --name="cluster-test" --command="sleep 5"`
++
++
++
++## Ansible roles
++- java - install oracle java 8
++- zookeeper - install zookeeper
++- mesos-master - install mesos mastera and marathon
++- mesos-slave - install mesos slave
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/group_vars/all.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/group_vars/all.yml
index 0000000,0000000..62c83dc
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/group_vars/all.yml
@@@ -1,0 -1,0 +1,19 @@@
++---
++ansible_connection: ssh
++ansible_ssh_user: centos
++ansible_ssh_private_key_file: /Users/syodage/Projects/airavata-ansible/shameera-aws.pem.txt
++
++user: centos
++group: centos
++
++#Zookeeper related properties
++zookeeper_client_port: 2181
++# zookeeper_servers:
++# - { index: 1, ip: "", ports: "2888:3888"}
++# - { index: 2, ip: "", ports: "2888:3888"}
++# - { index: 3, ip: "", ports: "2888:3888"}
++zk_1: "{{ groups['mesos-master'][0] }}"
++zk_2: "{{ groups['mesos-master'][1] }}"
++zk_3: "{{ groups['mesos-master'][2] }}"
++
++zk_quorum: 2
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/hosts
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/hosts
index 0000000,0000000..cc90239
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/hosts
@@@ -1,0 -1,0 +1,13 @@@
++---
++# inventory fie: production
++#[localhost]
++127.0.0.1
++
++[mesos-master]
++52.87.208.230 my_id=1
++54.172.21.138 my_id=2
++54.196.170.54 my_id=3
++
++
++[mesos-slave]
++54.175.143.203
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/ec2/tasks/main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/ec2/tasks/main.yml
index 0000000,0000000..7a6eb3d
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/ec2/tasks/main.yml
@@@ -1,0 -1,0 +1,30 @@@
++---
++- name: include credentials
++ include_vars: vars/aws-credential.yml
++
++- name: create a aws instace/s
++ ec2:
++ key_name: "{{ key_name }}"
++ region: us-east-1
++ instance_type: t2.small
++ image: "{{centos_ami_7_2}}"
++ group: "{{ security_group }}"
++ instance_tags:
++ Name: "{{ item }}"
++ assign_public_ip: yes
++ state: present
++ count: 1
++ vpc_subnet_id: subnet-54f8f70c
++ volumes:
++ - device_name: /dev/sda1
++ volume_type: io1
++ iops: 200
++ volume_size: 50
++ delete_on_termination: true
++ wait: yes
++ with_items:
++ - Mesos-master-1
++ - Mesos-master-2
++ - Mesos-master-3
++ - Mesos-slave-1
++ register: ec2
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/ec2/vars/aws-credential.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/ec2/vars/aws-credential.yml
index 0000000,0000000..2e4c703
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/ec2/vars/aws-credential.yml
@@@ -1,0 -1,0 +1,3 @@@
++---
++aws_access_key: xxxxxxxxx
++aws_secret_key: xxxxxxxxx
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/ec2/vars/main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/ec2/vars/main.yml
index 0000000,0000000..1f7d77a
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/ec2/vars/main.yml
@@@ -1,0 -1,0 +1,8 @@@
++---
++
++key_name: shameera
++ec2_name: Mesos28
++centos_ami_7_2: ami-6d1c2007
++ubutntu_ami_16_04: ami-29f96d3e
++
++security_group: Mesos-sec
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/java/tasks/main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/java/tasks/main.yml
index 0000000,0000000..92f2039
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/java/tasks/main.yml
@@@ -1,0 -1,0 +1,21 @@@
++---
++# Install Orcal Java
++- name: download oracle java 8 rpm
++ get_url: url="{{ java_rpm_url }}" dest="{{ java_dir_source }}" headers='Cookie:oraclelicense=accept-securebackup-cookie'
++ become: yes
++
++- name: Install oracle java 8
++ yum: name="{{ java_dir_source }}/{{ java_rpm_filename }}" state=present
++ become: yes
++
++- name: set Oracle Java {{ java_version_string }} as default
++ alternatives:
++ name="{{ item.exe }}"
++ link="/usr/bin/{{ item.exe }}"
++ path="{{ item.path }}/{{ item.exe }}"
++ with_items:
++ - { path: "{{ java_home }}/jre/bin", exe: 'java' }
++ - { path: "{{ java_home }}/jre/bin", exe: 'keytool' }
++ - { path: "{{ java_home }}/bin", exe: 'javac' }
++ - { path: "{{ java_home }}/bin", exe: 'javadoc' }
++ become: yes
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/java/vars/main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/java/vars/main.yml
index 0000000,0000000..a995684
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/java/vars/main.yml
@@@ -1,0 -1,0 +1,15 @@@
++---
++#Variables associated with this role
++# Oracle Java 8
++java_dir_source: "/usr/local/src"
++
++java_version: 8
++java_version_update: 91
++java_version_build: '14'
++java_version_string: "1.{{ java_version }}.0_{{ java_version_update }}"
++java_home: "/usr/java/jdk1.{{ java_version }}.0_{{ java_version_update }}"
++
++java_rpm_filename: "jdk-{{ java_version }}u{{ java_version_update }}-linux-x64.rpm"
++java_rpm_url: "http://download.oracle.com/otn-pub/java/jdk/{{ java_version }}u{{ java_version_update }}-b{{ java_version_build }}/{{ java_rpm_filename }}"
++
++...
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/mesos-master/tasks/backup_main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/mesos-master/tasks/backup_main.yml
index 0000000,0000000..1d0abd1
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/mesos-master/tasks/backup_main.yml
@@@ -1,0 -1,0 +1,117 @@@
++---
++
++- name: wait untill instance is up and running
++ local_action: wait_for port=22
++ host="{{ ansible_ssh_host | default(inventory_hostname) }}"
++ search_regex=OpenSSH delay=10
++- name: create group
++ group: name="{{group}}"
++ become: yes
++
++- name: create user
++ user: name="{{user}}"
++ become: yes
++
++# RedHat related things
++- name: install required packages
++ yum: name="{{item}}" state=latest
++ with_items:
++ - wget
++ - tar
++ - git
++ - epel-release
++ - firewalld
++ become: yes
++
++- name: updage systemd
++ yum: name=systemd state=latest
++ become: yes
++
++- name: get apache maven
++ get_url:
++ url: http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo
++ dest: /etc/yum.repos.d/epel-apache-maven.repo
++ become: yes
++
++- name: copy svn repo file
++ copy:
++ src: wandisco-svn.repo
++ dest: /etc/yum.repos.d/wandisco-svn.repo
++ become: yes
++
++- name: install development tools
++ yum: name="@Development tools" state=present
++ become: yes
++
++#Install other Mesos dependencies.
++- name: install mesos dependencies
++ yum: name="{{item}}" state=latest
++ with_items:
++ - apache-maven
++ - python-devel
++ - java-1.8.0-openjdk-devel
++ - zlib-devel
++ - libcurl-devel
++ - openssl-devel
++ - cyrus-sasl-devel
++ - cyrus-sasl-md5
++ - apr-devel
++ - apr-util-devel
++ - subversion-devel
++ become: yes
++
++- name: start firewall service
++ service: name=firewalld state=started enabled=yes
++ become: yes
++
++- name: download and unarchive mesos latest
++ unarchive:
++ src="{{mesos_url}}"
++ dest="{{user_home}}"
++ remote_src=yes
++ copy=no
++ creates="{{mesos_deploy_dir}}/configure"
++ owner="{{user}}"
++ group="{{group}}"
++
++ # Bootstrap (Only required if building from git repository).
++ # $ ./bootstrap
++
++- name: make build dir
++ file: path="{{mesos_build_dir}}" state=directory owner={{user}} group={{group}}
++
++# - name: create simlink to configure script
++# file: src="{{mesos_deploy_dir}}/configure"
++# dest="{{mesos_build_dir}}/configure"
++# state=link
++# owner={{user}}
++# group={{group}}
++# tags: test
++
++- name: configure and build mesos
++ command: ../configure chdir="{{mesos_build_dir}}"
++ register: build
++
++- name: run make
++ command: make chdir="{{mesos_build_dir}}"
++ when: build|success
++ register: make
++
++# - name: run make tests
++# command: make check chdir="{{mesos_build_dir}}"
++# when: make|success
++
++- name: open firewall port
++ firewalld: port=5050/tcp zone=public permanent=true state=enabled immediate=yes
++ become: yes
++
++- name: create working dir
++ file: path="{{mesos_work_dir}}" state=directory user={{user}} group={{group}}
++
++- name: start mesos master
++ command: ./bin/mesos-master.sh --ip={{inventory_hostname}} --work_dir={{mesos_work_dir}} &
++ chdir={{mesos_build_dir}}
++
++- name: start mesos agent
++ command: /bin/mesos-agent.sh --master={{inventory_hostname}}:5050 --work_dir={{mesos_work_dir}} &
++ chdir={{mesos_build_dir}}
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/mesos-master/tasks/main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/mesos-master/tasks/main.yml
index 0000000,0000000..effb35a
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/mesos-master/tasks/main.yml
@@@ -1,0 -1,0 +1,86 @@@
++---
++# reference: https://open.mesosphere.com/getting-started/install/
++- name: install firewalld
++ yum:
++ name: firewalld
++ state: latest
++ become: yes
++
++- name: start firewalld
++ service:
++ name: firewalld
++ state: started
++ enabled: yes
++ become: yes
++
++- name: open ports
++ firewalld:
++ port: "{{ item }}"
++ zone: public
++ permanent: true
++ state: enabled
++ immediate: yes
++ with_items:
++ - "5050/tcp"
++ - "8080/tcp"
++ become: yes
++
++- name: install utility - TODO delete this
++ yum:
++ name: "{{ item }}"
++ state: latest
++ with_items:
++ - vim
++ become: yes
++
++# Setup repository
++- name: add mesosphere rpm
++ yum:
++ name: http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
++ state: present
++ become: yes
++
++- name: install mesos and marathon
++ yum:
++ name: "{{ item }}"
++ state: latest
++ with_items:
++ - mesos
++ - marathon
++ # - mesosphere-zookeeper
++ become: yes
++
++# configurations
++
++- name: set zk file for mesos master and marathon
++ template:
++ src: zk.j2
++ dest: /etc/mesos/zk
++ force: yes
++ become: yes
++
++- name: set zookeeper quorum
++ template:
++ src: quorum.j2
++ dest: /etc/mesos-master/quorum
++ force: yes
++ become: yes
++
++- name: stop and disable mesos slave
++ service:
++ name: mesos-slave
++ enabled: no
++ state: stopped
++ become: yes
++
++- name: restart mesos master
++ service:
++ name: mesos-master
++ state: restarted
++ become: yes
++
++- name: restart marathon
++ service:
++ name: marathon
++ state: restarted
++ become: yes
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/mesos-master/templates/quorum.j2
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/mesos-master/templates/quorum.j2
index 0000000,0000000..c1d54ef
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/mesos-master/templates/quorum.j2
@@@ -1,0 -1,0 +1,1 @@@
++{{zk_quorum}}
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/mesos-master/templates/zk.j2
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/mesos-master/templates/zk.j2
index 0000000,0000000..e82fe65
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/mesos-master/templates/zk.j2
@@@ -1,0 -1,0 +1,1 @@@
++zk://{{zk_1}}:{{zookeeper_client_port}},{{zk_2}}:{{zookeeper_client_port}},{{zk_3}}:{{zookeeper_client_port}}/mesos
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/mesos-slave/tasks/main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/mesos-slave/tasks/main.yml
index 0000000,0000000..f6890fe
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/mesos-slave/tasks/main.yml
@@@ -1,0 -1,0 +1,66 @@@
++---
++- name: install firewalld
++ yum:
++ name: firewalld
++ state: latest
++ update_cache: yes
++ become: yes
++
++- name: start firewalld
++ service:
++ name: firewalld
++ state: started
++ enabled: yes
++ become: yes
++
++- name: open ports
++ firewalld:
++ port: "{{ item }}"
++ zone: public
++ permanent: true
++ state: enabled
++ immediate: yes
++ with_items:
++ - "5051/tcp"
++ become: yes
++
++# setup mesos slave
++- name: add mesosphere rpm
++ yum:
++ name: http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
++ state: present
++ become: yes
++
++- name: install mesos
++ yum:
++ name: mesos
++ state: latest
++ become: yes
++
++# zookeeper will be pulled in and isntall as a depencency automatically
++# - name: disable zookeeper
++# service:
++# name: mesosphere-zookeeper
++# state: stopped
++# enabled: no
++# become: yes
++
++- name: set zk file for mesos master and marathon
++ template:
++ src: zk.j2
++ dest: /etc/mesos/zk
++ force: yes
++ become: yes
++
++- name: disable mesos-master
++ service:
++ name: mesos-master
++ state: stopped
++ enabled: no
++ become: yes
++
++- name: restart mesos slave
++ service:
++ name: mesos-slave
++ state: restarted
++ become: yes
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/mesos-slave/templates/zk.j2
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/mesos-slave/templates/zk.j2
index 0000000,0000000..e82fe65
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/mesos-slave/templates/zk.j2
@@@ -1,0 -1,0 +1,1 @@@
++zk://{{zk_1}}:{{zookeeper_client_port}},{{zk_2}}:{{zookeeper_client_port}},{{zk_3}}:{{zookeeper_client_port}}/mesos
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/test/tasks/main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/test/tasks/main.yml
index 0000000,0000000..f0548b1
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/test/tasks/main.yml
@@@ -1,0 -1,0 +1,3 @@@
++- debug: msg="ding {{ inventory_hostname }}"
++
++- debug: msg="{{ inventory }}"
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/zookeeper/handlers/main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/zookeeper/handlers/main.yml
index 0000000,0000000..f1ea4be
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/zookeeper/handlers/main.yml
@@@ -1,0 -1,0 +1,15 @@@
++---
++- name: start zookeeper
++ command: bin/zkServer.sh start
++ args:
++ chdir: "{{zookeeper_dir}}"
++
++- name: restart zookeeper
++ command: bin/zkServer.sh restart
++ args:
++ chdir: "{{zookeeper_dir}}"
++
++- name: stop zookeeper
++ command: bin/zkServer.sh stop
++ args:
++ chdir: "{{zookeeper_dir}}"
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/zookeeper/tasks/main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/zookeeper/tasks/main.yml
index 0000000,0000000..02c246f
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/zookeeper/tasks/main.yml
@@@ -1,0 -1,0 +1,98 @@@
++---
++- name: install pre-requities
++ yum:
++ name: "{{item}}"
++ state: latest
++ with_items:
++ - firewalld
++ - vim
++ become: yes
++
++- name: start firewalld
++ service:
++ name: firewalld
++ state: started
++ become: yes
++
++- name: open zookeeper port
++ firewalld:
++ port: "{{item}}"
++ zone: public
++ permanent: true
++ state: enabled
++ immediate: yes
++ with_items:
++ - "{{zookeeper_client_port}}/tcp"
++ - 2888/tcp
++ - 3888/tcp
++ become: yes
++
++# - name: get ip4 address
++# command: hostname -i
++# register: _ip4
++# tags: ttt
++
++# - debug:
++# msg: "{{_ip4.stdout}}"
++# tags: ttt
++
++######################
++- name: Download and unarchive zookeeper
++ unarchive:
++ src: "{{ zookeeper_url }}"
++ dest: /opt
++ copy: no
++ creates: /opt/zookeeper-{{ zookeeper_version }}
++ owner: "{{ user }}"
++ group: "{{ group }}"
++ become: yes
++
++- name: Link to Zookeeper Directory
++ file:
++ src: "/opt/zookeeper-{{ zookeeper_version }}"
++ dest: "{{zookeeper_dir}}"
++ state: link
++ force: yes
++ become: yes
++
++- name: Create Zookeeper Log Dir
++ file:
++ path: "{{zookeeper_log_dir}}"
++ owner: "{{ user }}"
++ group: "{{ group }}"
++ recurse: yes
++ state: directory
++ mode: 0644
++ become: yes
++
++- name: Create Zookeeper DataStore Dir
++ file:
++ path: "{{zookeeper_datastore_dir}}"
++ owner: "{{ user }}"
++ group: "{{ group }}"
++ recurse: yes
++ state: directory
++ mode: 0755
++ become: yes
++
++- name: Copy zoo.cfg file
++ template:
++ src: zoo.cfg.j2
++ # dest: "{{zookeeper_conf_dir}}/zoo.cfg"
++ dest: "{{zookeeper_dir}}/conf/zoo.cfg"
++ owner: "{{ user }}"
++ group: "{{ group }}"
++ mode: "u=rw,g=r,o=r"
++ notify:
++ - restart zookeeper
++
++
++- name: Copy myid file
++ template:
++ src: myid.j2
++ dest: "{{zookeeper_datastore_dir}}/myid"
++ owner: "{{ user }}"
++ group: "{{ group }}"
++ mode: "u=rw,g=r,o=r"
++ notify:
++ - restart zookeeper
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/zookeeper/templates/myid.j2
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/zookeeper/templates/myid.j2
index 0000000,0000000..78a0bd7
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/zookeeper/templates/myid.j2
@@@ -1,0 -1,0 +1,1 @@@
++{{ my_id }}
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/zookeeper/templates/zoo.cfg.j2
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/zookeeper/templates/zoo.cfg.j2
index 0000000,0000000..4845a71
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/zookeeper/templates/zoo.cfg.j2
@@@ -1,0 -1,0 +1,32 @@@
++# Licensed to the Apache Software Foundation (ASF) under one or more
++# contributor license agreements. See the NOTICE file distributed with
++# this work for additional information regarding copyright ownership.
++# The ASF licenses this file to You under the Apache License, Version 2.0
++# (the "License"); you may not use this file except in compliance with
++# the License. You may obtain a copy of the License at
++#
++# http://www.apache.org/licenses/LICENSE-2.0
++#
++# Unless required by applicable law or agreed to in writing, software
++# distributed under the License is distributed on an "AS IS" BASIS,
++# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++# See the License for the specific language governing permissions and
++# limitations under the License.
++
++maxClientCnxns=50
++# The number of milliseconds of each tick
++tickTime=2000
++# The number of ticks that the initial
++# synchronization phase can take
++initLimit=10
++# The number of ticks that can pass between
++# sending a request and getting an acknowledgement
++syncLimit=5
++# the directory where the snapshot is stored.
++dataDir={{zookeeper_datastore_dir}}
++# the port at which the clients will connect
++clientPort={{zookeeper_client_port}}
++
++{% for server in zookeeper_servers %}
++server.{{ server.id }}={{ server.ip }}:2888:3888
++{% endfor %}
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/roles/zookeeper/vars/main.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/roles/zookeeper/vars/main.yml
index 0000000,0000000..07f9ca1
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/roles/zookeeper/vars/main.yml
@@@ -1,0 -1,0 +1,15 @@@
++---
++zookeeper_version: 3.4.8
++#zookeeper_conf_dir: /etc/zookeeper/conf
++zookeeper_log_dir: /var/log/zookeeper
++zookeeper_datastore_dir: /var/lib/zookeeper
++zookeeper_url: http://www.us.apache.org/dist/zookeeper/zookeeper-{{zookeeper_version}}/zookeeper-{{zookeeper_version}}.tar.gz
++zookeeper_dir: /opt/zookeeper
++#zookeeper_client_port: 2181
++
++zookeeper_servers:
++ - {id: "1", ip: "172.31.24.149"}
++ - {id: "2", ip: "172.31.19.49"}
++ - {id: "3", ip: "172.31.25.80"}
++#exhibitor_version: 1.5.5
++#exhibitor_install_dir: /opt/exhibitor
http://git-wip-us.apache.org/repos/asf/airavata/blob/7071ef17/modules/cloud/ansible-playbooks/site.yml
----------------------------------------------------------------------
diff --cc modules/cloud/ansible-playbooks/site.yml
index 0000000,0000000..17258bd
new file mode 100644
--- /dev/null
+++ b/modules/cloud/ansible-playbooks/site.yml
@@@ -1,0 -1,0 +1,19 @@@
++---
++- hosts: localhost
++ tags: ec2
++ roles:
++ - ec2
++
++- hosts: mesos-master
++ tags: mesos, mesos-master
++ roles:
++ - java
++ - zookeeper
++ - mesos-master
++
++- hosts: mesos-slave
++ tags: mesos, mesos-slave
++ roles:
++ - mesos-slave
++
++...
[2/9] airavata git commit: Added zookeeper and mesos roles
Posted by sh...@apache.org.
Added zookeeper and mesos roles
Project: http://git-wip-us.apache.org/repos/asf/airavata/repo
Commit: http://git-wip-us.apache.org/repos/asf/airavata/commit/7429aa65
Tree: http://git-wip-us.apache.org/repos/asf/airavata/tree/7429aa65
Diff: http://git-wip-us.apache.org/repos/asf/airavata/diff/7429aa65
Branch: refs/heads/develop
Commit: 7429aa65b0eff708debde76aadd65a04c179c157
Parents: dc6012e
Author: Shameera Rathnayaka <sh...@gmail.com>
Authored: Tue Aug 30 15:30:27 2016 -0400
Committer: Shameera Rathnayaka <sh...@gmail.com>
Committed: Tue Aug 30 15:30:27 2016 -0400
----------------------------------------------------------------------
hosts | 8 ++
roles/group_vars/all.yml | 13 ++++
roles/mesos/tasks/RedHat.yml | 18 +++++
roles/mesos/tasks/main.yml | 84 +++++++++++++++++++++
roles/test/tasks/main.yml | 1 +
roles/zookeeper/handlers/main.yml | 12 +++
roles/zookeeper/tasks/main.yml | 34 +++++++++
roles/zookeeper/templates/zoo.cfg.j2 | 28 +++++++
roles/zookeeper/templates/zookeeper.service.j2 | 15 ++++
roles/zookeeper/vars/main.yml | 17 +++++
site.yml | 11 +++
11 files changed, 241 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/hosts
----------------------------------------------------------------------
diff --git a/hosts b/hosts
new file mode 100644
index 0000000..949fb79
--- /dev/null
+++ b/hosts
@@ -0,0 +1,8 @@
+---
+[localhost]
+127.0.0.1
+
+[mesos-master]
+
+
+[mesos-agent]
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/roles/group_vars/all.yml
----------------------------------------------------------------------
diff --git a/roles/group_vars/all.yml b/roles/group_vars/all.yml
new file mode 100644
index 0000000..c041787
--- /dev/null
+++ b/roles/group_vars/all.yml
@@ -0,0 +1,13 @@
+---
+
+user: centos
+group: centos
+user_home: "home/{{user}}"
+
+# Mesos related properties
+mesos_dir: "{{user_home}}/mesos"
+mesos_build_dir: "{{mesos_dir}}/build"
+
+
+#Zookeeper related properties
+zookeeper_client_port: 2181
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/roles/mesos/tasks/RedHat.yml
----------------------------------------------------------------------
diff --git a/roles/mesos/tasks/RedHat.yml b/roles/mesos/tasks/RedHat.yml
new file mode 100644
index 0000000..4c7ace9
--- /dev/null
+++ b/roles/mesos/tasks/RedHat.yml
@@ -0,0 +1,18 @@
+---
+- name: create group
+ group: name="{{group}}"
+ become: yes
+
+- name: create user
+ user: name="{{user}}"
+ become: yes
+
+- name: install required packages
+ yum: name="{{item}}" state=latest
+ with_item:
+ - wget
+ - tar
+ - git
+ - epel-release
+ - subversion
+ - firewalld
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/roles/mesos/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/mesos/tasks/main.yml b/roles/mesos/tasks/main.yml
new file mode 100644
index 0000000..f0fa90c
--- /dev/null
+++ b/roles/mesos/tasks/main.yml
@@ -0,0 +1,84 @@
+---
+- name: create group
+ group: name="{{group}}"
+ become: yes
+
+- name: create user
+ user: name="{{user}}"
+ become: yes
+
+# RedHat related things
+- name: install required packages
+ yum: name="{{item}}" state=latest
+ with_items:
+ - wget
+ - tar
+ - git
+ - epel-release
+ - firewalld
+ become: yes
+
+- name: updage systemd
+ yum: name=systemd state=latest enabled=yes
+ become: yes
+
+- name: install development tools
+ yum: name="@Development tools" state=latest
+ become: yes
+
+ # Install other Mesos dependencies.
+- name: install mesos dependencies
+ yum: name="{{item}}" state=latest
+ with_items:
+ - apache-maven
+ - python-devel
+ - java-1.8.0-openjdk-devel
+ - zlib-devel
+ - libcurl-devel
+ - openssl-devel
+ - cyrus-sasl-devel
+ - cyrus-sasl-md5
+ - apr-devel
+ - apr-util-devel
+ - subversion-devel
+
+- name: start firewall service
+ service: name=firewalld state=started enabled=yes
+ become: yes
+
+- name: download and unarchive mesos latest
+ unarchive:
+ src=http://www.apache.org/dist/mesos/1.0.1/mesos-1.0.1.tar.gz
+ dest="{{mesos_dir}}"
+ owner="{{user}}"
+ group="{{group}}"
+
+ # Bootstrap (Only required if building from git repository).
+ # $ ./bootstrap
+
+- name: make build dir
+ file: path="{{mesos_dir}}/build" state=directory user={{user}} group={{group}}
+
+- name: configure and build mesos
+ command: ../configure chdir="{{mesos_build_dir}}"
+ register: build
+
+- name: run make
+ command: make chdir="{{mesos_build_dir}}"
+ when: build|success
+ register: make
+
+- name: run make tests
+ command: make check chdir="{{mesos_build_dir}}"
+ when: make|success
+
+- name: create working dir
+ file: path="{{mesos_work_dir}}" state=directory user={{user}} group={{group}}
+
+- name: start mesos master
+ command: ./bin/mesos-master.sh --ip={{inventory_hostname}} --work_dir={{mesos_work_dir}} &
+ chdir={{mesos_build_dir}}
+
+- name: start mesos agent
+ command: /bin/mesos-agent.sh --master={{inventory_hostname}}:5050 --work_dir={{mesos_work_dir}} &
+ chdir={{mesos_build_dir}}
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/roles/test/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/test/tasks/main.yml b/roles/test/tasks/main.yml
new file mode 100644
index 0000000..b3cb5ca
--- /dev/null
+++ b/roles/test/tasks/main.yml
@@ -0,0 +1 @@
+- debug: msg="ding {{ inventory_hostname }}"
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/roles/zookeeper/handlers/main.yml
----------------------------------------------------------------------
diff --git a/roles/zookeeper/handlers/main.yml b/roles/zookeeper/handlers/main.yml
new file mode 100644
index 0000000..daefebe
--- /dev/null
+++ b/roles/zookeeper/handlers/main.yml
@@ -0,0 +1,12 @@
+---
+- name: start zookeeper
+ service: name=zookeeper state=started enabled=yes
+ become: yes
+
+- name: stop zookeeper
+ service: name=zookeeper state=stopped
+ become: yes
+
+- name: restart zookeeper
+ service: name=zookeeper state=restarted enabled=yes
+ become: yes
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/roles/zookeeper/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/zookeeper/tasks/main.yml b/roles/zookeeper/tasks/main.yml
new file mode 100644
index 0000000..78bcec2
--- /dev/null
+++ b/roles/zookeeper/tasks/main.yml
@@ -0,0 +1,34 @@
+---
+################################################################################
+# Setup and run Zookeeper
+- name: Download and unarchive zookeeper
+ unarchive: src="{{ zookeeper_url }}"
+ dest="{{ user_home }}"
+ copy=no
+ owner="{{ user }}"
+ group="{{ group }}"
+
+- name: open zookeeper port
+ firewalld: port=2181/tcp zone=public permanent=true state=enabled immediate=yes
+ become: yes
+
+- name: Copy zoo.cfg file
+ template: src=zoo.cfg.j2 dest="{{ zookeeper_dir }}/conf/zoo.cfg" owner="{{ user }}" group="{{ group }}" mode="u=rw,g=r,o=r"
+ notify:
+ - restart zookeeper
+
+- name: Check if systemd exists
+ stat: path=/usr/lib/systemd/system/
+ register: systemd_check
+
+- name: Systemd script.
+ template: src=zookeeper.service.j2 dest=/usr/lib/systemd/system/zookeeper.service
+ when: systemd_check.stat.exists == true
+ notify:
+ - restart zookeeper
+ become: yes
+
+- name: reload systemd daemons
+ command: systemctl daemon-reload
+ become: yes
+...
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/roles/zookeeper/templates/zoo.cfg.j2
----------------------------------------------------------------------
diff --git a/roles/zookeeper/templates/zoo.cfg.j2 b/roles/zookeeper/templates/zoo.cfg.j2
new file mode 100644
index 0000000..8426b98
--- /dev/null
+++ b/roles/zookeeper/templates/zoo.cfg.j2
@@ -0,0 +1,28 @@
+# The number of milliseconds of each tick
+tickTime={{tick_time}}
+# The number of ticks that the initial
+# synchronization phase can take
+initLimit={{init_limit}}
+# The number of ticks that can pass between
+# sending a request and getting an acknowledgement
+syncLimit={{sync_limit}}
+# the directory where the snapshot is stored.
+# do not use /tmp for storage, /tmp here is just
+# example sakes.
+dataDir={{zookeeper_data_dir}}
+# the port at which the clients will connect
+clientPort={{ client_port }}
+# the maximum number of client connections.
+# increase this if you need to handle more clients
+#maxClientCnxns=60
+#
+# Be sure to read the maintenance section of the
+# administrator guide before turning on autopurge.
+#
+# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
+#
+# The number of snapshots to retain in dataDir
+#autopurge.snapRetainCount=3
+# Purge task interval in hours
+# Set to "0" to disable auto purge feature
+#autopurge.purgeInterval=1
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/roles/zookeeper/templates/zookeeper.service.j2
----------------------------------------------------------------------
diff --git a/roles/zookeeper/templates/zookeeper.service.j2 b/roles/zookeeper/templates/zookeeper.service.j2
new file mode 100644
index 0000000..19c3718
--- /dev/null
+++ b/roles/zookeeper/templates/zookeeper.service.j2
@@ -0,0 +1,15 @@
+# {{ansible_managed}}
+
+[Unit]
+Description=ZooKeeper
+
+
+[Service]
+Type=forking
+ExecStart={{zookeeper_dir}}/bin/zkServer.sh start
+ExecStop={{ zookeeper_dir }}/bin/zkServer.sh stop
+Restart=always
+TimeoutSec=300
+
+[Install]
+WantedBy=multi-user.target
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/roles/zookeeper/vars/main.yml
----------------------------------------------------------------------
diff --git a/roles/zookeeper/vars/main.yml b/roles/zookeeper/vars/main.yml
new file mode 100644
index 0000000..a3e10db
--- /dev/null
+++ b/roles/zookeeper/vars/main.yml
@@ -0,0 +1,17 @@
+---
+#Variables associated with this role
+# zookeeper related variable
+zookeeper_version: 3.4.8
+zookeeper_url: http://www.us.apache.org/dist/zookeeper/zookeeper-{{zookeeper_version}}/zookeeper-{{zookeeper_version}}.tar.gz
+
+apt_cache_timeout: 3600
+client_port: "{{ zookeeper_client_port }}"
+init_limit: 5
+sync_limit: 2
+tick_time: 2000
+data_dir: /var/lib/zookeeper
+log_dir: /var/log/zookeeper
+zookeeper_dir: "{{ user_home }}/zookeeper-{{zookeeper_version}}"
+zookeeper_data_dir: "{{ zookeeper_dir }}/data"
+
+...
http://git-wip-us.apache.org/repos/asf/airavata/blob/7429aa65/site.yml
----------------------------------------------------------------------
diff --git a/site.yml b/site.yml
new file mode 100644
index 0000000..2fc7293
--- /dev/null
+++ b/site.yml
@@ -0,0 +1,11 @@
+---
+- hosts: localhost
+ tags: test
+ roles:
+ - test
+
+
+- hosts: mesos-master
+ tags: mesos-master
+ roles:
+ - mesos
[6/9] airavata git commit: successfully started mesos-master with
zookeeper ring
Posted by sh...@apache.org.
successfully started mesos-master with zookeeper ring
Project: http://git-wip-us.apache.org/repos/asf/airavata/repo
Commit: http://git-wip-us.apache.org/repos/asf/airavata/commit/1ae82630
Tree: http://git-wip-us.apache.org/repos/asf/airavata/tree/1ae82630
Diff: http://git-wip-us.apache.org/repos/asf/airavata/diff/1ae82630
Branch: refs/heads/develop
Commit: 1ae8263079fc45fbff6be88706675ed239c973f3
Parents: 223ea3a
Author: Shameera Rathnayaka <sh...@gmail.com>
Authored: Sun Sep 11 01:23:50 2016 -0400
Committer: Shameera Rathnayaka <sh...@gmail.com>
Committed: Sun Sep 11 01:23:50 2016 -0400
----------------------------------------------------------------------
group_vars/all.yml | 2 +-
hosts | 10 ++-
roles/java/tasks/main.yml | 21 ++++++
roles/java/vars/main.yml | 15 +++++
roles/mesos-master/tasks/main.yml | 38 ++++-------
roles/mesos-master/templates/myid.j2 | 1 -
roles/mesos-master/templates/zk.j2 | 2 +-
roles/mesos-master/templates/zoo.cfg.j2 | 32 ---------
roles/mesos-slave/templates/zk.j2 | 2 +-
roles/zookeeper/handlers/main.yml | 15 +++++
roles/zookeeper/tasks/main.yml | 98 ++++++++++++++++++++++++++++
roles/zookeeper/templates/myid.j2 | 1 +
roles/zookeeper/templates/zoo.cfg.j2 | 32 +++++++++
roles/zookeeper/vars/main.yml | 15 +++++
site.yml | 16 +++--
15 files changed, 229 insertions(+), 71 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/group_vars/all.yml
----------------------------------------------------------------------
diff --git a/group_vars/all.yml b/group_vars/all.yml
index b0a6cf2..f7d7ae5 100644
--- a/group_vars/all.yml
+++ b/group_vars/all.yml
@@ -19,7 +19,7 @@ user_home: "/home/{{user}}"
#Zookeeper related properties
-zk_client_port: 2181
+zookeeper_client_port: 2181
# zookeeper_servers:
# - { index: 1, ip: "", ports: "2888:3888"}
# - { index: 2, ip: "", ports: "2888:3888"}
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/hosts
----------------------------------------------------------------------
diff --git a/hosts b/hosts
index 7ceb1f7..cc90239 100644
--- a/hosts
+++ b/hosts
@@ -1,9 +1,13 @@
---
# inventory fie: production
#[localhost]
-#127.0.0.1
+127.0.0.1
[mesos-master]
-52.91.222.138
+52.87.208.230 my_id=1
+54.172.21.138 my_id=2
+54.196.170.54 my_id=3
-#[mesos-agent]
+
+[mesos-slave]
+54.175.143.203
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/java/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/java/tasks/main.yml b/roles/java/tasks/main.yml
new file mode 100644
index 0000000..92f2039
--- /dev/null
+++ b/roles/java/tasks/main.yml
@@ -0,0 +1,21 @@
+---
+# Install Orcal Java
+- name: download oracle java 8 rpm
+ get_url: url="{{ java_rpm_url }}" dest="{{ java_dir_source }}" headers='Cookie:oraclelicense=accept-securebackup-cookie'
+ become: yes
+
+- name: Install oracle java 8
+ yum: name="{{ java_dir_source }}/{{ java_rpm_filename }}" state=present
+ become: yes
+
+- name: set Oracle Java {{ java_version_string }} as default
+ alternatives:
+ name="{{ item.exe }}"
+ link="/usr/bin/{{ item.exe }}"
+ path="{{ item.path }}/{{ item.exe }}"
+ with_items:
+ - { path: "{{ java_home }}/jre/bin", exe: 'java' }
+ - { path: "{{ java_home }}/jre/bin", exe: 'keytool' }
+ - { path: "{{ java_home }}/bin", exe: 'javac' }
+ - { path: "{{ java_home }}/bin", exe: 'javadoc' }
+ become: yes
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/java/vars/main.yml
----------------------------------------------------------------------
diff --git a/roles/java/vars/main.yml b/roles/java/vars/main.yml
new file mode 100644
index 0000000..a995684
--- /dev/null
+++ b/roles/java/vars/main.yml
@@ -0,0 +1,15 @@
+---
+#Variables associated with this role
+# Oracle Java 8
+java_dir_source: "/usr/local/src"
+
+java_version: 8
+java_version_update: 91
+java_version_build: '14'
+java_version_string: "1.{{ java_version }}.0_{{ java_version_update }}"
+java_home: "/usr/java/jdk1.{{ java_version }}.0_{{ java_version_update }}"
+
+java_rpm_filename: "jdk-{{ java_version }}u{{ java_version_update }}-linux-x64.rpm"
+java_rpm_url: "http://download.oracle.com/otn-pub/java/jdk/{{ java_version }}u{{ java_version_update }}-b{{ java_version_build }}/{{ java_rpm_filename }}"
+
+...
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/mesos-master/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/mesos-master/tasks/main.yml b/roles/mesos-master/tasks/main.yml
index e132915..6afefe1 100644
--- a/roles/mesos-master/tasks/main.yml
+++ b/roles/mesos-master/tasks/main.yml
@@ -21,13 +21,21 @@
state: enabled
immediate: yes
with_items:
- - "{{ zk_client_port }}/tcp"
- - "2888/tcp"
- - "3888/tcp"
+ # - "{{ zk_client_port }}/tcp"
+ # - "2888/tcp"
+ # - "3888/tcp"
- "5050/tcp"
# - "marathon/tcp"
become: yes
+- name: install utility - TODO delete this
+ yum:
+ name: "{{ item }}"
+ state: latest
+ with_items:
+ - vim
+ become: yes
+
# Setup repository
- name: add mesosphere rpm
yum:
@@ -35,37 +43,17 @@
state: present
become: yes
-- name: install mesos , marathon and mesosphere-zookeeper
+- name: install mesos and marathon
yum:
name: "{{ item }}"
state: latest
with_items:
- mesos
- marathon
- - mesosphere-zookeeper
+ # - mesosphere-zookeeper
become: yes
# configurations
-#setting ID
-- name: copy myid file
- template:
- src: myid.j2
- dest: /var/lib/zookeeper/myid
- force: yes
- become: yes
-
-- name: append zookeeper ring ips
- template:
- src: zoo.cfg.j2
- dest: /etc/zookeeper/conf/zoo.cfg
- force: yes
- become: yes
-
-- name: start Zookeeper
- service:
- name: zookeeper
- state: started
- become: yes
- name: set zk file for mesos master and marathon
template:
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/mesos-master/templates/myid.j2
----------------------------------------------------------------------
diff --git a/roles/mesos-master/templates/myid.j2 b/roles/mesos-master/templates/myid.j2
deleted file mode 100644
index 78a0bd7..0000000
--- a/roles/mesos-master/templates/myid.j2
+++ /dev/null
@@ -1 +0,0 @@
-{{ my_id }}
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/mesos-master/templates/zk.j2
----------------------------------------------------------------------
diff --git a/roles/mesos-master/templates/zk.j2 b/roles/mesos-master/templates/zk.j2
index 9c7747b..e82fe65 100644
--- a/roles/mesos-master/templates/zk.j2
+++ b/roles/mesos-master/templates/zk.j2
@@ -1 +1 @@
-zk://{{zk_1}}:{{zk_client_port}},{{zk_2}}:{{zk_client_port}},{{zk_3}}:{{zk_client_port}}/mesos
+zk://{{zk_1}}:{{zookeeper_client_port}},{{zk_2}}:{{zookeeper_client_port}},{{zk_3}}:{{zookeeper_client_port}}/mesos
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/mesos-master/templates/zoo.cfg.j2
----------------------------------------------------------------------
diff --git a/roles/mesos-master/templates/zoo.cfg.j2 b/roles/mesos-master/templates/zoo.cfg.j2
deleted file mode 100644
index 922d0b2..0000000
--- a/roles/mesos-master/templates/zoo.cfg.j2
+++ /dev/null
@@ -1,32 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-maxClientCnxns=50
-# The number of milliseconds of each tick
-tickTime=2000
-# The number of ticks that the initial
-# synchronization phase can take
-initLimit=10
-# The number of ticks that can pass between
-# sending a request and getting an acknowledgement
-syncLimit=5
-# the directory where the snapshot is stored.
-dataDir=/var/lib/zookeeper
-# the port at which the clients will connect
-clientPort={{ zk_client_port }}
-
-server.1={{zk_1}}:2888:3888
-server.2={{zk_2}}:2888:3888
-server.3={{zk_3}}:2888:3888
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/mesos-slave/templates/zk.j2
----------------------------------------------------------------------
diff --git a/roles/mesos-slave/templates/zk.j2 b/roles/mesos-slave/templates/zk.j2
index 9c7747b..e82fe65 100644
--- a/roles/mesos-slave/templates/zk.j2
+++ b/roles/mesos-slave/templates/zk.j2
@@ -1 +1 @@
-zk://{{zk_1}}:{{zk_client_port}},{{zk_2}}:{{zk_client_port}},{{zk_3}}:{{zk_client_port}}/mesos
+zk://{{zk_1}}:{{zookeeper_client_port}},{{zk_2}}:{{zookeeper_client_port}},{{zk_3}}:{{zookeeper_client_port}}/mesos
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/zookeeper/handlers/main.yml
----------------------------------------------------------------------
diff --git a/roles/zookeeper/handlers/main.yml b/roles/zookeeper/handlers/main.yml
new file mode 100644
index 0000000..f1ea4be
--- /dev/null
+++ b/roles/zookeeper/handlers/main.yml
@@ -0,0 +1,15 @@
+---
+- name: start zookeeper
+ command: bin/zkServer.sh start
+ args:
+ chdir: "{{zookeeper_dir}}"
+
+- name: restart zookeeper
+ command: bin/zkServer.sh restart
+ args:
+ chdir: "{{zookeeper_dir}}"
+
+- name: stop zookeeper
+ command: bin/zkServer.sh stop
+ args:
+ chdir: "{{zookeeper_dir}}"
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/zookeeper/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/zookeeper/tasks/main.yml b/roles/zookeeper/tasks/main.yml
new file mode 100644
index 0000000..02c246f
--- /dev/null
+++ b/roles/zookeeper/tasks/main.yml
@@ -0,0 +1,98 @@
+---
+- name: install pre-requities
+ yum:
+ name: "{{item}}"
+ state: latest
+ with_items:
+ - firewalld
+ - vim
+ become: yes
+
+- name: start firewalld
+ service:
+ name: firewalld
+ state: started
+ become: yes
+
+- name: open zookeeper port
+ firewalld:
+ port: "{{item}}"
+ zone: public
+ permanent: true
+ state: enabled
+ immediate: yes
+ with_items:
+ - "{{zookeeper_client_port}}/tcp"
+ - 2888/tcp
+ - 3888/tcp
+ become: yes
+
+# - name: get ip4 address
+# command: hostname -i
+# register: _ip4
+# tags: ttt
+
+# - debug:
+# msg: "{{_ip4.stdout}}"
+# tags: ttt
+
+######################
+- name: Download and unarchive zookeeper
+ unarchive:
+ src: "{{ zookeeper_url }}"
+ dest: /opt
+ copy: no
+ creates: /opt/zookeeper-{{ zookeeper_version }}
+ owner: "{{ user }}"
+ group: "{{ group }}"
+ become: yes
+
+- name: Link to Zookeeper Directory
+ file:
+ src: "/opt/zookeeper-{{ zookeeper_version }}"
+ dest: "{{zookeeper_dir}}"
+ state: link
+ force: yes
+ become: yes
+
+- name: Create Zookeeper Log Dir
+ file:
+ path: "{{zookeeper_log_dir}}"
+ owner: "{{ user }}"
+ group: "{{ group }}"
+ recurse: yes
+ state: directory
+ mode: 0644
+ become: yes
+
+- name: Create Zookeeper DataStore Dir
+ file:
+ path: "{{zookeeper_datastore_dir}}"
+ owner: "{{ user }}"
+ group: "{{ group }}"
+ recurse: yes
+ state: directory
+ mode: 0755
+ become: yes
+
+- name: Copy zoo.cfg file
+ template:
+ src: zoo.cfg.j2
+ # dest: "{{zookeeper_conf_dir}}/zoo.cfg"
+ dest: "{{zookeeper_dir}}/conf/zoo.cfg"
+ owner: "{{ user }}"
+ group: "{{ group }}"
+ mode: "u=rw,g=r,o=r"
+ notify:
+ - restart zookeeper
+
+
+- name: Copy myid file
+ template:
+ src: myid.j2
+ dest: "{{zookeeper_datastore_dir}}/myid"
+ owner: "{{ user }}"
+ group: "{{ group }}"
+ mode: "u=rw,g=r,o=r"
+ notify:
+ - restart zookeeper
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/zookeeper/templates/myid.j2
----------------------------------------------------------------------
diff --git a/roles/zookeeper/templates/myid.j2 b/roles/zookeeper/templates/myid.j2
new file mode 100644
index 0000000..78a0bd7
--- /dev/null
+++ b/roles/zookeeper/templates/myid.j2
@@ -0,0 +1 @@
+{{ my_id }}
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/zookeeper/templates/zoo.cfg.j2
----------------------------------------------------------------------
diff --git a/roles/zookeeper/templates/zoo.cfg.j2 b/roles/zookeeper/templates/zoo.cfg.j2
new file mode 100644
index 0000000..4845a71
--- /dev/null
+++ b/roles/zookeeper/templates/zoo.cfg.j2
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+maxClientCnxns=50
+# The number of milliseconds of each tick
+tickTime=2000
+# The number of ticks that the initial
+# synchronization phase can take
+initLimit=10
+# The number of ticks that can pass between
+# sending a request and getting an acknowledgement
+syncLimit=5
+# the directory where the snapshot is stored.
+dataDir={{zookeeper_datastore_dir}}
+# the port at which the clients will connect
+clientPort={{zookeeper_client_port}}
+
+{% for server in zookeeper_servers %}
+server.{{ server.id }}={{ server.ip }}:2888:3888
+{% endfor %}
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/roles/zookeeper/vars/main.yml
----------------------------------------------------------------------
diff --git a/roles/zookeeper/vars/main.yml b/roles/zookeeper/vars/main.yml
new file mode 100644
index 0000000..07f9ca1
--- /dev/null
+++ b/roles/zookeeper/vars/main.yml
@@ -0,0 +1,15 @@
+---
+zookeeper_version: 3.4.8
+#zookeeper_conf_dir: /etc/zookeeper/conf
+zookeeper_log_dir: /var/log/zookeeper
+zookeeper_datastore_dir: /var/lib/zookeeper
+zookeeper_url: http://www.us.apache.org/dist/zookeeper/zookeeper-{{zookeeper_version}}/zookeeper-{{zookeeper_version}}.tar.gz
+zookeeper_dir: /opt/zookeeper
+#zookeeper_client_port: 2181
+
+zookeeper_servers:
+ - {id: "1", ip: "172.31.24.149"}
+ - {id: "2", ip: "172.31.19.49"}
+ - {id: "3", ip: "172.31.25.80"}
+#exhibitor_version: 1.5.5
+#exhibitor_install_dir: /opt/exhibitor
http://git-wip-us.apache.org/repos/asf/airavata/blob/1ae82630/site.yml
----------------------------------------------------------------------
diff --git a/site.yml b/site.yml
index 4020f15..17258bd 100644
--- a/site.yml
+++ b/site.yml
@@ -1,17 +1,19 @@
---
- hosts: localhost
- tags: test
+ tags: ec2
roles:
- - test
+ - ec2
- hosts: mesos-master
- tags: mesos
+ tags: mesos, mesos-master
roles:
- - mesos
+ - java
+ - zookeeper
+ - mesos-master
-- hosts: mesos-master
- tags: mesosyum
+- hosts: mesos-slave
+ tags: mesos, mesos-slave
roles:
- - mesos-yum
+ - mesos-slave
...
[5/9] airavata git commit: Fixed security issue
Posted by sh...@apache.org.
Fixed security issue
Project: http://git-wip-us.apache.org/repos/asf/airavata/repo
Commit: http://git-wip-us.apache.org/repos/asf/airavata/commit/223ea3a8
Tree: http://git-wip-us.apache.org/repos/asf/airavata/tree/223ea3a8
Diff: http://git-wip-us.apache.org/repos/asf/airavata/diff/223ea3a8
Branch: refs/heads/develop
Commit: 223ea3a8d18511b6dbe7cbdf5fc3957f7d9b9df2
Parents: 03db1df
Author: Shameera Rathnayaka <sh...@gmail.com>
Authored: Sat Sep 10 19:58:37 2016 -0400
Committer: Shameera Rathnayaka <sh...@gmail.com>
Committed: Sat Sep 10 19:58:37 2016 -0400
----------------------------------------------------------------------
roles/ec2/vars/aws-credential.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/airavata/blob/223ea3a8/roles/ec2/vars/aws-credential.yml
----------------------------------------------------------------------
diff --git a/roles/ec2/vars/aws-credential.yml b/roles/ec2/vars/aws-credential.yml
index 687cfbb..2e4c703 100644
--- a/roles/ec2/vars/aws-credential.yml
+++ b/roles/ec2/vars/aws-credential.yml
@@ -1,3 +1,3 @@
---
-aws_access_key: AKIAIKBSNQUKEZN2VLYA
-aws_secret_key: 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
+aws_access_key: xxxxxxxxx
+aws_secret_key: xxxxxxxxx
[7/9] airavata git commit: Updage README.md
Posted by sh...@apache.org.
Updage README.md
Project: http://git-wip-us.apache.org/repos/asf/airavata/repo
Commit: http://git-wip-us.apache.org/repos/asf/airavata/commit/d5a9c8aa
Tree: http://git-wip-us.apache.org/repos/asf/airavata/tree/d5a9c8aa
Diff: http://git-wip-us.apache.org/repos/asf/airavata/diff/d5a9c8aa
Branch: refs/heads/develop
Commit: d5a9c8aadd699dbf0679e933a08382c6c0e51daf
Parents: 1ae8263
Author: Shameera Rathnayaka <sh...@gmail.com>
Authored: Sun Sep 11 02:30:36 2016 -0400
Committer: Shameera Rathnayaka <sh...@gmail.com>
Committed: Sun Sep 11 02:30:36 2016 -0400
----------------------------------------------------------------------
README.md | 81 +++++++++++++++++++++++++++++++++-
group_vars/all.yml | 12 -----
roles/mesos-master/tasks/main.yml | 5 +--
3 files changed, 81 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/airavata/blob/d5a9c8aa/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 70fcd39..794099a 100644
--- a/README.md
+++ b/README.md
@@ -1,2 +1,81 @@
# mesos-deployment
-Setup Apache Mesos on Cloud Infrastructures using Anisble
+This project setup Apache Mesos master and slave clusters on Cloud Infrastructures using Anisble
+
+
+## Configurations
+
+To run this ansible script you must have at least 4 instances. 3 instances to setup Mesos master , marathon and zookeeper clusters. Along with Mesos master we install marathon and zookeeper. Another one or more instance/s to setup mesos agent/s. You can use ec2 ansible role provided with this playbook to spin up aws instances. To do that you need to set valid aws credentials.
+
+1. set valid aws credentials in `roles/ec2/vars/aws-credential.yml` if it doesn't work add folowing to ec2 task in `roles/ec2/tasks/main.yml`
+
+ `aws_access_key: <your_valid_access_key>`
+
+ `aws_secret_key: <your_valid_secret_key?`
+
+2. Set ec2 instance names under ec2 task `with_items:` configurations
+
+ After you set valid aws credentials and instance names run following ansible playbook command to spin up require aws ec2 instances.
+
+ `ansible-palybook -i hosts site.yml -t "ec2"`
+
+3. You need to know public ips of all the nodes and private ips of all mesos-master nodes. List all mesos master public ips under `[mesos-master]` section in `hosts` file. List all mesos-slave *(mesos-agent)* public ips under `[mesos-slave]` section. You need to set `my_id` hosts variable along with mesos-master ip addresses. This will use to set zookeeper id.
+
+ > [mesos-master]
+
+ > *1.2.3.4 my_id=1*
+
+ > *11.22.33.44 my_id=2*
+
+ > ...
+
+ > [mesos-slave]
+
+ > *123.123.123.123*
+
+ > ...
+
+4. Use mesos-master private ip addresses to set zookeeper servers properties in `roles/zookeeper/vars/main.yml`
+
+ > zookeeper_servers:
+
+ > \- {id: "1", ip: "172.31.24.149"}
+
+ > \- {id: "2", ip: "172.31.19.49"}
+
+ > \- {id: "3", ip: "172.31.25.80"}
+
+5. Set Mesos-master zookeeper quorum value in `group_vars/all.yml` file
+
+ > zk_quorum: 2
+
+6. Now we are set to deploy Apache Mesos cluster (Mesos-master, marthon, zookeeper, mesos-slave). Following ansible playbook command respectively setup mesos-master cluster , Mesos-slaves, and both mesos-master cluster and mesos-slave
+
+ `ansible-playbook -i hosts site.yml -t "mesos-master"`
+
+ `ansible-playbook -i hosts site.yml -t "mesos-slave"`
+
+ `ansible-playbook -i hosts site.yml -t "mesos"`
+
+ If everything works without any error, now you have running mesos cluster on aws instances.
+
+## verifying installation
+
+1. If your plays works without any error. Now you can access Mesos master console using `http://<master-ip>:5050` run following in one of Mesos master node to find which instance is the leader of cluster.
+
+ ``mesos-resolve `cat /etc/mesos/zk` `` this will return master-ip. Then use it to access console
+
+2. To access marathon console use `http://<master-ip>:8080`
+
+3. If you want run simple task on mesos. run following command on one of master node.
+
+ ``MASTER=$(mesos-resolve `cat /etc/mesos/zk`)``
+
+ `mesos-execute --master=$MASTER --name="cluster-test" --command="sleep 5"`
+
+
+
+## Ansible roles
+- java - install oracle java 8
+- zookeeper - install zookeeper
+- mesos-master - install mesos mastera and marathon
+- mesos-slave - install mesos slave
http://git-wip-us.apache.org/repos/asf/airavata/blob/d5a9c8aa/group_vars/all.yml
----------------------------------------------------------------------
diff --git a/group_vars/all.yml b/group_vars/all.yml
index f7d7ae5..62c83dc 100644
--- a/group_vars/all.yml
+++ b/group_vars/all.yml
@@ -1,22 +1,10 @@
---
ansible_connection: ssh
-#ansible_ssh_user: root
ansible_ssh_user: centos
-#ansible_ssh_private_key_file: /Users/syodage/Projects/mesos-deployment/jetcloud.key
ansible_ssh_private_key_file: /Users/syodage/Projects/airavata-ansible/shameera-aws.pem.txt
user: centos
group: centos
-user_home: "/home/{{user}}"
-
-# Mesos related properties
-#mesos_version: "1.0.1"
-# mesos_version: "0.28.2"
-# mesos_name: "mesos"
-# mesos_url: "http://www.apache.org/dist/mesos/{{mesos_version}}/{{mesos_name}}-{{mesos_version}}.tar.gz"
-# mesos_deploy_dir: "{{user_home}}/{{mesos_name}}-{{mesos_version}}"
-# mesos_build_dir: "{{mesos_deploy_dir}}/build"
-
#Zookeeper related properties
zookeeper_client_port: 2181
http://git-wip-us.apache.org/repos/asf/airavata/blob/d5a9c8aa/roles/mesos-master/tasks/main.yml
----------------------------------------------------------------------
diff --git a/roles/mesos-master/tasks/main.yml b/roles/mesos-master/tasks/main.yml
index 6afefe1..effb35a 100644
--- a/roles/mesos-master/tasks/main.yml
+++ b/roles/mesos-master/tasks/main.yml
@@ -21,11 +21,8 @@
state: enabled
immediate: yes
with_items:
- # - "{{ zk_client_port }}/tcp"
- # - "2888/tcp"
- # - "3888/tcp"
- "5050/tcp"
- # - "marathon/tcp"
+ - "8080/tcp"
become: yes
- name: install utility - TODO delete this
[3/9] airavata git commit: added .gitignore file
Posted by sh...@apache.org.
added .gitignore file
Project: http://git-wip-us.apache.org/repos/asf/airavata/repo
Commit: http://git-wip-us.apache.org/repos/asf/airavata/commit/8868abc4
Tree: http://git-wip-us.apache.org/repos/asf/airavata/tree/8868abc4
Diff: http://git-wip-us.apache.org/repos/asf/airavata/diff/8868abc4
Branch: refs/heads/develop
Commit: 8868abc4f17b6085c7f183f8919a38e20c83bffb
Parents: 7429aa6
Author: Shameera Rathnayaka <sh...@gmail.com>
Authored: Tue Aug 30 15:31:41 2016 -0400
Committer: Shameera Rathnayaka <sh...@gmail.com>
Committed: Tue Aug 30 15:31:41 2016 -0400
----------------------------------------------------------------------
.gitignore | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/airavata/blob/8868abc4/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..c996e50
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1 @@
+*.key