You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by "Shenoy, Gourav Ganesh" <go...@indiana.edu> on 2017/03/09 18:17:52 UTC

List of Devops Scripts for Apache Airavata

Hi devs,

I wanted to start a thread to track efforts made towards writing devops scrips (including Ansible/Terraform) for Apache Airavata. Currently these scripts seem to reside in more than a few places. This can naturally lead to redundancy, and therefore it would be helpful if we can have a list of these existing scripts along with their purpose.

Let me start – I have worked with Shameera in writing cloud provisioning and Mesos/Marathon cluster deployment scripts in Ansible. Here are the details:

Script location

Branch

Purpose of Script

modules/cloud/ansible-playbooks

develop

1. Provisions instances on EC2 and OpenStack (Jetstream)
2. Deploys a Mesos/Marathon cluster on the provisioned instances










Anyone who is working/has worked on any devops scripts, please add to the above table. This will make sure we have a central list of existing contributions and avoid duplication in efforts.

Thanks and Regards,
Gourav Shenoy

Re: List of Devops Scripts for Apache Airavata

Posted by Suresh Marru <sm...@apache.org>.
Thanks Gourav for starting this discussion. 

Do we need a separate DevOps repo or should we have the packing and deployment scripts as part of individual repositories? 

Suresh

> On Mar 9, 2017, at 1:17 PM, Shenoy, Gourav Ganesh <go...@indiana.edu> wrote:
> 
> Hi devs,
>  
> I wanted to start a thread to track efforts made towards writing devops scrips (including Ansible/Terraform) for Apache Airavata. Currently these scripts seem to reside in more than a few places. This can naturally lead to redundancy, and therefore it would be helpful if we can have a list of these existing scripts along with their purpose.
>  
> Let me start – I have worked with Shameera in writing cloud provisioning and Mesos/Marathon cluster deployment scripts in Ansible. Here are the details:
>  
> Script location
> Branch
> Purpose of Script
> modules/cloud/ansible-playbooks
> develop
> 1. Provisions instances on EC2 and OpenStack (Jetstream)
> 2. Deploys a Mesos/Marathon cluster on the provisioned instances
>  
>  
>  
>  
>  
>  
>  
>  
>  
> Anyone who is working/has worked on any devops scripts, please add to the above table. This will make sure we have a central list of existing contributions and avoid duplication in efforts.
>  
> Thanks and Regards,
> Gourav Shenoy

Re: List of Devops Scripts for Apache Airavata

Posted by "Christie, Marcus Aaron" <ma...@iu.edu>.
On Mar 10, 2017, at 6:36 PM, Shameera Rathnayaka <sh...@gmail.com>> wrote:

Hi,

I second Marlon's point about burden on CI/CD by having totally two repositories, also we should have good discipline to keep two separate repos in sync which is kind of hard IMO. Moving all scripts to one place and organizing them in different subdirectories make more sense. That is why I created dev-tools and dev-tools/ansible in the first place.

Hey Shameera,

Yeah, I agree. This was also my concern and I now think the advantage is with keeping the scripts in the same repo.

This makes me think that the PGA Ansible scripts should really be in the PGA repo.


We only need to maintain two different scripts, one for development and one for master, that is the only thing we will officially maintain.(feature developers can have their own variation of scripts, that is not our concern). We need to have proper code review sessions to avoid our scripts being obsolete. As all in one repo help us to do this in same PR. We can tag dev-tools along with new release branch to have proper versioning.


Regarding scripts becoming obsolete, I would like to see us using these scripts as part of CI/CD eventually, so that any breakage is immediately obvious.

Thanks,

Marcus

Re: List of Devops Scripts for Apache Airavata

Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi,

I second Marlon's point about burden on CI/CD by having totally two
repositories, also we should have good discipline to keep two separate
repos in sync which is kind of hard IMO. Moving all scripts to one place
and organizing them in different subdirectories make more sense. That is
why I created dev-tools and dev-tools/ansible in the first place.

We only need to maintain two different scripts, one for development and one
for master, that is the only thing we will officially maintain.(feature
developers can have their own variation of scripts, that is not our
concern). We need to have proper code review sessions to avoid our scripts
being obsolete. As all in one repo help us to do this in same PR. We can
tag dev-tools along with new release branch to have proper versioning.

Thank you,
Shameera.

On Fri, Mar 10, 2017 at 1:14 PM Shenoy, Gourav Ganesh <go...@indiana.edu>
wrote:

Hi Marcus,



Although it is not being actively used, but it still is in working shape.



Thanks and Regards,

Gourav Shenoy



*From: *"Christie, Marcus Aaron" <ma...@iu.edu>


*Date: *Friday, March 10, 2017 at 11:54 AM
*To: *"dev@airavata.apache.org" <de...@airavata.apache.org>, "Shenoy, Gourav
Ganesh" <go...@indiana.edu>
*Subject: *Re: List of Devops Scripts for Apache Airavata



Dev,



Eric and I met this morning to discuss how to consolidate these various
efforts.  Here’s what we came up with:



1. (Marcus) Create an airavata-devops repo for holding all of the Ansible
scripts.  To start with, we’ll use the dev-tools/ansible scripts.

2. (Eric) Once the airavata-devops repo is in place, Eric will move his
Jetstream provisioning scripts onto a branch there

3. (Eric) Eric is also going to start working on Jetstream scripts for
other servers besides PGA



A question for Gourav: are the modules/cloud/ansible-playbooks still being
actively used?



Some other things we discussed:

* How to securely store secret information?  We discussed using
ansible-vault versus having a separate, private repo for Ansible
inventories.  We settled on using ansible-vault since it has the advantage
of keeping everything in one repo. The disadvantage is in needing to share
a password file out-of-band with the rest of the team.  However, we felt
the sharing of the password would be mitigated as we move toward more
automated systems for running deployments (for example, users who only need
to run a deploy won’t need the password file if we use Jenkins to run
deploys; only Jenkins would really need to have the password file).

* One advantage to NOT having a separate repo for devops scripts is that it
keeps the devops scripts versioned together with the code those scripts can
build and deploy.  Having a separate repo means we will need to have some
discipline when updating devops scripts.  I think it makes sense to have
the devops repo branch structure reflect the airavata branch structure.
That is, the ‘develop’ branch of the devops repo should be good for
deploying the ‘develop’ branch of airavata and likewise for ‘master’.  If
code is updated in airavata on a separate branch that needs corresponding
devops script updates, that should likewise be done on a same-named
separate branch in the devops repo.



Eric, let me know if I missed anything.





Thanks,



Marcus





On Mar 9, 2017, at 3:37 PM, Anuj Bhandar <bh...@gmail.com> wrote:



Gourav,



It is a great initiative, I vote for a separate repository for devops, the
Airavata repository plays many roles already.



A separate repository would aid active development and bring modularity.



Thanks,

Anuj Bhandar



On 3/9/17 3:14 PM, Coulter, John Eric wrote:

Thanks for starting this, Gourav!

I'm inclined to vote for a separate DevOps repo, to keep things more
modular.



I know Marcus and I have done some work/testing starting from scripts that
I think Shameera created in dev-tools/ansible. I've got a side-repo up
which contains playbooks for provisioning and deploying the PGA from
scratch as a proof of concept, using dynamic inventory instead of a static
file (so, on an empty jetstream allocation, you get working instance
running the PGA, with router, public ip with a single ansible-playbook
command). Next step there is to add provisioning for VMs to run airavata,
etc.


*Script location*

*Branch*

*Purpose of Script*

modules/cloud/ansible-playbooks​​​

develop

1. Provisions instances on EC2 and OpenStack (Jetstream)

2. Deploys a Mesos/Marathon cluster on the provisioned i​nstances

 ​

​dev-tools/ansible/

​develop

​1. Deploy/update airavata services on existing machines​/instances
2. todo - include provisioning scripts for cloud resources

https://github.com/ECoulter/airavata-vms

​

​1. POC for provisioning and deployment of PGA on Jetstream, with dynamic
inventory. (intended to merge with dev-tools/ansible when complete/tested
by others)





---------------



Cheers,

---------------------
Eric Coulter        jecoulte@iu.edu
XSEDE Capabilities and Resource Integration Engineer
IU Campus Bridging & Research Infrastructure
RT/PTI/UITS
812-856-3250 <(812)%20856-3250>





-- 
Shameera Rathnayaka

Re: List of Devops Scripts for Apache Airavata

Posted by "Christie, Marcus Aaron" <ma...@iu.edu>.
On Mar 10, 2017, at 1:14 PM, Shenoy, Gourav Ganesh <go...@indiana.edu>> wrote:

Although it is not being actively used, but it still is in working shape.


Gourav,

Thanks, good to know. I think for now I’ll just leave it where it is. If we start to use it more, I think at that point we would want to put it in the devops repo.

Thanks,

Marcus

Re: List of Devops Scripts for Apache Airavata

Posted by "Shenoy, Gourav Ganesh" <go...@indiana.edu>.
Hi Marcus,

Although it is not being actively used, but it still is in working shape.

Thanks and Regards,
Gourav Shenoy

From: "Christie, Marcus Aaron" <ma...@iu.edu>
Date: Friday, March 10, 2017 at 11:54 AM
To: "dev@airavata.apache.org" <de...@airavata.apache.org>, "Shenoy, Gourav Ganesh" <go...@indiana.edu>
Subject: Re: List of Devops Scripts for Apache Airavata

Dev,

Eric and I met this morning to discuss how to consolidate these various efforts.  Here’s what we came up with:

1. (Marcus) Create an airavata-devops repo for holding all of the Ansible scripts.  To start with, we’ll use the dev-tools/ansible scripts.
2. (Eric) Once the airavata-devops repo is in place, Eric will move his Jetstream provisioning scripts onto a branch there
3. (Eric) Eric is also going to start working on Jetstream scripts for other servers besides PGA

A question for Gourav: are the modules/cloud/ansible-playbooks still being actively used?

Some other things we discussed:
* How to securely store secret information?  We discussed using ansible-vault versus having a separate, private repo for Ansible inventories.  We settled on using ansible-vault since it has the advantage of keeping everything in one repo. The disadvantage is in needing to share a password file out-of-band with the rest of the team.  However, we felt the sharing of the password would be mitigated as we move toward more automated systems for running deployments (for example, users who only need to run a deploy won’t need the password file if we use Jenkins to run deploys; only Jenkins would really need to have the password file).
* One advantage to NOT having a separate repo for devops scripts is that it keeps the devops scripts versioned together with the code those scripts can build and deploy.  Having a separate repo means we will need to have some discipline when updating devops scripts.  I think it makes sense to have the devops repo branch structure reflect the airavata branch structure.  That is, the ‘develop’ branch of the devops repo should be good for deploying the ‘develop’ branch of airavata and likewise for ‘master’.  If code is updated in airavata on a separate branch that needs corresponding devops script updates, that should likewise be done on a same-named separate branch in the devops repo.

Eric, let me know if I missed anything.


Thanks,

Marcus


On Mar 9, 2017, at 3:37 PM, Anuj Bhandar <bh...@gmail.com>> wrote:

Gourav,

It is a great initiative, I vote for a separate repository for devops, the Airavata repository plays many roles already.

A separate repository would aid active development and bring modularity.

Thanks,
Anuj Bhandar

On 3/9/17 3:14 PM, Coulter, John Eric wrote:
Thanks for starting this, Gourav!
I'm inclined to vote for a separate DevOps repo, to keep things more modular.

I know Marcus and I have done some work/testing starting from scripts that I think Shameera created in dev-tools/ansible. I've got a side-repo up which contains playbooks for provisioning and deploying the PGA from scratch as a proof of concept, using dynamic inventory instead of a static file (so, on an empty jetstream allocation, you get working instance running the PGA, with router, public ip with a single ansible-playbook command). Next step there is to add provisioning for VMs to run airavata, etc.

Script location

Branch

Purpose of Script

modules/cloud/ansible-playbooks​​​

develop

1. Provisions instances on EC2 and OpenStack (Jetstream)
2. Deploys a Mesos/Marathon cluster on the provisioned i​nstances
 ​

​dev-tools/ansible/

​develop

​1. Deploy/update airavata services on existing machines​/instances
2. todo - include provisioning scripts for cloud resources

https://github.com/ECoulter/airavata-vms

​

​1. POC for provisioning and deployment of PGA on Jetstream, with dynamic inventory. (intended to merge with dev-tools/ansible when complete/tested by others)



---------------

Cheers,
---------------------
Eric Coulter        jecoulte@iu.edu<ma...@iu.edu>
XSEDE Capabilities and Resource Integration Engineer
IU Campus Bridging & Research Infrastructure
RT/PTI/UITS
812-856-3250

Re: List of Devops Scripts for Apache Airavata

Posted by "Christie, Marcus Aaron" <ma...@iu.edu>.
On Mar 10, 2017, at 11:57 AM, Pierce, Marlon <ma...@iu.edu>> wrote:

Re: Marcus’s second bullet. Separating these into separate branches puts the burden on using CI/CD.


Marlon,

Could you expand on this? I’m not sure I follow what you are saying.

Thanks,

Marcus

Re: List of Devops Scripts for Apache Airavata

Posted by "Pierce, Marlon" <ma...@iu.edu>.
Re: Marcus’s second bullet. Separating these into separate branches puts the burden on using CI/CD.

 

Marlon

 

 

From: "Christie, Marcus Aaron" <ma...@iu.edu>
Reply-To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Date: Friday, March 10, 2017 at 11:54 AM
To: "dev@airavata.apache.org" <de...@airavata.apache.org>, "Shenoy, Gourav Ganesh" <go...@indiana.edu>
Subject: Re: List of Devops Scripts for Apache Airavata

 

Dev, 

 

Eric and I met this morning to discuss how to consolidate these various efforts.  Here’s what we came up with:

 

1. (Marcus) Create an airavata-devops repo for holding all of the Ansible scripts.  To start with, we’ll use the dev-tools/ansible scripts.

2. (Eric) Once the airavata-devops repo is in place, Eric will move his Jetstream provisioning scripts onto a branch there

3. (Eric) Eric is also going to start working on Jetstream scripts for other servers besides PGA

 

A question for Gourav: are the modules/cloud/ansible-playbooks still being actively used?

 

Some other things we discussed:

* How to securely store secret information?  We discussed using ansible-vault versus having a separate, private repo for Ansible inventories.  We settled on using ansible-vault since it has the advantage of keeping everything in one repo. The disadvantage is in needing to share a password file out-of-band with the rest of the team.  However, we felt the sharing of the password would be mitigated as we move toward more automated systems for running deployments (for example, users who only need to run a deploy won’t need the password file if we use Jenkins to run deploys; only Jenkins would really need to have the password file).

* One advantage to NOT having a separate repo for devops scripts is that it keeps the devops scripts versioned together with the code those scripts can build and deploy.  Having a separate repo means we will need to have some discipline when updating devops scripts.  I think it makes sense to have the devops repo branch structure reflect the airavata branch structure.  That is, the ‘develop’ branch of the devops repo should be good for deploying the ‘develop’ branch of airavata and likewise for ‘master’.  If code is updated in airavata on a separate branch that needs corresponding devops script updates, that should likewise be done on a same-named separate branch in the devops repo.

 

Eric, let me know if I missed anything.

 

 

Thanks,

 

Marcus

 

 

On Mar 9, 2017, at 3:37 PM, Anuj Bhandar <bh...@gmail.com> wrote:

 

Gourav,

 

It is a great initiative, I vote for a separate repository for devops, the Airavata repository plays many roles already.

 

A separate repository would aid active development and bring modularity.

 

Thanks,

Anuj Bhandar

 

On 3/9/17 3:14 PM, Coulter, John Eric wrote:

Thanks for starting this, Gourav! 

I'm inclined to vote for a separate DevOps repo, to keep things more modular. 

 

I know Marcus and I have done some work/testing starting from scripts that I think Shameera created in dev-tools/ansible. I've got a side-repo up which contains playbooks for provisioning and deploying the PGA from scratch as a proof of concept, using dynamic inventory instead of a static file (so, on an empty jetstream allocation, you get working instance running the PGA, with router, public ip with a single ansible-playbook command). Next step there is to add provisioning for VMs to run airavata, etc. 
 

Script locationBranchPurpose of Script
modules/cloud/ansible-playbooks​​​
develop
1. Provisions instances on EC2 and OpenStack (Jetstream)
2. Deploys a Mesos/Marathon cluster on the provisioned i​nstances
 ​
​dev-tools/ansible/​develop​1. Deploy/update airavata services on existing machines​/instances
2. todo - include provisioning scripts for cloud resources
https://github.com/ECoulter/airavata-vms​​1. POC for provisioning and deployment of PGA on Jetstream, with dynamic inventory. (intended to merge with dev-tools/ansible when complete/tested by others)
 

 

---------------

 

Cheers,

---------------------
Eric Coulter        jecoulte@iu.edu
XSEDE Capabilities and Resource Integration Engineer 
IU Campus Bridging & Research Infrastructure
RT/PTI/UITS
812-856-3250

Re: List of Devops Scripts for Apache Airavata

Posted by "Christie, Marcus Aaron" <ma...@iu.edu>.
Dev,

Eric and I met this morning to discuss how to consolidate these various efforts.  Here’s what we came up with:

1. (Marcus) Create an airavata-devops repo for holding all of the Ansible scripts.  To start with, we’ll use the dev-tools/ansible scripts.
2. (Eric) Once the airavata-devops repo is in place, Eric will move his Jetstream provisioning scripts onto a branch there
3. (Eric) Eric is also going to start working on Jetstream scripts for other servers besides PGA

A question for Gourav: are the modules/cloud/ansible-playbooks still being actively used?

Some other things we discussed:
* How to securely store secret information?  We discussed using ansible-vault versus having a separate, private repo for Ansible inventories.  We settled on using ansible-vault since it has the advantage of keeping everything in one repo. The disadvantage is in needing to share a password file out-of-band with the rest of the team.  However, we felt the sharing of the password would be mitigated as we move toward more automated systems for running deployments (for example, users who only need to run a deploy won’t need the password file if we use Jenkins to run deploys; only Jenkins would really need to have the password file).
* One advantage to NOT having a separate repo for devops scripts is that it keeps the devops scripts versioned together with the code those scripts can build and deploy.  Having a separate repo means we will need to have some discipline when updating devops scripts.  I think it makes sense to have the devops repo branch structure reflect the airavata branch structure.  That is, the ‘develop’ branch of the devops repo should be good for deploying the ‘develop’ branch of airavata and likewise for ‘master’.  If code is updated in airavata on a separate branch that needs corresponding devops script updates, that should likewise be done on a same-named separate branch in the devops repo.

Eric, let me know if I missed anything.


Thanks,

Marcus


On Mar 9, 2017, at 3:37 PM, Anuj Bhandar <bh...@gmail.com>> wrote:

Gourav,

It is a great initiative, I vote for a separate repository for devops, the Airavata repository plays many roles already.

A separate repository would aid active development and bring modularity.

Thanks,
Anuj Bhandar

On 3/9/17 3:14 PM, Coulter, John Eric wrote:
Thanks for starting this, Gourav!

I'm inclined to vote for a separate DevOps repo, to keep things more modular.

I know Marcus and I have done some work/testing starting from scripts that I think Shameera created in dev-tools/ansible. I've got a side-repo up which contains playbooks for provisioning and deploying the PGA from scratch as a proof of concept, using dynamic inventory instead of a static file (so, on an empty jetstream allocation, you get working instance running the PGA, with router, public ip with a single ansible-playbook command). Next step there is to add provisioning for VMs to run airavata, etc.

Script location

Branch

Purpose of Script

modules/cloud/ansible-playbooks​​​

develop

1. Provisions instances on EC2 and OpenStack (Jetstream)
2. Deploys a Mesos/Marathon cluster on the provisioned i​nstances
 ​

​dev-tools/ansible/
        ​develop
        ​1. Deploy/update airavata services on existing machines​/instances
2. todo - include provisioning scripts for cloud resources

https://github.com/ECoulter/airavata-vms
        ​
        ​1. POC for provisioning and deployment of PGA on Jetstream, with dynamic inventory. (intended to merge with dev-tools/ansible when complete/tested by others)



---------------

Cheers,
---------------------
Eric Coulter        jecoulte@iu.edu<ma...@iu.edu>
XSEDE Capabilities and Resource Integration Engineer
IU Campus Bridging & Research Infrastructure
RT/PTI/UITS
812-856-3250

Re: List of Devops Scripts for Apache Airavata

Posted by Anuj Bhandar <bh...@gmail.com>.
Gourav,


It is a great initiative, I vote for a separate repository for devops, 
the Airavata repository plays many roles already.


A separate repository would aid active development and bring modularity.


Thanks,

Anuj Bhandar


On 3/9/17 3:14 PM, Coulter, John Eric wrote:
>
> Thanks for starting this, Gourav!
>
> I'm inclined to vote for a separate DevOps repo, to keep things more 
> modular.
>
>
> I know Marcus and I have done some work/testing starting from scripts 
> that I think Shameera created in dev-tools/ansible. I've got a 
> side-repo up which contains playbooks for provisioning and deploying 
> the PGA from scratch as a proof of concept, using dynamic inventory 
> instead of a static file (so, on an empty jetstream allocation, you 
> get working instance running the PGA, with router, public ip with a 
> single ansible-playbook command). Next step there is to add 
> provisioning for VMs to run airavata, etc.
>
> *Script location*
>
> 	
>
> *Branch*
>
> 	
>
> *Purpose of Script*
>
> modules/cloud/ansible-playbooks\u200b\u200b\u200b
>
> 	
>
> develop
>
> 	
>
> 1. Provisions instances on EC2 and OpenStack (Jetstream)
>
> 2. Deploys a Mesos/Marathon cluster on the provisioned i\u200bnstances
>
>  \u200b
>
> \u200bdev-tools/ansible/
> 	\u200bdevelop
> 	\u200b1. Deploy/update airavata services on existing machines\u200b/instances
> 2. todo - include provisioning scripts for cloud resources
> https://github.com/ECoulter/airavata-vms
> 	\u200b
> 	\u200b1. POC for provisioning and deployment of PGA on Jetstream, with 
> dynamic inventory. (intended to merge with dev-tools/ansible when 
> complete/tested by others)
>
>
>
> ---------------
>
> Cheers,
> ---------------------
> Eric Coulter jecoulte@iu.edu
> XSEDE Capabilities and Resource Integration Engineer
> IU Campus Bridging & Research Infrastructure
> RT/PTI/UITS
> 812-856-3250
>

Re: List of Devops Scripts for Apache Airavata

Posted by "Coulter, John Eric" <je...@iu.edu>.
Thanks for starting this, Gourav!

I'm inclined to vote for a separate DevOps repo, to keep things more modular.

I know Marcus and I have done some work/testing starting from scripts that I think Shameera created in dev-tools/ansible. I've got a side-repo up which contains playbooks for provisioning and deploying the PGA from scratch as a proof of concept, using dynamic inventory instead of a static file (so, on an empty jetstream allocation, you get working instance running the PGA, with router, public ip with a single ansible-playbook command). Next step there is to add provisioning for VMs to run airavata, etc.

Script location

Branch

Purpose of Script

modules/cloud/ansible-playbooks???

develop

1. Provisions instances on EC2 and OpenStack (Jetstream)
2. Deploys a Mesos/Marathon cluster on the provisioned i?nstances
 ?

?dev-tools/ansible/
        ?develop
        ?1. Deploy/update airavata services on existing machines?/instances
2. todo - include provisioning scripts for cloud resources

https://github.com/ECoulter/airavata-vms
        ?
        ?1. POC for provisioning and deployment of PGA on Jetstream, with dynamic inventory. (intended to merge with dev-tools/ansible when complete/tested by others)




---------------

Cheers,
---------------------
Eric Coulter         jecoulte@iu.edu
XSEDE Capabilities and Resource Integration Engineer
IU Campus Bridging & Research Infrastructure
RT/PTI/UITS
812-856-3250