[DISCUSS] Automatic Deployment Script

[Sean Schulte <Se...@rackspace.com>]

There are two needs here. One is that we should be able to verify that the current codebase can be deployed. The other is that we want to be able to click a button and get a running Metron.

Verify that we can deploy: I don't think this is necessarily the same as a real Ansible playbook that builds an entire environment, but rather simply verifies that each topology can start up -- this way we can avoid submitting code that compiles and is tested, but can't actually run within Storm with the default configuration. This probably involves running the topology in "local" mode.

Get a running Metron: We've seen some progress here with Ansible playbooks. I think it would be helpful to the project if these were in place within the repository, to make it simple for a new Metron cluster to be deployed.

Since the deployment-verification script could potentially take a little while to run, we may not want it to be part of the CI process. But if we don't run it automatically, we may risk merging code that can't deploy. How do we all feel about this?

Thanks, Sean

Re: COMMERCIAL:Re: [DISCUSS] Automatic Deployment Script

[Sean Schulte <Se...@rackspace.com>]

Mark,

Awesome, I’m looking forward to those Ansible scripts getting in.

And I believe they’ll have to change a bit when the Storm Flux changes go in, because those make substantial changes to the files in the OpenSOC_Config directory. (It should reduce the number of config files we need.)

> On Jan 6, 2016, at 11:32 AM, Mark Bittmann <ma...@b23.io> wrote:
> 
> Sean,
> 
> Totally agree with your point about separating the automation of deployment vs build/CI. 
> 
> For deployment (not build/CI), I submitted a PR in mid Dec. with ansible scripts to get the Hadoop ecosystem up and running with the required Metron components (Kafka, Storm, HDFS, Hbase). Based on my experience deploying Metron, I think Ansible is the fastest and most flexible way to deploy, especially in templating the OpenSOC_Config files to inject hostnames as inventory variables. I know Ryan Merman has been spending some time reviewing it and told me about a few issues with the README. I'll push those in today. I would appreciate any additional review to get it pulled into the project repo. I agree that a single click deployment is the way to go and will really drive developer adoption. I'm working on getting the other stuff donated. 
> 
> Here's the PR: https://github.com/apache/incubator-metron/pull/3 and the ticket: https://issues.apache.org/jira/browse/METRON-2
> 
> 
> 
> 
> On 1/6/16, 11:33 AM, "Sean Schulte" <Se...@rackspace.com> wrote:
> 
>> There are two needs here. One is that we should be able to verify that the current codebase can be deployed. The other is that we want to be able to click a button and get a running Metron.
>> 
>> Verify that we can deploy: I don't think this is necessarily the same as a real Ansible playbook that builds an entire environment, but rather simply verifies that each topology can start up -- this way we can avoid submitting code that compiles and is tested, but can't actually run within Storm with the default configuration. This probably involves running the topology in "local" mode.
>> 
>> Get a running Metron: We've seen some progress here with Ansible playbooks. I think it would be helpful to the project if these were in place within the repository, to make it simple for a new Metron cluster to be deployed.
>> 
>> Since the deployment-verification script could potentially take a little while to run, we may not want it to be part of the CI process. But if we don't run it automatically, we may risk merging code that can't deploy. How do we all feel about this?
>> 
>> Thanks, Sean

Re: [DISCUSS] Automatic Deployment Script

[Mark Bittmann <ma...@b23.io>]

Sean,

Totally agree with your point about separating the automation of deployment vs build/CI. 

For deployment (not build/CI), I submitted a PR in mid Dec. with ansible scripts to get the Hadoop ecosystem up and running with the required Metron components (Kafka, Storm, HDFS, Hbase). Based on my experience deploying Metron, I think Ansible is the fastest and most flexible way to deploy, especially in templating the OpenSOC_Config files to inject hostnames as inventory variables. I know Ryan Merman has been spending some time reviewing it and told me about a few issues with the README. I'll push those in today. I would appreciate any additional review to get it pulled into the project repo. I agree that a single click deployment is the way to go and will really drive developer adoption. I'm working on getting the other stuff donated. 

Here's the PR: https://github.com/apache/incubator-metron/pull/3 and the ticket: https://issues.apache.org/jira/browse/METRON-2




On 1/6/16, 11:33 AM, "Sean Schulte" <Se...@rackspace.com> wrote:

>There are two needs here. One is that we should be able to verify that the current codebase can be deployed. The other is that we want to be able to click a button and get a running Metron.
>
>Verify that we can deploy: I don't think this is necessarily the same as a real Ansible playbook that builds an entire environment, but rather simply verifies that each topology can start up -- this way we can avoid submitting code that compiles and is tested, but can't actually run within Storm with the default configuration. This probably involves running the topology in "local" mode.
>
>Get a running Metron: We've seen some progress here with Ansible playbooks. I think it would be helpful to the project if these were in place within the repository, to make it simple for a new Metron cluster to be deployed.
>
>Since the deployment-verification script could potentially take a little while to run, we may not want it to be part of the CI process. But if we don't run it automatically, we may risk merging code that can't deploy. How do we all feel about this?
>
>Thanks, Sean